{"thread":{"id":"65383","subject":"[PATCH 00/16] repack: incremental MIDX/bitmap-based repacking","startedAt":"2026-03-29T21:40:52Z","lastAt":"2026-06-12T20:08:07Z","messageCount":113,"participants":["Taylor Blau","Jeff King","Elijah Newren","SZEDER Gábor","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":16},"messages":[{"id":"540336","messageId":"cover.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":null,"subject":"[PATCH 00/16] repack: incremental MIDX/bitmap-based repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:40:50Z","receivedAt":"2026-03-29T21:40:52Z","isPatch":true,"body":"Note to the maintainer:\n\n * Now that tb/incremental-midx-part-3.2 is merged, this series is\n rebased on current 'master', which is 5361983c075 (The 22nd batch,\n 2026-03-27) at the time of writing.\n\n * I suggest queueing as 'tb/incremental-midx-part-3.3'.\n\nThe below is a nearly identical cover letter to the one I sent for the\nRFC-version[1] of this series. The only changes between the RFC version\nand this one are:\n\n * A new commit to correctly handle certain cases when optimizing out\n MIDX writes.\n\n * Defined `strvec_init()` as a special case of `strvec_init_alloc()`\n based on feedback from Junio.\n\n * Minor ODB source-related fallout as a consequence of rebasing.\n\n * Add trace2 instrumentation to help follow what repack is doing in\n response to different packing configurations.\n\n * Fix various memory leaks.\n\nThe original cover letter is below:\n\n------------------------------------------------------------------------\n\nOVERVIEW\n========\n\nThis series implements the third and final major component for an\nincremental MIDX/bitmap-based repacking strategy. As a refresher, those\nare:\n\n 1. Refactoring code out of builtin/repack.c into individual compilation\n units outside of the builtin directory.\n\n 2. Implementing MIDX layer compaction, i.e., the ability to combine a\n contiguous sequence of MIDX layers in an existing chain and replace\n them with a single layer representing the union of objects/packs\n among the compacted layers.\n\n 3. (this series) An incremental repacking strategy that, unlike our\n current '--geometric' approach, does not rely on periodic\n all-into-one repacks.\n\nBACKGROUND\n==========\n\nToday, a '--geometric' repack with '--write-midx' rewrites the entire\nMIDX (and, when enabled, its reachability bitmap) on every invocation.\nFor most repositories this is acceptable. In large monorepos, these\ncosts can add up significantly, especially when those repositories are\nrepacked frequently.\n\nThe incremental MIDX support introduced in the earlier parts of this\neffort allows us to append new layers to a MIDX chain without rewriting\nanything. Combined with the support for MIDX compaction in 3.2, we now\nhave all of the building blocks needed to maintain an incrementally\ngrowing and shrinking MIDX chain as part of the repack cycle.\n\nSTRATEGY\n========\n\nThe incremental repacking strategy implemented in this series works\n(roughly) as follows:\n\n 1. Repack non-MIDX'd packs using an ordinary '--geometric' repack,\n optionally including packs from the tip MIDX layer if and only if it\n contains more than 'repack.midxNewLayerThreshold' number of packs.\n\n 2. Prepare to write a new MIDX layer containing the freshly written\n pack(s) (and any surviving packs from the tip layer, if it was\n rewritten).\n\n 3. Perform a compaction pass over adjacent MIDX layers to restore a\n geometric progression on object count among layers in the chain\n (determined by 'repack.midxSplitFactor').\n\n 4. Write the new MIDX chain, link it into place, and remove redundant\n layers.\n\nIn effect, this approach encourages MIDX chains where:\n\n * older layers contain fewer, larger packs, and\n\n * newer layers contain many smaller packs.\n\nAs a result of the compaction pass, we prevent the chain itself from\never growing too long.\n\nThis roughly matches the description of this algorithm I gave at Git\nMerge last year, which is covered on slides 80-131 of this presentation:\n\n https://ttaylorr.com/presentations/git-merge-2025.pdf\n\nTHIS SERIES\n===========\n\nThis series is organized roughly as follows:\n\n * The first three patches perform small-ish quality-of-life refactors\n within the MIDX machinery.\n\n * The next two patches introduce `--checksum-only` and `--base`\n options for `git multi-pack-index write` and `compact`, which are\n needed by the incremental repacking machinery to assemble MIDX\n chains without prematurely updating the chain file.\n\n * The next six patches prepare the repack infrastructure and pack\n geometry code for the new repacking strategy.\n\n * The final three patches introduce the new repacking machinery, expose\n it to users, and then extend it to work with non-geometric repacks in\n that order. The first two are the substantive patches of this series.\n\nWHERE WE'RE AT\n==============\n\nThis series delivers the final substantive component of this overall\neffort to enable the new repacking strategy implemented here. There are\na couple of (comparatively much smaller) items that will be useful as\nfollow-on items, including:\n\n * \"Reachability-infused\" geometric repacking to emit small cruft packs\n to introduce other ways to update the set of cruft objects beyond a\n whole-repository traversal.\n\n * Richer bitmap configuration to determine which bitmap(s) to carry\n forward between adjacent MIDX layers when doing an incremental\n repack, to prevent an endless accumulation of reachability bitmaps.\n\nLike I mentioned earlier, I think that both of those are significantly\nsmaller challenges than this and the previous series.\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/cover.1771978829.git.me@ttaylorr.com/\n\nTaylor Blau (16):\n midx-write: handle noop writes when converting incremental chains\n midx: use `string_list` for retained MIDX files\n strvec: introduce `strvec_init_alloc()`\n midx: use `strvec` for `keep_hashes`\n midx: introduce `--checksum-only` for incremental MIDX writes\n midx: support custom `--base` for incremental MIDX writes\n repack: track the ODB source via existing_packs\n midx: expose `midx_layer_contains_pack()`\n repack-midx: factor out `repack_prepare_midx_command()`\n repack-midx: extract `repack_fill_midx_stdin_packs()`\n repack-geometry: prepare for incremental MIDX repacking\n builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`\n packfile: ensure `close_pack_revindex()` frees in-memory revindex\n repack: implement incremental MIDX repacking\n repack: introduce `--write-midx=incremental`\n repack: allow `--write-midx=incremental` without `--geometric`\n\n Documentation/config/repack.adoc | 18 +\n Documentation/git-multi-pack-index.adoc | 19 +-\n Documentation/git-repack.adoc | 44 +-\n builtin/multi-pack-index.c | 48 +-\n builtin/repack.c | 90 ++-\n midx-write.c | 137 +++--\n midx.c | 106 ++--\n midx.h | 11 +-\n packfile.c | 2 +\n repack-geometry.c | 50 +-\n repack-midx.c | 717 +++++++++++++++++++++++-\n repack.c | 23 +-\n repack.h | 25 +-\n strvec.c | 15 +-\n strvec.h | 5 +\n t/meson.build | 1 +\n t/t5334-incremental-multi-pack-index.sh | 63 +++\n t/t5335-compact-multi-pack-index.sh | 113 ++++\n t/t7705-repack-incremental-midx.sh | 461 +++++++++++++++\n 19 files changed, 1792 insertions(+), 156 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\n\nbase-commit: 5361983c075154725be47b65cca9a2421789e410\n-- \n2.53.0.729.g817728289e1.dirty\n"},{"id":"540337","messageId":"0c7a68ca5a777cce9c0c951643ea2220b1b56a8c.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 01/16] midx-write: handle noop writes when converting incremental chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:40:53Z","receivedAt":"2026-03-29T21:40:55Z","isPatch":true,"body":"When updating a MIDX, we optimize out writes that will result in an\nidentical MIDX as the one we already have on disk. See b3bab9d2729\n(midx-write: extract function to test whether MIDX needs updating,\n2025-12-10) for more details on exactly which writes are optimized out.\n\nIf `midx_needs_update()` can't rule out any of the obvious cases (e.g.,\nthe checksum is invalid, we're requesting a different version, or\nperforming compaction which always requires an update), then we compare\nthe packs we're writing to the packs we already know about. If there are\nan equal number of packs being written as there are in any existing\nMIDX layer(s), then we compare the packs by their name.\n\nThis comparison fails when we have an incremental MIDX chain with\nat least two layers, since we do not recursively peel through earlier\nlayers, instead treating the `->pack_names` array of the tip MIDX layer\nas containing all `m->num_packs + m->num_packs_in_base` packs.\n\nAdjust this to instead look through the MIDX layers one by one when\ncomparing pack names. While we're at it, fix a typo above in the same\nfunction.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 18 ++++++++++--------\n t/t5334-incremental-multi-pack-index.sh | 16 ++++++++++++++++\n 2 files changed, 26 insertions(+), 8 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 0ff2e45aa7a..d0351bde493 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1152,7 +1152,7 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \n \t/*\n \t * Ensure that we have a valid checksum before consulting the\n-\t * exisiting MIDX in order to determine if we can avoid an\n+\t * existing MIDX in order to determine if we can avoid an\n \t * update.\n \t *\n \t * This is necessary because the given MIDX is loaded directly\n@@ -1208,14 +1208,16 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \t\t\tBUG(\"same pack added twice?\");\n \t}\n \n-\tfor (uint32_t i = 0; i < ctx->nr; i++) {\n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addstr(&buf, midx->pack_names[i]);\n-\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\tfor (struct multi_pack_index *m = midx; m; m = m->base_midx) {\n+\t\tfor (uint32_t i = 0; i < m->num_packs; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, m->pack_names[i]);\n+\t\t\tstrbuf_strip_suffix(&buf, \".idx\");\n \n-\t\tif (!strset_contains(&packs, buf.buf))\n-\t\t\tgoto out;\n-\t\tstrset_remove(&packs, buf.buf);\n+\t\t\tif (!strset_contains(&packs, buf.buf))\n+\t\t\t\tgoto out;\n+\t\t\tstrset_remove(&packs, buf.buf);\n+\t\t}\n \t}\n \n \tneeded = false;\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 99c7d44d8e9..c9f5b4e87aa 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -132,4 +132,20 @@ test_expect_success 'relink existing MIDX layer' '\n \n '\n \n+test_expect_success 'non-incremental write with existing incremental chain' '\n+\tgit init non-incremental-write-with-existing &&\n+\ttest_when_finished \"rm -fr non-incremental-write-with-existing\" &&\n+\n+\t(\n+\t\tcd non-incremental-write-with-existing &&\n+\n+\t\tgit config set maintenance.auto false &&\n+\n+\t\twrite_midx_layer &&\n+\t\twrite_midx_layer &&\n+\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n test_done\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540338","messageId":"1391552dfc6624374bed89b25d1892d9b9d080f9.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 02/16] midx: use `string_list` for retained MIDX files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:40:55Z","receivedAt":"2026-03-29T21:40:57Z","isPatch":true,"body":"Both `clear_midx_files_ext()` and `clear_incremental_midx_files_ext()`\nbuild a list of filenames to keep while pruning stale MIDX files. Today\nthey hand-roll an array instead of using a `string_list`, thus requiring\nus to pass an additional length parameter, and makes lookups linear.\n\nReplace the bare array with a `string_list` which can be passed around\nas a single parameter. Though it improves lookup performance, the\ndifference is likely immeasurable given how small the keep_hashes array\ntypically is.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 53 +++++++++++++++++++++++------------------------------\n 1 file changed, 23 insertions(+), 30 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 81d6ab11e6e..d322743f2db 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -758,8 +758,7 @@ int midx_checksum_valid(struct multi_pack_index *m)\n }\n \n struct clear_midx_data {\n-\tchar **keep;\n-\tuint32_t keep_nr;\n+\tstruct string_list keep;\n \tconst char *ext;\n };\n \n@@ -767,15 +766,12 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\t\t\tconst char *file_name, void *_data)\n {\n \tstruct clear_midx_data *data = _data;\n-\tuint32_t i;\n \n \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n \t ends_with(file_name, data->ext)))\n \t\treturn;\n-\tfor (i = 0; i < data->keep_nr; i++) {\n-\t\tif (!strcmp(data->keep[i], file_name))\n-\t\t\treturn;\n-\t}\n+\tif (string_list_has_string(&data->keep, file_name))\n+\t\treturn;\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n@@ -783,48 +779,45 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash)\n {\n-\tstruct clear_midx_data data;\n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRING_LIST_INIT_DUP,\n+\t\t.ext = ext,\n+\t};\n \n \tif (keep_hash) {\n-\t\tALLOC_ARRAY(data.keep, 1);\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hash, ext);\n \n-\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n-\t\tdata.keep_nr = 1;\n+\t\tstring_list_insert(&data.keep, buf.buf);\n+\n+\t\tstrbuf_release(&buf);\n \t}\n-\tdata.ext = ext;\n \n-\tfor_each_file_in_pack_dir(source->path,\n-\t\t\t\t clear_midx_file_ext,\n-\t\t\t\t &data);\n+\tfor_each_file_in_pack_dir(source->path, clear_midx_file_ext, &data);\n \n-\tif (keep_hash)\n-\t\tfree(data.keep[0]);\n-\tfree(data.keep);\n+\tstring_list_clear(&data.keep, 0);\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t char **keep_hashes,\n \t\t\t\t uint32_t hashes_nr)\n {\n-\tstruct clear_midx_data data;\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRING_LIST_INIT_NODUP,\n+\t\t.ext = ext,\n+\t};\n \tuint32_t i;\n \n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n-\n-\tALLOC_ARRAY(data.keep, hashes_nr);\n \tfor (i = 0; i < hashes_nr; i++)\n-\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n-\t\t\t\t ext);\n-\tdata.keep_nr = hashes_nr;\n-\tdata.ext = ext;\n+\t\tstring_list_append(&data.keep,\n+\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n+\t\t\t\t\t keep_hashes[i], ext));\n+\tstring_list_sort(&data.keep);\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n \n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tfree(data.keep[i]);\n-\tfree(data.keep);\n+\tstring_list_clear(&data.keep, 0);\n }\n \n void clear_midx_file(struct repository *r)\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540339","messageId":"aea6aad695336a974df9c90ce7bd02ddcc33582f.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 03/16] strvec: introduce `strvec_init_alloc()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:40:58Z","receivedAt":"2026-03-29T21:41:00Z","isPatch":true,"body":"When the caller knows upfront how many elements will be pushed onto a\n`strvec`, it is useful to pre-allocate enough space in the array to fit\nthat many elements (and one additional slot to store NULL, indicating\nthe end of the list.)\n\nIntroduce `strvec_init_alloc()`, which allocates the backing array large\nenough to hold `alloc` elements and the termination marker without\nfurther reallocation. Reimplement `strvec_init()` as a special case of\n`strvec_init_alloc()`, namely when `alloc` is zero.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n strvec.c | 15 +++++++++++++--\n strvec.h | 5 +++++\n 2 files changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/strvec.c b/strvec.c\nindex f8de79f5579..15371980b84 100644\n--- a/strvec.c\n+++ b/strvec.c\n@@ -6,8 +6,19 @@ const char *empty_strvec[] = { NULL };\n \n void strvec_init(struct strvec *array)\n {\n-\tstruct strvec blank = STRVEC_INIT;\n-\tmemcpy(array, &blank, sizeof(*array));\n+\tstrvec_init_alloc(array, 0);\n+}\n+\n+void strvec_init_alloc(struct strvec *array, size_t alloc)\n+{\n+\tif (!alloc) {\n+\t\tstruct strvec blank = STRVEC_INIT;\n+\t\tmemcpy(array, &blank, sizeof(*array));\n+\t} else {\n+\t\tCALLOC_ARRAY(array->v, st_add(alloc, 1));\n+\t\tarray->nr = 0;\n+\t\tarray->alloc = alloc + 1;\n+\t}\n }\n \n void strvec_push_nodup(struct strvec *array, char *value)\ndiff --git a/strvec.h b/strvec.h\nindex f74e061e141..34cb1f939f0 100644\n--- a/strvec.h\n+++ b/strvec.h\n@@ -43,6 +43,11 @@ struct strvec {\n */\n void strvec_init(struct strvec *);\n \n+/*\n+ * Initializes an array large enough to store `alloc` elements.\n+ */\n+void strvec_init_alloc(struct strvec *, size_t alloc);\n+\n /* Push a copy of a string onto the end of the array. */\n const char *strvec_push(struct strvec *, const char *);\n \n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540340","messageId":"5fc72d5049a602ae5ede6bb243f44546f02d995d.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:00Z","receivedAt":"2026-03-29T21:41:02Z","isPatch":true,"body":"The `keep_hashes` array in `write_midx_internal()` accumulates the\nchecksums of MIDX files that should be retained when pruning stale\nentries from the MIDX chain. For similar reasons as in a previous\ncommit, rewrite this using a strvec, requiring us to pass one fewer\nparameter.\n\nUnlike the aforementioned previous commit, use a `strvec` instead of a\n`string_list`, which provides a more ergonomic interface to adjust the\nvalues at a particular index. The ordering is important here, as this\nvalue is used to determine the contents of the resulting\n`multi-pack-index-chain` file when writing with \"--incremental\".\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 51 ++++++++++++++++++++++++++-------------------------\n midx.c | 28 +++++++++++++++++-----------\n 2 files changed, 43 insertions(+), 36 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex d0351bde493..9f7d2bbf4cb 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -29,8 +29,7 @@ extern void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t const char *keep_hash);\n extern void clear_incremental_midx_files_ext(struct odb_source *source,\n \t\t\t\t\t const char *ext,\n-\t\t\t\t\t const char **keep_hashes,\n-\t\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t\t const struct strvec *keep_hashes);\n extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t\tconst char *idx_name);\n \n@@ -1109,8 +1108,7 @@ static int link_midx_to_chain(struct multi_pack_index *m)\n }\n \n static void clear_midx_files(struct odb_source *source,\n-\t\t\t const char **hashes, uint32_t hashes_nr,\n-\t\t\t unsigned incremental)\n+\t\t\t const struct strvec *hashes, unsigned incremental)\n {\n \t/*\n \t * if incremental:\n@@ -1124,13 +1122,15 @@ static void clear_midx_files(struct odb_source *source,\n \t */\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX };\n-\tuint32_t i, j;\n+\tuint32_t i;\n \n \tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tclear_incremental_midx_files_ext(source, exts[i],\n-\t\t\t\t\t\t hashes, hashes_nr);\n-\t\tfor (j = 0; j < hashes_nr; j++)\n-\t\t\tclear_midx_files_ext(source, exts[i], hashes[j]);\n+\t\tclear_incremental_midx_files_ext(source, exts[i], hashes);\n+\t\tif (hashes) {\n+\t\t\tfor (size_t j = 0; j < hashes->nr; j++)\n+\t\t\t\tclear_midx_files_ext(source, exts[i],\n+\t\t\t\t\t\t hashes->v[j]);\n+\t\t}\n \t}\n \n \tif (incremental)\n@@ -1268,7 +1268,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = -1;\n-\tconst char **keep_hashes = NULL;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tsize_t keep_hashes_nr = 0;\n \tstruct chunkfile *cf;\n \n@@ -1723,7 +1723,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t} else {\n \t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n \t}\n-\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n+\tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n \n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n@@ -1760,39 +1760,45 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n \t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n \n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.nr++;\n \t\t\t\tm = m->base_midx;\n \t\t\t}\n \n-\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\tkeep_hashes.v[i] = xstrdup(hash_to_hex_algop(midx_hash,\n \t\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes.nr++;\n \n \t\t\ti = 0;\n \t\t\tfor (m = ctx.m;\n \t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n \t\t\t m = m->base_midx) {\n-\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n+\t\t\t\tkeep_hashes.v[keep_hashes_nr - i - 1] =\n \t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.nr++;\n \t\t\t\ti++;\n \t\t\t}\n \t\t} else {\n-\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n \t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n \t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes.nr++;\n \n \t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n \t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n \n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tkeep_hashes.nr++;\n \t\t\t\tm = m->base_midx;\n \t\t\t}\n \t\t}\n \n \t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n+\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n \t} else {\n-\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n \t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n+\t\tkeep_hashes.nr++;\n \t}\n \n \tif (ctx.m || ctx.base_midx)\n@@ -1801,8 +1807,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, keep_hashes, keep_hashes_nr,\n-\t\t\t ctx.incremental);\n+\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n \tresult = 0;\n \n cleanup:\n@@ -1818,11 +1823,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n-\tif (keep_hashes) {\n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n-\t\t\tfree((char *)keep_hashes[i]);\n-\t\tfree(keep_hashes);\n-\t}\n+\tstrvec_clear(&keep_hashes);\n \tstrbuf_release(&midx_name);\n \tclose_midx(midx_to_free);\n \ndiff --git a/midx.c b/midx.c\nindex d322743f2db..e6b1fbe37d7 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -12,6 +12,7 @@\n #include \"chunk-format.h\"\n #include \"pack-bitmap.h\"\n #include \"pack-revindex.h\"\n+#include \"strvec.h\"\n \n #define MIDX_PACK_ERROR ((void *)(intptr_t)-1)\n \n@@ -19,8 +20,7 @@ int midx_checksum_valid(struct multi_pack_index *m);\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash);\n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t const struct strvec *keep_hashes);\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n@@ -799,20 +799,26 @@ void clear_midx_files_ext(struct odb_source *source, const char *ext,\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr)\n+\t\t\t\t const struct strvec *keep_hashes)\n {\n \tstruct clear_midx_data data = {\n-\t\t.keep = STRING_LIST_INIT_NODUP,\n+\t\t.keep = STRING_LIST_INIT_DUP,\n \t\t.ext = ext,\n \t};\n-\tuint32_t i;\n \n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tstring_list_append(&data.keep,\n-\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n-\t\t\t\t\t keep_hashes[i], ext));\n-\tstring_list_sort(&data.keep);\n+\tif (keep_hashes) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tfor (size_t i = 0; i < keep_hashes->nr; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\n+\t\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\",\n+\t\t\t\t keep_hashes->v[i], ext);\n+\t\t\tstring_list_append(&data.keep, buf.buf);\n+\t\t}\n+\n+\t\tstring_list_sort(&data.keep);\n+\t\tstrbuf_release(&buf);\n+\t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540341","messageId":"41cb5471bb96d38a1b9ae75d022e2c5356318939.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 05/16] midx: introduce `--checksum-only` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:03Z","receivedAt":"2026-03-29T21:41:05Z","isPatch":true,"body":"When writing an incremental MIDX layer, the MIDX machinery writes the\nnew layer into the multi-pack-index.d directory and then updates the\nmulti-pack-index-chain file to include the freshly written layer.\n\nFuture callers however may not wish to immediately update the MIDX chain\nitself, preferring instead to write out new layer(s) themselves before\natomically updating the chain. Concretely, the new incremental\nMIDX-based repacking strategy will want to do exactly this (that is,\nassemble the new MIDX chain itself before writing a new chain file and\natomically linking it into place).\n\nIntroduce a `--checksum-only` flag that:\n\n * writes the new MIDX layer into the multi-pack-index.d directory\n\n * prints its checksum\n\n * does not update the multi-pack-index-chain file.\n\nThe MIDX chain file (and thus, the lock protecting it) remain untouched,\nallowing callers to assemble the chain themselves. This flag requires\n`--incremental`, since the notion of a separate layer only makes sense\nfor incremental MIDXs.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 4 +--\n builtin/multi-pack-index.c | 28 ++++++++++++++++--\n midx-write.c | 38 ++++++++++++++++---------\n midx.h | 1 +\n t/t5334-incremental-multi-pack-index.sh | 17 +++++++++++\n t/t5335-compact-multi-pack-index.sh | 36 +++++++++++++++++++++++\n 6 files changed, 107 insertions(+), 17 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 61256830141..657e0639f6a 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -11,9 +11,9 @@ SYNOPSIS\n [verse]\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n-\t\t [--refs-snapshot=]\n+\t\t [--refs-snapshot=] [--[no-]checksum-only]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] \n+\t\t [--[no-]bitmap] [--[no-]checksum-only] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 0f72d96c02d..17ab1525454 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -16,11 +16,11 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=]\")\n+\t \" [--refs-snapshot=] [--[no-]checksum-only]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] \")\n+\t \" [--[no-]bitmap] [--[no-]checksum-only] \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -153,6 +153,9 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n+\t\t\tN_(\"write a MIDX layer without updating the MIDX chain\"),\n+\t\t\tMIDX_WRITE_CHECKSUM_ONLY),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n@@ -178,6 +181,15 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \tif (argc)\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--checksum-only\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -221,6 +233,9 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n+\t\t\tN_(\"write a MIDX layer without updating the MIDX chain\"),\n+\t\t\tMIDX_WRITE_CHECKSUM_ONLY),\n \t\tOPT_END(),\n \t};\n \n@@ -239,6 +254,15 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \tif (argc != 2)\n \t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--checksum-only\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(the_repository);\n \n \tFREE_AND_NULL(options);\ndiff --git a/midx-write.c b/midx-write.c\nindex 9f7d2bbf4cb..2c6905173ba 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1600,11 +1600,14 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \n \tif (ctx.incremental) {\n-\t\tstruct strbuf lock_name = STRBUF_INIT;\n+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n+\t\t\tstruct strbuf lock_name = STRBUF_INIT;\n \n-\t\tget_midx_chain_filename(opts->source, &lock_name);\n-\t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n-\t\tstrbuf_release(&lock_name);\n+\t\t\tget_midx_chain_filename(opts->source, &lock_name);\n+\t\t\thold_lock_file_for_update(&lk, lock_name.buf,\n+\t\t\t\t\t\t LOCK_DIE_ON_ERROR);\n+\t\t\tstrbuf_release(&lock_name);\n+\t\t}\n \n \t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n \t\tif (!incr) {\n@@ -1725,14 +1728,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n \n+\tif (opts->flags & MIDX_WRITE_CHECKSUM_ONLY)\n+\t\tprintf(\"%s\\n\", hash_to_hex_algop(midx_hash, r->hash_algo));\n+\n \tif (ctx.incremental) {\n-\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \n-\t\tif (!chainf) {\n-\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n-\t\t\tgoto cleanup;\n+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n+\t\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\t\tif (!chainf) {\n+\t\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t}\n \n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n@@ -1793,8 +1801,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t}\n \t\t}\n \n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY))\n+\t\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n+\t\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\",\n+\t\t\t\t\tkeep_hashes.v[i]);\n \t} else {\n \t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n \t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n@@ -1804,10 +1814,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.m || ctx.base_midx)\n \t\todb_close(ctx.repo->objects);\n \n-\tif (commit_lock_file(&lk) < 0)\n-\t\tdie_errno(_(\"could not write multi-pack-index\"));\n+\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n+\t\tif (commit_lock_file(&lk) < 0)\n+\t\t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t}\n \tresult = 0;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex 08f3728e520..9f1acd7ace4 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -83,6 +83,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n #define MIDX_WRITE_COMPACT (1 << 6)\n+#define MIDX_WRITE_CHECKSUM_ONLY (1 << 7)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex c9f5b4e87aa..ac249365f2b 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -96,6 +96,23 @@ test_expect_success 'show object from second pack' '\n \tgit cat-file -p 2.2\n '\n \n+test_expect_success 'write MIDX layer with --checksum-only' '\n+\ttest_commit checksum-only &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--checksum-only)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write non-incremental MIDX layer with --checksum-only' '\n+\ttest_must_fail git multi-pack-index write --bitmap --checksum-only 2>err &&\n+\ttest_grep \"cannot use --checksum-only without --incremental\" err\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 40f3844282f..3ad910a9d1c 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -290,4 +290,40 @@ test_expect_success 'MIDX compaction with bitmaps (non-trivial)' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --checksum-only' '\n+\tgit init midx-compact-with--checksum-only &&\n+\t(\n+\t\tcd midx-compact-with--checksum-only &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\tlayer=\"$(git multi-pack-index compact --incremental \\\n+\t\t\t--checksum-only \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n+\n+\t\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\n+\t\t# After writing the new layer, insert it into the chain\n+\t\t# manually. This is done in order to make $layer visible\n+\t\t# to the read-midx test helper below, and matches what\n+\t\t# the MIDX command would do without --checksum-only.\n+\t\t{\n+\t\t\tnth_line 1 \"$midx_chain.bak\" &&\n+\t\t\techo $layer &&\n+\t\t\tnth_line 4 \"$midx_chain.bak\"\n+\t\t} >$midx_chain &&\n+\n+\t\ttest-tool read-midx $objdir $layer >midx.data &&\n+\t\tgrep \"^pack-B-.*\\.idx\" midx.data &&\n+\t\tgrep \"^pack-C-.*\\.idx\" midx.data\n+\n+\t)\n+'\n+\n test_done\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540342","messageId":"6b42d284b79a25e97d506e22c709e4dbe6fddd58.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:06Z","receivedAt":"2026-03-29T21:41:07Z","isPatch":true,"body":"Both `compact` and `write --incremental` fix the base of the resulting\nMIDX layer: `compact` always places the compacted result on top of\n\"from's\" immediate parent in the chain, and `write --incremental` always\nappends a new layer to the existing tip. In both cases the base is not\nconfigurable.\n\nFuture callers need additional flexibility. For instance, the incremental\nMIDX-based repacking code may wish to write a layer based on some\nintermediate ancestor rather than the current tip, or produce a root\nlayer when replacing the bottommost entries in the chain.\n\nIntroduce a new `--base` option for both subcommands to specify the\nchecksum of the MIDX layer to use as the base. The given checksum must\nrefer to a valid layer in the MIDX chain that is an ancestor of the\ntopmost layer being written or compacted.\n\nThe special value \"none\" is accepted to produce a root layer with no\nparent. This will be needed when the incremental repacking machinery\ndetermines that the bottommost layers of the chain should be replaced.\n\nIf no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\nimmediate parent in the chain, and `write` appends to the existing tip.\n\nFor the `write` subcommand, `--base` requires `--checksum-only`. A plain\n`write --incremental` appends a new layer to the live chain tip with no\nmechanism to atomically replace it; overriding the base would produce a\nlayer that does not extend the tip, breaking chain invariants. With\n`--checksum-only` the chain is left unmodified and the caller is\nresponsible for assembling a valid chain.\n\nFor `compact`, no such restriction applies. The compaction operation\natomically replaces the compacted range in the chain file, so writing\nthe result on top of any valid ancestor preserves chain invariants.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 17 +++++-\n builtin/multi-pack-index.c | 24 ++++++--\n midx-write.c | 34 ++++++++++-\n midx.h | 5 +-\n t/t5334-incremental-multi-pack-index.sh | 30 ++++++++++\n t/t5335-compact-multi-pack-index.sh | 77 +++++++++++++++++++++++++\n 6 files changed, 178 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 657e0639f6a..635105ad801 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -12,8 +12,10 @@ SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=] [--[no-]checksum-only]\n+\t\t\t [--base=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] [--[no-]checksum-only] \n+\t\t [--[no-]bitmap] [--base=] [--[no-]checksum-only]\n+\t\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -83,6 +85,13 @@ marker).\n \t\tand packs not present in an existing MIDX layer.\n \t\tMigrates non-incremental MIDXs to incremental ones when\n \t\tnecessary.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base when writing a new incremental layer.\n+\t\tThe special value `none` indicates that the new layer\n+\t\tshould have no base (i.e., it becomes a root layer).\n+\t\tRequires `--checksum-only`.\n --\n \n compact::\n@@ -97,6 +106,12 @@ compact::\n \n \t--[no-]bitmap::\n \t\tControl whether or not a multi-pack bitmap is written.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base for the compacted result, instead of using\n+\t\tthe immediate parent of ``. The special value\n+\t\t`none` indicates that the result should have no base.\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 17ab1525454..3a12baacb56 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -16,11 +16,13 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=] [--[no-]checksum-only]\")\n+\t \" [--refs-snapshot=] [--[no-]checksum-only]\\n\" \\\n+\t \" [--base=]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] [--[no-]checksum-only] \")\n+\t \" [--[no-]bitmap] [--base=] [--[no-]checksum-only]\\n\" \\\n+\t \" \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -63,6 +65,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tchar *object_dir;\n \tconst char *preferred_pack;\n+\tconst char *incremental_base;\n \tchar *refs_snapshot;\n \tunsigned long batch_size;\n \tunsigned flags;\n@@ -151,6 +154,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n@@ -190,6 +195,13 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t\t options);\n \t}\n \n+\tif (opts.incremental_base &&\n+\t !(opts.flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n+\t\terror(_(\"cannot use --base without --checksum-only\"));\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -201,7 +213,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \n \t\tret = write_midx_file_only(source, &packs,\n \t\t\t\t\t opts.preferred_pack,\n-\t\t\t\t\t opts.refs_snapshot, opts.flags);\n+\t\t\t\t\t opts.refs_snapshot,\n+\t\t\t\t\t opts.incremental_base, opts.flags);\n \n \t\tstring_list_clear(&packs, 0);\n \t\tfree(opts.refs_snapshot);\n@@ -229,6 +242,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n@@ -290,7 +305,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n \t}\n \n-\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n+\tret = write_midx_file_compact(source, from_midx, to_midx,\n+\t\t\t\t opts.incremental_base, opts.flags);\n \n \treturn ret;\n }\ndiff --git a/midx-write.c b/midx-write.c\nindex 2c6905173ba..9298a5140fd 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1247,6 +1247,7 @@ struct write_midx_opts {\n \n \tconst char *preferred_pack_name;\n \tconst char *refs_snapshot;\n+\tconst char *incremental_base;\n \tunsigned flags;\n };\n \n@@ -1329,11 +1330,32 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t/*\n \t * If compacting MIDX layer(s) in the range [from, to], then the\n-\t * compacted MIDX will share the same base MIDX as 'from'.\n+\t * compacted MIDX will share the same base MIDX as 'from',\n+\t * unless a custom --base is specified (see below).\n \t */\n \tif (ctx.compact)\n \t\tctx.base_midx = ctx.compact_from->base_midx;\n \n+\tif (opts->incremental_base) {\n+\t\tif (!strcmp(opts->incremental_base, \"none\")) {\n+\t\t\tctx.base_midx = NULL;\n+\t\t} else {\n+\t\t\twhile (ctx.base_midx) {\n+\t\t\t\tconst char *cmp = midx_get_checksum_hex(ctx.base_midx);\n+\t\t\t\tif (!strcmp(opts->incremental_base, cmp))\n+\t\t\t\t\tbreak;\n+\n+\t\t\t\tctx.base_midx = ctx.base_midx->base_midx;\n+\t\t\t}\n+\n+\t\t\tif (!ctx.base_midx) {\n+\t\t\t\terror(_(\"could not find base MIDX '%s'\"),\n+\t\t\t\t opts->incremental_base);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n@@ -1846,7 +1868,8 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n int write_midx_file(struct odb_source *source,\n \t\t const char *preferred_pack_name,\n-\t\t const char *refs_snapshot, unsigned flags)\n+\t\t const char *refs_snapshot,\n+\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n@@ -1861,13 +1884,16 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags)\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.packs_to_include = packs_to_include,\n \t\t.preferred_pack_name = preferred_pack_name,\n \t\t.refs_snapshot = refs_snapshot,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags,\n \t};\n \n@@ -1877,12 +1903,14 @@ int write_midx_file_only(struct odb_source *source,\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.compact_from = from,\n \t\t.compact_to = to,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags | MIDX_WRITE_COMPACT,\n \t};\n \ndiff --git a/midx.h b/midx.h\nindex 9f1acd7ace4..e4a75ff2bef 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -132,10 +132,13 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags);\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags);\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct odb_source *source, unsigned flags);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex ac249365f2b..7733cafb647 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -113,6 +113,36 @@ test_expect_success 'write non-incremental MIDX layer with --checksum-only' '\n \ttest_grep \"cannot use --checksum-only without --incremental\" err\n '\n \n+test_expect_success 'write MIDX layer with --base without --checksum-only' '\n+\ttest_must_fail git multi-pack-index write --bitmap --incremental \\\n+\t\t--base=none 2>err &&\n+\ttest_grep \"cannot use --base without --checksum-only\" err\n+'\n+\n+test_expect_success 'write MIDX layer with --base=none and --checksum-only' '\n+\ttest_commit base-none &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--checksum-only --base=none)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write MIDX layer with --base= and --checksum-only' '\n+\ttest_commit base-hash &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--checksum-only --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 3ad910a9d1c..d8e1b03669a 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -304,6 +304,7 @@ test_expect_success 'MIDX compaction with --checksum-only' '\n \n \t\tlayer=\"$(git multi-pack-index compact --incremental \\\n \t\t\t--checksum-only \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n \n@@ -326,4 +327,80 @@ test_expect_success 'MIDX compaction with --checksum-only' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --base' '\n+\tgit init midx-compact-with--base &&\n+\t(\n+\t\tcd midx-compact-with--base &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tnth_line 1 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 1 \"$midx_chain\" >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with --base=none' '\n+\tgit init midx-compact-base-none &&\n+\t(\n+\t\tcd midx-compact-base-none &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\t# Compact the two bottommost layers (A and B) into a new\n+\t\t# root layer with no parent.\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=none \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" &&\n+\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\t# The upper layers (C and D) should be preserved\n+\t\t# unchanged.\n+\t\tnth_line 3 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 4 \"$midx_chain.bak\" >>expect &&\n+\t\tnth_line 2 \"$midx_chain\" >actual &&\n+\t\tnth_line 3 \"$midx_chain\" >>actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bogus --base checksum' '\n+\tgit init midx-compact-bogus-base &&\n+\t(\n+\t\tcd midx-compact-bogus-base &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t--base=deadbeef \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" 2>err &&\n+\t\ttest_grep \"could not find base MIDX\" err\n+\t)\n+'\n+\n test_done\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540343","messageId":"2d377f534071ad9def3fb4860b2581197d02726b.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 07/16] repack: track the ODB source via existing_packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:08Z","receivedAt":"2026-03-29T21:41:10Z","isPatch":true,"body":"Store the ODB source in the `existing_packs` struct and use that in\nplace of the raw `repo->objects->sources` access within `cmd_repack()`.\n\nThe source used is still assigned from the first source in the list, so\nthere are no functional changes in this commit. The changes instead\nserve two purposes (one immediate, one not):\n\n - The incremental MIDX-based repacking machinery will need to know what\n source is being used to read the existing MIDX/chain (should one\n exist).\n\n - In the future, if \"git repack\" is taught how to operate on other\n object sources, this field will serve as the authoritative value for\n that source.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 5 ++---\n repack.c | 2 ++\n repack.h | 1 +\n 3 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f6bb04bef72..44a95b56f23 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -402,7 +402,7 @@ int cmd_repack(int argc,\n \t\t * midx_has_unknown_packs() will make the decision for\n \t\t * us.\n \t\t */\n-\t\tif (!get_multi_pack_index(repo->objects->sources))\n+\t\tif (!get_multi_pack_index(existing.source))\n \t\t\tmidx_must_contain_cruft = 1;\n \t}\n \n@@ -549,8 +549,7 @@ int cmd_repack(int argc,\n \t\tunsigned flags = 0;\n \t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))\n \t\t\tflags |= MIDX_WRITE_INCREMENTAL;\n-\t\twrite_midx_file(repo->objects->sources,\n-\t\t\t\tNULL, NULL, flags);\n+\t\twrite_midx_file(existing.source, NULL, NULL, flags);\n \t}\n \n cleanup:\ndiff --git a/repack.c b/repack.c\nindex 596841027af..2ee6b51420a 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -154,6 +154,8 @@ void existing_packs_collect(struct existing_packs *existing,\n \t\t\tstring_list_append(&existing->non_kept_packs, buf.buf);\n \t}\n \n+\texisting->source = existing->repo->objects->sources;\n+\n \tstring_list_sort(&existing->kept_packs);\n \tstring_list_sort(&existing->non_kept_packs);\n \tstring_list_sort(&existing->cruft_packs);\ndiff --git a/repack.h b/repack.h\nindex bc9f2e1a5de..c0e9f0ca647 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -56,6 +56,7 @@ struct packed_git;\n \n struct existing_packs {\n \tstruct repository *repo;\n+\tstruct odb_source *source;\n \tstruct string_list kept_packs;\n \tstruct string_list non_kept_packs;\n \tstruct string_list cruft_packs;\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540344","messageId":"6af23849f1d499e7a23d1ef4a70ac3d29748e1ce.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 08/16] midx: expose `midx_layer_contains_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:11Z","receivedAt":"2026-03-29T21:41:12Z","isPatch":true,"body":"Rename the function `midx_contains_pack_1()` to instead be called\n`midx_layer_contains_pack()` and make it accessible. Unlike\n`midx_contains_pack()` (which recurses through the entire chain), this\nfunction checks only a single MIDX layer.\n\nThis will be used by a subsequent commit to determine whether a given\npack belongs to the tip MIDX layer specifically, rather than to any\nlayer in the chain.\n\nNo functional changes are present in this commit.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 +++---\n midx.h | 2 ++\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex e6b1fbe37d7..7d23338aa3a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -667,8 +667,8 @@ static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n \t\t m->pack_names[*(const size_t *)b]);\n }\n \n-static int midx_contains_pack_1(struct multi_pack_index *m,\n-\t\t\t\tconst char *idx_or_pack_name)\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -709,7 +709,7 @@ static int midx_contains_pack_1(struct multi_pack_index *m,\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n {\n \tfor (; m; m = m->base_midx)\n-\t\tif (midx_contains_pack_1(m, idx_or_pack_name))\n+\t\tif (midx_layer_contains_pack(m, idx_or_pack_name))\n \t\t\treturn 1;\n \treturn 0;\n }\ndiff --git a/midx.h b/midx.h\nindex e4a75ff2bef..f211a38b9e7 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -119,6 +119,8 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n int fill_midx_entry(struct multi_pack_index *m, const struct object_id *oid, struct pack_entry *e);\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t const char *idx_or_pack_name);\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n int prepare_multi_pack_index_one(struct odb_source *source);\n \n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540345","messageId":"5c9fea0fc788a883903f1dafe4965bc23d018f91.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 09/16] repack-midx: factor out `repack_prepare_midx_command()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:13Z","receivedAt":"2026-03-29T21:41:15Z","isPatch":true,"body":"The `write_midx_included_packs()` function assembles and executes a\n`git multi-pack-index write` command, constructing the argument list\ninline.\n\nFuture commits will introduce additional callers that need to construct\nsimilar `git multi-pack-index` commands (for both `write` and `compact`\nsubcommands), so extract the common portions of the command setup into a\nreusable `repack_prepare_midx_command()` helper.\n\nThe extracted helper sets `git_cmd`, pushes the `multi-pack-index`\nsubcommand and verb, and handles `--progress`/`--no-progress` and\n`--bitmap` flags. The remaining arguments that are specific to the\n`write` subcommand (such as `--stdin-packs`) are left to the caller.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 30 +++++++++++++++++++-----------\n 1 file changed, 19 insertions(+), 11 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 0682b80c427..83151d4734a 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -275,6 +275,23 @@ static void remove_redundant_bitmaps(struct string_list *include,\n \tstrbuf_release(&path);\n }\n \n+static void repack_prepare_midx_command(struct child_process *cmd,\n+\t\t\t\t\tstruct repack_write_midx_opts *opts,\n+\t\t\t\t\tconst char *verb)\n+{\n+\tcmd->git_cmd = 1;\n+\n+\tstrvec_pushl(&cmd->args, \"multi-pack-index\", verb, NULL);\n+\n+\tif (opts->show_progress)\n+\t\tstrvec_push(&cmd->args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd->args, \"--no-progress\");\n+\n+\tif (opts->write_bitmaps)\n+\t\tstrvec_push(&cmd->args, \"--bitmap\");\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -289,18 +306,9 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \n \tcmd.in = -1;\n-\tcmd.git_cmd = 1;\n \n-\tstrvec_push(&cmd.args, \"multi-pack-index\");\n-\tstrvec_pushl(&cmd.args, \"write\", \"--stdin-packs\", NULL);\n-\n-\tif (opts->show_progress)\n-\t\tstrvec_push(&cmd.args, \"--progress\");\n-\telse\n-\t\tstrvec_push(&cmd.args, \"--no-progress\");\n-\n-\tif (opts->write_bitmaps)\n-\t\tstrvec_push(&cmd.args, \"--bitmap\");\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540346","messageId":"cd5eba0f665a4d3a4febc47f1fe5f13200a4f941.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 10/16] repack-midx: extract `repack_fill_midx_stdin_packs()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:16Z","receivedAt":"2026-03-29T21:41:18Z","isPatch":true,"body":"The function `write_midx_included_packs()` manages the lifecycle of\nwriting packs to stdin when running `git multi-pack-index write` as a\nchild process.\n\nExtract a standalone `repack_fill_midx_stdin_packs()` helper, which\nhandles `--stdin-packs` argument setup, starting the command, writing\npack names to its standard input, and finishing the command.\n\nThis simplifies `write_midx_included_packs()` and prepares for a\nsubsequent commit where the same helper is called with `cmd->out = -1`\nto capture the MIDX's checksum from the command's standard output,\nwhich is needed when writing MIDX layers with `--checksum-only`.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 38 ++++++++++++++++++++++++--------------\n 1 file changed, 24 insertions(+), 14 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 83151d4734a..78f069c2151 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -292,23 +292,42 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n \t\tstrvec_push(&cmd->args, \"--bitmap\");\n }\n \n+static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n+\t\t\t\t\tstruct string_list *include)\n+{\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\n+\tcmd->in = -1;\n+\n+\tstrvec_push(&cmd->args, \"--stdin-packs\");\n+\n+\tret = start_command(cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tin = xfdopen(cmd->in, \"w\");\n+\tfor_each_string_list_item(item, include)\n+\t\tfprintf(in, \"%s\\n\", item->string);\n+\tfclose(in);\n+\n+\treturn finish_command(cmd);\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n \tstruct string_list_item *item;\n \tstruct packed_git *preferred = pack_geometry_preferred_pack(opts->geometry);\n-\tFILE *in;\n \tint ret = 0;\n \n \tmidx_included_packs(&include, opts);\n \tif (!include.nr)\n \t\tgoto done;\n \n-\tcmd.in = -1;\n-\n \trepack_prepare_midx_command(&cmd, opts, \"write\");\n-\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n@@ -350,16 +369,7 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = start_command(&cmd);\n-\tif (ret)\n-\t\tgoto done;\n-\n-\tin = xfdopen(cmd.in, \"w\");\n-\tfor_each_string_list_item(item, &include)\n-\t\tfprintf(in, \"%s\\n\", item->string);\n-\tfclose(in);\n-\n-\tret = finish_command(&cmd);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540347","messageId":"d2c667e841300c3310e3a23493ac27eb2b08992b.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 11/16] repack-geometry: prepare for incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:18Z","receivedAt":"2026-03-29T21:41:20Z","isPatch":true,"body":"Teach `pack_geometry_init()` to optionally restrict the set of\nrepacking candidates to only packs in the tip MIDX layer when a\n`midx_layer_threshold` is configured. If the tip layer has fewer packs\nthan the threshold, those packs are excluded entirely; otherwise only\npacks in that layer participate in the geometric repack.\n\nAlso track whether any tip-layer packs were included in the rollup\n(`midx_tip_rewritten`), which a subsequent commit will use to decide\nhow to update the MIDX chain after repacking.\n\nSigned-off-by: Taylor Blau \n---\n repack-geometry.c | 37 +++++++++++++++++++++++++++++++++++++\n repack.h | 4 ++++\n 2 files changed, 41 insertions(+)\n\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex 7cebd0cb45f..d2065205f87 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -4,6 +4,7 @@\n #include \"repack.h\"\n #include \"repository.h\"\n #include \"hex.h\"\n+#include \"midx.h\"\n #include \"packfile.h\"\n \n static uint32_t pack_geometry_weight(struct packed_git *p)\n@@ -31,8 +32,30 @@ void pack_geometry_init(struct pack_geometry *geometry,\n {\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n+\tstruct multi_pack_index *m = get_multi_pack_index(existing->source);\n \n \trepo_for_each_pack(existing->repo, p) {\n+\t\tif (geometry->midx_layer_threshold_set && m &&\n+\t\t p->multi_pack_index) {\n+\t\t\t/*\n+\t\t\t * When writing MIDX layers incrementally,\n+\t\t\t * ignore packs unless they are in the most\n+\t\t\t * recent MIDX layer *and* there are at least\n+\t\t\t * 'midx_layer_threshold' packs in that layer.\n+\t\t\t *\n+\t\t\t * Otherwise 'p' is either in an older layer, or\n+\t\t\t * the youngest layer does not have enough packs\n+\t\t\t * to consider its packs as candidates for\n+\t\t\t * repacking. In either of those cases we want\n+\t\t\t * to ignore the pack.\n+\t\t\t */\n+\t\t\tif (m->num_packs > geometry->midx_layer_threshold &&\n+\t\t\t midx_layer_contains_pack(m, pack_basename(p)))\n+\t\t\t\t;\n+\t\t\telse\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -173,6 +196,20 @@ void pack_geometry_split(struct pack_geometry *geometry)\n \tgeometry->promisor_split = compute_pack_geometry_split(geometry->promisor_pack,\n \t\t\t\t\t\t\t geometry->promisor_pack_nr,\n \t\t\t\t\t\t\t geometry->split_factor);\n+\tfor (uint32_t i = 0; i < geometry->split; i++) {\n+\t\tstruct packed_git *p = geometry->pack[i];\n+\t\t/*\n+\t\t * During incremental MIDX/bitmap repacking, any packs\n+\t\t * included in the rollup are either (a) not MIDX'd, or\n+\t\t * (b) contained in the tip layer iff it has more than\n+\t\t * the threshold number of packs.\n+\t\t *\n+\t\t * In the latter case, we can safely conclude that the\n+\t\t * tip of the MIDX chain will be rewritten.\n+\t\t */\n+\t\tif (p->multi_pack_index)\n+\t\t\tgeometry->midx_tip_rewritten = true;\n+\t}\n }\n \n struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry)\ndiff --git a/repack.h b/repack.h\nindex c0e9f0ca647..77d24ee45fb 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -108,6 +108,10 @@ struct pack_geometry {\n \tuint32_t promisor_pack_nr, promisor_pack_alloc;\n \tuint32_t promisor_split;\n \n+\tuint32_t midx_layer_threshold;\n+\tbool midx_layer_threshold_set;\n+\tbool midx_tip_rewritten;\n+\n \tint split_factor;\n };\n \n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540348","messageId":"56dc6d55974ffe08647d57915d0861181e1d7d41.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 12/16] builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:21Z","receivedAt":"2026-03-29T21:41:23Z","isPatch":true,"body":"Change the --write-midx (-m) flag from an OPT_BOOL to an OPT_CALLBACK\nthat accepts an optional mode argument. Introduce an enum with\nREPACK_WRITE_MIDX_NONE and REPACK_WRITE_MIDX_DEFAULT to distinguish\nbetween the two states, and update all existing boolean checks\naccordingly.\n\nFor now, passing no argument (or just `-m`) selects the default mode,\npreserving existing behavior. A subsequent commit will add a new mode\nfor writing incremental MIDXs.\n\nExtract repack_write_midx() as a dispatcher that selects the\nappropriate MIDX-writing implementation based on the mode.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 50 ++++++++++++++++++++++++++++++++++++------------\n repack-midx.c | 14 +++++++++++++-\n repack.h | 8 +++++++-\n 3 files changed, 58 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 44a95b56f23..3a5042491d6 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -97,6 +97,24 @@ static int repack_config(const char *var, const char *value,\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n+static int option_parse_write_midx(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum repack_write_midx_mode *cfg = opt->value;\n+\n+\tif (unset) {\n+\t\t*cfg = REPACK_WRITE_MIDX_NONE;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!arg || !*arg)\n+\t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse\n+\t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_repack(int argc,\n \t const char **argv,\n \t const char *prefix,\n@@ -119,7 +137,7 @@ int cmd_repack(int argc,\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct pack_objects_args po_args = PACK_OBJECTS_ARGS_INIT;\n \tstruct pack_objects_args cruft_po_args = PACK_OBJECTS_ARGS_INIT;\n-\tint write_midx = 0;\n+\tenum repack_write_midx_mode write_midx = REPACK_WRITE_MIDX_NONE;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n \tconst char *filter_to = NULL;\n@@ -185,8 +203,14 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_INTEGER('g', \"geometric\", &geometry.split_factor,\n \t\t\t N_(\"find a geometric progression with factor \")),\n-\t\tOPT_BOOL('m', \"write-midx\", &write_midx,\n-\t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n+\t\tOPT_CALLBACK_F(0, \"write-midx\", &write_midx,\n+\t\t\t N_(\"mode\"),\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t PARSE_OPT_OPTARG, option_parse_write_midx),\n+\t\tOPT_SET_INT_F('m', NULL, &write_midx,\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t REPACK_WRITE_MIDX_DEFAULT,\n+\t\t\t PARSE_OPT_HIDDEN),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n \t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n@@ -221,14 +245,16 @@ int cmd_repack(int argc,\n \t\tpack_everything |= ALL_INTO_ONE;\n \n \tif (write_bitmaps < 0) {\n-\t\tif (!write_midx &&\n+\t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n \t}\n \tif (po_args.pack_kept_objects < 0)\n-\t\tpo_args.pack_kept_objects = write_bitmaps > 0 && !write_midx;\n+\t\tpo_args.pack_kept_objects = write_bitmaps > 0 &&\n+\t\t\twrite_midx == REPACK_WRITE_MIDX_NONE;\n \n-\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n+\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) &&\n+\t write_midx == REPACK_WRITE_MIDX_NONE)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n \tif (write_bitmaps && po_args.local &&\n@@ -244,7 +270,7 @@ int cmd_repack(int argc,\n \t\twrite_bitmaps = 0;\n \t}\n \n-\tif (write_midx && write_bitmaps) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \n \t\tstrbuf_addf(&path, \"%s/%s_XXXXXX\",\n@@ -297,7 +323,7 @@ int cmd_repack(int argc,\n \t}\n \tif (repo_has_promisor_remote(repo))\n \t\tstrvec_push(&cmd.args, \"--exclude-promisor-objects\");\n-\tif (!write_midx) {\n+\tif (write_midx == REPACK_WRITE_MIDX_NONE) {\n \t\tif (write_bitmaps > 0)\n \t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n \t\telse if (write_bitmaps < 0)\n@@ -504,7 +530,7 @@ int cmd_repack(int argc,\n \tif (delete_redundant && pack_everything & ALL_INTO_ONE)\n \t\texisting_packs_mark_for_deletion(&existing, &names);\n \n-\tif (write_midx) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE) {\n \t\tstruct repack_write_midx_opts opts = {\n \t\t\t.existing = &existing,\n \t\t\t.geometry = &geometry,\n@@ -513,11 +539,11 @@ int cmd_repack(int argc,\n \t\t\t.packdir = packdir,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n-\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft\n+\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.mode = write_midx,\n \t\t};\n \n-\t\tret = write_midx_included_packs(&opts);\n-\n+\t\tret = repack_write_midx(&opts);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \t}\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 78f069c2151..4a568a2a9b8 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -315,7 +315,7 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n-int write_midx_included_packs(struct repack_write_midx_opts *opts)\n+static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n@@ -378,3 +378,15 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \n \treturn ret;\n }\n+\n+int repack_write_midx(struct repack_write_midx_opts *opts)\n+{\n+\tswitch (opts->mode) {\n+\tcase REPACK_WRITE_MIDX_NONE:\n+\t\tBUG(\"write_midx mode is NONE?\");\n+\tcase REPACK_WRITE_MIDX_DEFAULT:\n+\t\treturn write_midx_included_packs(opts);\n+\tdefault:\n+\t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n+\t}\n+}\ndiff --git a/repack.h b/repack.h\nindex 77d24ee45fb..81907fcce7f 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -134,6 +134,11 @@ void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\n \n+enum repack_write_midx_mode {\n+\tREPACK_WRITE_MIDX_NONE,\n+\tREPACK_WRITE_MIDX_DEFAULT,\n+};\n+\n struct repack_write_midx_opts {\n \tstruct existing_packs *existing;\n \tstruct pack_geometry *geometry;\n@@ -143,10 +148,11 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tenum repack_write_midx_mode mode;\n };\n \n void midx_snapshot_refs(struct repository *repo, struct tempfile *f);\n-int write_midx_included_packs(struct repack_write_midx_opts *opts);\n+int repack_write_midx(struct repack_write_midx_opts *opts);\n \n int write_filtered_pack(const struct write_pack_opts *opts,\n \t\t\tstruct existing_packs *existing,\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540349","messageId":"e304e316ac4d4e92125e2e57b32fa445acf66e3e.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 13/16] packfile: ensure `close_pack_revindex()` frees in-memory revindex","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:23Z","receivedAt":"2026-03-29T21:41:25Z","isPatch":true,"body":"The following commit will introduce a case where we write a MIDX bitmap\nover packs that do not themselves have on-disk *.rev files.\n\nThis case is supported within Git, and we will simply fall back to\ngenerating the revindex in memory. But we don't ever release that\nmemory, causing a leak that is exposed by a test introduced in the\nfollowing commit.\n\n(As far as I could find, we never free()'d memory allocated as a\nbyproduct of creating an in-memory revindex, likely because that code\npredates the leak-checking niceties we have in the test suite now.)\n\nRectify this by calling `FREE_AND_NULL()` on the `p->revindex` field\nwhen calling `close_pack_revindex()`.\n\nSigned-off-by: Taylor Blau \n---\n packfile.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/packfile.c b/packfile.c\nindex d4de9f3ffe8..97f7662e079 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -420,6 +420,8 @@ void close_pack_index(struct packed_git *p)\n \n static void close_pack_revindex(struct packed_git *p)\n {\n+\tFREE_AND_NULL(p->revindex);\n+\n \tif (!p->revindex_map)\n \t\treturn;\n \n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540350","messageId":"8d5aa3d6fa70d408fc02561f2203ae1a48186c91.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 14/16] repack: implement incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:26Z","receivedAt":"2026-03-29T21:41:28Z","isPatch":true,"body":"Implement the `write_midx_incremental()` function, which builds and\nmaintains an incremental MIDX chain as part of the geometric repacking\nprocess.\n\nUnlike the default mode which writes a single flat MIDX, the incremental\nmode constructs a compaction plan that determines which MIDX layers to\nwrite, compact, or copy, and then executes each step using `git\nmulti-pack-index` subcommands with the --checksum-only flag.\n\nThe repacking strategy works as follows:\n\n * Acquire the lock guarding the multi-pack-index-chain.\n\n * A new MIDX layer is always written containing the newly created\n pack(s). If the tip MIDX layer was rewritten during geometric\n repacking, any surviving packs from that layer are also included.\n\n * Starting from the new layer, adjacent MIDX layers are merged together\n as long as the accumulated object count exceeds half the object count\n of the next deeper layer (controlled by 'repack.midxSplitFactor').\n\n * Remaining layers in the chain are evaluated pairwise and either\n compacted or copied as-is, following the same merging condition.\n\n * Write the contents of the new multi-pack-index chain, atomically move\n it into place, and then release the lock.\n\n * Delete any now-unused MIDX layers.\n\nAfter writing the new layer, the strategy is evaluated among the\nexisting MIDX layers in order from oldest to newest. Each step that\nwrites a new MIDX layer uses \"--checksum-only\" to avoid updating the\nmulti-pack-index-chain file. After all steps are complete, the new chain\nfile is written and then atomically moved into place.\n\nAt present, this functionality is exposed behind a new enum value,\n`REPACK_WRITE_MIDX_INCREMENTAL`, but has no external callers. A\nsubsequent commit will expose this mode via `git repack\n--write-midx=incremental`.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 578 +++++++++++++++++++++++++++++++++++++++++++++++++-\n repack.h | 3 +\n 2 files changed, 579 insertions(+), 2 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 4a568a2a9b8..a159b879775 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -2,12 +2,16 @@\n #include \"repack.h\"\n #include \"hash.h\"\n #include \"hex.h\"\n+#include \"lockfile.h\"\n+#include \"midx.h\"\n #include \"odb.h\"\n #include \"oidset.h\"\n #include \"pack-bitmap.h\"\n+#include \"path.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n #include \"tempfile.h\"\n+#include \"trace2.h\"\n \n struct midx_snapshot_ref_data {\n \tstruct repository *repo;\n@@ -293,13 +297,16 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n }\n \n static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n-\t\t\t\t\tstruct string_list *include)\n+\t\t\t\t\tstruct string_list *include,\n+\t\t\t\t\tstruct string_list *out)\n {\n \tstruct string_list_item *item;\n \tFILE *in;\n \tint ret;\n \n \tcmd->in = -1;\n+\tif (out)\n+\t\tcmd->out = -1;\n \n \tstrvec_push(&cmd->args, \"--stdin-packs\");\n \n@@ -312,6 +319,17 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n \t\tfprintf(in, \"%s\\n\", item->string);\n \tfclose(in);\n \n+\tif (out) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tFILE *outf = xfdopen(cmd->out, \"r\");\n+\n+\t\twhile (strbuf_getline(&buf, outf) != EOF)\n+\t\t\tstring_list_append(out, buf.buf);\n+\t\tstrbuf_release(&buf);\n+\n+\t\tfclose(outf);\n+\t}\n+\n \treturn finish_command(cmd);\n }\n \n@@ -369,7 +387,7 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include, NULL);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n@@ -379,6 +397,560 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \treturn ret;\n }\n \n+struct midx_compaction_step {\n+\tunion {\n+\t\tstruct multi_pack_index *copy;\n+\t\tstruct string_list write;\n+\t\tstruct {\n+\t\t\tstruct multi_pack_index *from;\n+\t\t\tstruct multi_pack_index *to;\n+\t\t} compact;\n+\t} u;\n+\n+\tuint32_t objects_nr;\n+\tchar *csum;\n+\n+\tenum {\n+\t\tMIDX_COMPACTION_STEP_UNKNOWN,\n+\t\tMIDX_COMPACTION_STEP_COPY,\n+\t\tMIDX_COMPACTION_STEP_WRITE,\n+\t\tMIDX_COMPACTION_STEP_COMPACT,\n+\t} type;\n+};\n+\n+static const char *midx_compaction_step_base(const struct midx_compaction_step *step)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot use UNKNOWN step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_get_checksum_hex(step->u.copy);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\tBUG(\"cannot use WRITE step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_get_checksum_hex(step->u.compact.to);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static int midx_compaction_step_exec_copy(struct midx_compaction_step *step)\n+{\n+\tstep->csum = xstrdup(midx_get_checksum_hex(step->u.copy));\n+\treturn 0;\n+}\n+\n+static int midx_compaction_step_exec_write(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t\t const char *base)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list hash = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\tconst char *preferred_pack = NULL;\n+\tint ret = 0;\n+\n+\tif (!step->u.write.nr) {\n+\t\tret = error(_(\"no packs to write MIDX during compaction\"));\n+\t\tgoto out;\n+\t}\n+\n+\tfor_each_string_list_item(item, &step->u.write) {\n+\t\tif (item->util)\n+\t\t\tpreferred_pack = item->string;\n+\t}\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--checksum-only\", NULL);\n+\tstrvec_pushf(&cmd.args, \"--base=%s\", base ? base : \"none\");\n+\n+\tif (preferred_pack) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\n+\t\tstrbuf_addstr(&buf, preferred_pack);\n+\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\t\tstrbuf_addstr(&buf, \".pack\");\n+\n+\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\", buf.buf);\n+\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\tret = repack_fill_midx_stdin_packs(&cmd, &step->u.write, &hash);\n+\tif (hash.nr != 1) {\n+\t\tret = error(_(\"expected exactly one line during MIDX write, \"\n+\t\t\t \"got: %\"PRIuMAX),\n+\t\t\t (uintmax_t)hash.nr);\n+\t\tgoto out;\n+\t}\n+\n+\tstep->csum = xstrdup(hash.items[0].string);\n+\n+out:\n+\tstring_list_clear(&hash, 0);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec_compact(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tFILE *out = NULL;\n+\tint ret;\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"compact\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--checksum-only\",\n+\t\t midx_get_checksum_hex(step->u.compact.from),\n+\t\t midx_get_checksum_hex(step->u.compact.to), NULL);\n+\n+\tcmd.out = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\tgoto out;\n+\n+\tout = xfdopen(cmd.out, \"r\");\n+\twhile (strbuf_getline_lf(&buf, out) != EOF) {\n+\t\tif (step->csum) {\n+\t\t\tret = error(_(\"unexpected MIDX output: '%s'\"), buf.buf);\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep->csum = strbuf_detach(&buf, NULL);\n+\t}\n+\n+\tret = finish_command(&cmd);\n+\n+out:\n+\tif (out)\n+\t\tfclose(out);\n+\tstrbuf_release(&buf);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec(struct midx_compaction_step *step,\n+\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t const char *base)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot execute UNKNOWN midx compaction step\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_compaction_step_exec_copy(step);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\treturn midx_compaction_step_exec_write(step, opts, base);\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_compaction_step_exec_compact(step, opts);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static void midx_compaction_step_release(struct midx_compaction_step *step)\n+{\n+\tif (step->type == MIDX_COMPACTION_STEP_WRITE)\n+\t\tstring_list_clear(&step->u.write, 0);\n+\tfree(step->csum);\n+}\n+\n+static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step step = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\tuint32_t i;\n+\tint ret = 0;\n+\n+\ttrace2_region_enter(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tfor (i = 0; m && i < m->num_packs + m->num_packs_in_base; i++) {\n+\t\tif (prepare_midx_pack(m, i)) {\n+\t\t\tret = error(_(\"could not load pack %\"PRIu32\" from MIDX\"),\n+\t\t\t\t i);\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\ttrace2_region_enter(\"repack\", \"steps:write\", opts->existing->repo);\n+\n+\t/*\n+\t * The first MIDX in the resulting chain is always going to be\n+\t * new.\n+\t *\n+\t * At a minimum, it will include all of the newly written packs.\n+\t * If there is an existing MIDX whose tip layer contains packs\n+\t * that were repacked, it will also include any of its packs\n+\t * which were *not* rolled up as part of the geometric repack\n+\t * (if any), and the previous tip will be replaced.\n+\t *\n+\t * It may grow to include the packs from zero or more MIDXs from\n+\t * the old chain, beginning either at the old tip (if the MIDX\n+\t * was *not* rewritten) or the old tip's base MIDX layer\n+\t * (otherwise).\n+\t */\n+\tstep.type = MIDX_COMPACTION_STEP_WRITE;\n+\tstring_list_init_dup(&step.u.write);\n+\n+\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"pack-%s.idx\", opts->names->items[i].string);\n+\t\tstring_list_append(&step.u.write, buf.buf);\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:fresh\",\n+\t\t\t\t step.u.write.items[step.u.write.nr - 1].string);\n+\t}\n+\tfor (i = 0; i < opts->geometry->split; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:fresh:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n+\n+\t/*\n+\t * Now handle any existing packs which were *not* rewritten.\n+\t *\n+\t * The list of packs in opts->geometry only contains MIDX'd\n+\t * packs from the newest layer when that layer has more than\n+\t * 'repack.midxNewLayerThreshold' number of packs.\n+\t *\n+\t * If the MIDX tip was rewritten (that is, one or more of those\n+\t * packs appear below the split line), then add all packs above\n+\t * the split line to the new layer, as the old one is no longer\n+\t * usable.\n+\t *\n+\t * If the MIDX tip was not rewritten (that is, all MIDX'd packs\n+\t * from the youngest layer appear below the split line, or were\n+\t * not included in the geometric repack at all because there\n+\t * were too few of them), ignore them since we'll retain the\n+\t * existing layer as-is.\n+\t */\n+\tfor (i = opts->geometry->split; i < opts->geometry->pack_nr; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tstruct string_list_item *item;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\tif (p->multi_pack_index &&\n+\t\t !opts->geometry->midx_tip_rewritten) {\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"exclude:unmodified\", buf.buf);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified\", buf.buf);\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified:midx\",\n+\t\t\t\t p->multi_pack_index ? \"true\" : \"false\");\n+\n+\t\titem = string_list_append(&step.u.write, buf.buf);\n+\t\tif (p->multi_pack_index || i == opts->geometry->pack_nr - 1)\n+\t\t\titem->util = (void *)1; /* mark as preferred */\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:unmodified:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n+\n+\t/*\n+\t * If the MIDX tip was rewritten, then we no longer consider it\n+\t * a candidate for compaction, since it will not exist in the\n+\t * MIDX chain being built.\n+\t */\n+\tif (opts->geometry->midx_tip_rewritten)\n+\t\tm = m->base_midx;\n+\n+\ttrace2_data_string(\"repack\", opts->existing->repo, \"midx:rewrote-tip\",\n+\t\t\t opts->geometry->midx_tip_rewritten ? \"true\" : \"false\");\n+\n+\ttrace2_region_enter(\"repack\", \"compact\", opts->existing->repo);\n+\n+\t/*\n+\t * Compact additional MIDX layers into this proposed one until\n+\t * the merging condition is violated.\n+\t */\n+\twhile (m) {\n+\t\tuint32_t preferred_pack_idx;\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"candidate\", midx_get_checksum_hex(m));\n+\n+\t\tif (step.objects_nr < m->num_objects / opts->midx_split_factor) {\n+\t\t\t/*\n+\t\t\t * Stop compacting MIDX layer as soon as the\n+\t\t\t * merged size is less than half the size of the\n+\t\t\t * next layer in the chain.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"compact\", \"violated\");\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"next_objects_nr\",\n+\t\t\t\t\t (uintmax_t)m->num_objects);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"split_factor\",\n+\t\t\t\t\t (uintmax_t)opts->midx_split_factor);\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (midx_preferred_pack(m, &preferred_pack_idx) < 0) {\n+\t\t\tret = error(_(\"could not find preferred pack for MIDX \"\n+\t\t\t\t \"%s\"), midx_get_checksum_hex(m));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\tstruct string_list_item *item;\n+\t\t\tuint32_t pack_int_id = i + m->num_packs_in_base;\n+\t\t\tstruct packed_git *p = nth_midxed_pack(m, pack_int_id);\n+\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"midx:pack\", buf.buf);\n+\n+\t\t\titem = string_list_append(&step.u.write, buf.buf);\n+\t\t\tif (pack_int_id == preferred_pack_idx)\n+\t\t\t\titem->util = (void *)1; /* mark as preferred */\n+\t\t}\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, m->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep.objects_nr += m->num_objects;\n+\n+\t\tm = m->base_midx;\n+\t}\n+\n+\tif (step.u.write.nr > 0) {\n+\t\t/*\n+\t\t * As long as there is at least one new pack to write\n+\t\t * (and thus the MIDX is non-empty), add it to the plan.\n+\t\t */\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\t\tsteps[steps_nr++] = step;\n+\t}\n+\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:objects_nr\", (uintmax_t)step.objects_nr);\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:packs_nr\", (uintmax_t)step.u.write.nr);\n+\n+\ttrace2_region_leave(\"repack\", \"compact\", opts->existing->repo);\n+\ttrace2_region_leave(\"repack\", \"steps:write\", opts->existing->repo);\n+\n+\ttrace2_region_enter(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n+\t/*\n+\t * Then start over, repeat, and either compact or keep as-is\n+\t * each MIDX layer until we have exhausted the chain.\n+\t *\n+\t * Finally, evaluate the remainder of the chain (if any) and\n+\t * either compact a sequence of adjacent layers, or keep\n+\t * individual layers as-is according to the same merging\n+\t * condition as above.\n+\t */\n+\twhile (m) {\n+\t\tstruct multi_pack_index *next = m;\n+\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\n+\t\tmemset(&step, 0, sizeof(step));\n+\t\tstep.type = MIDX_COMPACTION_STEP_UNKNOWN;\n+\n+\t\ttrace2_region_enter(\"repack\", \"step\", opts->existing->repo);\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"from\", midx_get_checksum_hex(m));\n+\n+\t\twhile (next) {\n+\t\t\tuint32_t proposed_objects_nr;\n+\t\t\tif (unsigned_add_overflows(step.objects_nr, next->num_objects)) {\n+\t\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tproposed_objects_nr = step.objects_nr + next->num_objects;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed\",\n+\t\t\t\t\t midx_get_checksum_hex(next));\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed:objects_nr\",\n+\t\t\t\t\t (uintmax_t)next->num_objects);\n+\n+\t\t\tif (!next->base_midx) {\n+\t\t\t\t/*\n+\t\t\t\t * If we are at the end of the MIDX\n+\t\t\t\t * chain, there is nothing to compact,\n+\t\t\t\t * so mark it and stop.\n+\t\t\t\t */\n+\t\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (proposed_objects_nr < next->base_midx->num_objects / opts->midx_split_factor) {\n+\t\t\t\t/*\n+\t\t\t\t * If there is a MIDX following this\n+\t\t\t\t * one, but our accumulated size is less\n+\t\t\t\t * than half of its size, compacting\n+\t\t\t\t * them would violate the merging\n+\t\t\t\t * condition, so stop here.\n+\t\t\t\t */\n+\n+\t\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at\",\n+\t\t\t\t\t\t midx_get_checksum_hex(next->base_midx));\n+\t\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at:objects_nr\",\n+\t\t\t\t\t\t (uintmax_t)next->base_midx->num_objects);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * Otherwise, it is OK to compact the next layer\n+\t\t\t * into this one. Do so, and then continue\n+\t\t\t * through the remainder of the chain.\n+\t\t\t */\n+\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"step:objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\tnext = next->base_midx;\n+\t\t}\n+\n+\t\tif (m == next) {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COPY;\n+\t\t\tstep.u.copy = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"copy\");\n+\t\t} else {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COMPACT;\n+\t\t\tstep.u.compact.from = next;\n+\t\t\tstep.u.compact.to = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"to\", midx_get_checksum_hex(m));\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"compact\");\n+\t\t}\n+\n+\t\tm = next->base_midx;\n+\t\tsteps[steps_nr++] = step;\n+\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n+\t}\n+\n+\ttrace2_region_leave(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n+out:\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+\n+\tstrbuf_release(&buf);\n+\n+\ttrace2_region_leave(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n+\treturn ret;\n+}\n+\n+static int write_midx_incremental(struct repack_write_midx_opts *opts)\n+{\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct strbuf lock_name = STRBUF_INIT;\n+\tstruct lock_file lf;\n+\tsize_t steps_nr = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tget_midx_chain_filename(opts->existing->source, &lock_name);\n+\tif (safe_create_leading_directories(opts->existing->repo,\n+\t\t\t\t\t lock_name.buf))\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t lock_name.buf);\n+\thold_lock_file_for_update(&lf, lock_name.buf, LOCK_DIE_ON_ERROR);\n+\n+\tif (!fdopen_lock_file(&lf, \"w\")) {\n+\t\tret = error_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\tgoto done;\n+\t}\n+\n+\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < steps_nr; i++) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tchar *base = NULL;\n+\n+\t\tif (i + 1 < steps_nr)\n+\t\t\tbase = xstrdup(midx_compaction_step_base(&steps[i + 1]));\n+\n+\t\tif (midx_compaction_step_exec(step, opts, base) < 0) {\n+\t\t\tret = error(_(\"unable to execute compaction step %\"PRIuMAX),\n+\t\t\t\t (uintmax_t)i);\n+\t\t\tfree(base);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfree(base);\n+\t}\n+\n+\ti = steps_nr;\n+\twhile (i--) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tif (!step->csum)\n+\t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n+\t\t\t (uintmax_t)i);\n+\t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t}\n+\n+\tcommit_lock_file(&lf);\n+\n+done:\n+\tstrbuf_release(&lock_name);\n+\tfor (i = 0; i < steps_nr; i++)\n+\t\tmidx_compaction_step_release(&steps[i]);\n+\tfree(steps);\n+\treturn ret;\n+}\n+\n int repack_write_midx(struct repack_write_midx_opts *opts)\n {\n \tswitch (opts->mode) {\n@@ -386,6 +958,8 @@ int repack_write_midx(struct repack_write_midx_opts *opts)\n \t\tBUG(\"write_midx mode is NONE?\");\n \tcase REPACK_WRITE_MIDX_DEFAULT:\n \t\treturn write_midx_included_packs(opts);\n+\tcase REPACK_WRITE_MIDX_INCREMENTAL:\n+\t\treturn write_midx_incremental(opts);\n \tdefault:\n \t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n \t}\ndiff --git a/repack.h b/repack.h\nindex 81907fcce7f..831ccfb1c6c 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -137,6 +137,7 @@ struct tempfile;\n enum repack_write_midx_mode {\n \tREPACK_WRITE_MIDX_NONE,\n \tREPACK_WRITE_MIDX_DEFAULT,\n+\tREPACK_WRITE_MIDX_INCREMENTAL,\n };\n \n struct repack_write_midx_opts {\n@@ -148,6 +149,8 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n \tenum repack_write_midx_mode mode;\n };\n \n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540351","messageId":"e8e22d7958d513c3b7d4165857392a19277df65e.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 15/16] repack: introduce `--write-midx=incremental`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:29Z","receivedAt":"2026-03-29T21:41:31Z","isPatch":true,"body":"Expose the incremental MIDX repacking mode (implemented in an earlier\ncommit) via a new --write-midx=incremental option for `git repack`.\n\nAdd \"incremental\" as a recognized argument to the --write-midx\nOPT_CALLBACK, mapping it to REPACK_WRITE_MIDX_INCREMENTAL. When this\nmode is active and --geometric is in use, set the midx_layer_threshold\non the pack geometry so that only packs in sufficiently large tip layers\nare considered for repacking.\n\nTwo new configuration options control the compaction behavior:\n\n - repack.midxSplitFactor (default: 2): the factor used in the\n geometric merging condition for MIDX layers.\n\n - repack.midxNewLayerThreshold (default: 8): the minimum number of\n packs in the tip MIDX layer before its packs are considered as\n candidates for geometric repacking.\n\nAdd tests exercising the new mode across a variety of scenarios\nincluding basic geometric violations, multi-round chain integrity,\nbranching and merging histories, cross-layer object uniqueness, and\nthreshold-based compaction.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/repack.adoc | 18 ++\n Documentation/git-repack.adoc | 39 ++-\n builtin/repack.c | 38 ++-\n midx.c | 31 ++\n midx.h | 3 +\n repack-geometry.c | 13 +-\n repack-midx.c | 5 +\n repack.c | 21 +-\n repack.h | 9 +-\n t/meson.build | 1 +\n t/t7705-repack-incremental-midx.sh | 436 +++++++++++++++++++++++++++++\n 11 files changed, 593 insertions(+), 21 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex e9e78dcb198..054de9f8795 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -46,3 +46,21 @@ repack.midxMustContainCruft::\n \t`--write-midx`. When false, cruft packs are only included in the MIDX\n \twhen necessary (e.g., because they might be required to form a\n \treachability closure with MIDX bitmaps). Defaults to true.\n+\n+repack.midxSplitFactor::\n+\tThe factor used in the geometric merging condition when\n+\tcompacting incremental MIDX layers during `git repack` when\n+\tinvoked with the `--write-midx=incremental` option.\n++\n+Adjacent layers are merged when the accumulated object count of the\n+newer layer exceeds `1/` of the object count of the next deeper\n+layer. Defaults to 2.\n+\n+repack.midxNewLayerThreshold::\n+\tThe minimum number of packs in the tip MIDX layer before those\n+\tpacks are considered as candidates for geometric repacking\n+\tduring `git repack --write-midx=incremental`.\n++\n+When the tip layer has fewer packs than this threshold, those packs are\n+excluded from the geometric repack entirely, and are thus left\n+unmodified. Defaults to 8.\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 673ce910837..27a99cc46f4 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -11,7 +11,7 @@ SYNOPSIS\n [verse]\n 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n \t[--window=] [--depth=] [--threads=] [--keep-pack=]\n-\t[--write-midx] [--name-hash-version=] [--path-walk]\n+\t[--write-midx[=]] [--name-hash-version=] [--path-walk]\n \n DESCRIPTION\n -----------\n@@ -250,9 +250,42 @@ pack as the preferred pack for object selection by the MIDX (see\n linkgit:git-multi-pack-index[1]).\n \n -m::\n---write-midx::\n+--write-midx[=]::\n \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n-\tcontaining the non-redundant packs.\n+\tcontaining the non-redundant packs. The following modes are\n+\tavailable:\n++\n+--\n+\t`default`;;\n+\t\tWrite a single MIDX covering all packs. This is the\n+\t\tdefault when `--write-midx` is given without an\n+\t\texplicit mode.\n+\n+\t`incremental`;;\n+\t\tWrite an incremental MIDX chain instead of a single\n+\t\tflat MIDX. This mode requires `--geometric`.\n++\n+The incremental mode maintains a chain of MIDX layers that is compacted\n+over time using a geometric merging strategy. Each repack creates a new\n+tip layer containing the newly written pack(s). Adjacent layers are then\n+merged whenever the newer layer's object count exceeds\n+`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n+that do not meet this condition are retained as-is.\n++\n+The result is that newer (tip) layers tend to contain many small packs\n+with relatively few objects, while older (deeper) layers contain fewer,\n+larger packs covering more objects. Because compaction is driven by the\n+tip of the chain, newer layers are also rewritten more frequently than\n+older ones, which are only touched when enough objects have accumulated\n+to justify merging into them. This keeps the total number of layers\n+logarithmic relative to the total number of objects.\n++\n+Only packs in the tip MIDX layer are considered as candidates for the\n+geometric repack; packs in deeper layers are left untouched. If the tip\n+layer contains fewer packs than `repack.midxNewLayerThreshold`, those\n+packs are excluded from the geometry entirely, and a new layer is\n+created for any new pack(s) without disturbing the existing chain.\n+--\n \n --name-hash-version=::\n \tProvide this argument to the underlying `git pack-objects` process.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 3a5042491d6..9e070f35868 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -33,7 +33,7 @@ static int midx_must_contain_cruft = 1;\n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n \t \"[--window=] [--depth=] [--threads=] [--keep-pack=]\\n\"\n-\t \"[--write-midx] [--name-hash-version=] [--path-walk]\"),\n+\t \"[--write-midx[=]] [--name-hash-version=] [--path-walk]\"),\n \tNULL\n };\n \n@@ -42,9 +42,14 @@ static const char incremental_bitmap_conflict_error[] = N_(\n \"--no-write-bitmap-index or disable the pack.writeBitmaps configuration.\"\n );\n \n+#define DEFAULT_MIDX_SPLIT_FACTOR 2\n+#define DEFAULT_MIDX_NEW_LAYER_THRESHOLD 8\n+\n struct repack_config_ctx {\n \tstruct pack_objects_args *po_args;\n \tstruct pack_objects_args *cruft_po_args;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -94,6 +99,16 @@ static int repack_config(const char *var, const char *value,\n \t\tmidx_must_contain_cruft = git_config_bool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"repack.midxsplitfactor\")) {\n+\t\trepack_ctx->midx_split_factor = git_config_int(var, value,\n+\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(var, \"repack.midxnewlayerthreshold\")) {\n+\t\trepack_ctx->midx_new_layer_threshold = git_config_int(var, value,\n+\t\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -109,6 +124,8 @@ static int option_parse_write_midx(const struct option *opt, const char *arg,\n \n \tif (!arg || !*arg)\n \t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse if (!strcmp(arg, \"incremental\"))\n+\t\t*cfg = REPACK_WRITE_MIDX_INCREMENTAL;\n \telse\n \t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n \n@@ -223,6 +240,8 @@ int cmd_repack(int argc,\n \tmemset(&config_ctx, 0, sizeof(config_ctx));\n \tconfig_ctx.po_args = &po_args;\n \tconfig_ctx.cruft_po_args = &cruft_po_args;\n+\tconfig_ctx.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR;\n+\tconfig_ctx.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD;\n \n \trepo_config(repo, repack_config, &config_ctx);\n \n@@ -244,6 +263,9 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n+\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n+\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n+\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n@@ -293,6 +315,10 @@ int cmd_repack(int argc,\n \tif (geometry.split_factor) {\n \t\tif (pack_everything)\n \t\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--geometric\", \"-A/-a\");\n+\t\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL) {\n+\t\t\tgeometry.midx_layer_threshold = config_ctx.midx_new_layer_threshold;\n+\t\t\tgeometry.midx_layer_threshold_set = true;\n+\t\t}\n \t\tpack_geometry_init(&geometry, &existing, &po_args);\n \t\tpack_geometry_split(&geometry);\n \t}\n@@ -540,6 +566,8 @@ int cmd_repack(int argc,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.midx_split_factor = config_ctx.midx_split_factor,\n+\t\t\t.midx_new_layer_threshold = config_ctx.midx_new_layer_threshold,\n \t\t\t.mode = write_midx,\n \t\t};\n \n@@ -552,11 +580,15 @@ int cmd_repack(int argc,\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\n-\t\texisting_packs_remove_redundant(&existing, packdir);\n+\t\tbool wrote_incremental_midx = write_midx == REPACK_WRITE_MIDX_INCREMENTAL;\n+\n+\t\texisting_packs_remove_redundant(&existing, packdir,\n+\t\t\t\t\t\twrote_incremental_midx);\n \n \t\tif (geometry.split_factor)\n \t\t\tpack_geometry_remove_redundant(&geometry, &names,\n-\t\t\t\t\t\t &existing, packdir);\n+\t\t\t\t\t\t &existing, packdir,\n+\t\t\t\t\t\t wrote_incremental_midx);\n \t\tif (show_progress)\n \t\t\topts |= PRUNE_PACKED_VERBOSE;\n \t\tprune_packed_objects(opts);\ndiff --git a/midx.c b/midx.c\nindex 7d23338aa3a..fd9e72a3941 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -852,6 +852,37 @@ void clear_midx_file(struct repository *r)\n \tstrbuf_release(&midx);\n }\n \n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes)\n+{\n+\tstruct strbuf chain = STRBUF_INIT;\n+\n+\tget_midx_chain_filename(r->objects->sources, &chain);\n+\n+\tif (r->objects) {\n+\t\tstruct odb_source *source = r->objects->sources;\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\tif (files->packed->midx)\n+\t\t\t\tclose_midx(files->packed->midx);\n+\t\t\tfiles->packed->midx = NULL;\n+\t\t}\n+\t}\n+\n+\tif (!keep_hashes && remove_path(chain.buf))\n+\t\tdie(_(\"failed to clear multi-pack-index chain at %s\"),\n+\t\t chain.buf);\n+\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_BITMAP,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_REV,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_MIDX,\n+\t\t\t\t\t keep_hashes);\n+\n+\tstrbuf_release(&chain);\n+}\n+\n static int verify_midx_error;\n \n __attribute__((format (printf, 1, 2)))\ndiff --git a/midx.h b/midx.h\nindex f211a38b9e7..b45da0a3144 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -9,6 +9,7 @@ struct repository;\n struct bitmapped_pack;\n struct git_hash_algo;\n struct odb_source;\n+struct strvec;\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION_V1 1\n@@ -143,6 +144,8 @@ int write_midx_file_compact(struct odb_source *source,\n \t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes);\n int verify_midx_file(struct odb_source *source, unsigned flags);\n int expire_midx_packs(struct odb_source *source, unsigned flags);\n int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags);\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex d2065205f87..5b554da89f1 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -251,7 +251,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t\t\t uint32_t pack_nr,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tconst struct git_hash_algo *algop = existing->repo->hash_algo;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -271,7 +272,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t (string_list_has_string(&existing->kept_packs, buf.buf)))\n \t\t\tcontinue;\n \n-\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf);\n+\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n \n \tstrbuf_release(&buf);\n@@ -280,12 +282,13 @@ static void remove_redundant_packs(struct packed_git **pack,\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs(geometry->pack, geometry->split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n \tremove_redundant_packs(geometry->promisor_pack, geometry->promisor_split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n }\n \n void pack_geometry_release(struct pack_geometry *geometry)\ndiff --git a/repack-midx.c b/repack-midx.c\nindex a159b879775..5bf5df03d5d 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -894,6 +894,7 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \tstruct midx_compaction_step *steps = NULL;\n \tstruct strbuf lock_name = STRBUF_INIT;\n \tstruct lock_file lf;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tsize_t steps_nr = 0;\n \tsize_t i;\n \tint ret = 0;\n@@ -939,11 +940,15 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n \t\t\t (uintmax_t)i);\n \t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t\tstrvec_push(&keep_hashes, step->csum);\n \t}\n \n \tcommit_lock_file(&lf);\n \n+\tclear_incremental_midx_files(opts->existing->repo, &keep_hashes);\n+\n done:\n+\tstrvec_clear(&keep_hashes);\n \tstrbuf_release(&lock_name);\n \tfor (i = 0; i < steps_nr; i++)\n \t\tmidx_compaction_step_release(&steps[i]);\ndiff --git a/repack.c b/repack.c\nindex 2ee6b51420a..be2cc0e9d44 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -55,14 +55,18 @@ void pack_objects_args_release(struct pack_objects_args *args)\n }\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name)\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct odb_source *source = repo->objects->sources;\n \tstruct multi_pack_index *m = get_multi_pack_index(source);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && source->local && midx_contains_pack(m, buf.buf))\n+\tif (m && source->local && midx_contains_pack(m, buf.buf)) {\n \t\tclear_midx_file(repo);\n+\t\tif (!wrote_incremental_midx)\n+\t\t\tclear_incremental_midx_files(repo, NULL);\n+\t}\n \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n \tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n@@ -252,23 +256,26 @@ void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \n static void remove_redundant_packs_1(struct repository *repo,\n \t\t\t\t struct string_list *packs,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct string_list_item *item;\n \tfor_each_string_list_item(item, packs) {\n \t\tif (!existing_pack_is_marked_for_deletion(item))\n \t\t\tcontinue;\n-\t\trepack_remove_redundant_pack(repo, packdir, item->string);\n+\t\trepack_remove_redundant_pack(repo, packdir, item->string,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n }\n \n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs_1(existing->repo, &existing->non_kept_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n \tremove_redundant_packs_1(existing->repo, &existing->cruft_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n }\n \n void existing_packs_release(struct existing_packs *existing)\ndiff --git a/repack.h b/repack.h\nindex 831ccfb1c6c..d2876f569a6 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -34,7 +34,8 @@ void prepare_pack_objects(struct child_process *cmd,\n void pack_objects_args_release(struct pack_objects_args *args);\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name);\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx);\n \n struct write_pack_opts {\n \tstruct pack_objects_args *po_args;\n@@ -84,7 +85,8 @@ void existing_packs_retain_cruft(struct existing_packs *existing,\n void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \t\t\t\t struct string_list *names);\n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void existing_packs_release(struct existing_packs *existing);\n \n struct generated_pack;\n@@ -129,7 +131,8 @@ struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry);\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\ndiff --git a/t/meson.build b/t/meson.build\nindex 919a0ae4cbb..3fc38d123b1 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -950,6 +950,7 @@ integration_tests = [\n 't7702-repack-cyclic-alternate.sh',\n 't7703-repack-geometric.sh',\n 't7704-repack-cruft.sh',\n+ 't7705-repack-incremental-midx.sh',\n 't7800-difftool.sh',\n 't7810-grep.sh',\n 't7811-grep-open.sh',\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nnew file mode 100755\nindex 00000000000..f81c2c67060\n--- /dev/null\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -0,0 +1,436 @@\n+#!/bin/sh\n+\n+test_description='git repack --write-midx=incremental'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+midxdir=$packdir/multi-pack-index.d\n+midx_chain=$midxdir/multi-pack-index-chain\n+\n+# incrementally_repack N\n+#\n+# Make \"N\" new commits, each stored in their own pack, and then repacked\n+# with the --write-midx=incremental strategy.\n+incrementally_repack () {\n+\tfor i in $(test_seq 1 \"$1\")\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify || return 1\n+\tdone\n+}\n+\n+# Create packs with geometrically increasing sizes so that they\n+# satisfy the geometric progression and survive a --geometric=2\n+# repack without being rolled up. Creates 3 packs containing 1,\n+# 2, and 6 commits (3, 6, and 18 objects) respectively.\n+create_geometric_packs () {\n+\ttest_commit \"small\" &&\n+\tgit repack -d &&\n+\n+\ttest_commit_bulk --message=\"medium\" 2 &&\n+\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t--write-bitmap-index\n+}\n+\n+# create_layer \n+#\n+# Creates a new MIDX layer with the contents of \"test_commit_bulk $@\".\n+create_layer () {\n+\ttest_commit_bulk \"$@\" &&\n+\n+\tgit multi-pack-index write --incremental --bitmap\n+}\n+\n+# create_layers\n+#\n+# Reads lines of \" \" from stdin and creates a new MIDX\n+# layer for each line. See create_layer above for more.\n+create_layers () {\n+\twhile read msg nr\n+\tdo\n+\t\tcreate_layer --message=\"$msg\" \"$nr\" || return 1\n+\tdone\n+}\n+\n+test_expect_success '--write-midx=incremental requires --geometric' '\n+\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+\n+\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+'\n+\n+test_expect_success 'below layer threshold, tip packs excluded' '\n+\tgit init below-layer-threshold-tip-packs-excluded &&\n+\t(\n+\t\tcd below-layer-threshold-tip-packs-excluded &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 4 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create 3 packs forming a geometric progression by\n+\t\t# object count such that they are unmodified by the\n+\t\t# initial repack. The MIDX chain thusly contains a\n+\t\t# single layer with three packs.\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a new commit. Since the layer threshold is\n+\t\t# unmet, a new MIDX layer is added on top of the\n+\t\t# existing one.\n+\t\ttest_commit extra &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs repacked' '\n+\tgit init above-layer-threshold-tip-packs-repacked &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-repacked &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Same setup, but with the layer threshold set to 2.\n+\t\t# Since the tip MIDX layer meets that threshold, its\n+\t\t# packs are considered repack candidates.\n+\t\tcreate_geometric_packs &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Perturb the existing progression such that it is\n+\t\t# rolled up into a single new pack, invalidating the\n+\t\t# existing MIDX layer and replacing it with a new one.\n+\t\ttest_commit extra &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip layer preserved' '\n+\tgit init above-layer-threshold-tip-layer-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-layer-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\ttest_commit_bulk --message=\"medium\" 2 &&\n+\t\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Create objects to form a pack satisfying the geometric\n+\t\t# progression (thus preserving the tip layer), but not\n+\t\t# so large that it meets the layer merging condition.\n+\t\ttest_commit_bulk --message=\"small\" 1 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\ttest_line_count = 1 packs.new &&\n+\t\ttest_line_count = 3 packs.after &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs preserved' '\n+\tgit init above-layer-threshold-tip-packs-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Same setup as above, but this time the new objects do\n+\t\t# not satisfy the new layer merging condition, resulting\n+\t\t# in a new tip layer.\n+\t\ttest_commit_bulk --message=\"huge\" 18 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'new tip absorbs multiple layers' '\n+\tgit init new-tip-absorbs-multiple-layers &&\n+\t(\n+\t\tcd new-tip-absorbs-multiple-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a 4-layer chain where each layer is too small to\n+\t\t# absorb the one below it. The sizes must satisfy L(n) <\n+\t\t# L(n-1)/2 for each adjacent pair:\n+\t\t#\n+\t\t# L0 (oldest): 75 obj (25 commits)\n+\t\t# L1: 21 obj (7 commits, 21 < 75/2)\n+\t\t# L2: 9 obj (3 commits, 9 < 21/2)\n+\t\t# L3 (tip): 3 obj (1 commit, 3 < 9/2)\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tL0 25\n+\t\tL1 7\n+\t\tL2 3\n+\t\tL3 1\n+\t\tEOF\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Now add a new commit. The merging condition is\n+\t\t# satisfied between L3-L1, but violated at L0, which is\n+\t\t# too large relative to the accumulated size.\n+\t\t#\n+\t\t# As a result, the chain shrinks from 4 to 2 layers.\n+\t\ttest_commit new &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'compaction of older layers' '\n+\tgit init compaction-of-older-layers &&\n+\t(\n+\t\tcd compaction-of-older-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a chain with two small layers at the bottom\n+\t\t# and a larger barrier layer on top, producing a\n+\t\t# chain that violates the compaction invariant, since\n+\t\t# the two small layers would normally have been merged.\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 2\n+\t\ttwo 4\n+\t\tbarrier 54\n+\t\tEOF\n+\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running an incremental repack compacts the two\n+\t\t# small layers at the bottom of the chain as a\n+\t\t# separate step in the compaction plan.\n+\t\ttest_commit another &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'geometric rollup with surviving tip packs' '\n+\tgit init geometric-rollup-with-surviving-tip-packs &&\n+\t(\n+\t\tcd geometric-rollup-with-surviving-tip-packs &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create a pack large enough to anchor the geometric\n+\t\t# progression when small packs are added alongside it.\n+\t\tcreate_layer --message=\"big\" 5 &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a small number of objects such that the\n+\t\t# progression is unbothered. Note that the existing pack\n+\t\t# is considered a repack candidate as the new layer\n+\t\t# threshold is set to 1.\n+\t\ttest_commit small-1 &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before\n+\t)\n+'\n+\n+test_expect_success 'kept packs are excluded from repack' '\n+\tgit init kept-packs-excluded-from-repack &&\n+\t(\n+\t\tcd kept-packs-excluded-from-repack &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create two equal-sized packs, marking one as kept.\n+\t\tfor i in A B\n+\t\tdo\n+\t\t\ttest_commit \"$i\" && git repack -d || return 1\n+\t\tdone &&\n+\n+\t\tkeep=$(ls $packdir/pack-*.idx | head -n 1) &&\n+\t\ttouch \"${keep%.idx}.keep\" &&\n+\n+\t\t# The kept pack is excluded as a repacking candidate\n+\t\t# entirely, so no rollup occurs as there is only one\n+\t\t# non-kept pack. A new MIDX layer is written containing\n+\t\t# that pack.\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count = 1 actual.packs &&\n+\t\ttest_grep ! \"$keep\" actual.packs &&\n+\n+\t\tgit multi-pack-index verify &&\n+\n+\t\t# All objects (from both kept and non-kept packs)\n+\t\t# must still be accessible.\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'incremental MIDX with --max-pack-size' '\n+\tgit init incremental-midx-with--max-pack-size &&\n+\t(\n+\t\tcd incremental-midx-with--max-pack-size &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\t# Now add enough data that a small --max-pack-size will\n+\t\t# cause pack-objects to split its output. Create objects\n+\t\t# large enough to fill multiple packs.\n+\t\ttest-tool genrandom foo 1M >big1 &&\n+\t\ttest-tool genrandom bar 1M >big2 &&\n+\t\tgit add big1 big2 &&\n+\t\ttest_tick &&\n+\t\tgit commit -a -m \"big blobs\" &&\n+\t\tgit repack -d &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index --max-pack-size=1M &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count -gt 1 actual.packs &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'noop repack preserves valid MIDX chain' '\n+\tgit init noop-repack-preserves-valid-midx-chain &&\n+\t(\n+\t\tcd noop-repack-preserves-valid-midx-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running again with no new objects should not break\n+\t\t# the MIDX chain. It produces \"Nothing new to pack.\"\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_cmp $midx_chain.before $midx_chain &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'repack -ad removes stale incremental chain' '\n+\tgit init repack--ad-removes-stale-incremental-chain &&\n+\t(\n+\t\tcd repack--ad-removes-stale-incremental-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 1\n+\t\ttwo 1\n+\t\tEOF\n+\n+\t\ttest_path_is_file $midx_chain &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tgit repack -ad &&\n+\n+\t\ttest_path_is_missing $packdir/multi-pack-index &&\n+\t\ttest_dir_is_empty $midxdir\n+\t)\n+'\n+\n+test_done\n-- \n2.53.0.729.g817728289e1.dirty\n\n"},{"id":"540352","messageId":"79d5a12a3902fc58f4ff824403603e252b3cf62a.1774820449.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH 16/16] repack: allow `--write-midx=incremental` without `--geometric`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-29T21:41:31Z","receivedAt":"2026-03-29T21:41:33Z","isPatch":true,"body":"Previously, `--write-midx=incremental` required `--geometric` and would\ndie() without it. Relax this restriction so that incremental MIDX\nrepacking can be used independently.\n\nWithout `--geometric`, the behavior is append-only: a single new MIDX\nlayer is created containing whatever packs were written by the repack\nand appended to the existing chain (or a new chain is started). Existing\nlayers are preserved as-is with no compaction or merging.\n\nImplement this via a new repack_make_midx_append_plan() that builds a\nplan consisting of a WRITE step for the freshly written packs followed\nby COPY steps for every existing MIDX layer. The existing compaction\nplan (repack_make_midx_compaction_plan) is used only when `--geometric`\nis active.\n\nUpdate the documentation to describe the behavior with and without\n`--geometric`, and replace the test that enforced the old restriction\nwith one exercising append-only incremental MIDX repacking.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-repack.adoc | 19 +++++----\n builtin/repack.c | 3 --\n repack-midx.c | 64 ++++++++++++++++++++++++++++--\n t/t7705-repack-incremental-midx.sh | 35 +++++++++++++---\n 4 files changed, 103 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 27a99cc46f4..72c42015e23 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -263,14 +263,19 @@ linkgit:git-multi-pack-index[1]).\n \n \t`incremental`;;\n \t\tWrite an incremental MIDX chain instead of a single\n-\t\tflat MIDX. This mode requires `--geometric`.\n+\t\tflat MIDX.\n +\n-The incremental mode maintains a chain of MIDX layers that is compacted\n-over time using a geometric merging strategy. Each repack creates a new\n-tip layer containing the newly written pack(s). Adjacent layers are then\n-merged whenever the newer layer's object count exceeds\n-`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n-that do not meet this condition are retained as-is.\n+Without `--geometric`, a new MIDX layer is appended to the existing\n+chain (or a new chain is started) containing whatever packs were written\n+by the repack. Existing layers are preserved as-is.\n++\n+When combined with `--geometric`, the incremental mode maintains a chain\n+of MIDX layers that is compacted over time using a geometric merging\n+strategy. Each repack creates a new tip layer containing the newly\n+written pack(s). Adjacent layers are then merged whenever the newer\n+layer's object count exceeds `1/repack.midxSplitFactor` of the next\n+deeper layer's count. Layers that do not meet this condition are\n+retained as-is.\n +\n The result is that newer (tip) layers tend to contain many small packs\n with relatively few objects, while older (deeper) layers contain fewer,\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 9e070f35868..8e187322fe4 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -263,9 +263,6 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n-\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n-\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n-\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 5bf5df03d5d..055aa4c1356 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -555,6 +555,60 @@ static void midx_compaction_step_release(struct midx_compaction_step *step)\n \tfree(step->csum);\n }\n \n+/*\n+ * Build an append-only MIDX plan: a single WRITE step for the freshly\n+ * written packs, plus COPY steps for every existing layer. No\n+ * compaction or merging is performed.\n+ */\n+static void repack_make_midx_append_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step *step;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tif (opts->names->nr) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tuint32_t i;\n+\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_WRITE;\n+\t\tstring_list_init_dup(&step->u.write);\n+\n+\t\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"pack-%s.idx\",\n+\t\t\t\t opts->names->items[i].string);\n+\t\t\tstring_list_append(&step->u.write, buf.buf);\n+\t\t}\n+\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\tfor (; m; m = m->base_midx) {\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_COPY;\n+\t\tstep->u.copy = m;\n+\t\tstep->objects_nr = m->num_objects;\n+\t}\n+\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+}\n+\n static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\t\t\t struct midx_compaction_step **steps_p,\n \t\t\t\t\t size_t *steps_nr_p)\n@@ -911,9 +965,13 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \t}\n \n-\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n-\t\tret = error(_(\"unable to generate compaction plan\"));\n-\t\tgoto done;\n+\tif (opts->geometry->split_factor) {\n+\t\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\t\tgoto done;\n+\t\t}\n+\t} else {\n+\t\trepack_make_midx_append_plan(opts, &steps, &steps_nr);\n \t}\n \n \tfor (i = 0; i < steps_nr; i++) {\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nindex f81c2c67060..562554e69b4 100755\n--- a/t/t7705-repack-incremental-midx.sh\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -63,10 +63,36 @@ create_layers () {\n \tdone\n }\n \n-test_expect_success '--write-midx=incremental requires --geometric' '\n-\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+test_expect_success '--write-midx=incremental without --geometric' '\n+\tgit init incremental-without-geometric &&\n+\t(\n+\t\tcd incremental-without-geometric &&\n \n-\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+\t\tgit config maintenance.auto false &&\n+\n+\t\ttest_commit first &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_commit second &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# A second repack appends a new layer without\n+\t\t# disturbing the existing one.\n+\t\ttest_commit third &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\t\thead -n 1 $midx_chain.before >expect &&\n+\t\thead -n 1 $midx_chain >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit fsck\n+\t)\n '\n \n test_expect_success 'below layer threshold, tip packs excluded' '\n@@ -334,8 +360,7 @@ test_expect_success 'kept packs are excluded from repack' '\n \t\t# entirely, so no rollup occurs as there is only one\n \t\t# non-kept pack. A new MIDX layer is written containing\n \t\t# that pack.\n-\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n-\t\t\t--write-bitmap-index &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental &&\n \n \t\ttest-tool read-midx $objdir >actual &&\n \t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n-- \n2.53.0.729.g817728289e1.dirty\n"},{"id":"540435","messageId":"20260330223346.GA41843@coredump.intra.peff.net","threadId":"65383","inReplyTo":"0c7a68ca5a777cce9c0c951643ea2220b1b56a8c.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 01/16] midx-write: handle noop writes when converting incremental chains","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-03-30T22:33:46Z","receivedAt":"2026-03-30T22:33:54Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:40:53PM -0400, Taylor Blau wrote:\n\n> This comparison fails when we have an incremental MIDX chain with\n> at least two layers, since we do not recursively peel through earlier\n> layers, instead treating the `->pack_names` array of the tip MIDX layer\n> as containing all `m->num_packs + m->num_packs_in_base` packs.\n> \n> Adjust this to instead look through the MIDX layers one by one when\n> comparing pack names. While we're at it, fix a typo above in the same\n> function.\n\nMakes sense.\n\n> @@ -1208,14 +1208,16 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n> \t\t\tBUG(\"same pack added twice?\");\n> \t}\n> \n> -\tfor (uint32_t i = 0; i < ctx->nr; i++) {\n> -\t\tstrbuf_reset(&buf);\n> -\t\tstrbuf_addstr(&buf, midx->pack_names[i]);\n> -\t\tstrbuf_strip_suffix(&buf, \".idx\");\n> +\tfor (struct multi_pack_index *m = midx; m; m = m->base_midx) {\n> +\t\tfor (uint32_t i = 0; i < m->num_packs; i++) {\n> +\t\t\tstrbuf_reset(&buf);\n> +\t\t\tstrbuf_addstr(&buf, m->pack_names[i]);\n> +\t\t\tstrbuf_strip_suffix(&buf, \".idx\");\n> \n> -\t\tif (!strset_contains(&packs, buf.buf))\n> -\t\t\tgoto out;\n> -\t\tstrset_remove(&packs, buf.buf);\n> +\t\t\tif (!strset_contains(&packs, buf.buf))\n> +\t\t\t\tgoto out;\n> +\t\t\tstrset_remove(&packs, buf.buf);\n> +\t\t}\n\nI wondered here if we would also need to adjust the check of the count,\nbut it is already OK. Just above the context is:\n\n if (ctx->nr != midx->num_packs + midx->num_packs_in_base)\n goto out;\n\nwhich handles the layered midx. Since we were counting up to ctx->nr\nbefore, which includes packs in base midx layers, I guess these old\nlines:\n\n> -\tfor (uint32_t i = 0; i < ctx->nr; i++) {\n> [...]\n> -\t\tstrbuf_addstr(&buf, midx->pack_names[i]);\n\nwas probably reading garbage off the end of the array.\n\nNow we iterate through each layer individually. We assume that the total\ncount matches ctx->nr, otherwise the midx struct is totally corrupt. We\ncould count them and double-check, I guess, but I don't think there's\nmuch point (it would be a BUG() if they were mismatched).\n\nSo looks good to me.\n\n-Peff\n"},{"id":"540436","messageId":"20260330223849.GB41843@coredump.intra.peff.net","threadId":"65383","inReplyTo":"1391552dfc6624374bed89b25d1892d9b9d080f9.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 02/16] midx: use `string_list` for retained MIDX files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-03-30T22:38:49Z","receivedAt":"2026-03-30T22:38:54Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:40:55PM -0400, Taylor Blau wrote:\n\n> @@ -767,15 +766,12 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n> \t\t\t\tconst char *file_name, void *_data)\n> {\n> \tstruct clear_midx_data *data = _data;\n> -\tuint32_t i;\n> \n> \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n> \t ends_with(file_name, data->ext)))\n> \t\treturn;\n> -\tfor (i = 0; i < data->keep_nr; i++) {\n> -\t\tif (!strcmp(data->keep[i], file_name))\n> -\t\t\treturn;\n> -\t}\n> +\tif (string_list_has_string(&data->keep, file_name))\n> +\t\treturn;\n\nWe are using has_string() here, so &data->keep will need to be sorted\n(just keeping it in mind as we read further).\n\n> \tif (keep_hash) {\n> -\t\tALLOC_ARRAY(data.keep, 1);\n> +\t\tstruct strbuf buf = STRBUF_INIT;\n> +\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hash, ext);\n> \n> -\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n> -\t\tdata.keep_nr = 1;\n> +\t\tstring_list_insert(&data.keep, buf.buf);\n\nOK, we use insert() here. But there is no difference between insert and\nappend, since the list only has one element. It is sorted either way. ;)\n\nAnd we do not have to worry about being accidentally-quadratic because\nthere is no loop.\n\n> -\tALLOC_ARRAY(data.keep, hashes_nr);\n> \tfor (i = 0; i < hashes_nr; i++)\n> -\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n> -\t\t\t\t ext);\n> -\tdata.keep_nr = hashes_nr;\n> -\tdata.ext = ext;\n> +\t\tstring_list_append(&data.keep,\n> +\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n> +\t\t\t\t\t keep_hashes[i], ext));\n> +\tstring_list_sort(&data.keep);\n\nOK, and here we really do have a list, but we do append+sort to avoid\nbeing quadratic. Good.\n\nSince AFAICT we only care about set inclusion, I probably would have\nused a strset instead, but for the numbers we are talking about it is\nunlikely to matter much either way.\n\nSo looking good.\n\n-Peff\n"},{"id":"540437","messageId":"20260330224631.GC41843@coredump.intra.peff.net","threadId":"65383","inReplyTo":"aea6aad695336a974df9c90ce7bd02ddcc33582f.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 03/16] strvec: introduce `strvec_init_alloc()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-03-30T22:46:31Z","receivedAt":"2026-03-30T22:46:32Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:40:58PM -0400, Taylor Blau wrote:\n\n> When the caller knows upfront how many elements will be pushed onto a\n> `strvec`, it is useful to pre-allocate enough space in the array to fit\n> that many elements (and one additional slot to store NULL, indicating\n> the end of the list.)\n\nThe main reason I didn't add an \"alloc\" parameter to the initializer\noriginally is that if you know how many elements you are going to push,\nyou probably don't need a strvec in the first place (you can just make\nan array). But let's wait and see how it gets used in a future patch.\n\n> +void strvec_init_alloc(struct strvec *array, size_t alloc)\n> +{\n> +\tif (!alloc) {\n> +\t\tstruct strvec blank = STRVEC_INIT;\n> +\t\tmemcpy(array, &blank, sizeof(*array));\n> +\t} else {\n> +\t\tCALLOC_ARRAY(array->v, st_add(alloc, 1));\n> +\t\tarray->nr = 0;\n> +\t\tarray->alloc = alloc + 1;\n> +\t}\n> }\n\nWe use the blank+memcpy approach so that we don't duplicate knowledge of\nwhich elements are in the struct (and which might need to be\ninitialized). But now there's an implicit zero-ing happening when alloc\nis non-zero. Can we write this as:\n\n struct strvec blank = STRVEC_INIT;\n memcpy(array, &blank, sizeof(*array));\n if (alloc) {\n\tCALLOC_ARRAY(array->v, st_add(alloc, 1));\n\tarray->alloc = alloc + 1;\n }\n\nThe worst case is a useless write of a few struct elements, but I\nsuspect that an optimizing compiler can even get rid of those.\n\nI doubt we are going to add new elements to a strvec, so this is\narguably just pedantry. It just feels like it's easy enough to do now\nwhile we're thinking about it.\n\n As an aside, it is sad that we have to CALLOC() ourselves here, as\n there is no reusable \"grow\" operation. Probably not worth refactoring\n it out, though, though.\n\n-Peff\n"},{"id":"540441","messageId":"20260330230130.GD41843@coredump.intra.peff.net","threadId":"65383","inReplyTo":"5fc72d5049a602ae5ede6bb243f44546f02d995d.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-03-30T23:01:30Z","receivedAt":"2026-03-30T23:01:32Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:41:00PM -0400, Taylor Blau wrote:\n\n> -\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n> +\tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n> \n> \tif (ctx.incremental) {\n> \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n> @@ -1760,39 +1760,45 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n> \t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n> \n> -\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n> +\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n> +\t\t\t\tkeep_hashes.nr++;\n\nGross, we are just manipulating the innards of the strvec ourselves?\n\nIs it really worth doing this (and adding init_alloc()) versus just:\n\n strvec_init(&keep_hashes);\n for (...)\n\tstrvec_push(midx_get_checksum_hex(m));\n\n? That's amortized linear-time, and it's not like the number of midx\nlayers is going to be large anyway.\n\n> \t\t\t\tm = m->base_midx;\n> \t\t\t}\n> \n> -\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n> +\t\t\tkeep_hashes.v[i] = xstrdup(hash_to_hex_algop(midx_hash,\n> \t\t\t\t\t\t\t\t r->hash_algo));\n> +\t\t\tkeep_hashes.nr++;\n\nOK, this could be a push, too.\n\n> \n> \t\t\ti = 0;\n> \t\t\tfor (m = ctx.m;\n> \t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n> \t\t\t m = m->base_midx) {\n> -\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n> +\t\t\t\tkeep_hashes.v[keep_hashes_nr - i - 1] =\n> \t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n> +\t\t\t\tkeep_hashes.nr++;\n\nBut what is this? We're filling in from the back side of the array? I\nmean...yeah, that's something that strvec_push() can't do. But I can't\nhelp but feel like it might be simpler and more obvious to adjust the\niteration to build the array in order.\n\nI dunno. Maybe that is hard to do. But if so, I question whether moving\nto a strvec is worth it here, since we are not treating it as an opaque\ntype anymore. And it is not buying us much to use it (we get to pass one\nparameter versus two, though that is easily solved with a struct, and we\nget to use _clear() instead of our own free loop).\n\n> void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n> -\t\t\t\t char **keep_hashes,\n> -\t\t\t\t uint32_t hashes_nr)\n> +\t\t\t\t const struct strvec *keep_hashes)\n> {\n> \tstruct clear_midx_data data = {\n> -\t\t.keep = STRING_LIST_INIT_NODUP,\n> +\t\t.keep = STRING_LIST_INIT_DUP,\n> \t\t.ext = ext,\n> \t};\n> -\tuint32_t i;\n> \n> -\tfor (i = 0; i < hashes_nr; i++)\n> -\t\tstring_list_append(&data.keep,\n> -\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n> -\t\t\t\t\t keep_hashes[i], ext));\n> -\tstring_list_sort(&data.keep);\n> +\tif (keep_hashes) {\n> +\t\tstruct strbuf buf = STRBUF_INIT;\n> +\t\tfor (size_t i = 0; i < keep_hashes->nr; i++) {\n> +\t\t\tstrbuf_reset(&buf);\n> +\n> +\t\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\",\n> +\t\t\t\t keep_hashes->v[i], ext);\n> +\t\t\tstring_list_append(&data.keep, buf.buf);\n> +\t\t}\n> +\n> +\t\tstring_list_sort(&data.keep);\n> +\t\tstrbuf_release(&buf);\n> +\t}\n\nThis hunk was unexpected. We move from xstrfmt() to a strbuf, but does\nthat have anything to do with the rest of the patch?\n\nAlso, I don't think using a strbuf really buys us anything. We are\nreusing the strbuf for each format operation, but then we copy into the\nstring list anyway. So there is one allocation per loop iteration either\nway.\n\nAlso also, the original was leaking the strings, right? The string_list\nwas initialized as NODUP, but we assigned allocate xstrfmt() results to\nit. But because of the nodup, string_list_clear() won't free them.\nIt should have been:\n\n .keep = STRING_LIST_INIT_DUP,\n [...]\n string_list_append_nodup(&data.keep, xstfmrt(...));\n [...]\n string_list_clear(&data.keep);\n\nin patch 2.\n\n-Peff\n"},{"id":"540444","messageId":"20260330231547.GE41843@coredump.intra.peff.net","threadId":"65383","inReplyTo":"41cb5471bb96d38a1b9ae75d022e2c5356318939.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 05/16] midx: introduce `--checksum-only` for incremental MIDX writes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-03-30T23:15:47Z","receivedAt":"2026-03-30T23:15:49Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:41:03PM -0400, Taylor Blau wrote:\n\n> When writing an incremental MIDX layer, the MIDX machinery writes the\n> new layer into the multi-pack-index.d directory and then updates the\n> multi-pack-index-chain file to include the freshly written layer.\n> \n> Future callers however may not wish to immediately update the MIDX chain\n> itself, preferring instead to write out new layer(s) themselves before\n> atomically updating the chain. Concretely, the new incremental\n> MIDX-based repacking strategy will want to do exactly this (that is,\n> assemble the new MIDX chain itself before writing a new chain file and\n> atomically linking it into place).\n> \n> Introduce a `--checksum-only` flag that:\n> \n> * writes the new MIDX layer into the multi-pack-index.d directory\n> \n> * prints its checksum\n> \n> * does not update the multi-pack-index-chain file.\n\nYour goal makes sense, but that is not at all what I would have expected\na \"--checksum-only\" option to do. From the name I expected it to just\ncompute or verify checksums and not write anything to disk. I dunno.\nMaybe it should be \"--no-write-chain-file\" or something?\n\nThis is perhaps on the border of bike-shedding, and maybe the fact that\nit is an option of \"multi-pack-index write\" would be enough to clarify.\n\n> @@ -221,6 +233,9 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n> \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n> \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n> \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n> +\t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n> +\t\t\tN_(\"write a MIDX layer without updating the MIDX chain\"),\n> +\t\t\tMIDX_WRITE_CHECKSUM_ONLY),\n> \t\tOPT_END(),\n> \t};\n> \n> @@ -239,6 +254,15 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n> \tif (argc != 2)\n> \t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n> \t\t\t\t options);\n> +\n> +\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n> +\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n> +\t\terror(_(\"cannot use %s without %s\"),\n> +\t\t \"--checksum-only\", \"--incremental\");\n> +\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n> +\t\t\t\t options);\n> +\t}\n> +\n\nOK. It's sad that we have to duplicate these bits between the write and\ncompact operations, but I'm not sure how much sense it would make to\nfactor out common options (both how much work it is, and how much\nduplication we could actually drop in practice).\n\n> \tsource = handle_object_dir_option(the_repository);\n> \n> \tFREE_AND_NULL(options);\n> diff --git a/midx-write.c b/midx-write.c\n> index 9f7d2bbf4cb..2c6905173ba 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -1600,11 +1600,14 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \t}\n> \n> \tif (ctx.incremental) {\n> -\t\tstruct strbuf lock_name = STRBUF_INIT;\n> +\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n> +\t\t\tstruct strbuf lock_name = STRBUF_INIT;\n> \n> -\t\tget_midx_chain_filename(opts->source, &lock_name);\n> -\t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n> -\t\tstrbuf_release(&lock_name);\n> +\t\t\tget_midx_chain_filename(opts->source, &lock_name);\n> +\t\t\thold_lock_file_for_update(&lk, lock_name.buf,\n> +\t\t\t\t\t\t LOCK_DIE_ON_ERROR);\n> +\t\t\tstrbuf_release(&lock_name);\n> +\t\t}\n\nOK, we skip the chain-file write here (or least the start of it).\n\n> @@ -1725,14 +1728,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \t}\n> \tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n> \n> +\tif (opts->flags & MIDX_WRITE_CHECKSUM_ONLY)\n> +\t\tprintf(\"%s\\n\", hash_to_hex_algop(midx_hash, r->hash_algo));\n\nAnd then output the hash here instead.\n\nJust thinking on the name I suggested, it implies the action from the\nfirst hunk above, but not this one (whereas your name sort of does the\nopposite). I don't think there is any reason to have _two_ options,\nsince you'd always want to do them together (or not at all). I still\nlike my suggestion better. ;)\n\n\n> \tif (ctx.incremental) {\n> -\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n> \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n> \t\tstruct multi_pack_index *m = ctx.base_midx;\n> \n> -\t\tif (!chainf) {\n> -\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n> -\t\t\tgoto cleanup;\n> +\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n> +\t\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n> +\t\t\tif (!chainf) {\n> +\t\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n> +\t\t\t\tgoto cleanup;\n> +\t\t\t}\n> \t\t}\n\nOK, and this is the actual write which is skipped. I guess this could\nalso be conditional on is_lock_file_locked(&lk), assuming we initialize\nit with LOCK_INIT. In some ways that seems easier to analyze to me,\nsince you don't have to realize that \"lk\" is only active when\nCHECKSUM_ONLY is not set. But I don't think there's another reason we'd\nchoose not to take a lock file, so I think it may be six of one and half\na dozen of the other.\n\n> @@ -1793,8 +1801,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \t\t\t}\n> \t\t}\n> \n> -\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n> -\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n> +\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY))\n> +\t\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n> +\t\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\",\n> +\t\t\t\t\tkeep_hashes.v[i]);\n> \t} else {\n> \t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n> \t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n> @@ -1804,10 +1814,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \tif (ctx.m || ctx.base_midx)\n> \t\todb_close(ctx.repo->objects);\n> \n> -\tif (commit_lock_file(&lk) < 0)\n> -\t\tdie_errno(_(\"could not write multi-pack-index\"));\n> +\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n> +\t\tif (commit_lock_file(&lk) < 0)\n> +\t\t\tdie_errno(_(\"could not write multi-pack-index\"));\n> \n> -\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n> +\t\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n> +\t}\n> \tresult = 0;\n\nDitto on these conditionals.\n\nI'm out of time to keep reviewing, so I'll stop here for now. I'll try\nto continue with the rest late tonight or tomorrow.\n\n-Peff\n"},{"id":"540564","messageId":"acw/79TEXFs2cZyJ@nand.local","threadId":"65383","inReplyTo":"20260330223346.GA41843@coredump.intra.peff.net","subject":"Re: [PATCH 01/16] midx-write: handle noop writes when converting incremental chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-31T21:43:11Z","receivedAt":"2026-03-31T21:43:13Z","isPatch":true,"body":"On Mon, Mar 30, 2026 at 06:33:46PM -0400, Jeff King wrote:\n> Now we iterate through each layer individually. We assume that the total\n> count matches ctx->nr, otherwise the midx struct is totally corrupt. We\n> could count them and double-check, I guess, but I don't think there's\n> much point (it would be a BUG() if they were mismatched).\n\nYeah, I agree. The omitted context:\n\n if (ctx->nr != midx->num_packs + midx->num_packs_in_base)\n goto out;\n\ntells us that if we get to this spot, we expect our 'ctx->nr' to be equal\nto 'midx->num_packs + midx->num_packs_in_base'.\n\nAs soon as we find a non-already-MIDX'd pack, we 'goto out', so I don't\nthink it makes much sense to build up a count and check it at the end\neven after we know we can't optimize out the write.\n\nWe could check that the ->num_packs of each of the tip's base MIDXs\nmatch ->num_packs_in_base, but the MIDX internals handle that for us by\nhow we construct that field, since num_packs_in_base is *not* read from\nthe on-disk MIDX.\n\nThanks,\nTaylor\n"},{"id":"540565","messageId":"acxBUkHDolY9VCnR@nand.local","threadId":"65383","inReplyTo":"20260330223849.GB41843@coredump.intra.peff.net","subject":"Re: [PATCH 02/16] midx: use `string_list` for retained MIDX files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-31T21:49:06Z","receivedAt":"2026-03-31T21:49:09Z","isPatch":true,"body":"On Mon, Mar 30, 2026 at 06:38:49PM -0400, Jeff King wrote:\n> Since AFAICT we only care about set inclusion, I probably would have\n> used a strset instead, but for the numbers we are talking about it is\n> unlikely to matter much either way.\n\nYeah, a strset makes more sense here. I adjusted it locally to use that\nover a string_list. Thanks!\n\nThanks,\nTaylor\n"},{"id":"540576","messageId":"acxJ/NfLNloCv3o+@nand.local","threadId":"65383","inReplyTo":"20260330230130.GD41843@coredump.intra.peff.net","subject":"Re: [PATCH 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-31T22:26:04Z","receivedAt":"2026-03-31T22:26:06Z","isPatch":true,"body":"On Mon, Mar 30, 2026 at 07:01:30PM -0400, Jeff King wrote:\n> On Sun, Mar 29, 2026 at 05:41:00PM -0400, Taylor Blau wrote:\n>\n> > -\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n> > +\tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n> >\n> > \tif (ctx.incremental) {\n> > \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n> > @@ -1760,39 +1760,45 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> > \t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n> > \t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n> >\n> > -\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n> > +\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n> > +\t\t\t\tkeep_hashes.nr++;\n>\n> Gross, we are just manipulating the innards of the strvec ourselves?\n>\n> Is it really worth doing this (and adding init_alloc()) versus just:\n>\n> strvec_init(&keep_hashes);\n> for (...)\n> \tstrvec_push(midx_get_checksum_hex(m));\n>\n> ? That's amortized linear-time, and it's not like the number of midx\n> layers is going to be large anyway.\n\nYeah, this is all pretty disgusting. It's in service of what you noted\nbelow where we need to fill in the array out-of-order, but that's really\nbending strvec around MIDX-specific awkwardness, which I dislike.\n\n> I dunno. Maybe that is hard to do. But if so, I question whether moving\n> to a strvec is worth it here, since we are not treating it as an opaque\n> type anymore. And it is not buying us much to use it (we get to pass one\n> parameter versus two, though that is easily solved with a struct, and we\n> get to use _clear() instead of our own free loop).\n\nI think it is worth it to move to a strvec here to avoid having to\nmanage our own memory and pass the length around separately, but I\ndislike the way that I did it in this patch.\n\nI tried adjusting this patch to juggle the MIDX layers in a way that\nwould allow us to just push to the strvec. It ended up being less\nawkward/difficult than I thought, so I think we should go with that and\ndrop the previous patch.\n\n> > void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n> > -\t\t\t\t char **keep_hashes,\n> > -\t\t\t\t uint32_t hashes_nr)\n> > +\t\t\t\t const struct strvec *keep_hashes)\n> > {\n> > \tstruct clear_midx_data data = {\n> > -\t\t.keep = STRING_LIST_INIT_NODUP,\n> > +\t\t.keep = STRING_LIST_INIT_DUP,\n> > \t\t.ext = ext,\n> > \t};\n> > -\tuint32_t i;\n> >\n> > -\tfor (i = 0; i < hashes_nr; i++)\n> > -\t\tstring_list_append(&data.keep,\n> > -\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n> > -\t\t\t\t\t keep_hashes[i], ext));\n> > -\tstring_list_sort(&data.keep);\n> > +\tif (keep_hashes) {\n> > +\t\tstruct strbuf buf = STRBUF_INIT;\n> > +\t\tfor (size_t i = 0; i < keep_hashes->nr; i++) {\n> > +\t\t\tstrbuf_reset(&buf);\n> > +\n> > +\t\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\",\n> > +\t\t\t\t keep_hashes->v[i], ext);\n> > +\t\t\tstring_list_append(&data.keep, buf.buf);\n> > +\t\t}\n> > +\n> > +\t\tstring_list_sort(&data.keep);\n> > +\t\tstrbuf_release(&buf);\n> > +\t}\n>\n> This hunk was unexpected. We move from xstrfmt() to a strbuf, but does\n> that have anything to do with the rest of the patch?\n\nYeah, this looks like it was left over from when I was writing this\npatch and trying out different approaches, and I didn't notice that I\nleft it in when sending this patch out.\n\n> Also also, the original was leaking the strings, right? The string_list\n> was initialized as NODUP, but we assigned allocate xstrfmt() results to\n> it. But because of the nodup, string_list_clear() won't free them.\n> It should have been:\n>\n> .keep = STRING_LIST_INIT_DUP,\n> [...]\n> string_list_append_nodup(&data.keep, xstfmrt(...));\n> [...]\n> string_list_clear(&data.keep);\n>\n> in patch 2.\n\nGood catch, that's right, but partially obviated by the fact that we're moving to\nstrset here.\n\nThanks,\nTaylor\n"},{"id":"540579","messageId":"acxPyXnV8HtxtEHl@nand.local","threadId":"65383","inReplyTo":"acxJ/NfLNloCv3o+@nand.local","subject":"Re: [PATCH 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-03-31T22:50:49Z","receivedAt":"2026-03-31T22:50:51Z","isPatch":true,"body":"On Tue, Mar 31, 2026 at 06:26:04PM -0400, Taylor Blau wrote:\n> > Also also, the original was leaking the strings, right? The string_list\n> > was initialized as NODUP, but we assigned allocate xstrfmt() results to\n> > it. But because of the nodup, string_list_clear() won't free them.\n> > It should have been:\n> >\n> > .keep = STRING_LIST_INIT_DUP,\n> > [...]\n> > string_list_append_nodup(&data.keep, xstfmrt(...));\n> > [...]\n> > string_list_clear(&data.keep);\n> >\n> > in patch 2.\n>\n> Good catch, that's right, but partially obviated by the fact that we're moving to\n> strset here.\n\n...actually, I think this is a little more subtle than I was hoping for.\n\nIf we move to strset, we obviously still need to ensure that\nconstructing .keep doesn't leak strings. The strset API has a\nstrdup_strings option (via the strmap underneath), but does not\nexplicitly free keys on clear.\n\nAs far as I can tell, the strset API is not designed to take ownership\nof the strings themselves, so to avoid leaks we would have to do\nsomething like:\n\n--- 8< ---\ndiff --git a/midx.c b/midx.c\nindex 4976ed4fac3..1c17a8657ff 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -802,12 +802,14 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n \t\t\t\t char **keep_hashes,\n \t\t\t\t uint32_t hashes_nr)\n {\n-\tstruct clear_midx_data data = {\n-\t\t.keep = STRSET_INIT,\n-\t\t.ext = ext,\n-\t};\n+\tstruct clear_midx_data data = { .ext = ext };\n+\tstruct hashmap_iter iter;\n+\tstruct strmap_entry *e;\n+\n \tuint32_t i;\n\n+\tstrset_init_with_options(&data.keep, NULL, 0);\n+\n \tfor (i = 0; i < hashes_nr; i++)\n \t\tstrset_add(&data.keep, xstrfmt(\"multi-pack-index-%s.%s\",\n \t\t\t\t\t keep_hashes[i], ext));\n@@ -815,6 +817,9 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n\n+\thashmap_for_each_entry((struct hashmap*)&data.keep.map, &iter, e, ent) {\n+\t\tfree((void *)e->key);\n+\t}\n \tstrset_clear(&data.keep);\n }\n--- >8 ---\n\nwhich is... pretty gross. I think we may be better off sticking with a\nstring_list for 'keep' here internally.\n\nThanks,\nTaylor\n"},{"id":"540581","messageId":"20260331231713.GA2325145@coredump.intra.peff.net","threadId":"65383","inReplyTo":"acxPyXnV8HtxtEHl@nand.local","subject":"Re: [PATCH 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-03-31T23:17:13Z","receivedAt":"2026-03-31T23:17:21Z","isPatch":true,"body":"On Tue, Mar 31, 2026 at 06:50:49PM -0400, Taylor Blau wrote:\n\n> If we move to strset, we obviously still need to ensure that\n> constructing .keep doesn't leak strings. The strset API has a\n> strdup_strings option (via the strmap underneath), but does not\n> explicitly free keys on clear.\n> \n> As far as I can tell, the strset API is not designed to take ownership\n> of the strings themselves, so to avoid leaks we would have to do\n> something like:\n\nIt's based on strmap, which handles ownership. If you use STRMAP_INIT or\nstrmap_init(), then strdup_strings is set to 1.\n\nOr do you mean that you can't transfer ownership into the strset from an\nexisting string? I think that is true, and if you are generating the\nstrings with xstrfmt() then yeah, you have two copies in the loop and\nhave to free one.\n\nThat is probably actually fine for the sizes we are talking about, but\nif you want to care you can either do the reusable-strbuf thing, or\nwe can add a strset_add_nodup().\n\n> @@ -815,6 +817,9 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n> \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n> \t\t\t\t clear_midx_file_ext, &data);\n> \n> +\thashmap_for_each_entry((struct hashmap*)&data.keep.map, &iter, e, ent) {\n> +\t\tfree((void *)e->key);\n> +\t}\n> \tstrset_clear(&data.keep);\n> }\n> --- >8 ---\n> \n> which is... pretty gross. I think we may be better off sticking with a\n> string_list for 'keep' here internally.\n\nYeah, this is horrible, and this kind of \"we own it, but only kind of\nand strdup_strings is not set\" game has been the cause of many bugs on\nthe string-list side.\n\nHaving a \"nodup\" variant is less awful, but still error prone and leads\nto complications. For example, if strset_add_nodup() sees a duplicate,\nwhat does it do with the input?\n\nSo I would probably just either do:\n\n for (...) {\n\titem = xstrfmt(...);\n\tstrset_add(&set, item);\n\tfree(item);\n }\n\nor if you care about the extra allocation, then:\n\n struct strbuf item = STRBUF_INIT;\n for (...) {\n\tstrbuf_reset(&item);\n\tstrbuf_addf(&item, ...);\n\tstrset_add(&set, item.buf);\n }\n strbuf_release(&item);\n\n-Peff\n"},{"id":"540649","messageId":"ac08qYDcQ7z/2/bC@nand.local","threadId":"65383","inReplyTo":"20260331231713.GA2325145@coredump.intra.peff.net","subject":"Re: [PATCH 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-01T15:41:29Z","receivedAt":"2026-04-01T15:41:31Z","isPatch":true,"body":"On Tue, Mar 31, 2026 at 07:17:13PM -0400, Jeff King wrote:\n> > @@ -815,6 +817,9 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n> > \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n> > \t\t\t\t clear_midx_file_ext, &data);\n> >\n> > +\thashmap_for_each_entry((struct hashmap*)&data.keep.map, &iter, e, ent) {\n> > +\t\tfree((void *)e->key);\n> > +\t}\n> > \tstrset_clear(&data.keep);\n> > }\n> > --- >8 ---\n> >\n> > which is... pretty gross. I think we may be better off sticking with a\n> > string_list for 'keep' here internally.\n>\n> Yeah, this is horrible, and this kind of \"we own it, but only kind of\n> and strdup_strings is not set\" game has been the cause of many bugs on\n> the string-list side.\n\nIndeed ;-).\n\n> Having a \"nodup\" variant is less awful, but still error prone and leads\n> to complications. For example, if strset_add_nodup() sees a duplicate,\n> what does it do with the input?\n>\n> So I would probably just either do:\n>\n> for (...) {\n> \titem = xstrfmt(...);\n> \tstrset_add(&set, item);\n> \tfree(item);\n> }\n>\n> or if you care about the extra allocation, then:\n>\n> struct strbuf item = STRBUF_INIT;\n> for (...) {\n> \tstrbuf_reset(&item);\n> \tstrbuf_addf(&item, ...);\n> \tstrset_add(&set, item.buf);\n> }\n> strbuf_release(&item);\n\nI think the latter of these two makes sense to me.\n\nI'm not necessarily of the opinion that the strset API should *never*\nhave DUP/NODUP semantics like the string_list API does. But I do think\nthat we should punt on taking something like this up as long as\npossible.\n\nThanks,\nTaylor\n"},{"id":"540664","messageId":"20260401192526.GB2905896@coredump.intra.peff.net","threadId":"65383","inReplyTo":"ac08qYDcQ7z/2/bC@nand.local","subject":"Re: [PATCH 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-01T19:25:26Z","receivedAt":"2026-04-01T19:25:28Z","isPatch":true,"body":"On Wed, Apr 01, 2026 at 11:41:29AM -0400, Taylor Blau wrote:\n\n> I think the latter of these two makes sense to me.\n> \n> I'm not necessarily of the opinion that the strset API should *never*\n> have DUP/NODUP semantics like the string_list API does. But I do think\n> that we should punt on taking something like this up as long as\n> possible.\n\nYup, I completely agree.\n\n-Peff\n"},{"id":"540807","messageId":"ac7y7t+Qgm0htU3c@nand.local","threadId":"65383","inReplyTo":"20260330231547.GE41843@coredump.intra.peff.net","subject":"Re: [PATCH 05/16] midx: introduce `--checksum-only` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-02T22:51:26Z","receivedAt":"2026-04-02T22:51:30Z","isPatch":true,"body":"On Mon, Mar 30, 2026 at 07:15:47PM -0400, Jeff King wrote:\n> Your goal makes sense, but that is not at all what I would have expected\n> a \"--checksum-only\" option to do. From the name I expected it to just\n> compute or verify checksums and not write anything to disk. I dunno.\n> Maybe it should be \"--no-write-chain-file\" or something?\n>\n> This is perhaps on the border of bike-shedding, and maybe the fact that\n> it is an option of \"multi-pack-index write\" would be enough to clarify.\n\nHeh :-).\n\nI think that it definitely qualifies as bike-shedding, but I'm glad you\nbrought it up. I hadn't considered the name being potentially confusing,\nbut now that you mention it I can't un-see it.\n\nCalling this something like '--no-write-chain-file' feels like a good\nchoice to me, and doesn't paint us into a corner if we want to later add\nan option that does something like you describe above.\n\n(As an aside, we already have 'git multi-pack-index verify', which is\nprobably the closest thing to what you're describing, so perhaps the\nconcern is lessened by that fact. But even if we already have that\nfunctionality laying around, I think that \"--no-write-chain-file\" is a\nmore descriptive name for the command-line flag than \"--checksum-only\".)\n\n> > @@ -239,6 +254,15 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n> > \tif (argc != 2)\n> > \t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n> > \t\t\t\t options);\n> > +\n> > +\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n> > +\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n> > +\t\terror(_(\"cannot use %s without %s\"),\n> > +\t\t \"--checksum-only\", \"--incremental\");\n> > +\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n> > +\t\t\t\t options);\n> > +\t}\n> > +\n>\n> OK. It's sad that we have to duplicate these bits between the write and\n> compact operations, but I'm not sure how much sense it would make to\n> factor out common options (both how much work it is, and how much\n> duplication we could actually drop in practice).\n\nI agree. I looked at unifying these a little further, but decided\nagainst it as it ended up making things more complex than it was worth\nIMHO.\n\n> > \tif (ctx.incremental) {\n> > -\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n> > \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n> > \t\tstruct multi_pack_index *m = ctx.base_midx;\n> >\n> > -\t\tif (!chainf) {\n> > -\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n> > -\t\t\tgoto cleanup;\n> > +\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n> > +\t\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n> > +\t\t\tif (!chainf) {\n> > +\t\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n> > +\t\t\t\tgoto cleanup;\n> > +\t\t\t}\n> > \t\t}\n>\n> OK, and this is the actual write which is skipped. I guess this could\n> also be conditional on is_lock_file_locked(&lk), assuming we initialize\n> it with LOCK_INIT. In some ways that seems easier to analyze to me,\n> since you don't have to realize that \"lk\" is only active when\n> CHECKSUM_ONLY is not set. But I don't think there's another reason we'd\n> choose not to take a lock file, so I think it may be six of one and half\n> a dozen of the other.\n\nYeah, I think this is more or less equivalent. In some ways I like the\nexplicitness of the conditional as it is currently written, but I think\nyour suggestion reads clearer.\n\nIt ends up producing nice code like:\n\n if (is_lock_file_locked(&lk)) {\n FILE *chainf = fdopen_lock_file(&lk, \"w\");\n /* ... */\n }\n\nwhich is pleasing to me in the sense that it reads like \"if the lockfile\nis held, then do something with it\".\n\nI think there is a small risk of coupling the state of the lockfile with\nthe MIDX_NO_WRITE_CHAIN bit in the flags, but it's easily mitigated by\nan ASSERT(opts->flags & MIDX_NO_WRITE_CHAIN) in the else branch of the\nconditional above.\n\nThere are a few such conditionals, and repeatedly checking that the flag\nbits are set is wasteful, so I check it only once in the big \"if\n(ctx->incremental)\" block, and then once again in the top-level\nwrite_midx_internal() function before we do the final commit and\ncleanup.\n\nThanks,\nTaylor\n"},{"id":"541039","messageId":"20260407055753.GA714878@coredump.intra.peff.net","threadId":"65383","inReplyTo":"6b42d284b79a25e97d506e22c709e4dbe6fddd58.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-07T05:57:53Z","receivedAt":"2026-04-07T05:57:54Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:41:06PM -0400, Taylor Blau wrote:\n\n> Introduce a new `--base` option for both subcommands to specify the\n> checksum of the MIDX layer to use as the base. The given checksum must\n> refer to a valid layer in the MIDX chain that is an ancestor of the\n> topmost layer being written or compacted.\n\nOK, that makes sense.\n\n> The special value \"none\" is accepted to produce a root layer with no\n> parent. This will be needed when the incremental repacking machinery\n> determines that the bottommost layers of the chain should be replaced.\n\nMy gut feeling on reading is that a chain based on \"none\" is really just\na non-incremental midx, so we could just write that. But:\n\n 1. I don't recall off-hand how expensive it is to convert a\n non-incremental to an incremental one. I.e., can we just rename the\n file and write out a chain link and it becomes incremental?\n\n 2. Even if it is cheap to convert, I can believe it makes geometric\n repacking easier to think about if we just always think about it as\n an incremental midx.\n\nAnd perhaps (3), I am just wrong that they are the same thing. ;) I\nhaven't thought it through all that carefully.\n\n> If no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\n> immediate parent in the chain, and `write` appends to the existing tip.\n> \n> For the `write` subcommand, `--base` requires `--checksum-only`. A plain\n> `write --incremental` appends a new layer to the live chain tip with no\n> mechanism to atomically replace it; overriding the base would produce a\n> layer that does not extend the tip, breaking chain invariants. With\n> `--checksum-only` the chain is left unmodified and the caller is\n> responsible for assembling a valid chain.\n\nOK, makes sense. If we are using --base we are doing something clever\nand weird, and the chain tip should be left untouched. I guess \"midx\nwrite\" would never delete an incremental (it is always just adding one\nmore to the chain), so we don't have to worry about that. Presumably\nwe'll eventually write out the new chain tip from the caller, which will\nthen have to remember to delete the now-orphaned chain elements. But\nthat's kind of the point of the new geometric approach, so it seems like\nthings are falling into place.\n\n> For `compact`, no such restriction applies. The compaction operation\n> atomically replaces the compacted range in the chain file, so writing\n> the result on top of any valid ancestor preserves chain invariants.\n\nOK, so there's an implicit base..tip compaction here (as opposed to\nwithout --base, an implicit root..trip compaction?). Makes sense.\n\n> @@ -83,6 +85,13 @@ marker).\n> \t\tand packs not present in an existing MIDX layer.\n> \t\tMigrates non-incremental MIDXs to incremental ones when\n> \t\tnecessary.\n> +\n> +\t--base=::\n> +\t\tSpecify the checksum of an existing MIDX layer to use\n> +\t\tas the base when writing a new incremental layer.\n> +\t\tThe special value `none` indicates that the new layer\n> +\t\tshould have no base (i.e., it becomes a root layer).\n> +\t\tRequires `--checksum-only`.\n\nNot really a critique of this patch, but I'd guess that a lot of these\noptions are getting esoteric enough that it is hard to understand \n\n> [...]\n\nThe patch looked as advertised, and I didn't find anything in it to\ncomment on.\n\n-Peff\n"},{"id":"541040","messageId":"20260407060439.GB714878@coredump.intra.peff.net","threadId":"65383","inReplyTo":"2d377f534071ad9def3fb4860b2581197d02726b.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 07/16] repack: track the ODB source via existing_packs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-07T06:04:39Z","receivedAt":"2026-04-07T06:04:40Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:41:08PM -0400, Taylor Blau wrote:\n\n> Store the ODB source in the `existing_packs` struct and use that in\n> place of the raw `repo->objects->sources` access within `cmd_repack()`.\n> \n> The source used is still assigned from the first source in the list, so\n> there are no functional changes in this commit. The changes instead\n> serve two purposes (one immediate, one not):\n> \n> - The incremental MIDX-based repacking machinery will need to know what\n> source is being used to read the existing MIDX/chain (should one\n> exist).\n> \n> - In the future, if \"git repack\" is taught how to operate on other\n> object sources, this field will serve as the authoritative value for\n> that source.\n\nMakes sense. I wonder if we still need the repo field here:\n\n> diff --git a/repack.h b/repack.h\n> index bc9f2e1a5de..c0e9f0ca647 100644\n> --- a/repack.h\n> +++ b/repack.h\n> @@ -56,6 +56,7 @@ struct packed_git;\n> \n> struct existing_packs {\n> \tstruct repository *repo;\n> +\tstruct odb_source *source;\n> \tstruct string_list kept_packs;\n> \tstruct string_list non_kept_packs;\n> \tstruct string_list cruft_packs;\n\nsince we are going to mostly be operating on one odb source. But I think\nit gets weird asking about things like \"which hash algorithm to use\",\nbecause then you end up back at source->odb->repo. Plus there are some\nparts that seem to look at the whole repo.\n\nSo I think it gets complicated and is well off the path to our goal, so\nnot worth thinking further about for now.\n\n-Peff\n"},{"id":"541041","messageId":"20260407060554.GC714878@coredump.intra.peff.net","threadId":"65383","inReplyTo":"6af23849f1d499e7a23d1ef4a70ac3d29748e1ce.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 08/16] midx: expose `midx_layer_contains_pack()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-07T06:05:54Z","receivedAt":"2026-04-07T06:05:56Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:41:11PM -0400, Taylor Blau wrote:\n\n> Rename the function `midx_contains_pack_1()` to instead be called\n> `midx_layer_contains_pack()` and make it accessible. Unlike\n> `midx_contains_pack()` (which recurses through the entire chain), this\n> function checks only a single MIDX layer.\n> \n> This will be used by a subsequent commit to determine whether a given\n> pack belongs to the tip MIDX layer specifically, rather than to any\n> layer in the chain.\n\nYup, that is a much better name.\n\n-Peff\n"},{"id":"541042","messageId":"20260407060842.GD714878@coredump.intra.peff.net","threadId":"65383","inReplyTo":"cd5eba0f665a4d3a4febc47f1fe5f13200a4f941.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 10/16] repack-midx: extract `repack_fill_midx_stdin_packs()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-07T06:08:42Z","receivedAt":"2026-04-07T06:08:43Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:41:16PM -0400, Taylor Blau wrote:\n\n> This simplifies `write_midx_included_packs()` and prepares for a\n> subsequent commit where the same helper is called with `cmd->out = -1`\n> to capture the MIDX's checksum from the command's standard output,\n> which is needed when writing MIDX layers with `--checksum-only`.\n\nHmm, that will involve us feeding both \"in\" and \"out\", so we'll have to\nconsider deadlock. We're not there yet, though... :)\n\n-Peff\n"},{"id":"541043","messageId":"20260407061024.GE714878@coredump.intra.peff.net","threadId":"65383","inReplyTo":"d2c667e841300c3310e3a23493ac27eb2b08992b.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 11/16] repack-geometry: prepare for incremental MIDX repacking","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-07T06:10:24Z","receivedAt":"2026-04-07T06:10:26Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:41:18PM -0400, Taylor Blau wrote:\n\n> Teach `pack_geometry_init()` to optionally restrict the set of\n> repacking candidates to only packs in the tip MIDX layer when a\n> `midx_layer_threshold` is configured. If the tip layer has fewer packs\n> than the threshold, those packs are excluded entirely; otherwise only\n> packs in that layer participate in the geometric repack.\n\nMy first question of course is \"so how do we set that threshold?\". But\nit looks like that is not done yet in this commit, so this is mostly\ndead code preparing for later (and thus no tests). OK.\n\n-Peff\n"},{"id":"541044","messageId":"20260407061837.GF714878@coredump.intra.peff.net","threadId":"65383","inReplyTo":"56dc6d55974ffe08647d57915d0861181e1d7d41.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 12/16] builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-07T06:18:37Z","receivedAt":"2026-04-07T06:18:42Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:41:21PM -0400, Taylor Blau wrote:\n\n> diff --git a/repack.h b/repack.h\n> index 77d24ee45fb..81907fcce7f 100644\n> --- a/repack.h\n> +++ b/repack.h\n> @@ -134,6 +134,11 @@ void pack_geometry_release(struct pack_geometry *geometry);\n> \n> struct tempfile;\n> \n> +enum repack_write_midx_mode {\n> +\tREPACK_WRITE_MIDX_NONE,\n> +\tREPACK_WRITE_MIDX_DEFAULT,\n> +};\n\nSince the name of the variable did not change, the compiler would not\ncatch any old cases that did:\n\n if (write_midx)\n\nfor us. Which made me wonder if we should set _NONE to 0 explicitly here\nso that continues to work, in case we missed a spot.\n\nBut after manually checking each spot that uses the enum (and there are\nnot that many), they have all been faithfully converted. So it's\nprobably not worth trying to preserve that compatibility (and anyway,\nonce we introduce a third mode, it's not clear that such a site would be\ncorrect anymore anyway).\n\n-Peff\n"},{"id":"541045","messageId":"20260407062954.GG714878@coredump.intra.peff.net","threadId":"65383","inReplyTo":"e304e316ac4d4e92125e2e57b32fa445acf66e3e.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 13/16] packfile: ensure `close_pack_revindex()` frees in-memory revindex","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-07T06:29:54Z","receivedAt":"2026-04-07T06:29:56Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:41:23PM -0400, Taylor Blau wrote:\n\n> The following commit will introduce a case where we write a MIDX bitmap\n> over packs that do not themselves have on-disk *.rev files.\n> \n> This case is supported within Git, and we will simply fall back to\n> generating the revindex in memory. But we don't ever release that\n> memory, causing a leak that is exposed by a test introduced in the\n> following commit.\n> \n> (As far as I could find, we never free()'d memory allocated as a\n> byproduct of creating an in-memory revindex, likely because that code\n> predates the leak-checking niceties we have in the test suite now.)\n\nHmmm, I'm surprised we don't trigger this leak during the course of\nodb_free(). But I guess we do that fairly rarely. It mostly happens via\nrepo_clear(), and we never do that for the \"main\" repo (just for\nsubmodules). And it's unlikely that we'd have loaded a revindex for the\nsubmodule for most operations (mostly we need them for generating packs\nor when using bitmaps).\n\nSo I can't see an obvious way that this would be triggered in practice\nwith the current code. But I agree it is a leak waiting to happen, as\nnobody ever clears p->revindex.\n\n> diff --git a/packfile.c b/packfile.c\n> index d4de9f3ffe8..97f7662e079 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -420,6 +420,8 @@ void close_pack_index(struct packed_git *p)\n> \n> static void close_pack_revindex(struct packed_git *p)\n> {\n> +\tFREE_AND_NULL(p->revindex);\n> +\n> \tif (!p->revindex_map)\n> \t\treturn;\n\nThe fact that this is called \"close\" made me wonder if some code might\nwant to just relinquish access to on-disk resources, and then pay the\ncost when we have to unexpectedly re-compute the in-memory revindex.\n\nBut this function and its siblings are really about resetting the whole\nstate of the packed_git struct. So the behavior you're proposing matches\nthat.\n\n-Peff\n"},{"id":"541598","messageId":"ad67LgVcCeQubMbO@nand.local","threadId":"65383","inReplyTo":"20260407055753.GA714878@coredump.intra.peff.net","subject":"Re: [PATCH 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-14T22:09:50Z","receivedAt":"2026-04-14T22:09:52Z","isPatch":true,"body":"On Tue, Apr 07, 2026 at 01:57:53AM -0400, Jeff King wrote:\n> On Sun, Mar 29, 2026 at 05:41:06PM -0400, Taylor Blau wrote:\n>\n> > Introduce a new `--base` option for both subcommands to specify the\n> > checksum of the MIDX layer to use as the base. The given checksum must\n> > refer to a valid layer in the MIDX chain that is an ancestor of the\n> > topmost layer being written or compacted.\n>\n> OK, that makes sense.\n>\n> > The special value \"none\" is accepted to produce a root layer with no\n> > parent. This will be needed when the incremental repacking machinery\n> > determines that the bottommost layers of the chain should be replaced.\n>\n> My gut feeling on reading is that a chain based on \"none\" is really just\n> a non-incremental midx, so we could just write that. But:\n>\n> 1. I don't recall off-hand how expensive it is to convert a\n> non-incremental to an incremental one. I.e., can we just rename the\n> file and write out a chain link and it becomes incremental?\n>\n> 2. Even if it is cheap to convert, I can believe it makes geometric\n> repacking easier to think about if we just always think about it as\n> an incremental midx.\n>\n> And perhaps (3), I am just wrong that they are the same thing. ;) I\n> haven't thought it through all that carefully.\n\nIn this case we are truly writing an incremental MIDX layer without any\nparent. So it's logically similar from a non-incremental MIDX in the\nsense that there is only a single layer, but it is still an incremental\nMIDX nonetheless.\n\nWe have to write these as incremental layers, since we can't replace the\nmain $GIT_DIR/objects/pack/multi-pack-index, so that takes (1) off of\nthe table.\n\n> > If no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\n> > immediate parent in the chain, and `write` appends to the existing tip.\n> >\n> > For the `write` subcommand, `--base` requires `--checksum-only`. A plain\n> > `write --incremental` appends a new layer to the live chain tip with no\n> > mechanism to atomically replace it; overriding the base would produce a\n> > layer that does not extend the tip, breaking chain invariants. With\n> > `--checksum-only` the chain is left unmodified and the caller is\n> > responsible for assembling a valid chain.\n>\n> OK, makes sense. If we are using --base we are doing something clever\n> and weird, and the chain tip should be left untouched. I guess \"midx\n> write\" would never delete an incremental (it is always just adding one\n> more to the chain), so we don't have to worry about that. Presumably\n> we'll eventually write out the new chain tip from the caller, which will\n> then have to remember to delete the now-orphaned chain elements. But\n> that's kind of the point of the new geometric approach, so it seems like\n> things are falling into place.\n\nYup, that is driven at the 'repack' layer and will appear later on in\nthis series.\n\nThanks,\nTaylor\n"},{"id":"541603","messageId":"ad6+jMHcwatRRcQa@nand.local","threadId":"65383","inReplyTo":"20260407060439.GB714878@coredump.intra.peff.net","subject":"Re: [PATCH 07/16] repack: track the ODB source via existing_packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-14T22:24:12Z","receivedAt":"2026-04-14T22:24:14Z","isPatch":true,"body":"On Tue, Apr 07, 2026 at 02:04:39AM -0400, Jeff King wrote:\n> Makes sense. I wonder if we still need the repo field here:\n>\n> > diff --git a/repack.h b/repack.h\n> > index bc9f2e1a5de..c0e9f0ca647 100644\n> > --- a/repack.h\n> > +++ b/repack.h\n> > @@ -56,6 +56,7 @@ struct packed_git;\n> >\n> > struct existing_packs {\n> > \tstruct repository *repo;\n> > +\tstruct odb_source *source;\n> > \tstruct string_list kept_packs;\n> > \tstruct string_list non_kept_packs;\n> > \tstruct string_list cruft_packs;\n>\n> since we are going to mostly be operating on one odb source. But I think\n> it gets weird asking about things like \"which hash algorithm to use\",\n> because then you end up back at source->odb->repo. Plus there are some\n> parts that seem to look at the whole repo.\n\nI think that doing so is possible, though it's not totally\nstraightforward. There are a handful of places where you really truly\ndo want to work with the repo (e.g., when calling trace2_region_enter(),\nwhich requires a repository pointer as its final argument).\n\nI do think that we are probably over-relying on the repository pointer\nin the existing_packs struct after this patch, especially with the ODB\nsource refactorings that have been going on.\n\n> So I think it gets complicated and is well off the path to our goal, so\n> not worth thinking further about for now.\n\nI agree.\n\nThanks,\nTaylor\n"},{"id":"541604","messageId":"ad7B7cxKP7tNCXUf@nand.local","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 00/16] repack: incremental MIDX/bitmap-based repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-14T22:38:37Z","receivedAt":"2026-04-14T22:38:41Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 05:40:50PM -0400, Taylor Blau wrote:\n> OVERVIEW\n> ========\n>\n> This series implements the third and final major component for an\n> incremental MIDX/bitmap-based repacking strategy. As a refresher, those\n> are:\n\nThanks for the review on the first round. Review seems to have\nstabilized (I somehow missed the handful of messages that Peff sent on\nthe 7th, but just read through them and only came up with minimal\nchanges as a result).\n\nFor now, I'll hold off on sending a v2 until we are on the other side of\n2.54 next week. In the meantime, here is a quick summary of the main\nchanges that that version will contain:\n\n - Use a strset instead of a string_list for keeping track of the MIDX\n layers to retain when calling `either clear_midx_files_ext()` or\n `clear_incremental_midx_files_ext()`.\n\n - A new patch to rewrite the logic for determining which MIDX layers\n comprise the new chain via keep_hashes to build the array in order.\n The subsequent patch converts that into a strvec, which no longer\n requires direct manipulation.\n\n - The new \"--checksum-only\" option has been renamed to\n \"--no-write-chain-file\", and various small implementation tweaks\n (e.g., relying on `is_lock_file_locked()` to determine whether we\n should update the chain file as opposed to reading the flags).\n\nFor the curious, the v2 that I'll send is currently in my fork\n(available from git@github.com:ttaylorr/git.git) in a branch called\n'tb/incremental-midx-part-3.3'. A range-diff of what I will likely send\nis below:\n\n--- 8< ---\n 1: 0c7a68ca5a7 = 1: 2db38d41b96 midx-write: handle noop writes when converting incremental chains\n 2: 1391552dfc6 ! 2: 99e2c8f0cd6 midx: use `string_list` for retained MIDX files\n @@ Metadata\n Author: Taylor Blau \n\n ## Commit message ##\n - midx: use `string_list` for retained MIDX files\n + midx: use `strset` for retained MIDX files\n\n Both `clear_midx_files_ext()` and `clear_incremental_midx_files_ext()`\n build a list of filenames to keep while pruning stale MIDX files. Today\n - they hand-roll an array instead of using a `string_list`, thus requiring\n - us to pass an additional length parameter, and makes lookups linear.\n + they hand-roll an array instead of using a `strset`, thus requiring us\n + to pass an additional length parameter, and makes lookups linear.\n\n - Replace the bare array with a `string_list` which can be passed around\n - as a single parameter. Though it improves lookup performance, the\n - difference is likely immeasurable given how small the keep_hashes array\n - typically is.\n + Replace the bare array with a `strset` which can be passed around as a\n + single parameter. Though it improves lookup performance, the difference\n + is likely immeasurable given how small the keep_hashes array typically\n + is.\n\n Signed-off-by: Taylor Blau \n\n @@ midx.c: int midx_checksum_valid(struct multi_pack_index *m)\n struct clear_midx_data {\n -\tchar **keep;\n -\tuint32_t keep_nr;\n -+\tstruct string_list keep;\n ++\tstruct strset keep;\n \tconst char *ext;\n };\n\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n -\t\tif (!strcmp(data->keep[i], file_name))\n -\t\t\treturn;\n -\t}\n -+\tif (string_list_has_string(&data->keep, file_name))\n ++\tif (strset_contains(&data->keep, file_name))\n +\t\treturn;\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n -\tstruct clear_midx_data data;\n -\tmemset(&data, 0, sizeof(struct clear_midx_data));\n +\tstruct clear_midx_data data = {\n -+\t\t.keep = STRING_LIST_INIT_DUP,\n ++\t\t.keep = STRSET_INIT,\n +\t\t.ext = ext,\n +\t};\n\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n\n -\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n -\t\tdata.keep_nr = 1;\n -+\t\tstring_list_insert(&data.keep, buf.buf);\n ++\t\tstrset_add(&data.keep, buf.buf);\n +\n +\t\tstrbuf_release(&buf);\n \t}\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n -\tif (keep_hash)\n -\t\tfree(data.keep[0]);\n -\tfree(data.keep);\n -+\tstring_list_clear(&data.keep, 0);\n ++\tstrset_clear(&data.keep);\n }\n\n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n {\n -\tstruct clear_midx_data data;\n +\tstruct clear_midx_data data = {\n -+\t\t.keep = STRING_LIST_INIT_NODUP,\n ++\t\t.keep = STRSET_INIT,\n +\t\t.ext = ext,\n +\t};\n ++\tstruct strbuf buf = STRBUF_INIT;\n \tuint32_t i;\n\n -\tmemset(&data, 0, sizeof(struct clear_midx_data));\n --\n ++\tfor (i = 0; i < hashes_nr; i++) {\n ++\t\tstrbuf_reset(&buf);\n ++\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n ++\t\t\t ext);\n +\n -\tALLOC_ARRAY(data.keep, hashes_nr);\n - \tfor (i = 0; i < hashes_nr; i++)\n +-\tfor (i = 0; i < hashes_nr; i++)\n -\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n -\t\t\t\t ext);\n -\tdata.keep_nr = hashes_nr;\n -\tdata.ext = ext;\n -+\t\tstring_list_append(&data.keep,\n -+\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n -+\t\t\t\t\t keep_hashes[i], ext));\n -+\tstring_list_sort(&data.keep);\n ++\t\tstrset_add(&data.keep, buf.buf);\n ++\t}\n\n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n -\tfor (i = 0; i < hashes_nr; i++)\n -\t\tfree(data.keep[i]);\n -\tfree(data.keep);\n -+\tstring_list_clear(&data.keep, 0);\n ++\tstrbuf_release(&buf);\n ++\tstrset_clear(&data.keep);\n }\n\n void clear_midx_file(struct repository *r)\n 3: aea6aad6953 < -: ----------- strvec: introduce `strvec_init_alloc()`\n -: ----------- > 3: 04f1c23884c midx: build `keep_hashes` array in order\n 4: 5fc72d5049a ! 4: a71c8024c1f midx: use `strvec` for `keep_hashes`\n @@ Commit message\n value is used to determine the contents of the resulting\n `multi-pack-index-chain` file when writing with \"--incremental\".\n\n + Since the previous commit already builds the array in forward order, the\n + conversion is straightforward: replace indexed assignments with\n + `strvec_push()`, drop the pre-counting and `CALLOC_ARRAY()`, and\n + simplify cleanup via `strvec_clear()`.\n +\n Signed-off-by: Taylor Blau \n\n ## midx-write.c ##\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \tint dropped_packs = 0;\n \tint result = -1;\n -\tconst char **keep_hashes = NULL;\n +-\tsize_t keep_hashes_nr = 0;\n +\tstruct strvec keep_hashes = STRVEC_INIT;\n - \tsize_t keep_hashes_nr = 0;\n \tstruct chunkfile *cf;\n\n + \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n - \t} else {\n - \t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n - \t}\n + \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n + \t\tdie(_(\"too many multi-pack-indexes\"));\n +\n +-\tif (ctx.compact) {\n +-\t\tstruct multi_pack_index *m;\n +-\n +-\t\t/*\n +-\t\t * Keep all MIDX layers excluding those in the range [from, to].\n +-\t\t */\n +-\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n +-\t\t\tkeep_hashes_nr++;\n +-\t\tfor (m = ctx.m;\n +-\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n +-\t\t m = m->base_midx)\n +-\t\t\tkeep_hashes_nr++;\n +-\n +-\t\tkeep_hashes_nr++; /* include the compacted layer */\n +-\t} else {\n +-\t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n +-\t}\n -\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n -+\tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n -\n +-\n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n -@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n - \t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n - \t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n + \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n + \t\tstruct multi_pack_index *m = ctx.base_midx;\n + \t\tstruct multi_pack_index **layers = NULL;\n + \t\tsize_t layers_nr = 0, layers_alloc = 0;\n +-\t\tsize_t j = 0;\n\n --\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.nr++;\n - \t\t\t\tm = m->base_midx;\n + \t\tif (!chainf) {\n + \t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n + \t\t\twhile (layers_nr)\n +-\t\t\t\tkeep_hashes[j++] =\n +-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n ++\t\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n\n --\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n -+\t\t\tkeep_hashes.v[i] = xstrdup(hash_to_hex_algop(midx_hash,\n - \t\t\t\t\t\t\t\t r->hash_algo));\n -+\t\t\tkeep_hashes.nr++;\n +-\t\t\tkeep_hashes[j++] =\n +-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n +-\t\t\t\t\t\t\t r->hash_algo));\n ++\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t hash_to_hex_algop(midx_hash,\n ++\t\t\t\t\t\t r->hash_algo));\n\n - \t\t\ti = 0;\n - \t\t\tfor (m = ctx.m;\n - \t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n - \t\t\t m = m->base_midx) {\n --\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n -+\t\t\t\tkeep_hashes.v[keep_hashes_nr - i - 1] =\n - \t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.nr++;\n - \t\t\t\ti++;\n + \t\t\tfor (mp = ctx.m;\n + \t\t\t mp && midx_hashcmp(mp, ctx.compact_to,\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n + \t\t\twhile (layers_nr)\n +-\t\t\t\tkeep_hashes[j++] =\n +-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n ++\t\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \t\t} else {\n --\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n -+\t\t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n - \t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n - \t\t\t\t\t\t\t r->hash_algo));\n -+\t\t\tkeep_hashes.nr++;\n -\n - \t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n - \t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n -\n --\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.nr++;\n - \t\t\t\tm = m->base_midx;\n + \t\t\tfor (; m; m = m->base_midx) {\n + \t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n + \t\t\t\tlayers[layers_nr++] = m;\n \t\t\t}\n + \t\t\twhile (layers_nr)\n +-\t\t\t\tkeep_hashes[j++] =\n +-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n ++\t\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n +\n +-\t\t\tkeep_hashes[j++] =\n +-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n +-\t\t\t\t\t\t\t r->hash_algo));\n ++\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t hash_to_hex_algop(midx_hash,\n ++\t\t\t\t\t\t r->hash_algo));\n \t\t}\n\n - \t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n +-\t\tASSERT(j == keep_hashes_nr);\n +-\n + \t\tfree(layers);\n +\n +-\t\tfor (uint32_t i = 0; i < j; i++)\n -\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n ++\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n +\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n \t} else {\n -\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n -+\t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n - \t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n -+\t\tkeep_hashes.nr++;\n +-\t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n ++\t\tstrvec_push(&keep_hashes,\n ++\t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n \t}\n\n \tif (ctx.m || ctx.base_midx)\n @@ midx.c: void clear_midx_files_ext(struct odb_source *source, const char *ext,\n +\t\t\t\t const struct strvec *keep_hashes)\n {\n \tstruct clear_midx_data data = {\n --\t\t.keep = STRING_LIST_INIT_NODUP,\n -+\t\t.keep = STRING_LIST_INIT_DUP,\n + \t\t.keep = STRSET_INIT,\n \t\t.ext = ext,\n \t};\n + \tstruct strbuf buf = STRBUF_INIT;\n -\tuint32_t i;\n\n --\tfor (i = 0; i < hashes_nr; i++)\n --\t\tstring_list_append(&data.keep,\n --\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n --\t\t\t\t\t keep_hashes[i], ext));\n --\tstring_list_sort(&data.keep);\n +-\tfor (i = 0; i < hashes_nr; i++) {\n +-\t\tstrbuf_reset(&buf);\n +-\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n +-\t\t\t ext);\n +\tif (keep_hashes) {\n -+\t\tstruct strbuf buf = STRBUF_INIT;\n +\t\tfor (size_t i = 0; i < keep_hashes->nr; i++) {\n +\t\t\tstrbuf_reset(&buf);\n -+\n +\t\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\",\n +\t\t\t\t keep_hashes->v[i], ext);\n -+\t\t\tstring_list_append(&data.keep, buf.buf);\n +\n +-\t\tstrset_add(&data.keep, buf.buf);\n ++\t\t\tstrset_add(&data.keep, buf.buf);\n +\t\t}\n -+\n -+\t\tstring_list_sort(&data.keep);\n -+\t\tstrbuf_release(&buf);\n -+\t}\n + \t}\n\n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n - \t\t\t\t clear_midx_file_ext, &data);\n 5: 41cb5471bb9 ! 5: 51ffdb1382d midx: introduce `--checksum-only` for incremental MIDX writes\n @@ Metadata\n Author: Taylor Blau \n\n ## Commit message ##\n - midx: introduce `--checksum-only` for incremental MIDX writes\n + midx: introduce `--no-write-chain-file` for incremental MIDX writes\n\n When writing an incremental MIDX layer, the MIDX machinery writes the\n new layer into the multi-pack-index.d directory and then updates the\n @@ Commit message\n assemble the new MIDX chain itself before writing a new chain file and\n atomically linking it into place).\n\n - Introduce a `--checksum-only` flag that:\n + Introduce a `--no-write-chain-file` flag that:\n\n * writes the new MIDX layer into the multi-pack-index.d directory\n\n @@ Documentation/git-multi-pack-index.adoc: SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n -\t\t [--refs-snapshot=]\n -+\t\t [--refs-snapshot=] [--[no-]checksum-only]\n ++\t\t [--refs-snapshot=] [--[no-]write-chain-file]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n -\t\t [--[no-]bitmap] \n -+\t\t [--[no-]bitmap] [--[no-]checksum-only] \n ++\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n +@@ Documentation/git-multi-pack-index.adoc: marker).\n + \t\tand packs not present in an existing MIDX layer.\n + \t\tMigrates non-incremental MIDXs to incremental ones when\n + \t\tnecessary.\n ++\n ++\t--[no-]write-chain-file::\n ++\t\tWhen used with `--incremental`, write a new MIDX layer\n ++\t\tbut do not update the multi-pack-index-chain file.\n ++\t\tThe checksum of the new layer is printed to standard\n ++\t\toutput, allowing the caller to assemble and write the\n ++\t\tchain itself. Requires `--incremental`.\n + --\n +\n + compact::\n +@@ Documentation/git-multi-pack-index.adoc: compact::\n +\n + \t--[no-]bitmap::\n + \t\tControl whether or not a multi-pack bitmap is written.\n ++\n ++\t--[no-]write-chain-file::\n ++\t\tWhen used with `--incremental`, write a new compacted\n ++\t\tMIDX layer but do not update the multi-pack-index-chain\n ++\t\tfile. The checksum of the new layer is printed to\n ++\t\tstandard output. Requires `--incremental`.\n + --\n +\n + verify::\n\n ## builtin/multi-pack-index.c ##\n @@\n @@ builtin/multi-pack-index.c\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n -\t \" [--refs-snapshot=]\")\n -+\t \" [--refs-snapshot=] [--[no-]checksum-only]\")\n ++\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n\n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n -\t \" [--[no-]bitmap] \")\n -+\t \" [--[no-]bitmap] [--[no-]checksum-only] \")\n ++\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n\n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n -+\t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n -+\t\t\tN_(\"write a MIDX layer without updating the MIDX chain\"),\n -+\t\t\tMIDX_WRITE_CHECKSUM_ONLY),\n ++\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n ++\t\t\tN_(\"write the multi-pack-index chain file\"),\n ++\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n +\n -+\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n ++\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n +\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n +\t\terror(_(\"cannot use %s without %s\"),\n -+\t\t \"--checksum-only\", \"--incremental\");\n ++\t\t \"--no-write-chain-file\", \"--incremental\");\n +\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n +\t\t\t\t options);\n +\t}\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_compact(int argc, co\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n -+\t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n -+\t\t\tN_(\"write a MIDX layer without updating the MIDX chain\"),\n -+\t\t\tMIDX_WRITE_CHECKSUM_ONLY),\n ++\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n ++\t\t\tN_(\"write the multi-pack-index chain file\"),\n ++\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_END(),\n \t};\n\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_compact(int argc, co\n \t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n \t\t\t\t options);\n +\n -+\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n ++\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n +\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n +\t\terror(_(\"cannot use %s without %s\"),\n -+\t\t \"--checksum-only\", \"--incremental\");\n ++\t\t \"--no-write-chain-file\", \"--incremental\");\n +\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n +\t\t\t\t options);\n +\t}\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_compact(int argc, co\n \tFREE_AND_NULL(options);\n\n ## midx-write.c ##\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n + \tuint32_t start_pack;\n + \tstruct hashfile *f = NULL;\n +-\tstruct lock_file lk;\n ++\tstruct lock_file lk = LOCK_INIT;\n + \tstruct tempfile *incr;\n + \tstruct write_midx_context ctx = {\n + \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n\n \tif (ctx.incremental) {\n -\t\tstruct strbuf lock_name = STRBUF_INIT;\n -+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n ++\t\tif (!(opts->flags & MIDX_WRITE_NO_CHAIN)) {\n +\t\t\tstruct strbuf lock_name = STRBUF_INIT;\n\n -\t\tget_midx_chain_filename(opts->source, &lock_name);\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n \t\tif (!incr) {\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n - \t}\n - \tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n + \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n + \t\tdie(_(\"too many multi-pack-indexes\"));\n\n -+\tif (opts->flags & MIDX_WRITE_CHECKSUM_ONLY)\n ++\tif (!is_lock_file_locked(&lk))\n +\t\tprintf(\"%s\\n\", hash_to_hex_algop(midx_hash, r->hash_algo));\n ++\telse if (opts->flags & MIDX_WRITE_NO_CHAIN)\n ++\t\tBUG(\"lockfile held with MIDX_WRITE_NO_CHAIN set?\");\n +\n \tif (ctx.incremental) {\n -\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n + \t\tstruct multi_pack_index **layers = NULL;\n + \t\tsize_t layers_nr = 0, layers_alloc = 0;\n\n -\t\tif (!chainf) {\n -\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n -\t\t\tgoto cleanup;\n -+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n ++\t\tif (is_lock_file_locked(&lk)){\n +\t\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n +\t\t\tif (!chainf) {\n +\t\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n\n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n - \t\t\t}\n - \t\t}\n\n --\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n + \t\tfree(layers);\n +\n +-\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n -\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n -+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY))\n -+\t\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n ++\t\tif (is_lock_file_locked(&lk))\n ++\t\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n +\t\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\",\n +\t\t\t\t\tkeep_hashes.v[i]);\n \t} else {\n - \t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n - \t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n + \t\tstrvec_push(&keep_hashes,\n + \t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.m || ctx.base_midx)\n \t\todb_close(ctx.repo->objects);\n\n -\tif (commit_lock_file(&lk) < 0)\n -\t\tdie_errno(_(\"could not write multi-pack-index\"));\n -+\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n ++\tif (is_lock_file_locked(&lk)) {\n +\t\tif (commit_lock_file(&lk) < 0)\n +\t\t\tdie_errno(_(\"could not write multi-pack-index\"));\n\n @@ midx.h: struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n #define MIDX_WRITE_COMPACT (1 << 6)\n -+#define MIDX_WRITE_CHECKSUM_ONLY (1 << 7)\n ++#define MIDX_WRITE_NO_CHAIN (1 << 7)\n\n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n @@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'show object from s\n \tgit cat-file -p 2.2\n '\n\n -+test_expect_success 'write MIDX layer with --checksum-only' '\n -+\ttest_commit checksum-only &&\n ++test_expect_success 'write MIDX layer with --no-write-chain-file' '\n ++\ttest_commit no-write-chain-file &&\n +\tgit repack -d &&\n +\n +\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n -+\t\t--checksum-only)\" &&\n ++\t\t--no-write-chain-file)\" &&\n +\n +\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n +\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n +'\n +\n -+test_expect_success 'write non-incremental MIDX layer with --checksum-only' '\n -+\ttest_must_fail git multi-pack-index write --bitmap --checksum-only 2>err &&\n -+\ttest_grep \"cannot use --checksum-only without --incremental\" err\n ++test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file' '\n ++\ttest_must_fail git multi-pack-index write --bitmap --no-write-chain-file 2>err &&\n ++\ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n +'\n +\n for reuse in false single multi\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with b\n \t)\n '\n\n -+test_expect_success 'MIDX compaction with --checksum-only' '\n -+\tgit init midx-compact-with--checksum-only &&\n ++test_expect_success 'MIDX compaction with --no-write-chain-file' '\n ++\tgit init midx-compact-with--no-write-chain-file &&\n +\t(\n -+\t\tcd midx-compact-with--checksum-only &&\n ++\t\tcd midx-compact-with--no-write-chain-file &&\n +\n +\t\tgit config maintenance.auto false &&\n +\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with b\n +\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n +\n +\t\tlayer=\"$(git multi-pack-index compact --incremental \\\n -+\t\t\t--checksum-only \\\n ++\t\t\t--no-write-chain-file \\\n +\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n +\t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n +\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with b\n +\t\t# After writing the new layer, insert it into the chain\n +\t\t# manually. This is done in order to make $layer visible\n +\t\t# to the read-midx test helper below, and matches what\n -+\t\t# the MIDX command would do without --checksum-only.\n ++\t\t# the MIDX command would do without --no-write-chain-file.\n +\t\t{\n +\t\t\tnth_line 1 \"$midx_chain.bak\" &&\n +\t\t\techo $layer &&\n 6: 6b42d284b79 ! 6: 48893baa339 midx: support custom `--base` for incremental MIDX writes\n @@ Commit message\n If no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\n immediate parent in the chain, and `write` appends to the existing tip.\n\n - For the `write` subcommand, `--base` requires `--checksum-only`. A plain\n + For the `write` subcommand, `--base` requires `--no-write-chain-file`. A plain\n `write --incremental` appends a new layer to the live chain tip with no\n mechanism to atomically replace it; overriding the base would produce a\n layer that does not extend the tip, breaking chain invariants. With\n - `--checksum-only` the chain is left unmodified and the caller is\n + `--no-write-chain-file` the chain is left unmodified and the caller is\n responsible for assembling a valid chain.\n\n For `compact`, no such restriction applies. The compaction operation\n @@ Documentation/git-multi-pack-index.adoc\n @@ Documentation/git-multi-pack-index.adoc: SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n - \t\t [--refs-snapshot=] [--[no-]checksum-only]\n + \t\t [--refs-snapshot=] [--[no-]write-chain-file]\n +\t\t\t [--base=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n --\t\t [--[no-]bitmap] [--[no-]checksum-only] \n -+\t\t [--[no-]bitmap] [--base=] [--[no-]checksum-only]\n +-\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n ++\t\t [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\n +\t\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n @@ Documentation/git-multi-pack-index.adoc: marker).\n - \t\tand packs not present in an existing MIDX layer.\n - \t\tMigrates non-incremental MIDXs to incremental ones when\n - \t\tnecessary.\n + \t\tThe checksum of the new layer is printed to standard\n + \t\toutput, allowing the caller to assemble and write the\n + \t\tchain itself. Requires `--incremental`.\n +\n +\t--base=::\n +\t\tSpecify the checksum of an existing MIDX layer to use\n +\t\tas the base when writing a new incremental layer.\n +\t\tThe special value `none` indicates that the new layer\n +\t\tshould have no base (i.e., it becomes a root layer).\n -+\t\tRequires `--checksum-only`.\n ++\t\tRequires `--no-write-chain-file`.\n --\n\n compact::\n @@ Documentation/git-multi-pack-index.adoc: compact::\n -\n - \t--[no-]bitmap::\n - \t\tControl whether or not a multi-pack bitmap is written.\n + \t\tMIDX layer but do not update the multi-pack-index-chain\n + \t\tfile. The checksum of the new layer is printed to\n + \t\tstandard output. Requires `--incremental`.\n +\n +\t--base=::\n +\t\tSpecify the checksum of an existing MIDX layer to use\n @@ builtin/multi-pack-index.c\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n --\t \" [--refs-snapshot=] [--[no-]checksum-only]\")\n -+\t \" [--refs-snapshot=] [--[no-]checksum-only]\\n\" \\\n +-\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n ++\t \" [--refs-snapshot=] [--[no-]write-chain-file]\\n\" \\\n +\t \" [--base=]\")\n\n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n --\t \" [--[no-]bitmap] [--[no-]checksum-only] \")\n -+\t \" [--[no-]bitmap] [--base=] [--[no-]checksum-only]\\n\" \\\n +-\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n ++\t \" [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\\n\" \\\n +\t \" \")\n\n #define BUILTIN_MIDX_VERIFY_USAGE \\\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n +\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n - \t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n + \t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t\t options);\n \t}\n\n +\tif (opts.incremental_base &&\n -+\t !(opts.flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n -+\t\terror(_(\"cannot use --base without --checksum-only\"));\n ++\t !(opts.flags & MIDX_WRITE_NO_CHAIN)) {\n ++\t\terror(_(\"cannot use --base without --no-write-chain-file\"));\n +\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n +\t\t\t\t options);\n +\t}\n @@ midx.h: int write_midx_file(struct odb_source *source,\n int verify_midx_file(struct odb_source *source, unsigned flags);\n\n ## t/t5334-incremental-multi-pack-index.sh ##\n -@@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'write non-incremental MIDX layer with --checksum-only' '\n - \ttest_grep \"cannot use --checksum-only without --incremental\" err\n +@@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file\n + \ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n '\n\n -+test_expect_success 'write MIDX layer with --base without --checksum-only' '\n ++test_expect_success 'write MIDX layer with --base without --no-write-chain-file' '\n +\ttest_must_fail git multi-pack-index write --bitmap --incremental \\\n +\t\t--base=none 2>err &&\n -+\ttest_grep \"cannot use --base without --checksum-only\" err\n ++\ttest_grep \"cannot use --base without --no-write-chain-file\" err\n +'\n +\n -+test_expect_success 'write MIDX layer with --base=none and --checksum-only' '\n ++test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '\n +\ttest_commit base-none &&\n +\tgit repack -d &&\n +\n +\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n -+\t\t--checksum-only --base=none)\" &&\n ++\t\t--no-write-chain-file --base=none)\" &&\n +\n +\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n +\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n +'\n +\n -+test_expect_success 'write MIDX layer with --base= and --checksum-only' '\n ++test_expect_success 'write MIDX layer with --base= and --no-write-chain-file' '\n +\ttest_commit base-hash &&\n +\tgit repack -d &&\n +\n +\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n -+\t\t--checksum-only --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n ++\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n +\n +\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n +\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n @@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'write non-incremen\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\n\n ## t/t5335-compact-multi-pack-index.sh ##\n -@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --checksum-only' '\n +@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --no-write-chain-file' '\n\n \t\tlayer=\"$(git multi-pack-index compact --incremental \\\n - \t\t\t--checksum-only \\\n + \t\t\t--no-write-chain-file \\\n +\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n\n -@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --checksum-only' '\n +@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \t)\n '\n\n 7: 2d377f53407 = 7: b4bfaa699ba repack: track the ODB source via existing_packs\n 8: 6af23849f1d = 8: 901f1401bb1 midx: expose `midx_layer_contains_pack()`\n 9: 5c9fea0fc78 = 9: dced4f38fda repack-midx: factor out `repack_prepare_midx_command()`\n10: cd5eba0f665 = 10: 42a70e4edbe repack-midx: extract `repack_fill_midx_stdin_packs()`\n11: d2c667e8413 = 11: 6eb1277edca repack-geometry: prepare for incremental MIDX repacking\n12: 56dc6d55974 = 12: a1272a25fca builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`\n13: e304e316ac4 = 13: f0c3152c085 packfile: ensure `close_pack_revindex()` frees in-memory revindex\n14: 8d5aa3d6fa7 ! 14: 961413b89f3 repack: implement incremental MIDX repacking\n @@ repack-midx.c: static int write_midx_included_packs(struct repack_write_midx_opt\n +\t}\n +\n +\trepack_prepare_midx_command(&cmd, opts, \"write\");\n -+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--checksum-only\", NULL);\n ++\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\", NULL);\n +\tstrvec_pushf(&cmd.args, \"--base=%s\", base ? base : \"none\");\n +\n +\tif (preferred_pack) {\n @@ repack-midx.c: static int write_midx_included_packs(struct repack_write_midx_opt\n +\tint ret;\n +\n +\trepack_prepare_midx_command(&cmd, opts, \"compact\");\n -+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--checksum-only\",\n ++\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\",\n +\t\t midx_get_checksum_hex(step->u.compact.from),\n +\t\t midx_get_checksum_hex(step->u.compact.to), NULL);\n +\n15: e8e22d7958d = 15: eade33bf8cb repack: introduce `--write-midx=incremental`\n16: 79d5a12a390 = 16: e674ba7741b repack: allow `--write-midx=incremental` without `--geometric`\n--- >8 ---\n\n\nThanks,\nTaylor\n"},{"id":"541785","messageId":"CABPp-BGhXzo7yLQ5z_pPztSNf4vEiPM2x2pMxty99rak_NU-CQ@mail.gmail.com","threadId":"65383","inReplyTo":"d2c667e841300c3310e3a23493ac27eb2b08992b.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 11/16] repack-geometry: prepare for incremental MIDX repacking","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2026-04-16T22:51:32Z","receivedAt":"2026-04-16T22:51:45Z","isPatch":true,"body":"Hi,\n\nOn Sun, Mar 29, 2026 at 2:41 PM Taylor Blau wrote:\n>\n> ...If the tip layer has fewer packs\n> than the threshold, those packs are excluded entirely; otherwise only\n> packs in that layer participate in the geometric repack.\n\n\"fewer packs than the threshold\"; I'll come back to this below...\n\n> repo_for_each_pack(existing->repo, p) {\n> + if (geometry->midx_layer_threshold_set && m &&\n> + p->multi_pack_index) {\n> + /*\n> + * When writing MIDX layers incrementally,\n> + * ignore packs unless they are in the most\n> + * recent MIDX layer *and* there are at least\n> + * 'midx_layer_threshold' packs in that layer.\n> + *\n> + * Otherwise 'p' is either in an older layer, or\n> + * the youngest layer does not have enough packs\n> + * to consider its packs as candidates for\n> + * repacking. In either of those cases we want\n> + * to ignore the pack.\n> + */\n> + if (m->num_packs > geometry->midx_layer_threshold &&\n> + midx_layer_contains_pack(m, pack_basename(p)))\n> + ;\n> + else\n> + continue;\n\nShould the commit message use \"no more packs than\" rather than \"fewer\npacks than\"? Alternatively, should this be >= rather than >? Or am\nI just mis-reading?\n\nAlso, the empty-if looks a bit weird; would it make sense to invert\nthe condition to get rid of the empty-if? In other words, something\nlike:\n\n if (m->num_packs <= geometry->midx_layer_threshold ||\n !midx_layer_contains_pack(m, pack_basename(p)))\n continue;\n\n> @@ -173,6 +196,20 @@ void pack_geometry_split(struct pack_geometry *geometry)\n> geometry->promisor_split = compute_pack_geometry_split(geometry->promisor_pack,\n> geometry->promisor_pack_nr,\n> geometry->split_factor);\n> + for (uint32_t i = 0; i < geometry->split; i++) {\n> + struct packed_git *p = geometry->pack[i];\n> + /*\n> + * During incremental MIDX/bitmap repacking, any packs\n> + * included in the rollup are either (a) not MIDX'd, or\n> + * (b) contained in the tip layer iff it has more than\n> + * the threshold number of packs.\n\nThis wording looks consistent with the previous check, but not the\ncommit message. If the commit message is deemed correct, then you'd\nprobably need to update not only the above comparison but this comment\nas well.\n"},{"id":"541786","messageId":"CABPp-BHuj8UKG3Mh9opniHMrgNkKiWLKv7pqVAex3PeAX1+Ydw@mail.gmail.com","threadId":"65383","inReplyTo":"8d5aa3d6fa70d408fc02561f2203ae1a48186c91.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 14/16] repack: implement incremental MIDX repacking","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2026-04-16T22:53:10Z","receivedAt":"2026-04-16T22:53:22Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 2:41 PM Taylor Blau wrote:\n>\n> At present, this functionality is exposed behind a new enum value,\n> `REPACK_WRITE_MIDX_INCREMENTAL`, but has no external callers. A\n> subsequent commit will expose this mode via `git repack\n> --write-midx=incremental`.\n\nNoted, although...\n\n> + /*\n> + * Compact additional MIDX layers into this proposed one until\n> + * the merging condition is violated.\n> + */\n> + while (m) {\n> + uint32_t preferred_pack_idx;\n> +\n> + trace2_data_string(\"repack\", opts->existing->repo,\n> + \"candidate\", midx_get_checksum_hex(m));\n> +\n> + if (step.objects_nr < m->num_objects / opts->midx_split_factor) {\n\nHere the new value is used...\n\n> + if (proposed_objects_nr < next->base_midx->num_objects / opts->midx_split_factor) {\n\n...and here...\n\n> struct repack_write_midx_opts {\n> @@ -148,6 +149,8 @@ struct repack_write_midx_opts {\n> int show_progress;\n> int write_bitmaps;\n> int midx_must_contain_cruft;\n> + int midx_split_factor;\n> + int midx_new_layer_threshold;\n> enum repack_write_midx_mode mode;\n> };\n\n...and here it is defined. There is also another new value defined\nhere (midx_new_layer_threshold) that isn't used anywhere in this patch\nyet, but I'll focus on midx_split_factor.\n\nAlthough, midx_split_factor is defined here, it isn't explicitly\ninitialized anywhere. As far as I can tell, it's only initialized\nthrough a designated initializer in builtin/repack.c with\nmidx_split_factor not appearing as one of the designated fields and\nthus getting a value of 0. That would result in a divide by\nzero...although, as you noted in the commit message, this code isn't\nyet active.\n\nStill, would it make sense to initialize opts.midx_split_factor in\nbuiltin/repack.c to something non-zero as part of this patch?\n"},{"id":"541787","messageId":"CABPp-BH-Wd5vwxHCx8BXJvCANxTzD=54DzB+YCME7bL+xnQG_g@mail.gmail.com","threadId":"65383","inReplyTo":"e8e22d7958d513c3b7d4165857392a19277df65e.1774820449.git.me@ttaylorr.com","subject":"Re: [PATCH 15/16] repack: introduce `--write-midx=incremental`","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2026-04-16T22:53:28Z","receivedAt":"2026-04-16T22:53:43Z","isPatch":true,"body":"On Sun, Mar 29, 2026 at 2:41 PM Taylor Blau wrote:\n>\n> + if (!strcmp(var, \"repack.midxsplitfactor\")) {\n> + repack_ctx->midx_split_factor = git_config_int(var, value,\n> + ctx->kvi);\n> + return 0;\n> + }\n> + if (!strcmp(var, \"repack.midxnewlayerthreshold\")) {\n> + repack_ctx->midx_new_layer_threshold = git_config_int(var, value,\n> + ctx->kvi);\n> + return 0;\n> + }\n\nShould there be any sanity checking on appropriate values beyond\nis-an-integer, such as requiring midx_split_factor > 1 ?\n"},{"id":"542054","messageId":"aefRKvNO1XXtIe1K@nand.local","threadId":"65383","inReplyTo":"CABPp-BGhXzo7yLQ5z_pPztSNf4vEiPM2x2pMxty99rak_NU-CQ@mail.gmail.com","subject":"Re: [PATCH 11/16] repack-geometry: prepare for incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T19:34:02Z","receivedAt":"2026-04-21T19:34:04Z","isPatch":true,"body":"On Thu, Apr 16, 2026 at 03:51:32PM -0700, Elijah Newren wrote:\n> Hi,\n>\n> On Sun, Mar 29, 2026 at 2:41 PM Taylor Blau wrote:\n> >\n> > ...If the tip layer has fewer packs\n> > than the threshold, those packs are excluded entirely; otherwise only\n> > packs in that layer participate in the geometric repack.\n>\n> \"fewer packs than the threshold\"; I'll come back to this below...\n>\n> > repo_for_each_pack(existing->repo, p) {\n> > + if (geometry->midx_layer_threshold_set && m &&\n> > + p->multi_pack_index) {\n> > + /*\n> > + * When writing MIDX layers incrementally,\n> > + * ignore packs unless they are in the most\n> > + * recent MIDX layer *and* there are at least\n> > + * 'midx_layer_threshold' packs in that layer.\n> > + *\n> > + * Otherwise 'p' is either in an older layer, or\n> > + * the youngest layer does not have enough packs\n> > + * to consider its packs as candidates for\n> > + * repacking. In either of those cases we want\n> > + * to ignore the pack.\n> > + */\n> > + if (m->num_packs > geometry->midx_layer_threshold &&\n> > + midx_layer_contains_pack(m, pack_basename(p)))\n> > + ;\n> > + else\n> > + continue;\n>\n> Should the commit message use \"no more packs than\" rather than \"fewer\n> packs than\"? Alternatively, should this be >= rather than >? Or am\n> I just mis-reading?\n\nI think the goal was to have \"fewer packs than\" so the commit message is\nright here and the implementation is wrong.\n\n> Also, the empty-if looks a bit weird; would it make sense to invert\n> the condition to get rid of the empty-if? In other words, something\n> like:\n>\n> if (m->num_packs <= geometry->midx_layer_threshold ||\n> !midx_layer_contains_pack(m, pack_basename(p)))\n> continue;\n\nI sometimes will use the empty-if style when the conditional is easier\nto explain in the positive than the negative, but reading it out like\nthis I think that the conditional as written here is still clear. Thanks\nfor the suggestion!\n\nWith the above \"no more than\" vs \"fewer than\", I changed this as\nfollows:\n\n--- 8< ---\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex 5b554da89f1..0469dc6496a 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -49,10 +49,8 @@ void pack_geometry_init(struct pack_geometry *geometry,\n \t\t\t * repacking. In either of those cases we want\n \t\t\t * to ignore the pack.\n \t\t\t */\n-\t\t\tif (m->num_packs > geometry->midx_layer_threshold &&\n-\t\t\t midx_layer_contains_pack(m, pack_basename(p)))\n-\t\t\t\t;\n-\t\t\telse\n+\t\t\tif (m->num_packs < geometry->midx_layer_threshold ||\n+\t\t\t !midx_layer_contains_pack(m, pack_basename(p)))\n \t\t\t\tcontinue;\n \t\t}\n--- >8 ---\n\n> > @@ -173,6 +196,20 @@ void pack_geometry_split(struct pack_geometry *geometry)\n> > geometry->promisor_split = compute_pack_geometry_split(geometry->promisor_pack,\n> > geometry->promisor_pack_nr,\n> > geometry->split_factor);\n> > + for (uint32_t i = 0; i < geometry->split; i++) {\n> > + struct packed_git *p = geometry->pack[i];\n> > + /*\n> > + * During incremental MIDX/bitmap repacking, any packs\n> > + * included in the rollup are either (a) not MIDX'd, or\n> > + * (b) contained in the tip layer iff it has more than\n> > + * the threshold number of packs.\n>\n> This wording looks consistent with the previous check, but not the\n> commit message. If the commit message is deemed correct, then you'd\n> probably need to update not only the above comparison but this comment\n> as well.\n\nGreat catch, I would have definitely missed this if you hadn't pointed\nit out.\n\nThanks,\nTaylor\n"},{"id":"542055","messageId":"aefSy92LC3Uo3hNW@nand.local","threadId":"65383","inReplyTo":"CABPp-BHuj8UKG3Mh9opniHMrgNkKiWLKv7pqVAex3PeAX1+Ydw@mail.gmail.com","subject":"Re: [PATCH 14/16] repack: implement incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T19:40:59Z","receivedAt":"2026-04-21T19:41:01Z","isPatch":true,"body":"On Thu, Apr 16, 2026 at 03:53:10PM -0700, Elijah Newren wrote:\n> Although, midx_split_factor is defined here, it isn't explicitly\n> initialized anywhere. As far as I can tell, it's only initialized\n> through a designated initializer in builtin/repack.c with\n> midx_split_factor not appearing as one of the designated fields and\n> thus getting a value of 0. That would result in a divide by\n> zero...although, as you noted in the commit message, this code isn't\n> yet active.\n>\n> Still, would it make sense to initialize opts.midx_split_factor in\n> builtin/repack.c to something non-zero as part of this patch?\n\nGood point. I had avoided initializing it because this code is all dead\nas of this point in the series, and I didn't want to touch the repack\nbuiltin until we actually wire it all up.\n\nBut I think the point you raise here is very fair, so I introduced the\nconstants in this patch and used them in the initializer, then adjust\nthem in the subsequent patch to use the given values instead of the\ndefaults.\n\nThanks,\nTaylor\n"},{"id":"542056","messageId":"aefVj2fSB5M3VWF7@nand.local","threadId":"65383","inReplyTo":"CABPp-BH-Wd5vwxHCx8BXJvCANxTzD=54DzB+YCME7bL+xnQG_g@mail.gmail.com","subject":"Re: [PATCH 15/16] repack: introduce `--write-midx=incremental`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T19:52:47Z","receivedAt":"2026-04-21T19:52:50Z","isPatch":true,"body":"On Thu, Apr 16, 2026 at 03:53:28PM -0700, Elijah Newren wrote:\n> On Sun, Mar 29, 2026 at 2:41 PM Taylor Blau wrote:\n> >\n> > + if (!strcmp(var, \"repack.midxsplitfactor\")) {\n> > + repack_ctx->midx_split_factor = git_config_int(var, value,\n> > + ctx->kvi);\n> > + return 0;\n> > + }\n> > + if (!strcmp(var, \"repack.midxnewlayerthreshold\")) {\n> > + repack_ctx->midx_new_layer_threshold = git_config_int(var, value,\n> > + ctx->kvi);\n> > + return 0;\n> > + }\n>\n> Should there be any sanity checking on appropriate values beyond\n> is-an-integer, such as requiring midx_split_factor > 1 ?\n\nVery reasonable, I added some basic sanity checking for both.\n\nThanks,\nTaylor\n"},{"id":"542068","messageId":"cover.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH v2 00/16] repack: incremental MIDX/bitmap-based repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:08Z","receivedAt":"2026-04-21T20:37:11Z","isPatch":true,"body":"[Note to the maintainer, this is rebased onto current 'master', which is\n94f057755b7 (Git 2.54, 2026-04-19) at the time of writing.]\n\nThis a modest-sized reroll of my series to implement the last remaining\ncomponent of the incremental MIDX/bitmap-based repacking strategy that I\nhave been working on.\n\nThe main changes since last time are described in [1], which I sent to\nthe list last week just before the 2.54 release was tagged. For\nconvenience, a copy of the main changes are below:\n\n - Use a strset instead of a string_list for keeping track of the MIDX\n layers to retain when calling `either clear_midx_files_ext()` or\n `clear_incremental_midx_files_ext()`.\n\n - A new patch to rewrite the logic for determining which MIDX layers\n comprise the new chain via keep_hashes to build the array in order.\n The subsequent patch converts that into a strvec, which no longer\n requires direct manipulation.\n\n - The new \"--checksum-only\" option has been renamed to\n \"--no-write-chain-file\", and various small implementation tweaks\n (e.g., relying on `is_lock_file_locked()` to determine whether we\n should update the chain file as opposed to reading the flags).\n\nAs usual, a range-diff is included below as well for convenience. Thanks\nin advance for reviewing!\n\n[1]: https://lore.kernel.org/git/ad7B7cxKP7tNCXUf@nand.local\n\nTaylor Blau (16):\n midx-write: handle noop writes when converting incremental chains\n midx: use `strset` for retained MIDX files\n midx: build `keep_hashes` array in order\n midx: use `strvec` for `keep_hashes`\n midx: introduce `--no-write-chain-file` for incremental MIDX writes\n midx: support custom `--base` for incremental MIDX writes\n repack: track the ODB source via existing_packs\n midx: expose `midx_layer_contains_pack()`\n repack-midx: factor out `repack_prepare_midx_command()`\n repack-midx: extract `repack_fill_midx_stdin_packs()`\n repack-geometry: prepare for incremental MIDX repacking\n builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`\n packfile: ensure `close_pack_revindex()` frees in-memory revindex\n repack: implement incremental MIDX repacking\n repack: introduce `--write-midx=incremental`\n repack: allow `--write-midx=incremental` without `--geometric`\n\n Documentation/config/repack.adoc | 18 +\n Documentation/git-multi-pack-index.adoc | 32 +-\n Documentation/git-repack.adoc | 44 +-\n builtin/multi-pack-index.c | 48 +-\n builtin/repack.c | 102 +++-\n midx-write.c | 206 ++++---\n midx.c | 104 ++--\n midx.h | 11 +-\n packfile.c | 2 +\n repack-geometry.c | 48 +-\n repack-midx.c | 717 +++++++++++++++++++++++-\n repack.c | 58 +-\n repack.h | 26 +-\n t/meson.build | 1 +\n t/t5334-incremental-multi-pack-index.sh | 63 +++\n t/t5335-compact-multi-pack-index.sh | 113 ++++\n t/t7705-repack-incremental-midx.sh | 525 +++++++++++++++++\n 17 files changed, 1918 insertions(+), 200 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\nRange-diff against v1:\n 1: 0c7a68ca5a7 = 1: d6c27317c25 midx-write: handle noop writes when converting incremental chains\n 2: 1391552dfc6 ! 2: 629c8d23116 midx: use `string_list` for retained MIDX files\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - midx: use `string_list` for retained MIDX files\n + midx: use `strset` for retained MIDX files\n \n Both `clear_midx_files_ext()` and `clear_incremental_midx_files_ext()`\n build a list of filenames to keep while pruning stale MIDX files. Today\n - they hand-roll an array instead of using a `string_list`, thus requiring\n - us to pass an additional length parameter, and makes lookups linear.\n + they hand-roll an array instead of using a `strset`, thus requiring us\n + to pass an additional length parameter, and makes lookups linear.\n \n - Replace the bare array with a `string_list` which can be passed around\n - as a single parameter. Though it improves lookup performance, the\n - difference is likely immeasurable given how small the keep_hashes array\n - typically is.\n + Replace the bare array with a `strset` which can be passed around as a\n + single parameter. Though it improves lookup performance, the difference\n + is likely immeasurable given how small the keep_hashes array typically\n + is.\n \n Signed-off-by: Taylor Blau \n \n @@ midx.c: int midx_checksum_valid(struct multi_pack_index *m)\n struct clear_midx_data {\n -\tchar **keep;\n -\tuint32_t keep_nr;\n -+\tstruct string_list keep;\n ++\tstruct strset keep;\n \tconst char *ext;\n };\n \n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n -\t\tif (!strcmp(data->keep[i], file_name))\n -\t\t\treturn;\n -\t}\n -+\tif (string_list_has_string(&data->keep, file_name))\n ++\tif (strset_contains(&data->keep, file_name))\n +\t\treturn;\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n -\tstruct clear_midx_data data;\n -\tmemset(&data, 0, sizeof(struct clear_midx_data));\n +\tstruct clear_midx_data data = {\n -+\t\t.keep = STRING_LIST_INIT_DUP,\n ++\t\t.keep = STRSET_INIT,\n +\t\t.ext = ext,\n +\t};\n \n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n \n -\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n -\t\tdata.keep_nr = 1;\n -+\t\tstring_list_insert(&data.keep, buf.buf);\n ++\t\tstrset_add(&data.keep, buf.buf);\n +\n +\t\tstrbuf_release(&buf);\n \t}\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n -\tif (keep_hash)\n -\t\tfree(data.keep[0]);\n -\tfree(data.keep);\n -+\tstring_list_clear(&data.keep, 0);\n ++\tstrset_clear(&data.keep);\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n {\n -\tstruct clear_midx_data data;\n +\tstruct clear_midx_data data = {\n -+\t\t.keep = STRING_LIST_INIT_NODUP,\n ++\t\t.keep = STRSET_INIT,\n +\t\t.ext = ext,\n +\t};\n ++\tstruct strbuf buf = STRBUF_INIT;\n \tuint32_t i;\n \n -\tmemset(&data, 0, sizeof(struct clear_midx_data));\n --\n ++\tfor (i = 0; i < hashes_nr; i++) {\n ++\t\tstrbuf_reset(&buf);\n ++\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n ++\t\t\t ext);\n + \n -\tALLOC_ARRAY(data.keep, hashes_nr);\n - \tfor (i = 0; i < hashes_nr; i++)\n +-\tfor (i = 0; i < hashes_nr; i++)\n -\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n -\t\t\t\t ext);\n -\tdata.keep_nr = hashes_nr;\n -\tdata.ext = ext;\n -+\t\tstring_list_append(&data.keep,\n -+\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n -+\t\t\t\t\t keep_hashes[i], ext));\n -+\tstring_list_sort(&data.keep);\n ++\t\tstrset_add(&data.keep, buf.buf);\n ++\t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n -\tfor (i = 0; i < hashes_nr; i++)\n -\t\tfree(data.keep[i]);\n -\tfree(data.keep);\n -+\tstring_list_clear(&data.keep, 0);\n ++\tstrbuf_release(&buf);\n ++\tstrset_clear(&data.keep);\n }\n \n void clear_midx_file(struct repository *r)\n 3: aea6aad6953 ! 3: e303bf6a4ac strvec: introduce `strvec_init_alloc()`\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - strvec: introduce `strvec_init_alloc()`\n + midx: build `keep_hashes` array in order\n \n - When the caller knows upfront how many elements will be pushed onto a\n - `strvec`, it is useful to pre-allocate enough space in the array to fit\n - that many elements (and one additional slot to store NULL, indicating\n - the end of the list.)\n + Instead of filling the keep_hashes array using reverse indexing (e.g.,\n + `keep_hashes[count - i - 1]`) while traversing linked lists forward,\n + collect linked list nodes into a temporary `layers` array and then\n + iterate it backwards to fill `keep_hashes` sequentially.\n \n - Introduce `strvec_init_alloc()`, which allocates the backing array large\n - enough to hold `alloc` elements and the termination marker without\n - further reallocation. Reimplement `strvec_init()` as a special case of\n - `strvec_init_alloc()`, namely when `alloc` is zero.\n + This makes the filling logic easier to follow, since each segment of the\n + array is filled with a simple forward-marching index. Moreover, this\n + change prepares us for a subsequent commit that will switch to using a\n + `strvec`.\n \n - Helped-by: Junio C Hamano \n Signed-off-by: Taylor Blau \n \n - ## strvec.c ##\n -@@ strvec.c: const char *empty_strvec[] = { NULL };\n + ## midx-write.c ##\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n + \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n + \t\tstruct multi_pack_index *m = ctx.base_midx;\n ++\t\tstruct multi_pack_index **layers = NULL;\n ++\t\tsize_t layers_nr = 0, layers_alloc = 0;\n ++\t\tsize_t j = 0;\n \n - void strvec_init(struct strvec *array)\n - {\n --\tstruct strvec blank = STRVEC_INIT;\n --\tmemcpy(array, &blank, sizeof(*array));\n -+\tstrvec_init_alloc(array, 0);\n -+}\n -+\n -+void strvec_init_alloc(struct strvec *array, size_t alloc)\n -+{\n -+\tif (!alloc) {\n -+\t\tstruct strvec blank = STRVEC_INIT;\n -+\t\tmemcpy(array, &blank, sizeof(*array));\n -+\t} else {\n -+\t\tCALLOC_ARRAY(array->v, st_add(alloc, 1));\n -+\t\tarray->nr = 0;\n -+\t\tarray->alloc = alloc + 1;\n -+\t}\n - }\n + \t\tif (!chainf) {\n + \t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \t\tstrbuf_release(&final_midx_name);\n + \n + \t\tif (ctx.compact) {\n +-\t\t\tstruct multi_pack_index *m;\n +-\t\t\tuint32_t num_layers_before_from = 0;\n +-\t\t\tuint32_t i;\n ++\t\t\tstruct multi_pack_index *mp;\n + \n +-\t\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n +-\t\t\t\tnum_layers_before_from++;\n +-\n +-\t\t\tm = ctx.base_midx;\n +-\t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n +-\t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n +-\n +-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n +-\t\t\t\tm = m->base_midx;\n ++\t\t\tfor (mp = ctx.base_midx; mp; mp = mp->base_midx) {\n ++\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n ++\t\t\t\tlayers[layers_nr++] = mp;\n + \t\t\t}\n ++\t\t\twhile (layers_nr)\n ++\t\t\t\tkeep_hashes[j++] =\n ++\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n \n - void strvec_push_nodup(struct strvec *array, char *value)\n -\n - ## strvec.h ##\n -@@ strvec.h: struct strvec {\n - */\n - void strvec_init(struct strvec *);\n +-\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n +-\t\t\t\t\t\t\t\t r->hash_algo));\n ++\t\t\tkeep_hashes[j++] =\n ++\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n ++\t\t\t\t\t\t\t r->hash_algo));\n \n -+/*\n -+ * Initializes an array large enough to store `alloc` elements.\n -+ */\n -+void strvec_init_alloc(struct strvec *, size_t alloc);\n +-\t\t\ti = 0;\n +-\t\t\tfor (m = ctx.m;\n +-\t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n +-\t\t\t m = m->base_midx) {\n +-\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n +-\t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n +-\t\t\t\ti++;\n ++\t\t\tfor (mp = ctx.m;\n ++\t\t\t mp && midx_hashcmp(mp, ctx.compact_to,\n ++\t\t\t\t\t\tr->hash_algo);\n ++\t\t\t mp = mp->base_midx) {\n ++\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n ++\t\t\t\tlayers[layers_nr++] = mp;\n + \t\t\t}\n ++\t\t\twhile (layers_nr)\n ++\t\t\t\tkeep_hashes[j++] =\n ++\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n + \t\t} else {\n +-\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n ++\t\t\tfor (; m; m = m->base_midx) {\n ++\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n ++\t\t\t\tlayers[layers_nr++] = m;\n ++\t\t\t}\n ++\t\t\twhile (layers_nr)\n ++\t\t\t\tkeep_hashes[j++] =\n ++\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n +\n - /* Push a copy of a string onto the end of the array. */\n - const char *strvec_push(struct strvec *, const char *);\n ++\t\t\tkeep_hashes[j++] =\n + \t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n + \t\t\t\t\t\t\t r->hash_algo));\n +-\n +-\t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n +-\t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n +-\n +-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n +-\t\t\t\tm = m->base_midx;\n +-\t\t\t}\n + \t\t}\n \n +-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n ++\t\tASSERT(j == keep_hashes_nr);\n ++\n ++\t\tfree(layers);\n ++\n ++\t\tfor (uint32_t i = 0; i < j; i++)\n + \t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n + \t} else {\n + \t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n 4: 5fc72d5049a ! 4: 42d76c70060 midx: use `strvec` for `keep_hashes`\n @@ Commit message\n value is used to determine the contents of the resulting\n `multi-pack-index-chain` file when writing with \"--incremental\".\n \n + Since the previous commit already builds the array in forward order, the\n + conversion is straightforward: replace indexed assignments with\n + `strvec_push()`, drop the pre-counting and `CALLOC_ARRAY()`, and\n + simplify cleanup via `strvec_clear()`.\n +\n Signed-off-by: Taylor Blau \n \n ## midx-write.c ##\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \tint dropped_packs = 0;\n \tint result = -1;\n -\tconst char **keep_hashes = NULL;\n +-\tsize_t keep_hashes_nr = 0;\n +\tstruct strvec keep_hashes = STRVEC_INIT;\n - \tsize_t keep_hashes_nr = 0;\n \tstruct chunkfile *cf;\n \n + \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n - \t} else {\n - \t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n - \t}\n + \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n + \t\tdie(_(\"too many multi-pack-indexes\"));\n + \n +-\tif (ctx.compact) {\n +-\t\tstruct multi_pack_index *m;\n +-\n +-\t\t/*\n +-\t\t * Keep all MIDX layers excluding those in the range [from, to].\n +-\t\t */\n +-\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n +-\t\t\tkeep_hashes_nr++;\n +-\t\tfor (m = ctx.m;\n +-\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n +-\t\t m = m->base_midx)\n +-\t\t\tkeep_hashes_nr++;\n +-\n +-\t\tkeep_hashes_nr++; /* include the compacted layer */\n +-\t} else {\n +-\t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n +-\t}\n -\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n -+\tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n - \n +-\n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n -@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n - \t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n - \t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n + \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n + \t\tstruct multi_pack_index *m = ctx.base_midx;\n + \t\tstruct multi_pack_index **layers = NULL;\n + \t\tsize_t layers_nr = 0, layers_alloc = 0;\n +-\t\tsize_t j = 0;\n \n --\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.nr++;\n - \t\t\t\tm = m->base_midx;\n + \t\tif (!chainf) {\n + \t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n + \t\t\twhile (layers_nr)\n +-\t\t\t\tkeep_hashes[j++] =\n +-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n ++\t\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \n --\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n -+\t\t\tkeep_hashes.v[i] = xstrdup(hash_to_hex_algop(midx_hash,\n - \t\t\t\t\t\t\t\t r->hash_algo));\n -+\t\t\tkeep_hashes.nr++;\n +-\t\t\tkeep_hashes[j++] =\n +-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n +-\t\t\t\t\t\t\t r->hash_algo));\n ++\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t hash_to_hex_algop(midx_hash,\n ++\t\t\t\t\t\t r->hash_algo));\n \n - \t\t\ti = 0;\n - \t\t\tfor (m = ctx.m;\n - \t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n - \t\t\t m = m->base_midx) {\n --\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n -+\t\t\t\tkeep_hashes.v[keep_hashes_nr - i - 1] =\n - \t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.nr++;\n - \t\t\t\ti++;\n + \t\t\tfor (mp = ctx.m;\n + \t\t\t mp && midx_hashcmp(mp, ctx.compact_to,\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n + \t\t\twhile (layers_nr)\n +-\t\t\t\tkeep_hashes[j++] =\n +-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n ++\t\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \t\t} else {\n --\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n -+\t\t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n - \t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n - \t\t\t\t\t\t\t r->hash_algo));\n -+\t\t\tkeep_hashes.nr++;\n - \n - \t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n - \t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n - \n --\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.v[j] = xstrdup(midx_get_checksum_hex(m));\n -+\t\t\t\tkeep_hashes.nr++;\n - \t\t\t\tm = m->base_midx;\n + \t\t\tfor (; m; m = m->base_midx) {\n + \t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n + \t\t\t\tlayers[layers_nr++] = m;\n \t\t\t}\n + \t\t\twhile (layers_nr)\n +-\t\t\t\tkeep_hashes[j++] =\n +-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n ++\t\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n + \n +-\t\t\tkeep_hashes[j++] =\n +-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n +-\t\t\t\t\t\t\t r->hash_algo));\n ++\t\t\tstrvec_push(&keep_hashes,\n ++\t\t\t\t hash_to_hex_algop(midx_hash,\n ++\t\t\t\t\t\t r->hash_algo));\n \t\t}\n \n - \t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n +-\t\tASSERT(j == keep_hashes_nr);\n +-\n + \t\tfree(layers);\n + \n +-\t\tfor (uint32_t i = 0; i < j; i++)\n -\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n ++\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n +\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n \t} else {\n -\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n -+\t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n - \t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n -+\t\tkeep_hashes.nr++;\n +-\t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n ++\t\tstrvec_push(&keep_hashes,\n ++\t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n \t}\n \n \tif (ctx.m || ctx.base_midx)\n @@ midx.c: void clear_midx_files_ext(struct odb_source *source, const char *ext,\n +\t\t\t\t const struct strvec *keep_hashes)\n {\n \tstruct clear_midx_data data = {\n --\t\t.keep = STRING_LIST_INIT_NODUP,\n -+\t\t.keep = STRING_LIST_INIT_DUP,\n + \t\t.keep = STRSET_INIT,\n \t\t.ext = ext,\n \t};\n + \tstruct strbuf buf = STRBUF_INIT;\n -\tuint32_t i;\n \n --\tfor (i = 0; i < hashes_nr; i++)\n --\t\tstring_list_append(&data.keep,\n --\t\t\t\t xstrfmt(\"multi-pack-index-%s.%s\",\n --\t\t\t\t\t keep_hashes[i], ext));\n --\tstring_list_sort(&data.keep);\n +-\tfor (i = 0; i < hashes_nr; i++) {\n +-\t\tstrbuf_reset(&buf);\n +-\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n +-\t\t\t ext);\n +\tif (keep_hashes) {\n -+\t\tstruct strbuf buf = STRBUF_INIT;\n +\t\tfor (size_t i = 0; i < keep_hashes->nr; i++) {\n +\t\t\tstrbuf_reset(&buf);\n -+\n +\t\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\",\n +\t\t\t\t keep_hashes->v[i], ext);\n -+\t\t\tstring_list_append(&data.keep, buf.buf);\n + \n +-\t\tstrset_add(&data.keep, buf.buf);\n ++\t\t\tstrset_add(&data.keep, buf.buf);\n +\t\t}\n -+\n -+\t\tstring_list_sort(&data.keep);\n -+\t\tstrbuf_release(&buf);\n -+\t}\n + \t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n - \t\t\t\t clear_midx_file_ext, &data);\n 5: 41cb5471bb9 ! 5: 2c80aa34fac midx: introduce `--checksum-only` for incremental MIDX writes\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - midx: introduce `--checksum-only` for incremental MIDX writes\n + midx: introduce `--no-write-chain-file` for incremental MIDX writes\n \n When writing an incremental MIDX layer, the MIDX machinery writes the\n new layer into the multi-pack-index.d directory and then updates the\n @@ Commit message\n assemble the new MIDX chain itself before writing a new chain file and\n atomically linking it into place).\n \n - Introduce a `--checksum-only` flag that:\n + Introduce a `--no-write-chain-file` flag that:\n \n * writes the new MIDX layer into the multi-pack-index.d directory\n \n @@ Documentation/git-multi-pack-index.adoc: SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n -\t\t [--refs-snapshot=]\n -+\t\t [--refs-snapshot=] [--[no-]checksum-only]\n ++\t\t [--refs-snapshot=] [--[no-]write-chain-file]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n -\t\t [--[no-]bitmap] \n -+\t\t [--[no-]bitmap] [--[no-]checksum-only] \n ++\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n +@@ Documentation/git-multi-pack-index.adoc: marker).\n + \t\tand packs not present in an existing MIDX layer.\n + \t\tMigrates non-incremental MIDXs to incremental ones when\n + \t\tnecessary.\n ++\n ++\t--[no-]write-chain-file::\n ++\t\tWhen used with `--incremental`, write a new MIDX layer\n ++\t\tbut do not update the multi-pack-index-chain file.\n ++\t\tThe checksum of the new layer is printed to standard\n ++\t\toutput, allowing the caller to assemble and write the\n ++\t\tchain itself. Requires `--incremental`.\n + --\n + \n + compact::\n +@@ Documentation/git-multi-pack-index.adoc: compact::\n + \n + \t--[no-]bitmap::\n + \t\tControl whether or not a multi-pack bitmap is written.\n ++\n ++\t--[no-]write-chain-file::\n ++\t\tWhen used with `--incremental`, write a new compacted\n ++\t\tMIDX layer but do not update the multi-pack-index-chain\n ++\t\tfile. The checksum of the new layer is printed to\n ++\t\tstandard output. Requires `--incremental`.\n + --\n + +\n + Note that the compact command requires writing a version-2 midx that\n \n ## builtin/multi-pack-index.c ##\n @@\n @@ builtin/multi-pack-index.c\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n -\t \" [--refs-snapshot=]\")\n -+\t \" [--refs-snapshot=] [--[no-]checksum-only]\")\n ++\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n -\t \" [--[no-]bitmap] \")\n -+\t \" [--[no-]bitmap] [--[no-]checksum-only] \")\n ++\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n -+\t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n -+\t\t\tN_(\"write a MIDX layer without updating the MIDX chain\"),\n -+\t\t\tMIDX_WRITE_CHECKSUM_ONLY),\n ++\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n ++\t\t\tN_(\"write the multi-pack-index chain file\"),\n ++\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n +\n -+\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n ++\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n +\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n +\t\terror(_(\"cannot use %s without %s\"),\n -+\t\t \"--checksum-only\", \"--incremental\");\n ++\t\t \"--no-write-chain-file\", \"--incremental\");\n +\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n +\t\t\t\t options);\n +\t}\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_compact(int argc, co\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n -+\t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n -+\t\t\tN_(\"write a MIDX layer without updating the MIDX chain\"),\n -+\t\t\tMIDX_WRITE_CHECKSUM_ONLY),\n ++\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n ++\t\t\tN_(\"write the multi-pack-index chain file\"),\n ++\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_END(),\n \t};\n \n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_compact(int argc, co\n \t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n \t\t\t\t options);\n +\n -+\tif (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&\n ++\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n +\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n +\t\terror(_(\"cannot use %s without %s\"),\n -+\t\t \"--checksum-only\", \"--incremental\");\n ++\t\t \"--no-write-chain-file\", \"--incremental\");\n +\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n +\t\t\t\t options);\n +\t}\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_compact(int argc, co\n \tFREE_AND_NULL(options);\n \n ## midx-write.c ##\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n + \tuint32_t start_pack;\n + \tstruct hashfile *f = NULL;\n +-\tstruct lock_file lk;\n ++\tstruct lock_file lk = LOCK_INIT;\n + \tstruct tempfile *incr;\n + \tstruct write_midx_context ctx = {\n + \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \n \tif (ctx.incremental) {\n -\t\tstruct strbuf lock_name = STRBUF_INIT;\n -+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n ++\t\tif (!(opts->flags & MIDX_WRITE_NO_CHAIN)) {\n +\t\t\tstruct strbuf lock_name = STRBUF_INIT;\n \n -\t\tget_midx_chain_filename(opts->source, &lock_name);\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n \t\tif (!incr) {\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n - \t}\n - \tstrvec_init_alloc(&keep_hashes, keep_hashes_nr);\n + \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n + \t\tdie(_(\"too many multi-pack-indexes\"));\n \n -+\tif (opts->flags & MIDX_WRITE_CHECKSUM_ONLY)\n ++\tif (!is_lock_file_locked(&lk))\n +\t\tprintf(\"%s\\n\", hash_to_hex_algop(midx_hash, r->hash_algo));\n ++\telse if (opts->flags & MIDX_WRITE_NO_CHAIN)\n ++\t\tBUG(\"lockfile held with MIDX_WRITE_NO_CHAIN set?\");\n +\n \tif (ctx.incremental) {\n -\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n + \t\tstruct multi_pack_index **layers = NULL;\n + \t\tsize_t layers_nr = 0, layers_alloc = 0;\n \n -\t\tif (!chainf) {\n -\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n -\t\t\tgoto cleanup;\n -+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n ++\t\tif (is_lock_file_locked(&lk)){\n +\t\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n +\t\t\tif (!chainf) {\n +\t\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n - \t\t\t}\n - \t\t}\n \n --\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n + \t\tfree(layers);\n + \n +-\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n -\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n -+\t\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY))\n -+\t\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n ++\t\tif (is_lock_file_locked(&lk))\n ++\t\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n +\t\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\",\n +\t\t\t\t\tkeep_hashes.v[i]);\n \t} else {\n - \t\tkeep_hashes.v[ctx.num_multi_pack_indexes_before] =\n - \t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n + \t\tstrvec_push(&keep_hashes,\n + \t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.m || ctx.base_midx)\n \t\todb_close(ctx.repo->objects);\n \n -\tif (commit_lock_file(&lk) < 0)\n -\t\tdie_errno(_(\"could not write multi-pack-index\"));\n -+\tif (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n ++\tif (is_lock_file_locked(&lk)) {\n +\t\tif (commit_lock_file(&lk) < 0)\n +\t\t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n @@ midx.h: struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n #define MIDX_WRITE_COMPACT (1 << 6)\n -+#define MIDX_WRITE_CHECKSUM_ONLY (1 << 7)\n ++#define MIDX_WRITE_NO_CHAIN (1 << 7)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n @@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'show object from s\n \tgit cat-file -p 2.2\n '\n \n -+test_expect_success 'write MIDX layer with --checksum-only' '\n -+\ttest_commit checksum-only &&\n ++test_expect_success 'write MIDX layer with --no-write-chain-file' '\n ++\ttest_commit no-write-chain-file &&\n +\tgit repack -d &&\n +\n +\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n -+\t\t--checksum-only)\" &&\n ++\t\t--no-write-chain-file)\" &&\n +\n +\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n +\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n +'\n +\n -+test_expect_success 'write non-incremental MIDX layer with --checksum-only' '\n -+\ttest_must_fail git multi-pack-index write --bitmap --checksum-only 2>err &&\n -+\ttest_grep \"cannot use --checksum-only without --incremental\" err\n ++test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file' '\n ++\ttest_must_fail git multi-pack-index write --bitmap --no-write-chain-file 2>err &&\n ++\ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n +'\n +\n for reuse in false single multi\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with b\n \t)\n '\n \n -+test_expect_success 'MIDX compaction with --checksum-only' '\n -+\tgit init midx-compact-with--checksum-only &&\n ++test_expect_success 'MIDX compaction with --no-write-chain-file' '\n ++\tgit init midx-compact-with--no-write-chain-file &&\n +\t(\n -+\t\tcd midx-compact-with--checksum-only &&\n ++\t\tcd midx-compact-with--no-write-chain-file &&\n +\n +\t\tgit config maintenance.auto false &&\n +\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with b\n +\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n +\n +\t\tlayer=\"$(git multi-pack-index compact --incremental \\\n -+\t\t\t--checksum-only \\\n ++\t\t\t--no-write-chain-file \\\n +\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n +\t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n +\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with b\n +\t\t# After writing the new layer, insert it into the chain\n +\t\t# manually. This is done in order to make $layer visible\n +\t\t# to the read-midx test helper below, and matches what\n -+\t\t# the MIDX command would do without --checksum-only.\n ++\t\t# the MIDX command would do without --no-write-chain-file.\n +\t\t{\n +\t\t\tnth_line 1 \"$midx_chain.bak\" &&\n +\t\t\techo $layer &&\n 6: 6b42d284b79 ! 6: 2a05f4b86f3 midx: support custom `--base` for incremental MIDX writes\n @@ Commit message\n If no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\n immediate parent in the chain, and `write` appends to the existing tip.\n \n - For the `write` subcommand, `--base` requires `--checksum-only`. A plain\n + For the `write` subcommand, `--base` requires `--no-write-chain-file`. A plain\n `write --incremental` appends a new layer to the live chain tip with no\n mechanism to atomically replace it; overriding the base would produce a\n layer that does not extend the tip, breaking chain invariants. With\n - `--checksum-only` the chain is left unmodified and the caller is\n + `--no-write-chain-file` the chain is left unmodified and the caller is\n responsible for assembling a valid chain.\n \n For `compact`, no such restriction applies. The compaction operation\n @@ Documentation/git-multi-pack-index.adoc\n @@ Documentation/git-multi-pack-index.adoc: SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n - \t\t [--refs-snapshot=] [--[no-]checksum-only]\n + \t\t [--refs-snapshot=] [--[no-]write-chain-file]\n +\t\t\t [--base=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n --\t\t [--[no-]bitmap] [--[no-]checksum-only] \n -+\t\t [--[no-]bitmap] [--base=] [--[no-]checksum-only]\n +-\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n ++\t\t [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\n +\t\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n @@ Documentation/git-multi-pack-index.adoc: marker).\n - \t\tand packs not present in an existing MIDX layer.\n - \t\tMigrates non-incremental MIDXs to incremental ones when\n - \t\tnecessary.\n + \t\tThe checksum of the new layer is printed to standard\n + \t\toutput, allowing the caller to assemble and write the\n + \t\tchain itself. Requires `--incremental`.\n +\n +\t--base=::\n +\t\tSpecify the checksum of an existing MIDX layer to use\n +\t\tas the base when writing a new incremental layer.\n +\t\tThe special value `none` indicates that the new layer\n +\t\tshould have no base (i.e., it becomes a root layer).\n -+\t\tRequires `--checksum-only`.\n ++\t\tRequires `--no-write-chain-file`.\n --\n \n compact::\n @@ Documentation/git-multi-pack-index.adoc: compact::\n - \n - \t--[no-]bitmap::\n - \t\tControl whether or not a multi-pack bitmap is written.\n + \t\tMIDX layer but do not update the multi-pack-index-chain\n + \t\tfile. The checksum of the new layer is printed to\n + \t\tstandard output. Requires `--incremental`.\n +\n +\t--base=::\n +\t\tSpecify the checksum of an existing MIDX layer to use\n @@ Documentation/git-multi-pack-index.adoc: compact::\n +\t\tthe immediate parent of ``. The special value\n +\t\t`none` indicates that the result should have no base.\n --\n - \n - verify::\n + +\n + Note that the compact command requires writing a version-2 midx that\n \n ## builtin/multi-pack-index.c ##\n @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n --\t \" [--refs-snapshot=] [--[no-]checksum-only]\")\n -+\t \" [--refs-snapshot=] [--[no-]checksum-only]\\n\" \\\n +-\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n ++\t \" [--refs-snapshot=] [--[no-]write-chain-file]\\n\" \\\n +\t \" [--base=]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n --\t \" [--[no-]bitmap] [--[no-]checksum-only] \")\n -+\t \" [--[no-]bitmap] [--base=] [--[no-]checksum-only]\\n\" \\\n +-\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n ++\t \" [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\\n\" \\\n +\t \" \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n +\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n - \t\tOPT_BIT(0, \"checksum-only\", &opts.flags,\n + \t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t\t options);\n \t}\n \n +\tif (opts.incremental_base &&\n -+\t !(opts.flags & MIDX_WRITE_CHECKSUM_ONLY)) {\n -+\t\terror(_(\"cannot use --base without --checksum-only\"));\n ++\t !(opts.flags & MIDX_WRITE_NO_CHAIN)) {\n ++\t\terror(_(\"cannot use --base without --no-write-chain-file\"));\n +\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n +\t\t\t\t options);\n +\t}\n @@ midx.h: int write_midx_file(struct odb_source *source,\n int verify_midx_file(struct odb_source *source, unsigned flags);\n \n ## t/t5334-incremental-multi-pack-index.sh ##\n -@@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'write non-incremental MIDX layer with --checksum-only' '\n - \ttest_grep \"cannot use --checksum-only without --incremental\" err\n +@@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file\n + \ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n '\n \n -+test_expect_success 'write MIDX layer with --base without --checksum-only' '\n ++test_expect_success 'write MIDX layer with --base without --no-write-chain-file' '\n +\ttest_must_fail git multi-pack-index write --bitmap --incremental \\\n +\t\t--base=none 2>err &&\n -+\ttest_grep \"cannot use --base without --checksum-only\" err\n ++\ttest_grep \"cannot use --base without --no-write-chain-file\" err\n +'\n +\n -+test_expect_success 'write MIDX layer with --base=none and --checksum-only' '\n ++test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '\n +\ttest_commit base-none &&\n +\tgit repack -d &&\n +\n +\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n -+\t\t--checksum-only --base=none)\" &&\n ++\t\t--no-write-chain-file --base=none)\" &&\n +\n +\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n +\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n +'\n +\n -+test_expect_success 'write MIDX layer with --base= and --checksum-only' '\n ++test_expect_success 'write MIDX layer with --base= and --no-write-chain-file' '\n +\ttest_commit base-hash &&\n +\tgit repack -d &&\n +\n +\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n -+\t\t--checksum-only --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n ++\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n +\n +\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n +\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n @@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'write non-incremen\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\n \n ## t/t5335-compact-multi-pack-index.sh ##\n -@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --checksum-only' '\n +@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \n \t\tlayer=\"$(git multi-pack-index compact --incremental \\\n - \t\t\t--checksum-only \\\n + \t\t\t--no-write-chain-file \\\n +\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n \n -@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --checksum-only' '\n +@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \t)\n '\n \n 7: 2d377f53407 = 7: 92aba3d366f repack: track the ODB source via existing_packs\n 8: 6af23849f1d = 8: d3ac65c1f11 midx: expose `midx_layer_contains_pack()`\n 9: 5c9fea0fc78 = 9: 1bd2f194c6f repack-midx: factor out `repack_prepare_midx_command()`\n10: cd5eba0f665 = 10: 44f522ea04d repack-midx: extract `repack_fill_midx_stdin_packs()`\n11: d2c667e8413 ! 11: f5642a46bbd repack-geometry: prepare for incremental MIDX repacking\n @@ repack-geometry.c: void pack_geometry_init(struct pack_geometry *geometry,\n +\t\t\t * repacking. In either of those cases we want\n +\t\t\t * to ignore the pack.\n +\t\t\t */\n -+\t\t\tif (m->num_packs > geometry->midx_layer_threshold &&\n -+\t\t\t midx_layer_contains_pack(m, pack_basename(p)))\n -+\t\t\t\t;\n -+\t\t\telse\n ++\t\t\tif (m->num_packs < geometry->midx_layer_threshold ||\n ++\t\t\t !midx_layer_contains_pack(m, pack_basename(p)))\n +\t\t\t\tcontinue;\n +\t\t}\n +\n @@ repack-geometry.c: void pack_geometry_split(struct pack_geometry *geometry)\n +\t\t/*\n +\t\t * During incremental MIDX/bitmap repacking, any packs\n +\t\t * included in the rollup are either (a) not MIDX'd, or\n -+\t\t * (b) contained in the tip layer iff it has more than\n ++\t\t * (b) contained in the tip layer iff it has at least\n +\t\t * the threshold number of packs.\n +\t\t *\n +\t\t * In the latter case, we can safely conclude that the\n12: 56dc6d55974 = 12: 9fdcb253a96 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`\n13: e304e316ac4 = 13: 1e1b957bf12 packfile: ensure `close_pack_revindex()` frees in-memory revindex\n14: 8d5aa3d6fa7 ! 14: 93e152fb6aa repack: implement incremental MIDX repacking\n @@ Commit message\n \n Signed-off-by: Taylor Blau \n \n + ## builtin/repack.c ##\n +@@ builtin/repack.c: static const char incremental_bitmap_conflict_error[] = N_(\n + \"--no-write-bitmap-index or disable the pack.writeBitmaps configuration.\"\n + );\n + \n ++#define DEFAULT_MIDX_SPLIT_FACTOR 2\n ++#define DEFAULT_MIDX_NEW_LAYER_THRESHOLD 8\n ++\n + struct repack_config_ctx {\n + \tstruct pack_objects_args *po_args;\n + \tstruct pack_objects_args *cruft_po_args;\n +@@ builtin/repack.c: int cmd_repack(int argc,\n + \t\t\t.show_progress = show_progress,\n + \t\t\t.write_bitmaps = write_bitmaps > 0,\n + \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n ++\t\t\t.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,\n ++\t\t\t.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,\n + \t\t\t.mode = write_midx,\n + \t\t};\n + \n +\n ## repack-midx.c ##\n @@\n #include \"repack.h\"\n @@ repack-midx.c: static int write_midx_included_packs(struct repack_write_midx_opt\n +\t}\n +\n +\trepack_prepare_midx_command(&cmd, opts, \"write\");\n -+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--checksum-only\", NULL);\n ++\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\", NULL);\n +\tstrvec_pushf(&cmd.args, \"--base=%s\", base ? base : \"none\");\n +\n +\tif (preferred_pack) {\n @@ repack-midx.c: static int write_midx_included_packs(struct repack_write_midx_opt\n +\tint ret;\n +\n +\trepack_prepare_midx_command(&cmd, opts, \"compact\");\n -+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--checksum-only\",\n ++\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\",\n +\t\t midx_get_checksum_hex(step->u.compact.from),\n +\t\t midx_get_checksum_hex(step->u.compact.to), NULL);\n +\n15: e8e22d7958d ! 15: 6119f15d3e8 repack: introduce `--write-midx=incremental`\n @@ Documentation/config/repack.adoc: repack.midxMustContainCruft::\n ++\n +Adjacent layers are merged when the accumulated object count of the\n +newer layer exceeds `1/` of the object count of the next deeper\n -+layer. Defaults to 2.\n ++layer. Must be at least 2. Defaults to 2.\n +\n +repack.midxNewLayerThreshold::\n +\tThe minimum number of packs in the tip MIDX layer before those\n @@ Documentation/config/repack.adoc: repack.midxMustContainCruft::\n ++\n +When the tip layer has fewer packs than this threshold, those packs are\n +excluded from the geometric repack entirely, and are thus left\n -+unmodified. Defaults to 8.\n ++unmodified. Must be at least 1. Defaults to 8.\n \n ## Documentation/git-repack.adoc ##\n @@ Documentation/git-repack.adoc: SYNOPSIS\n @@ builtin/repack.c: static int midx_must_contain_cruft = 1;\n };\n \n @@ builtin/repack.c: static const char incremental_bitmap_conflict_error[] = N_(\n - \"--no-write-bitmap-index or disable the pack.writeBitmaps configuration.\"\n - );\n - \n -+#define DEFAULT_MIDX_SPLIT_FACTOR 2\n -+#define DEFAULT_MIDX_NEW_LAYER_THRESHOLD 8\n -+\n struct repack_config_ctx {\n \tstruct pack_objects_args *po_args;\n \tstruct pack_objects_args *cruft_po_args;\n @@ builtin/repack.c: int cmd_repack(int argc,\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n +@@ builtin/repack.c: int cmd_repack(int argc,\n + \t\twrite_bitmaps = 0;\n + \t}\n + \n ++\tif (config_ctx.midx_split_factor < 2)\n ++\t\tdie(_(\"invalid value for %s: %d\"), \"--midx-split-factor\",\n ++\t\t config_ctx.midx_split_factor);\n ++\tif (config_ctx.midx_new_layer_threshold < 1)\n ++\t\tdie(_(\"invalid value for %s: %d\"), \"--midx-new-layer-threshold\",\n ++\t\t config_ctx.midx_new_layer_threshold);\n ++\n + \tif (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {\n + \t\tstruct strbuf path = STRBUF_INIT;\n + \n @@ builtin/repack.c: int cmd_repack(int argc,\n \tif (geometry.split_factor) {\n \t\tif (pack_everything)\n @@ builtin/repack.c: int cmd_repack(int argc,\n \t\tpack_geometry_init(&geometry, &existing, &po_args);\n \t\tpack_geometry_split(&geometry);\n \t}\n +@@ builtin/repack.c: int cmd_repack(int argc,\n + \t\t\t\t packtmp);\n + \t/* End of pack replacement. */\n + \n +-\tif (delete_redundant && pack_everything & ALL_INTO_ONE)\n ++\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n ++\t\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL)\n ++\t\t\texisting_packs_retain_midx_packs(&existing);\n + \t\texisting_packs_mark_for_deletion(&existing, &names);\n ++\t}\n + \n + \tif (write_midx != REPACK_WRITE_MIDX_NONE) {\n + \t\tstruct repack_write_midx_opts opts = {\n @@ builtin/repack.c: int cmd_repack(int argc,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n +-\t\t\t.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,\n +-\t\t\t.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,\n +\t\t\t.midx_split_factor = config_ctx.midx_split_factor,\n +\t\t\t.midx_new_layer_threshold = config_ctx.midx_new_layer_threshold,\n \t\t\t.mode = write_midx,\n @@ repack.c: void pack_objects_args_release(struct pack_objects_args *args)\n \tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n @@ repack.c: void existing_packs_mark_for_deletion(struct existing_packs *existing,\n + \t\t\t\t\t &existing->cruft_packs);\n + }\n \n ++/*\n ++ * Mark every pack that is referenced by the existing MIDX chain as\n ++ * retained, so that a subsequent call to\n ++ * existing_packs_mark_for_deletion() will not mark them for deletion.\n ++ *\n ++ * This is used when writing an incremental MIDX layer on top of an\n ++ * existing chain: retained layers continue to reference the same\n ++ * packs on disk, so those packs must not be unlinked even if the\n ++ * freshly-written pack supersedes them.\n ++ */\n ++void existing_packs_retain_midx_packs(struct existing_packs *existing)\n ++{\n ++\tstruct string_list_item *item;\n ++\tstruct strbuf buf = STRBUF_INIT;\n ++\n ++\tfor_each_string_list_item(item, &existing->midx_packs) {\n ++\t\tstruct string_list_item *found;\n ++\n ++\t\tstrbuf_reset(&buf);\n ++\t\tstrbuf_addstr(&buf, item->string);\n ++\t\tstrbuf_strip_suffix(&buf, \".pack\");\n ++\t\tstrbuf_strip_suffix(&buf, \".idx\");\n ++\n ++\t\tfound = string_list_lookup(&existing->non_kept_packs, buf.buf);\n ++\t\tif (found)\n ++\t\t\texisting_packs_mark_retained(found);\n ++\n ++\t\tfound = string_list_lookup(&existing->cruft_packs, buf.buf);\n ++\t\tif (found)\n ++\t\t\texisting_packs_mark_retained(found);\n ++\t}\n ++\n ++\tstrbuf_release(&buf);\n ++}\n ++\n static void remove_redundant_packs_1(struct repository *repo,\n \t\t\t\t struct string_list *packs,\n -\t\t\t\t const char *packdir)\n @@ repack.h: void prepare_pack_objects(struct child_process *cmd,\n struct write_pack_opts {\n \tstruct pack_objects_args *po_args;\n @@ repack.h: void existing_packs_retain_cruft(struct existing_packs *existing,\n + \t\t\t\t struct packed_git *cruft);\n void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \t\t\t\t struct string_list *names);\n ++void existing_packs_retain_midx_packs(struct existing_packs *existing);\n void existing_packs_remove_redundant(struct existing_packs *existing,\n -\t\t\t\t const char *packdir);\n +\t\t\t\t const char *packdir,\n @@ t/t7705-repack-incremental-midx.sh (new)\n +\t)\n +'\n +\n ++test_expect_success 'repack -ad --write-midx=incremental is safe' '\n ++\tgit init ad-incremental-midx &&\n ++\t(\n ++\t\tcd ad-incremental-midx &&\n ++\n ++\t\tgit config maintenance.auto false &&\n ++\n ++\t\t# Build a MIDX chain with multiple layers referencing\n ++\t\t# distinct packs.\n ++\t\ttest_commit first &&\n ++\t\tgit repack -d &&\n ++\n ++\t\ttest_commit second &&\n ++\t\tgit repack -d --write-midx=incremental &&\n ++\n ++\t\tgit multi-pack-index verify &&\n ++\t\ttest_line_count = 1 $midx_chain &&\n ++\n ++\t\t# Now do a full -ad repack. The new pack contains all\n ++\t\t# objects, but any retained MIDX layers still reference\n ++\t\t# the now-deleted packs.\n ++\t\ttest_commit third &&\n ++\t\tgit repack -ad --write-midx=incremental &&\n ++\n ++\t\tgit multi-pack-index verify &&\n ++\t\tgit fsck &&\n ++\t\tgit rev-list --all --objects >/dev/null\n ++\t)\n ++'\n ++\n ++test_expect_success 'repack rejects invalid midxSplitFactor' '\n ++\ttest_when_finished \"rm -fr bad-split-factor\" &&\n ++\tgit init bad-split-factor &&\n ++\t(\n ++\t\tcd bad-split-factor &&\n ++\t\ttest_commit base &&\n ++\n ++\t\tfor v in 0 1 -1\n ++\t\tdo\n ++\t\t\ttest_must_fail git -c repack.midxSplitFactor=$v \\\n ++\t\t\t\trepack -d --geometric=2 --write-midx=incremental 2>err &&\n ++\t\t\ttest_grep \"invalid value for --midx-split-factor\" err ||\n ++\t\t\treturn 1\n ++\t\tdone\n ++\t)\n ++'\n ++\n ++test_expect_success 'repack rejects invalid midxNewLayerThreshold' '\n ++\ttest_when_finished \"rm -fr bad-layer-threshold\" &&\n ++\tgit init bad-layer-threshold &&\n ++\t(\n ++\t\tcd bad-layer-threshold &&\n ++\t\ttest_commit base &&\n ++\n ++\t\tfor v in 0 -1\n ++\t\tdo\n ++\t\t\ttest_must_fail git -c repack.midxNewLayerThreshold=$v \\\n ++\t\t\t\trepack -d --geometric=2 --write-midx=incremental 2>err &&\n ++\t\t\ttest_grep \"invalid value for --midx-new-layer-threshold\" err ||\n ++\t\t\treturn 1\n ++\t\tdone\n ++\t)\n ++'\n ++\n +test_done\n16: 79d5a12a390 = 16: d9acef1334a repack: allow `--write-midx=incremental` without `--geometric`\n\nbase-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0\n-- \n2.54.0.9.gb905fd5d0ae\n"},{"id":"542069","messageId":"d6c27317c25895da223906e07b82615afb6f633d.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 01/16] midx-write: handle noop writes when converting incremental chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:14Z","receivedAt":"2026-04-21T20:37:16Z","isPatch":true,"body":"When updating a MIDX, we optimize out writes that will result in an\nidentical MIDX as the one we already have on disk. See b3bab9d2729\n(midx-write: extract function to test whether MIDX needs updating,\n2025-12-10) for more details on exactly which writes are optimized out.\n\nIf `midx_needs_update()` can't rule out any of the obvious cases (e.g.,\nthe checksum is invalid, we're requesting a different version, or\nperforming compaction which always requires an update), then we compare\nthe packs we're writing to the packs we already know about. If there are\nan equal number of packs being written as there are in any existing\nMIDX layer(s), then we compare the packs by their name.\n\nThis comparison fails when we have an incremental MIDX chain with\nat least two layers, since we do not recursively peel through earlier\nlayers, instead treating the `->pack_names` array of the tip MIDX layer\nas containing all `m->num_packs + m->num_packs_in_base` packs.\n\nAdjust this to instead look through the MIDX layers one by one when\ncomparing pack names. While we're at it, fix a typo above in the same\nfunction.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 18 ++++++++++--------\n t/t5334-incremental-multi-pack-index.sh | 16 ++++++++++++++++\n 2 files changed, 26 insertions(+), 8 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex a25cab75aba..9328f65a201 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1152,7 +1152,7 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \n \t/*\n \t * Ensure that we have a valid checksum before consulting the\n-\t * exisiting MIDX in order to determine if we can avoid an\n+\t * existing MIDX in order to determine if we can avoid an\n \t * update.\n \t *\n \t * This is necessary because the given MIDX is loaded directly\n@@ -1208,14 +1208,16 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \t\t\tBUG(\"same pack added twice?\");\n \t}\n \n-\tfor (uint32_t i = 0; i < ctx->nr; i++) {\n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addstr(&buf, midx->pack_names[i]);\n-\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\tfor (struct multi_pack_index *m = midx; m; m = m->base_midx) {\n+\t\tfor (uint32_t i = 0; i < m->num_packs; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, m->pack_names[i]);\n+\t\t\tstrbuf_strip_suffix(&buf, \".idx\");\n \n-\t\tif (!strset_contains(&packs, buf.buf))\n-\t\t\tgoto out;\n-\t\tstrset_remove(&packs, buf.buf);\n+\t\t\tif (!strset_contains(&packs, buf.buf))\n+\t\t\t\tgoto out;\n+\t\t\tstrset_remove(&packs, buf.buf);\n+\t\t}\n \t}\n \n \tneeded = false;\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 99c7d44d8e9..c9f5b4e87aa 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -132,4 +132,20 @@ test_expect_success 'relink existing MIDX layer' '\n \n '\n \n+test_expect_success 'non-incremental write with existing incremental chain' '\n+\tgit init non-incremental-write-with-existing &&\n+\ttest_when_finished \"rm -fr non-incremental-write-with-existing\" &&\n+\n+\t(\n+\t\tcd non-incremental-write-with-existing &&\n+\n+\t\tgit config set maintenance.auto false &&\n+\n+\t\twrite_midx_layer &&\n+\t\twrite_midx_layer &&\n+\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n test_done\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542070","messageId":"629c8d2311647fdde69cc56d946cdd957fa8c16e.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 02/16] midx: use `strset` for retained MIDX files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:17Z","receivedAt":"2026-04-21T20:37:19Z","isPatch":true,"body":"Both `clear_midx_files_ext()` and `clear_incremental_midx_files_ext()`\nbuild a list of filenames to keep while pruning stale MIDX files. Today\nthey hand-roll an array instead of using a `strset`, thus requiring us\nto pass an additional length parameter, and makes lookups linear.\n\nReplace the bare array with a `strset` which can be passed around as a\nsingle parameter. Though it improves lookup performance, the difference\nis likely immeasurable given how small the keep_hashes array typically\nis.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 57 +++++++++++++++++++++++++++------------------------------\n 1 file changed, 27 insertions(+), 30 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 81d6ab11e6e..f75e3c9fa6d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -758,8 +758,7 @@ int midx_checksum_valid(struct multi_pack_index *m)\n }\n \n struct clear_midx_data {\n-\tchar **keep;\n-\tuint32_t keep_nr;\n+\tstruct strset keep;\n \tconst char *ext;\n };\n \n@@ -767,15 +766,12 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\t\t\tconst char *file_name, void *_data)\n {\n \tstruct clear_midx_data *data = _data;\n-\tuint32_t i;\n \n \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n \t ends_with(file_name, data->ext)))\n \t\treturn;\n-\tfor (i = 0; i < data->keep_nr; i++) {\n-\t\tif (!strcmp(data->keep[i], file_name))\n-\t\t\treturn;\n-\t}\n+\tif (strset_contains(&data->keep, file_name))\n+\t\treturn;\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n@@ -783,48 +779,49 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash)\n {\n-\tstruct clear_midx_data data;\n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRSET_INIT,\n+\t\t.ext = ext,\n+\t};\n \n \tif (keep_hash) {\n-\t\tALLOC_ARRAY(data.keep, 1);\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hash, ext);\n \n-\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n-\t\tdata.keep_nr = 1;\n+\t\tstrset_add(&data.keep, buf.buf);\n+\n+\t\tstrbuf_release(&buf);\n \t}\n-\tdata.ext = ext;\n \n-\tfor_each_file_in_pack_dir(source->path,\n-\t\t\t\t clear_midx_file_ext,\n-\t\t\t\t &data);\n+\tfor_each_file_in_pack_dir(source->path, clear_midx_file_ext, &data);\n \n-\tif (keep_hash)\n-\t\tfree(data.keep[0]);\n-\tfree(data.keep);\n+\tstrset_clear(&data.keep);\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t char **keep_hashes,\n \t\t\t\t uint32_t hashes_nr)\n {\n-\tstruct clear_midx_data data;\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRSET_INIT,\n+\t\t.ext = ext,\n+\t};\n+\tstruct strbuf buf = STRBUF_INIT;\n \tuint32_t i;\n \n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\tfor (i = 0; i < hashes_nr; i++) {\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n+\t\t\t ext);\n \n-\tALLOC_ARRAY(data.keep, hashes_nr);\n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n-\t\t\t\t ext);\n-\tdata.keep_nr = hashes_nr;\n-\tdata.ext = ext;\n+\t\tstrset_add(&data.keep, buf.buf);\n+\t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n \n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tfree(data.keep[i]);\n-\tfree(data.keep);\n+\tstrbuf_release(&buf);\n+\tstrset_clear(&data.keep);\n }\n \n void clear_midx_file(struct repository *r)\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542071","messageId":"e303bf6a4ace587259f09755b257b2f5df793af9.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 03/16] midx: build `keep_hashes` array in order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:20Z","receivedAt":"2026-04-21T20:37:22Z","isPatch":true,"body":"Instead of filling the keep_hashes array using reverse indexing (e.g.,\n`keep_hashes[count - i - 1]`) while traversing linked lists forward,\ncollect linked list nodes into a temporary `layers` array and then\niterate it backwards to fill `keep_hashes` sequentially.\n\nThis makes the filling logic easier to follow, since each segment of the\narray is filled with a simple forward-marching index. Moreover, this\nchange prepares us for a subsequent commit that will switch to using a\n`strvec`.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 66 ++++++++++++++++++++++++++++------------------------\n 1 file changed, 36 insertions(+), 30 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 9328f65a201..55c778a97cb 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1731,6 +1731,9 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n+\t\tstruct multi_pack_index **layers = NULL;\n+\t\tsize_t layers_nr = 0, layers_alloc = 0;\n+\t\tsize_t j = 0;\n \n \t\tif (!chainf) {\n \t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n@@ -1751,46 +1754,49 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\tstrbuf_release(&final_midx_name);\n \n \t\tif (ctx.compact) {\n-\t\t\tstruct multi_pack_index *m;\n-\t\t\tuint32_t num_layers_before_from = 0;\n-\t\t\tuint32_t i;\n+\t\t\tstruct multi_pack_index *mp;\n \n-\t\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n-\t\t\t\tnum_layers_before_from++;\n-\n-\t\t\tm = ctx.base_midx;\n-\t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n-\t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n-\n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n-\t\t\t\tm = m->base_midx;\n+\t\t\tfor (mp = ctx.base_midx; mp; mp = mp->base_midx) {\n+\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n+\t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n+\t\t\twhile (layers_nr)\n+\t\t\t\tkeep_hashes[j++] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n \n-\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes[j++] =\n+\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t\t r->hash_algo));\n \n-\t\t\ti = 0;\n-\t\t\tfor (m = ctx.m;\n-\t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n-\t\t\t m = m->base_midx) {\n-\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n-\t\t\t\ti++;\n+\t\t\tfor (mp = ctx.m;\n+\t\t\t mp && midx_hashcmp(mp, ctx.compact_to,\n+\t\t\t\t\t\tr->hash_algo);\n+\t\t\t mp = mp->base_midx) {\n+\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n+\t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n+\t\t\twhile (layers_nr)\n+\t\t\t\tkeep_hashes[j++] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n \t\t} else {\n-\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\tfor (; m; m = m->base_midx) {\n+\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n+\t\t\t\tlayers[layers_nr++] = m;\n+\t\t\t}\n+\t\t\twhile (layers_nr)\n+\t\t\t\tkeep_hashes[j++] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\n+\t\t\tkeep_hashes[j++] =\n \t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n \t\t\t\t\t\t\t r->hash_algo));\n-\n-\t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n-\t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n-\n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n-\t\t\t\tm = m->base_midx;\n-\t\t\t}\n \t\t}\n \n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n+\t\tASSERT(j == keep_hashes_nr);\n+\n+\t\tfree(layers);\n+\n+\t\tfor (uint32_t i = 0; i < j; i++)\n \t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n \t} else {\n \t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542072","messageId":"42d76c7006096125a741faa7edb1286f5ea20612.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:24Z","receivedAt":"2026-04-21T20:37:27Z","isPatch":true,"body":"The `keep_hashes` array in `write_midx_internal()` accumulates the\nchecksums of MIDX files that should be retained when pruning stale\nentries from the MIDX chain. For similar reasons as in a previous\ncommit, rewrite this using a strvec, requiring us to pass one fewer\nparameter.\n\nUnlike the aforementioned previous commit, use a `strvec` instead of a\n`string_list`, which provides a more ergonomic interface to adjust the\nvalues at a particular index. The ordering is important here, as this\nvalue is used to determine the contents of the resulting\n`multi-pack-index-chain` file when writing with \"--incremental\".\n\nSince the previous commit already builds the array in forward order, the\nconversion is straightforward: replace indexed assignments with\n`strvec_push()`, drop the pre-counting and `CALLOC_ARRAY()`, and\nsimplify cleanup via `strvec_clear()`.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 84 ++++++++++++++++++----------------------------------\n midx.c | 20 ++++++-------\n 2 files changed, 38 insertions(+), 66 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 55c778a97cb..5d9409a9741 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -29,8 +29,7 @@ extern void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t const char *keep_hash);\n extern void clear_incremental_midx_files_ext(struct odb_source *source,\n \t\t\t\t\t const char *ext,\n-\t\t\t\t\t const char **keep_hashes,\n-\t\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t\t const struct strvec *keep_hashes);\n extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t\tconst char *idx_name);\n \n@@ -1109,8 +1108,7 @@ static int link_midx_to_chain(struct multi_pack_index *m)\n }\n \n static void clear_midx_files(struct odb_source *source,\n-\t\t\t const char **hashes, uint32_t hashes_nr,\n-\t\t\t unsigned incremental)\n+\t\t\t const struct strvec *hashes, unsigned incremental)\n {\n \t/*\n \t * if incremental:\n@@ -1124,13 +1122,15 @@ static void clear_midx_files(struct odb_source *source,\n \t */\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX };\n-\tuint32_t i, j;\n+\tuint32_t i;\n \n \tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tclear_incremental_midx_files_ext(source, exts[i],\n-\t\t\t\t\t\t hashes, hashes_nr);\n-\t\tfor (j = 0; j < hashes_nr; j++)\n-\t\t\tclear_midx_files_ext(source, exts[i], hashes[j]);\n+\t\tclear_incremental_midx_files_ext(source, exts[i], hashes);\n+\t\tif (hashes) {\n+\t\t\tfor (size_t j = 0; j < hashes->nr; j++)\n+\t\t\t\tclear_midx_files_ext(source, exts[i],\n+\t\t\t\t\t\t hashes->v[j]);\n+\t\t}\n \t}\n \n \tif (incremental)\n@@ -1267,8 +1267,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = -1;\n-\tconst char **keep_hashes = NULL;\n-\tsize_t keep_hashes_nr = 0;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tstruct chunkfile *cf;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n@@ -1708,32 +1707,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n \t\tdie(_(\"too many multi-pack-indexes\"));\n \n-\tif (ctx.compact) {\n-\t\tstruct multi_pack_index *m;\n-\n-\t\t/*\n-\t\t * Keep all MIDX layers excluding those in the range [from, to].\n-\t\t */\n-\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n-\t\t\tkeep_hashes_nr++;\n-\t\tfor (m = ctx.m;\n-\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n-\t\t m = m->base_midx)\n-\t\t\tkeep_hashes_nr++;\n-\n-\t\tkeep_hashes_nr++; /* include the compacted layer */\n-\t} else {\n-\t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n-\t}\n-\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n-\n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\tstruct multi_pack_index **layers = NULL;\n \t\tsize_t layers_nr = 0, layers_alloc = 0;\n-\t\tsize_t j = 0;\n \n \t\tif (!chainf) {\n \t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n@@ -1761,12 +1740,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n \t\t\twhile (layers_nr)\n-\t\t\t\tkeep_hashes[j++] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\t\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \n-\t\t\tkeep_hashes[j++] =\n-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t r->hash_algo));\n \n \t\t\tfor (mp = ctx.m;\n \t\t\t mp && midx_hashcmp(mp, ctx.compact_to,\n@@ -1776,31 +1755,29 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n \t\t\twhile (layers_nr)\n-\t\t\t\tkeep_hashes[j++] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\t\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \t\t} else {\n \t\t\tfor (; m; m = m->base_midx) {\n \t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n \t\t\t\tlayers[layers_nr++] = m;\n \t\t\t}\n \t\t\twhile (layers_nr)\n-\t\t\t\tkeep_hashes[j++] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\t\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \n-\t\t\tkeep_hashes[j++] =\n-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t r->hash_algo));\n \t\t}\n \n-\t\tASSERT(j == keep_hashes_nr);\n-\n \t\tfree(layers);\n \n-\t\tfor (uint32_t i = 0; i < j; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n+\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n+\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n \t} else {\n-\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n-\t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n+\t\tstrvec_push(&keep_hashes,\n+\t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n \t}\n \n \tif (ctx.m || ctx.base_midx)\n@@ -1809,8 +1786,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, keep_hashes, keep_hashes_nr,\n-\t\t\t ctx.incremental);\n+\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n \tresult = 0;\n \n cleanup:\n@@ -1826,11 +1802,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n-\tif (keep_hashes) {\n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n-\t\t\tfree((char *)keep_hashes[i]);\n-\t\tfree(keep_hashes);\n-\t}\n+\tstrvec_clear(&keep_hashes);\n \tstrbuf_release(&midx_name);\n \tclose_midx(midx_to_free);\n \ndiff --git a/midx.c b/midx.c\nindex f75e3c9fa6d..bcb8c999015 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -12,6 +12,7 @@\n #include \"chunk-format.h\"\n #include \"pack-bitmap.h\"\n #include \"pack-revindex.h\"\n+#include \"strvec.h\"\n \n #define MIDX_PACK_ERROR ((void *)(intptr_t)-1)\n \n@@ -19,8 +20,7 @@ int midx_checksum_valid(struct multi_pack_index *m);\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash);\n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t const struct strvec *keep_hashes);\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n@@ -799,22 +799,22 @@ void clear_midx_files_ext(struct odb_source *source, const char *ext,\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr)\n+\t\t\t\t const struct strvec *keep_hashes)\n {\n \tstruct clear_midx_data data = {\n \t\t.keep = STRSET_INIT,\n \t\t.ext = ext,\n \t};\n \tstruct strbuf buf = STRBUF_INIT;\n-\tuint32_t i;\n \n-\tfor (i = 0; i < hashes_nr; i++) {\n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n-\t\t\t ext);\n+\tif (keep_hashes) {\n+\t\tfor (size_t i = 0; i < keep_hashes->nr; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\",\n+\t\t\t\t keep_hashes->v[i], ext);\n \n-\t\tstrset_add(&data.keep, buf.buf);\n+\t\t\tstrset_add(&data.keep, buf.buf);\n+\t\t}\n \t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542073","messageId":"2c80aa34fac76a7755d3cdcd0d9bdc659b5d3795.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 05/16] midx: introduce `--no-write-chain-file` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:27Z","receivedAt":"2026-04-21T20:37:30Z","isPatch":true,"body":"When writing an incremental MIDX layer, the MIDX machinery writes the\nnew layer into the multi-pack-index.d directory and then updates the\nmulti-pack-index-chain file to include the freshly written layer.\n\nFuture callers however may not wish to immediately update the MIDX chain\nitself, preferring instead to write out new layer(s) themselves before\natomically updating the chain. Concretely, the new incremental\nMIDX-based repacking strategy will want to do exactly this (that is,\nassemble the new MIDX chain itself before writing a new chain file and\natomically linking it into place).\n\nIntroduce a `--no-write-chain-file` flag that:\n\n * writes the new MIDX layer into the multi-pack-index.d directory\n\n * prints its checksum\n\n * does not update the multi-pack-index-chain file.\n\nThe MIDX chain file (and thus, the lock protecting it) remain untouched,\nallowing callers to assemble the chain themselves. This flag requires\n`--incremental`, since the notion of a separate layer only makes sense\nfor incremental MIDXs.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 17 ++++++++--\n builtin/multi-pack-index.c | 28 +++++++++++++++--\n midx-write.c | 42 ++++++++++++++++---------\n midx.h | 1 +\n t/t5334-incremental-multi-pack-index.sh | 17 ++++++++++\n t/t5335-compact-multi-pack-index.sh | 36 +++++++++++++++++++++\n 6 files changed, 123 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 3a5aa227784..c26196815e2 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -11,9 +11,9 @@ SYNOPSIS\n [verse]\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n-\t\t [--refs-snapshot=]\n+\t\t [--refs-snapshot=] [--[no-]write-chain-file]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] \n+\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -83,6 +83,13 @@ marker).\n \t\tand packs not present in an existing MIDX layer.\n \t\tMigrates non-incremental MIDXs to incremental ones when\n \t\tnecessary.\n+\n+\t--[no-]write-chain-file::\n+\t\tWhen used with `--incremental`, write a new MIDX layer\n+\t\tbut do not update the multi-pack-index-chain file.\n+\t\tThe checksum of the new layer is printed to standard\n+\t\toutput, allowing the caller to assemble and write the\n+\t\tchain itself. Requires `--incremental`.\n --\n \n compact::\n@@ -97,6 +104,12 @@ compact::\n \n \t--[no-]bitmap::\n \t\tControl whether or not a multi-pack bitmap is written.\n+\n+\t--[no-]write-chain-file::\n+\t\tWhen used with `--incremental`, write a new compacted\n+\t\tMIDX layer but do not update the multi-pack-index-chain\n+\t\tfile. The checksum of the new layer is printed to\n+\t\tstandard output. Requires `--incremental`.\n --\n +\n Note that the compact command requires writing a version-2 midx that\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 0f72d96c02d..f861b4b8394 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -16,11 +16,11 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=]\")\n+\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] \")\n+\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -153,6 +153,9 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n+\t\t\tN_(\"write the multi-pack-index chain file\"),\n+\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n@@ -178,6 +181,15 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \tif (argc)\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--no-write-chain-file\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -221,6 +233,9 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n+\t\t\tN_(\"write the multi-pack-index chain file\"),\n+\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_END(),\n \t};\n \n@@ -239,6 +254,15 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \tif (argc != 2)\n \t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--no-write-chain-file\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(the_repository);\n \n \tFREE_AND_NULL(options);\ndiff --git a/midx-write.c b/midx-write.c\nindex 5d9409a9741..38c898e5ff5 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1257,7 +1257,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t start_pack;\n \tstruct hashfile *f = NULL;\n-\tstruct lock_file lk;\n+\tstruct lock_file lk = LOCK_INIT;\n \tstruct tempfile *incr;\n \tstruct write_midx_context ctx = {\n \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n@@ -1601,11 +1601,14 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \n \tif (ctx.incremental) {\n-\t\tstruct strbuf lock_name = STRBUF_INIT;\n+\t\tif (!(opts->flags & MIDX_WRITE_NO_CHAIN)) {\n+\t\t\tstruct strbuf lock_name = STRBUF_INIT;\n \n-\t\tget_midx_chain_filename(opts->source, &lock_name);\n-\t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n-\t\tstrbuf_release(&lock_name);\n+\t\t\tget_midx_chain_filename(opts->source, &lock_name);\n+\t\t\thold_lock_file_for_update(&lk, lock_name.buf,\n+\t\t\t\t\t\t LOCK_DIE_ON_ERROR);\n+\t\t\tstrbuf_release(&lock_name);\n+\t\t}\n \n \t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n \t\tif (!incr) {\n@@ -1707,16 +1710,23 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n \t\tdie(_(\"too many multi-pack-indexes\"));\n \n+\tif (!is_lock_file_locked(&lk))\n+\t\tprintf(\"%s\\n\", hash_to_hex_algop(midx_hash, r->hash_algo));\n+\telse if (opts->flags & MIDX_WRITE_NO_CHAIN)\n+\t\tBUG(\"lockfile held with MIDX_WRITE_NO_CHAIN set?\");\n+\n \tif (ctx.incremental) {\n-\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\tstruct multi_pack_index **layers = NULL;\n \t\tsize_t layers_nr = 0, layers_alloc = 0;\n \n-\t\tif (!chainf) {\n-\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n-\t\t\tgoto cleanup;\n+\t\tif (is_lock_file_locked(&lk)){\n+\t\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\t\tif (!chainf) {\n+\t\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t}\n \n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n@@ -1773,8 +1783,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t\tfree(layers);\n \n-\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n+\t\tif (is_lock_file_locked(&lk))\n+\t\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n+\t\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\",\n+\t\t\t\t\tkeep_hashes.v[i]);\n \t} else {\n \t\tstrvec_push(&keep_hashes,\n \t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n@@ -1783,10 +1795,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.m || ctx.base_midx)\n \t\todb_close(ctx.repo->objects);\n \n-\tif (commit_lock_file(&lk) < 0)\n-\t\tdie_errno(_(\"could not write multi-pack-index\"));\n+\tif (is_lock_file_locked(&lk)) {\n+\t\tif (commit_lock_file(&lk) < 0)\n+\t\t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t}\n \tresult = 0;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex 08f3728e520..5b193882dcf 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -83,6 +83,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n #define MIDX_WRITE_COMPACT (1 << 6)\n+#define MIDX_WRITE_NO_CHAIN (1 << 7)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex c9f5b4e87aa..66d6894761b 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -96,6 +96,23 @@ test_expect_success 'show object from second pack' '\n \tgit cat-file -p 2.2\n '\n \n+test_expect_success 'write MIDX layer with --no-write-chain-file' '\n+\ttest_commit no-write-chain-file &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--no-write-chain-file)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file' '\n+\ttest_must_fail git multi-pack-index write --bitmap --no-write-chain-file 2>err &&\n+\ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 40f3844282f..1a65d48b62b 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -290,4 +290,40 @@ test_expect_success 'MIDX compaction with bitmaps (non-trivial)' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --no-write-chain-file' '\n+\tgit init midx-compact-with--no-write-chain-file &&\n+\t(\n+\t\tcd midx-compact-with--no-write-chain-file &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\tlayer=\"$(git multi-pack-index compact --incremental \\\n+\t\t\t--no-write-chain-file \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n+\n+\t\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\n+\t\t# After writing the new layer, insert it into the chain\n+\t\t# manually. This is done in order to make $layer visible\n+\t\t# to the read-midx test helper below, and matches what\n+\t\t# the MIDX command would do without --no-write-chain-file.\n+\t\t{\n+\t\t\tnth_line 1 \"$midx_chain.bak\" &&\n+\t\t\techo $layer &&\n+\t\t\tnth_line 4 \"$midx_chain.bak\"\n+\t\t} >$midx_chain &&\n+\n+\t\ttest-tool read-midx $objdir $layer >midx.data &&\n+\t\tgrep \"^pack-B-.*\\.idx\" midx.data &&\n+\t\tgrep \"^pack-C-.*\\.idx\" midx.data\n+\n+\t)\n+'\n+\n test_done\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542074","messageId":"2a05f4b86f30ccd1f54e65c5da2d9234d405cbec.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:30Z","receivedAt":"2026-04-21T20:37:33Z","isPatch":true,"body":"Both `compact` and `write --incremental` fix the base of the resulting\nMIDX layer: `compact` always places the compacted result on top of\n\"from's\" immediate parent in the chain, and `write --incremental` always\nappends a new layer to the existing tip. In both cases the base is not\nconfigurable.\n\nFuture callers need additional flexibility. For instance, the incremental\nMIDX-based repacking code may wish to write a layer based on some\nintermediate ancestor rather than the current tip, or produce a root\nlayer when replacing the bottommost entries in the chain.\n\nIntroduce a new `--base` option for both subcommands to specify the\nchecksum of the MIDX layer to use as the base. The given checksum must\nrefer to a valid layer in the MIDX chain that is an ancestor of the\ntopmost layer being written or compacted.\n\nThe special value \"none\" is accepted to produce a root layer with no\nparent. This will be needed when the incremental repacking machinery\ndetermines that the bottommost layers of the chain should be replaced.\n\nIf no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\nimmediate parent in the chain, and `write` appends to the existing tip.\n\nFor the `write` subcommand, `--base` requires `--no-write-chain-file`. A plain\n`write --incremental` appends a new layer to the live chain tip with no\nmechanism to atomically replace it; overriding the base would produce a\nlayer that does not extend the tip, breaking chain invariants. With\n`--no-write-chain-file` the chain is left unmodified and the caller is\nresponsible for assembling a valid chain.\n\nFor `compact`, no such restriction applies. The compaction operation\natomically replaces the compacted range in the chain file, so writing\nthe result on top of any valid ancestor preserves chain invariants.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 17 +++++-\n builtin/multi-pack-index.c | 24 ++++++--\n midx-write.c | 34 ++++++++++-\n midx.h | 5 +-\n t/t5334-incremental-multi-pack-index.sh | 30 ++++++++++\n t/t5335-compact-multi-pack-index.sh | 77 +++++++++++++++++++++++++\n 6 files changed, 178 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex c26196815e2..c6d23aeeb9a 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -12,8 +12,10 @@ SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=] [--[no-]write-chain-file]\n+\t\t\t [--base=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n+\t\t [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\n+\t\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -90,6 +92,13 @@ marker).\n \t\tThe checksum of the new layer is printed to standard\n \t\toutput, allowing the caller to assemble and write the\n \t\tchain itself. Requires `--incremental`.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base when writing a new incremental layer.\n+\t\tThe special value `none` indicates that the new layer\n+\t\tshould have no base (i.e., it becomes a root layer).\n+\t\tRequires `--no-write-chain-file`.\n --\n \n compact::\n@@ -110,6 +119,12 @@ compact::\n \t\tMIDX layer but do not update the multi-pack-index-chain\n \t\tfile. The checksum of the new layer is printed to\n \t\tstandard output. Requires `--incremental`.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base for the compacted result, instead of using\n+\t\tthe immediate parent of ``. The special value\n+\t\t`none` indicates that the result should have no base.\n --\n +\n Note that the compact command requires writing a version-2 midx that\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex f861b4b8394..00ffb36394d 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -16,11 +16,13 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n+\t \" [--refs-snapshot=] [--[no-]write-chain-file]\\n\" \\\n+\t \" [--base=]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n+\t \" [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\\n\" \\\n+\t \" \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -63,6 +65,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tchar *object_dir;\n \tconst char *preferred_pack;\n+\tconst char *incremental_base;\n \tchar *refs_snapshot;\n \tunsigned long batch_size;\n \tunsigned flags;\n@@ -151,6 +154,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n@@ -190,6 +195,13 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t\t options);\n \t}\n \n+\tif (opts.incremental_base &&\n+\t !(opts.flags & MIDX_WRITE_NO_CHAIN)) {\n+\t\terror(_(\"cannot use --base without --no-write-chain-file\"));\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -201,7 +213,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \n \t\tret = write_midx_file_only(source, &packs,\n \t\t\t\t\t opts.preferred_pack,\n-\t\t\t\t\t opts.refs_snapshot, opts.flags);\n+\t\t\t\t\t opts.refs_snapshot,\n+\t\t\t\t\t opts.incremental_base, opts.flags);\n \n \t\tstring_list_clear(&packs, 0);\n \t\tfree(opts.refs_snapshot);\n@@ -229,6 +242,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n@@ -290,7 +305,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n \t}\n \n-\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n+\tret = write_midx_file_compact(source, from_midx, to_midx,\n+\t\t\t\t opts.incremental_base, opts.flags);\n \n \treturn ret;\n }\ndiff --git a/midx-write.c b/midx-write.c\nindex 38c898e5ff5..561e9eedc0e 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1247,6 +1247,7 @@ struct write_midx_opts {\n \n \tconst char *preferred_pack_name;\n \tconst char *refs_snapshot;\n+\tconst char *incremental_base;\n \tunsigned flags;\n };\n \n@@ -1330,11 +1331,32 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t/*\n \t * If compacting MIDX layer(s) in the range [from, to], then the\n-\t * compacted MIDX will share the same base MIDX as 'from'.\n+\t * compacted MIDX will share the same base MIDX as 'from',\n+\t * unless a custom --base is specified (see below).\n \t */\n \tif (ctx.compact)\n \t\tctx.base_midx = ctx.compact_from->base_midx;\n \n+\tif (opts->incremental_base) {\n+\t\tif (!strcmp(opts->incremental_base, \"none\")) {\n+\t\t\tctx.base_midx = NULL;\n+\t\t} else {\n+\t\t\twhile (ctx.base_midx) {\n+\t\t\t\tconst char *cmp = midx_get_checksum_hex(ctx.base_midx);\n+\t\t\t\tif (!strcmp(opts->incremental_base, cmp))\n+\t\t\t\t\tbreak;\n+\n+\t\t\t\tctx.base_midx = ctx.base_midx->base_midx;\n+\t\t\t}\n+\n+\t\t\tif (!ctx.base_midx) {\n+\t\t\t\terror(_(\"could not find base MIDX '%s'\"),\n+\t\t\t\t opts->incremental_base);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n@@ -1827,7 +1849,8 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n int write_midx_file(struct odb_source *source,\n \t\t const char *preferred_pack_name,\n-\t\t const char *refs_snapshot, unsigned flags)\n+\t\t const char *refs_snapshot,\n+\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n@@ -1842,13 +1865,16 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags)\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.packs_to_include = packs_to_include,\n \t\t.preferred_pack_name = preferred_pack_name,\n \t\t.refs_snapshot = refs_snapshot,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags,\n \t};\n \n@@ -1858,12 +1884,14 @@ int write_midx_file_only(struct odb_source *source,\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.compact_from = from,\n \t\t.compact_to = to,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags | MIDX_WRITE_COMPACT,\n \t};\n \ndiff --git a/midx.h b/midx.h\nindex 5b193882dcf..77dd66de02b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -132,10 +132,13 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags);\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags);\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct odb_source *source, unsigned flags);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 66d6894761b..68a103d13d2 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -113,6 +113,36 @@ test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file\n \ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n '\n \n+test_expect_success 'write MIDX layer with --base without --no-write-chain-file' '\n+\ttest_must_fail git multi-pack-index write --bitmap --incremental \\\n+\t\t--base=none 2>err &&\n+\ttest_grep \"cannot use --base without --no-write-chain-file\" err\n+'\n+\n+test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '\n+\ttest_commit base-none &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--no-write-chain-file --base=none)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write MIDX layer with --base= and --no-write-chain-file' '\n+\ttest_commit base-hash &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 1a65d48b62b..ec1dafe89fc 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -304,6 +304,7 @@ test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \n \t\tlayer=\"$(git multi-pack-index compact --incremental \\\n \t\t\t--no-write-chain-file \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n \n@@ -326,4 +327,80 @@ test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --base' '\n+\tgit init midx-compact-with--base &&\n+\t(\n+\t\tcd midx-compact-with--base &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tnth_line 1 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 1 \"$midx_chain\" >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with --base=none' '\n+\tgit init midx-compact-base-none &&\n+\t(\n+\t\tcd midx-compact-base-none &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\t# Compact the two bottommost layers (A and B) into a new\n+\t\t# root layer with no parent.\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=none \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" &&\n+\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\t# The upper layers (C and D) should be preserved\n+\t\t# unchanged.\n+\t\tnth_line 3 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 4 \"$midx_chain.bak\" >>expect &&\n+\t\tnth_line 2 \"$midx_chain\" >actual &&\n+\t\tnth_line 3 \"$midx_chain\" >>actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bogus --base checksum' '\n+\tgit init midx-compact-bogus-base &&\n+\t(\n+\t\tcd midx-compact-bogus-base &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t--base=deadbeef \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" 2>err &&\n+\t\ttest_grep \"could not find base MIDX\" err\n+\t)\n+'\n+\n test_done\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542075","messageId":"92aba3d366f78fbc0f5a0928107ed8f5791fb58a.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 07/16] repack: track the ODB source via existing_packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:33Z","receivedAt":"2026-04-21T20:37:35Z","isPatch":true,"body":"Store the ODB source in the `existing_packs` struct and use that in\nplace of the raw `repo->objects->sources` access within `cmd_repack()`.\n\nThe source used is still assigned from the first source in the list, so\nthere are no functional changes in this commit. The changes instead\nserve two purposes (one immediate, one not):\n\n - The incremental MIDX-based repacking machinery will need to know what\n source is being used to read the existing MIDX/chain (should one\n exist).\n\n - In the future, if \"git repack\" is taught how to operate on other\n object sources, this field will serve as the authoritative value for\n that source.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 5 ++---\n repack.c | 2 ++\n repack.h | 1 +\n 3 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 4c5a82c2c8d..24be147d39a 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -417,7 +417,7 @@ int cmd_repack(int argc,\n \t\t * midx_has_unknown_packs() will make the decision for\n \t\t * us.\n \t\t */\n-\t\tif (!get_multi_pack_index(repo->objects->sources))\n+\t\tif (!get_multi_pack_index(existing.source))\n \t\t\tmidx_must_contain_cruft = 1;\n \t}\n \n@@ -564,8 +564,7 @@ int cmd_repack(int argc,\n \t\tunsigned flags = 0;\n \t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))\n \t\t\tflags |= MIDX_WRITE_INCREMENTAL;\n-\t\twrite_midx_file(repo->objects->sources,\n-\t\t\t\tNULL, NULL, flags);\n+\t\twrite_midx_file(existing.source, NULL, NULL, flags);\n \t}\n \n cleanup:\ndiff --git a/repack.c b/repack.c\nindex 596841027af..2ee6b51420a 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -154,6 +154,8 @@ void existing_packs_collect(struct existing_packs *existing,\n \t\t\tstring_list_append(&existing->non_kept_packs, buf.buf);\n \t}\n \n+\texisting->source = existing->repo->objects->sources;\n+\n \tstring_list_sort(&existing->kept_packs);\n \tstring_list_sort(&existing->non_kept_packs);\n \tstring_list_sort(&existing->cruft_packs);\ndiff --git a/repack.h b/repack.h\nindex bc9f2e1a5de..c0e9f0ca647 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -56,6 +56,7 @@ struct packed_git;\n \n struct existing_packs {\n \tstruct repository *repo;\n+\tstruct odb_source *source;\n \tstruct string_list kept_packs;\n \tstruct string_list non_kept_packs;\n \tstruct string_list cruft_packs;\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542076","messageId":"d3ac65c1f11656d11ff6c17fe4e5b02a916585d5.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 08/16] midx: expose `midx_layer_contains_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:36Z","receivedAt":"2026-04-21T20:37:39Z","isPatch":true,"body":"Rename the function `midx_contains_pack_1()` to instead be called\n`midx_layer_contains_pack()` and make it accessible. Unlike\n`midx_contains_pack()` (which recurses through the entire chain), this\nfunction checks only a single MIDX layer.\n\nThis will be used by a subsequent commit to determine whether a given\npack belongs to the tip MIDX layer specifically, rather than to any\nlayer in the chain.\n\nNo functional changes are present in this commit.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 +++---\n midx.h | 2 ++\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex bcb8c999015..dc86c8e7fee 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -667,8 +667,8 @@ static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n \t\t m->pack_names[*(const size_t *)b]);\n }\n \n-static int midx_contains_pack_1(struct multi_pack_index *m,\n-\t\t\t\tconst char *idx_or_pack_name)\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -709,7 +709,7 @@ static int midx_contains_pack_1(struct multi_pack_index *m,\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n {\n \tfor (; m; m = m->base_midx)\n-\t\tif (midx_contains_pack_1(m, idx_or_pack_name))\n+\t\tif (midx_layer_contains_pack(m, idx_or_pack_name))\n \t\t\treturn 1;\n \treturn 0;\n }\ndiff --git a/midx.h b/midx.h\nindex 77dd66de02b..3ee12dd08ec 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -119,6 +119,8 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n int fill_midx_entry(struct multi_pack_index *m, const struct object_id *oid, struct pack_entry *e);\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t const char *idx_or_pack_name);\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n int prepare_multi_pack_index_one(struct odb_source *source);\n \n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542077","messageId":"1bd2f194c6f7f64f2ff1e7b55a3a69defcb6a344.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 09/16] repack-midx: factor out `repack_prepare_midx_command()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:39Z","receivedAt":"2026-04-21T20:37:41Z","isPatch":true,"body":"The `write_midx_included_packs()` function assembles and executes a\n`git multi-pack-index write` command, constructing the argument list\ninline.\n\nFuture commits will introduce additional callers that need to construct\nsimilar `git multi-pack-index` commands (for both `write` and `compact`\nsubcommands), so extract the common portions of the command setup into a\nreusable `repack_prepare_midx_command()` helper.\n\nThe extracted helper sets `git_cmd`, pushes the `multi-pack-index`\nsubcommand and verb, and handles `--progress`/`--no-progress` and\n`--bitmap` flags. The remaining arguments that are specific to the\n`write` subcommand (such as `--stdin-packs`) are left to the caller.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 30 +++++++++++++++++++-----------\n 1 file changed, 19 insertions(+), 11 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 0682b80c427..83151d4734a 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -275,6 +275,23 @@ static void remove_redundant_bitmaps(struct string_list *include,\n \tstrbuf_release(&path);\n }\n \n+static void repack_prepare_midx_command(struct child_process *cmd,\n+\t\t\t\t\tstruct repack_write_midx_opts *opts,\n+\t\t\t\t\tconst char *verb)\n+{\n+\tcmd->git_cmd = 1;\n+\n+\tstrvec_pushl(&cmd->args, \"multi-pack-index\", verb, NULL);\n+\n+\tif (opts->show_progress)\n+\t\tstrvec_push(&cmd->args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd->args, \"--no-progress\");\n+\n+\tif (opts->write_bitmaps)\n+\t\tstrvec_push(&cmd->args, \"--bitmap\");\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -289,18 +306,9 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \n \tcmd.in = -1;\n-\tcmd.git_cmd = 1;\n \n-\tstrvec_push(&cmd.args, \"multi-pack-index\");\n-\tstrvec_pushl(&cmd.args, \"write\", \"--stdin-packs\", NULL);\n-\n-\tif (opts->show_progress)\n-\t\tstrvec_push(&cmd.args, \"--progress\");\n-\telse\n-\t\tstrvec_push(&cmd.args, \"--no-progress\");\n-\n-\tif (opts->write_bitmaps)\n-\t\tstrvec_push(&cmd.args, \"--bitmap\");\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542078","messageId":"44f522ea04df5176ec332b60a38d1eab9b649c30.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 10/16] repack-midx: extract `repack_fill_midx_stdin_packs()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:42Z","receivedAt":"2026-04-21T20:37:44Z","isPatch":true,"body":"The function `write_midx_included_packs()` manages the lifecycle of\nwriting packs to stdin when running `git multi-pack-index write` as a\nchild process.\n\nExtract a standalone `repack_fill_midx_stdin_packs()` helper, which\nhandles `--stdin-packs` argument setup, starting the command, writing\npack names to its standard input, and finishing the command.\n\nThis simplifies `write_midx_included_packs()` and prepares for a\nsubsequent commit where the same helper is called with `cmd->out = -1`\nto capture the MIDX's checksum from the command's standard output,\nwhich is needed when writing MIDX layers with `--checksum-only`.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 38 ++++++++++++++++++++++++--------------\n 1 file changed, 24 insertions(+), 14 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 83151d4734a..78f069c2151 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -292,23 +292,42 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n \t\tstrvec_push(&cmd->args, \"--bitmap\");\n }\n \n+static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n+\t\t\t\t\tstruct string_list *include)\n+{\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\n+\tcmd->in = -1;\n+\n+\tstrvec_push(&cmd->args, \"--stdin-packs\");\n+\n+\tret = start_command(cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tin = xfdopen(cmd->in, \"w\");\n+\tfor_each_string_list_item(item, include)\n+\t\tfprintf(in, \"%s\\n\", item->string);\n+\tfclose(in);\n+\n+\treturn finish_command(cmd);\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n \tstruct string_list_item *item;\n \tstruct packed_git *preferred = pack_geometry_preferred_pack(opts->geometry);\n-\tFILE *in;\n \tint ret = 0;\n \n \tmidx_included_packs(&include, opts);\n \tif (!include.nr)\n \t\tgoto done;\n \n-\tcmd.in = -1;\n-\n \trepack_prepare_midx_command(&cmd, opts, \"write\");\n-\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n@@ -350,16 +369,7 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = start_command(&cmd);\n-\tif (ret)\n-\t\tgoto done;\n-\n-\tin = xfdopen(cmd.in, \"w\");\n-\tfor_each_string_list_item(item, &include)\n-\t\tfprintf(in, \"%s\\n\", item->string);\n-\tfclose(in);\n-\n-\tret = finish_command(&cmd);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542079","messageId":"f5642a46bbd1cb092c525a14b5f7e2233db3ea56.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 11/16] repack-geometry: prepare for incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:45Z","receivedAt":"2026-04-21T20:37:47Z","isPatch":true,"body":"Teach `pack_geometry_init()` to optionally restrict the set of\nrepacking candidates to only packs in the tip MIDX layer when a\n`midx_layer_threshold` is configured. If the tip layer has fewer packs\nthan the threshold, those packs are excluded entirely; otherwise only\npacks in that layer participate in the geometric repack.\n\nAlso track whether any tip-layer packs were included in the rollup\n(`midx_tip_rewritten`), which a subsequent commit will use to decide\nhow to update the MIDX chain after repacking.\n\nSigned-off-by: Taylor Blau \n---\n repack-geometry.c | 35 +++++++++++++++++++++++++++++++++++\n repack.h | 4 ++++\n 2 files changed, 39 insertions(+)\n\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex 7cebd0cb45f..2408b8a3cc2 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -4,6 +4,7 @@\n #include \"repack.h\"\n #include \"repository.h\"\n #include \"hex.h\"\n+#include \"midx.h\"\n #include \"packfile.h\"\n \n static uint32_t pack_geometry_weight(struct packed_git *p)\n@@ -31,8 +32,28 @@ void pack_geometry_init(struct pack_geometry *geometry,\n {\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n+\tstruct multi_pack_index *m = get_multi_pack_index(existing->source);\n \n \trepo_for_each_pack(existing->repo, p) {\n+\t\tif (geometry->midx_layer_threshold_set && m &&\n+\t\t p->multi_pack_index) {\n+\t\t\t/*\n+\t\t\t * When writing MIDX layers incrementally,\n+\t\t\t * ignore packs unless they are in the most\n+\t\t\t * recent MIDX layer *and* there are at least\n+\t\t\t * 'midx_layer_threshold' packs in that layer.\n+\t\t\t *\n+\t\t\t * Otherwise 'p' is either in an older layer, or\n+\t\t\t * the youngest layer does not have enough packs\n+\t\t\t * to consider its packs as candidates for\n+\t\t\t * repacking. In either of those cases we want\n+\t\t\t * to ignore the pack.\n+\t\t\t */\n+\t\t\tif (m->num_packs < geometry->midx_layer_threshold ||\n+\t\t\t !midx_layer_contains_pack(m, pack_basename(p)))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -173,6 +194,20 @@ void pack_geometry_split(struct pack_geometry *geometry)\n \tgeometry->promisor_split = compute_pack_geometry_split(geometry->promisor_pack,\n \t\t\t\t\t\t\t geometry->promisor_pack_nr,\n \t\t\t\t\t\t\t geometry->split_factor);\n+\tfor (uint32_t i = 0; i < geometry->split; i++) {\n+\t\tstruct packed_git *p = geometry->pack[i];\n+\t\t/*\n+\t\t * During incremental MIDX/bitmap repacking, any packs\n+\t\t * included in the rollup are either (a) not MIDX'd, or\n+\t\t * (b) contained in the tip layer iff it has at least\n+\t\t * the threshold number of packs.\n+\t\t *\n+\t\t * In the latter case, we can safely conclude that the\n+\t\t * tip of the MIDX chain will be rewritten.\n+\t\t */\n+\t\tif (p->multi_pack_index)\n+\t\t\tgeometry->midx_tip_rewritten = true;\n+\t}\n }\n \n struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry)\ndiff --git a/repack.h b/repack.h\nindex c0e9f0ca647..77d24ee45fb 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -108,6 +108,10 @@ struct pack_geometry {\n \tuint32_t promisor_pack_nr, promisor_pack_alloc;\n \tuint32_t promisor_split;\n \n+\tuint32_t midx_layer_threshold;\n+\tbool midx_layer_threshold_set;\n+\tbool midx_tip_rewritten;\n+\n \tint split_factor;\n };\n \n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542080","messageId":"9fdcb253a9631adb467593fc1d59c594c1a350b9.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 12/16] builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:48Z","receivedAt":"2026-04-21T20:37:50Z","isPatch":true,"body":"Change the --write-midx (-m) flag from an OPT_BOOL to an OPT_CALLBACK\nthat accepts an optional mode argument. Introduce an enum with\nREPACK_WRITE_MIDX_NONE and REPACK_WRITE_MIDX_DEFAULT to distinguish\nbetween the two states, and update all existing boolean checks\naccordingly.\n\nFor now, passing no argument (or just `-m`) selects the default mode,\npreserving existing behavior. A subsequent commit will add a new mode\nfor writing incremental MIDXs.\n\nExtract repack_write_midx() as a dispatcher that selects the\nappropriate MIDX-writing implementation based on the mode.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 50 ++++++++++++++++++++++++++++++++++++------------\n repack-midx.c | 14 +++++++++++++-\n repack.h | 8 +++++++-\n 3 files changed, 58 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 24be147d39a..5d366340c34 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -97,6 +97,24 @@ static int repack_config(const char *var, const char *value,\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n+static int option_parse_write_midx(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum repack_write_midx_mode *cfg = opt->value;\n+\n+\tif (unset) {\n+\t\t*cfg = REPACK_WRITE_MIDX_NONE;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!arg || !*arg)\n+\t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse\n+\t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_repack(int argc,\n \t const char **argv,\n \t const char *prefix,\n@@ -119,7 +137,7 @@ int cmd_repack(int argc,\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct pack_objects_args po_args = PACK_OBJECTS_ARGS_INIT;\n \tstruct pack_objects_args cruft_po_args = PACK_OBJECTS_ARGS_INIT;\n-\tint write_midx = 0;\n+\tenum repack_write_midx_mode write_midx = REPACK_WRITE_MIDX_NONE;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n \tconst char *filter_to = NULL;\n@@ -185,8 +203,14 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_INTEGER('g', \"geometric\", &geometry.split_factor,\n \t\t\t N_(\"find a geometric progression with factor \")),\n-\t\tOPT_BOOL('m', \"write-midx\", &write_midx,\n-\t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n+\t\tOPT_CALLBACK_F(0, \"write-midx\", &write_midx,\n+\t\t\t N_(\"mode\"),\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t PARSE_OPT_OPTARG, option_parse_write_midx),\n+\t\tOPT_SET_INT_F('m', NULL, &write_midx,\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t REPACK_WRITE_MIDX_DEFAULT,\n+\t\t\t PARSE_OPT_HIDDEN),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n \t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n@@ -221,14 +245,16 @@ int cmd_repack(int argc,\n \t\tpack_everything |= ALL_INTO_ONE;\n \n \tif (write_bitmaps < 0) {\n-\t\tif (!write_midx &&\n+\t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n \t}\n \tif (po_args.pack_kept_objects < 0)\n-\t\tpo_args.pack_kept_objects = write_bitmaps > 0 && !write_midx;\n+\t\tpo_args.pack_kept_objects = write_bitmaps > 0 &&\n+\t\t\twrite_midx == REPACK_WRITE_MIDX_NONE;\n \n-\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n+\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) &&\n+\t write_midx == REPACK_WRITE_MIDX_NONE)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n \tif (write_bitmaps && po_args.local &&\n@@ -244,7 +270,7 @@ int cmd_repack(int argc,\n \t\twrite_bitmaps = 0;\n \t}\n \n-\tif (write_midx && write_bitmaps) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \n \t\tstrbuf_addf(&path, \"%s/%s_XXXXXX\",\n@@ -297,7 +323,7 @@ int cmd_repack(int argc,\n \t}\n \tif (repo_has_promisor_remote(repo))\n \t\tstrvec_push(&cmd.args, \"--exclude-promisor-objects\");\n-\tif (!write_midx) {\n+\tif (write_midx == REPACK_WRITE_MIDX_NONE) {\n \t\tif (write_bitmaps > 0)\n \t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n \t\telse if (write_bitmaps < 0)\n@@ -519,7 +545,7 @@ int cmd_repack(int argc,\n \tif (delete_redundant && pack_everything & ALL_INTO_ONE)\n \t\texisting_packs_mark_for_deletion(&existing, &names);\n \n-\tif (write_midx) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE) {\n \t\tstruct repack_write_midx_opts opts = {\n \t\t\t.existing = &existing,\n \t\t\t.geometry = &geometry,\n@@ -528,11 +554,11 @@ int cmd_repack(int argc,\n \t\t\t.packdir = packdir,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n-\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft\n+\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.mode = write_midx,\n \t\t};\n \n-\t\tret = write_midx_included_packs(&opts);\n-\n+\t\tret = repack_write_midx(&opts);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \t}\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 78f069c2151..4a568a2a9b8 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -315,7 +315,7 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n-int write_midx_included_packs(struct repack_write_midx_opts *opts)\n+static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n@@ -378,3 +378,15 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \n \treturn ret;\n }\n+\n+int repack_write_midx(struct repack_write_midx_opts *opts)\n+{\n+\tswitch (opts->mode) {\n+\tcase REPACK_WRITE_MIDX_NONE:\n+\t\tBUG(\"write_midx mode is NONE?\");\n+\tcase REPACK_WRITE_MIDX_DEFAULT:\n+\t\treturn write_midx_included_packs(opts);\n+\tdefault:\n+\t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n+\t}\n+}\ndiff --git a/repack.h b/repack.h\nindex 77d24ee45fb..81907fcce7f 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -134,6 +134,11 @@ void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\n \n+enum repack_write_midx_mode {\n+\tREPACK_WRITE_MIDX_NONE,\n+\tREPACK_WRITE_MIDX_DEFAULT,\n+};\n+\n struct repack_write_midx_opts {\n \tstruct existing_packs *existing;\n \tstruct pack_geometry *geometry;\n@@ -143,10 +148,11 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tenum repack_write_midx_mode mode;\n };\n \n void midx_snapshot_refs(struct repository *repo, struct tempfile *f);\n-int write_midx_included_packs(struct repack_write_midx_opts *opts);\n+int repack_write_midx(struct repack_write_midx_opts *opts);\n \n int write_filtered_pack(const struct write_pack_opts *opts,\n \t\t\tstruct existing_packs *existing,\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542081","messageId":"1e1b957bf1266cee1608981d1a2960d4577aec79.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 13/16] packfile: ensure `close_pack_revindex()` frees in-memory revindex","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:51Z","receivedAt":"2026-04-21T20:37:53Z","isPatch":true,"body":"The following commit will introduce a case where we write a MIDX bitmap\nover packs that do not themselves have on-disk *.rev files.\n\nThis case is supported within Git, and we will simply fall back to\ngenerating the revindex in memory. But we don't ever release that\nmemory, causing a leak that is exposed by a test introduced in the\nfollowing commit.\n\n(As far as I could find, we never free()'d memory allocated as a\nbyproduct of creating an in-memory revindex, likely because that code\npredates the leak-checking niceties we have in the test suite now.)\n\nRectify this by calling `FREE_AND_NULL()` on the `p->revindex` field\nwhen calling `close_pack_revindex()`.\n\nSigned-off-by: Taylor Blau \n---\n packfile.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/packfile.c b/packfile.c\nindex b012d648ada..a1e88fdb223 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -420,6 +420,8 @@ void close_pack_index(struct packed_git *p)\n \n static void close_pack_revindex(struct packed_git *p)\n {\n+\tFREE_AND_NULL(p->revindex);\n+\n \tif (!p->revindex_map)\n \t\treturn;\n \n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542082","messageId":"93e152fb6aa3bbd450d46c6cdd219dfef14d41fc.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 14/16] repack: implement incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:54Z","receivedAt":"2026-04-21T20:37:57Z","isPatch":true,"body":"Implement the `write_midx_incremental()` function, which builds and\nmaintains an incremental MIDX chain as part of the geometric repacking\nprocess.\n\nUnlike the default mode which writes a single flat MIDX, the incremental\nmode constructs a compaction plan that determines which MIDX layers to\nwrite, compact, or copy, and then executes each step using `git\nmulti-pack-index` subcommands with the --checksum-only flag.\n\nThe repacking strategy works as follows:\n\n * Acquire the lock guarding the multi-pack-index-chain.\n\n * A new MIDX layer is always written containing the newly created\n pack(s). If the tip MIDX layer was rewritten during geometric\n repacking, any surviving packs from that layer are also included.\n\n * Starting from the new layer, adjacent MIDX layers are merged together\n as long as the accumulated object count exceeds half the object count\n of the next deeper layer (controlled by 'repack.midxSplitFactor').\n\n * Remaining layers in the chain are evaluated pairwise and either\n compacted or copied as-is, following the same merging condition.\n\n * Write the contents of the new multi-pack-index chain, atomically move\n it into place, and then release the lock.\n\n * Delete any now-unused MIDX layers.\n\nAfter writing the new layer, the strategy is evaluated among the\nexisting MIDX layers in order from oldest to newest. Each step that\nwrites a new MIDX layer uses \"--checksum-only\" to avoid updating the\nmulti-pack-index-chain file. After all steps are complete, the new chain\nfile is written and then atomically moved into place.\n\nAt present, this functionality is exposed behind a new enum value,\n`REPACK_WRITE_MIDX_INCREMENTAL`, but has no external callers. A\nsubsequent commit will expose this mode via `git repack\n--write-midx=incremental`.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 5 +\n repack-midx.c | 578 ++++++++++++++++++++++++++++++++++++++++++++++-\n repack.h | 3 +\n 3 files changed, 584 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5d366340c34..75c57736780 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -42,6 +42,9 @@ static const char incremental_bitmap_conflict_error[] = N_(\n \"--no-write-bitmap-index or disable the pack.writeBitmaps configuration.\"\n );\n \n+#define DEFAULT_MIDX_SPLIT_FACTOR 2\n+#define DEFAULT_MIDX_NEW_LAYER_THRESHOLD 8\n+\n struct repack_config_ctx {\n \tstruct pack_objects_args *po_args;\n \tstruct pack_objects_args *cruft_po_args;\n@@ -555,6 +558,8 @@ int cmd_repack(int argc,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,\n+\t\t\t.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,\n \t\t\t.mode = write_midx,\n \t\t};\n \ndiff --git a/repack-midx.c b/repack-midx.c\nindex 4a568a2a9b8..8f3720772b8 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -2,12 +2,16 @@\n #include \"repack.h\"\n #include \"hash.h\"\n #include \"hex.h\"\n+#include \"lockfile.h\"\n+#include \"midx.h\"\n #include \"odb.h\"\n #include \"oidset.h\"\n #include \"pack-bitmap.h\"\n+#include \"path.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n #include \"tempfile.h\"\n+#include \"trace2.h\"\n \n struct midx_snapshot_ref_data {\n \tstruct repository *repo;\n@@ -293,13 +297,16 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n }\n \n static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n-\t\t\t\t\tstruct string_list *include)\n+\t\t\t\t\tstruct string_list *include,\n+\t\t\t\t\tstruct string_list *out)\n {\n \tstruct string_list_item *item;\n \tFILE *in;\n \tint ret;\n \n \tcmd->in = -1;\n+\tif (out)\n+\t\tcmd->out = -1;\n \n \tstrvec_push(&cmd->args, \"--stdin-packs\");\n \n@@ -312,6 +319,17 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n \t\tfprintf(in, \"%s\\n\", item->string);\n \tfclose(in);\n \n+\tif (out) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tFILE *outf = xfdopen(cmd->out, \"r\");\n+\n+\t\twhile (strbuf_getline(&buf, outf) != EOF)\n+\t\t\tstring_list_append(out, buf.buf);\n+\t\tstrbuf_release(&buf);\n+\n+\t\tfclose(outf);\n+\t}\n+\n \treturn finish_command(cmd);\n }\n \n@@ -369,7 +387,7 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include, NULL);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n@@ -379,6 +397,560 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \treturn ret;\n }\n \n+struct midx_compaction_step {\n+\tunion {\n+\t\tstruct multi_pack_index *copy;\n+\t\tstruct string_list write;\n+\t\tstruct {\n+\t\t\tstruct multi_pack_index *from;\n+\t\t\tstruct multi_pack_index *to;\n+\t\t} compact;\n+\t} u;\n+\n+\tuint32_t objects_nr;\n+\tchar *csum;\n+\n+\tenum {\n+\t\tMIDX_COMPACTION_STEP_UNKNOWN,\n+\t\tMIDX_COMPACTION_STEP_COPY,\n+\t\tMIDX_COMPACTION_STEP_WRITE,\n+\t\tMIDX_COMPACTION_STEP_COMPACT,\n+\t} type;\n+};\n+\n+static const char *midx_compaction_step_base(const struct midx_compaction_step *step)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot use UNKNOWN step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_get_checksum_hex(step->u.copy);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\tBUG(\"cannot use WRITE step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_get_checksum_hex(step->u.compact.to);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static int midx_compaction_step_exec_copy(struct midx_compaction_step *step)\n+{\n+\tstep->csum = xstrdup(midx_get_checksum_hex(step->u.copy));\n+\treturn 0;\n+}\n+\n+static int midx_compaction_step_exec_write(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t\t const char *base)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list hash = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\tconst char *preferred_pack = NULL;\n+\tint ret = 0;\n+\n+\tif (!step->u.write.nr) {\n+\t\tret = error(_(\"no packs to write MIDX during compaction\"));\n+\t\tgoto out;\n+\t}\n+\n+\tfor_each_string_list_item(item, &step->u.write) {\n+\t\tif (item->util)\n+\t\t\tpreferred_pack = item->string;\n+\t}\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\", NULL);\n+\tstrvec_pushf(&cmd.args, \"--base=%s\", base ? base : \"none\");\n+\n+\tif (preferred_pack) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\n+\t\tstrbuf_addstr(&buf, preferred_pack);\n+\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\t\tstrbuf_addstr(&buf, \".pack\");\n+\n+\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\", buf.buf);\n+\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\tret = repack_fill_midx_stdin_packs(&cmd, &step->u.write, &hash);\n+\tif (hash.nr != 1) {\n+\t\tret = error(_(\"expected exactly one line during MIDX write, \"\n+\t\t\t \"got: %\"PRIuMAX),\n+\t\t\t (uintmax_t)hash.nr);\n+\t\tgoto out;\n+\t}\n+\n+\tstep->csum = xstrdup(hash.items[0].string);\n+\n+out:\n+\tstring_list_clear(&hash, 0);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec_compact(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tFILE *out = NULL;\n+\tint ret;\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"compact\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\",\n+\t\t midx_get_checksum_hex(step->u.compact.from),\n+\t\t midx_get_checksum_hex(step->u.compact.to), NULL);\n+\n+\tcmd.out = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\tgoto out;\n+\n+\tout = xfdopen(cmd.out, \"r\");\n+\twhile (strbuf_getline_lf(&buf, out) != EOF) {\n+\t\tif (step->csum) {\n+\t\t\tret = error(_(\"unexpected MIDX output: '%s'\"), buf.buf);\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep->csum = strbuf_detach(&buf, NULL);\n+\t}\n+\n+\tret = finish_command(&cmd);\n+\n+out:\n+\tif (out)\n+\t\tfclose(out);\n+\tstrbuf_release(&buf);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec(struct midx_compaction_step *step,\n+\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t const char *base)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot execute UNKNOWN midx compaction step\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_compaction_step_exec_copy(step);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\treturn midx_compaction_step_exec_write(step, opts, base);\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_compaction_step_exec_compact(step, opts);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static void midx_compaction_step_release(struct midx_compaction_step *step)\n+{\n+\tif (step->type == MIDX_COMPACTION_STEP_WRITE)\n+\t\tstring_list_clear(&step->u.write, 0);\n+\tfree(step->csum);\n+}\n+\n+static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step step = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\tuint32_t i;\n+\tint ret = 0;\n+\n+\ttrace2_region_enter(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tfor (i = 0; m && i < m->num_packs + m->num_packs_in_base; i++) {\n+\t\tif (prepare_midx_pack(m, i)) {\n+\t\t\tret = error(_(\"could not load pack %\"PRIu32\" from MIDX\"),\n+\t\t\t\t i);\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\ttrace2_region_enter(\"repack\", \"steps:write\", opts->existing->repo);\n+\n+\t/*\n+\t * The first MIDX in the resulting chain is always going to be\n+\t * new.\n+\t *\n+\t * At a minimum, it will include all of the newly written packs.\n+\t * If there is an existing MIDX whose tip layer contains packs\n+\t * that were repacked, it will also include any of its packs\n+\t * which were *not* rolled up as part of the geometric repack\n+\t * (if any), and the previous tip will be replaced.\n+\t *\n+\t * It may grow to include the packs from zero or more MIDXs from\n+\t * the old chain, beginning either at the old tip (if the MIDX\n+\t * was *not* rewritten) or the old tip's base MIDX layer\n+\t * (otherwise).\n+\t */\n+\tstep.type = MIDX_COMPACTION_STEP_WRITE;\n+\tstring_list_init_dup(&step.u.write);\n+\n+\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"pack-%s.idx\", opts->names->items[i].string);\n+\t\tstring_list_append(&step.u.write, buf.buf);\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:fresh\",\n+\t\t\t\t step.u.write.items[step.u.write.nr - 1].string);\n+\t}\n+\tfor (i = 0; i < opts->geometry->split; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:fresh:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n+\n+\t/*\n+\t * Now handle any existing packs which were *not* rewritten.\n+\t *\n+\t * The list of packs in opts->geometry only contains MIDX'd\n+\t * packs from the newest layer when that layer has more than\n+\t * 'repack.midxNewLayerThreshold' number of packs.\n+\t *\n+\t * If the MIDX tip was rewritten (that is, one or more of those\n+\t * packs appear below the split line), then add all packs above\n+\t * the split line to the new layer, as the old one is no longer\n+\t * usable.\n+\t *\n+\t * If the MIDX tip was not rewritten (that is, all MIDX'd packs\n+\t * from the youngest layer appear below the split line, or were\n+\t * not included in the geometric repack at all because there\n+\t * were too few of them), ignore them since we'll retain the\n+\t * existing layer as-is.\n+\t */\n+\tfor (i = opts->geometry->split; i < opts->geometry->pack_nr; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tstruct string_list_item *item;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\tif (p->multi_pack_index &&\n+\t\t !opts->geometry->midx_tip_rewritten) {\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"exclude:unmodified\", buf.buf);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified\", buf.buf);\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified:midx\",\n+\t\t\t\t p->multi_pack_index ? \"true\" : \"false\");\n+\n+\t\titem = string_list_append(&step.u.write, buf.buf);\n+\t\tif (p->multi_pack_index || i == opts->geometry->pack_nr - 1)\n+\t\t\titem->util = (void *)1; /* mark as preferred */\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:unmodified:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n+\n+\t/*\n+\t * If the MIDX tip was rewritten, then we no longer consider it\n+\t * a candidate for compaction, since it will not exist in the\n+\t * MIDX chain being built.\n+\t */\n+\tif (opts->geometry->midx_tip_rewritten)\n+\t\tm = m->base_midx;\n+\n+\ttrace2_data_string(\"repack\", opts->existing->repo, \"midx:rewrote-tip\",\n+\t\t\t opts->geometry->midx_tip_rewritten ? \"true\" : \"false\");\n+\n+\ttrace2_region_enter(\"repack\", \"compact\", opts->existing->repo);\n+\n+\t/*\n+\t * Compact additional MIDX layers into this proposed one until\n+\t * the merging condition is violated.\n+\t */\n+\twhile (m) {\n+\t\tuint32_t preferred_pack_idx;\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"candidate\", midx_get_checksum_hex(m));\n+\n+\t\tif (step.objects_nr < m->num_objects / opts->midx_split_factor) {\n+\t\t\t/*\n+\t\t\t * Stop compacting MIDX layer as soon as the\n+\t\t\t * merged size is less than half the size of the\n+\t\t\t * next layer in the chain.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"compact\", \"violated\");\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"next_objects_nr\",\n+\t\t\t\t\t (uintmax_t)m->num_objects);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"split_factor\",\n+\t\t\t\t\t (uintmax_t)opts->midx_split_factor);\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (midx_preferred_pack(m, &preferred_pack_idx) < 0) {\n+\t\t\tret = error(_(\"could not find preferred pack for MIDX \"\n+\t\t\t\t \"%s\"), midx_get_checksum_hex(m));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\tstruct string_list_item *item;\n+\t\t\tuint32_t pack_int_id = i + m->num_packs_in_base;\n+\t\t\tstruct packed_git *p = nth_midxed_pack(m, pack_int_id);\n+\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"midx:pack\", buf.buf);\n+\n+\t\t\titem = string_list_append(&step.u.write, buf.buf);\n+\t\t\tif (pack_int_id == preferred_pack_idx)\n+\t\t\t\titem->util = (void *)1; /* mark as preferred */\n+\t\t}\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, m->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep.objects_nr += m->num_objects;\n+\n+\t\tm = m->base_midx;\n+\t}\n+\n+\tif (step.u.write.nr > 0) {\n+\t\t/*\n+\t\t * As long as there is at least one new pack to write\n+\t\t * (and thus the MIDX is non-empty), add it to the plan.\n+\t\t */\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\t\tsteps[steps_nr++] = step;\n+\t}\n+\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:objects_nr\", (uintmax_t)step.objects_nr);\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:packs_nr\", (uintmax_t)step.u.write.nr);\n+\n+\ttrace2_region_leave(\"repack\", \"compact\", opts->existing->repo);\n+\ttrace2_region_leave(\"repack\", \"steps:write\", opts->existing->repo);\n+\n+\ttrace2_region_enter(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n+\t/*\n+\t * Then start over, repeat, and either compact or keep as-is\n+\t * each MIDX layer until we have exhausted the chain.\n+\t *\n+\t * Finally, evaluate the remainder of the chain (if any) and\n+\t * either compact a sequence of adjacent layers, or keep\n+\t * individual layers as-is according to the same merging\n+\t * condition as above.\n+\t */\n+\twhile (m) {\n+\t\tstruct multi_pack_index *next = m;\n+\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\n+\t\tmemset(&step, 0, sizeof(step));\n+\t\tstep.type = MIDX_COMPACTION_STEP_UNKNOWN;\n+\n+\t\ttrace2_region_enter(\"repack\", \"step\", opts->existing->repo);\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"from\", midx_get_checksum_hex(m));\n+\n+\t\twhile (next) {\n+\t\t\tuint32_t proposed_objects_nr;\n+\t\t\tif (unsigned_add_overflows(step.objects_nr, next->num_objects)) {\n+\t\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tproposed_objects_nr = step.objects_nr + next->num_objects;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed\",\n+\t\t\t\t\t midx_get_checksum_hex(next));\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed:objects_nr\",\n+\t\t\t\t\t (uintmax_t)next->num_objects);\n+\n+\t\t\tif (!next->base_midx) {\n+\t\t\t\t/*\n+\t\t\t\t * If we are at the end of the MIDX\n+\t\t\t\t * chain, there is nothing to compact,\n+\t\t\t\t * so mark it and stop.\n+\t\t\t\t */\n+\t\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (proposed_objects_nr < next->base_midx->num_objects / opts->midx_split_factor) {\n+\t\t\t\t/*\n+\t\t\t\t * If there is a MIDX following this\n+\t\t\t\t * one, but our accumulated size is less\n+\t\t\t\t * than half of its size, compacting\n+\t\t\t\t * them would violate the merging\n+\t\t\t\t * condition, so stop here.\n+\t\t\t\t */\n+\n+\t\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at\",\n+\t\t\t\t\t\t midx_get_checksum_hex(next->base_midx));\n+\t\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at:objects_nr\",\n+\t\t\t\t\t\t (uintmax_t)next->base_midx->num_objects);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * Otherwise, it is OK to compact the next layer\n+\t\t\t * into this one. Do so, and then continue\n+\t\t\t * through the remainder of the chain.\n+\t\t\t */\n+\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"step:objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\tnext = next->base_midx;\n+\t\t}\n+\n+\t\tif (m == next) {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COPY;\n+\t\t\tstep.u.copy = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"copy\");\n+\t\t} else {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COMPACT;\n+\t\t\tstep.u.compact.from = next;\n+\t\t\tstep.u.compact.to = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"to\", midx_get_checksum_hex(m));\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"compact\");\n+\t\t}\n+\n+\t\tm = next->base_midx;\n+\t\tsteps[steps_nr++] = step;\n+\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n+\t}\n+\n+\ttrace2_region_leave(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n+out:\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+\n+\tstrbuf_release(&buf);\n+\n+\ttrace2_region_leave(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n+\treturn ret;\n+}\n+\n+static int write_midx_incremental(struct repack_write_midx_opts *opts)\n+{\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct strbuf lock_name = STRBUF_INIT;\n+\tstruct lock_file lf;\n+\tsize_t steps_nr = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tget_midx_chain_filename(opts->existing->source, &lock_name);\n+\tif (safe_create_leading_directories(opts->existing->repo,\n+\t\t\t\t\t lock_name.buf))\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t lock_name.buf);\n+\thold_lock_file_for_update(&lf, lock_name.buf, LOCK_DIE_ON_ERROR);\n+\n+\tif (!fdopen_lock_file(&lf, \"w\")) {\n+\t\tret = error_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\tgoto done;\n+\t}\n+\n+\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < steps_nr; i++) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tchar *base = NULL;\n+\n+\t\tif (i + 1 < steps_nr)\n+\t\t\tbase = xstrdup(midx_compaction_step_base(&steps[i + 1]));\n+\n+\t\tif (midx_compaction_step_exec(step, opts, base) < 0) {\n+\t\t\tret = error(_(\"unable to execute compaction step %\"PRIuMAX),\n+\t\t\t\t (uintmax_t)i);\n+\t\t\tfree(base);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfree(base);\n+\t}\n+\n+\ti = steps_nr;\n+\twhile (i--) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tif (!step->csum)\n+\t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n+\t\t\t (uintmax_t)i);\n+\t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t}\n+\n+\tcommit_lock_file(&lf);\n+\n+done:\n+\tstrbuf_release(&lock_name);\n+\tfor (i = 0; i < steps_nr; i++)\n+\t\tmidx_compaction_step_release(&steps[i]);\n+\tfree(steps);\n+\treturn ret;\n+}\n+\n int repack_write_midx(struct repack_write_midx_opts *opts)\n {\n \tswitch (opts->mode) {\n@@ -386,6 +958,8 @@ int repack_write_midx(struct repack_write_midx_opts *opts)\n \t\tBUG(\"write_midx mode is NONE?\");\n \tcase REPACK_WRITE_MIDX_DEFAULT:\n \t\treturn write_midx_included_packs(opts);\n+\tcase REPACK_WRITE_MIDX_INCREMENTAL:\n+\t\treturn write_midx_incremental(opts);\n \tdefault:\n \t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n \t}\ndiff --git a/repack.h b/repack.h\nindex 81907fcce7f..831ccfb1c6c 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -137,6 +137,7 @@ struct tempfile;\n enum repack_write_midx_mode {\n \tREPACK_WRITE_MIDX_NONE,\n \tREPACK_WRITE_MIDX_DEFAULT,\n+\tREPACK_WRITE_MIDX_INCREMENTAL,\n };\n \n struct repack_write_midx_opts {\n@@ -148,6 +149,8 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n \tenum repack_write_midx_mode mode;\n };\n \n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542083","messageId":"6119f15d3e8619f75c7dfb2e5dcf67e9f5fafc41.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 15/16] repack: introduce `--write-midx=incremental`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:37:57Z","receivedAt":"2026-04-21T20:38:00Z","isPatch":true,"body":"Expose the incremental MIDX repacking mode (implemented in an earlier\ncommit) via a new --write-midx=incremental option for `git repack`.\n\nAdd \"incremental\" as a recognized argument to the --write-midx\nOPT_CALLBACK, mapping it to REPACK_WRITE_MIDX_INCREMENTAL. When this\nmode is active and --geometric is in use, set the midx_layer_threshold\non the pack geometry so that only packs in sufficiently large tip layers\nare considered for repacking.\n\nTwo new configuration options control the compaction behavior:\n\n - repack.midxSplitFactor (default: 2): the factor used in the\n geometric merging condition for MIDX layers.\n\n - repack.midxNewLayerThreshold (default: 8): the minimum number of\n packs in the tip MIDX layer before its packs are considered as\n candidates for geometric repacking.\n\nAdd tests exercising the new mode across a variety of scenarios\nincluding basic geometric violations, multi-round chain integrity,\nbranching and merging histories, cross-layer object uniqueness, and\nthreshold-based compaction.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/repack.adoc | 18 ++\n Documentation/git-repack.adoc | 39 ++-\n builtin/repack.c | 49 ++-\n midx.c | 31 ++\n midx.h | 3 +\n repack-geometry.c | 13 +-\n repack-midx.c | 5 +\n repack.c | 56 +++-\n repack.h | 10 +-\n t/meson.build | 1 +\n t/t7705-repack-incremental-midx.sh | 500 +++++++++++++++++++++++++++++\n 11 files changed, 701 insertions(+), 24 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex e9e78dcb198..4c22a499f62 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -46,3 +46,21 @@ repack.midxMustContainCruft::\n \t`--write-midx`. When false, cruft packs are only included in the MIDX\n \twhen necessary (e.g., because they might be required to form a\n \treachability closure with MIDX bitmaps). Defaults to true.\n+\n+repack.midxSplitFactor::\n+\tThe factor used in the geometric merging condition when\n+\tcompacting incremental MIDX layers during `git repack` when\n+\tinvoked with the `--write-midx=incremental` option.\n++\n+Adjacent layers are merged when the accumulated object count of the\n+newer layer exceeds `1/` of the object count of the next deeper\n+layer. Must be at least 2. Defaults to 2.\n+\n+repack.midxNewLayerThreshold::\n+\tThe minimum number of packs in the tip MIDX layer before those\n+\tpacks are considered as candidates for geometric repacking\n+\tduring `git repack --write-midx=incremental`.\n++\n+When the tip layer has fewer packs than this threshold, those packs are\n+excluded from the geometric repack entirely, and are thus left\n+unmodified. Must be at least 1. Defaults to 8.\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 673ce910837..27a99cc46f4 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -11,7 +11,7 @@ SYNOPSIS\n [verse]\n 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n \t[--window=] [--depth=] [--threads=] [--keep-pack=]\n-\t[--write-midx] [--name-hash-version=] [--path-walk]\n+\t[--write-midx[=]] [--name-hash-version=] [--path-walk]\n \n DESCRIPTION\n -----------\n@@ -250,9 +250,42 @@ pack as the preferred pack for object selection by the MIDX (see\n linkgit:git-multi-pack-index[1]).\n \n -m::\n---write-midx::\n+--write-midx[=]::\n \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n-\tcontaining the non-redundant packs.\n+\tcontaining the non-redundant packs. The following modes are\n+\tavailable:\n++\n+--\n+\t`default`;;\n+\t\tWrite a single MIDX covering all packs. This is the\n+\t\tdefault when `--write-midx` is given without an\n+\t\texplicit mode.\n+\n+\t`incremental`;;\n+\t\tWrite an incremental MIDX chain instead of a single\n+\t\tflat MIDX. This mode requires `--geometric`.\n++\n+The incremental mode maintains a chain of MIDX layers that is compacted\n+over time using a geometric merging strategy. Each repack creates a new\n+tip layer containing the newly written pack(s). Adjacent layers are then\n+merged whenever the newer layer's object count exceeds\n+`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n+that do not meet this condition are retained as-is.\n++\n+The result is that newer (tip) layers tend to contain many small packs\n+with relatively few objects, while older (deeper) layers contain fewer,\n+larger packs covering more objects. Because compaction is driven by the\n+tip of the chain, newer layers are also rewritten more frequently than\n+older ones, which are only touched when enough objects have accumulated\n+to justify merging into them. This keeps the total number of layers\n+logarithmic relative to the total number of objects.\n++\n+Only packs in the tip MIDX layer are considered as candidates for the\n+geometric repack; packs in deeper layers are left untouched. If the tip\n+layer contains fewer packs than `repack.midxNewLayerThreshold`, those\n+packs are excluded from the geometry entirely, and a new layer is\n+created for any new pack(s) without disturbing the existing chain.\n+--\n \n --name-hash-version=::\n \tProvide this argument to the underlying `git pack-objects` process.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 75c57736780..5ffa18e085e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -33,7 +33,7 @@ static int midx_must_contain_cruft = 1;\n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n \t \"[--window=] [--depth=] [--threads=] [--keep-pack=]\\n\"\n-\t \"[--write-midx] [--name-hash-version=] [--path-walk]\"),\n+\t \"[--write-midx[=]] [--name-hash-version=] [--path-walk]\"),\n \tNULL\n };\n \n@@ -48,6 +48,8 @@ static const char incremental_bitmap_conflict_error[] = N_(\n struct repack_config_ctx {\n \tstruct pack_objects_args *po_args;\n \tstruct pack_objects_args *cruft_po_args;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -97,6 +99,16 @@ static int repack_config(const char *var, const char *value,\n \t\tmidx_must_contain_cruft = git_config_bool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"repack.midxsplitfactor\")) {\n+\t\trepack_ctx->midx_split_factor = git_config_int(var, value,\n+\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(var, \"repack.midxnewlayerthreshold\")) {\n+\t\trepack_ctx->midx_new_layer_threshold = git_config_int(var, value,\n+\t\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -112,6 +124,8 @@ static int option_parse_write_midx(const struct option *opt, const char *arg,\n \n \tif (!arg || !*arg)\n \t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse if (!strcmp(arg, \"incremental\"))\n+\t\t*cfg = REPACK_WRITE_MIDX_INCREMENTAL;\n \telse\n \t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n \n@@ -226,6 +240,8 @@ int cmd_repack(int argc,\n \tmemset(&config_ctx, 0, sizeof(config_ctx));\n \tconfig_ctx.po_args = &po_args;\n \tconfig_ctx.cruft_po_args = &cruft_po_args;\n+\tconfig_ctx.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR;\n+\tconfig_ctx.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD;\n \n \trepo_config(repo, repack_config, &config_ctx);\n \n@@ -247,6 +263,9 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n+\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n+\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n+\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n@@ -273,6 +292,13 @@ int cmd_repack(int argc,\n \t\twrite_bitmaps = 0;\n \t}\n \n+\tif (config_ctx.midx_split_factor < 2)\n+\t\tdie(_(\"invalid value for %s: %d\"), \"--midx-split-factor\",\n+\t\t config_ctx.midx_split_factor);\n+\tif (config_ctx.midx_new_layer_threshold < 1)\n+\t\tdie(_(\"invalid value for %s: %d\"), \"--midx-new-layer-threshold\",\n+\t\t config_ctx.midx_new_layer_threshold);\n+\n \tif (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \n@@ -296,6 +322,10 @@ int cmd_repack(int argc,\n \tif (geometry.split_factor) {\n \t\tif (pack_everything)\n \t\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--geometric\", \"-A/-a\");\n+\t\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL) {\n+\t\t\tgeometry.midx_layer_threshold = config_ctx.midx_new_layer_threshold;\n+\t\t\tgeometry.midx_layer_threshold_set = true;\n+\t\t}\n \t\tpack_geometry_init(&geometry, &existing, &po_args);\n \t\tpack_geometry_split(&geometry);\n \t}\n@@ -545,8 +575,11 @@ int cmd_repack(int argc,\n \t\t\t\t packtmp);\n \t/* End of pack replacement. */\n \n-\tif (delete_redundant && pack_everything & ALL_INTO_ONE)\n+\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n+\t\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL)\n+\t\t\texisting_packs_retain_midx_packs(&existing);\n \t\texisting_packs_mark_for_deletion(&existing, &names);\n+\t}\n \n \tif (write_midx != REPACK_WRITE_MIDX_NONE) {\n \t\tstruct repack_write_midx_opts opts = {\n@@ -558,8 +591,8 @@ int cmd_repack(int argc,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n-\t\t\t.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,\n-\t\t\t.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,\n+\t\t\t.midx_split_factor = config_ctx.midx_split_factor,\n+\t\t\t.midx_new_layer_threshold = config_ctx.midx_new_layer_threshold,\n \t\t\t.mode = write_midx,\n \t\t};\n \n@@ -572,11 +605,15 @@ int cmd_repack(int argc,\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\n-\t\texisting_packs_remove_redundant(&existing, packdir);\n+\t\tbool wrote_incremental_midx = write_midx == REPACK_WRITE_MIDX_INCREMENTAL;\n+\n+\t\texisting_packs_remove_redundant(&existing, packdir,\n+\t\t\t\t\t\twrote_incremental_midx);\n \n \t\tif (geometry.split_factor)\n \t\t\tpack_geometry_remove_redundant(&geometry, &names,\n-\t\t\t\t\t\t &existing, packdir);\n+\t\t\t\t\t\t &existing, packdir,\n+\t\t\t\t\t\t wrote_incremental_midx);\n \t\tif (show_progress)\n \t\t\topts |= PRUNE_PACKED_VERBOSE;\n \t\tprune_packed_objects(opts);\ndiff --git a/midx.c b/midx.c\nindex dc86c8e7fee..cd31fa20788 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -850,6 +850,37 @@ void clear_midx_file(struct repository *r)\n \tstrbuf_release(&midx);\n }\n \n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes)\n+{\n+\tstruct strbuf chain = STRBUF_INIT;\n+\n+\tget_midx_chain_filename(r->objects->sources, &chain);\n+\n+\tif (r->objects) {\n+\t\tstruct odb_source *source = r->objects->sources;\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\tif (files->packed->midx)\n+\t\t\t\tclose_midx(files->packed->midx);\n+\t\t\tfiles->packed->midx = NULL;\n+\t\t}\n+\t}\n+\n+\tif (!keep_hashes && remove_path(chain.buf))\n+\t\tdie(_(\"failed to clear multi-pack-index chain at %s\"),\n+\t\t chain.buf);\n+\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_BITMAP,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_REV,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_MIDX,\n+\t\t\t\t\t keep_hashes);\n+\n+\tstrbuf_release(&chain);\n+}\n+\n static int verify_midx_error;\n \n __attribute__((format (printf, 1, 2)))\ndiff --git a/midx.h b/midx.h\nindex 3ee12dd08ec..63853a03a47 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -9,6 +9,7 @@ struct repository;\n struct bitmapped_pack;\n struct git_hash_algo;\n struct odb_source;\n+struct strvec;\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION_V1 1\n@@ -143,6 +144,8 @@ int write_midx_file_compact(struct odb_source *source,\n \t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes);\n int verify_midx_file(struct odb_source *source, unsigned flags);\n int expire_midx_packs(struct odb_source *source, unsigned flags);\n int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags);\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex 2408b8a3cc2..2064683dcfe 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -249,7 +249,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t\t\t uint32_t pack_nr,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tconst struct git_hash_algo *algop = existing->repo->hash_algo;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -269,7 +270,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t (string_list_has_string(&existing->kept_packs, buf.buf)))\n \t\t\tcontinue;\n \n-\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf);\n+\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n \n \tstrbuf_release(&buf);\n@@ -278,12 +280,13 @@ static void remove_redundant_packs(struct packed_git **pack,\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs(geometry->pack, geometry->split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n \tremove_redundant_packs(geometry->promisor_pack, geometry->promisor_split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n }\n \n void pack_geometry_release(struct pack_geometry *geometry)\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 8f3720772b8..6a0358096ec 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -894,6 +894,7 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \tstruct midx_compaction_step *steps = NULL;\n \tstruct strbuf lock_name = STRBUF_INIT;\n \tstruct lock_file lf;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tsize_t steps_nr = 0;\n \tsize_t i;\n \tint ret = 0;\n@@ -939,11 +940,15 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n \t\t\t (uintmax_t)i);\n \t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t\tstrvec_push(&keep_hashes, step->csum);\n \t}\n \n \tcommit_lock_file(&lf);\n \n+\tclear_incremental_midx_files(opts->existing->repo, &keep_hashes);\n+\n done:\n+\tstrvec_clear(&keep_hashes);\n \tstrbuf_release(&lock_name);\n \tfor (i = 0; i < steps_nr; i++)\n \t\tmidx_compaction_step_release(&steps[i]);\ndiff --git a/repack.c b/repack.c\nindex 2ee6b51420a..571dabb665e 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -55,14 +55,18 @@ void pack_objects_args_release(struct pack_objects_args *args)\n }\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name)\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct odb_source *source = repo->objects->sources;\n \tstruct multi_pack_index *m = get_multi_pack_index(source);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && source->local && midx_contains_pack(m, buf.buf))\n+\tif (m && source->local && midx_contains_pack(m, buf.buf)) {\n \t\tclear_midx_file(repo);\n+\t\tif (!wrote_incremental_midx)\n+\t\t\tclear_incremental_midx_files(repo, NULL);\n+\t}\n \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n \tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n@@ -250,25 +254,63 @@ void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \t\t\t\t\t &existing->cruft_packs);\n }\n \n+/*\n+ * Mark every pack that is referenced by the existing MIDX chain as\n+ * retained, so that a subsequent call to\n+ * existing_packs_mark_for_deletion() will not mark them for deletion.\n+ *\n+ * This is used when writing an incremental MIDX layer on top of an\n+ * existing chain: retained layers continue to reference the same\n+ * packs on disk, so those packs must not be unlinked even if the\n+ * freshly-written pack supersedes them.\n+ */\n+void existing_packs_retain_midx_packs(struct existing_packs *existing)\n+{\n+\tstruct string_list_item *item;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tfor_each_string_list_item(item, &existing->midx_packs) {\n+\t\tstruct string_list_item *found;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addstr(&buf, item->string);\n+\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\n+\t\tfound = string_list_lookup(&existing->non_kept_packs, buf.buf);\n+\t\tif (found)\n+\t\t\texisting_packs_mark_retained(found);\n+\n+\t\tfound = string_list_lookup(&existing->cruft_packs, buf.buf);\n+\t\tif (found)\n+\t\t\texisting_packs_mark_retained(found);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+}\n+\n static void remove_redundant_packs_1(struct repository *repo,\n \t\t\t\t struct string_list *packs,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct string_list_item *item;\n \tfor_each_string_list_item(item, packs) {\n \t\tif (!existing_pack_is_marked_for_deletion(item))\n \t\t\tcontinue;\n-\t\trepack_remove_redundant_pack(repo, packdir, item->string);\n+\t\trepack_remove_redundant_pack(repo, packdir, item->string,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n }\n \n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs_1(existing->repo, &existing->non_kept_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n \tremove_redundant_packs_1(existing->repo, &existing->cruft_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n }\n \n void existing_packs_release(struct existing_packs *existing)\ndiff --git a/repack.h b/repack.h\nindex 831ccfb1c6c..f9fbc895f02 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -34,7 +34,8 @@ void prepare_pack_objects(struct child_process *cmd,\n void pack_objects_args_release(struct pack_objects_args *args);\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name);\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx);\n \n struct write_pack_opts {\n \tstruct pack_objects_args *po_args;\n@@ -83,8 +84,10 @@ void existing_packs_retain_cruft(struct existing_packs *existing,\n \t\t\t\t struct packed_git *cruft);\n void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \t\t\t\t struct string_list *names);\n+void existing_packs_retain_midx_packs(struct existing_packs *existing);\n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void existing_packs_release(struct existing_packs *existing);\n \n struct generated_pack;\n@@ -129,7 +132,8 @@ struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry);\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\ndiff --git a/t/meson.build b/t/meson.build\nindex 7528e5cda5f..25f0d823d8e 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -951,6 +951,7 @@ integration_tests = [\n 't7702-repack-cyclic-alternate.sh',\n 't7703-repack-geometric.sh',\n 't7704-repack-cruft.sh',\n+ 't7705-repack-incremental-midx.sh',\n 't7800-difftool.sh',\n 't7810-grep.sh',\n 't7811-grep-open.sh',\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nnew file mode 100755\nindex 00000000000..af1a1483b8c\n--- /dev/null\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -0,0 +1,500 @@\n+#!/bin/sh\n+\n+test_description='git repack --write-midx=incremental'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+midxdir=$packdir/multi-pack-index.d\n+midx_chain=$midxdir/multi-pack-index-chain\n+\n+# incrementally_repack N\n+#\n+# Make \"N\" new commits, each stored in their own pack, and then repacked\n+# with the --write-midx=incremental strategy.\n+incrementally_repack () {\n+\tfor i in $(test_seq 1 \"$1\")\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify || return 1\n+\tdone\n+}\n+\n+# Create packs with geometrically increasing sizes so that they\n+# satisfy the geometric progression and survive a --geometric=2\n+# repack without being rolled up. Creates 3 packs containing 1,\n+# 2, and 6 commits (3, 6, and 18 objects) respectively.\n+create_geometric_packs () {\n+\ttest_commit \"small\" &&\n+\tgit repack -d &&\n+\n+\ttest_commit_bulk --message=\"medium\" 2 &&\n+\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t--write-bitmap-index\n+}\n+\n+# create_layer \n+#\n+# Creates a new MIDX layer with the contents of \"test_commit_bulk $@\".\n+create_layer () {\n+\ttest_commit_bulk \"$@\" &&\n+\n+\tgit multi-pack-index write --incremental --bitmap\n+}\n+\n+# create_layers\n+#\n+# Reads lines of \" \" from stdin and creates a new MIDX\n+# layer for each line. See create_layer above for more.\n+create_layers () {\n+\twhile read msg nr\n+\tdo\n+\t\tcreate_layer --message=\"$msg\" \"$nr\" || return 1\n+\tdone\n+}\n+\n+test_expect_success '--write-midx=incremental requires --geometric' '\n+\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+\n+\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+'\n+\n+test_expect_success 'below layer threshold, tip packs excluded' '\n+\tgit init below-layer-threshold-tip-packs-excluded &&\n+\t(\n+\t\tcd below-layer-threshold-tip-packs-excluded &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 4 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create 3 packs forming a geometric progression by\n+\t\t# object count such that they are unmodified by the\n+\t\t# initial repack. The MIDX chain thusly contains a\n+\t\t# single layer with three packs.\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a new commit. Since the layer threshold is\n+\t\t# unmet, a new MIDX layer is added on top of the\n+\t\t# existing one.\n+\t\ttest_commit extra &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs repacked' '\n+\tgit init above-layer-threshold-tip-packs-repacked &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-repacked &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Same setup, but with the layer threshold set to 2.\n+\t\t# Since the tip MIDX layer meets that threshold, its\n+\t\t# packs are considered repack candidates.\n+\t\tcreate_geometric_packs &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Perturb the existing progression such that it is\n+\t\t# rolled up into a single new pack, invalidating the\n+\t\t# existing MIDX layer and replacing it with a new one.\n+\t\ttest_commit extra &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip layer preserved' '\n+\tgit init above-layer-threshold-tip-layer-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-layer-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\ttest_commit_bulk --message=\"medium\" 2 &&\n+\t\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Create objects to form a pack satisfying the geometric\n+\t\t# progression (thus preserving the tip layer), but not\n+\t\t# so large that it meets the layer merging condition.\n+\t\ttest_commit_bulk --message=\"small\" 1 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\ttest_line_count = 1 packs.new &&\n+\t\ttest_line_count = 3 packs.after &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs preserved' '\n+\tgit init above-layer-threshold-tip-packs-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Same setup as above, but this time the new objects do\n+\t\t# not satisfy the new layer merging condition, resulting\n+\t\t# in a new tip layer.\n+\t\ttest_commit_bulk --message=\"huge\" 18 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'new tip absorbs multiple layers' '\n+\tgit init new-tip-absorbs-multiple-layers &&\n+\t(\n+\t\tcd new-tip-absorbs-multiple-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a 4-layer chain where each layer is too small to\n+\t\t# absorb the one below it. The sizes must satisfy L(n) <\n+\t\t# L(n-1)/2 for each adjacent pair:\n+\t\t#\n+\t\t# L0 (oldest): 75 obj (25 commits)\n+\t\t# L1: 21 obj (7 commits, 21 < 75/2)\n+\t\t# L2: 9 obj (3 commits, 9 < 21/2)\n+\t\t# L3 (tip): 3 obj (1 commit, 3 < 9/2)\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tL0 25\n+\t\tL1 7\n+\t\tL2 3\n+\t\tL3 1\n+\t\tEOF\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Now add a new commit. The merging condition is\n+\t\t# satisfied between L3-L1, but violated at L0, which is\n+\t\t# too large relative to the accumulated size.\n+\t\t#\n+\t\t# As a result, the chain shrinks from 4 to 2 layers.\n+\t\ttest_commit new &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'compaction of older layers' '\n+\tgit init compaction-of-older-layers &&\n+\t(\n+\t\tcd compaction-of-older-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a chain with two small layers at the bottom\n+\t\t# and a larger barrier layer on top, producing a\n+\t\t# chain that violates the compaction invariant, since\n+\t\t# the two small layers would normally have been merged.\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 2\n+\t\ttwo 4\n+\t\tbarrier 54\n+\t\tEOF\n+\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running an incremental repack compacts the two\n+\t\t# small layers at the bottom of the chain as a\n+\t\t# separate step in the compaction plan.\n+\t\ttest_commit another &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'geometric rollup with surviving tip packs' '\n+\tgit init geometric-rollup-with-surviving-tip-packs &&\n+\t(\n+\t\tcd geometric-rollup-with-surviving-tip-packs &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create a pack large enough to anchor the geometric\n+\t\t# progression when small packs are added alongside it.\n+\t\tcreate_layer --message=\"big\" 5 &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a small number of objects such that the\n+\t\t# progression is unbothered. Note that the existing pack\n+\t\t# is considered a repack candidate as the new layer\n+\t\t# threshold is set to 1.\n+\t\ttest_commit small-1 &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before\n+\t)\n+'\n+\n+test_expect_success 'kept packs are excluded from repack' '\n+\tgit init kept-packs-excluded-from-repack &&\n+\t(\n+\t\tcd kept-packs-excluded-from-repack &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create two equal-sized packs, marking one as kept.\n+\t\tfor i in A B\n+\t\tdo\n+\t\t\ttest_commit \"$i\" && git repack -d || return 1\n+\t\tdone &&\n+\n+\t\tkeep=$(ls $packdir/pack-*.idx | head -n 1) &&\n+\t\ttouch \"${keep%.idx}.keep\" &&\n+\n+\t\t# The kept pack is excluded as a repacking candidate\n+\t\t# entirely, so no rollup occurs as there is only one\n+\t\t# non-kept pack. A new MIDX layer is written containing\n+\t\t# that pack.\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count = 1 actual.packs &&\n+\t\ttest_grep ! \"$keep\" actual.packs &&\n+\n+\t\tgit multi-pack-index verify &&\n+\n+\t\t# All objects (from both kept and non-kept packs)\n+\t\t# must still be accessible.\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'incremental MIDX with --max-pack-size' '\n+\tgit init incremental-midx-with--max-pack-size &&\n+\t(\n+\t\tcd incremental-midx-with--max-pack-size &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\t# Now add enough data that a small --max-pack-size will\n+\t\t# cause pack-objects to split its output. Create objects\n+\t\t# large enough to fill multiple packs.\n+\t\ttest-tool genrandom foo 1M >big1 &&\n+\t\ttest-tool genrandom bar 1M >big2 &&\n+\t\tgit add big1 big2 &&\n+\t\ttest_tick &&\n+\t\tgit commit -a -m \"big blobs\" &&\n+\t\tgit repack -d &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index --max-pack-size=1M &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count -gt 1 actual.packs &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'noop repack preserves valid MIDX chain' '\n+\tgit init noop-repack-preserves-valid-midx-chain &&\n+\t(\n+\t\tcd noop-repack-preserves-valid-midx-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running again with no new objects should not break\n+\t\t# the MIDX chain. It produces \"Nothing new to pack.\"\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_cmp $midx_chain.before $midx_chain &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'repack -ad removes stale incremental chain' '\n+\tgit init repack--ad-removes-stale-incremental-chain &&\n+\t(\n+\t\tcd repack--ad-removes-stale-incremental-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 1\n+\t\ttwo 1\n+\t\tEOF\n+\n+\t\ttest_path_is_file $midx_chain &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tgit repack -ad &&\n+\n+\t\ttest_path_is_missing $packdir/multi-pack-index &&\n+\t\ttest_dir_is_empty $midxdir\n+\t)\n+'\n+\n+test_expect_success 'repack -ad --write-midx=incremental is safe' '\n+\tgit init ad-incremental-midx &&\n+\t(\n+\t\tcd ad-incremental-midx &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\t# Build a MIDX chain with multiple layers referencing\n+\t\t# distinct packs.\n+\t\ttest_commit first &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_commit second &&\n+\t\tgit repack -d --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\n+\t\t# Now do a full -ad repack. The new pack contains all\n+\t\t# objects, but any retained MIDX layers still reference\n+\t\t# the now-deleted packs.\n+\t\ttest_commit third &&\n+\t\tgit repack -ad --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck &&\n+\t\tgit rev-list --all --objects >/dev/null\n+\t)\n+'\n+\n+test_expect_success 'repack rejects invalid midxSplitFactor' '\n+\ttest_when_finished \"rm -fr bad-split-factor\" &&\n+\tgit init bad-split-factor &&\n+\t(\n+\t\tcd bad-split-factor &&\n+\t\ttest_commit base &&\n+\n+\t\tfor v in 0 1 -1\n+\t\tdo\n+\t\t\ttest_must_fail git -c repack.midxSplitFactor=$v \\\n+\t\t\t\trepack -d --geometric=2 --write-midx=incremental 2>err &&\n+\t\t\ttest_grep \"invalid value for --midx-split-factor\" err ||\n+\t\t\treturn 1\n+\t\tdone\n+\t)\n+'\n+\n+test_expect_success 'repack rejects invalid midxNewLayerThreshold' '\n+\ttest_when_finished \"rm -fr bad-layer-threshold\" &&\n+\tgit init bad-layer-threshold &&\n+\t(\n+\t\tcd bad-layer-threshold &&\n+\t\ttest_commit base &&\n+\n+\t\tfor v in 0 -1\n+\t\tdo\n+\t\t\ttest_must_fail git -c repack.midxNewLayerThreshold=$v \\\n+\t\t\t\trepack -d --geometric=2 --write-midx=incremental 2>err &&\n+\t\t\ttest_grep \"invalid value for --midx-new-layer-threshold\" err ||\n+\t\t\treturn 1\n+\t\tdone\n+\t)\n+'\n+\n+test_done\n-- \n2.54.0.9.gb905fd5d0ae\n\n"},{"id":"542084","messageId":"d9acef1334ae42df45b8a4ec1e1b771b48249006.1776803827.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"[PATCH v2 16/16] repack: allow `--write-midx=incremental` without `--geometric`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T20:38:03Z","receivedAt":"2026-04-21T20:38:06Z","isPatch":true,"body":"Previously, `--write-midx=incremental` required `--geometric` and would\ndie() without it. Relax this restriction so that incremental MIDX\nrepacking can be used independently.\n\nWithout `--geometric`, the behavior is append-only: a single new MIDX\nlayer is created containing whatever packs were written by the repack\nand appended to the existing chain (or a new chain is started). Existing\nlayers are preserved as-is with no compaction or merging.\n\nImplement this via a new repack_make_midx_append_plan() that builds a\nplan consisting of a WRITE step for the freshly written packs followed\nby COPY steps for every existing MIDX layer. The existing compaction\nplan (repack_make_midx_compaction_plan) is used only when `--geometric`\nis active.\n\nUpdate the documentation to describe the behavior with and without\n`--geometric`, and replace the test that enforced the old restriction\nwith one exercising append-only incremental MIDX repacking.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-repack.adoc | 19 +++++----\n builtin/repack.c | 3 --\n repack-midx.c | 64 ++++++++++++++++++++++++++++--\n t/t7705-repack-incremental-midx.sh | 35 +++++++++++++---\n 4 files changed, 103 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 27a99cc46f4..72c42015e23 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -263,14 +263,19 @@ linkgit:git-multi-pack-index[1]).\n \n \t`incremental`;;\n \t\tWrite an incremental MIDX chain instead of a single\n-\t\tflat MIDX. This mode requires `--geometric`.\n+\t\tflat MIDX.\n +\n-The incremental mode maintains a chain of MIDX layers that is compacted\n-over time using a geometric merging strategy. Each repack creates a new\n-tip layer containing the newly written pack(s). Adjacent layers are then\n-merged whenever the newer layer's object count exceeds\n-`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n-that do not meet this condition are retained as-is.\n+Without `--geometric`, a new MIDX layer is appended to the existing\n+chain (or a new chain is started) containing whatever packs were written\n+by the repack. Existing layers are preserved as-is.\n++\n+When combined with `--geometric`, the incremental mode maintains a chain\n+of MIDX layers that is compacted over time using a geometric merging\n+strategy. Each repack creates a new tip layer containing the newly\n+written pack(s). Adjacent layers are then merged whenever the newer\n+layer's object count exceeds `1/repack.midxSplitFactor` of the next\n+deeper layer's count. Layers that do not meet this condition are\n+retained as-is.\n +\n The result is that newer (tip) layers tend to contain many small packs\n with relatively few objects, while older (deeper) layers contain fewer,\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5ffa18e085e..1524a9c13ad 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -263,9 +263,6 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n-\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n-\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n-\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 6a0358096ec..b1c429b8cee 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -555,6 +555,60 @@ static void midx_compaction_step_release(struct midx_compaction_step *step)\n \tfree(step->csum);\n }\n \n+/*\n+ * Build an append-only MIDX plan: a single WRITE step for the freshly\n+ * written packs, plus COPY steps for every existing layer. No\n+ * compaction or merging is performed.\n+ */\n+static void repack_make_midx_append_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step *step;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tif (opts->names->nr) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tuint32_t i;\n+\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_WRITE;\n+\t\tstring_list_init_dup(&step->u.write);\n+\n+\t\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"pack-%s.idx\",\n+\t\t\t\t opts->names->items[i].string);\n+\t\t\tstring_list_append(&step->u.write, buf.buf);\n+\t\t}\n+\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\tfor (; m; m = m->base_midx) {\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_COPY;\n+\t\tstep->u.copy = m;\n+\t\tstep->objects_nr = m->num_objects;\n+\t}\n+\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+}\n+\n static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\t\t\t struct midx_compaction_step **steps_p,\n \t\t\t\t\t size_t *steps_nr_p)\n@@ -911,9 +965,13 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \t}\n \n-\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n-\t\tret = error(_(\"unable to generate compaction plan\"));\n-\t\tgoto done;\n+\tif (opts->geometry->split_factor) {\n+\t\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\t\tgoto done;\n+\t\t}\n+\t} else {\n+\t\trepack_make_midx_append_plan(opts, &steps, &steps_nr);\n \t}\n \n \tfor (i = 0; i < steps_nr; i++) {\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nindex af1a1483b8c..25a8c40e8ee 100755\n--- a/t/t7705-repack-incremental-midx.sh\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -63,10 +63,36 @@ create_layers () {\n \tdone\n }\n \n-test_expect_success '--write-midx=incremental requires --geometric' '\n-\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+test_expect_success '--write-midx=incremental without --geometric' '\n+\tgit init incremental-without-geometric &&\n+\t(\n+\t\tcd incremental-without-geometric &&\n \n-\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+\t\tgit config maintenance.auto false &&\n+\n+\t\ttest_commit first &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_commit second &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# A second repack appends a new layer without\n+\t\t# disturbing the existing one.\n+\t\ttest_commit third &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\t\thead -n 1 $midx_chain.before >expect &&\n+\t\thead -n 1 $midx_chain >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit fsck\n+\t)\n '\n \n test_expect_success 'below layer threshold, tip packs excluded' '\n@@ -334,8 +360,7 @@ test_expect_success 'kept packs are excluded from repack' '\n \t\t# entirely, so no rollup occurs as there is only one\n \t\t# non-kept pack. A new MIDX layer is written containing\n \t\t# that pack.\n-\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n-\t\t\t--write-bitmap-index &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental &&\n \n \t\ttest-tool read-midx $objdir >actual &&\n \t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n-- \n2.54.0.9.gb905fd5d0ae\n"},{"id":"542085","messageId":"aefl0ZQ0H3w7xi8a@nand.local","threadId":"65383","inReplyTo":"6119f15d3e8619f75c7dfb2e5dcf67e9f5fafc41.1776803827.git.me@ttaylorr.com","subject":"Re: [PATCH v2 15/16] repack: introduce `--write-midx=incremental`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-21T21:02:09Z","receivedAt":"2026-04-21T21:02:12Z","isPatch":true,"body":"On Tue, Apr 21, 2026 at 04:37:57PM -0400, Taylor Blau wrote:\n> ---\n> Documentation/config/repack.adoc | 18 ++\n> Documentation/git-repack.adoc | 39 ++-\n> builtin/repack.c | 49 ++-\n> midx.c | 31 ++\n> midx.h | 3 +\n> repack-geometry.c | 13 +-\n> repack-midx.c | 5 +\n> repack.c | 56 +++-\n> repack.h | 10 +-\n> t/meson.build | 1 +\n> t/t7705-repack-incremental-midx.sh | 500 +++++++++++++++++++++++++++++\n> 11 files changed, 701 insertions(+), 24 deletions(-)\n> create mode 100755 t/t7705-repack-incremental-midx.sh\n\nI squashed a new test into this commit, but meant to squash it into the\nlatter. That test depends on functionality that is not yet introduced\nuntil the subsequent commit, hence t7705 fails on this commit.\n\nSince the maintainer is offline currently, I'll avoid re-submitting for\nnow since this won't get queued anyways. Once there is some review, I'll\nre-submit the series with the test squashed into the right commit,\nalongside any changes from that review.\n\nSorry about that.\n\nThanks,\nTaylor\n"},{"id":"542136","messageId":"CABPp-BEoXTukGMQNMyj1P2DNxU6KrYasq=3uWcjOgkstqeK06A@mail.gmail.com","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/16] repack: incremental MIDX/bitmap-based repacking","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2026-04-22T14:45:24Z","receivedAt":"2026-04-22T14:45:39Z","isPatch":true,"body":"On Tue, Apr 21, 2026 at 1:37 PM Taylor Blau wrote:\n>\n> [Note to the maintainer, this is rebased onto current 'master', which is\n> 94f057755b7 (Git 2.54, 2026-04-19) at the time of writing.]\n>\n> This a modest-sized reroll of my series to implement the last remaining\n> component of the incremental MIDX/bitmap-based repacking strategy that I\n> have been working on.\n>\n> The main changes since last time are described in [1], which I sent to\n> the list last week just before the 2.54 release was tagged. For\n> convenience, a copy of the main changes are below:\n>\n> - Use a strset instead of a string_list for keeping track of the MIDX\n> layers to retain when calling `either clear_midx_files_ext()` or\n> `clear_incremental_midx_files_ext()`.\n>\n> - A new patch to rewrite the logic for determining which MIDX layers\n> comprise the new chain via keep_hashes to build the array in order.\n> The subsequent patch converts that into a strvec, which no longer\n> requires direct manipulation.\n>\n> - The new \"--checksum-only\" option has been renamed to\n> \"--no-write-chain-file\", and various small implementation tweaks\n> (e.g., relying on `is_lock_file_locked()` to determine whether we\n> should update the chain file as opposed to reading the flags).\n>\n> As usual, a range-diff is included below as well for convenience. Thanks\n> in advance for reviewing!\n[...]\n> Range-diff against v1:\n[...]\n\nThis round addresses all the (admittedly minor) pieces of feedback I\nhad on v1, and I didn't spot anything new to comment on in looking\nover the changes in this round.\n"},{"id":"542461","messageId":"20260429075150.GA1267476@coredump.intra.peff.net","threadId":"65383","inReplyTo":"93e152fb6aa3bbd450d46c6cdd219dfef14d41fc.1776803827.git.me@ttaylorr.com","subject":"Re: [PATCH v2 14/16] repack: implement incremental MIDX repacking","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-29T07:51:50Z","receivedAt":"2026-04-29T07:51:52Z","isPatch":true,"body":"On Tue, Apr 21, 2026 at 04:37:54PM -0400, Taylor Blau wrote:\n\n> static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n> -\t\t\t\t\tstruct string_list *include)\n> +\t\t\t\t\tstruct string_list *include,\n> +\t\t\t\t\tstruct string_list *out)\n> {\n> \tstruct string_list_item *item;\n> \tFILE *in;\n> \tint ret;\n> \n> \tcmd->in = -1;\n> +\tif (out)\n> +\t\tcmd->out = -1;\n\nWe are already piping to the sub-command, and now we will read the\noutput. Looks like we do them sequentially:\n\n> @@ -312,6 +319,17 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n> \t\tfprintf(in, \"%s\\n\", item->string);\n> \tfclose(in);\n> \n> +\tif (out) {\n> +\t\tstruct strbuf buf = STRBUF_INIT;\n> +\t\tFILE *outf = xfdopen(cmd->out, \"r\");\n> +\n> +\t\twhile (strbuf_getline(&buf, outf) != EOF)\n> +\t\t\tstring_list_append(out, buf.buf);\n> +\t\tstrbuf_release(&buf);\n> +\n> +\t\tfclose(outf);\n> +\t}\n\nIs it possible to deadlock here where we block writing to the child, but\nthe child is blocked trying to write back to us. It's probably quite\nunlikely as it implies both pipe buffers are filled up (and we are\ncounting packs and midx hashes here, neither of which we'd expect to be\ntoo numerous).\n\nUsing pipe_command() would solve this, but it also might be impossible\nto trigger if the child reads all input before generating any output. I\n_think_ that's the case looking at cmd_multi_pack_index_write(). So\nwe're OK, but you might want to double check.\n\n> +static int midx_compaction_step_exec_compact(struct midx_compaction_step *step,\n> +\t\t\t\t\t struct repack_write_midx_opts *opts)\n> +{\n> [...]\n> +\tret = start_command(&cmd);\n> +\tif (ret)\n> +\t\tgoto out;\n\nOK, if we get an error here we'll jump to \"out\", but run-command.c will\nhave cleaned up everything about \"cmd\" that we need.\n\nBut later...\n\n> +\tout = xfdopen(cmd.out, \"r\");\n> +\twhile (strbuf_getline_lf(&buf, out) != EOF) {\n> +\t\tif (step->csum) {\n> +\t\t\tret = error(_(\"unexpected MIDX output: '%s'\"), buf.buf);\n> +\t\t\tgoto out;\n> +\t\t}\n\n...here we jump to \"out\" but the command is left running. I guess it\nwill eventually get SIGPIPE when we fclose(out), but of course we won't\nwait() for it and we're probably leaking some memory, too.\n\n> +\t}\n> +\n> +\tret = finish_command(&cmd);\n> +\n> +out:\n> +\tif (out)\n> +\t\tfclose(out);\n> +\tstrbuf_release(&buf);\n\nIt would be nice if we had an idempotent version of finish_command(),\nthat would do nothing when the program was already done. But I don't\nthink we do. We don't even clear cmd->pid when we finish the command, so\nyou can't even do something like:\n\n out:\n if (cmd->pid)\n\tfinish_command(&cmd);\n\nSo probably that \"goto out\" in the loop needs to call fclose() and\nfinish_command() itself.\n\n-Peff\n"},{"id":"542462","messageId":"20260429080821.GA1269182@coredump.intra.peff.net","threadId":"65383","inReplyTo":"44f522ea04df5176ec332b60a38d1eab9b649c30.1776803827.git.me@ttaylorr.com","subject":"Re: [PATCH v2 10/16] repack-midx: extract `repack_fill_midx_stdin_packs()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-29T08:08:21Z","receivedAt":"2026-04-29T08:08:23Z","isPatch":true,"body":"On Tue, Apr 21, 2026 at 04:37:42PM -0400, Taylor Blau wrote:\n\n> This simplifies `write_midx_included_packs()` and prepares for a\n> subsequent commit where the same helper is called with `cmd->out = -1`\n> to capture the MIDX's checksum from the command's standard output,\n> which is needed when writing MIDX layers with `--checksum-only`.\n\nThis should be --no-write-chain-file now, right? It's not used in the\ncode here, so it's just a commit message fixup.\n\n-Peff\n"},{"id":"542463","messageId":"20260429081017.GB1269182@coredump.intra.peff.net","threadId":"65383","inReplyTo":"93e152fb6aa3bbd450d46c6cdd219dfef14d41fc.1776803827.git.me@ttaylorr.com","subject":"Re: [PATCH v2 14/16] repack: implement incremental MIDX repacking","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-29T08:10:17Z","receivedAt":"2026-04-29T08:10:18Z","isPatch":true,"body":"On Tue, Apr 21, 2026 at 04:37:54PM -0400, Taylor Blau wrote:\n\n> Unlike the default mode which writes a single flat MIDX, the incremental\n> mode constructs a compaction plan that determines which MIDX layers to\n> write, compact, or copy, and then executes each step using `git\n> multi-pack-index` subcommands with the --checksum-only flag.\n\nThis should be --no-write-chain-file, I think.\n\nDitto here:\n\n> After writing the new layer, the strategy is evaluated among the\n> existing MIDX layers in order from oldest to newest. Each step that\n> writes a new MIDX layer uses \"--checksum-only\" to avoid updating the\n> multi-pack-index-chain file. After all steps are complete, the new chain\n> file is written and then atomically moved into place.\n\nIn the code I think it is all good, though:\n\n> +\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\", NULL);\n\n-Peff\n"},{"id":"542464","messageId":"20260429081030.GA1268740@coredump.intra.peff.net","threadId":"65383","inReplyTo":"cover.1776803827.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/16] repack: incremental MIDX/bitmap-based repacking","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-29T08:10:30Z","receivedAt":"2026-04-29T08:10:32Z","isPatch":true,"body":"On Tue, Apr 21, 2026 at 04:37:08PM -0400, Taylor Blau wrote:\n\n> The main changes since last time are described in [1], which I sent to\n> the list last week just before the 2.54 release was tagged. For\n> convenience, a copy of the main changes are below:\n> \n> - Use a strset instead of a string_list for keeping track of the MIDX\n> layers to retain when calling `either clear_midx_files_ext()` or\n> `clear_incremental_midx_files_ext()`.\n> \n> - A new patch to rewrite the logic for determining which MIDX layers\n> comprise the new chain via keep_hashes to build the array in order.\n> The subsequent patch converts that into a strvec, which no longer\n> requires direct manipulation.\n> \n> - The new \"--checksum-only\" option has been renamed to\n> \"--no-write-chain-file\", and various small implementation tweaks\n> (e.g., relying on `is_lock_file_locked()` to determine whether we\n> should update the chain file as opposed to reading the flags).\n\nThanks, I think that addresses my review from the previous round.\n\nI looked over the final few patches. Aside from a small nit about a\npossible leak, I didn't notice anything obviously wrong. There's enough\nnew and tricky concepts that I wouldn't guarantee there's not a subtle\ncorner case lurking, but I don't think me staring at it any longer is\ngoing to yield any further insight.\n\n-Peff\n"},{"id":"542489","messageId":"afKI6rWBr5A7QdLc@nand.local","threadId":"65383","inReplyTo":"20260429080821.GA1269182@coredump.intra.peff.net","subject":"Re: [PATCH v2 10/16] repack-midx: extract `repack_fill_midx_stdin_packs()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-29T22:40:42Z","receivedAt":"2026-04-29T22:40:45Z","isPatch":true,"body":"On Wed, Apr 29, 2026 at 04:08:21AM -0400, Jeff King wrote:\n> On Tue, Apr 21, 2026 at 04:37:42PM -0400, Taylor Blau wrote:\n>\n> > This simplifies `write_midx_included_packs()` and prepares for a\n> > subsequent commit where the same helper is called with `cmd->out = -1`\n> > to capture the MIDX's checksum from the command's standard output,\n> > which is needed when writing MIDX layers with `--checksum-only`.\n>\n> This should be --no-write-chain-file now, right? It's not used in the\n> code here, so it's just a commit message fixup.\n\nYup, good spotting. Fixed.\n\nThanks,\nTaylor\n"},{"id":"542491","messageId":"afKV/SthrwsHDuWI@nand.local","threadId":"65383","inReplyTo":"20260429075150.GA1267476@coredump.intra.peff.net","subject":"Re: [PATCH v2 14/16] repack: implement incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-29T23:36:29Z","receivedAt":"2026-04-29T23:36:31Z","isPatch":true,"body":"On Wed, Apr 29, 2026 at 03:51:50AM -0400, Jeff King wrote:\n> > @@ -312,6 +319,17 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n> > \t\tfprintf(in, \"%s\\n\", item->string);\n> > \tfclose(in);\n> >\n> > +\tif (out) {\n> > +\t\tstruct strbuf buf = STRBUF_INIT;\n> > +\t\tFILE *outf = xfdopen(cmd->out, \"r\");\n> > +\n> > +\t\twhile (strbuf_getline(&buf, outf) != EOF)\n> > +\t\t\tstring_list_append(out, buf.buf);\n> > +\t\tstrbuf_release(&buf);\n> > +\n> > +\t\tfclose(outf);\n> > +\t}\n>\n> Is it possible to deadlock here where we block writing to the child, but\n> the child is blocked trying to write back to us. It's probably quite\n> unlikely as it implies both pipe buffers are filled up (and we are\n> counting packs and midx hashes here, neither of which we'd expect to be\n> too numerous).\n>\n> Using pipe_command() would solve this, but it also might be impossible\n> to trigger if the child reads all input before generating any output. I\n> _think_ that's the case looking at cmd_multi_pack_index_write(). So\n> we're OK, but you might want to double check.\n\nI think in theory this could deadlock, though in practice it is highly\nunlikely. Regardless, using pipe_command() here is straightforward, and\nguarantees that we'll avoid a nasty deadlock, so let's do that.\n\nThe resulting code is a lot easier to read, too, which is nice:\n\n--- 8< ---\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 8f3720772b8..9db59b18334 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -300,37 +300,27 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n \t\t\t\t\tstruct string_list *include,\n \t\t\t\t\tstruct string_list *out)\n {\n+\tstruct strbuf in_buf = STRBUF_INIT;\n+\tstruct strbuf out_buf = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in;\n \tint ret;\n\n-\tcmd->in = -1;\n-\tif (out)\n-\t\tcmd->out = -1;\n-\n \tstrvec_push(&cmd->args, \"--stdin-packs\");\n\n-\tret = start_command(cmd);\n-\tif (ret)\n-\t\treturn ret;\n-\n-\tin = xfdopen(cmd->in, \"w\");\n \tfor_each_string_list_item(item, include)\n-\t\tfprintf(in, \"%s\\n\", item->string);\n-\tfclose(in);\n+\t\tstrbuf_addf(&in_buf, \"%s\\n\", item->string);\n\n-\tif (out) {\n-\t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tFILE *outf = xfdopen(cmd->out, \"r\");\n+\tret = pipe_command(cmd, in_buf.buf, in_buf.len,\n+\t\t\t out ? &out_buf : NULL, 0, NULL, 0);\n\n-\t\twhile (strbuf_getline(&buf, outf) != EOF)\n-\t\t\tstring_list_append(out, buf.buf);\n-\t\tstrbuf_release(&buf);\n+\tif (out)\n+\t\tstring_list_split_f(out, out_buf.buf, \"\\n\", -1,\n+\t\t\t\t STRING_LIST_SPLIT_NONEMPTY);\n\n-\t\tfclose(outf);\n-\t}\n+\tstrbuf_release(&in_buf);\n+\tstrbuf_release(&out_buf);\n\n-\treturn finish_command(cmd);\n+\treturn ret;\n }\n\n static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n--- >8 ---\n\n> > +static int midx_compaction_step_exec_compact(struct midx_compaction_step *step,\n> > +\t\t\t\t\t struct repack_write_midx_opts *opts)\n> > +{\n> > [...]\n> > +\tret = start_command(&cmd);\n> > +\tif (ret)\n> > +\t\tgoto out;\n>\n> OK, if we get an error here we'll jump to \"out\", but run-command.c will\n> have cleaned up everything about \"cmd\" that we need.\n>\n> But later...\n>\n> > +\tout = xfdopen(cmd.out, \"r\");\n> > +\twhile (strbuf_getline_lf(&buf, out) != EOF) {\n> > +\t\tif (step->csum) {\n> > +\t\t\tret = error(_(\"unexpected MIDX output: '%s'\"), buf.buf);\n> > +\t\t\tgoto out;\n> > +\t\t}\n>\n> ...here we jump to \"out\" but the command is left running. I guess it\n> will eventually get SIGPIPE when we fclose(out), but of course we won't\n> wait() for it and we're probably leaking some memory, too.\n\nGood catch, I fixed this up in the way you suggested to call\n`fclose(out)` and `finish_command()` within this loop before jumping\nout.\n\nThanks,\nTaylor\n"},{"id":"542492","messageId":"afKWrukfb6WNcE1T@nand.local","threadId":"65383","inReplyTo":"20260429081017.GB1269182@coredump.intra.peff.net","subject":"Re: [PATCH v2 14/16] repack: implement incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-29T23:39:26Z","receivedAt":"2026-04-29T23:39:28Z","isPatch":true,"body":"On Wed, Apr 29, 2026 at 04:10:17AM -0400, Jeff King wrote:\n> On Tue, Apr 21, 2026 at 04:37:54PM -0400, Taylor Blau wrote:\n>\n> > Unlike the default mode which writes a single flat MIDX, the incremental\n> > mode constructs a compaction plan that determines which MIDX layers to\n> > write, compact, or copy, and then executes each step using `git\n> > multi-pack-index` subcommands with the --checksum-only flag.\n>\n> This should be --no-write-chain-file, I think.\n>\n> Ditto here:\n\nYup, thanks for spotting.\n\n> > After writing the new layer, the strategy is evaluated among the\n> > existing MIDX layers in order from oldest to newest. Each step that\n> > writes a new MIDX layer uses \"--checksum-only\" to avoid updating the\n> > multi-pack-index-chain file. After all steps are complete, the new chain\n> > file is written and then atomically moved into place.\n>\n> In the code I think it is all good, though:\n>\n> > +\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\", NULL);\n\nHeh. Clearly I 'git grep'd through the code, but didn't adjust the\ncommit messages. These should all be fixed, and now by the end of the\nseries we have:\n\n $ git log -p @{u}.. | grep -c ..checksum.only\n 0\n\nThanks,\nTaylor\n"},{"id":"542493","messageId":"cover.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH v3 00/16] repack: incremental MIDX/bitmap-based repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:04Z","receivedAt":"2026-04-30T00:13:07Z","isPatch":true,"body":"This a small reroll of my series to implement the last remaining\ncomponent of the incremental MIDX/bitmap-based repacking strategy that I\nhave been working on.\n\nI expect that this should be the final reroll absent any late-breaking\nfeedback. The only changes since last time are the following:\n\n - Various stale references to \"--checksum-only\" have been eradicated\n from commit message(s).\n\n - `repack_prepare_midx_command()` now uses `pipe_command()` to\n eliminate the possibility of a deadlock.\n\n - `write_midx_included_packs()` now cleans up after itself properly\n when receiving multiple lines of output.\n\n - The errant test in t7705 was moved to the final commit (where it\n belongs) instead of the penultimate one.\n\nAs usual, a range-diff is included below as well for convenience. Thanks\nin advance for reviewing!\n\nTaylor Blau (16):\n midx-write: handle noop writes when converting incremental chains\n midx: use `strset` for retained MIDX files\n midx: build `keep_hashes` array in order\n midx: use `strvec` for `keep_hashes`\n midx: introduce `--no-write-chain-file` for incremental MIDX writes\n midx: support custom `--base` for incremental MIDX writes\n repack: track the ODB source via existing_packs\n midx: expose `midx_layer_contains_pack()`\n repack-midx: factor out `repack_prepare_midx_command()`\n repack-midx: extract `repack_fill_midx_stdin_packs()`\n repack-geometry: prepare for incremental MIDX repacking\n builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`\n packfile: ensure `close_pack_revindex()` frees in-memory revindex\n repack: implement incremental MIDX repacking\n repack: introduce `--write-midx=incremental`\n repack: allow `--write-midx=incremental` without `--geometric`\n\n Documentation/config/repack.adoc | 18 +\n Documentation/git-multi-pack-index.adoc | 32 +-\n Documentation/git-repack.adoc | 44 +-\n builtin/multi-pack-index.c | 48 +-\n builtin/repack.c | 102 +++-\n midx-write.c | 206 ++++---\n midx.c | 104 ++--\n midx.h | 11 +-\n packfile.c | 2 +\n repack-geometry.c | 48 +-\n repack-midx.c | 710 +++++++++++++++++++++++-\n repack.c | 58 +-\n repack.h | 26 +-\n t/meson.build | 1 +\n t/t5334-incremental-multi-pack-index.sh | 63 +++\n t/t5335-compact-multi-pack-index.sh | 113 ++++\n t/t7705-repack-incremental-midx.sh | 525 ++++++++++++++++++\n 17 files changed, 1911 insertions(+), 200 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\nRange-diff against v2:\n -: ----------- > 1: d6c27317c25 midx-write: handle noop writes when converting incremental chains\n -: ----------- > 2: 629c8d23116 midx: use `strset` for retained MIDX files\n -: ----------- > 3: e303bf6a4ac midx: build `keep_hashes` array in order\n -: ----------- > 4: 42d76c70060 midx: use `strvec` for `keep_hashes`\n -: ----------- > 5: 2c80aa34fac midx: introduce `--no-write-chain-file` for incremental MIDX writes\n 7: d9acef1334a ! 6: 2a05f4b86f3 repack: allow `--write-midx=incremental` without `--geometric`\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - repack: allow `--write-midx=incremental` without `--geometric`\n + midx: support custom `--base` for incremental MIDX writes\n \n - Previously, `--write-midx=incremental` required `--geometric` and would\n - die() without it. Relax this restriction so that incremental MIDX\n - repacking can be used independently.\n + Both `compact` and `write --incremental` fix the base of the resulting\n + MIDX layer: `compact` always places the compacted result on top of\n + \"from's\" immediate parent in the chain, and `write --incremental` always\n + appends a new layer to the existing tip. In both cases the base is not\n + configurable.\n \n - Without `--geometric`, the behavior is append-only: a single new MIDX\n - layer is created containing whatever packs were written by the repack\n - and appended to the existing chain (or a new chain is started). Existing\n - layers are preserved as-is with no compaction or merging.\n + Future callers need additional flexibility. For instance, the incremental\n + MIDX-based repacking code may wish to write a layer based on some\n + intermediate ancestor rather than the current tip, or produce a root\n + layer when replacing the bottommost entries in the chain.\n \n - Implement this via a new repack_make_midx_append_plan() that builds a\n - plan consisting of a WRITE step for the freshly written packs followed\n - by COPY steps for every existing MIDX layer. The existing compaction\n - plan (repack_make_midx_compaction_plan) is used only when `--geometric`\n - is active.\n + Introduce a new `--base` option for both subcommands to specify the\n + checksum of the MIDX layer to use as the base. The given checksum must\n + refer to a valid layer in the MIDX chain that is an ancestor of the\n + topmost layer being written or compacted.\n \n - Update the documentation to describe the behavior with and without\n - `--geometric`, and replace the test that enforced the old restriction\n - with one exercising append-only incremental MIDX repacking.\n + The special value \"none\" is accepted to produce a root layer with no\n + parent. This will be needed when the incremental repacking machinery\n + determines that the bottommost layers of the chain should be replaced.\n +\n + If no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\n + immediate parent in the chain, and `write` appends to the existing tip.\n +\n + For the `write` subcommand, `--base` requires `--no-write-chain-file`. A plain\n + `write --incremental` appends a new layer to the live chain tip with no\n + mechanism to atomically replace it; overriding the base would produce a\n + layer that does not extend the tip, breaking chain invariants. With\n + `--no-write-chain-file` the chain is left unmodified and the caller is\n + responsible for assembling a valid chain.\n +\n + For `compact`, no such restriction applies. The compaction operation\n + atomically replaces the compacted range in the chain file, so writing\n + the result on top of any valid ancestor preserves chain invariants.\n \n Signed-off-by: Taylor Blau \n \n - ## Documentation/git-repack.adoc ##\n -@@ Documentation/git-repack.adoc: linkgit:git-multi-pack-index[1]).\n + ## Documentation/git-multi-pack-index.adoc ##\n +@@ Documentation/git-multi-pack-index.adoc: SYNOPSIS\n + 'git multi-pack-index' [] write [--preferred-pack=]\n + \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n + \t\t [--refs-snapshot=] [--[no-]write-chain-file]\n ++\t\t\t [--base=]\n + 'git multi-pack-index' [] compact [--[no-]incremental]\n +-\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n ++\t\t [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\n ++\t\t\t \n + 'git multi-pack-index' [] verify\n + 'git multi-pack-index' [] expire\n + 'git multi-pack-index' [] repack [--batch-size=]\n +@@ Documentation/git-multi-pack-index.adoc: marker).\n + \t\tThe checksum of the new layer is printed to standard\n + \t\toutput, allowing the caller to assemble and write the\n + \t\tchain itself. Requires `--incremental`.\n ++\n ++\t--base=::\n ++\t\tSpecify the checksum of an existing MIDX layer to use\n ++\t\tas the base when writing a new incremental layer.\n ++\t\tThe special value `none` indicates that the new layer\n ++\t\tshould have no base (i.e., it becomes a root layer).\n ++\t\tRequires `--no-write-chain-file`.\n + --\n \n - \t`incremental`;;\n - \t\tWrite an incremental MIDX chain instead of a single\n --\t\tflat MIDX. This mode requires `--geometric`.\n -+\t\tflat MIDX.\n - +\n --The incremental mode maintains a chain of MIDX layers that is compacted\n --over time using a geometric merging strategy. Each repack creates a new\n --tip layer containing the newly written pack(s). Adjacent layers are then\n --merged whenever the newer layer's object count exceeds\n --`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n --that do not meet this condition are retained as-is.\n -+Without `--geometric`, a new MIDX layer is appended to the existing\n -+chain (or a new chain is started) containing whatever packs were written\n -+by the repack. Existing layers are preserved as-is.\n -++\n -+When combined with `--geometric`, the incremental mode maintains a chain\n -+of MIDX layers that is compacted over time using a geometric merging\n -+strategy. Each repack creates a new tip layer containing the newly\n -+written pack(s). Adjacent layers are then merged whenever the newer\n -+layer's object count exceeds `1/repack.midxSplitFactor` of the next\n -+deeper layer's count. Layers that do not meet this condition are\n -+retained as-is.\n + compact::\n +@@ Documentation/git-multi-pack-index.adoc: compact::\n + \t\tMIDX layer but do not update the multi-pack-index-chain\n + \t\tfile. The checksum of the new layer is printed to\n + \t\tstandard output. Requires `--incremental`.\n ++\n ++\t--base=::\n ++\t\tSpecify the checksum of an existing MIDX layer to use\n ++\t\tas the base for the compacted result, instead of using\n ++\t\tthe immediate parent of ``. The special value\n ++\t\t`none` indicates that the result should have no base.\n + --\n +\n - The result is that newer (tip) layers tend to contain many small packs\n - with relatively few objects, while older (deeper) layers contain fewer,\n + Note that the compact command requires writing a version-2 midx that\n \n - ## builtin/repack.c ##\n -@@ builtin/repack.c: int cmd_repack(int argc,\n - \tif (pack_everything & PACK_CRUFT)\n - \t\tpack_everything |= ALL_INTO_ONE;\n - \n --\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n --\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n --\n - \tif (write_bitmaps < 0) {\n - \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n - \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n -\n - ## repack-midx.c ##\n -@@ repack-midx.c: static void midx_compaction_step_release(struct midx_compaction_step *step)\n - \tfree(step->csum);\n - }\n + ## builtin/multi-pack-index.c ##\n +@@\n + #define BUILTIN_MIDX_WRITE_USAGE \\\n + \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n + \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n +-\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n ++\t \" [--refs-snapshot=] [--[no-]write-chain-file]\\n\" \\\n ++\t \" [--base=]\")\n \n -+/*\n -+ * Build an append-only MIDX plan: a single WRITE step for the freshly\n -+ * written packs, plus COPY steps for every existing layer. No\n -+ * compaction or merging is performed.\n -+ */\n -+static void repack_make_midx_append_plan(struct repack_write_midx_opts *opts,\n -+\t\t\t\t\t struct midx_compaction_step **steps_p,\n -+\t\t\t\t\t size_t *steps_nr_p)\n -+{\n -+\tstruct multi_pack_index *m;\n -+\tstruct midx_compaction_step *steps = NULL;\n -+\tstruct midx_compaction_step *step;\n -+\tsize_t steps_nr = 0, steps_alloc = 0;\n -+\n -+\todb_reprepare(opts->existing->repo->objects);\n -+\tm = get_multi_pack_index(opts->existing->source);\n + #define BUILTIN_MIDX_COMPACT_USAGE \\\n + \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n +-\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n ++\t \" [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\\n\" \\\n ++\t \" \")\n + \n + #define BUILTIN_MIDX_VERIFY_USAGE \\\n + \tN_(\"git multi-pack-index [] verify\")\n +@@ builtin/multi-pack-index.c: static char const * const builtin_multi_pack_index_usage[] = {\n + static struct opts_multi_pack_index {\n + \tchar *object_dir;\n + \tconst char *preferred_pack;\n ++\tconst char *incremental_base;\n + \tchar *refs_snapshot;\n + \tunsigned long batch_size;\n + \tunsigned flags;\n +@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, const char **argv,\n + \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n + \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n + \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n ++\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n ++\t\t\t N_(\"base MIDX for incremental writes\")),\n + \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n + \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n + \t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n +@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, const char **argv,\n + \t\t\t\t options);\n + \t}\n + \n ++\tif (opts.incremental_base &&\n ++\t !(opts.flags & MIDX_WRITE_NO_CHAIN)) {\n ++\t\terror(_(\"cannot use --base without --no-write-chain-file\"));\n ++\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n ++\t\t\t\t options);\n ++\t}\n +\n -+\tif (opts->names->nr) {\n -+\t\tstruct strbuf buf = STRBUF_INIT;\n -+\t\tuint32_t i;\n + \tsource = handle_object_dir_option(repo);\n + \n + \tFREE_AND_NULL(options);\n +@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, const char **argv,\n + \n + \t\tret = write_midx_file_only(source, &packs,\n + \t\t\t\t\t opts.preferred_pack,\n +-\t\t\t\t\t opts.refs_snapshot, opts.flags);\n ++\t\t\t\t\t opts.refs_snapshot,\n ++\t\t\t\t\t opts.incremental_base, opts.flags);\n + \n + \t\tstring_list_clear(&packs, 0);\n + \t\tfree(opts.refs_snapshot);\n +@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_compact(int argc, const char **argv,\n + \n + \tstruct option *options;\n + \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n ++\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n ++\t\t\t N_(\"base MIDX for incremental writes\")),\n + \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n + \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n + \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n +@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_compact(int argc, const char **argv,\n + \t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n + \t}\n + \n +-\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n ++\tret = write_midx_file_compact(source, from_midx, to_midx,\n ++\t\t\t\t opts.incremental_base, opts.flags);\n + \n + \treturn ret;\n + }\n +\n + ## midx-write.c ##\n +@@ midx-write.c: struct write_midx_opts {\n + \n + \tconst char *preferred_pack_name;\n + \tconst char *refs_snapshot;\n ++\tconst char *incremental_base;\n + \tunsigned flags;\n + };\n + \n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \n + \t/*\n + \t * If compacting MIDX layer(s) in the range [from, to], then the\n +-\t * compacted MIDX will share the same base MIDX as 'from'.\n ++\t * compacted MIDX will share the same base MIDX as 'from',\n ++\t * unless a custom --base is specified (see below).\n + \t */\n + \tif (ctx.compact)\n + \t\tctx.base_midx = ctx.compact_from->base_midx;\n + \n ++\tif (opts->incremental_base) {\n ++\t\tif (!strcmp(opts->incremental_base, \"none\")) {\n ++\t\t\tctx.base_midx = NULL;\n ++\t\t} else {\n ++\t\t\twhile (ctx.base_midx) {\n ++\t\t\t\tconst char *cmp = midx_get_checksum_hex(ctx.base_midx);\n ++\t\t\t\tif (!strcmp(opts->incremental_base, cmp))\n ++\t\t\t\t\tbreak;\n +\n -+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n ++\t\t\t\tctx.base_midx = ctx.base_midx->base_midx;\n ++\t\t\t}\n +\n -+\t\tstep = &steps[steps_nr++];\n -+\t\tmemset(step, 0, sizeof(*step));\n ++\t\t\tif (!ctx.base_midx) {\n ++\t\t\t\terror(_(\"could not find base MIDX '%s'\"),\n ++\t\t\t\t opts->incremental_base);\n ++\t\t\t\tgoto cleanup;\n ++\t\t\t}\n ++\t\t}\n ++\t}\n +\n -+\t\tstep->type = MIDX_COMPACTION_STEP_WRITE;\n -+\t\tstring_list_init_dup(&step->u.write);\n + \tctx.nr = 0;\n + \tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n + \tctx.info = NULL;\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \n + int write_midx_file(struct odb_source *source,\n + \t\t const char *preferred_pack_name,\n +-\t\t const char *refs_snapshot, unsigned flags)\n ++\t\t const char *refs_snapshot,\n ++\t\t unsigned flags)\n + {\n + \tstruct write_midx_opts opts = {\n + \t\t.source = source,\n +@@ midx-write.c: int write_midx_file(struct odb_source *source,\n + int write_midx_file_only(struct odb_source *source,\n + \t\t\t struct string_list *packs_to_include,\n + \t\t\t const char *preferred_pack_name,\n +-\t\t\t const char *refs_snapshot, unsigned flags)\n ++\t\t\t const char *refs_snapshot,\n ++\t\t\t const char *incremental_base,\n ++\t\t\t unsigned flags)\n + {\n + \tstruct write_midx_opts opts = {\n + \t\t.source = source,\n + \t\t.packs_to_include = packs_to_include,\n + \t\t.preferred_pack_name = preferred_pack_name,\n + \t\t.refs_snapshot = refs_snapshot,\n ++\t\t.incremental_base = incremental_base,\n + \t\t.flags = flags,\n + \t};\n + \n +@@ midx-write.c: int write_midx_file_only(struct odb_source *source,\n + int write_midx_file_compact(struct odb_source *source,\n + \t\t\t struct multi_pack_index *from,\n + \t\t\t struct multi_pack_index *to,\n ++\t\t\t const char *incremental_base,\n + \t\t\t unsigned flags)\n + {\n + \tstruct write_midx_opts opts = {\n + \t\t.source = source,\n + \t\t.compact_from = from,\n + \t\t.compact_to = to,\n ++\t\t.incremental_base = incremental_base,\n + \t\t.flags = flags | MIDX_WRITE_COMPACT,\n + \t};\n + \n +\n + ## midx.h ##\n +@@ midx.h: int write_midx_file(struct odb_source *source,\n + int write_midx_file_only(struct odb_source *source,\n + \t\t\t struct string_list *packs_to_include,\n + \t\t\t const char *preferred_pack_name,\n +-\t\t\t const char *refs_snapshot, unsigned flags);\n ++\t\t\t const char *refs_snapshot,\n ++\t\t\t const char *incremental_base,\n ++\t\t\t unsigned flags);\n + int write_midx_file_compact(struct odb_source *source,\n + \t\t\t struct multi_pack_index *from,\n + \t\t\t struct multi_pack_index *to,\n ++\t\t\t const char *incremental_base,\n + \t\t\t unsigned flags);\n + void clear_midx_file(struct repository *r);\n + int verify_midx_file(struct odb_source *source, unsigned flags);\n +\n + ## t/t5334-incremental-multi-pack-index.sh ##\n +@@ t/t5334-incremental-multi-pack-index.sh: test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file\n + \ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n + '\n + \n ++test_expect_success 'write MIDX layer with --base without --no-write-chain-file' '\n ++\ttest_must_fail git multi-pack-index write --bitmap --incremental \\\n ++\t\t--base=none 2>err &&\n ++\ttest_grep \"cannot use --base without --no-write-chain-file\" err\n ++'\n +\n -+\t\tfor (i = 0; i < opts->names->nr; i++) {\n -+\t\t\tstrbuf_reset(&buf);\n -+\t\t\tstrbuf_addf(&buf, \"pack-%s.idx\",\n -+\t\t\t\t opts->names->items[i].string);\n -+\t\t\tstring_list_append(&step->u.write, buf.buf);\n -+\t\t}\n ++test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '\n ++\ttest_commit base-none &&\n ++\tgit repack -d &&\n +\n -+\t\tstrbuf_release(&buf);\n -+\t}\n ++\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n ++\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n ++\t\t--no-write-chain-file --base=none)\" &&\n +\n -+\tfor (; m; m = m->base_midx) {\n -+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n ++\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n ++\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n ++'\n +\n -+\t\tstep = &steps[steps_nr++];\n -+\t\tmemset(step, 0, sizeof(*step));\n ++test_expect_success 'write MIDX layer with --base= and --no-write-chain-file' '\n ++\ttest_commit base-hash &&\n ++\tgit repack -d &&\n +\n -+\t\tstep->type = MIDX_COMPACTION_STEP_COPY;\n -+\t\tstep->u.copy = m;\n -+\t\tstep->objects_nr = m->num_objects;\n -+\t}\n ++\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n ++\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n ++\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n +\n -+\t*steps_p = steps;\n -+\t*steps_nr_p = steps_nr;\n -+}\n ++\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n ++\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n ++'\n +\n - static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n - \t\t\t\t\t struct midx_compaction_step **steps_p,\n - \t\t\t\t\t size_t *steps_nr_p)\n -@@ repack-midx.c: static int write_midx_incremental(struct repack_write_midx_opts *opts)\n - \t\tgoto done;\n - \t}\n + for reuse in false single multi\n + do\n + \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\n +\n + ## t/t5335-compact-multi-pack-index.sh ##\n +@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \n --\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n --\t\tret = error(_(\"unable to generate compaction plan\"));\n --\t\tgoto done;\n -+\tif (opts->geometry->split_factor) {\n -+\t\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n -+\t\t\tret = error(_(\"unable to generate compaction plan\"));\n -+\t\t\tgoto done;\n -+\t\t}\n -+\t} else {\n -+\t\trepack_make_midx_append_plan(opts, &steps, &steps_nr);\n - \t}\n + \t\tlayer=\"$(git multi-pack-index compact --incremental \\\n + \t\t\t--no-write-chain-file \\\n ++\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n + \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n + \t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n \n - \tfor (i = 0; i < steps_nr; i++) {\n -\n - ## t/t7705-repack-incremental-midx.sh ##\n -@@ t/t7705-repack-incremental-midx.sh: create_layers () {\n - \tdone\n - }\n +@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with --no-write-chain-file' '\n + \t)\n + '\n \n --test_expect_success '--write-midx=incremental requires --geometric' '\n --\ttest_must_fail git repack --write-midx=incremental 2>err &&\n -+test_expect_success '--write-midx=incremental without --geometric' '\n -+\tgit init incremental-without-geometric &&\n ++test_expect_success 'MIDX compaction with --base' '\n ++\tgit init midx-compact-with--base &&\n +\t(\n -+\t\tcd incremental-without-geometric &&\n - \n --\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n ++\t\tcd midx-compact-with--base &&\n ++\n +\t\tgit config maintenance.auto false &&\n +\n -+\t\ttest_commit first &&\n -+\t\tgit repack -d &&\n ++\t\twrite_packs A B C D &&\n +\n -+\t\ttest_commit second &&\n -+\t\tgit repack --write-midx=incremental &&\n ++\t\ttest_line_count = 4 \"$midx_chain\" &&\n +\n -+\t\tgit multi-pack-index verify &&\n -+\t\ttest_line_count = 1 $midx_chain &&\n -+\t\tcp $midx_chain $midx_chain.before &&\n ++\t\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n +\n -+\t\t# A second repack appends a new layer without\n -+\t\t# disturbing the existing one.\n -+\t\ttest_commit third &&\n -+\t\tgit repack --write-midx=incremental &&\n -+\n -+\t\tgit multi-pack-index verify &&\n ++\t\tgit multi-pack-index compact --incremental \\\n ++\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n ++\t\t\t\"$(nth_line 3 \"$midx_chain\")\" \\\n ++\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n +\t\ttest_line_count = 2 $midx_chain &&\n -+\t\thead -n 1 $midx_chain.before >expect &&\n -+\t\thead -n 1 $midx_chain >actual &&\n -+\t\ttest_cmp expect actual &&\n +\n -+\t\tgit fsck\n ++\t\tnth_line 1 \"$midx_chain.bak\" >expect &&\n ++\t\tnth_line 1 \"$midx_chain\" >actual &&\n ++\n ++\t\ttest_cmp expect actual\n +\t)\n - '\n - \n - test_expect_success 'below layer threshold, tip packs excluded' '\n -@@ t/t7705-repack-incremental-midx.sh: test_expect_success 'kept packs are excluded from repack' '\n - \t\t# entirely, so no rollup occurs as there is only one\n - \t\t# non-kept pack. A new MIDX layer is written containing\n - \t\t# that pack.\n --\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n --\t\t\t--write-bitmap-index &&\n -+\t\tgit repack --geometric=2 -d --write-midx=incremental &&\n - \n - \t\ttest-tool read-midx $objdir >actual &&\n - \t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n ++'\n ++\n ++test_expect_success 'MIDX compaction with --base=none' '\n ++\tgit init midx-compact-base-none &&\n ++\t(\n ++\t\tcd midx-compact-base-none &&\n ++\n ++\t\tgit config maintenance.auto false &&\n ++\n ++\t\twrite_packs A B C D &&\n ++\n ++\t\ttest_line_count = 4 $midx_chain &&\n ++\n ++\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n ++\n ++\t\t# Compact the two bottommost layers (A and B) into a new\n ++\t\t# root layer with no parent.\n ++\t\tgit multi-pack-index compact --incremental \\\n ++\t\t\t--base=none \\\n ++\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n ++\t\t\t\"$(nth_line 2 \"$midx_chain\")\" &&\n ++\n ++\t\ttest_line_count = 3 $midx_chain &&\n ++\n ++\t\t# The upper layers (C and D) should be preserved\n ++\t\t# unchanged.\n ++\t\tnth_line 3 \"$midx_chain.bak\" >expect &&\n ++\t\tnth_line 4 \"$midx_chain.bak\" >>expect &&\n ++\t\tnth_line 2 \"$midx_chain\" >actual &&\n ++\t\tnth_line 3 \"$midx_chain\" >>actual &&\n ++\n ++\t\ttest_cmp expect actual\n ++\t)\n ++'\n ++\n ++test_expect_success 'MIDX compaction with bogus --base checksum' '\n ++\tgit init midx-compact-bogus-base &&\n ++\t(\n ++\t\tcd midx-compact-bogus-base &&\n ++\n ++\t\tgit config maintenance.auto false &&\n ++\n ++\t\twrite_packs A B C &&\n ++\n ++\t\ttest_must_fail git multi-pack-index compact --incremental \\\n ++\t\t\t--base=deadbeef \\\n ++\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n ++\t\t\t\"$(nth_line 3 \"$midx_chain\")\" 2>err &&\n ++\t\ttest_grep \"could not find base MIDX\" err\n ++\t)\n ++'\n ++\n + test_done\n -: ----------- > 7: 92aba3d366f repack: track the ODB source via existing_packs\n -: ----------- > 8: d3ac65c1f11 midx: expose `midx_layer_contains_pack()`\n -: ----------- > 9: 1bd2f194c6f repack-midx: factor out `repack_prepare_midx_command()`\n 1: 44f522ea04d ! 10: 2a87a1e4561 repack-midx: extract `repack_fill_midx_stdin_packs()`\n @@ Commit message\n This simplifies `write_midx_included_packs()` and prepares for a\n subsequent commit where the same helper is called with `cmd->out = -1`\n to capture the MIDX's checksum from the command's standard output,\n - which is needed when writing MIDX layers with `--checksum-only`.\n + which is needed when writing MIDX layers with `--no-write-chain-file`.\n \n No functional changes are included in this patch.\n \n 2: f5642a46bbd = 11: 3d32b9c88da repack-geometry: prepare for incremental MIDX repacking\n 3: 9fdcb253a96 = 12: 1f7a5479bb8 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`\n 4: 1e1b957bf12 = 13: b155f25d53c packfile: ensure `close_pack_revindex()` frees in-memory revindex\n 5: 93e152fb6aa ! 14: ef012314930 repack: implement incremental MIDX repacking\n @@ Commit message\n Unlike the default mode which writes a single flat MIDX, the incremental\n mode constructs a compaction plan that determines which MIDX layers to\n write, compact, or copy, and then executes each step using `git\n - multi-pack-index` subcommands with the --checksum-only flag.\n + multi-pack-index` subcommands with the --no-write-chain-file flag.\n \n The repacking strategy works as follows:\n \n @@ Commit message\n \n After writing the new layer, the strategy is evaluated among the\n existing MIDX layers in order from oldest to newest. Each step that\n - writes a new MIDX layer uses \"--checksum-only\" to avoid updating the\n - multi-pack-index-chain file. After all steps are complete, the new chain\n - file is written and then atomically moved into place.\n + writes a new MIDX layer uses \"--no-write-chain-file\" to avoid updating\n + the multi-pack-index-chain file. After all steps are complete, the new\n + chain file is written and then atomically moved into place.\n \n At present, this functionality is exposed behind a new enum value,\n `REPACK_WRITE_MIDX_INCREMENTAL`, but has no external callers. A\n @@ repack-midx.c: static void repack_prepare_midx_command(struct child_process *cmd\n +\t\t\t\t\tstruct string_list *include,\n +\t\t\t\t\tstruct string_list *out)\n {\n ++\tstruct strbuf in_buf = STRBUF_INIT;\n ++\tstruct strbuf out_buf = STRBUF_INIT;\n \tstruct string_list_item *item;\n - \tFILE *in;\n +-\tFILE *in;\n \tint ret;\n \n - \tcmd->in = -1;\n -+\tif (out)\n -+\t\tcmd->out = -1;\n - \n +-\tcmd->in = -1;\n +-\n \tstrvec_push(&cmd->args, \"--stdin-packs\");\n \n -@@ repack-midx.c: static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n - \t\tfprintf(in, \"%s\\n\", item->string);\n - \tfclose(in);\n +-\tret = start_command(cmd);\n +-\tif (ret)\n +-\t\treturn ret;\n +-\n +-\tin = xfdopen(cmd->in, \"w\");\n + \tfor_each_string_list_item(item, include)\n +-\t\tfprintf(in, \"%s\\n\", item->string);\n +-\tfclose(in);\n ++\t\tstrbuf_addf(&in_buf, \"%s\\n\", item->string);\n \n -+\tif (out) {\n -+\t\tstruct strbuf buf = STRBUF_INIT;\n -+\t\tFILE *outf = xfdopen(cmd->out, \"r\");\n +-\treturn finish_command(cmd);\n ++\tret = pipe_command(cmd, in_buf.buf, in_buf.len,\n ++\t\t\t out ? &out_buf : NULL, 0, NULL, 0);\n +\n -+\t\twhile (strbuf_getline(&buf, outf) != EOF)\n -+\t\t\tstring_list_append(out, buf.buf);\n -+\t\tstrbuf_release(&buf);\n ++\tif (out)\n ++\t\tstring_list_split_f(out, out_buf.buf, \"\\n\", -1,\n ++\t\t\t\t STRING_LIST_SPLIT_NONEMPTY);\n +\n -+\t\tfclose(outf);\n -+\t}\n ++\tstrbuf_release(&in_buf);\n ++\tstrbuf_release(&out_buf);\n +\n - \treturn finish_command(cmd);\n ++\treturn ret;\n }\n \n + static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n @@ repack-midx.c: static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n @@ repack-midx.c: static int write_midx_included_packs(struct repack_write_midx_opt\n +\twhile (strbuf_getline_lf(&buf, out) != EOF) {\n +\t\tif (step->csum) {\n +\t\t\tret = error(_(\"unexpected MIDX output: '%s'\"), buf.buf);\n ++\t\t\tfclose(out);\n ++\t\t\tout = NULL;\n ++\t\t\tfinish_command(&cmd);\n +\t\t\tgoto out;\n +\t\t}\n +\t\tstep->csum = strbuf_detach(&buf, NULL);\n 6: 6119f15d3e8 ! 15: 04cfecd5136 repack: introduce `--write-midx=incremental`\n @@ t/t7705-repack-incremental-midx.sh (new)\n +\t)\n +'\n +\n -+test_expect_success 'repack -ad --write-midx=incremental is safe' '\n -+\tgit init ad-incremental-midx &&\n -+\t(\n -+\t\tcd ad-incremental-midx &&\n -+\n -+\t\tgit config maintenance.auto false &&\n -+\n -+\t\t# Build a MIDX chain with multiple layers referencing\n -+\t\t# distinct packs.\n -+\t\ttest_commit first &&\n -+\t\tgit repack -d &&\n -+\n -+\t\ttest_commit second &&\n -+\t\tgit repack -d --write-midx=incremental &&\n -+\n -+\t\tgit multi-pack-index verify &&\n -+\t\ttest_line_count = 1 $midx_chain &&\n -+\n -+\t\t# Now do a full -ad repack. The new pack contains all\n -+\t\t# objects, but any retained MIDX layers still reference\n -+\t\t# the now-deleted packs.\n -+\t\ttest_commit third &&\n -+\t\tgit repack -ad --write-midx=incremental &&\n -+\n -+\t\tgit multi-pack-index verify &&\n -+\t\tgit fsck &&\n -+\t\tgit rev-list --all --objects >/dev/null\n -+\t)\n -+'\n -+\n +test_expect_success 'repack rejects invalid midxSplitFactor' '\n +\ttest_when_finished \"rm -fr bad-split-factor\" &&\n +\tgit init bad-split-factor &&\n -: ----------- > 16: 1c05dfce579 repack: allow `--write-midx=incremental` without `--geometric`\n\nbase-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0\n-- \n2.54.0.16.g1c05dfce579\n"},{"id":"542494","messageId":"d6c27317c25895da223906e07b82615afb6f633d.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 01/16] midx-write: handle noop writes when converting incremental chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:08Z","receivedAt":"2026-04-30T00:13:10Z","isPatch":true,"body":"When updating a MIDX, we optimize out writes that will result in an\nidentical MIDX as the one we already have on disk. See b3bab9d2729\n(midx-write: extract function to test whether MIDX needs updating,\n2025-12-10) for more details on exactly which writes are optimized out.\n\nIf `midx_needs_update()` can't rule out any of the obvious cases (e.g.,\nthe checksum is invalid, we're requesting a different version, or\nperforming compaction which always requires an update), then we compare\nthe packs we're writing to the packs we already know about. If there are\nan equal number of packs being written as there are in any existing\nMIDX layer(s), then we compare the packs by their name.\n\nThis comparison fails when we have an incremental MIDX chain with\nat least two layers, since we do not recursively peel through earlier\nlayers, instead treating the `->pack_names` array of the tip MIDX layer\nas containing all `m->num_packs + m->num_packs_in_base` packs.\n\nAdjust this to instead look through the MIDX layers one by one when\ncomparing pack names. While we're at it, fix a typo above in the same\nfunction.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 18 ++++++++++--------\n t/t5334-incremental-multi-pack-index.sh | 16 ++++++++++++++++\n 2 files changed, 26 insertions(+), 8 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex a25cab75aba..9328f65a201 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1152,7 +1152,7 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \n \t/*\n \t * Ensure that we have a valid checksum before consulting the\n-\t * exisiting MIDX in order to determine if we can avoid an\n+\t * existing MIDX in order to determine if we can avoid an\n \t * update.\n \t *\n \t * This is necessary because the given MIDX is loaded directly\n@@ -1208,14 +1208,16 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \t\t\tBUG(\"same pack added twice?\");\n \t}\n \n-\tfor (uint32_t i = 0; i < ctx->nr; i++) {\n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addstr(&buf, midx->pack_names[i]);\n-\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\tfor (struct multi_pack_index *m = midx; m; m = m->base_midx) {\n+\t\tfor (uint32_t i = 0; i < m->num_packs; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, m->pack_names[i]);\n+\t\t\tstrbuf_strip_suffix(&buf, \".idx\");\n \n-\t\tif (!strset_contains(&packs, buf.buf))\n-\t\t\tgoto out;\n-\t\tstrset_remove(&packs, buf.buf);\n+\t\t\tif (!strset_contains(&packs, buf.buf))\n+\t\t\t\tgoto out;\n+\t\t\tstrset_remove(&packs, buf.buf);\n+\t\t}\n \t}\n \n \tneeded = false;\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 99c7d44d8e9..c9f5b4e87aa 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -132,4 +132,20 @@ test_expect_success 'relink existing MIDX layer' '\n \n '\n \n+test_expect_success 'non-incremental write with existing incremental chain' '\n+\tgit init non-incremental-write-with-existing &&\n+\ttest_when_finished \"rm -fr non-incremental-write-with-existing\" &&\n+\n+\t(\n+\t\tcd non-incremental-write-with-existing &&\n+\n+\t\tgit config set maintenance.auto false &&\n+\n+\t\twrite_midx_layer &&\n+\t\twrite_midx_layer &&\n+\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n test_done\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542495","messageId":"629c8d2311647fdde69cc56d946cdd957fa8c16e.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 02/16] midx: use `strset` for retained MIDX files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:11Z","receivedAt":"2026-04-30T00:13:13Z","isPatch":true,"body":"Both `clear_midx_files_ext()` and `clear_incremental_midx_files_ext()`\nbuild a list of filenames to keep while pruning stale MIDX files. Today\nthey hand-roll an array instead of using a `strset`, thus requiring us\nto pass an additional length parameter, and makes lookups linear.\n\nReplace the bare array with a `strset` which can be passed around as a\nsingle parameter. Though it improves lookup performance, the difference\nis likely immeasurable given how small the keep_hashes array typically\nis.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 57 +++++++++++++++++++++++++++------------------------------\n 1 file changed, 27 insertions(+), 30 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 81d6ab11e6e..f75e3c9fa6d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -758,8 +758,7 @@ int midx_checksum_valid(struct multi_pack_index *m)\n }\n \n struct clear_midx_data {\n-\tchar **keep;\n-\tuint32_t keep_nr;\n+\tstruct strset keep;\n \tconst char *ext;\n };\n \n@@ -767,15 +766,12 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\t\t\tconst char *file_name, void *_data)\n {\n \tstruct clear_midx_data *data = _data;\n-\tuint32_t i;\n \n \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n \t ends_with(file_name, data->ext)))\n \t\treturn;\n-\tfor (i = 0; i < data->keep_nr; i++) {\n-\t\tif (!strcmp(data->keep[i], file_name))\n-\t\t\treturn;\n-\t}\n+\tif (strset_contains(&data->keep, file_name))\n+\t\treturn;\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n@@ -783,48 +779,49 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash)\n {\n-\tstruct clear_midx_data data;\n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRSET_INIT,\n+\t\t.ext = ext,\n+\t};\n \n \tif (keep_hash) {\n-\t\tALLOC_ARRAY(data.keep, 1);\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hash, ext);\n \n-\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n-\t\tdata.keep_nr = 1;\n+\t\tstrset_add(&data.keep, buf.buf);\n+\n+\t\tstrbuf_release(&buf);\n \t}\n-\tdata.ext = ext;\n \n-\tfor_each_file_in_pack_dir(source->path,\n-\t\t\t\t clear_midx_file_ext,\n-\t\t\t\t &data);\n+\tfor_each_file_in_pack_dir(source->path, clear_midx_file_ext, &data);\n \n-\tif (keep_hash)\n-\t\tfree(data.keep[0]);\n-\tfree(data.keep);\n+\tstrset_clear(&data.keep);\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t char **keep_hashes,\n \t\t\t\t uint32_t hashes_nr)\n {\n-\tstruct clear_midx_data data;\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRSET_INIT,\n+\t\t.ext = ext,\n+\t};\n+\tstruct strbuf buf = STRBUF_INIT;\n \tuint32_t i;\n \n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\tfor (i = 0; i < hashes_nr; i++) {\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n+\t\t\t ext);\n \n-\tALLOC_ARRAY(data.keep, hashes_nr);\n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n-\t\t\t\t ext);\n-\tdata.keep_nr = hashes_nr;\n-\tdata.ext = ext;\n+\t\tstrset_add(&data.keep, buf.buf);\n+\t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n \n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tfree(data.keep[i]);\n-\tfree(data.keep);\n+\tstrbuf_release(&buf);\n+\tstrset_clear(&data.keep);\n }\n \n void clear_midx_file(struct repository *r)\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542496","messageId":"e303bf6a4ace587259f09755b257b2f5df793af9.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 03/16] midx: build `keep_hashes` array in order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:14Z","receivedAt":"2026-04-30T00:13:16Z","isPatch":true,"body":"Instead of filling the keep_hashes array using reverse indexing (e.g.,\n`keep_hashes[count - i - 1]`) while traversing linked lists forward,\ncollect linked list nodes into a temporary `layers` array and then\niterate it backwards to fill `keep_hashes` sequentially.\n\nThis makes the filling logic easier to follow, since each segment of the\narray is filled with a simple forward-marching index. Moreover, this\nchange prepares us for a subsequent commit that will switch to using a\n`strvec`.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 66 ++++++++++++++++++++++++++++------------------------\n 1 file changed, 36 insertions(+), 30 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 9328f65a201..55c778a97cb 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1731,6 +1731,9 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n+\t\tstruct multi_pack_index **layers = NULL;\n+\t\tsize_t layers_nr = 0, layers_alloc = 0;\n+\t\tsize_t j = 0;\n \n \t\tif (!chainf) {\n \t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n@@ -1751,46 +1754,49 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\tstrbuf_release(&final_midx_name);\n \n \t\tif (ctx.compact) {\n-\t\t\tstruct multi_pack_index *m;\n-\t\t\tuint32_t num_layers_before_from = 0;\n-\t\t\tuint32_t i;\n+\t\t\tstruct multi_pack_index *mp;\n \n-\t\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n-\t\t\t\tnum_layers_before_from++;\n-\n-\t\t\tm = ctx.base_midx;\n-\t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n-\t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n-\n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n-\t\t\t\tm = m->base_midx;\n+\t\t\tfor (mp = ctx.base_midx; mp; mp = mp->base_midx) {\n+\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n+\t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n+\t\t\twhile (layers_nr)\n+\t\t\t\tkeep_hashes[j++] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n \n-\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes[j++] =\n+\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t\t r->hash_algo));\n \n-\t\t\ti = 0;\n-\t\t\tfor (m = ctx.m;\n-\t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n-\t\t\t m = m->base_midx) {\n-\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n-\t\t\t\ti++;\n+\t\t\tfor (mp = ctx.m;\n+\t\t\t mp && midx_hashcmp(mp, ctx.compact_to,\n+\t\t\t\t\t\tr->hash_algo);\n+\t\t\t mp = mp->base_midx) {\n+\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n+\t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n+\t\t\twhile (layers_nr)\n+\t\t\t\tkeep_hashes[j++] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n \t\t} else {\n-\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\tfor (; m; m = m->base_midx) {\n+\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n+\t\t\t\tlayers[layers_nr++] = m;\n+\t\t\t}\n+\t\t\twhile (layers_nr)\n+\t\t\t\tkeep_hashes[j++] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\n+\t\t\tkeep_hashes[j++] =\n \t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n \t\t\t\t\t\t\t r->hash_algo));\n-\n-\t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n-\t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n-\n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n-\t\t\t\tm = m->base_midx;\n-\t\t\t}\n \t\t}\n \n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n+\t\tASSERT(j == keep_hashes_nr);\n+\n+\t\tfree(layers);\n+\n+\t\tfor (uint32_t i = 0; i < j; i++)\n \t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n \t} else {\n \t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542497","messageId":"42d76c7006096125a741faa7edb1286f5ea20612.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:17Z","receivedAt":"2026-04-30T00:13:19Z","isPatch":true,"body":"The `keep_hashes` array in `write_midx_internal()` accumulates the\nchecksums of MIDX files that should be retained when pruning stale\nentries from the MIDX chain. For similar reasons as in a previous\ncommit, rewrite this using a strvec, requiring us to pass one fewer\nparameter.\n\nUnlike the aforementioned previous commit, use a `strvec` instead of a\n`string_list`, which provides a more ergonomic interface to adjust the\nvalues at a particular index. The ordering is important here, as this\nvalue is used to determine the contents of the resulting\n`multi-pack-index-chain` file when writing with \"--incremental\".\n\nSince the previous commit already builds the array in forward order, the\nconversion is straightforward: replace indexed assignments with\n`strvec_push()`, drop the pre-counting and `CALLOC_ARRAY()`, and\nsimplify cleanup via `strvec_clear()`.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 84 ++++++++++++++++++----------------------------------\n midx.c | 20 ++++++-------\n 2 files changed, 38 insertions(+), 66 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 55c778a97cb..5d9409a9741 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -29,8 +29,7 @@ extern void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t const char *keep_hash);\n extern void clear_incremental_midx_files_ext(struct odb_source *source,\n \t\t\t\t\t const char *ext,\n-\t\t\t\t\t const char **keep_hashes,\n-\t\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t\t const struct strvec *keep_hashes);\n extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t\tconst char *idx_name);\n \n@@ -1109,8 +1108,7 @@ static int link_midx_to_chain(struct multi_pack_index *m)\n }\n \n static void clear_midx_files(struct odb_source *source,\n-\t\t\t const char **hashes, uint32_t hashes_nr,\n-\t\t\t unsigned incremental)\n+\t\t\t const struct strvec *hashes, unsigned incremental)\n {\n \t/*\n \t * if incremental:\n@@ -1124,13 +1122,15 @@ static void clear_midx_files(struct odb_source *source,\n \t */\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX };\n-\tuint32_t i, j;\n+\tuint32_t i;\n \n \tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tclear_incremental_midx_files_ext(source, exts[i],\n-\t\t\t\t\t\t hashes, hashes_nr);\n-\t\tfor (j = 0; j < hashes_nr; j++)\n-\t\t\tclear_midx_files_ext(source, exts[i], hashes[j]);\n+\t\tclear_incremental_midx_files_ext(source, exts[i], hashes);\n+\t\tif (hashes) {\n+\t\t\tfor (size_t j = 0; j < hashes->nr; j++)\n+\t\t\t\tclear_midx_files_ext(source, exts[i],\n+\t\t\t\t\t\t hashes->v[j]);\n+\t\t}\n \t}\n \n \tif (incremental)\n@@ -1267,8 +1267,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = -1;\n-\tconst char **keep_hashes = NULL;\n-\tsize_t keep_hashes_nr = 0;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tstruct chunkfile *cf;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n@@ -1708,32 +1707,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n \t\tdie(_(\"too many multi-pack-indexes\"));\n \n-\tif (ctx.compact) {\n-\t\tstruct multi_pack_index *m;\n-\n-\t\t/*\n-\t\t * Keep all MIDX layers excluding those in the range [from, to].\n-\t\t */\n-\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n-\t\t\tkeep_hashes_nr++;\n-\t\tfor (m = ctx.m;\n-\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n-\t\t m = m->base_midx)\n-\t\t\tkeep_hashes_nr++;\n-\n-\t\tkeep_hashes_nr++; /* include the compacted layer */\n-\t} else {\n-\t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n-\t}\n-\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n-\n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\tstruct multi_pack_index **layers = NULL;\n \t\tsize_t layers_nr = 0, layers_alloc = 0;\n-\t\tsize_t j = 0;\n \n \t\tif (!chainf) {\n \t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n@@ -1761,12 +1740,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n \t\t\twhile (layers_nr)\n-\t\t\t\tkeep_hashes[j++] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\t\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \n-\t\t\tkeep_hashes[j++] =\n-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t r->hash_algo));\n \n \t\t\tfor (mp = ctx.m;\n \t\t\t mp && midx_hashcmp(mp, ctx.compact_to,\n@@ -1776,31 +1755,29 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n \t\t\twhile (layers_nr)\n-\t\t\t\tkeep_hashes[j++] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\t\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \t\t} else {\n \t\t\tfor (; m; m = m->base_midx) {\n \t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n \t\t\t\tlayers[layers_nr++] = m;\n \t\t\t}\n \t\t\twhile (layers_nr)\n-\t\t\t\tkeep_hashes[j++] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\t\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \n-\t\t\tkeep_hashes[j++] =\n-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t r->hash_algo));\n \t\t}\n \n-\t\tASSERT(j == keep_hashes_nr);\n-\n \t\tfree(layers);\n \n-\t\tfor (uint32_t i = 0; i < j; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n+\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n+\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n \t} else {\n-\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n-\t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n+\t\tstrvec_push(&keep_hashes,\n+\t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n \t}\n \n \tif (ctx.m || ctx.base_midx)\n@@ -1809,8 +1786,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, keep_hashes, keep_hashes_nr,\n-\t\t\t ctx.incremental);\n+\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n \tresult = 0;\n \n cleanup:\n@@ -1826,11 +1802,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n-\tif (keep_hashes) {\n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n-\t\t\tfree((char *)keep_hashes[i]);\n-\t\tfree(keep_hashes);\n-\t}\n+\tstrvec_clear(&keep_hashes);\n \tstrbuf_release(&midx_name);\n \tclose_midx(midx_to_free);\n \ndiff --git a/midx.c b/midx.c\nindex f75e3c9fa6d..bcb8c999015 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -12,6 +12,7 @@\n #include \"chunk-format.h\"\n #include \"pack-bitmap.h\"\n #include \"pack-revindex.h\"\n+#include \"strvec.h\"\n \n #define MIDX_PACK_ERROR ((void *)(intptr_t)-1)\n \n@@ -19,8 +20,7 @@ int midx_checksum_valid(struct multi_pack_index *m);\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash);\n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t const struct strvec *keep_hashes);\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n@@ -799,22 +799,22 @@ void clear_midx_files_ext(struct odb_source *source, const char *ext,\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr)\n+\t\t\t\t const struct strvec *keep_hashes)\n {\n \tstruct clear_midx_data data = {\n \t\t.keep = STRSET_INIT,\n \t\t.ext = ext,\n \t};\n \tstruct strbuf buf = STRBUF_INIT;\n-\tuint32_t i;\n \n-\tfor (i = 0; i < hashes_nr; i++) {\n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n-\t\t\t ext);\n+\tif (keep_hashes) {\n+\t\tfor (size_t i = 0; i < keep_hashes->nr; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\",\n+\t\t\t\t keep_hashes->v[i], ext);\n \n-\t\tstrset_add(&data.keep, buf.buf);\n+\t\t\tstrset_add(&data.keep, buf.buf);\n+\t\t}\n \t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542498","messageId":"2c80aa34fac76a7755d3cdcd0d9bdc659b5d3795.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 05/16] midx: introduce `--no-write-chain-file` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:19Z","receivedAt":"2026-04-30T00:13:22Z","isPatch":true,"body":"When writing an incremental MIDX layer, the MIDX machinery writes the\nnew layer into the multi-pack-index.d directory and then updates the\nmulti-pack-index-chain file to include the freshly written layer.\n\nFuture callers however may not wish to immediately update the MIDX chain\nitself, preferring instead to write out new layer(s) themselves before\natomically updating the chain. Concretely, the new incremental\nMIDX-based repacking strategy will want to do exactly this (that is,\nassemble the new MIDX chain itself before writing a new chain file and\natomically linking it into place).\n\nIntroduce a `--no-write-chain-file` flag that:\n\n * writes the new MIDX layer into the multi-pack-index.d directory\n\n * prints its checksum\n\n * does not update the multi-pack-index-chain file.\n\nThe MIDX chain file (and thus, the lock protecting it) remain untouched,\nallowing callers to assemble the chain themselves. This flag requires\n`--incremental`, since the notion of a separate layer only makes sense\nfor incremental MIDXs.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 17 ++++++++--\n builtin/multi-pack-index.c | 28 +++++++++++++++--\n midx-write.c | 42 ++++++++++++++++---------\n midx.h | 1 +\n t/t5334-incremental-multi-pack-index.sh | 17 ++++++++++\n t/t5335-compact-multi-pack-index.sh | 36 +++++++++++++++++++++\n 6 files changed, 123 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 3a5aa227784..c26196815e2 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -11,9 +11,9 @@ SYNOPSIS\n [verse]\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n-\t\t [--refs-snapshot=]\n+\t\t [--refs-snapshot=] [--[no-]write-chain-file]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] \n+\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -83,6 +83,13 @@ marker).\n \t\tand packs not present in an existing MIDX layer.\n \t\tMigrates non-incremental MIDXs to incremental ones when\n \t\tnecessary.\n+\n+\t--[no-]write-chain-file::\n+\t\tWhen used with `--incremental`, write a new MIDX layer\n+\t\tbut do not update the multi-pack-index-chain file.\n+\t\tThe checksum of the new layer is printed to standard\n+\t\toutput, allowing the caller to assemble and write the\n+\t\tchain itself. Requires `--incremental`.\n --\n \n compact::\n@@ -97,6 +104,12 @@ compact::\n \n \t--[no-]bitmap::\n \t\tControl whether or not a multi-pack bitmap is written.\n+\n+\t--[no-]write-chain-file::\n+\t\tWhen used with `--incremental`, write a new compacted\n+\t\tMIDX layer but do not update the multi-pack-index-chain\n+\t\tfile. The checksum of the new layer is printed to\n+\t\tstandard output. Requires `--incremental`.\n --\n +\n Note that the compact command requires writing a version-2 midx that\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 0f72d96c02d..f861b4b8394 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -16,11 +16,11 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=]\")\n+\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] \")\n+\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -153,6 +153,9 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n+\t\t\tN_(\"write the multi-pack-index chain file\"),\n+\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n@@ -178,6 +181,15 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \tif (argc)\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--no-write-chain-file\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -221,6 +233,9 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n+\t\t\tN_(\"write the multi-pack-index chain file\"),\n+\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_END(),\n \t};\n \n@@ -239,6 +254,15 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \tif (argc != 2)\n \t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--no-write-chain-file\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(the_repository);\n \n \tFREE_AND_NULL(options);\ndiff --git a/midx-write.c b/midx-write.c\nindex 5d9409a9741..38c898e5ff5 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1257,7 +1257,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t start_pack;\n \tstruct hashfile *f = NULL;\n-\tstruct lock_file lk;\n+\tstruct lock_file lk = LOCK_INIT;\n \tstruct tempfile *incr;\n \tstruct write_midx_context ctx = {\n \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n@@ -1601,11 +1601,14 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \n \tif (ctx.incremental) {\n-\t\tstruct strbuf lock_name = STRBUF_INIT;\n+\t\tif (!(opts->flags & MIDX_WRITE_NO_CHAIN)) {\n+\t\t\tstruct strbuf lock_name = STRBUF_INIT;\n \n-\t\tget_midx_chain_filename(opts->source, &lock_name);\n-\t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n-\t\tstrbuf_release(&lock_name);\n+\t\t\tget_midx_chain_filename(opts->source, &lock_name);\n+\t\t\thold_lock_file_for_update(&lk, lock_name.buf,\n+\t\t\t\t\t\t LOCK_DIE_ON_ERROR);\n+\t\t\tstrbuf_release(&lock_name);\n+\t\t}\n \n \t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n \t\tif (!incr) {\n@@ -1707,16 +1710,23 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n \t\tdie(_(\"too many multi-pack-indexes\"));\n \n+\tif (!is_lock_file_locked(&lk))\n+\t\tprintf(\"%s\\n\", hash_to_hex_algop(midx_hash, r->hash_algo));\n+\telse if (opts->flags & MIDX_WRITE_NO_CHAIN)\n+\t\tBUG(\"lockfile held with MIDX_WRITE_NO_CHAIN set?\");\n+\n \tif (ctx.incremental) {\n-\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\tstruct multi_pack_index **layers = NULL;\n \t\tsize_t layers_nr = 0, layers_alloc = 0;\n \n-\t\tif (!chainf) {\n-\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n-\t\t\tgoto cleanup;\n+\t\tif (is_lock_file_locked(&lk)){\n+\t\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\t\tif (!chainf) {\n+\t\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t}\n \n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n@@ -1773,8 +1783,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t\tfree(layers);\n \n-\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n+\t\tif (is_lock_file_locked(&lk))\n+\t\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n+\t\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\",\n+\t\t\t\t\tkeep_hashes.v[i]);\n \t} else {\n \t\tstrvec_push(&keep_hashes,\n \t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n@@ -1783,10 +1795,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.m || ctx.base_midx)\n \t\todb_close(ctx.repo->objects);\n \n-\tif (commit_lock_file(&lk) < 0)\n-\t\tdie_errno(_(\"could not write multi-pack-index\"));\n+\tif (is_lock_file_locked(&lk)) {\n+\t\tif (commit_lock_file(&lk) < 0)\n+\t\t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t}\n \tresult = 0;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex 08f3728e520..5b193882dcf 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -83,6 +83,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n #define MIDX_WRITE_COMPACT (1 << 6)\n+#define MIDX_WRITE_NO_CHAIN (1 << 7)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex c9f5b4e87aa..66d6894761b 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -96,6 +96,23 @@ test_expect_success 'show object from second pack' '\n \tgit cat-file -p 2.2\n '\n \n+test_expect_success 'write MIDX layer with --no-write-chain-file' '\n+\ttest_commit no-write-chain-file &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--no-write-chain-file)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file' '\n+\ttest_must_fail git multi-pack-index write --bitmap --no-write-chain-file 2>err &&\n+\ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 40f3844282f..1a65d48b62b 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -290,4 +290,40 @@ test_expect_success 'MIDX compaction with bitmaps (non-trivial)' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --no-write-chain-file' '\n+\tgit init midx-compact-with--no-write-chain-file &&\n+\t(\n+\t\tcd midx-compact-with--no-write-chain-file &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\tlayer=\"$(git multi-pack-index compact --incremental \\\n+\t\t\t--no-write-chain-file \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n+\n+\t\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\n+\t\t# After writing the new layer, insert it into the chain\n+\t\t# manually. This is done in order to make $layer visible\n+\t\t# to the read-midx test helper below, and matches what\n+\t\t# the MIDX command would do without --no-write-chain-file.\n+\t\t{\n+\t\t\tnth_line 1 \"$midx_chain.bak\" &&\n+\t\t\techo $layer &&\n+\t\t\tnth_line 4 \"$midx_chain.bak\"\n+\t\t} >$midx_chain &&\n+\n+\t\ttest-tool read-midx $objdir $layer >midx.data &&\n+\t\tgrep \"^pack-B-.*\\.idx\" midx.data &&\n+\t\tgrep \"^pack-C-.*\\.idx\" midx.data\n+\n+\t)\n+'\n+\n test_done\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542499","messageId":"2a05f4b86f30ccd1f54e65c5da2d9234d405cbec.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:23Z","receivedAt":"2026-04-30T00:13:25Z","isPatch":true,"body":"Both `compact` and `write --incremental` fix the base of the resulting\nMIDX layer: `compact` always places the compacted result on top of\n\"from's\" immediate parent in the chain, and `write --incremental` always\nappends a new layer to the existing tip. In both cases the base is not\nconfigurable.\n\nFuture callers need additional flexibility. For instance, the incremental\nMIDX-based repacking code may wish to write a layer based on some\nintermediate ancestor rather than the current tip, or produce a root\nlayer when replacing the bottommost entries in the chain.\n\nIntroduce a new `--base` option for both subcommands to specify the\nchecksum of the MIDX layer to use as the base. The given checksum must\nrefer to a valid layer in the MIDX chain that is an ancestor of the\ntopmost layer being written or compacted.\n\nThe special value \"none\" is accepted to produce a root layer with no\nparent. This will be needed when the incremental repacking machinery\ndetermines that the bottommost layers of the chain should be replaced.\n\nIf no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\nimmediate parent in the chain, and `write` appends to the existing tip.\n\nFor the `write` subcommand, `--base` requires `--no-write-chain-file`. A plain\n`write --incremental` appends a new layer to the live chain tip with no\nmechanism to atomically replace it; overriding the base would produce a\nlayer that does not extend the tip, breaking chain invariants. With\n`--no-write-chain-file` the chain is left unmodified and the caller is\nresponsible for assembling a valid chain.\n\nFor `compact`, no such restriction applies. The compaction operation\natomically replaces the compacted range in the chain file, so writing\nthe result on top of any valid ancestor preserves chain invariants.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 17 +++++-\n builtin/multi-pack-index.c | 24 ++++++--\n midx-write.c | 34 ++++++++++-\n midx.h | 5 +-\n t/t5334-incremental-multi-pack-index.sh | 30 ++++++++++\n t/t5335-compact-multi-pack-index.sh | 77 +++++++++++++++++++++++++\n 6 files changed, 178 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex c26196815e2..c6d23aeeb9a 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -12,8 +12,10 @@ SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=] [--[no-]write-chain-file]\n+\t\t\t [--base=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n+\t\t [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\n+\t\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -90,6 +92,13 @@ marker).\n \t\tThe checksum of the new layer is printed to standard\n \t\toutput, allowing the caller to assemble and write the\n \t\tchain itself. Requires `--incremental`.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base when writing a new incremental layer.\n+\t\tThe special value `none` indicates that the new layer\n+\t\tshould have no base (i.e., it becomes a root layer).\n+\t\tRequires `--no-write-chain-file`.\n --\n \n compact::\n@@ -110,6 +119,12 @@ compact::\n \t\tMIDX layer but do not update the multi-pack-index-chain\n \t\tfile. The checksum of the new layer is printed to\n \t\tstandard output. Requires `--incremental`.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base for the compacted result, instead of using\n+\t\tthe immediate parent of ``. The special value\n+\t\t`none` indicates that the result should have no base.\n --\n +\n Note that the compact command requires writing a version-2 midx that\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex f861b4b8394..00ffb36394d 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -16,11 +16,13 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n+\t \" [--refs-snapshot=] [--[no-]write-chain-file]\\n\" \\\n+\t \" [--base=]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n+\t \" [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\\n\" \\\n+\t \" \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -63,6 +65,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tchar *object_dir;\n \tconst char *preferred_pack;\n+\tconst char *incremental_base;\n \tchar *refs_snapshot;\n \tunsigned long batch_size;\n \tunsigned flags;\n@@ -151,6 +154,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n@@ -190,6 +195,13 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t\t options);\n \t}\n \n+\tif (opts.incremental_base &&\n+\t !(opts.flags & MIDX_WRITE_NO_CHAIN)) {\n+\t\terror(_(\"cannot use --base without --no-write-chain-file\"));\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -201,7 +213,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \n \t\tret = write_midx_file_only(source, &packs,\n \t\t\t\t\t opts.preferred_pack,\n-\t\t\t\t\t opts.refs_snapshot, opts.flags);\n+\t\t\t\t\t opts.refs_snapshot,\n+\t\t\t\t\t opts.incremental_base, opts.flags);\n \n \t\tstring_list_clear(&packs, 0);\n \t\tfree(opts.refs_snapshot);\n@@ -229,6 +242,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n@@ -290,7 +305,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n \t}\n \n-\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n+\tret = write_midx_file_compact(source, from_midx, to_midx,\n+\t\t\t\t opts.incremental_base, opts.flags);\n \n \treturn ret;\n }\ndiff --git a/midx-write.c b/midx-write.c\nindex 38c898e5ff5..561e9eedc0e 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1247,6 +1247,7 @@ struct write_midx_opts {\n \n \tconst char *preferred_pack_name;\n \tconst char *refs_snapshot;\n+\tconst char *incremental_base;\n \tunsigned flags;\n };\n \n@@ -1330,11 +1331,32 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t/*\n \t * If compacting MIDX layer(s) in the range [from, to], then the\n-\t * compacted MIDX will share the same base MIDX as 'from'.\n+\t * compacted MIDX will share the same base MIDX as 'from',\n+\t * unless a custom --base is specified (see below).\n \t */\n \tif (ctx.compact)\n \t\tctx.base_midx = ctx.compact_from->base_midx;\n \n+\tif (opts->incremental_base) {\n+\t\tif (!strcmp(opts->incremental_base, \"none\")) {\n+\t\t\tctx.base_midx = NULL;\n+\t\t} else {\n+\t\t\twhile (ctx.base_midx) {\n+\t\t\t\tconst char *cmp = midx_get_checksum_hex(ctx.base_midx);\n+\t\t\t\tif (!strcmp(opts->incremental_base, cmp))\n+\t\t\t\t\tbreak;\n+\n+\t\t\t\tctx.base_midx = ctx.base_midx->base_midx;\n+\t\t\t}\n+\n+\t\t\tif (!ctx.base_midx) {\n+\t\t\t\terror(_(\"could not find base MIDX '%s'\"),\n+\t\t\t\t opts->incremental_base);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n@@ -1827,7 +1849,8 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n int write_midx_file(struct odb_source *source,\n \t\t const char *preferred_pack_name,\n-\t\t const char *refs_snapshot, unsigned flags)\n+\t\t const char *refs_snapshot,\n+\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n@@ -1842,13 +1865,16 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags)\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.packs_to_include = packs_to_include,\n \t\t.preferred_pack_name = preferred_pack_name,\n \t\t.refs_snapshot = refs_snapshot,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags,\n \t};\n \n@@ -1858,12 +1884,14 @@ int write_midx_file_only(struct odb_source *source,\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.compact_from = from,\n \t\t.compact_to = to,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags | MIDX_WRITE_COMPACT,\n \t};\n \ndiff --git a/midx.h b/midx.h\nindex 5b193882dcf..77dd66de02b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -132,10 +132,13 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags);\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags);\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct odb_source *source, unsigned flags);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 66d6894761b..68a103d13d2 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -113,6 +113,36 @@ test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file\n \ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n '\n \n+test_expect_success 'write MIDX layer with --base without --no-write-chain-file' '\n+\ttest_must_fail git multi-pack-index write --bitmap --incremental \\\n+\t\t--base=none 2>err &&\n+\ttest_grep \"cannot use --base without --no-write-chain-file\" err\n+'\n+\n+test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '\n+\ttest_commit base-none &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--no-write-chain-file --base=none)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write MIDX layer with --base= and --no-write-chain-file' '\n+\ttest_commit base-hash &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 1a65d48b62b..ec1dafe89fc 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -304,6 +304,7 @@ test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \n \t\tlayer=\"$(git multi-pack-index compact --incremental \\\n \t\t\t--no-write-chain-file \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n \n@@ -326,4 +327,80 @@ test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --base' '\n+\tgit init midx-compact-with--base &&\n+\t(\n+\t\tcd midx-compact-with--base &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tnth_line 1 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 1 \"$midx_chain\" >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with --base=none' '\n+\tgit init midx-compact-base-none &&\n+\t(\n+\t\tcd midx-compact-base-none &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\t# Compact the two bottommost layers (A and B) into a new\n+\t\t# root layer with no parent.\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=none \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" &&\n+\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\t# The upper layers (C and D) should be preserved\n+\t\t# unchanged.\n+\t\tnth_line 3 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 4 \"$midx_chain.bak\" >>expect &&\n+\t\tnth_line 2 \"$midx_chain\" >actual &&\n+\t\tnth_line 3 \"$midx_chain\" >>actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bogus --base checksum' '\n+\tgit init midx-compact-bogus-base &&\n+\t(\n+\t\tcd midx-compact-bogus-base &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t--base=deadbeef \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" 2>err &&\n+\t\ttest_grep \"could not find base MIDX\" err\n+\t)\n+'\n+\n test_done\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542500","messageId":"92aba3d366f78fbc0f5a0928107ed8f5791fb58a.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 07/16] repack: track the ODB source via existing_packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:26Z","receivedAt":"2026-04-30T00:13:28Z","isPatch":true,"body":"Store the ODB source in the `existing_packs` struct and use that in\nplace of the raw `repo->objects->sources` access within `cmd_repack()`.\n\nThe source used is still assigned from the first source in the list, so\nthere are no functional changes in this commit. The changes instead\nserve two purposes (one immediate, one not):\n\n - The incremental MIDX-based repacking machinery will need to know what\n source is being used to read the existing MIDX/chain (should one\n exist).\n\n - In the future, if \"git repack\" is taught how to operate on other\n object sources, this field will serve as the authoritative value for\n that source.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 5 ++---\n repack.c | 2 ++\n repack.h | 1 +\n 3 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 4c5a82c2c8d..24be147d39a 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -417,7 +417,7 @@ int cmd_repack(int argc,\n \t\t * midx_has_unknown_packs() will make the decision for\n \t\t * us.\n \t\t */\n-\t\tif (!get_multi_pack_index(repo->objects->sources))\n+\t\tif (!get_multi_pack_index(existing.source))\n \t\t\tmidx_must_contain_cruft = 1;\n \t}\n \n@@ -564,8 +564,7 @@ int cmd_repack(int argc,\n \t\tunsigned flags = 0;\n \t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))\n \t\t\tflags |= MIDX_WRITE_INCREMENTAL;\n-\t\twrite_midx_file(repo->objects->sources,\n-\t\t\t\tNULL, NULL, flags);\n+\t\twrite_midx_file(existing.source, NULL, NULL, flags);\n \t}\n \n cleanup:\ndiff --git a/repack.c b/repack.c\nindex 596841027af..2ee6b51420a 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -154,6 +154,8 @@ void existing_packs_collect(struct existing_packs *existing,\n \t\t\tstring_list_append(&existing->non_kept_packs, buf.buf);\n \t}\n \n+\texisting->source = existing->repo->objects->sources;\n+\n \tstring_list_sort(&existing->kept_packs);\n \tstring_list_sort(&existing->non_kept_packs);\n \tstring_list_sort(&existing->cruft_packs);\ndiff --git a/repack.h b/repack.h\nindex bc9f2e1a5de..c0e9f0ca647 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -56,6 +56,7 @@ struct packed_git;\n \n struct existing_packs {\n \tstruct repository *repo;\n+\tstruct odb_source *source;\n \tstruct string_list kept_packs;\n \tstruct string_list non_kept_packs;\n \tstruct string_list cruft_packs;\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542501","messageId":"d3ac65c1f11656d11ff6c17fe4e5b02a916585d5.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 08/16] midx: expose `midx_layer_contains_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:28Z","receivedAt":"2026-04-30T00:13:30Z","isPatch":true,"body":"Rename the function `midx_contains_pack_1()` to instead be called\n`midx_layer_contains_pack()` and make it accessible. Unlike\n`midx_contains_pack()` (which recurses through the entire chain), this\nfunction checks only a single MIDX layer.\n\nThis will be used by a subsequent commit to determine whether a given\npack belongs to the tip MIDX layer specifically, rather than to any\nlayer in the chain.\n\nNo functional changes are present in this commit.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 +++---\n midx.h | 2 ++\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex bcb8c999015..dc86c8e7fee 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -667,8 +667,8 @@ static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n \t\t m->pack_names[*(const size_t *)b]);\n }\n \n-static int midx_contains_pack_1(struct multi_pack_index *m,\n-\t\t\t\tconst char *idx_or_pack_name)\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -709,7 +709,7 @@ static int midx_contains_pack_1(struct multi_pack_index *m,\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n {\n \tfor (; m; m = m->base_midx)\n-\t\tif (midx_contains_pack_1(m, idx_or_pack_name))\n+\t\tif (midx_layer_contains_pack(m, idx_or_pack_name))\n \t\t\treturn 1;\n \treturn 0;\n }\ndiff --git a/midx.h b/midx.h\nindex 77dd66de02b..3ee12dd08ec 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -119,6 +119,8 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n int fill_midx_entry(struct multi_pack_index *m, const struct object_id *oid, struct pack_entry *e);\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t const char *idx_or_pack_name);\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n int prepare_multi_pack_index_one(struct odb_source *source);\n \n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542502","messageId":"1bd2f194c6f7f64f2ff1e7b55a3a69defcb6a344.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 09/16] repack-midx: factor out `repack_prepare_midx_command()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:31Z","receivedAt":"2026-04-30T00:13:33Z","isPatch":true,"body":"The `write_midx_included_packs()` function assembles and executes a\n`git multi-pack-index write` command, constructing the argument list\ninline.\n\nFuture commits will introduce additional callers that need to construct\nsimilar `git multi-pack-index` commands (for both `write` and `compact`\nsubcommands), so extract the common portions of the command setup into a\nreusable `repack_prepare_midx_command()` helper.\n\nThe extracted helper sets `git_cmd`, pushes the `multi-pack-index`\nsubcommand and verb, and handles `--progress`/`--no-progress` and\n`--bitmap` flags. The remaining arguments that are specific to the\n`write` subcommand (such as `--stdin-packs`) are left to the caller.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 30 +++++++++++++++++++-----------\n 1 file changed, 19 insertions(+), 11 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 0682b80c427..83151d4734a 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -275,6 +275,23 @@ static void remove_redundant_bitmaps(struct string_list *include,\n \tstrbuf_release(&path);\n }\n \n+static void repack_prepare_midx_command(struct child_process *cmd,\n+\t\t\t\t\tstruct repack_write_midx_opts *opts,\n+\t\t\t\t\tconst char *verb)\n+{\n+\tcmd->git_cmd = 1;\n+\n+\tstrvec_pushl(&cmd->args, \"multi-pack-index\", verb, NULL);\n+\n+\tif (opts->show_progress)\n+\t\tstrvec_push(&cmd->args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd->args, \"--no-progress\");\n+\n+\tif (opts->write_bitmaps)\n+\t\tstrvec_push(&cmd->args, \"--bitmap\");\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -289,18 +306,9 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \n \tcmd.in = -1;\n-\tcmd.git_cmd = 1;\n \n-\tstrvec_push(&cmd.args, \"multi-pack-index\");\n-\tstrvec_pushl(&cmd.args, \"write\", \"--stdin-packs\", NULL);\n-\n-\tif (opts->show_progress)\n-\t\tstrvec_push(&cmd.args, \"--progress\");\n-\telse\n-\t\tstrvec_push(&cmd.args, \"--no-progress\");\n-\n-\tif (opts->write_bitmaps)\n-\t\tstrvec_push(&cmd.args, \"--bitmap\");\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542503","messageId":"2a87a1e45619d8c0c00cd816ca6614937f6ea205.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 10/16] repack-midx: extract `repack_fill_midx_stdin_packs()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:34Z","receivedAt":"2026-04-30T00:13:36Z","isPatch":true,"body":"The function `write_midx_included_packs()` manages the lifecycle of\nwriting packs to stdin when running `git multi-pack-index write` as a\nchild process.\n\nExtract a standalone `repack_fill_midx_stdin_packs()` helper, which\nhandles `--stdin-packs` argument setup, starting the command, writing\npack names to its standard input, and finishing the command.\n\nThis simplifies `write_midx_included_packs()` and prepares for a\nsubsequent commit where the same helper is called with `cmd->out = -1`\nto capture the MIDX's checksum from the command's standard output,\nwhich is needed when writing MIDX layers with `--no-write-chain-file`.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 38 ++++++++++++++++++++++++--------------\n 1 file changed, 24 insertions(+), 14 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 83151d4734a..78f069c2151 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -292,23 +292,42 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n \t\tstrvec_push(&cmd->args, \"--bitmap\");\n }\n \n+static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n+\t\t\t\t\tstruct string_list *include)\n+{\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\n+\tcmd->in = -1;\n+\n+\tstrvec_push(&cmd->args, \"--stdin-packs\");\n+\n+\tret = start_command(cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tin = xfdopen(cmd->in, \"w\");\n+\tfor_each_string_list_item(item, include)\n+\t\tfprintf(in, \"%s\\n\", item->string);\n+\tfclose(in);\n+\n+\treturn finish_command(cmd);\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n \tstruct string_list_item *item;\n \tstruct packed_git *preferred = pack_geometry_preferred_pack(opts->geometry);\n-\tFILE *in;\n \tint ret = 0;\n \n \tmidx_included_packs(&include, opts);\n \tif (!include.nr)\n \t\tgoto done;\n \n-\tcmd.in = -1;\n-\n \trepack_prepare_midx_command(&cmd, opts, \"write\");\n-\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n@@ -350,16 +369,7 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = start_command(&cmd);\n-\tif (ret)\n-\t\tgoto done;\n-\n-\tin = xfdopen(cmd.in, \"w\");\n-\tfor_each_string_list_item(item, &include)\n-\t\tfprintf(in, \"%s\\n\", item->string);\n-\tfclose(in);\n-\n-\tret = finish_command(&cmd);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542504","messageId":"3d32b9c88daaf632845f0ceb097ce95fbe576c7d.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 11/16] repack-geometry: prepare for incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:37Z","receivedAt":"2026-04-30T00:13:39Z","isPatch":true,"body":"Teach `pack_geometry_init()` to optionally restrict the set of\nrepacking candidates to only packs in the tip MIDX layer when a\n`midx_layer_threshold` is configured. If the tip layer has fewer packs\nthan the threshold, those packs are excluded entirely; otherwise only\npacks in that layer participate in the geometric repack.\n\nAlso track whether any tip-layer packs were included in the rollup\n(`midx_tip_rewritten`), which a subsequent commit will use to decide\nhow to update the MIDX chain after repacking.\n\nSigned-off-by: Taylor Blau \n---\n repack-geometry.c | 35 +++++++++++++++++++++++++++++++++++\n repack.h | 4 ++++\n 2 files changed, 39 insertions(+)\n\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex 7cebd0cb45f..2408b8a3cc2 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -4,6 +4,7 @@\n #include \"repack.h\"\n #include \"repository.h\"\n #include \"hex.h\"\n+#include \"midx.h\"\n #include \"packfile.h\"\n \n static uint32_t pack_geometry_weight(struct packed_git *p)\n@@ -31,8 +32,28 @@ void pack_geometry_init(struct pack_geometry *geometry,\n {\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n+\tstruct multi_pack_index *m = get_multi_pack_index(existing->source);\n \n \trepo_for_each_pack(existing->repo, p) {\n+\t\tif (geometry->midx_layer_threshold_set && m &&\n+\t\t p->multi_pack_index) {\n+\t\t\t/*\n+\t\t\t * When writing MIDX layers incrementally,\n+\t\t\t * ignore packs unless they are in the most\n+\t\t\t * recent MIDX layer *and* there are at least\n+\t\t\t * 'midx_layer_threshold' packs in that layer.\n+\t\t\t *\n+\t\t\t * Otherwise 'p' is either in an older layer, or\n+\t\t\t * the youngest layer does not have enough packs\n+\t\t\t * to consider its packs as candidates for\n+\t\t\t * repacking. In either of those cases we want\n+\t\t\t * to ignore the pack.\n+\t\t\t */\n+\t\t\tif (m->num_packs < geometry->midx_layer_threshold ||\n+\t\t\t !midx_layer_contains_pack(m, pack_basename(p)))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -173,6 +194,20 @@ void pack_geometry_split(struct pack_geometry *geometry)\n \tgeometry->promisor_split = compute_pack_geometry_split(geometry->promisor_pack,\n \t\t\t\t\t\t\t geometry->promisor_pack_nr,\n \t\t\t\t\t\t\t geometry->split_factor);\n+\tfor (uint32_t i = 0; i < geometry->split; i++) {\n+\t\tstruct packed_git *p = geometry->pack[i];\n+\t\t/*\n+\t\t * During incremental MIDX/bitmap repacking, any packs\n+\t\t * included in the rollup are either (a) not MIDX'd, or\n+\t\t * (b) contained in the tip layer iff it has at least\n+\t\t * the threshold number of packs.\n+\t\t *\n+\t\t * In the latter case, we can safely conclude that the\n+\t\t * tip of the MIDX chain will be rewritten.\n+\t\t */\n+\t\tif (p->multi_pack_index)\n+\t\t\tgeometry->midx_tip_rewritten = true;\n+\t}\n }\n \n struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry)\ndiff --git a/repack.h b/repack.h\nindex c0e9f0ca647..77d24ee45fb 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -108,6 +108,10 @@ struct pack_geometry {\n \tuint32_t promisor_pack_nr, promisor_pack_alloc;\n \tuint32_t promisor_split;\n \n+\tuint32_t midx_layer_threshold;\n+\tbool midx_layer_threshold_set;\n+\tbool midx_tip_rewritten;\n+\n \tint split_factor;\n };\n \n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542505","messageId":"1f7a5479bb8a4bdb8ff00afd2cd2f05dc6a629e4.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 12/16] builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:40Z","receivedAt":"2026-04-30T00:13:42Z","isPatch":true,"body":"Change the --write-midx (-m) flag from an OPT_BOOL to an OPT_CALLBACK\nthat accepts an optional mode argument. Introduce an enum with\nREPACK_WRITE_MIDX_NONE and REPACK_WRITE_MIDX_DEFAULT to distinguish\nbetween the two states, and update all existing boolean checks\naccordingly.\n\nFor now, passing no argument (or just `-m`) selects the default mode,\npreserving existing behavior. A subsequent commit will add a new mode\nfor writing incremental MIDXs.\n\nExtract repack_write_midx() as a dispatcher that selects the\nappropriate MIDX-writing implementation based on the mode.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 50 ++++++++++++++++++++++++++++++++++++------------\n repack-midx.c | 14 +++++++++++++-\n repack.h | 8 +++++++-\n 3 files changed, 58 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 24be147d39a..5d366340c34 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -97,6 +97,24 @@ static int repack_config(const char *var, const char *value,\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n+static int option_parse_write_midx(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum repack_write_midx_mode *cfg = opt->value;\n+\n+\tif (unset) {\n+\t\t*cfg = REPACK_WRITE_MIDX_NONE;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!arg || !*arg)\n+\t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse\n+\t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_repack(int argc,\n \t const char **argv,\n \t const char *prefix,\n@@ -119,7 +137,7 @@ int cmd_repack(int argc,\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct pack_objects_args po_args = PACK_OBJECTS_ARGS_INIT;\n \tstruct pack_objects_args cruft_po_args = PACK_OBJECTS_ARGS_INIT;\n-\tint write_midx = 0;\n+\tenum repack_write_midx_mode write_midx = REPACK_WRITE_MIDX_NONE;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n \tconst char *filter_to = NULL;\n@@ -185,8 +203,14 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_INTEGER('g', \"geometric\", &geometry.split_factor,\n \t\t\t N_(\"find a geometric progression with factor \")),\n-\t\tOPT_BOOL('m', \"write-midx\", &write_midx,\n-\t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n+\t\tOPT_CALLBACK_F(0, \"write-midx\", &write_midx,\n+\t\t\t N_(\"mode\"),\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t PARSE_OPT_OPTARG, option_parse_write_midx),\n+\t\tOPT_SET_INT_F('m', NULL, &write_midx,\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t REPACK_WRITE_MIDX_DEFAULT,\n+\t\t\t PARSE_OPT_HIDDEN),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n \t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n@@ -221,14 +245,16 @@ int cmd_repack(int argc,\n \t\tpack_everything |= ALL_INTO_ONE;\n \n \tif (write_bitmaps < 0) {\n-\t\tif (!write_midx &&\n+\t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n \t}\n \tif (po_args.pack_kept_objects < 0)\n-\t\tpo_args.pack_kept_objects = write_bitmaps > 0 && !write_midx;\n+\t\tpo_args.pack_kept_objects = write_bitmaps > 0 &&\n+\t\t\twrite_midx == REPACK_WRITE_MIDX_NONE;\n \n-\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n+\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) &&\n+\t write_midx == REPACK_WRITE_MIDX_NONE)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n \tif (write_bitmaps && po_args.local &&\n@@ -244,7 +270,7 @@ int cmd_repack(int argc,\n \t\twrite_bitmaps = 0;\n \t}\n \n-\tif (write_midx && write_bitmaps) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \n \t\tstrbuf_addf(&path, \"%s/%s_XXXXXX\",\n@@ -297,7 +323,7 @@ int cmd_repack(int argc,\n \t}\n \tif (repo_has_promisor_remote(repo))\n \t\tstrvec_push(&cmd.args, \"--exclude-promisor-objects\");\n-\tif (!write_midx) {\n+\tif (write_midx == REPACK_WRITE_MIDX_NONE) {\n \t\tif (write_bitmaps > 0)\n \t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n \t\telse if (write_bitmaps < 0)\n@@ -519,7 +545,7 @@ int cmd_repack(int argc,\n \tif (delete_redundant && pack_everything & ALL_INTO_ONE)\n \t\texisting_packs_mark_for_deletion(&existing, &names);\n \n-\tif (write_midx) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE) {\n \t\tstruct repack_write_midx_opts opts = {\n \t\t\t.existing = &existing,\n \t\t\t.geometry = &geometry,\n@@ -528,11 +554,11 @@ int cmd_repack(int argc,\n \t\t\t.packdir = packdir,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n-\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft\n+\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.mode = write_midx,\n \t\t};\n \n-\t\tret = write_midx_included_packs(&opts);\n-\n+\t\tret = repack_write_midx(&opts);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \t}\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 78f069c2151..4a568a2a9b8 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -315,7 +315,7 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n-int write_midx_included_packs(struct repack_write_midx_opts *opts)\n+static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n@@ -378,3 +378,15 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \n \treturn ret;\n }\n+\n+int repack_write_midx(struct repack_write_midx_opts *opts)\n+{\n+\tswitch (opts->mode) {\n+\tcase REPACK_WRITE_MIDX_NONE:\n+\t\tBUG(\"write_midx mode is NONE?\");\n+\tcase REPACK_WRITE_MIDX_DEFAULT:\n+\t\treturn write_midx_included_packs(opts);\n+\tdefault:\n+\t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n+\t}\n+}\ndiff --git a/repack.h b/repack.h\nindex 77d24ee45fb..81907fcce7f 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -134,6 +134,11 @@ void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\n \n+enum repack_write_midx_mode {\n+\tREPACK_WRITE_MIDX_NONE,\n+\tREPACK_WRITE_MIDX_DEFAULT,\n+};\n+\n struct repack_write_midx_opts {\n \tstruct existing_packs *existing;\n \tstruct pack_geometry *geometry;\n@@ -143,10 +148,11 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tenum repack_write_midx_mode mode;\n };\n \n void midx_snapshot_refs(struct repository *repo, struct tempfile *f);\n-int write_midx_included_packs(struct repack_write_midx_opts *opts);\n+int repack_write_midx(struct repack_write_midx_opts *opts);\n \n int write_filtered_pack(const struct write_pack_opts *opts,\n \t\t\tstruct existing_packs *existing,\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542506","messageId":"b155f25d53c20b59c662c012a70bb33f8446e620.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 13/16] packfile: ensure `close_pack_revindex()` frees in-memory revindex","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:43Z","receivedAt":"2026-04-30T00:13:45Z","isPatch":true,"body":"The following commit will introduce a case where we write a MIDX bitmap\nover packs that do not themselves have on-disk *.rev files.\n\nThis case is supported within Git, and we will simply fall back to\ngenerating the revindex in memory. But we don't ever release that\nmemory, causing a leak that is exposed by a test introduced in the\nfollowing commit.\n\n(As far as I could find, we never free()'d memory allocated as a\nbyproduct of creating an in-memory revindex, likely because that code\npredates the leak-checking niceties we have in the test suite now.)\n\nRectify this by calling `FREE_AND_NULL()` on the `p->revindex` field\nwhen calling `close_pack_revindex()`.\n\nSigned-off-by: Taylor Blau \n---\n packfile.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/packfile.c b/packfile.c\nindex b012d648ada..a1e88fdb223 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -420,6 +420,8 @@ void close_pack_index(struct packed_git *p)\n \n static void close_pack_revindex(struct packed_git *p)\n {\n+\tFREE_AND_NULL(p->revindex);\n+\n \tif (!p->revindex_map)\n \t\treturn;\n \n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542507","messageId":"ef0123149301df184b5b6e858a88530615fde0c7.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 14/16] repack: implement incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:46Z","receivedAt":"2026-04-30T00:13:48Z","isPatch":true,"body":"Implement the `write_midx_incremental()` function, which builds and\nmaintains an incremental MIDX chain as part of the geometric repacking\nprocess.\n\nUnlike the default mode which writes a single flat MIDX, the incremental\nmode constructs a compaction plan that determines which MIDX layers to\nwrite, compact, or copy, and then executes each step using `git\nmulti-pack-index` subcommands with the --no-write-chain-file flag.\n\nThe repacking strategy works as follows:\n\n * Acquire the lock guarding the multi-pack-index-chain.\n\n * A new MIDX layer is always written containing the newly created\n pack(s). If the tip MIDX layer was rewritten during geometric\n repacking, any surviving packs from that layer are also included.\n\n * Starting from the new layer, adjacent MIDX layers are merged together\n as long as the accumulated object count exceeds half the object count\n of the next deeper layer (controlled by 'repack.midxSplitFactor').\n\n * Remaining layers in the chain are evaluated pairwise and either\n compacted or copied as-is, following the same merging condition.\n\n * Write the contents of the new multi-pack-index chain, atomically move\n it into place, and then release the lock.\n\n * Delete any now-unused MIDX layers.\n\nAfter writing the new layer, the strategy is evaluated among the\nexisting MIDX layers in order from oldest to newest. Each step that\nwrites a new MIDX layer uses \"--no-write-chain-file\" to avoid updating\nthe multi-pack-index-chain file. After all steps are complete, the new\nchain file is written and then atomically moved into place.\n\nAt present, this functionality is exposed behind a new enum value,\n`REPACK_WRITE_MIDX_INCREMENTAL`, but has no external callers. A\nsubsequent commit will expose this mode via `git repack\n--write-midx=incremental`.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 5 +\n repack-midx.c | 593 +++++++++++++++++++++++++++++++++++++++++++++--\n repack.h | 3 +\n 3 files changed, 588 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5d366340c34..75c57736780 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -42,6 +42,9 @@ static const char incremental_bitmap_conflict_error[] = N_(\n \"--no-write-bitmap-index or disable the pack.writeBitmaps configuration.\"\n );\n \n+#define DEFAULT_MIDX_SPLIT_FACTOR 2\n+#define DEFAULT_MIDX_NEW_LAYER_THRESHOLD 8\n+\n struct repack_config_ctx {\n \tstruct pack_objects_args *po_args;\n \tstruct pack_objects_args *cruft_po_args;\n@@ -555,6 +558,8 @@ int cmd_repack(int argc,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,\n+\t\t\t.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,\n \t\t\t.mode = write_midx,\n \t\t};\n \ndiff --git a/repack-midx.c b/repack-midx.c\nindex 4a568a2a9b8..405f4662174 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -2,12 +2,16 @@\n #include \"repack.h\"\n #include \"hash.h\"\n #include \"hex.h\"\n+#include \"lockfile.h\"\n+#include \"midx.h\"\n #include \"odb.h\"\n #include \"oidset.h\"\n #include \"pack-bitmap.h\"\n+#include \"path.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n #include \"tempfile.h\"\n+#include \"trace2.h\"\n \n struct midx_snapshot_ref_data {\n \tstruct repository *repo;\n@@ -293,26 +297,30 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n }\n \n static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n-\t\t\t\t\tstruct string_list *include)\n+\t\t\t\t\tstruct string_list *include,\n+\t\t\t\t\tstruct string_list *out)\n {\n+\tstruct strbuf in_buf = STRBUF_INIT;\n+\tstruct strbuf out_buf = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in;\n \tint ret;\n \n-\tcmd->in = -1;\n-\n \tstrvec_push(&cmd->args, \"--stdin-packs\");\n \n-\tret = start_command(cmd);\n-\tif (ret)\n-\t\treturn ret;\n-\n-\tin = xfdopen(cmd->in, \"w\");\n \tfor_each_string_list_item(item, include)\n-\t\tfprintf(in, \"%s\\n\", item->string);\n-\tfclose(in);\n+\t\tstrbuf_addf(&in_buf, \"%s\\n\", item->string);\n \n-\treturn finish_command(cmd);\n+\tret = pipe_command(cmd, in_buf.buf, in_buf.len,\n+\t\t\t out ? &out_buf : NULL, 0, NULL, 0);\n+\n+\tif (out)\n+\t\tstring_list_split_f(out, out_buf.buf, \"\\n\", -1,\n+\t\t\t\t STRING_LIST_SPLIT_NONEMPTY);\n+\n+\tstrbuf_release(&in_buf);\n+\tstrbuf_release(&out_buf);\n+\n+\treturn ret;\n }\n \n static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n@@ -369,7 +377,7 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include, NULL);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n@@ -379,6 +387,563 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \treturn ret;\n }\n \n+struct midx_compaction_step {\n+\tunion {\n+\t\tstruct multi_pack_index *copy;\n+\t\tstruct string_list write;\n+\t\tstruct {\n+\t\t\tstruct multi_pack_index *from;\n+\t\t\tstruct multi_pack_index *to;\n+\t\t} compact;\n+\t} u;\n+\n+\tuint32_t objects_nr;\n+\tchar *csum;\n+\n+\tenum {\n+\t\tMIDX_COMPACTION_STEP_UNKNOWN,\n+\t\tMIDX_COMPACTION_STEP_COPY,\n+\t\tMIDX_COMPACTION_STEP_WRITE,\n+\t\tMIDX_COMPACTION_STEP_COMPACT,\n+\t} type;\n+};\n+\n+static const char *midx_compaction_step_base(const struct midx_compaction_step *step)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot use UNKNOWN step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_get_checksum_hex(step->u.copy);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\tBUG(\"cannot use WRITE step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_get_checksum_hex(step->u.compact.to);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static int midx_compaction_step_exec_copy(struct midx_compaction_step *step)\n+{\n+\tstep->csum = xstrdup(midx_get_checksum_hex(step->u.copy));\n+\treturn 0;\n+}\n+\n+static int midx_compaction_step_exec_write(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t\t const char *base)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list hash = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\tconst char *preferred_pack = NULL;\n+\tint ret = 0;\n+\n+\tif (!step->u.write.nr) {\n+\t\tret = error(_(\"no packs to write MIDX during compaction\"));\n+\t\tgoto out;\n+\t}\n+\n+\tfor_each_string_list_item(item, &step->u.write) {\n+\t\tif (item->util)\n+\t\t\tpreferred_pack = item->string;\n+\t}\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\", NULL);\n+\tstrvec_pushf(&cmd.args, \"--base=%s\", base ? base : \"none\");\n+\n+\tif (preferred_pack) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\n+\t\tstrbuf_addstr(&buf, preferred_pack);\n+\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\t\tstrbuf_addstr(&buf, \".pack\");\n+\n+\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\", buf.buf);\n+\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\tret = repack_fill_midx_stdin_packs(&cmd, &step->u.write, &hash);\n+\tif (hash.nr != 1) {\n+\t\tret = error(_(\"expected exactly one line during MIDX write, \"\n+\t\t\t \"got: %\"PRIuMAX),\n+\t\t\t (uintmax_t)hash.nr);\n+\t\tgoto out;\n+\t}\n+\n+\tstep->csum = xstrdup(hash.items[0].string);\n+\n+out:\n+\tstring_list_clear(&hash, 0);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec_compact(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tFILE *out = NULL;\n+\tint ret;\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"compact\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\",\n+\t\t midx_get_checksum_hex(step->u.compact.from),\n+\t\t midx_get_checksum_hex(step->u.compact.to), NULL);\n+\n+\tcmd.out = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\tgoto out;\n+\n+\tout = xfdopen(cmd.out, \"r\");\n+\twhile (strbuf_getline_lf(&buf, out) != EOF) {\n+\t\tif (step->csum) {\n+\t\t\tret = error(_(\"unexpected MIDX output: '%s'\"), buf.buf);\n+\t\t\tfclose(out);\n+\t\t\tout = NULL;\n+\t\t\tfinish_command(&cmd);\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep->csum = strbuf_detach(&buf, NULL);\n+\t}\n+\n+\tret = finish_command(&cmd);\n+\n+out:\n+\tif (out)\n+\t\tfclose(out);\n+\tstrbuf_release(&buf);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec(struct midx_compaction_step *step,\n+\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t const char *base)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot execute UNKNOWN midx compaction step\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_compaction_step_exec_copy(step);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\treturn midx_compaction_step_exec_write(step, opts, base);\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_compaction_step_exec_compact(step, opts);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static void midx_compaction_step_release(struct midx_compaction_step *step)\n+{\n+\tif (step->type == MIDX_COMPACTION_STEP_WRITE)\n+\t\tstring_list_clear(&step->u.write, 0);\n+\tfree(step->csum);\n+}\n+\n+static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step step = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\tuint32_t i;\n+\tint ret = 0;\n+\n+\ttrace2_region_enter(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tfor (i = 0; m && i < m->num_packs + m->num_packs_in_base; i++) {\n+\t\tif (prepare_midx_pack(m, i)) {\n+\t\t\tret = error(_(\"could not load pack %\"PRIu32\" from MIDX\"),\n+\t\t\t\t i);\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\ttrace2_region_enter(\"repack\", \"steps:write\", opts->existing->repo);\n+\n+\t/*\n+\t * The first MIDX in the resulting chain is always going to be\n+\t * new.\n+\t *\n+\t * At a minimum, it will include all of the newly written packs.\n+\t * If there is an existing MIDX whose tip layer contains packs\n+\t * that were repacked, it will also include any of its packs\n+\t * which were *not* rolled up as part of the geometric repack\n+\t * (if any), and the previous tip will be replaced.\n+\t *\n+\t * It may grow to include the packs from zero or more MIDXs from\n+\t * the old chain, beginning either at the old tip (if the MIDX\n+\t * was *not* rewritten) or the old tip's base MIDX layer\n+\t * (otherwise).\n+\t */\n+\tstep.type = MIDX_COMPACTION_STEP_WRITE;\n+\tstring_list_init_dup(&step.u.write);\n+\n+\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"pack-%s.idx\", opts->names->items[i].string);\n+\t\tstring_list_append(&step.u.write, buf.buf);\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:fresh\",\n+\t\t\t\t step.u.write.items[step.u.write.nr - 1].string);\n+\t}\n+\tfor (i = 0; i < opts->geometry->split; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:fresh:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n+\n+\t/*\n+\t * Now handle any existing packs which were *not* rewritten.\n+\t *\n+\t * The list of packs in opts->geometry only contains MIDX'd\n+\t * packs from the newest layer when that layer has more than\n+\t * 'repack.midxNewLayerThreshold' number of packs.\n+\t *\n+\t * If the MIDX tip was rewritten (that is, one or more of those\n+\t * packs appear below the split line), then add all packs above\n+\t * the split line to the new layer, as the old one is no longer\n+\t * usable.\n+\t *\n+\t * If the MIDX tip was not rewritten (that is, all MIDX'd packs\n+\t * from the youngest layer appear below the split line, or were\n+\t * not included in the geometric repack at all because there\n+\t * were too few of them), ignore them since we'll retain the\n+\t * existing layer as-is.\n+\t */\n+\tfor (i = opts->geometry->split; i < opts->geometry->pack_nr; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tstruct string_list_item *item;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\tif (p->multi_pack_index &&\n+\t\t !opts->geometry->midx_tip_rewritten) {\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"exclude:unmodified\", buf.buf);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified\", buf.buf);\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified:midx\",\n+\t\t\t\t p->multi_pack_index ? \"true\" : \"false\");\n+\n+\t\titem = string_list_append(&step.u.write, buf.buf);\n+\t\tif (p->multi_pack_index || i == opts->geometry->pack_nr - 1)\n+\t\t\titem->util = (void *)1; /* mark as preferred */\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:unmodified:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n+\n+\t/*\n+\t * If the MIDX tip was rewritten, then we no longer consider it\n+\t * a candidate for compaction, since it will not exist in the\n+\t * MIDX chain being built.\n+\t */\n+\tif (opts->geometry->midx_tip_rewritten)\n+\t\tm = m->base_midx;\n+\n+\ttrace2_data_string(\"repack\", opts->existing->repo, \"midx:rewrote-tip\",\n+\t\t\t opts->geometry->midx_tip_rewritten ? \"true\" : \"false\");\n+\n+\ttrace2_region_enter(\"repack\", \"compact\", opts->existing->repo);\n+\n+\t/*\n+\t * Compact additional MIDX layers into this proposed one until\n+\t * the merging condition is violated.\n+\t */\n+\twhile (m) {\n+\t\tuint32_t preferred_pack_idx;\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"candidate\", midx_get_checksum_hex(m));\n+\n+\t\tif (step.objects_nr < m->num_objects / opts->midx_split_factor) {\n+\t\t\t/*\n+\t\t\t * Stop compacting MIDX layer as soon as the\n+\t\t\t * merged size is less than half the size of the\n+\t\t\t * next layer in the chain.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"compact\", \"violated\");\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"next_objects_nr\",\n+\t\t\t\t\t (uintmax_t)m->num_objects);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"split_factor\",\n+\t\t\t\t\t (uintmax_t)opts->midx_split_factor);\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (midx_preferred_pack(m, &preferred_pack_idx) < 0) {\n+\t\t\tret = error(_(\"could not find preferred pack for MIDX \"\n+\t\t\t\t \"%s\"), midx_get_checksum_hex(m));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\tstruct string_list_item *item;\n+\t\t\tuint32_t pack_int_id = i + m->num_packs_in_base;\n+\t\t\tstruct packed_git *p = nth_midxed_pack(m, pack_int_id);\n+\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"midx:pack\", buf.buf);\n+\n+\t\t\titem = string_list_append(&step.u.write, buf.buf);\n+\t\t\tif (pack_int_id == preferred_pack_idx)\n+\t\t\t\titem->util = (void *)1; /* mark as preferred */\n+\t\t}\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, m->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep.objects_nr += m->num_objects;\n+\n+\t\tm = m->base_midx;\n+\t}\n+\n+\tif (step.u.write.nr > 0) {\n+\t\t/*\n+\t\t * As long as there is at least one new pack to write\n+\t\t * (and thus the MIDX is non-empty), add it to the plan.\n+\t\t */\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\t\tsteps[steps_nr++] = step;\n+\t}\n+\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:objects_nr\", (uintmax_t)step.objects_nr);\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:packs_nr\", (uintmax_t)step.u.write.nr);\n+\n+\ttrace2_region_leave(\"repack\", \"compact\", opts->existing->repo);\n+\ttrace2_region_leave(\"repack\", \"steps:write\", opts->existing->repo);\n+\n+\ttrace2_region_enter(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n+\t/*\n+\t * Then start over, repeat, and either compact or keep as-is\n+\t * each MIDX layer until we have exhausted the chain.\n+\t *\n+\t * Finally, evaluate the remainder of the chain (if any) and\n+\t * either compact a sequence of adjacent layers, or keep\n+\t * individual layers as-is according to the same merging\n+\t * condition as above.\n+\t */\n+\twhile (m) {\n+\t\tstruct multi_pack_index *next = m;\n+\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\n+\t\tmemset(&step, 0, sizeof(step));\n+\t\tstep.type = MIDX_COMPACTION_STEP_UNKNOWN;\n+\n+\t\ttrace2_region_enter(\"repack\", \"step\", opts->existing->repo);\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"from\", midx_get_checksum_hex(m));\n+\n+\t\twhile (next) {\n+\t\t\tuint32_t proposed_objects_nr;\n+\t\t\tif (unsigned_add_overflows(step.objects_nr, next->num_objects)) {\n+\t\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tproposed_objects_nr = step.objects_nr + next->num_objects;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed\",\n+\t\t\t\t\t midx_get_checksum_hex(next));\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed:objects_nr\",\n+\t\t\t\t\t (uintmax_t)next->num_objects);\n+\n+\t\t\tif (!next->base_midx) {\n+\t\t\t\t/*\n+\t\t\t\t * If we are at the end of the MIDX\n+\t\t\t\t * chain, there is nothing to compact,\n+\t\t\t\t * so mark it and stop.\n+\t\t\t\t */\n+\t\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (proposed_objects_nr < next->base_midx->num_objects / opts->midx_split_factor) {\n+\t\t\t\t/*\n+\t\t\t\t * If there is a MIDX following this\n+\t\t\t\t * one, but our accumulated size is less\n+\t\t\t\t * than half of its size, compacting\n+\t\t\t\t * them would violate the merging\n+\t\t\t\t * condition, so stop here.\n+\t\t\t\t */\n+\n+\t\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at\",\n+\t\t\t\t\t\t midx_get_checksum_hex(next->base_midx));\n+\t\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at:objects_nr\",\n+\t\t\t\t\t\t (uintmax_t)next->base_midx->num_objects);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * Otherwise, it is OK to compact the next layer\n+\t\t\t * into this one. Do so, and then continue\n+\t\t\t * through the remainder of the chain.\n+\t\t\t */\n+\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"step:objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\tnext = next->base_midx;\n+\t\t}\n+\n+\t\tif (m == next) {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COPY;\n+\t\t\tstep.u.copy = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"copy\");\n+\t\t} else {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COMPACT;\n+\t\t\tstep.u.compact.from = next;\n+\t\t\tstep.u.compact.to = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"to\", midx_get_checksum_hex(m));\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"compact\");\n+\t\t}\n+\n+\t\tm = next->base_midx;\n+\t\tsteps[steps_nr++] = step;\n+\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n+\t}\n+\n+\ttrace2_region_leave(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n+out:\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+\n+\tstrbuf_release(&buf);\n+\n+\ttrace2_region_leave(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n+\treturn ret;\n+}\n+\n+static int write_midx_incremental(struct repack_write_midx_opts *opts)\n+{\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct strbuf lock_name = STRBUF_INIT;\n+\tstruct lock_file lf;\n+\tsize_t steps_nr = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tget_midx_chain_filename(opts->existing->source, &lock_name);\n+\tif (safe_create_leading_directories(opts->existing->repo,\n+\t\t\t\t\t lock_name.buf))\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t lock_name.buf);\n+\thold_lock_file_for_update(&lf, lock_name.buf, LOCK_DIE_ON_ERROR);\n+\n+\tif (!fdopen_lock_file(&lf, \"w\")) {\n+\t\tret = error_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\tgoto done;\n+\t}\n+\n+\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < steps_nr; i++) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tchar *base = NULL;\n+\n+\t\tif (i + 1 < steps_nr)\n+\t\t\tbase = xstrdup(midx_compaction_step_base(&steps[i + 1]));\n+\n+\t\tif (midx_compaction_step_exec(step, opts, base) < 0) {\n+\t\t\tret = error(_(\"unable to execute compaction step %\"PRIuMAX),\n+\t\t\t\t (uintmax_t)i);\n+\t\t\tfree(base);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfree(base);\n+\t}\n+\n+\ti = steps_nr;\n+\twhile (i--) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tif (!step->csum)\n+\t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n+\t\t\t (uintmax_t)i);\n+\t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t}\n+\n+\tcommit_lock_file(&lf);\n+\n+done:\n+\tstrbuf_release(&lock_name);\n+\tfor (i = 0; i < steps_nr; i++)\n+\t\tmidx_compaction_step_release(&steps[i]);\n+\tfree(steps);\n+\treturn ret;\n+}\n+\n int repack_write_midx(struct repack_write_midx_opts *opts)\n {\n \tswitch (opts->mode) {\n@@ -386,6 +951,8 @@ int repack_write_midx(struct repack_write_midx_opts *opts)\n \t\tBUG(\"write_midx mode is NONE?\");\n \tcase REPACK_WRITE_MIDX_DEFAULT:\n \t\treturn write_midx_included_packs(opts);\n+\tcase REPACK_WRITE_MIDX_INCREMENTAL:\n+\t\treturn write_midx_incremental(opts);\n \tdefault:\n \t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n \t}\ndiff --git a/repack.h b/repack.h\nindex 81907fcce7f..831ccfb1c6c 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -137,6 +137,7 @@ struct tempfile;\n enum repack_write_midx_mode {\n \tREPACK_WRITE_MIDX_NONE,\n \tREPACK_WRITE_MIDX_DEFAULT,\n+\tREPACK_WRITE_MIDX_INCREMENTAL,\n };\n \n struct repack_write_midx_opts {\n@@ -148,6 +149,8 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n \tenum repack_write_midx_mode mode;\n };\n \n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542508","messageId":"04cfecd5136b2e5a3f76d538f7adbd697b6f5abf.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 15/16] repack: introduce `--write-midx=incremental`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:49Z","receivedAt":"2026-04-30T00:13:51Z","isPatch":true,"body":"Expose the incremental MIDX repacking mode (implemented in an earlier\ncommit) via a new --write-midx=incremental option for `git repack`.\n\nAdd \"incremental\" as a recognized argument to the --write-midx\nOPT_CALLBACK, mapping it to REPACK_WRITE_MIDX_INCREMENTAL. When this\nmode is active and --geometric is in use, set the midx_layer_threshold\non the pack geometry so that only packs in sufficiently large tip layers\nare considered for repacking.\n\nTwo new configuration options control the compaction behavior:\n\n - repack.midxSplitFactor (default: 2): the factor used in the\n geometric merging condition for MIDX layers.\n\n - repack.midxNewLayerThreshold (default: 8): the minimum number of\n packs in the tip MIDX layer before its packs are considered as\n candidates for geometric repacking.\n\nAdd tests exercising the new mode across a variety of scenarios\nincluding basic geometric violations, multi-round chain integrity,\nbranching and merging histories, cross-layer object uniqueness, and\nthreshold-based compaction.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/repack.adoc | 18 ++\n Documentation/git-repack.adoc | 39 ++-\n builtin/repack.c | 49 ++-\n midx.c | 31 ++\n midx.h | 3 +\n repack-geometry.c | 13 +-\n repack-midx.c | 5 +\n repack.c | 56 +++-\n repack.h | 10 +-\n t/meson.build | 1 +\n t/t7705-repack-incremental-midx.sh | 470 +++++++++++++++++++++++++++++\n 11 files changed, 671 insertions(+), 24 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex e9e78dcb198..4c22a499f62 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -46,3 +46,21 @@ repack.midxMustContainCruft::\n \t`--write-midx`. When false, cruft packs are only included in the MIDX\n \twhen necessary (e.g., because they might be required to form a\n \treachability closure with MIDX bitmaps). Defaults to true.\n+\n+repack.midxSplitFactor::\n+\tThe factor used in the geometric merging condition when\n+\tcompacting incremental MIDX layers during `git repack` when\n+\tinvoked with the `--write-midx=incremental` option.\n++\n+Adjacent layers are merged when the accumulated object count of the\n+newer layer exceeds `1/` of the object count of the next deeper\n+layer. Must be at least 2. Defaults to 2.\n+\n+repack.midxNewLayerThreshold::\n+\tThe minimum number of packs in the tip MIDX layer before those\n+\tpacks are considered as candidates for geometric repacking\n+\tduring `git repack --write-midx=incremental`.\n++\n+When the tip layer has fewer packs than this threshold, those packs are\n+excluded from the geometric repack entirely, and are thus left\n+unmodified. Must be at least 1. Defaults to 8.\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 673ce910837..27a99cc46f4 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -11,7 +11,7 @@ SYNOPSIS\n [verse]\n 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n \t[--window=] [--depth=] [--threads=] [--keep-pack=]\n-\t[--write-midx] [--name-hash-version=] [--path-walk]\n+\t[--write-midx[=]] [--name-hash-version=] [--path-walk]\n \n DESCRIPTION\n -----------\n@@ -250,9 +250,42 @@ pack as the preferred pack for object selection by the MIDX (see\n linkgit:git-multi-pack-index[1]).\n \n -m::\n---write-midx::\n+--write-midx[=]::\n \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n-\tcontaining the non-redundant packs.\n+\tcontaining the non-redundant packs. The following modes are\n+\tavailable:\n++\n+--\n+\t`default`;;\n+\t\tWrite a single MIDX covering all packs. This is the\n+\t\tdefault when `--write-midx` is given without an\n+\t\texplicit mode.\n+\n+\t`incremental`;;\n+\t\tWrite an incremental MIDX chain instead of a single\n+\t\tflat MIDX. This mode requires `--geometric`.\n++\n+The incremental mode maintains a chain of MIDX layers that is compacted\n+over time using a geometric merging strategy. Each repack creates a new\n+tip layer containing the newly written pack(s). Adjacent layers are then\n+merged whenever the newer layer's object count exceeds\n+`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n+that do not meet this condition are retained as-is.\n++\n+The result is that newer (tip) layers tend to contain many small packs\n+with relatively few objects, while older (deeper) layers contain fewer,\n+larger packs covering more objects. Because compaction is driven by the\n+tip of the chain, newer layers are also rewritten more frequently than\n+older ones, which are only touched when enough objects have accumulated\n+to justify merging into them. This keeps the total number of layers\n+logarithmic relative to the total number of objects.\n++\n+Only packs in the tip MIDX layer are considered as candidates for the\n+geometric repack; packs in deeper layers are left untouched. If the tip\n+layer contains fewer packs than `repack.midxNewLayerThreshold`, those\n+packs are excluded from the geometry entirely, and a new layer is\n+created for any new pack(s) without disturbing the existing chain.\n+--\n \n --name-hash-version=::\n \tProvide this argument to the underlying `git pack-objects` process.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 75c57736780..5ffa18e085e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -33,7 +33,7 @@ static int midx_must_contain_cruft = 1;\n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n \t \"[--window=] [--depth=] [--threads=] [--keep-pack=]\\n\"\n-\t \"[--write-midx] [--name-hash-version=] [--path-walk]\"),\n+\t \"[--write-midx[=]] [--name-hash-version=] [--path-walk]\"),\n \tNULL\n };\n \n@@ -48,6 +48,8 @@ static const char incremental_bitmap_conflict_error[] = N_(\n struct repack_config_ctx {\n \tstruct pack_objects_args *po_args;\n \tstruct pack_objects_args *cruft_po_args;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -97,6 +99,16 @@ static int repack_config(const char *var, const char *value,\n \t\tmidx_must_contain_cruft = git_config_bool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"repack.midxsplitfactor\")) {\n+\t\trepack_ctx->midx_split_factor = git_config_int(var, value,\n+\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(var, \"repack.midxnewlayerthreshold\")) {\n+\t\trepack_ctx->midx_new_layer_threshold = git_config_int(var, value,\n+\t\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -112,6 +124,8 @@ static int option_parse_write_midx(const struct option *opt, const char *arg,\n \n \tif (!arg || !*arg)\n \t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse if (!strcmp(arg, \"incremental\"))\n+\t\t*cfg = REPACK_WRITE_MIDX_INCREMENTAL;\n \telse\n \t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n \n@@ -226,6 +240,8 @@ int cmd_repack(int argc,\n \tmemset(&config_ctx, 0, sizeof(config_ctx));\n \tconfig_ctx.po_args = &po_args;\n \tconfig_ctx.cruft_po_args = &cruft_po_args;\n+\tconfig_ctx.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR;\n+\tconfig_ctx.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD;\n \n \trepo_config(repo, repack_config, &config_ctx);\n \n@@ -247,6 +263,9 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n+\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n+\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n+\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n@@ -273,6 +292,13 @@ int cmd_repack(int argc,\n \t\twrite_bitmaps = 0;\n \t}\n \n+\tif (config_ctx.midx_split_factor < 2)\n+\t\tdie(_(\"invalid value for %s: %d\"), \"--midx-split-factor\",\n+\t\t config_ctx.midx_split_factor);\n+\tif (config_ctx.midx_new_layer_threshold < 1)\n+\t\tdie(_(\"invalid value for %s: %d\"), \"--midx-new-layer-threshold\",\n+\t\t config_ctx.midx_new_layer_threshold);\n+\n \tif (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \n@@ -296,6 +322,10 @@ int cmd_repack(int argc,\n \tif (geometry.split_factor) {\n \t\tif (pack_everything)\n \t\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--geometric\", \"-A/-a\");\n+\t\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL) {\n+\t\t\tgeometry.midx_layer_threshold = config_ctx.midx_new_layer_threshold;\n+\t\t\tgeometry.midx_layer_threshold_set = true;\n+\t\t}\n \t\tpack_geometry_init(&geometry, &existing, &po_args);\n \t\tpack_geometry_split(&geometry);\n \t}\n@@ -545,8 +575,11 @@ int cmd_repack(int argc,\n \t\t\t\t packtmp);\n \t/* End of pack replacement. */\n \n-\tif (delete_redundant && pack_everything & ALL_INTO_ONE)\n+\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n+\t\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL)\n+\t\t\texisting_packs_retain_midx_packs(&existing);\n \t\texisting_packs_mark_for_deletion(&existing, &names);\n+\t}\n \n \tif (write_midx != REPACK_WRITE_MIDX_NONE) {\n \t\tstruct repack_write_midx_opts opts = {\n@@ -558,8 +591,8 @@ int cmd_repack(int argc,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n-\t\t\t.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,\n-\t\t\t.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,\n+\t\t\t.midx_split_factor = config_ctx.midx_split_factor,\n+\t\t\t.midx_new_layer_threshold = config_ctx.midx_new_layer_threshold,\n \t\t\t.mode = write_midx,\n \t\t};\n \n@@ -572,11 +605,15 @@ int cmd_repack(int argc,\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\n-\t\texisting_packs_remove_redundant(&existing, packdir);\n+\t\tbool wrote_incremental_midx = write_midx == REPACK_WRITE_MIDX_INCREMENTAL;\n+\n+\t\texisting_packs_remove_redundant(&existing, packdir,\n+\t\t\t\t\t\twrote_incremental_midx);\n \n \t\tif (geometry.split_factor)\n \t\t\tpack_geometry_remove_redundant(&geometry, &names,\n-\t\t\t\t\t\t &existing, packdir);\n+\t\t\t\t\t\t &existing, packdir,\n+\t\t\t\t\t\t wrote_incremental_midx);\n \t\tif (show_progress)\n \t\t\topts |= PRUNE_PACKED_VERBOSE;\n \t\tprune_packed_objects(opts);\ndiff --git a/midx.c b/midx.c\nindex dc86c8e7fee..cd31fa20788 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -850,6 +850,37 @@ void clear_midx_file(struct repository *r)\n \tstrbuf_release(&midx);\n }\n \n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes)\n+{\n+\tstruct strbuf chain = STRBUF_INIT;\n+\n+\tget_midx_chain_filename(r->objects->sources, &chain);\n+\n+\tif (r->objects) {\n+\t\tstruct odb_source *source = r->objects->sources;\n+\t\tfor (source = r->objects->sources; source; source = source->next) {\n+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\t\tif (files->packed->midx)\n+\t\t\t\tclose_midx(files->packed->midx);\n+\t\t\tfiles->packed->midx = NULL;\n+\t\t}\n+\t}\n+\n+\tif (!keep_hashes && remove_path(chain.buf))\n+\t\tdie(_(\"failed to clear multi-pack-index chain at %s\"),\n+\t\t chain.buf);\n+\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_BITMAP,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_REV,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_MIDX,\n+\t\t\t\t\t keep_hashes);\n+\n+\tstrbuf_release(&chain);\n+}\n+\n static int verify_midx_error;\n \n __attribute__((format (printf, 1, 2)))\ndiff --git a/midx.h b/midx.h\nindex 3ee12dd08ec..63853a03a47 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -9,6 +9,7 @@ struct repository;\n struct bitmapped_pack;\n struct git_hash_algo;\n struct odb_source;\n+struct strvec;\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION_V1 1\n@@ -143,6 +144,8 @@ int write_midx_file_compact(struct odb_source *source,\n \t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes);\n int verify_midx_file(struct odb_source *source, unsigned flags);\n int expire_midx_packs(struct odb_source *source, unsigned flags);\n int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags);\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex 2408b8a3cc2..2064683dcfe 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -249,7 +249,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t\t\t uint32_t pack_nr,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tconst struct git_hash_algo *algop = existing->repo->hash_algo;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -269,7 +270,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t (string_list_has_string(&existing->kept_packs, buf.buf)))\n \t\t\tcontinue;\n \n-\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf);\n+\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n \n \tstrbuf_release(&buf);\n@@ -278,12 +280,13 @@ static void remove_redundant_packs(struct packed_git **pack,\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs(geometry->pack, geometry->split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n \tremove_redundant_packs(geometry->promisor_pack, geometry->promisor_split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n }\n \n void pack_geometry_release(struct pack_geometry *geometry)\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 405f4662174..bb3bee03ace 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -887,6 +887,7 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \tstruct midx_compaction_step *steps = NULL;\n \tstruct strbuf lock_name = STRBUF_INIT;\n \tstruct lock_file lf;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tsize_t steps_nr = 0;\n \tsize_t i;\n \tint ret = 0;\n@@ -932,11 +933,15 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n \t\t\t (uintmax_t)i);\n \t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t\tstrvec_push(&keep_hashes, step->csum);\n \t}\n \n \tcommit_lock_file(&lf);\n \n+\tclear_incremental_midx_files(opts->existing->repo, &keep_hashes);\n+\n done:\n+\tstrvec_clear(&keep_hashes);\n \tstrbuf_release(&lock_name);\n \tfor (i = 0; i < steps_nr; i++)\n \t\tmidx_compaction_step_release(&steps[i]);\ndiff --git a/repack.c b/repack.c\nindex 2ee6b51420a..571dabb665e 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -55,14 +55,18 @@ void pack_objects_args_release(struct pack_objects_args *args)\n }\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name)\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct odb_source *source = repo->objects->sources;\n \tstruct multi_pack_index *m = get_multi_pack_index(source);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && source->local && midx_contains_pack(m, buf.buf))\n+\tif (m && source->local && midx_contains_pack(m, buf.buf)) {\n \t\tclear_midx_file(repo);\n+\t\tif (!wrote_incremental_midx)\n+\t\t\tclear_incremental_midx_files(repo, NULL);\n+\t}\n \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n \tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n@@ -250,25 +254,63 @@ void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \t\t\t\t\t &existing->cruft_packs);\n }\n \n+/*\n+ * Mark every pack that is referenced by the existing MIDX chain as\n+ * retained, so that a subsequent call to\n+ * existing_packs_mark_for_deletion() will not mark them for deletion.\n+ *\n+ * This is used when writing an incremental MIDX layer on top of an\n+ * existing chain: retained layers continue to reference the same\n+ * packs on disk, so those packs must not be unlinked even if the\n+ * freshly-written pack supersedes them.\n+ */\n+void existing_packs_retain_midx_packs(struct existing_packs *existing)\n+{\n+\tstruct string_list_item *item;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tfor_each_string_list_item(item, &existing->midx_packs) {\n+\t\tstruct string_list_item *found;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addstr(&buf, item->string);\n+\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\n+\t\tfound = string_list_lookup(&existing->non_kept_packs, buf.buf);\n+\t\tif (found)\n+\t\t\texisting_packs_mark_retained(found);\n+\n+\t\tfound = string_list_lookup(&existing->cruft_packs, buf.buf);\n+\t\tif (found)\n+\t\t\texisting_packs_mark_retained(found);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+}\n+\n static void remove_redundant_packs_1(struct repository *repo,\n \t\t\t\t struct string_list *packs,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct string_list_item *item;\n \tfor_each_string_list_item(item, packs) {\n \t\tif (!existing_pack_is_marked_for_deletion(item))\n \t\t\tcontinue;\n-\t\trepack_remove_redundant_pack(repo, packdir, item->string);\n+\t\trepack_remove_redundant_pack(repo, packdir, item->string,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n }\n \n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs_1(existing->repo, &existing->non_kept_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n \tremove_redundant_packs_1(existing->repo, &existing->cruft_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n }\n \n void existing_packs_release(struct existing_packs *existing)\ndiff --git a/repack.h b/repack.h\nindex 831ccfb1c6c..f9fbc895f02 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -34,7 +34,8 @@ void prepare_pack_objects(struct child_process *cmd,\n void pack_objects_args_release(struct pack_objects_args *args);\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name);\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx);\n \n struct write_pack_opts {\n \tstruct pack_objects_args *po_args;\n@@ -83,8 +84,10 @@ void existing_packs_retain_cruft(struct existing_packs *existing,\n \t\t\t\t struct packed_git *cruft);\n void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \t\t\t\t struct string_list *names);\n+void existing_packs_retain_midx_packs(struct existing_packs *existing);\n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void existing_packs_release(struct existing_packs *existing);\n \n struct generated_pack;\n@@ -129,7 +132,8 @@ struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry);\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\ndiff --git a/t/meson.build b/t/meson.build\nindex 7528e5cda5f..25f0d823d8e 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -951,6 +951,7 @@ integration_tests = [\n 't7702-repack-cyclic-alternate.sh',\n 't7703-repack-geometric.sh',\n 't7704-repack-cruft.sh',\n+ 't7705-repack-incremental-midx.sh',\n 't7800-difftool.sh',\n 't7810-grep.sh',\n 't7811-grep-open.sh',\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nnew file mode 100755\nindex 00000000000..9e317ff6e8f\n--- /dev/null\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -0,0 +1,470 @@\n+#!/bin/sh\n+\n+test_description='git repack --write-midx=incremental'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+midxdir=$packdir/multi-pack-index.d\n+midx_chain=$midxdir/multi-pack-index-chain\n+\n+# incrementally_repack N\n+#\n+# Make \"N\" new commits, each stored in their own pack, and then repacked\n+# with the --write-midx=incremental strategy.\n+incrementally_repack () {\n+\tfor i in $(test_seq 1 \"$1\")\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify || return 1\n+\tdone\n+}\n+\n+# Create packs with geometrically increasing sizes so that they\n+# satisfy the geometric progression and survive a --geometric=2\n+# repack without being rolled up. Creates 3 packs containing 1,\n+# 2, and 6 commits (3, 6, and 18 objects) respectively.\n+create_geometric_packs () {\n+\ttest_commit \"small\" &&\n+\tgit repack -d &&\n+\n+\ttest_commit_bulk --message=\"medium\" 2 &&\n+\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t--write-bitmap-index\n+}\n+\n+# create_layer \n+#\n+# Creates a new MIDX layer with the contents of \"test_commit_bulk $@\".\n+create_layer () {\n+\ttest_commit_bulk \"$@\" &&\n+\n+\tgit multi-pack-index write --incremental --bitmap\n+}\n+\n+# create_layers\n+#\n+# Reads lines of \" \" from stdin and creates a new MIDX\n+# layer for each line. See create_layer above for more.\n+create_layers () {\n+\twhile read msg nr\n+\tdo\n+\t\tcreate_layer --message=\"$msg\" \"$nr\" || return 1\n+\tdone\n+}\n+\n+test_expect_success '--write-midx=incremental requires --geometric' '\n+\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+\n+\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+'\n+\n+test_expect_success 'below layer threshold, tip packs excluded' '\n+\tgit init below-layer-threshold-tip-packs-excluded &&\n+\t(\n+\t\tcd below-layer-threshold-tip-packs-excluded &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 4 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create 3 packs forming a geometric progression by\n+\t\t# object count such that they are unmodified by the\n+\t\t# initial repack. The MIDX chain thusly contains a\n+\t\t# single layer with three packs.\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a new commit. Since the layer threshold is\n+\t\t# unmet, a new MIDX layer is added on top of the\n+\t\t# existing one.\n+\t\ttest_commit extra &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs repacked' '\n+\tgit init above-layer-threshold-tip-packs-repacked &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-repacked &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Same setup, but with the layer threshold set to 2.\n+\t\t# Since the tip MIDX layer meets that threshold, its\n+\t\t# packs are considered repack candidates.\n+\t\tcreate_geometric_packs &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Perturb the existing progression such that it is\n+\t\t# rolled up into a single new pack, invalidating the\n+\t\t# existing MIDX layer and replacing it with a new one.\n+\t\ttest_commit extra &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip layer preserved' '\n+\tgit init above-layer-threshold-tip-layer-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-layer-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\ttest_commit_bulk --message=\"medium\" 2 &&\n+\t\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Create objects to form a pack satisfying the geometric\n+\t\t# progression (thus preserving the tip layer), but not\n+\t\t# so large that it meets the layer merging condition.\n+\t\ttest_commit_bulk --message=\"small\" 1 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\ttest_line_count = 1 packs.new &&\n+\t\ttest_line_count = 3 packs.after &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs preserved' '\n+\tgit init above-layer-threshold-tip-packs-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Same setup as above, but this time the new objects do\n+\t\t# not satisfy the new layer merging condition, resulting\n+\t\t# in a new tip layer.\n+\t\ttest_commit_bulk --message=\"huge\" 18 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'new tip absorbs multiple layers' '\n+\tgit init new-tip-absorbs-multiple-layers &&\n+\t(\n+\t\tcd new-tip-absorbs-multiple-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a 4-layer chain where each layer is too small to\n+\t\t# absorb the one below it. The sizes must satisfy L(n) <\n+\t\t# L(n-1)/2 for each adjacent pair:\n+\t\t#\n+\t\t# L0 (oldest): 75 obj (25 commits)\n+\t\t# L1: 21 obj (7 commits, 21 < 75/2)\n+\t\t# L2: 9 obj (3 commits, 9 < 21/2)\n+\t\t# L3 (tip): 3 obj (1 commit, 3 < 9/2)\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tL0 25\n+\t\tL1 7\n+\t\tL2 3\n+\t\tL3 1\n+\t\tEOF\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Now add a new commit. The merging condition is\n+\t\t# satisfied between L3-L1, but violated at L0, which is\n+\t\t# too large relative to the accumulated size.\n+\t\t#\n+\t\t# As a result, the chain shrinks from 4 to 2 layers.\n+\t\ttest_commit new &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'compaction of older layers' '\n+\tgit init compaction-of-older-layers &&\n+\t(\n+\t\tcd compaction-of-older-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a chain with two small layers at the bottom\n+\t\t# and a larger barrier layer on top, producing a\n+\t\t# chain that violates the compaction invariant, since\n+\t\t# the two small layers would normally have been merged.\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 2\n+\t\ttwo 4\n+\t\tbarrier 54\n+\t\tEOF\n+\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running an incremental repack compacts the two\n+\t\t# small layers at the bottom of the chain as a\n+\t\t# separate step in the compaction plan.\n+\t\ttest_commit another &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'geometric rollup with surviving tip packs' '\n+\tgit init geometric-rollup-with-surviving-tip-packs &&\n+\t(\n+\t\tcd geometric-rollup-with-surviving-tip-packs &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create a pack large enough to anchor the geometric\n+\t\t# progression when small packs are added alongside it.\n+\t\tcreate_layer --message=\"big\" 5 &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a small number of objects such that the\n+\t\t# progression is unbothered. Note that the existing pack\n+\t\t# is considered a repack candidate as the new layer\n+\t\t# threshold is set to 1.\n+\t\ttest_commit small-1 &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before\n+\t)\n+'\n+\n+test_expect_success 'kept packs are excluded from repack' '\n+\tgit init kept-packs-excluded-from-repack &&\n+\t(\n+\t\tcd kept-packs-excluded-from-repack &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create two equal-sized packs, marking one as kept.\n+\t\tfor i in A B\n+\t\tdo\n+\t\t\ttest_commit \"$i\" && git repack -d || return 1\n+\t\tdone &&\n+\n+\t\tkeep=$(ls $packdir/pack-*.idx | head -n 1) &&\n+\t\ttouch \"${keep%.idx}.keep\" &&\n+\n+\t\t# The kept pack is excluded as a repacking candidate\n+\t\t# entirely, so no rollup occurs as there is only one\n+\t\t# non-kept pack. A new MIDX layer is written containing\n+\t\t# that pack.\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count = 1 actual.packs &&\n+\t\ttest_grep ! \"$keep\" actual.packs &&\n+\n+\t\tgit multi-pack-index verify &&\n+\n+\t\t# All objects (from both kept and non-kept packs)\n+\t\t# must still be accessible.\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'incremental MIDX with --max-pack-size' '\n+\tgit init incremental-midx-with--max-pack-size &&\n+\t(\n+\t\tcd incremental-midx-with--max-pack-size &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\t# Now add enough data that a small --max-pack-size will\n+\t\t# cause pack-objects to split its output. Create objects\n+\t\t# large enough to fill multiple packs.\n+\t\ttest-tool genrandom foo 1M >big1 &&\n+\t\ttest-tool genrandom bar 1M >big2 &&\n+\t\tgit add big1 big2 &&\n+\t\ttest_tick &&\n+\t\tgit commit -a -m \"big blobs\" &&\n+\t\tgit repack -d &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index --max-pack-size=1M &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count -gt 1 actual.packs &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'noop repack preserves valid MIDX chain' '\n+\tgit init noop-repack-preserves-valid-midx-chain &&\n+\t(\n+\t\tcd noop-repack-preserves-valid-midx-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running again with no new objects should not break\n+\t\t# the MIDX chain. It produces \"Nothing new to pack.\"\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_cmp $midx_chain.before $midx_chain &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'repack -ad removes stale incremental chain' '\n+\tgit init repack--ad-removes-stale-incremental-chain &&\n+\t(\n+\t\tcd repack--ad-removes-stale-incremental-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 1\n+\t\ttwo 1\n+\t\tEOF\n+\n+\t\ttest_path_is_file $midx_chain &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tgit repack -ad &&\n+\n+\t\ttest_path_is_missing $packdir/multi-pack-index &&\n+\t\ttest_dir_is_empty $midxdir\n+\t)\n+'\n+\n+test_expect_success 'repack rejects invalid midxSplitFactor' '\n+\ttest_when_finished \"rm -fr bad-split-factor\" &&\n+\tgit init bad-split-factor &&\n+\t(\n+\t\tcd bad-split-factor &&\n+\t\ttest_commit base &&\n+\n+\t\tfor v in 0 1 -1\n+\t\tdo\n+\t\t\ttest_must_fail git -c repack.midxSplitFactor=$v \\\n+\t\t\t\trepack -d --geometric=2 --write-midx=incremental 2>err &&\n+\t\t\ttest_grep \"invalid value for --midx-split-factor\" err ||\n+\t\t\treturn 1\n+\t\tdone\n+\t)\n+'\n+\n+test_expect_success 'repack rejects invalid midxNewLayerThreshold' '\n+\ttest_when_finished \"rm -fr bad-layer-threshold\" &&\n+\tgit init bad-layer-threshold &&\n+\t(\n+\t\tcd bad-layer-threshold &&\n+\t\ttest_commit base &&\n+\n+\t\tfor v in 0 -1\n+\t\tdo\n+\t\t\ttest_must_fail git -c repack.midxNewLayerThreshold=$v \\\n+\t\t\t\trepack -d --geometric=2 --write-midx=incremental 2>err &&\n+\t\t\ttest_grep \"invalid value for --midx-new-layer-threshold\" err ||\n+\t\t\treturn 1\n+\t\tdone\n+\t)\n+'\n+\n+test_done\n-- \n2.54.0.16.g1c05dfce579\n\n"},{"id":"542509","messageId":"1c05dfce579fe7067adc148e02557493924adc64.1777507303.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"[PATCH v3 16/16] repack: allow `--write-midx=incremental` without `--geometric`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-04-30T00:13:52Z","receivedAt":"2026-04-30T00:13:54Z","isPatch":true,"body":"Previously, `--write-midx=incremental` required `--geometric` and would\ndie() without it. Relax this restriction so that incremental MIDX\nrepacking can be used independently.\n\nWithout `--geometric`, the behavior is append-only: a single new MIDX\nlayer is created containing whatever packs were written by the repack\nand appended to the existing chain (or a new chain is started). Existing\nlayers are preserved as-is with no compaction or merging.\n\nImplement this via a new repack_make_midx_append_plan() that builds a\nplan consisting of a WRITE step for the freshly written packs followed\nby COPY steps for every existing MIDX layer. The existing compaction\nplan (repack_make_midx_compaction_plan) is used only when `--geometric`\nis active.\n\nUpdate the documentation to describe the behavior with and without\n`--geometric`, and replace the test that enforced the old restriction\nwith one exercising append-only incremental MIDX repacking.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-repack.adoc | 19 +++++----\n builtin/repack.c | 3 --\n repack-midx.c | 64 +++++++++++++++++++++++++++--\n t/t7705-repack-incremental-midx.sh | 65 +++++++++++++++++++++++++++---\n 4 files changed, 133 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 27a99cc46f4..72c42015e23 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -263,14 +263,19 @@ linkgit:git-multi-pack-index[1]).\n \n \t`incremental`;;\n \t\tWrite an incremental MIDX chain instead of a single\n-\t\tflat MIDX. This mode requires `--geometric`.\n+\t\tflat MIDX.\n +\n-The incremental mode maintains a chain of MIDX layers that is compacted\n-over time using a geometric merging strategy. Each repack creates a new\n-tip layer containing the newly written pack(s). Adjacent layers are then\n-merged whenever the newer layer's object count exceeds\n-`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n-that do not meet this condition are retained as-is.\n+Without `--geometric`, a new MIDX layer is appended to the existing\n+chain (or a new chain is started) containing whatever packs were written\n+by the repack. Existing layers are preserved as-is.\n++\n+When combined with `--geometric`, the incremental mode maintains a chain\n+of MIDX layers that is compacted over time using a geometric merging\n+strategy. Each repack creates a new tip layer containing the newly\n+written pack(s). Adjacent layers are then merged whenever the newer\n+layer's object count exceeds `1/repack.midxSplitFactor` of the next\n+deeper layer's count. Layers that do not meet this condition are\n+retained as-is.\n +\n The result is that newer (tip) layers tend to contain many small packs\n with relatively few objects, while older (deeper) layers contain fewer,\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5ffa18e085e..1524a9c13ad 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -263,9 +263,6 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n-\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n-\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n-\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\ndiff --git a/repack-midx.c b/repack-midx.c\nindex bb3bee03ace..799f2b03697 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -548,6 +548,60 @@ static void midx_compaction_step_release(struct midx_compaction_step *step)\n \tfree(step->csum);\n }\n \n+/*\n+ * Build an append-only MIDX plan: a single WRITE step for the freshly\n+ * written packs, plus COPY steps for every existing layer. No\n+ * compaction or merging is performed.\n+ */\n+static void repack_make_midx_append_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step *step;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tif (opts->names->nr) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tuint32_t i;\n+\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_WRITE;\n+\t\tstring_list_init_dup(&step->u.write);\n+\n+\t\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"pack-%s.idx\",\n+\t\t\t\t opts->names->items[i].string);\n+\t\t\tstring_list_append(&step->u.write, buf.buf);\n+\t\t}\n+\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\tfor (; m; m = m->base_midx) {\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_COPY;\n+\t\tstep->u.copy = m;\n+\t\tstep->objects_nr = m->num_objects;\n+\t}\n+\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+}\n+\n static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\t\t\t struct midx_compaction_step **steps_p,\n \t\t\t\t\t size_t *steps_nr_p)\n@@ -904,9 +958,13 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \t}\n \n-\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n-\t\tret = error(_(\"unable to generate compaction plan\"));\n-\t\tgoto done;\n+\tif (opts->geometry->split_factor) {\n+\t\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\t\tgoto done;\n+\t\t}\n+\t} else {\n+\t\trepack_make_midx_append_plan(opts, &steps, &steps_nr);\n \t}\n \n \tfor (i = 0; i < steps_nr; i++) {\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nindex 9e317ff6e8f..25a8c40e8ee 100755\n--- a/t/t7705-repack-incremental-midx.sh\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -63,10 +63,36 @@ create_layers () {\n \tdone\n }\n \n-test_expect_success '--write-midx=incremental requires --geometric' '\n-\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+test_expect_success '--write-midx=incremental without --geometric' '\n+\tgit init incremental-without-geometric &&\n+\t(\n+\t\tcd incremental-without-geometric &&\n \n-\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+\t\tgit config maintenance.auto false &&\n+\n+\t\ttest_commit first &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_commit second &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# A second repack appends a new layer without\n+\t\t# disturbing the existing one.\n+\t\ttest_commit third &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\t\thead -n 1 $midx_chain.before >expect &&\n+\t\thead -n 1 $midx_chain >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit fsck\n+\t)\n '\n \n test_expect_success 'below layer threshold, tip packs excluded' '\n@@ -334,8 +360,7 @@ test_expect_success 'kept packs are excluded from repack' '\n \t\t# entirely, so no rollup occurs as there is only one\n \t\t# non-kept pack. A new MIDX layer is written containing\n \t\t# that pack.\n-\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n-\t\t\t--write-bitmap-index &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental &&\n \n \t\ttest-tool read-midx $objdir >actual &&\n \t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n@@ -433,6 +458,36 @@ test_expect_success 'repack -ad removes stale incremental chain' '\n \t)\n '\n \n+test_expect_success 'repack -ad --write-midx=incremental is safe' '\n+\tgit init ad-incremental-midx &&\n+\t(\n+\t\tcd ad-incremental-midx &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\t# Build a MIDX chain with multiple layers referencing\n+\t\t# distinct packs.\n+\t\ttest_commit first &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_commit second &&\n+\t\tgit repack -d --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\n+\t\t# Now do a full -ad repack. The new pack contains all\n+\t\t# objects, but any retained MIDX layers still reference\n+\t\t# the now-deleted packs.\n+\t\ttest_commit third &&\n+\t\tgit repack -ad --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck &&\n+\t\tgit rev-list --all --objects >/dev/null\n+\t)\n+'\n+\n test_expect_success 'repack rejects invalid midxSplitFactor' '\n \ttest_when_finished \"rm -fr bad-split-factor\" &&\n \tgit init bad-split-factor &&\n-- \n2.54.0.16.g1c05dfce579\n"},{"id":"542541","messageId":"20260501064640.GB2038915@coredump.intra.peff.net","threadId":"65383","inReplyTo":"cover.1777507303.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/16] repack: incremental MIDX/bitmap-based repacking","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-05-01T06:46:40Z","receivedAt":"2026-05-01T06:46:42Z","isPatch":true,"body":"On Wed, Apr 29, 2026 at 08:13:04PM -0400, Taylor Blau wrote:\n\n> I expect that this should be the final reroll absent any late-breaking\n> feedback. The only changes since last time are the following:\n> \n> - Various stale references to \"--checksum-only\" have been eradicated\n> from commit message(s).\n> \n> - `repack_prepare_midx_command()` now uses `pipe_command()` to\n> eliminate the possibility of a deadlock.\n> \n> - `write_midx_included_packs()` now cleans up after itself properly\n> when receiving multiple lines of output.\n> \n> - The errant test in t7705 was moved to the final commit (where it\n> belongs) instead of the penultimate one.\n\nThanks, the new pipe_command() and leak-fix for patch 14 look good to\nme.\n\n> As usual, a range-diff is included below as well for convenience. Thanks\n> in advance for reviewing!\n\nSomething funny going on here:\n\n> Range-diff against v2:\n> -: ----------- > 1: d6c27317c25 midx-write: handle noop writes when converting incremental chains\n> -: ----------- > 2: 629c8d23116 midx: use `strset` for retained MIDX files\n> -: ----------- > 3: e303bf6a4ac midx: build `keep_hashes` array in order\n> -: ----------- > 4: 42d76c70060 midx: use `strvec` for `keep_hashes`\n> -: ----------- > 5: 2c80aa34fac midx: introduce `--no-write-chain-file` for incremental MIDX writes\n> 7: d9acef1334a ! 6: 2a05f4b86f3 repack: allow `--write-midx=incremental` without `--geometric`\n> @@ Metadata\n> Author: Taylor Blau \n> \n> ## Commit message ##\n> - repack: allow `--write-midx=incremental` without `--geometric`\n> + midx: support custom `--base` for incremental MIDX writes\n> [...gigantic range-diff for this commit...]\n\nI'm not sure what's happening here. The 6th patch in v2 was \"midx:\nsupport custom --base...\". The series you sent looks right, I think, but\nmaybe your range-diff endpoints were wrong or something?\n\n-Peff\n"},{"id":"543284","messageId":"agTw579yuy4iHoMq@szeder.dev","threadId":"65383","inReplyTo":"1bd2f194c6f7f64f2ff1e7b55a3a69defcb6a344.1777507303.git.me@ttaylorr.com","subject":"Re: [PATCH v3 09/16] repack-midx: factor out `repack_prepare_midx_command()`","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2026-05-13T21:45:11Z","receivedAt":"2026-05-13T21:45:22Z","isPatch":true,"body":"On Wed, Apr 29, 2026 at 08:13:31PM -0400, Taylor Blau wrote:\n> The `write_midx_included_packs()` function assembles and executes a\n> `git multi-pack-index write` command, constructing the argument list\n> inline.\n> \n> Future commits will introduce additional callers that need to construct\n> similar `git multi-pack-index` commands (for both `write` and `compact`\n> subcommands), so extract the common portions of the command setup into a\n> reusable `repack_prepare_midx_command()` helper.\n> \n> The extracted helper sets `git_cmd`, pushes the `multi-pack-index`\n> subcommand and verb,\n\nWe don't have \"verbs\" in Git, \"multi-pack-index\" is the name of the\ngit command, and \"write\"/\"compact\"/etc. are the subcommands.\n\n> and handles `--progress`/`--no-progress` and\n> `--bitmap` flags. The remaining arguments that are specific to the\n> `write` subcommand (such as `--stdin-packs`) are left to the caller.\n> \n> No functional changes are included in this patch.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> repack-midx.c | 30 +++++++++++++++++++-----------\n> 1 file changed, 19 insertions(+), 11 deletions(-)\n> \n> diff --git a/repack-midx.c b/repack-midx.c\n> index 0682b80c427..83151d4734a 100644\n> --- a/repack-midx.c\n> +++ b/repack-midx.c\n> @@ -275,6 +275,23 @@ static void remove_redundant_bitmaps(struct string_list *include,\n> \tstrbuf_release(&path);\n> }\n> \n> +static void repack_prepare_midx_command(struct child_process *cmd,\n> +\t\t\t\t\tstruct repack_write_midx_opts *opts,\n> +\t\t\t\t\tconst char *verb)\n> +{\n> +\tcmd->git_cmd = 1;\n> +\n> +\tstrvec_pushl(&cmd->args, \"multi-pack-index\", verb, NULL);\n> +\n> +\tif (opts->show_progress)\n> +\t\tstrvec_push(&cmd->args, \"--progress\");\n> +\telse\n> +\t\tstrvec_push(&cmd->args, \"--no-progress\");\n> +\n> +\tif (opts->write_bitmaps)\n> +\t\tstrvec_push(&cmd->args, \"--bitmap\");\n> +}\n> +\n> int write_midx_included_packs(struct repack_write_midx_opts *opts)\n> {\n> \tstruct child_process cmd = CHILD_PROCESS_INIT;\n> @@ -289,18 +306,9 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n> \t\tgoto done;\n> \n> \tcmd.in = -1;\n> -\tcmd.git_cmd = 1;\n> \n> -\tstrvec_push(&cmd.args, \"multi-pack-index\");\n> -\tstrvec_pushl(&cmd.args, \"write\", \"--stdin-packs\", NULL);\n> -\n> -\tif (opts->show_progress)\n> -\t\tstrvec_push(&cmd.args, \"--progress\");\n> -\telse\n> -\t\tstrvec_push(&cmd.args, \"--no-progress\");\n> -\n> -\tif (opts->write_bitmaps)\n> -\t\tstrvec_push(&cmd.args, \"--bitmap\");\n> +\trepack_prepare_midx_command(&cmd, opts, \"write\");\n> +\tstrvec_push(&cmd.args, \"--stdin-packs\");\n> \n> \tif (preferred)\n> \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n> -- \n> 2.54.0.16.g1c05dfce579\n> \n"},{"id":"543286","messageId":"20260513230825.GA1378716@coredump.intra.peff.net","threadId":"65383","inReplyTo":"04cfecd5136b2e5a3f76d538f7adbd697b6f5abf.1777507303.git.me@ttaylorr.com","subject":"Re: [PATCH v3 15/16] repack: introduce `--write-midx=incremental`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-05-13T23:08:25Z","receivedAt":"2026-05-13T23:08:27Z","isPatch":true,"body":"On Wed, Apr 29, 2026 at 08:13:49PM -0400, Taylor Blau wrote:\n\n> +void clear_incremental_midx_files(struct repository *r,\n> +\t\t\t\t const struct strvec *keep_hashes)\n> +{\n> +\tstruct strbuf chain = STRBUF_INIT;\n> +\n> +\tget_midx_chain_filename(r->objects->sources, &chain);\n> +\n> +\tif (r->objects) {\n> +\t\tstruct odb_source *source = r->objects->sources;\n> +\t\tfor (source = r->objects->sources; source; source = source->next) {\n> +\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n> +\t\t\tif (files->packed->midx)\n> +\t\t\t\tclose_midx(files->packed->midx);\n> +\t\t\tfiles->packed->midx = NULL;\n> +\t\t}\n> +\t}\n\nCan r->objects be NULL here? If so, then the get_midx_chain_filename()\ncall will segfault, since we look at r->objects->sources. If not, then\nthis conditional is tautological and can be dropped.\n\nLooking at the two callers, the one in repack_remove_redundant_pack()\nunconditionally looks at repo->objects->sources itself, so we know it is\nnot NULL there.\n\nThe other is in write_midx_incremental(), but the flow there is less\nclear.\n\nI'd guess it is probably the case that it is always non-NULL so there is\nno bug, but dropping the conditional would make the code more clear.\n\n-Peff\n"},{"id":"543654","messageId":"cover.1779206239.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1774820449.git.me@ttaylorr.com","subject":"[PATCH v4 00/16] repack: incremental MIDX/bitmap-based repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:57:36Z","receivedAt":"2026-05-19T15:57:39Z","isPatch":true,"body":"Here is another small reroll of my series to implement the last\nremaining component of the incremental MIDX/bitmap-based repacking\nstrategy [1].\n\nThe changes since v3 are fairly small:\n\n - `repack_prepare_midx_command()` now calls its third parameter\n \"subcommand\" instead of \"verb\".\n\n - `clear_incremental_midx_files()` now avoids a redundant conditional\n around `r->objects` and reuses a local `struct odb_source *` when\n closing any open incremental MIDX handles.\n\nAs usual, a range-diff is included below for convenience. Thanks in\nadvance for reviewing!\n\n[1]: https://lore.kernel.org/git/cover.1777507303.git.me@ttaylorr.com/\n\nTaylor Blau (16):\n midx-write: handle noop writes when converting incremental chains\n midx: use `strset` for retained MIDX files\n midx: build `keep_hashes` array in order\n midx: use `strvec` for `keep_hashes`\n midx: introduce `--no-write-chain-file` for incremental MIDX writes\n midx: support custom `--base` for incremental MIDX writes\n repack: track the ODB source via existing_packs\n midx: expose `midx_layer_contains_pack()`\n repack-midx: factor out `repack_prepare_midx_command()`\n repack-midx: extract `repack_fill_midx_stdin_packs()`\n repack-geometry: prepare for incremental MIDX repacking\n builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`\n packfile: ensure `close_pack_revindex()` frees in-memory revindex\n repack: implement incremental MIDX repacking\n repack: introduce `--write-midx=incremental`\n repack: allow `--write-midx=incremental` without `--geometric`\n\n Documentation/config/repack.adoc | 18 +\n Documentation/git-multi-pack-index.adoc | 32 +-\n Documentation/git-repack.adoc | 44 +-\n builtin/multi-pack-index.c | 48 +-\n builtin/repack.c | 102 +++-\n midx-write.c | 206 ++++---\n midx.c | 102 ++--\n midx.h | 11 +-\n packfile.c | 2 +\n repack-geometry.c | 48 +-\n repack-midx.c | 710 +++++++++++++++++++++++-\n repack.c | 58 +-\n repack.h | 26 +-\n t/meson.build | 1 +\n t/t5334-incremental-multi-pack-index.sh | 63 +++\n t/t5335-compact-multi-pack-index.sh | 113 ++++\n t/t7705-repack-incremental-midx.sh | 525 ++++++++++++++++++\n 17 files changed, 1909 insertions(+), 200 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\nRange-diff against v3:\n 1: d6c27317c25 = 1: ead11e610c8 midx-write: handle noop writes when converting incremental chains\n 2: 629c8d23116 = 2: ece55bf2957 midx: use `strset` for retained MIDX files\n 3: e303bf6a4ac = 3: 5609d1941e6 midx: build `keep_hashes` array in order\n 4: 42d76c70060 = 4: 13b7c808860 midx: use `strvec` for `keep_hashes`\n 5: 2c80aa34fac = 5: cac3fd54bf0 midx: introduce `--no-write-chain-file` for incremental MIDX writes\n 6: 2a05f4b86f3 = 6: 1bbb387d6b6 midx: support custom `--base` for incremental MIDX writes\n 7: 92aba3d366f = 7: 4a93adb3ad3 repack: track the ODB source via existing_packs\n 8: d3ac65c1f11 = 8: 8d1b8b1d301 midx: expose `midx_layer_contains_pack()`\n 9: 1bd2f194c6f ! 9: 42111e5f75d repack-midx: factor out `repack_prepare_midx_command()`\n @@ Commit message\n subcommands), so extract the common portions of the command setup into a\n reusable `repack_prepare_midx_command()` helper.\n \n - The extracted helper sets `git_cmd`, pushes the `multi-pack-index`\n - subcommand and verb, and handles `--progress`/`--no-progress` and\n - `--bitmap` flags. The remaining arguments that are specific to the\n - `write` subcommand (such as `--stdin-packs`) are left to the caller.\n + The extracted helper sets `git_cmd`, pushes `multi-pack-index` and a\n + subcommand, and handles `--progress`/`--no-progress` and `--bitmap`\n + flags. The remaining arguments that are specific to the `write`\n + subcommand (such as `--stdin-packs`) are left to the caller.\n \n No functional changes are included in this patch.\n \n @@ repack-midx.c: static void remove_redundant_bitmaps(struct string_list *include,\n \n +static void repack_prepare_midx_command(struct child_process *cmd,\n +\t\t\t\t\tstruct repack_write_midx_opts *opts,\n -+\t\t\t\t\tconst char *verb)\n ++\t\t\t\t\tconst char *subcommand)\n +{\n +\tcmd->git_cmd = 1;\n +\n -+\tstrvec_pushl(&cmd->args, \"multi-pack-index\", verb, NULL);\n ++\tstrvec_pushl(&cmd->args, \"multi-pack-index\", subcommand, NULL);\n +\n +\tif (opts->show_progress)\n +\t\tstrvec_push(&cmd->args, \"--progress\");\n10: 2a87a1e4561 = 10: ed76e6efd1c repack-midx: extract `repack_fill_midx_stdin_packs()`\n11: 3d32b9c88da = 11: 9665f1b3a64 repack-geometry: prepare for incremental MIDX repacking\n12: 1f7a5479bb8 = 12: e0db62b9f10 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`\n13: b155f25d53c = 13: c8c846b1ac1 packfile: ensure `close_pack_revindex()` frees in-memory revindex\n14: ef012314930 = 14: 78f1a98c1ea repack: implement incremental MIDX repacking\n15: 04cfecd5136 ! 15: 95bf8b21fff repack: introduce `--write-midx=incremental`\n @@ midx.c: void clear_midx_file(struct repository *r)\n +void clear_incremental_midx_files(struct repository *r,\n +\t\t\t\t const struct strvec *keep_hashes)\n +{\n ++\tstruct odb_source *source = r->objects->sources;\n +\tstruct strbuf chain = STRBUF_INIT;\n +\n -+\tget_midx_chain_filename(r->objects->sources, &chain);\n ++\tget_midx_chain_filename(source, &chain);\n +\n -+\tif (r->objects) {\n -+\t\tstruct odb_source *source = r->objects->sources;\n -+\t\tfor (source = r->objects->sources; source; source = source->next) {\n -+\t\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n -+\t\t\tif (files->packed->midx)\n -+\t\t\t\tclose_midx(files->packed->midx);\n -+\t\t\tfiles->packed->midx = NULL;\n -+\t\t}\n ++\tfor (; source; source = source->next) {\n ++\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n ++\t\tif (files->packed->midx)\n ++\t\t\tclose_midx(files->packed->midx);\n ++\t\tfiles->packed->midx = NULL;\n +\t}\n +\n +\tif (!keep_hashes && remove_path(chain.buf))\n16: 1c05dfce579 = 16: 8bd0ec98dc3 repack: allow `--write-midx=incremental` without `--geometric`\n\nbase-commit: 7bcaabddcf68bd0702697da5904c3b68c52f94cf\n-- \n2.54.0.175.g8bd0ec98dc3\n"},{"id":"543655","messageId":"ead11e610c8c619de08cfd2736c12daeb2f45e20.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 01/16] midx-write: handle noop writes when converting incremental chains","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:57:39Z","receivedAt":"2026-05-19T15:57:42Z","isPatch":true,"body":"When updating a MIDX, we optimize out writes that will result in an\nidentical MIDX as the one we already have on disk. See b3bab9d2729\n(midx-write: extract function to test whether MIDX needs updating,\n2025-12-10) for more details on exactly which writes are optimized out.\n\nIf `midx_needs_update()` can't rule out any of the obvious cases (e.g.,\nthe checksum is invalid, we're requesting a different version, or\nperforming compaction which always requires an update), then we compare\nthe packs we're writing to the packs we already know about. If there are\nan equal number of packs being written as there are in any existing\nMIDX layer(s), then we compare the packs by their name.\n\nThis comparison fails when we have an incremental MIDX chain with\nat least two layers, since we do not recursively peel through earlier\nlayers, instead treating the `->pack_names` array of the tip MIDX layer\nas containing all `m->num_packs + m->num_packs_in_base` packs.\n\nAdjust this to instead look through the MIDX layers one by one when\ncomparing pack names. While we're at it, fix a typo above in the same\nfunction.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 18 ++++++++++--------\n t/t5334-incremental-multi-pack-index.sh | 16 ++++++++++++++++\n 2 files changed, 26 insertions(+), 8 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex a25cab75aba..9328f65a201 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1152,7 +1152,7 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \n \t/*\n \t * Ensure that we have a valid checksum before consulting the\n-\t * exisiting MIDX in order to determine if we can avoid an\n+\t * existing MIDX in order to determine if we can avoid an\n \t * update.\n \t *\n \t * This is necessary because the given MIDX is loaded directly\n@@ -1208,14 +1208,16 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \t\t\tBUG(\"same pack added twice?\");\n \t}\n \n-\tfor (uint32_t i = 0; i < ctx->nr; i++) {\n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addstr(&buf, midx->pack_names[i]);\n-\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\tfor (struct multi_pack_index *m = midx; m; m = m->base_midx) {\n+\t\tfor (uint32_t i = 0; i < m->num_packs; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, m->pack_names[i]);\n+\t\t\tstrbuf_strip_suffix(&buf, \".idx\");\n \n-\t\tif (!strset_contains(&packs, buf.buf))\n-\t\t\tgoto out;\n-\t\tstrset_remove(&packs, buf.buf);\n+\t\t\tif (!strset_contains(&packs, buf.buf))\n+\t\t\t\tgoto out;\n+\t\t\tstrset_remove(&packs, buf.buf);\n+\t\t}\n \t}\n \n \tneeded = false;\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 99c7d44d8e9..c9f5b4e87aa 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -132,4 +132,20 @@ test_expect_success 'relink existing MIDX layer' '\n \n '\n \n+test_expect_success 'non-incremental write with existing incremental chain' '\n+\tgit init non-incremental-write-with-existing &&\n+\ttest_when_finished \"rm -fr non-incremental-write-with-existing\" &&\n+\n+\t(\n+\t\tcd non-incremental-write-with-existing &&\n+\n+\t\tgit config set maintenance.auto false &&\n+\n+\t\twrite_midx_layer &&\n+\t\twrite_midx_layer &&\n+\n+\t\tgit multi-pack-index write\n+\t)\n+'\n+\n test_done\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543656","messageId":"ece55bf29579cddd5a245ce01c142d7ad49768a3.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 02/16] midx: use `strset` for retained MIDX files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:57:42Z","receivedAt":"2026-05-19T15:57:45Z","isPatch":true,"body":"Both `clear_midx_files_ext()` and `clear_incremental_midx_files_ext()`\nbuild a list of filenames to keep while pruning stale MIDX files. Today\nthey hand-roll an array instead of using a `strset`, thus requiring us\nto pass an additional length parameter, and makes lookups linear.\n\nReplace the bare array with a `strset` which can be passed around as a\nsingle parameter. Though it improves lookup performance, the difference\nis likely immeasurable given how small the keep_hashes array typically\nis.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 57 +++++++++++++++++++++++++++------------------------------\n 1 file changed, 27 insertions(+), 30 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 81d6ab11e6e..f75e3c9fa6d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -758,8 +758,7 @@ int midx_checksum_valid(struct multi_pack_index *m)\n }\n \n struct clear_midx_data {\n-\tchar **keep;\n-\tuint32_t keep_nr;\n+\tstruct strset keep;\n \tconst char *ext;\n };\n \n@@ -767,15 +766,12 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\t\t\tconst char *file_name, void *_data)\n {\n \tstruct clear_midx_data *data = _data;\n-\tuint32_t i;\n \n \tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n \t ends_with(file_name, data->ext)))\n \t\treturn;\n-\tfor (i = 0; i < data->keep_nr; i++) {\n-\t\tif (!strcmp(data->keep[i], file_name))\n-\t\t\treturn;\n-\t}\n+\tif (strset_contains(&data->keep, file_name))\n+\t\treturn;\n \tif (unlink(full_path))\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n@@ -783,48 +779,49 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash)\n {\n-\tstruct clear_midx_data data;\n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRSET_INIT,\n+\t\t.ext = ext,\n+\t};\n \n \tif (keep_hash) {\n-\t\tALLOC_ARRAY(data.keep, 1);\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hash, ext);\n \n-\t\tdata.keep[0] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hash, ext);\n-\t\tdata.keep_nr = 1;\n+\t\tstrset_add(&data.keep, buf.buf);\n+\n+\t\tstrbuf_release(&buf);\n \t}\n-\tdata.ext = ext;\n \n-\tfor_each_file_in_pack_dir(source->path,\n-\t\t\t\t clear_midx_file_ext,\n-\t\t\t\t &data);\n+\tfor_each_file_in_pack_dir(source->path, clear_midx_file_ext, &data);\n \n-\tif (keep_hash)\n-\t\tfree(data.keep[0]);\n-\tfree(data.keep);\n+\tstrset_clear(&data.keep);\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t char **keep_hashes,\n \t\t\t\t uint32_t hashes_nr)\n {\n-\tstruct clear_midx_data data;\n+\tstruct clear_midx_data data = {\n+\t\t.keep = STRSET_INIT,\n+\t\t.ext = ext,\n+\t};\n+\tstruct strbuf buf = STRBUF_INIT;\n \tuint32_t i;\n \n-\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\tfor (i = 0; i < hashes_nr; i++) {\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n+\t\t\t ext);\n \n-\tALLOC_ARRAY(data.keep, hashes_nr);\n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tdata.keep[i] = xstrfmt(\"multi-pack-index-%s.%s\", keep_hashes[i],\n-\t\t\t\t ext);\n-\tdata.keep_nr = hashes_nr;\n-\tdata.ext = ext;\n+\t\tstrset_add(&data.keep, buf.buf);\n+\t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n \t\t\t\t clear_midx_file_ext, &data);\n \n-\tfor (i = 0; i < hashes_nr; i++)\n-\t\tfree(data.keep[i]);\n-\tfree(data.keep);\n+\tstrbuf_release(&buf);\n+\tstrset_clear(&data.keep);\n }\n \n void clear_midx_file(struct repository *r)\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543657","messageId":"5609d1941e6339b3819ea8d07e15ab9e9108f079.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 03/16] midx: build `keep_hashes` array in order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:57:45Z","receivedAt":"2026-05-19T15:57:47Z","isPatch":true,"body":"Instead of filling the keep_hashes array using reverse indexing (e.g.,\n`keep_hashes[count - i - 1]`) while traversing linked lists forward,\ncollect linked list nodes into a temporary `layers` array and then\niterate it backwards to fill `keep_hashes` sequentially.\n\nThis makes the filling logic easier to follow, since each segment of the\narray is filled with a simple forward-marching index. Moreover, this\nchange prepares us for a subsequent commit that will switch to using a\n`strvec`.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 66 ++++++++++++++++++++++++++++------------------------\n 1 file changed, 36 insertions(+), 30 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 9328f65a201..55c778a97cb 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1731,6 +1731,9 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n+\t\tstruct multi_pack_index **layers = NULL;\n+\t\tsize_t layers_nr = 0, layers_alloc = 0;\n+\t\tsize_t j = 0;\n \n \t\tif (!chainf) {\n \t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n@@ -1751,46 +1754,49 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\tstrbuf_release(&final_midx_name);\n \n \t\tif (ctx.compact) {\n-\t\t\tstruct multi_pack_index *m;\n-\t\t\tuint32_t num_layers_before_from = 0;\n-\t\t\tuint32_t i;\n+\t\t\tstruct multi_pack_index *mp;\n \n-\t\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n-\t\t\t\tnum_layers_before_from++;\n-\n-\t\t\tm = ctx.base_midx;\n-\t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n-\t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n-\n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n-\t\t\t\tm = m->base_midx;\n+\t\t\tfor (mp = ctx.base_midx; mp; mp = mp->base_midx) {\n+\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n+\t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n+\t\t\twhile (layers_nr)\n+\t\t\t\tkeep_hashes[j++] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n \n-\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes[j++] =\n+\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t\t r->hash_algo));\n \n-\t\t\ti = 0;\n-\t\t\tfor (m = ctx.m;\n-\t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n-\t\t\t m = m->base_midx) {\n-\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n-\t\t\t\ti++;\n+\t\t\tfor (mp = ctx.m;\n+\t\t\t mp && midx_hashcmp(mp, ctx.compact_to,\n+\t\t\t\t\t\tr->hash_algo);\n+\t\t\t mp = mp->base_midx) {\n+\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n+\t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n+\t\t\twhile (layers_nr)\n+\t\t\t\tkeep_hashes[j++] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n \t\t} else {\n-\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\tfor (; m; m = m->base_midx) {\n+\t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n+\t\t\t\tlayers[layers_nr++] = m;\n+\t\t\t}\n+\t\t\twhile (layers_nr)\n+\t\t\t\tkeep_hashes[j++] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\n+\t\t\tkeep_hashes[j++] =\n \t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n \t\t\t\t\t\t\t r->hash_algo));\n-\n-\t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n-\t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n-\n-\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n-\t\t\t\tm = m->base_midx;\n-\t\t\t}\n \t\t}\n \n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n+\t\tASSERT(j == keep_hashes_nr);\n+\n+\t\tfree(layers);\n+\n+\t\tfor (uint32_t i = 0; i < j; i++)\n \t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n \t} else {\n \t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543658","messageId":"13b7c808860e18d7e70aea00d58bf8faaf0d3a36.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 04/16] midx: use `strvec` for `keep_hashes`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:57:48Z","receivedAt":"2026-05-19T15:57:51Z","isPatch":true,"body":"The `keep_hashes` array in `write_midx_internal()` accumulates the\nchecksums of MIDX files that should be retained when pruning stale\nentries from the MIDX chain. For similar reasons as in a previous\ncommit, rewrite this using a strvec, requiring us to pass one fewer\nparameter.\n\nUnlike the aforementioned previous commit, use a `strvec` instead of a\n`string_list`, which provides a more ergonomic interface to adjust the\nvalues at a particular index. The ordering is important here, as this\nvalue is used to determine the contents of the resulting\n`multi-pack-index-chain` file when writing with \"--incremental\".\n\nSince the previous commit already builds the array in forward order, the\nconversion is straightforward: replace indexed assignments with\n`strvec_push()`, drop the pre-counting and `CALLOC_ARRAY()`, and\nsimplify cleanup via `strvec_clear()`.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 84 ++++++++++++++++++----------------------------------\n midx.c | 20 ++++++-------\n 2 files changed, 38 insertions(+), 66 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 55c778a97cb..5d9409a9741 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -29,8 +29,7 @@ extern void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t\t const char *keep_hash);\n extern void clear_incremental_midx_files_ext(struct odb_source *source,\n \t\t\t\t\t const char *ext,\n-\t\t\t\t\t const char **keep_hashes,\n-\t\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t\t const struct strvec *keep_hashes);\n extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t\tconst char *idx_name);\n \n@@ -1109,8 +1108,7 @@ static int link_midx_to_chain(struct multi_pack_index *m)\n }\n \n static void clear_midx_files(struct odb_source *source,\n-\t\t\t const char **hashes, uint32_t hashes_nr,\n-\t\t\t unsigned incremental)\n+\t\t\t const struct strvec *hashes, unsigned incremental)\n {\n \t/*\n \t * if incremental:\n@@ -1124,13 +1122,15 @@ static void clear_midx_files(struct odb_source *source,\n \t */\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX };\n-\tuint32_t i, j;\n+\tuint32_t i;\n \n \tfor (i = 0; i < ARRAY_SIZE(exts); i++) {\n-\t\tclear_incremental_midx_files_ext(source, exts[i],\n-\t\t\t\t\t\t hashes, hashes_nr);\n-\t\tfor (j = 0; j < hashes_nr; j++)\n-\t\t\tclear_midx_files_ext(source, exts[i], hashes[j]);\n+\t\tclear_incremental_midx_files_ext(source, exts[i], hashes);\n+\t\tif (hashes) {\n+\t\t\tfor (size_t j = 0; j < hashes->nr; j++)\n+\t\t\t\tclear_midx_files_ext(source, exts[i],\n+\t\t\t\t\t\t hashes->v[j]);\n+\t\t}\n \t}\n \n \tif (incremental)\n@@ -1267,8 +1267,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = -1;\n-\tconst char **keep_hashes = NULL;\n-\tsize_t keep_hashes_nr = 0;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tstruct chunkfile *cf;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n@@ -1708,32 +1707,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n \t\tdie(_(\"too many multi-pack-indexes\"));\n \n-\tif (ctx.compact) {\n-\t\tstruct multi_pack_index *m;\n-\n-\t\t/*\n-\t\t * Keep all MIDX layers excluding those in the range [from, to].\n-\t\t */\n-\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n-\t\t\tkeep_hashes_nr++;\n-\t\tfor (m = ctx.m;\n-\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n-\t\t m = m->base_midx)\n-\t\t\tkeep_hashes_nr++;\n-\n-\t\tkeep_hashes_nr++; /* include the compacted layer */\n-\t} else {\n-\t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n-\t}\n-\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n-\n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\tstruct multi_pack_index **layers = NULL;\n \t\tsize_t layers_nr = 0, layers_alloc = 0;\n-\t\tsize_t j = 0;\n \n \t\tif (!chainf) {\n \t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n@@ -1761,12 +1740,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n \t\t\twhile (layers_nr)\n-\t\t\t\tkeep_hashes[j++] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\t\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \n-\t\t\tkeep_hashes[j++] =\n-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t r->hash_algo));\n \n \t\t\tfor (mp = ctx.m;\n \t\t\t mp && midx_hashcmp(mp, ctx.compact_to,\n@@ -1776,31 +1755,29 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t\tlayers[layers_nr++] = mp;\n \t\t\t}\n \t\t\twhile (layers_nr)\n-\t\t\t\tkeep_hashes[j++] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\t\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \t\t} else {\n \t\t\tfor (; m; m = m->base_midx) {\n \t\t\t\tALLOC_GROW(layers, layers_nr + 1, layers_alloc);\n \t\t\t\tlayers[layers_nr++] = m;\n \t\t\t}\n \t\t\twhile (layers_nr)\n-\t\t\t\tkeep_hashes[j++] =\n-\t\t\t\t\txstrdup(midx_get_checksum_hex(layers[--layers_nr]));\n+\t\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t\t midx_get_checksum_hex(layers[--layers_nr]));\n \n-\t\t\tkeep_hashes[j++] =\n-\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n-\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tstrvec_push(&keep_hashes,\n+\t\t\t\t hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t r->hash_algo));\n \t\t}\n \n-\t\tASSERT(j == keep_hashes_nr);\n-\n \t\tfree(layers);\n \n-\t\tfor (uint32_t i = 0; i < j; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n+\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n+\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n \t} else {\n-\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n-\t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n+\t\tstrvec_push(&keep_hashes,\n+\t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n \t}\n \n \tif (ctx.m || ctx.base_midx)\n@@ -1809,8 +1786,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, keep_hashes, keep_hashes_nr,\n-\t\t\t ctx.incremental);\n+\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n \tresult = 0;\n \n cleanup:\n@@ -1826,11 +1802,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n-\tif (keep_hashes) {\n-\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n-\t\t\tfree((char *)keep_hashes[i]);\n-\t\tfree(keep_hashes);\n-\t}\n+\tstrvec_clear(&keep_hashes);\n \tstrbuf_release(&midx_name);\n \tclose_midx(midx_to_free);\n \ndiff --git a/midx.c b/midx.c\nindex f75e3c9fa6d..bcb8c999015 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -12,6 +12,7 @@\n #include \"chunk-format.h\"\n #include \"pack-bitmap.h\"\n #include \"pack-revindex.h\"\n+#include \"strvec.h\"\n \n #define MIDX_PACK_ERROR ((void *)(intptr_t)-1)\n \n@@ -19,8 +20,7 @@ int midx_checksum_valid(struct multi_pack_index *m);\n void clear_midx_files_ext(struct odb_source *source, const char *ext,\n \t\t\t const char *keep_hash);\n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr);\n+\t\t\t\t const struct strvec *keep_hashes);\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n@@ -799,22 +799,22 @@ void clear_midx_files_ext(struct odb_source *source, const char *ext,\n }\n \n void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext,\n-\t\t\t\t char **keep_hashes,\n-\t\t\t\t uint32_t hashes_nr)\n+\t\t\t\t const struct strvec *keep_hashes)\n {\n \tstruct clear_midx_data data = {\n \t\t.keep = STRSET_INIT,\n \t\t.ext = ext,\n \t};\n \tstruct strbuf buf = STRBUF_INIT;\n-\tuint32_t i;\n \n-\tfor (i = 0; i < hashes_nr; i++) {\n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\", keep_hashes[i],\n-\t\t\t ext);\n+\tif (keep_hashes) {\n+\t\tfor (size_t i = 0; i < keep_hashes->nr; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"multi-pack-index-%s.%s\",\n+\t\t\t\t keep_hashes->v[i], ext);\n \n-\t\tstrset_add(&data.keep, buf.buf);\n+\t\t\tstrset_add(&data.keep, buf.buf);\n+\t\t}\n \t}\n \n \tfor_each_file_in_pack_subdir(source->path, \"multi-pack-index.d\",\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543659","messageId":"cac3fd54bf0c5fe23ca40eb9e13f850ae0a1996f.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 05/16] midx: introduce `--no-write-chain-file` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:57:51Z","receivedAt":"2026-05-19T15:57:54Z","isPatch":true,"body":"When writing an incremental MIDX layer, the MIDX machinery writes the\nnew layer into the multi-pack-index.d directory and then updates the\nmulti-pack-index-chain file to include the freshly written layer.\n\nFuture callers however may not wish to immediately update the MIDX chain\nitself, preferring instead to write out new layer(s) themselves before\natomically updating the chain. Concretely, the new incremental\nMIDX-based repacking strategy will want to do exactly this (that is,\nassemble the new MIDX chain itself before writing a new chain file and\natomically linking it into place).\n\nIntroduce a `--no-write-chain-file` flag that:\n\n * writes the new MIDX layer into the multi-pack-index.d directory\n\n * prints its checksum\n\n * does not update the multi-pack-index-chain file.\n\nThe MIDX chain file (and thus, the lock protecting it) remain untouched,\nallowing callers to assemble the chain themselves. This flag requires\n`--incremental`, since the notion of a separate layer only makes sense\nfor incremental MIDXs.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 17 ++++++++--\n builtin/multi-pack-index.c | 28 +++++++++++++++--\n midx-write.c | 42 ++++++++++++++++---------\n midx.h | 1 +\n t/t5334-incremental-multi-pack-index.sh | 17 ++++++++++\n t/t5335-compact-multi-pack-index.sh | 36 +++++++++++++++++++++\n 6 files changed, 123 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 3a5aa227784..c26196815e2 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -11,9 +11,9 @@ SYNOPSIS\n [verse]\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n-\t\t [--refs-snapshot=]\n+\t\t [--refs-snapshot=] [--[no-]write-chain-file]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] \n+\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -83,6 +83,13 @@ marker).\n \t\tand packs not present in an existing MIDX layer.\n \t\tMigrates non-incremental MIDXs to incremental ones when\n \t\tnecessary.\n+\n+\t--[no-]write-chain-file::\n+\t\tWhen used with `--incremental`, write a new MIDX layer\n+\t\tbut do not update the multi-pack-index-chain file.\n+\t\tThe checksum of the new layer is printed to standard\n+\t\toutput, allowing the caller to assemble and write the\n+\t\tchain itself. Requires `--incremental`.\n --\n \n compact::\n@@ -97,6 +104,12 @@ compact::\n \n \t--[no-]bitmap::\n \t\tControl whether or not a multi-pack bitmap is written.\n+\n+\t--[no-]write-chain-file::\n+\t\tWhen used with `--incremental`, write a new compacted\n+\t\tMIDX layer but do not update the multi-pack-index-chain\n+\t\tfile. The checksum of the new layer is printed to\n+\t\tstandard output. Requires `--incremental`.\n --\n +\n Note that the compact command requires writing a version-2 midx that\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 0f72d96c02d..f861b4b8394 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -16,11 +16,11 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=]\")\n+\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] \")\n+\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -153,6 +153,9 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n+\t\t\tN_(\"write the multi-pack-index chain file\"),\n+\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n@@ -178,6 +181,15 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \tif (argc)\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--no-write-chain-file\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -221,6 +233,9 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n+\t\t\tN_(\"write the multi-pack-index chain file\"),\n+\t\t\tMIDX_WRITE_NO_CHAIN),\n \t\tOPT_END(),\n \t};\n \n@@ -239,6 +254,15 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \tif (argc != 2)\n \t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n \t\t\t\t options);\n+\n+\tif (opts.flags & MIDX_WRITE_NO_CHAIN &&\n+\t !(opts.flags & MIDX_WRITE_INCREMENTAL)) {\n+\t\terror(_(\"cannot use %s without %s\"),\n+\t\t \"--no-write-chain-file\", \"--incremental\");\n+\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(the_repository);\n \n \tFREE_AND_NULL(options);\ndiff --git a/midx-write.c b/midx-write.c\nindex 5d9409a9741..38c898e5ff5 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1257,7 +1257,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t start_pack;\n \tstruct hashfile *f = NULL;\n-\tstruct lock_file lk;\n+\tstruct lock_file lk = LOCK_INIT;\n \tstruct tempfile *incr;\n \tstruct write_midx_context ctx = {\n \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n@@ -1601,11 +1601,14 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \n \tif (ctx.incremental) {\n-\t\tstruct strbuf lock_name = STRBUF_INIT;\n+\t\tif (!(opts->flags & MIDX_WRITE_NO_CHAIN)) {\n+\t\t\tstruct strbuf lock_name = STRBUF_INIT;\n \n-\t\tget_midx_chain_filename(opts->source, &lock_name);\n-\t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n-\t\tstrbuf_release(&lock_name);\n+\t\t\tget_midx_chain_filename(opts->source, &lock_name);\n+\t\t\thold_lock_file_for_update(&lk, lock_name.buf,\n+\t\t\t\t\t\t LOCK_DIE_ON_ERROR);\n+\t\t\tstrbuf_release(&lock_name);\n+\t\t}\n \n \t\tincr = mks_tempfile_m(midx_name.buf, 0444);\n \t\tif (!incr) {\n@@ -1707,16 +1710,23 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n \t\tdie(_(\"too many multi-pack-indexes\"));\n \n+\tif (!is_lock_file_locked(&lk))\n+\t\tprintf(\"%s\\n\", hash_to_hex_algop(midx_hash, r->hash_algo));\n+\telse if (opts->flags & MIDX_WRITE_NO_CHAIN)\n+\t\tBUG(\"lockfile held with MIDX_WRITE_NO_CHAIN set?\");\n+\n \tif (ctx.incremental) {\n-\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n \t\tstruct strbuf final_midx_name = STRBUF_INIT;\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\tstruct multi_pack_index **layers = NULL;\n \t\tsize_t layers_nr = 0, layers_alloc = 0;\n \n-\t\tif (!chainf) {\n-\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n-\t\t\tgoto cleanup;\n+\t\tif (is_lock_file_locked(&lk)){\n+\t\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\t\tif (!chainf) {\n+\t\t\t\terror_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t}\n \n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n@@ -1773,8 +1783,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t\tfree(layers);\n \n-\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n-\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes.v[i]);\n+\t\tif (is_lock_file_locked(&lk))\n+\t\t\tfor (size_t i = 0; i < keep_hashes.nr; i++)\n+\t\t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\",\n+\t\t\t\t\tkeep_hashes.v[i]);\n \t} else {\n \t\tstrvec_push(&keep_hashes,\n \t\t\t hash_to_hex_algop(midx_hash, r->hash_algo));\n@@ -1783,10 +1795,12 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.m || ctx.base_midx)\n \t\todb_close(ctx.repo->objects);\n \n-\tif (commit_lock_file(&lk) < 0)\n-\t\tdie_errno(_(\"could not write multi-pack-index\"));\n+\tif (is_lock_file_locked(&lk)) {\n+\t\tif (commit_lock_file(&lk) < 0)\n+\t\t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t\tclear_midx_files(opts->source, &keep_hashes, ctx.incremental);\n+\t}\n \tresult = 0;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex 08f3728e520..5b193882dcf 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -83,6 +83,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n #define MIDX_WRITE_COMPACT (1 << 6)\n+#define MIDX_WRITE_NO_CHAIN (1 << 7)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex c9f5b4e87aa..66d6894761b 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -96,6 +96,23 @@ test_expect_success 'show object from second pack' '\n \tgit cat-file -p 2.2\n '\n \n+test_expect_success 'write MIDX layer with --no-write-chain-file' '\n+\ttest_commit no-write-chain-file &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--no-write-chain-file)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file' '\n+\ttest_must_fail git multi-pack-index write --bitmap --no-write-chain-file 2>err &&\n+\ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 40f3844282f..1a65d48b62b 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -290,4 +290,40 @@ test_expect_success 'MIDX compaction with bitmaps (non-trivial)' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --no-write-chain-file' '\n+\tgit init midx-compact-with--no-write-chain-file &&\n+\t(\n+\t\tcd midx-compact-with--no-write-chain-file &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\tlayer=\"$(git multi-pack-index compact --incremental \\\n+\t\t\t--no-write-chain-file \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n+\n+\t\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\n+\t\t# After writing the new layer, insert it into the chain\n+\t\t# manually. This is done in order to make $layer visible\n+\t\t# to the read-midx test helper below, and matches what\n+\t\t# the MIDX command would do without --no-write-chain-file.\n+\t\t{\n+\t\t\tnth_line 1 \"$midx_chain.bak\" &&\n+\t\t\techo $layer &&\n+\t\t\tnth_line 4 \"$midx_chain.bak\"\n+\t\t} >$midx_chain &&\n+\n+\t\ttest-tool read-midx $objdir $layer >midx.data &&\n+\t\tgrep \"^pack-B-.*\\.idx\" midx.data &&\n+\t\tgrep \"^pack-C-.*\\.idx\" midx.data\n+\n+\t)\n+'\n+\n test_done\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543660","messageId":"1bbb387d6b6204045d97882fd8775dbff12dedbb.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:57:54Z","receivedAt":"2026-05-19T15:57:57Z","isPatch":true,"body":"Both `compact` and `write --incremental` fix the base of the resulting\nMIDX layer: `compact` always places the compacted result on top of\n\"from's\" immediate parent in the chain, and `write --incremental` always\nappends a new layer to the existing tip. In both cases the base is not\nconfigurable.\n\nFuture callers need additional flexibility. For instance, the incremental\nMIDX-based repacking code may wish to write a layer based on some\nintermediate ancestor rather than the current tip, or produce a root\nlayer when replacing the bottommost entries in the chain.\n\nIntroduce a new `--base` option for both subcommands to specify the\nchecksum of the MIDX layer to use as the base. The given checksum must\nrefer to a valid layer in the MIDX chain that is an ancestor of the\ntopmost layer being written or compacted.\n\nThe special value \"none\" is accepted to produce a root layer with no\nparent. This will be needed when the incremental repacking machinery\ndetermines that the bottommost layers of the chain should be replaced.\n\nIf no `--base` is given, behavior is unchanged: `compact` uses \"from's\"\nimmediate parent in the chain, and `write` appends to the existing tip.\n\nFor the `write` subcommand, `--base` requires `--no-write-chain-file`. A plain\n`write --incremental` appends a new layer to the live chain tip with no\nmechanism to atomically replace it; overriding the base would produce a\nlayer that does not extend the tip, breaking chain invariants. With\n`--no-write-chain-file` the chain is left unmodified and the caller is\nresponsible for assembling a valid chain.\n\nFor `compact`, no such restriction applies. The compaction operation\natomically replaces the compacted range in the chain file, so writing\nthe result on top of any valid ancestor preserves chain invariants.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 17 +++++-\n builtin/multi-pack-index.c | 24 ++++++--\n midx-write.c | 34 ++++++++++-\n midx.h | 5 +-\n t/t5334-incremental-multi-pack-index.sh | 30 ++++++++++\n t/t5335-compact-multi-pack-index.sh | 77 +++++++++++++++++++++++++\n 6 files changed, 178 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex c26196815e2..c6d23aeeb9a 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -12,8 +12,10 @@ SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=] [--[no-]write-chain-file]\n+\t\t\t [--base=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t [--[no-]bitmap] [--[no-]write-chain-file] \n+\t\t [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\n+\t\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -90,6 +92,13 @@ marker).\n \t\tThe checksum of the new layer is printed to standard\n \t\toutput, allowing the caller to assemble and write the\n \t\tchain itself. Requires `--incremental`.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base when writing a new incremental layer.\n+\t\tThe special value `none` indicates that the new layer\n+\t\tshould have no base (i.e., it becomes a root layer).\n+\t\tRequires `--no-write-chain-file`.\n --\n \n compact::\n@@ -110,6 +119,12 @@ compact::\n \t\tMIDX layer but do not update the multi-pack-index-chain\n \t\tfile. The checksum of the new layer is printed to\n \t\tstandard output. Requires `--incremental`.\n+\n+\t--base=::\n+\t\tSpecify the checksum of an existing MIDX layer to use\n+\t\tas the base for the compacted result, instead of using\n+\t\tthe immediate parent of ``. The special value\n+\t\t`none` indicates that the result should have no base.\n --\n +\n Note that the compact command requires writing a version-2 midx that\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex f861b4b8394..00ffb36394d 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -16,11 +16,13 @@\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n-\t \" [--refs-snapshot=] [--[no-]write-chain-file]\")\n+\t \" [--refs-snapshot=] [--[no-]write-chain-file]\\n\" \\\n+\t \" [--base=]\")\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" [--[no-]bitmap] [--[no-]write-chain-file] \")\n+\t \" [--[no-]bitmap] [--base=] [--[no-]write-chain-file]\\n\" \\\n+\t \" \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -63,6 +65,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tchar *object_dir;\n \tconst char *preferred_pack;\n+\tconst char *incremental_base;\n \tchar *refs_snapshot;\n \tunsigned long batch_size;\n \tunsigned flags;\n@@ -151,6 +154,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_NEGBIT(0, \"write-chain-file\", &opts.flags,\n@@ -190,6 +195,13 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t\t options);\n \t}\n \n+\tif (opts.incremental_base &&\n+\t !(opts.flags & MIDX_WRITE_NO_CHAIN)) {\n+\t\terror(_(\"cannot use --base without --no-write-chain-file\"));\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\t}\n+\n \tsource = handle_object_dir_option(repo);\n \n \tFREE_AND_NULL(options);\n@@ -201,7 +213,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \n \t\tret = write_midx_file_only(source, &packs,\n \t\t\t\t\t opts.preferred_pack,\n-\t\t\t\t\t opts.refs_snapshot, opts.flags);\n+\t\t\t\t\t opts.refs_snapshot,\n+\t\t\t\t\t opts.incremental_base, opts.flags);\n \n \t\tstring_list_clear(&packs, 0);\n \t\tfree(opts.refs_snapshot);\n@@ -229,6 +242,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_STRING(0, \"base\", &opts.incremental_base, N_(\"checksum\"),\n+\t\t\t N_(\"base MIDX for incremental writes\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n@@ -290,7 +305,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n \t}\n \n-\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n+\tret = write_midx_file_compact(source, from_midx, to_midx,\n+\t\t\t\t opts.incremental_base, opts.flags);\n \n \treturn ret;\n }\ndiff --git a/midx-write.c b/midx-write.c\nindex 38c898e5ff5..561e9eedc0e 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1247,6 +1247,7 @@ struct write_midx_opts {\n \n \tconst char *preferred_pack_name;\n \tconst char *refs_snapshot;\n+\tconst char *incremental_base;\n \tunsigned flags;\n };\n \n@@ -1330,11 +1331,32 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t/*\n \t * If compacting MIDX layer(s) in the range [from, to], then the\n-\t * compacted MIDX will share the same base MIDX as 'from'.\n+\t * compacted MIDX will share the same base MIDX as 'from',\n+\t * unless a custom --base is specified (see below).\n \t */\n \tif (ctx.compact)\n \t\tctx.base_midx = ctx.compact_from->base_midx;\n \n+\tif (opts->incremental_base) {\n+\t\tif (!strcmp(opts->incremental_base, \"none\")) {\n+\t\t\tctx.base_midx = NULL;\n+\t\t} else {\n+\t\t\twhile (ctx.base_midx) {\n+\t\t\t\tconst char *cmp = midx_get_checksum_hex(ctx.base_midx);\n+\t\t\t\tif (!strcmp(opts->incremental_base, cmp))\n+\t\t\t\t\tbreak;\n+\n+\t\t\t\tctx.base_midx = ctx.base_midx->base_midx;\n+\t\t\t}\n+\n+\t\t\tif (!ctx.base_midx) {\n+\t\t\t\terror(_(\"could not find base MIDX '%s'\"),\n+\t\t\t\t opts->incremental_base);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n@@ -1827,7 +1849,8 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n int write_midx_file(struct odb_source *source,\n \t\t const char *preferred_pack_name,\n-\t\t const char *refs_snapshot, unsigned flags)\n+\t\t const char *refs_snapshot,\n+\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n@@ -1842,13 +1865,16 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags)\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.packs_to_include = packs_to_include,\n \t\t.preferred_pack_name = preferred_pack_name,\n \t\t.refs_snapshot = refs_snapshot,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags,\n \t};\n \n@@ -1858,12 +1884,14 @@ int write_midx_file_only(struct odb_source *source,\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags)\n {\n \tstruct write_midx_opts opts = {\n \t\t.source = source,\n \t\t.compact_from = from,\n \t\t.compact_to = to,\n+\t\t.incremental_base = incremental_base,\n \t\t.flags = flags | MIDX_WRITE_COMPACT,\n \t};\n \ndiff --git a/midx.h b/midx.h\nindex 5b193882dcf..77dd66de02b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -132,10 +132,13 @@ int write_midx_file(struct odb_source *source,\n int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot, unsigned flags);\n+\t\t\t const char *refs_snapshot,\n+\t\t\t const char *incremental_base,\n+\t\t\t unsigned flags);\n int write_midx_file_compact(struct odb_source *source,\n \t\t\t struct multi_pack_index *from,\n \t\t\t struct multi_pack_index *to,\n+\t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct odb_source *source, unsigned flags);\ndiff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\nindex 66d6894761b..68a103d13d2 100755\n--- a/t/t5334-incremental-multi-pack-index.sh\n+++ b/t/t5334-incremental-multi-pack-index.sh\n@@ -113,6 +113,36 @@ test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file\n \ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n '\n \n+test_expect_success 'write MIDX layer with --base without --no-write-chain-file' '\n+\ttest_must_fail git multi-pack-index write --bitmap --incremental \\\n+\t\t--base=none 2>err &&\n+\ttest_grep \"cannot use --base without --no-write-chain-file\" err\n+'\n+\n+test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '\n+\ttest_commit base-none &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--no-write-chain-file --base=none)\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n+test_expect_success 'write MIDX layer with --base= and --no-write-chain-file' '\n+\ttest_commit base-hash &&\n+\tgit repack -d &&\n+\n+\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n+\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n+\n+\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n+\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n+'\n+\n for reuse in false single multi\n do\n \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 1a65d48b62b..ec1dafe89fc 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -304,6 +304,7 @@ test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \n \t\tlayer=\"$(git multi-pack-index compact --incremental \\\n \t\t\t--no-write-chain-file \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 3 \"$midx_chain\")\")\" &&\n \n@@ -326,4 +327,80 @@ test_expect_success 'MIDX compaction with --no-write-chain-file' '\n \t)\n '\n \n+test_expect_success 'MIDX compaction with --base' '\n+\tgit init midx-compact-with--base &&\n+\t(\n+\t\tcd midx-compact-with--base &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tnth_line 1 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 1 \"$midx_chain\" >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with --base=none' '\n+\tgit init midx-compact-base-none &&\n+\t(\n+\t\tcd midx-compact-base-none &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D &&\n+\n+\t\ttest_line_count = 4 $midx_chain &&\n+\n+\t\tcp \"$midx_chain\" \"$midx_chain\".bak &&\n+\n+\t\t# Compact the two bottommost layers (A and B) into a new\n+\t\t# root layer with no parent.\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t--base=none \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" &&\n+\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\t# The upper layers (C and D) should be preserved\n+\t\t# unchanged.\n+\t\tnth_line 3 \"$midx_chain.bak\" >expect &&\n+\t\tnth_line 4 \"$midx_chain.bak\" >>expect &&\n+\t\tnth_line 2 \"$midx_chain\" >actual &&\n+\t\tnth_line 3 \"$midx_chain\" >>actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bogus --base checksum' '\n+\tgit init midx-compact-bogus-base &&\n+\t(\n+\t\tcd midx-compact-bogus-base &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t--base=deadbeef \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 3 \"$midx_chain\")\" 2>err &&\n+\t\ttest_grep \"could not find base MIDX\" err\n+\t)\n+'\n+\n test_done\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543661","messageId":"4a93adb3ad3f2d408c03db885c698fbcbf39d109.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 07/16] repack: track the ODB source via existing_packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:57:57Z","receivedAt":"2026-05-19T15:58:00Z","isPatch":true,"body":"Store the ODB source in the `existing_packs` struct and use that in\nplace of the raw `repo->objects->sources` access within `cmd_repack()`.\n\nThe source used is still assigned from the first source in the list, so\nthere are no functional changes in this commit. The changes instead\nserve two purposes (one immediate, one not):\n\n - The incremental MIDX-based repacking machinery will need to know what\n source is being used to read the existing MIDX/chain (should one\n exist).\n\n - In the future, if \"git repack\" is taught how to operate on other\n object sources, this field will serve as the authoritative value for\n that source.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 5 ++---\n repack.c | 2 ++\n repack.h | 1 +\n 3 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 4c5a82c2c8d..24be147d39a 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -417,7 +417,7 @@ int cmd_repack(int argc,\n \t\t * midx_has_unknown_packs() will make the decision for\n \t\t * us.\n \t\t */\n-\t\tif (!get_multi_pack_index(repo->objects->sources))\n+\t\tif (!get_multi_pack_index(existing.source))\n \t\t\tmidx_must_contain_cruft = 1;\n \t}\n \n@@ -564,8 +564,7 @@ int cmd_repack(int argc,\n \t\tunsigned flags = 0;\n \t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))\n \t\t\tflags |= MIDX_WRITE_INCREMENTAL;\n-\t\twrite_midx_file(repo->objects->sources,\n-\t\t\t\tNULL, NULL, flags);\n+\t\twrite_midx_file(existing.source, NULL, NULL, flags);\n \t}\n \n cleanup:\ndiff --git a/repack.c b/repack.c\nindex 596841027af..2ee6b51420a 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -154,6 +154,8 @@ void existing_packs_collect(struct existing_packs *existing,\n \t\t\tstring_list_append(&existing->non_kept_packs, buf.buf);\n \t}\n \n+\texisting->source = existing->repo->objects->sources;\n+\n \tstring_list_sort(&existing->kept_packs);\n \tstring_list_sort(&existing->non_kept_packs);\n \tstring_list_sort(&existing->cruft_packs);\ndiff --git a/repack.h b/repack.h\nindex bc9f2e1a5de..c0e9f0ca647 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -56,6 +56,7 @@ struct packed_git;\n \n struct existing_packs {\n \tstruct repository *repo;\n+\tstruct odb_source *source;\n \tstruct string_list kept_packs;\n \tstruct string_list non_kept_packs;\n \tstruct string_list cruft_packs;\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543662","messageId":"8d1b8b1d301313cda1a952b02b162ccd02f40c01.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 08/16] midx: expose `midx_layer_contains_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:58:00Z","receivedAt":"2026-05-19T15:58:03Z","isPatch":true,"body":"Rename the function `midx_contains_pack_1()` to instead be called\n`midx_layer_contains_pack()` and make it accessible. Unlike\n`midx_contains_pack()` (which recurses through the entire chain), this\nfunction checks only a single MIDX layer.\n\nThis will be used by a subsequent commit to determine whether a given\npack belongs to the tip MIDX layer specifically, rather than to any\nlayer in the chain.\n\nNo functional changes are present in this commit.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 6 +++---\n midx.h | 2 ++\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex bcb8c999015..dc86c8e7fee 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -667,8 +667,8 @@ static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n \t\t m->pack_names[*(const size_t *)b]);\n }\n \n-static int midx_contains_pack_1(struct multi_pack_index *m,\n-\t\t\t\tconst char *idx_or_pack_name)\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -709,7 +709,7 @@ static int midx_contains_pack_1(struct multi_pack_index *m,\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n {\n \tfor (; m; m = m->base_midx)\n-\t\tif (midx_contains_pack_1(m, idx_or_pack_name))\n+\t\tif (midx_layer_contains_pack(m, idx_or_pack_name))\n \t\t\treturn 1;\n \treturn 0;\n }\ndiff --git a/midx.h b/midx.h\nindex 77dd66de02b..3ee12dd08ec 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -119,6 +119,8 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n int fill_midx_entry(struct multi_pack_index *m, const struct object_id *oid, struct pack_entry *e);\n int midx_contains_pack(struct multi_pack_index *m,\n \t\t const char *idx_or_pack_name);\n+int midx_layer_contains_pack(struct multi_pack_index *m,\n+\t\t\t const char *idx_or_pack_name);\n int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n int prepare_multi_pack_index_one(struct odb_source *source);\n \n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543663","messageId":"42111e5f75d57d377480616505ad25c2b70f8870.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 09/16] repack-midx: factor out `repack_prepare_midx_command()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:58:03Z","receivedAt":"2026-05-19T15:58:06Z","isPatch":true,"body":"The `write_midx_included_packs()` function assembles and executes a\n`git multi-pack-index write` command, constructing the argument list\ninline.\n\nFuture commits will introduce additional callers that need to construct\nsimilar `git multi-pack-index` commands (for both `write` and `compact`\nsubcommands), so extract the common portions of the command setup into a\nreusable `repack_prepare_midx_command()` helper.\n\nThe extracted helper sets `git_cmd`, pushes `multi-pack-index` and a\nsubcommand, and handles `--progress`/`--no-progress` and `--bitmap`\nflags. The remaining arguments that are specific to the `write`\nsubcommand (such as `--stdin-packs`) are left to the caller.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 30 +++++++++++++++++++-----------\n 1 file changed, 19 insertions(+), 11 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 0682b80c427..5634dc186d0 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -275,6 +275,23 @@ static void remove_redundant_bitmaps(struct string_list *include,\n \tstrbuf_release(&path);\n }\n \n+static void repack_prepare_midx_command(struct child_process *cmd,\n+\t\t\t\t\tstruct repack_write_midx_opts *opts,\n+\t\t\t\t\tconst char *subcommand)\n+{\n+\tcmd->git_cmd = 1;\n+\n+\tstrvec_pushl(&cmd->args, \"multi-pack-index\", subcommand, NULL);\n+\n+\tif (opts->show_progress)\n+\t\tstrvec_push(&cmd->args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd->args, \"--no-progress\");\n+\n+\tif (opts->write_bitmaps)\n+\t\tstrvec_push(&cmd->args, \"--bitmap\");\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -289,18 +306,9 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \n \tcmd.in = -1;\n-\tcmd.git_cmd = 1;\n \n-\tstrvec_push(&cmd.args, \"multi-pack-index\");\n-\tstrvec_pushl(&cmd.args, \"write\", \"--stdin-packs\", NULL);\n-\n-\tif (opts->show_progress)\n-\t\tstrvec_push(&cmd.args, \"--progress\");\n-\telse\n-\t\tstrvec_push(&cmd.args, \"--no-progress\");\n-\n-\tif (opts->write_bitmaps)\n-\t\tstrvec_push(&cmd.args, \"--bitmap\");\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543664","messageId":"ed76e6efd1c9a4ba304a768734ef5065e00c1957.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 10/16] repack-midx: extract `repack_fill_midx_stdin_packs()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:58:07Z","receivedAt":"2026-05-19T15:58:09Z","isPatch":true,"body":"The function `write_midx_included_packs()` manages the lifecycle of\nwriting packs to stdin when running `git multi-pack-index write` as a\nchild process.\n\nExtract a standalone `repack_fill_midx_stdin_packs()` helper, which\nhandles `--stdin-packs` argument setup, starting the command, writing\npack names to its standard input, and finishing the command.\n\nThis simplifies `write_midx_included_packs()` and prepares for a\nsubsequent commit where the same helper is called with `cmd->out = -1`\nto capture the MIDX's checksum from the command's standard output,\nwhich is needed when writing MIDX layers with `--no-write-chain-file`.\n\nNo functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n repack-midx.c | 38 ++++++++++++++++++++++++--------------\n 1 file changed, 24 insertions(+), 14 deletions(-)\n\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 5634dc186d0..3fe83715da4 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -292,23 +292,42 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n \t\tstrvec_push(&cmd->args, \"--bitmap\");\n }\n \n+static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n+\t\t\t\t\tstruct string_list *include)\n+{\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\n+\tcmd->in = -1;\n+\n+\tstrvec_push(&cmd->args, \"--stdin-packs\");\n+\n+\tret = start_command(cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tin = xfdopen(cmd->in, \"w\");\n+\tfor_each_string_list_item(item, include)\n+\t\tfprintf(in, \"%s\\n\", item->string);\n+\tfclose(in);\n+\n+\treturn finish_command(cmd);\n+}\n+\n int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n \tstruct string_list_item *item;\n \tstruct packed_git *preferred = pack_geometry_preferred_pack(opts->geometry);\n-\tFILE *in;\n \tint ret = 0;\n \n \tmidx_included_packs(&include, opts);\n \tif (!include.nr)\n \t\tgoto done;\n \n-\tcmd.in = -1;\n-\n \trepack_prepare_midx_command(&cmd, opts, \"write\");\n-\tstrvec_push(&cmd.args, \"--stdin-packs\");\n \n \tif (preferred)\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n@@ -350,16 +369,7 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = start_command(&cmd);\n-\tif (ret)\n-\t\tgoto done;\n-\n-\tin = xfdopen(cmd.in, \"w\");\n-\tfor_each_string_list_item(item, &include)\n-\t\tfprintf(in, \"%s\\n\", item->string);\n-\tfclose(in);\n-\n-\tret = finish_command(&cmd);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543665","messageId":"9665f1b3a6404f1e69939db218a59cd2f3a14417.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 11/16] repack-geometry: prepare for incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:58:10Z","receivedAt":"2026-05-19T15:58:13Z","isPatch":true,"body":"Teach `pack_geometry_init()` to optionally restrict the set of\nrepacking candidates to only packs in the tip MIDX layer when a\n`midx_layer_threshold` is configured. If the tip layer has fewer packs\nthan the threshold, those packs are excluded entirely; otherwise only\npacks in that layer participate in the geometric repack.\n\nAlso track whether any tip-layer packs were included in the rollup\n(`midx_tip_rewritten`), which a subsequent commit will use to decide\nhow to update the MIDX chain after repacking.\n\nSigned-off-by: Taylor Blau \n---\n repack-geometry.c | 35 +++++++++++++++++++++++++++++++++++\n repack.h | 4 ++++\n 2 files changed, 39 insertions(+)\n\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex 7cebd0cb45f..2408b8a3cc2 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -4,6 +4,7 @@\n #include \"repack.h\"\n #include \"repository.h\"\n #include \"hex.h\"\n+#include \"midx.h\"\n #include \"packfile.h\"\n \n static uint32_t pack_geometry_weight(struct packed_git *p)\n@@ -31,8 +32,28 @@ void pack_geometry_init(struct pack_geometry *geometry,\n {\n \tstruct packed_git *p;\n \tstruct strbuf buf = STRBUF_INIT;\n+\tstruct multi_pack_index *m = get_multi_pack_index(existing->source);\n \n \trepo_for_each_pack(existing->repo, p) {\n+\t\tif (geometry->midx_layer_threshold_set && m &&\n+\t\t p->multi_pack_index) {\n+\t\t\t/*\n+\t\t\t * When writing MIDX layers incrementally,\n+\t\t\t * ignore packs unless they are in the most\n+\t\t\t * recent MIDX layer *and* there are at least\n+\t\t\t * 'midx_layer_threshold' packs in that layer.\n+\t\t\t *\n+\t\t\t * Otherwise 'p' is either in an older layer, or\n+\t\t\t * the youngest layer does not have enough packs\n+\t\t\t * to consider its packs as candidates for\n+\t\t\t * repacking. In either of those cases we want\n+\t\t\t * to ignore the pack.\n+\t\t\t */\n+\t\t\tif (m->num_packs < geometry->midx_layer_threshold ||\n+\t\t\t !midx_layer_contains_pack(m, pack_basename(p)))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (args->local && !p->pack_local)\n \t\t\t/*\n \t\t\t * When asked to only repack local packfiles we skip\n@@ -173,6 +194,20 @@ void pack_geometry_split(struct pack_geometry *geometry)\n \tgeometry->promisor_split = compute_pack_geometry_split(geometry->promisor_pack,\n \t\t\t\t\t\t\t geometry->promisor_pack_nr,\n \t\t\t\t\t\t\t geometry->split_factor);\n+\tfor (uint32_t i = 0; i < geometry->split; i++) {\n+\t\tstruct packed_git *p = geometry->pack[i];\n+\t\t/*\n+\t\t * During incremental MIDX/bitmap repacking, any packs\n+\t\t * included in the rollup are either (a) not MIDX'd, or\n+\t\t * (b) contained in the tip layer iff it has at least\n+\t\t * the threshold number of packs.\n+\t\t *\n+\t\t * In the latter case, we can safely conclude that the\n+\t\t * tip of the MIDX chain will be rewritten.\n+\t\t */\n+\t\tif (p->multi_pack_index)\n+\t\t\tgeometry->midx_tip_rewritten = true;\n+\t}\n }\n \n struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry)\ndiff --git a/repack.h b/repack.h\nindex c0e9f0ca647..77d24ee45fb 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -108,6 +108,10 @@ struct pack_geometry {\n \tuint32_t promisor_pack_nr, promisor_pack_alloc;\n \tuint32_t promisor_split;\n \n+\tuint32_t midx_layer_threshold;\n+\tbool midx_layer_threshold_set;\n+\tbool midx_tip_rewritten;\n+\n \tint split_factor;\n };\n \n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543666","messageId":"e0db62b9f10946eed38bb4925194821c699449fa.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 12/16] builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:58:13Z","receivedAt":"2026-05-19T15:58:16Z","isPatch":true,"body":"Change the --write-midx (-m) flag from an OPT_BOOL to an OPT_CALLBACK\nthat accepts an optional mode argument. Introduce an enum with\nREPACK_WRITE_MIDX_NONE and REPACK_WRITE_MIDX_DEFAULT to distinguish\nbetween the two states, and update all existing boolean checks\naccordingly.\n\nFor now, passing no argument (or just `-m`) selects the default mode,\npreserving existing behavior. A subsequent commit will add a new mode\nfor writing incremental MIDXs.\n\nExtract repack_write_midx() as a dispatcher that selects the\nappropriate MIDX-writing implementation based on the mode.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 50 ++++++++++++++++++++++++++++++++++++------------\n repack-midx.c | 14 +++++++++++++-\n repack.h | 8 +++++++-\n 3 files changed, 58 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 24be147d39a..5d366340c34 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -97,6 +97,24 @@ static int repack_config(const char *var, const char *value,\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n+static int option_parse_write_midx(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum repack_write_midx_mode *cfg = opt->value;\n+\n+\tif (unset) {\n+\t\t*cfg = REPACK_WRITE_MIDX_NONE;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!arg || !*arg)\n+\t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse\n+\t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_repack(int argc,\n \t const char **argv,\n \t const char *prefix,\n@@ -119,7 +137,7 @@ int cmd_repack(int argc,\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct pack_objects_args po_args = PACK_OBJECTS_ARGS_INIT;\n \tstruct pack_objects_args cruft_po_args = PACK_OBJECTS_ARGS_INIT;\n-\tint write_midx = 0;\n+\tenum repack_write_midx_mode write_midx = REPACK_WRITE_MIDX_NONE;\n \tconst char *cruft_expiration = NULL;\n \tconst char *expire_to = NULL;\n \tconst char *filter_to = NULL;\n@@ -185,8 +203,14 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_INTEGER('g', \"geometric\", &geometry.split_factor,\n \t\t\t N_(\"find a geometric progression with factor \")),\n-\t\tOPT_BOOL('m', \"write-midx\", &write_midx,\n-\t\t\t N_(\"write a multi-pack index of the resulting packs\")),\n+\t\tOPT_CALLBACK_F(0, \"write-midx\", &write_midx,\n+\t\t\t N_(\"mode\"),\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t PARSE_OPT_OPTARG, option_parse_write_midx),\n+\t\tOPT_SET_INT_F('m', NULL, &write_midx,\n+\t\t\t N_(\"write a multi-pack index of the resulting packs\"),\n+\t\t\t REPACK_WRITE_MIDX_DEFAULT,\n+\t\t\t PARSE_OPT_HIDDEN),\n \t\tOPT_STRING(0, \"expire-to\", &expire_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n \t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n@@ -221,14 +245,16 @@ int cmd_repack(int argc,\n \t\tpack_everything |= ALL_INTO_ONE;\n \n \tif (write_bitmaps < 0) {\n-\t\tif (!write_midx &&\n+\t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n \t}\n \tif (po_args.pack_kept_objects < 0)\n-\t\tpo_args.pack_kept_objects = write_bitmaps > 0 && !write_midx;\n+\t\tpo_args.pack_kept_objects = write_bitmaps > 0 &&\n+\t\t\twrite_midx == REPACK_WRITE_MIDX_NONE;\n \n-\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n+\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) &&\n+\t write_midx == REPACK_WRITE_MIDX_NONE)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n \tif (write_bitmaps && po_args.local &&\n@@ -244,7 +270,7 @@ int cmd_repack(int argc,\n \t\twrite_bitmaps = 0;\n \t}\n \n-\tif (write_midx && write_bitmaps) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \n \t\tstrbuf_addf(&path, \"%s/%s_XXXXXX\",\n@@ -297,7 +323,7 @@ int cmd_repack(int argc,\n \t}\n \tif (repo_has_promisor_remote(repo))\n \t\tstrvec_push(&cmd.args, \"--exclude-promisor-objects\");\n-\tif (!write_midx) {\n+\tif (write_midx == REPACK_WRITE_MIDX_NONE) {\n \t\tif (write_bitmaps > 0)\n \t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n \t\telse if (write_bitmaps < 0)\n@@ -519,7 +545,7 @@ int cmd_repack(int argc,\n \tif (delete_redundant && pack_everything & ALL_INTO_ONE)\n \t\texisting_packs_mark_for_deletion(&existing, &names);\n \n-\tif (write_midx) {\n+\tif (write_midx != REPACK_WRITE_MIDX_NONE) {\n \t\tstruct repack_write_midx_opts opts = {\n \t\t\t.existing = &existing,\n \t\t\t.geometry = &geometry,\n@@ -528,11 +554,11 @@ int cmd_repack(int argc,\n \t\t\t.packdir = packdir,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n-\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft\n+\t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.mode = write_midx,\n \t\t};\n \n-\t\tret = write_midx_included_packs(&opts);\n-\n+\t\tret = repack_write_midx(&opts);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \t}\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 3fe83715da4..b1ca3797080 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -315,7 +315,7 @@ static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n \treturn finish_command(cmd);\n }\n \n-int write_midx_included_packs(struct repack_write_midx_opts *opts)\n+static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list include = STRING_LIST_INIT_DUP;\n@@ -378,3 +378,15 @@ int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \n \treturn ret;\n }\n+\n+int repack_write_midx(struct repack_write_midx_opts *opts)\n+{\n+\tswitch (opts->mode) {\n+\tcase REPACK_WRITE_MIDX_NONE:\n+\t\tBUG(\"write_midx mode is NONE?\");\n+\tcase REPACK_WRITE_MIDX_DEFAULT:\n+\t\treturn write_midx_included_packs(opts);\n+\tdefault:\n+\t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n+\t}\n+}\ndiff --git a/repack.h b/repack.h\nindex 77d24ee45fb..81907fcce7f 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -134,6 +134,11 @@ void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\n \n+enum repack_write_midx_mode {\n+\tREPACK_WRITE_MIDX_NONE,\n+\tREPACK_WRITE_MIDX_DEFAULT,\n+};\n+\n struct repack_write_midx_opts {\n \tstruct existing_packs *existing;\n \tstruct pack_geometry *geometry;\n@@ -143,10 +148,11 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tenum repack_write_midx_mode mode;\n };\n \n void midx_snapshot_refs(struct repository *repo, struct tempfile *f);\n-int write_midx_included_packs(struct repack_write_midx_opts *opts);\n+int repack_write_midx(struct repack_write_midx_opts *opts);\n \n int write_filtered_pack(const struct write_pack_opts *opts,\n \t\t\tstruct existing_packs *existing,\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543667","messageId":"c8c846b1ac17caea4a4ee62d9150ab8014b96fb4.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 13/16] packfile: ensure `close_pack_revindex()` frees in-memory revindex","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:58:16Z","receivedAt":"2026-05-19T15:58:18Z","isPatch":true,"body":"The following commit will introduce a case where we write a MIDX bitmap\nover packs that do not themselves have on-disk *.rev files.\n\nThis case is supported within Git, and we will simply fall back to\ngenerating the revindex in memory. But we don't ever release that\nmemory, causing a leak that is exposed by a test introduced in the\nfollowing commit.\n\n(As far as I could find, we never free()'d memory allocated as a\nbyproduct of creating an in-memory revindex, likely because that code\npredates the leak-checking niceties we have in the test suite now.)\n\nRectify this by calling `FREE_AND_NULL()` on the `p->revindex` field\nwhen calling `close_pack_revindex()`.\n\nSigned-off-by: Taylor Blau \n---\n packfile.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/packfile.c b/packfile.c\nindex b012d648ada..a1e88fdb223 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -420,6 +420,8 @@ void close_pack_index(struct packed_git *p)\n \n static void close_pack_revindex(struct packed_git *p)\n {\n+\tFREE_AND_NULL(p->revindex);\n+\n \tif (!p->revindex_map)\n \t\treturn;\n \n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543668","messageId":"78f1a98c1ea14f7af051b2162d83ef655dc44a8b.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 14/16] repack: implement incremental MIDX repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:58:19Z","receivedAt":"2026-05-19T15:58:22Z","isPatch":true,"body":"Implement the `write_midx_incremental()` function, which builds and\nmaintains an incremental MIDX chain as part of the geometric repacking\nprocess.\n\nUnlike the default mode which writes a single flat MIDX, the incremental\nmode constructs a compaction plan that determines which MIDX layers to\nwrite, compact, or copy, and then executes each step using `git\nmulti-pack-index` subcommands with the --no-write-chain-file flag.\n\nThe repacking strategy works as follows:\n\n * Acquire the lock guarding the multi-pack-index-chain.\n\n * A new MIDX layer is always written containing the newly created\n pack(s). If the tip MIDX layer was rewritten during geometric\n repacking, any surviving packs from that layer are also included.\n\n * Starting from the new layer, adjacent MIDX layers are merged together\n as long as the accumulated object count exceeds half the object count\n of the next deeper layer (controlled by 'repack.midxSplitFactor').\n\n * Remaining layers in the chain are evaluated pairwise and either\n compacted or copied as-is, following the same merging condition.\n\n * Write the contents of the new multi-pack-index chain, atomically move\n it into place, and then release the lock.\n\n * Delete any now-unused MIDX layers.\n\nAfter writing the new layer, the strategy is evaluated among the\nexisting MIDX layers in order from oldest to newest. Each step that\nwrites a new MIDX layer uses \"--no-write-chain-file\" to avoid updating\nthe multi-pack-index-chain file. After all steps are complete, the new\nchain file is written and then atomically moved into place.\n\nAt present, this functionality is exposed behind a new enum value,\n`REPACK_WRITE_MIDX_INCREMENTAL`, but has no external callers. A\nsubsequent commit will expose this mode via `git repack\n--write-midx=incremental`.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 5 +\n repack-midx.c | 593 +++++++++++++++++++++++++++++++++++++++++++++--\n repack.h | 3 +\n 3 files changed, 588 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5d366340c34..75c57736780 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -42,6 +42,9 @@ static const char incremental_bitmap_conflict_error[] = N_(\n \"--no-write-bitmap-index or disable the pack.writeBitmaps configuration.\"\n );\n \n+#define DEFAULT_MIDX_SPLIT_FACTOR 2\n+#define DEFAULT_MIDX_NEW_LAYER_THRESHOLD 8\n+\n struct repack_config_ctx {\n \tstruct pack_objects_args *po_args;\n \tstruct pack_objects_args *cruft_po_args;\n@@ -555,6 +558,8 @@ int cmd_repack(int argc,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n+\t\t\t.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,\n+\t\t\t.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,\n \t\t\t.mode = write_midx,\n \t\t};\n \ndiff --git a/repack-midx.c b/repack-midx.c\nindex b1ca3797080..f97331fb1b7 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -2,12 +2,16 @@\n #include \"repack.h\"\n #include \"hash.h\"\n #include \"hex.h\"\n+#include \"lockfile.h\"\n+#include \"midx.h\"\n #include \"odb.h\"\n #include \"oidset.h\"\n #include \"pack-bitmap.h\"\n+#include \"path.h\"\n #include \"refs.h\"\n #include \"run-command.h\"\n #include \"tempfile.h\"\n+#include \"trace2.h\"\n \n struct midx_snapshot_ref_data {\n \tstruct repository *repo;\n@@ -293,26 +297,30 @@ static void repack_prepare_midx_command(struct child_process *cmd,\n }\n \n static int repack_fill_midx_stdin_packs(struct child_process *cmd,\n-\t\t\t\t\tstruct string_list *include)\n+\t\t\t\t\tstruct string_list *include,\n+\t\t\t\t\tstruct string_list *out)\n {\n+\tstruct strbuf in_buf = STRBUF_INIT;\n+\tstruct strbuf out_buf = STRBUF_INIT;\n \tstruct string_list_item *item;\n-\tFILE *in;\n \tint ret;\n \n-\tcmd->in = -1;\n-\n \tstrvec_push(&cmd->args, \"--stdin-packs\");\n \n-\tret = start_command(cmd);\n-\tif (ret)\n-\t\treturn ret;\n-\n-\tin = xfdopen(cmd->in, \"w\");\n \tfor_each_string_list_item(item, include)\n-\t\tfprintf(in, \"%s\\n\", item->string);\n-\tfclose(in);\n+\t\tstrbuf_addf(&in_buf, \"%s\\n\", item->string);\n \n-\treturn finish_command(cmd);\n+\tret = pipe_command(cmd, in_buf.buf, in_buf.len,\n+\t\t\t out ? &out_buf : NULL, 0, NULL, 0);\n+\n+\tif (out)\n+\t\tstring_list_split_f(out, out_buf.buf, \"\\n\", -1,\n+\t\t\t\t STRING_LIST_SPLIT_NONEMPTY);\n+\n+\tstrbuf_release(&in_buf);\n+\tstrbuf_release(&out_buf);\n+\n+\treturn ret;\n }\n \n static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n@@ -369,7 +377,7 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\",\n \t\t\t opts->refs_snapshot);\n \n-\tret = repack_fill_midx_stdin_packs(&cmd, &include);\n+\tret = repack_fill_midx_stdin_packs(&cmd, &include, NULL);\n done:\n \tif (!ret && opts->write_bitmaps)\n \t\tremove_redundant_bitmaps(&include, opts->packdir);\n@@ -379,6 +387,563 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)\n \treturn ret;\n }\n \n+struct midx_compaction_step {\n+\tunion {\n+\t\tstruct multi_pack_index *copy;\n+\t\tstruct string_list write;\n+\t\tstruct {\n+\t\t\tstruct multi_pack_index *from;\n+\t\t\tstruct multi_pack_index *to;\n+\t\t} compact;\n+\t} u;\n+\n+\tuint32_t objects_nr;\n+\tchar *csum;\n+\n+\tenum {\n+\t\tMIDX_COMPACTION_STEP_UNKNOWN,\n+\t\tMIDX_COMPACTION_STEP_COPY,\n+\t\tMIDX_COMPACTION_STEP_WRITE,\n+\t\tMIDX_COMPACTION_STEP_COMPACT,\n+\t} type;\n+};\n+\n+static const char *midx_compaction_step_base(const struct midx_compaction_step *step)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot use UNKNOWN step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_get_checksum_hex(step->u.copy);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\tBUG(\"cannot use WRITE step as a base\");\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_get_checksum_hex(step->u.compact.to);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static int midx_compaction_step_exec_copy(struct midx_compaction_step *step)\n+{\n+\tstep->csum = xstrdup(midx_get_checksum_hex(step->u.copy));\n+\treturn 0;\n+}\n+\n+static int midx_compaction_step_exec_write(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t\t const char *base)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list hash = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\tconst char *preferred_pack = NULL;\n+\tint ret = 0;\n+\n+\tif (!step->u.write.nr) {\n+\t\tret = error(_(\"no packs to write MIDX during compaction\"));\n+\t\tgoto out;\n+\t}\n+\n+\tfor_each_string_list_item(item, &step->u.write) {\n+\t\tif (item->util)\n+\t\t\tpreferred_pack = item->string;\n+\t}\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"write\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\", NULL);\n+\tstrvec_pushf(&cmd.args, \"--base=%s\", base ? base : \"none\");\n+\n+\tif (preferred_pack) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\n+\t\tstrbuf_addstr(&buf, preferred_pack);\n+\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\t\tstrbuf_addstr(&buf, \".pack\");\n+\n+\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\", buf.buf);\n+\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\tret = repack_fill_midx_stdin_packs(&cmd, &step->u.write, &hash);\n+\tif (hash.nr != 1) {\n+\t\tret = error(_(\"expected exactly one line during MIDX write, \"\n+\t\t\t \"got: %\"PRIuMAX),\n+\t\t\t (uintmax_t)hash.nr);\n+\t\tgoto out;\n+\t}\n+\n+\tstep->csum = xstrdup(hash.items[0].string);\n+\n+out:\n+\tstring_list_clear(&hash, 0);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec_compact(struct midx_compaction_step *step,\n+\t\t\t\t\t struct repack_write_midx_opts *opts)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tFILE *out = NULL;\n+\tint ret;\n+\n+\trepack_prepare_midx_command(&cmd, opts, \"compact\");\n+\tstrvec_pushl(&cmd.args, \"--incremental\", \"--no-write-chain-file\",\n+\t\t midx_get_checksum_hex(step->u.compact.from),\n+\t\t midx_get_checksum_hex(step->u.compact.to), NULL);\n+\n+\tcmd.out = -1;\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\tgoto out;\n+\n+\tout = xfdopen(cmd.out, \"r\");\n+\twhile (strbuf_getline_lf(&buf, out) != EOF) {\n+\t\tif (step->csum) {\n+\t\t\tret = error(_(\"unexpected MIDX output: '%s'\"), buf.buf);\n+\t\t\tfclose(out);\n+\t\t\tout = NULL;\n+\t\t\tfinish_command(&cmd);\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep->csum = strbuf_detach(&buf, NULL);\n+\t}\n+\n+\tret = finish_command(&cmd);\n+\n+out:\n+\tif (out)\n+\t\tfclose(out);\n+\tstrbuf_release(&buf);\n+\n+\treturn ret;\n+}\n+\n+static int midx_compaction_step_exec(struct midx_compaction_step *step,\n+\t\t\t\t struct repack_write_midx_opts *opts,\n+\t\t\t\t const char *base)\n+{\n+\tswitch (step->type) {\n+\tcase MIDX_COMPACTION_STEP_UNKNOWN:\n+\t\tBUG(\"cannot execute UNKNOWN midx compaction step\");\n+\tcase MIDX_COMPACTION_STEP_COPY:\n+\t\treturn midx_compaction_step_exec_copy(step);\n+\tcase MIDX_COMPACTION_STEP_WRITE:\n+\t\treturn midx_compaction_step_exec_write(step, opts, base);\n+\tcase MIDX_COMPACTION_STEP_COMPACT:\n+\t\treturn midx_compaction_step_exec_compact(step, opts);\n+\tdefault:\n+\t\tBUG(\"unhandled midx compaction step type %d\", step->type);\n+\t}\n+}\n+\n+static void midx_compaction_step_release(struct midx_compaction_step *step)\n+{\n+\tif (step->type == MIDX_COMPACTION_STEP_WRITE)\n+\t\tstring_list_clear(&step->u.write, 0);\n+\tfree(step->csum);\n+}\n+\n+static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step step = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\tuint32_t i;\n+\tint ret = 0;\n+\n+\ttrace2_region_enter(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tfor (i = 0; m && i < m->num_packs + m->num_packs_in_base; i++) {\n+\t\tif (prepare_midx_pack(m, i)) {\n+\t\t\tret = error(_(\"could not load pack %\"PRIu32\" from MIDX\"),\n+\t\t\t\t i);\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\ttrace2_region_enter(\"repack\", \"steps:write\", opts->existing->repo);\n+\n+\t/*\n+\t * The first MIDX in the resulting chain is always going to be\n+\t * new.\n+\t *\n+\t * At a minimum, it will include all of the newly written packs.\n+\t * If there is an existing MIDX whose tip layer contains packs\n+\t * that were repacked, it will also include any of its packs\n+\t * which were *not* rolled up as part of the geometric repack\n+\t * (if any), and the previous tip will be replaced.\n+\t *\n+\t * It may grow to include the packs from zero or more MIDXs from\n+\t * the old chain, beginning either at the old tip (if the MIDX\n+\t * was *not* rewritten) or the old tip's base MIDX layer\n+\t * (otherwise).\n+\t */\n+\tstep.type = MIDX_COMPACTION_STEP_WRITE;\n+\tstring_list_init_dup(&step.u.write);\n+\n+\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"pack-%s.idx\", opts->names->items[i].string);\n+\t\tstring_list_append(&step.u.write, buf.buf);\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:fresh\",\n+\t\t\t\t step.u.write.items[step.u.write.nr - 1].string);\n+\t}\n+\tfor (i = 0; i < opts->geometry->split; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:fresh:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n+\n+\t/*\n+\t * Now handle any existing packs which were *not* rewritten.\n+\t *\n+\t * The list of packs in opts->geometry only contains MIDX'd\n+\t * packs from the newest layer when that layer has more than\n+\t * 'repack.midxNewLayerThreshold' number of packs.\n+\t *\n+\t * If the MIDX tip was rewritten (that is, one or more of those\n+\t * packs appear below the split line), then add all packs above\n+\t * the split line to the new layer, as the old one is no longer\n+\t * usable.\n+\t *\n+\t * If the MIDX tip was not rewritten (that is, all MIDX'd packs\n+\t * from the youngest layer appear below the split line, or were\n+\t * not included in the geometric repack at all because there\n+\t * were too few of them), ignore them since we'll retain the\n+\t * existing layer as-is.\n+\t */\n+\tfor (i = opts->geometry->split; i < opts->geometry->pack_nr; i++) {\n+\t\tstruct packed_git *p = opts->geometry->pack[i];\n+\t\tstruct string_list_item *item;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\tif (p->multi_pack_index &&\n+\t\t !opts->geometry->midx_tip_rewritten) {\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"exclude:unmodified\", buf.buf);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified\", buf.buf);\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"include:unmodified:midx\",\n+\t\t\t\t p->multi_pack_index ? \"true\" : \"false\");\n+\n+\t\titem = string_list_append(&step.u.write, buf.buf);\n+\t\tif (p->multi_pack_index || i == opts->geometry->pack_nr - 1)\n+\t\t\titem->util = (void *)1; /* mark as preferred */\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, p->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstep.objects_nr += p->num_objects;\n+\t}\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"include:unmodified:objects_nr\",\n+\t\t\t (uintmax_t)step.objects_nr);\n+\n+\t/*\n+\t * If the MIDX tip was rewritten, then we no longer consider it\n+\t * a candidate for compaction, since it will not exist in the\n+\t * MIDX chain being built.\n+\t */\n+\tif (opts->geometry->midx_tip_rewritten)\n+\t\tm = m->base_midx;\n+\n+\ttrace2_data_string(\"repack\", opts->existing->repo, \"midx:rewrote-tip\",\n+\t\t\t opts->geometry->midx_tip_rewritten ? \"true\" : \"false\");\n+\n+\ttrace2_region_enter(\"repack\", \"compact\", opts->existing->repo);\n+\n+\t/*\n+\t * Compact additional MIDX layers into this proposed one until\n+\t * the merging condition is violated.\n+\t */\n+\twhile (m) {\n+\t\tuint32_t preferred_pack_idx;\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"candidate\", midx_get_checksum_hex(m));\n+\n+\t\tif (step.objects_nr < m->num_objects / opts->midx_split_factor) {\n+\t\t\t/*\n+\t\t\t * Stop compacting MIDX layer as soon as the\n+\t\t\t * merged size is less than half the size of the\n+\t\t\t * next layer in the chain.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"compact\", \"violated\");\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"next_objects_nr\",\n+\t\t\t\t\t (uintmax_t)m->num_objects);\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"split_factor\",\n+\t\t\t\t\t (uintmax_t)opts->midx_split_factor);\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (midx_preferred_pack(m, &preferred_pack_idx) < 0) {\n+\t\t\tret = error(_(\"could not find preferred pack for MIDX \"\n+\t\t\t\t \"%s\"), midx_get_checksum_hex(m));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\t\tstruct string_list_item *item;\n+\t\t\tuint32_t pack_int_id = i + m->num_packs_in_base;\n+\t\t\tstruct packed_git *p = nth_midxed_pack(m, pack_int_id);\n+\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"midx:pack\", buf.buf);\n+\n+\t\t\titem = string_list_append(&step.u.write, buf.buf);\n+\t\t\tif (pack_int_id == preferred_pack_idx)\n+\t\t\t\titem->util = (void *)1; /* mark as preferred */\n+\t\t}\n+\n+\t\tif (unsigned_add_overflows(step.objects_nr, m->num_objects)) {\n+\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\tgoto out;\n+\t\t}\n+\t\tstep.objects_nr += m->num_objects;\n+\n+\t\tm = m->base_midx;\n+\t}\n+\n+\tif (step.u.write.nr > 0) {\n+\t\t/*\n+\t\t * As long as there is at least one new pack to write\n+\t\t * (and thus the MIDX is non-empty), add it to the plan.\n+\t\t */\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\t\tsteps[steps_nr++] = step;\n+\t}\n+\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:objects_nr\", (uintmax_t)step.objects_nr);\n+\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t \"step:packs_nr\", (uintmax_t)step.u.write.nr);\n+\n+\ttrace2_region_leave(\"repack\", \"compact\", opts->existing->repo);\n+\ttrace2_region_leave(\"repack\", \"steps:write\", opts->existing->repo);\n+\n+\ttrace2_region_enter(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n+\t/*\n+\t * Then start over, repeat, and either compact or keep as-is\n+\t * each MIDX layer until we have exhausted the chain.\n+\t *\n+\t * Finally, evaluate the remainder of the chain (if any) and\n+\t * either compact a sequence of adjacent layers, or keep\n+\t * individual layers as-is according to the same merging\n+\t * condition as above.\n+\t */\n+\twhile (m) {\n+\t\tstruct multi_pack_index *next = m;\n+\n+\t\tALLOC_GROW(steps, steps_nr + 1, steps_alloc);\n+\n+\t\tmemset(&step, 0, sizeof(step));\n+\t\tstep.type = MIDX_COMPACTION_STEP_UNKNOWN;\n+\n+\t\ttrace2_region_enter(\"repack\", \"step\", opts->existing->repo);\n+\n+\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t \"from\", midx_get_checksum_hex(m));\n+\n+\t\twhile (next) {\n+\t\t\tuint32_t proposed_objects_nr;\n+\t\t\tif (unsigned_add_overflows(step.objects_nr, next->num_objects)) {\n+\t\t\t\tret = error(_(\"too many objects in MIDX compaction step\"));\n+\t\t\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tproposed_objects_nr = step.objects_nr + next->num_objects;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed\",\n+\t\t\t\t\t midx_get_checksum_hex(next));\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"proposed:objects_nr\",\n+\t\t\t\t\t (uintmax_t)next->num_objects);\n+\n+\t\t\tif (!next->base_midx) {\n+\t\t\t\t/*\n+\t\t\t\t * If we are at the end of the MIDX\n+\t\t\t\t * chain, there is nothing to compact,\n+\t\t\t\t * so mark it and stop.\n+\t\t\t\t */\n+\t\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (proposed_objects_nr < next->base_midx->num_objects / opts->midx_split_factor) {\n+\t\t\t\t/*\n+\t\t\t\t * If there is a MIDX following this\n+\t\t\t\t * one, but our accumulated size is less\n+\t\t\t\t * than half of its size, compacting\n+\t\t\t\t * them would violate the merging\n+\t\t\t\t * condition, so stop here.\n+\t\t\t\t */\n+\n+\t\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at\",\n+\t\t\t\t\t\t midx_get_checksum_hex(next->base_midx));\n+\t\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t\t \"compact:violated:at:objects_nr\",\n+\t\t\t\t\t\t (uintmax_t)next->base_midx->num_objects);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * Otherwise, it is OK to compact the next layer\n+\t\t\t * into this one. Do so, and then continue\n+\t\t\t * through the remainder of the chain.\n+\t\t\t */\n+\t\t\tstep.objects_nr = proposed_objects_nr;\n+\t\t\ttrace2_data_intmax(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"step:objects_nr\",\n+\t\t\t\t\t (uintmax_t)step.objects_nr);\n+\t\t\tnext = next->base_midx;\n+\t\t}\n+\n+\t\tif (m == next) {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COPY;\n+\t\t\tstep.u.copy = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"copy\");\n+\t\t} else {\n+\t\t\tstep.type = MIDX_COMPACTION_STEP_COMPACT;\n+\t\t\tstep.u.compact.from = next;\n+\t\t\tstep.u.compact.to = m;\n+\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"to\", midx_get_checksum_hex(m));\n+\t\t\ttrace2_data_string(\"repack\", opts->existing->repo,\n+\t\t\t\t\t \"type\", \"compact\");\n+\t\t}\n+\n+\t\tm = next->base_midx;\n+\t\tsteps[steps_nr++] = step;\n+\t\ttrace2_region_leave(\"repack\", \"step\", opts->existing->repo);\n+\t}\n+\n+\ttrace2_region_leave(\"repack\", \"steps:rest\", opts->existing->repo);\n+\n+out:\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+\n+\tstrbuf_release(&buf);\n+\n+\ttrace2_region_leave(\"repack\", \"make_midx_compaction_plan\",\n+\t\t\t opts->existing->repo);\n+\n+\treturn ret;\n+}\n+\n+static int write_midx_incremental(struct repack_write_midx_opts *opts)\n+{\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct strbuf lock_name = STRBUF_INIT;\n+\tstruct lock_file lf;\n+\tsize_t steps_nr = 0;\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tget_midx_chain_filename(opts->existing->source, &lock_name);\n+\tif (safe_create_leading_directories(opts->existing->repo,\n+\t\t\t\t\t lock_name.buf))\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t lock_name.buf);\n+\thold_lock_file_for_update(&lf, lock_name.buf, LOCK_DIE_ON_ERROR);\n+\n+\tif (!fdopen_lock_file(&lf, \"w\")) {\n+\t\tret = error_errno(_(\"unable to open multi-pack-index chain file\"));\n+\t\tgoto done;\n+\t}\n+\n+\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < steps_nr; i++) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tchar *base = NULL;\n+\n+\t\tif (i + 1 < steps_nr)\n+\t\t\tbase = xstrdup(midx_compaction_step_base(&steps[i + 1]));\n+\n+\t\tif (midx_compaction_step_exec(step, opts, base) < 0) {\n+\t\t\tret = error(_(\"unable to execute compaction step %\"PRIuMAX),\n+\t\t\t\t (uintmax_t)i);\n+\t\t\tfree(base);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfree(base);\n+\t}\n+\n+\ti = steps_nr;\n+\twhile (i--) {\n+\t\tstruct midx_compaction_step *step = &steps[i];\n+\t\tif (!step->csum)\n+\t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n+\t\t\t (uintmax_t)i);\n+\t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t}\n+\n+\tcommit_lock_file(&lf);\n+\n+done:\n+\tstrbuf_release(&lock_name);\n+\tfor (i = 0; i < steps_nr; i++)\n+\t\tmidx_compaction_step_release(&steps[i]);\n+\tfree(steps);\n+\treturn ret;\n+}\n+\n int repack_write_midx(struct repack_write_midx_opts *opts)\n {\n \tswitch (opts->mode) {\n@@ -386,6 +951,8 @@ int repack_write_midx(struct repack_write_midx_opts *opts)\n \t\tBUG(\"write_midx mode is NONE?\");\n \tcase REPACK_WRITE_MIDX_DEFAULT:\n \t\treturn write_midx_included_packs(opts);\n+\tcase REPACK_WRITE_MIDX_INCREMENTAL:\n+\t\treturn write_midx_incremental(opts);\n \tdefault:\n \t\tBUG(\"unhandled write_midx mode: %d\", opts->mode);\n \t}\ndiff --git a/repack.h b/repack.h\nindex 81907fcce7f..831ccfb1c6c 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -137,6 +137,7 @@ struct tempfile;\n enum repack_write_midx_mode {\n \tREPACK_WRITE_MIDX_NONE,\n \tREPACK_WRITE_MIDX_DEFAULT,\n+\tREPACK_WRITE_MIDX_INCREMENTAL,\n };\n \n struct repack_write_midx_opts {\n@@ -148,6 +149,8 @@ struct repack_write_midx_opts {\n \tint show_progress;\n \tint write_bitmaps;\n \tint midx_must_contain_cruft;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n \tenum repack_write_midx_mode mode;\n };\n \n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543669","messageId":"95bf8b21fff6a29a09bdb784323b794cdd2777c3.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 15/16] repack: introduce `--write-midx=incremental`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:58:22Z","receivedAt":"2026-05-19T15:58:27Z","isPatch":true,"body":"Expose the incremental MIDX repacking mode (implemented in an earlier\ncommit) via a new --write-midx=incremental option for `git repack`.\n\nAdd \"incremental\" as a recognized argument to the --write-midx\nOPT_CALLBACK, mapping it to REPACK_WRITE_MIDX_INCREMENTAL. When this\nmode is active and --geometric is in use, set the midx_layer_threshold\non the pack geometry so that only packs in sufficiently large tip layers\nare considered for repacking.\n\nTwo new configuration options control the compaction behavior:\n\n - repack.midxSplitFactor (default: 2): the factor used in the\n geometric merging condition for MIDX layers.\n\n - repack.midxNewLayerThreshold (default: 8): the minimum number of\n packs in the tip MIDX layer before its packs are considered as\n candidates for geometric repacking.\n\nAdd tests exercising the new mode across a variety of scenarios\nincluding basic geometric violations, multi-round chain integrity,\nbranching and merging histories, cross-layer object uniqueness, and\nthreshold-based compaction.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/repack.adoc | 18 ++\n Documentation/git-repack.adoc | 39 ++-\n builtin/repack.c | 49 ++-\n midx.c | 29 ++\n midx.h | 3 +\n repack-geometry.c | 13 +-\n repack-midx.c | 5 +\n repack.c | 56 +++-\n repack.h | 10 +-\n t/meson.build | 1 +\n t/t7705-repack-incremental-midx.sh | 470 +++++++++++++++++++++++++++++\n 11 files changed, 669 insertions(+), 24 deletions(-)\n create mode 100755 t/t7705-repack-incremental-midx.sh\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex e9e78dcb198..4c22a499f62 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -46,3 +46,21 @@ repack.midxMustContainCruft::\n \t`--write-midx`. When false, cruft packs are only included in the MIDX\n \twhen necessary (e.g., because they might be required to form a\n \treachability closure with MIDX bitmaps). Defaults to true.\n+\n+repack.midxSplitFactor::\n+\tThe factor used in the geometric merging condition when\n+\tcompacting incremental MIDX layers during `git repack` when\n+\tinvoked with the `--write-midx=incremental` option.\n++\n+Adjacent layers are merged when the accumulated object count of the\n+newer layer exceeds `1/` of the object count of the next deeper\n+layer. Must be at least 2. Defaults to 2.\n+\n+repack.midxNewLayerThreshold::\n+\tThe minimum number of packs in the tip MIDX layer before those\n+\tpacks are considered as candidates for geometric repacking\n+\tduring `git repack --write-midx=incremental`.\n++\n+When the tip layer has fewer packs than this threshold, those packs are\n+excluded from the geometric repack entirely, and are thus left\n+unmodified. Must be at least 1. Defaults to 8.\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 673ce910837..27a99cc46f4 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -11,7 +11,7 @@ SYNOPSIS\n [verse]\n 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n \t[--window=] [--depth=] [--threads=] [--keep-pack=]\n-\t[--write-midx] [--name-hash-version=] [--path-walk]\n+\t[--write-midx[=]] [--name-hash-version=] [--path-walk]\n \n DESCRIPTION\n -----------\n@@ -250,9 +250,42 @@ pack as the preferred pack for object selection by the MIDX (see\n linkgit:git-multi-pack-index[1]).\n \n -m::\n---write-midx::\n+--write-midx[=]::\n \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n-\tcontaining the non-redundant packs.\n+\tcontaining the non-redundant packs. The following modes are\n+\tavailable:\n++\n+--\n+\t`default`;;\n+\t\tWrite a single MIDX covering all packs. This is the\n+\t\tdefault when `--write-midx` is given without an\n+\t\texplicit mode.\n+\n+\t`incremental`;;\n+\t\tWrite an incremental MIDX chain instead of a single\n+\t\tflat MIDX. This mode requires `--geometric`.\n++\n+The incremental mode maintains a chain of MIDX layers that is compacted\n+over time using a geometric merging strategy. Each repack creates a new\n+tip layer containing the newly written pack(s). Adjacent layers are then\n+merged whenever the newer layer's object count exceeds\n+`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n+that do not meet this condition are retained as-is.\n++\n+The result is that newer (tip) layers tend to contain many small packs\n+with relatively few objects, while older (deeper) layers contain fewer,\n+larger packs covering more objects. Because compaction is driven by the\n+tip of the chain, newer layers are also rewritten more frequently than\n+older ones, which are only touched when enough objects have accumulated\n+to justify merging into them. This keeps the total number of layers\n+logarithmic relative to the total number of objects.\n++\n+Only packs in the tip MIDX layer are considered as candidates for the\n+geometric repack; packs in deeper layers are left untouched. If the tip\n+layer contains fewer packs than `repack.midxNewLayerThreshold`, those\n+packs are excluded from the geometry entirely, and a new layer is\n+created for any new pack(s) without disturbing the existing chain.\n+--\n \n --name-hash-version=::\n \tProvide this argument to the underlying `git pack-objects` process.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 75c57736780..5ffa18e085e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -33,7 +33,7 @@ static int midx_must_contain_cruft = 1;\n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n \t \"[--window=] [--depth=] [--threads=] [--keep-pack=]\\n\"\n-\t \"[--write-midx] [--name-hash-version=] [--path-walk]\"),\n+\t \"[--write-midx[=]] [--name-hash-version=] [--path-walk]\"),\n \tNULL\n };\n \n@@ -48,6 +48,8 @@ static const char incremental_bitmap_conflict_error[] = N_(\n struct repack_config_ctx {\n \tstruct pack_objects_args *po_args;\n \tstruct pack_objects_args *cruft_po_args;\n+\tint midx_split_factor;\n+\tint midx_new_layer_threshold;\n };\n \n static int repack_config(const char *var, const char *value,\n@@ -97,6 +99,16 @@ static int repack_config(const char *var, const char *value,\n \t\tmidx_must_contain_cruft = git_config_bool(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"repack.midxsplitfactor\")) {\n+\t\trepack_ctx->midx_split_factor = git_config_int(var, value,\n+\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(var, \"repack.midxnewlayerthreshold\")) {\n+\t\trepack_ctx->midx_new_layer_threshold = git_config_int(var, value,\n+\t\t\t\t\t\t\t\t ctx->kvi);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -112,6 +124,8 @@ static int option_parse_write_midx(const struct option *opt, const char *arg,\n \n \tif (!arg || !*arg)\n \t\t*cfg = REPACK_WRITE_MIDX_DEFAULT;\n+\telse if (!strcmp(arg, \"incremental\"))\n+\t\t*cfg = REPACK_WRITE_MIDX_INCREMENTAL;\n \telse\n \t\treturn error(_(\"unknown value for %s: %s\"), opt->long_name, arg);\n \n@@ -226,6 +240,8 @@ int cmd_repack(int argc,\n \tmemset(&config_ctx, 0, sizeof(config_ctx));\n \tconfig_ctx.po_args = &po_args;\n \tconfig_ctx.cruft_po_args = &cruft_po_args;\n+\tconfig_ctx.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR;\n+\tconfig_ctx.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD;\n \n \trepo_config(repo, repack_config, &config_ctx);\n \n@@ -247,6 +263,9 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n+\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n+\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n+\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n@@ -273,6 +292,13 @@ int cmd_repack(int argc,\n \t\twrite_bitmaps = 0;\n \t}\n \n+\tif (config_ctx.midx_split_factor < 2)\n+\t\tdie(_(\"invalid value for %s: %d\"), \"--midx-split-factor\",\n+\t\t config_ctx.midx_split_factor);\n+\tif (config_ctx.midx_new_layer_threshold < 1)\n+\t\tdie(_(\"invalid value for %s: %d\"), \"--midx-new-layer-threshold\",\n+\t\t config_ctx.midx_new_layer_threshold);\n+\n \tif (write_midx != REPACK_WRITE_MIDX_NONE && write_bitmaps) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \n@@ -296,6 +322,10 @@ int cmd_repack(int argc,\n \tif (geometry.split_factor) {\n \t\tif (pack_everything)\n \t\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--geometric\", \"-A/-a\");\n+\t\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL) {\n+\t\t\tgeometry.midx_layer_threshold = config_ctx.midx_new_layer_threshold;\n+\t\t\tgeometry.midx_layer_threshold_set = true;\n+\t\t}\n \t\tpack_geometry_init(&geometry, &existing, &po_args);\n \t\tpack_geometry_split(&geometry);\n \t}\n@@ -545,8 +575,11 @@ int cmd_repack(int argc,\n \t\t\t\t packtmp);\n \t/* End of pack replacement. */\n \n-\tif (delete_redundant && pack_everything & ALL_INTO_ONE)\n+\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n+\t\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL)\n+\t\t\texisting_packs_retain_midx_packs(&existing);\n \t\texisting_packs_mark_for_deletion(&existing, &names);\n+\t}\n \n \tif (write_midx != REPACK_WRITE_MIDX_NONE) {\n \t\tstruct repack_write_midx_opts opts = {\n@@ -558,8 +591,8 @@ int cmd_repack(int argc,\n \t\t\t.show_progress = show_progress,\n \t\t\t.write_bitmaps = write_bitmaps > 0,\n \t\t\t.midx_must_contain_cruft = midx_must_contain_cruft,\n-\t\t\t.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,\n-\t\t\t.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,\n+\t\t\t.midx_split_factor = config_ctx.midx_split_factor,\n+\t\t\t.midx_new_layer_threshold = config_ctx.midx_new_layer_threshold,\n \t\t\t.mode = write_midx,\n \t\t};\n \n@@ -572,11 +605,15 @@ int cmd_repack(int argc,\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\n-\t\texisting_packs_remove_redundant(&existing, packdir);\n+\t\tbool wrote_incremental_midx = write_midx == REPACK_WRITE_MIDX_INCREMENTAL;\n+\n+\t\texisting_packs_remove_redundant(&existing, packdir,\n+\t\t\t\t\t\twrote_incremental_midx);\n \n \t\tif (geometry.split_factor)\n \t\t\tpack_geometry_remove_redundant(&geometry, &names,\n-\t\t\t\t\t\t &existing, packdir);\n+\t\t\t\t\t\t &existing, packdir,\n+\t\t\t\t\t\t wrote_incremental_midx);\n \t\tif (show_progress)\n \t\t\topts |= PRUNE_PACKED_VERBOSE;\n \t\tprune_packed_objects(opts);\ndiff --git a/midx.c b/midx.c\nindex dc86c8e7fee..efbfbb13f41 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -850,6 +850,35 @@ void clear_midx_file(struct repository *r)\n \tstrbuf_release(&midx);\n }\n \n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes)\n+{\n+\tstruct odb_source *source = r->objects->sources;\n+\tstruct strbuf chain = STRBUF_INIT;\n+\n+\tget_midx_chain_filename(source, &chain);\n+\n+\tfor (; source; source = source->next) {\n+\t\tstruct odb_source_files *files = odb_source_files_downcast(source);\n+\t\tif (files->packed->midx)\n+\t\t\tclose_midx(files->packed->midx);\n+\t\tfiles->packed->midx = NULL;\n+\t}\n+\n+\tif (!keep_hashes && remove_path(chain.buf))\n+\t\tdie(_(\"failed to clear multi-pack-index chain at %s\"),\n+\t\t chain.buf);\n+\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_BITMAP,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_REV,\n+\t\t\t\t\t keep_hashes);\n+\tclear_incremental_midx_files_ext(r->objects->sources, MIDX_EXT_MIDX,\n+\t\t\t\t\t keep_hashes);\n+\n+\tstrbuf_release(&chain);\n+}\n+\n static int verify_midx_error;\n \n __attribute__((format (printf, 1, 2)))\ndiff --git a/midx.h b/midx.h\nindex 3ee12dd08ec..63853a03a47 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -9,6 +9,7 @@ struct repository;\n struct bitmapped_pack;\n struct git_hash_algo;\n struct odb_source;\n+struct strvec;\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION_V1 1\n@@ -143,6 +144,8 @@ int write_midx_file_compact(struct odb_source *source,\n \t\t\t const char *incremental_base,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n+void clear_incremental_midx_files(struct repository *r,\n+\t\t\t\t const struct strvec *keep_hashes);\n int verify_midx_file(struct odb_source *source, unsigned flags);\n int expire_midx_packs(struct odb_source *source, unsigned flags);\n int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags);\ndiff --git a/repack-geometry.c b/repack-geometry.c\nindex 2408b8a3cc2..2064683dcfe 100644\n--- a/repack-geometry.c\n+++ b/repack-geometry.c\n@@ -249,7 +249,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t\t\t uint32_t pack_nr,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tconst struct git_hash_algo *algop = existing->repo->hash_algo;\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -269,7 +270,8 @@ static void remove_redundant_packs(struct packed_git **pack,\n \t\t (string_list_has_string(&existing->kept_packs, buf.buf)))\n \t\t\tcontinue;\n \n-\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf);\n+\t\trepack_remove_redundant_pack(existing->repo, packdir, buf.buf,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n \n \tstrbuf_release(&buf);\n@@ -278,12 +280,13 @@ static void remove_redundant_packs(struct packed_git **pack,\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs(geometry->pack, geometry->split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n \tremove_redundant_packs(geometry->promisor_pack, geometry->promisor_split,\n-\t\t\t names, existing, packdir);\n+\t\t\t names, existing, packdir, wrote_incremental_midx);\n }\n \n void pack_geometry_release(struct pack_geometry *geometry)\ndiff --git a/repack-midx.c b/repack-midx.c\nindex f97331fb1b7..4f5deeb97bf 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -887,6 +887,7 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \tstruct midx_compaction_step *steps = NULL;\n \tstruct strbuf lock_name = STRBUF_INIT;\n \tstruct lock_file lf;\n+\tstruct strvec keep_hashes = STRVEC_INIT;\n \tsize_t steps_nr = 0;\n \tsize_t i;\n \tint ret = 0;\n@@ -932,11 +933,15 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\t\tBUG(\"missing result for compaction step %\"PRIuMAX,\n \t\t\t (uintmax_t)i);\n \t\tfprintf(get_lock_file_fp(&lf), \"%s\\n\", step->csum);\n+\t\tstrvec_push(&keep_hashes, step->csum);\n \t}\n \n \tcommit_lock_file(&lf);\n \n+\tclear_incremental_midx_files(opts->existing->repo, &keep_hashes);\n+\n done:\n+\tstrvec_clear(&keep_hashes);\n \tstrbuf_release(&lock_name);\n \tfor (i = 0; i < steps_nr; i++)\n \t\tmidx_compaction_step_release(&steps[i]);\ndiff --git a/repack.c b/repack.c\nindex 2ee6b51420a..571dabb665e 100644\n--- a/repack.c\n+++ b/repack.c\n@@ -55,14 +55,18 @@ void pack_objects_args_release(struct pack_objects_args *args)\n }\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name)\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct odb_source *source = repo->objects->sources;\n \tstruct multi_pack_index *m = get_multi_pack_index(source);\n \tstrbuf_addf(&buf, \"%s.pack\", base_name);\n-\tif (m && source->local && midx_contains_pack(m, buf.buf))\n+\tif (m && source->local && midx_contains_pack(m, buf.buf)) {\n \t\tclear_midx_file(repo);\n+\t\tif (!wrote_incremental_midx)\n+\t\t\tclear_incremental_midx_files(repo, NULL);\n+\t}\n \tstrbuf_insertf(&buf, 0, \"%s/\", dir_name);\n \tunlink_pack_path(buf.buf, 1);\n \tstrbuf_release(&buf);\n@@ -250,25 +254,63 @@ void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \t\t\t\t\t &existing->cruft_packs);\n }\n \n+/*\n+ * Mark every pack that is referenced by the existing MIDX chain as\n+ * retained, so that a subsequent call to\n+ * existing_packs_mark_for_deletion() will not mark them for deletion.\n+ *\n+ * This is used when writing an incremental MIDX layer on top of an\n+ * existing chain: retained layers continue to reference the same\n+ * packs on disk, so those packs must not be unlinked even if the\n+ * freshly-written pack supersedes them.\n+ */\n+void existing_packs_retain_midx_packs(struct existing_packs *existing)\n+{\n+\tstruct string_list_item *item;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tfor_each_string_list_item(item, &existing->midx_packs) {\n+\t\tstruct string_list_item *found;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addstr(&buf, item->string);\n+\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\tstrbuf_strip_suffix(&buf, \".idx\");\n+\n+\t\tfound = string_list_lookup(&existing->non_kept_packs, buf.buf);\n+\t\tif (found)\n+\t\t\texisting_packs_mark_retained(found);\n+\n+\t\tfound = string_list_lookup(&existing->cruft_packs, buf.buf);\n+\t\tif (found)\n+\t\t\texisting_packs_mark_retained(found);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+}\n+\n static void remove_redundant_packs_1(struct repository *repo,\n \t\t\t\t struct string_list *packs,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tstruct string_list_item *item;\n \tfor_each_string_list_item(item, packs) {\n \t\tif (!existing_pack_is_marked_for_deletion(item))\n \t\t\tcontinue;\n-\t\trepack_remove_redundant_pack(repo, packdir, item->string);\n+\t\trepack_remove_redundant_pack(repo, packdir, item->string,\n+\t\t\t\t\t wrote_incremental_midx);\n \t}\n }\n \n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir)\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx)\n {\n \tremove_redundant_packs_1(existing->repo, &existing->non_kept_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n \tremove_redundant_packs_1(existing->repo, &existing->cruft_packs,\n-\t\t\t\t packdir);\n+\t\t\t\t packdir, wrote_incremental_midx);\n }\n \n void existing_packs_release(struct existing_packs *existing)\ndiff --git a/repack.h b/repack.h\nindex 831ccfb1c6c..f9fbc895f02 100644\n--- a/repack.h\n+++ b/repack.h\n@@ -34,7 +34,8 @@ void prepare_pack_objects(struct child_process *cmd,\n void pack_objects_args_release(struct pack_objects_args *args);\n \n void repack_remove_redundant_pack(struct repository *repo, const char *dir_name,\n-\t\t\t\t const char *base_name);\n+\t\t\t\t const char *base_name,\n+\t\t\t\t bool wrote_incremental_midx);\n \n struct write_pack_opts {\n \tstruct pack_objects_args *po_args;\n@@ -83,8 +84,10 @@ void existing_packs_retain_cruft(struct existing_packs *existing,\n \t\t\t\t struct packed_git *cruft);\n void existing_packs_mark_for_deletion(struct existing_packs *existing,\n \t\t\t\t struct string_list *names);\n+void existing_packs_retain_midx_packs(struct existing_packs *existing);\n void existing_packs_remove_redundant(struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void existing_packs_release(struct existing_packs *existing);\n \n struct generated_pack;\n@@ -129,7 +132,8 @@ struct packed_git *pack_geometry_preferred_pack(struct pack_geometry *geometry);\n void pack_geometry_remove_redundant(struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t struct existing_packs *existing,\n-\t\t\t\t const char *packdir);\n+\t\t\t\t const char *packdir,\n+\t\t\t\t bool wrote_incremental_midx);\n void pack_geometry_release(struct pack_geometry *geometry);\n \n struct tempfile;\ndiff --git a/t/meson.build b/t/meson.build\nindex 7528e5cda5f..25f0d823d8e 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -951,6 +951,7 @@ integration_tests = [\n 't7702-repack-cyclic-alternate.sh',\n 't7703-repack-geometric.sh',\n 't7704-repack-cruft.sh',\n+ 't7705-repack-incremental-midx.sh',\n 't7800-difftool.sh',\n 't7810-grep.sh',\n 't7811-grep-open.sh',\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nnew file mode 100755\nindex 00000000000..9e317ff6e8f\n--- /dev/null\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -0,0 +1,470 @@\n+#!/bin/sh\n+\n+test_description='git repack --write-midx=incremental'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+midxdir=$packdir/multi-pack-index.d\n+midx_chain=$midxdir/multi-pack-index-chain\n+\n+# incrementally_repack N\n+#\n+# Make \"N\" new commits, each stored in their own pack, and then repacked\n+# with the --write-midx=incremental strategy.\n+incrementally_repack () {\n+\tfor i in $(test_seq 1 \"$1\")\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify || return 1\n+\tdone\n+}\n+\n+# Create packs with geometrically increasing sizes so that they\n+# satisfy the geometric progression and survive a --geometric=2\n+# repack without being rolled up. Creates 3 packs containing 1,\n+# 2, and 6 commits (3, 6, and 18 objects) respectively.\n+create_geometric_packs () {\n+\ttest_commit \"small\" &&\n+\tgit repack -d &&\n+\n+\ttest_commit_bulk --message=\"medium\" 2 &&\n+\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t--write-bitmap-index\n+}\n+\n+# create_layer \n+#\n+# Creates a new MIDX layer with the contents of \"test_commit_bulk $@\".\n+create_layer () {\n+\ttest_commit_bulk \"$@\" &&\n+\n+\tgit multi-pack-index write --incremental --bitmap\n+}\n+\n+# create_layers\n+#\n+# Reads lines of \" \" from stdin and creates a new MIDX\n+# layer for each line. See create_layer above for more.\n+create_layers () {\n+\twhile read msg nr\n+\tdo\n+\t\tcreate_layer --message=\"$msg\" \"$nr\" || return 1\n+\tdone\n+}\n+\n+test_expect_success '--write-midx=incremental requires --geometric' '\n+\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+\n+\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+'\n+\n+test_expect_success 'below layer threshold, tip packs excluded' '\n+\tgit init below-layer-threshold-tip-packs-excluded &&\n+\t(\n+\t\tcd below-layer-threshold-tip-packs-excluded &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 4 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create 3 packs forming a geometric progression by\n+\t\t# object count such that they are unmodified by the\n+\t\t# initial repack. The MIDX chain thusly contains a\n+\t\t# single layer with three packs.\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a new commit. Since the layer threshold is\n+\t\t# unmet, a new MIDX layer is added on top of the\n+\t\t# existing one.\n+\t\ttest_commit extra &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\t\tgit multi-pack-index verify &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs repacked' '\n+\tgit init above-layer-threshold-tip-packs-repacked &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-repacked &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Same setup, but with the layer threshold set to 2.\n+\t\t# Since the tip MIDX layer meets that threshold, its\n+\t\t# packs are considered repack candidates.\n+\t\tcreate_geometric_packs &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Perturb the existing progression such that it is\n+\t\t# rolled up into a single new pack, invalidating the\n+\t\t# existing MIDX layer and replacing it with a new one.\n+\t\ttest_commit extra &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip layer preserved' '\n+\tgit init above-layer-threshold-tip-layer-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-layer-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\ttest_commit_bulk --message=\"medium\" 2 &&\n+\t\ttest_commit_bulk --message=\"large\" 6 &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Create objects to form a pack satisfying the geometric\n+\t\t# progression (thus preserving the tip layer), but not\n+\t\t# so large that it meets the layer merging condition.\n+\t\ttest_commit_bulk --message=\"small\" 1 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\ttest_line_count = 1 packs.new &&\n+\t\ttest_line_count = 3 packs.after &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\thead -n 1 \"$midx_chain.before\" >expect &&\n+\t\thead -n 1 \"$midx_chain\" >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'above layer threshold, tip packs preserved' '\n+\tgit init above-layer-threshold-tip-packs-preserved &&\n+\t(\n+\t\tcd above-layer-threshold-tip-packs-preserved &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 2 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_geometric_packs &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Same setup as above, but this time the new objects do\n+\t\t# not satisfy the new layer merging condition, resulting\n+\t\t# in a new tip layer.\n+\t\ttest_commit_bulk --message=\"huge\" 18 &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'new tip absorbs multiple layers' '\n+\tgit init new-tip-absorbs-multiple-layers &&\n+\t(\n+\t\tcd new-tip-absorbs-multiple-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a 4-layer chain where each layer is too small to\n+\t\t# absorb the one below it. The sizes must satisfy L(n) <\n+\t\t# L(n-1)/2 for each adjacent pair:\n+\t\t#\n+\t\t# L0 (oldest): 75 obj (25 commits)\n+\t\t# L1: 21 obj (7 commits, 21 < 75/2)\n+\t\t# L2: 9 obj (3 commits, 9 < 21/2)\n+\t\t# L3 (tip): 3 obj (1 commit, 3 < 9/2)\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tL0 25\n+\t\tL1 7\n+\t\tL2 3\n+\t\tL3 1\n+\t\tEOF\n+\n+\t\ttest_line_count = 4 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Now add a new commit. The merging condition is\n+\t\t# satisfied between L3-L1, but violated at L0, which is\n+\t\t# too large relative to the accumulated size.\n+\t\t#\n+\t\t# As a result, the chain shrinks from 4 to 2 layers.\n+\t\ttest_commit new &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'compaction of older layers' '\n+\tgit init compaction-of-older-layers &&\n+\t(\n+\t\tcd compaction-of-older-layers &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Build a chain with two small layers at the bottom\n+\t\t# and a larger barrier layer on top, producing a\n+\t\t# chain that violates the compaction invariant, since\n+\t\t# the two small layers would normally have been merged.\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 2\n+\t\ttwo 4\n+\t\tbarrier 54\n+\t\tEOF\n+\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running an incremental repack compacts the two\n+\t\t# small layers at the bottom of the chain as a\n+\t\t# separate step in the compaction plan.\n+\t\ttest_commit another &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_line_count = 2 \"$midx_chain\" &&\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'geometric rollup with surviving tip packs' '\n+\tgit init geometric-rollup-with-surviving-tip-packs &&\n+\t(\n+\t\tcd geometric-rollup-with-surviving-tip-packs &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create a pack large enough to anchor the geometric\n+\t\t# progression when small packs are added alongside it.\n+\t\tcreate_layer --message=\"big\" 5 &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Repack a small number of objects such that the\n+\t\t# progression is unbothered. Note that the existing pack\n+\t\t# is considered a repack candidate as the new layer\n+\t\t# threshold is set to 1.\n+\t\ttest_commit small-1 &&\n+\t\tgit repack -d &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\t! test_cmp $midx_chain.before $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before\n+\t)\n+'\n+\n+test_expect_success 'kept packs are excluded from repack' '\n+\tgit init kept-packs-excluded-from-repack &&\n+\t(\n+\t\tcd kept-packs-excluded-from-repack &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\t# Create two equal-sized packs, marking one as kept.\n+\t\tfor i in A B\n+\t\tdo\n+\t\t\ttest_commit \"$i\" && git repack -d || return 1\n+\t\tdone &&\n+\n+\t\tkeep=$(ls $packdir/pack-*.idx | head -n 1) &&\n+\t\ttouch \"${keep%.idx}.keep\" &&\n+\n+\t\t# The kept pack is excluded as a repacking candidate\n+\t\t# entirely, so no rollup occurs as there is only one\n+\t\t# non-kept pack. A new MIDX layer is written containing\n+\t\t# that pack.\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count = 1 actual.packs &&\n+\t\ttest_grep ! \"$keep\" actual.packs &&\n+\n+\t\tgit multi-pack-index verify &&\n+\n+\t\t# All objects (from both kept and non-kept packs)\n+\t\t# must still be accessible.\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'incremental MIDX with --max-pack-size' '\n+\tgit init incremental-midx-with--max-pack-size &&\n+\t(\n+\t\tcd incremental-midx-with--max-pack-size &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\t# Now add enough data that a small --max-pack-size will\n+\t\t# cause pack-objects to split its output. Create objects\n+\t\t# large enough to fill multiple packs.\n+\t\ttest-tool genrandom foo 1M >big1 &&\n+\t\ttest-tool genrandom bar 1M >big2 &&\n+\t\tgit add big1 big2 &&\n+\t\ttest_tick &&\n+\t\tgit commit -a -m \"big blobs\" &&\n+\t\tgit repack -d &&\n+\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index --max-pack-size=1M &&\n+\n+\t\ttest_line_count = 1 \"$midx_chain\" &&\n+\t\ttest-tool read-midx $objdir >actual &&\n+\t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n+\t\ttest_line_count -gt 1 actual.packs &&\n+\n+\t\tgit multi-pack-index verify\n+\t)\n+'\n+\n+test_expect_success 'noop repack preserves valid MIDX chain' '\n+\tgit init noop-repack-preserves-valid-midx-chain &&\n+\t(\n+\t\tcd noop-repack-preserves-valid-midx-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layer --message=\"base\" 1 &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# Running again with no new objects should not break\n+\t\t# the MIDX chain. It produces \"Nothing new to pack.\"\n+\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n+\t\t\t--write-bitmap-index &&\n+\n+\t\ttest_cmp $midx_chain.before $midx_chain &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck\n+\t)\n+'\n+\n+test_expect_success 'repack -ad removes stale incremental chain' '\n+\tgit init repack--ad-removes-stale-incremental-chain &&\n+\t(\n+\t\tcd repack--ad-removes-stale-incremental-chain &&\n+\n+\t\tgit config maintenance.auto false &&\n+\t\tgit config repack.midxnewlayerthreshold 1 &&\n+\t\tgit config repack.midxsplitfactor 2 &&\n+\n+\t\tcreate_layers <<-\\EOF &&\n+\t\tone 1\n+\t\ttwo 1\n+\t\tEOF\n+\n+\t\ttest_path_is_file $midx_chain &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\n+\t\tgit repack -ad &&\n+\n+\t\ttest_path_is_missing $packdir/multi-pack-index &&\n+\t\ttest_dir_is_empty $midxdir\n+\t)\n+'\n+\n+test_expect_success 'repack rejects invalid midxSplitFactor' '\n+\ttest_when_finished \"rm -fr bad-split-factor\" &&\n+\tgit init bad-split-factor &&\n+\t(\n+\t\tcd bad-split-factor &&\n+\t\ttest_commit base &&\n+\n+\t\tfor v in 0 1 -1\n+\t\tdo\n+\t\t\ttest_must_fail git -c repack.midxSplitFactor=$v \\\n+\t\t\t\trepack -d --geometric=2 --write-midx=incremental 2>err &&\n+\t\t\ttest_grep \"invalid value for --midx-split-factor\" err ||\n+\t\t\treturn 1\n+\t\tdone\n+\t)\n+'\n+\n+test_expect_success 'repack rejects invalid midxNewLayerThreshold' '\n+\ttest_when_finished \"rm -fr bad-layer-threshold\" &&\n+\tgit init bad-layer-threshold &&\n+\t(\n+\t\tcd bad-layer-threshold &&\n+\t\ttest_commit base &&\n+\n+\t\tfor v in 0 -1\n+\t\tdo\n+\t\t\ttest_must_fail git -c repack.midxNewLayerThreshold=$v \\\n+\t\t\t\trepack -d --geometric=2 --write-midx=incremental 2>err &&\n+\t\t\ttest_grep \"invalid value for --midx-new-layer-threshold\" err ||\n+\t\t\treturn 1\n+\t\tdone\n+\t)\n+'\n+\n+test_done\n-- \n2.54.0.175.g8bd0ec98dc3\n\n"},{"id":"543670","messageId":"8bd0ec98dc31e2ece72d01c7efe6f1508729e3fd.1779206240.git.me@ttaylorr.com","threadId":"65383","inReplyTo":"cover.1779206239.git.me@ttaylorr.com","subject":"[PATCH v4 16/16] repack: allow `--write-midx=incremental` without `--geometric`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-05-19T15:58:25Z","receivedAt":"2026-05-19T15:58:29Z","isPatch":true,"body":"Previously, `--write-midx=incremental` required `--geometric` and would\ndie() without it. Relax this restriction so that incremental MIDX\nrepacking can be used independently.\n\nWithout `--geometric`, the behavior is append-only: a single new MIDX\nlayer is created containing whatever packs were written by the repack\nand appended to the existing chain (or a new chain is started). Existing\nlayers are preserved as-is with no compaction or merging.\n\nImplement this via a new repack_make_midx_append_plan() that builds a\nplan consisting of a WRITE step for the freshly written packs followed\nby COPY steps for every existing MIDX layer. The existing compaction\nplan (repack_make_midx_compaction_plan) is used only when `--geometric`\nis active.\n\nUpdate the documentation to describe the behavior with and without\n`--geometric`, and replace the test that enforced the old restriction\nwith one exercising append-only incremental MIDX repacking.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-repack.adoc | 19 +++++----\n builtin/repack.c | 3 --\n repack-midx.c | 64 +++++++++++++++++++++++++++--\n t/t7705-repack-incremental-midx.sh | 65 +++++++++++++++++++++++++++---\n 4 files changed, 133 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 27a99cc46f4..72c42015e23 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -263,14 +263,19 @@ linkgit:git-multi-pack-index[1]).\n \n \t`incremental`;;\n \t\tWrite an incremental MIDX chain instead of a single\n-\t\tflat MIDX. This mode requires `--geometric`.\n+\t\tflat MIDX.\n +\n-The incremental mode maintains a chain of MIDX layers that is compacted\n-over time using a geometric merging strategy. Each repack creates a new\n-tip layer containing the newly written pack(s). Adjacent layers are then\n-merged whenever the newer layer's object count exceeds\n-`1/repack.midxSplitFactor` of the next deeper layer's count. Layers\n-that do not meet this condition are retained as-is.\n+Without `--geometric`, a new MIDX layer is appended to the existing\n+chain (or a new chain is started) containing whatever packs were written\n+by the repack. Existing layers are preserved as-is.\n++\n+When combined with `--geometric`, the incremental mode maintains a chain\n+of MIDX layers that is compacted over time using a geometric merging\n+strategy. Each repack creates a new tip layer containing the newly\n+written pack(s). Adjacent layers are then merged whenever the newer\n+layer's object count exceeds `1/repack.midxSplitFactor` of the next\n+deeper layer's count. Layers that do not meet this condition are\n+retained as-is.\n +\n The result is that newer (tip) layers tend to contain many small packs\n with relatively few objects, while older (deeper) layers contain fewer,\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5ffa18e085e..1524a9c13ad 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -263,9 +263,6 @@ int cmd_repack(int argc,\n \tif (pack_everything & PACK_CRUFT)\n \t\tpack_everything |= ALL_INTO_ONE;\n \n-\tif (write_midx == REPACK_WRITE_MIDX_INCREMENTAL && !geometry.split_factor)\n-\t\tdie(_(\"--write-midx=incremental requires --geometric\"));\n-\n \tif (write_bitmaps < 0) {\n \t\tif (write_midx == REPACK_WRITE_MIDX_NONE &&\n \t\t (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\ndiff --git a/repack-midx.c b/repack-midx.c\nindex 4f5deeb97bf..b6b1de71805 100644\n--- a/repack-midx.c\n+++ b/repack-midx.c\n@@ -548,6 +548,60 @@ static void midx_compaction_step_release(struct midx_compaction_step *step)\n \tfree(step->csum);\n }\n \n+/*\n+ * Build an append-only MIDX plan: a single WRITE step for the freshly\n+ * written packs, plus COPY steps for every existing layer. No\n+ * compaction or merging is performed.\n+ */\n+static void repack_make_midx_append_plan(struct repack_write_midx_opts *opts,\n+\t\t\t\t\t struct midx_compaction_step **steps_p,\n+\t\t\t\t\t size_t *steps_nr_p)\n+{\n+\tstruct multi_pack_index *m;\n+\tstruct midx_compaction_step *steps = NULL;\n+\tstruct midx_compaction_step *step;\n+\tsize_t steps_nr = 0, steps_alloc = 0;\n+\n+\todb_reprepare(opts->existing->repo->objects);\n+\tm = get_multi_pack_index(opts->existing->source);\n+\n+\tif (opts->names->nr) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tuint32_t i;\n+\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_WRITE;\n+\t\tstring_list_init_dup(&step->u.write);\n+\n+\t\tfor (i = 0; i < opts->names->nr; i++) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"pack-%s.idx\",\n+\t\t\t\t opts->names->items[i].string);\n+\t\t\tstring_list_append(&step->u.write, buf.buf);\n+\t\t}\n+\n+\t\tstrbuf_release(&buf);\n+\t}\n+\n+\tfor (; m; m = m->base_midx) {\n+\t\tALLOC_GROW(steps, st_add(steps_nr, 1), steps_alloc);\n+\n+\t\tstep = &steps[steps_nr++];\n+\t\tmemset(step, 0, sizeof(*step));\n+\n+\t\tstep->type = MIDX_COMPACTION_STEP_COPY;\n+\t\tstep->u.copy = m;\n+\t\tstep->objects_nr = m->num_objects;\n+\t}\n+\n+\t*steps_p = steps;\n+\t*steps_nr_p = steps_nr;\n+}\n+\n static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,\n \t\t\t\t\t struct midx_compaction_step **steps_p,\n \t\t\t\t\t size_t *steps_nr_p)\n@@ -904,9 +958,13 @@ static int write_midx_incremental(struct repack_write_midx_opts *opts)\n \t\tgoto done;\n \t}\n \n-\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n-\t\tret = error(_(\"unable to generate compaction plan\"));\n-\t\tgoto done;\n+\tif (opts->geometry->split_factor) {\n+\t\tif (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {\n+\t\t\tret = error(_(\"unable to generate compaction plan\"));\n+\t\t\tgoto done;\n+\t\t}\n+\t} else {\n+\t\trepack_make_midx_append_plan(opts, &steps, &steps_nr);\n \t}\n \n \tfor (i = 0; i < steps_nr; i++) {\ndiff --git a/t/t7705-repack-incremental-midx.sh b/t/t7705-repack-incremental-midx.sh\nindex 9e317ff6e8f..25a8c40e8ee 100755\n--- a/t/t7705-repack-incremental-midx.sh\n+++ b/t/t7705-repack-incremental-midx.sh\n@@ -63,10 +63,36 @@ create_layers () {\n \tdone\n }\n \n-test_expect_success '--write-midx=incremental requires --geometric' '\n-\ttest_must_fail git repack --write-midx=incremental 2>err &&\n+test_expect_success '--write-midx=incremental without --geometric' '\n+\tgit init incremental-without-geometric &&\n+\t(\n+\t\tcd incremental-without-geometric &&\n \n-\ttest_grep -- \"--write-midx=incremental requires --geometric\" err\n+\t\tgit config maintenance.auto false &&\n+\n+\t\ttest_commit first &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_commit second &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\t\tcp $midx_chain $midx_chain.before &&\n+\n+\t\t# A second repack appends a new layer without\n+\t\t# disturbing the existing one.\n+\t\ttest_commit third &&\n+\t\tgit repack --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 2 $midx_chain &&\n+\t\thead -n 1 $midx_chain.before >expect &&\n+\t\thead -n 1 $midx_chain >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit fsck\n+\t)\n '\n \n test_expect_success 'below layer threshold, tip packs excluded' '\n@@ -334,8 +360,7 @@ test_expect_success 'kept packs are excluded from repack' '\n \t\t# entirely, so no rollup occurs as there is only one\n \t\t# non-kept pack. A new MIDX layer is written containing\n \t\t# that pack.\n-\t\tgit repack --geometric=2 -d --write-midx=incremental \\\n-\t\t\t--write-bitmap-index &&\n+\t\tgit repack --geometric=2 -d --write-midx=incremental &&\n \n \t\ttest-tool read-midx $objdir >actual &&\n \t\tgrep \"^pack-.*\\.idx$\" actual >actual.packs &&\n@@ -433,6 +458,36 @@ test_expect_success 'repack -ad removes stale incremental chain' '\n \t)\n '\n \n+test_expect_success 'repack -ad --write-midx=incremental is safe' '\n+\tgit init ad-incremental-midx &&\n+\t(\n+\t\tcd ad-incremental-midx &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\t# Build a MIDX chain with multiple layers referencing\n+\t\t# distinct packs.\n+\t\ttest_commit first &&\n+\t\tgit repack -d &&\n+\n+\t\ttest_commit second &&\n+\t\tgit repack -d --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\ttest_line_count = 1 $midx_chain &&\n+\n+\t\t# Now do a full -ad repack. The new pack contains all\n+\t\t# objects, but any retained MIDX layers still reference\n+\t\t# the now-deleted packs.\n+\t\ttest_commit third &&\n+\t\tgit repack -ad --write-midx=incremental &&\n+\n+\t\tgit multi-pack-index verify &&\n+\t\tgit fsck &&\n+\t\tgit rev-list --all --objects >/dev/null\n+\t)\n+'\n+\n test_expect_success 'repack rejects invalid midxSplitFactor' '\n \ttest_when_finished \"rm -fr bad-split-factor\" &&\n \tgit init bad-split-factor &&\n-- \n2.54.0.175.g8bd0ec98dc3\n"},{"id":"545329","messageId":"aiuaf3fKJ6kIITrf@szeder.dev","threadId":"65383","inReplyTo":"1bbb387d6b6204045d97882fd8775dbff12dedbb.1779206240.git.me@ttaylorr.com","subject":"Re: [PATCH v4 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2026-06-12T05:34:55Z","receivedAt":"2026-06-12T05:35:18Z","isPatch":true,"body":"On Tue, May 19, 2026 at 11:57:54AM -0400, Taylor Blau wrote:\n> diff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh\n> index 66d6894761b..68a103d13d2 100755\n> --- a/t/t5334-incremental-multi-pack-index.sh\n> +++ b/t/t5334-incremental-multi-pack-index.sh\n> @@ -113,6 +113,36 @@ test_expect_success 'write non-incremental MIDX layer with --no-write-chain-file\n> \ttest_grep \"cannot use --no-write-chain-file without --incremental\" err\n> '\n> \n> +test_expect_success 'write MIDX layer with --base without --no-write-chain-file' '\n> +\ttest_must_fail git multi-pack-index write --bitmap --incremental \\\n> +\t\t--base=none 2>err &&\n> +\ttest_grep \"cannot use --base without --no-write-chain-file\" err\n> +'\n> +\n> +test_expect_success 'write MIDX layer with --base=none and --no-write-chain-file' '\n> +\ttest_commit base-none &&\n> +\tgit repack -d &&\n> +\n> +\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n> +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n> +\t\t--no-write-chain-file --base=none)\" &&\n> +\n> +\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n> +\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n> +'\n> +\n> +test_expect_success 'write MIDX layer with --base= and --no-write-chain-file' '\n> +\ttest_commit base-hash &&\n> +\tgit repack -d &&\n> +\n> +\tcp \"$midx_chain\" \"$midx_chain.bak\" &&\n> +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n> +\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n\nThere is no 'nth_line' helper function in this test script.\n\n> +\n> +\ttest_cmp \"$midx_chain.bak\" \"$midx_chain\" &&\n> +\ttest_path_is_file \"$midxdir/multi-pack-index-$layer.midx\"\n> +'\n> +\n> for reuse in false single multi\n> do\n> \ttest_expect_success \"full clone (pack.allowPackReuse=$reuse)\" '\n"},{"id":"545373","messageId":"xmqqqzmbj3mb.fsf@gitster.g","threadId":"65383","inReplyTo":"aiuaf3fKJ6kIITrf@szeder.dev","subject":"Re: [PATCH v4 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-06-12T13:21:48Z","receivedAt":"2026-06-12T13:21:52Z","isPatch":true,"body":"SZEDER Gábor writes:\n\n>> +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n>> +\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n>\n> There is no 'nth_line' helper function in this test script.\n\nGood eyes. It has been there in the file next door t5335 since\nFebruary, but not available here in t5334.\n\n"},{"id":"545406","messageId":"aixNXOxfPZnAVLgK@nand.local","threadId":"65383","inReplyTo":"xmqqqzmbj3mb.fsf@gitster.g","subject":"Re: [PATCH v4 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-06-12T18:18:04Z","receivedAt":"2026-06-12T18:18:06Z","isPatch":true,"body":"On Fri, Jun 12, 2026 at 06:21:48AM -0700, Junio C Hamano wrote:\n> SZEDER Gábor writes:\n>\n> >> +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n> >> +\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n> >\n> > There is no 'nth_line' helper function in this test script.\n>\n> Good eyes. It has been there in the file next door t5335 since\n> February, but not available here in t5334.\n\nGood spotting indeed. Fortunately or unfortunately for us, pulling on\nthis thread revealed a bit of a rabbit hole. Patches forthcoming..\n\nThanks,\nTaylor\n"},{"id":"545418","messageId":"aixnJJuDLOo/FsGt@nand.local","threadId":"65383","inReplyTo":"aixNXOxfPZnAVLgK@nand.local","subject":"Re: [PATCH v4 06/16] midx: support custom `--base` for incremental MIDX writes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-06-12T20:08:04Z","receivedAt":"2026-06-12T20:08:07Z","isPatch":true,"body":"On Fri, Jun 12, 2026 at 02:18:04PM -0400, Taylor Blau wrote:\n> On Fri, Jun 12, 2026 at 06:21:48AM -0700, Junio C Hamano wrote:\n> > SZEDER Gábor writes:\n> >\n> > >> +\tlayer=\"$(git multi-pack-index write --bitmap --incremental \\\n> > >> +\t\t--no-write-chain-file --base=\"$(nth_line 1 \"$midx_chain\")\")\" &&\n> > >\n> > > There is no 'nth_line' helper function in this test script.\n> >\n> > Good eyes. It has been there in the file next door t5335 since\n> > February, but not available here in t5334.\n>\n> Good spotting indeed. Fortunately or unfortunately for us, pulling on\n> this thread revealed a bit of a rabbit hole. Patches forthcoming..\n\n https://lore.kernel.org/git/cover.1781294771.git.me@ttaylorr.com/\n\nThanks,\nTaylor\n"}]}