{"thread":{"id":"64585","subject":"[PATCH 01/17] midx: mark `get_midx_checksum()` arguments as const","startedAt":"2025-12-06T20:31:00Z","lastAt":"2026-02-24T19:08:11Z","messageCount":99,"participants":["Taylor Blau","Patrick Steinhardt","Junio C Hamano","rsbecker@nexbridge.com","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":17},"messages":[{"id":"531765","messageId":"cover.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":null,"subject":"[PATCH 00/17] midx: incremental MIDX/bitmap layer compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:30:57Z","receivedAt":"2025-12-06T20:31:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Note to the maintainer:\n\n * This series is the spiritual successor to my earlier series queued\n as 'tb/incremental-midx-part-3.1', but is based on 'master'\n (currently f0ef5b6d9bc (The fifth batch, 2025-11-30) at the time of\n writing). I suggest queueing as 'tb/incremental-midx-part-3.2'.\n\nOVERVIEW\n========\n\nThis series implements the second of three major components for an\nincremental MIDX/bitmap-based repacking strategy. Those components\nare:\n\n 1. Refactoring code out of builtin/repack.c into individual\n compilation units outside of the builtin directory.\n\n 2. (this series) Implementing MIDX layer compaction, i.e. the ability\n to take an contiguous sequence of MIDX layers in an existing MIDX\n chain and replace them with a single layer representing the union\n of objects/packs among the compacted layers.\n\n 3. An incremental repacking strategy that, unlike our current\n '--geometric' approach, does not require periodic all-into-one\n repacks.\n\nBACKGROUND\n==========\n\nI'll briefly discuss that repacking strategy here in order to provide\nsome context on why MIDX layer compaction is important. When doing a\n--geometric repack today, any update to the MIDX and/or its .bitmap\nrequires a full rewrite. For most repositories this is not such a big\ndeal, but in large monorepos these costs can add up significantly over\ntime.\n\nTo deal with this, incremental MIDXs (and corresponding support for\nreachability bitmaps) were introduced as a way to append updates to\nthe end of a MIDX chain without having to rewrite anything.\n\nOf course, growing a MIDX chain infinitely without ever shrinking it\nwould be pointless. So any repacking strategy which leverages\nincremental MIDXs must take some action which encourages the MIDX\nchain to avoid growing too large.\n\nThe strategy that I will share in the forthcoming topic related to\nthis effort does (roughly) the following:\n\n 1. Repack all non-MIDX'd packs geometrically, optionally including\n any packs in the most recent MIDX layer iff it has more than some\n threshold of packs.\n\n 2. Write out a new MIDX layer which is the result of the previous\n step, adding it onto the end of the chain (or replacing the\n previous end, depending on whether the pack threshold condition\n was met).\n\n 3. Perform a compaction pass, combining adjacent MIDX layers to\n restore a geometric progression on object count among MIDX layers\n in the chain.\n\n(As an aside, this description is not the most efficient way to\nimplement this behavior, and so the actual implementation varies\nslightly, though achieves the same effect.)\n\nThis strategy encourages MIDX chains where older layers have fewer,\nlarger packs, and newer layers have a greater number of smaller packs.\nThe compaction pass prevents the MIDX chain from ever growing too\nlarge. (In my simulation[1], even a large number of simulated pushes\nyields a MIDX chain which is consistently fewer than ~10 or so layers\nlong.)\n\nCOMPACTION\n==========\n\nMIDX compaction works by producing a new MIDX layer which is the\nlogical union of any contiguous sequence within the existing MIDX\nchain. Importantly, the packs/objects from those layers within the\ncompaction range have their order preserved. That means that the\npseudo-pack order of the compacted layer is guaranteed to be the same\nas the concatenated pseudo-pack order across all layers in the\ncompaction range.\n\nAs a result, we can reuse any selected bitmaps within the compaction\nrange verbatim. More importantly, any bitmaps belonging to layers\nwhich are descendants of the compacted one do not need to permute any\nexisting bitmaps in order for them to be valid post-compaction.\n\nThis allows us to cheaply combine parts of the MIDX chain without\nneeding to do any work outside of the combine layers, which is\nessential in the above-described repacking strategy.\n\nTHIS SERIES\n===========\n\nThis series implements MIDX compaction, and is organized roughly as\nfollows:\n\n * The first two patches are stray code clean-up and minor refactoring\n changes that I made while writing this series.\n\n * The next three patches clean up the SYNOPSIS for in the manual page\n for git-multi-pack-index(1), and removes the builtin from the list\n of known failures in t0450.\n\n * The next patch is a typofix, and the one following that is a\n preemptive bugfix.\n\n * The eighth patch is a code refactoring patch that makes the\n argument list for midx-write.c::write_midx_internal() easier to\n reason about.\n\n * The ninth patch relaxes a MIDX constraint that forces all packs\n within a layer to be sorted lexicographically.\n\n * The remaining patches prepare for and introduce MIDX compaction.\n\nThe test coverage in the new t5335-compact-multi-pack-index.sh is\nfairly sparse in my opinion, though most of the interesting behavior\nis already exercised via t5319 and t5334. If others have ideas on how\nto improve test coverage here, please let me know!\n\nThis series is somewhat lengthy, though the individual changes are\nrelatively small for the most part, so I am hoping that they aren't\ntoo bad to review.\n\nThanks in advance for your review!\n\n[1]: https://gist.github.com/ttaylorr/7ac0305f130197744f3583096c8198a1\n\nTaylor Blau (17):\n midx: mark `get_midx_checksum()` arguments as const\n midx: split `get_midx_checksum()` by adding `get_midx_hash()`\n builtin/multi-pack-index.c: make '--progress' a common option\n git-multi-pack-index(1): remove non-existent incompatibility\n git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'\n t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39\n midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`\n midx-write.c: introduce `struct write_midx_opts`\n midx: do not require packs to be sorted in lexicographic order\n git-compat-util.h: introduce `u32_add()`\n midx-write.c: introduce `midx_pack_perm()` helper\n midx-write.c: extract `fill_pack_from_midx()`\n midx-write.c: enumerate `pack_int_id` values directly\n midx-write.c: factor fanout layering from `compute_sorted_entries()`\n t/helper/test-read-midx.c: plug memory leak when selecting layer\n midx: implement MIDX compaction\n midx: enable reachability bitmaps during MIDX compaction\n\n Documentation/git-multi-pack-index.adoc | 24 +-\n builtin/multi-pack-index.c | 84 ++++-\n git-compat-util.h | 8 +\n midx-write.c | 463 ++++++++++++++++++------\n midx.c | 36 +-\n midx.h | 9 +-\n pack-bitmap.c | 9 +-\n pack-revindex.c | 4 +-\n t/helper/test-read-midx.c | 21 +-\n t/meson.build | 1 +\n t/t0450/adoc-help-mismatches | 1 -\n t/t5319-multi-pack-index.sh | 7 +-\n t/t5335-compact-multi-pack-index.sh | 218 +++++++++++\n 13 files changed, 737 insertions(+), 148 deletions(-)\n create mode 100755 t/t5335-compact-multi-pack-index.sh\n\n\nbase-commit: f0ef5b6d9bcc258e4cbef93839d1b7465d5212b9\n-- \n2.52.0.171.gd6a4e6b6955\n"},{"id":"531764","messageId":"8cd8f6e8420a52030f92166d5a20387183cdbb6b.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 01/17] midx: mark `get_midx_checksum()` arguments as const","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:01Z","receivedAt":"2025-12-06T20:31:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To make clear that the fucntion `get_midx_checksum()` does not do\nanything to modify its argument, mark the MIDX pointer as const.\n\nThe following commit will rename this function altogether to make clear\nthat it returns the raw bytes of the checksum, not a hex-encoded copy of\nit.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 2 +-\n midx.h | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 24e1e721754..6c01f0fa522 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -24,7 +24,7 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n-const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+const unsigned char *get_midx_checksum(const struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\n }\ndiff --git a/midx.h b/midx.h\nindex 6e54d73503d..7c7e0b59121 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -85,7 +85,7 @@ struct multi_pack_index {\n #define MIDX_EXT_BITMAP \"bitmap\"\n #define MIDX_EXT_MIDX \"midx\"\n \n-const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n+const unsigned char *get_midx_checksum(const struct multi_pack_index *m);\n void get_midx_filename(struct odb_source *source, struct strbuf *out);\n void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\n \t\t\t const unsigned char *hash, const char *ext);\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531766","messageId":"d583a169f08eea9d750622c6e3cf79774a2a68e7.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 02/17] midx: split `get_midx_checksum()` by adding `get_midx_hash()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:04Z","receivedAt":"2025-12-06T20:31:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When trying to print out, say, the hexadecimal representation of a\nMIDX's hash, our code will do something like:\n\n hash_to_hex_algop(get_midx_checksum(m),\n m->source->odb->repo->hash_algo);\n\n, which is both cumbersome and repetitive. In fact, all but a handful of\ncallers to `get_midx_checksum()` do exactly the above. Reduce the\nrepetitive nature of calling `get_midx_checksum()` by having it return a\npointer into a static buffer containing the above result.\n\nFor the handful of callers that do need to compare the raw bytes and\ndon't want to deal with an encoded copy (e.g., because they are passing\nit to hasheq() or similar), introduce `get_midx_hash()` which returns\nthe raw bytes.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 8 +++-----\n midx.c | 8 +++++++-\n midx.h | 3 ++-\n pack-bitmap.c | 9 ++++-----\n pack-revindex.c | 4 ++--\n t/helper/test-read-midx.c | 4 ++--\n 6 files changed, 20 insertions(+), 16 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 23e61cb0001..73d24fabbc6 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -955,7 +955,7 @@ static int link_midx_to_chain(struct multi_pack_index *m)\n \t}\n \n \tfor (i = 0; i < ARRAY_SIZE(midx_exts); i++) {\n-\t\tconst unsigned char *hash = get_midx_checksum(m);\n+\t\tconst unsigned char *hash = get_midx_hash(m);\n \n \t\tget_midx_filename_ext(m->source, &from,\n \t\t\t\t hash, midx_exts[i].non_split);\n@@ -1086,8 +1086,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\twhile (m) {\n \t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n-\t\t\t\t hash_to_hex_algop(get_midx_checksum(m),\n-\t\t\t\t\t\t\tm->source->odb->repo->hash_algo));\n+\t\t\t\t get_midx_checksum(m));\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \t\t\tctx.num_multi_pack_indexes_before++;\n@@ -1445,8 +1444,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n \t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n \n-\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(get_midx_checksum(m),\n-\t\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes[j] = xstrdup(get_midx_checksum(m));\n \t\t\tm = m->base_midx;\n \t\t}\n \ndiff --git a/midx.c b/midx.c\nindex 6c01f0fa522..f9b11de9ca9 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -24,7 +24,13 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n-const unsigned char *get_midx_checksum(const struct multi_pack_index *m)\n+const char *get_midx_checksum(const struct multi_pack_index *m)\n+{\n+\treturn hash_to_hex_algop(get_midx_hash(m),\n+\t\t\t\t m->source->odb->repo->hash_algo);\n+}\n+\n+const unsigned char *get_midx_hash(const struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\n }\ndiff --git a/midx.h b/midx.h\nindex 7c7e0b59121..e188ffeb578 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -85,7 +85,8 @@ struct multi_pack_index {\n #define MIDX_EXT_BITMAP \"bitmap\"\n #define MIDX_EXT_MIDX \"midx\"\n \n-const unsigned char *get_midx_checksum(const struct multi_pack_index *m);\n+const char *get_midx_checksum(const struct multi_pack_index *m) /* static buffer */;\n+const unsigned char *get_midx_hash(const struct multi_pack_index *m);\n void get_midx_filename(struct odb_source *source, struct strbuf *out);\n void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\n \t\t\t const unsigned char *hash, const char *ext);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 8ca79725b1d..f466ed2ddcb 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -441,11 +441,11 @@ char *midx_bitmap_filename(struct multi_pack_index *midx)\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (midx->has_chain)\n \t\tget_split_midx_filename_ext(midx->source, &buf,\n-\t\t\t\t\t get_midx_checksum(midx),\n+\t\t\t\t\t get_midx_hash(midx),\n \t\t\t\t\t MIDX_EXT_BITMAP);\n \telse\n \t\tget_midx_filename_ext(midx->source, &buf,\n-\t\t\t\t get_midx_checksum(midx),\n+\t\t\t\t get_midx_hash(midx),\n \t\t\t\t MIDX_EXT_BITMAP);\n \n \treturn strbuf_detach(&buf, NULL);\n@@ -502,7 +502,7 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \tif (load_bitmap_header(bitmap_git) < 0)\n \t\tgoto cleanup;\n \n-\tif (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum,\n+\tif (!hasheq(get_midx_hash(bitmap_git->midx), bitmap_git->checksum,\n \t\t bitmap_repo(bitmap_git)->hash_algo)) {\n \t\terror(_(\"checksum doesn't match in MIDX and bitmap\"));\n \t\tgoto cleanup;\n@@ -2820,8 +2820,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \t\tif (bitmap_is_midx(found))\n \t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n-\t\t\t\t hash_to_hex_algop(get_midx_checksum(found->midx),\n-\t\t\t\t\t\t revs->repo->hash_algo));\n+\t\t\t\t get_midx_checksum(found->midx));\n \t\telse\n \t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n \t\t\t\t hash_to_hex_algop(found->pack->hash,\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex d0791cc4938..016195ceb93 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -390,11 +390,11 @@ int load_midx_revindex(struct multi_pack_index *m)\n \n \tif (m->has_chain)\n \t\tget_split_midx_filename_ext(m->source, &revindex_name,\n-\t\t\t\t\t get_midx_checksum(m),\n+\t\t\t\t\t get_midx_hash(m),\n \t\t\t\t\t MIDX_EXT_REV);\n \telse\n \t\tget_midx_filename_ext(m->source, &revindex_name,\n-\t\t\t\t get_midx_checksum(m),\n+\t\t\t\t get_midx_hash(m),\n \t\t\t\t MIDX_EXT_REV);\n \n \tret = load_revindex_from_disk(m->source->odb->repo->hash_algo,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 6de5d1665af..dee603b3cd0 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -34,7 +34,7 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\treturn 1;\n \n \tif (checksum) {\n-\t\twhile (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum))\n+\t\twhile (m && strcmp(get_midx_checksum(m), checksum))\n \t\t\tm = m->base_midx;\n \t\tif (!m)\n \t\t\treturn 1;\n@@ -94,7 +94,7 @@ static int read_midx_checksum(const char *object_dir)\n \tm = setup_midx(object_dir);\n \tif (!m)\n \t\treturn 1;\n-\tprintf(\"%s\\n\", hash_to_hex(get_midx_checksum(m)));\n+\tprintf(\"%s\\n\", get_midx_checksum(m));\n \n \tclose_midx(m);\n \treturn 0;\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531767","messageId":"35377d47c36b4ea28428cfa562dd7b1e2cb00656.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 03/17] builtin/multi-pack-index.c: make '--progress' a common option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:07Z","receivedAt":"2025-12-06T20:31:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"All multi-pack-index sub-commands (write, verify, repack, and expire)\nsupport a '--progress' command-line option, despite not listing it as\none of the common options in `common_opts`.\n\nAs a result each sub-command declares its own `OPT_BIT()` for a\n\"--progress\" command-line option. Centralize this within the\n`common_opts` to avoid re-declaring it in each sub-command.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 2 ++\n builtin/multi-pack-index.c | 10 ++--------\n 2 files changed, 4 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 2f642697e9e..a4550e28bed 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -18,6 +18,8 @@ Write or verify a multi-pack-index (MIDX) file.\n OPTIONS\n -------\n \n+The following command-line options are applicable to all sub-commands:\n+\n --object-dir=::\n \tUse given directory for the location of Git objects. We check\n \t`/packs/multi-pack-index` for the current MIDX file, and\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5f364aa816b..ca98d4c3ba3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -84,6 +84,8 @@ static struct option common_opts[] = {\n \t N_(\"directory\"),\n \t N_(\"object directory containing set of packfile and pack-index pairs\"),\n \t parse_object_dir),\n+\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"),\n+\t\tMIDX_PROGRESS),\n \tOPT_END(),\n };\n \n@@ -138,8 +140,6 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n@@ -200,8 +200,6 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv,\n {\n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_verify_options[] = {\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_END(),\n \t};\n \tstruct odb_source *source;\n@@ -231,8 +229,6 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv,\n {\n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_expire_options[] = {\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_END(),\n \t};\n \tstruct odb_source *source;\n@@ -264,8 +260,6 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_repack_options[] = {\n \t\tOPT_UNSIGNED(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t N_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_END(),\n \t};\n \tstruct odb_source *source;\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531768","messageId":"66411e25bb49546500f10a3d3734a2277ddbd633.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 04/17] git-multi-pack-index(1): remove non-existent incompatibility","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:10Z","receivedAt":"2025-12-06T20:31:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since fcb2205b774 (midx: implement support for writing incremental MIDX\nchains, 2024-08-06), the command-line options '--incremental' and\n'--bitmap' were declared to be incompatible with one another when\nrunning 'git multi-pack-index write'.\n\nHowever, since 27afc272c49 (midx: implement writing incremental MIDX\nbitmaps, 2025-03-20), that incompatibility no longer exists, despite the\ndocumentation saying so. Correct this by removing the stale reference to\ntheir incompatibility.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex a4550e28bed..a502819fc38 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -75,7 +75,7 @@ marker).\n \t\tWrite an incremental MIDX file containing only objects\n \t\tand packs not present in an existing MIDX layer.\n \t\tMigrates non-incremental MIDXs to incremental ones when\n-\t\tnecessary. Incompatible with `--bitmap`.\n+\t\tnecessary.\n --\n \n verify::\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531769","messageId":"e19fcd92622c282194fd60d4a8e8feb135c50551.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 05/17] git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:13Z","receivedAt":"2025-12-06T20:31:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since c39fffc1c90 (tests: start asserting that *.txt SYNOPSIS matches -h\noutput, 2022-10-13), the manual page for 'git multi-pack-index' has a\nSYNOPSIS section which differs from 'git multi-pack-index -h'.\n\nCorrect this while also documenting additional options accepted by the\n'write' sub-command.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 7 ++++++-\n builtin/multi-pack-index.c | 5 +++--\n t/t0450/adoc-help-mismatches | 1 -\n 3 files changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex a502819fc38..164cf1f2291 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -9,7 +9,12 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] [--[no-]bitmap] \n+'git multi-pack-index' [] write [--preferred-pack=]\n+\t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n+\t\t [--refs-snapshot=]\n+'git multi-pack-index' [] verify\n+'git multi-pack-index' [] expire\n+'git multi-pack-index' [] repack [--batch-size=]\n \n DESCRIPTION\n -----------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex ca98d4c3ba3..c0c6c1760c0 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -13,8 +13,9 @@\n #include \"repository.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n-\tN_(\"git multi-pack-index [] write [--preferred-pack=]\" \\\n-\t \"[--refs-snapshot=]\")\n+\tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n+\t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n+\t \" [--refs-snapshot=]\")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\ndiff --git a/t/t0450/adoc-help-mismatches b/t/t0450/adoc-help-mismatches\nindex 8ee2d3f7c81..e8d6c13ccd0 100644\n--- a/t/t0450/adoc-help-mismatches\n+++ b/t/t0450/adoc-help-mismatches\n@@ -33,7 +33,6 @@ merge\n merge-file\n merge-index\n merge-one-file\n-multi-pack-index\n name-rev\n notes\n push\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531770","messageId":"7bd2604561a1470e5e4ef902aa2ba3130baa821f.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 06/17] t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:16Z","receivedAt":"2025-12-06T20:31:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Commit d4bf1d88b90 (multi-pack-index: verify missing pack, 2018-09-13)\nadds a new test to the MIDX test script to test how we handle missing\npacks.\n\nWhile the commit itself describes the test as \"verify missing pack[s]\",\nthe test itself is actually called \"verify packnames out of order\",\ndespite that not being what it tests.\n\nLikely this was a copy-and-paste of the test immediately above it of the\nsame name. Correct this by renaming the test to match the commit\nmessage.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 93f319a4b29..ca020091dda 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -455,7 +455,7 @@ test_expect_success 'verify packnames out of order' '\n \t\t\"pack names out of order\"\n '\n \n-test_expect_success 'verify packnames out of order' '\n+test_expect_success 'verify missing pack' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"a\" $objdir \\\n \t\t\"failed to load pack\"\n '\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531771","messageId":"1fc359d9a98cdd61f6986fa1d9a1190f57dacb9b.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 07/17] midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:19Z","receivedAt":"2025-12-06T20:31:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In midx_pack_order(), we compute for each bitampped pack the first bit\nto correspond to an object in that pack, along with how many bits were\nassigned to object(s) in that pack.\n\nInitially, each bitmap_nr value is set to zero, and each bitmap_pos\nvalue is set to the sentinel BITMAP_POS_UNKNOWN. This is done to ensure\nthat there are no packs who have an unknown bit position but a somehow\nnon-zero number of objects (cf. `write_midx_bitmapped_packs()` in\nmidx-write.c).\n\nOnce the pack order is fully determined, midx_pack_order() sets the\nbitmap_pos field for any bitmapped packs to zero if they are still\nlisted as BITMAP_POS_UNKNOWN.\n\nHowever, we enumerate the bitmapped packs in order of `ctx->pack_perm`.\nThis is fine for existing cases, since the only time the\n`ctx->pack_perm` array holds a value outside of the addressable range of\n`ctx->info` is when there are expired packs, which only occurs via 'git\nmulti-pack-index expire', which does not support writing MIDX bitmaps.\nAs a result, the range of ctx->pack_perm covers all values in [0,\n`ctx->nr`), so enumerating in this order isn't an issue.\n\nA future change necessary for compaction will complicate this further by\nintroducing a wrapper around the `ctx->pack_perm` array, which turns the\ngiven `pack_int_id` into one that is relative to the lower end of the\ncompaction range. As a result, indexing into `ctx->pack_perm` through\nthis helper, say, with \"0\" will produce a crash when the lower end of\nthe compaction range has >0 pack(s) in its base layer, since the\nsubtraction will wrap around the 32-bit unsigned range, resulting in an\nuninitialized read.\n\nBut the process is completely unnecessary in the first place: we are\nenumerating all values of `ctx->info`, and there is no reason to process\nthem in a different order than they appear in memory. Index `ctx->info`\ndirectly to reflect that.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 73d24fabbc6..c30f6a70d37 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -637,7 +637,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \t\tpack_order[i] = data[i].nr;\n \t}\n \tfor (i = 0; i < ctx->nr; i++) {\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n+\t\tstruct pack_info *pack = &ctx->info[i];\n \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n \t\t\tpack->bitmap_pos = 0;\n \t}\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531772","messageId":"47aae3bf2a83a2724aecd3314f8cc5d47e8013f1.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 08/17] midx-write.c: introduce `struct write_midx_opts`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:22Z","receivedAt":"2025-12-06T20:31:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the MIDX writing code, there are four functions which perform some\nsort of MIDX write operation. They are:\n\n - write_midx_file()\n - write_midx_file_only()\n - expire_midx_packs()\n - midx_repack()\n\nAll of these functions are thin wrappers over `write_midx_internal()`,\nwhich implements the bulk of these routines. As a result, the\n`write_midx_internal()` function takes six arguments.\n\nFuture commits in this series will want to add additional arguments, and\nin general this function's signature will be the union of parameters\namong *all* possible ways to write a MIDX.\n\nInstead of adding yet more arguments to this function to support MIDX\ncompaction, introduce a `struct write_midx_opts`, which has the same\nstruct members as `write_midx_internal()`'s arguments.\n\nAdding additional fields to the `write_midx_opts` struct is preferable\nto adding additional arguments to `write_midx_internal()`. This is\nbecause the callers below all zero-initialize the struct, so each time\nwe add a new piece of information, we do not have to pass the zero value\nfor it in all other call-sites that do not care about it.\n\nFor now, no functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 129 ++++++++++++++++++++++++++++++---------------------\n 1 file changed, 77 insertions(+), 52 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex c30f6a70d37..b262631ae45 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1014,14 +1014,20 @@ static void clear_midx_files(struct odb_source *source,\n \tstrbuf_release(&buf);\n }\n \n-static int write_midx_internal(struct odb_source *source,\n-\t\t\t struct string_list *packs_to_include,\n-\t\t\t struct string_list *packs_to_drop,\n-\t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot,\n-\t\t\t unsigned flags)\n+struct write_midx_opts {\n+\tstruct odb_source *source;\n+\n+\tstruct string_list *packs_to_include;\n+\tstruct string_list *packs_to_drop;\n+\n+\tconst char *preferred_pack_name;\n+\tconst char *refs_snapshot;\n+\tunsigned flags;\n+};\n+\n+static int write_midx_internal(struct write_midx_opts *opts)\n {\n-\tstruct repository *r = source->odb->repo;\n+\tstruct repository *r = opts->source->odb->repo;\n \tstruct strbuf midx_name = STRBUF_INIT;\n \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t start_pack;\n@@ -1041,22 +1047,22 @@ static int write_midx_internal(struct odb_source *source,\n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n \n \tctx.repo = r;\n-\tctx.source = source;\n+\tctx.source = opts->source;\n \n-\tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n+\tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n \t\t\t \"%s/pack/multi-pack-index.d/tmp_midx_XXXXXX\",\n-\t\t\t source->path);\n+\t\t\t opts->source->path);\n \telse\n-\t\tget_midx_filename(source, &midx_name);\n+\t\tget_midx_filename(opts->source, &midx_name);\n \tif (safe_create_leading_directories(r, midx_name.buf))\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name.buf);\n \n-\tif (!packs_to_include || ctx.incremental) {\n-\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\tif (!opts->packs_to_include || ctx.incremental) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(opts->source);\n \t\tif (m && !midx_checksum_valid(m)) {\n \t\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n \t\t\tm = NULL;\n@@ -1071,7 +1077,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t */\n \t\t\tif (ctx.incremental)\n \t\t\t\tctx.base_midx = m;\n-\t\t\telse if (!packs_to_include)\n+\t\t\telse if (!opts->packs_to_include)\n \t\t\t\tctx.m = m;\n \t\t}\n \t}\n@@ -1084,7 +1090,7 @@ static int write_midx_internal(struct odb_source *source,\n \tif (ctx.incremental) {\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\twhile (m) {\n-\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n+\t\t\tif (opts->flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n \t\t\t\t get_midx_checksum(m));\n \t\t\t\tgoto cleanup;\n@@ -1099,23 +1105,23 @@ static int write_midx_internal(struct odb_source *source,\n \tstart_pack = ctx.nr;\n \n \tctx.pack_paths_checked = 0;\n-\tif (flags & MIDX_PROGRESS)\n+\tif (opts->flags & MIDX_PROGRESS)\n \t\tctx.progress = start_delayed_progress(r,\n \t\t\t\t\t\t _(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n \t\tctx.progress = NULL;\n \n-\tctx.to_include = packs_to_include;\n+\tctx.to_include = opts->packs_to_include;\n \n-\tfor_each_file_in_pack_dir(source->path, add_pack_to_midx, &ctx);\n+\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n \tif ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) &&\n \t !ctx.incremental &&\n-\t !(packs_to_include || packs_to_drop)) {\n+\t !(opts->packs_to_include || opts->packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\n \t\tint bitmap_exists;\n-\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\t\tint want_bitmap = opts->flags & MIDX_WRITE_BITMAP;\n \n \t\tbitmap_git = prepare_midx_bitmap_git(ctx.m);\n \t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n@@ -1127,7 +1133,8 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t * corresponding bitmap (or one wasn't requested).\n \t\t\t */\n \t\t\tif (!want_bitmap)\n-\t\t\t\tclear_midx_files_ext(source, \"bitmap\", NULL);\n+\t\t\t\tclear_midx_files_ext(opts->source, \"bitmap\",\n+\t\t\t\t\t\t NULL);\n \t\t\tresult = 0;\n \t\t\tgoto cleanup;\n \t\t}\n@@ -1138,11 +1145,11 @@ static int write_midx_internal(struct odb_source *source,\n \t\tgoto cleanup; /* nothing to do */\n \t}\n \n-\tif (preferred_pack_name) {\n+\tif (opts->preferred_pack_name) {\n \t\tctx.preferred_pack_idx = NO_PREFERRED_PACK;\n \n \t\tfor (size_t i = 0; i < ctx.nr; i++) {\n-\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n+\t\t\tif (!cmp_idx_or_pack_name(opts->preferred_pack_name,\n \t\t\t\t\t\t ctx.info[i].pack_name)) {\n \t\t\t\tctx.preferred_pack_idx = i;\n \t\t\t\tbreak;\n@@ -1151,9 +1158,9 @@ static int write_midx_internal(struct odb_source *source,\n \n \t\tif (ctx.preferred_pack_idx == NO_PREFERRED_PACK)\n \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n-\t\t\t\tpreferred_pack_name);\n+\t\t\t\topts->preferred_pack_name);\n \t} else if (ctx.nr &&\n-\t\t (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n+\t\t (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n \t\tstruct packed_git *oldest = ctx.info[0].p;\n \t\tctx.preferred_pack_idx = 0;\n \n@@ -1164,7 +1171,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t */\n \t\topen_pack_index(oldest);\n \n-\t\tif (packs_to_drop && packs_to_drop->nr)\n+\t\tif (opts->packs_to_drop && opts->packs_to_drop->nr)\n \t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n \n \t\t/*\n@@ -1226,20 +1233,21 @@ static int write_midx_internal(struct odb_source *source,\n \n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n-\tif (packs_to_drop && packs_to_drop->nr) {\n+\tif (opts->packs_to_drop && opts->packs_to_drop->nr) {\n \t\tsize_t drop_index = 0;\n \t\tint missing_drops = 0;\n \n-\t\tfor (size_t i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\tfor (size_t i = 0;\n+\t\t i < ctx.nr && drop_index < opts->packs_to_drop->nr; i++) {\n \t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n-\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\t\t opts->packs_to_drop->items[drop_index].string);\n \n \t\t\tif (!cmp) {\n \t\t\t\tdrop_index++;\n \t\t\t\tctx.info[i].expired = 1;\n \t\t\t} else if (cmp > 0) {\n \t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n-\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\t opts->packs_to_drop->items[drop_index].string);\n \t\t\t\tdrop_index++;\n \t\t\t\tmissing_drops++;\n \t\t\t\ti--;\n@@ -1276,8 +1284,8 @@ static int write_midx_internal(struct odb_source *source,\n \t}\n \n \t/* Check that the preferred pack wasn't expired (if given). */\n-\tif (preferred_pack_name) {\n-\t\tstruct pack_info *preferred = bsearch(preferred_pack_name,\n+\tif (opts->preferred_pack_name) {\n+\t\tstruct pack_info *preferred = bsearch(opts->preferred_pack_name,\n \t\t\t\t\t\t ctx.info, ctx.nr,\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n@@ -1285,7 +1293,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n-\t\t\t\t\tpreferred_pack_name);\n+\t\t\t\t\topts->preferred_pack_name);\n \t\t}\n \t}\n \n@@ -1299,15 +1307,15 @@ static int write_midx_internal(struct odb_source *source,\n \t}\n \n \tif (!ctx.entries_nr) {\n-\t\tif (flags & MIDX_WRITE_BITMAP)\n+\t\tif (opts->flags & MIDX_WRITE_BITMAP)\n \t\t\twarning(_(\"refusing to write multi-pack .bitmap without any objects\"));\n-\t\tflags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n+\t\topts->flags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n \t}\n \n \tif (ctx.incremental) {\n \t\tstruct strbuf lock_name = STRBUF_INIT;\n \n-\t\tget_midx_chain_filename(source, &lock_name);\n+\t\tget_midx_chain_filename(opts->source, &lock_name);\n \t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n \t\tstrbuf_release(&lock_name);\n \n@@ -1350,7 +1358,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t\tMIDX_CHUNK_LARGE_OFFSET_WIDTH),\n \t\t\twrite_midx_large_offsets);\n \n-\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n+\tif (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n \t\tctx.pack_order = midx_pack_order(&ctx);\n \t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n \t\t\t st_mult(ctx.entries_nr, sizeof(uint32_t)),\n@@ -1368,11 +1376,11 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX &&\n+\tif (opts->flags & MIDX_WRITE_REV_INDEX &&\n \t git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n \t\twrite_midx_reverse_index(&ctx, midx_hash);\n \n-\tif (flags & MIDX_WRITE_BITMAP) {\n+\tif (opts->flags & MIDX_WRITE_BITMAP) {\n \t\tstruct packing_data pdata;\n \t\tstruct commit **commits;\n \t\tuint32_t commits_nr;\n@@ -1382,7 +1390,7 @@ static int write_midx_internal(struct odb_source *source,\n \n \t\tprepare_midx_packing_data(&pdata, &ctx);\n \n-\t\tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, &ctx);\n+\t\tcommits = find_commits_for_midx_bitmap(&commits_nr, opts->refs_snapshot, &ctx);\n \n \t\t/*\n \t\t * The previous steps translated the information from\n@@ -1395,7 +1403,7 @@ static int write_midx_internal(struct odb_source *source,\n \n \t\tif (write_midx_bitmap(&ctx,\n \t\t\t\t midx_hash, &pdata, commits, commits_nr,\n-\t\t\t\t flags) < 0) {\n+\t\t\t\t opts->flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tclear_packing_data(&pdata);\n \t\t\tfree(commits);\n@@ -1428,7 +1436,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n \t\t\tgoto cleanup;\n \n-\t\tget_split_midx_filename_ext(source, &final_midx_name,\n+\t\tget_split_midx_filename_ext(opts->source, &final_midx_name,\n \t\t\t\t\t midx_hash, MIDX_EXT_MIDX);\n \n \t\tif (rename_tempfile(&incr, final_midx_name.buf) < 0) {\n@@ -1461,7 +1469,7 @@ static int write_midx_internal(struct odb_source *source,\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(source, keep_hashes,\n+\tclear_midx_files(opts->source, keep_hashes,\n \t\t\t ctx.num_multi_pack_indexes_before + 1,\n \t\t\t ctx.incremental);\n \tresult = 0;\n@@ -1495,9 +1503,14 @@ int write_midx_file(struct odb_source *source,\n \t\t const char *preferred_pack_name,\n \t\t const char *refs_snapshot, unsigned flags)\n {\n-\treturn write_midx_internal(source, NULL, NULL,\n-\t\t\t\t preferred_pack_name, refs_snapshot,\n-\t\t\t\t flags);\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.preferred_pack_name = preferred_pack_name,\n+\t\t.refs_snapshot = refs_snapshot,\n+\t\t.flags = flags,\n+\t};\n+\n+\treturn write_midx_internal(&opts);\n }\n \n int write_midx_file_only(struct odb_source *source,\n@@ -1505,8 +1518,15 @@ int write_midx_file_only(struct odb_source *source,\n \t\t\t const char *preferred_pack_name,\n \t\t\t const char *refs_snapshot, unsigned flags)\n {\n-\treturn write_midx_internal(source, packs_to_include, NULL,\n-\t\t\t\t preferred_pack_name, refs_snapshot, flags);\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.packs_to_include = packs_to_include,\n+\t\t.preferred_pack_name = preferred_pack_name,\n+\t\t.refs_snapshot = refs_snapshot,\n+\t\t.flags = flags,\n+\t};\n+\n+\treturn write_midx_internal(&opts);\n }\n \n int expire_midx_packs(struct odb_source *source, unsigned flags)\n@@ -1566,8 +1586,11 @@ int expire_midx_packs(struct odb_source *source, unsigned flags)\n \tfree(count);\n \n \tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(source, NULL,\n-\t\t\t\t\t &packs_to_drop, NULL, NULL, flags);\n+\t\tresult = write_midx_internal(&(struct write_midx_opts) {\n+\t\t\t\t\t .source = source,\n+\t\t\t\t\t .packs_to_drop = &packs_to_drop,\n+\t\t\t\t\t .flags = flags & MIDX_PROGRESS,\n+\t\t\t\t\t });\n \n \tstring_list_clear(&packs_to_drop, 0);\n \n@@ -1774,8 +1797,10 @@ int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags)\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(source, NULL, NULL, NULL, NULL,\n-\t\t\t\t flags);\n+\tresult = write_midx_internal(&(struct write_midx_opts) {\n+\t\t\t\t .source = source,\n+\t\t\t\t .flags = flags,\n+\t\t\t\t });\n \n cleanup:\n \tfree(include_pack);\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531773","messageId":"d5389a93b16a4933b0c54f78f2d5ce84b9ecac53.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 09/17] midx: do not require packs to be sorted in lexicographic order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:25Z","receivedAt":"2025-12-06T20:31:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The MIDX file format currently requires that pack files be identified by\nthe lexicographic ordering of their names (that is, a pack having a\nchecksum beginning with \"abc\" would have a numeric pack_int_id which is\nsmaller than the same value for a pack beginning with \"bcd\").\n\nAs a result, it is impossible to combine adjacent MIDX layers together\nwithout permuting bits from bitmaps that are in more recent layer(s).\n\nTo see why, consider the following example:\n\n | packs | preferred pack\n --------+-------------+---------------\n MIDX #0 | { X, Y, Z } | Y\n MIDX #1 | { A, B, C } | B\n MIDX #2 | { D, E, F } | D\n\n, where MIDX #2's base MIDX is MIDX #1, and so on. Suppose that we want\nto combine MIDX layers #0 and #1, to create a new layer #0' containing\nthe packs from both layers. With the original three MIDX layers, objects\nare laid out in the bitmap in the order they appear in their source\npack, and the packs themselves are arranged according to the pseudo-pack\norder. In this case, that ordering is Y, X, Z, B, A, C.\n\nBut recall that the pseudo-pack ordering is defined by the order that\npacks appear in the MIDX, with the exception of the preferred pack,\nwhich sorts ahead of all other packs regardless of its position within\nthe MIDX. In the above example, that means that pack 'Y' could be placed\nanywhere (so long as it is designated as preferred), however, all other\npacks must be placed in the location listed above.\n\nBecause that ordering isn't sorted lexicographically, it is impossible\nto compact MIDX layers in the above configuration without permuting the\nobject-to-bit-position mapping. Changing this mapping would affect all\nbitmaps belonging to newer layers, rendering the bitmaps associated with\nMIDX #2 unreadable.\n\nOne of the goals of MIDX compaction is that we are able to shrink the\nlength of the MIDX chain *without* invalidating bitmaps that belong to\nnewer layers, and the lexicographic ordering constraint is at odds with\nthis goal.\n\nHowever, packs do not *need* to be lexicographically ordered within the\nMIDX. As far as I can gather, the only reason they are sorted lexically\nis to make it possible to perform a binary search over the pack names in\na MIDX, necessary to make `midx_contains_pack()`'s performance\nlogarithmic in the number of packs rather than linear.\n\nRelax this constraint by allowing MIDX writes to proceed with packs that\nare not arranged in lexicographic order. `midx_contains_pack()` will\nlazily instantiate a `pack_names_sorted` array on the MIDX, which will\nbe used to implement the binary search over pack names.\n\nNote that this produces MIDXs which may be incompatible with earlier\nversions of Git that have stricter requirements on the layout of packs\nwithin a MIDX. This patch does *not* modify the version number of the\nMIDX format, since existing versions of Git already know to gracefully\nignore a MIDX with packs that appear out-of-order.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 5 -----\n midx.c | 28 ++++++++++++++++++++++------\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 5 -----\n 4 files changed, 23 insertions(+), 16 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex b262631ae45..55342fcb6dd 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -410,11 +410,6 @@ static int write_midx_pack_names(struct hashfile *f, void *data)\n \t\tif (ctx->info[i].expired)\n \t\t\tcontinue;\n \n-\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n-\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t ctx->info[i - 1].pack_name,\n-\t\t\t ctx->info[i].pack_name);\n-\n \t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n \t\thashwrite(f, ctx->info[i].pack_name, writelen);\n \t\twritten += writelen;\ndiff --git a/midx.c b/midx.c\nindex f9b11de9ca9..4d5fe880649 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -209,11 +209,6 @@ static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *sou\n \t\tif (!end)\n \t\t\tdie(_(\"multi-pack-index pack-name chunk is too short\"));\n \t\tcur_pack_name = end + 1;\n-\n-\t\tif (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0)\n-\t\t\tdie(_(\"multi-pack-index pack names out of order: '%s' before '%s'\"),\n-\t\t\t m->pack_names[i - 1],\n-\t\t\t m->pack_names[i]);\n \t}\n \n \ttrace2_data_intmax(\"midx\", r, \"load/num_packs\", m->num_packs);\n@@ -411,6 +406,7 @@ void close_midx(struct multi_pack_index *m)\n \t}\n \tFREE_AND_NULL(m->packs);\n \tFREE_AND_NULL(m->pack_names);\n+\tFREE_AND_NULL(m->pack_names_sorted);\n \tfree(m);\n }\n \n@@ -656,17 +652,37 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n+\n+static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n+{\n+\tstruct multi_pack_index *m = m_;\n+\treturn strcmp(m->pack_names[*(const size_t *)a],\n+\t\t m->pack_names[*(const size_t *)b]);\n+}\n+\n static int midx_contains_pack_1(struct multi_pack_index *m,\n \t\t\t\tconst char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n+\tif (!m->pack_names_sorted) {\n+\t\tuint32_t i;\n+\n+\t\tALLOC_ARRAY(m->pack_names_sorted, m->num_packs);\n+\n+\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\t\tm->pack_names_sorted[i] = i;\n+\n+\t\tQSORT_S(m->pack_names_sorted, m->num_packs, midx_pack_names_cmp,\n+\t\t\tm);\n+\t}\n+\n \twhile (first < last) {\n \t\tuint32_t mid = first + (last - first) / 2;\n \t\tconst char *current;\n \t\tint cmp;\n \n-\t\tcurrent = m->pack_names[mid];\n+\t\tcurrent = m->pack_names[m->pack_names_sorted[mid]];\n \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n \t\tif (!cmp)\n \t\t\treturn 1;\ndiff --git a/midx.h b/midx.h\nindex e188ffeb578..39bf04b18e5 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -71,6 +71,7 @@ struct multi_pack_index {\n \tuint32_t num_packs_in_base;\n \n \tconst char **pack_names;\n+\tsize_t *pack_names_sorted;\n \tstruct packed_git **packs;\n };\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex ca020091dda..03676d37b98 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -450,11 +450,6 @@ test_expect_success 'verify invalid chunk offset' '\n \t\t\"improper chunk offset(s)\"\n '\n \n-test_expect_success 'verify packnames out of order' '\n-\tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"z\" $objdir \\\n-\t\t\"pack names out of order\"\n-'\n-\n test_expect_success 'verify missing pack' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"a\" $objdir \\\n \t\t\"failed to load pack\"\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531774","messageId":"776eb88905f53dd929e5b7cb23593976a2f1aad3.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 10/17] git-compat-util.h: introduce `u32_add()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:28Z","receivedAt":"2025-12-06T20:31:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A future commit will want to add two 32-bit unsigned values together\nwhile checking for overflow. Introduce a variant of the u64_add()\nfunction for operating on 32-bit inputs.\n\nSigned-off-by: Taylor Blau \n---\n git-compat-util.h | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 398e0fac4fa..a7aa5f05fc9 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -670,6 +670,14 @@ static inline int cast_size_t_to_int(size_t a)\n \treturn (int)a;\n }\n \n+static inline uint32_t u32_add(uint32_t a, uint32_t b)\n+{\n+\tif (unsigned_add_overflows(a, b))\n+\t\tdie(\"uint32_t overflow: %\"PRIuMAX\" + %\"PRIuMAX,\n+\t\t (uintmax_t)a, (uintmax_t)b);\n+\treturn a + b;\n+}\n+\n static inline uint64_t u64_mult(uint64_t a, uint64_t b)\n {\n \tif (unsigned_mult_overflows(a, b))\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531775","messageId":"d4f0b369b6f6b37ffe3823fec1f64b0e542e080d.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 11/17] midx-write.c: introduce `midx_pack_perm()` helper","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:31Z","receivedAt":"2025-12-06T20:31:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `ctx->pack_perm` array can be considered as a permutation between\nthe original `pack_int_id` of some given pack to its position in the\n`ctx->info` array containing all packs.\n\nToday we can always index into this array with any known `pack_int_id`,\nsince there is never a `pack_int_id` which is greater than or equal to\nthe value `ctx->nr`.\n\nThat is not necessarily the case with MIDX compaction. For example,\nsuppose we have a MIDX chain with three layers, each containing three\npacks. The base of the MIDX chain will have packs with IDs 0, 1, and 2,\nthe next layer 3, 4, and 5, and so on. If we are compacting the topmost\ntwo layers, we'll have input `pack_int_id` values between [3, 8], but\n`ctx->nr` will only be 6.\n\nIn that example, if we want to know where the pack whose original\n`pack_int_id` value was, say, 7, we would compute `ctx->pack_perm[7]`,\nleading to an uninitialized read, since there are only 6 entries\nallocated in that array.\n\nTo address this, there are a couple of options:\n\n - We could allocate enough entries in `ctx->pack_perm` to accommodate\n the largest `orig_pack_int_id` value.\n\n - Or, we could internally shift the input values by the number of packs\n in the base layer of the lower end of the MIDX compaction range.\n\nThis patch prepare us to take the latter approach, since it does not\nallocate more memory than strictly necessary. (In our above example, the\nbase of the lower end of the compaction range is the first MIDX layer\n(having three packs), so we would end up indexing `ctx->pack_perm[7-3]`,\nwhich is a valid read.)\n\nNote that this patch does not actually implement that approach yet, but\nmerely performs a behavior-preserving refactoring which will make the\nchange easier to carry out in the future.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 55342fcb6dd..4a1a16431a6 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -114,6 +114,12 @@ struct write_midx_context {\n \tstruct odb_source *source;\n };\n \n+static uint32_t midx_pack_perm(struct write_midx_context *ctx,\n+\t\t\t uint32_t orig_pack_int_id)\n+{\n+\treturn ctx->pack_perm[orig_pack_int_id];\n+}\n+\n static int should_include_pack(const struct write_midx_context *ctx,\n \t\t\t const char *file_name)\n {\n@@ -509,12 +515,12 @@ static int write_midx_object_offsets(struct hashfile *f,\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\tif (midx_pack_perm(ctx, obj->pack_int_id) == PACK_EXPIRED)\n \t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->pack_int_id);\n \n-\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n+\t\thashwrite_be32(f, midx_pack_perm(ctx, obj->pack_int_id));\n \n \t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n@@ -615,7 +621,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[i];\n \t\tdata[i].nr = i;\n-\t\tdata[i].pack = ctx->pack_perm[e->pack_int_id];\n+\t\tdata[i].pack = midx_pack_perm(ctx, e->pack_int_id);\n \t\tif (!e->preferred)\n \t\t\tdata[i].pack |= (1U << 31);\n \t\tdata[i].offset = e->offset;\n@@ -625,7 +631,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n+\t\tstruct pack_info *pack = &ctx->info[midx_pack_perm(ctx, e->pack_int_id)];\n \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n \t\t\tpack->bitmap_pos = i + base_objects;\n \t\tpack->bitmap_nr++;\n@@ -686,7 +692,7 @@ static void prepare_midx_packing_data(struct packing_data *pdata,\n \t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n \n \t\toe_set_in_pack(pdata, to,\n-\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t\t\t ctx->info[midx_pack_perm(ctx, from->pack_int_id)].p);\n \t}\n \n \ttrace2_region_leave(\"midx\", \"prepare_midx_packing_data\", ctx->repo);\n@@ -1285,7 +1291,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n \t\tif (preferred) {\n-\t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n+\t\t\tuint32_t perm = midx_pack_perm(&ctx, preferred->orig_pack_int_id);\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n \t\t\t\t\topts->preferred_pack_name);\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531776","messageId":"d13fce5c79578720912d779d73287c786bb54368.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 12/17] midx-write.c: extract `fill_pack_from_midx()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:34Z","receivedAt":"2025-12-06T20:31:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When filling packs from an existing MIDX, `fill_packs_from_midx()`\nhandles preparing a MIDX'd pack, and reading out its pack name from the\nexisting MIDX.\n\nMIDX compaction will want to perform an identical operation, though the\ncaller will look quite different than `fill_packs_from_midx()`. To\nreduce any future code duplication, extract `fill_pack_from_midx()`\nfrom `fill_packs_from_midx()` to prepare to call our new helper function\nin a future change.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 27 +++++++++++++++++++++------\n 1 file changed, 21 insertions(+), 6 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 4a1a16431a6..5927691f6a0 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -910,6 +910,21 @@ static int write_midx_bitmap(struct write_midx_context *ctx,\n \treturn ret;\n }\n \n+static int fill_pack_from_midx(struct pack_info *info,\n+\t\t\t struct multi_pack_index *m,\n+\t\t\t uint32_t pack_int_id)\n+{\n+\tif (prepare_midx_pack(m, pack_int_id))\n+\t\treturn error(_(\"could not load pack %d\"), pack_int_id);\n+\n+\tfill_pack_info(info,\n+\t\t m->packs[pack_int_id - m->num_packs_in_base],\n+\t\t m->pack_names[pack_int_id - m->num_packs_in_base],\n+\t\t pack_int_id);\n+\n+\treturn 0;\n+}\n+\n static int fill_packs_from_midx(struct write_midx_context *ctx)\n {\n \tstruct multi_pack_index *m;\n@@ -918,13 +933,13 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n \t\tuint32_t i;\n \n \t\tfor (i = 0; i < m->num_packs; i++) {\n-\t\t\tif (prepare_midx_pack(m, m->num_packs_in_base + i))\n-\t\t\t\treturn error(_(\"could not load pack\"));\n-\n \t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n-\t\t\tfill_pack_info(&ctx->info[ctx->nr++], m->packs[i],\n-\t\t\t\t m->pack_names[i],\n-\t\t\t\t m->num_packs_in_base + i);\n+\n+\t\t\tif (fill_pack_from_midx(&ctx->info[ctx->nr], m,\n+\t\t\t\t\t\tm->num_packs_in_base + i) < 0)\n+\t\t\t\treturn -1;\n+\n+\t\t\tctx->nr++;\n \t\t}\n \t}\n \treturn 0;\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531777","messageId":"8288afd8cc40a56999353643f49c48d610e6b3aa.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 13/17] midx-write.c: enumerate `pack_int_id` values directly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:37Z","receivedAt":"2025-12-06T20:31:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Our `midx-write.c::fill_packs_from_midx()` function currently enumerates\nthe range [0, m->num_packs), and then shifts its index variable up by\n`m->num_packs_in_base` to produce a valid `pack_int_id`.\n\nInstead, directly enumerate the range:\n\n [m->num_packs_in_base, m->num_packs_in_base + m->num_packs)\n\n, which are the original pack_int_ids themselves as opposed to the\nindexes of those packs relative to the MIDX layer they are contained\nwithin.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 5927691f6a0..d3644276aad 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -932,11 +932,11 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n \tfor (m = ctx->m; m; m = m->base_midx) {\n \t\tuint32_t i;\n \n-\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tfor (i = m->num_packs_in_base;\n+\t\t i < m->num_packs_in_base + m->num_packs; i++) {\n \t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\t\tif (fill_pack_from_midx(&ctx->info[ctx->nr], m,\n-\t\t\t\t\t\tm->num_packs_in_base + i) < 0)\n+\t\t\tif (fill_pack_from_midx(&ctx->info[ctx->nr], m, i) < 0)\n \t\t\t\treturn -1;\n \n \t\t\tctx->nr++;\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531778","messageId":"d94222ba33050e69ed59d363873cd98ee31f8b6b.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 14/17] midx-write.c: factor fanout layering from `compute_sorted_entries()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:40Z","receivedAt":"2025-12-06T20:31:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When computing the set of objects to appear in a MIDX, we use\ncompute_sorted_entries(), which handles objects from various existing\nsources one fanout layer at a time.\n\nThe process for computing this set is slightly different during MIDX\ncompaction, so factor out the existing functionality into its own\nroutine to prevent `compute_sorted_entries()` from becoming too\ndifficult to read.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex d3644276aad..7854561359d 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -323,6 +323,30 @@ static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n \t}\n }\n \n+static void midx_fanout_add(struct midx_fanout *fanout,\n+\t\t\t struct write_midx_context *ctx,\n+\t\t\t uint32_t start_pack,\n+\t\t\t uint32_t cur_fanout)\n+{\n+\tuint32_t cur_pack;\n+\n+\tif (ctx->m && !ctx->incremental)\n+\t\tmidx_fanout_add_midx_fanout(fanout, ctx->m, cur_fanout,\n+\t\t\t\t\t ctx->preferred_pack_idx);\n+\n+\tfor (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) {\n+\t\tint preferred = cur_pack == ctx->preferred_pack_idx;\n+\t\tmidx_fanout_add_pack_fanout(fanout, ctx->info, cur_pack,\n+\t\t\t\t\t preferred, cur_fanout);\n+\t}\n+\n+\tif (ctx->preferred_pack_idx != NO_PREFERRED_PACK &&\n+\t ctx->preferred_pack_idx < start_pack)\n+\t\tmidx_fanout_add_pack_fanout(fanout, ctx->info,\n+\t\t\t\t\t ctx->preferred_pack_idx, 1,\n+\t\t\t\t\t cur_fanout);\n+}\n+\n /*\n * It is possible to artificially get into a state where there are many\n * duplicate copies of objects. That can create high memory pressure if\n@@ -359,23 +383,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n \n-\t\tif (ctx->m && !ctx->incremental)\n-\t\t\tmidx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout,\n-\t\t\t\t\t\t ctx->preferred_pack_idx);\n-\n-\t\tfor (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) {\n-\t\t\tint preferred = cur_pack == ctx->preferred_pack_idx;\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout,\n-\t\t\t\t\t\t ctx->info, cur_pack,\n-\t\t\t\t\t\t preferred, cur_fanout);\n-\t\t}\n-\n-\t\tif (ctx->preferred_pack_idx != NO_PREFERRED_PACK &&\n-\t\t ctx->preferred_pack_idx < start_pack)\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout, ctx->info,\n-\t\t\t\t\t\t ctx->preferred_pack_idx, 1,\n-\t\t\t\t\t\t cur_fanout);\n-\n+\t\tmidx_fanout_add(&fanout, ctx, start_pack, cur_fanout);\n \t\tmidx_fanout_sort(&fanout);\n \n \t\t/*\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531779","messageId":"43ef3d0013f1534c7ff8b1a7c4a17fb0c9b68684.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 15/17] t/helper/test-read-midx.c: plug memory leak when selecting layer","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:43Z","receivedAt":"2025-12-06T20:31:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Though our 'read-midx' test tool is capable of printing information\nabout a single MIDX layer identified by its checksum, no caller in our\ntest suite exercises this path.\n\nUnfortunately, there is a memory leak lurking in this (currently) unused\npath that would otherwise be exposed by the following commit.\n\nThis occurs when providing a MIDX layer checksum other than the tip. As\nwe walk over the MIDX chain trying to find the matching layer, we drop\nour reference to the top-most MIDX layer. Thus, our call to\n'close_midx()' later on leaks memory between the top-most MIDX layer and\nthe MIDX layer immediately following the specified one.\n\nPlug this leak by holding a reference to the tip of the MIDX chain, and\nensure that we call `close_midx()` before terminating the test tool.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 17 +++++++++++------\n 1 file changed, 11 insertions(+), 6 deletions(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex dee603b3cd0..6e03aabca79 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -26,9 +26,10 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\t\t int show_objects)\n {\n \tuint32_t i;\n-\tstruct multi_pack_index *m;\n+\tstruct multi_pack_index *m, *tip;\n+\tint ret = 0;\n \n-\tm = setup_midx(object_dir);\n+\tm = tip = setup_midx(object_dir);\n \n \tif (!m)\n \t\treturn 1;\n@@ -36,8 +37,11 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \tif (checksum) {\n \t\twhile (m && strcmp(get_midx_checksum(m), checksum))\n \t\t\tm = m->base_midx;\n-\t\tif (!m)\n-\t\t\treturn 1;\n+\t\tif (!m) {\n+\t\t\tret = error(_(\"could not find MIDX with checksum %s\"),\n+\t\t\t\t checksum);\n+\t\t\tgoto out;\n+\t\t}\n \t}\n \n \tprintf(\"header: %08x %d %d %d %d\\n\",\n@@ -82,9 +86,10 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\t}\n \t}\n \n-\tclose_midx(m);\n+out:\n+\tclose_midx(tip);\n \n-\treturn 0;\n+\treturn ret;\n }\n \n static int read_midx_checksum(const char *object_dir)\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531780","messageId":"c136b2e179d02321de7e7b3f1b6c748cb434d68d.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 16/17] midx: implement MIDX compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:47Z","receivedAt":"2025-12-06T20:31:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When managing a MIDX chain with many layers, it is convenient to combine\na sequence of adjacent layers into a single layer to prevent the chain\nfrom growing too long.\n\nWhile it is conceptually possible to \"compact\" a sequence of MIDX layers\ntogether by running \"git multi-pack-index write --stdin-packs\", there\nare a few drawbacks that make this less than desirable:\n\n - Preserving the MIDX chain is impossible, since there is no way to\n write a MIDX layer that contains objects or packs found in an earlier\n MIDX layer already part of the chain. So callers would have to write\n an entirely new (non-incremental) MIDX containing only the compacted\n layers, discarding all other objects/packs from the MIDX.\n\n - There is (currently) no way to write a MIDX layer outside of the MIDX\n chain to work around the above, such that the MIDX chain could be\n reassembled substituting the compacted layers with the MIDX that was\n written.\n\n - The `--stdin-packs` command-line option does not allow us to specify\n the order of packs as they appear in the MIDX. Therefore, even if\n there were workarounds for the previous two challenges, any bitmaps\n belonging to layers which come after the compacted layer(s) would no\n longer be valid.\n\nThis commit introduces a way to compact a sequence of adjacent MIDX\nlayers into a single layer while preserving the MIDX chain, as well as\nany bitmap(s) in layers which are newer than the compacted ones.\n\nImplementing MIDX compaction does not require a significant number of\nchanges to how MIDX layers are written. The main changes are as follows:\n\n - Instead of calling `fill_packs_from_midx()`, we call a new function\n `fill_packs_from_midx_range()`, which walks backwards along the\n portion of the MIDX chain which we are compacting, and adds packs one\n layer a time.\n\n In order to preserve the pseudo-pack order, the concatenated pack\n order is preserved, with the exception of preferred packs which are\n always added first.\n\n - After adding entries from the set of packs in the compaction range,\n `compute_sorted_entries()` must adjust the `pack_int_id`'s for all\n objects added in each fanout layer to match their original\n `pack_int_id`'s (as opposed to the index at which each pack appears\n in `ctx.info`).\n\n - When writing out the new 'multi-pack-index-chain' file, discard any\n layers in the compaction range, replacing them with the newly written\n layer, instead of keeping them and placing the new layer at the end\n of the chain.\n\nThis ends up being sufficient to implement MIDX compaction in such a way\nthat preserves bitmaps corresponding to more recent layers in the MIDX\nchain.\n\nThe tests for MIDX compaction are so far fairly spartan, since the main\ninteresting behavior here is ensuring that the right packs/objects are\nselected from each layer, and that the pack order is preserved despite\nwhether or not they are sorted in lexicographic order in the original\nMIDX chain.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 13 ++\n builtin/multi-pack-index.c | 67 +++++++\n midx-write.c | 242 ++++++++++++++++++++++--\n midx.h | 5 +\n t/meson.build | 1 +\n t/t5335-compact-multi-pack-index.sh | 102 ++++++++++\n 6 files changed, 411 insertions(+), 19 deletions(-)\n create mode 100755 t/t5335-compact-multi-pack-index.sh\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 164cf1f2291..a9664e77411 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -12,6 +12,8 @@ SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=]\n+'git multi-pack-index' [] compact [--[no-]incremental]\n+\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -83,6 +85,17 @@ marker).\n \t\tnecessary.\n --\n \n+compact::\n+\tWrite a new MIDX layer containing only objects and packs present\n+\tin the range `` to ``, where both arguments are\n+\tchecksums of existing layers in the MIDX chain.\n++\n+--\n+\t--incremental::\n+\t\tWrite the result to a MIDX chain instead of writing a\n+\t\tstand-alone MIDX. Incompatible with `--bitmap`.\n+--\n+\n verify::\n \tVerify the contents of the MIDX file.\n \ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex c0c6c1760c0..9b0c2082cb3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -17,6 +17,10 @@\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n \t \" [--refs-snapshot=]\")\n \n+#define BUILTIN_MIDX_COMPACT_USAGE \\\n+\tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n+\t \" \")\n+\n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n \n@@ -30,6 +34,10 @@ static char const * const builtin_multi_pack_index_write_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n \tNULL\n };\n+static char const * const builtin_multi_pack_index_compact_usage[] = {\n+\tBUILTIN_MIDX_COMPACT_USAGE,\n+\tNULL\n+};\n static char const * const builtin_multi_pack_index_verify_usage[] = {\n \tBUILTIN_MIDX_VERIFY_USAGE,\n \tNULL\n@@ -44,6 +52,7 @@ static char const * const builtin_multi_pack_index_repack_usage[] = {\n };\n static char const * const builtin_multi_pack_index_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n+\tBUILTIN_MIDX_COMPACT_USAGE,\n \tBUILTIN_MIDX_VERIFY_USAGE,\n \tBUILTIN_MIDX_EXPIRE_USAGE,\n \tBUILTIN_MIDX_REPACK_USAGE,\n@@ -195,6 +204,63 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \treturn ret;\n }\n \n+static int cmd_multi_pack_index_compact(int argc, const char **argv,\n+\t\t\t\t\tconst char *prefix,\n+\t\t\t\t\tstruct repository *repo)\n+{\n+\tstruct multi_pack_index *m, *cur;\n+\tstruct multi_pack_index *from_midx = NULL;\n+\tstruct multi_pack_index *to_midx = NULL;\n+\tstruct odb_source *source;\n+\tint ret;\n+\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_BIT(0, \"incremental\", &opts.flags,\n+\t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_END(),\n+\t};\n+\n+\trepo_config(repo, git_multi_pack_index_write_config, NULL);\n+\n+\toptions = add_common_options(builtin_multi_pack_index_compact_options);\n+\n+\ttrace2_cmd_mode(argv[0]);\n+\n+\tif (isatty(2))\n+\t\topts.flags |= MIDX_PROGRESS;\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t options, builtin_multi_pack_index_compact_usage,\n+\t\t\t 0);\n+\n+\tif (argc != 2)\n+\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n+\t\t\t\t options);\n+\tsource = handle_object_dir_option(the_repository);\n+\n+\tFREE_AND_NULL(options);\n+\n+\tm = get_multi_pack_index(source);\n+\n+\tfor (cur = m; cur && !(from_midx && to_midx); cur = cur->base_midx) {\n+\t\tconst char *midx_csum = get_midx_checksum(cur);\n+\n+\t\tif (!from_midx && !strcmp(midx_csum, argv[0]))\n+\t\t\tfrom_midx = cur;\n+\t\tif (!to_midx && !strcmp(midx_csum, argv[1]))\n+\t\t\tto_midx = cur;\n+\t}\n+\n+\tif (!from_midx)\n+\t\tdie(_(\"could not find MIDX 'from': %s\"), argv[0]);\n+\tif (!to_midx)\n+\t\tdie(_(\"could not find MIDX 'to': %s\"), argv[1]);\n+\n+\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n+\n+\treturn ret;\n+}\n+\n static int cmd_multi_pack_index_verify(int argc, const char **argv,\n \t\t\t\t const char *prefix,\n \t\t\t\t struct repository *repo UNUSED)\n@@ -295,6 +361,7 @@ int cmd_multi_pack_index(int argc,\n \tstruct option builtin_multi_pack_index_options[] = {\n \t\tOPT_SUBCOMMAND(\"repack\", &fn, cmd_multi_pack_index_repack),\n \t\tOPT_SUBCOMMAND(\"write\", &fn, cmd_multi_pack_index_write),\n+\t\tOPT_SUBCOMMAND(\"compact\", &fn, cmd_multi_pack_index_compact),\n \t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_multi_pack_index_verify),\n \t\tOPT_SUBCOMMAND(\"expire\", &fn, cmd_multi_pack_index_expire),\n \t\tOPT_END(),\ndiff --git a/midx-write.c b/midx-write.c\nindex 7854561359d..fcbfedcd913 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -108,6 +108,10 @@ struct write_midx_context {\n \tint incremental;\n \tuint32_t num_multi_pack_indexes_before;\n \n+\tstruct multi_pack_index *compact_from;\n+\tstruct multi_pack_index *compact_to;\n+\tint compact;\n+\n \tstruct string_list *to_include;\n \n \tstruct repository *repo;\n@@ -117,6 +121,8 @@ struct write_midx_context {\n static uint32_t midx_pack_perm(struct write_midx_context *ctx,\n \t\t\t uint32_t orig_pack_int_id)\n {\n+\tif (ctx->compact)\n+\t\torig_pack_int_id -= ctx->compact_from->num_packs_in_base;\n \treturn ctx->pack_perm[orig_pack_int_id];\n }\n \n@@ -347,6 +353,21 @@ static void midx_fanout_add(struct midx_fanout *fanout,\n \t\t\t\t\t cur_fanout);\n }\n \n+static void midx_fanout_add_compact(struct midx_fanout *fanout,\n+\t\t\t\t struct write_midx_context *ctx,\n+\t\t\t\t uint32_t cur_fanout)\n+{\n+\tstruct multi_pack_index *m = ctx->compact_to;\n+\n+\tASSERT(ctx->compact);\n+\n+\twhile (m && m != ctx->compact_from->base_midx) {\n+\t\tmidx_fanout_add_midx_fanout(fanout, m, cur_fanout,\n+\t\t\t\t\t NO_PREFERRED_PACK);\n+\t\tm = m->base_midx;\n+\t}\n+}\n+\n /*\n * It is possible to artificially get into a state where there are many\n * duplicate copies of objects. That can create high memory pressure if\n@@ -365,6 +386,9 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tsize_t alloc_objects, total_objects = 0;\n \tstruct midx_fanout fanout = { 0 };\n \n+\tif (ctx->compact)\n+\t\tASSERT(!start_pack);\n+\n \tfor (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++)\n \t\ttotal_objects = st_add(total_objects,\n \t\t\t\t ctx->info[cur_pack].p->num_objects);\n@@ -383,7 +407,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n \n-\t\tmidx_fanout_add(&fanout, ctx, start_pack, cur_fanout);\n+\t\tif (ctx->compact)\n+\t\t\tmidx_fanout_add_compact(&fanout, ctx, cur_fanout);\n+\t\telse\n+\t\t\tmidx_fanout_add(&fanout, ctx, start_pack, cur_fanout);\n \t\tmidx_fanout_sort(&fanout);\n \n \t\t/*\n@@ -953,6 +980,72 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n \treturn 0;\n }\n \n+static uint32_t compactible_packs_between(const struct multi_pack_index *from,\n+\t\t\t\t\t const struct multi_pack_index *to)\n+{\n+\tuint32_t nr;\n+\n+\tASSERT(from && to);\n+\n+\tnr = u32_add(to->num_packs, to->num_packs_in_base);\n+\tif (nr < from->num_packs_in_base)\n+\t\tBUG(\"unexpected number of packs in base during compaction: \"\n+\t\t \"%\"PRIu32\" < %\"PRIu32, nr, from->num_packs_in_base);\n+\n+\treturn nr - from->num_packs_in_base;\n+}\n+\n+static int fill_packs_from_midx_range(struct write_midx_context *ctx,\n+\t\t\t\t int bitmap_order)\n+{\n+\tstruct multi_pack_index *m = ctx->compact_to;\n+\tuint32_t packs_nr;\n+\n+\tASSERT(ctx->compact && !ctx->nr);\n+\tASSERT(ctx->compact_from);\n+\tASSERT(ctx->compact_to);\n+\n+\tpacks_nr = compactible_packs_between(ctx->compact_from,\n+\t\t\t\t\t ctx->compact_to);\n+\n+\tALLOC_GROW(ctx->info, packs_nr, ctx->alloc);\n+\n+\twhile (m != ctx->compact_from->base_midx) {\n+\t\tuint32_t pack_int_id, preferred_pack_id;\n+\t\tuint32_t i;\n+\n+\t\tif (bitmap_order) {\n+\t\t\tif (midx_preferred_pack(m, &preferred_pack_id) < 0)\n+\t\t\t\tdie(_(\"could not determine preferred pack\"));\n+\t\t} else {\n+\t\t\tpreferred_pack_id = m->num_packs_in_base;\n+\t\t}\n+\n+\t\tpack_int_id = m->num_packs_in_base - ctx->compact_from->num_packs_in_base;\n+\n+\t\tif (fill_pack_from_midx(&ctx->info[pack_int_id++], m,\n+\t\t\t\t\tpreferred_pack_id) < 0)\n+\t\t\treturn -1;\n+\n+\t\tfor (i = m->num_packs_in_base;\n+\t\t i < m->num_packs_in_base + m->num_packs; i++) {\n+\t\t\tif (preferred_pack_id == i)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (fill_pack_from_midx(&ctx->info[pack_int_id++], m,\n+\t\t\t\t\t\ti) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\n+\t\tctx->nr += m->num_packs;\n+\t\tm = m->base_midx;\n+\t}\n+\n+\tASSERT(ctx->nr == packs_nr);\n+\n+\treturn 0;\n+}\n+\n static struct {\n \tconst char *non_split;\n \tconst char *split;\n@@ -1038,12 +1131,22 @@ static void clear_midx_files(struct odb_source *source,\n \tstrbuf_release(&buf);\n }\n \n+static int midx_hashcmp(const struct multi_pack_index *a,\n+\t\t\tconst struct multi_pack_index *b,\n+\t\t\tconst struct git_hash_algo *algop)\n+{\n+\treturn hashcmp(get_midx_hash(a), get_midx_hash(b), algop);\n+}\n+\n struct write_midx_opts {\n \tstruct odb_source *source;\n \n \tstruct string_list *packs_to_include;\n \tstruct string_list *packs_to_drop;\n \n+\tstruct multi_pack_index *compact_from;\n+\tstruct multi_pack_index *compact_to;\n+\n \tconst char *preferred_pack_name;\n \tconst char *refs_snapshot;\n \tunsigned flags;\n@@ -1066,6 +1169,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tint dropped_packs = 0;\n \tint result = -1;\n \tconst char **keep_hashes = NULL;\n+\tsize_t keep_hashes_nr = 0;\n \tstruct chunkfile *cf;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n@@ -1074,6 +1178,17 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tctx.source = opts->source;\n \n \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n+\tctx.compact = !!(opts->flags & MIDX_WRITE_COMPACT);\n+\n+\tif (ctx.compact) {\n+\t\tif (!opts->compact_from)\n+\t\t\tBUG(\"expected non-NULL 'from' MIDX during compaction\");\n+\t\tif (!opts->compact_to)\n+\t\t\tBUG(\"expected non-NULL 'to' MIDX during compaction\");\n+\n+\t\tctx.compact_from = opts->compact_from;\n+\t\tctx.compact_to = opts->compact_to;\n+\t}\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n@@ -1101,11 +1216,18 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t */\n \t\t\tif (ctx.incremental)\n \t\t\t\tctx.base_midx = m;\n-\t\t\telse if (!opts->packs_to_include)\n+\t\t\tif (!opts->packs_to_include)\n \t\t\t\tctx.m = m;\n \t\t}\n \t}\n \n+\t/*\n+\t * If compacting MIDX layer(s) in the range [from, to], then the\n+\t * compacted MIDX will share the same base MIDX as 'from'.\n+\t */\n+\tif (ctx.compact)\n+\t\tctx.base_midx = ctx.compact_from->base_midx;\n+\n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n@@ -1122,7 +1244,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\tctx.num_multi_pack_indexes_before++;\n \t\t\tm = m->base_midx;\n \t\t}\n-\t} else if (ctx.m && fill_packs_from_midx(&ctx)) {\n+\t} else if (ctx.m && !ctx.compact && fill_packs_from_midx(&ctx)) {\n \t\tgoto cleanup;\n \t}\n \n@@ -1135,13 +1257,23 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \telse\n \t\tctx.progress = NULL;\n \n-\tctx.to_include = opts->packs_to_include;\n+\tif (ctx.compact) {\n+\t\tint bitmap_order = 0;\n+\t\tif (opts->preferred_pack_name)\n+\t\t\tbitmap_order |= 1;\n+\t\telse if (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n+\t\t\tbitmap_order |= 1;\n \n-\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n+\t\tfill_packs_from_midx_range(&ctx, bitmap_order);\n+\t} else {\n+\t\tctx.to_include = opts->packs_to_include;\n+\t\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n+\t}\n \tstop_progress(&ctx.progress);\n \n \tif ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) &&\n \t !ctx.incremental &&\n+\t !ctx.compact &&\n \t !(opts->packs_to_include || opts->packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\n \t\tint bitmap_exists;\n@@ -1255,12 +1387,15 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\tctx.large_offsets_needed = 1;\n \t}\n \n-\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n+\tif (!ctx.compact)\n+\t\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n \tif (opts->packs_to_drop && opts->packs_to_drop->nr) {\n \t\tsize_t drop_index = 0;\n \t\tint missing_drops = 0;\n \n+\t\tASSERT(!ctx.compact);\n+\n \t\tfor (size_t i = 0;\n \t\t i < ctx.nr && drop_index < opts->packs_to_drop->nr; i++) {\n \t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n@@ -1292,12 +1427,20 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t */\n \tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n \tfor (size_t i = 0; i < ctx.nr; i++) {\n+\t\tuint32_t from = ctx.info[i].orig_pack_int_id;\n+\t\tuint32_t to;\n+\n \t\tif (ctx.info[i].expired) {\n+\t\t\tto = PACK_EXPIRED;\n \t\t\tdropped_packs++;\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tto = i - dropped_packs;\n \t\t}\n+\n+\t\tif (ctx.compact)\n+\t\t\tfrom -= ctx.compact_from->num_packs_in_base;\n+\n+\t\tctx.pack_perm[from] = to;\n \t}\n \n \tfor (size_t i = 0; i < ctx.nr; i++) {\n@@ -1445,7 +1588,24 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n \t\tdie(_(\"too many multi-pack-indexes\"));\n \n-\tCALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);\n+\tif (ctx.compact) {\n+\t\tstruct multi_pack_index *m;\n+\n+\t\t/*\n+\t\t * Keep all MIDX layers excluding those in the range [from, to].\n+\t\t */\n+\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n+\t\t\tkeep_hashes_nr++;\n+\t\tfor (m = ctx.m;\n+\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n+\t\t m = m->base_midx)\n+\t\t\tkeep_hashes_nr++;\n+\n+\t\tkeep_hashes_nr++; /* include the compacted layer */\n+\t} else {\n+\t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n+\t}\n+\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n \n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n@@ -1470,17 +1630,47 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t\tstrbuf_release(&final_midx_name);\n \n-\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n-\t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n+\t\tif (ctx.compact) {\n+\t\t\tstruct multi_pack_index *m;\n+\t\t\tuint32_t num_layers_before_from = 0;\n+\t\t\tuint32_t i;\n \n-\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n-\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n+\t\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n+\t\t\t\tnum_layers_before_from++;\n \n-\t\t\tkeep_hashes[j] = xstrdup(get_midx_checksum(m));\n-\t\t\tm = m->base_midx;\n+\t\t\tm = ctx.base_midx;\n+\t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n+\t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n+\n+\t\t\t\tkeep_hashes[j] = xstrdup(get_midx_checksum(m));\n+\t\t\t\tm = m->base_midx;\n+\t\t\t}\n+\n+\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t\t\t r->hash_algo));\n+\n+\t\t\ti = 0;\n+\t\t\tfor (m = ctx.m;\n+\t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n+\t\t\t m = m->base_midx) {\n+\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n+\t\t\t\t\txstrdup(get_midx_checksum(m));\n+\t\t\t\ti++;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t\t r->hash_algo));\n+\n+\t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n+\t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n+\n+\t\t\t\tkeep_hashes[j] = xstrdup(get_midx_checksum(m));\n+\t\t\t\tm = m->base_midx;\n+\t\t\t}\n \t\t}\n \n-\t\tfor (uint32_t i = 0; i <= ctx.num_multi_pack_indexes_before; i++)\n+\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n \t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n \t} else {\n \t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n@@ -1493,8 +1683,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, keep_hashes,\n-\t\t\t ctx.num_multi_pack_indexes_before + 1,\n+\tclear_midx_files(opts->source, keep_hashes, keep_hashes_nr,\n \t\t\t ctx.incremental);\n \tresult = 0;\n \n@@ -1512,7 +1701,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tif (keep_hashes) {\n-\t\tfor (uint32_t i = 0; i <= ctx.num_multi_pack_indexes_before; i++)\n+\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n \t\t\tfree((char *)keep_hashes[i]);\n \t\tfree(keep_hashes);\n \t}\n@@ -1553,6 +1742,21 @@ int write_midx_file_only(struct odb_source *source,\n \treturn write_midx_internal(&opts);\n }\n \n+int write_midx_file_compact(struct odb_source *source,\n+\t\t\t struct multi_pack_index *from,\n+\t\t\t struct multi_pack_index *to,\n+\t\t\t unsigned flags)\n+{\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.compact_from = from,\n+\t\t.compact_to = to,\n+\t\t.flags = flags | MIDX_WRITE_COMPACT,\n+\t};\n+\n+\treturn write_midx_internal(&opts);\n+}\n+\n int expire_midx_packs(struct odb_source *source, unsigned flags)\n {\n \tuint32_t i, *count, result = 0;\ndiff --git a/midx.h b/midx.h\nindex 39bf04b18e5..61f9809b8c9 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -81,6 +81,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n+#define MIDX_WRITE_COMPACT (1 << 6)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n@@ -130,6 +131,10 @@ int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n \t\t\t const char *refs_snapshot, unsigned flags);\n+int write_midx_file_compact(struct odb_source *source,\n+\t\t\t struct multi_pack_index *from,\n+\t\t\t struct multi_pack_index *to,\n+\t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct odb_source *source, unsigned flags);\n int expire_midx_packs(struct odb_source *source, unsigned flags);\ndiff --git a/t/meson.build b/t/meson.build\nindex 7c994d4643e..2d1926faaf2 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -613,6 +613,7 @@ integration_tests = [\n 't5332-multi-pack-reuse.sh',\n 't5333-pseudo-merge-bitmaps.sh',\n 't5334-incremental-multi-pack-index.sh',\n+ 't5335-compact-multi-pack-index.sh',\n 't5351-unpack-large-objects.sh',\n 't5400-send-pack.sh',\n 't5401-update-hooks.sh',\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nnew file mode 100755\nindex 00000000000..f889af7fb1d\n--- /dev/null\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -0,0 +1,102 @@\n+#!/bin/sh\n+\n+test_description='multi-pack-index compaction'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+midxdir=$packdir/multi-pack-index.d\n+midx_chain=$midxdir/multi-pack-index-chain\n+\n+nth_line() {\n+\tlocal n=\"$1\"\n+\tshift\n+\tawk \"NR==$n\" \"$@\"\n+}\n+\n+write_packs () {\n+\tfor c in \"$@\"\n+\tdo\n+\t\ttest_commit \"$c\" &&\n+\n+\t\tgit pack-objects --all --unpacked $packdir/pack-$c &&\n+\t\tgit prune-packed &&\n+\n+\t\tgit multi-pack-index write --incremental --bitmap || return 1\n+\tdone\n+}\n+\n+test_midx_layer_packs () {\n+\tlocal checksum=\"$1\" &&\n+\tshift &&\n+\n+\ttest-tool read-midx $objdir \"$checksum\" >out &&\n+\n+\tprintf \"%s\\n\" \"$@\" >expect &&\n+\t# NOTE: do *not* pipe through sort here, we want to ensure the\n+\t# order of packs is preserved during compaction.\n+\tgrep \"^pack-\" out | cut -d\"-\" -f2 >actual &&\n+\n+\ttest_cmp expect actual\n+}\n+\n+test_midx_layer_object_uniqueness () {\n+\t: >objs.all\n+\twhile read layer\n+\tdo\n+\t\ttest-tool read-midx --show-objects $objdir \"$layer\" >out &&\n+\t\tgrep \"\\.pack$\" out | cut -d\" \" -f1 | sort >objs.layer &&\n+\t\ttest_stdout_line_count = 0 comm -12 objs.all objs.layer &&\n+\t\tcat objs.all objs.layer | sort >objs.tmp &&\n+\t\tmv objs.tmp objs.all || return 1\n+\tdone <$midx_chain\n+}\n+\n+test_expect_success 'MIDX compaction with lex-ordered pack names' '\n+\tgit init midx-compact-lex-order &&\n+\t(\n+\t\tcd midx-compact-lex-order &&\n+\n+\t\twrite_packs A B C D E &&\n+\t\ttest_line_count = 5 $midx_chain &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\ttest_midx_layer_packs \"$(nth_line 1 \"$midx_chain\")\" A &&\n+\t\ttest_midx_layer_packs \"$(nth_line 2 \"$midx_chain\")\" B C D &&\n+\t\ttest_midx_layer_packs \"$(nth_line 3 \"$midx_chain\")\" E &&\n+\n+\t\ttest_midx_layer_object_uniqueness\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with non-lex-ordered pack names' '\n+\tgit init midx-compact-non-lex-order &&\n+\t(\n+\t\tcd midx-compact-non-lex-order &&\n+\n+\t\twrite_packs D C A B E &&\n+\t\ttest_line_count = 5 $midx_chain &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\ttest_midx_layer_packs \"$(nth_line 1 \"$midx_chain\")\" D &&\n+\t\ttest_midx_layer_packs \"$(nth_line 2 \"$midx_chain\")\" C A B &&\n+\t\ttest_midx_layer_packs \"$(nth_line 3 \"$midx_chain\")\" E &&\n+\n+\t\ttest_midx_layer_object_uniqueness\n+\t)\n+'\n+\n+test_done\n-- \n2.52.0.171.gd6a4e6b6955\n\n"},{"id":"531781","messageId":"d6a4e6b69552d8626e6df2d7cd042b9fe151914a.1765053054.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH 17/17] midx: enable reachability bitmaps during MIDX compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-06T20:31:50Z","receivedAt":"2025-12-06T20:31:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Enable callers to generate reachability bitmaps when performing MIDX\nlayer compaction by combining all existing bitmaps from the compacted\nlayers.\n\nNote that the because of the object/pack ordering described by the\nprevious commit, the pseudo-pack order for the compacted MIDX is the\nsame as concatenating the individual pseudo-pack orderings for each\nlayer in the compaction range.\n\nAs a result, the only non-test or documentation change necessary is to\ntreat all objects as non-preferred during compaction so as not to\ndisturb the object ordering.\n\nIn the future, we may want to adjust which commit(s) receive\nreachability bitmaps when compacting multiple .bitmap files into one, or\neven generate new bitmaps (e.g., if the references have moved\nsignificantly since the .bitmap was generated). This commit only\nimplements combining all existing bitmaps in range together in order to\ndemonstrate and lay the groundwork for more exotic strategies.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 2 +-\n builtin/multi-pack-index.c | 4 +-\n midx-write.c | 2 +-\n t/t5335-compact-multi-pack-index.sh | 120 +++++++++++++++++++++++-\n 4 files changed, 123 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex a9664e77411..458bb873633 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t \n+\t\t [--[no-]bitmap] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 9b0c2082cb3..40afa8f1ed8 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -19,7 +19,7 @@\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" \")\n+\t \" [--[no-]bitmap] \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -216,6 +216,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n+\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_END(),\ndiff --git a/midx-write.c b/midx-write.c\nindex fcbfedcd913..f2dbacef4cd 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -657,7 +657,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \t\tstruct pack_midx_entry *e = &ctx->entries[i];\n \t\tdata[i].nr = i;\n \t\tdata[i].pack = midx_pack_perm(ctx, e->pack_int_id);\n-\t\tif (!e->preferred)\n+\t\tif (!e->preferred || ctx->compact)\n \t\t\tdata[i].pack |= (1U << 31);\n \t\tdata[i].offset = e->offset;\n \t}\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex f889af7fb1d..a306f504305 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -65,7 +65,7 @@ test_expect_success 'MIDX compaction with lex-ordered pack names' '\n \t\twrite_packs A B C D E &&\n \t\ttest_line_count = 5 $midx_chain &&\n \n-\t\tgit multi-pack-index compact --incremental \\\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n \t\ttest_line_count = 3 $midx_chain &&\n@@ -86,7 +86,7 @@ test_expect_success 'MIDX compaction with non-lex-ordered pack names' '\n \t\twrite_packs D C A B E &&\n \t\ttest_line_count = 5 $midx_chain &&\n \n-\t\tgit multi-pack-index compact --incremental \\\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n \t\ttest_line_count = 3 $midx_chain &&\n@@ -99,4 +99,120 @@ test_expect_success 'MIDX compaction with non-lex-ordered pack names' '\n \t)\n '\n \n+midx_objs_by_pack () {\n+\tawk '/\\.pack$/ { split($3, a, \"-\"); print a[2], $1 }' | sort\n+}\n+\n+tag_objs_from_pack () {\n+\tobjs=\"$(git rev-list --objects --no-object-names \"$2\")\" &&\n+\tprintf \"$1 %s\\n\" $objs | sort\n+}\n+\n+test_expect_success 'MIDX compaction preserves pack object selection' '\n+\tgit init midx-compact-preserve-selection &&\n+\t(\n+\t\tcd midx-compact-preserve-selection &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\t# Create two packs, one containing just the objects from\n+\t\t# A, and another containing all objects from the\n+\t\t# repository.\n+\t\tp1=\"$(echo A | git pack-objects --revs --delta-base-offset \\\n+\t\t\t$packdir/pack-1)\" &&\n+\t\tp0=\"$(echo B | git pack-objects --revs --delta-base-offset \\\n+\t\t\t$packdir/pack-0)\" &&\n+\n+\t\techo \"pack-1-$p1.idx\" | git multi-pack-index write \\\n+\t\t\t--incremental --bitmap --stdin-packs &&\n+\t\techo \"pack-0-$p0.idx\" | git multi-pack-index write \\\n+\t\t\t--incremental --bitmap --stdin-packs &&\n+\n+\t\twrite_packs C &&\n+\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" &&\n+\n+\n+\t\ttest-tool read-midx --show-objects $objdir \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" >AB.info &&\n+\t\ttest-tool read-midx --show-objects $objdir \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" >C.info &&\n+\n+\t\tmidx_objs_by_pack AB.actual &&\n+\t\tmidx_objs_by_pack C.actual &&\n+\n+\t\t{\n+\t\t\ttag_objs_from_pack 1 A &&\n+\t\t\ttag_objs_from_pack 0 A..B\n+\t\t} | sort >AB.expect &&\n+\t\ttag_objs_from_pack C B..C >C.expect &&\n+\n+\t\ttest_cmp AB.expect AB.actual &&\n+\t\ttest_cmp C.expect C.actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bitmaps' '\n+\tgit init midx-compact-with-bitmaps &&\n+\t(\n+\t\tcd midx-compact-with-bitmaps &&\n+\n+\t\twrite_packs foo bar baz quux woot &&\n+\n+\t\ttest-tool read-midx --bitmap $objdir >bitmap.expect &&\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest-tool read-midx --bitmap $objdir >bitmap.actual &&\n+\n+\t\ttest_cmp bitmap.expect bitmap.actual &&\n+\n+\t\ttrue\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bitmaps (non-trivial)' '\n+\tgit init midx-compact-with-bitmaps-non-trivial &&\n+\t(\n+\t\tcd midx-compact-with-bitmaps-non-trivial &&\n+\n+\t\tgit branch -m main &&\n+\n+\t\t# D(4)\n+\t\t# /\n+\t\t# A(1) --- B(2) --- C(3) --- G(7)\n+\t\t# \\\n+\t\t# E(5) --- F(6)\n+\t\twrite_packs A B C &&\n+\t\tgit checkout -b side &&\n+\t\twrite_packs D &&\n+\t\tgit checkout -b other B &&\n+\t\twrite_packs E F &&\n+\t\tgit checkout main &&\n+\t\twrite_packs G &&\n+\n+\t\tcat $midx_chain &&\n+\n+\t\t# Compact layers 2-4, leaving us with:\n+\t\t#\n+\t\t# [A, [B, C, D], E, F, G]\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\n+\t\t# Then compact the top two layers, condensing the above\n+\t\t# such that the new 4th layer contains F and G.\n+\t\t#\n+\t\t# [A, [B, C, D], E, [F, G]]\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 5 \"$midx_chain\")\" &&\n+\n+\t\tcat $midx_chain\n+\t)\n+'\n+\n test_done\n-- \n2.52.0.171.gd6a4e6b6955\n"},{"id":"531843","messageId":"aTcYAdJ6lNCST7PT@pks.im","threadId":"64585","inReplyTo":"d583a169f08eea9d750622c6e3cf79774a2a68e7.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 02/17] midx: split `get_midx_checksum()` by adding `get_midx_hash()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-08T18:25:05Z","receivedAt":"2025-12-08T18:25:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:04PM -0500, Taylor Blau wrote:\n> When trying to print out, say, the hexadecimal representation of a\n> MIDX's hash, our code will do something like:\n> \n> hash_to_hex_algop(get_midx_checksum(m),\n> m->source->odb->repo->hash_algo);\n> \n> , which is both cumbersome and repetitive. In fact, all but a handful of\n> callers to `get_midx_checksum()` do exactly the above. Reduce the\n> repetitive nature of calling `get_midx_checksum()` by having it return a\n> pointer into a static buffer containing the above result.\n> \n> For the handful of callers that do need to compare the raw bytes and\n> don't want to deal with an encoded copy (e.g., because they are passing\n> it to hasheq() or similar), introduce `get_midx_hash()` which returns\n> the raw bytes.\n\nThis is a welcome change indeed.\n\n> diff --git a/midx.h b/midx.h\n> index 7c7e0b59121..e188ffeb578 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -85,7 +85,8 @@ struct multi_pack_index {\n> #define MIDX_EXT_BITMAP \"bitmap\"\n> #define MIDX_EXT_MIDX \"midx\"\n> \n> -const unsigned char *get_midx_checksum(const struct multi_pack_index *m);\n> +const char *get_midx_checksum(const struct multi_pack_index *m) /* static buffer */;\n> +const unsigned char *get_midx_hash(const struct multi_pack_index *m);\n> void get_midx_filename(struct odb_source *source, struct strbuf *out);\n> void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\n> \t\t\t const unsigned char *hash, const char *ext);\n\nIf I didn't have the context of this patch series I would be wondering\nwhat the actual difference between `get_midx_checksum()` and\n`get_midx_hash()` is. The way the functions are named seems to rather\nindicate that we talk about two different kinds of hashes, rather than\ntwo different ways to encode them.\n\nWould it maybe be preferable to call them `get_midx_checksum()` and\n`get_midx_checksum_hex()`? While at it, we could go even further and\nrename them to `midx_get_checksum()` and `midx_get_checksum_hex()` to\nconform to our modern best practices.\n\nPatrick\n"},{"id":"531844","messageId":"aTcYU_yVYyXL9TXv@pks.im","threadId":"64585","inReplyTo":"1fc359d9a98cdd61f6986fa1d9a1190f57dacb9b.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 07/17] midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-08T18:26:27Z","receivedAt":"2025-12-08T18:26:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:19PM -0500, Taylor Blau wrote:\n> In midx_pack_order(), we compute for each bitampped pack the first bit\n\ns/bitampped/bitmapped/\n\n> to correspond to an object in that pack, along with how many bits were\n> assigned to object(s) in that pack.\n> \n> Initially, each bitmap_nr value is set to zero, and each bitmap_pos\n\nI assume `bitmap_nr` is the number of bits, whereas `bitmap_pos` is the\nposition of the first bit?\n\n> value is set to the sentinel BITMAP_POS_UNKNOWN. This is done to ensure\n> that there are no packs who have an unknown bit position but a somehow\n> non-zero number of objects (cf. `write_midx_bitmapped_packs()` in\n> midx-write.c).\n> \n> Once the pack order is fully determined, midx_pack_order() sets the\n> bitmap_pos field for any bitmapped packs to zero if they are still\n> listed as BITMAP_POS_UNKNOWN.\n\nIf so, it feels somewhat weird that we'd set the `bitmap_pos` to zero.\nBut I guess it doesn't matter anyway, as I assume that the `bitmap_nr`\nwould be zero anyway?\n\nAnyway, reading on.\n\n> However, we enumerate the bitmapped packs in order of `ctx->pack_perm`.\n\nWhich is the \"permutation between pack-int-ids from the previous\nmulti-pack-index to the new one we are writing\"'. So it's basically\ntracking which new packs correspond to the old packs.\n\n> This is fine for existing cases, since the only time the\n> `ctx->pack_perm` array holds a value outside of the addressable range of\n> `ctx->info` is when there are expired packs, which only occurs via 'git\n> multi-pack-index expire', which does not support writing MIDX bitmaps.\n> As a result, the range of ctx->pack_perm covers all values in [0,\n> `ctx->nr`), so enumerating in this order isn't an issue.\n> \n> A future change necessary for compaction will complicate this further by\n> introducing a wrapper around the `ctx->pack_perm` array, which turns the\n> given `pack_int_id` into one that is relative to the lower end of the\n> compaction range. As a result, indexing into `ctx->pack_perm` through\n> this helper, say, with \"0\" will produce a crash when the lower end of\n> the compaction range has >0 pack(s) in its base layer, since the\n> subtraction will wrap around the 32-bit unsigned range, resulting in an\n> uninitialized read.\n> \n> But the process is completely unnecessary in the first place: we are\n> enumerating all values of `ctx->info`, and there is no reason to process\n> them in a different order than they appear in memory. Index `ctx->info`\n> directly to reflect that.\n\nFair. We do initialize the permutations like this:\n\n\tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n\tfor (size_t i = 0; i < ctx.nr; i++) {\n\t\tif (ctx.info[i].expired) {\n\t\t\tdropped_packs++;\n\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n\t\t} else {\n\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n\t\t}\n\t}\n\nSo obviously, the permutation will only ever be different in case we've\ngot at least one dropped pack, and that only happens when we expire any\npacks. So the explanation matches.\n\nOf course it may be a bit more fragile now if we ever added a caller\nof this function that _does_ expire data. But we don't have any, so that\nenters the territory of overthinking things.\n\n> diff --git a/midx-write.c b/midx-write.c\n> index 73d24fabbc6..c30f6a70d37 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -637,7 +637,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n> \t\tpack_order[i] = data[i].nr;\n> \t}\n> \tfor (i = 0; i < ctx->nr; i++) {\n> -\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n> +\t\tstruct pack_info *pack = &ctx->info[i];\n> \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n> \t\t\tpack->bitmap_pos = 0;\n> \t}\n\nThe change looks simple enough.\n\nPatrick\n"},{"id":"531845","messageId":"aTcYXJr_-IxPmC65@pks.im","threadId":"64585","inReplyTo":"47aae3bf2a83a2724aecd3314f8cc5d47e8013f1.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 08/17] midx-write.c: introduce `struct write_midx_opts`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-08T18:26:36Z","receivedAt":"2025-12-08T18:26:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:22PM -0500, Taylor Blau wrote:\n> diff --git a/midx-write.c b/midx-write.c\n> index c30f6a70d37..b262631ae45 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -1014,14 +1014,20 @@ static void clear_midx_files(struct odb_source *source,\n> \tstrbuf_release(&buf);\n> }\n> \n> -static int write_midx_internal(struct odb_source *source,\n> -\t\t\t struct string_list *packs_to_include,\n> -\t\t\t struct string_list *packs_to_drop,\n> -\t\t\t const char *preferred_pack_name,\n> -\t\t\t const char *refs_snapshot,\n> -\t\t\t unsigned flags)\n> +struct write_midx_opts {\n> +\tstruct odb_source *source;\n> +\n> +\tstruct string_list *packs_to_include;\n> +\tstruct string_list *packs_to_drop;\n> +\n> +\tconst char *preferred_pack_name;\n> +\tconst char *refs_snapshot;\n> +\tunsigned flags;\n> +};\n> +\n> +static int write_midx_internal(struct write_midx_opts *opts)\n> {\n> -\tstruct repository *r = source->odb->repo;\n> +\tstruct repository *r = opts->source->odb->repo;\n> \tstruct strbuf midx_name = STRBUF_INIT;\n> \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n> \tuint32_t start_pack;\n\nOne might argue that parameters which _must_ be passed could be moved\nout of the structure and into the function signature, and as far as I\nunderstand, that would only be the `struct odb_source`. After all, we\nare talking about options, and a mandatory field is not really an option\nin my book. It also makes the interface at least a tiny bit more self\ndocumenting.\n\nOther than that this patch looks like a nice improvement to me.\n\n> @@ -1566,8 +1586,11 @@ int expire_midx_packs(struct odb_source *source, unsigned flags)\n> \tfree(count);\n> \n> \tif (packs_to_drop.nr)\n> -\t\tresult = write_midx_internal(source, NULL,\n> -\t\t\t\t\t &packs_to_drop, NULL, NULL, flags);\n> +\t\tresult = write_midx_internal(&(struct write_midx_opts) {\n> +\t\t\t\t\t .source = source,\n> +\t\t\t\t\t .packs_to_drop = &packs_to_drop,\n> +\t\t\t\t\t .flags = flags & MIDX_PROGRESS,\n> +\t\t\t\t\t });\n> \n> \tstring_list_clear(&packs_to_drop, 0);\n> \n\nI think this syntax is not allowed in our codebase except for a test\nballoon just yet. See aso 9b2527caa4 (CodingGuidelines: document test\nballoons in flight, 2025-07-23):\n\n since late 2024 with v2.48.0-rc0~20, we have test balloons for\n compound literal syntax, e.g., (struct foo){ .member = value };\n our hope is that no platforms we care about have trouble using\n them, and officially adopt its wider use in mid 2026. Do not add\n more use of the syntax until that happens.\n\n> @@ -1774,8 +1797,10 @@ int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags)\n> \t\tgoto cleanup;\n> \t}\n> \n> -\tresult = write_midx_internal(source, NULL, NULL, NULL, NULL,\n> -\t\t\t\t flags);\n> +\tresult = write_midx_internal(&(struct write_midx_opts) {\n> +\t\t\t\t .source = source,\n> +\t\t\t\t .flags = flags,\n> +\t\t\t\t });\n\nSame here.\n\nPatrick\n"},{"id":"531846","messageId":"aTcXKE9ZW2Um92n3@pks.im","threadId":"64585","inReplyTo":"8cd8f6e8420a52030f92166d5a20387183cdbb6b.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 01/17] midx: mark `get_midx_checksum()` arguments as const","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-08T18:26:45Z","receivedAt":"2025-12-08T18:26:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:01PM -0500, Taylor Blau wrote:\n> To make clear that the fucntion `get_midx_checksum()` does not do\n\nNit: s/fucntion/functino\n\n> anything to modify its argument, mark the MIDX pointer as const.\n> \n> The following commit will rename this function altogether to make clear\n> that it returns the raw bytes of the checksum, not a hex-encoded copy of\n> it.\n\nPatrick\n"},{"id":"531847","messageId":"aTcYbRt-aUIcym77@pks.im","threadId":"64585","inReplyTo":"d5389a93b16a4933b0c54f78f2d5ce84b9ecac53.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 09/17] midx: do not require packs to be sorted in lexicographic order","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-08T18:26:53Z","receivedAt":"2025-12-08T18:27:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:25PM -0500, Taylor Blau wrote:\n> Note that this produces MIDXs which may be incompatible with earlier\n> versions of Git that have stricter requirements on the layout of packs\n> within a MIDX. This patch does *not* modify the version number of the\n> MIDX format, since existing versions of Git already know to gracefully\n> ignore a MIDX with packs that appear out-of-order.\n\nInteresting. Did you verify how other implementations of Git behave if\nwe start to relax this requirement? It seems like a somewhat dangerous\nassumption to me that this will just continue to work.\n\nAlso, is there a reason why you prefer this over bumping the version\nnumber?\n\n> @@ -656,17 +652,37 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n> \treturn strcmp(idx_or_pack_name, idx_name);\n> }\n> \n> +\n> +static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n> +{\n> +\tstruct multi_pack_index *m = m_;\n> +\treturn strcmp(m->pack_names[*(const size_t *)a],\n> +\t\t m->pack_names[*(const size_t *)b]);\n> +}\n> +\n> static int midx_contains_pack_1(struct multi_pack_index *m,\n> \t\t\t\tconst char *idx_or_pack_name)\n> {\n> \tuint32_t first = 0, last = m->num_packs;\n> \n> +\tif (!m->pack_names_sorted) {\n> +\t\tuint32_t i;\n> +\n> +\t\tALLOC_ARRAY(m->pack_names_sorted, m->num_packs);\n> +\n> +\t\tfor (i = 0; i < m->num_packs; i++)\n> +\t\t\tm->pack_names_sorted[i] = i;\n> +\n> +\t\tQSORT_S(m->pack_names_sorted, m->num_packs, midx_pack_names_cmp,\n> +\t\t\tm);\n> +\t}\n> +\n> \twhile (first < last) {\n> \t\tuint32_t mid = first + (last - first) / 2;\n> \t\tconst char *current;\n> \t\tint cmp;\n> \n> -\t\tcurrent = m->pack_names[mid];\n> +\t\tcurrent = m->pack_names[m->pack_names_sorted[mid]];\n> \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n> \t\tif (!cmp)\n> \t\t\treturn 1;\n\nI assume that it cannot happen that we append to the array of MIDX'd\npacks after we have sorted. It would mean that the MIDX somehow changed\nits representation or was amended to, which isn't possible.\n\nPatrick \n"},{"id":"531848","messageId":"aTcYdebOuUfrAAJJ@pks.im","threadId":"64585","inReplyTo":"776eb88905f53dd929e5b7cb23593976a2f1aad3.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 10/17] git-compat-util.h: introduce `u32_add()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-08T18:27:01Z","receivedAt":"2025-12-08T18:27:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:28PM -0500, Taylor Blau wrote:\n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index 398e0fac4fa..a7aa5f05fc9 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -670,6 +670,14 @@ static inline int cast_size_t_to_int(size_t a)\n> \treturn (int)a;\n> }\n> \n> +static inline uint32_t u32_add(uint32_t a, uint32_t b)\n> +{\n> +\tif (unsigned_add_overflows(a, b))\n> +\t\tdie(\"uint32_t overflow: %\"PRIuMAX\" + %\"PRIuMAX,\n> +\t\t (uintmax_t)a, (uintmax_t)b);\n> +\treturn a + b;\n> +}\n\nWe already use PRIu32 in our codebase, so why is the cast necessary?\n\nPatrick\n"},{"id":"531849","messageId":"aTcYfOZ4_GdIwM9b@pks.im","threadId":"64585","inReplyTo":"8288afd8cc40a56999353643f49c48d610e6b3aa.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 13/17] midx-write.c: enumerate `pack_int_id` values directly","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-08T18:27:08Z","receivedAt":"2025-12-08T18:27:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:37PM -0500, Taylor Blau wrote:\n> Our `midx-write.c::fill_packs_from_midx()` function currently enumerates\n> the range [0, m->num_packs), and then shifts its index variable up by\n> `m->num_packs_in_base` to produce a valid `pack_int_id`.\n> \n> Instead, directly enumerate the range:\n> \n> [m->num_packs_in_base, m->num_packs_in_base + m->num_packs)\n> \n> , which are the original pack_int_ids themselves as opposed to the\n> indexes of those packs relative to the MIDX layer they are contained\n> within.\n\nSensible. I was confused a bit by the previous change because I couldn't\nquite spot the shift happening. I think this makes things a bit easier\nto read.\n\nPatrick\n"},{"id":"531850","messageId":"aTcYhKOIu7ebJ_xV@pks.im","threadId":"64585","inReplyTo":"43ef3d0013f1534c7ff8b1a7c4a17fb0c9b68684.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 15/17] t/helper/test-read-midx.c: plug memory leak when selecting layer","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-08T18:27:16Z","receivedAt":"2025-12-08T18:27:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:43PM -0500, Taylor Blau wrote:\n> diff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\n> index dee603b3cd0..6e03aabca79 100644\n> --- a/t/helper/test-read-midx.c\n> +++ b/t/helper/test-read-midx.c\n> @@ -26,9 +26,10 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n> \t\t\t int show_objects)\n> {\n> \tuint32_t i;\n> -\tstruct multi_pack_index *m;\n> +\tstruct multi_pack_index *m, *tip;\n> +\tint ret = 0;\n> \n> -\tm = setup_midx(object_dir);\n> +\tm = tip = setup_midx(object_dir);\n> \n> \tif (!m)\n> \t\treturn 1;\n\nI was briefly wondering whether we should also convert this into a `goto\nout`. It's of course not required, as setting up the MIDX has just\nfailed. But it would simplfy things a bit as we now have a single exit\npath, only.\n\nI don't mind this too much though.\n\n> @@ -36,8 +37,11 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n> \tif (checksum) {\n> \t\twhile (m && strcmp(get_midx_checksum(m), checksum))\n> \t\t\tm = m->base_midx;\n> -\t\tif (!m)\n> -\t\t\treturn 1;\n> +\t\tif (!m) {\n> +\t\t\tret = error(_(\"could not find MIDX with checksum %s\"),\n> +\t\t\t\t checksum);\n> +\t\t\tgoto out;\n> +\t\t}\n> \t}\n> \n> \tprintf(\"header: %08x %d %d %d %d\\n\",\n\nWe change the return code from 1 to -1, but that ultimately shouldn't\nmatter much.\n\nI'll stop reviewing here and will have a look at the remaining two\npatches with some fresh eyes. But so far this was a nice read, thanks!\n\nPatrick\n"},{"id":"531866","messageId":"aTd+RZpVxwRTM+1h@nand.local","threadId":"64585","inReplyTo":"aTcXKE9ZW2Um92n3@pks.im","subject":"Re: [PATCH 01/17] midx: mark `get_midx_checksum()` arguments as const","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T01:41:25Z","receivedAt":"2025-12-09T01:41:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 07:26:45PM +0100, Patrick Steinhardt wrote:\n> On Sat, Dec 06, 2025 at 03:31:01PM -0500, Taylor Blau wrote:\n> > To make clear that the fucntion `get_midx_checksum()` does not do\n>\n> Nit: s/fucntion/functino\n\ns/functino/function, but otherwise ;-).\n\nThanks,\nTaylor\n"},{"id":"531867","messageId":"aTd+fxpflcLuJXDn@nand.local","threadId":"64585","inReplyTo":"aTcYAdJ6lNCST7PT@pks.im","subject":"Re: [PATCH 02/17] midx: split `get_midx_checksum()` by adding `get_midx_hash()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T01:42:23Z","receivedAt":"2025-12-09T01:42:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 07:25:05PM +0100, Patrick Steinhardt wrote:\n> If I didn't have the context of this patch series I would be wondering\n> what the actual difference between `get_midx_checksum()` and\n> `get_midx_hash()` is. The way the functions are named seems to rather\n> indicate that we talk about two different kinds of hashes, rather than\n> two different ways to encode them.\n>\n> Would it maybe be preferable to call them `get_midx_checksum()` and\n> `get_midx_checksum_hex()`? While at it, we could go even further and\n> rename them to `midx_get_checksum()` and `midx_get_checksum_hex()` to\n> conform to our modern best practices.\n\nYeah, I think those are both reasonable suggestions; I'll apply those\nlocally, thanks!\n\n> Patrick\n\nThanks,\nTaylor\n"},{"id":"531872","messageId":"aTeAUKrB89RM9p+i@nand.local","threadId":"64585","inReplyTo":"aTd+fxpflcLuJXDn@nand.local","subject":"Re: [PATCH 02/17] midx: split `get_midx_checksum()` by adding `get_midx_hash()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T01:50:08Z","receivedAt":"2025-12-09T01:50:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 08:42:23PM -0500, Taylor Blau wrote:\n> On Mon, Dec 08, 2025 at 07:25:05PM +0100, Patrick Steinhardt wrote:\n> > If I didn't have the context of this patch series I would be wondering\n> > what the actual difference between `get_midx_checksum()` and\n> > `get_midx_hash()` is. The way the functions are named seems to rather\n> > indicate that we talk about two different kinds of hashes, rather than\n> > two different ways to encode them.\n> >\n> > Would it maybe be preferable to call them `get_midx_checksum()` and\n> > `get_midx_checksum_hex()`? While at it, we could go even further and\n> > rename them to `midx_get_checksum()` and `midx_get_checksum_hex()` to\n> > conform to our modern best practices.\n>\n> Yeah, I think those are both reasonable suggestions; I'll apply those\n> locally, thanks!\n\nHmm. Upon further thinking, I wonder which function should be named\nwhich.\n\nI think the _checksum() variant suggests that it returns the non-hex\nencoded form, while the _hex() variant suggests the opposite.\nUnfortunately, the latter is both more commonly used and more characters\nto type ;-).\n\nI wonder if there are shorter names available. Perhaps\nmidx_get_checksum() and midx_get_checksum_raw()?\n\nThanks,\nTaylor\n"},{"id":"531873","messageId":"aTeCmrLnRtYRm/ah@nand.local","threadId":"64585","inReplyTo":"aTcYU_yVYyXL9TXv@pks.im","subject":"Re: [PATCH 07/17] midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T01:59:54Z","receivedAt":"2025-12-09T01:59:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 07:26:27PM +0100, Patrick Steinhardt wrote:\n> On Sat, Dec 06, 2025 at 03:31:19PM -0500, Taylor Blau wrote:\n> > In midx_pack_order(), we compute for each bitampped pack the first bit\n>\n> s/bitampped/bitmapped/\n\nUgh. My \"bitamp\" typo strikes again, thanks for spotting!\n\n> > to correspond to an object in that pack, along with how many bits were\n> > assigned to object(s) in that pack.\n> >\n> > Initially, each bitmap_nr value is set to zero, and each bitmap_pos\n>\n> I assume `bitmap_nr` is the number of bits, whereas `bitmap_pos` is the\n> position of the first bit?\n\nThat's right!\n\n> > However, we enumerate the bitmapped packs in order of `ctx->pack_perm`.\n>\n> Which is the \"permutation between pack-int-ids from the previous\n> multi-pack-index to the new one we are writing\"'. So it's basically\n> tracking which new packs correspond to the old packs.\n\nDitto.\n\n> So obviously, the permutation will only ever be different in case we've\n> got at least one dropped pack, and that only happens when we expire any\n> packs. So the explanation matches.\n>\n> Of course it may be a bit more fragile now if we ever added a caller\n> of this function that _does_ expire data. But we don't have any, so that\n> enters the territory of overthinking things.\n\nI think that with incremental MIDXs we will never have such a caller\nwithout a mechanism to tombstone objects in existing packs, but\ndefinitely worth calling out.\n\n> > diff --git a/midx-write.c b/midx-write.c\n> > index 73d24fabbc6..c30f6a70d37 100644\n> > --- a/midx-write.c\n> > +++ b/midx-write.c\n> > @@ -637,7 +637,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n> > \t\tpack_order[i] = data[i].nr;\n> > \t}\n> > \tfor (i = 0; i < ctx->nr; i++) {\n> > -\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n> > +\t\tstruct pack_info *pack = &ctx->info[i];\n> > \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n> > \t\t\tpack->bitmap_pos = 0;\n> > \t}\n>\n> The change looks simple enough.\n\nYeah, I almost wonder if the commit message was more harmful than not.\nThe main points that I wanted to get across were:\n\n - Ultimately we want to enumerate a list, and there's no reason to do\n that in a permuted order.\n\n - Iterating in that permuted order is fine today because the array of\n values in ctx->pack_perm are always addressable indices into\n ctx->info.\n\n - That won't be the case in the future when we are combining packs from\n MIDX layers that have a non-zero m->num_packs_in_base, so adjusting\n the implementation now prevents us from running into that pitfall in\n such a future.\n\nLet me know if you think that I should adjust the commit message here.\nIt's hard to know whether something resembling the above is better or\nworse than the current version of the commit message from a reviewer's\nperspective, so I'm happy to do whatever you think is cleaner ;-).\n\nThanks,\nTaylor\n"},{"id":"531874","messageId":"aTeDqfOlDK9terAD@nand.local","threadId":"64585","inReplyTo":"aTcYXJr_-IxPmC65@pks.im","subject":"Re: [PATCH 08/17] midx-write.c: introduce `struct write_midx_opts`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T02:04:25Z","receivedAt":"2025-12-09T02:04:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 07:26:36PM +0100, Patrick Steinhardt wrote:\n> One might argue that parameters which _must_ be passed could be moved\n> out of the structure and into the function signature, and as far as I\n> understand, that would only be the `struct odb_source`. After all, we\n> are talking about options, and a mandatory field is not really an option\n> in my book. It also makes the interface at least a tiny bit more self\n> documenting.\n>\n> Other than that this patch looks like a nice improvement to me.\n\nI think that's a reasonable consideration. My preference would be to\nkeep everything contained within the 'struct write_midx_opts' since it\nmakes it really easy to pass everything you might need around to other\nsub-routines by just passing a single pointer.\n\nSo I'm inclined to keep the new API as-is presented here, but I'm happy\nto discuss changing it around if you feel strongly about it.\n\nAs a reasonable middle-ground, I added a \"/* non-optional */\" next to\nthe \"source\" member within the new structure.\n\n> > @@ -1566,8 +1586,11 @@ int expire_midx_packs(struct odb_source *source, unsigned flags)\n> > \tfree(count);\n> >\n> > \tif (packs_to_drop.nr)\n> > -\t\tresult = write_midx_internal(source, NULL,\n> > -\t\t\t\t\t &packs_to_drop, NULL, NULL, flags);\n> > +\t\tresult = write_midx_internal(&(struct write_midx_opts) {\n> > +\t\t\t\t\t .source = source,\n> > +\t\t\t\t\t .packs_to_drop = &packs_to_drop,\n> > +\t\t\t\t\t .flags = flags & MIDX_PROGRESS,\n> > +\t\t\t\t\t });\n> >\n> > \tstring_list_clear(&packs_to_drop, 0);\n> >\n>\n> I think this syntax is not allowed in our codebase except for a test\n> balloon just yet. See aso 9b2527caa4 (CodingGuidelines: document test\n> balloons in flight, 2025-07-23):\n>\n> since late 2024 with v2.48.0-rc0~20, we have test balloons for\n> compound literal syntax, e.g., (struct foo){ .member = value };\n> our hope is that no platforms we care about have trouble using\n> them, and officially adopt its wider use in mid 2026. Do not add\n> more use of the syntax until that happens.\n>\n> > @@ -1774,8 +1797,10 @@ int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags)\n> > \t\tgoto cleanup;\n> > \t}\n> >\n> > -\tresult = write_midx_internal(source, NULL, NULL, NULL, NULL,\n> > -\t\t\t\t flags);\n> > +\tresult = write_midx_internal(&(struct write_midx_opts) {\n> > +\t\t\t\t .source = source,\n> > +\t\t\t\t .flags = flags,\n> > +\t\t\t\t });\n>\n> Same here.\n\nHah, I even remember checking to make sure there wasn't such a test\nballoon and then thinking that I'd need to adjust before sending. That\nmust have been just before I got up from my desk, and I must have\nforgotten about it until now. Fixed up locally, thanks for spotting!\n\nThanks,\nTaylor\n"},{"id":"531875","messageId":"aTeEZX4036A9YecX@nand.local","threadId":"64585","inReplyTo":"aTcYbRt-aUIcym77@pks.im","subject":"Re: [PATCH 09/17] midx: do not require packs to be sorted in lexicographic order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T02:07:33Z","receivedAt":"2025-12-09T02:07:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 07:26:53PM +0100, Patrick Steinhardt wrote:\n> On Sat, Dec 06, 2025 at 03:31:25PM -0500, Taylor Blau wrote:\n> > Note that this produces MIDXs which may be incompatible with earlier\n> > versions of Git that have stricter requirements on the layout of packs\n> > within a MIDX. This patch does *not* modify the version number of the\n> > MIDX format, since existing versions of Git already know to gracefully\n> > ignore a MIDX with packs that appear out-of-order.\n>\n> Interesting. Did you verify how other implementations of Git behave if\n> we start to relax this requirement? It seems like a somewhat dangerous\n> assumption to me that this will just continue to work.\n\nThat's a great point. It looks like current libgit2 assumes[1] that the\nlist is sorted and complains loudly if it is not. Presumably other\nimplementations behave similarly.\n\nI think that is a compelling enough argument to swing us towards\nbumping the version number to avoid compatibility issues.\n\n> Also, is there a reason why you prefer this over bumping the version\n> number?\n\nI was trying to avoid having all existing Git clients be unable to read\nv2 MIDXs for such a seemingly minor change, but I think the above\ncompels us to.\n\n> I assume that it cannot happen that we append to the array of MIDX'd\n> packs after we have sorted. It would mean that the MIDX somehow changed\n> its representation or was amended to, which isn't possible.\n\nThat's right.\n\nThanks,\nTaylor\n\n[1]: https://github.com/libgit2/libgit2/blob/v1.9.2/src/libgit2/midx.c#L75-L76\n"},{"id":"531877","messageId":"aTeFVDGo69ljiQP9@nand.local","threadId":"64585","inReplyTo":"aTeEZX4036A9YecX@nand.local","subject":"Re: [PATCH 09/17] midx: do not require packs to be sorted in lexicographic order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T02:11:32Z","receivedAt":"2025-12-09T02:11:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 09:07:33PM -0500, Taylor Blau wrote:\n> On Mon, Dec 08, 2025 at 07:26:53PM +0100, Patrick Steinhardt wrote:\n> > On Sat, Dec 06, 2025 at 03:31:25PM -0500, Taylor Blau wrote:\n> > > Note that this produces MIDXs which may be incompatible with earlier\n> > > versions of Git that have stricter requirements on the layout of packs\n> > > within a MIDX. This patch does *not* modify the version number of the\n> > > MIDX format, since existing versions of Git already know to gracefully\n> > > ignore a MIDX with packs that appear out-of-order.\n> >\n> > Interesting. Did you verify how other implementations of Git behave if\n> > we start to relax this requirement? It seems like a somewhat dangerous\n> > assumption to me that this will just continue to work.\n>\n> That's a great point. It looks like current libgit2 assumes[1] that the\n> list is sorted and complains loudly if it is not. Presumably other\n> implementations behave similarly.\n>\n> I think that is a compelling enough argument to swing us towards\n> bumping the version number to avoid compatibility issues.\n\nI had another thought about how we might work around this without\nforcing a compatibility issue, but it's a non-starter. I wanted to share\nit on the list for posterity regardless.\n\nI was going to add that we could instead consider adding a new chunk to\nthe MIDX format that lists the pack names in the order that they should\nappear in the pseudo-pack order. Absent of that chunk, the pseudo-pack\norder would be defined by the lexicographic order of pack names. If the\nchunk exists, it would supersede that ordering.\n\nBut that just kicks the can down the road, since implementations like\nlibgit2 would think that they could read a *.midx file, but then they'd\nproduce all sorts of errors when trying to read its corresponding\n*.bitmap file by permuting its bits out-of-order.\n\n(I'm not sure off-hand whether or not libgit2 supports reading MIDX\nbitmaps to begin with. Regardless, we should not introduce the\npossibility for such a breakage in clients that *do* support reading\nMIDX bitmaps, whether or not libgit2 is such a client.)\n\nThanks,\nTaylor\n"},{"id":"531878","messageId":"aTeF4rn6h1gc6Kgl@nand.local","threadId":"64585","inReplyTo":"aTcYdebOuUfrAAJJ@pks.im","subject":"Re: [PATCH 10/17] git-compat-util.h: introduce `u32_add()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T02:13:54Z","receivedAt":"2025-12-09T02:13:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 07:27:01PM +0100, Patrick Steinhardt wrote:\n> On Sat, Dec 06, 2025 at 03:31:28PM -0500, Taylor Blau wrote:\n> > diff --git a/git-compat-util.h b/git-compat-util.h\n> > index 398e0fac4fa..a7aa5f05fc9 100644\n> > --- a/git-compat-util.h\n> > +++ b/git-compat-util.h\n> > @@ -670,6 +670,14 @@ static inline int cast_size_t_to_int(size_t a)\n> > \treturn (int)a;\n> > }\n> >\n> > +static inline uint32_t u32_add(uint32_t a, uint32_t b)\n> > +{\n> > +\tif (unsigned_add_overflows(a, b))\n> > +\t\tdie(\"uint32_t overflow: %\"PRIuMAX\" + %\"PRIuMAX,\n> > +\t\t (uintmax_t)a, (uintmax_t)b);\n> > +\treturn a + b;\n> > +}\n>\n> We already use PRIu32 in our codebase, so why is the cast necessary?\n\nI don't think it is; we could easily write this as:\n\n die(\"uint32_t overflow: %\"PRIu32\" + %\"PRIu32, a, b);\n\ninstead, but this matches the convention of other similar functions in\nthe compat-util header.\n\n(It's possible that there is some reasoning here that using PRIuMAX\nreally *is* necessary, but it isn't clear to me that's the case.)\n\nThanks,\nTaylor\n"},{"id":"531879","messageId":"aTeGE2qplVC+kE/c@nand.local","threadId":"64585","inReplyTo":"aTcYfOZ4_GdIwM9b@pks.im","subject":"Re: [PATCH 13/17] midx-write.c: enumerate `pack_int_id` values directly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T02:14:43Z","receivedAt":"2025-12-09T02:14:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 07:27:08PM +0100, Patrick Steinhardt wrote:\n> On Sat, Dec 06, 2025 at 03:31:37PM -0500, Taylor Blau wrote:\n> > Our `midx-write.c::fill_packs_from_midx()` function currently enumerates\n> > the range [0, m->num_packs), and then shifts its index variable up by\n> > `m->num_packs_in_base` to produce a valid `pack_int_id`.\n> >\n> > Instead, directly enumerate the range:\n> >\n> > [m->num_packs_in_base, m->num_packs_in_base + m->num_packs)\n> >\n> > , which are the original pack_int_ids themselves as opposed to the\n> > indexes of those packs relative to the MIDX layer they are contained\n> > within.\n>\n> Sensible. I was confused a bit by the previous change because I couldn't\n> quite spot the shift happening. I think this makes things a bit easier\n> to read.\n\nI'm glad that the end result was more pleasing. I have gone back and\nforth whether to enumerate [0, m->num_packs) and shift, or to enumerate\nthe pack_int_ids directly, so it's helpful to know what style others\nprefer.\n\nThanks,\nTaylor\n"},{"id":"531880","messageId":"aTeGgqxVO4xcuk6y@nand.local","threadId":"64585","inReplyTo":"aTcYhKOIu7ebJ_xV@pks.im","subject":"Re: [PATCH 15/17] t/helper/test-read-midx.c: plug memory leak when selecting layer","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-12-09T02:16:34Z","receivedAt":"2025-12-09T02:16:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 08, 2025 at 07:27:16PM +0100, Patrick Steinhardt wrote:\n> > @@ -36,8 +37,11 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n> > \tif (checksum) {\n> > \t\twhile (m && strcmp(get_midx_checksum(m), checksum))\n> > \t\t\tm = m->base_midx;\n> > -\t\tif (!m)\n> > -\t\t\treturn 1;\n> > +\t\tif (!m) {\n> > +\t\t\tret = error(_(\"could not find MIDX with checksum %s\"),\n> > +\t\t\t\t checksum);\n> > +\t\t\tgoto out;\n> > +\t\t}\n> > \t}\n> >\n> > \tprintf(\"header: %08x %d %d %d %d\\n\",\n>\n> We change the return code from 1 to -1, but that ultimately shouldn't\n> matter much.\n\nYeah; I think that returning negative values here makes more sense, and\nuse of error() encourages that pattern, hence the change here.\n\n> I'll stop reviewing here and will have a look at the remaining two\n> patches with some fresh eyes. But so far this was a nice read, thanks!\n\nThanks for the review thus far! I look forward to your thoughts on the\nremainder of the series. In related news, I owe you some review on your\n'pks/skip-noop-rewrite' patches, which I hope to get to tomorrow.\n\nThanks,\nTaylor\n"},{"id":"531887","messageId":"aTfBZVo6Hq8Nf7uW@pks.im","threadId":"64585","inReplyTo":"aTeAUKrB89RM9p+i@nand.local","subject":"Re: [PATCH 02/17] midx: split `get_midx_checksum()` by adding `get_midx_hash()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-09T06:27:49Z","receivedAt":"2025-12-09T06:27:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Dec 08, 2025 at 08:50:08PM -0500, Taylor Blau wrote:\n> On Mon, Dec 08, 2025 at 08:42:23PM -0500, Taylor Blau wrote:\n> > On Mon, Dec 08, 2025 at 07:25:05PM +0100, Patrick Steinhardt wrote:\n> > > If I didn't have the context of this patch series I would be wondering\n> > > what the actual difference between `get_midx_checksum()` and\n> > > `get_midx_hash()` is. The way the functions are named seems to rather\n> > > indicate that we talk about two different kinds of hashes, rather than\n> > > two different ways to encode them.\n> > >\n> > > Would it maybe be preferable to call them `get_midx_checksum()` and\n> > > `get_midx_checksum_hex()`? While at it, we could go even further and\n> > > rename them to `midx_get_checksum()` and `midx_get_checksum_hex()` to\n> > > conform to our modern best practices.\n> >\n> > Yeah, I think those are both reasonable suggestions; I'll apply those\n> > locally, thanks!\n> \n> Hmm. Upon further thinking, I wonder which function should be named\n> which.\n> \n> I think the _checksum() variant suggests that it returns the non-hex\n> encoded form, while the _hex() variant suggests the opposite.\n> Unfortunately, the latter is both more commonly used and more characters\n> to type ;-).\n> \n> I wonder if there are shorter names available. Perhaps\n> midx_get_checksum() and midx_get_checksum_raw()?\n\nIt's only four more characters to type the `_hex()` variant, and it is\nin line with the interfaces we've got in \"hex.h\". So personally I'd\nstill prefer to go with `_hex()`.\n\nPatrick\n"},{"id":"531888","messageId":"aTfN_PycU9ag8c0u@pks.im","threadId":"64585","inReplyTo":"c136b2e179d02321de7e7b3f1b6c748cb434d68d.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 16/17] midx: implement MIDX compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-09T07:21:32Z","receivedAt":"2025-12-09T07:21:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:47PM -0500, Taylor Blau wrote:\n> diff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\n> index 164cf1f2291..a9664e77411 100644\n> --- a/Documentation/git-multi-pack-index.adoc\n> +++ b/Documentation/git-multi-pack-index.adoc\n> @@ -12,6 +12,8 @@ SYNOPSIS\n> 'git multi-pack-index' [] write [--preferred-pack=]\n> \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n> \t\t [--refs-snapshot=]\n> +'git multi-pack-index' [] compact [--[no-]incremental]\n> +\t\t \n> 'git multi-pack-index' [] verify\n> 'git multi-pack-index' [] expire\n> 'git multi-pack-index' [] repack [--batch-size=]\n> @@ -83,6 +85,17 @@ marker).\n> \t\tnecessary.\n> --\n> \n> +compact::\n> +\tWrite a new MIDX layer containing only objects and packs present\n> +\tin the range `` to ``, where both arguments are\n> +\tchecksums of existing layers in the MIDX chain.\n> ++\n> +--\n> +\t--incremental::\n> +\t\tWrite the result to a MIDX chain instead of writing a\n> +\t\tstand-alone MIDX. Incompatible with `--bitmap`.\n\nInteresting. What would happen if you compact a subrange of the MIDX\nchain without incremental? Would the MIDX be completely replaced with a\nMIDX that only covers these packs?\n\nAlso, the \"--bitmap\" flag does not exist yet, so the second sentence\nprobably needs to be introduced in the next commit.\n\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index c0c6c1760c0..9b0c2082cb3 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -195,6 +204,63 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n> \treturn ret;\n> }\n> \n> +static int cmd_multi_pack_index_compact(int argc, const char **argv,\n> +\t\t\t\t\tconst char *prefix,\n> +\t\t\t\t\tstruct repository *repo)\n> +{\n> +\tstruct multi_pack_index *m, *cur;\n> +\tstruct multi_pack_index *from_midx = NULL;\n> +\tstruct multi_pack_index *to_midx = NULL;\n> +\tstruct odb_source *source;\n> +\tint ret;\n> +\n> +\tstruct option *options;\n> +\tstatic struct option builtin_multi_pack_index_compact_options[] = {\n> +\t\tOPT_BIT(0, \"incremental\", &opts.flags,\n> +\t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n> +\t\tOPT_END(),\n> +\t};\n> +\n> +\trepo_config(repo, git_multi_pack_index_write_config, NULL);\n> +\n> +\toptions = add_common_options(builtin_multi_pack_index_compact_options);\n> +\n> +\ttrace2_cmd_mode(argv[0]);\n> +\n> +\tif (isatty(2))\n> +\t\topts.flags |= MIDX_PROGRESS;\n> +\targc = parse_options(argc, argv, prefix,\n> +\t\t\t options, builtin_multi_pack_index_compact_usage,\n> +\t\t\t 0);\n> +\n> +\tif (argc != 2)\n> +\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n> +\t\t\t\t options);\n> +\tsource = handle_object_dir_option(the_repository);\n> +\n> +\tFREE_AND_NULL(options);\n> +\n> +\tm = get_multi_pack_index(source);\n> +\n> +\tfor (cur = m; cur && !(from_midx && to_midx); cur = cur->base_midx) {\n> +\t\tconst char *midx_csum = get_midx_checksum(cur);\n> +\n> +\t\tif (!from_midx && !strcmp(midx_csum, argv[0]))\n> +\t\t\tfrom_midx = cur;\n> +\t\tif (!to_midx && !strcmp(midx_csum, argv[1]))\n> +\t\t\tto_midx = cur;\n> +\t}\n> +\n> +\tif (!from_midx)\n> +\t\tdie(_(\"could not find MIDX 'from': %s\"), argv[0]);\n> +\tif (!to_midx)\n> +\t\tdie(_(\"could not find MIDX 'to': %s\"), argv[1]);\n> +\n> +\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n> +\n> +\treturn ret;\n> +}\n\nIs it valid if `from_midx == to_midx`?\n\n> diff --git a/midx-write.c b/midx-write.c\n> index 7854561359d..fcbfedcd913 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -953,6 +980,72 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n> \treturn 0;\n> }\n> \n> +static uint32_t compactible_packs_between(const struct multi_pack_index *from,\n> +\t\t\t\t\t const struct multi_pack_index *to)\n> +{\n> +\tuint32_t nr;\n> +\n> +\tASSERT(from && to);\n> +\n> +\tnr = u32_add(to->num_packs, to->num_packs_in_base);\n> +\tif (nr < from->num_packs_in_base)\n> +\t\tBUG(\"unexpected number of packs in base during compaction: \"\n> +\t\t \"%\"PRIu32\" < %\"PRIu32, nr, from->num_packs_in_base);\n> +\n> +\treturn nr - from->num_packs_in_base;\n> +}\n> +\n> +static int fill_packs_from_midx_range(struct write_midx_context *ctx,\n> +\t\t\t\t int bitmap_order)\n> +{\n> +\tstruct multi_pack_index *m = ctx->compact_to;\n> +\tuint32_t packs_nr;\n> +\n> +\tASSERT(ctx->compact && !ctx->nr);\n> +\tASSERT(ctx->compact_from);\n> +\tASSERT(ctx->compact_to);\n> +\n> +\tpacks_nr = compactible_packs_between(ctx->compact_from,\n> +\t\t\t\t\t ctx->compact_to);\n> +\n> +\tALLOC_GROW(ctx->info, packs_nr, ctx->alloc);\n> +\n> +\twhile (m != ctx->compact_from->base_midx) {\n> +\t\tuint32_t pack_int_id, preferred_pack_id;\n> +\t\tuint32_t i;\n> +\n> +\t\tif (bitmap_order) {\n> +\t\t\tif (midx_preferred_pack(m, &preferred_pack_id) < 0)\n> +\t\t\t\tdie(_(\"could not determine preferred pack\"));\n\n`midx_preferred_pack()` only returns a valid pack ID in case we've got a\nreverse index, and as far as I understand we seem to only generate those\nwhen computing bitmaps. I assume that this means that we can only\ncompact MIDX layers in bitmap order if they already were in bitmap order\nbefore?\n\nThat would at least also make sense. We of course cannot randomly change\nthe order in the middle of our layers, as that would break later layers\nthat build on top.\n\n> +\t\t} else {\n> +\t\t\tpreferred_pack_id = m->num_packs_in_base;\n> +\t\t}\n> +\n> +\t\tpack_int_id = m->num_packs_in_base - ctx->compact_from->num_packs_in_base;\n> +\n> +\t\tif (fill_pack_from_midx(&ctx->info[pack_int_id++], m,\n> +\t\t\t\t\tpreferred_pack_id) < 0)\n> +\t\t\treturn -1;\n> +\n> +\t\tfor (i = m->num_packs_in_base;\n> +\t\t i < m->num_packs_in_base + m->num_packs; i++) {\n> +\t\t\tif (preferred_pack_id == i)\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\tif (fill_pack_from_midx(&ctx->info[pack_int_id++], m,\n> +\t\t\t\t\t\ti) < 0)\n> +\t\t\t\treturn -1;\n> +\t\t}\n> +\n\nSo the condition that should hold after this loop is `pack_int_id ==\nm->num_packs`. Which is somewhat obvious: we skip one pack, but that\npack is the preferred pack that we have populated first.\n\n> @@ -1101,11 +1216,18 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \t\t\t */\n> \t\t\tif (ctx.incremental)\n> \t\t\t\tctx.base_midx = m;\n> -\t\t\telse if (!opts->packs_to_include)\n> +\t\t\tif (!opts->packs_to_include)\n> \t\t\t\tctx.m = m;\n\nI'm a bit surprised by this change here. I would've expected that we\nnever pass `packs_to_include` when compacting, so why is this change\nnecessary?\n\n> \t\t}\n> \t}\n> \n> +\t/*\n> +\t * If compacting MIDX layer(s) in the range [from, to], then the\n> +\t * compacted MIDX will share the same base MIDX as 'from'.\n> +\t */\n> +\tif (ctx.compact)\n> +\t\tctx.base_midx = ctx.compact_from->base_midx;\n\nOkay, here we overwrite `ctx.base_midx` that we might've already set in\nthe condition above. It's a bit confusing, doubly so because we may be\nwarning about the invalid MIDX and claim to ignore it, but ultimately we\ndon't.\n\n> diff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\n> new file mode 100755\n> index 00000000000..f889af7fb1d\n> --- /dev/null\n> +++ b/t/t5335-compact-multi-pack-index.sh\n> @@ -0,0 +1,102 @@\n> +#!/bin/sh\n> +\n> +test_description='multi-pack-index compaction'\n> +\n> +. ./test-lib.sh\n> +\n> +GIT_TEST_MULTI_PACK_INDEX=0\n> +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n> +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n> +\n> +objdir=.git/objects\n> +packdir=$objdir/pack\n> +midxdir=$packdir/multi-pack-index.d\n> +midx_chain=$midxdir/multi-pack-index-chain\n> +\n> +nth_line() {\n> +\tlocal n=\"$1\"\n> +\tshift\n> +\tawk \"NR==$n\" \"$@\"\n> +}\n> +\n> +write_packs () {\n> +\tfor c in \"$@\"\n> +\tdo\n> +\t\ttest_commit \"$c\" &&\n\nNit: it might be sensible to disable housekeeping here. You strongly\ndepend on the on-disk shape of the objects, so if you by chance wrote\ntwo objects starting with \"17\" we'd end up repacking and racing.\n\nI've also got an upcoming patch series in mindthat I've got cooking to\nmake geometric compaction the default for auto-maintenance. We've got\nmany test suites that implicitly rely on the current algorithm used by\ngit-gc(1), so I'd love to avoid adding more.\n\n[snip]\n> +test_expect_success 'MIDX compaction with lex-ordered pack names' '\n> +\tgit init midx-compact-lex-order &&\n> +\t(\n> +\t\tcd midx-compact-lex-order &&\n> +\n> +\t\twrite_packs A B C D E &&\n> +\t\ttest_line_count = 5 $midx_chain &&\n> +\n> +\t\tgit multi-pack-index compact --incremental \\\n> +\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n> +\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n> +\t\ttest_line_count = 3 $midx_chain &&\n> +\n> +\t\ttest_midx_layer_packs \"$(nth_line 1 \"$midx_chain\")\" A &&\n> +\t\ttest_midx_layer_packs \"$(nth_line 2 \"$midx_chain\")\" B C D &&\n> +\t\ttest_midx_layer_packs \"$(nth_line 3 \"$midx_chain\")\" E &&\n> +\n> +\t\ttest_midx_layer_object_uniqueness\n> +\t)\n> +'\n\nIt would be nice to also test for requests that don't make sense: \"from\"\nlarger than \"to\", \"from == to\", missing \"from\" or \"foo\" and so on.\n\nPatrick\n"},{"id":"531889","messageId":"aTfOBz3ElQYi6j8i@pks.im","threadId":"64585","inReplyTo":"d6a4e6b69552d8626e6df2d7cd042b9fe151914a.1765053054.git.me@ttaylorr.com","subject":"Re: [PATCH 17/17] midx: enable reachability bitmaps during MIDX compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-12-09T07:21:43Z","receivedAt":"2025-12-09T07:21:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Dec 06, 2025 at 03:31:50PM -0500, Taylor Blau wrote:\n> Enable callers to generate reachability bitmaps when performing MIDX\n> layer compaction by combining all existing bitmaps from the compacted\n> layers.\n> \n> Note that the because of the object/pack ordering described by the\n\ns/that the because/that because/\n\n> previous commit, the pseudo-pack order for the compacted MIDX is the\n> same as concatenating the individual pseudo-pack orderings for each\n> layer in the compaction range.\n> \n> As a result, the only non-test or documentation change necessary is to\n> treat all objects as non-preferred during compaction so as not to\n> disturb the object ordering.\n> \n> In the future, we may want to adjust which commit(s) receive\n> reachability bitmaps when compacting multiple .bitmap files into one, or\n> even generate new bitmaps (e.g., if the references have moved\n> significantly since the .bitmap was generated). This commit only\n> implements combining all existing bitmaps in range together in order to\n> demonstrate and lay the groundwork for more exotic strategies.\n\nWill there also be a follow-up patch series that introduces geometric\nrepacking for multi-pack indices?\n\n> @@ -216,6 +216,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n> \n> \tstruct option *options;\n> \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n> +\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n> +\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n> \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n> \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n> \t\tOPT_END(),\n\nIs this new flag actually incompatible with the incremental flag like\nyou claimed in the preceding commit? I had the impression that it should\nbe possible to write incremental bitmaps now.\n\nIf that's not the case, we should probably have a call to\n`die_for_imcopatible_opt2()` somewhere.\n\nPatrick\n"},{"id":"533778","messageId":"aWbLYCvzqKNziLfZ@nand.local","threadId":"64585","inReplyTo":"aTfBZVo6Hq8Nf7uW@pks.im","subject":"Re: [PATCH 02/17] midx: split `get_midx_checksum()` by adding `get_midx_hash()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-13T22:46:56Z","receivedAt":"2026-01-13T22:47:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Dec 09, 2025 at 07:27:49AM +0100, Patrick Steinhardt wrote:\n> It's only four more characters to type the `_hex()` variant, and it is\n> in line with the interfaces we've got in \"hex.h\". So personally I'd\n> still prefer to go with `_hex()`.\n\nI think that's a fair point. I am not sure that I go along with the\n\"it's only four more characters\" argument, but I would like to be\nconsistent with existing APIs above all else. I think the choice we made\nin \"hex.h\" is a good one, so let's stick with that.\n\nThanks,\nTaylor\n"},{"id":"533780","messageId":"aWbWC4nTnP52lxSg@nand.local","threadId":"64585","inReplyTo":"aTfN_PycU9ag8c0u@pks.im","subject":"Re: [PATCH 16/17] midx: implement MIDX compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-13T23:32:27Z","receivedAt":"2026-01-13T23:32:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Dec 09, 2025 at 08:21:32AM +0100, Patrick Steinhardt wrote:\n> On Sat, Dec 06, 2025 at 03:31:47PM -0500, Taylor Blau wrote:\n> > diff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\n> > index 164cf1f2291..a9664e77411 100644\n> > --- a/Documentation/git-multi-pack-index.adoc\n> > +++ b/Documentation/git-multi-pack-index.adoc\n> > @@ -12,6 +12,8 @@ SYNOPSIS\n> > 'git multi-pack-index' [] write [--preferred-pack=]\n> > \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n> > \t\t [--refs-snapshot=]\n> > +'git multi-pack-index' [] compact [--[no-]incremental]\n> > +\t\t \n> > 'git multi-pack-index' [] verify\n> > 'git multi-pack-index' [] expire\n> > 'git multi-pack-index' [] repack [--batch-size=]\n> > @@ -83,6 +85,17 @@ marker).\n> > \t\tnecessary.\n> > --\n> >\n> > +compact::\n> > +\tWrite a new MIDX layer containing only objects and packs present\n> > +\tin the range `` to ``, where both arguments are\n> > +\tchecksums of existing layers in the MIDX chain.\n> > ++\n> > +--\n> > +\t--incremental::\n> > +\t\tWrite the result to a MIDX chain instead of writing a\n> > +\t\tstand-alone MIDX. Incompatible with `--bitmap`.\n>\n> Interesting. What would happen if you compact a subrange of the MIDX\n> chain without incremental? Would the MIDX be completely replaced with a\n> MIDX that only covers these packs?\n\nThat's right.\n\n> Also, the \"--bitmap\" flag does not exist yet, so the second sentence\n> probably needs to be introduced in the next commit.\n\nAh, great catch -- I removed that line here. I don't think it needs to\nbe readded in the following commit, though, since that patch introduces\n\"--bitmap\" and makes it compatible with MIDX compaction.\n\n> > +\tif (!from_midx)\n> > +\t\tdie(_(\"could not find MIDX 'from': %s\"), argv[0]);\n> > +\tif (!to_midx)\n> > +\t\tdie(_(\"could not find MIDX 'to': %s\"), argv[1]);\n> > +\n> > +\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n> > +\n> > +\treturn ret;\n> > +}\n>\n> Is it valid if `from_midx == to_midx`?\n\nYes, that would result in a noop write.\n\n> > +\twhile (m != ctx->compact_from->base_midx) {\n> > +\t\tuint32_t pack_int_id, preferred_pack_id;\n> > +\t\tuint32_t i;\n> > +\n> > +\t\tif (bitmap_order) {\n> > +\t\t\tif (midx_preferred_pack(m, &preferred_pack_id) < 0)\n> > +\t\t\t\tdie(_(\"could not determine preferred pack\"));\n>\n> `midx_preferred_pack()` only returns a valid pack ID in case we've got a\n> reverse index, and as far as I understand we seem to only generate those\n> when computing bitmaps. I assume that this means that we can only\n> compact MIDX layers in bitmap order if they already were in bitmap order\n> before?\n>\n> That would at least also make sense. We of course cannot randomly change\n> the order in the middle of our layers, as that would break later layers\n> that build on top.\n\nIndeed, we only generate a reverse index for a MIDX if we are writing it\nwith bitmaps, since there is no other purpose for having a revindex\noutside of reachability bitmaps.\n\nSo if we have a bitmap and are compacting, then we need to retain the\norder of the packs as they appear in the pre-compaction pseudo-pack\norder to avoid permuting the bits corresponding to those objects. In\nother words, you're correct in saying that we cannot start writing\nbitmaps during compaction if we did not have bitmaps to begin with\npre-compaction.\n\n> > +\t\tfor (i = m->num_packs_in_base;\n> > +\t\t i < m->num_packs_in_base + m->num_packs; i++) {\n> > +\t\t\tif (preferred_pack_id == i)\n> > +\t\t\t\tcontinue;\n> > +\n> > +\t\t\tif (fill_pack_from_midx(&ctx->info[pack_int_id++], m,\n> > +\t\t\t\t\t\ti) < 0)\n> > +\t\t\t\treturn -1;\n> > +\t\t}\n> > +\n>\n> So the condition that should hold after this loop is `pack_int_id ==\n> m->num_packs`. Which is somewhat obvious: we skip one pack, but that\n> pack is the preferred pack that we have populated first.\n\nExactly!\n\n> > @@ -1101,11 +1216,18 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> > \t\t\t */\n> > \t\t\tif (ctx.incremental)\n> > \t\t\t\tctx.base_midx = m;\n> > -\t\t\telse if (!opts->packs_to_include)\n> > +\t\t\tif (!opts->packs_to_include)\n> > \t\t\t\tctx.m = m;\n>\n> I'm a bit surprised by this change here. I would've expected that we\n> never pass `packs_to_include` when compacting, so why is this change\n> necessary?\n\nRight, we do not pass packs_to_include here during compaction. But if we\nare doing an incremental compaction, then we do want to assign ctx.m in\naddition to ctx.base_midx.\n\n> > diff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\n> > new file mode 100755\n> > index 00000000000..f889af7fb1d\n> > --- /dev/null\n> > +++ b/t/t5335-compact-multi-pack-index.sh\n> > @@ -0,0 +1,102 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='multi-pack-index compaction'\n> > +\n> > +. ./test-lib.sh\n> > +\n> > +GIT_TEST_MULTI_PACK_INDEX=0\n> > +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n> > +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n> > +\n> > +objdir=.git/objects\n> > +packdir=$objdir/pack\n> > +midxdir=$packdir/multi-pack-index.d\n> > +midx_chain=$midxdir/multi-pack-index-chain\n> > +\n> > +nth_line() {\n> > +\tlocal n=\"$1\"\n> > +\tshift\n> > +\tawk \"NR==$n\" \"$@\"\n> > +}\n> > +\n> > +write_packs () {\n> > +\tfor c in \"$@\"\n> > +\tdo\n> > +\t\ttest_commit \"$c\" &&\n>\n> Nit: it might be sensible to disable housekeeping here. You strongly\n> depend on the on-disk shape of the objects, so if you by chance wrote\n> two objects starting with \"17\" we'd end up repacking and racing.\n>\n> I've also got an upcoming patch series in mindthat I've got cooking to\n> make geometric compaction the default for auto-maintenance. We've got\n> many test suites that implicitly rely on the current algorithm used by\n> git-gc(1), so I'd love to avoid adding more.\n\nI'm not sure I follow what you mean by \"housekeeping\" here. Are you\nreferring to maintenance.auto? If so, we shouldn't be writing so many\npacks as to trigger that during these tests, but I can disable it as a\nsanity check just in case.\n\n> [snip]\n> > +test_expect_success 'MIDX compaction with lex-ordered pack names' '\n> > +\tgit init midx-compact-lex-order &&\n> > +\t(\n> > +\t\tcd midx-compact-lex-order &&\n> > +\n> > +\t\twrite_packs A B C D E &&\n> > +\t\ttest_line_count = 5 $midx_chain &&\n> > +\n> > +\t\tgit multi-pack-index compact --incremental \\\n> > +\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n> > +\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n> > +\t\ttest_line_count = 3 $midx_chain &&\n> > +\n> > +\t\ttest_midx_layer_packs \"$(nth_line 1 \"$midx_chain\")\" A &&\n> > +\t\ttest_midx_layer_packs \"$(nth_line 2 \"$midx_chain\")\" B C D &&\n> > +\t\ttest_midx_layer_packs \"$(nth_line 3 \"$midx_chain\")\" E &&\n> > +\n> > +\t\ttest_midx_layer_object_uniqueness\n> > +\t)\n> > +'\n>\n> It would be nice to also test for requests that don't make sense: \"from\"\n> larger than \"to\", \"from == to\", missing \"from\" or \"foo\" and so on.\n\nAll good suggestions, thanks!\n\nThanks,\nTaylor\n"},{"id":"533782","messageId":"aWbZopq71ZUXvUsr@nand.local","threadId":"64585","inReplyTo":"aTfOBz3ElQYi6j8i@pks.im","subject":"Re: [PATCH 17/17] midx: enable reachability bitmaps during MIDX compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-13T23:47:46Z","receivedAt":"2026-01-13T23:47:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Dec 09, 2025 at 08:21:43AM +0100, Patrick Steinhardt wrote:\n> On Sat, Dec 06, 2025 at 03:31:50PM -0500, Taylor Blau wrote:\n> > Enable callers to generate reachability bitmaps when performing MIDX\n> > layer compaction by combining all existing bitmaps from the compacted\n> > layers.\n> >\n> > Note that the because of the object/pack ordering described by the\n>\n> s/that the because/that because/\n\n;-) great catch!\n\n> > previous commit, the pseudo-pack order for the compacted MIDX is the\n> > same as concatenating the individual pseudo-pack orderings for each\n> > layer in the compaction range.\n> >\n> > As a result, the only non-test or documentation change necessary is to\n> > treat all objects as non-preferred during compaction so as not to\n> > disturb the object ordering.\n> >\n> > In the future, we may want to adjust which commit(s) receive\n> > reachability bitmaps when compacting multiple .bitmap files into one, or\n> > even generate new bitmaps (e.g., if the references have moved\n> > significantly since the .bitmap was generated). This commit only\n> > implements combining all existing bitmaps in range together in order to\n> > demonstrate and lay the groundwork for more exotic strategies.\n>\n> Will there also be a follow-up patch series that introduces geometric\n> repacking for multi-pack indices?\n\nThat's the plan, though I have been calling it \"incremental MIDX/bitmap\"\nrepacking. The idea is roughly what I presented on slides 80-90 of my\nGit Merge talk from last year[1].\n\nThe gist is the following:\n\n 1. Perform a geometric repack of all non-MIDX'd packs, optionally\n including any packs in the top-most MIDX layer if it has more than M\n packs.\n\n 2. Generate a new MIDX layer containing the result of that geometric\n repack.\n\n 3. Perform layer compaction on the new chain in order to ensure that\n each layer has no more than 2x the number of packs as the layer\n above it.\n\nThis is a conceptual overview, so there are a few white lies here.\nNamely, we do not write out a MIDX containing the result of the\ngeometric repack from the first step in all cases. If it would be\ncompacted in the following step, we'll optimize out that write and\ninstead write a layer whose contents is the result of the geometric\nrepack and the contents of the layer(s) below.\n\nCompaction is also done in a single pass, so we don't compact the same\nlayer multiple times during a single repack.\n\nThe result is that we end up with a MIDX chain with two properties (see\nslide 131 for more details):\n\n - Newer layers contain a greater number of packs which tend to be\n smaller by object count than older layers.\n\n - Older layers contain a smaller number of packs which tend to be\n larger by object count than newer layers.\n\nI have that working in a WIP-quality branch[2], which I have been using\nas a base to pull patches out of (which has yielded parts 3.1 and 3.2\nthus far).\n\n> > @@ -216,6 +216,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n> >\n> > \tstruct option *options;\n> > \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n> > +\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n> > +\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n> > \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n> > \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n> > \t\tOPT_END(),\n>\n> Is this new flag actually incompatible with the incremental flag like\n> you claimed in the preceding commit? I had the impression that it should\n> be possible to write incremental bitmaps now.\n>\n> If that's not the case, we should probably have a call to\n> `die_for_imcopatible_opt2()` somewhere.\n\nIt's not incompatible, the comment is just stale, thanks for pointing it\nout!\n\nThanks,\nTaylor\n\n[1]: https://ttaylorr.com/presentations/git-merge-2025.pdf\n\n[2]: 'tb/incremental-midx-part-3.wip' from my 'ttaylorr/git' fork on\n GitHub.\n"},{"id":"533869","messageId":"cover.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH v2 00/18] midx: incremental MIDX/bitmap layer compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:13Z","receivedAt":"2026-01-14T19:54:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"[Note to the maintainer: this is based on 'master' with my\n'tb/midx-write-corrupt-checksum-fix' merged in and should produce zero\nconflicts when applied on top.]\n\nThis is a reroll of my series to implement MIDX layer compaction\nadjusted in response to reviewer feedback.\n\nAs usual, a range-diff is included below for convenience. The main\nchanges since last time are as follows:\n\n - The MIDX file format has been bumped to version 2 to avoid breaking\n external implementations which may not be prepared to gracefully\n degrade away from the MIDX when encountering a non-sorted PNAM\n chunk.\n\n - I adjusted the names of the two functions which were split out from\n \"get_midx_checksum()\" to be \"midx_get_checksum_hash()\" and\n \"midx_get_checksum_hex()\" to match the hex.h API convention.\n\nAlong with those, there are a handful of more minor/cosmetic changes\nthat are also included in this round:\n\n - Explicitly declared the write_midx_opts struct from\n expire_midx_packs() instead of making use of compound literal\n syntax, for which we have an in-flight weather balloon.\n\n - Docfix to remove an incorrect statement suggesting that the options\n \"--bitmap\" and \"--incremental\" are incompatible with one another\n when used in conjunction with \"git multi-pack-index compact\".\n\n - Explicit checks and tests for bogus compaction scenarios (e.g.,\n missing from/to, to as an ancestor of from, etc.).\n\n - Various wording changes and a small handful of typofixes.\n\nThe original cover letter may be found here[1].\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/cover.1765053054.git.me@ttaylorr.com/\n\nTaylor Blau (18):\n midx: mark `get_midx_checksum()` arguments as const\n midx: rename `get_midx_checksum()` to `midx_get_checksum_hash()`\n midx: introduce `midx_get_checksum_hex()`\n builtin/multi-pack-index.c: make '--progress' a common option\n git-multi-pack-index(1): remove non-existent incompatibility\n git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'\n t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39\n midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`\n midx-write.c: introduce `struct write_midx_opts`\n midx: do not require packs to be sorted in lexicographic order\n git-compat-util.h: introduce `u32_add()`\n midx-write.c: introduce `midx_pack_perm()` helper\n midx-write.c: extract `fill_pack_from_midx()`\n midx-write.c: enumerate `pack_int_id` values directly\n midx-write.c: factor fanout layering from `compute_sorted_entries()`\n t/helper/test-read-midx.c: plug memory leak when selecting layer\n midx: implement MIDX compaction\n midx: enable reachability bitmaps during MIDX compaction\n\n Documentation/git-multi-pack-index.adoc | 27 +-\n Documentation/gitformat-pack.adoc | 6 +-\n builtin/multi-pack-index.c | 91 ++++-\n git-compat-util.h | 8 +\n midx-write.c | 499 +++++++++++++++++++-----\n midx.c | 39 +-\n midx.h | 12 +-\n pack-bitmap.c | 9 +-\n pack-revindex.c | 4 +-\n t/helper/test-read-midx.c | 21 +-\n t/meson.build | 1 +\n t/t0450/adoc-help-mismatches | 1 -\n t/t5319-multi-pack-index.sh | 18 +-\n t/t5335-compact-multi-pack-index.sh | 293 ++++++++++++++\n 14 files changed, 879 insertions(+), 150 deletions(-)\n create mode 100755 t/t5335-compact-multi-pack-index.sh\n\nRange-diff against v1:\n 1: 1ee8b752279 ! 1: 2e549ea6443 midx: mark `get_midx_checksum()` arguments as const\n @@ Metadata\n ## Commit message ##\n midx: mark `get_midx_checksum()` arguments as const\n \n - To make clear that the fucntion `get_midx_checksum()` does not do\n + To make clear that the function `get_midx_checksum()` does not do\n anything to modify its argument, mark the MIDX pointer as const.\n \n The following commit will rename this function altogether to make clear\n 2: 57125303758 ! 2: 7255adafe70 midx: split `get_midx_checksum()` by adding `get_midx_hash()`\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - midx: split `get_midx_checksum()` by adding `get_midx_hash()`\n + midx: rename `get_midx_checksum()` to `midx_get_checksum_hash()`\n \n - When trying to print out, say, the hexadecimal representation of a\n - MIDX's hash, our code will do something like:\n + Since 541204aabea (Documentation: document naming schema for structs and\n + their functions, 2024-07-30), we have adopted a naming convention for\n + functions that would prefer a name like, say, `midx_get_checksum()` over\n + `get_midx_checksum()`.\n \n - hash_to_hex_algop(get_midx_checksum(m),\n - m->source->odb->repo->hash_algo);\n -\n - , which is both cumbersome and repetitive. In fact, all but a handful of\n - callers to `get_midx_checksum()` do exactly the above. Reduce the\n - repetitive nature of calling `get_midx_checksum()` by having it return a\n - pointer into a static buffer containing the above result.\n -\n - For the handful of callers that do need to compare the raw bytes and\n - don't want to deal with an encoded copy (e.g., because they are passing\n - it to hasheq() or similar), introduce `get_midx_hash()` which returns\n - the raw bytes.\n + Adopt this convention throughout the midx.h API. Since this function\n + returns a raw (that is, non-hex encoded) hash, let's suffix the function\n + with \"_hash()\" to make this clear. As a side effect, this prepares us\n + for the subsequent change which will introduce a \"_hex()\" variant that\n + encodes the checksum itself.\n \n + Suggested-by: Patrick Steinhardt \n Signed-off-by: Taylor Blau \n \n ## midx-write.c ##\n @@ midx-write.c: static int link_midx_to_chain(struct multi_pack_index *m)\n \n \tfor (i = 0; i < ARRAY_SIZE(midx_exts); i++) {\n -\t\tconst unsigned char *hash = get_midx_checksum(m);\n -+\t\tconst unsigned char *hash = get_midx_hash(m);\n ++\t\tconst unsigned char *hash = midx_get_checksum_hash(m);\n \n \t\tget_midx_filename_ext(m->source, &from,\n \t\t\t\t hash, midx_exts[i].non_split);\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n \t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n -\t\t\t\t hash_to_hex_algop(get_midx_checksum(m),\n --\t\t\t\t\t\t\tm->source->odb->repo->hash_algo));\n -+\t\t\t\t get_midx_checksum(m));\n ++\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(m),\n + \t\t\t\t\t\t\tm->source->odb->repo->hash_algo));\n \t\t\t\tgoto cleanup;\n \t\t\t}\n - \t\t\tctx.num_multi_pack_indexes_before++;\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n \t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n \t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n \n -\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(get_midx_checksum(m),\n --\t\t\t\t\t\t\t\t r->hash_algo));\n -+\t\t\tkeep_hashes[j] = xstrdup(get_midx_checksum(m));\n ++\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(midx_get_checksum_hash(m),\n + \t\t\t\t\t\t\t\t r->hash_algo));\n \t\t\tm = m->base_midx;\n \t\t}\n - \n \n ## midx.c ##\n @@ midx.c: void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n @@ midx.c: void clear_incremental_midx_files_ext(struct odb_source *source, const c\n \t\t\t const char *idx_name);\n \n -const unsigned char *get_midx_checksum(const struct multi_pack_index *m)\n -+const char *get_midx_checksum(const struct multi_pack_index *m)\n -+{\n -+\treturn hash_to_hex_algop(get_midx_hash(m),\n -+\t\t\t\t m->source->odb->repo->hash_algo);\n -+}\n -+\n -+const unsigned char *get_midx_hash(const struct multi_pack_index *m)\n ++const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\n }\n @@ midx.h: struct multi_pack_index {\n #define MIDX_EXT_MIDX \"midx\"\n \n -const unsigned char *get_midx_checksum(const struct multi_pack_index *m);\n -+const char *get_midx_checksum(const struct multi_pack_index *m) /* static buffer */;\n -+const unsigned char *get_midx_hash(const struct multi_pack_index *m);\n ++const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m);\n void get_midx_filename(struct odb_source *source, struct strbuf *out);\n void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\n \t\t\t const unsigned char *hash, const char *ext);\n @@ pack-bitmap.c: char *midx_bitmap_filename(struct multi_pack_index *midx)\n \tif (midx->has_chain)\n \t\tget_split_midx_filename_ext(midx->source, &buf,\n -\t\t\t\t\t get_midx_checksum(midx),\n -+\t\t\t\t\t get_midx_hash(midx),\n ++\t\t\t\t\t midx_get_checksum_hash(midx),\n \t\t\t\t\t MIDX_EXT_BITMAP);\n \telse\n \t\tget_midx_filename_ext(midx->source, &buf,\n -\t\t\t\t get_midx_checksum(midx),\n -+\t\t\t\t get_midx_hash(midx),\n ++\t\t\t\t midx_get_checksum_hash(midx),\n \t\t\t\t MIDX_EXT_BITMAP);\n \n \treturn strbuf_detach(&buf, NULL);\n @@ pack-bitmap.c: static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\tgoto cleanup;\n \n -\tif (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum,\n -+\tif (!hasheq(get_midx_hash(bitmap_git->midx), bitmap_git->checksum,\n ++\tif (!hasheq(midx_get_checksum_hash(bitmap_git->midx), bitmap_git->checksum,\n \t\t bitmap_repo(bitmap_git)->hash_algo)) {\n \t\terror(_(\"checksum doesn't match in MIDX and bitmap\"));\n \t\tgoto cleanup;\n @@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n \t\tif (bitmap_is_midx(found))\n \t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n -\t\t\t\t hash_to_hex_algop(get_midx_checksum(found->midx),\n --\t\t\t\t\t\t revs->repo->hash_algo));\n -+\t\t\t\t get_midx_checksum(found->midx));\n ++\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(found->midx),\n + \t\t\t\t\t\t revs->repo->hash_algo));\n \t\telse\n \t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n - \t\t\t\t hash_to_hex_algop(found->pack->hash,\n \n ## pack-revindex.c ##\n @@ pack-revindex.c: int load_midx_revindex(struct multi_pack_index *m)\n @@ pack-revindex.c: int load_midx_revindex(struct multi_pack_index *m)\n \tif (m->has_chain)\n \t\tget_split_midx_filename_ext(m->source, &revindex_name,\n -\t\t\t\t\t get_midx_checksum(m),\n -+\t\t\t\t\t get_midx_hash(m),\n ++\t\t\t\t\t midx_get_checksum_hash(m),\n \t\t\t\t\t MIDX_EXT_REV);\n \telse\n \t\tget_midx_filename_ext(m->source, &revindex_name,\n -\t\t\t\t get_midx_checksum(m),\n -+\t\t\t\t get_midx_hash(m),\n ++\t\t\t\t midx_get_checksum_hash(m),\n \t\t\t\t MIDX_EXT_REV);\n \n \tret = load_revindex_from_disk(m->source->odb->repo->hash_algo,\n @@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir, con\n \n \tif (checksum) {\n -\t\twhile (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum))\n -+\t\twhile (m && strcmp(get_midx_checksum(m), checksum))\n ++\t\twhile (m && strcmp(hash_to_hex(midx_get_checksum_hash(m)), checksum))\n \t\t\tm = m->base_midx;\n \t\tif (!m)\n \t\t\treturn 1;\n @@ t/helper/test-read-midx.c: static int read_midx_checksum(const char *object_dir)\n \tif (!m)\n \t\treturn 1;\n -\tprintf(\"%s\\n\", hash_to_hex(get_midx_checksum(m)));\n -+\tprintf(\"%s\\n\", get_midx_checksum(m));\n ++\tprintf(\"%s\\n\", hash_to_hex(midx_get_checksum_hash(m)));\n \n \tclose_midx(m);\n \treturn 0;\n15: c2149ae5bc5 ! 3: 25b628fda97 t/helper/test-read-midx.c: plug memory leak when selecting layer\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - t/helper/test-read-midx.c: plug memory leak when selecting layer\n + midx: introduce `midx_get_checksum_hex()`\n \n - Though our 'read-midx' test tool is capable of printing information\n - about a single MIDX layer identified by its checksum, no caller in our\n - test suite exercises this path.\n + When trying to print out, say, the hexadecimal representation of a\n + MIDX's hash, our code will do something like:\n \n - Unfortunately, there is a memory leak lurking in this (currently) unused\n - path that would otherwise be exposed by the following commit.\n + hash_to_hex_algop(midx_get_checksum_hash(m),\n + m->source->odb->repo->hash_algo);\n \n - This occurs when providing a MIDX layer checksum other than the tip. As\n - we walk over the MIDX chain trying to find the matching layer, we drop\n - our reference to the top-most MIDX layer. Thus, our call to\n - 'close_midx()' later on leaks memory between the top-most MIDX layer and\n - the MIDX layer immediately following the specified one.\n + , which is both cumbersome and repetitive. In fact, all but a handful of\n + callers to `midx_get_checksum_hash()` do exactly the above. Reduce the\n + repetitive nature of calling `midx_get_checksum_hash()` by having it\n + return a pointer into a static buffer containing the above result.\n \n - Plug this leak by holding a reference to the tip of the MIDX chain, and\n - ensure that we call `close_midx()` before terminating the test tool.\n + For the handful of callers that do need to compare the raw bytes and\n + don't want to deal with an encoded copy (e.g., because they are passing\n + it to hasheq() or similar), they may still rely on\n + `midx_get_checksum_hash()` which returns the raw bytes.\n \n Signed-off-by: Taylor Blau \n \n - ## t/helper/test-read-midx.c ##\n -@@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir, const char *checksum,\n - \t\t\t int show_objects)\n + ## midx-write.c ##\n +@@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n + \t\twhile (m) {\n + \t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n + \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n +-\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(m),\n +-\t\t\t\t\t\t\tm->source->odb->repo->hash_algo));\n ++\t\t\t\t midx_get_checksum_hex(m));\n + \t\t\t\tgoto cleanup;\n + \t\t\t}\n + \t\t\tctx.num_multi_pack_indexes_before++;\n +@@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n + \t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n + \t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n + \n +-\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(midx_get_checksum_hash(m),\n +-\t\t\t\t\t\t\t\t r->hash_algo));\n ++\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n + \t\t\tm = m->base_midx;\n + \t\t}\n + \n +\n + ## midx.c ##\n +@@ midx.c: void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n + int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n + \t\t\t const char *idx_name);\n + \n ++const char *midx_get_checksum_hex(const struct multi_pack_index *m)\n ++{\n ++\treturn hash_to_hex_algop(midx_get_checksum_hash(m),\n ++\t\t\t\t m->source->odb->repo->hash_algo);\n ++}\n ++\n + const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m)\n {\n - \tuint32_t i;\n --\tstruct multi_pack_index *m;\n -+\tstruct multi_pack_index *m, *tip;\n -+\tint ret = 0;\n + \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\n +\n + ## midx.h ##\n +@@ midx.h: struct multi_pack_index {\n + #define MIDX_EXT_BITMAP \"bitmap\"\n + #define MIDX_EXT_MIDX \"midx\"\n \n --\tm = setup_midx(object_dir);\n -+\tm = tip = setup_midx(object_dir);\n ++const char *midx_get_checksum_hex(const struct multi_pack_index *m) /* static buffer */;\n + const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m);\n + void get_midx_filename(struct odb_source *source, struct strbuf *out);\n + void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\n +\n + ## pack-bitmap.c ##\n +@@ pack-bitmap.c: void test_bitmap_walk(struct rev_info *revs)\n \n - \tif (!m)\n - \t\treturn 1;\n + \t\tif (bitmap_is_midx(found))\n + \t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n +-\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(found->midx),\n +-\t\t\t\t\t\t revs->repo->hash_algo));\n ++\t\t\t\t midx_get_checksum_hex(found->midx));\n + \t\telse\n + \t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n + \t\t\t\t hash_to_hex_algop(found->pack->hash,\n +\n + ## t/helper/test-read-midx.c ##\n @@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir, const char *checksum,\n + \t\treturn 1;\n + \n \tif (checksum) {\n - \t\twhile (m && strcmp(get_midx_checksum(m), checksum))\n +-\t\twhile (m && strcmp(hash_to_hex(midx_get_checksum_hash(m)), checksum))\n ++\t\twhile (m && strcmp(midx_get_checksum_hex(m), checksum))\n \t\t\tm = m->base_midx;\n --\t\tif (!m)\n --\t\t\treturn 1;\n -+\t\tif (!m) {\n -+\t\t\tret = error(_(\"could not find MIDX with checksum %s\"),\n -+\t\t\t\t checksum);\n -+\t\t\tgoto out;\n -+\t\t}\n - \t}\n - \n - \tprintf(\"header: %08x %d %d %d %d\\n\",\n -@@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir, const char *checksum,\n - \t\t}\n - \t}\n - \n --\tclose_midx(m);\n -+out:\n -+\tclose_midx(tip);\n - \n --\treturn 0;\n -+\treturn ret;\n - }\n - \n - static int read_midx_checksum(const char *object_dir)\n + \t\tif (!m)\n + \t\t\treturn 1;\n +@@ t/helper/test-read-midx.c: static int read_midx_checksum(const char *object_dir)\n + \tm = setup_midx(object_dir);\n + \tif (!m)\n + \t\treturn 1;\n +-\tprintf(\"%s\\n\", hash_to_hex(midx_get_checksum_hash(m)));\n ++\tprintf(\"%s\\n\", midx_get_checksum_hex(m));\n + \n + \tclose_midx(m);\n + \treturn 0;\n 3: d35c709632b = 4: 2aedd72db8c builtin/multi-pack-index.c: make '--progress' a common option\n 4: 6111a1f5abd = 5: a00598a36a3 git-multi-pack-index(1): remove non-existent incompatibility\n 5: 630a53adbd6 = 6: 92e6d868a45 git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'\n 6: 6dd59967876 = 7: ff599c11f68 t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39\n 7: 7f498a02ce7 ! 8: 315a0ea2985 midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`\n @@ Metadata\n ## Commit message ##\n midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`\n \n - In midx_pack_order(), we compute for each bitampped pack the first bit\n + In midx_pack_order(), we compute for each bitmapped pack the first bit\n to correspond to an object in that pack, along with how many bits were\n assigned to object(s) in that pack.\n \n 8: 687a2f50337 ! 9: af174e22e1e midx-write.c: introduce `struct write_midx_opts`\n @@ Commit message\n Signed-off-by: Taylor Blau \n \n ## midx-write.c ##\n -@@ midx-write.c: static void clear_midx_files(struct odb_source *source,\n - \tstrbuf_release(&buf);\n +@@ midx-write.c: static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n + \treturn needed;\n }\n \n -static int write_midx_internal(struct odb_source *source,\n @@ midx-write.c: static void clear_midx_files(struct odb_source *source,\n -\t\t\t const char *refs_snapshot,\n -\t\t\t unsigned flags)\n +struct write_midx_opts {\n -+\tstruct odb_source *source;\n ++\tstruct odb_source *source; /* non-optional */\n +\n +\tstruct string_list *packs_to_include;\n +\tstruct string_list *packs_to_drop;\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n -\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n +\t\t\tif (opts->flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n - \t\t\t\t get_midx_checksum(m));\n + \t\t\t\t midx_get_checksum_hex(m));\n \t\t\t\tgoto cleanup;\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n \tstart_pack = ctx.nr;\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n +\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n - \tif ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) &&\n - \t !ctx.incremental &&\n --\t !(packs_to_include || packs_to_drop)) {\n -+\t !(opts->packs_to_include || opts->packs_to_drop)) {\n - \t\tstruct bitmap_index *bitmap_git;\n - \t\tint bitmap_exists;\n --\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n -+\t\tint want_bitmap = opts->flags & MIDX_WRITE_BITMAP;\n +-\tif (!packs_to_drop) {\n ++\tif (!opts->packs_to_drop) {\n + \t\t/*\n + \t\t * If there is no MIDX then either it doesn't exist, or we're\n + \t\t * doing a geometric repack. Try to load it from the source to\n +@@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n + \t\tif (midx && !midx_needs_update(midx, &ctx)) {\n + \t\t\tstruct bitmap_index *bitmap_git;\n + \t\t\tint bitmap_exists;\n +-\t\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n ++\t\t\tint want_bitmap = opts->flags & MIDX_WRITE_BITMAP;\n \n - \t\tbitmap_git = prepare_midx_bitmap_git(ctx.m);\n - \t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n + \t\t\tbitmap_git = prepare_midx_bitmap_git(midx);\n + \t\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n - \t\t\t * corresponding bitmap (or one wasn't requested).\n - \t\t\t */\n - \t\t\tif (!want_bitmap)\n --\t\t\t\tclear_midx_files_ext(source, \"bitmap\", NULL);\n -+\t\t\t\tclear_midx_files_ext(opts->source, \"bitmap\",\n -+\t\t\t\t\t\t NULL);\n - \t\t\tresult = 0;\n - \t\t\tgoto cleanup;\n - \t\t}\n + \t\t\t\t * corresponding bitmap (or one wasn't requested).\n + \t\t\t\t */\n + \t\t\t\tif (!want_bitmap)\n +-\t\t\t\t\tclear_midx_files_ext(source, \"bitmap\", NULL);\n ++\t\t\t\t\tclear_midx_files_ext(ctx.source, \"bitmap\", NULL);\n + \t\t\t\tresult = 0;\n + \t\t\t\tgoto cleanup;\n + \t\t\t}\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n \t\tgoto cleanup; /* nothing to do */\n \t}\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n -\tif (flags & MIDX_WRITE_BITMAP) {\n +\tif (opts->flags & MIDX_WRITE_BITMAP) {\n \t\tstruct packing_data pdata;\n - \t\tstruct commit **commits;\n - \t\tuint32_t commits_nr;\n + \t\tstruct commit_stack commits = COMMIT_STACK_INIT;\n + \n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n \n \t\tprepare_midx_packing_data(&pdata, &ctx);\n \n --\t\tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, &ctx);\n -+\t\tcommits = find_commits_for_midx_bitmap(&commits_nr, opts->refs_snapshot, &ctx);\n +-\t\tfind_commits_for_midx_bitmap(&commits, refs_snapshot, &ctx);\n ++\t\tfind_commits_for_midx_bitmap(&commits, opts->refs_snapshot, &ctx);\n \n \t\t/*\n \t\t * The previous steps translated the information from\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n + \t\tFREE_AND_NULL(ctx.entries);\n + \t\tctx.entries_nr = 0;\n \n - \t\tif (write_midx_bitmap(&ctx,\n - \t\t\t\t midx_hash, &pdata, commits, commits_nr,\n --\t\t\t\t flags) < 0) {\n -+\t\t\t\t opts->flags) < 0) {\n +-\t\tif (write_midx_bitmap(&ctx, midx_hash, &pdata,\n +-\t\t\t\t commits.items, commits.nr, flags) < 0) {\n ++\t\tif (write_midx_bitmap(&ctx, midx_hash, &pdata, commits.items,\n ++\t\t\t\t commits.nr, opts->flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tclear_packing_data(&pdata);\n - \t\t\tfree(commits);\n + \t\t\tcommit_stack_clear(&commits);\n @@ midx-write.c: static int write_midx_internal(struct odb_source *source,\n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n \t\t\tgoto cleanup;\n @@ midx-write.c: int write_midx_file_only(struct odb_source *source,\n \n int expire_midx_packs(struct odb_source *source, unsigned flags)\n @@ midx-write.c: int expire_midx_packs(struct odb_source *source, unsigned flags)\n + \n \tfree(count);\n \n - \tif (packs_to_drop.nr)\n +-\tif (packs_to_drop.nr)\n -\t\tresult = write_midx_internal(source, NULL,\n -\t\t\t\t\t &packs_to_drop, NULL, NULL, flags);\n -+\t\tresult = write_midx_internal(&(struct write_midx_opts) {\n -+\t\t\t\t\t .source = source,\n -+\t\t\t\t\t .packs_to_drop = &packs_to_drop,\n -+\t\t\t\t\t .flags = flags & MIDX_PROGRESS,\n -+\t\t\t\t\t });\n ++\tif (packs_to_drop.nr) {\n ++\t\tstruct write_midx_opts opts = {\n ++\t\t\t.source = source,\n ++\t\t\t.packs_to_drop = &packs_to_drop,\n ++\t\t\t.flags = flags & MIDX_PROGRESS,\n ++\t\t};\n ++\t\tresult = write_midx_internal(&opts);\n ++\t}\n \n \tstring_list_clear(&packs_to_drop, 0);\n \n +@@ midx-write.c: int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags)\n + \tstruct child_process cmd = CHILD_PROCESS_INIT;\n + \tFILE *cmd_in;\n + \tstruct multi_pack_index *m = get_multi_pack_index(source);\n ++\tstruct write_midx_opts opts = {\n ++\t\t.source = source,\n ++\t\t.flags = flags,\n ++\t};\n + \n + \t/*\n + \t * When updating the default for these configuration\n @@ midx-write.c: int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags)\n \t\tgoto cleanup;\n \t}\n \n -\tresult = write_midx_internal(source, NULL, NULL, NULL, NULL,\n -\t\t\t\t flags);\n -+\tresult = write_midx_internal(&(struct write_midx_opts) {\n -+\t\t\t\t .source = source,\n -+\t\t\t\t .flags = flags,\n -+\t\t\t\t });\n ++\tresult = write_midx_internal(&opts);\n \n cleanup:\n \tfree(include_pack);\n 9: 66ae4bc8c0a ! 10: 72bcd4ed6c7 midx: do not require packs to be sorted in lexicographic order\n @@ Commit message\n lazily instantiate a `pack_names_sorted` array on the MIDX, which will\n be used to implement the binary search over pack names.\n \n - Note that this produces MIDXs which may be incompatible with earlier\n - versions of Git that have stricter requirements on the layout of packs\n - within a MIDX. This patch does *not* modify the version number of the\n - MIDX format, since existing versions of Git already know to gracefully\n - ignore a MIDX with packs that appear out-of-order.\n + Because this change produces MIDXs which may not be correctly read with\n + external tools or older versions of Git. Though older versions of Git\n + know how to gracefully degrade and ignore any MIDX(s) they consider\n + corrupt, external tools may not be as robust. To avoid unintentionally\n + breaking any such tools, guard this change behind a version bump in the\n + MIDX's on-disk format.\n \n Signed-off-by: Taylor Blau \n \n + ## Documentation/gitformat-pack.adoc ##\n +@@ Documentation/gitformat-pack.adoc: HEADER:\n + \t The signature is: {'M', 'I', 'D', 'X'}\n + \n + \t1-byte version number:\n +-\t Git only writes or recognizes version 1.\n ++\t Git only writes version 2, but recognizes versions 1 and 2.\n + \n + \t1-byte Object Id Version\n + \t We infer the length of object IDs (OIDs) from this value:\n +@@ Documentation/gitformat-pack.adoc: CHUNK DATA:\n + \t strings. There is no extra padding between the filenames,\n + \t and they are listed in lexicographic order. The chunk itself\n + \t is padded at the end with between 0 and 3 NUL bytes to make the\n +-\t chunk size a multiple of 4 bytes.\n ++\t chunk size a multiple of 4 bytes. Version 1 MIDXs are required to\n ++\t list their packs in lexicographic order, but version 2 MIDXs may\n ++\t list their packs in any arbitrary order.\n + \n + \tBitmapped Packfiles (ID: {'B', 'T', 'M', 'P'})\n + \t Stores a table of two 4-byte unsigned integers in network order.\n +\n ## midx-write.c ##\n +@@ midx-write.c: extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n + \n + static size_t write_midx_header(const struct git_hash_algo *hash_algo,\n + \t\t\t\tstruct hashfile *f, unsigned char num_chunks,\n +-\t\t\t\tuint32_t num_packs)\n ++\t\t\t\tuint32_t num_packs, int version)\n + {\n ++\tif (version != MIDX_VERSION_V1 && version != MIDX_VERSION_V2)\n ++\t\tBUG(\"unexpected MIDX version: %d\", version);\n ++\n + \thashwrite_be32(f, MIDX_SIGNATURE);\n +-\thashwrite_u8(f, MIDX_VERSION);\n ++\thashwrite_u8(f, version);\n + \thashwrite_u8(f, oid_version(hash_algo));\n + \thashwrite_u8(f, num_chunks);\n + \thashwrite_u8(f, 0); /* unused */\n +@@ midx-write.c: struct write_midx_context {\n + \n + \tuint32_t preferred_pack_idx;\n + \n ++\tint version; /* must be MIDX_VERSION_V1 or _V2 */\n ++\n + \tint incremental;\n + \tuint32_t num_multi_pack_indexes_before;\n + \n @@ midx-write.c: static int write_midx_pack_names(struct hashfile *f, void *data)\n \t\tif (ctx->info[i].expired)\n \t\t\tcontinue;\n \n -\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n --\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n --\t\t\t ctx->info[i - 1].pack_name,\n --\t\t\t ctx->info[i].pack_name);\n --\n - \t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n - \t\thashwrite(f, ctx->info[i].pack_name, writelen);\n - \t\twritten += writelen;\n ++\t\tif (ctx->version == MIDX_VERSION_V1 &&\n ++\t\t i && strcmp(ctx->info[i].pack_name,\n ++\t\t\t\tctx->info[i - 1].pack_name) <= 0)\n + \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n + \t\t\t ctx->info[i - 1].pack_name,\n + \t\t\t ctx->info[i].pack_name);\n +@@ midx-write.c: static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n + \tif (!midx_checksum_valid(midx))\n + \t\tgoto out;\n + \n ++\t/*\n ++\t * If the version differs, we need to update.\n ++\t */\n ++\tif (midx->version != ctx->version)\n ++\t\tgoto out;\n ++\n + \t/*\n + \t * Ignore incremental updates for now. The assumption is that any\n + \t * incremental update would be either empty (in which case we will bail\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \tstruct tempfile *incr;\n + \tstruct write_midx_context ctx = {\n + \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n ++\t\t.version = MIDX_VERSION_V2,\n + \t };\n + \tstruct multi_pack_index *midx_to_free = NULL;\n + \tint bitmapped_packs_concat_len = 0;\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \tctx.repo = r;\n + \tctx.source = opts->source;\n + \n ++\trepo_config_get_int(ctx.repo, \"midx.version\", &ctx.version);\n ++\tif (ctx.version != MIDX_VERSION_V1 && ctx.version != MIDX_VERSION_V2)\n ++\t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n ++\n + \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n + \n + \tif (ctx.incremental)\n +@@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n + \t}\n + \n + \twrite_midx_header(r->hash_algo, f, get_num_chunks(cf),\n +-\t\t\t ctx.nr - dropped_packs);\n ++\t\t\t ctx.nr - dropped_packs, ctx.version);\n + \twrite_chunkfile(cf, &ctx);\n + \n + \tfinalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,\n \n ## midx.c ##\n @@ midx.c: static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *sou\n - \t\tif (!end)\n + \t\t m->signature, MIDX_SIGNATURE);\n + \n + \tm->version = m->data[MIDX_BYTE_FILE_VERSION];\n +-\tif (m->version != MIDX_VERSION)\n ++\tif (m->version != MIDX_VERSION_V1 && m->version != MIDX_VERSION_V2)\n + \t\tdie(_(\"multi-pack-index version %d not recognized\"),\n + \t\t m->version);\n + \n +@@ midx.c: static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *sou\n \t\t\tdie(_(\"multi-pack-index pack-name chunk is too short\"));\n \t\tcur_pack_name = end + 1;\n --\n + \n -\t\tif (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0)\n --\t\t\tdie(_(\"multi-pack-index pack names out of order: '%s' before '%s'\"),\n --\t\t\t m->pack_names[i - 1],\n --\t\t\t m->pack_names[i]);\n - \t}\n - \n - \ttrace2_data_intmax(\"midx\", r, \"load/num_packs\", m->num_packs);\n ++\t\tif (m->version == MIDX_VERSION_V1 &&\n ++\t\t i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0)\n + \t\t\tdie(_(\"multi-pack-index pack names out of order: '%s' before '%s'\"),\n + \t\t\t m->pack_names[i - 1],\n + \t\t\t m->pack_names[i]);\n @@ midx.c: void close_midx(struct multi_pack_index *m)\n \t}\n \tFREE_AND_NULL(m->packs);\n @@ midx.c: int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n -+\tif (!m->pack_names_sorted) {\n ++\tif (m->version == MIDX_VERSION_V2 && !m->pack_names_sorted) {\n +\t\tuint32_t i;\n +\n +\t\tALLOC_ARRAY(m->pack_names_sorted, m->num_packs);\n @@ midx.c: int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\tint cmp;\n \n -\t\tcurrent = m->pack_names[mid];\n -+\t\tcurrent = m->pack_names[m->pack_names_sorted[mid]];\n ++\t\tif (m->pack_names_sorted)\n ++\t\t\tcurrent = m->pack_names[m->pack_names_sorted[mid]];\n ++\t\telse\n ++\t\t\tcurrent = m->pack_names[mid];\n \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n \t\tif (!cmp)\n \t\t\treturn 1;\n \n ## midx.h ##\n +@@ midx.h: struct git_hash_algo;\n + struct odb_source;\n + \n + #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n +-#define MIDX_VERSION 1\n ++#define MIDX_VERSION_V1 1\n ++#define MIDX_VERSION_V2 2\n + #define MIDX_BYTE_FILE_VERSION 4\n + #define MIDX_BYTE_HASH_VERSION 5\n + #define MIDX_BYTE_NUM_CHUNKS 6\n @@ midx.h: struct multi_pack_index {\n \tuint32_t num_packs_in_base;\n \n @@ midx.h: struct multi_pack_index {\n \n \n ## t/t5319-multi-pack-index.sh ##\n +@@ t/t5319-multi-pack-index.sh: midx_read_expect () {\n + \tEXTRA_CHUNKS=\"$5\"\n + \t{\n + \t\tcat <<-EOF &&\n +-\t\theader: 4d494458 1 $HASH_LEN $NUM_CHUNKS $NUM_PACKS\n ++\t\theader: 4d494458 2 $HASH_LEN $NUM_CHUNKS $NUM_PACKS\n + \t\tchunks: pack-names oid-fanout oid-lookup object-offsets$EXTRA_CHUNKS\n + \t\tnum_objects: $NUM_OBJECTS\n + \t\tpacks:\n @@ t/t5319-multi-pack-index.sh: test_expect_success 'verify invalid chunk offset' '\n \t\t\"improper chunk offset(s)\"\n '\n @@ t/t5319-multi-pack-index.sh: test_expect_success 'verify invalid chunk offset' '\n test_expect_success 'verify missing pack' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"a\" $objdir \\\n \t\t\"failed to load pack\"\n +@@ t/t5319-multi-pack-index.sh: test_expect_success 'verify incorrect checksum' '\n + \t\t$objdir \"incorrect checksum\"\n + '\n + \n ++test_expect_success 'setup for v1-specific fsck tests' '\n ++\tgit -c midx.version=1 multi-pack-index write\n ++'\n ++\n ++test_expect_success 'verify packnames out of order (v1)' '\n ++\tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"z\" $objdir \\\n ++\t\t\"pack names out of order\"\n ++'\n ++\n + test_expect_success 'repack progress off for redirected stderr' '\n + \tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir repack 2>err &&\n + \ttest_line_count = 0 err\n10: 9231fdca112 = 11: c0c1769464b git-compat-util.h: introduce `u32_add()`\n11: 12211c9ad53 = 12: c11214a51f0 midx-write.c: introduce `midx_pack_perm()` helper\n12: 978d7bf8bbc = 13: b9244a04297 midx-write.c: extract `fill_pack_from_midx()`\n13: 61deab3f731 = 14: c6f8d323477 midx-write.c: enumerate `pack_int_id` values directly\n14: 4f920a328ef = 15: e71aa575463 midx-write.c: factor fanout layering from `compute_sorted_entries()`\n -: ----------- > 16: dbbcb494563 t/helper/test-read-midx.c: plug memory leak when selecting layer\n16: 088673c762d ! 17: 13336e864f4 midx: implement MIDX compaction\n @@ Documentation/git-multi-pack-index.adoc: marker).\n +--\n +\t--incremental::\n +\t\tWrite the result to a MIDX chain instead of writing a\n -+\t\tstand-alone MIDX. Incompatible with `--bitmap`.\n ++\t\tstand-alone MIDX.\n +--\n +\n verify::\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n +\tm = get_multi_pack_index(source);\n +\n +\tfor (cur = m; cur && !(from_midx && to_midx); cur = cur->base_midx) {\n -+\t\tconst char *midx_csum = get_midx_checksum(cur);\n ++\t\tconst char *midx_csum = midx_get_checksum_hex(cur);\n +\n +\t\tif (!from_midx && !strcmp(midx_csum, argv[0]))\n +\t\t\tfrom_midx = cur;\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n +\t}\n +\n +\tif (!from_midx)\n -+\t\tdie(_(\"could not find MIDX 'from': %s\"), argv[0]);\n ++\t\tdie(_(\"could not find MIDX: %s\"), argv[0]);\n +\tif (!to_midx)\n -+\t\tdie(_(\"could not find MIDX 'to': %s\"), argv[1]);\n ++\t\tdie(_(\"could not find MIDX: %s\"), argv[1]);\n ++\tif (from_midx == to_midx)\n ++\t\tdie(_(\"MIDX compaction endpoints must be unique\"));\n ++\n ++\tfor (m = from_midx; m; m = m->base_midx) {\n ++\t\tif (m == to_midx)\n ++\t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n ++\t}\n +\n +\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n +\n @@ midx-write.c: static int fill_packs_from_midx(struct write_midx_context *ctx)\n static struct {\n \tconst char *non_split;\n \tconst char *split;\n -@@ midx-write.c: static void clear_midx_files(struct odb_source *source,\n - \tstrbuf_release(&buf);\n +@@ midx-write.c: static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n + \tif (ctx->incremental)\n + \t\tgoto out;\n + \n ++\tif (ctx->compact)\n ++\t\tgoto out; /* Compaction always requires an update. */\n ++\n + \t/*\n + \t * Otherwise, we need to verify that the packs covered by the existing\n + \t * MIDX match the packs that we already have. The logic to do so is way\n +@@ midx-write.c: static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n + \treturn needed;\n }\n \n +static int midx_hashcmp(const struct multi_pack_index *a,\n +\t\t\tconst struct multi_pack_index *b,\n +\t\t\tconst struct git_hash_algo *algop)\n +{\n -+\treturn hashcmp(get_midx_hash(a), get_midx_hash(b), algop);\n ++\treturn hashcmp(midx_get_checksum_hash(a), midx_get_checksum_hash(b),\n ++\t\t algop);\n +}\n +\n struct write_midx_opts {\n - \tstruct odb_source *source;\n + \tstruct odb_source *source; /* non-optional */\n \n \tstruct string_list *packs_to_include;\n \tstruct string_list *packs_to_drop;\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n - \tctx.source = opts->source;\n + \t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n \n \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n +\tctx.compact = !!(opts->flags & MIDX_WRITE_COMPACT);\n +\n +\tif (ctx.compact) {\n ++\t\tif (ctx.version != MIDX_VERSION_V2)\n ++\t\t\tdie(_(\"cannot perform MIDX compaction with v1 format\"));\n +\t\tif (!opts->compact_from)\n +\t\t\tBUG(\"expected non-NULL 'from' MIDX during compaction\");\n +\t\tif (!opts->compact_to)\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n +\t}\n \tstop_progress(&ctx.progress);\n \n - \tif ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) &&\n - \t !ctx.incremental &&\n -+\t !ctx.compact &&\n - \t !(opts->packs_to_include || opts->packs_to_drop)) {\n - \t\tstruct bitmap_index *bitmap_git;\n - \t\tint bitmap_exists;\n + \tif (!opts->packs_to_drop) {\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\tctx.large_offsets_needed = 1;\n \t}\n \n -\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n -+\tif (!ctx.compact)\n ++\tif (ctx.compact) {\n ++\t\tif (ctx.version != MIDX_VERSION_V2)\n ++\t\t\tBUG(\"performing MIDX compaction with v1 MIDX\");\n ++\t} else {\n +\t\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n ++\t}\n \n \tif (opts->packs_to_drop && opts->packs_to_drop->nr) {\n \t\tsize_t drop_index = 0;\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n +\t\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n +\t\t\t\tnum_layers_before_from++;\n \n --\t\t\tkeep_hashes[j] = xstrdup(get_midx_checksum(m));\n +-\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n -\t\t\tm = m->base_midx;\n +\t\t\tm = ctx.base_midx;\n +\t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n +\t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n +\n -+\t\t\t\tkeep_hashes[j] = xstrdup(get_midx_checksum(m));\n ++\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n +\t\t\t\tm = m->base_midx;\n +\t\t\t}\n +\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n +\t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n +\t\t\t m = m->base_midx) {\n +\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n -+\t\t\t\t\txstrdup(get_midx_checksum(m));\n ++\t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n +\t\t\t\ti++;\n +\t\t\t}\n +\t\t} else {\n @@ midx-write.c: static int write_midx_internal(struct write_midx_opts *opts)\n +\t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n +\t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n +\n -+\t\t\t\tkeep_hashes[j] = xstrdup(get_midx_checksum(m));\n ++\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n +\t\t\t\tm = m->base_midx;\n +\t\t\t}\n \t\t}\n @@ t/t5335-compact-multi-pack-index.sh (new)\n +\t(\n +\t\tcd midx-compact-lex-order &&\n +\n ++\t\tgit config maintenance.auto false &&\n ++\n +\t\twrite_packs A B C D E &&\n +\t\ttest_line_count = 5 $midx_chain &&\n +\n @@ t/t5335-compact-multi-pack-index.sh (new)\n +\t(\n +\t\tcd midx-compact-non-lex-order &&\n +\n ++\t\tgit config maintenance.auto false &&\n ++\n +\t\twrite_packs D C A B E &&\n +\t\ttest_line_count = 5 $midx_chain &&\n +\n @@ t/t5335-compact-multi-pack-index.sh (new)\n +\t)\n +'\n +\n ++test_expect_success 'setup for bogus MIDX compaction scenarios' '\n ++\tgit init midx-compact-bogus &&\n ++\t(\n ++\t\tcd midx-compact-bogus &&\n ++\n ++\t\tgit config maintenance.auto false &&\n ++\n ++\t\twrite_packs A B C\n ++\t)\n ++'\n ++\n ++test_expect_success 'MIDX compaction with missing endpoints' '\n ++\t(\n ++\t\tcd midx-compact-bogus &&\n ++\n ++\t\ttest_must_fail git multi-pack-index compact --incremental \\\n ++\t\t\t\"\" \"\" 2>err &&\n ++\t\ttest_grep \"could not find MIDX: \" err &&\n ++\n ++\t\ttest_must_fail git multi-pack-index compact --incremental \\\n ++\t\t\t\"\" \"$(nth_line 2 \"$midx_chain\")\" 2>err &&\n ++\t\ttest_grep \"could not find MIDX: \" err &&\n ++\n ++\t\ttest_must_fail git multi-pack-index compact --incremental \\\n ++\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \"\" 2>err &&\n ++\t\ttest_grep \"could not find MIDX: \" err\n ++\t)\n ++'\n ++\n ++test_expect_success 'MIDX compaction with reversed endpoints' '\n ++\t(\n ++\t\tcd midx-compact-bogus &&\n ++\n ++\t\tfrom=\"$(nth_line 3 \"$midx_chain\")\" &&\n ++\t\tto=\"$(nth_line 1 \"$midx_chain\")\" &&\n ++\n ++\t\ttest_must_fail git multi-pack-index compact --incremental \\\n ++\t\t\t\"$from\" \"$to\" 2>err &&\n ++\n ++\t\ttest_grep \"MIDX $from must be an ancestor of $to\" err\n ++\t)\n ++'\n ++\n ++test_expect_success 'MIDX compaction with identical endpoints' '\n ++\t(\n ++\t\tcd midx-compact-bogus &&\n ++\n ++\t\tfrom=\"$(nth_line 3 \"$midx_chain\")\" &&\n ++\t\tto=\"$(nth_line 3 \"$midx_chain\")\" &&\n ++\n ++\t\ttest_must_fail git multi-pack-index compact --incremental \\\n ++\t\t\t\"$from\" \"$to\" 2>err &&\n ++\n ++\t\ttest_grep \"MIDX compaction endpoints must be unique\" err\n ++\t)\n ++'\n ++\n ++test_expect_success 'MIDX compaction with midx.version=1' '\n ++\t(\n ++\t\tcd midx-compact-bogus &&\n ++\n ++\t\ttest_must_fail git -c midx.version=1 multi-pack-index compact \\\n ++\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n ++\t\t\t\"$(nth_line 2 \"$midx_chain\")\" 2>err &&\n ++\n ++\t\ttest_grep \"fatal: cannot perform MIDX compaction with v1 format\" err\n ++\t)\n ++'\n ++\n +test_done\n17: b96c4e04266 ! 18: b599f1ad4b0 midx: enable reachability bitmaps during MIDX compaction\n @@ Commit message\n layer compaction by combining all existing bitmaps from the compacted\n layers.\n \n - Note that the because of the object/pack ordering described by the\n - previous commit, the pseudo-pack order for the compacted MIDX is the\n - same as concatenating the individual pseudo-pack orderings for each\n - layer in the compaction range.\n + Note that because of the object/pack ordering described by the previous\n + commit, the pseudo-pack order for the compacted MIDX is the same as\n + concatenating the individual pseudo-pack orderings for each layer in the\n + compaction range.\n \n As a result, the only non-test or documentation change necessary is to\n treat all objects as non-preferred during compaction so as not to\n @@ Documentation/git-multi-pack-index.adoc: SYNOPSIS\n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n +@@ Documentation/git-multi-pack-index.adoc: compact::\n + \t--incremental::\n + \t\tWrite the result to a MIDX chain instead of writing a\n + \t\tstand-alone MIDX.\n ++\n ++\t--[no-]bitmap::\n ++\t\tControl whether or not a multi-pack bitmap is written.\n + --\n + \n + verify::\n \n ## builtin/multi-pack-index.c ##\n @@\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with n\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n \t\ttest_line_count = 3 $midx_chain &&\n -@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with non-lex-ordered pack names' '\n +@@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with midx.version=1' '\n \t)\n '\n \n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with n\n +\t(\n +\t\tcd midx-compact-preserve-selection &&\n +\n ++\t\tgit config maintenance.auto false &&\n ++\n +\t\ttest_commit A &&\n +\t\ttest_commit B &&\n +\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with n\n +\t(\n +\t\tcd midx-compact-with-bitmaps &&\n +\n ++\t\tgit config maintenance.auto false &&\n ++\n +\t\twrite_packs foo bar baz quux woot &&\n +\n +\t\ttest-tool read-midx --bitmap $objdir >bitmap.expect &&\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with n\n +\t(\n +\t\tcd midx-compact-with-bitmaps-non-trivial &&\n +\n ++\t\tgit config maintenance.auto false &&\n ++\n +\t\tgit branch -m main &&\n +\n +\t\t# D(4)\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with n\n +\t\tgit checkout main &&\n +\t\twrite_packs G &&\n +\n -+\t\tcat $midx_chain &&\n -+\n +\t\t# Compact layers 2-4, leaving us with:\n +\t\t#\n +\t\t# [A, [B, C, D], E, F, G]\n @@ t/t5335-compact-multi-pack-index.sh: test_expect_success 'MIDX compaction with n\n +\t\t# [A, [B, C, D], E, [F, G]]\n +\t\tgit multi-pack-index compact --incremental --bitmap \\\n +\t\t\t\"$(nth_line 4 \"$midx_chain\")\" \\\n -+\t\t\t\"$(nth_line 5 \"$midx_chain\")\" &&\n -+\n -+\t\tcat $midx_chain\n ++\t\t\t\"$(nth_line 5 \"$midx_chain\")\"\n +\t)\n +'\n +\n-- \n2.52.0.457.gb599f1ad4b0\n"},{"id":"533870","messageId":"2e549ea6443e8c4189a7a1c1f59812f9cd22df74.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 01/18] midx: mark `get_midx_checksum()` arguments as const","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:18Z","receivedAt":"2026-01-14T19:54:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To make clear that the function `get_midx_checksum()` does not do\nanything to modify its argument, mark the MIDX pointer as const.\n\nThe following commit will rename this function altogether to make clear\nthat it returns the raw bytes of the checksum, not a hex-encoded copy of\nit.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 2 +-\n midx.h | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex b681b18fc19..4c90eb04c64 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -24,7 +24,7 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n-const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+const unsigned char *get_midx_checksum(const struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\n }\ndiff --git a/midx.h b/midx.h\nindex 6e54d73503d..7c7e0b59121 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -85,7 +85,7 @@ struct multi_pack_index {\n #define MIDX_EXT_BITMAP \"bitmap\"\n #define MIDX_EXT_MIDX \"midx\"\n \n-const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n+const unsigned char *get_midx_checksum(const struct multi_pack_index *m);\n void get_midx_filename(struct odb_source *source, struct strbuf *out);\n void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\n \t\t\t const unsigned char *hash, const char *ext);\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533871","messageId":"7255adafe70a4a49bd83ce2db4203d8e331f49ef.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 02/18] midx: rename `get_midx_checksum()` to `midx_get_checksum_hash()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:21Z","receivedAt":"2026-01-14T19:54:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since 541204aabea (Documentation: document naming schema for structs and\ntheir functions, 2024-07-30), we have adopted a naming convention for\nfunctions that would prefer a name like, say, `midx_get_checksum()` over\n`get_midx_checksum()`.\n\nAdopt this convention throughout the midx.h API. Since this function\nreturns a raw (that is, non-hex encoded) hash, let's suffix the function\nwith \"_hash()\" to make this clear. As a side effect, this prepares us\nfor the subsequent change which will introduce a \"_hex()\" variant that\nencodes the checksum itself.\n\nSuggested-by: Patrick Steinhardt \nSigned-off-by: Taylor Blau \n---\n midx-write.c | 6 +++---\n midx.c | 2 +-\n midx.h | 2 +-\n pack-bitmap.c | 8 ++++----\n pack-revindex.c | 4 ++--\n t/helper/test-read-midx.c | 4 ++--\n 6 files changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 6485cb67068..73d33752ef1 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -946,7 +946,7 @@ static int link_midx_to_chain(struct multi_pack_index *m)\n \t}\n \n \tfor (i = 0; i < ARRAY_SIZE(midx_exts); i++) {\n-\t\tconst unsigned char *hash = get_midx_checksum(m);\n+\t\tconst unsigned char *hash = midx_get_checksum_hash(m);\n \n \t\tget_midx_filename_ext(m->source, &from,\n \t\t\t\t hash, midx_exts[i].non_split);\n@@ -1151,7 +1151,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\twhile (m) {\n \t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n-\t\t\t\t hash_to_hex_algop(get_midx_checksum(m),\n+\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(m),\n \t\t\t\t\t\t\tm->source->odb->repo->hash_algo));\n \t\t\t\tgoto cleanup;\n \t\t\t}\n@@ -1520,7 +1520,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n \t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n \n-\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(get_midx_checksum(m),\n+\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(midx_get_checksum_hash(m),\n \t\t\t\t\t\t\t\t r->hash_algo));\n \t\t\tm = m->base_midx;\n \t\t}\ndiff --git a/midx.c b/midx.c\nindex 4c90eb04c64..554bdfc505b 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -24,7 +24,7 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n-const unsigned char *get_midx_checksum(const struct multi_pack_index *m)\n+const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\n }\ndiff --git a/midx.h b/midx.h\nindex 7c7e0b59121..62d6105195f 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -85,7 +85,7 @@ struct multi_pack_index {\n #define MIDX_EXT_BITMAP \"bitmap\"\n #define MIDX_EXT_MIDX \"midx\"\n \n-const unsigned char *get_midx_checksum(const struct multi_pack_index *m);\n+const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m);\n void get_midx_filename(struct odb_source *source, struct strbuf *out);\n void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\n \t\t\t const unsigned char *hash, const char *ext);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 8ca79725b1d..90ae63e6804 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -441,11 +441,11 @@ char *midx_bitmap_filename(struct multi_pack_index *midx)\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (midx->has_chain)\n \t\tget_split_midx_filename_ext(midx->source, &buf,\n-\t\t\t\t\t get_midx_checksum(midx),\n+\t\t\t\t\t midx_get_checksum_hash(midx),\n \t\t\t\t\t MIDX_EXT_BITMAP);\n \telse\n \t\tget_midx_filename_ext(midx->source, &buf,\n-\t\t\t\t get_midx_checksum(midx),\n+\t\t\t\t midx_get_checksum_hash(midx),\n \t\t\t\t MIDX_EXT_BITMAP);\n \n \treturn strbuf_detach(&buf, NULL);\n@@ -502,7 +502,7 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \tif (load_bitmap_header(bitmap_git) < 0)\n \t\tgoto cleanup;\n \n-\tif (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum,\n+\tif (!hasheq(midx_get_checksum_hash(bitmap_git->midx), bitmap_git->checksum,\n \t\t bitmap_repo(bitmap_git)->hash_algo)) {\n \t\terror(_(\"checksum doesn't match in MIDX and bitmap\"));\n \t\tgoto cleanup;\n@@ -2820,7 +2820,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \t\tif (bitmap_is_midx(found))\n \t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n-\t\t\t\t hash_to_hex_algop(get_midx_checksum(found->midx),\n+\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(found->midx),\n \t\t\t\t\t\t revs->repo->hash_algo));\n \t\telse\n \t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 8598b941c8c..70731071ab6 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -390,11 +390,11 @@ int load_midx_revindex(struct multi_pack_index *m)\n \n \tif (m->has_chain)\n \t\tget_split_midx_filename_ext(m->source, &revindex_name,\n-\t\t\t\t\t get_midx_checksum(m),\n+\t\t\t\t\t midx_get_checksum_hash(m),\n \t\t\t\t\t MIDX_EXT_REV);\n \telse\n \t\tget_midx_filename_ext(m->source, &revindex_name,\n-\t\t\t\t get_midx_checksum(m),\n+\t\t\t\t midx_get_checksum_hash(m),\n \t\t\t\t MIDX_EXT_REV);\n \n \tret = load_revindex_from_disk(m->source->odb->repo->hash_algo,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 6de5d1665af..b8fefb1a124 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -34,7 +34,7 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\treturn 1;\n \n \tif (checksum) {\n-\t\twhile (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum))\n+\t\twhile (m && strcmp(hash_to_hex(midx_get_checksum_hash(m)), checksum))\n \t\t\tm = m->base_midx;\n \t\tif (!m)\n \t\t\treturn 1;\n@@ -94,7 +94,7 @@ static int read_midx_checksum(const char *object_dir)\n \tm = setup_midx(object_dir);\n \tif (!m)\n \t\treturn 1;\n-\tprintf(\"%s\\n\", hash_to_hex(get_midx_checksum(m)));\n+\tprintf(\"%s\\n\", hash_to_hex(midx_get_checksum_hash(m)));\n \n \tclose_midx(m);\n \treturn 0;\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533872","messageId":"25b628fda97cb4c31319ebe1585b9136bb0a57af.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 03/18] midx: introduce `midx_get_checksum_hex()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:24Z","receivedAt":"2026-01-14T19:54:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When trying to print out, say, the hexadecimal representation of a\nMIDX's hash, our code will do something like:\n\n hash_to_hex_algop(midx_get_checksum_hash(m),\n m->source->odb->repo->hash_algo);\n\n, which is both cumbersome and repetitive. In fact, all but a handful of\ncallers to `midx_get_checksum_hash()` do exactly the above. Reduce the\nrepetitive nature of calling `midx_get_checksum_hash()` by having it\nreturn a pointer into a static buffer containing the above result.\n\nFor the handful of callers that do need to compare the raw bytes and\ndon't want to deal with an encoded copy (e.g., because they are passing\nit to hasheq() or similar), they may still rely on\n`midx_get_checksum_hash()` which returns the raw bytes.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 6 ++----\n midx.c | 6 ++++++\n midx.h | 1 +\n pack-bitmap.c | 3 +--\n t/helper/test-read-midx.c | 4 ++--\n 5 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 73d33752ef1..13171d7e9c4 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1151,8 +1151,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\twhile (m) {\n \t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n-\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(m),\n-\t\t\t\t\t\t\tm->source->odb->repo->hash_algo));\n+\t\t\t\t midx_get_checksum_hex(m));\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \t\t\tctx.num_multi_pack_indexes_before++;\n@@ -1520,8 +1519,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n \t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n \n-\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(midx_get_checksum_hash(m),\n-\t\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n \t\t\tm = m->base_midx;\n \t\t}\n \ndiff --git a/midx.c b/midx.c\nindex 554bdfc505b..19ef230d3fd 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -24,6 +24,12 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n+const char *midx_get_checksum_hex(const struct multi_pack_index *m)\n+{\n+\treturn hash_to_hex_algop(midx_get_checksum_hash(m),\n+\t\t\t\t m->source->odb->repo->hash_algo);\n+}\n+\n const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\ndiff --git a/midx.h b/midx.h\nindex 62d6105195f..a39bcc9d03f 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -85,6 +85,7 @@ struct multi_pack_index {\n #define MIDX_EXT_BITMAP \"bitmap\"\n #define MIDX_EXT_MIDX \"midx\"\n \n+const char *midx_get_checksum_hex(const struct multi_pack_index *m) /* static buffer */;\n const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m);\n void get_midx_filename(struct odb_source *source, struct strbuf *out);\n void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 90ae63e6804..b688531fd2f 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2820,8 +2820,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \t\tif (bitmap_is_midx(found))\n \t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n-\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(found->midx),\n-\t\t\t\t\t\t revs->repo->hash_algo));\n+\t\t\t\t midx_get_checksum_hex(found->midx));\n \t\telse\n \t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n \t\t\t\t hash_to_hex_algop(found->pack->hash,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex b8fefb1a124..9d42c587564 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -34,7 +34,7 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\treturn 1;\n \n \tif (checksum) {\n-\t\twhile (m && strcmp(hash_to_hex(midx_get_checksum_hash(m)), checksum))\n+\t\twhile (m && strcmp(midx_get_checksum_hex(m), checksum))\n \t\t\tm = m->base_midx;\n \t\tif (!m)\n \t\t\treturn 1;\n@@ -94,7 +94,7 @@ static int read_midx_checksum(const char *object_dir)\n \tm = setup_midx(object_dir);\n \tif (!m)\n \t\treturn 1;\n-\tprintf(\"%s\\n\", hash_to_hex(midx_get_checksum_hash(m)));\n+\tprintf(\"%s\\n\", midx_get_checksum_hex(m));\n \n \tclose_midx(m);\n \treturn 0;\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533873","messageId":"2aedd72db8c6a6bc776e003e306a5d9ef446bc32.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 04/18] builtin/multi-pack-index.c: make '--progress' a common option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:27Z","receivedAt":"2026-01-14T19:54:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"All multi-pack-index sub-commands (write, verify, repack, and expire)\nsupport a '--progress' command-line option, despite not listing it as\none of the common options in `common_opts`.\n\nAs a result each sub-command declares its own `OPT_BIT()` for a\n\"--progress\" command-line option. Centralize this within the\n`common_opts` to avoid re-declaring it in each sub-command.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 2 ++\n builtin/multi-pack-index.c | 10 ++--------\n 2 files changed, 4 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 2f642697e9e..a4550e28bed 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -18,6 +18,8 @@ Write or verify a multi-pack-index (MIDX) file.\n OPTIONS\n -------\n \n+The following command-line options are applicable to all sub-commands:\n+\n --object-dir=::\n \tUse given directory for the location of Git objects. We check\n \t`/packs/multi-pack-index` for the current MIDX file, and\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5f364aa816b..ca98d4c3ba3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -84,6 +84,8 @@ static struct option common_opts[] = {\n \t N_(\"directory\"),\n \t N_(\"object directory containing set of packfile and pack-index pairs\"),\n \t parse_object_dir),\n+\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"),\n+\t\tMIDX_PROGRESS),\n \tOPT_END(),\n };\n \n@@ -138,8 +140,6 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n@@ -200,8 +200,6 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv,\n {\n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_verify_options[] = {\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_END(),\n \t};\n \tstruct odb_source *source;\n@@ -231,8 +229,6 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv,\n {\n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_expire_options[] = {\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_END(),\n \t};\n \tstruct odb_source *source;\n@@ -264,8 +260,6 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_repack_options[] = {\n \t\tOPT_UNSIGNED(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t N_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_END(),\n \t};\n \tstruct odb_source *source;\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533874","messageId":"a00598a36a31e3efcf45f35b2cf4fc2e3a4de958.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 05/18] git-multi-pack-index(1): remove non-existent incompatibility","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:30Z","receivedAt":"2026-01-14T19:54:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since fcb2205b774 (midx: implement support for writing incremental MIDX\nchains, 2024-08-06), the command-line options '--incremental' and\n'--bitmap' were declared to be incompatible with one another when\nrunning 'git multi-pack-index write'.\n\nHowever, since 27afc272c49 (midx: implement writing incremental MIDX\nbitmaps, 2025-03-20), that incompatibility no longer exists, despite the\ndocumentation saying so. Correct this by removing the stale reference to\ntheir incompatibility.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex a4550e28bed..a502819fc38 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -75,7 +75,7 @@ marker).\n \t\tWrite an incremental MIDX file containing only objects\n \t\tand packs not present in an existing MIDX layer.\n \t\tMigrates non-incremental MIDXs to incremental ones when\n-\t\tnecessary. Incompatible with `--bitmap`.\n+\t\tnecessary.\n --\n \n verify::\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533876","messageId":"92e6d868a45444eda33a9ea3deedfab088bbc62e.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 06/18] git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:33Z","receivedAt":"2026-01-14T19:54:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since c39fffc1c90 (tests: start asserting that *.txt SYNOPSIS matches -h\noutput, 2022-10-13), the manual page for 'git multi-pack-index' has a\nSYNOPSIS section which differs from 'git multi-pack-index -h'.\n\nCorrect this while also documenting additional options accepted by the\n'write' sub-command.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 7 ++++++-\n builtin/multi-pack-index.c | 5 +++--\n t/t0450/adoc-help-mismatches | 1 -\n 3 files changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex a502819fc38..164cf1f2291 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -9,7 +9,12 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] [--[no-]bitmap] \n+'git multi-pack-index' [] write [--preferred-pack=]\n+\t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n+\t\t [--refs-snapshot=]\n+'git multi-pack-index' [] verify\n+'git multi-pack-index' [] expire\n+'git multi-pack-index' [] repack [--batch-size=]\n \n DESCRIPTION\n -----------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex ca98d4c3ba3..c0c6c1760c0 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -13,8 +13,9 @@\n #include \"repository.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n-\tN_(\"git multi-pack-index [] write [--preferred-pack=]\" \\\n-\t \"[--refs-snapshot=]\")\n+\tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n+\t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n+\t \" [--refs-snapshot=]\")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\ndiff --git a/t/t0450/adoc-help-mismatches b/t/t0450/adoc-help-mismatches\nindex 8ee2d3f7c81..e8d6c13ccd0 100644\n--- a/t/t0450/adoc-help-mismatches\n+++ b/t/t0450/adoc-help-mismatches\n@@ -33,7 +33,6 @@ merge\n merge-file\n merge-index\n merge-one-file\n-multi-pack-index\n name-rev\n notes\n push\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533875","messageId":"ff599c11f68859a6189a5b67a174784e90fe4a7e.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 07/18] t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:36Z","receivedAt":"2026-01-14T19:54:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Commit d4bf1d88b90 (multi-pack-index: verify missing pack, 2018-09-13)\nadds a new test to the MIDX test script to test how we handle missing\npacks.\n\nWhile the commit itself describes the test as \"verify missing pack[s]\",\nthe test itself is actually called \"verify packnames out of order\",\ndespite that not being what it tests.\n\nLikely this was a copy-and-paste of the test immediately above it of the\nsame name. Correct this by renaming the test to match the commit\nmessage.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex faae98c7e76..efeab4d22b7 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -517,7 +517,7 @@ test_expect_success 'verify packnames out of order' '\n \t\t\"pack names out of order\"\n '\n \n-test_expect_success 'verify packnames out of order' '\n+test_expect_success 'verify missing pack' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"a\" $objdir \\\n \t\t\"failed to load pack\"\n '\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533877","messageId":"315a0ea2985894f5ae5f166bdd1685ce0981fb23.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 08/18] midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:39Z","receivedAt":"2026-01-14T19:54:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In midx_pack_order(), we compute for each bitmapped pack the first bit\nto correspond to an object in that pack, along with how many bits were\nassigned to object(s) in that pack.\n\nInitially, each bitmap_nr value is set to zero, and each bitmap_pos\nvalue is set to the sentinel BITMAP_POS_UNKNOWN. This is done to ensure\nthat there are no packs who have an unknown bit position but a somehow\nnon-zero number of objects (cf. `write_midx_bitmapped_packs()` in\nmidx-write.c).\n\nOnce the pack order is fully determined, midx_pack_order() sets the\nbitmap_pos field for any bitmapped packs to zero if they are still\nlisted as BITMAP_POS_UNKNOWN.\n\nHowever, we enumerate the bitmapped packs in order of `ctx->pack_perm`.\nThis is fine for existing cases, since the only time the\n`ctx->pack_perm` array holds a value outside of the addressable range of\n`ctx->info` is when there are expired packs, which only occurs via 'git\nmulti-pack-index expire', which does not support writing MIDX bitmaps.\nAs a result, the range of ctx->pack_perm covers all values in [0,\n`ctx->nr`), so enumerating in this order isn't an issue.\n\nA future change necessary for compaction will complicate this further by\nintroducing a wrapper around the `ctx->pack_perm` array, which turns the\ngiven `pack_int_id` into one that is relative to the lower end of the\ncompaction range. As a result, indexing into `ctx->pack_perm` through\nthis helper, say, with \"0\" will produce a crash when the lower end of\nthe compaction range has >0 pack(s) in its base layer, since the\nsubtraction will wrap around the 32-bit unsigned range, resulting in an\nuninitialized read.\n\nBut the process is completely unnecessary in the first place: we are\nenumerating all values of `ctx->info`, and there is no reason to process\nthem in a different order than they appear in memory. Index `ctx->info`\ndirectly to reflect that.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 13171d7e9c4..da9c5a7c295 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -637,7 +637,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \t\tpack_order[i] = data[i].nr;\n \t}\n \tfor (i = 0; i < ctx->nr; i++) {\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n+\t\tstruct pack_info *pack = &ctx->info[i];\n \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n \t\t\tpack->bitmap_pos = 0;\n \t}\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533878","messageId":"af174e22e1e3c59956b5f554b3515bb4b3f2cdba.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 09/18] midx-write.c: introduce `struct write_midx_opts`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:42Z","receivedAt":"2026-01-14T19:54:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the MIDX writing code, there are four functions which perform some\nsort of MIDX write operation. They are:\n\n - write_midx_file()\n - write_midx_file_only()\n - expire_midx_packs()\n - midx_repack()\n\nAll of these functions are thin wrappers over `write_midx_internal()`,\nwhich implements the bulk of these routines. As a result, the\n`write_midx_internal()` function takes six arguments.\n\nFuture commits in this series will want to add additional arguments, and\nin general this function's signature will be the union of parameters\namong *all* possible ways to write a MIDX.\n\nInstead of adding yet more arguments to this function to support MIDX\ncompaction, introduce a `struct write_midx_opts`, which has the same\nstruct members as `write_midx_internal()`'s arguments.\n\nAdding additional fields to the `write_midx_opts` struct is preferable\nto adding additional arguments to `write_midx_internal()`. This is\nbecause the callers below all zero-initialize the struct, so each time\nwe add a new piece of information, we do not have to pass the zero value\nfor it in all other call-sites that do not care about it.\n\nFor now, no functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 135 ++++++++++++++++++++++++++++++---------------------\n 1 file changed, 81 insertions(+), 54 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex da9c5a7c295..8a54644e427 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1078,14 +1078,20 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \treturn needed;\n }\n \n-static int write_midx_internal(struct odb_source *source,\n-\t\t\t struct string_list *packs_to_include,\n-\t\t\t struct string_list *packs_to_drop,\n-\t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot,\n-\t\t\t unsigned flags)\n+struct write_midx_opts {\n+\tstruct odb_source *source; /* non-optional */\n+\n+\tstruct string_list *packs_to_include;\n+\tstruct string_list *packs_to_drop;\n+\n+\tconst char *preferred_pack_name;\n+\tconst char *refs_snapshot;\n+\tunsigned flags;\n+};\n+\n+static int write_midx_internal(struct write_midx_opts *opts)\n {\n-\tstruct repository *r = source->odb->repo;\n+\tstruct repository *r = opts->source->odb->repo;\n \tstruct strbuf midx_name = STRBUF_INIT;\n \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t start_pack;\n@@ -1106,22 +1112,22 @@ static int write_midx_internal(struct odb_source *source,\n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n \n \tctx.repo = r;\n-\tctx.source = source;\n+\tctx.source = opts->source;\n \n-\tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n+\tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n \t\t\t \"%s/pack/multi-pack-index.d/tmp_midx_XXXXXX\",\n-\t\t\t source->path);\n+\t\t\t opts->source->path);\n \telse\n-\t\tget_midx_filename(source, &midx_name);\n+\t\tget_midx_filename(opts->source, &midx_name);\n \tif (safe_create_leading_directories(r, midx_name.buf))\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name.buf);\n \n-\tif (!packs_to_include || ctx.incremental) {\n-\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\tif (!opts->packs_to_include || ctx.incremental) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(opts->source);\n \t\tif (m && !midx_checksum_valid(m)) {\n \t\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n \t\t\tm = NULL;\n@@ -1136,7 +1142,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t */\n \t\t\tif (ctx.incremental)\n \t\t\t\tctx.base_midx = m;\n-\t\t\telse if (!packs_to_include)\n+\t\t\telse if (!opts->packs_to_include)\n \t\t\t\tctx.m = m;\n \t\t}\n \t}\n@@ -1149,7 +1155,7 @@ static int write_midx_internal(struct odb_source *source,\n \tif (ctx.incremental) {\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\twhile (m) {\n-\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n+\t\t\tif (opts->flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n \t\t\t\t midx_get_checksum_hex(m));\n \t\t\t\tgoto cleanup;\n@@ -1164,18 +1170,18 @@ static int write_midx_internal(struct odb_source *source,\n \tstart_pack = ctx.nr;\n \n \tctx.pack_paths_checked = 0;\n-\tif (flags & MIDX_PROGRESS)\n+\tif (opts->flags & MIDX_PROGRESS)\n \t\tctx.progress = start_delayed_progress(r,\n \t\t\t\t\t\t _(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n \t\tctx.progress = NULL;\n \n-\tctx.to_include = packs_to_include;\n+\tctx.to_include = opts->packs_to_include;\n \n-\tfor_each_file_in_pack_dir(source->path, add_pack_to_midx, &ctx);\n+\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (!packs_to_drop) {\n+\tif (!opts->packs_to_drop) {\n \t\t/*\n \t\t * If there is no MIDX then either it doesn't exist, or we're\n \t\t * doing a geometric repack. Try to load it from the source to\n@@ -1188,7 +1194,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tif (midx && !midx_needs_update(midx, &ctx)) {\n \t\t\tstruct bitmap_index *bitmap_git;\n \t\t\tint bitmap_exists;\n-\t\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\t\t\tint want_bitmap = opts->flags & MIDX_WRITE_BITMAP;\n \n \t\t\tbitmap_git = prepare_midx_bitmap_git(midx);\n \t\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n@@ -1200,7 +1206,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t\t * corresponding bitmap (or one wasn't requested).\n \t\t\t\t */\n \t\t\t\tif (!want_bitmap)\n-\t\t\t\t\tclear_midx_files_ext(source, \"bitmap\", NULL);\n+\t\t\t\t\tclear_midx_files_ext(ctx.source, \"bitmap\", NULL);\n \t\t\t\tresult = 0;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n@@ -1215,11 +1221,11 @@ static int write_midx_internal(struct odb_source *source,\n \t\tgoto cleanup; /* nothing to do */\n \t}\n \n-\tif (preferred_pack_name) {\n+\tif (opts->preferred_pack_name) {\n \t\tctx.preferred_pack_idx = NO_PREFERRED_PACK;\n \n \t\tfor (size_t i = 0; i < ctx.nr; i++) {\n-\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n+\t\t\tif (!cmp_idx_or_pack_name(opts->preferred_pack_name,\n \t\t\t\t\t\t ctx.info[i].pack_name)) {\n \t\t\t\tctx.preferred_pack_idx = i;\n \t\t\t\tbreak;\n@@ -1228,9 +1234,9 @@ static int write_midx_internal(struct odb_source *source,\n \n \t\tif (ctx.preferred_pack_idx == NO_PREFERRED_PACK)\n \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n-\t\t\t\tpreferred_pack_name);\n+\t\t\t\topts->preferred_pack_name);\n \t} else if (ctx.nr &&\n-\t\t (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n+\t\t (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n \t\tstruct packed_git *oldest = ctx.info[0].p;\n \t\tctx.preferred_pack_idx = 0;\n \n@@ -1241,7 +1247,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t */\n \t\topen_pack_index(oldest);\n \n-\t\tif (packs_to_drop && packs_to_drop->nr)\n+\t\tif (opts->packs_to_drop && opts->packs_to_drop->nr)\n \t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n \n \t\t/*\n@@ -1303,20 +1309,21 @@ static int write_midx_internal(struct odb_source *source,\n \n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n-\tif (packs_to_drop && packs_to_drop->nr) {\n+\tif (opts->packs_to_drop && opts->packs_to_drop->nr) {\n \t\tsize_t drop_index = 0;\n \t\tint missing_drops = 0;\n \n-\t\tfor (size_t i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\tfor (size_t i = 0;\n+\t\t i < ctx.nr && drop_index < opts->packs_to_drop->nr; i++) {\n \t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n-\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\t\t opts->packs_to_drop->items[drop_index].string);\n \n \t\t\tif (!cmp) {\n \t\t\t\tdrop_index++;\n \t\t\t\tctx.info[i].expired = 1;\n \t\t\t} else if (cmp > 0) {\n \t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n-\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\t opts->packs_to_drop->items[drop_index].string);\n \t\t\t\tdrop_index++;\n \t\t\t\tmissing_drops++;\n \t\t\t\ti--;\n@@ -1353,8 +1360,8 @@ static int write_midx_internal(struct odb_source *source,\n \t}\n \n \t/* Check that the preferred pack wasn't expired (if given). */\n-\tif (preferred_pack_name) {\n-\t\tstruct pack_info *preferred = bsearch(preferred_pack_name,\n+\tif (opts->preferred_pack_name) {\n+\t\tstruct pack_info *preferred = bsearch(opts->preferred_pack_name,\n \t\t\t\t\t\t ctx.info, ctx.nr,\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n@@ -1362,7 +1369,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n-\t\t\t\t\tpreferred_pack_name);\n+\t\t\t\t\topts->preferred_pack_name);\n \t\t}\n \t}\n \n@@ -1376,15 +1383,15 @@ static int write_midx_internal(struct odb_source *source,\n \t}\n \n \tif (!ctx.entries_nr) {\n-\t\tif (flags & MIDX_WRITE_BITMAP)\n+\t\tif (opts->flags & MIDX_WRITE_BITMAP)\n \t\t\twarning(_(\"refusing to write multi-pack .bitmap without any objects\"));\n-\t\tflags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n+\t\topts->flags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n \t}\n \n \tif (ctx.incremental) {\n \t\tstruct strbuf lock_name = STRBUF_INIT;\n \n-\t\tget_midx_chain_filename(source, &lock_name);\n+\t\tget_midx_chain_filename(opts->source, &lock_name);\n \t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n \t\tstrbuf_release(&lock_name);\n \n@@ -1427,7 +1434,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t\tMIDX_CHUNK_LARGE_OFFSET_WIDTH),\n \t\t\twrite_midx_large_offsets);\n \n-\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n+\tif (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n \t\tctx.pack_order = midx_pack_order(&ctx);\n \t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n \t\t\t st_mult(ctx.entries_nr, sizeof(uint32_t)),\n@@ -1445,11 +1452,11 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX &&\n+\tif (opts->flags & MIDX_WRITE_REV_INDEX &&\n \t git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n \t\twrite_midx_reverse_index(&ctx, midx_hash);\n \n-\tif (flags & MIDX_WRITE_BITMAP) {\n+\tif (opts->flags & MIDX_WRITE_BITMAP) {\n \t\tstruct packing_data pdata;\n \t\tstruct commit_stack commits = COMMIT_STACK_INIT;\n \n@@ -1458,7 +1465,7 @@ static int write_midx_internal(struct odb_source *source,\n \n \t\tprepare_midx_packing_data(&pdata, &ctx);\n \n-\t\tfind_commits_for_midx_bitmap(&commits, refs_snapshot, &ctx);\n+\t\tfind_commits_for_midx_bitmap(&commits, opts->refs_snapshot, &ctx);\n \n \t\t/*\n \t\t * The previous steps translated the information from\n@@ -1469,8 +1476,8 @@ static int write_midx_internal(struct odb_source *source,\n \t\tFREE_AND_NULL(ctx.entries);\n \t\tctx.entries_nr = 0;\n \n-\t\tif (write_midx_bitmap(&ctx, midx_hash, &pdata,\n-\t\t\t\t commits.items, commits.nr, flags) < 0) {\n+\t\tif (write_midx_bitmap(&ctx, midx_hash, &pdata, commits.items,\n+\t\t\t\t commits.nr, opts->flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tclear_packing_data(&pdata);\n \t\t\tcommit_stack_clear(&commits);\n@@ -1503,7 +1510,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n \t\t\tgoto cleanup;\n \n-\t\tget_split_midx_filename_ext(source, &final_midx_name,\n+\t\tget_split_midx_filename_ext(opts->source, &final_midx_name,\n \t\t\t\t\t midx_hash, MIDX_EXT_MIDX);\n \n \t\tif (rename_tempfile(&incr, final_midx_name.buf) < 0) {\n@@ -1536,7 +1543,7 @@ static int write_midx_internal(struct odb_source *source,\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(source, keep_hashes,\n+\tclear_midx_files(opts->source, keep_hashes,\n \t\t\t ctx.num_multi_pack_indexes_before + 1,\n \t\t\t ctx.incremental);\n \tresult = 0;\n@@ -1571,9 +1578,14 @@ int write_midx_file(struct odb_source *source,\n \t\t const char *preferred_pack_name,\n \t\t const char *refs_snapshot, unsigned flags)\n {\n-\treturn write_midx_internal(source, NULL, NULL,\n-\t\t\t\t preferred_pack_name, refs_snapshot,\n-\t\t\t\t flags);\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.preferred_pack_name = preferred_pack_name,\n+\t\t.refs_snapshot = refs_snapshot,\n+\t\t.flags = flags,\n+\t};\n+\n+\treturn write_midx_internal(&opts);\n }\n \n int write_midx_file_only(struct odb_source *source,\n@@ -1581,8 +1593,15 @@ int write_midx_file_only(struct odb_source *source,\n \t\t\t const char *preferred_pack_name,\n \t\t\t const char *refs_snapshot, unsigned flags)\n {\n-\treturn write_midx_internal(source, packs_to_include, NULL,\n-\t\t\t\t preferred_pack_name, refs_snapshot, flags);\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.packs_to_include = packs_to_include,\n+\t\t.preferred_pack_name = preferred_pack_name,\n+\t\t.refs_snapshot = refs_snapshot,\n+\t\t.flags = flags,\n+\t};\n+\n+\treturn write_midx_internal(&opts);\n }\n \n int expire_midx_packs(struct odb_source *source, unsigned flags)\n@@ -1641,9 +1660,14 @@ int expire_midx_packs(struct odb_source *source, unsigned flags)\n \n \tfree(count);\n \n-\tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(source, NULL,\n-\t\t\t\t\t &packs_to_drop, NULL, NULL, flags);\n+\tif (packs_to_drop.nr) {\n+\t\tstruct write_midx_opts opts = {\n+\t\t\t.source = source,\n+\t\t\t.packs_to_drop = &packs_to_drop,\n+\t\t\t.flags = flags & MIDX_PROGRESS,\n+\t\t};\n+\t\tresult = write_midx_internal(&opts);\n+\t}\n \n \tstring_list_clear(&packs_to_drop, 0);\n \n@@ -1776,6 +1800,10 @@ int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags)\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tFILE *cmd_in;\n \tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.flags = flags,\n+\t};\n \n \t/*\n \t * When updating the default for these configuration\n@@ -1850,8 +1878,7 @@ int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags)\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(source, NULL, NULL, NULL, NULL,\n-\t\t\t\t flags);\n+\tresult = write_midx_internal(&opts);\n \n cleanup:\n \tfree(include_pack);\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533879","messageId":"72bcd4ed6c7f685f58bb3b905fe553173abe1845.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 10/18] midx: do not require packs to be sorted in lexicographic order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:45Z","receivedAt":"2026-01-14T19:54:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The MIDX file format currently requires that pack files be identified by\nthe lexicographic ordering of their names (that is, a pack having a\nchecksum beginning with \"abc\" would have a numeric pack_int_id which is\nsmaller than the same value for a pack beginning with \"bcd\").\n\nAs a result, it is impossible to combine adjacent MIDX layers together\nwithout permuting bits from bitmaps that are in more recent layer(s).\n\nTo see why, consider the following example:\n\n | packs | preferred pack\n --------+-------------+---------------\n MIDX #0 | { X, Y, Z } | Y\n MIDX #1 | { A, B, C } | B\n MIDX #2 | { D, E, F } | D\n\n, where MIDX #2's base MIDX is MIDX #1, and so on. Suppose that we want\nto combine MIDX layers #0 and #1, to create a new layer #0' containing\nthe packs from both layers. With the original three MIDX layers, objects\nare laid out in the bitmap in the order they appear in their source\npack, and the packs themselves are arranged according to the pseudo-pack\norder. In this case, that ordering is Y, X, Z, B, A, C.\n\nBut recall that the pseudo-pack ordering is defined by the order that\npacks appear in the MIDX, with the exception of the preferred pack,\nwhich sorts ahead of all other packs regardless of its position within\nthe MIDX. In the above example, that means that pack 'Y' could be placed\nanywhere (so long as it is designated as preferred), however, all other\npacks must be placed in the location listed above.\n\nBecause that ordering isn't sorted lexicographically, it is impossible\nto compact MIDX layers in the above configuration without permuting the\nobject-to-bit-position mapping. Changing this mapping would affect all\nbitmaps belonging to newer layers, rendering the bitmaps associated with\nMIDX #2 unreadable.\n\nOne of the goals of MIDX compaction is that we are able to shrink the\nlength of the MIDX chain *without* invalidating bitmaps that belong to\nnewer layers, and the lexicographic ordering constraint is at odds with\nthis goal.\n\nHowever, packs do not *need* to be lexicographically ordered within the\nMIDX. As far as I can gather, the only reason they are sorted lexically\nis to make it possible to perform a binary search over the pack names in\na MIDX, necessary to make `midx_contains_pack()`'s performance\nlogarithmic in the number of packs rather than linear.\n\nRelax this constraint by allowing MIDX writes to proceed with packs that\nare not arranged in lexicographic order. `midx_contains_pack()` will\nlazily instantiate a `pack_names_sorted` array on the MIDX, which will\nbe used to implement the binary search over pack names.\n\nBecause this change produces MIDXs which may not be correctly read with\nexternal tools or older versions of Git. Though older versions of Git\nknow how to gracefully degrade and ignore any MIDX(s) they consider\ncorrupt, external tools may not be as robust. To avoid unintentionally\nbreaking any such tools, guard this change behind a version bump in the\nMIDX's on-disk format.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/gitformat-pack.adoc | 6 ++++--\n midx-write.c | 26 ++++++++++++++++++++++----\n midx.c | 31 ++++++++++++++++++++++++++++---\n midx.h | 4 +++-\n t/t5319-multi-pack-index.sh | 16 ++++++++++------\n 5 files changed, 67 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/gitformat-pack.adoc b/Documentation/gitformat-pack.adoc\nindex 1b4db4aa611..5be2206b20b 100644\n--- a/Documentation/gitformat-pack.adoc\n+++ b/Documentation/gitformat-pack.adoc\n@@ -374,7 +374,7 @@ HEADER:\n \t The signature is: {'M', 'I', 'D', 'X'}\n \n \t1-byte version number:\n-\t Git only writes or recognizes version 1.\n+\t Git only writes version 2, but recognizes versions 1 and 2.\n \n \t1-byte Object Id Version\n \t We infer the length of object IDs (OIDs) from this value:\n@@ -413,7 +413,9 @@ CHUNK DATA:\n \t strings. There is no extra padding between the filenames,\n \t and they are listed in lexicographic order. The chunk itself\n \t is padded at the end with between 0 and 3 NUL bytes to make the\n-\t chunk size a multiple of 4 bytes.\n+\t chunk size a multiple of 4 bytes. Version 1 MIDXs are required to\n+\t list their packs in lexicographic order, but version 2 MIDXs may\n+\t list their packs in any arbitrary order.\n \n \tBitmapped Packfiles (ID: {'B', 'T', 'M', 'P'})\n \t Stores a table of two 4-byte unsigned integers in network order.\ndiff --git a/midx-write.c b/midx-write.c\nindex 8a54644e427..5c8700065a1 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -36,10 +36,13 @@ extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \n static size_t write_midx_header(const struct git_hash_algo *hash_algo,\n \t\t\t\tstruct hashfile *f, unsigned char num_chunks,\n-\t\t\t\tuint32_t num_packs)\n+\t\t\t\tuint32_t num_packs, int version)\n {\n+\tif (version != MIDX_VERSION_V1 && version != MIDX_VERSION_V2)\n+\t\tBUG(\"unexpected MIDX version: %d\", version);\n+\n \thashwrite_be32(f, MIDX_SIGNATURE);\n-\thashwrite_u8(f, MIDX_VERSION);\n+\thashwrite_u8(f, version);\n \thashwrite_u8(f, oid_version(hash_algo));\n \thashwrite_u8(f, num_chunks);\n \thashwrite_u8(f, 0); /* unused */\n@@ -105,6 +108,8 @@ struct write_midx_context {\n \n \tuint32_t preferred_pack_idx;\n \n+\tint version; /* must be MIDX_VERSION_V1 or _V2 */\n+\n \tint incremental;\n \tuint32_t num_multi_pack_indexes_before;\n \n@@ -410,7 +415,9 @@ static int write_midx_pack_names(struct hashfile *f, void *data)\n \t\tif (ctx->info[i].expired)\n \t\t\tcontinue;\n \n-\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n+\t\tif (ctx->version == MIDX_VERSION_V1 &&\n+\t\t i && strcmp(ctx->info[i].pack_name,\n+\t\t\t\tctx->info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n \t\t\t ctx->info[i - 1].pack_name,\n \t\t\t ctx->info[i].pack_name);\n@@ -1025,6 +1032,12 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \tif (!midx_checksum_valid(midx))\n \t\tgoto out;\n \n+\t/*\n+\t * If the version differs, we need to update.\n+\t */\n+\tif (midx->version != ctx->version)\n+\t\tgoto out;\n+\n \t/*\n \t * Ignore incremental updates for now. The assumption is that any\n \t * incremental update would be either empty (in which case we will bail\n@@ -1100,6 +1113,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tstruct tempfile *incr;\n \tstruct write_midx_context ctx = {\n \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n+\t\t.version = MIDX_VERSION_V2,\n \t };\n \tstruct multi_pack_index *midx_to_free = NULL;\n \tint bitmapped_packs_concat_len = 0;\n@@ -1114,6 +1128,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tctx.repo = r;\n \tctx.source = opts->source;\n \n+\trepo_config_get_int(ctx.repo, \"midx.version\", &ctx.version);\n+\tif (ctx.version != MIDX_VERSION_V1 && ctx.version != MIDX_VERSION_V2)\n+\t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n+\n \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n \n \tif (ctx.incremental)\n@@ -1445,7 +1463,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \n \twrite_midx_header(r->hash_algo, f, get_num_chunks(cf),\n-\t\t\t ctx.nr - dropped_packs);\n+\t\t\t ctx.nr - dropped_packs, ctx.version);\n \twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,\ndiff --git a/midx.c b/midx.c\nindex 19ef230d3fd..1327d0a3695 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -149,7 +149,7 @@ static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *sou\n \t\t m->signature, MIDX_SIGNATURE);\n \n \tm->version = m->data[MIDX_BYTE_FILE_VERSION];\n-\tif (m->version != MIDX_VERSION)\n+\tif (m->version != MIDX_VERSION_V1 && m->version != MIDX_VERSION_V2)\n \t\tdie(_(\"multi-pack-index version %d not recognized\"),\n \t\t m->version);\n \n@@ -210,7 +210,8 @@ static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *sou\n \t\t\tdie(_(\"multi-pack-index pack-name chunk is too short\"));\n \t\tcur_pack_name = end + 1;\n \n-\t\tif (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0)\n+\t\tif (m->version == MIDX_VERSION_V1 &&\n+\t\t i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0)\n \t\t\tdie(_(\"multi-pack-index pack names out of order: '%s' before '%s'\"),\n \t\t\t m->pack_names[i - 1],\n \t\t\t m->pack_names[i]);\n@@ -411,6 +412,7 @@ void close_midx(struct multi_pack_index *m)\n \t}\n \tFREE_AND_NULL(m->packs);\n \tFREE_AND_NULL(m->pack_names);\n+\tFREE_AND_NULL(m->pack_names_sorted);\n \tfree(m);\n }\n \n@@ -656,17 +658,40 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n+\n+static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n+{\n+\tstruct multi_pack_index *m = m_;\n+\treturn strcmp(m->pack_names[*(const size_t *)a],\n+\t\t m->pack_names[*(const size_t *)b]);\n+}\n+\n static int midx_contains_pack_1(struct multi_pack_index *m,\n \t\t\t\tconst char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n+\tif (m->version == MIDX_VERSION_V2 && !m->pack_names_sorted) {\n+\t\tuint32_t i;\n+\n+\t\tALLOC_ARRAY(m->pack_names_sorted, m->num_packs);\n+\n+\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\t\tm->pack_names_sorted[i] = i;\n+\n+\t\tQSORT_S(m->pack_names_sorted, m->num_packs, midx_pack_names_cmp,\n+\t\t\tm);\n+\t}\n+\n \twhile (first < last) {\n \t\tuint32_t mid = first + (last - first) / 2;\n \t\tconst char *current;\n \t\tint cmp;\n \n-\t\tcurrent = m->pack_names[mid];\n+\t\tif (m->pack_names_sorted)\n+\t\t\tcurrent = m->pack_names[m->pack_names_sorted[mid]];\n+\t\telse\n+\t\t\tcurrent = m->pack_names[mid];\n \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n \t\tif (!cmp)\n \t\t\treturn 1;\ndiff --git a/midx.h b/midx.h\nindex a39bcc9d03f..aa99a6cb215 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -11,7 +11,8 @@ struct git_hash_algo;\n struct odb_source;\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n-#define MIDX_VERSION 1\n+#define MIDX_VERSION_V1 1\n+#define MIDX_VERSION_V2 2\n #define MIDX_BYTE_FILE_VERSION 4\n #define MIDX_BYTE_HASH_VERSION 5\n #define MIDX_BYTE_NUM_CHUNKS 6\n@@ -71,6 +72,7 @@ struct multi_pack_index {\n \tuint32_t num_packs_in_base;\n \n \tconst char **pack_names;\n+\tsize_t *pack_names_sorted;\n \tstruct packed_git **packs;\n };\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex efeab4d22b7..250d21dbd67 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -21,7 +21,7 @@ midx_read_expect () {\n \tEXTRA_CHUNKS=\"$5\"\n \t{\n \t\tcat <<-EOF &&\n-\t\theader: 4d494458 1 $HASH_LEN $NUM_CHUNKS $NUM_PACKS\n+\t\theader: 4d494458 2 $HASH_LEN $NUM_CHUNKS $NUM_PACKS\n \t\tchunks: pack-names oid-fanout oid-lookup object-offsets$EXTRA_CHUNKS\n \t\tnum_objects: $NUM_OBJECTS\n \t\tpacks:\n@@ -512,11 +512,6 @@ test_expect_success 'verify invalid chunk offset' '\n \t\t\"improper chunk offset(s)\"\n '\n \n-test_expect_success 'verify packnames out of order' '\n-\tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"z\" $objdir \\\n-\t\t\"pack names out of order\"\n-'\n-\n test_expect_success 'verify missing pack' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"a\" $objdir \\\n \t\t\"failed to load pack\"\n@@ -578,6 +573,15 @@ test_expect_success 'verify incorrect checksum' '\n \t\t$objdir \"incorrect checksum\"\n '\n \n+test_expect_success 'setup for v1-specific fsck tests' '\n+\tgit -c midx.version=1 multi-pack-index write\n+'\n+\n+test_expect_success 'verify packnames out of order (v1)' '\n+\tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"z\" $objdir \\\n+\t\t\"pack names out of order\"\n+'\n+\n test_expect_success 'repack progress off for redirected stderr' '\n \tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir repack 2>err &&\n \ttest_line_count = 0 err\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533880","messageId":"c0c1769464b1c8065c2cea59dfd85a1d37de9dd1.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:48Z","receivedAt":"2026-01-14T19:54:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A future commit will want to add two 32-bit unsigned values together\nwhile checking for overflow. Introduce a variant of the u64_add()\nfunction for operating on 32-bit inputs.\n\nSigned-off-by: Taylor Blau \n---\n git-compat-util.h | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b0673d1a450..db62a6f25c5 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -641,6 +641,14 @@ static inline int cast_size_t_to_int(size_t a)\n \treturn (int)a;\n }\n \n+static inline uint32_t u32_add(uint32_t a, uint32_t b)\n+{\n+\tif (unsigned_add_overflows(a, b))\n+\t\tdie(\"uint32_t overflow: %\"PRIuMAX\" + %\"PRIuMAX,\n+\t\t (uintmax_t)a, (uintmax_t)b);\n+\treturn a + b;\n+}\n+\n static inline uint64_t u64_mult(uint64_t a, uint64_t b)\n {\n \tif (unsigned_mult_overflows(a, b))\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533881","messageId":"c11214a51f003cb45f5c48084f17b989b2cb5443.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 12/18] midx-write.c: introduce `midx_pack_perm()` helper","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:52Z","receivedAt":"2026-01-14T19:54:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `ctx->pack_perm` array can be considered as a permutation between\nthe original `pack_int_id` of some given pack to its position in the\n`ctx->info` array containing all packs.\n\nToday we can always index into this array with any known `pack_int_id`,\nsince there is never a `pack_int_id` which is greater than or equal to\nthe value `ctx->nr`.\n\nThat is not necessarily the case with MIDX compaction. For example,\nsuppose we have a MIDX chain with three layers, each containing three\npacks. The base of the MIDX chain will have packs with IDs 0, 1, and 2,\nthe next layer 3, 4, and 5, and so on. If we are compacting the topmost\ntwo layers, we'll have input `pack_int_id` values between [3, 8], but\n`ctx->nr` will only be 6.\n\nIn that example, if we want to know where the pack whose original\n`pack_int_id` value was, say, 7, we would compute `ctx->pack_perm[7]`,\nleading to an uninitialized read, since there are only 6 entries\nallocated in that array.\n\nTo address this, there are a couple of options:\n\n - We could allocate enough entries in `ctx->pack_perm` to accommodate\n the largest `orig_pack_int_id` value.\n\n - Or, we could internally shift the input values by the number of packs\n in the base layer of the lower end of the MIDX compaction range.\n\nThis patch prepare us to take the latter approach, since it does not\nallocate more memory than strictly necessary. (In our above example, the\nbase of the lower end of the compaction range is the first MIDX layer\n(having three packs), so we would end up indexing `ctx->pack_perm[7-3]`,\nwhich is a valid read.)\n\nNote that this patch does not actually implement that approach yet, but\nmerely performs a behavior-preserving refactoring which will make the\nchange easier to carry out in the future.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 5c8700065a1..9d345fb4737 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -119,6 +119,12 @@ struct write_midx_context {\n \tstruct odb_source *source;\n };\n \n+static uint32_t midx_pack_perm(struct write_midx_context *ctx,\n+\t\t\t uint32_t orig_pack_int_id)\n+{\n+\treturn ctx->pack_perm[orig_pack_int_id];\n+}\n+\n static int should_include_pack(const struct write_midx_context *ctx,\n \t\t\t const char *file_name)\n {\n@@ -521,12 +527,12 @@ static int write_midx_object_offsets(struct hashfile *f,\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\tif (midx_pack_perm(ctx, obj->pack_int_id) == PACK_EXPIRED)\n \t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->pack_int_id);\n \n-\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n+\t\thashwrite_be32(f, midx_pack_perm(ctx, obj->pack_int_id));\n \n \t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n@@ -627,7 +633,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[i];\n \t\tdata[i].nr = i;\n-\t\tdata[i].pack = ctx->pack_perm[e->pack_int_id];\n+\t\tdata[i].pack = midx_pack_perm(ctx, e->pack_int_id);\n \t\tif (!e->preferred)\n \t\t\tdata[i].pack |= (1U << 31);\n \t\tdata[i].offset = e->offset;\n@@ -637,7 +643,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n+\t\tstruct pack_info *pack = &ctx->info[midx_pack_perm(ctx, e->pack_int_id)];\n \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n \t\t\tpack->bitmap_pos = i + base_objects;\n \t\tpack->bitmap_nr++;\n@@ -698,7 +704,7 @@ static void prepare_midx_packing_data(struct packing_data *pdata,\n \t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n \n \t\toe_set_in_pack(pdata, to,\n-\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t\t\t ctx->info[midx_pack_perm(ctx, from->pack_int_id)].p);\n \t}\n \n \ttrace2_region_leave(\"midx\", \"prepare_midx_packing_data\", ctx->repo);\n@@ -1384,7 +1390,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n \t\tif (preferred) {\n-\t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n+\t\t\tuint32_t perm = midx_pack_perm(&ctx, preferred->orig_pack_int_id);\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n \t\t\t\t\topts->preferred_pack_name);\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533882","messageId":"b9244a0429730badb6472e827a941a33855add3a.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 13/18] midx-write.c: extract `fill_pack_from_midx()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:55Z","receivedAt":"2026-01-14T19:54:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When filling packs from an existing MIDX, `fill_packs_from_midx()`\nhandles preparing a MIDX'd pack, and reading out its pack name from the\nexisting MIDX.\n\nMIDX compaction will want to perform an identical operation, though the\ncaller will look quite different than `fill_packs_from_midx()`. To\nreduce any future code duplication, extract `fill_pack_from_midx()`\nfrom `fill_packs_from_midx()` to prepare to call our new helper function\nin a future change.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 27 +++++++++++++++++++++------\n 1 file changed, 21 insertions(+), 6 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 9d345fb4737..c54113cdc84 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -913,6 +913,21 @@ static int write_midx_bitmap(struct write_midx_context *ctx,\n \treturn ret;\n }\n \n+static int fill_pack_from_midx(struct pack_info *info,\n+\t\t\t struct multi_pack_index *m,\n+\t\t\t uint32_t pack_int_id)\n+{\n+\tif (prepare_midx_pack(m, pack_int_id))\n+\t\treturn error(_(\"could not load pack %d\"), pack_int_id);\n+\n+\tfill_pack_info(info,\n+\t\t m->packs[pack_int_id - m->num_packs_in_base],\n+\t\t m->pack_names[pack_int_id - m->num_packs_in_base],\n+\t\t pack_int_id);\n+\n+\treturn 0;\n+}\n+\n static int fill_packs_from_midx(struct write_midx_context *ctx)\n {\n \tstruct multi_pack_index *m;\n@@ -921,13 +936,13 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n \t\tuint32_t i;\n \n \t\tfor (i = 0; i < m->num_packs; i++) {\n-\t\t\tif (prepare_midx_pack(m, m->num_packs_in_base + i))\n-\t\t\t\treturn error(_(\"could not load pack\"));\n-\n \t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n-\t\t\tfill_pack_info(&ctx->info[ctx->nr++], m->packs[i],\n-\t\t\t\t m->pack_names[i],\n-\t\t\t\t m->num_packs_in_base + i);\n+\n+\t\t\tif (fill_pack_from_midx(&ctx->info[ctx->nr], m,\n+\t\t\t\t\t\tm->num_packs_in_base + i) < 0)\n+\t\t\t\treturn -1;\n+\n+\t\t\tctx->nr++;\n \t\t}\n \t}\n \treturn 0;\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533883","messageId":"c6f8d323477e9e74ce038b111d400c706f535676.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 14/18] midx-write.c: enumerate `pack_int_id` values directly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:54:58Z","receivedAt":"2026-01-14T19:55:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Our `midx-write.c::fill_packs_from_midx()` function currently enumerates\nthe range [0, m->num_packs), and then shifts its index variable up by\n`m->num_packs_in_base` to produce a valid `pack_int_id`.\n\nInstead, directly enumerate the range:\n\n [m->num_packs_in_base, m->num_packs_in_base + m->num_packs)\n\n, which are the original pack_int_ids themselves as opposed to the\nindexes of those packs relative to the MIDX layer they are contained\nwithin.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex c54113cdc84..80334914d3e 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -935,11 +935,11 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n \tfor (m = ctx->m; m; m = m->base_midx) {\n \t\tuint32_t i;\n \n-\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tfor (i = m->num_packs_in_base;\n+\t\t i < m->num_packs_in_base + m->num_packs; i++) {\n \t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\t\tif (fill_pack_from_midx(&ctx->info[ctx->nr], m,\n-\t\t\t\t\t\tm->num_packs_in_base + i) < 0)\n+\t\t\tif (fill_pack_from_midx(&ctx->info[ctx->nr], m, i) < 0)\n \t\t\t\treturn -1;\n \n \t\t\tctx->nr++;\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533884","messageId":"e71aa5754631d6458658220390021cea7dc2bd56.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 15/18] midx-write.c: factor fanout layering from `compute_sorted_entries()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:55:01Z","receivedAt":"2026-01-14T19:55:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When computing the set of objects to appear in a MIDX, we use\ncompute_sorted_entries(), which handles objects from various existing\nsources one fanout layer at a time.\n\nThe process for computing this set is slightly different during MIDX\ncompaction, so factor out the existing functionality into its own\nroutine to prevent `compute_sorted_entries()` from becoming too\ndifficult to read.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 80334914d3e..ca2469213e6 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -328,6 +328,30 @@ static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n \t}\n }\n \n+static void midx_fanout_add(struct midx_fanout *fanout,\n+\t\t\t struct write_midx_context *ctx,\n+\t\t\t uint32_t start_pack,\n+\t\t\t uint32_t cur_fanout)\n+{\n+\tuint32_t cur_pack;\n+\n+\tif (ctx->m && !ctx->incremental)\n+\t\tmidx_fanout_add_midx_fanout(fanout, ctx->m, cur_fanout,\n+\t\t\t\t\t ctx->preferred_pack_idx);\n+\n+\tfor (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) {\n+\t\tint preferred = cur_pack == ctx->preferred_pack_idx;\n+\t\tmidx_fanout_add_pack_fanout(fanout, ctx->info, cur_pack,\n+\t\t\t\t\t preferred, cur_fanout);\n+\t}\n+\n+\tif (ctx->preferred_pack_idx != NO_PREFERRED_PACK &&\n+\t ctx->preferred_pack_idx < start_pack)\n+\t\tmidx_fanout_add_pack_fanout(fanout, ctx->info,\n+\t\t\t\t\t ctx->preferred_pack_idx, 1,\n+\t\t\t\t\t cur_fanout);\n+}\n+\n /*\n * It is possible to artificially get into a state where there are many\n * duplicate copies of objects. That can create high memory pressure if\n@@ -364,23 +388,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n \n-\t\tif (ctx->m && !ctx->incremental)\n-\t\t\tmidx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout,\n-\t\t\t\t\t\t ctx->preferred_pack_idx);\n-\n-\t\tfor (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) {\n-\t\t\tint preferred = cur_pack == ctx->preferred_pack_idx;\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout,\n-\t\t\t\t\t\t ctx->info, cur_pack,\n-\t\t\t\t\t\t preferred, cur_fanout);\n-\t\t}\n-\n-\t\tif (ctx->preferred_pack_idx != NO_PREFERRED_PACK &&\n-\t\t ctx->preferred_pack_idx < start_pack)\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout, ctx->info,\n-\t\t\t\t\t\t ctx->preferred_pack_idx, 1,\n-\t\t\t\t\t\t cur_fanout);\n-\n+\t\tmidx_fanout_add(&fanout, ctx, start_pack, cur_fanout);\n \t\tmidx_fanout_sort(&fanout);\n \n \t\t/*\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533885","messageId":"dbbcb49456385265eceb6dd77e054df1e26677c8.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 16/18] t/helper/test-read-midx.c: plug memory leak when selecting layer","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:55:07Z","receivedAt":"2026-01-14T19:55:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Though our 'read-midx' test tool is capable of printing information\nabout a single MIDX layer identified by its checksum, no caller in our\ntest suite exercises this path.\n\nUnfortunately, there is a memory leak lurking in this (currently) unused\npath that would otherwise be exposed by the following commit.\n\nThis occurs when providing a MIDX layer checksum other than the tip. As\nwe walk over the MIDX chain trying to find the matching layer, we drop\nour reference to the top-most MIDX layer. Thus, our call to\n'close_midx()' later on leaks memory between the top-most MIDX layer and\nthe MIDX layer immediately following the specified one.\n\nPlug this leak by holding a reference to the tip of the MIDX chain, and\nensure that we call `close_midx()` before terminating the test tool.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 17 +++++++++++------\n 1 file changed, 11 insertions(+), 6 deletions(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 9d42c587564..388d29e2b53 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -26,9 +26,10 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\t\t int show_objects)\n {\n \tuint32_t i;\n-\tstruct multi_pack_index *m;\n+\tstruct multi_pack_index *m, *tip;\n+\tint ret = 0;\n \n-\tm = setup_midx(object_dir);\n+\tm = tip = setup_midx(object_dir);\n \n \tif (!m)\n \t\treturn 1;\n@@ -36,8 +37,11 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \tif (checksum) {\n \t\twhile (m && strcmp(midx_get_checksum_hex(m), checksum))\n \t\t\tm = m->base_midx;\n-\t\tif (!m)\n-\t\t\treturn 1;\n+\t\tif (!m) {\n+\t\t\tret = error(_(\"could not find MIDX with checksum %s\"),\n+\t\t\t\t checksum);\n+\t\t\tgoto out;\n+\t\t}\n \t}\n \n \tprintf(\"header: %08x %d %d %d %d\\n\",\n@@ -82,9 +86,10 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\t}\n \t}\n \n-\tclose_midx(m);\n+out:\n+\tclose_midx(tip);\n \n-\treturn 0;\n+\treturn ret;\n }\n \n static int read_midx_checksum(const char *object_dir)\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533886","messageId":"13336e864f4ed3a6954b782f0bcc090d92ac722c.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 17/18] midx: implement MIDX compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:55:10Z","receivedAt":"2026-01-14T19:55:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When managing a MIDX chain with many layers, it is convenient to combine\na sequence of adjacent layers into a single layer to prevent the chain\nfrom growing too long.\n\nWhile it is conceptually possible to \"compact\" a sequence of MIDX layers\ntogether by running \"git multi-pack-index write --stdin-packs\", there\nare a few drawbacks that make this less than desirable:\n\n - Preserving the MIDX chain is impossible, since there is no way to\n write a MIDX layer that contains objects or packs found in an earlier\n MIDX layer already part of the chain. So callers would have to write\n an entirely new (non-incremental) MIDX containing only the compacted\n layers, discarding all other objects/packs from the MIDX.\n\n - There is (currently) no way to write a MIDX layer outside of the MIDX\n chain to work around the above, such that the MIDX chain could be\n reassembled substituting the compacted layers with the MIDX that was\n written.\n\n - The `--stdin-packs` command-line option does not allow us to specify\n the order of packs as they appear in the MIDX. Therefore, even if\n there were workarounds for the previous two challenges, any bitmaps\n belonging to layers which come after the compacted layer(s) would no\n longer be valid.\n\nThis commit introduces a way to compact a sequence of adjacent MIDX\nlayers into a single layer while preserving the MIDX chain, as well as\nany bitmap(s) in layers which are newer than the compacted ones.\n\nImplementing MIDX compaction does not require a significant number of\nchanges to how MIDX layers are written. The main changes are as follows:\n\n - Instead of calling `fill_packs_from_midx()`, we call a new function\n `fill_packs_from_midx_range()`, which walks backwards along the\n portion of the MIDX chain which we are compacting, and adds packs one\n layer a time.\n\n In order to preserve the pseudo-pack order, the concatenated pack\n order is preserved, with the exception of preferred packs which are\n always added first.\n\n - After adding entries from the set of packs in the compaction range,\n `compute_sorted_entries()` must adjust the `pack_int_id`'s for all\n objects added in each fanout layer to match their original\n `pack_int_id`'s (as opposed to the index at which each pack appears\n in `ctx.info`).\n\n - When writing out the new 'multi-pack-index-chain' file, discard any\n layers in the compaction range, replacing them with the newly written\n layer, instead of keeping them and placing the new layer at the end\n of the chain.\n\nThis ends up being sufficient to implement MIDX compaction in such a way\nthat preserves bitmaps corresponding to more recent layers in the MIDX\nchain.\n\nThe tests for MIDX compaction are so far fairly spartan, since the main\ninteresting behavior here is ensuring that the right packs/objects are\nselected from each layer, and that the pack order is preserved despite\nwhether or not they are sorted in lexicographic order in the original\nMIDX chain.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 13 ++\n builtin/multi-pack-index.c | 74 +++++++\n midx-write.c | 251 ++++++++++++++++++++++--\n midx.h | 5 +\n t/meson.build | 1 +\n t/t5335-compact-multi-pack-index.sh | 175 +++++++++++++++++\n 6 files changed, 500 insertions(+), 19 deletions(-)\n create mode 100755 t/t5335-compact-multi-pack-index.sh\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 164cf1f2291..883a0529741 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -12,6 +12,8 @@ SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=]\n+'git multi-pack-index' [] compact [--[no-]incremental]\n+\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -83,6 +85,17 @@ marker).\n \t\tnecessary.\n --\n \n+compact::\n+\tWrite a new MIDX layer containing only objects and packs present\n+\tin the range `` to ``, where both arguments are\n+\tchecksums of existing layers in the MIDX chain.\n++\n+--\n+\t--incremental::\n+\t\tWrite the result to a MIDX chain instead of writing a\n+\t\tstand-alone MIDX.\n+--\n+\n verify::\n \tVerify the contents of the MIDX file.\n \ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex c0c6c1760c0..043ee8c478a 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -17,6 +17,10 @@\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n \t \" [--refs-snapshot=]\")\n \n+#define BUILTIN_MIDX_COMPACT_USAGE \\\n+\tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n+\t \" \")\n+\n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n \n@@ -30,6 +34,10 @@ static char const * const builtin_multi_pack_index_write_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n \tNULL\n };\n+static char const * const builtin_multi_pack_index_compact_usage[] = {\n+\tBUILTIN_MIDX_COMPACT_USAGE,\n+\tNULL\n+};\n static char const * const builtin_multi_pack_index_verify_usage[] = {\n \tBUILTIN_MIDX_VERIFY_USAGE,\n \tNULL\n@@ -44,6 +52,7 @@ static char const * const builtin_multi_pack_index_repack_usage[] = {\n };\n static char const * const builtin_multi_pack_index_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n+\tBUILTIN_MIDX_COMPACT_USAGE,\n \tBUILTIN_MIDX_VERIFY_USAGE,\n \tBUILTIN_MIDX_EXPIRE_USAGE,\n \tBUILTIN_MIDX_REPACK_USAGE,\n@@ -195,6 +204,70 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \treturn ret;\n }\n \n+static int cmd_multi_pack_index_compact(int argc, const char **argv,\n+\t\t\t\t\tconst char *prefix,\n+\t\t\t\t\tstruct repository *repo)\n+{\n+\tstruct multi_pack_index *m, *cur;\n+\tstruct multi_pack_index *from_midx = NULL;\n+\tstruct multi_pack_index *to_midx = NULL;\n+\tstruct odb_source *source;\n+\tint ret;\n+\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_BIT(0, \"incremental\", &opts.flags,\n+\t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_END(),\n+\t};\n+\n+\trepo_config(repo, git_multi_pack_index_write_config, NULL);\n+\n+\toptions = add_common_options(builtin_multi_pack_index_compact_options);\n+\n+\ttrace2_cmd_mode(argv[0]);\n+\n+\tif (isatty(2))\n+\t\topts.flags |= MIDX_PROGRESS;\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t options, builtin_multi_pack_index_compact_usage,\n+\t\t\t 0);\n+\n+\tif (argc != 2)\n+\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n+\t\t\t\t options);\n+\tsource = handle_object_dir_option(the_repository);\n+\n+\tFREE_AND_NULL(options);\n+\n+\tm = get_multi_pack_index(source);\n+\n+\tfor (cur = m; cur && !(from_midx && to_midx); cur = cur->base_midx) {\n+\t\tconst char *midx_csum = midx_get_checksum_hex(cur);\n+\n+\t\tif (!from_midx && !strcmp(midx_csum, argv[0]))\n+\t\t\tfrom_midx = cur;\n+\t\tif (!to_midx && !strcmp(midx_csum, argv[1]))\n+\t\t\tto_midx = cur;\n+\t}\n+\n+\tif (!from_midx)\n+\t\tdie(_(\"could not find MIDX: %s\"), argv[0]);\n+\tif (!to_midx)\n+\t\tdie(_(\"could not find MIDX: %s\"), argv[1]);\n+\tif (from_midx == to_midx)\n+\t\tdie(_(\"MIDX compaction endpoints must be unique\"));\n+\n+\tfor (m = from_midx; m; m = m->base_midx) {\n+\t\tif (m == to_midx)\n+\t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n+\t}\n+\n+\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n+\n+\treturn ret;\n+}\n+\n static int cmd_multi_pack_index_verify(int argc, const char **argv,\n \t\t\t\t const char *prefix,\n \t\t\t\t struct repository *repo UNUSED)\n@@ -295,6 +368,7 @@ int cmd_multi_pack_index(int argc,\n \tstruct option builtin_multi_pack_index_options[] = {\n \t\tOPT_SUBCOMMAND(\"repack\", &fn, cmd_multi_pack_index_repack),\n \t\tOPT_SUBCOMMAND(\"write\", &fn, cmd_multi_pack_index_write),\n+\t\tOPT_SUBCOMMAND(\"compact\", &fn, cmd_multi_pack_index_compact),\n \t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_multi_pack_index_verify),\n \t\tOPT_SUBCOMMAND(\"expire\", &fn, cmd_multi_pack_index_expire),\n \t\tOPT_END(),\ndiff --git a/midx-write.c b/midx-write.c\nindex ca2469213e6..afa077a09cc 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -113,6 +113,10 @@ struct write_midx_context {\n \tint incremental;\n \tuint32_t num_multi_pack_indexes_before;\n \n+\tstruct multi_pack_index *compact_from;\n+\tstruct multi_pack_index *compact_to;\n+\tint compact;\n+\n \tstruct string_list *to_include;\n \n \tstruct repository *repo;\n@@ -122,6 +126,8 @@ struct write_midx_context {\n static uint32_t midx_pack_perm(struct write_midx_context *ctx,\n \t\t\t uint32_t orig_pack_int_id)\n {\n+\tif (ctx->compact)\n+\t\torig_pack_int_id -= ctx->compact_from->num_packs_in_base;\n \treturn ctx->pack_perm[orig_pack_int_id];\n }\n \n@@ -352,6 +358,21 @@ static void midx_fanout_add(struct midx_fanout *fanout,\n \t\t\t\t\t cur_fanout);\n }\n \n+static void midx_fanout_add_compact(struct midx_fanout *fanout,\n+\t\t\t\t struct write_midx_context *ctx,\n+\t\t\t\t uint32_t cur_fanout)\n+{\n+\tstruct multi_pack_index *m = ctx->compact_to;\n+\n+\tASSERT(ctx->compact);\n+\n+\twhile (m && m != ctx->compact_from->base_midx) {\n+\t\tmidx_fanout_add_midx_fanout(fanout, m, cur_fanout,\n+\t\t\t\t\t NO_PREFERRED_PACK);\n+\t\tm = m->base_midx;\n+\t}\n+}\n+\n /*\n * It is possible to artificially get into a state where there are many\n * duplicate copies of objects. That can create high memory pressure if\n@@ -370,6 +391,9 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tsize_t alloc_objects, total_objects = 0;\n \tstruct midx_fanout fanout = { 0 };\n \n+\tif (ctx->compact)\n+\t\tASSERT(!start_pack);\n+\n \tfor (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++)\n \t\ttotal_objects = st_add(total_objects,\n \t\t\t\t ctx->info[cur_pack].p->num_objects);\n@@ -388,7 +412,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n \n-\t\tmidx_fanout_add(&fanout, ctx, start_pack, cur_fanout);\n+\t\tif (ctx->compact)\n+\t\t\tmidx_fanout_add_compact(&fanout, ctx, cur_fanout);\n+\t\telse\n+\t\t\tmidx_fanout_add(&fanout, ctx, start_pack, cur_fanout);\n \t\tmidx_fanout_sort(&fanout);\n \n \t\t/*\n@@ -956,6 +983,72 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n \treturn 0;\n }\n \n+static uint32_t compactible_packs_between(const struct multi_pack_index *from,\n+\t\t\t\t\t const struct multi_pack_index *to)\n+{\n+\tuint32_t nr;\n+\n+\tASSERT(from && to);\n+\n+\tnr = u32_add(to->num_packs, to->num_packs_in_base);\n+\tif (nr < from->num_packs_in_base)\n+\t\tBUG(\"unexpected number of packs in base during compaction: \"\n+\t\t \"%\"PRIu32\" < %\"PRIu32, nr, from->num_packs_in_base);\n+\n+\treturn nr - from->num_packs_in_base;\n+}\n+\n+static int fill_packs_from_midx_range(struct write_midx_context *ctx,\n+\t\t\t\t int bitmap_order)\n+{\n+\tstruct multi_pack_index *m = ctx->compact_to;\n+\tuint32_t packs_nr;\n+\n+\tASSERT(ctx->compact && !ctx->nr);\n+\tASSERT(ctx->compact_from);\n+\tASSERT(ctx->compact_to);\n+\n+\tpacks_nr = compactible_packs_between(ctx->compact_from,\n+\t\t\t\t\t ctx->compact_to);\n+\n+\tALLOC_GROW(ctx->info, packs_nr, ctx->alloc);\n+\n+\twhile (m != ctx->compact_from->base_midx) {\n+\t\tuint32_t pack_int_id, preferred_pack_id;\n+\t\tuint32_t i;\n+\n+\t\tif (bitmap_order) {\n+\t\t\tif (midx_preferred_pack(m, &preferred_pack_id) < 0)\n+\t\t\t\tdie(_(\"could not determine preferred pack\"));\n+\t\t} else {\n+\t\t\tpreferred_pack_id = m->num_packs_in_base;\n+\t\t}\n+\n+\t\tpack_int_id = m->num_packs_in_base - ctx->compact_from->num_packs_in_base;\n+\n+\t\tif (fill_pack_from_midx(&ctx->info[pack_int_id++], m,\n+\t\t\t\t\tpreferred_pack_id) < 0)\n+\t\t\treturn -1;\n+\n+\t\tfor (i = m->num_packs_in_base;\n+\t\t i < m->num_packs_in_base + m->num_packs; i++) {\n+\t\t\tif (preferred_pack_id == i)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (fill_pack_from_midx(&ctx->info[pack_int_id++], m,\n+\t\t\t\t\t\ti) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\n+\t\tctx->nr += m->num_packs;\n+\t\tm = m->base_midx;\n+\t}\n+\n+\tASSERT(ctx->nr == packs_nr);\n+\n+\treturn 0;\n+}\n+\n static struct {\n \tconst char *non_split;\n \tconst char *split;\n@@ -1075,6 +1168,9 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \tif (ctx->incremental)\n \t\tgoto out;\n \n+\tif (ctx->compact)\n+\t\tgoto out; /* Compaction always requires an update. */\n+\n \t/*\n \t * Otherwise, we need to verify that the packs covered by the existing\n \t * MIDX match the packs that we already have. The logic to do so is way\n@@ -1120,12 +1216,23 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \treturn needed;\n }\n \n+static int midx_hashcmp(const struct multi_pack_index *a,\n+\t\t\tconst struct multi_pack_index *b,\n+\t\t\tconst struct git_hash_algo *algop)\n+{\n+\treturn hashcmp(midx_get_checksum_hash(a), midx_get_checksum_hash(b),\n+\t\t algop);\n+}\n+\n struct write_midx_opts {\n \tstruct odb_source *source; /* non-optional */\n \n \tstruct string_list *packs_to_include;\n \tstruct string_list *packs_to_drop;\n \n+\tstruct multi_pack_index *compact_from;\n+\tstruct multi_pack_index *compact_to;\n+\n \tconst char *preferred_pack_name;\n \tconst char *refs_snapshot;\n \tunsigned flags;\n@@ -1150,6 +1257,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tint dropped_packs = 0;\n \tint result = -1;\n \tconst char **keep_hashes = NULL;\n+\tsize_t keep_hashes_nr = 0;\n \tstruct chunkfile *cf;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n@@ -1162,6 +1270,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n \n \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n+\tctx.compact = !!(opts->flags & MIDX_WRITE_COMPACT);\n+\n+\tif (ctx.compact) {\n+\t\tif (ctx.version != MIDX_VERSION_V2)\n+\t\t\tdie(_(\"cannot perform MIDX compaction with v1 format\"));\n+\t\tif (!opts->compact_from)\n+\t\t\tBUG(\"expected non-NULL 'from' MIDX during compaction\");\n+\t\tif (!opts->compact_to)\n+\t\t\tBUG(\"expected non-NULL 'to' MIDX during compaction\");\n+\n+\t\tctx.compact_from = opts->compact_from;\n+\t\tctx.compact_to = opts->compact_to;\n+\t}\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n@@ -1189,11 +1310,18 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t */\n \t\t\tif (ctx.incremental)\n \t\t\t\tctx.base_midx = m;\n-\t\t\telse if (!opts->packs_to_include)\n+\t\t\tif (!opts->packs_to_include)\n \t\t\t\tctx.m = m;\n \t\t}\n \t}\n \n+\t/*\n+\t * If compacting MIDX layer(s) in the range [from, to], then the\n+\t * compacted MIDX will share the same base MIDX as 'from'.\n+\t */\n+\tif (ctx.compact)\n+\t\tctx.base_midx = ctx.compact_from->base_midx;\n+\n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n@@ -1210,7 +1338,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\tctx.num_multi_pack_indexes_before++;\n \t\t\tm = m->base_midx;\n \t\t}\n-\t} else if (ctx.m && fill_packs_from_midx(&ctx)) {\n+\t} else if (ctx.m && !ctx.compact && fill_packs_from_midx(&ctx)) {\n \t\tgoto cleanup;\n \t}\n \n@@ -1223,9 +1351,18 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \telse\n \t\tctx.progress = NULL;\n \n-\tctx.to_include = opts->packs_to_include;\n+\tif (ctx.compact) {\n+\t\tint bitmap_order = 0;\n+\t\tif (opts->preferred_pack_name)\n+\t\t\tbitmap_order |= 1;\n+\t\telse if (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n+\t\t\tbitmap_order |= 1;\n \n-\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n+\t\tfill_packs_from_midx_range(&ctx, bitmap_order);\n+\t} else {\n+\t\tctx.to_include = opts->packs_to_include;\n+\t\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n+\t}\n \tstop_progress(&ctx.progress);\n \n \tif (!opts->packs_to_drop) {\n@@ -1354,12 +1491,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\tctx.large_offsets_needed = 1;\n \t}\n \n-\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n+\tif (ctx.compact) {\n+\t\tif (ctx.version != MIDX_VERSION_V2)\n+\t\t\tBUG(\"performing MIDX compaction with v1 MIDX\");\n+\t} else {\n+\t\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n+\t}\n \n \tif (opts->packs_to_drop && opts->packs_to_drop->nr) {\n \t\tsize_t drop_index = 0;\n \t\tint missing_drops = 0;\n \n+\t\tASSERT(!ctx.compact);\n+\n \t\tfor (size_t i = 0;\n \t\t i < ctx.nr && drop_index < opts->packs_to_drop->nr; i++) {\n \t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n@@ -1391,12 +1535,20 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t */\n \tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n \tfor (size_t i = 0; i < ctx.nr; i++) {\n+\t\tuint32_t from = ctx.info[i].orig_pack_int_id;\n+\t\tuint32_t to;\n+\n \t\tif (ctx.info[i].expired) {\n+\t\t\tto = PACK_EXPIRED;\n \t\t\tdropped_packs++;\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tto = i - dropped_packs;\n \t\t}\n+\n+\t\tif (ctx.compact)\n+\t\t\tfrom -= ctx.compact_from->num_packs_in_base;\n+\n+\t\tctx.pack_perm[from] = to;\n \t}\n \n \tfor (size_t i = 0; i < ctx.nr; i++) {\n@@ -1542,7 +1694,24 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n \t\tdie(_(\"too many multi-pack-indexes\"));\n \n-\tCALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);\n+\tif (ctx.compact) {\n+\t\tstruct multi_pack_index *m;\n+\n+\t\t/*\n+\t\t * Keep all MIDX layers excluding those in the range [from, to].\n+\t\t */\n+\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n+\t\t\tkeep_hashes_nr++;\n+\t\tfor (m = ctx.m;\n+\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n+\t\t m = m->base_midx)\n+\t\t\tkeep_hashes_nr++;\n+\n+\t\tkeep_hashes_nr++; /* include the compacted layer */\n+\t} else {\n+\t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n+\t}\n+\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n \n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n@@ -1567,17 +1736,47 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t\tstrbuf_release(&final_midx_name);\n \n-\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n-\t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n+\t\tif (ctx.compact) {\n+\t\t\tstruct multi_pack_index *m;\n+\t\t\tuint32_t num_layers_before_from = 0;\n+\t\t\tuint32_t i;\n \n-\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n-\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n+\t\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n+\t\t\t\tnum_layers_before_from++;\n \n-\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n-\t\t\tm = m->base_midx;\n+\t\t\tm = ctx.base_midx;\n+\t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n+\t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n+\n+\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tm = m->base_midx;\n+\t\t\t}\n+\n+\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t\t\t r->hash_algo));\n+\n+\t\t\ti = 0;\n+\t\t\tfor (m = ctx.m;\n+\t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n+\t\t\t m = m->base_midx) {\n+\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n+\t\t\t\ti++;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t\t r->hash_algo));\n+\n+\t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n+\t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n+\n+\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tm = m->base_midx;\n+\t\t\t}\n \t\t}\n \n-\t\tfor (uint32_t i = 0; i <= ctx.num_multi_pack_indexes_before; i++)\n+\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n \t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n \t} else {\n \t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n@@ -1590,8 +1789,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, keep_hashes,\n-\t\t\t ctx.num_multi_pack_indexes_before + 1,\n+\tclear_midx_files(opts->source, keep_hashes, keep_hashes_nr,\n \t\t\t ctx.incremental);\n \tresult = 0;\n \n@@ -1609,7 +1807,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tif (keep_hashes) {\n-\t\tfor (uint32_t i = 0; i <= ctx.num_multi_pack_indexes_before; i++)\n+\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n \t\t\tfree((char *)keep_hashes[i]);\n \t\tfree(keep_hashes);\n \t}\n@@ -1651,6 +1849,21 @@ int write_midx_file_only(struct odb_source *source,\n \treturn write_midx_internal(&opts);\n }\n \n+int write_midx_file_compact(struct odb_source *source,\n+\t\t\t struct multi_pack_index *from,\n+\t\t\t struct multi_pack_index *to,\n+\t\t\t unsigned flags)\n+{\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.compact_from = from,\n+\t\t.compact_to = to,\n+\t\t.flags = flags | MIDX_WRITE_COMPACT,\n+\t};\n+\n+\treturn write_midx_internal(&opts);\n+}\n+\n int expire_midx_packs(struct odb_source *source, unsigned flags)\n {\n \tuint32_t i, *count, result = 0;\ndiff --git a/midx.h b/midx.h\nindex aa99a6cb215..08f3728e520 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -82,6 +82,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n+#define MIDX_WRITE_COMPACT (1 << 6)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n@@ -131,6 +132,10 @@ int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n \t\t\t const char *refs_snapshot, unsigned flags);\n+int write_midx_file_compact(struct odb_source *source,\n+\t\t\t struct multi_pack_index *from,\n+\t\t\t struct multi_pack_index *to,\n+\t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct odb_source *source, unsigned flags);\n int expire_midx_packs(struct odb_source *source, unsigned flags);\ndiff --git a/t/meson.build b/t/meson.build\nindex 459c52a4897..5393322a8d2 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -614,6 +614,7 @@ integration_tests = [\n 't5332-multi-pack-reuse.sh',\n 't5333-pseudo-merge-bitmaps.sh',\n 't5334-incremental-multi-pack-index.sh',\n+ 't5335-compact-multi-pack-index.sh',\n 't5351-unpack-large-objects.sh',\n 't5400-send-pack.sh',\n 't5401-update-hooks.sh',\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nnew file mode 100755\nindex 00000000000..797ae05c3bd\n--- /dev/null\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -0,0 +1,175 @@\n+#!/bin/sh\n+\n+test_description='multi-pack-index compaction'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+midxdir=$packdir/multi-pack-index.d\n+midx_chain=$midxdir/multi-pack-index-chain\n+\n+nth_line() {\n+\tlocal n=\"$1\"\n+\tshift\n+\tawk \"NR==$n\" \"$@\"\n+}\n+\n+write_packs () {\n+\tfor c in \"$@\"\n+\tdo\n+\t\ttest_commit \"$c\" &&\n+\n+\t\tgit pack-objects --all --unpacked $packdir/pack-$c &&\n+\t\tgit prune-packed &&\n+\n+\t\tgit multi-pack-index write --incremental --bitmap || return 1\n+\tdone\n+}\n+\n+test_midx_layer_packs () {\n+\tlocal checksum=\"$1\" &&\n+\tshift &&\n+\n+\ttest-tool read-midx $objdir \"$checksum\" >out &&\n+\n+\tprintf \"%s\\n\" \"$@\" >expect &&\n+\t# NOTE: do *not* pipe through sort here, we want to ensure the\n+\t# order of packs is preserved during compaction.\n+\tgrep \"^pack-\" out | cut -d\"-\" -f2 >actual &&\n+\n+\ttest_cmp expect actual\n+}\n+\n+test_midx_layer_object_uniqueness () {\n+\t: >objs.all\n+\twhile read layer\n+\tdo\n+\t\ttest-tool read-midx --show-objects $objdir \"$layer\" >out &&\n+\t\tgrep \"\\.pack$\" out | cut -d\" \" -f1 | sort >objs.layer &&\n+\t\ttest_stdout_line_count = 0 comm -12 objs.all objs.layer &&\n+\t\tcat objs.all objs.layer | sort >objs.tmp &&\n+\t\tmv objs.tmp objs.all || return 1\n+\tdone <$midx_chain\n+}\n+\n+test_expect_success 'MIDX compaction with lex-ordered pack names' '\n+\tgit init midx-compact-lex-order &&\n+\t(\n+\t\tcd midx-compact-lex-order &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D E &&\n+\t\ttest_line_count = 5 $midx_chain &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\ttest_midx_layer_packs \"$(nth_line 1 \"$midx_chain\")\" A &&\n+\t\ttest_midx_layer_packs \"$(nth_line 2 \"$midx_chain\")\" B C D &&\n+\t\ttest_midx_layer_packs \"$(nth_line 3 \"$midx_chain\")\" E &&\n+\n+\t\ttest_midx_layer_object_uniqueness\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with non-lex-ordered pack names' '\n+\tgit init midx-compact-non-lex-order &&\n+\t(\n+\t\tcd midx-compact-non-lex-order &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs D C A B E &&\n+\t\ttest_line_count = 5 $midx_chain &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\ttest_midx_layer_packs \"$(nth_line 1 \"$midx_chain\")\" D &&\n+\t\ttest_midx_layer_packs \"$(nth_line 2 \"$midx_chain\")\" C A B &&\n+\t\ttest_midx_layer_packs \"$(nth_line 3 \"$midx_chain\")\" E &&\n+\n+\t\ttest_midx_layer_object_uniqueness\n+\t)\n+'\n+\n+test_expect_success 'setup for bogus MIDX compaction scenarios' '\n+\tgit init midx-compact-bogus &&\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with missing endpoints' '\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"\" \"\" 2>err &&\n+\t\ttest_grep \"could not find MIDX: \" err &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"\" \"$(nth_line 2 \"$midx_chain\")\" 2>err &&\n+\t\ttest_grep \"could not find MIDX: \" err &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \"\" 2>err &&\n+\t\ttest_grep \"could not find MIDX: \" err\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with reversed endpoints' '\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\tfrom=\"$(nth_line 3 \"$midx_chain\")\" &&\n+\t\tto=\"$(nth_line 1 \"$midx_chain\")\" &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"$from\" \"$to\" 2>err &&\n+\n+\t\ttest_grep \"MIDX $from must be an ancestor of $to\" err\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with identical endpoints' '\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\tfrom=\"$(nth_line 3 \"$midx_chain\")\" &&\n+\t\tto=\"$(nth_line 3 \"$midx_chain\")\" &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"$from\" \"$to\" 2>err &&\n+\n+\t\ttest_grep \"MIDX compaction endpoints must be unique\" err\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with midx.version=1' '\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\ttest_must_fail git -c midx.version=1 multi-pack-index compact \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" 2>err &&\n+\n+\t\ttest_grep \"fatal: cannot perform MIDX compaction with v1 format\" err\n+\t)\n+'\n+\n+test_done\n-- \n2.52.0.457.gb599f1ad4b0\n\n"},{"id":"533887","messageId":"b599f1ad4b0e0c9be5f027417bf343b52aed6e84.1768420450.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"[PATCH v2 18/18] midx: enable reachability bitmaps during MIDX compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T19:55:13Z","receivedAt":"2026-01-14T19:55:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Enable callers to generate reachability bitmaps when performing MIDX\nlayer compaction by combining all existing bitmaps from the compacted\nlayers.\n\nNote that because of the object/pack ordering described by the previous\ncommit, the pseudo-pack order for the compacted MIDX is the same as\nconcatenating the individual pseudo-pack orderings for each layer in the\ncompaction range.\n\nAs a result, the only non-test or documentation change necessary is to\ntreat all objects as non-preferred during compaction so as not to\ndisturb the object ordering.\n\nIn the future, we may want to adjust which commit(s) receive\nreachability bitmaps when compacting multiple .bitmap files into one, or\neven generate new bitmaps (e.g., if the references have moved\nsignificantly since the .bitmap was generated). This commit only\nimplements combining all existing bitmaps in range together in order to\ndemonstrate and lay the groundwork for more exotic strategies.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 5 +-\n builtin/multi-pack-index.c | 4 +-\n midx-write.c | 2 +-\n t/t5335-compact-multi-pack-index.sh | 122 +++++++++++++++++++++++-\n 4 files changed, 128 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 883a0529741..61256830141 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t \n+\t\t [--[no-]bitmap] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -94,6 +94,9 @@ compact::\n \t--incremental::\n \t\tWrite the result to a MIDX chain instead of writing a\n \t\tstand-alone MIDX.\n+\n+\t--[no-]bitmap::\n+\t\tControl whether or not a multi-pack bitmap is written.\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 043ee8c478a..2f24c113c8f 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -19,7 +19,7 @@\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" \")\n+\t \" [--[no-]bitmap] \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -216,6 +216,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n+\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_END(),\ndiff --git a/midx-write.c b/midx-write.c\nindex afa077a09cc..4c52843db9b 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -669,7 +669,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \t\tstruct pack_midx_entry *e = &ctx->entries[i];\n \t\tdata[i].nr = i;\n \t\tdata[i].pack = midx_pack_perm(ctx, e->pack_int_id);\n-\t\tif (!e->preferred)\n+\t\tif (!e->preferred || ctx->compact)\n \t\t\tdata[i].pack |= (1U << 31);\n \t\tdata[i].offset = e->offset;\n \t}\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 797ae05c3bd..40f3844282f 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -67,7 +67,7 @@ test_expect_success 'MIDX compaction with lex-ordered pack names' '\n \t\twrite_packs A B C D E &&\n \t\ttest_line_count = 5 $midx_chain &&\n \n-\t\tgit multi-pack-index compact --incremental \\\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n \t\ttest_line_count = 3 $midx_chain &&\n@@ -90,7 +90,7 @@ test_expect_success 'MIDX compaction with non-lex-ordered pack names' '\n \t\twrite_packs D C A B E &&\n \t\ttest_line_count = 5 $midx_chain &&\n \n-\t\tgit multi-pack-index compact --incremental \\\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n \t\ttest_line_count = 3 $midx_chain &&\n@@ -172,4 +172,122 @@ test_expect_success 'MIDX compaction with midx.version=1' '\n \t)\n '\n \n+midx_objs_by_pack () {\n+\tawk '/\\.pack$/ { split($3, a, \"-\"); print a[2], $1 }' | sort\n+}\n+\n+tag_objs_from_pack () {\n+\tobjs=\"$(git rev-list --objects --no-object-names \"$2\")\" &&\n+\tprintf \"$1 %s\\n\" $objs | sort\n+}\n+\n+test_expect_success 'MIDX compaction preserves pack object selection' '\n+\tgit init midx-compact-preserve-selection &&\n+\t(\n+\t\tcd midx-compact-preserve-selection &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\t# Create two packs, one containing just the objects from\n+\t\t# A, and another containing all objects from the\n+\t\t# repository.\n+\t\tp1=\"$(echo A | git pack-objects --revs --delta-base-offset \\\n+\t\t\t$packdir/pack-1)\" &&\n+\t\tp0=\"$(echo B | git pack-objects --revs --delta-base-offset \\\n+\t\t\t$packdir/pack-0)\" &&\n+\n+\t\techo \"pack-1-$p1.idx\" | git multi-pack-index write \\\n+\t\t\t--incremental --bitmap --stdin-packs &&\n+\t\techo \"pack-0-$p0.idx\" | git multi-pack-index write \\\n+\t\t\t--incremental --bitmap --stdin-packs &&\n+\n+\t\twrite_packs C &&\n+\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" &&\n+\n+\n+\t\ttest-tool read-midx --show-objects $objdir \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" >AB.info &&\n+\t\ttest-tool read-midx --show-objects $objdir \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" >C.info &&\n+\n+\t\tmidx_objs_by_pack AB.actual &&\n+\t\tmidx_objs_by_pack C.actual &&\n+\n+\t\t{\n+\t\t\ttag_objs_from_pack 1 A &&\n+\t\t\ttag_objs_from_pack 0 A..B\n+\t\t} | sort >AB.expect &&\n+\t\ttag_objs_from_pack C B..C >C.expect &&\n+\n+\t\ttest_cmp AB.expect AB.actual &&\n+\t\ttest_cmp C.expect C.actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bitmaps' '\n+\tgit init midx-compact-with-bitmaps &&\n+\t(\n+\t\tcd midx-compact-with-bitmaps &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs foo bar baz quux woot &&\n+\n+\t\ttest-tool read-midx --bitmap $objdir >bitmap.expect &&\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest-tool read-midx --bitmap $objdir >bitmap.actual &&\n+\n+\t\ttest_cmp bitmap.expect bitmap.actual &&\n+\n+\t\ttrue\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bitmaps (non-trivial)' '\n+\tgit init midx-compact-with-bitmaps-non-trivial &&\n+\t(\n+\t\tcd midx-compact-with-bitmaps-non-trivial &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\tgit branch -m main &&\n+\n+\t\t# D(4)\n+\t\t# /\n+\t\t# A(1) --- B(2) --- C(3) --- G(7)\n+\t\t# \\\n+\t\t# E(5) --- F(6)\n+\t\twrite_packs A B C &&\n+\t\tgit checkout -b side &&\n+\t\twrite_packs D &&\n+\t\tgit checkout -b other B &&\n+\t\twrite_packs E F &&\n+\t\tgit checkout main &&\n+\t\twrite_packs G &&\n+\n+\t\t# Compact layers 2-4, leaving us with:\n+\t\t#\n+\t\t# [A, [B, C, D], E, F, G]\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\n+\t\t# Then compact the top two layers, condensing the above\n+\t\t# such that the new 4th layer contains F and G.\n+\t\t#\n+\t\t# [A, [B, C, D], E, [F, G]]\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 5 \"$midx_chain\")\"\n+\t)\n+'\n+\n test_done\n-- \n2.52.0.457.gb599f1ad4b0\n"},{"id":"533891","messageId":"xmqq4iongaoi.fsf@gitster.g","threadId":"64585","inReplyTo":"315a0ea2985894f5ae5f166bdd1685ce0981fb23.1768420450.git.me@ttaylorr.com","subject":"Re: [PATCH v2 08/18] midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-14T21:13:17Z","receivedAt":"2026-01-14T21:13:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> But the process is completely unnecessary in the first place: we are\n> enumerating all values of `ctx->info`, and there is no reason to process\n> them in a different order than they appear in memory. Index `ctx->info`\n> directly to reflect that.\n\nAs long as pack_perm[] is permutations of [0..ctx->nr-1], it does\nnot matter, as we are clearing all the [0..ctx->nr-1] range anyway,\nin other words?\n\n>\n> Signed-off-by: Taylor Blau \n> ---\n> midx-write.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/midx-write.c b/midx-write.c\n> index 13171d7e9c4..da9c5a7c295 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -637,7 +637,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n> \t\tpack_order[i] = data[i].nr;\n> \t}\n> \tfor (i = 0; i < ctx->nr; i++) {\n> -\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n> +\t\tstruct pack_info *pack = &ctx->info[i];\n> \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n> \t\t\tpack->bitmap_pos = 0;\n> \t}\n"},{"id":"533895","messageId":"xmqqtswnevfc.fsf@gitster.g","threadId":"64585","inReplyTo":"72bcd4ed6c7f685f58bb3b905fe553173abe1845.1768420450.git.me@ttaylorr.com","subject":"Re: [PATCH v2 10/18] midx: do not require packs to be sorted in lexicographic order","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-14T21:28:07Z","receivedAt":"2026-01-14T21:28:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> @@ -374,7 +374,7 @@ HEADER:\n> \t The signature is: {'M', 'I', 'D', 'X'}\n> \n> \t1-byte version number:\n> -\t Git only writes or recognizes version 1.\n> +\t Git only writes version 2, but recognizes versions 1 and 2.\n\nWe only write version 2, then ...\n\n> diff --git a/midx-write.c b/midx-write.c\n> index 8a54644e427..5c8700065a1 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -36,10 +36,13 @@ extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n> \n> static size_t write_midx_header(const struct git_hash_algo *hash_algo,\n> \t\t\t\tstruct hashfile *f, unsigned char num_chunks,\n> -\t\t\t\tuint32_t num_packs)\n> +\t\t\t\tuint32_t num_packs, int version)\n> {\n> +\tif (version != MIDX_VERSION_V1 && version != MIDX_VERSION_V2)\n> +\t\tBUG(\"unexpected MIDX version: %d\", version);\n> +\n\n... do we need to add version parameter to this function? Unless\nthe writer that calls this helper function demotes version to 1 when\nit realizes that the pack files we write midx for happens to be\nsorted, in which case it may need to call this function with version\nset to 1, I do not quite see why we need it.\n\n> \thashwrite_be32(f, MIDX_SIGNATURE);\n> -\thashwrite_u8(f, MIDX_VERSION);\n> +\thashwrite_u8(f, version);\n> \thashwrite_u8(f, oid_version(hash_algo));\n> \thashwrite_u8(f, num_chunks);\n> \thashwrite_u8(f, 0); /* unused */\n> @@ -105,6 +108,8 @@ struct write_midx_context {\n> \n> \tuint32_t preferred_pack_idx;\n> \n> +\tint version; /* must be MIDX_VERSION_V1 or _V2 */\n> +\n\nDitto. When we are writing it out, shouldn't this always be 2\nanyway?\n\n> @@ -410,7 +415,9 @@ static int write_midx_pack_names(struct hashfile *f, void *data)\n> \t\tif (ctx->info[i].expired)\n> \t\t\tcontinue;\n> \n> -\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n> +\t\tif (ctx->version == MIDX_VERSION_V1 &&\n> +\t\t i && strcmp(ctx->info[i].pack_name,\n> +\t\t\t\tctx->info[i - 1].pack_name) <= 0)\n> \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n> \t\t\t ctx->info[i - 1].pack_name,\n> \t\t\t ctx->info[i].pack_name);\n\nDitto.\n\n> @@ -1025,6 +1032,12 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n> \tif (!midx_checksum_valid(midx))\n> \t\tgoto out;\n> \n> +\t/*\n> +\t * If the version differs, we need to update.\n> +\t */\n> +\tif (midx->version != ctx->version)\n> +\t\tgoto out;\n> +\n\nOK.\n\n> @@ -1100,6 +1113,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \tstruct tempfile *incr;\n> \tstruct write_midx_context ctx = {\n> \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n> +\t\t.version = MIDX_VERSION_V2,\n> \t };\n> \tstruct multi_pack_index *midx_to_free = NULL;\n> \tint bitmapped_packs_concat_len = 0;\n> @@ -1114,6 +1128,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \tctx.repo = r;\n> \tctx.source = opts->source;\n> \n> +\trepo_config_get_int(ctx.repo, \"midx.version\", &ctx.version);\n> +\tif (ctx.version != MIDX_VERSION_V1 && ctx.version != MIDX_VERSION_V2)\n> +\t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n> +\n\nDitto.\n\n> @@ -1445,7 +1463,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \t}\n> \n> \twrite_midx_header(r->hash_algo, f, get_num_chunks(cf),\n> -\t\t\t ctx.nr - dropped_packs);\n> +\t\t\t ctx.nr - dropped_packs, ctx.version);\n> \twrite_chunkfile(cf, &ctx);\n> \n> \tfinalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,\n\nDitto.\n\n> diff --git a/midx.h b/midx.h\n> index a39bcc9d03f..aa99a6cb215 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -11,7 +11,8 @@ struct git_hash_algo;\n> struct odb_source;\n> \n> #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n> -#define MIDX_VERSION 1\n> +#define MIDX_VERSION_V1 1\n> +#define MIDX_VERSION_V2 2\n> #define MIDX_BYTE_FILE_VERSION 4\n> #define MIDX_BYTE_HASH_VERSION 5\n> #define MIDX_BYTE_NUM_CHUNKS 6\n> @@ -71,6 +72,7 @@ struct multi_pack_index {\n> \tuint32_t num_packs_in_base;\n> \n> \tconst char **pack_names;\n> +\tsize_t *pack_names_sorted;\n> \tstruct packed_git **packs;\n> };\n\nThis does make sense. The code paths that reads existing on-disk\nfiles must notice when they are dealing with v1 format and need to\nact differently.\n\nThanks.\n"},{"id":"533900","messageId":"aWgNVjCej8wO85iD@nand.local","threadId":"64585","inReplyTo":"xmqq4iongaoi.fsf@gitster.g","subject":"Re: [PATCH v2 08/18] midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T21:40:38Z","receivedAt":"2026-01-14T21:40:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 14, 2026 at 01:13:17PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > But the process is completely unnecessary in the first place: we are\n> > enumerating all values of `ctx->info`, and there is no reason to process\n> > them in a different order than they appear in memory. Index `ctx->info`\n> > directly to reflect that.\n>\n> As long as pack_perm[] is permutations of [0..ctx->nr-1], it does\n> not matter, as we are clearing all the [0..ctx->nr-1] range anyway,\n> in other words?\n\nThat's right. The contents of ctx->pack_perm here aren't so important\nsince it doesn't matter what order we consider the pack_info structs in.\nAs long as we see all of them we are OK, hence iterating over the\nctx->info array directly rather than in the permuted order given by\npack_perm.\n\nThanks,\nTaylor\n"},{"id":"533901","messageId":"aWgOVnp+zkGiR8p6@nand.local","threadId":"64585","inReplyTo":"xmqqtswnevfc.fsf@gitster.g","subject":"Re: [PATCH v2 10/18] midx: do not require packs to be sorted in lexicographic order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T21:44:54Z","receivedAt":"2026-01-14T21:44:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 14, 2026 at 01:28:07PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > @@ -374,7 +374,7 @@ HEADER:\n> > \t The signature is: {'M', 'I', 'D', 'X'}\n> >\n> > \t1-byte version number:\n> > -\t Git only writes or recognizes version 1.\n> > +\t Git only writes version 2, but recognizes versions 1 and 2.\n>\n> We only write version 2, then ...\n\nAh, thanks for spotting.\n\nThis comment is outdated, and I didn't catch it when proof-reading\nthe patches. I wrote this line before adding the \"midx.version\" config\nescape hatch, and doing so made this comment stale.\n\nIt should probably look something like:\n\n Git writes the version specified by the \"midx.version\"\n configuration option, which defaults to 2. It recognizes\n both versions 1 and 2.\n\nI'll update it and include it in the subsequent round.\n\n> > \thashwrite_be32(f, MIDX_SIGNATURE);\n> > -\thashwrite_u8(f, MIDX_VERSION);\n> > +\thashwrite_u8(f, version);\n> > \thashwrite_u8(f, oid_version(hash_algo));\n> > \thashwrite_u8(f, num_chunks);\n> > \thashwrite_u8(f, 0); /* unused */\n> > @@ -105,6 +108,8 @@ struct write_midx_context {\n> >\n> > \tuint32_t preferred_pack_idx;\n> >\n> > +\tint version; /* must be MIDX_VERSION_V1 or _V2 */\n> > +\n>\n> Ditto. When we are writing it out, shouldn't this always be 2\n> anyway?\n\nFor this and below, the code here is right and the documentation is\nwrong (per above).\n\n> > @@ -410,7 +415,9 @@ static int write_midx_pack_names(struct hashfile *f, void *data)\n> > \t\tif (ctx->info[i].expired)\n> > \t\t\tcontinue;\n> >\n> > -\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n> > +\t\tif (ctx->version == MIDX_VERSION_V1 &&\n> > +\t\t i && strcmp(ctx->info[i].pack_name,\n> > +\t\t\t\tctx->info[i - 1].pack_name) <= 0)\n> > \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n> > \t\t\t ctx->info[i - 1].pack_name,\n> > \t\t\t ctx->info[i].pack_name);\n>\n> Ditto.\n\nDitto (and so on) ;-).\n\nThanks,\nTaylor\n"},{"id":"533903","messageId":"xmqqpl7beugj.fsf@gitster.g","threadId":"64585","inReplyTo":"c0c1769464b1c8065c2cea59dfd85a1d37de9dd1.1768420450.git.me@ttaylorr.com","subject":"Re: [PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-14T21:49:00Z","receivedAt":"2026-01-14T21:49:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> A future commit will want to add two 32-bit unsigned values together\n> while checking for overflow. Introduce a variant of the u64_add()\n> function for operating on 32-bit inputs.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> git-compat-util.h | 8 ++++++++\n> 1 file changed, 8 insertions(+)\n>\n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index b0673d1a450..db62a6f25c5 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -641,6 +641,14 @@ static inline int cast_size_t_to_int(size_t a)\n> \treturn (int)a;\n> }\n> \n> +static inline uint32_t u32_add(uint32_t a, uint32_t b)\n> +{\n> +\tif (unsigned_add_overflows(a, b))\n> +\t\tdie(\"uint32_t overflow: %\"PRIuMAX\" + %\"PRIuMAX,\n> +\t\t (uintmax_t)a, (uintmax_t)b);\n> +\treturn a + b;\n> +}\n\nNeither this one, nor the original u64_add(), seem to me a\nparticularly good API.\n\nWhen things might overflow, it is a given that we should give a\ncontrolled death rather than nonsense behaviour and/or corrupt\noutput, but shouldn't the diagnosis message given to the end-user\nwhen we find an overflow be given at a bit higher layer? At this\nlevel, you do not even know what quantities you are adding together\n*means*. Even though your caller might be able to give a more\nintelligible message like \"the number of packs to combine exceeds\n2^32 that is too many\". But dying in these functions means the\ncallers have no chance to do so. The only thing the user sees is\nsome code tried to add two u32 and overflowed---without any hint\nwhat these quantities were or what the addition was trying to\ncompute.\n\n\n\n\n"},{"id":"533904","messageId":"aWgSzI30k0BZfZ4Q@nand.local","threadId":"64585","inReplyTo":"xmqqpl7beugj.fsf@gitster.g","subject":"Re: [PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-14T22:03:56Z","receivedAt":"2026-01-14T22:04:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 14, 2026 at 01:49:00PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > A future commit will want to add two 32-bit unsigned values together\n> > while checking for overflow. Introduce a variant of the u64_add()\n> > function for operating on 32-bit inputs.\n> >\n> > Signed-off-by: Taylor Blau \n> > ---\n> > git-compat-util.h | 8 ++++++++\n> > 1 file changed, 8 insertions(+)\n> >\n> > diff --git a/git-compat-util.h b/git-compat-util.h\n> > index b0673d1a450..db62a6f25c5 100644\n> > --- a/git-compat-util.h\n> > +++ b/git-compat-util.h\n> > @@ -641,6 +641,14 @@ static inline int cast_size_t_to_int(size_t a)\n> > \treturn (int)a;\n> > }\n> >\n> > +static inline uint32_t u32_add(uint32_t a, uint32_t b)\n> > +{\n> > +\tif (unsigned_add_overflows(a, b))\n> > +\t\tdie(\"uint32_t overflow: %\"PRIuMAX\" + %\"PRIuMAX,\n> > +\t\t (uintmax_t)a, (uintmax_t)b);\n> > +\treturn a + b;\n> > +}\n>\n> Neither this one, nor the original u64_add(), seem to me a\n> particularly good API.\n>\n> When things might overflow, it is a given that we should give a\n> controlled death rather than nonsense behaviour and/or corrupt\n> output, but shouldn't the diagnosis message given to the end-user\n> when we find an overflow be given at a bit higher layer? At this\n> level, you do not even know what quantities you are adding together\n> *means*. Even though your caller might be able to give a more\n> intelligible message like \"the number of packs to combine exceeds\n> 2^32 that is too many\". But dying in these functions means the\n> callers have no chance to do so. The only thing the user sees is\n> some code tried to add two u32 and overflowed---without any hint\n> what these quantities were or what the addition was trying to\n> compute.\n\nI agree with your sentiment here. Let's avoid proliferating an API\npattern that encourages non-descriptive error message by dropping the\nsole caller of u32_add() in this series with the following:\n\n--- 8< ---\ndiff --git a/midx-write.c b/midx-write.c\nindex afa077a09cc..99e7116f4d0 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -990,7 +990,10 @@ static uint32_t compactible_packs_between(const struct multi_pack_index *from,\n\n \tASSERT(from && to);\n\n-\tnr = u32_add(to->num_packs, to->num_packs_in_base);\n+\tif (unsigned_add_overflows(to->num_packs, to->num_packs_in_base))\n+\t\tdie(_(\"too many packs, unable to compact\"));\n+\n+\tnr = to->num_packs + to->num_packs_in_base;\n \tif (nr < from->num_packs_in_base)\n \t\tBUG(\"unexpected number of packs in base during compaction: \"\n \t\t \"%\"PRIu32\" < %\"PRIu32, nr, from->num_packs_in_base);\n--- >8 ---\n\nAs for removing u64_add(), that should be straightforward as well since\nthere is also a single caller. Let me know if you think that makes sense\nto take up as part of this series, or if you would prefer it done\nseparately. I tend to prefer the latter, since the state after applying\nthe above is that we avoid adding any new callers.\n\nRemoving st_add() and st_mult(), on the other hand, will be much more\ninvolved since they are extremely widely used by comparison.\n\nThanks,\nTaylor\n"},{"id":"533915","messageId":"aWgwn2rk/qw+fRoA@nand.local","threadId":"64585","inReplyTo":"aWgSzI30k0BZfZ4Q@nand.local","subject":"Re: [PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-15T00:11:11Z","receivedAt":"2026-01-15T00:11:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 14, 2026 at 05:03:56PM -0500, Taylor Blau wrote:\n> As for removing u64_add(), that should be straightforward as well since\n> there is also a single caller. Let me know if you think that makes sense\n> to take up as part of this series, or if you would prefer it done\n> separately. I tend to prefer the latter, since the state after applying\n> the above is that we avoid adding any new callers.\n\nThis appears to be easy enough. The following applies on top of 'master'\nif you want to pick it up separately:\n\n--- 8< ---\n\nSubject: [PATCH] git-compat-util.h: drop u64_add(), u64_mult() helpers\n\nThe u64_add() and u64_mult() helper functions were introduced in\nb103881d4f4 (midx repack: avoid integer overflow on 32 bit systems,\n2025-05-22) to implement overflow checks during a fixed-point\ncalculation when estimating pack sizes in the MIDX writing code.\n\nHowever, those functions call die() when either the addition or\nmultiplication of their operands (depending on which function is being\ncalled) would cause an overflow. This does not allow the caller to\nprovide a more detailed message, presenting the user with an opaque\nmessage like:\n\n fatal: uint64_t overflow: M * N\n\nLet's discourage these opaque error messages by dropping these functions\nentirely and instead having the caller use unsigned_mult_overflows() or\nunsigned_add_overflows() themselves, providing the caller the\nopportunity to come up with their own die() message.\n\nSuggested-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n git-compat-util.h | 16 ----------------\n midx-write.c | 15 +++++++++++++--\n 2 files changed, 13 insertions(+), 18 deletions(-)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b0673d1a450..24edd68c671 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -641,22 +641,6 @@ static inline int cast_size_t_to_int(size_t a)\n \treturn (int)a;\n }\n\n-static inline uint64_t u64_mult(uint64_t a, uint64_t b)\n-{\n-\tif (unsigned_mult_overflows(a, b))\n-\t\tdie(\"uint64_t overflow: %\"PRIuMAX\" * %\"PRIuMAX,\n-\t\t (uintmax_t)a, (uintmax_t)b);\n-\treturn a * b;\n-}\n-\n-static inline uint64_t u64_add(uint64_t a, uint64_t b)\n-{\n-\tif (unsigned_add_overflows(a, b))\n-\t\tdie(\"uint64_t overflow: %\"PRIuMAX\" + %\"PRIuMAX,\n-\t\t (uintmax_t)a, (uintmax_t)b);\n-\treturn a + b;\n-}\n-\n /*\n * Limit size of IO chunks, because huge chunks only cause pain. OS X\n * 64-bit is buggy, returning EINVAL if len >= INT_MAX; and even in\ndiff --git a/midx-write.c b/midx-write.c\nindex 87b97c70872..6006b6569c8 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1738,8 +1738,19 @@ static void fill_included_packs_batch(struct repository *r,\n \t\t */\n \t\texpected_size = (uint64_t)pack_info[i].referenced_objects << 14;\n \t\texpected_size /= p->num_objects;\n-\t\texpected_size = u64_mult(expected_size, p->pack_size);\n-\t\texpected_size = u64_add(expected_size, 1u << 13) >> 14;\n+\n+\t\tif (unsigned_mult_overflows(expected_size,\n+\t\t\t\t\t (uint64_t)p->pack_size))\n+\t\t\tdie(_(\"overflow during fixed-point multiply (%\"PRIu64\" \"\n+\t\t\t \"* %\"PRIu64\")\"),\n+\t\t\t expected_size, (uint64_t)p->pack_size);\n+\t\texpected_size = expected_size * p->pack_size;\n+\n+\t\tif (unsigned_add_overflows(expected_size, 1u << 13))\n+\t\t\tdie(_(\"overflow during fixed-point rounding (%\"PRIu64\" \"\n+\t\t\t \" + %\"PRIu64\")\"),\n+\t\t\t expected_size, (uint64_t)(1ul << 13));\n+\t\texpected_size = (expected_size + (1u << 13)) >> 14;\n\n \t\tif (expected_size >= batch_size)\n \t\t\tcontinue;\n\nbase-commit: 8745eae506f700657882b9e32b2aa00f234a6fb6\n--\n2.52.0.436.g7dc2c5478ff\n"},{"id":"534320","messageId":"aXCTkVpjJkTabx_0@pks.im","threadId":"64585","inReplyTo":"aWgwn2rk/qw+fRoA@nand.local","subject":"Re: [PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-21T08:51:36Z","receivedAt":"2026-01-21T08:51:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 14, 2026 at 07:11:11PM -0500, Taylor Blau wrote:\n> On Wed, Jan 14, 2026 at 05:03:56PM -0500, Taylor Blau wrote:\n> > As for removing u64_add(), that should be straightforward as well since\n> > there is also a single caller. Let me know if you think that makes sense\n> > to take up as part of this series, or if you would prefer it done\n> > separately. I tend to prefer the latter, since the state after applying\n> > the above is that we avoid adding any new callers.\n> \n> This appears to be easy enough. The following applies on top of 'master'\n> if you want to pick it up separately:\n> \n> --- 8< ---\n> \n> Subject: [PATCH] git-compat-util.h: drop u64_add(), u64_mult() helpers\n> \n> The u64_add() and u64_mult() helper functions were introduced in\n> b103881d4f4 (midx repack: avoid integer overflow on 32 bit systems,\n> 2025-05-22) to implement overflow checks during a fixed-point\n> calculation when estimating pack sizes in the MIDX writing code.\n> \n> However, those functions call die() when either the addition or\n> multiplication of their operands (depending on which function is being\n> called) would cause an overflow. This does not allow the caller to\n> provide a more detailed message, presenting the user with an opaque\n> message like:\n> \n> fatal: uint64_t overflow: M * N\n> \n> Let's discourage these opaque error messages by dropping these functions\n> entirely and instead having the caller use unsigned_mult_overflows() or\n> unsigned_add_overflows() themselves, providing the caller the\n> opportunity to come up with their own die() message.\n> \n> Suggested-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> ---\n> git-compat-util.h | 16 ----------------\n> midx-write.c | 15 +++++++++++++--\n> 2 files changed, 13 insertions(+), 18 deletions(-)\n> \n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index b0673d1a450..24edd68c671 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -641,22 +641,6 @@ static inline int cast_size_t_to_int(size_t a)\n> \treturn (int)a;\n> }\n> \n> -static inline uint64_t u64_mult(uint64_t a, uint64_t b)\n> -{\n> -\tif (unsigned_mult_overflows(a, b))\n> -\t\tdie(\"uint64_t overflow: %\"PRIuMAX\" * %\"PRIuMAX,\n> -\t\t (uintmax_t)a, (uintmax_t)b);\n> -\treturn a * b;\n> -}\n> -\n> -static inline uint64_t u64_add(uint64_t a, uint64_t b)\n> -{\n> -\tif (unsigned_add_overflows(a, b))\n> -\t\tdie(\"uint64_t overflow: %\"PRIuMAX\" + %\"PRIuMAX,\n> -\t\t (uintmax_t)a, (uintmax_t)b);\n> -\treturn a + b;\n> -}\n> -\n> /*\n> * Limit size of IO chunks, because huge chunks only cause pain. OS X\n> * 64-bit is buggy, returning EINVAL if len >= INT_MAX; and even in\n> diff --git a/midx-write.c b/midx-write.c\n> index 87b97c70872..6006b6569c8 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -1738,8 +1738,19 @@ static void fill_included_packs_batch(struct repository *r,\n> \t\t */\n> \t\texpected_size = (uint64_t)pack_info[i].referenced_objects << 14;\n> \t\texpected_size /= p->num_objects;\n> -\t\texpected_size = u64_mult(expected_size, p->pack_size);\n> -\t\texpected_size = u64_add(expected_size, 1u << 13) >> 14;\n> +\n> +\t\tif (unsigned_mult_overflows(expected_size,\n> +\t\t\t\t\t (uint64_t)p->pack_size))\n> +\t\t\tdie(_(\"overflow during fixed-point multiply (%\"PRIu64\" \"\n> +\t\t\t \"* %\"PRIu64\")\"),\n> +\t\t\t expected_size, (uint64_t)p->pack_size);\n> +\t\texpected_size = expected_size * p->pack_size;\n> +\n> +\t\tif (unsigned_add_overflows(expected_size, 1u << 13))\n> +\t\t\tdie(_(\"overflow during fixed-point rounding (%\"PRIu64\" \"\n> +\t\t\t \" + %\"PRIu64\")\"),\n> +\t\t\t expected_size, (uint64_t)(1ul << 13));\n> +\t\texpected_size = (expected_size + (1u << 13)) >> 14;\n\nOne downside this pattern has is that we repeat the computation, which\nmakes it easy to get it wrong or forget updating either the check or the\ncomputation.\n\nI think ideally, we would have interfaces that combine the two\napproaches in `u64_mult()` and `unsigned_mult_overflows()`. Something\nlike this for example:\n\n static intline bool u64_mult(uint64_t a, uint64_t b, uint64_t *out)\n {\n if (unsigned_mult_overflows(a, b))\n return false;\n *out = a * b;\n return true;\n }\n\nThis would let the caller handle the failure and is thus quite flexible,\nwhich results in the following code:\n\n\tif (!u64_mult(expected_size, (uint64_t)p->pack_size, &expected_size))\n\t\tdie(_(\"overflow during fixed-point multiply (%\"PRIu64\" \"\n\t\t \"* %\"PRIu64\")\"), expected_size, (uint64_t)p->pack_size);\n\nPatrick\n"},{"id":"534417","messageId":"aXFni2tE7vn1dKFp@nand.local","threadId":"64585","inReplyTo":"aXCTkVpjJkTabx_0@pks.im","subject":"Re: [PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-21T23:55:55Z","receivedAt":"2026-01-21T23:55:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 21, 2026 at 09:51:36AM +0100, Patrick Steinhardt wrote:\n> > diff --git a/midx-write.c b/midx-write.c\n> > index 87b97c70872..6006b6569c8 100644\n> > --- a/midx-write.c\n> > +++ b/midx-write.c\n> > @@ -1738,8 +1738,19 @@ static void fill_included_packs_batch(struct repository *r,\n> > \t\t */\n> > \t\texpected_size = (uint64_t)pack_info[i].referenced_objects << 14;\n> > \t\texpected_size /= p->num_objects;\n> > -\t\texpected_size = u64_mult(expected_size, p->pack_size);\n> > -\t\texpected_size = u64_add(expected_size, 1u << 13) >> 14;\n> > +\n> > +\t\tif (unsigned_mult_overflows(expected_size,\n> > +\t\t\t\t\t (uint64_t)p->pack_size))\n> > +\t\t\tdie(_(\"overflow during fixed-point multiply (%\"PRIu64\" \"\n> > +\t\t\t \"* %\"PRIu64\")\"),\n> > +\t\t\t expected_size, (uint64_t)p->pack_size);\n> > +\t\texpected_size = expected_size * p->pack_size;\n> > +\n> > +\t\tif (unsigned_add_overflows(expected_size, 1u << 13))\n> > +\t\t\tdie(_(\"overflow during fixed-point rounding (%\"PRIu64\" \"\n> > +\t\t\t \" + %\"PRIu64\")\"),\n> > +\t\t\t expected_size, (uint64_t)(1ul << 13));\n> > +\t\texpected_size = (expected_size + (1u << 13)) >> 14;\n>\n> One downside this pattern has is that we repeat the computation, which\n> makes it easy to get it wrong or forget updating either the check or the\n> computation.\n>\n> I think ideally, we would have interfaces that combine the two\n> approaches in `u64_mult()` and `unsigned_mult_overflows()`. Something\n> like this for example:\n>\n> static intline bool u64_mult(uint64_t a, uint64_t b, uint64_t *out)\n> {\n> if (unsigned_mult_overflows(a, b))\n> return false;\n> *out = a * b;\n> return true;\n> }\n\nI had considered this approach when writing, but ultimately decided\nagainst it, since it felt a little clunky to have to pass a pointer in\nto do a simple arithmetic operation. But I think your point about\nensuring that we actually do:\n\n if (unsigned_mult_overflows(a, b))\n die(...);\n result = a * b;\n\nand not \"result = a * c\" or some other expression which is not \"a * b\"\nis a good one.\n\nI dunno. The spots in this patch are the only uses of u64_mult() and\nu64_add(), so I'm hesitant to keep a helper function around just for\nthat sole use-case. I wonder if we should do what you suggest here for\nthe much more frequently used st_add() / st_mult() / st_sub() functions?\n\n> This would let the caller handle the failure and is thus quite flexible,\n> which results in the following code:\n>\n> \tif (!u64_mult(expected_size, (uint64_t)p->pack_size, &expected_size))\n> \t\tdie(_(\"overflow during fixed-point multiply (%\"PRIu64\" \"\n> \t\t \"* %\"PRIu64\")\"), expected_size, (uint64_t)p->pack_size);\n\nIt does read quite cleanly, so I think I'm convinced.\n\nThanks,\nTaylor\n"},{"id":"534424","messageId":"014c01dc8b46$7a2997b0$6e7cc710$@nexbridge.com","threadId":"64585","inReplyTo":"aXFni2tE7vn1dKFp@nand.local","subject":"RE: [PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2026-01-22T02:26:06Z","receivedAt":"2026-01-22T02:26:25Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On Taylor Blau January 21, 2026 6:56 PM writes:\n>On Wed, Jan 21, 2026 at 09:51:36AM +0100, Patrick Steinhardt wrote:\n>> > diff --git a/midx-write.c b/midx-write.c index\n>> > 87b97c70872..6006b6569c8 100644\n>> > --- a/midx-write.c\n>> > +++ b/midx-write.c\n>> > @@ -1738,8 +1738,19 @@ static void fill_included_packs_batch(struct\n>repository *r,\n>> > \t\t */\n>> > \t\texpected_size = (uint64_t)pack_info[i].referenced_objects << 14;\n>> > \t\texpected_size /= p->num_objects;\n>> > -\t\texpected_size = u64_mult(expected_size, p->pack_size);\n>> > -\t\texpected_size = u64_add(expected_size, 1u << 13) >> 14;\n>> > +\n>> > +\t\tif (unsigned_mult_overflows(expected_size,\n>> > +\t\t\t\t\t (uint64_t)p->pack_size))\n>> > +\t\t\tdie(_(\"overflow during fixed-point multiply (%\"PRIu64\" \"\n>> > +\t\t\t \"* %\"PRIu64\")\"),\n>> > +\t\t\t expected_size, (uint64_t)p->pack_size);\n>> > +\t\texpected_size = expected_size * p->pack_size;\n>> > +\n>> > +\t\tif (unsigned_add_overflows(expected_size, 1u << 13))\n>> > +\t\t\tdie(_(\"overflow during fixed-point rounding (%\"PRIu64\" \"\n>> > +\t\t\t \" + %\"PRIu64\")\"),\n>> > +\t\t\t expected_size, (uint64_t)(1ul << 13));\n>> > +\t\texpected_size = (expected_size + (1u << 13)) >> 14;\n>>\n>> One downside this pattern has is that we repeat the computation, which\n>> makes it easy to get it wrong or forget updating either the check or\n>> the computation.\n>>\n>> I think ideally, we would have interfaces that combine the two\n>> approaches in `u64_mult()` and `unsigned_mult_overflows()`. Something\n>> like this for example:\n>>\n>> static intline bool u64_mult(uint64_t a, uint64_t b, uint64_t *out)\n>> {\n>> if (unsigned_mult_overflows(a, b))\n>> return false;\n>> *out = a * b;\n>> return true;\n>> }\n>\n>I had considered this approach when writing, but ultimately decided against it, since\n>it felt a little clunky to have to pass a pointer in to do a simple arithmetic operation.\n>But I think your point about ensuring that we actually do:\n>\n> if (unsigned_mult_overflows(a, b))\n> die(...);\n> result = a * b;\n>\n>and not \"result = a * c\" or some other expression which is not \"a * b\"\n>is a good one.\n>\n>I dunno. The spots in this patch are the only uses of u64_mult() and u64_add(), so\n>I'm hesitant to keep a helper function around just for that sole use-case. I wonder if\n>we should do what you suggest here for the much more frequently used st_add() /\n>st_mult() / st_sub() functions?\n>\n>> This would let the caller handle the failure and is thus quite\n>> flexible, which results in the following code:\n>>\n>> \tif (!u64_mult(expected_size, (uint64_t)p->pack_size, &expected_size))\n>> \t\tdie(_(\"overflow during fixed-point multiply (%\"PRIu64\" \"\n>> \t\t \"* %\"PRIu64\")\"), expected_size, (uint64_t)p->pack_size);\n>\n>It does read quite cleanly, so I think I'm convinced.\n\nAre all of these changes endian-safe?\n\n"},{"id":"534471","messageId":"xmqq4iodpodf.fsf@gitster.g","threadId":"64585","inReplyTo":"014c01dc8b46$7a2997b0$6e7cc710$@nexbridge.com","subject":"Re: [PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-01-22T17:07:40Z","receivedAt":"2026-01-22T17:07:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":" writes:\n\n>>> This would let the caller handle the failure and is thus quite\n>>> flexible, which results in the following code:\n>>>\n>>> \tif (!u64_mult(expected_size, (uint64_t)p->pack_size, &expected_size))\n>>> \t\tdie(_(\"overflow during fixed-point multiply (%\"PRIu64\" \"\n>>> \t\t \"* %\"PRIu64\")\"), expected_size, (uint64_t)p->pack_size);\n>>\n>>It does read quite cleanly, so I think I'm convinced.\n>\n> Are all of these changes endian-safe?\n\nI do not think of any endian-ness issue around here; these \"mult\"\nhelpers are checking if the result would overflow using regular\ndivision, without doing any funny byte-shuffling ourselves.\n has definitions of unsigned_mult_overflows(),\nand maximum_unsigned_value_of_type(), if you are curious.\n\n"},{"id":"534703","messageId":"aXhqo3f-NeUcO2IM@pks.im","threadId":"64585","inReplyTo":"72bcd4ed6c7f685f58bb3b905fe553173abe1845.1768420450.git.me@ttaylorr.com","subject":"Re: [PATCH v2 10/18] midx: do not require packs to be sorted in lexicographic order","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-27T07:34:59Z","receivedAt":"2026-01-27T07:35:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 14, 2026 at 02:54:45PM -0500, Taylor Blau wrote:\n[snip]\n> Because this change produces MIDXs which may not be correctly read with\n> external tools or older versions of Git. Though older versions of Git\n> know how to gracefully degrade and ignore any MIDX(s) they consider\n> corrupt, external tools may not be as robust. To avoid unintentionally\n> breaking any such tools, guard this change behind a version bump in the\n> MIDX's on-disk format.\n\ns/Because t/T/?\n\n> diff --git a/midx-write.c b/midx-write.c\n> index 8a54644e427..5c8700065a1 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -36,10 +36,13 @@ extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n> \n> static size_t write_midx_header(const struct git_hash_algo *hash_algo,\n> \t\t\t\tstruct hashfile *f, unsigned char num_chunks,\n> -\t\t\t\tuint32_t num_packs)\n> +\t\t\t\tuint32_t num_packs, int version)\n> {\n> +\tif (version != MIDX_VERSION_V1 && version != MIDX_VERSION_V2)\n> +\t\tBUG(\"unexpected MIDX version: %d\", version);\n\nAsking the function to write a MIDX header of an unsupported version\nwould be a bug indeed.\n\n> @@ -105,6 +108,8 @@ struct write_midx_context {\n> \n> \tuint32_t preferred_pack_idx;\n> \n> +\tint version; /* must be MIDX_VERSION_V1 or _V2 */\n\nTiny nit: this could be converted into an `enum` for implicit\ndocumentation.\n\n> @@ -1100,6 +1113,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \tstruct tempfile *incr;\n> \tstruct write_midx_context ctx = {\n> \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n> +\t\t.version = MIDX_VERSION_V2,\n> \t };\n> \tstruct multi_pack_index *midx_to_free = NULL;\n> \tint bitmapped_packs_concat_len = 0;\n\nOkay. Here we set the default version, ...\n\n> @@ -1114,6 +1128,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \tctx.repo = r;\n> \tctx.source = opts->source;\n> \n> +\trepo_config_get_int(ctx.repo, \"midx.version\", &ctx.version);\n> +\tif (ctx.version != MIDX_VERSION_V1 && ctx.version != MIDX_VERSION_V2)\n> +\t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n> +\n> \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n> \n> \tif (ctx.incremental)\n\n... but the user can change it. Good.\n\n> diff --git a/midx.c b/midx.c\n> index 19ef230d3fd..1327d0a3695 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -656,17 +658,40 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n> \treturn strcmp(idx_or_pack_name, idx_name);\n> }\n> \n> +\n> +static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n> +{\n> +\tstruct multi_pack_index *m = m_;\n> +\treturn strcmp(m->pack_names[*(const size_t *)a],\n> +\t\t m->pack_names[*(const size_t *)b]);\n> +}\n\nOkay, this took a second to figure out. The `pack_names_sorted` is an\narray of `size_t` indexes into `m->pack_names`. So what we get here are\nthese indices, and we can compare by using those indices via\n`m->pack_names`. Makes sense.\n\nI was wondering whether this would be easier to follow if\n`pack_names_sorted` was a simple array of unowned pointers. So it would\ncontain the same pointers as `pack_names`, but properly sorted. It would\nhave the downside of more confusing ownership semantics though.\n\nI assume we cannot live with a simple `bool sorted` field and then sort\n`pack_names` lazily?\n\nPatrick\n"},{"id":"534704","messageId":"aXhqroubXFbnBgJI@pks.im","threadId":"64585","inReplyTo":"13336e864f4ed3a6954b782f0bcc090d92ac722c.1768420450.git.me@ttaylorr.com","subject":"Re: [PATCH v2 17/18] midx: implement MIDX compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-01-27T07:35:10Z","receivedAt":"2026-01-27T07:35:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 14, 2026 at 02:55:10PM -0500, Taylor Blau wrote:\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index c0c6c1760c0..043ee8c478a 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -195,6 +204,70 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n[snip]\n> +\tif (!from_midx)\n> +\t\tdie(_(\"could not find MIDX: %s\"), argv[0]);\n> +\tif (!to_midx)\n> +\t\tdie(_(\"could not find MIDX: %s\"), argv[1]);\n> +\tif (from_midx == to_midx)\n> +\t\tdie(_(\"MIDX compaction endpoints must be unique\"));\n> +\n> +\tfor (m = from_midx; m; m = m->base_midx) {\n> +\t\tif (m == to_midx)\n> +\t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n> +\t}\n\nThese new checks all feel sensible to me.\n\n> diff --git a/midx-write.c b/midx-write.c\n> index ca2469213e6..afa077a09cc 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -1120,12 +1216,23 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n> @@ -1162,6 +1270,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n> \n> \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n> +\tctx.compact = !!(opts->flags & MIDX_WRITE_COMPACT);\n> +\n> +\tif (ctx.compact) {\n> +\t\tif (ctx.version != MIDX_VERSION_V2)\n> +\t\t\tdie(_(\"cannot perform MIDX compaction with v1 format\"));\n\nRight. So if the user has configured \"midx.version=1\" they cannot\ncompact.\n\n> @@ -1354,12 +1491,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> \t\t\tctx.large_offsets_needed = 1;\n> \t}\n> \n> -\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n> +\tif (ctx.compact) {\n> +\t\tif (ctx.version != MIDX_VERSION_V2)\n> +\t\t\tBUG(\"performing MIDX compaction with v1 MIDX\");\n\nIsn't this `BUG()` redundant with the above call to `die()`?\n\nPatrick\n"},{"id":"534731","messageId":"aXk4iIHRu3mPxY6S@nand.local","threadId":"64585","inReplyTo":"aXhqroubXFbnBgJI@pks.im","subject":"Re: [PATCH v2 17/18] midx: implement MIDX compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-01-27T22:13:28Z","receivedAt":"2026-01-27T22:13:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 27, 2026 at 08:35:10AM +0100, Patrick Steinhardt wrote:\n> > +\tif (!from_midx)\n> > +\t\tdie(_(\"could not find MIDX: %s\"), argv[0]);\n> > +\tif (!to_midx)\n> > +\t\tdie(_(\"could not find MIDX: %s\"), argv[1]);\n> > +\tif (from_midx == to_midx)\n> > +\t\tdie(_(\"MIDX compaction endpoints must be unique\"));\n> > +\n> > +\tfor (m = from_midx; m; m = m->base_midx) {\n> > +\t\tif (m == to_midx)\n> > +\t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n> > +\t}\n>\n> These new checks all feel sensible to me.\n\nThanks for taking a look and suggesting them in the first place!\n\n> > diff --git a/midx-write.c b/midx-write.c\n> > index ca2469213e6..afa077a09cc 100644\n> > --- a/midx-write.c\n> > +++ b/midx-write.c\n> > @@ -1120,12 +1216,23 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n> > @@ -1162,6 +1270,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> > \t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n> >\n> > \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n> > +\tctx.compact = !!(opts->flags & MIDX_WRITE_COMPACT);\n> > +\n> > +\tif (ctx.compact) {\n> > +\t\tif (ctx.version != MIDX_VERSION_V2)\n> > +\t\t\tdie(_(\"cannot perform MIDX compaction with v1 format\"));\n>\n> Right. So if the user has configured \"midx.version=1\" they cannot\n> compact.\n\nExactly. I think the limitation here is a fundamental one, too, since by\nits nature compaction *must* retain the pseudo-pack order concatenated\nacross each MIDX layer in the compaction range. With midx.version=1, we\ndon't have a way to express that information in a backwards-compatible\nway, so midx.version=2 here is a requirement.\n\n> > @@ -1354,12 +1491,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n> > \t\t\tctx.large_offsets_needed = 1;\n> > \t}\n> >\n> > -\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n> > +\tif (ctx.compact) {\n> > +\t\tif (ctx.version != MIDX_VERSION_V2)\n> > +\t\t\tBUG(\"performing MIDX compaction with v1 MIDX\");\n>\n> Isn't this `BUG()` redundant with the above call to `die()`?\n\nTechnically, though I put it in here as a sanity check to ensure that\nany potential regressions with the above die() don't cause us to get\ninto a worse situation that would result in bitmap corruption.\n\nThanks,\nTaylor\n"},{"id":"536553","messageId":"xmqqbjhjvysm.fsf@gitster.g","threadId":"64585","inReplyTo":"cover.1768420450.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/18] midx: incremental MIDX/bitmap layer compaction","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-02-20T22:24:25Z","receivedAt":"2026-02-20T22:24:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> [Note to the maintainer: this is based on 'master' with my\n> 'tb/midx-write-corrupt-checksum-fix' merged in and should produce zero\n> conflicts when applied on top.]\n>\n> This is a reroll of my series to implement MIDX layer compaction\n> adjusted in response to reviewer feedback.\n\nHaven't seen a lot happen since the end of last month or so.\nAnything left to do here?\n\nThanks.\n"},{"id":"536791","messageId":"20260223134935.GA271392@coredump.intra.peff.net","threadId":"64585","inReplyTo":"aXFni2tE7vn1dKFp@nand.local","subject":"Re: [PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-02-23T13:49:35Z","receivedAt":"2026-02-23T13:49:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 21, 2026 at 06:55:55PM -0500, Taylor Blau wrote:\n\n> I had considered this approach when writing, but ultimately decided\n> against it, since it felt a little clunky to have to pass a pointer in\n> to do a simple arithmetic operation. But I think your point about\n> ensuring that we actually do:\n> \n> if (unsigned_mult_overflows(a, b))\n> die(...);\n> result = a * b;\n> \n> and not \"result = a * c\" or some other expression which is not \"a * b\"\n> is a good one.\n\nIt is clunky, but it's how the compiler intrinsics work (if we ever\nchose to use them).\n\n> I dunno. The spots in this patch are the only uses of u64_mult() and\n> u64_add(), so I'm hesitant to keep a helper function around just for\n> that sole use-case. I wonder if we should do what you suggest here for\n> the much more frequently used st_add() / st_mult() / st_sub() functions?\n\nI am to blame for the crappy interface of the st_add() etc functions. It\ndid make conversion _much_ easier, because you can do stuff like:\n\n -foo = malloc(nr * size);\n +foo = malloc(st_mult(nr, size));\n\nas opposed to:\n\n size_t total;\n ...\n st_mult(&total, nr, size));\n foo = malloc(total);\n\nMy rationale was that size_t computations like this are OK to die() with\nvery little useful error reporting up the chain because:\n\n 1. The result is generally just passed along to malloc() anyway, where\n we likewise find it OK to die() without much info. So you can\n imagine a world where we just do 128-bit size computations and then\n let malloc() fail, and it would look the same. ;)\n\n 2. They don't happen in practice unless there is a bug or a malicious\n input. Which is mostly true for 64-bit systems. Maybe less so for\n 32-bit ones, where you might conceivably wish to have 4 billion of\n something.\n\nI don't think any of that holds true for u32 values like counts of\nobjects. It's conceivable that you might want to try to write a midx for\ntwo packs with 2.1 billion objects each (though from my experience, such\na repo would be unusable).\n\nAnyway. My point is mostly that I think we can design u32_add() to be\nwhat we want and not worry too much about going back to fix st_add(),\netc.\n\n-Peff\n"},{"id":"536794","messageId":"20260223140847.GB271392@coredump.intra.peff.net","threadId":"64585","inReplyTo":"xmqqbjhjvysm.fsf@gitster.g","subject":"Re: [PATCH v2 00/18] midx: incremental MIDX/bitmap layer compaction","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-02-23T14:08:47Z","receivedAt":"2026-02-23T14:08:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 20, 2026 at 02:24:25PM -0800, Junio C Hamano wrote:\n\n> Taylor Blau writes:\n> \n> > [Note to the maintainer: this is based on 'master' with my\n> > 'tb/midx-write-corrupt-checksum-fix' merged in and should produce zero\n> > conflicts when applied on top.]\n> >\n> > This is a reroll of my series to implement MIDX layer compaction\n> > adjusted in response to reviewer feedback.\n> \n> Haven't seen a lot happen since the end of last month or so.\n> Anything left to do here?\n\nSorry, I promised Taylor I would review it and I have been dragging my\nfeet.\n\nI just read through the whole series, and didn't find anything that\nhadn't already been brought up. The scariest / most complicated bits\nare:\n\n 1. Dropping the sort-order requirement for midx pack lists in patch\n 10. But I think the approach with the version bump is the maximally\n conservative one, and I feel good about that. IIRC we discussed\n previously the possibility of using new chunks to give the\n alternate ordering, but I think this keeps the complexity in the\n code to a minimum.\n\n 2. The actual compaction bits in the final two patches. I didn't see\n anything wrong here, but this is exactly the kind of spot where I\n think review fails, because you don't realize the corner case that\n you missed (speaking from experience with midx and bitmap code).\n But the nice thing here is that it should be quite unlikely to\n cause a regression if you're not using compaction. As this is\n mostly a building block for \"part 3\" that starts doing compaction\n as part of a broader repacking strategy, I think it is OK to\n consider it somewhat-experimental, build the next layer, and then\n eventually let it see more exercise in a production environment.\n\n That's how we've traditionally found those corner cases, and I\n think trying to spend more time staring at it in review is not\n likely to produce more insights.\n\nThere were a couple minor issues brought up in review, like out-dated\ncomments and the u32_add interface. So I think we might need a v3 with a\nfew touch-ups, but that's it.\n\n-Peff\n"},{"id":"536912","messageId":"aZ02Y170p/yEjohh@nand.local","threadId":"64585","inReplyTo":"20260223140847.GB271392@coredump.intra.peff.net","subject":"Re: [PATCH v2 00/18] midx: incremental MIDX/bitmap layer compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T05:25:55Z","receivedAt":"2026-02-24T05:25:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 23, 2026 at 09:08:47AM -0500, Jeff King wrote:\n> On Fri, Feb 20, 2026 at 02:24:25PM -0800, Junio C Hamano wrote:\n>\n> > Taylor Blau writes:\n> >\n> > > [Note to the maintainer: this is based on 'master' with my\n> > > 'tb/midx-write-corrupt-checksum-fix' merged in and should produce zero\n> > > conflicts when applied on top.]\n> > >\n> > > This is a reroll of my series to implement MIDX layer compaction\n> > > adjusted in response to reviewer feedback.\n> >\n> > Haven't seen a lot happen since the end of last month or so.\n> > Anything left to do here?\n>\n> Sorry, I promised Taylor I would review it and I have been dragging my\n> feet.\n>\n> I just read through the whole series, and didn't find anything that\n> hadn't already been brought up. The scariest / most complicated bits\n> are:\n>\n> 1. Dropping the sort-order requirement for midx pack lists in patch\n> 10. But I think the approach with the version bump is the maximally\n> conservative one, and I feel good about that. IIRC we discussed\n> previously the possibility of using new chunks to give the\n> alternate ordering, but I think this keeps the complexity in the\n> code to a minimum.\n\nThanks, I think the \"midx.version\" approach is the one that I'm most\ncomfortable with too.\n\nFWIW, I think that it's impossible to supplement the existing version\nwith an additional chunk to indicate the new ordering while maintaining\ncompatibility with out-of-tree MIDX readers. If others have thoughts on\nhow to do this, I'd be very curious to discuss it more, but I think what\nwe have in this series is likely the best we can do without breaking,\ne.g., libgit2.\n\n> 2. The actual compaction bits in the final two patches. I didn't see\n> anything wrong here, but this is exactly the kind of spot where I\n> think review fails, because you don't realize the corner case that\n> you missed (speaking from experience with midx and bitmap code).\n\nIndeed.\n\n> But the nice thing here is that it should be quite unlikely to\n> cause a regression if you're not using compaction. As this is\n> mostly a building block for \"part 3\" that starts doing compaction\n> as part of a broader repacking strategy, I think it is OK to\n> consider it somewhat-experimental, build the next layer, and then\n> eventually let it see more exercise in a production environment.\n>\n> That's how we've traditionally found those corner cases, and I\n> think trying to spend more time staring at it in review is not\n> likely to produce more insights.\n\nAs you note, compaction on its own is not all that interesting, since\nusers are unlikely to be managing their MIDX layers manually. Compaction\nreally is the final building block for the new repacking strategy, at\nwhich point this code becomes a lot more accessible and therefore easy\nto test and experiment with.\n\nI'm going to deploy this slowly to GitHub's infrastructure in the coming\nweeks[^1] and opting personal repositories into the new code to flush\nout any bug(s).\n\nAs I find and fix those, I'll of course share the results here.\n\n> There were a couple minor issues brought up in review, like out-dated\n> comments and the u32_add interface. So I think we might need a v3 with a\n> few touch-ups, but that's it.\n\nThanks. I have the following range-diff prepared locally, but after a\nday of traveling I am too tired to confidently declare it free of any\ntypos/thinkos ;-).\n\n--- 8< ---\n 1: 2e549ea6443 = 1: 275960bf36f midx: mark `get_midx_checksum()` arguments as const\n 2: 7255adafe70 = 2: 6b769517656 midx: rename `get_midx_checksum()` to `midx_get_checksum_hash()`\n 3: 25b628fda97 = 3: 64a56ee1506 midx: introduce `midx_get_checksum_hex()`\n 4: 2aedd72db8c = 4: f6d3ecbb283 builtin/multi-pack-index.c: make '--progress' a common option\n 5: a00598a36a3 = 5: 7d9f5766cda git-multi-pack-index(1): remove non-existent incompatibility\n 6: 92e6d868a45 = 6: 8ba8529fe32 git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'\n 7: ff599c11f68 = 7: cd2b9f6f94f t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39\n 8: 315a0ea2985 = 8: 983cff2f1b7 midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`\n 9: af174e22e1e = 9: 5f03fda72ef midx-write.c: introduce `struct write_midx_opts`\n10: 72bcd4ed6c7 ! 10: 9b02544b9ae midx: do not require packs to be sorted in lexicographic order\n @@ Documentation/gitformat-pack.adoc: HEADER:\n\n \t1-byte version number:\n -\t Git only writes or recognizes version 1.\n -+\t Git only writes version 2, but recognizes versions 1 and 2.\n ++\t Git writes the version specified by the \"midx.version\"\n ++\t configuration option, which defaults to 2. It recognizes\n ++\t both versions 1 and 2.\n\n \t1-byte Object Id Version\n \t We infer the length of object IDs (OIDs) from this value:\n11: c0c1769464b < -: ----------- git-compat-util.h: introduce `u32_add()`\n12: c11214a51f0 = 11: 06536b2081a midx-write.c: introduce `midx_pack_perm()` helper\n13: b9244a04297 = 12: ee92de1795e midx-write.c: extract `fill_pack_from_midx()`\n14: c6f8d323477 = 13: 217cd5799a5 midx-write.c: enumerate `pack_int_id` values directly\n15: e71aa575463 = 14: 1193e9b3c49 midx-write.c: factor fanout layering from `compute_sorted_entries()`\n16: dbbcb494563 = 15: 73f2151609b t/helper/test-read-midx.c: plug memory leak when selecting layer\n17: 13336e864f4 ! 16: 7344ce8e533 midx: implement MIDX compaction\n @@ Commit message\n `pack_int_id`'s (as opposed to the index at which each pack appears\n in `ctx.info`).\n\n + Note that we cannot reuse `midx_fanout_add_midx_fanout()` directly\n + here, as it unconditionally recurs through the `->base_midx`. Factor\n + out a `_1()` variant that operates on a single layer, reimplement\n + the existing function in terms of it, and use the new variant from\n + `midx_fanout_add_compact()`.\n +\n + Since we are sorting the list of objects ourselves, the order we add\n + them in does not matter.\n +\n - When writing out the new 'multi-pack-index-chain' file, discard any\n layers in the compaction range, replacing them with the newly written\n layer, instead of keeping them and placing the new layer at the end\n @@ midx-write.c: struct write_midx_context {\n \treturn ctx->pack_perm[orig_pack_int_id];\n }\n\n +@@ midx-write.c: static void midx_fanout_sort(struct midx_fanout *fanout)\n + \tQSORT(fanout->entries, fanout->nr, midx_oid_compare);\n + }\n +\n +-static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n +-\t\t\t\t\tstruct multi_pack_index *m,\n +-\t\t\t\t\tuint32_t cur_fanout,\n +-\t\t\t\t\tuint32_t preferred_pack)\n ++static void midx_fanout_add_midx_fanout_1(struct midx_fanout *fanout,\n ++\t\t\t\t\t struct multi_pack_index *m,\n ++\t\t\t\t\t uint32_t cur_fanout,\n ++\t\t\t\t\t uint32_t preferred_pack)\n + {\n + \tuint32_t start = m->num_objects_in_base, end;\n + \tuint32_t cur_object;\n +\n +-\tif (m->base_midx)\n +-\t\tmidx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,\n +-\t\t\t\t\t preferred_pack);\n +-\n + \tif (cur_fanout)\n + \t\tstart += ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n + \tend = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);\n +@@ midx-write.c: static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n + \t}\n + }\n +\n ++static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n ++\t\t\t\t\tstruct multi_pack_index *m,\n ++\t\t\t\t\tuint32_t cur_fanout,\n ++\t\t\t\t\tuint32_t preferred_pack)\n ++{\n ++\tif (m->base_midx)\n ++\t\tmidx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,\n ++\t\t\t\t\t preferred_pack);\n ++\tmidx_fanout_add_midx_fanout_1(fanout, m, cur_fanout, preferred_pack);\n ++}\n ++\n + static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n + \t\t\t\t\tstruct pack_info *info,\n + \t\t\t\t\tuint32_t cur_pack,\n @@ midx-write.c: static void midx_fanout_add(struct midx_fanout *fanout,\n \t\t\t\t\t cur_fanout);\n }\n @@ midx-write.c: static void midx_fanout_add(struct midx_fanout *fanout,\n +\tASSERT(ctx->compact);\n +\n +\twhile (m && m != ctx->compact_from->base_midx) {\n -+\t\tmidx_fanout_add_midx_fanout(fanout, m, cur_fanout,\n -+\t\t\t\t\t NO_PREFERRED_PACK);\n ++\t\tmidx_fanout_add_midx_fanout_1(fanout, m, cur_fanout,\n ++\t\t\t\t\t NO_PREFERRED_PACK);\n +\t\tm = m->base_midx;\n +\t}\n +}\n @@ midx-write.c: static int fill_packs_from_midx(struct write_midx_context *ctx)\n +\n +\tASSERT(from && to);\n +\n -+\tnr = u32_add(to->num_packs, to->num_packs_in_base);\n ++\tif (unsigned_add_overflows(to->num_packs, to->num_packs_in_base))\n ++\t\tdie(_(\"too many packs, unable to compact\"));\n ++\n ++\tnr = to->num_packs + to->num_packs_in_base;\n +\tif (nr < from->num_packs_in_base)\n +\t\tBUG(\"unexpected number of packs in base during compaction: \"\n +\t\t \"%\"PRIu32\" < %\"PRIu32, nr, from->num_packs_in_base);\n18: b599f1ad4b0 = 17: b9c1b3992c5 midx: enable reachability bitmaps during MIDX compaction\n--- >8 ---\n\nThanks,\nTaylor\n\n[^1]: Worth noting that I am at an offsite for work this week, and then\n taking a planned vacation for the following two, so I may be a little\n less available than usual.\n"},{"id":"536988","messageId":"aZ3yQ7D54jcyQhLY@nand.local","threadId":"64585","inReplyTo":"aXhqo3f-NeUcO2IM@pks.im","subject":"Re: [PATCH v2 10/18] midx: do not require packs to be sorted in lexicographic order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T18:47:31Z","receivedAt":"2026-02-24T18:47:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 27, 2026 at 08:34:59AM +0100, Patrick Steinhardt wrote:\n> On Wed, Jan 14, 2026 at 02:54:45PM -0500, Taylor Blau wrote:\n> [snip]\n> > Because this change produces MIDXs which may not be correctly read with\n> > external tools or older versions of Git. Though older versions of Git\n> > know how to gracefully degrade and ignore any MIDX(s) they consider\n> > corrupt, external tools may not be as robust. To avoid unintentionally\n> > breaking any such tools, guard this change behind a version bump in the\n> > MIDX's on-disk format.\n>\n> s/Because t/T/?\n\nNice spotting, thank you.\n\n> > @@ -105,6 +108,8 @@ struct write_midx_context {\n> >\n> > \tuint32_t preferred_pack_idx;\n> >\n> > +\tint version; /* must be MIDX_VERSION_V1 or _V2 */\n>\n> Tiny nit: this could be converted into an `enum` for implicit\n> documentation.\n\nI wrote it this way initially, but ended up dropping the type down to\n\"int\" because the enum values named \"_V1\" and \"_V2\" would have integral\nvalues of \"1\", and \"2\", respectively.\n\n> > diff --git a/midx.c b/midx.c\n> > index 19ef230d3fd..1327d0a3695 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -656,17 +658,40 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n> > \treturn strcmp(idx_or_pack_name, idx_name);\n> > }\n> >\n> > +\n> > +static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n> > +{\n> > +\tstruct multi_pack_index *m = m_;\n> > +\treturn strcmp(m->pack_names[*(const size_t *)a],\n> > +\t\t m->pack_names[*(const size_t *)b]);\n> > +}\n>\n> Okay, this took a second to figure out. The `pack_names_sorted` is an\n> array of `size_t` indexes into `m->pack_names`. So what we get here are\n> these indices, and we can compare by using those indices via\n> `m->pack_names`. Makes sense.\n>\n> I was wondering whether this would be easier to follow if\n> `pack_names_sorted` was a simple array of unowned pointers. So it would\n> contain the same pointers as `pack_names`, but properly sorted. It would\n> have the downside of more confusing ownership semantics though.\n\nExactly. For that reason, I generally prefer holding onto a permutation\narray (of indices) rather than a permuted form of the existing array.\nThe ownership of the former is unambiguous, but the latter is IMHO not.\n\n> I assume we cannot live with a simple `bool sorted` field and then sort\n> `pack_names` lazily?\n\nUnfortunately not, because the pack_names array is indexed by the\npack_int_id, which we have to preserve.\n\nThanks,\nTaylor\n"},{"id":"536989","messageId":"aZ3zq0QEd8LeBxUG@nand.local","threadId":"64585","inReplyTo":"20260223134935.GA271392@coredump.intra.peff.net","subject":"Re: [PATCH v2 11/18] git-compat-util.h: introduce `u32_add()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T18:53:31Z","receivedAt":"2026-02-24T18:53:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 23, 2026 at 08:49:35AM -0500, Jeff King wrote:\n> I am to blame for the crappy interface of the st_add() etc functions. It\n> did make conversion _much_ easier, because you can do stuff like:\n>\n> -foo = malloc(nr * size);\n> +foo = malloc(st_mult(nr, size));\n>\n> as opposed to:\n>\n> size_t total;\n> ...\n> st_mult(&total, nr, size));\n> foo = malloc(total);\n\nYeah, I agree that the latter is far less readable and more verbose than\nthe former. I think that your reasoning about why it's OK to just die()\nhere\n\n> My rationale was that size_t computations like this are OK to die() with\n> very little useful error reporting up the chain because:\n\nAll very sensible.\n\nThanks,\nTaylor\n"},{"id":"536991","messageId":"cover.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1765053054.git.me@ttaylorr.com","subject":"[PATCH v3 00/17] midx: incremental MIDX/bitmap layer compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T18:59:30Z","receivedAt":"2026-02-24T18:59:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"[Note to the maintainer: this is based on 'master', which is 7c02d39fc2e\n(The 6th batch, 2026-02-20) at the time of writing.]\n\nThis is a tiny reroll of my series to implement MIDX layer compaction\nadjusted in response to reviewer feedback.\n\nAs usual, a range-diff is included below for convenience. The main\nchanges since last time are as follows:\n\n - Some typo and documentation fixes around the new \"midx.version\"\n setting.\n\n - Bugfix when adding objects from MIDXs in the compaction range by\n fanout layer where we recur beyond the lower end of the compaction\n boundary.\n\n - Drop u32_add().\n\nI think that this should be fairly close to done, at which point I can\nsend the next series (tb/incremental-midx-part-3.3) which introduces the\nnew compaction-based repacking strategy on top of the tools introduced\nin this series.\n\nLike Peff wrote in his review of the last round, the proof here will be\nin the pudding, and I am sure that as we continue to test and use these\npatches we'll find and fix interesting bugs. But these changes are all\nfairly well isolated to the compaction logic itself, so the risk of\nregression among any existing use-cases should be low.\n\nThe original cover letter may be found here[1].\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/cover.1765053054.git.me@ttaylorr.com/\n\nTaylor Blau (17):\n midx: mark `get_midx_checksum()` arguments as const\n midx: rename `get_midx_checksum()` to `midx_get_checksum_hash()`\n midx: introduce `midx_get_checksum_hex()`\n builtin/multi-pack-index.c: make '--progress' a common option\n git-multi-pack-index(1): remove non-existent incompatibility\n git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'\n t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39\n midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`\n midx-write.c: introduce `struct write_midx_opts`\n midx: do not require packs to be sorted in lexicographic order\n midx-write.c: introduce `midx_pack_perm()` helper\n midx-write.c: extract `fill_pack_from_midx()`\n midx-write.c: enumerate `pack_int_id` values directly\n midx-write.c: factor fanout layering from `compute_sorted_entries()`\n t/helper/test-read-midx.c: plug memory leak when selecting layer\n midx: implement MIDX compaction\n midx: enable reachability bitmaps during MIDX compaction\n\n Documentation/git-multi-pack-index.adoc | 27 +-\n Documentation/gitformat-pack.adoc | 8 +-\n builtin/multi-pack-index.c | 91 +++-\n midx-write.c | 525 ++++++++++++++++++------\n midx.c | 39 +-\n midx.h | 12 +-\n pack-bitmap.c | 9 +-\n pack-revindex.c | 4 +-\n t/helper/test-read-midx.c | 21 +-\n t/meson.build | 1 +\n t/t0450/adoc-help-mismatches | 1 -\n t/t5319-multi-pack-index.sh | 18 +-\n t/t5335-compact-multi-pack-index.sh | 293 +++++++++++++\n 13 files changed, 891 insertions(+), 158 deletions(-)\n create mode 100755 t/t5335-compact-multi-pack-index.sh\n\nRange-diff against v2:\n 1: 2e549ea6443 = 1: 61045c604c0 midx: mark `get_midx_checksum()` arguments as const\n 2: 7255adafe70 = 2: 61dd8e65d1a midx: rename `get_midx_checksum()` to `midx_get_checksum_hash()`\n 3: 25b628fda97 = 3: cc5c7783062 midx: introduce `midx_get_checksum_hex()`\n 4: 2aedd72db8c = 4: 016420264d2 builtin/multi-pack-index.c: make '--progress' a common option\n 5: a00598a36a3 = 5: feb9ca5538e git-multi-pack-index(1): remove non-existent incompatibility\n 6: 92e6d868a45 = 6: 1e86068046d git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'\n 7: ff599c11f68 = 7: 06a96e16c37 t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39\n 8: 315a0ea2985 = 8: 84d9d1ef7ba midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`\n 9: af174e22e1e = 9: 132e823e758 midx-write.c: introduce `struct write_midx_opts`\n10: 72bcd4ed6c7 ! 10: 3682bfd0e08 midx: do not require packs to be sorted in lexicographic order\n @@ Commit message\n lazily instantiate a `pack_names_sorted` array on the MIDX, which will\n be used to implement the binary search over pack names.\n \n - Because this change produces MIDXs which may not be correctly read with\n - external tools or older versions of Git. Though older versions of Git\n - know how to gracefully degrade and ignore any MIDX(s) they consider\n - corrupt, external tools may not be as robust. To avoid unintentionally\n - breaking any such tools, guard this change behind a version bump in the\n - MIDX's on-disk format.\n + This change produces MIDXs which may not be correctly read with external\n + tools or older versions of Git. Though older versions of Git know how to\n + gracefully degrade and ignore any MIDX(s) they consider corrupt,\n + external tools may not be as robust. To avoid unintentionally breaking\n + any such tools, guard this change behind a version bump in the MIDX's\n + on-disk format.\n \n Signed-off-by: Taylor Blau \n \n @@ Documentation/gitformat-pack.adoc: HEADER:\n \n \t1-byte version number:\n -\t Git only writes or recognizes version 1.\n -+\t Git only writes version 2, but recognizes versions 1 and 2.\n ++\t Git writes the version specified by the \"midx.version\"\n ++\t configuration option, which defaults to 2. It recognizes\n ++\t both versions 1 and 2.\n \n \t1-byte Object Id Version\n \t We infer the length of object IDs (OIDs) from this value:\n11: c0c1769464b < -: ----------- git-compat-util.h: introduce `u32_add()`\n12: c11214a51f0 = 11: 7f99d3d728a midx-write.c: introduce `midx_pack_perm()` helper\n13: b9244a04297 = 12: f3952f7db36 midx-write.c: extract `fill_pack_from_midx()`\n14: c6f8d323477 = 13: 238bd203eaa midx-write.c: enumerate `pack_int_id` values directly\n15: e71aa575463 = 14: 3a139575b15 midx-write.c: factor fanout layering from `compute_sorted_entries()`\n16: dbbcb494563 = 15: 505c8d72aa1 t/helper/test-read-midx.c: plug memory leak when selecting layer\n17: 13336e864f4 ! 16: 19c983138c7 midx: implement MIDX compaction\n @@ Commit message\n `pack_int_id`'s (as opposed to the index at which each pack appears\n in `ctx.info`).\n \n + Note that we cannot reuse `midx_fanout_add_midx_fanout()` directly\n + here, as it unconditionally recurs through the `->base_midx`. Factor\n + out a `_1()` variant that operates on a single layer, reimplement\n + the existing function in terms of it, and use the new variant from\n + `midx_fanout_add_compact()`.\n +\n + Since we are sorting the list of objects ourselves, the order we add\n + them in does not matter.\n +\n - When writing out the new 'multi-pack-index-chain' file, discard any\n layers in the compaction range, replacing them with the newly written\n layer, instead of keeping them and placing the new layer at the end\n @@ midx-write.c: struct write_midx_context {\n \treturn ctx->pack_perm[orig_pack_int_id];\n }\n \n +@@ midx-write.c: static void midx_fanout_sort(struct midx_fanout *fanout)\n + \tQSORT(fanout->entries, fanout->nr, midx_oid_compare);\n + }\n + \n +-static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n +-\t\t\t\t\tstruct multi_pack_index *m,\n +-\t\t\t\t\tuint32_t cur_fanout,\n +-\t\t\t\t\tuint32_t preferred_pack)\n ++static void midx_fanout_add_midx_fanout_1(struct midx_fanout *fanout,\n ++\t\t\t\t\t struct multi_pack_index *m,\n ++\t\t\t\t\t uint32_t cur_fanout,\n ++\t\t\t\t\t uint32_t preferred_pack)\n + {\n + \tuint32_t start = m->num_objects_in_base, end;\n + \tuint32_t cur_object;\n + \n +-\tif (m->base_midx)\n +-\t\tmidx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,\n +-\t\t\t\t\t preferred_pack);\n +-\n + \tif (cur_fanout)\n + \t\tstart += ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n + \tend = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);\n +@@ midx-write.c: static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n + \t}\n + }\n + \n ++static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n ++\t\t\t\t\tstruct multi_pack_index *m,\n ++\t\t\t\t\tuint32_t cur_fanout,\n ++\t\t\t\t\tuint32_t preferred_pack)\n ++{\n ++\tif (m->base_midx)\n ++\t\tmidx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,\n ++\t\t\t\t\t preferred_pack);\n ++\tmidx_fanout_add_midx_fanout_1(fanout, m, cur_fanout, preferred_pack);\n ++}\n ++\n + static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n + \t\t\t\t\tstruct pack_info *info,\n + \t\t\t\t\tuint32_t cur_pack,\n @@ midx-write.c: static void midx_fanout_add(struct midx_fanout *fanout,\n \t\t\t\t\t cur_fanout);\n }\n @@ midx-write.c: static void midx_fanout_add(struct midx_fanout *fanout,\n +\tASSERT(ctx->compact);\n +\n +\twhile (m && m != ctx->compact_from->base_midx) {\n -+\t\tmidx_fanout_add_midx_fanout(fanout, m, cur_fanout,\n -+\t\t\t\t\t NO_PREFERRED_PACK);\n ++\t\tmidx_fanout_add_midx_fanout_1(fanout, m, cur_fanout,\n ++\t\t\t\t\t NO_PREFERRED_PACK);\n +\t\tm = m->base_midx;\n +\t}\n +}\n @@ midx-write.c: static int fill_packs_from_midx(struct write_midx_context *ctx)\n +\n +\tASSERT(from && to);\n +\n -+\tnr = u32_add(to->num_packs, to->num_packs_in_base);\n ++\tif (unsigned_add_overflows(to->num_packs, to->num_packs_in_base))\n ++\t\tdie(_(\"too many packs, unable to compact\"));\n ++\n ++\tnr = to->num_packs + to->num_packs_in_base;\n +\tif (nr < from->num_packs_in_base)\n +\t\tBUG(\"unexpected number of packs in base during compaction: \"\n +\t\t \"%\"PRIu32\" < %\"PRIu32, nr, from->num_packs_in_base);\n18: b599f1ad4b0 = 17: ac6221686db midx: enable reachability bitmaps during MIDX compaction\n\nbase-commit: 7c02d39fc2ed2702223c7674f73150d9a7e61ba4\n-- \n2.53.0.171.gde83996e422\n"},{"id":"536992","messageId":"61045c604c046c40434f98b81cc039be06f79f2d.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 01/17] midx: mark `get_midx_checksum()` arguments as const","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T18:59:34Z","receivedAt":"2026-02-24T18:59:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To make clear that the function `get_midx_checksum()` does not do\nanything to modify its argument, mark the MIDX pointer as const.\n\nThe following commit will rename this function altogether to make clear\nthat it returns the raw bytes of the checksum, not a hex-encoded copy of\nit.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 2 +-\n midx.h | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex a75ea99a0d4..2a6b18954c5 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -24,7 +24,7 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n-const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+const unsigned char *get_midx_checksum(const struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\n }\ndiff --git a/midx.h b/midx.h\nindex 6e54d73503d..7c7e0b59121 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -85,7 +85,7 @@ struct multi_pack_index {\n #define MIDX_EXT_BITMAP \"bitmap\"\n #define MIDX_EXT_MIDX \"midx\"\n \n-const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n+const unsigned char *get_midx_checksum(const struct multi_pack_index *m);\n void get_midx_filename(struct odb_source *source, struct strbuf *out);\n void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\n \t\t\t const unsigned char *hash, const char *ext);\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"536993","messageId":"61dd8e65d1a6d4c68648f73de1e4f991c9d6f2cf.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 02/17] midx: rename `get_midx_checksum()` to `midx_get_checksum_hash()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T18:59:39Z","receivedAt":"2026-02-24T18:59:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since 541204aabea (Documentation: document naming schema for structs and\ntheir functions, 2024-07-30), we have adopted a naming convention for\nfunctions that would prefer a name like, say, `midx_get_checksum()` over\n`get_midx_checksum()`.\n\nAdopt this convention throughout the midx.h API. Since this function\nreturns a raw (that is, non-hex encoded) hash, let's suffix the function\nwith \"_hash()\" to make this clear. As a side effect, this prepares us\nfor the subsequent change which will introduce a \"_hex()\" variant that\nencodes the checksum itself.\n\nSuggested-by: Patrick Steinhardt \nSigned-off-by: Taylor Blau \n---\n midx-write.c | 6 +++---\n midx.c | 2 +-\n midx.h | 2 +-\n pack-bitmap.c | 8 ++++----\n pack-revindex.c | 4 ++--\n t/helper/test-read-midx.c | 4 ++--\n 6 files changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 6485cb67068..73d33752ef1 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -946,7 +946,7 @@ static int link_midx_to_chain(struct multi_pack_index *m)\n \t}\n \n \tfor (i = 0; i < ARRAY_SIZE(midx_exts); i++) {\n-\t\tconst unsigned char *hash = get_midx_checksum(m);\n+\t\tconst unsigned char *hash = midx_get_checksum_hash(m);\n \n \t\tget_midx_filename_ext(m->source, &from,\n \t\t\t\t hash, midx_exts[i].non_split);\n@@ -1151,7 +1151,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\twhile (m) {\n \t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n-\t\t\t\t hash_to_hex_algop(get_midx_checksum(m),\n+\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(m),\n \t\t\t\t\t\t\tm->source->odb->repo->hash_algo));\n \t\t\t\tgoto cleanup;\n \t\t\t}\n@@ -1520,7 +1520,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n \t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n \n-\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(get_midx_checksum(m),\n+\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(midx_get_checksum_hash(m),\n \t\t\t\t\t\t\t\t r->hash_algo));\n \t\t\tm = m->base_midx;\n \t\t}\ndiff --git a/midx.c b/midx.c\nindex 2a6b18954c5..1d072bd9931 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -24,7 +24,7 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n-const unsigned char *get_midx_checksum(const struct multi_pack_index *m)\n+const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\n }\ndiff --git a/midx.h b/midx.h\nindex 7c7e0b59121..62d6105195f 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -85,7 +85,7 @@ struct multi_pack_index {\n #define MIDX_EXT_BITMAP \"bitmap\"\n #define MIDX_EXT_MIDX \"midx\"\n \n-const unsigned char *get_midx_checksum(const struct multi_pack_index *m);\n+const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m);\n void get_midx_filename(struct odb_source *source, struct strbuf *out);\n void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\n \t\t\t const unsigned char *hash, const char *ext);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 972203f12b6..6307bbdf1e1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -441,11 +441,11 @@ char *midx_bitmap_filename(struct multi_pack_index *midx)\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (midx->has_chain)\n \t\tget_split_midx_filename_ext(midx->source, &buf,\n-\t\t\t\t\t get_midx_checksum(midx),\n+\t\t\t\t\t midx_get_checksum_hash(midx),\n \t\t\t\t\t MIDX_EXT_BITMAP);\n \telse\n \t\tget_midx_filename_ext(midx->source, &buf,\n-\t\t\t\t get_midx_checksum(midx),\n+\t\t\t\t midx_get_checksum_hash(midx),\n \t\t\t\t MIDX_EXT_BITMAP);\n \n \treturn strbuf_detach(&buf, NULL);\n@@ -502,7 +502,7 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \tif (load_bitmap_header(bitmap_git) < 0)\n \t\tgoto cleanup;\n \n-\tif (!hasheq(get_midx_checksum(bitmap_git->midx), bitmap_git->checksum,\n+\tif (!hasheq(midx_get_checksum_hash(bitmap_git->midx), bitmap_git->checksum,\n \t\t bitmap_repo(bitmap_git)->hash_algo)) {\n \t\terror(_(\"checksum doesn't match in MIDX and bitmap\"));\n \t\tgoto cleanup;\n@@ -2819,7 +2819,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \t\tif (bitmap_is_midx(found))\n \t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n-\t\t\t\t hash_to_hex_algop(get_midx_checksum(found->midx),\n+\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(found->midx),\n \t\t\t\t\t\t revs->repo->hash_algo));\n \t\telse\n \t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 56cd803a679..294b802d402 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -390,11 +390,11 @@ int load_midx_revindex(struct multi_pack_index *m)\n \n \tif (m->has_chain)\n \t\tget_split_midx_filename_ext(m->source, &revindex_name,\n-\t\t\t\t\t get_midx_checksum(m),\n+\t\t\t\t\t midx_get_checksum_hash(m),\n \t\t\t\t\t MIDX_EXT_REV);\n \telse\n \t\tget_midx_filename_ext(m->source, &revindex_name,\n-\t\t\t\t get_midx_checksum(m),\n+\t\t\t\t midx_get_checksum_hash(m),\n \t\t\t\t MIDX_EXT_REV);\n \n \tret = load_revindex_from_disk(m->source->odb->repo->hash_algo,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 6de5d1665af..b8fefb1a124 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -34,7 +34,7 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\treturn 1;\n \n \tif (checksum) {\n-\t\twhile (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum))\n+\t\twhile (m && strcmp(hash_to_hex(midx_get_checksum_hash(m)), checksum))\n \t\t\tm = m->base_midx;\n \t\tif (!m)\n \t\t\treturn 1;\n@@ -94,7 +94,7 @@ static int read_midx_checksum(const char *object_dir)\n \tm = setup_midx(object_dir);\n \tif (!m)\n \t\treturn 1;\n-\tprintf(\"%s\\n\", hash_to_hex(get_midx_checksum(m)));\n+\tprintf(\"%s\\n\", hash_to_hex(midx_get_checksum_hash(m)));\n \n \tclose_midx(m);\n \treturn 0;\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"536994","messageId":"cc5c77830625997e141156fcc2bf86085254fe12.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 03/17] midx: introduce `midx_get_checksum_hex()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T18:59:44Z","receivedAt":"2026-02-24T18:59:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When trying to print out, say, the hexadecimal representation of a\nMIDX's hash, our code will do something like:\n\n hash_to_hex_algop(midx_get_checksum_hash(m),\n m->source->odb->repo->hash_algo);\n\n, which is both cumbersome and repetitive. In fact, all but a handful of\ncallers to `midx_get_checksum_hash()` do exactly the above. Reduce the\nrepetitive nature of calling `midx_get_checksum_hash()` by having it\nreturn a pointer into a static buffer containing the above result.\n\nFor the handful of callers that do need to compare the raw bytes and\ndon't want to deal with an encoded copy (e.g., because they are passing\nit to hasheq() or similar), they may still rely on\n`midx_get_checksum_hash()` which returns the raw bytes.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 6 ++----\n midx.c | 6 ++++++\n midx.h | 1 +\n pack-bitmap.c | 3 +--\n t/helper/test-read-midx.c | 4 ++--\n 5 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 73d33752ef1..13171d7e9c4 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1151,8 +1151,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\twhile (m) {\n \t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n-\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(m),\n-\t\t\t\t\t\t\tm->source->odb->repo->hash_algo));\n+\t\t\t\t midx_get_checksum_hex(m));\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \t\t\tctx.num_multi_pack_indexes_before++;\n@@ -1520,8 +1519,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n \t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n \n-\t\t\tkeep_hashes[j] = xstrdup(hash_to_hex_algop(midx_get_checksum_hash(m),\n-\t\t\t\t\t\t\t\t r->hash_algo));\n+\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n \t\t\tm = m->base_midx;\n \t\t}\n \ndiff --git a/midx.c b/midx.c\nindex 1d072bd9931..bae45892323 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -24,6 +24,12 @@ void clear_incremental_midx_files_ext(struct odb_source *source, const char *ext\n int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \t\t\t const char *idx_name);\n \n+const char *midx_get_checksum_hex(const struct multi_pack_index *m)\n+{\n+\treturn hash_to_hex_algop(midx_get_checksum_hash(m),\n+\t\t\t\t m->source->odb->repo->hash_algo);\n+}\n+\n const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - m->source->odb->repo->hash_algo->rawsz;\ndiff --git a/midx.h b/midx.h\nindex 62d6105195f..a39bcc9d03f 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -85,6 +85,7 @@ struct multi_pack_index {\n #define MIDX_EXT_BITMAP \"bitmap\"\n #define MIDX_EXT_MIDX \"midx\"\n \n+const char *midx_get_checksum_hex(const struct multi_pack_index *m) /* static buffer */;\n const unsigned char *midx_get_checksum_hash(const struct multi_pack_index *m);\n void get_midx_filename(struct odb_source *source, struct strbuf *out);\n void get_midx_filename_ext(struct odb_source *source, struct strbuf *out,\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 6307bbdf1e1..afc7fba0197 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2819,8 +2819,7 @@ void test_bitmap_walk(struct rev_info *revs)\n \n \t\tif (bitmap_is_midx(found))\n \t\t\tfprintf_ln(stderr, \"Located via MIDX '%s'.\",\n-\t\t\t\t hash_to_hex_algop(midx_get_checksum_hash(found->midx),\n-\t\t\t\t\t\t revs->repo->hash_algo));\n+\t\t\t\t midx_get_checksum_hex(found->midx));\n \t\telse\n \t\t\tfprintf_ln(stderr, \"Located via pack '%s'.\",\n \t\t\t\t hash_to_hex_algop(found->pack->hash,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex b8fefb1a124..9d42c587564 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -34,7 +34,7 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\treturn 1;\n \n \tif (checksum) {\n-\t\twhile (m && strcmp(hash_to_hex(midx_get_checksum_hash(m)), checksum))\n+\t\twhile (m && strcmp(midx_get_checksum_hex(m), checksum))\n \t\t\tm = m->base_midx;\n \t\tif (!m)\n \t\t\treturn 1;\n@@ -94,7 +94,7 @@ static int read_midx_checksum(const char *object_dir)\n \tm = setup_midx(object_dir);\n \tif (!m)\n \t\treturn 1;\n-\tprintf(\"%s\\n\", hash_to_hex(midx_get_checksum_hash(m)));\n+\tprintf(\"%s\\n\", midx_get_checksum_hex(m));\n \n \tclose_midx(m);\n \treturn 0;\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"536995","messageId":"016420264d2490d9fc5c6147ba889cdf60fd0395.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 04/17] builtin/multi-pack-index.c: make '--progress' a common option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T18:59:48Z","receivedAt":"2026-02-24T18:59:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"All multi-pack-index sub-commands (write, verify, repack, and expire)\nsupport a '--progress' command-line option, despite not listing it as\none of the common options in `common_opts`.\n\nAs a result each sub-command declares its own `OPT_BIT()` for a\n\"--progress\" command-line option. Centralize this within the\n`common_opts` to avoid re-declaring it in each sub-command.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 2 ++\n builtin/multi-pack-index.c | 10 ++--------\n 2 files changed, 4 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 2f642697e9e..a4550e28bed 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -18,6 +18,8 @@ Write or verify a multi-pack-index (MIDX) file.\n OPTIONS\n -------\n \n+The following command-line options are applicable to all sub-commands:\n+\n --object-dir=::\n \tUse given directory for the location of Git objects. We check\n \t`/packs/multi-pack-index` for the current MIDX file, and\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5f364aa816b..ca98d4c3ba3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -84,6 +84,8 @@ static struct option common_opts[] = {\n \t N_(\"directory\"),\n \t N_(\"object directory containing set of packfile and pack-index pairs\"),\n \t parse_object_dir),\n+\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"),\n+\t\tMIDX_PROGRESS),\n \tOPT_END(),\n };\n \n@@ -138,8 +140,6 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n@@ -200,8 +200,6 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv,\n {\n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_verify_options[] = {\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_END(),\n \t};\n \tstruct odb_source *source;\n@@ -231,8 +229,6 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv,\n {\n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_expire_options[] = {\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t\tN_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_END(),\n \t};\n \tstruct odb_source *source;\n@@ -264,8 +260,6 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_repack_options[] = {\n \t\tOPT_UNSIGNED(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags,\n-\t\t N_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_END(),\n \t};\n \tstruct odb_source *source;\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"536996","messageId":"feb9ca5538e9dcdc9f8bb1ec1370ae7c26cbd6fd.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 05/17] git-multi-pack-index(1): remove non-existent incompatibility","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T18:59:52Z","receivedAt":"2026-02-24T18:59:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since fcb2205b774 (midx: implement support for writing incremental MIDX\nchains, 2024-08-06), the command-line options '--incremental' and\n'--bitmap' were declared to be incompatible with one another when\nrunning 'git multi-pack-index write'.\n\nHowever, since 27afc272c49 (midx: implement writing incremental MIDX\nbitmaps, 2025-03-20), that incompatibility no longer exists, despite the\ndocumentation saying so. Correct this by removing the stale reference to\ntheir incompatibility.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex a4550e28bed..a502819fc38 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -75,7 +75,7 @@ marker).\n \t\tWrite an incremental MIDX file containing only objects\n \t\tand packs not present in an existing MIDX layer.\n \t\tMigrates non-incremental MIDXs to incremental ones when\n-\t\tnecessary. Incompatible with `--bitmap`.\n+\t\tnecessary.\n --\n \n verify::\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"536997","messageId":"1e86068046d953cfa410053e30c5aebb696f1e14.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 06/17] git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T18:59:56Z","receivedAt":"2026-02-24T19:00:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Since c39fffc1c90 (tests: start asserting that *.txt SYNOPSIS matches -h\noutput, 2022-10-13), the manual page for 'git multi-pack-index' has a\nSYNOPSIS section which differs from 'git multi-pack-index -h'.\n\nCorrect this while also documenting additional options accepted by the\n'write' sub-command.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 7 ++++++-\n builtin/multi-pack-index.c | 5 +++--\n t/t0450/adoc-help-mismatches | 1 -\n 3 files changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex a502819fc38..164cf1f2291 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -9,7 +9,12 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] [--[no-]bitmap] \n+'git multi-pack-index' [] write [--preferred-pack=]\n+\t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n+\t\t [--refs-snapshot=]\n+'git multi-pack-index' [] verify\n+'git multi-pack-index' [] expire\n+'git multi-pack-index' [] repack [--batch-size=]\n \n DESCRIPTION\n -----------\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex ca98d4c3ba3..c0c6c1760c0 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -13,8 +13,9 @@\n #include \"repository.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n-\tN_(\"git multi-pack-index [] write [--preferred-pack=]\" \\\n-\t \"[--refs-snapshot=]\")\n+\tN_(\"git multi-pack-index [] write [--preferred-pack=]\\n\" \\\n+\t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n+\t \" [--refs-snapshot=]\")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\ndiff --git a/t/t0450/adoc-help-mismatches b/t/t0450/adoc-help-mismatches\nindex 8ee2d3f7c81..e8d6c13ccd0 100644\n--- a/t/t0450/adoc-help-mismatches\n+++ b/t/t0450/adoc-help-mismatches\n@@ -33,7 +33,6 @@ merge\n merge-file\n merge-index\n merge-one-file\n-multi-pack-index\n name-rev\n notes\n push\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"536998","messageId":"06a96e16c371bcc04bf40c7c339781771e57df8d.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 07/17] t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:00Z","receivedAt":"2026-02-24T19:00:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Commit d4bf1d88b90 (multi-pack-index: verify missing pack, 2018-09-13)\nadds a new test to the MIDX test script to test how we handle missing\npacks.\n\nWhile the commit itself describes the test as \"verify missing pack[s]\",\nthe test itself is actually called \"verify packnames out of order\",\ndespite that not being what it tests.\n\nLikely this was a copy-and-paste of the test immediately above it of the\nsame name. Correct this by renaming the test to match the commit\nmessage.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex faae98c7e76..efeab4d22b7 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -517,7 +517,7 @@ test_expect_success 'verify packnames out of order' '\n \t\t\"pack names out of order\"\n '\n \n-test_expect_success 'verify packnames out of order' '\n+test_expect_success 'verify missing pack' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"a\" $objdir \\\n \t\t\"failed to load pack\"\n '\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"536999","messageId":"84d9d1ef7ba5258fe57a476621598a75e7952a5c.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 08/17] midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:05Z","receivedAt":"2026-02-24T19:00:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In midx_pack_order(), we compute for each bitmapped pack the first bit\nto correspond to an object in that pack, along with how many bits were\nassigned to object(s) in that pack.\n\nInitially, each bitmap_nr value is set to zero, and each bitmap_pos\nvalue is set to the sentinel BITMAP_POS_UNKNOWN. This is done to ensure\nthat there are no packs who have an unknown bit position but a somehow\nnon-zero number of objects (cf. `write_midx_bitmapped_packs()` in\nmidx-write.c).\n\nOnce the pack order is fully determined, midx_pack_order() sets the\nbitmap_pos field for any bitmapped packs to zero if they are still\nlisted as BITMAP_POS_UNKNOWN.\n\nHowever, we enumerate the bitmapped packs in order of `ctx->pack_perm`.\nThis is fine for existing cases, since the only time the\n`ctx->pack_perm` array holds a value outside of the addressable range of\n`ctx->info` is when there are expired packs, which only occurs via 'git\nmulti-pack-index expire', which does not support writing MIDX bitmaps.\nAs a result, the range of ctx->pack_perm covers all values in [0,\n`ctx->nr`), so enumerating in this order isn't an issue.\n\nA future change necessary for compaction will complicate this further by\nintroducing a wrapper around the `ctx->pack_perm` array, which turns the\ngiven `pack_int_id` into one that is relative to the lower end of the\ncompaction range. As a result, indexing into `ctx->pack_perm` through\nthis helper, say, with \"0\" will produce a crash when the lower end of\nthe compaction range has >0 pack(s) in its base layer, since the\nsubtraction will wrap around the 32-bit unsigned range, resulting in an\nuninitialized read.\n\nBut the process is completely unnecessary in the first place: we are\nenumerating all values of `ctx->info`, and there is no reason to process\nthem in a different order than they appear in memory. Index `ctx->info`\ndirectly to reflect that.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 13171d7e9c4..da9c5a7c295 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -637,7 +637,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \t\tpack_order[i] = data[i].nr;\n \t}\n \tfor (i = 0; i < ctx->nr; i++) {\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n+\t\tstruct pack_info *pack = &ctx->info[i];\n \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n \t\t\tpack->bitmap_pos = 0;\n \t}\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"537000","messageId":"132e823e758418f0d8b930940b2c836118692dae.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 09/17] midx-write.c: introduce `struct write_midx_opts`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:09Z","receivedAt":"2026-02-24T19:00:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the MIDX writing code, there are four functions which perform some\nsort of MIDX write operation. They are:\n\n - write_midx_file()\n - write_midx_file_only()\n - expire_midx_packs()\n - midx_repack()\n\nAll of these functions are thin wrappers over `write_midx_internal()`,\nwhich implements the bulk of these routines. As a result, the\n`write_midx_internal()` function takes six arguments.\n\nFuture commits in this series will want to add additional arguments, and\nin general this function's signature will be the union of parameters\namong *all* possible ways to write a MIDX.\n\nInstead of adding yet more arguments to this function to support MIDX\ncompaction, introduce a `struct write_midx_opts`, which has the same\nstruct members as `write_midx_internal()`'s arguments.\n\nAdding additional fields to the `write_midx_opts` struct is preferable\nto adding additional arguments to `write_midx_internal()`. This is\nbecause the callers below all zero-initialize the struct, so each time\nwe add a new piece of information, we do not have to pass the zero value\nfor it in all other call-sites that do not care about it.\n\nFor now, no functional changes are included in this patch.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 135 ++++++++++++++++++++++++++++++---------------------\n 1 file changed, 81 insertions(+), 54 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex da9c5a7c295..8a54644e427 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1078,14 +1078,20 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \treturn needed;\n }\n \n-static int write_midx_internal(struct odb_source *source,\n-\t\t\t struct string_list *packs_to_include,\n-\t\t\t struct string_list *packs_to_drop,\n-\t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot,\n-\t\t\t unsigned flags)\n+struct write_midx_opts {\n+\tstruct odb_source *source; /* non-optional */\n+\n+\tstruct string_list *packs_to_include;\n+\tstruct string_list *packs_to_drop;\n+\n+\tconst char *preferred_pack_name;\n+\tconst char *refs_snapshot;\n+\tunsigned flags;\n+};\n+\n+static int write_midx_internal(struct write_midx_opts *opts)\n {\n-\tstruct repository *r = source->odb->repo;\n+\tstruct repository *r = opts->source->odb->repo;\n \tstruct strbuf midx_name = STRBUF_INIT;\n \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t start_pack;\n@@ -1106,22 +1112,22 @@ static int write_midx_internal(struct odb_source *source,\n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n \n \tctx.repo = r;\n-\tctx.source = source;\n+\tctx.source = opts->source;\n \n-\tctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);\n+\tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n \t\t\t \"%s/pack/multi-pack-index.d/tmp_midx_XXXXXX\",\n-\t\t\t source->path);\n+\t\t\t opts->source->path);\n \telse\n-\t\tget_midx_filename(source, &midx_name);\n+\t\tget_midx_filename(opts->source, &midx_name);\n \tif (safe_create_leading_directories(r, midx_name.buf))\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t midx_name.buf);\n \n-\tif (!packs_to_include || ctx.incremental) {\n-\t\tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\tif (!opts->packs_to_include || ctx.incremental) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(opts->source);\n \t\tif (m && !midx_checksum_valid(m)) {\n \t\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n \t\t\tm = NULL;\n@@ -1136,7 +1142,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t */\n \t\t\tif (ctx.incremental)\n \t\t\t\tctx.base_midx = m;\n-\t\t\telse if (!packs_to_include)\n+\t\t\telse if (!opts->packs_to_include)\n \t\t\t\tctx.m = m;\n \t\t}\n \t}\n@@ -1149,7 +1155,7 @@ static int write_midx_internal(struct odb_source *source,\n \tif (ctx.incremental) {\n \t\tstruct multi_pack_index *m = ctx.base_midx;\n \t\twhile (m) {\n-\t\t\tif (flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n+\t\t\tif (opts->flags & MIDX_WRITE_BITMAP && load_midx_revindex(m)) {\n \t\t\t\terror(_(\"could not load reverse index for MIDX %s\"),\n \t\t\t\t midx_get_checksum_hex(m));\n \t\t\t\tgoto cleanup;\n@@ -1164,18 +1170,18 @@ static int write_midx_internal(struct odb_source *source,\n \tstart_pack = ctx.nr;\n \n \tctx.pack_paths_checked = 0;\n-\tif (flags & MIDX_PROGRESS)\n+\tif (opts->flags & MIDX_PROGRESS)\n \t\tctx.progress = start_delayed_progress(r,\n \t\t\t\t\t\t _(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n \t\tctx.progress = NULL;\n \n-\tctx.to_include = packs_to_include;\n+\tctx.to_include = opts->packs_to_include;\n \n-\tfor_each_file_in_pack_dir(source->path, add_pack_to_midx, &ctx);\n+\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (!packs_to_drop) {\n+\tif (!opts->packs_to_drop) {\n \t\t/*\n \t\t * If there is no MIDX then either it doesn't exist, or we're\n \t\t * doing a geometric repack. Try to load it from the source to\n@@ -1188,7 +1194,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tif (midx && !midx_needs_update(midx, &ctx)) {\n \t\t\tstruct bitmap_index *bitmap_git;\n \t\t\tint bitmap_exists;\n-\t\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\t\t\tint want_bitmap = opts->flags & MIDX_WRITE_BITMAP;\n \n \t\t\tbitmap_git = prepare_midx_bitmap_git(midx);\n \t\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n@@ -1200,7 +1206,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t\t * corresponding bitmap (or one wasn't requested).\n \t\t\t\t */\n \t\t\t\tif (!want_bitmap)\n-\t\t\t\t\tclear_midx_files_ext(source, \"bitmap\", NULL);\n+\t\t\t\t\tclear_midx_files_ext(ctx.source, \"bitmap\", NULL);\n \t\t\t\tresult = 0;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n@@ -1215,11 +1221,11 @@ static int write_midx_internal(struct odb_source *source,\n \t\tgoto cleanup; /* nothing to do */\n \t}\n \n-\tif (preferred_pack_name) {\n+\tif (opts->preferred_pack_name) {\n \t\tctx.preferred_pack_idx = NO_PREFERRED_PACK;\n \n \t\tfor (size_t i = 0; i < ctx.nr; i++) {\n-\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n+\t\t\tif (!cmp_idx_or_pack_name(opts->preferred_pack_name,\n \t\t\t\t\t\t ctx.info[i].pack_name)) {\n \t\t\t\tctx.preferred_pack_idx = i;\n \t\t\t\tbreak;\n@@ -1228,9 +1234,9 @@ static int write_midx_internal(struct odb_source *source,\n \n \t\tif (ctx.preferred_pack_idx == NO_PREFERRED_PACK)\n \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n-\t\t\t\tpreferred_pack_name);\n+\t\t\t\topts->preferred_pack_name);\n \t} else if (ctx.nr &&\n-\t\t (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n+\t\t (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n \t\tstruct packed_git *oldest = ctx.info[0].p;\n \t\tctx.preferred_pack_idx = 0;\n \n@@ -1241,7 +1247,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t */\n \t\topen_pack_index(oldest);\n \n-\t\tif (packs_to_drop && packs_to_drop->nr)\n+\t\tif (opts->packs_to_drop && opts->packs_to_drop->nr)\n \t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n \n \t\t/*\n@@ -1303,20 +1309,21 @@ static int write_midx_internal(struct odb_source *source,\n \n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n-\tif (packs_to_drop && packs_to_drop->nr) {\n+\tif (opts->packs_to_drop && opts->packs_to_drop->nr) {\n \t\tsize_t drop_index = 0;\n \t\tint missing_drops = 0;\n \n-\t\tfor (size_t i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\tfor (size_t i = 0;\n+\t\t i < ctx.nr && drop_index < opts->packs_to_drop->nr; i++) {\n \t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n-\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\t\t opts->packs_to_drop->items[drop_index].string);\n \n \t\t\tif (!cmp) {\n \t\t\t\tdrop_index++;\n \t\t\t\tctx.info[i].expired = 1;\n \t\t\t} else if (cmp > 0) {\n \t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n-\t\t\t\t packs_to_drop->items[drop_index].string);\n+\t\t\t\t opts->packs_to_drop->items[drop_index].string);\n \t\t\t\tdrop_index++;\n \t\t\t\tmissing_drops++;\n \t\t\t\ti--;\n@@ -1353,8 +1360,8 @@ static int write_midx_internal(struct odb_source *source,\n \t}\n \n \t/* Check that the preferred pack wasn't expired (if given). */\n-\tif (preferred_pack_name) {\n-\t\tstruct pack_info *preferred = bsearch(preferred_pack_name,\n+\tif (opts->preferred_pack_name) {\n+\t\tstruct pack_info *preferred = bsearch(opts->preferred_pack_name,\n \t\t\t\t\t\t ctx.info, ctx.nr,\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n@@ -1362,7 +1369,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n-\t\t\t\t\tpreferred_pack_name);\n+\t\t\t\t\topts->preferred_pack_name);\n \t\t}\n \t}\n \n@@ -1376,15 +1383,15 @@ static int write_midx_internal(struct odb_source *source,\n \t}\n \n \tif (!ctx.entries_nr) {\n-\t\tif (flags & MIDX_WRITE_BITMAP)\n+\t\tif (opts->flags & MIDX_WRITE_BITMAP)\n \t\t\twarning(_(\"refusing to write multi-pack .bitmap without any objects\"));\n-\t\tflags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n+\t\topts->flags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n \t}\n \n \tif (ctx.incremental) {\n \t\tstruct strbuf lock_name = STRBUF_INIT;\n \n-\t\tget_midx_chain_filename(source, &lock_name);\n+\t\tget_midx_chain_filename(opts->source, &lock_name);\n \t\thold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);\n \t\tstrbuf_release(&lock_name);\n \n@@ -1427,7 +1434,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t\tMIDX_CHUNK_LARGE_OFFSET_WIDTH),\n \t\t\twrite_midx_large_offsets);\n \n-\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n+\tif (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n \t\tctx.pack_order = midx_pack_order(&ctx);\n \t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n \t\t\t st_mult(ctx.entries_nr, sizeof(uint32_t)),\n@@ -1445,11 +1452,11 @@ static int write_midx_internal(struct odb_source *source,\n \t\t\t CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \n-\tif (flags & MIDX_WRITE_REV_INDEX &&\n+\tif (opts->flags & MIDX_WRITE_REV_INDEX &&\n \t git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n \t\twrite_midx_reverse_index(&ctx, midx_hash);\n \n-\tif (flags & MIDX_WRITE_BITMAP) {\n+\tif (opts->flags & MIDX_WRITE_BITMAP) {\n \t\tstruct packing_data pdata;\n \t\tstruct commit_stack commits = COMMIT_STACK_INIT;\n \n@@ -1458,7 +1465,7 @@ static int write_midx_internal(struct odb_source *source,\n \n \t\tprepare_midx_packing_data(&pdata, &ctx);\n \n-\t\tfind_commits_for_midx_bitmap(&commits, refs_snapshot, &ctx);\n+\t\tfind_commits_for_midx_bitmap(&commits, opts->refs_snapshot, &ctx);\n \n \t\t/*\n \t\t * The previous steps translated the information from\n@@ -1469,8 +1476,8 @@ static int write_midx_internal(struct odb_source *source,\n \t\tFREE_AND_NULL(ctx.entries);\n \t\tctx.entries_nr = 0;\n \n-\t\tif (write_midx_bitmap(&ctx, midx_hash, &pdata,\n-\t\t\t\t commits.items, commits.nr, flags) < 0) {\n+\t\tif (write_midx_bitmap(&ctx, midx_hash, &pdata, commits.items,\n+\t\t\t\t commits.nr, opts->flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tclear_packing_data(&pdata);\n \t\t\tcommit_stack_clear(&commits);\n@@ -1503,7 +1510,7 @@ static int write_midx_internal(struct odb_source *source,\n \t\tif (link_midx_to_chain(ctx.base_midx) < 0)\n \t\t\tgoto cleanup;\n \n-\t\tget_split_midx_filename_ext(source, &final_midx_name,\n+\t\tget_split_midx_filename_ext(opts->source, &final_midx_name,\n \t\t\t\t\t midx_hash, MIDX_EXT_MIDX);\n \n \t\tif (rename_tempfile(&incr, final_midx_name.buf) < 0) {\n@@ -1536,7 +1543,7 @@ static int write_midx_internal(struct odb_source *source,\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(source, keep_hashes,\n+\tclear_midx_files(opts->source, keep_hashes,\n \t\t\t ctx.num_multi_pack_indexes_before + 1,\n \t\t\t ctx.incremental);\n \tresult = 0;\n@@ -1571,9 +1578,14 @@ int write_midx_file(struct odb_source *source,\n \t\t const char *preferred_pack_name,\n \t\t const char *refs_snapshot, unsigned flags)\n {\n-\treturn write_midx_internal(source, NULL, NULL,\n-\t\t\t\t preferred_pack_name, refs_snapshot,\n-\t\t\t\t flags);\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.preferred_pack_name = preferred_pack_name,\n+\t\t.refs_snapshot = refs_snapshot,\n+\t\t.flags = flags,\n+\t};\n+\n+\treturn write_midx_internal(&opts);\n }\n \n int write_midx_file_only(struct odb_source *source,\n@@ -1581,8 +1593,15 @@ int write_midx_file_only(struct odb_source *source,\n \t\t\t const char *preferred_pack_name,\n \t\t\t const char *refs_snapshot, unsigned flags)\n {\n-\treturn write_midx_internal(source, packs_to_include, NULL,\n-\t\t\t\t preferred_pack_name, refs_snapshot, flags);\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.packs_to_include = packs_to_include,\n+\t\t.preferred_pack_name = preferred_pack_name,\n+\t\t.refs_snapshot = refs_snapshot,\n+\t\t.flags = flags,\n+\t};\n+\n+\treturn write_midx_internal(&opts);\n }\n \n int expire_midx_packs(struct odb_source *source, unsigned flags)\n@@ -1641,9 +1660,14 @@ int expire_midx_packs(struct odb_source *source, unsigned flags)\n \n \tfree(count);\n \n-\tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(source, NULL,\n-\t\t\t\t\t &packs_to_drop, NULL, NULL, flags);\n+\tif (packs_to_drop.nr) {\n+\t\tstruct write_midx_opts opts = {\n+\t\t\t.source = source,\n+\t\t\t.packs_to_drop = &packs_to_drop,\n+\t\t\t.flags = flags & MIDX_PROGRESS,\n+\t\t};\n+\t\tresult = write_midx_internal(&opts);\n+\t}\n \n \tstring_list_clear(&packs_to_drop, 0);\n \n@@ -1776,6 +1800,10 @@ int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags)\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tFILE *cmd_in;\n \tstruct multi_pack_index *m = get_multi_pack_index(source);\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.flags = flags,\n+\t};\n \n \t/*\n \t * When updating the default for these configuration\n@@ -1850,8 +1878,7 @@ int midx_repack(struct odb_source *source, size_t batch_size, unsigned flags)\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(source, NULL, NULL, NULL, NULL,\n-\t\t\t\t flags);\n+\tresult = write_midx_internal(&opts);\n \n cleanup:\n \tfree(include_pack);\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"537001","messageId":"3682bfd0e0872b9b8831d5c481db306e316c78ef.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 10/17] midx: do not require packs to be sorted in lexicographic order","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:13Z","receivedAt":"2026-02-24T19:00:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The MIDX file format currently requires that pack files be identified by\nthe lexicographic ordering of their names (that is, a pack having a\nchecksum beginning with \"abc\" would have a numeric pack_int_id which is\nsmaller than the same value for a pack beginning with \"bcd\").\n\nAs a result, it is impossible to combine adjacent MIDX layers together\nwithout permuting bits from bitmaps that are in more recent layer(s).\n\nTo see why, consider the following example:\n\n | packs | preferred pack\n --------+-------------+---------------\n MIDX #0 | { X, Y, Z } | Y\n MIDX #1 | { A, B, C } | B\n MIDX #2 | { D, E, F } | D\n\n, where MIDX #2's base MIDX is MIDX #1, and so on. Suppose that we want\nto combine MIDX layers #0 and #1, to create a new layer #0' containing\nthe packs from both layers. With the original three MIDX layers, objects\nare laid out in the bitmap in the order they appear in their source\npack, and the packs themselves are arranged according to the pseudo-pack\norder. In this case, that ordering is Y, X, Z, B, A, C.\n\nBut recall that the pseudo-pack ordering is defined by the order that\npacks appear in the MIDX, with the exception of the preferred pack,\nwhich sorts ahead of all other packs regardless of its position within\nthe MIDX. In the above example, that means that pack 'Y' could be placed\nanywhere (so long as it is designated as preferred), however, all other\npacks must be placed in the location listed above.\n\nBecause that ordering isn't sorted lexicographically, it is impossible\nto compact MIDX layers in the above configuration without permuting the\nobject-to-bit-position mapping. Changing this mapping would affect all\nbitmaps belonging to newer layers, rendering the bitmaps associated with\nMIDX #2 unreadable.\n\nOne of the goals of MIDX compaction is that we are able to shrink the\nlength of the MIDX chain *without* invalidating bitmaps that belong to\nnewer layers, and the lexicographic ordering constraint is at odds with\nthis goal.\n\nHowever, packs do not *need* to be lexicographically ordered within the\nMIDX. As far as I can gather, the only reason they are sorted lexically\nis to make it possible to perform a binary search over the pack names in\na MIDX, necessary to make `midx_contains_pack()`'s performance\nlogarithmic in the number of packs rather than linear.\n\nRelax this constraint by allowing MIDX writes to proceed with packs that\nare not arranged in lexicographic order. `midx_contains_pack()` will\nlazily instantiate a `pack_names_sorted` array on the MIDX, which will\nbe used to implement the binary search over pack names.\n\nThis change produces MIDXs which may not be correctly read with external\ntools or older versions of Git. Though older versions of Git know how to\ngracefully degrade and ignore any MIDX(s) they consider corrupt,\nexternal tools may not be as robust. To avoid unintentionally breaking\nany such tools, guard this change behind a version bump in the MIDX's\non-disk format.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/gitformat-pack.adoc | 8 ++++++--\n midx-write.c | 26 ++++++++++++++++++++++----\n midx.c | 31 ++++++++++++++++++++++++++++---\n midx.h | 4 +++-\n t/t5319-multi-pack-index.sh | 16 ++++++++++------\n 5 files changed, 69 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/gitformat-pack.adoc b/Documentation/gitformat-pack.adoc\nindex 1b4db4aa611..3416edceab8 100644\n--- a/Documentation/gitformat-pack.adoc\n+++ b/Documentation/gitformat-pack.adoc\n@@ -374,7 +374,9 @@ HEADER:\n \t The signature is: {'M', 'I', 'D', 'X'}\n \n \t1-byte version number:\n-\t Git only writes or recognizes version 1.\n+\t Git writes the version specified by the \"midx.version\"\n+\t configuration option, which defaults to 2. It recognizes\n+\t both versions 1 and 2.\n \n \t1-byte Object Id Version\n \t We infer the length of object IDs (OIDs) from this value:\n@@ -413,7 +415,9 @@ CHUNK DATA:\n \t strings. There is no extra padding between the filenames,\n \t and they are listed in lexicographic order. The chunk itself\n \t is padded at the end with between 0 and 3 NUL bytes to make the\n-\t chunk size a multiple of 4 bytes.\n+\t chunk size a multiple of 4 bytes. Version 1 MIDXs are required to\n+\t list their packs in lexicographic order, but version 2 MIDXs may\n+\t list their packs in any arbitrary order.\n \n \tBitmapped Packfiles (ID: {'B', 'T', 'M', 'P'})\n \t Stores a table of two 4-byte unsigned integers in network order.\ndiff --git a/midx-write.c b/midx-write.c\nindex 8a54644e427..5c8700065a1 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -36,10 +36,13 @@ extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \n static size_t write_midx_header(const struct git_hash_algo *hash_algo,\n \t\t\t\tstruct hashfile *f, unsigned char num_chunks,\n-\t\t\t\tuint32_t num_packs)\n+\t\t\t\tuint32_t num_packs, int version)\n {\n+\tif (version != MIDX_VERSION_V1 && version != MIDX_VERSION_V2)\n+\t\tBUG(\"unexpected MIDX version: %d\", version);\n+\n \thashwrite_be32(f, MIDX_SIGNATURE);\n-\thashwrite_u8(f, MIDX_VERSION);\n+\thashwrite_u8(f, version);\n \thashwrite_u8(f, oid_version(hash_algo));\n \thashwrite_u8(f, num_chunks);\n \thashwrite_u8(f, 0); /* unused */\n@@ -105,6 +108,8 @@ struct write_midx_context {\n \n \tuint32_t preferred_pack_idx;\n \n+\tint version; /* must be MIDX_VERSION_V1 or _V2 */\n+\n \tint incremental;\n \tuint32_t num_multi_pack_indexes_before;\n \n@@ -410,7 +415,9 @@ static int write_midx_pack_names(struct hashfile *f, void *data)\n \t\tif (ctx->info[i].expired)\n \t\t\tcontinue;\n \n-\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n+\t\tif (ctx->version == MIDX_VERSION_V1 &&\n+\t\t i && strcmp(ctx->info[i].pack_name,\n+\t\t\t\tctx->info[i - 1].pack_name) <= 0)\n \t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n \t\t\t ctx->info[i - 1].pack_name,\n \t\t\t ctx->info[i].pack_name);\n@@ -1025,6 +1032,12 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \tif (!midx_checksum_valid(midx))\n \t\tgoto out;\n \n+\t/*\n+\t * If the version differs, we need to update.\n+\t */\n+\tif (midx->version != ctx->version)\n+\t\tgoto out;\n+\n \t/*\n \t * Ignore incremental updates for now. The assumption is that any\n \t * incremental update would be either empty (in which case we will bail\n@@ -1100,6 +1113,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tstruct tempfile *incr;\n \tstruct write_midx_context ctx = {\n \t\t.preferred_pack_idx = NO_PREFERRED_PACK,\n+\t\t.version = MIDX_VERSION_V2,\n \t };\n \tstruct multi_pack_index *midx_to_free = NULL;\n \tint bitmapped_packs_concat_len = 0;\n@@ -1114,6 +1128,10 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tctx.repo = r;\n \tctx.source = opts->source;\n \n+\trepo_config_get_int(ctx.repo, \"midx.version\", &ctx.version);\n+\tif (ctx.version != MIDX_VERSION_V1 && ctx.version != MIDX_VERSION_V2)\n+\t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n+\n \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n \n \tif (ctx.incremental)\n@@ -1445,7 +1463,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t}\n \n \twrite_midx_header(r->hash_algo, f, get_num_chunks(cf),\n-\t\t\t ctx.nr - dropped_packs);\n+\t\t\t ctx.nr - dropped_packs, ctx.version);\n \twrite_chunkfile(cf, &ctx);\n \n \tfinalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,\ndiff --git a/midx.c b/midx.c\nindex bae45892323..c1b9658240d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -149,7 +149,7 @@ static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *sou\n \t\t m->signature, MIDX_SIGNATURE);\n \n \tm->version = m->data[MIDX_BYTE_FILE_VERSION];\n-\tif (m->version != MIDX_VERSION)\n+\tif (m->version != MIDX_VERSION_V1 && m->version != MIDX_VERSION_V2)\n \t\tdie(_(\"multi-pack-index version %d not recognized\"),\n \t\t m->version);\n \n@@ -210,7 +210,8 @@ static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *sou\n \t\t\tdie(_(\"multi-pack-index pack-name chunk is too short\"));\n \t\tcur_pack_name = end + 1;\n \n-\t\tif (i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0)\n+\t\tif (m->version == MIDX_VERSION_V1 &&\n+\t\t i && strcmp(m->pack_names[i], m->pack_names[i - 1]) <= 0)\n \t\t\tdie(_(\"multi-pack-index pack names out of order: '%s' before '%s'\"),\n \t\t\t m->pack_names[i - 1],\n \t\t\t m->pack_names[i]);\n@@ -411,6 +412,7 @@ void close_midx(struct multi_pack_index *m)\n \t}\n \tFREE_AND_NULL(m->packs);\n \tFREE_AND_NULL(m->pack_names);\n+\tFREE_AND_NULL(m->pack_names_sorted);\n \tfree(m);\n }\n \n@@ -655,17 +657,40 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n+\n+static int midx_pack_names_cmp(const void *a, const void *b, void *m_)\n+{\n+\tstruct multi_pack_index *m = m_;\n+\treturn strcmp(m->pack_names[*(const size_t *)a],\n+\t\t m->pack_names[*(const size_t *)b]);\n+}\n+\n static int midx_contains_pack_1(struct multi_pack_index *m,\n \t\t\t\tconst char *idx_or_pack_name)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n+\tif (m->version == MIDX_VERSION_V2 && !m->pack_names_sorted) {\n+\t\tuint32_t i;\n+\n+\t\tALLOC_ARRAY(m->pack_names_sorted, m->num_packs);\n+\n+\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\t\tm->pack_names_sorted[i] = i;\n+\n+\t\tQSORT_S(m->pack_names_sorted, m->num_packs, midx_pack_names_cmp,\n+\t\t\tm);\n+\t}\n+\n \twhile (first < last) {\n \t\tuint32_t mid = first + (last - first) / 2;\n \t\tconst char *current;\n \t\tint cmp;\n \n-\t\tcurrent = m->pack_names[mid];\n+\t\tif (m->pack_names_sorted)\n+\t\t\tcurrent = m->pack_names[m->pack_names_sorted[mid]];\n+\t\telse\n+\t\t\tcurrent = m->pack_names[mid];\n \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n \t\tif (!cmp)\n \t\t\treturn 1;\ndiff --git a/midx.h b/midx.h\nindex a39bcc9d03f..aa99a6cb215 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -11,7 +11,8 @@ struct git_hash_algo;\n struct odb_source;\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n-#define MIDX_VERSION 1\n+#define MIDX_VERSION_V1 1\n+#define MIDX_VERSION_V2 2\n #define MIDX_BYTE_FILE_VERSION 4\n #define MIDX_BYTE_HASH_VERSION 5\n #define MIDX_BYTE_NUM_CHUNKS 6\n@@ -71,6 +72,7 @@ struct multi_pack_index {\n \tuint32_t num_packs_in_base;\n \n \tconst char **pack_names;\n+\tsize_t *pack_names_sorted;\n \tstruct packed_git **packs;\n };\n \ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex efeab4d22b7..250d21dbd67 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -21,7 +21,7 @@ midx_read_expect () {\n \tEXTRA_CHUNKS=\"$5\"\n \t{\n \t\tcat <<-EOF &&\n-\t\theader: 4d494458 1 $HASH_LEN $NUM_CHUNKS $NUM_PACKS\n+\t\theader: 4d494458 2 $HASH_LEN $NUM_CHUNKS $NUM_PACKS\n \t\tchunks: pack-names oid-fanout oid-lookup object-offsets$EXTRA_CHUNKS\n \t\tnum_objects: $NUM_OBJECTS\n \t\tpacks:\n@@ -512,11 +512,6 @@ test_expect_success 'verify invalid chunk offset' '\n \t\t\"improper chunk offset(s)\"\n '\n \n-test_expect_success 'verify packnames out of order' '\n-\tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"z\" $objdir \\\n-\t\t\"pack names out of order\"\n-'\n-\n test_expect_success 'verify missing pack' '\n \tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"a\" $objdir \\\n \t\t\"failed to load pack\"\n@@ -578,6 +573,15 @@ test_expect_success 'verify incorrect checksum' '\n \t\t$objdir \"incorrect checksum\"\n '\n \n+test_expect_success 'setup for v1-specific fsck tests' '\n+\tgit -c midx.version=1 multi-pack-index write\n+'\n+\n+test_expect_success 'verify packnames out of order (v1)' '\n+\tcorrupt_midx_and_verify $MIDX_BYTE_PACKNAME_ORDER \"z\" $objdir \\\n+\t\t\"pack names out of order\"\n+'\n+\n test_expect_success 'repack progress off for redirected stderr' '\n \tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir repack 2>err &&\n \ttest_line_count = 0 err\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"537002","messageId":"7f99d3d728a41b7df37009def10dcd1a5bdfc9b3.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 11/17] midx-write.c: introduce `midx_pack_perm()` helper","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:17Z","receivedAt":"2026-02-24T19:00:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `ctx->pack_perm` array can be considered as a permutation between\nthe original `pack_int_id` of some given pack to its position in the\n`ctx->info` array containing all packs.\n\nToday we can always index into this array with any known `pack_int_id`,\nsince there is never a `pack_int_id` which is greater than or equal to\nthe value `ctx->nr`.\n\nThat is not necessarily the case with MIDX compaction. For example,\nsuppose we have a MIDX chain with three layers, each containing three\npacks. The base of the MIDX chain will have packs with IDs 0, 1, and 2,\nthe next layer 3, 4, and 5, and so on. If we are compacting the topmost\ntwo layers, we'll have input `pack_int_id` values between [3, 8], but\n`ctx->nr` will only be 6.\n\nIn that example, if we want to know where the pack whose original\n`pack_int_id` value was, say, 7, we would compute `ctx->pack_perm[7]`,\nleading to an uninitialized read, since there are only 6 entries\nallocated in that array.\n\nTo address this, there are a couple of options:\n\n - We could allocate enough entries in `ctx->pack_perm` to accommodate\n the largest `orig_pack_int_id` value.\n\n - Or, we could internally shift the input values by the number of packs\n in the base layer of the lower end of the MIDX compaction range.\n\nThis patch prepare us to take the latter approach, since it does not\nallocate more memory than strictly necessary. (In our above example, the\nbase of the lower end of the compaction range is the first MIDX layer\n(having three packs), so we would end up indexing `ctx->pack_perm[7-3]`,\nwhich is a valid read.)\n\nNote that this patch does not actually implement that approach yet, but\nmerely performs a behavior-preserving refactoring which will make the\nchange easier to carry out in the future.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 5c8700065a1..9d345fb4737 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -119,6 +119,12 @@ struct write_midx_context {\n \tstruct odb_source *source;\n };\n \n+static uint32_t midx_pack_perm(struct write_midx_context *ctx,\n+\t\t\t uint32_t orig_pack_int_id)\n+{\n+\treturn ctx->pack_perm[orig_pack_int_id];\n+}\n+\n static int should_include_pack(const struct write_midx_context *ctx,\n \t\t\t const char *file_name)\n {\n@@ -521,12 +527,12 @@ static int write_midx_object_offsets(struct hashfile *f,\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *obj = list++;\n \n-\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\tif (midx_pack_perm(ctx, obj->pack_int_id) == PACK_EXPIRED)\n \t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n \t\t\t oid_to_hex(&obj->oid),\n \t\t\t obj->pack_int_id);\n \n-\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n+\t\thashwrite_be32(f, midx_pack_perm(ctx, obj->pack_int_id));\n \n \t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n \t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n@@ -627,7 +633,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[i];\n \t\tdata[i].nr = i;\n-\t\tdata[i].pack = ctx->pack_perm[e->pack_int_id];\n+\t\tdata[i].pack = midx_pack_perm(ctx, e->pack_int_id);\n \t\tif (!e->preferred)\n \t\t\tdata[i].pack |= (1U << 31);\n \t\tdata[i].offset = e->offset;\n@@ -637,7 +643,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \n \tfor (i = 0; i < ctx->entries_nr; i++) {\n \t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n+\t\tstruct pack_info *pack = &ctx->info[midx_pack_perm(ctx, e->pack_int_id)];\n \t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n \t\t\tpack->bitmap_pos = i + base_objects;\n \t\tpack->bitmap_nr++;\n@@ -698,7 +704,7 @@ static void prepare_midx_packing_data(struct packing_data *pdata,\n \t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n \n \t\toe_set_in_pack(pdata, to,\n-\t\t\t ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t\t\t ctx->info[midx_pack_perm(ctx, from->pack_int_id)].p);\n \t}\n \n \ttrace2_region_leave(\"midx\", \"prepare_midx_packing_data\", ctx->repo);\n@@ -1384,7 +1390,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t\t\t\t sizeof(*ctx.info),\n \t\t\t\t\t\t idx_or_pack_name_cmp);\n \t\tif (preferred) {\n-\t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n+\t\t\tuint32_t perm = midx_pack_perm(&ctx, preferred->orig_pack_int_id);\n \t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n \t\t\t\t\topts->preferred_pack_name);\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"537003","messageId":"238bd203eaa7629baf95ea5c94593561848dc7bb.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 13/17] midx-write.c: enumerate `pack_int_id` values directly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:26Z","receivedAt":"2026-02-24T19:00:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Our `midx-write.c::fill_packs_from_midx()` function currently enumerates\nthe range [0, m->num_packs), and then shifts its index variable up by\n`m->num_packs_in_base` to produce a valid `pack_int_id`.\n\nInstead, directly enumerate the range:\n\n [m->num_packs_in_base, m->num_packs_in_base + m->num_packs)\n\n, which are the original pack_int_ids themselves as opposed to the\nindexes of those packs relative to the MIDX layer they are contained\nwithin.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex c54113cdc84..80334914d3e 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -935,11 +935,11 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n \tfor (m = ctx->m; m; m = m->base_midx) {\n \t\tuint32_t i;\n \n-\t\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tfor (i = m->num_packs_in_base;\n+\t\t i < m->num_packs_in_base + m->num_packs; i++) {\n \t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\t\tif (fill_pack_from_midx(&ctx->info[ctx->nr], m,\n-\t\t\t\t\t\tm->num_packs_in_base + i) < 0)\n+\t\t\tif (fill_pack_from_midx(&ctx->info[ctx->nr], m, i) < 0)\n \t\t\t\treturn -1;\n \n \t\t\tctx->nr++;\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"537004","messageId":"3a139575b1576a9ce4d4a2e1d928a60338636f1b.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 14/17] midx-write.c: factor fanout layering from `compute_sorted_entries()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:30Z","receivedAt":"2026-02-24T19:00:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When computing the set of objects to appear in a MIDX, we use\ncompute_sorted_entries(), which handles objects from various existing\nsources one fanout layer at a time.\n\nThe process for computing this set is slightly different during MIDX\ncompaction, so factor out the existing functionality into its own\nroutine to prevent `compute_sorted_entries()` from becoming too\ndifficult to read.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 42 +++++++++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 17 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 80334914d3e..ca2469213e6 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -328,6 +328,30 @@ static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n \t}\n }\n \n+static void midx_fanout_add(struct midx_fanout *fanout,\n+\t\t\t struct write_midx_context *ctx,\n+\t\t\t uint32_t start_pack,\n+\t\t\t uint32_t cur_fanout)\n+{\n+\tuint32_t cur_pack;\n+\n+\tif (ctx->m && !ctx->incremental)\n+\t\tmidx_fanout_add_midx_fanout(fanout, ctx->m, cur_fanout,\n+\t\t\t\t\t ctx->preferred_pack_idx);\n+\n+\tfor (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) {\n+\t\tint preferred = cur_pack == ctx->preferred_pack_idx;\n+\t\tmidx_fanout_add_pack_fanout(fanout, ctx->info, cur_pack,\n+\t\t\t\t\t preferred, cur_fanout);\n+\t}\n+\n+\tif (ctx->preferred_pack_idx != NO_PREFERRED_PACK &&\n+\t ctx->preferred_pack_idx < start_pack)\n+\t\tmidx_fanout_add_pack_fanout(fanout, ctx->info,\n+\t\t\t\t\t ctx->preferred_pack_idx, 1,\n+\t\t\t\t\t cur_fanout);\n+}\n+\n /*\n * It is possible to artificially get into a state where there are many\n * duplicate copies of objects. That can create high memory pressure if\n@@ -364,23 +388,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n \n-\t\tif (ctx->m && !ctx->incremental)\n-\t\t\tmidx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout,\n-\t\t\t\t\t\t ctx->preferred_pack_idx);\n-\n-\t\tfor (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) {\n-\t\t\tint preferred = cur_pack == ctx->preferred_pack_idx;\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout,\n-\t\t\t\t\t\t ctx->info, cur_pack,\n-\t\t\t\t\t\t preferred, cur_fanout);\n-\t\t}\n-\n-\t\tif (ctx->preferred_pack_idx != NO_PREFERRED_PACK &&\n-\t\t ctx->preferred_pack_idx < start_pack)\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout, ctx->info,\n-\t\t\t\t\t\t ctx->preferred_pack_idx, 1,\n-\t\t\t\t\t\t cur_fanout);\n-\n+\t\tmidx_fanout_add(&fanout, ctx, start_pack, cur_fanout);\n \t\tmidx_fanout_sort(&fanout);\n \n \t\t/*\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"537005","messageId":"505c8d72aa1befb07dda537945152b07b812dc25.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 15/17] t/helper/test-read-midx.c: plug memory leak when selecting layer","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:36Z","receivedAt":"2026-02-24T19:00:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Though our 'read-midx' test tool is capable of printing information\nabout a single MIDX layer identified by its checksum, no caller in our\ntest suite exercises this path.\n\nUnfortunately, there is a memory leak lurking in this (currently) unused\npath that would otherwise be exposed by the following commit.\n\nThis occurs when providing a MIDX layer checksum other than the tip. As\nwe walk over the MIDX chain trying to find the matching layer, we drop\nour reference to the top-most MIDX layer. Thus, our call to\n'close_midx()' later on leaks memory between the top-most MIDX layer and\nthe MIDX layer immediately following the specified one.\n\nPlug this leak by holding a reference to the tip of the MIDX chain, and\nensure that we call `close_midx()` before terminating the test tool.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 17 +++++++++++------\n 1 file changed, 11 insertions(+), 6 deletions(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 9d42c587564..388d29e2b53 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -26,9 +26,10 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\t\t int show_objects)\n {\n \tuint32_t i;\n-\tstruct multi_pack_index *m;\n+\tstruct multi_pack_index *m, *tip;\n+\tint ret = 0;\n \n-\tm = setup_midx(object_dir);\n+\tm = tip = setup_midx(object_dir);\n \n \tif (!m)\n \t\treturn 1;\n@@ -36,8 +37,11 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \tif (checksum) {\n \t\twhile (m && strcmp(midx_get_checksum_hex(m), checksum))\n \t\t\tm = m->base_midx;\n-\t\tif (!m)\n-\t\t\treturn 1;\n+\t\tif (!m) {\n+\t\t\tret = error(_(\"could not find MIDX with checksum %s\"),\n+\t\t\t\t checksum);\n+\t\t\tgoto out;\n+\t\t}\n \t}\n \n \tprintf(\"header: %08x %d %d %d %d\\n\",\n@@ -82,9 +86,10 @@ static int read_midx_file(const char *object_dir, const char *checksum,\n \t\t}\n \t}\n \n-\tclose_midx(m);\n+out:\n+\tclose_midx(tip);\n \n-\treturn 0;\n+\treturn ret;\n }\n \n static int read_midx_checksum(const char *object_dir)\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"537006","messageId":"19c983138c75728fcbb7b723feac1f604902794f.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 16/17] midx: implement MIDX compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:41Z","receivedAt":"2026-02-24T19:00:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When managing a MIDX chain with many layers, it is convenient to combine\na sequence of adjacent layers into a single layer to prevent the chain\nfrom growing too long.\n\nWhile it is conceptually possible to \"compact\" a sequence of MIDX layers\ntogether by running \"git multi-pack-index write --stdin-packs\", there\nare a few drawbacks that make this less than desirable:\n\n - Preserving the MIDX chain is impossible, since there is no way to\n write a MIDX layer that contains objects or packs found in an earlier\n MIDX layer already part of the chain. So callers would have to write\n an entirely new (non-incremental) MIDX containing only the compacted\n layers, discarding all other objects/packs from the MIDX.\n\n - There is (currently) no way to write a MIDX layer outside of the MIDX\n chain to work around the above, such that the MIDX chain could be\n reassembled substituting the compacted layers with the MIDX that was\n written.\n\n - The `--stdin-packs` command-line option does not allow us to specify\n the order of packs as they appear in the MIDX. Therefore, even if\n there were workarounds for the previous two challenges, any bitmaps\n belonging to layers which come after the compacted layer(s) would no\n longer be valid.\n\nThis commit introduces a way to compact a sequence of adjacent MIDX\nlayers into a single layer while preserving the MIDX chain, as well as\nany bitmap(s) in layers which are newer than the compacted ones.\n\nImplementing MIDX compaction does not require a significant number of\nchanges to how MIDX layers are written. The main changes are as follows:\n\n - Instead of calling `fill_packs_from_midx()`, we call a new function\n `fill_packs_from_midx_range()`, which walks backwards along the\n portion of the MIDX chain which we are compacting, and adds packs one\n layer a time.\n\n In order to preserve the pseudo-pack order, the concatenated pack\n order is preserved, with the exception of preferred packs which are\n always added first.\n\n - After adding entries from the set of packs in the compaction range,\n `compute_sorted_entries()` must adjust the `pack_int_id`'s for all\n objects added in each fanout layer to match their original\n `pack_int_id`'s (as opposed to the index at which each pack appears\n in `ctx.info`).\n\n Note that we cannot reuse `midx_fanout_add_midx_fanout()` directly\n here, as it unconditionally recurs through the `->base_midx`. Factor\n out a `_1()` variant that operates on a single layer, reimplement\n the existing function in terms of it, and use the new variant from\n `midx_fanout_add_compact()`.\n\n Since we are sorting the list of objects ourselves, the order we add\n them in does not matter.\n\n - When writing out the new 'multi-pack-index-chain' file, discard any\n layers in the compaction range, replacing them with the newly written\n layer, instead of keeping them and placing the new layer at the end\n of the chain.\n\nThis ends up being sufficient to implement MIDX compaction in such a way\nthat preserves bitmaps corresponding to more recent layers in the MIDX\nchain.\n\nThe tests for MIDX compaction are so far fairly spartan, since the main\ninteresting behavior here is ensuring that the right packs/objects are\nselected from each layer, and that the pack order is preserved despite\nwhether or not they are sorted in lexicographic order in the original\nMIDX chain.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 13 ++\n builtin/multi-pack-index.c | 74 +++++++\n midx-write.c | 277 +++++++++++++++++++++---\n midx.h | 5 +\n t/meson.build | 1 +\n t/t5335-compact-multi-pack-index.sh | 175 +++++++++++++++\n 6 files changed, 518 insertions(+), 27 deletions(-)\n create mode 100755 t/t5335-compact-multi-pack-index.sh\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 164cf1f2291..883a0529741 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -12,6 +12,8 @@ SYNOPSIS\n 'git multi-pack-index' [] write [--preferred-pack=]\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=]\n+'git multi-pack-index' [] compact [--[no-]incremental]\n+\t\t \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -83,6 +85,17 @@ marker).\n \t\tnecessary.\n --\n \n+compact::\n+\tWrite a new MIDX layer containing only objects and packs present\n+\tin the range `` to ``, where both arguments are\n+\tchecksums of existing layers in the MIDX chain.\n++\n+--\n+\t--incremental::\n+\t\tWrite the result to a MIDX chain instead of writing a\n+\t\tstand-alone MIDX.\n+--\n+\n verify::\n \tVerify the contents of the MIDX file.\n \ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex c0c6c1760c0..043ee8c478a 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -17,6 +17,10 @@\n \t \" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\\n\" \\\n \t \" [--refs-snapshot=]\")\n \n+#define BUILTIN_MIDX_COMPACT_USAGE \\\n+\tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n+\t \" \")\n+\n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n \n@@ -30,6 +34,10 @@ static char const * const builtin_multi_pack_index_write_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n \tNULL\n };\n+static char const * const builtin_multi_pack_index_compact_usage[] = {\n+\tBUILTIN_MIDX_COMPACT_USAGE,\n+\tNULL\n+};\n static char const * const builtin_multi_pack_index_verify_usage[] = {\n \tBUILTIN_MIDX_VERIFY_USAGE,\n \tNULL\n@@ -44,6 +52,7 @@ static char const * const builtin_multi_pack_index_repack_usage[] = {\n };\n static char const * const builtin_multi_pack_index_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n+\tBUILTIN_MIDX_COMPACT_USAGE,\n \tBUILTIN_MIDX_VERIFY_USAGE,\n \tBUILTIN_MIDX_EXPIRE_USAGE,\n \tBUILTIN_MIDX_REPACK_USAGE,\n@@ -195,6 +204,70 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \treturn ret;\n }\n \n+static int cmd_multi_pack_index_compact(int argc, const char **argv,\n+\t\t\t\t\tconst char *prefix,\n+\t\t\t\t\tstruct repository *repo)\n+{\n+\tstruct multi_pack_index *m, *cur;\n+\tstruct multi_pack_index *from_midx = NULL;\n+\tstruct multi_pack_index *to_midx = NULL;\n+\tstruct odb_source *source;\n+\tint ret;\n+\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_BIT(0, \"incremental\", &opts.flags,\n+\t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n+\t\tOPT_END(),\n+\t};\n+\n+\trepo_config(repo, git_multi_pack_index_write_config, NULL);\n+\n+\toptions = add_common_options(builtin_multi_pack_index_compact_options);\n+\n+\ttrace2_cmd_mode(argv[0]);\n+\n+\tif (isatty(2))\n+\t\topts.flags |= MIDX_PROGRESS;\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t options, builtin_multi_pack_index_compact_usage,\n+\t\t\t 0);\n+\n+\tif (argc != 2)\n+\t\tusage_with_options(builtin_multi_pack_index_compact_usage,\n+\t\t\t\t options);\n+\tsource = handle_object_dir_option(the_repository);\n+\n+\tFREE_AND_NULL(options);\n+\n+\tm = get_multi_pack_index(source);\n+\n+\tfor (cur = m; cur && !(from_midx && to_midx); cur = cur->base_midx) {\n+\t\tconst char *midx_csum = midx_get_checksum_hex(cur);\n+\n+\t\tif (!from_midx && !strcmp(midx_csum, argv[0]))\n+\t\t\tfrom_midx = cur;\n+\t\tif (!to_midx && !strcmp(midx_csum, argv[1]))\n+\t\t\tto_midx = cur;\n+\t}\n+\n+\tif (!from_midx)\n+\t\tdie(_(\"could not find MIDX: %s\"), argv[0]);\n+\tif (!to_midx)\n+\t\tdie(_(\"could not find MIDX: %s\"), argv[1]);\n+\tif (from_midx == to_midx)\n+\t\tdie(_(\"MIDX compaction endpoints must be unique\"));\n+\n+\tfor (m = from_midx; m; m = m->base_midx) {\n+\t\tif (m == to_midx)\n+\t\t\tdie(_(\"MIDX %s must be an ancestor of %s\"), argv[0], argv[1]);\n+\t}\n+\n+\tret = write_midx_file_compact(source, from_midx, to_midx, opts.flags);\n+\n+\treturn ret;\n+}\n+\n static int cmd_multi_pack_index_verify(int argc, const char **argv,\n \t\t\t\t const char *prefix,\n \t\t\t\t struct repository *repo UNUSED)\n@@ -295,6 +368,7 @@ int cmd_multi_pack_index(int argc,\n \tstruct option builtin_multi_pack_index_options[] = {\n \t\tOPT_SUBCOMMAND(\"repack\", &fn, cmd_multi_pack_index_repack),\n \t\tOPT_SUBCOMMAND(\"write\", &fn, cmd_multi_pack_index_write),\n+\t\tOPT_SUBCOMMAND(\"compact\", &fn, cmd_multi_pack_index_compact),\n \t\tOPT_SUBCOMMAND(\"verify\", &fn, cmd_multi_pack_index_verify),\n \t\tOPT_SUBCOMMAND(\"expire\", &fn, cmd_multi_pack_index_expire),\n \t\tOPT_END(),\ndiff --git a/midx-write.c b/midx-write.c\nindex ca2469213e6..bf53ad1c4b7 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -113,6 +113,10 @@ struct write_midx_context {\n \tint incremental;\n \tuint32_t num_multi_pack_indexes_before;\n \n+\tstruct multi_pack_index *compact_from;\n+\tstruct multi_pack_index *compact_to;\n+\tint compact;\n+\n \tstruct string_list *to_include;\n \n \tstruct repository *repo;\n@@ -122,6 +126,8 @@ struct write_midx_context {\n static uint32_t midx_pack_perm(struct write_midx_context *ctx,\n \t\t\t uint32_t orig_pack_int_id)\n {\n+\tif (ctx->compact)\n+\t\torig_pack_int_id -= ctx->compact_from->num_packs_in_base;\n \treturn ctx->pack_perm[orig_pack_int_id];\n }\n \n@@ -268,18 +274,14 @@ static void midx_fanout_sort(struct midx_fanout *fanout)\n \tQSORT(fanout->entries, fanout->nr, midx_oid_compare);\n }\n \n-static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n-\t\t\t\t\tstruct multi_pack_index *m,\n-\t\t\t\t\tuint32_t cur_fanout,\n-\t\t\t\t\tuint32_t preferred_pack)\n+static void midx_fanout_add_midx_fanout_1(struct midx_fanout *fanout,\n+\t\t\t\t\t struct multi_pack_index *m,\n+\t\t\t\t\t uint32_t cur_fanout,\n+\t\t\t\t\t uint32_t preferred_pack)\n {\n \tuint32_t start = m->num_objects_in_base, end;\n \tuint32_t cur_object;\n \n-\tif (m->base_midx)\n-\t\tmidx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,\n-\t\t\t\t\t preferred_pack);\n-\n \tif (cur_fanout)\n \t\tstart += ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n \tend = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);\n@@ -303,6 +305,17 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n \t}\n }\n \n+static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n+\t\t\t\t\tstruct multi_pack_index *m,\n+\t\t\t\t\tuint32_t cur_fanout,\n+\t\t\t\t\tuint32_t preferred_pack)\n+{\n+\tif (m->base_midx)\n+\t\tmidx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,\n+\t\t\t\t\t preferred_pack);\n+\tmidx_fanout_add_midx_fanout_1(fanout, m, cur_fanout, preferred_pack);\n+}\n+\n static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n \t\t\t\t\tstruct pack_info *info,\n \t\t\t\t\tuint32_t cur_pack,\n@@ -352,6 +365,21 @@ static void midx_fanout_add(struct midx_fanout *fanout,\n \t\t\t\t\t cur_fanout);\n }\n \n+static void midx_fanout_add_compact(struct midx_fanout *fanout,\n+\t\t\t\t struct write_midx_context *ctx,\n+\t\t\t\t uint32_t cur_fanout)\n+{\n+\tstruct multi_pack_index *m = ctx->compact_to;\n+\n+\tASSERT(ctx->compact);\n+\n+\twhile (m && m != ctx->compact_from->base_midx) {\n+\t\tmidx_fanout_add_midx_fanout_1(fanout, m, cur_fanout,\n+\t\t\t\t\t NO_PREFERRED_PACK);\n+\t\tm = m->base_midx;\n+\t}\n+}\n+\n /*\n * It is possible to artificially get into a state where there are many\n * duplicate copies of objects. That can create high memory pressure if\n@@ -370,6 +398,9 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tsize_t alloc_objects, total_objects = 0;\n \tstruct midx_fanout fanout = { 0 };\n \n+\tif (ctx->compact)\n+\t\tASSERT(!start_pack);\n+\n \tfor (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++)\n \t\ttotal_objects = st_add(total_objects,\n \t\t\t\t ctx->info[cur_pack].p->num_objects);\n@@ -388,7 +419,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx,\n \tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n \t\tfanout.nr = 0;\n \n-\t\tmidx_fanout_add(&fanout, ctx, start_pack, cur_fanout);\n+\t\tif (ctx->compact)\n+\t\t\tmidx_fanout_add_compact(&fanout, ctx, cur_fanout);\n+\t\telse\n+\t\t\tmidx_fanout_add(&fanout, ctx, start_pack, cur_fanout);\n \t\tmidx_fanout_sort(&fanout);\n \n \t\t/*\n@@ -956,6 +990,75 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n \treturn 0;\n }\n \n+static uint32_t compactible_packs_between(const struct multi_pack_index *from,\n+\t\t\t\t\t const struct multi_pack_index *to)\n+{\n+\tuint32_t nr;\n+\n+\tASSERT(from && to);\n+\n+\tif (unsigned_add_overflows(to->num_packs, to->num_packs_in_base))\n+\t\tdie(_(\"too many packs, unable to compact\"));\n+\n+\tnr = to->num_packs + to->num_packs_in_base;\n+\tif (nr < from->num_packs_in_base)\n+\t\tBUG(\"unexpected number of packs in base during compaction: \"\n+\t\t \"%\"PRIu32\" < %\"PRIu32, nr, from->num_packs_in_base);\n+\n+\treturn nr - from->num_packs_in_base;\n+}\n+\n+static int fill_packs_from_midx_range(struct write_midx_context *ctx,\n+\t\t\t\t int bitmap_order)\n+{\n+\tstruct multi_pack_index *m = ctx->compact_to;\n+\tuint32_t packs_nr;\n+\n+\tASSERT(ctx->compact && !ctx->nr);\n+\tASSERT(ctx->compact_from);\n+\tASSERT(ctx->compact_to);\n+\n+\tpacks_nr = compactible_packs_between(ctx->compact_from,\n+\t\t\t\t\t ctx->compact_to);\n+\n+\tALLOC_GROW(ctx->info, packs_nr, ctx->alloc);\n+\n+\twhile (m != ctx->compact_from->base_midx) {\n+\t\tuint32_t pack_int_id, preferred_pack_id;\n+\t\tuint32_t i;\n+\n+\t\tif (bitmap_order) {\n+\t\t\tif (midx_preferred_pack(m, &preferred_pack_id) < 0)\n+\t\t\t\tdie(_(\"could not determine preferred pack\"));\n+\t\t} else {\n+\t\t\tpreferred_pack_id = m->num_packs_in_base;\n+\t\t}\n+\n+\t\tpack_int_id = m->num_packs_in_base - ctx->compact_from->num_packs_in_base;\n+\n+\t\tif (fill_pack_from_midx(&ctx->info[pack_int_id++], m,\n+\t\t\t\t\tpreferred_pack_id) < 0)\n+\t\t\treturn -1;\n+\n+\t\tfor (i = m->num_packs_in_base;\n+\t\t i < m->num_packs_in_base + m->num_packs; i++) {\n+\t\t\tif (preferred_pack_id == i)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (fill_pack_from_midx(&ctx->info[pack_int_id++], m,\n+\t\t\t\t\t\ti) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\n+\t\tctx->nr += m->num_packs;\n+\t\tm = m->base_midx;\n+\t}\n+\n+\tASSERT(ctx->nr == packs_nr);\n+\n+\treturn 0;\n+}\n+\n static struct {\n \tconst char *non_split;\n \tconst char *split;\n@@ -1075,6 +1178,9 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \tif (ctx->incremental)\n \t\tgoto out;\n \n+\tif (ctx->compact)\n+\t\tgoto out; /* Compaction always requires an update. */\n+\n \t/*\n \t * Otherwise, we need to verify that the packs covered by the existing\n \t * MIDX match the packs that we already have. The logic to do so is way\n@@ -1120,12 +1226,23 @@ static bool midx_needs_update(struct multi_pack_index *midx, struct write_midx_c\n \treturn needed;\n }\n \n+static int midx_hashcmp(const struct multi_pack_index *a,\n+\t\t\tconst struct multi_pack_index *b,\n+\t\t\tconst struct git_hash_algo *algop)\n+{\n+\treturn hashcmp(midx_get_checksum_hash(a), midx_get_checksum_hash(b),\n+\t\t algop);\n+}\n+\n struct write_midx_opts {\n \tstruct odb_source *source; /* non-optional */\n \n \tstruct string_list *packs_to_include;\n \tstruct string_list *packs_to_drop;\n \n+\tstruct multi_pack_index *compact_from;\n+\tstruct multi_pack_index *compact_to;\n+\n \tconst char *preferred_pack_name;\n \tconst char *refs_snapshot;\n \tunsigned flags;\n@@ -1150,6 +1267,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tint dropped_packs = 0;\n \tint result = -1;\n \tconst char **keep_hashes = NULL;\n+\tsize_t keep_hashes_nr = 0;\n \tstruct chunkfile *cf;\n \n \ttrace2_region_enter(\"midx\", \"write_midx_internal\", r);\n@@ -1162,6 +1280,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\tdie(_(\"unknown MIDX version: %d\"), ctx.version);\n \n \tctx.incremental = !!(opts->flags & MIDX_WRITE_INCREMENTAL);\n+\tctx.compact = !!(opts->flags & MIDX_WRITE_COMPACT);\n+\n+\tif (ctx.compact) {\n+\t\tif (ctx.version != MIDX_VERSION_V2)\n+\t\t\tdie(_(\"cannot perform MIDX compaction with v1 format\"));\n+\t\tif (!opts->compact_from)\n+\t\t\tBUG(\"expected non-NULL 'from' MIDX during compaction\");\n+\t\tif (!opts->compact_to)\n+\t\t\tBUG(\"expected non-NULL 'to' MIDX during compaction\");\n+\n+\t\tctx.compact_from = opts->compact_from;\n+\t\tctx.compact_to = opts->compact_to;\n+\t}\n \n \tif (ctx.incremental)\n \t\tstrbuf_addf(&midx_name,\n@@ -1189,11 +1320,18 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\t */\n \t\t\tif (ctx.incremental)\n \t\t\t\tctx.base_midx = m;\n-\t\t\telse if (!opts->packs_to_include)\n+\t\t\tif (!opts->packs_to_include)\n \t\t\t\tctx.m = m;\n \t\t}\n \t}\n \n+\t/*\n+\t * If compacting MIDX layer(s) in the range [from, to], then the\n+\t * compacted MIDX will share the same base MIDX as 'from'.\n+\t */\n+\tif (ctx.compact)\n+\t\tctx.base_midx = ctx.compact_from->base_midx;\n+\n \tctx.nr = 0;\n \tctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;\n \tctx.info = NULL;\n@@ -1210,7 +1348,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\tctx.num_multi_pack_indexes_before++;\n \t\t\tm = m->base_midx;\n \t\t}\n-\t} else if (ctx.m && fill_packs_from_midx(&ctx)) {\n+\t} else if (ctx.m && !ctx.compact && fill_packs_from_midx(&ctx)) {\n \t\tgoto cleanup;\n \t}\n \n@@ -1223,9 +1361,18 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \telse\n \t\tctx.progress = NULL;\n \n-\tctx.to_include = opts->packs_to_include;\n+\tif (ctx.compact) {\n+\t\tint bitmap_order = 0;\n+\t\tif (opts->preferred_pack_name)\n+\t\t\tbitmap_order |= 1;\n+\t\telse if (opts->flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n+\t\t\tbitmap_order |= 1;\n \n-\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n+\t\tfill_packs_from_midx_range(&ctx, bitmap_order);\n+\t} else {\n+\t\tctx.to_include = opts->packs_to_include;\n+\t\tfor_each_file_in_pack_dir(opts->source->path, add_pack_to_midx, &ctx);\n+\t}\n \tstop_progress(&ctx.progress);\n \n \tif (!opts->packs_to_drop) {\n@@ -1354,12 +1501,19 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t\t\tctx.large_offsets_needed = 1;\n \t}\n \n-\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n+\tif (ctx.compact) {\n+\t\tif (ctx.version != MIDX_VERSION_V2)\n+\t\t\tBUG(\"performing MIDX compaction with v1 MIDX\");\n+\t} else {\n+\t\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n+\t}\n \n \tif (opts->packs_to_drop && opts->packs_to_drop->nr) {\n \t\tsize_t drop_index = 0;\n \t\tint missing_drops = 0;\n \n+\t\tASSERT(!ctx.compact);\n+\n \t\tfor (size_t i = 0;\n \t\t i < ctx.nr && drop_index < opts->packs_to_drop->nr; i++) {\n \t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n@@ -1391,12 +1545,20 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \t */\n \tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n \tfor (size_t i = 0; i < ctx.nr; i++) {\n+\t\tuint32_t from = ctx.info[i].orig_pack_int_id;\n+\t\tuint32_t to;\n+\n \t\tif (ctx.info[i].expired) {\n+\t\t\tto = PACK_EXPIRED;\n \t\t\tdropped_packs++;\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n \t\t} else {\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t\tto = i - dropped_packs;\n \t\t}\n+\n+\t\tif (ctx.compact)\n+\t\t\tfrom -= ctx.compact_from->num_packs_in_base;\n+\n+\t\tctx.pack_perm[from] = to;\n \t}\n \n \tfor (size_t i = 0; i < ctx.nr; i++) {\n@@ -1542,7 +1704,24 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (ctx.num_multi_pack_indexes_before == UINT32_MAX)\n \t\tdie(_(\"too many multi-pack-indexes\"));\n \n-\tCALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);\n+\tif (ctx.compact) {\n+\t\tstruct multi_pack_index *m;\n+\n+\t\t/*\n+\t\t * Keep all MIDX layers excluding those in the range [from, to].\n+\t\t */\n+\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n+\t\t\tkeep_hashes_nr++;\n+\t\tfor (m = ctx.m;\n+\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n+\t\t m = m->base_midx)\n+\t\t\tkeep_hashes_nr++;\n+\n+\t\tkeep_hashes_nr++; /* include the compacted layer */\n+\t} else {\n+\t\tkeep_hashes_nr = ctx.num_multi_pack_indexes_before + 1;\n+\t}\n+\tCALLOC_ARRAY(keep_hashes, keep_hashes_nr);\n \n \tif (ctx.incremental) {\n \t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n@@ -1567,17 +1746,47 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \n \t\tstrbuf_release(&final_midx_name);\n \n-\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n-\t\t\txstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));\n+\t\tif (ctx.compact) {\n+\t\t\tstruct multi_pack_index *m;\n+\t\t\tuint32_t num_layers_before_from = 0;\n+\t\t\tuint32_t i;\n \n-\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n-\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n+\t\t\tfor (m = ctx.base_midx; m; m = m->base_midx)\n+\t\t\t\tnum_layers_before_from++;\n \n-\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n-\t\t\tm = m->base_midx;\n+\t\t\tm = ctx.base_midx;\n+\t\t\tfor (i = 0; i < num_layers_before_from; i++) {\n+\t\t\t\tuint32_t j = num_layers_before_from - i - 1;\n+\n+\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tm = m->base_midx;\n+\t\t\t}\n+\n+\t\t\tkeep_hashes[i] = xstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t\t\t r->hash_algo));\n+\n+\t\t\ti = 0;\n+\t\t\tfor (m = ctx.m;\n+\t\t\t m && midx_hashcmp(m, ctx.compact_to, r->hash_algo);\n+\t\t\t m = m->base_midx) {\n+\t\t\t\tkeep_hashes[keep_hashes_nr - i - 1] =\n+\t\t\t\t\txstrdup(midx_get_checksum_hex(m));\n+\t\t\t\ti++;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n+\t\t\t\txstrdup(hash_to_hex_algop(midx_hash,\n+\t\t\t\t\t\t\t r->hash_algo));\n+\n+\t\t\tfor (uint32_t i = 0; i < ctx.num_multi_pack_indexes_before; i++) {\n+\t\t\t\tuint32_t j = ctx.num_multi_pack_indexes_before - i - 1;\n+\n+\t\t\t\tkeep_hashes[j] = xstrdup(midx_get_checksum_hex(m));\n+\t\t\t\tm = m->base_midx;\n+\t\t\t}\n \t\t}\n \n-\t\tfor (uint32_t i = 0; i <= ctx.num_multi_pack_indexes_before; i++)\n+\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n \t\t\tfprintf(get_lock_file_fp(&lk), \"%s\\n\", keep_hashes[i]);\n \t} else {\n \t\tkeep_hashes[ctx.num_multi_pack_indexes_before] =\n@@ -1590,8 +1799,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tif (commit_lock_file(&lk) < 0)\n \t\tdie_errno(_(\"could not write multi-pack-index\"));\n \n-\tclear_midx_files(opts->source, keep_hashes,\n-\t\t\t ctx.num_multi_pack_indexes_before + 1,\n+\tclear_midx_files(opts->source, keep_hashes, keep_hashes_nr,\n \t\t\t ctx.incremental);\n \tresult = 0;\n \n@@ -1609,7 +1817,7 @@ static int write_midx_internal(struct write_midx_opts *opts)\n \tfree(ctx.pack_perm);\n \tfree(ctx.pack_order);\n \tif (keep_hashes) {\n-\t\tfor (uint32_t i = 0; i <= ctx.num_multi_pack_indexes_before; i++)\n+\t\tfor (uint32_t i = 0; i < keep_hashes_nr; i++)\n \t\t\tfree((char *)keep_hashes[i]);\n \t\tfree(keep_hashes);\n \t}\n@@ -1651,6 +1859,21 @@ int write_midx_file_only(struct odb_source *source,\n \treturn write_midx_internal(&opts);\n }\n \n+int write_midx_file_compact(struct odb_source *source,\n+\t\t\t struct multi_pack_index *from,\n+\t\t\t struct multi_pack_index *to,\n+\t\t\t unsigned flags)\n+{\n+\tstruct write_midx_opts opts = {\n+\t\t.source = source,\n+\t\t.compact_from = from,\n+\t\t.compact_to = to,\n+\t\t.flags = flags | MIDX_WRITE_COMPACT,\n+\t};\n+\n+\treturn write_midx_internal(&opts);\n+}\n+\n int expire_midx_packs(struct odb_source *source, unsigned flags)\n {\n \tuint32_t i, *count, result = 0;\ndiff --git a/midx.h b/midx.h\nindex aa99a6cb215..08f3728e520 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -82,6 +82,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n #define MIDX_WRITE_INCREMENTAL (1 << 5)\n+#define MIDX_WRITE_COMPACT (1 << 6)\n \n #define MIDX_EXT_REV \"rev\"\n #define MIDX_EXT_BITMAP \"bitmap\"\n@@ -131,6 +132,10 @@ int write_midx_file_only(struct odb_source *source,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n \t\t\t const char *refs_snapshot, unsigned flags);\n+int write_midx_file_compact(struct odb_source *source,\n+\t\t\t struct multi_pack_index *from,\n+\t\t\t struct multi_pack_index *to,\n+\t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct odb_source *source, unsigned flags);\n int expire_midx_packs(struct odb_source *source, unsigned flags);\ndiff --git a/t/meson.build b/t/meson.build\nindex f80e366cff7..2421220917a 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -618,6 +618,7 @@ integration_tests = [\n 't5332-multi-pack-reuse.sh',\n 't5333-pseudo-merge-bitmaps.sh',\n 't5334-incremental-multi-pack-index.sh',\n+ 't5335-compact-multi-pack-index.sh',\n 't5351-unpack-large-objects.sh',\n 't5400-send-pack.sh',\n 't5401-update-hooks.sh',\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nnew file mode 100755\nindex 00000000000..797ae05c3bd\n--- /dev/null\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -0,0 +1,175 @@\n+#!/bin/sh\n+\n+test_description='multi-pack-index compaction'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+midxdir=$packdir/multi-pack-index.d\n+midx_chain=$midxdir/multi-pack-index-chain\n+\n+nth_line() {\n+\tlocal n=\"$1\"\n+\tshift\n+\tawk \"NR==$n\" \"$@\"\n+}\n+\n+write_packs () {\n+\tfor c in \"$@\"\n+\tdo\n+\t\ttest_commit \"$c\" &&\n+\n+\t\tgit pack-objects --all --unpacked $packdir/pack-$c &&\n+\t\tgit prune-packed &&\n+\n+\t\tgit multi-pack-index write --incremental --bitmap || return 1\n+\tdone\n+}\n+\n+test_midx_layer_packs () {\n+\tlocal checksum=\"$1\" &&\n+\tshift &&\n+\n+\ttest-tool read-midx $objdir \"$checksum\" >out &&\n+\n+\tprintf \"%s\\n\" \"$@\" >expect &&\n+\t# NOTE: do *not* pipe through sort here, we want to ensure the\n+\t# order of packs is preserved during compaction.\n+\tgrep \"^pack-\" out | cut -d\"-\" -f2 >actual &&\n+\n+\ttest_cmp expect actual\n+}\n+\n+test_midx_layer_object_uniqueness () {\n+\t: >objs.all\n+\twhile read layer\n+\tdo\n+\t\ttest-tool read-midx --show-objects $objdir \"$layer\" >out &&\n+\t\tgrep \"\\.pack$\" out | cut -d\" \" -f1 | sort >objs.layer &&\n+\t\ttest_stdout_line_count = 0 comm -12 objs.all objs.layer &&\n+\t\tcat objs.all objs.layer | sort >objs.tmp &&\n+\t\tmv objs.tmp objs.all || return 1\n+\tdone <$midx_chain\n+}\n+\n+test_expect_success 'MIDX compaction with lex-ordered pack names' '\n+\tgit init midx-compact-lex-order &&\n+\t(\n+\t\tcd midx-compact-lex-order &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C D E &&\n+\t\ttest_line_count = 5 $midx_chain &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\ttest_midx_layer_packs \"$(nth_line 1 \"$midx_chain\")\" A &&\n+\t\ttest_midx_layer_packs \"$(nth_line 2 \"$midx_chain\")\" B C D &&\n+\t\ttest_midx_layer_packs \"$(nth_line 3 \"$midx_chain\")\" E &&\n+\n+\t\ttest_midx_layer_object_uniqueness\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with non-lex-ordered pack names' '\n+\tgit init midx-compact-non-lex-order &&\n+\t(\n+\t\tcd midx-compact-non-lex-order &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs D C A B E &&\n+\t\ttest_line_count = 5 $midx_chain &&\n+\n+\t\tgit multi-pack-index compact --incremental \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest_line_count = 3 $midx_chain &&\n+\n+\t\ttest_midx_layer_packs \"$(nth_line 1 \"$midx_chain\")\" D &&\n+\t\ttest_midx_layer_packs \"$(nth_line 2 \"$midx_chain\")\" C A B &&\n+\t\ttest_midx_layer_packs \"$(nth_line 3 \"$midx_chain\")\" E &&\n+\n+\t\ttest_midx_layer_object_uniqueness\n+\t)\n+'\n+\n+test_expect_success 'setup for bogus MIDX compaction scenarios' '\n+\tgit init midx-compact-bogus &&\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs A B C\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with missing endpoints' '\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"\" \"\" 2>err &&\n+\t\ttest_grep \"could not find MIDX: \" err &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"\" \"$(nth_line 2 \"$midx_chain\")\" 2>err &&\n+\t\ttest_grep \"could not find MIDX: \" err &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \"\" 2>err &&\n+\t\ttest_grep \"could not find MIDX: \" err\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with reversed endpoints' '\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\tfrom=\"$(nth_line 3 \"$midx_chain\")\" &&\n+\t\tto=\"$(nth_line 1 \"$midx_chain\")\" &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"$from\" \"$to\" 2>err &&\n+\n+\t\ttest_grep \"MIDX $from must be an ancestor of $to\" err\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with identical endpoints' '\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\tfrom=\"$(nth_line 3 \"$midx_chain\")\" &&\n+\t\tto=\"$(nth_line 3 \"$midx_chain\")\" &&\n+\n+\t\ttest_must_fail git multi-pack-index compact --incremental \\\n+\t\t\t\"$from\" \"$to\" 2>err &&\n+\n+\t\ttest_grep \"MIDX compaction endpoints must be unique\" err\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with midx.version=1' '\n+\t(\n+\t\tcd midx-compact-bogus &&\n+\n+\t\ttest_must_fail git -c midx.version=1 multi-pack-index compact \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" 2>err &&\n+\n+\t\ttest_grep \"fatal: cannot perform MIDX compaction with v1 format\" err\n+\t)\n+'\n+\n+test_done\n-- \n2.53.0.171.gde83996e422\n\n"},{"id":"537007","messageId":"ac6221686db11ba802ced8ea6b8b2fa8388b21b2.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 17/17] midx: enable reachability bitmaps during MIDX compaction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:45Z","receivedAt":"2026-02-24T19:00:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Enable callers to generate reachability bitmaps when performing MIDX\nlayer compaction by combining all existing bitmaps from the compacted\nlayers.\n\nNote that because of the object/pack ordering described by the previous\ncommit, the pseudo-pack order for the compacted MIDX is the same as\nconcatenating the individual pseudo-pack orderings for each layer in the\ncompaction range.\n\nAs a result, the only non-test or documentation change necessary is to\ntreat all objects as non-preferred during compaction so as not to\ndisturb the object ordering.\n\nIn the future, we may want to adjust which commit(s) receive\nreachability bitmaps when compacting multiple .bitmap files into one, or\neven generate new bitmaps (e.g., if the references have moved\nsignificantly since the .bitmap was generated). This commit only\nimplements combining all existing bitmaps in range together in order to\ndemonstrate and lay the groundwork for more exotic strategies.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.adoc | 5 +-\n builtin/multi-pack-index.c | 4 +-\n midx-write.c | 2 +-\n t/t5335-compact-multi-pack-index.sh | 122 +++++++++++++++++++++++-\n 4 files changed, 128 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc\nindex 883a0529741..61256830141 100644\n--- a/Documentation/git-multi-pack-index.adoc\n+++ b/Documentation/git-multi-pack-index.adoc\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t\t [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n \t\t [--refs-snapshot=]\n 'git multi-pack-index' [] compact [--[no-]incremental]\n-\t\t \n+\t\t [--[no-]bitmap] \n 'git multi-pack-index' [] verify\n 'git multi-pack-index' [] expire\n 'git multi-pack-index' [] repack [--batch-size=]\n@@ -94,6 +94,9 @@ compact::\n \t--incremental::\n \t\tWrite the result to a MIDX chain instead of writing a\n \t\tstand-alone MIDX.\n+\n+\t--[no-]bitmap::\n+\t\tControl whether or not a multi-pack bitmap is written.\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 043ee8c478a..2f24c113c8f 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -19,7 +19,7 @@\n \n #define BUILTIN_MIDX_COMPACT_USAGE \\\n \tN_(\"git multi-pack-index [] compact [--[no-]incremental]\\n\" \\\n-\t \" \")\n+\t \" [--[no-]bitmap] \")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -216,6 +216,8 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,\n \n \tstruct option *options;\n \tstatic struct option builtin_multi_pack_index_compact_options[] = {\n+\t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n+\t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BIT(0, \"incremental\", &opts.flags,\n \t\t\tN_(\"write a new incremental MIDX\"), MIDX_WRITE_INCREMENTAL),\n \t\tOPT_END(),\ndiff --git a/midx-write.c b/midx-write.c\nindex bf53ad1c4b7..0ff2e45aa7a 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -676,7 +676,7 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \t\tstruct pack_midx_entry *e = &ctx->entries[i];\n \t\tdata[i].nr = i;\n \t\tdata[i].pack = midx_pack_perm(ctx, e->pack_int_id);\n-\t\tif (!e->preferred)\n+\t\tif (!e->preferred || ctx->compact)\n \t\t\tdata[i].pack |= (1U << 31);\n \t\tdata[i].offset = e->offset;\n \t}\ndiff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh\nindex 797ae05c3bd..40f3844282f 100755\n--- a/t/t5335-compact-multi-pack-index.sh\n+++ b/t/t5335-compact-multi-pack-index.sh\n@@ -67,7 +67,7 @@ test_expect_success 'MIDX compaction with lex-ordered pack names' '\n \t\twrite_packs A B C D E &&\n \t\ttest_line_count = 5 $midx_chain &&\n \n-\t\tgit multi-pack-index compact --incremental \\\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n \t\ttest_line_count = 3 $midx_chain &&\n@@ -90,7 +90,7 @@ test_expect_success 'MIDX compaction with non-lex-ordered pack names' '\n \t\twrite_packs D C A B E &&\n \t\ttest_line_count = 5 $midx_chain &&\n \n-\t\tgit multi-pack-index compact --incremental \\\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n \t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n \t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n \t\ttest_line_count = 3 $midx_chain &&\n@@ -172,4 +172,122 @@ test_expect_success 'MIDX compaction with midx.version=1' '\n \t)\n '\n \n+midx_objs_by_pack () {\n+\tawk '/\\.pack$/ { split($3, a, \"-\"); print a[2], $1 }' | sort\n+}\n+\n+tag_objs_from_pack () {\n+\tobjs=\"$(git rev-list --objects --no-object-names \"$2\")\" &&\n+\tprintf \"$1 %s\\n\" $objs | sort\n+}\n+\n+test_expect_success 'MIDX compaction preserves pack object selection' '\n+\tgit init midx-compact-preserve-selection &&\n+\t(\n+\t\tcd midx-compact-preserve-selection &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\t# Create two packs, one containing just the objects from\n+\t\t# A, and another containing all objects from the\n+\t\t# repository.\n+\t\tp1=\"$(echo A | git pack-objects --revs --delta-base-offset \\\n+\t\t\t$packdir/pack-1)\" &&\n+\t\tp0=\"$(echo B | git pack-objects --revs --delta-base-offset \\\n+\t\t\t$packdir/pack-0)\" &&\n+\n+\t\techo \"pack-1-$p1.idx\" | git multi-pack-index write \\\n+\t\t\t--incremental --bitmap --stdin-packs &&\n+\t\techo \"pack-0-$p0.idx\" | git multi-pack-index write \\\n+\t\t\t--incremental --bitmap --stdin-packs &&\n+\n+\t\twrite_packs C &&\n+\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" &&\n+\n+\n+\t\ttest-tool read-midx --show-objects $objdir \\\n+\t\t\t\"$(nth_line 1 \"$midx_chain\")\" >AB.info &&\n+\t\ttest-tool read-midx --show-objects $objdir \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" >C.info &&\n+\n+\t\tmidx_objs_by_pack AB.actual &&\n+\t\tmidx_objs_by_pack C.actual &&\n+\n+\t\t{\n+\t\t\ttag_objs_from_pack 1 A &&\n+\t\t\ttag_objs_from_pack 0 A..B\n+\t\t} | sort >AB.expect &&\n+\t\ttag_objs_from_pack C B..C >C.expect &&\n+\n+\t\ttest_cmp AB.expect AB.actual &&\n+\t\ttest_cmp C.expect C.actual\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bitmaps' '\n+\tgit init midx-compact-with-bitmaps &&\n+\t(\n+\t\tcd midx-compact-with-bitmaps &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\twrite_packs foo bar baz quux woot &&\n+\n+\t\ttest-tool read-midx --bitmap $objdir >bitmap.expect &&\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\t\ttest-tool read-midx --bitmap $objdir >bitmap.actual &&\n+\n+\t\ttest_cmp bitmap.expect bitmap.actual &&\n+\n+\t\ttrue\n+\t)\n+'\n+\n+test_expect_success 'MIDX compaction with bitmaps (non-trivial)' '\n+\tgit init midx-compact-with-bitmaps-non-trivial &&\n+\t(\n+\t\tcd midx-compact-with-bitmaps-non-trivial &&\n+\n+\t\tgit config maintenance.auto false &&\n+\n+\t\tgit branch -m main &&\n+\n+\t\t# D(4)\n+\t\t# /\n+\t\t# A(1) --- B(2) --- C(3) --- G(7)\n+\t\t# \\\n+\t\t# E(5) --- F(6)\n+\t\twrite_packs A B C &&\n+\t\tgit checkout -b side &&\n+\t\twrite_packs D &&\n+\t\tgit checkout -b other B &&\n+\t\twrite_packs E F &&\n+\t\tgit checkout main &&\n+\t\twrite_packs G &&\n+\n+\t\t# Compact layers 2-4, leaving us with:\n+\t\t#\n+\t\t# [A, [B, C, D], E, F, G]\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 2 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" &&\n+\n+\t\t# Then compact the top two layers, condensing the above\n+\t\t# such that the new 4th layer contains F and G.\n+\t\t#\n+\t\t# [A, [B, C, D], E, [F, G]]\n+\t\tgit multi-pack-index compact --incremental --bitmap \\\n+\t\t\t\"$(nth_line 4 \"$midx_chain\")\" \\\n+\t\t\t\"$(nth_line 5 \"$midx_chain\")\"\n+\t)\n+'\n+\n test_done\n-- \n2.53.0.171.gde83996e422\n"},{"id":"537008","messageId":"f3952f7db36585642145f5dc7c2192e5ddff7610.1771959555.git.me@ttaylorr.com","threadId":"64585","inReplyTo":"cover.1771959555.git.me@ttaylorr.com","subject":"[PATCH v3 12/17] midx-write.c: extract `fill_pack_from_midx()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2026-02-24T19:00:21Z","receivedAt":"2026-02-24T19:08:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When filling packs from an existing MIDX, `fill_packs_from_midx()`\nhandles preparing a MIDX'd pack, and reading out its pack name from the\nexisting MIDX.\n\nMIDX compaction will want to perform an identical operation, though the\ncaller will look quite different than `fill_packs_from_midx()`. To\nreduce any future code duplication, extract `fill_pack_from_midx()`\nfrom `fill_packs_from_midx()` to prepare to call our new helper function\nin a future change.\n\nSigned-off-by: Taylor Blau \n---\n midx-write.c | 27 +++++++++++++++++++++------\n 1 file changed, 21 insertions(+), 6 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 9d345fb4737..c54113cdc84 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -913,6 +913,21 @@ static int write_midx_bitmap(struct write_midx_context *ctx,\n \treturn ret;\n }\n \n+static int fill_pack_from_midx(struct pack_info *info,\n+\t\t\t struct multi_pack_index *m,\n+\t\t\t uint32_t pack_int_id)\n+{\n+\tif (prepare_midx_pack(m, pack_int_id))\n+\t\treturn error(_(\"could not load pack %d\"), pack_int_id);\n+\n+\tfill_pack_info(info,\n+\t\t m->packs[pack_int_id - m->num_packs_in_base],\n+\t\t m->pack_names[pack_int_id - m->num_packs_in_base],\n+\t\t pack_int_id);\n+\n+\treturn 0;\n+}\n+\n static int fill_packs_from_midx(struct write_midx_context *ctx)\n {\n \tstruct multi_pack_index *m;\n@@ -921,13 +936,13 @@ static int fill_packs_from_midx(struct write_midx_context *ctx)\n \t\tuint32_t i;\n \n \t\tfor (i = 0; i < m->num_packs; i++) {\n-\t\t\tif (prepare_midx_pack(m, m->num_packs_in_base + i))\n-\t\t\t\treturn error(_(\"could not load pack\"));\n-\n \t\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n-\t\t\tfill_pack_info(&ctx->info[ctx->nr++], m->packs[i],\n-\t\t\t\t m->pack_names[i],\n-\t\t\t\t m->num_packs_in_base + i);\n+\n+\t\t\tif (fill_pack_from_midx(&ctx->info[ctx->nr], m,\n+\t\t\t\t\t\tm->num_packs_in_base + i) < 0)\n+\t\t\t\treturn -1;\n+\n+\t\t\tctx->nr++;\n \t\t}\n \t}\n \treturn 0;\n-- \n2.53.0.171.gde83996e422\n\n"}]}