git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 14/16] repack: implement incremental MIDX repacking

From
Taylor Blau <me@ttaylorr.com>
Date
Apr 30, 2026, 00:13 UTC
Message-ID
<ef0123149301df184b5b6e858a88530615fde0c7.1777507303.git.me@ttaylorr.com>
In-Reply-To
<cover.1777507303.git.me@ttaylorr.com>

Implement the `write_midx_incremental()` function, which builds and maintains an incremental MIDX chain as part of the geometric repacking process.

Unlike the default mode which writes a single flat MIDX, the incremental mode constructs a compaction plan that determines which MIDX layers to write, compact, or copy, and then executes each step using `git multi-pack-index` subcommands with the --no-write-chain-file flag.

The repacking strategy works as follows:
 * Acquire the lock guarding the multi-pack-index-chain.
 * A new MIDX layer is always written containing the newly created
   pack(s). If the tip MIDX layer was rewritten during geometric
   repacking, any surviving packs from that layer are also included.
 * Starting from the new layer, adjacent MIDX layers are merged together
   as long as the accumulated object count exceeds half the object count
   of the next deeper layer (controlled by 'repack.midxSplitFactor').
 * Remaining layers in the chain are evaluated pairwise and either
   compacted or copied as-is, following the same merging condition.
 * Write the contents of the new multi-pack-index chain, atomically move
   it into place, and then release the lock.
 * Delete any now-unused MIDX layers.

After writing the new layer, the strategy is evaluated among the existing MIDX layers in order from oldest to newest. Each step that writes a new MIDX layer uses "--no-write-chain-file" to avoid updating the multi-pack-index-chain file. After all steps are complete, the new chain file is written and then atomically moved into place.

At present, this functionality is exposed behind a new enum value, `REPACK_WRITE_MIDX_INCREMENTAL`, but has no external callers. A subsequent commit will expose this mode via `git repack --write-midx=incremental`.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 builtin/repack.c |   5 +
 repack-midx.c    | 593 +++++++++++++++++++++++++++++++++++++++++++++--
 repack.h         |   3 +
 3 files changed, 588 insertions(+), 13 deletions(-)
diff --git a/builtin/repack.c b/builtin/repack.c
index 5d366340c34..75c57736780 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -42,6 +42,9 @@ static const char incremental_bitmap_conflict_error[] = N_(
 "--no-write-bitmap-index or disable the pack.writeBitmaps configuration."
 );
 
+#define DEFAULT_MIDX_SPLIT_FACTOR 2
+#define DEFAULT_MIDX_NEW_LAYER_THRESHOLD 8
+
 struct repack_config_ctx {
 	struct pack_objects_args *po_args;
 	struct pack_objects_args *cruft_po_args;
@@ -555,6 +558,8 @@ int cmd_repack(int argc,
 			.show_progress = show_progress,
 			.write_bitmaps = write_bitmaps > 0,
 			.midx_must_contain_cruft = midx_must_contain_cruft,
+			.midx_split_factor = DEFAULT_MIDX_SPLIT_FACTOR,
+			.midx_new_layer_threshold = DEFAULT_MIDX_NEW_LAYER_THRESHOLD,
 			.mode = write_midx,
 		};
 
diff --git a/repack-midx.c b/repack-midx.c
index 4a568a2a9b8..405f4662174 100644
--- a/repack-midx.c
+++ b/repack-midx.c
@@ -2,12 +2,16 @@
 #include "repack.h"
 #include "hash.h"
 #include "hex.h"
+#include "lockfile.h"
+#include "midx.h"
 #include "odb.h"
 #include "oidset.h"
 #include "pack-bitmap.h"
+#include "path.h"
 #include "refs.h"
 #include "run-command.h"
 #include "tempfile.h"
+#include "trace2.h"
 
 struct midx_snapshot_ref_data {
 	struct repository *repo;
@@ -293,26 +297,30 @@ static void repack_prepare_midx_command(struct child_process *cmd,
 }
 
 static int repack_fill_midx_stdin_packs(struct child_process *cmd,
-					struct string_list *include)
+					struct string_list *include,
+					struct string_list *out)
 {
+	struct strbuf in_buf = STRBUF_INIT;
+	struct strbuf out_buf = STRBUF_INIT;
 	struct string_list_item *item;
-	FILE *in;
 	int ret;
 
-	cmd->in = -1;
-
 	strvec_push(&cmd->args, "--stdin-packs");
 
-	ret = start_command(cmd);
-	if (ret)
-		return ret;
-
-	in = xfdopen(cmd->in, "w");
 	for_each_string_list_item(item, include)
-		fprintf(in, "%s\n", item->string);
-	fclose(in);
+		strbuf_addf(&in_buf, "%s\n", item->string);
 
-	return finish_command(cmd);
+	ret = pipe_command(cmd, in_buf.buf, in_buf.len,
+			   out ? &out_buf : NULL, 0, NULL, 0);
+
+	if (out)
+		string_list_split_f(out, out_buf.buf, "\n", -1,
+				    STRING_LIST_SPLIT_NONEMPTY);
+
+	strbuf_release(&in_buf);
+	strbuf_release(&out_buf);
+
+	return ret;
 }
 
 static int write_midx_included_packs(struct repack_write_midx_opts *opts)
@@ -369,7 +377,7 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)
 		strvec_pushf(&cmd.args, "--refs-snapshot=%s",
 			     opts->refs_snapshot);
 
-	ret = repack_fill_midx_stdin_packs(&cmd, &include);
+	ret = repack_fill_midx_stdin_packs(&cmd, &include, NULL);
 done:
 	if (!ret && opts->write_bitmaps)
 		remove_redundant_bitmaps(&include, opts->packdir);
@@ -379,6 +387,563 @@ static int write_midx_included_packs(struct repack_write_midx_opts *opts)
 	return ret;
 }
 
+struct midx_compaction_step {
+	union {
+		struct multi_pack_index *copy;
+		struct string_list write;
+		struct {
+			struct multi_pack_index *from;
+			struct multi_pack_index *to;
+		} compact;
+	} u;
+
+	uint32_t objects_nr;
+	char *csum;
+
+	enum {
+		MIDX_COMPACTION_STEP_UNKNOWN,
+		MIDX_COMPACTION_STEP_COPY,
+		MIDX_COMPACTION_STEP_WRITE,
+		MIDX_COMPACTION_STEP_COMPACT,
+	} type;
+};
+
+static const char *midx_compaction_step_base(const struct midx_compaction_step *step)
+{
+	switch (step->type) {
+	case MIDX_COMPACTION_STEP_UNKNOWN:
+		BUG("cannot use UNKNOWN step as a base");
+	case MIDX_COMPACTION_STEP_COPY:
+		return midx_get_checksum_hex(step->u.copy);
+	case MIDX_COMPACTION_STEP_WRITE:
+		BUG("cannot use WRITE step as a base");
+	case MIDX_COMPACTION_STEP_COMPACT:
+		return midx_get_checksum_hex(step->u.compact.to);
+	default:
+		BUG("unhandled midx compaction step type %d", step->type);
+	}
+}
+
+static int midx_compaction_step_exec_copy(struct midx_compaction_step *step)
+{
+	step->csum = xstrdup(midx_get_checksum_hex(step->u.copy));
+	return 0;
+}
+
+static int midx_compaction_step_exec_write(struct midx_compaction_step *step,
+					   struct repack_write_midx_opts *opts,
+					   const char *base)
+{
+	struct child_process cmd = CHILD_PROCESS_INIT;
+	struct string_list hash = STRING_LIST_INIT_DUP;
+	struct string_list_item *item;
+	const char *preferred_pack = NULL;
+	int ret = 0;
+
+	if (!step->u.write.nr) {
+		ret = error(_("no packs to write MIDX during compaction"));
+		goto out;
+	}
+
+	for_each_string_list_item(item, &step->u.write) {
+		if (item->util)
+			preferred_pack = item->string;
+	}
+
+	repack_prepare_midx_command(&cmd, opts, "write");
+	strvec_pushl(&cmd.args, "--incremental", "--no-write-chain-file", NULL);
+	strvec_pushf(&cmd.args, "--base=%s", base ? base : "none");
+
+	if (preferred_pack) {
+		struct strbuf buf = STRBUF_INIT;
+
+		strbuf_addstr(&buf, preferred_pack);
+		strbuf_strip_suffix(&buf, ".idx");
+		strbuf_addstr(&buf, ".pack");
+
+		strvec_pushf(&cmd.args, "--preferred-pack=%s", buf.buf);
+
+		strbuf_release(&buf);
+	}
+
+	ret = repack_fill_midx_stdin_packs(&cmd, &step->u.write, &hash);
+	if (hash.nr != 1) {
+		ret = error(_("expected exactly one line during MIDX write, "
+			      "got: %"PRIuMAX),
+			    (uintmax_t)hash.nr);
+		goto out;
+	}
+
+	step->csum = xstrdup(hash.items[0].string);
+
+out:
+	string_list_clear(&hash, 0);
+
+	return ret;
+}
+
+static int midx_compaction_step_exec_compact(struct midx_compaction_step *step,
+					     struct repack_write_midx_opts *opts)
+{
+	struct child_process cmd = CHILD_PROCESS_INIT;
+	struct strbuf buf = STRBUF_INIT;
+	FILE *out = NULL;
+	int ret;
+
+	repack_prepare_midx_command(&cmd, opts, "compact");
+	strvec_pushl(&cmd.args, "--incremental", "--no-write-chain-file",
+		     midx_get_checksum_hex(step->u.compact.from),
+		     midx_get_checksum_hex(step->u.compact.to), NULL);
+
+	cmd.out = -1;
+
+	ret = start_command(&cmd);
+	if (ret)
+		goto out;
+
+	out = xfdopen(cmd.out, "r");
+	while (strbuf_getline_lf(&buf, out) != EOF) {
+		if (step->csum) {
+			ret = error(_("unexpected MIDX output: '%s'"), buf.buf);
+			fclose(out);
+			out = NULL;
+			finish_command(&cmd);
+			goto out;
+		}
+		step->csum = strbuf_detach(&buf, NULL);
+	}
+
+	ret = finish_command(&cmd);
+
+out:
+	if (out)
+		fclose(out);
+	strbuf_release(&buf);
+
+	return ret;
+}
+
+static int midx_compaction_step_exec(struct midx_compaction_step *step,
+				     struct repack_write_midx_opts *opts,
+				     const char *base)
+{
+	switch (step->type) {
+	case MIDX_COMPACTION_STEP_UNKNOWN:
+		BUG("cannot execute UNKNOWN midx compaction step");
+	case MIDX_COMPACTION_STEP_COPY:
+		return midx_compaction_step_exec_copy(step);
+	case MIDX_COMPACTION_STEP_WRITE:
+		return midx_compaction_step_exec_write(step, opts, base);
+	case MIDX_COMPACTION_STEP_COMPACT:
+		return midx_compaction_step_exec_compact(step, opts);
+	default:
+		BUG("unhandled midx compaction step type %d", step->type);
+	}
+}
+
+static void midx_compaction_step_release(struct midx_compaction_step *step)
+{
+	if (step->type == MIDX_COMPACTION_STEP_WRITE)
+		string_list_clear(&step->u.write, 0);
+	free(step->csum);
+}
+
+static int repack_make_midx_compaction_plan(struct repack_write_midx_opts *opts,
+					    struct midx_compaction_step **steps_p,
+					    size_t *steps_nr_p)
+{
+	struct multi_pack_index *m;
+	struct midx_compaction_step *steps = NULL;
+	struct midx_compaction_step step = { 0 };
+	struct strbuf buf = STRBUF_INIT;
+	size_t steps_nr = 0, steps_alloc = 0;
+	uint32_t i;
+	int ret = 0;
+
+	trace2_region_enter("repack", "make_midx_compaction_plan",
+			    opts->existing->repo);
+
+	odb_reprepare(opts->existing->repo->objects);
+	m = get_multi_pack_index(opts->existing->source);
+
+	for (i = 0; m && i < m->num_packs + m->num_packs_in_base; i++) {
+		if (prepare_midx_pack(m, i)) {
+			ret = error(_("could not load pack %"PRIu32" from MIDX"),
+				    i);
+			goto out;
+		}
+	}
+
+	trace2_region_enter("repack", "steps:write", opts->existing->repo);
+
+	/*
+	 * The first MIDX in the resulting chain is always going to be
+	 * new.
+	 *
+	 * At a minimum, it will include all of the newly written packs.
+	 * If there is an existing MIDX whose tip layer contains packs
+	 * that were repacked, it will also include any of its packs
+	 * which were *not* rolled up as part of the geometric repack
+	 * (if any), and the previous tip will be replaced.
+	 *
+	 * It may grow to include the packs from zero or more MIDXs from
+	 * the old chain, beginning either at the old tip (if the MIDX
+	 * was *not* rewritten) or the old tip's base MIDX layer
+	 * (otherwise).
+	 */
+	step.type = MIDX_COMPACTION_STEP_WRITE;
+	string_list_init_dup(&step.u.write);
+
+	for (i = 0; i < opts->names->nr; i++) {
+		strbuf_reset(&buf);
+		strbuf_addf(&buf, "pack-%s.idx", opts->names->items[i].string);
+		string_list_append(&step.u.write, buf.buf);
+
+		trace2_data_string("repack", opts->existing->repo,
+				   "include:fresh",
+				   step.u.write.items[step.u.write.nr - 1].string);
+	}
+	for (i = 0; i < opts->geometry->split; i++) {
+		struct packed_git *p = opts->geometry->pack[i];
+		if (unsigned_add_overflows(step.objects_nr, p->num_objects)) {
+			ret = error(_("too many objects in MIDX compaction step"));
+			goto out;
+		}
+
+		step.objects_nr += p->num_objects;
+	}
+	trace2_data_intmax("repack", opts->existing->repo,
+			   "include:fresh:objects_nr",
+			   (uintmax_t)step.objects_nr);
+
+	/*
+	 * Now handle any existing packs which were *not* rewritten.
+	 *
+	 * The list of packs in opts->geometry only contains MIDX'd
+	 * packs from the newest layer when that layer has more than
+	 * 'repack.midxNewLayerThreshold' number of packs.
+	 *
+	 * If the MIDX tip was rewritten (that is, one or more of those
+	 * packs appear below the split line), then add all packs above
+	 * the split line to the new layer, as the old one is no longer
+	 * usable.
+	 *
+	 * If the MIDX tip was not rewritten (that is, all MIDX'd packs
+	 * from the youngest layer appear below the split line, or were
+	 * not included in the geometric repack at all because there
+	 * were too few of them), ignore them since we'll retain the
+	 * existing layer as-is.
+	 */
+	for (i = opts->geometry->split; i < opts->geometry->pack_nr; i++) {
+		struct packed_git *p = opts->geometry->pack[i];
+		struct string_list_item *item;
+
+		strbuf_reset(&buf);
+		strbuf_addstr(&buf, pack_basename(p));
+		strbuf_strip_suffix(&buf, ".pack");
+		strbuf_addstr(&buf, ".idx");
+
+		if (p->multi_pack_index &&
+		    !opts->geometry->midx_tip_rewritten) {
+			trace2_data_string("repack", opts->existing->repo,
+					   "exclude:unmodified", buf.buf);
+			continue;
+		}
+
+		trace2_data_string("repack", opts->existing->repo,
+				   "include:unmodified", buf.buf);
+		trace2_data_string("repack", opts->existing->repo,
+				   "include:unmodified:midx",
+				   p->multi_pack_index ? "true" : "false");
+
+		item = string_list_append(&step.u.write, buf.buf);
+		if (p->multi_pack_index || i == opts->geometry->pack_nr - 1)
+			item->util = (void *)1; /* mark as preferred */
+
+		if (unsigned_add_overflows(step.objects_nr, p->num_objects)) {
+			ret = error(_("too many objects in MIDX compaction step"));
+			goto out;
+		}
+
+		step.objects_nr += p->num_objects;
+	}
+	trace2_data_intmax("repack", opts->existing->repo,
+			   "include:unmodified:objects_nr",
+			   (uintmax_t)step.objects_nr);
+
+	/*
+	 * If the MIDX tip was rewritten, then we no longer consider it
+	 * a candidate for compaction, since it will not exist in the
+	 * MIDX chain being built.
+	 */
+	if (opts->geometry->midx_tip_rewritten)
+		m = m->base_midx;
+
+	trace2_data_string("repack", opts->existing->repo, "midx:rewrote-tip",
+			   opts->geometry->midx_tip_rewritten ? "true" : "false");
+
+	trace2_region_enter("repack", "compact", opts->existing->repo);
+
+	/*
+	 * Compact additional MIDX layers into this proposed one until
+	 * the merging condition is violated.
+	 */
+	while (m) {
+		uint32_t preferred_pack_idx;
+
+		trace2_data_string("repack", opts->existing->repo,
+				   "candidate", midx_get_checksum_hex(m));
+
+		if (step.objects_nr < m->num_objects / opts->midx_split_factor) {
+			/*
+			 * Stop compacting MIDX layer as soon as the
+			 * merged size is less than half the size of the
+			 * next layer in the chain.
+			 */
+			trace2_data_string("repack", opts->existing->repo,
+					   "compact", "violated");
+			trace2_data_intmax("repack", opts->existing->repo,
+					   "objects_nr",
+					   (uintmax_t)step.objects_nr);
+			trace2_data_intmax("repack", opts->existing->repo,
+					   "next_objects_nr",
+					   (uintmax_t)m->num_objects);
+			trace2_data_intmax("repack", opts->existing->repo,
+					   "split_factor",
+					   (uintmax_t)opts->midx_split_factor);
+
+			break;
+		}
+
+		if (midx_preferred_pack(m, &preferred_pack_idx) < 0) {
+			ret = error(_("could not find preferred pack for MIDX "
+				      "%s"), midx_get_checksum_hex(m));
+			goto out;
+		}
+
+		for (i = 0; i < m->num_packs; i++) {
+			struct string_list_item *item;
+			uint32_t pack_int_id = i + m->num_packs_in_base;
+			struct packed_git *p = nth_midxed_pack(m, pack_int_id);
+
+			strbuf_reset(&buf);
+			strbuf_addstr(&buf, pack_basename(p));
+			strbuf_strip_suffix(&buf, ".pack");
+			strbuf_addstr(&buf, ".idx");
+
+			trace2_data_string("repack", opts->existing->repo,
+					   "midx:pack", buf.buf);
+
+			item = string_list_append(&step.u.write, buf.buf);
+			if (pack_int_id == preferred_pack_idx)
+				item->util = (void *)1; /* mark as preferred */
+		}
+
+		if (unsigned_add_overflows(step.objects_nr, m->num_objects)) {
+			ret = error(_("too many objects in MIDX compaction step"));
+			goto out;
+		}
+		step.objects_nr += m->num_objects;
+
+		m = m->base_midx;
+	}
+
+	if (step.u.write.nr > 0) {
+		/*
+		 * As long as there is at least one new pack to write
+		 * (and thus the MIDX is non-empty), add it to the plan.
+		 */
+		ALLOC_GROW(steps, steps_nr + 1, steps_alloc);
+		steps[steps_nr++] = step;
+	}
+
+	trace2_data_intmax("repack", opts->existing->repo,
+			   "step:objects_nr", (uintmax_t)step.objects_nr);
+	trace2_data_intmax("repack", opts->existing->repo,
+			   "step:packs_nr", (uintmax_t)step.u.write.nr);
+
+	trace2_region_leave("repack", "compact", opts->existing->repo);
+	trace2_region_leave("repack", "steps:write", opts->existing->repo);
+
+	trace2_region_enter("repack", "steps:rest", opts->existing->repo);
+
+	/*
+	 * Then start over, repeat, and either compact or keep as-is
+	 * each MIDX layer until we have exhausted the chain.
+	 *
+	 * Finally, evaluate the remainder of the chain (if any) and
+	 * either compact a sequence of adjacent layers, or keep
+	 * individual layers as-is according to the same merging
+	 * condition as above.
+	 */
+	while (m) {
+		struct multi_pack_index *next = m;
+
+		ALLOC_GROW(steps, steps_nr + 1, steps_alloc);
+
+		memset(&step, 0, sizeof(step));
+		step.type = MIDX_COMPACTION_STEP_UNKNOWN;
+
+		trace2_region_enter("repack", "step", opts->existing->repo);
+
+		trace2_data_string("repack", opts->existing->repo,
+				   "from", midx_get_checksum_hex(m));
+
+		while (next) {
+			uint32_t proposed_objects_nr;
+			if (unsigned_add_overflows(step.objects_nr, next->num_objects)) {
+				ret = error(_("too many objects in MIDX compaction step"));
+				trace2_region_leave("repack", "step", opts->existing->repo);
+				goto out;
+			}
+
+			proposed_objects_nr = step.objects_nr + next->num_objects;
+
+			trace2_data_string("repack", opts->existing->repo,
+					   "proposed",
+					   midx_get_checksum_hex(next));
+			trace2_data_intmax("repack", opts->existing->repo,
+					   "proposed:objects_nr",
+					   (uintmax_t)next->num_objects);
+
+			if (!next->base_midx) {
+				/*
+				 * If we are at the end of the MIDX
+				 * chain, there is nothing to compact,
+				 * so mark it and stop.
+				 */
+				step.objects_nr = proposed_objects_nr;
+				break;
+			}
+
+			if (proposed_objects_nr < next->base_midx->num_objects / opts->midx_split_factor) {
+				/*
+				 * If there is a MIDX following this
+				 * one, but our accumulated size is less
+				 * than half of its size, compacting
+				 * them would violate the merging
+				 * condition, so stop here.
+				 */
+
+				trace2_data_string("repack", opts->existing->repo,
+						   "compact:violated:at",
+						   midx_get_checksum_hex(next->base_midx));
+				trace2_data_intmax("repack", opts->existing->repo,
+						   "compact:violated:at:objects_nr",
+						   (uintmax_t)next->base_midx->num_objects);
+				break;
+			}
+
+			/*
+			 * Otherwise, it is OK to compact the next layer
+			 * into this one. Do so, and then continue
+			 * through the remainder of the chain.
+			 */
+			step.objects_nr = proposed_objects_nr;
+			trace2_data_intmax("repack", opts->existing->repo,
+					   "step:objects_nr",
+					   (uintmax_t)step.objects_nr);
+			next = next->base_midx;
+		}
+
+		if (m == next) {
+			step.type = MIDX_COMPACTION_STEP_COPY;
+			step.u.copy = m;
+
+			trace2_data_string("repack", opts->existing->repo,
+					   "type", "copy");
+		} else {
+			step.type = MIDX_COMPACTION_STEP_COMPACT;
+			step.u.compact.from = next;
+			step.u.compact.to = m;
+
+			trace2_data_string("repack", opts->existing->repo,
+					   "to", midx_get_checksum_hex(m));
+			trace2_data_string("repack", opts->existing->repo,
+					   "type", "compact");
+		}
+
+		m = next->base_midx;
+		steps[steps_nr++] = step;
+		trace2_region_leave("repack", "step", opts->existing->repo);
+	}
+
+	trace2_region_leave("repack", "steps:rest", opts->existing->repo);
+
+out:
+	*steps_p = steps;
+	*steps_nr_p = steps_nr;
+
+	strbuf_release(&buf);
+
+	trace2_region_leave("repack", "make_midx_compaction_plan",
+			    opts->existing->repo);
+
+	return ret;
+}
+
+static int write_midx_incremental(struct repack_write_midx_opts *opts)
+{
+	struct midx_compaction_step *steps = NULL;
+	struct strbuf lock_name = STRBUF_INIT;
+	struct lock_file lf;
+	size_t steps_nr = 0;
+	size_t i;
+	int ret = 0;
+
+	get_midx_chain_filename(opts->existing->source, &lock_name);
+	if (safe_create_leading_directories(opts->existing->repo,
+					    lock_name.buf))
+		die_errno(_("unable to create leading directories of %s"),
+			  lock_name.buf);
+	hold_lock_file_for_update(&lf, lock_name.buf, LOCK_DIE_ON_ERROR);
+
+	if (!fdopen_lock_file(&lf, "w")) {
+		ret = error_errno(_("unable to open multi-pack-index chain file"));
+		goto done;
+	}
+
+	if (repack_make_midx_compaction_plan(opts, &steps, &steps_nr) < 0) {
+		ret = error(_("unable to generate compaction plan"));
+		goto done;
+	}
+
+	for (i = 0; i < steps_nr; i++) {
+		struct midx_compaction_step *step = &steps[i];
+		char *base = NULL;
+
+		if (i + 1 < steps_nr)
+			base = xstrdup(midx_compaction_step_base(&steps[i + 1]));
+
+		if (midx_compaction_step_exec(step, opts, base) < 0) {
+			ret = error(_("unable to execute compaction step %"PRIuMAX),
+				    (uintmax_t)i);
+			free(base);
+			goto done;
+		}
+
+		free(base);
+	}
+
+	i = steps_nr;
+	while (i--) {
+		struct midx_compaction_step *step = &steps[i];
+		if (!step->csum)
+			BUG("missing result for compaction step %"PRIuMAX,
+			    (uintmax_t)i);
+		fprintf(get_lock_file_fp(&lf), "%s\n", step->csum);
+	}
+
+	commit_lock_file(&lf);
+
+done:
+	strbuf_release(&lock_name);
+	for (i = 0; i < steps_nr; i++)
+		midx_compaction_step_release(&steps[i]);
+	free(steps);
+	return ret;
+}
+
 int repack_write_midx(struct repack_write_midx_opts *opts)
 {
 	switch (opts->mode) {
@@ -386,6 +951,8 @@ int repack_write_midx(struct repack_write_midx_opts *opts)
 		BUG("write_midx mode is NONE?");
 	case REPACK_WRITE_MIDX_DEFAULT:
 		return write_midx_included_packs(opts);
+	case REPACK_WRITE_MIDX_INCREMENTAL:
+		return write_midx_incremental(opts);
 	default:
 		BUG("unhandled write_midx mode: %d", opts->mode);
 	}
diff --git a/repack.h b/repack.h
index 81907fcce7f..831ccfb1c6c 100644
--- a/repack.h
+++ b/repack.h
@@ -137,6 +137,7 @@ struct tempfile;
 enum repack_write_midx_mode {
 	REPACK_WRITE_MIDX_NONE,
 	REPACK_WRITE_MIDX_DEFAULT,
+	REPACK_WRITE_MIDX_INCREMENTAL,
 };
 
 struct repack_write_midx_opts {
@@ -148,6 +149,8 @@ struct repack_write_midx_opts {
 	int show_progress;
 	int write_bitmaps;
 	int midx_must_contain_cruft;
+	int midx_split_factor;
+	int midx_new_layer_threshold;
 	enum repack_write_midx_mode mode;
 };
 
-- 
2.54.0.16.g1c05dfce579
Previous: Taylor BlauNext: Taylor Blau
Message 88 of 113 in “repack: incremental MIDX/bitmap-based repacking”
  1. 00/16 repack: incremental MIDX/bitmap-based repackingTaylor Blau, Mar 29, 2026
  2. 01/16 midx-write: handle noop writes when converting incremental chainsTaylor Blau, Mar 29, 2026
  3. Jeff KingMar 30, 2026
  4. Taylor BlauMar 31, 2026
  5. 02/16 midx: use `string_list` for retained MIDX filesTaylor Blau, Mar 29, 2026
  6. Jeff KingMar 30, 2026
  7. Taylor BlauMar 31, 2026
  8. 03/16 strvec: introduce `strvec_init_alloc()`Taylor Blau, Mar 29, 2026
  9. Jeff KingMar 30, 2026
  10. 04/16 midx: use `strvec` for `keep_hashes`Taylor Blau, Mar 29, 2026
  11. Jeff KingMar 30, 2026
  12. Taylor BlauMar 31, 2026
  13. Taylor BlauMar 31, 2026
  14. Jeff KingMar 31, 2026
  15. Taylor BlauApr 1, 2026
  16. Jeff KingApr 1, 2026
  17. 05/16 midx: introduce `--checksum-only` for incremental MIDX writesTaylor Blau, Mar 29, 2026
  18. Jeff KingMar 30, 2026
  19. Taylor BlauApr 2, 2026
  20. 06/16 midx: support custom `--base` for incremental MIDX writesTaylor Blau, Mar 29, 2026
  21. Jeff KingApr 7, 2026
  22. Taylor BlauApr 14, 2026
  23. 07/16 repack: track the ODB source via existing_packsTaylor Blau, Mar 29, 2026
  24. Jeff KingApr 7, 2026
  25. Taylor BlauApr 14, 2026
  26. 08/16 midx: expose `midx_layer_contains_pack()`Taylor Blau, Mar 29, 2026
  27. Jeff KingApr 7, 2026
  28. 09/16 repack-midx: factor out `repack_prepare_midx_command()`Taylor Blau, Mar 29, 2026
  29. 10/16 repack-midx: extract `repack_fill_midx_stdin_packs()`Taylor Blau, Mar 29, 2026
  30. Jeff KingApr 7, 2026
  31. 11/16 repack-geometry: prepare for incremental MIDX repackingTaylor Blau, Mar 29, 2026
  32. Jeff KingApr 7, 2026
  33. Elijah NewrenApr 16, 2026
  34. Taylor BlauApr 21, 2026
  35. 12/16 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`Taylor Blau, Mar 29, 2026
  36. Jeff KingApr 7, 2026
  37. 13/16 packfile: ensure `close_pack_revindex()` frees in-memory revindexTaylor Blau, Mar 29, 2026
  38. Jeff KingApr 7, 2026
  39. 14/16 repack: implement incremental MIDX repackingTaylor Blau, Mar 29, 2026
  40. Elijah NewrenApr 16, 2026
  41. Taylor BlauApr 21, 2026
  42. 15/16 repack: introduce `--write-midx=incremental`Taylor Blau, Mar 29, 2026
  43. Elijah NewrenApr 16, 2026
  44. Taylor BlauApr 21, 2026
  45. 16/16 repack: allow `--write-midx=incremental` without `--geometric`Taylor Blau, Mar 29, 2026
  46. Taylor BlauApr 14, 2026
  47. 00/16 repack: incremental MIDX/bitmap-based repackingTaylor Blau, Apr 21, 2026
  48. 01/16 midx-write: handle noop writes when converting incremental chainsTaylor Blau, Apr 21, 2026
  49. 02/16 midx: use `strset` for retained MIDX filesTaylor Blau, Apr 21, 2026
  50. 03/16 midx: build `keep_hashes` array in orderTaylor Blau, Apr 21, 2026
  51. 04/16 midx: use `strvec` for `keep_hashes`Taylor Blau, Apr 21, 2026
  52. 05/16 midx: introduce `--no-write-chain-file` for incremental MIDX writesTaylor Blau, Apr 21, 2026
  53. 06/16 midx: support custom `--base` for incremental MIDX writesTaylor Blau, Apr 21, 2026
  54. 07/16 repack: track the ODB source via existing_packsTaylor Blau, Apr 21, 2026
  55. 08/16 midx: expose `midx_layer_contains_pack()`Taylor Blau, Apr 21, 2026
  56. 09/16 repack-midx: factor out `repack_prepare_midx_command()`Taylor Blau, Apr 21, 2026
  57. 10/16 repack-midx: extract `repack_fill_midx_stdin_packs()`Taylor Blau, Apr 21, 2026
  58. Jeff KingApr 29, 2026
  59. Taylor BlauApr 29, 2026
  60. 11/16 repack-geometry: prepare for incremental MIDX repackingTaylor Blau, Apr 21, 2026
  61. 12/16 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`Taylor Blau, Apr 21, 2026
  62. 13/16 packfile: ensure `close_pack_revindex()` frees in-memory revindexTaylor Blau, Apr 21, 2026
  63. 14/16 repack: implement incremental MIDX repackingTaylor Blau, Apr 21, 2026
  64. Jeff KingApr 29, 2026
  65. Taylor BlauApr 29, 2026
  66. Jeff KingApr 29, 2026
  67. Taylor BlauApr 29, 2026
  68. 15/16 repack: introduce `--write-midx=incremental`Taylor Blau, Apr 21, 2026
  69. Taylor BlauApr 21, 2026
  70. 16/16 repack: allow `--write-midx=incremental` without `--geometric`Taylor Blau, Apr 21, 2026
  71. Elijah NewrenApr 22, 2026
  72. Jeff KingApr 29, 2026
  73. 00/16 repack: incremental MIDX/bitmap-based repackingTaylor Blau, Apr 30, 2026
  74. 01/16 midx-write: handle noop writes when converting incremental chainsTaylor Blau, Apr 30, 2026
  75. 02/16 midx: use `strset` for retained MIDX filesTaylor Blau, Apr 30, 2026
  76. 03/16 midx: build `keep_hashes` array in orderTaylor Blau, Apr 30, 2026
  77. 04/16 midx: use `strvec` for `keep_hashes`Taylor Blau, Apr 30, 2026
  78. 05/16 midx: introduce `--no-write-chain-file` for incremental MIDX writesTaylor Blau, Apr 30, 2026
  79. 06/16 midx: support custom `--base` for incremental MIDX writesTaylor Blau, Apr 30, 2026
  80. 07/16 repack: track the ODB source via existing_packsTaylor Blau, Apr 30, 2026
  81. 08/16 midx: expose `midx_layer_contains_pack()`Taylor Blau, Apr 30, 2026
  82. 09/16 repack-midx: factor out `repack_prepare_midx_command()`Taylor Blau, Apr 30, 2026
  83. SZEDER GáborMay 13, 2026
  84. 10/16 repack-midx: extract `repack_fill_midx_stdin_packs()`Taylor Blau, Apr 30, 2026
  85. 11/16 repack-geometry: prepare for incremental MIDX repackingTaylor Blau, Apr 30, 2026
  86. 12/16 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`Taylor Blau, Apr 30, 2026
  87. 13/16 packfile: ensure `close_pack_revindex()` frees in-memory revindexTaylor Blau, Apr 30, 2026
  88. 14/16 repack: implement incremental MIDX repackingTaylor Blau, Apr 30, 2026
  89. 15/16 repack: introduce `--write-midx=incremental`Taylor Blau, Apr 30, 2026
  90. Jeff KingMay 13, 2026
  91. 16/16 repack: allow `--write-midx=incremental` without `--geometric`Taylor Blau, Apr 30, 2026
  92. Jeff KingMay 1, 2026
  93. 00/16 repack: incremental MIDX/bitmap-based repackingTaylor Blau, May 19, 2026
  94. 01/16 midx-write: handle noop writes when converting incremental chainsTaylor Blau, May 19, 2026
  95. 02/16 midx: use `strset` for retained MIDX filesTaylor Blau, May 19, 2026
  96. 03/16 midx: build `keep_hashes` array in orderTaylor Blau, May 19, 2026
  97. 04/16 midx: use `strvec` for `keep_hashes`Taylor Blau, May 19, 2026
  98. 05/16 midx: introduce `--no-write-chain-file` for incremental MIDX writesTaylor Blau, May 19, 2026
  99. 06/16 midx: support custom `--base` for incremental MIDX writesTaylor Blau, May 19, 2026
  100. SZEDER GáborJun 12, 2026
  101. Junio C HamanoJun 12, 2026
  102. Taylor BlauJun 12, 2026
  103. Taylor BlauJun 12, 2026
  104. 07/16 repack: track the ODB source via existing_packsTaylor Blau, May 19, 2026
  105. 08/16 midx: expose `midx_layer_contains_pack()`Taylor Blau, May 19, 2026
  106. 09/16 repack-midx: factor out `repack_prepare_midx_command()`Taylor Blau, May 19, 2026
  107. 10/16 repack-midx: extract `repack_fill_midx_stdin_packs()`Taylor Blau, May 19, 2026
  108. 11/16 repack-geometry: prepare for incremental MIDX repackingTaylor Blau, May 19, 2026
  109. 12/16 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`Taylor Blau, May 19, 2026
  110. 13/16 packfile: ensure `close_pack_revindex()` frees in-memory revindexTaylor Blau, May 19, 2026
  111. 14/16 repack: implement incremental MIDX repackingTaylor Blau, May 19, 2026
  112. 15/16 repack: introduce `--write-midx=incremental`Taylor Blau, May 19, 2026
  113. 16/16 repack: allow `--write-midx=incremental` without `--geometric`Taylor Blau, May 19, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.