git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 00/16] repack: incremental MIDX/bitmap-based repacking

From
Taylor Blau <me@ttaylorr.com>
Date
May 19, 2026, 15:57 UTC
Message-ID
<cover.1779206239.git.me@ttaylorr.com>
In-Reply-To
<cover.1774820449.git.me@ttaylorr.com>

Here is another small reroll of my series to implement the last remaining component of the incremental MIDX/bitmap-based repacking strategy [1].

The changes since v3 are fairly small:
 - `repack_prepare_midx_command()` now calls its third parameter
   "subcommand" instead of "verb".
 - `clear_incremental_midx_files()` now avoids a redundant conditional
   around `r->objects` and reuses a local `struct odb_source *` when
   closing any open incremental MIDX handles.

As usual, a range-diff is included below for convenience. Thanks in advance for reviewing!

[1]: https://lore.kernel.org/git/cover.1777507303.git.me@ttaylorr.com/
Taylor Blau (16):
  midx-write: handle noop writes when converting incremental chains
  midx: use `strset` for retained MIDX files
  midx: build `keep_hashes` array in order
  midx: use `strvec` for `keep_hashes`
  midx: introduce `--no-write-chain-file` for incremental MIDX writes
  midx: support custom `--base` for incremental MIDX writes
  repack: track the ODB source via existing_packs
  midx: expose `midx_layer_contains_pack()`
  repack-midx: factor out `repack_prepare_midx_command()`
  repack-midx: extract `repack_fill_midx_stdin_packs()`
  repack-geometry: prepare for incremental MIDX repacking
  builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`
  packfile: ensure `close_pack_revindex()` frees in-memory revindex
  repack: implement incremental MIDX repacking
  repack: introduce `--write-midx=incremental`
  repack: allow `--write-midx=incremental` without `--geometric`
 Documentation/config/repack.adoc        |  18 +
 Documentation/git-multi-pack-index.adoc |  32 +-
 Documentation/git-repack.adoc           |  44 +-
 builtin/multi-pack-index.c              |  48 +-
 builtin/repack.c                        | 102 +++-
 midx-write.c                            | 206 ++++---
 midx.c                                  | 102 ++--
 midx.h                                  |  11 +-
 packfile.c                              |   2 +
 repack-geometry.c                       |  48 +-
 repack-midx.c                           | 710 +++++++++++++++++++++++-
 repack.c                                |  58 +-
 repack.h                                |  26 +-
 t/meson.build                           |   1 +
 t/t5334-incremental-multi-pack-index.sh |  63 +++
 t/t5335-compact-multi-pack-index.sh     | 113 ++++
 t/t7705-repack-incremental-midx.sh      | 525 ++++++++++++++++++
 17 files changed, 1909 insertions(+), 200 deletions(-)
 create mode 100755 t/t7705-repack-incremental-midx.sh
Range-diff against v3:
 1:  d6c27317c25 =  1:  ead11e610c8 midx-write: handle noop writes when converting incremental chains
 2:  629c8d23116 =  2:  ece55bf2957 midx: use `strset` for retained MIDX files
 3:  e303bf6a4ac =  3:  5609d1941e6 midx: build `keep_hashes` array in order
 4:  42d76c70060 =  4:  13b7c808860 midx: use `strvec` for `keep_hashes`
 5:  2c80aa34fac =  5:  cac3fd54bf0 midx: introduce `--no-write-chain-file` for incremental MIDX writes
 6:  2a05f4b86f3 =  6:  1bbb387d6b6 midx: support custom `--base` for incremental MIDX writes
 7:  92aba3d366f =  7:  4a93adb3ad3 repack: track the ODB source via existing_packs
 8:  d3ac65c1f11 =  8:  8d1b8b1d301 midx: expose `midx_layer_contains_pack()`
 9:  1bd2f194c6f !  9:  42111e5f75d repack-midx: factor out `repack_prepare_midx_command()`
    @@ Commit message
         subcommands), so extract the common portions of the command setup into a
         reusable `repack_prepare_midx_command()` helper.
     
    -    The extracted helper sets `git_cmd`, pushes the `multi-pack-index`
    -    subcommand and verb, and handles `--progress`/`--no-progress` and
    -    `--bitmap` flags. The remaining arguments that are specific to the
    -    `write` subcommand (such as `--stdin-packs`) are left to the caller.
    +    The extracted helper sets `git_cmd`, pushes `multi-pack-index` and a
    +    subcommand, and handles `--progress`/`--no-progress` and `--bitmap`
    +    flags. The remaining arguments that are specific to the `write`
    +    subcommand (such as `--stdin-packs`) are left to the caller.
     
         No functional changes are included in this patch.
     
    @@ repack-midx.c: static void remove_redundant_bitmaps(struct string_list *include,
      
     +static void repack_prepare_midx_command(struct child_process *cmd,
     +					struct repack_write_midx_opts *opts,
    -+					const char *verb)
    ++					const char *subcommand)
     +{
     +	cmd->git_cmd = 1;
     +
    -+	strvec_pushl(&cmd->args, "multi-pack-index", verb, NULL);
    ++	strvec_pushl(&cmd->args, "multi-pack-index", subcommand, NULL);
     +
     +	if (opts->show_progress)
     +		strvec_push(&cmd->args, "--progress");
10:  2a87a1e4561 = 10:  ed76e6efd1c repack-midx: extract `repack_fill_midx_stdin_packs()`
11:  3d32b9c88da = 11:  9665f1b3a64 repack-geometry: prepare for incremental MIDX repacking
12:  1f7a5479bb8 = 12:  e0db62b9f10 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`
13:  b155f25d53c = 13:  c8c846b1ac1 packfile: ensure `close_pack_revindex()` frees in-memory revindex
14:  ef012314930 = 14:  78f1a98c1ea repack: implement incremental MIDX repacking
15:  04cfecd5136 ! 15:  95bf8b21fff repack: introduce `--write-midx=incremental`
    @@ midx.c: void clear_midx_file(struct repository *r)
     +void clear_incremental_midx_files(struct repository *r,
     +				  const struct strvec *keep_hashes)
     +{
    ++	struct odb_source *source = r->objects->sources;
     +	struct strbuf chain = STRBUF_INIT;
     +
    -+	get_midx_chain_filename(r->objects->sources, &chain);
    ++	get_midx_chain_filename(source, &chain);
     +
    -+	if (r->objects) {
    -+		struct odb_source *source = r->objects->sources;
    -+		for (source = r->objects->sources; source; source = source->next) {
    -+			struct odb_source_files *files = odb_source_files_downcast(source);
    -+			if (files->packed->midx)
    -+				close_midx(files->packed->midx);
    -+			files->packed->midx = NULL;
    -+		}
    ++	for (; source; source = source->next) {
    ++		struct odb_source_files *files = odb_source_files_downcast(source);
    ++		if (files->packed->midx)
    ++			close_midx(files->packed->midx);
    ++		files->packed->midx = NULL;
     +	}
     +
     +	if (!keep_hashes && remove_path(chain.buf))
16:  1c05dfce579 = 16:  8bd0ec98dc3 repack: allow `--write-midx=incremental` without `--geometric`
base-commit: 7bcaabddcf68bd0702697da5904c3b68c52f94cf
-- 
2.54.0.175.g8bd0ec98dc3
Previous: Jeff KingNext: Taylor Blau
Message 93 of 113 in “repack: incremental MIDX/bitmap-based repacking”
  1. 00/16 repack: incremental MIDX/bitmap-based repackingTaylor Blau, Mar 29, 2026
  2. 01/16 midx-write: handle noop writes when converting incremental chainsTaylor Blau, Mar 29, 2026
  3. Jeff KingMar 30, 2026
  4. Taylor BlauMar 31, 2026
  5. 02/16 midx: use `string_list` for retained MIDX filesTaylor Blau, Mar 29, 2026
  6. Jeff KingMar 30, 2026
  7. Taylor BlauMar 31, 2026
  8. 03/16 strvec: introduce `strvec_init_alloc()`Taylor Blau, Mar 29, 2026
  9. Jeff KingMar 30, 2026
  10. 04/16 midx: use `strvec` for `keep_hashes`Taylor Blau, Mar 29, 2026
  11. Jeff KingMar 30, 2026
  12. Taylor BlauMar 31, 2026
  13. Taylor BlauMar 31, 2026
  14. Jeff KingMar 31, 2026
  15. Taylor BlauApr 1, 2026
  16. Jeff KingApr 1, 2026
  17. 05/16 midx: introduce `--checksum-only` for incremental MIDX writesTaylor Blau, Mar 29, 2026
  18. Jeff KingMar 30, 2026
  19. Taylor BlauApr 2, 2026
  20. 06/16 midx: support custom `--base` for incremental MIDX writesTaylor Blau, Mar 29, 2026
  21. Jeff KingApr 7, 2026
  22. Taylor BlauApr 14, 2026
  23. 07/16 repack: track the ODB source via existing_packsTaylor Blau, Mar 29, 2026
  24. Jeff KingApr 7, 2026
  25. Taylor BlauApr 14, 2026
  26. 08/16 midx: expose `midx_layer_contains_pack()`Taylor Blau, Mar 29, 2026
  27. Jeff KingApr 7, 2026
  28. 09/16 repack-midx: factor out `repack_prepare_midx_command()`Taylor Blau, Mar 29, 2026
  29. 10/16 repack-midx: extract `repack_fill_midx_stdin_packs()`Taylor Blau, Mar 29, 2026
  30. Jeff KingApr 7, 2026
  31. 11/16 repack-geometry: prepare for incremental MIDX repackingTaylor Blau, Mar 29, 2026
  32. Jeff KingApr 7, 2026
  33. Elijah NewrenApr 16, 2026
  34. Taylor BlauApr 21, 2026
  35. 12/16 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`Taylor Blau, Mar 29, 2026
  36. Jeff KingApr 7, 2026
  37. 13/16 packfile: ensure `close_pack_revindex()` frees in-memory revindexTaylor Blau, Mar 29, 2026
  38. Jeff KingApr 7, 2026
  39. 14/16 repack: implement incremental MIDX repackingTaylor Blau, Mar 29, 2026
  40. Elijah NewrenApr 16, 2026
  41. Taylor BlauApr 21, 2026
  42. 15/16 repack: introduce `--write-midx=incremental`Taylor Blau, Mar 29, 2026
  43. Elijah NewrenApr 16, 2026
  44. Taylor BlauApr 21, 2026
  45. 16/16 repack: allow `--write-midx=incremental` without `--geometric`Taylor Blau, Mar 29, 2026
  46. Taylor BlauApr 14, 2026
  47. 00/16 repack: incremental MIDX/bitmap-based repackingTaylor Blau, Apr 21, 2026
  48. 01/16 midx-write: handle noop writes when converting incremental chainsTaylor Blau, Apr 21, 2026
  49. 02/16 midx: use `strset` for retained MIDX filesTaylor Blau, Apr 21, 2026
  50. 03/16 midx: build `keep_hashes` array in orderTaylor Blau, Apr 21, 2026
  51. 04/16 midx: use `strvec` for `keep_hashes`Taylor Blau, Apr 21, 2026
  52. 05/16 midx: introduce `--no-write-chain-file` for incremental MIDX writesTaylor Blau, Apr 21, 2026
  53. 06/16 midx: support custom `--base` for incremental MIDX writesTaylor Blau, Apr 21, 2026
  54. 07/16 repack: track the ODB source via existing_packsTaylor Blau, Apr 21, 2026
  55. 08/16 midx: expose `midx_layer_contains_pack()`Taylor Blau, Apr 21, 2026
  56. 09/16 repack-midx: factor out `repack_prepare_midx_command()`Taylor Blau, Apr 21, 2026
  57. 10/16 repack-midx: extract `repack_fill_midx_stdin_packs()`Taylor Blau, Apr 21, 2026
  58. Jeff KingApr 29, 2026
  59. Taylor BlauApr 29, 2026
  60. 11/16 repack-geometry: prepare for incremental MIDX repackingTaylor Blau, Apr 21, 2026
  61. 12/16 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`Taylor Blau, Apr 21, 2026
  62. 13/16 packfile: ensure `close_pack_revindex()` frees in-memory revindexTaylor Blau, Apr 21, 2026
  63. 14/16 repack: implement incremental MIDX repackingTaylor Blau, Apr 21, 2026
  64. Jeff KingApr 29, 2026
  65. Taylor BlauApr 29, 2026
  66. Jeff KingApr 29, 2026
  67. Taylor BlauApr 29, 2026
  68. 15/16 repack: introduce `--write-midx=incremental`Taylor Blau, Apr 21, 2026
  69. Taylor BlauApr 21, 2026
  70. 16/16 repack: allow `--write-midx=incremental` without `--geometric`Taylor Blau, Apr 21, 2026
  71. Elijah NewrenApr 22, 2026
  72. Jeff KingApr 29, 2026
  73. 00/16 repack: incremental MIDX/bitmap-based repackingTaylor Blau, Apr 30, 2026
  74. 01/16 midx-write: handle noop writes when converting incremental chainsTaylor Blau, Apr 30, 2026
  75. 02/16 midx: use `strset` for retained MIDX filesTaylor Blau, Apr 30, 2026
  76. 03/16 midx: build `keep_hashes` array in orderTaylor Blau, Apr 30, 2026
  77. 04/16 midx: use `strvec` for `keep_hashes`Taylor Blau, Apr 30, 2026
  78. 05/16 midx: introduce `--no-write-chain-file` for incremental MIDX writesTaylor Blau, Apr 30, 2026
  79. 06/16 midx: support custom `--base` for incremental MIDX writesTaylor Blau, Apr 30, 2026
  80. 07/16 repack: track the ODB source via existing_packsTaylor Blau, Apr 30, 2026
  81. 08/16 midx: expose `midx_layer_contains_pack()`Taylor Blau, Apr 30, 2026
  82. 09/16 repack-midx: factor out `repack_prepare_midx_command()`Taylor Blau, Apr 30, 2026
  83. SZEDER GáborMay 13, 2026
  84. 10/16 repack-midx: extract `repack_fill_midx_stdin_packs()`Taylor Blau, Apr 30, 2026
  85. 11/16 repack-geometry: prepare for incremental MIDX repackingTaylor Blau, Apr 30, 2026
  86. 12/16 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`Taylor Blau, Apr 30, 2026
  87. 13/16 packfile: ensure `close_pack_revindex()` frees in-memory revindexTaylor Blau, Apr 30, 2026
  88. 14/16 repack: implement incremental MIDX repackingTaylor Blau, Apr 30, 2026
  89. 15/16 repack: introduce `--write-midx=incremental`Taylor Blau, Apr 30, 2026
  90. Jeff KingMay 13, 2026
  91. 16/16 repack: allow `--write-midx=incremental` without `--geometric`Taylor Blau, Apr 30, 2026
  92. Jeff KingMay 1, 2026
  93. 00/16 repack: incremental MIDX/bitmap-based repackingTaylor Blau, May 19, 2026
  94. 01/16 midx-write: handle noop writes when converting incremental chainsTaylor Blau, May 19, 2026
  95. 02/16 midx: use `strset` for retained MIDX filesTaylor Blau, May 19, 2026
  96. 03/16 midx: build `keep_hashes` array in orderTaylor Blau, May 19, 2026
  97. 04/16 midx: use `strvec` for `keep_hashes`Taylor Blau, May 19, 2026
  98. 05/16 midx: introduce `--no-write-chain-file` for incremental MIDX writesTaylor Blau, May 19, 2026
  99. 06/16 midx: support custom `--base` for incremental MIDX writesTaylor Blau, May 19, 2026
  100. SZEDER GáborJun 12, 2026
  101. Junio C HamanoJun 12, 2026
  102. Taylor BlauJun 12, 2026
  103. Taylor BlauJun 12, 2026
  104. 07/16 repack: track the ODB source via existing_packsTaylor Blau, May 19, 2026
  105. 08/16 midx: expose `midx_layer_contains_pack()`Taylor Blau, May 19, 2026
  106. 09/16 repack-midx: factor out `repack_prepare_midx_command()`Taylor Blau, May 19, 2026
  107. 10/16 repack-midx: extract `repack_fill_midx_stdin_packs()`Taylor Blau, May 19, 2026
  108. 11/16 repack-geometry: prepare for incremental MIDX repackingTaylor Blau, May 19, 2026
  109. 12/16 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`Taylor Blau, May 19, 2026
  110. 13/16 packfile: ensure `close_pack_revindex()` frees in-memory revindexTaylor Blau, May 19, 2026
  111. 14/16 repack: implement incremental MIDX repackingTaylor Blau, May 19, 2026
  112. 15/16 repack: introduce `--write-midx=incremental`Taylor Blau, May 19, 2026
  113. 16/16 repack: allow `--write-midx=incremental` without `--geometric`Taylor Blau, May 19, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.