From: Taylor Blau Date: Sun, 28 Sep 2025 22:07:10 GMT Subject: [PATCH 00/49] repack: prepare for incremental MIDX-based repacking Message-ID: Note(s) to the maintainer: * This series depends on ps/packfile-store at dd52a29b78 (packfile: refactor `get_packed_git_mru()` to work on packfile store, 2025-09-23). * This series is split out of my local copy of tb/incremental-midx-part-3. I suggest queueing it as tb/incremental-midx-part-3.1 ;-). When developing an incremental MIDX/bitmap-based repacking strategy, I thought that it might be useful to clean up builtin/repack.c before adding any amount of new functionality. At more than 1,700+ lines of code, the 'repack' builtin has grown quite unwieldy over the years. It has quite a bit of repack-related functionality, all implemented within the builtin itself. It knows how to: manage cruft packs, perform geometric repacking, write new MIDXs, repack promisor objects, keep track of existing packs, and much more. This series moves the vast majority of the builtin's functionality into several new compilation units, all grouped under a new repack.h API: * repack.c * repack-cruft.c * repack-filtered.c * repack-geometry.c * repack-midx.c * repack-promisor.c Moving this code around was less than straightforward, hence the length of this series. Likewise, since the repacking code can be rather fragile, I chose to go slower here than I otherwise would have (e.g., almost all code movement changes are strict moves with no rewrites in between). The structure of the series is roughly as follows: - The first 10 patches reduce the usage of the_repository, and the_hash_algo. They don't quite get us to being able to compile without USE_THE_REPOSITORY_VARIABLE for reasons described in the last commit of this section. - The next four commits introduces the new repack.o compilation unit, and moves a fairly isolated component of the repack builtin. - The next five commits move the existing_packs API from out of the builtin into the repack.h API. - The next five move the generated_pack API. - The next two move the repack_promisor_objects() API. - The next four move the pack_geometry API. - The next nine move various MIDX-related functionality out of the builtin. - The next nine move remaining functions that write packs out of the repack builtin. - The final commit cleans up the list of `#include`s within the builtin. My hope is that this produces a series that is easier to review, despite its rather long length. Each individual commit tries to do as little as possible, and most will benefit from being reviewed with --color-moved. The result, in my opinion, is a 'builtin/repack.c' is much cleaner than it was when we started. Nothing in this series is a strict requirement for implementing a repacking strategy based on incremental MIDX/bitmaps. But since the new functionality required for that is lengthy in and of itself, having a clean slate to start from seemed worthwhile to me. Thanks in advance for your review! Taylor Blau (49): builtin/repack.c: avoid "the_repository" in `cmd_repack()` builtin/repack.c: avoid "the_repository" in existing packs API builtin/repack.c: avoid "the_repository" when taking a ref snapshot builtin/repack.c: avoid "the_repository" when removing packs builtin/repack.c: avoid "the_repository" when repacking promisor objects builtin/repack.c: avoid "the_hash_algo" when deleting packs builtin/repack.c: avoid "the_hash_algo" in `write_oid()` builtin/repack: avoid "the_hash_algo" in `repack_promisor_objects()` builtin/repack.c: avoid "the_hash_algo" in `finish_pack_objects_cmd()` builtin/repack.c: avoid using `hash_to_hex()` in pack geometry repack: introduce new compilation unit builtin/repack.c: pass both pack_objects args to repack_config repack: move 'delta_base_offset' to 'struct pack_objects_args' repack: remove 'prepare_pack_objects' from the builtin builtin/repack.c: rename many 'struct existing_packs' functions repack: remove 'remove_redundant_pack' from the builtin builtin/repack.c: pass "packdir" when removing packs builtin/repack.c: avoid unnecessary numeric casts in existing_packs repack: remove 'existing_packs' API from the builtin builtin/repack.c: rename "struct generated_pack_data" builtin/repack.c: factor our "generated_pack_install" builtin/repack.c: pass "packtmp" to `generated_pack_populate()` builtin/repack.c: provide pack locations to `generated_pack_install()` repack: remove 'generated_pack' API from the builtin builtin/repack.c: pass "packtmp" to `repack_promisor_objects()` builtin/repack.c: remove "repack_promisor_objects()" from the builtin builtin/repack.c: rename various pack_geometry functions builtin/repack.c: pass 'pack_kept_objects' to `pack_geometry_init()` builtin/repack.c: pass 'packdir' to `pack_geometry_remove_redundant()` repack: remove pack_geometry API from the builtin builtin/repack.c: remove ref snapshotting from builtin builtin/repack.c: extract opts struct for 'write_midx_included_packs()' builtin/repack.c: use a string_list for 'midx_pack_names' repack: keep track of MIDX pack names using existing_packs builtin/repack.c: reorder `remove_redundant_bitmaps()` builtin/repack.c: inline `remove_redundant_bitmaps()` builtin/repack.c: pass `repack_write_midx_opts` to `midx_included_packs` builtin/repack.c: inline packs within `write_midx_included_packs()` repack: 'write_midx_included_packs' API from the builtin builtin/repack.c: introduce `struct write_pack_opts` builtin/repack.c: use `write_pack_opts` within `write_cruft_pack()` repack: move `find_pack_prefix()` out of the builtin repack: extract `write_pack_opts_is_local()` builtin/repack.c: pass `write_pack_opts` to `finish_pack_objects_cmd()` repack: move `finish_pack_objects_cmd()` out of the builtin repack: move `pack_kept_objects` to `struct pack_objects_args` repack: move `write_filtered_pack()` out of the builtin repack: move `write_cruft_pack()` out of the builtin builtin/repack.c: clean up unused `#include`s Makefile | 6 + builtin/repack.c | 1348 +++------------------------------------------ meson.build | 6 + repack-cruft.c | 99 ++++ repack-filtered.c | 51 ++ repack-geometry.c | 232 ++++++++ repack-midx.c | 372 +++++++++++++ repack-promisor.c | 102 ++++ repack.c | 361 ++++++++++++ repack.h | 146 +++++ 10 files changed, 1461 insertions(+), 1262 deletions(-) create mode 100644 repack-cruft.c create mode 100644 repack-filtered.c create mode 100644 repack-geometry.c create mode 100644 repack-midx.c create mode 100644 repack-promisor.c create mode 100644 repack.c create mode 100644 repack.h base-commit: dd52a29b78d80e425be660f3b443a42e0374a7d1 -- 2.51.0.243.g16eca91f2c0