From: Patrick Steinhardt Date: Fri, 24 Oct 2025 06:57:13 GMT Subject: [PATCH v3 00/10] builtin/maintenance: introduce "geometric" strategy Message-ID: <20251024-pks-maintenance-geometric-strategy-v3-0-9b5b3bdb4387@pks.im> In-Reply-To: <20251016-pks-maintenance-geometric-strategy-v1-0-18943d474203@pks.im> Hi, by default, git-maintenance(1) uses git-gc(1) to perform repository housekeeping. This tool has a couple of shortcomings, most importantly that it regularly does all-into-one repacks. This doesn't really work all that well in the context of monorepos, where you really want to avoid repacking all objects regularly. An alternative maintenance strategy is the "incremental" strategy, but this strategy has two downsides: - Strategies in general only apply to scheduled maintenance. So if you run git-maintenance(1), you still end up with git-gc(1). - The strategy is designed to not ever delete any data, but a full replacment for git-gc(1) needs to also prune reflogs, rereree caches and vanished worktrees. This patch series aims to fix both of these issues. First, the series introduces a new "geometric" maintenance task, which makes use of geometric repacking as exposed by git-repack(1) in the general case. In the case where a geometric repack ends up merging all packfiles into one we instead do an all-into-one repack with cruft packs so that we can still phase out objects over time. Second, the series extends maintenance strategies to also cover normal maintenance. If the user has configured the "geometric" strategy, we'll thus use it for both manual and scheduled maintenance. For backwards compatibility, the "incremental" strategy is changed so that it uses git-gc(1) for manual maintenance and the other tasks for scheduled maintenance. The series is built on top of b660e2dcb9 (Sync with 'maint', 2025-10-14) with tb/incremental-midx-part-3.1 at c886af90f8 (SQUASH??? play well with other topics by preemptively including "repository.h", 2025-09-29) merged into it. Changes in v3: - More line wrapping. - Improve readability of maintenance strategies by using nested designated initializers. - Use git-count-object(1) to count loose objects. - Link to v2: https://lore.kernel.org/r/20251021-pks-maintenance-geometric-strategy-v2-0-f0d727832b80@pks.im Changes in v2: - Make the geometric factor configurable via "maintenance.geometric-repack.splitFactor". - Wrap some overly long lines in our tests. - Link to v1: https://lore.kernel.org/r/20251016-pks-maintenance-geometric-strategy-v1-0-18943d474203@pks.im Thanks! Patrick --- Patrick Steinhardt (10): builtin/gc: remove global `repack` variable builtin/gc: make `too_many_loose_objects()` reusable without GC config builtin/maintenance: introduce "geometric-repack" task builtin/maintenance: make the geometric factor configurable builtin/maintenance: don't silently ignore invalid strategy builtin/maintenance: improve readability of strategies builtin/maintenance: run maintenance tasks depending on type builtin/maintenance: extend "maintenance.strategy" to manual maintenance builtin/maintenance: make "gc" strategy accessible builtin/maintenance: introduce "geometric" strategy Documentation/config/maintenance.adoc | 49 +++++- builtin/gc.c | 313 ++++++++++++++++++++++++++++------ t/t7900-maintenance.sh | 245 ++++++++++++++++++++++++++ 3 files changed, 544 insertions(+), 63 deletions(-) Range-diff versus v2: 1: b853ba54dca = 1: c35408a33d0 builtin/gc: remove global `repack` variable 2: 9bbdfe1b9e5 = 2: be572fe1542 builtin/gc: make `too_many_loose_objects()` reusable without GC config 3: bcd82ad038e ! 3: 5290f6d3e0f builtin/maintenance: introduce "geometric-repack" task @@ t/t7900-maintenance.sh: test_expect_success 'maintenance.incremental-repack.auto + test_line_count = "$EXPECTED_PACKS" packfiles && + + # And verify that there are no loose objects anymore. -+ cat >expect <<-\EOF && -+ info -+ pack -+ EOF -+ ls .git/objects >actual && -+ test_cmp expect actual ++ git count-objects -v >count && ++ test_grep '^count: 0$' count +} + +test_expect_success 'geometric repacking task' ' @@ t/t7900-maintenance.sh: test_expect_success 'maintenance.incremental-repack.auto + # The initial repack causes an all-into-one repack. + GIT_TRACE2_EVENT="$(pwd)/initial-repack.txt" \ + git maintenance run --task=geometric-repack 2>/dev/null && -+ test_subcommand git repack -d -l --cruft --cruft-expiration=2.weeks.ago --quiet --write-midx /dev/null && -+ test_subcommand git repack -d -l --cruft --cruft-expiration=2.weeks.ago --quiet --write-midx /dev/null && -+ test_subcommand git repack -d -l --cruft --cruft-expiration=2.weeks.ago --quiet --write-midx packs && + test_line_count = 2 packs && + ls .git/objects/pack/*.mtimes >cruft && 4: cb10031cc7c = 4: 7f2067fa4ec builtin/maintenance: make the geometric factor configurable 5: 7e8f83d4753 = 5: 7a76003215e builtin/maintenance: don't silently ignore invalid strategy -: ----------- > 6: a6383d121b2 builtin/maintenance: improve readability of strategies 6: 4217c37c0bf ! 7: e25c878a3ff builtin/maintenance: run maintenance tasks depending on type @@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts enum schedule_priority schedule; } tasks[TASK__COUNT]; }; - - static const struct maintenance_strategy none_strategy = { 0 }; -+ +@@ builtin/gc.c: static const struct maintenance_strategy none_strategy = { 0 }; static const struct maintenance_strategy default_strategy = { .tasks = { -- [TASK_GC].enabled = 1, -+ [TASK_GC].type = MAINTENANCE_TYPE_MANUAL, + [TASK_GC] = { +- .enabled = 1, ++ .type = MAINTENANCE_TYPE_MANUAL, + }, }, }; -+ +@@ builtin/gc.c: static const struct maintenance_strategy default_strategy = { static const struct maintenance_strategy incremental_strategy = { .tasks = { -- [TASK_COMMIT_GRAPH].enabled = 1, -+ [TASK_COMMIT_GRAPH].type = MAINTENANCE_TYPE_SCHEDULED, - [TASK_COMMIT_GRAPH].schedule = SCHEDULE_HOURLY, -- [TASK_PREFETCH].enabled = 1, -+ [TASK_PREFETCH].type = MAINTENANCE_TYPE_SCHEDULED, - [TASK_PREFETCH].schedule = SCHEDULE_HOURLY, -- [TASK_INCREMENTAL_REPACK].enabled = 1, -+ [TASK_INCREMENTAL_REPACK].type = MAINTENANCE_TYPE_SCHEDULED, - [TASK_INCREMENTAL_REPACK].schedule = SCHEDULE_DAILY, -- [TASK_LOOSE_OBJECTS].enabled = 1, -+ [TASK_LOOSE_OBJECTS].type = MAINTENANCE_TYPE_SCHEDULED, - [TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY, -- [TASK_PACK_REFS].enabled = 1, -+ [TASK_PACK_REFS].type = MAINTENANCE_TYPE_SCHEDULED, - [TASK_PACK_REFS].schedule = SCHEDULE_WEEKLY, + [TASK_COMMIT_GRAPH] = { +- .enabled = 1, ++ .type = MAINTENANCE_TYPE_SCHEDULED, + .schedule = SCHEDULE_HOURLY, + }, + [TASK_PREFETCH] = { +- .enabled = 1, ++ .type = MAINTENANCE_TYPE_SCHEDULED, + .schedule = SCHEDULE_HOURLY, + }, + [TASK_INCREMENTAL_REPACK] = { +- .enabled = 1, ++ .type = MAINTENANCE_TYPE_SCHEDULED, + .schedule = SCHEDULE_DAILY, + }, + [TASK_LOOSE_OBJECTS] = { +- .enabled = 1, ++ .type = MAINTENANCE_TYPE_SCHEDULED, + .schedule = SCHEDULE_DAILY, + }, + [TASK_PACK_REFS] = { +- .enabled = 1, ++ .type = MAINTENANCE_TYPE_SCHEDULED, + .schedule = SCHEDULE_WEEKLY, + }, }, - }; @@ builtin/gc.c: static void initialize_task_config(struct maintenance_run_opts *opts, { struct strbuf config_name = STRBUF_INIT; 7: 422b16a62a2 ! 8: ba147c3bf33 builtin/maintenance: extend "maintenance.strategy" to manual maintenance @@ Documentation/config/maintenance.adoc: detach. ## builtin/gc.c ## @@ builtin/gc.c: static const struct maintenance_strategy incremental_strategy = { - [TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY, - [TASK_PACK_REFS].type = MAINTENANCE_TYPE_SCHEDULED, - [TASK_PACK_REFS].schedule = SCHEDULE_WEEKLY, -+ + .type = MAINTENANCE_TYPE_SCHEDULED, + .schedule = SCHEDULE_WEEKLY, + }, + /* + * Historically, the "incremental" strategy was only available + * in the context of scheduled maintenance when set up via @@ builtin/gc.c: static const struct maintenance_strategy incremental_strategy = { + * requested. This is the same as the default strategy, which + * would have been in use beforehand. + */ -+ [TASK_GC].type = MAINTENANCE_TYPE_MANUAL, ++ [TASK_GC] = { ++ .type = MAINTENANCE_TYPE_MANUAL, ++ }, }, }; 8: 07f5b32a22e ! 9: eebfab4acda builtin/maintenance: make "gc" strategy accessible @@ builtin/gc.c: struct maintenance_strategy { -static const struct maintenance_strategy default_strategy = { +static const struct maintenance_strategy gc_strategy = { .tasks = { -- [TASK_GC].type = MAINTENANCE_TYPE_MANUAL, -+ [TASK_GC].type = MAINTENANCE_TYPE_MANUAL | MAINTENANCE_TYPE_SCHEDULED, -+ [TASK_GC].schedule = SCHEDULE_DAILY, + [TASK_GC] = { +- .type = MAINTENANCE_TYPE_MANUAL, ++ .type = MAINTENANCE_TYPE_MANUAL | MAINTENANCE_TYPE_SCHEDULED, ++ .schedule = SCHEDULE_DAILY, + }, }, }; - @@ builtin/gc.c: static struct maintenance_strategy parse_maintenance_strategy(const char *name) { if (!strcasecmp(name, "incremental")) 9: c597ae7f94a ! 10: 936358736f3 builtin/maintenance: introduce "geometric" strategy @@ builtin/gc.c: static const struct maintenance_strategy incremental_strategy = { +static const struct maintenance_strategy geometric_strategy = { + .tasks = { -+ [TASK_COMMIT_GRAPH].type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, -+ [TASK_COMMIT_GRAPH].schedule = SCHEDULE_HOURLY, -+ [TASK_GEOMETRIC_REPACK].type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, -+ [TASK_GEOMETRIC_REPACK].schedule = SCHEDULE_DAILY, -+ [TASK_PACK_REFS].type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, -+ [TASK_PACK_REFS].schedule = SCHEDULE_DAILY, -+ [TASK_RERERE_GC].type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, -+ [TASK_RERERE_GC].schedule = SCHEDULE_WEEKLY, -+ [TASK_REFLOG_EXPIRE].type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, -+ [TASK_REFLOG_EXPIRE].schedule = SCHEDULE_WEEKLY, -+ [TASK_WORKTREE_PRUNE].type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, -+ [TASK_WORKTREE_PRUNE].schedule = SCHEDULE_WEEKLY, ++ [TASK_COMMIT_GRAPH] = { ++ .type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, ++ .schedule = SCHEDULE_HOURLY, ++ }, ++ [TASK_GEOMETRIC_REPACK] = { ++ .type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, ++ .schedule = SCHEDULE_DAILY, ++ }, ++ [TASK_PACK_REFS] = { ++ .type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, ++ .schedule = SCHEDULE_DAILY, ++ }, ++ [TASK_RERERE_GC] = { ++ .type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, ++ .schedule = SCHEDULE_WEEKLY, ++ }, ++ [TASK_REFLOG_EXPIRE] = { ++ .type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, ++ .schedule = SCHEDULE_WEEKLY, ++ }, ++ [TASK_WORKTREE_PRUNE] = { ++ .type = MAINTENANCE_TYPE_SCHEDULED | MAINTENANCE_TYPE_MANUAL, ++ .schedule = SCHEDULE_WEEKLY, ++ }, + }, +}; + --- base-commit: 0bb2c786c2349dd6700727153c13d81cbfb41710 change-id: 20251015-pks-maintenance-geometric-strategy-580c58581b01