Patrick SteinhardtHi,
right now, it is left to the caller of `refs_pack_refs()` to decide whether or not to repack refs and how exactly to do it. This is inefficient at times given that the caller typically has no idea whether or not the refdb is close to optimal already. So even if the refdb is already in a close-to-optimal state we would end up repacking all of it.
This patch series aims to address this shortcoming. It introduces a new flag `PACK_REFS_AUTO` that tells the ref backend to repack refs as needed. For the "files" backend we don't honor this flag yet and thus end up behaving the exact same as if that flag wasn't set. But for the "reftable" backend we will use the same auto-compaction algorithm that is already used during writes to the refdb. Thus, in most of the cases it wouldn't actually do anything except when the refdb is suboptimally packed.
Eventually we'll probably also want to wire up heuristics for the "files" backend to honor the `PACK_REFS_AUTO` flag. Using something like a ratio of packed-refs size/number of loose refs might be viable. I punted on that though as I feared that it might lead to bikeshedding and thus distract from the main goal of this topic, which is to prepare the ref code and relevant commands to perform optimizations as required. I'm happy to add such a patch to this series though in case anybody feels strongly about this.
The `PACK_REFS_AUTO` flag is exposed via a new `git pack-refs --auto` flag. It is wired up in both `git gc --auto` and `git maintenance run --auto`.
The series is structured as follows:
- Patches 1 - 5: Bugfixes and improvements for the reftable-specific
compaction code. These are issues that I've found while working on
this series. - Patches 6 - 8: Refactorings to drop the `PACK_REFS_ALL` flag,
which isn't actually used by the ref backends anymore and confused
me multiple times. - Patches 9 - 15: Handling of `PACK_REFS_ALL` in git-pack-refs(1),
git-gc(1) and git-maintenance(1).The patch series is built on top of 2953d95d40 (The eighth batch, 2024-03-15) with "ps/reftable-stack-tempfile" at 60c4c42515 (reftable/stack: register compacted tables as tempfiles, 2024-03-07) merged into it due to a merge conflict in "reftable/stack.c".
Patrick
PS: I polished this patch series while traveling and am still out of
office until Thursday. I'll thus only get to respond to threads that
are waiting for my input at the end of this week.Patrick Steinhardt (15): reftable/stack: fix error handling in `reftable_stack_init_addition()` reftable/error: discern locked/outdated errors reftable/stack: use error codes when locking fails during compaction reftable/stack: gracefully handle failed auto-compaction due to locks refs/reftable: print errors on compaction failure t/helper: drop pack-refs wrapper refs: move `struct pack_refs_opts` to where it's used refs: remove `PACK_REFS_ALL` flag refs/reftable: expose auto compaction via new flag builtin/pack-refs: release allocated memory builtin/pack-refs: introduce new "--auto" flag builtin/gc: move `struct maintenance_run_opts` t6500: extract objects with "17" prefix builtin/gc: forward git-gc(1)'s `--auto` flag when packing refs builtin/gc: pack refs when using `git maintenance run --auto`
Documentation/git-pack-refs.txt | 15 +++++- builtin/gc.c | 86 +++++++++++++++++++-------------- builtin/pack-refs.c | 31 +++++++----- refs.h | 20 ++++---- refs/reftable-backend.c | 11 ++++- reftable/error.c | 4 +- reftable/reftable-error.h | 5 +- reftable/stack.c | 34 +++++++------ reftable/stack_test.c | 2 +- t/helper/test-ref-store.c | 20 -------- t/oid-info/hash-info | 12 +++++ t/t0601-reffiles-pack-refs.sh | 30 ++++++++++-- t/t0610-reftable-basics.sh | 79 ++++++++++++++++++++++++++++++ t/t6500-gc.sh | 30 +++--------- 14 files changed, 255 insertions(+), 124 deletions(-)
base-commit: f94fee98b456ef39a5f85fff5802f4e538a4dc7b
-- 2.44.0
