[PATCH 00/15] refs: introduce `--auto` to pack refs as needed

Hi,

right now, it is left to the caller of `refs_pack_refs()` to decide whether or not to repack refs and how exactly to do it. This is inefficient at times given that the caller typically has no idea whether or not the refdb is close to optimal already. So even if the refdb is already in a close-to-optimal state we would end up repacking all of it.

This patch series aims to address this shortcoming. It introduces a new flag `PACK_REFS_AUTO` that tells the ref backend to repack refs as needed. For the "files" backend we don't honor this flag yet and thus end up behaving the exact same as if that flag wasn't set. But for the "reftable" backend we will use the same auto-compaction algorithm that is already used during writes to the refdb. Thus, in most of the cases it wouldn't actually do anything except when the refdb is suboptimally packed.

Eventually we'll probably also want to wire up heuristics for the "files" backend to honor the `PACK_REFS_AUTO` flag. Using something like a ratio of packed-refs size/number of loose refs might be viable. I punted on that though as I feared that it might lead to bikeshedding and thus distract from the main goal of this topic, which is to prepare the ref code and relevant commands to perform optimizations as required. I'm happy to add such a patch to this series though in case anybody feels strongly about this.

The `PACK_REFS_AUTO` flag is exposed via a new `git pack-refs --auto` flag. It is wired up in both `git gc --auto` and `git maintenance run --auto`.

The series is structured as follows:

    - Patches 1 - 5: Bugfixes and improvements for the reftable-specific
      compaction code. These are issues that I've found while working on
      this series.

    - Patches 6 - 8: Refactorings to drop the `PACK_REFS_ALL` flag,
      which isn't actually used by the ref backends anymore and confused
      me multiple times.

    - Patches 9 - 15: Handling of `PACK_REFS_ALL` in git-pack-refs(1),
      git-gc(1) and git-maintenance(1).

The patch series is built on top of 2953d95d40 (The eighth batch, 2024-03-15) with "ps/reftable-stack-tempfile" at 60c4c42515 (reftable/stack: register compacted tables as tempfiles, 2024-03-07) merged into it due to a merge conflict in "reftable/stack.c".

Patrick

PS: I polished this patch series while traveling and am still out of
    office until Thursday. I'll thus only get to respond to threads that
    are waiting for my input at the end of this week.

Patrick Steinhardt (15):
  reftable/stack: fix error handling in `reftable_stack_init_addition()`
  reftable/error: discern locked/outdated errors
  reftable/stack: use error codes when locking fails during compaction
  reftable/stack: gracefully handle failed auto-compaction due to locks
  refs/reftable: print errors on compaction failure
  t/helper: drop pack-refs wrapper
  refs: move `struct pack_refs_opts` to where it's used
  refs: remove `PACK_REFS_ALL` flag
  refs/reftable: expose auto compaction via new flag
  builtin/pack-refs: release allocated memory
  builtin/pack-refs: introduce new "--auto" flag
  builtin/gc: move `struct maintenance_run_opts`
  t6500: extract objects with "17" prefix
  builtin/gc: forward git-gc(1)'s `--auto` flag when packing refs
  builtin/gc: pack refs when using `git maintenance run --auto`

 Documentation/git-pack-refs.txt | 15 +++++-
 builtin/gc.c                    | 86 +++++++++++++++++++--------------
 builtin/pack-refs.c             | 31 +++++++-----
 refs.h                          | 20 ++++----
 refs/reftable-backend.c         | 11 ++++-
 reftable/error.c                |  4 +-
 reftable/reftable-error.h       |  5 +-
 reftable/stack.c                | 34 +++++++------
 reftable/stack_test.c           |  2 +-
 t/helper/test-ref-store.c       | 20 --------
 t/oid-info/hash-info            | 12 +++++
 t/t0601-reffiles-pack-refs.sh   | 30 ++++++++++--
 t/t0610-reftable-basics.sh      | 79 ++++++++++++++++++++++++++++++
 t/t6500-gc.sh                   | 30 +++---------
 14 files changed, 255 insertions(+), 124 deletions(-)

base-commit: f94fee98b456ef39a5f85fff5802f4e538a4dc7b