[PATCH 0/9] refs: ref storage format migrations

Hi,

this patch series implements migration of ref storage formats such that it is for example possible to migrate a repository with "files" format to the "reftable" format. The migration logic comes in the form of a new `git refs migrate` subcommand. As mentioned in [1], I do have plans to extend the new git-refs(1) over time.

In the current form, the migration logic has three major limitations:

  - It does not work with repositories that have worktrees as we have to
    migrate multiple ref stores here, one for every worktree. I wanted
    to avoid making this series too complex right from the start.

  - It does not migrate reflogs, because we have no interfaces to do so.
    I want to eventually address this by adding log-only updates to
    transactions.

  - It is not safe with concurrent writers. This is the limitation that
    is most critical in my eyes. The root cause here is that it is
    inherently impossible to lock the "files" backend for writes. I have
    been thinking about this issue a lot and have not found any solution
    that works. There are partial solutions here:

      - Create a central lockfile for the "files" backend -- if there,
        the backend will refuse to write. If that lock needs to be
        acquired during the "commit" phase of transaction though then we
        would essentially start to sequentialize all writes in the
        "files" backend, which is a non-starter. If not, then processes
        which are running already may not have seen it, and thus the
        issue still exists.

     - Pack all loose refs, remove "refs/" and create a "refs.lock"
       file. This isn't safe though because root refs can still be
       written.

     - Create a log of concurrent writes and apply that to the migrated
       refs once done. This is a lot of complexity, and it's unclear
       whether it even solves the issue with already-running writers.

     - Create a temporary "extensions.refMigration" extension that is
       unhandled by Git. New processes will refuse to run in such a
       repo and thus cannot write to it. Again, unsafe with running
       writers.

    Another alternative is that we could just make this a best effort.
    The "reftable" backend supports locking, and for the "files" backend
    we could just lock "HEAD" and call it a day. I'm not sure whether it
    is preferable though to have a "partial" solution compared to having
    none at all, as it may cause users to be less mindful. That's why I
    decided to just have no solution at all and document the limitation
    accordingly.

    If anybody has ideas here I'd be very happy to hear them.

Anyway. The current state of this patch series is sufficient to migrate repos without reflogs and worktrees, and thus mostly applies to bare repositories, only. This is somewhat intentional -- as a server operator this is of course our primary usecase at GitLab. We do plan to also upstream support for writing reflogs though, but in a later step. We do not plan to implement support for migrating repositories with worktrees, but I'd be happy to help out with the effort in case somebody else wants to.

The series is built on top of 4365c6fcf9 (The sixth batch, 2024-05-20). It pulls in the following two dependencies:

  - ps/refs-without-the-repository-updates at 00892786b8 (refs/packed:
    remove references to `the_hash_algo`, 2024-05-17). This is mostly to
    avoid conflicts.

  - ps/pseudo-ref-terminology at 8e4f5c2dc2 (refs: refuse to write
    pseudorefs, 2024-05-15). This is a functional prerequisite because
    the migration logic relies on the new definition of pseudorefs.

There are two minor textual conflicts when merged to "next" or "seen". One is a conflicting added header in "refs/reftable-backend.c", and one is a conflict with added functions in "refs.c". Both of these conflicts are trivial to solve by accepting both sides.

Thanks!

Patrick

[1]: https://lore.kernel.org/git/ZkNJaaKTTKbns8wo@tanuki/#t

Patrick Steinhardt (9):
  setup: unset ref storage when reinitializing repository version
  refs: convert ref storage format to an enum
  refs: pass storage format to `ref_store_init()` explicitly
  refs: allow to skip creation of reflog entries
  refs/files: refactor `add_pseudoref_and_head_entries()`
  refs/files: extract function to iterate through root refs
  refs: implement removal of ref storages
  refs: implement logic to migrate between ref storage formats
  builtin/refs: new command to migrate ref storage formats

 .gitignore                 |   1 +
 Documentation/git-refs.txt |  59 +++++++
 Makefile                   |   1 +
 builtin.h                  |   1 +
 builtin/clone.c            |   2 +-
 builtin/init-db.c          |   2 +-
 builtin/refs.c             |  75 +++++++++
 git.c                      |   1 +
 refs.c                     | 319 +++++++++++++++++++++++++++++++++++--
 refs.h                     |  41 ++++-
 refs/files-backend.c       | 121 ++++++++++++--
 refs/packed-backend.c      |  15 ++
 refs/refs-internal.h       |   7 +
 refs/reftable-backend.c    |  37 ++++-
 repository.c               |   3 +-
 repository.h               |  10 +-
 setup.c                    |  10 +-
 setup.h                    |   9 +-
 t/helper/test-ref-store.c  |   1 +
 t/t1460-refs-migrate.sh    | 243 ++++++++++++++++++++++++++++
 20 files changed, 913 insertions(+), 45 deletions(-)
 create mode 100644 Documentation/git-refs.txt
 create mode 100644 builtin/refs.c
 create mode 100755 t/t1460-refs-migrate.sh