From: Karthik Nayak Date: Wed, 25 Feb 2026 09:40:40 GMT Subject: [PATCH v9 0/6] refs: allow setting the reference directory Message-ID: <20260225-kn-alternate-ref-dir-v9-0-3fe118e40e28@gmail.com> In-Reply-To: <20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com> While Git allows users to select different reference backends, unlike with objects, there is no flexibility in selecting the reference directory. Currently, the reference format is obtained from the config of the repository and the reference directory is set to the $GIT_DIR. This patch extends the config 'extensions.refStorage' to take in multiple forms of inputs: - A format name alone (e.g., `reftable` or `files`). - A URI format `://` explicitly specifies both the format and payload (e.g., `reftable:///foo/bar`). We also add in a new ENV variable GIT_REFERENCE_BACKEND which can be used to override the config. One use case for this is migration between different backends. On the server side, migrating from the files backend to the newly introduced reftable backend can be achieved by running 'git refs migrate'. However, for large repositories with millions of references, this migration can take from seconds to minutes. For some background, at GitLab, the criteria for our migration was to reduce the downtime of the migrate ideally to zero. So running 'git refs migrate --ref-format=reftable' by itself wouldn't work, since it scales with the number of references and we have repos with millions of references, so we need to migrate without loosing any information. We came up with the following plan: 1. Run git-pack-refs(1) and note timestamp of the generated packed-refs file. 2. Run git refs migrate –dry-run. 3. If there are no ongoing reference requests (read/write) a. Lock the repository by blocking incoming requests (done on a layer above git, in Gitaly [1]). b. If the timestamp of the packed-refs file has changed, unlock the repo and repeat from step 1. c. Apply all the loose refs to the dry-run reftable folder (this requires support in Git to write refs to arbitrary folder). d. Move the reftable dry-run folder into the GIT_DIR. e. Swap the repo config f. Unlock repo access Using such a route, scales much better since we only have to worry about blocking the repository by O(ref written between #1 and #3a) and not O(refs in repo). But for doing so, we need to be able to write to a arbitrary reference backend + path. This is to add the missing references to the dry-run reftable folder. This series, achieves that. Since there was a long gap between v3 <> v4, the version 4 onward is based on top of 2258446484 (RelNotes: correct "fast-import" option name, 2026-01-30). [1]: https://gitlab.com/gitlab-org/gitaly --- Changes in v9: - There was a issue with how the docs were parsed, causing the list to not be rendered correctly. - Some small other nits. - Link to v8: https://patch.msgid.link/20260223-kn-alternate-ref-dir-v8-0-0509c132a203@gmail.com Changes in v8: - Fix a typo/grammar in commit 4. - In the final commits tests, avoid creating a file for text comparison. - Link to v7: https://patch.msgid.link/20260219-kn-alternate-ref-dir-v7-0-16f27860dbdf@gmail.com Changes in v7: - Add more details in the commit messages. - Cleanup some whitespace. - Reorder the commits to be group related changes together. - Add checks for stubs in the tests when creating new repos. - Link to v6: https://patch.msgid.link/20260214-kn-alternate-ref-dir-v6-0-86a82c77cf59@gmail.com Changes in v6: - The biggest change in this version is that we now support using the environment variable with 'git-clone(1)' and 'git-init(1)'. In such situations, the alternate reference directory is created and the config is added to the repository. - Add a new commit which moves stub creation/removal to the generic layer. - Cleanup logic flow in `refs_compute_filesystem_location()`. - Add more tests for usage with 'git-clone(1)', 'git-init(1)' and migration of repositories using alternate refs backend. - Fixup documentation, commit messages and typos. - Link to v5: https://patch.msgid.link/20260209-kn-alternate-ref-dir-v5-0-740899834ceb@gmail.com Changes in v5: - Moved around the commits, to ensure that the code to handle the config in the backend is first. Previously, we added the config first, which meant the commit allowed users to provide a URI but it was simply ignore. - Fix typos and grammar and rename variables. - Clean up the description and documentation to actually specify protocol over location. - Avoid an extra memory allocation by detaching the strbuf value. - Link to v4: https://patch.msgid.link/20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com Changes in v4: - Mostly re-wrote the code to also support worktree. Now, the existing backends will store worktree references in 'ref_dir/worktrees/wt_id' and add corresponding stubs in 'git_dir/worktrees/wt_id'. - We also support relative paths in the reference directories. These relative paths are resolved relative to the GIT_DIR. - Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com Changes in v3: - Cleanup some stale code which wasn't removed. - Localize strings which will be output to the user. - Remove additional defensive checks which are not needed. - Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com Changes in v2: - Added more clarification and proper intent in the cover message. - Changed the format from '://' to `://` as it much clearer. - Added logic to check for the '//' in the provided URI and a test for the same. - In the tests: - Use test_must_fail() instead of ! git - Fix looped tests not using the variables correctly and ensure that the test description is correct. - Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com --- Documentation/config/extensions.adoc | 16 +- Documentation/git.adoc | 5 + builtin/clone.c | 9 +- builtin/worktree.c | 34 +++++ environment.h | 1 + refs.c | 126 +++++++++++++++- refs.h | 13 ++ refs/files-backend.c | 23 ++- refs/packed-backend.c | 5 + refs/packed-backend.h | 1 + refs/refs-internal.h | 14 ++ refs/reftable-backend.c | 61 ++------ repository.c | 9 +- repository.h | 8 +- setup.c | 96 ++++++++++-- setup.h | 4 +- t/meson.build | 1 + t/t1423-ref-backend.sh | 280 +++++++++++++++++++++++++++++++++++ 18 files changed, 625 insertions(+), 81 deletions(-) Karthik Nayak (6): setup: don't modify repo in `create_reference_database()` refs: extract out `refs_create_refdir_stubs()` refs: move out stub modification to generic layer refs: receive and use the reference storage payload refs: allow reference location in refstorage config refs: add GIT_REFERENCE_BACKEND to specify reference backend Range-diff versus v8: 1: 9f1978e991 = 1: 71504213ca setup: don't modify repo in `create_reference_database()` 2: 75013d6874 ! 2: f372287ec6 refs: extract out `refs_create_refdir_stubs()` @@ refs.h: void ref_iterator_free(struct ref_iterator *ref_iterator); + * While it is necessary within the files backend, newer backends may not + * follow the same structure. To go around this, we create stubs as necessary. + * -+ * If provided with a 'refs_heads_msg', we create the 'refs/heads/head' file ++ * If provided with a 'refs_heads_content', we create the 'refs/heads/head' file + * with the provided message. + */ +void refs_create_refdir_stubs(struct repository *repo, const char *refdir, -+ const char *refs_heads_msg); ++ const char *refs_heads_content); + #endif /* REFS_H */ 3: 97bef9c5c0 = 3: b79ac00a3d refs: move out stub modification to generic layer 4: ed55f79701 = 4: d0ffa07dfc refs: receive and use the reference storage payload 5: 5902a4588c ! 5: bf17494952 refs: allow reference location in refstorage config @@ Documentation/config/extensions.adoc: For historical reasons, this extension is + format and payload (e.g., `reftable:///foo/bar`). + +Supported format names are: -++ ++ include::../ref-storage-format.adoc[] -++ ++ +The payload is passed directly to the reference backend. For the files and +reftable backends, this must be a filesystem path where the references will +be stored. Defaulting to the commondir when no payload is provided. Relative -+paths are resolved relative to the $GIT_DIR. Future backends may support ++paths are resolved relative to the `$GIT_DIR`. Future backends may support +other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo. -- + 6: 7b0f103dbf ! 6: d1f3323df6 refs: add GIT_REFERENCE_BACKEND to specify reference backend @@ t/t1423-ref-backend.sh: do + ( + cd repo && + -+ git config get extensions.refstorage >expect && -+ echo $BACKEND >actual && ++ git config get extensions.refstorage >actual && ++ echo $BACKEND >expect && + test_cmp expect actual && + + test_commit 1 && @@ t/t1423-ref-backend.sh: do + BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" && + GIT_REFERENCE_BACKEND=$BACKEND git clone source repo && + -+ git -C repo config get extensions.refstorage >expect && -+ echo $BACKEND >actual && ++ git -C repo config get extensions.refstorage >actual && ++ echo $BACKEND >expect && + test_cmp expect actual && + + verify_files_exist repo/.git refdir && base-commit: 22584464849815268419fd9d2eba307362360db1 change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef Thanks - Karthik