Karthik Nayak[PATCH v3 0/2] refs: allow setting the reference directory
While Git allows users to select different reference backends, unlike with objects, there is no flexibility in selecting the reference directory. Currently, the reference format is obtained from the config of the repository and the reference directory is set to the $GIT_DIR.
This patch series adds a new ENV variable 'GIT_REF_URI' which takes the reference backend and path in a URI form:
<reference_backend>://<URI-for-resource>
For e.g. 'reftable:///foo' or 'files://$GIT_DIR/ref_migration.0xBsa0'.
One use case for this is migration between different backends. On the server side, migrating from the files backend to the newly introduced reftable backend can be achieved by running 'git refs migrate'. However, for large repositories with millions of references, this migration can take from seconds to minutes.
For some background, at GitLab, the criteria for our migration was to reduce the downtime of the migrate ideally to zero. So running 'git refs migrate --ref-format=reftable' by itself wouldn't work, since it scales with the number of references and we have repos with millions of references, so we need to migrate without loosing any information. We came up with the following plan:
1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
file.
2. Run git refs migrate –dry-run.
3. If there are no ongoing reference requests (read/write)
a. Lock the repository by blocking incoming requests (done on a
layer above git, in Gitaly [1]).
b. If the timestamp of the packed-refs file has changed, unlock
the repo and repeat from step 1.
c. Apply all the loose refs to the dry-run reftable folder (this
requires support in Git to write refs to arbitrary folder).
d. Move the reftable dry-run folder into the GIT_DIR.
e. Swap the repo config
f. Unlock repo accessUsing such a route, scales much better since we only have to worry about blocking the repository by O(ref written between #1 and #3a) and not O(refs in repo). But for doing so, we need to be able to write to a arbitrary reference backend + path. This is to add the missing references to the dry-run reftable folder. This series, achieves that.
The first commit adds the required changes to create a 'ref_store' for a given path. The second commit parses the URI if available when creating the main ref store.
This is based on top of 9a2fb147f2 (Git 2.52, 2025-11-17).
[1]: https://gitlab.com/gitlab-org/gitaly
--- Changes in v3: - Cleanup some stale code which wasn't removed. - Localize strings which will be output to the user. - Remove additional defensive checks which are not needed. - Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com
Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
`<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
the same.
- In the tests:
- Use test_must_fail() instead of ! git
- Fix looped tests not using the variables correctly and ensure that
the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com--- Documentation/git.adoc | 8 ++++ environment.h | 1 + refs.c | 68 +++++++++++++++++++++++++-- t/meson.build | 1 + t/t1423-ref-backend.sh | 121 +++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 196 insertions(+), 3 deletions(-)
Karthik Nayak (2):
refs: support obtaining ref_store for given dir
refs: add GIT_REF_URI to specify reference backend and directoryRange-diff versus v2:
1: 5d37b2f0be = 1: 2b65f93e56 refs: support obtaining ref_store for given dir
2: 493c7ca098 ! 2: d5dbb2f112 refs: add GIT_REF_URI to specify reference backend and directory
@@ refs.c: static struct ref_store *get_ref_store_for_dir(struct repository *r,
+ char *format_string;
+ char *dir;
+
-+ if (!uri || !uri[0]) {
-+ error("reference backend uri is empty");
++ if (!uri) {
++ error(_("reference backend uri is not provided"));
+ goto cleanup;
+ }
+
+ if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
-+ error("invalid reference backend uri format '%s'", uri);
++ error(_("invalid reference backend uri format '%s'"), uri);
+ goto cleanup;
+ }
+
+ format_string = ref_backend_info.items[0].string;
+ if (!starts_with(ref_backend_info.items[1].string, "//")) {
-+ error("invalid reference backend uri format '%s'", uri);
++ error(_("invalid reference backend uri format '%s'"), uri);
+ goto cleanup;
+ }
+ dir = ref_backend_info.items[1].string + 2;
+
-+ format_string = ref_backend_info.items[0].string;
-+ dir = ref_backend_info.items[1].string + 2;
-+
-+ if (!dir || !dir[0]) {
-+ error("invalid path in uri '%s'", uri);
++ if (!dir[0]) {
++ error(_("invalid path in uri '%s'"), uri);
+ goto cleanup;
+ }
+
+ format = ref_storage_format_by_name(format_string);
+ if (format == REF_STORAGE_FORMAT_UNKNOWN) {
-+ error("unknown reference backend '%s'", format_string);
++ error(_("unknown reference backend '%s'"), format_string);
+ goto cleanup;
+ }
+
@@ t/t1423-ref-backend.sh (new)
+ GIT_REF_URI="" &&
+ export GIT_REF_URI &&
+ test_must_fail git refs list 2>err &&
-+ test_grep "reference backend uri is empty" err
++ test_grep "invalid reference backend uri format" err
+ )
+'
+base-commit: 9a2fb147f2c61d0cab52c883e7e26f5b7948e3ed change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef
Thanks - Karthik
