Re: [PATCH 0/2] refs: allow setting the reference directory

From: Patrick Steinhardt <ps@pks.im>
Date: Dec 1, 2025, 13:19 UTC
Message-ID: <aS2V4TKeS4V_oxAb@pks.im>
In-Reply-To: <xmqq34651ie5.fsf@gitster.g>

On Sat, Nov 22, 2025 at 08:29:22PM -0800, Junio C Hamano wrote:

Show 24 quoted lines

> Karthik Nayak <karthik.188@gmail.com> writes:
> 
> > While Git allows users to select different reference backends, unlike
> > with objects, there is no flexibility in selecting the reference
> > directory. Currently, the reference format is obtained from the config
> > of the repository and the reference directory is set to the $GIT_DIR.
> 
> I actually am not sure if I like the proposed environment variable.
> 
> The proposal is based on an assumption that any reference backend
> should be able to move their backing store anywhere, and they should
> be able to express the location of their backing store as a single
> string <path>.  For a new backend, "where is your backing store" may
> not even be a question that does not make much sense (as "somewhere
> in the cloud that you do not even have to know" is certainly
> possible), and even for a new backend design that does allow such a
> question to have a meaningful answer, this "you have to be able to
> use a random place specified by this environment variable as your
> backing storage" is an additional requirement that its implementors
> may not need to satisfy in order to please their user base.
> 
> For reftable and files backends, these assumptions may be true, but
> then it is not too cumbersome if these stay to be backend specific,
> as there are only two backends.

I think it's a reasonable assumption to make that the path _can_ be represented as a single string. For now, we don't really require any configuration for the backend in the first place. So all you need to do is to say:

    [extension]
    refStorage = reftable

This implicitly identifies the location of the backend, too, as we derive it from the commondir/gitdir. As you say that's sufficient for the "files" and "reftable" backends, but it may be insufficient for other backends.

Suppose that we for example have a Postgres database to store data. It's clearly not sufficient to specify "extension.refStorage=postgres", as that wouldn't give you enough information to also know how to connect to the database.

It's a problem I have been thinking about quite a lot in the context of pluggable object databases, as well. Ultimately, the solution I arrived at is to extend the extension format itself. For pluggable ODBs this would look like this:

    [extension]
    objectStorage = postgres://127.0.0.1:5432?database=myrepo

This is similar to a normal URI with a schema: everything before the "://" identifies the format that is to be used, and everything after is then passed as-is to the backend itself. I think this should give us enough flexibility for any future formats and it is easy enough to configure. The added benefit is that this can also work in contexts like the GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES environment variables, even though their naming is off now.

For the reference storage I think we should be moving into a similar direction. Sure, for the current formats that we know its sufficient to only specify their directory. But I think we should treat the directory as an opaque string and then let the reference backend handle it, same as with the proposed format for object databases:

    # A schema-only variable will be treated as if we specified the
    # common directory.
    [extension]
    refStorage = reftable

    # It's also possible to explicitly specify a different location for
    # the backend.
    [extension]
    refStorage = reftable:///foo/bar

    # And same as above, we can also specify non-locations.
    [extension]
    refStorage = postgres://127.0.0.1:5432?database=myrepo

As said, the important thing here is that the reference backends get the string after the schema as opaque blobs that they can self-interpret.

Show 7 quoted lines

> So I dunno.  In addition, if this is designed to help migration
> (which is the impression I am getting from the cover letter
> description), don't you need a way to specify more than one (i.e.,
> source to migrate from and destination to migrate to)?  With a
> single GIT_REF_URI, it would not be obvious what it refers to,
> whether it is an additional place to write to, to read from, or
> something completely unrelated.  For example ...

I think we cannot easily retrofit handling of multiple refdbs into Git at this point in time anymore. The way to drive this would be that we have two processes:

  - One `git refs list` process in the repository that uses the old
    format.

  - One `git update-ref --stdin` process in the repository that uses the
    new format specified via GIT_REF_URI.

This allows us to do an online migration of data into a separate ref store.

Show 31 quoted lines

> > This patch series adds a new ENV variable 'GIT_REF_URI' which takes the
> > reference backend and path in a URI form:
> >
> >     <reference_backend>://<path>
> >
> > For e.g. 'reftable:///foo' or 'files://$GIT_DIR/ref_migration.0xBsa0'.
> >
> > One use case for this is migration between different backends. On the
> > server side, migrating from the files backend to the newly introduced
> > reftable backend can be achieved by running 'git refs migrate'. However,
> > for large repositories with millions of references, this migration can
> > take from seconds to minutes.
> >
> > We could make the migration non-blocking by running the migration in the
> > background and capturing and replaying updates to both backends. This
> > would require Git to support writing references to different reference
> > backends and paths.
> 
> ... I am reading that the above is saying that the system will write
> to whatever reference backend specified in the extension.refStorage,
> plus also where GIT_REF_URI points at, but if that is the way how
> the mechanism works, the variable should be named more specific to
> what it does, no?  It is not just a random "REF URI"; it is an
> additional ref backend that the updates are dumped to.  Maybe there
> would be a different use case where you may want to read from two
> reference backends, and you'd need to specify the secondary one with
> an environment variable, but if the system behaves one specific way
> for GIT_REF_URI (say, all updates are also copied to this additional
> ref backend at the specified ref backing store), a different
> environment variable name needs to be chosen to serve such a
> different use case, no?

Truth be told, I'm not realy a huge fan of the name, either. But as said, I don't think we can easily "overlay" multiple refdbs, as it would lead to various different questions due to our hierarchical layout of references.

That being said, I personally would prefer `GIT_REFERENCE_BACKEND` as variable name that accepts exactly the same kind of strings as the `extension.refStorage` values I have proposed above.

Thanks!

Patrick