Re: [RFC/WIP] Pluggable reference backends

From: Jeff King <peff@peff.net>
Date: Mar 10, 2014, 19:42 UTC
Message-ID: <20140310194247.GA24568@sigill.intra.peff.net>
In-Reply-To: <87k3c2820l.fsf@fencepost.gnu.org>

On Mon, Mar 10, 2014 at 05:14:02PM +0100, David Kastrup wrote:

Show 6 quoted lines

> [storing refs in sqlite]
>
> Of course, the basic premise for this feature is "let's assume that our
> file and/or operating system suck at providing file system functionality
> at file name granularity".  There have been two historically approaches
> to that problem that are not independent: a) use Linux b) kick Linus.

You didn't define "suck" here, but there are a number of issues with the current ref storage system. Here is a sampling:

  1. The filesystem does not present an atomic view of the data (e.g.,
     you read "a", then while you are reading "b", somebody else updates
     "a"; your view is one that never existed at any point in time).

  2. Using the filesystem creates D/F conflicts between branches "foo"
     and "foo/bar". Because this name is a primary key even for the
     reflogs, we cannot easily persist reflogs after the ref is removed.

  3. We use packed-refs in conjunction with loose ones to achieve
     reasonable performance when there are a large number of refs. The
     scheme for determining the current value of a ref is complicated
     and error-prone (we had several race conditions that caused real
     data loss).

Those things can be solved through better support from the filesystem. But they were also solved decades ago by relational databases.

I generally avoid databases where possible. They lock your data up in a binary format that you can't easily touch with standard unix tools. And they introduce complexity and opportunity for bugs.

But they are also a proven technology for solving exactly the sorts of problems that some people are having with git. I do not see a reason not to consider them as an option for a pluggable refs system. But I also do not see a reason to inflict their costs on people who do not have those problems. And that is why Michael's email is about _pluggable_ ref backends, and not "let's convert git to sqlite".

I do not even know if sqlite is going to end up as an interesting option. But it will be nice to be able to experiment with it easily due to git's ref code becoming more modular.

-Peff