Re: [RFC/WIP] Pluggable reference backends

From: Jeff King <peff@peff.net>
Date: Mar 10, 2014, 15:52 UTC
Message-ID: <20140310155230.GA29801@sigill.intra.peff.net>
In-Reply-To: <CAJo=hJtiPgByhk9M4ZKD98DARzgeU6z2mmw7fcLTEbBza-_h6A@mail.gmail.com>

On Mon, Mar 10, 2014 at 07:30:45AM -0700, Shawn Pearce wrote:

Show 7 quoted lines

> > * Store references in a SQLite database, to get correct transaction
> >   handling.
> 
> No to SQLLite in git-core. Using it from JGit requires building
> SQLLite and a JNI wrapper, which makes JGit significantly less
> portable. I know SQLLite is pretty amazing, but implementing
> compatibility with it from JGit will be a big nightmare for us.

That seems like a poor reason not to implement a pluggable feature for git-core. If we implement it, then a site using only git-core can take advantage of it. Sites with JGit cannot, and would use a different pluggable storage mechanism that's supported by both. But if we don't implement, it hurts people using only git-core, and it does not help sites using JGit at all.

That's assuming that attention spent on implementing the feature does not take away from implementing some other parallel scheme that does the same thing but does not use SQLite. I don't know what that would be offhand; mapping the ref and reflog into a relational database is pretty simple, and we get a lot of robustness and efficiency benefits for free. We could perhaps have some kind of "relational" backend could use an ODBC-like abstraction to point to a database. I have no idea if people would want to ever store refs in a "real" server-backend RDBMS, but I suspect Java has native support for such things.

Certainly I think we should aim for compatibility where we can, but if there's not a compatible way to do something, I don't think the limitations of one platform should drag other ones down. And that goes both ways; we had to reimplement disk-compatible EWAH from scratch in C for git-core to have bitmaps, whereas JGit just got to use a ready-made library. I don't think that was a bad thing. People in mixed-implementation environments couldn't use it, but people with JGit-only environments were free to take advantage of it.

At any rate, the repository needs to advertise "this is the ref storage mechanism I use" in the config. We're going to need to bump core.repositoryformatversion for such cases (because an old version of git should not blindly lock and write to a refs/ directory that nobody else is ever going to look at). And I'd suggest with that bump adding in something like core.refstorage, so that an implementation can say "foobar ref storage? Never heard of it" and barf. Whether it's because that implementation doesn't support "foobar", because it's an old version that doesn't understand "foobar" yet, or because it was simply built without "foobar" support.

-Peff