Re: reftable [v5]: new ref storage format

From: Jeff King <peff@peff.net>
Date: Aug 8, 2017, 07:52 UTC
Message-ID: <20170808075254.sfazy7a6oc5bcc7b@sigill.intra.peff.net>
In-Reply-To: <833bc420609e452e8837862eb90f6a43@exmbdft7.ad.twosigma.com>

On Mon, Aug 07, 2017 at 03:40:48PM +0000, David Turner wrote:

Show 14 quoted lines

> > -----Original Message-----
> > From: Shawn Pearce [mailto:spearce@spearce.org]
> > In git-core, I'm worried about the caveats related to locking. Git tries to work
> > nicely on NFS, and it seems LMDB wouldn't. Git also runs fine on a read-only
> > filesystem, and LMDB gets a little weird about that. Finally, Git doesn't have
> > nearly the risks LMDB has about a crashed reader or writer locking out future
> > operations until the locks have been resolved. This is especially true with shared
> > user repositories, where another user might setup and own the semaphore.
> 
> FWIW, git has problems with stale lock file in the event of a crash (refs/foo.lock 
> might still exist, and git does nothing to clean it up).
> 
> In my testing (which involved a *lot* of crashing), I never once had to clean up a
> stale LMDB lock.  That said, I didn't test on a RO filesystem.

Yeah, I'd expect LMDB to do much better than Git in a crash, because it relies on flock. So when the kernel goes away, so too does your lock (ditto if a git process dies without remembering to remove the lock, though I don't think we've ever had such a bug).

But that's also why it may not work well over NFS (though my impression is that flock _does_ work on modern NFS; I've been lucky enough not to ever use it). Lack of NFS support wouldn't be a show-stopper for most people, but it would be for totally replacing the existing code, I'd think. I'm just not clear on what the state of lmdb-on-nfs is.

Assuming it could work, the interesting tradeoffs to me are:

  - something like reftable is hyper-optimized for high-latency
    block-oriented access. It's not clear to me if lmdb would even be
    usable for the distributed storage case Shawn has.

  - reftable is more code for us to implement, but we'd "own" the whole
    stack down to the filesystem. That could be a big win for debugging
    and optimizing for our use case.

  - reftable is re-inventing a lot of the database wheel. lmdb really is
    a debugged, turn-key solution.

I'm not opposed to a world where lmdb becomes the standard solution and Google does their own bespoke thing. But that's easy for me to say because I'm not Google. I do care about keeping complexity and bugs to a minimum for most users, and it's possible that lmdb could do that. But if it can't become the baseline standard (due to NFS issues), then we'd still want something to replace the current loose/packed storage. And if reftable does that, then lmdb becomes a lot less interesting.

-Peff