Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff

From: Duy Nguyen <pclouds@gmail.com>
Date: Mar 11, 2016, 01:11 UTC
Message-ID: <CACsJy8CT7x3p2TLvMbJ2OkNbd=PZ7bxZQJv1wSU1j_8s826J2A@mail.gmail.com>
In-Reply-To: <1457641341.13557.32.camel@twopensource.com>

On Fri, Mar 11, 2016 at 3:22 AM, David Turner <dturner@twopensource.com> wrote:

Show 5 quoted lines

> Reads (ignoring SHA verification) will be slightly slower (due to the
> btree overhead).  If, in general, we only had to read part of the
> index, that would be faster. But a fair amount of git code is written
> to assume that it is reasonable to iterate over the whole index.  So I
> don't see a win from just switching to LMDB without other changes.

I didn't think of the btree overhead. Will need to run some test.. I think lmdb just gives us pointers to mmap'd file, so an entire index reading can be fast. We just record where the on-disk data is. We need to make index-on-lmdb use native byte endian though to avoid conversion and actually reading all index entries because of that.

Show 11 quoted lines

> (I guess we could parallelize index deserialization more easily with
> lmdb, but I don't know
>
> The original intent of the index was to be a "staging area" to build up
> a tree before committing it.  This implies an alternate view of the
> index as a diff against some (possibly-empty) tree.  An index diff or
> status operation, then, just requires iterating over the changes.
>
> I haven't really dug into merges to understand whether it would be
> reasonable for them to use a representation like this, and what that
> would do to speed there.

I think we keep the same internal representation of the index, a list of paths with extra info like stat and some flags. Minimal impacts to the code base that way. There are some cases (I think) where we just need to read a portion of the index (filtered by pathspec). If we don't update the index afterwards, lmdb partial reads definitely help. I need to check if those cases are common (and worth optimizing for) though.

Show 5 quoted lines

> In general, git takes the commit side of the commit-diff duality. This
> is generally a good thing, because commits are simpler to reason about.
> But I wonder if we could do better for this specific case.
>
> But I don't think switching to LMDB would replace index-helper.

The point of index-helper is reduce reading cost (let's leave watchman out for now). But it looks like we're not certain if using lmdb can reduce reading cost at lower complexity. I'll give it some thought over the weekend. Maybe in the end we better just go with index-helper...

-- 
Duy