Re: [PATCH v2] read_index_from(): Skip verification of the cache entry order to speed index loading

From: Ben Peart <peartben@gmail.com>
Date: Oct 31, 2017, 13:01 UTC
Message-ID: <f671ea09-d4aa-64aa-8225-c1fbf2eac175@gmail.com>
In-Reply-To: <alpine.DEB.2.10.1710301727160.10801@alexmv-linux>

On 10/30/2017 8:33 PM, Alex Vandiver wrote:

Show 32 quoted lines

> On Mon, 30 Oct 2017, Jeff King wrote:
>> On Mon, Oct 30, 2017 at 08:48:48AM -0400, Ben Peart wrote:
>>
>>> Any updates or thoughts on this one?  While the patch has become quite
>>> trivial, it does results in a savings of 5%-15% in index load time.
>>
>> I like the general direction of avoiding the check during each read.
> 
> Same -- the savings here are well worth it, IMHO.
> 
>>> I thought the compromise of having this test only run when DEBUG is defined
>>> should limit it to developer builds (hopefully everyone developing on git is
>>> running DEBUG builds :)).  Since the test is trying to detect buggy code
>>> when writing the index, I thought that was the right time to test/catch any
>>> issues.
>>
>> I certainly don't build with DEBUG. It traditionally hasn't done
>> anything useful. But I'm also not convinced that this is a likely way to
>> find bugs in the first place, so I'm OK missing out on it.
> 
> I also don't compile with DEBUG -- there's no documentation that
> mentions it, and I don't think I'd considered going poking for what
> was `#ifdef`d.  I think it'd be reasonable to provide some
> configure-time setting that adds `CFLAGS="-ggdb3 -O0 -DDEBUG"` or
> similar, but that seems possibly moot for this particular change (see
> below).
> 
>> But what we probably _do_ need is to make sure that "git fsck" would
>> detect such an out-of-order index. So that developers and users alike
>> can diagnose suspected problems.
> 
> Agree -- that seems like a better home for this logic.

That is how version 1 of this patch worked but the feedback to that patch was to remove it "not only during the normal operation but also in fsck."

Show 8 quoted lines

> 
>>> I am working on other, more substantial savings for index load times
>>> (stay tuned) but this seemed like a small simple way to help speed
>>> things up.
> 
> I'm interested to hear more about what direction you're looking in here.
>   - Alex
>

I'm working on parallelizing the index load process across multiple threads/cpu cores. Specifically the loop that calls create_from_disk() and set_index_entry(). The serial nature of the index formats makes that difficult but I believe I've come up with a way to make it work across all existing index formats.