Re: [PATCH v2 0/4] Speed up unpack_trees()

From: Ben Peart <peartben@gmail.com>
Date: Jul 31, 2018, 16:50 UTC
Message-ID: <cc3c4dbb-d545-6a6c-b20e-6a8ca66fc210@gmail.com>
In-Reply-To: <CACsJy8BUBjPngHz=icHomor-LJOkMLwZ9bQ6YJDxnoXGg++vjg@mail.gmail.com>

On 7/31/2018 11:31 AM, Duy Nguyen wrote:

Show 32 quoted lines

> On Mon, Jul 30, 2018 at 8:10 PM Ben Peart <peartben@gmail.com> wrote:
>> I ran "git checkout" on a large repo and averaged the results of 3 runs.
>>    This clearly demonstrates the benefit of the optimized unpack_trees()
>> as even the final "diff-index" is essentially a 3rd call to unpack_trees().
>>
>> baseline        new
>> ----------------------------------------------------------------------
>> 0.535510167     0.556558733     s: read cache .git/index
>> 0.3057373       0.3147105       s: initialize name hash
>> 0.0184082       0.023558433     s: preload index
>> 0.086910967     0.089085967     s: refresh index
>> 7.889590767     2.191554433     s: unpack trees
>> 0.120760833     0.131941267     s: update worktree after a merge
>> 2.2583504       2.572663167     s: repair cache-tree
>> 0.8916137       0.959495233     s: write index, changed mask = 28
>> 3.405199233     0.2710663       s: unpack trees
>> 0.000999667     0.0021554       s: update worktree after a merge
>> 3.4063306       0.273318333     s: diff-index
>> 16.9524923      9.462943133     s: git command:
>> 'c:\git-sdk-64\usr\src\git\git.exe' checkout
>>
>> The first call to unpack_trees() saves 72%
>> The 2nd and 3rd call save 92%
> 
> By the 3rd I guess you meant "diff-index" line. I think it's the same
> with the second call. diff-index triggers the second unpack-trees but
> there's no indent here and it's misleading to read this as diff-index
> and unpack-trees execute one after the other.
> 
>> Total time savings for the entire command was 44%
> 
> Wow.. I guess you have more trees since I could only save 30% on gcc.git.

Yes, with over 500K trees, this optimization really pays off for us. I can't wait to see how this works out in the wild (vs my "lab" based performance testing).

Thank you!  I definitely owe you lunch. :)

Show 12 quoted lines

> 
>> In the performance game of whack-a-mole, that call to repair cache-tree
>> is now looking quite expensive...
> 
> Yeah and I think we can whack that mole too. I did some measurement.
> Best case possible, we just need to scan through two indexes (one with
> many good cache-tree, one with no cache-tree), compare and copy
> cache-tree over. The scanning takes like 1% time of current repair
> step and I suspect it's the hashing that takes most of the time. Of
> course real world won't have such nice numbers, but I guess we could
> maybe half cache-tree update/repair time.
>

I have some great profiling tools available so will take a look at this next and see exactly where the time is being spent.