Re: [PATCH 2/5] add: allow operating on a sparse-only index
- From
Derrick Stolee <stolee@gmail.com>
- Date
- Jul 26, 2021, 13:11 UTC
- Message-ID
- <8a7f504a-99af-ce89-307e-5a662d69f5fa@gmail.com>
- In-Reply-To
- <CABPp-BHQ52WpCKc8dxaV+u1QZCeT-YieQynTmK_w84r_Tc=VGQ@mail.gmail.com>
On 7/23/2021 1:45 PM, Elijah Newren wrote:
> On Wed, Jul 21, 2021 at 2:07 PM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote:
...
Show 17 quoted lines
>> Test HEAD~1 HEAD >> ------------------------------------------------------------------------------ >> 2000.6: git add -A (full-index-v3) 0.35(0.30+0.05) 0.37(0.29+0.06) +5.7% >> 2000.7: git add -A (full-index-v4) 0.31(0.26+0.06) 0.33(0.27+0.06) +6.5% >> 2000.8: git add -A (sparse-index-v3) 0.57(0.53+0.07) 0.05(0.04+0.08) -91.2% >> 2000.9: git add -A (sparse-index-v4) 0.58(0.55+0.06) 0.05(0.05+0.06) -91.4% >> >> While the 91% improvement seems impressive, it's important to recognize >> that previously we had significant overhead for expanding the >> sparse-index. Comparing to the full index case, 'git add -A' goes from >> 0.37s to 0.05s, which is "only" an 86% improvement. > > Hehe. Yep, it's so "disappointing" to "only" have the code be 7x faster. :-) > > Out of curiosity, IIRC any operation involving the index took ~10s on > some of the Microsoft repos. What does the speedup look like over > there for these changes to git-add?
The latest numbers I have for a repo with ~2 million tracked files is that index reads take about half a second (because of the threaded reads) and writes take at least one second. There was a lot of work by Ben Peart, Jeff Hostetler, and Kevin Willford to reduce this cost as much as possible a few years ago. VFS for Git is still limited by this bottleneck, but Scalar's use of sparse-checkout enables the use of the sparse index.
We have an experimental release [1] out to users right now, and I will report to the mailing list about how that went after we get sufficient adoption that the data can be significant. When focusing on individual users I can find things like one user seeing "git commit" going from 4.3s to 0.35s and "git add" going from 6.1s to 0.13s. (The "git add" time might also be conflated with a change from the FS Monitor hook to the builtin FS Monitor.)
[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp
Thanks, -Stolee