Re: [GSoC] Designing a faster index format - Progress report week 14

From: Thomas Gummerer <t.gummerer@gmail.com>
Date: Jul 24, 2012, 07:52 UTC
Message-ID: <20120724075252.GB40532@tgummerer.surfnet.iacbox>
In-Reply-To: <CACsJy8A8J-FXtJezOZrmqfUPX5unbGG15A6BuZnDW+164n6-kw@mail.gmail.com>

On 07/24, Nguyen Thai Ngoc Duy wrote:

Show 8 quoted lines

> On Tue, Jul 24, 2012 at 2:08 AM, Thomas Gummerer <t.gummerer@gmail.com> wrote:
>
> Is ls-files improvement not drastic because you do not limit subdir
> like grep? I see Thomas Rast put similar partial loading hack to
> ls-files.c so I assume it can partial load too. Is partial loading
> still fast with when a lot of unmerged entries are present?
> -- 
> Duy

Yes, exactly, the ls-files was to show the overall performance of the index for a full read. The improvement when limiting it to a subdir is about the same as with git grep. Here are some more timings. I'm using update-index instead of ls-files in this tests, with a --force-rewrite option I added which writes the index even if there are no changes, to test the performance of both the reader and the writer.

Test this tree ----------------------------------------------------------- 0002.2: v[23]: update-index 0.29(0.21+0.06) 0002.3: v[23]: grep nonexistent -- subdir 0.13(0.12+0.01) 0002.5: v4: update-index 0.26(0.20+0.05) 0002.6: v4: grep nonexistent -- subdir 0.11(0.08+0.02) 0002.7: v4: ls-files -- subdir 0.10(0.07+0.02) 0002.9: v5: update-index 0.19(0.11+0.07) 0002.10: v5: grep nonexistent -- subdir 0.01(0.00+0.00) 0002.11: v5: ls-files -- subdir 0.01(0.00+0.00)

Partial loading is still fast with unmerged entries, since we only need to load files that belong to a specific directory there too. I've created about 15,000 conflicts on the webkit repository, and got the following times:

Test this tree ----------------------------------------------------------- 0002.2: v[23]: update-index 0.30(0.18+0.10) 0002.3: v[23]: grep nonexistent -- subdir 0.13(0.09+0.04) 0002.5: v4: update-index 0.26(0.22+0.04) 0002.6: v4: grep nonexistent -- subdir 0.11(0.07+0.03) 0002.7: v4: ls-files -- subdir 0.10(0.09+0.01) 0002.9: v5: update-index 0.21(0.16+0.05) 0002.10: v5: grep nonexistent -- subdir 0.01(0.00+0.00) 0002.11: v5: ls-files -- subdir 0.01(0.00+0.00)

I could create more conflicts (~180,000 files are in the index), but I think 15,000 already is a number that's very unlikely to be reached.