Re: [PATCH v1] read-cache: speed up index load through parallelization

From: Duy Nguyen <pclouds@gmail.com>
Date: Aug 24, 2018, 18:20 UTC
Message-ID: <CACsJy8BXy_7QbDtF8bY5YzwJf=JUwiODv0zKxoSXeu4rJ+xjwg@mail.gmail.com>
In-Reply-To: <CACsJy8CUPGUhR3girstdqD6YVxOQ6_xE+gacT98KXgqOSPz0dw@mail.gmail.com>

On Fri, Aug 24, 2018 at 5:37 PM Duy Nguyen <pclouds@gmail.com> wrote:

Show 13 quoted lines

> On Thu, Aug 23, 2018 at 10:36 PM Ben Peart <peartben@gmail.com> wrote:
> > > Nice to see this done without a new index extension that records
> > > offsets, so that we can load existing index files in parallel.
> > >
> >
> > Yes, I prefer this simpler model as well.  I wasn't sure it would
> > produce a significant improvement given the primary thread still has to
> > run through the variable length cache entries but was pleasantly surprised.
>
> Out of curiosity, how much time saving could we gain by recording
> offsets as an extension (I assume we need, like 4 offsets if the
> system has 4 cores)? Much much more than this simpler model (which may
> justify the complexity) or just "meh" compared to this?

To answer my own question, I ran a patched git to precalculate individual thread parameters, removed the scheduler code and hard coded these parameters (I ran just 4 threads, one per core). I got 0m2.949s (webkit.git, 275k files, 100 read-cache runs). Compared to 0m4.996s from Ben's patch (same test settings of course) I think it's definitely worth adding some extra complexity.

-- 
Duy