Re: [PATCH v2] read_directory: avoid invoking exclude machinery on tracked files

From: Pete Wyckoff <pw@padd.com>
Date: Feb 17, 2013, 15:49 UTC
Message-ID: <20130217154956.GA2395@padd.com>
In-Reply-To: <CACsJy8C9SGJxwnm1E2N_KyEMg5-MzDt2B+SrX7rygn8X1qq4Wg@mail.gmail.com>

pclouds@gmail.com wrote on Sun, 17 Feb 2013 11:39 +0700:

Show 18 quoted lines

> On Sun, Feb 17, 2013 at 1:11 AM, Pete Wyckoff <pw@padd.com> wrote:
> > pclouds@gmail.com wrote on Sat, 16 Feb 2013 14:17 +0700:
> >> Finally some numbers (best of 20 runs) that shows why it's worth all
> >> the hassle:
> >>
> >> git status   | webkit linux-2.6 libreoffice-core gentoo-x86
> >> -------------+----------------------------------------------
> >> before       | 1.097s    0.208s           0.399s     0.539s
> >> after        | 0.736s    0.159s           0.248s     0.501s
> >> nr. patterns |    89       376               19          0
> >> nr. tracked  |   182k       40k              63k       101k
> >
> > Thanks for this work.  I repeated some of the tests across NFS,
> > where I'd expect to see bigger differences.
> 
> This is about reducing CPU processing time, not I/O time. So no bigger
> differences is expected. I/O time can be reduced with inotify, or fam
> in nfs case because inotify does not support nfs.

Numbers from the last mail were core.preloadindex=true. Here's "time" output from average runs:

    stock = 0m2.28s user 0m4.18s sys 0m11.28s elapsed 57.39 %CPU
    duy   = 0m1.25s user 0m4.43s sys 0m7.45s elapsed 76.41 %CPU

With this huge repo, preloadindex may be stressing directory cache behavior on the NFS server or client. Your patch helps both CPU and wait time by avoiding the 6000-odd open() of non-existent .gitignore.

With core.preloadindex=false, it's a 1 sec speedup, all from CPU:

    stock = 0m2.18s user 0m1.59s sys 0m7.78s elapsed 48.45 %CPU
    duy   = 0m1.17s user 0m1.63s sys 0m6.91s elapsed 40.59 %CPU

		-- Pete