Re: [PATCH] grep: enable multi-threading for -p and -W

From: Thomas Rast <trast@student.ethz.ch>
Date: Nov 29, 2011, 14:07 UTC
Message-ID: <201111291507.04754.trast@student.ethz.ch>
In-Reply-To: <4ED4E2D7.9090804@lsrfire.ath.cx>

René Scharfe wrote:

Show 7 quoted lines

> > * none of the patches:
> >   git grep --cached INITRAMFS_ROOT_UID
> >     2.13user 0.14system 0:02.10elapsed
> >   git grep --cached -W INITRAMFS_ROOT_UID    # note: broken!
> >     2.23user 0.18system 0:02.21elapsed 
> 
> Broken is a tad too hard a word

Sorry, I just wanted to mark it as: this is unattainable since we're now doing extra work.

Show 19 quoted lines

> > * my patch, but not yours:
> >   git grep --cached -W INITRAMFS_ROOT_UID
> >     3.01user 0.05system 0:03.07elapsed
> > 
> > * my patch + your patch:
> >   git grep --cached -W INITRAMFS_ROOT_UID
> >     4.45user 0.22system 0:02.67elapsed
> > 
> > So while it ends up being faster overall, it also uses way more CPU
> > and would presumably be *slower* on a single-processor system.
> > Apparently those attribute lookups really hurt.
> 
> Hmm, why are they *that* expensive?
> 
> callgrind tells me that userdiff_find_by_path() contributes only 0.18%
> to the total cost with your first patch.  Timings in my virtual machine
> are very volatile, but it seems that here the difference is in the
> system time while user is basically the same for all combinations of
> patches.

Not sure, perhaps callgrind does not model syscalls as expensive enough. I also suspect (from my very cursory look at the attributes machinery) that loading attributes for all paths *in random order* (i.e., threaded) causes a lot of discards of the existing attributes data. (The order is still random with my new patch, but we only load them for files that have matches.)

> I wonder what caused the slight speedup for the case without -W.  It's
> probably just noise, though.

Yeah, it's very noisy, but I was too lazy for best-of-50 or something ;-)

[...]

> > +#ifndef NO_PTHREADS
> > +static inline void grep_attr_lock(struct grep_opt *opt)

[...]

> We'd need stubs here for the case of NO_PTHREADS being defined.

Right, thanks.  Sorry for not testing with NO_PTHREADS to begin with.

> Perhaps leave the thread stuff in builtin/grep.c and export a function
> from there that does [the userdiff lookup], with locking and all?

That seems like a layering violation to me. Isn't builtin/grep.c supposed to call out to grep.c, but not the other way around?

Maybe it would be less messy if we had a global (across all of git) flag that says whether threads are on. Then subsystems that are used from threaded code, but cannot handle it, can learn to lock themselves around their work. But it would be pretty much the opposite of what git-grep now does.

Thanks for the review.  I'll reroll as a proper patch later.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch