git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 3/3] grep: disable threading in all but worktree case

From
Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date
Dec 24, 2011, 10:49 UTC
Message-ID
<CACsJy8C1o_Ryf0QDJXz8xEqFYxprWx01AeNT4CC=3DwbAT1JFg@mail.gmail.com>
In-Reply-To
<20111224070715.GA32267@sigill.intra.peff.net>
On Sat, Dec 24, 2011 at 2:07 PM, Jeff King <peff@peff.net> wrote:
Show 34 quoted lines
> I tried to get some timings for this, but ran across some quite
> surprising results. Here's a simple grep of the linux-2.6 working tree,
> using a single-threaded grep:
>
>  $ time git grep SIMPLE >/dev/null
>  real    0m0.439s
>  user    0m0.272s
>  sys     0m0.160s
>
> and then the same thing, via xargs, without even turning on
> parallelization. This should give us a measurement of the overhead for
> going through xargs at all. We'd expect it to be slower, but not too
> much so:
>
>  $ time git ls-files | xargs git grep SIMPLE -- >/dev/null
>  real    0m11.989s
>  user    0m11.769s
>  sys     0m0.268s
>
> Twenty-five times slower! Running 'perf' reports the culprit as pathspec
> matching:
>
>  +  63.23%    git  git                 [.] match_pathspec_depth
>  +  28.60%    git  libc-2.13.so        [.] __strncmp_sse42
>  +   2.22%    git  git                 [.] strncmp@plt
>  +   1.67%    git  git                 [.] kwsexec
>
> where the strncmps are called as part of match_pathspec_depth. So over
> 90% of the CPU time is spent on matching the pathspecs, compared to less
> than 2% actually grepping.
>
> Which really makes me wonder if our pathspec matching could stand to be
> faster. True, giving a bunch of single files is the least efficient way
> to use pathspecs, but that's pretty amazingly slow.

We could eliminate get_pathspec_depth() in grep_directory() when read_directory() learns to filter path properly using (and at the cost of) tree_entry_interesting(). The latter function has more optimizaions built in and should be faster than the former. This is a good test case for my read_directory() rewrite. Thanks.

get_pathspec_depth() can still use some optimizations though for grep_cache() case.

-- 
Duy
Previous: Jeff KingNext: Nguyen Thai Ngoc Duy
Message 27 of 35 in “grep multithreading and scaling”
  1. 0/3 grep multithreading and scalingThomas Rast, Dec 2, 2011
  2. 1/3 grep: load funcname patterns for -WThomas Rast, Dec 2, 2011
  3. 2/3 grep: enable threading with -p and -W using lazy attribute lookupThomas Rast, Dec 2, 2011
  4. 3/3 grep: disable threading in all but worktree caseThomas Rast, Dec 2, 2011
  5. René ScharfeDec 2, 2011
  6. Thomas RastDec 5, 2011
  7. René ScharfeDec 6, 2011
  8. 4/2 grep: turn off threading for non-worktreeRené Scharfe, Dec 6, 2011
  9. Jeff KingDec 7, 2011
  10. René ScharfeDec 7, 2011
  11. Jeff KingDec 7, 2011
  12. J. Bruce FieldsDec 7, 2011
  13. Jeff KingDec 7, 2011
  14. Thomas RastDec 7, 2011
  15. René ScharfeDec 7, 2011
  16. Pete WyckoffDec 10, 2011
  17. René ScharfeDec 12, 2011
  18. Jeff KingDec 7, 2011
  19. René ScharfeDec 7, 2011
  20. Jeff KingDec 7, 2011
  21. Thomas RastDec 7, 2011
  22. René ScharfeDec 7, 2011
  23. Ævar Arnfjörð BjarmasonDec 23, 2011
  24. Thomas RastDec 23, 2011
  25. Ævar Arnfjörð BjarmasonDec 24, 2011
  26. Jeff KingDec 24, 2011
  27. Nguyen Thai Ngoc DuyDec 24, 2011
  28. Nguyen Thai Ngoc DuyDec 24, 2011
  29. Jeff KingDec 24, 2011
  30. Nguyen Thai Ngoc DuyDec 25, 2011
  31. Jeff KingDec 2, 2011
  32. Thomas RastDec 5, 2011
  33. Thomas RastDec 5, 2011
  34. Jeff KingDec 6, 2011
  35. Eric HermanDec 2, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.