git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v4] Add git-grep threads param

From
Linus Torvalds <torvalds@linux-foundation.org>
Date
Nov 9, 2015, 19:11 UTC
Message-ID
<CA+55aFwV7c6=4mXPuB0c21rK3TSVWEw9JT-kiu35RuMzuHxoVg@mail.gmail.com>
In-Reply-To
<6AE1604EE3EC5F4296C096518C6B77EE5D0FDABA19@mail.accesssoftek.com>

On Mon, Nov 9, 2015 at 10:32 AM, Victor Leschuk <vleschuk@accesssoftek.com> wrote:

> On Mon, Nov 9, 2015 at 9:28 AM, Victor Leschuk
>>   num_threads = online_cpus() <= 1 ? 0 : GREP_NUM_THREADS_DEFAULT;
>
> Actually I have never said the nCPUs played main role in it. T

The pseudo-code you sent disagrees. Not that "online_cpus() <= 1" is likely to ever be really an issue on any development platform from the last decade.

However, I do have to admit that that "online_cpus()" check goes back a long time, so I guess I can't really blame you.

At least in the index preloading, I was very conscious of the IO issues. It doesn't actually make a big difference on traditional disks (seek times dominate, and concurrent IO often doesn't help at all), but the reason I keep on bringing up NFS is that back when I used CVS (oh, the horrors), I *also* worked at a company that did everything over NFS. CPU ended up almost never being the limiting factor for any SCM operation.

So I don't have a very good idea of *what* we should use for automatic thread detection, but I'm pretty sure online_cpu's should not be it. Except, like Jeff mentioned, for pack formation (which does tend to be all about CPU).

Sadly, detecting what kind of filesystem you are on and how well cached it is, is really pretty hard. Even when you have OS-specific knowledge, and can look up the *type* of the filesystem, what often matters more is things like "is the filesystem on a rotational media or using flash?" etc.

In the meantime I'd argue for just getting rid of the online_cpu's check, because

 (a) I think it's actively misleading
 (b) the threaded grep probably doesn't hurt much even on a single
CPU, and the _potential_ upside from IO could easily dwarf the cost.
 (c) do developers actually have single-core machines any more?

But if somebody can come up with a smarter model, that would certainly be good too. The IO advantages really don't tend to be there for rotational media, but for both flash and network filesystems, threaded IO can be a huge deal.

                  Linus
Previous: Victor LeschukNext: Victor Leschuk
Message 16 of 19 in “Add git-grep threads param”
  1. Add git-grep threads paramVictor Leschuk, Oct 27, 2015
  2. Victor LeschukNov 2, 2015
  3. Junio C HamanoNov 2, 2015
  4. Victor LeschukNov 3, 2015
  5. Junio C HamanoNov 3, 2015
  6. Jeff KingNov 4, 2015
  7. Victor LeschukNov 9, 2015
  8. Jeff KingNov 9, 2015
  9. Victor LeschukNov 9, 2015
  10. Jeff KingNov 9, 2015
  11. Victor LeschukNov 9, 2015
  12. Jeff KingNov 9, 2015
  13. Linus TorvaldsNov 9, 2015
  14. Jeff KingNov 9, 2015
  15. Victor LeschukNov 9, 2015
  16. Linus TorvaldsNov 9, 2015
  17. Victor LeschukNov 9, 2015
  18. Stefan BellerNov 9, 2015
  19. Victor LeschukNov 9, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.