git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 1/4] fsmonitor: use fsmonitor data in `git diff`

From
Taylor Blau <me@ttaylorr.com>
Date
Oct 18, 2020, 23:43 UTC
Message-ID
<20201018234344.GC4204@nand.local>
In-Reply-To
<xmqq1rhw86ur.fsf@gitster.c.googlers.com>
On Sat, Oct 17, 2020 at 10:02:04PM -0700, Junio C Hamano wrote:
Show 27 quoted lines
> Taylor Blau <me@ttaylorr.com> writes:
>
> > Hmm. I do agree that I'd like to stay out of the business of trying to
> > figure out exactly what that trade-off is (although I'm sure that it
> > exists), only because it seems likely to vary to a large extent from
> > repository to repository. (That is, 20% may be a good number for some
> > repository, but a terrible choice for another).
>
> I think both of you misunderstood me.
>
> My question was a simple yes/no "does there a trade off exist?"
> question and the sentences with 20% in it were mere example of
> possible trade-off I had in mind that _could_ exist.  I wasn't even
> suggesting to figure out what the optimum cut-off heuristics would
> be (e.g. solving "when more than N% paths are subject to diff
> fsmonitor is faster" for N).
>
> I was hoping that we can show that even having to lstat just a
> single path is expensive enough---IOW, "there is no trade-off worth
> worrying about, because talking to fsmonitor is so cheap compared to
> the cost of even a single lstst" would have been a valid and happy
> answer.  With such a number, there is no risk of introducing an
> unwarranted performance regression to use cases that we did not
> anticipate by adding an unconditional call to refresh_fsmonitor().
>
> But without any rationale, the performance implication of adding an
> unconditional call to refresh_fsmonitor() would become much muddier.

Aha; thanks for clarifying. I'm glad we agree that finding 'N' would not be worth it, or at least that showing that talking to fsmonitor is cheaper than a single lstat would be more worthwhile.

Nipunn - I don't have fsmonitor/watchman setup on my workstation, but if you do, some numbers (or an interpretation of the numbers you already provided) on this would be really useful. If you don't have it set up, or don't have time to measure it, let me know, and I'd be happy to take a look.

Show 16 quoted lines
> > But, I think that we can invoke watchman better here; the
> > fsmonitor-watchman hook has no notion of a "pathspec", so every query
> > just asks for everything that isn't in '$GIT_DIR'. Is there anything
> > preventing us from taking an optional pathspec and building up a more
> > targeted query?
>
> Yup, it is what I had in mind when I brought up the pathspec.  It
> may be something worth pursuing longer term, but not within the
> scope of this patch.
>
> > There is some overhead to invoke the hook and talk to watchman, but
> > I'd expect that to be dwarfed by not having to issue O(# files)
> > syscalls.
>
> "invoke the hook"---is that a pipe+fork+exec, or something else that
> is far lighter-weight?
The former; see 'fsmonitor.c:query_fsmonitor()'.

Thanks, Taylor

Previous: Junio C HamanoNext: Junio C Hamano
Message 7 of 52 in “use fsmonitor data in git diff eliminating O(num_files) calls to lstat”
  1. 0/4 use fsmonitor data in git diff eliminating O(num_files) calls to lstatNipunn Koorapati via GitGitGadget, Oct 17, 2020
  2. 1/4 fsmonitor: use fsmonitor data in `git diff`Alex Vandiver via GitGitGadget, Oct 17, 2020
  3. Junio C HamanoOct 17, 2020
  4. Nipunn KoorapatiOct 18, 2020
  5. Taylor BlauOct 18, 2020
  6. Junio C HamanoOct 18, 2020
  7. Taylor BlauOct 18, 2020
  8. Junio C HamanoOct 19, 2020
  9. Taylor BlauOct 19, 2020
  10. Nipunn KoorapatiOct 19, 2020
  11. 2/4 t/perf/README: elaborate on output formatNipunn Koorapati via GitGitGadget, Oct 17, 2020
  12. 4/4 t/perf: add fsmonitor perf test for git diffNipunn Koorapati via GitGitGadget, Oct 17, 2020
  13. Junio C HamanoOct 17, 2020
  14. 3/4 t/perf/p7519-fsmonitor.sh: warm cache on first git statusNipunn Koorapati via GitGitGadget, Oct 17, 2020
  15. Taylor BlauOct 18, 2020
  16. 0/4 use fsmonitor data in git diff eliminating O(num_files) calls to lstatNipunn Koorapati via GitGitGadget, Oct 19, 2020
  17. 1/4 fsmonitor: use fsmonitor data in `git diff`Alex Vandiver via GitGitGadget, Oct 19, 2020
  18. 2/4 t/perf/README: elaborate on output formatNipunn Koorapati via GitGitGadget, Oct 19, 2020
  19. 3/4 t/perf/p7519-fsmonitor.sh: warm cache on first git statusNipunn Koorapati via GitGitGadget, Oct 19, 2020
  20. 4/4 t/perf: add fsmonitor perf test for git diffNipunn Koorapati via GitGitGadget, Oct 19, 2020
  21. Taylor BlauOct 19, 2020
  22. Taylor BlauOct 19, 2020
  23. Nipunn KoorapatiOct 19, 2020
  24. Taylor BlauOct 19, 2020
  25. Nipunn KoorapatiOct 19, 2020
  26. 0/7 use fsmonitor data in git diff eliminating O(num_files) calls to lstatNipunn Koorapati via GitGitGadget, Oct 19, 2020
  27. 3/7 t/perf/p7519-fsmonitor.sh: warm cache on first git statusNipunn Koorapati via GitGitGadget, Oct 19, 2020
  28. 1/7 fsmonitor: use fsmonitor data in `git diff`Alex Vandiver via GitGitGadget, Oct 19, 2020
  29. 2/7 t/perf/README: elaborate on output formatNipunn Koorapati via GitGitGadget, Oct 19, 2020
  30. 5/7 perf lint: check test-lint-shell-syntax in perf testsNipunn Koorapati via GitGitGadget, Oct 19, 2020
  31. Taylor BlauOct 20, 2020
  32. Junio C HamanoOct 20, 2020
  33. Taylor BlauOct 20, 2020
  34. Nipunn KoorapatiOct 20, 2020
  35. Nipunn KoorapatiOct 20, 2020
  36. 7/7 p7519-fsmonitor: add a git add benchmarkNipunn Koorapati via GitGitGadget, Oct 19, 2020
  37. Nipunn KoorapatiOct 19, 2020
  38. Taylor BlauOct 20, 2020
  39. 4/7 t/perf: add fsmonitor perf test for git diffNipunn Koorapati via GitGitGadget, Oct 19, 2020
  40. 6/7 p7519-fsmonitor: refactor to avoid code duplicationNipunn Koorapati via GitGitGadget, Oct 19, 2020
  41. Taylor BlauOct 20, 2020
  42. 0/7 use fsmonitor data in git diff eliminating O(num_files) calls to lstatNipunn Koorapati via GitGitGadget, Oct 20, 2020
  43. 1/7 fsmonitor: use fsmonitor data in `git diff`Alex Vandiver via GitGitGadget, Oct 20, 2020
  44. 2/7 t/perf/README: elaborate on output formatNipunn Koorapati via GitGitGadget, Oct 20, 2020
  45. 6/7 p7519-fsmonitor: refactor to avoid code duplicationNipunn Koorapati via GitGitGadget, Oct 20, 2020
  46. 3/7 t/perf/p7519-fsmonitor.sh: warm cache on first git statusNipunn Koorapati via GitGitGadget, Oct 20, 2020
  47. 5/7 perf lint: add make test-lint to perf testsNipunn Koorapati via GitGitGadget, Oct 20, 2020
  48. Taylor BlauOct 20, 2020
  49. Nipunn KoorapatiOct 20, 2020
  50. Taylor BlauOct 20, 2020
  51. 4/7 t/perf: add fsmonitor perf test for git diffNipunn Koorapati via GitGitGadget, Oct 20, 2020
  52. 7/7 p7519-fsmonitor: add a git add benchmarkNipunn Koorapati via GitGitGadget, Oct 20, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.