git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Question about fsmonitor and --untracked-files=all

From
Tao Klerks <tao@klerks.biz>
Date
Sep 22, 2020, 11:35 UTC
Message-ID
<CAPMMpoj+UhKCW_k34-cGkiWFghOOu13GhPgA0V-y4ZpLVppuiA@mail.gmail.com>
Hi folks,

I've got a couple questions about the "fsmonitor" functionality, untracked files, and multithreading.

Background:
In a repo with:
 * A couple hundred thousand tracked files, and a couple hundred
thousand .gitignored files, across a few thousand directories
 * The --untracked-cache setting, tested and working
 * core.fsmonitor set up with watchman (with the sample integration
script from january)
 * Git version 2.27.0.windows.1

"git status" takes about 2s "git status --untracked-files=all" takes about 20s

When I turn off "core.fsmonitor", the numbers change to something like: "git status": 8s "git status --untracked-files=all": 9s

Using windows' "procmon" to observe git.exe's behavior from outside, I
think I've understood a couple things that surprise me:
1. when you specify "--untracked-files=all", git scans the entire
folder tree regardless of the "fsmonitor" hook
2. when you specify the "fsmonitor" hook, git does any
filesystem-scanning in a single-threaded fashion (as opposed to
multi-threaded without "fsmonitor" / normally)

These two things combine so that with "fsmonitor" set, normal command-line git status performance is great, but the performance in tools that eagerly look for untracked files (like "Git Extensions" on windows) actually suffers - it takes twice as long to run the 'git -c diff.ignoreSubModules=none status --porcelain=2 -z --untracked-files=all' command that this UI wants (and blocks on, when you go to a commit dialog).

Questions:
1. Is there a reason "--untracked-files=all" causes a full directory
tree scan even with the "fsmonitor" hook active, or is this
accidental?
2. Assuming that the full directory tree scan is indeed necessary even
with "fsmonitor" (when requesting all untracked files), could it be
made multithreaded?

(my apologies for the simplistic "outside-in" observations; I don't feel qualified to attempt to understand the git source code)

Thanks for any help understanding the optimization opportunities here!
Tao Klerks
Next: Johannes Schindelin
Message 1 of 3 in “Question about fsmonitor and --untracked-files=all”
  1. Tao KlerksSep 22, 2020
  2. Johannes SchindelinSep 23, 2020
  3. Tao KlerksSep 24, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.