git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?

From
Tao Klerks <tao@klerks.biz>
Date
Jun 10, 2021, 10:24 UTC
Message-ID
<CAPMMpog7bNNPm3suZKu6OppHA+KDYgCfmaxW4HqTAr7_tTVAPQ@mail.gmail.com>
Hi folks,

With the new "core.useBuiltinFSMonitor" support in the Windows installer, I think this subject is worth calling out explicitly (and my apologies if I missed prior discussion):

TL;DR:
 - I believe "core.untrackedcache" should be enabled by default in
Windows (and it does not appear to be)
 - If a user enables "core.useBuiltinFSMonitor" (eg in the installer)
in the hopes of getting snappy "git status" on a repo with a large
deep working tree, they will be *unnecessarily* disappointed if
"core.untrackedcache" is not enabled
 - There is also a lingering "problem" with "git status -uall", with
both "core.useBuiltinFSMonitor" and "core.fsmonitor", but that seems
far less trivial to address
Detail:

I just started testing the new "core.useBuiltinFSMonitor" option in the new installer, and it's amazing, thanks Ben, Alex, Johannes and Kevin!

However, when I first enabled it, I was getting slightly *worse* git status times than without it... and those worse git status times were accompanied by a message along the lines of: --- It took 5.88 seconds to enumerate untracked files. 'status -uno' may speed it up, but you have to be careful not to forget to add new files yourself (see 'git help status'). ---

For context, this is in a repo with 200,000 or so files, within 40,000 folders (avg path depth 4 I think?), with a reasonably-intricate set of .gitignore patterns. Obviously that's not "your average user", but I would imagine it matches "the target audience for 'core.useBuiltinFSMonitor'" pretty well.

After a little head-scratching, I recalled an exchange with Johannes from last year: https://lore.kernel.org/git/CAPMMpohJicVeCaKsPvommYbGEH-D1V02TTMaiVTV8ux+9z9vkQ@mail.gmail.com/

I never did understand the relevant code paths in much detail, but the
practical conclusions were:
 - Without "core.untrackedcache" enabled, git ends up iterating
through the entire path structure of the working tree *even if
"core.fsmonitor" (and now "core.useBuiltinFSMonitor") is enabled*,
looking for untracked files to report
 - Even with "core.untrackedcache" enabled, if "core.fsmonitor" (and
now "core.useBuiltinFSMonitor") is enabled, git iterates through the
entire path structure of the working tree *single-threaded* when the
"--untracked-files" mode is set to "all" (by config or command-line)
Now, I imagine that addressing/improving these behaviors is very
non-trivial, but the impact could be reasonably limited if:
 - core.untrackedcache were defaulted to "true", at least under
Windows, at least when the installer is asked to set
core.useBuiltinFSMonitor
 - The "It took N.NN seconds to enumerate untracked files" message
were to include a hint about core.untrackedcache, at least when the
"--untracked-files" mode is set to "normal".

Final note: I personally would love to see "core.useBuiltinFSMonitor actually makes things slower, when --untracked-files=all is specified" behavior be addressed, because common windows git integrations or front-ends like Git Extensions or IntelliJ IDEA commonly use those options, and therefore "suffer" a performance degradation on at least some operations when core.useBuiltinFSMonitor is enabled.

I don't know whether this is the right place to report Windows-centric concerns, if not, my apologies.

Thanks, Tao

Next: Johannes Schindelin
Message 1 of 8 in “Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?”
  1. Tao KlerksJun 10, 2021
  2. Johannes SchindelinJun 11, 2021
  3. Tao KlerksJun 21, 2021
  4. Jeff HostetlerJun 21, 2021
  5. Tao KlerksJun 21, 2021
  6. Tao KlerksJun 24, 2021
  7. Tao KlerksJun 24, 2021
  8. Jeff HostetlerJun 24, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.