{"thread":{"id":"55894","subject":"Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?","startedAt":"2021-06-10T10:25:18Z","lastAt":"2021-06-24T18:52:08Z","messageCount":8,"participants":["Tao Klerks","Johannes Schindelin","Jeff Hostetler"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"426976","messageId":"CAPMMpog7bNNPm3suZKu6OppHA+KDYgCfmaxW4HqTAr7_tTVAPQ@mail.gmail.com","threadId":"55894","inReplyTo":null,"subject":"Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-10T10:24:04Z","receivedAt":"2021-06-10T10:25:18Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi folks,\n\nWith the new \"core.useBuiltinFSMonitor\" support in the Windows\ninstaller, I think this subject is worth calling out explicitly (and\nmy apologies if I missed prior discussion):\n\nTL;DR:\n - I believe \"core.untrackedcache\" should be enabled by default in\nWindows (and it does not appear to be)\n - If a user enables \"core.useBuiltinFSMonitor\" (eg in the installer)\nin the hopes of getting snappy \"git status\" on a repo with a large\ndeep working tree, they will be *unnecessarily* disappointed if\n\"core.untrackedcache\" is not enabled\n - There is also a lingering \"problem\" with \"git status -uall\", with\nboth \"core.useBuiltinFSMonitor\" and \"core.fsmonitor\", but that seems\nfar less trivial to address\n\nDetail:\n\nI just started testing the new \"core.useBuiltinFSMonitor\" option in\nthe new installer, and it's amazing, thanks Ben, Alex, Johannes and\nKevin!\n\nHowever, when I first enabled it, I was getting slightly *worse* git\nstatus times than without it... and those worse git status times were\naccompanied by a message along the lines of:\n---\nIt took 5.88 seconds to enumerate untracked files. 'status -uno' may\nspeed it up, but you have to be careful not to forget to add new files\nyourself (see 'git help status').\n---\n\nFor context, this is in a repo with 200,000 or so files, within 40,000\nfolders (avg path depth 4 I think?), with a reasonably-intricate set\nof .gitignore patterns. Obviously that's not \"your average user\", but\nI would imagine it matches \"the target audience for\n'core.useBuiltinFSMonitor'\" pretty well.\n\nAfter a little head-scratching, I recalled an exchange with Johannes\nfrom last year:\nhttps://lore.kernel.org/git/CAPMMpohJicVeCaKsPvommYbGEH-D1V02TTMaiVTV8ux+9z9vkQ@mail.gmail.com/\n\nI never did understand the relevant code paths in much detail, but the\npractical conclusions were:\n - Without \"core.untrackedcache\" enabled, git ends up iterating\nthrough the entire path structure of the working tree *even if\n\"core.fsmonitor\" (and now \"core.useBuiltinFSMonitor\") is enabled*,\nlooking for untracked files to report\n - Even with \"core.untrackedcache\" enabled, if \"core.fsmonitor\" (and\nnow \"core.useBuiltinFSMonitor\") is enabled, git iterates through the\nentire path structure of the working tree *single-threaded* when the\n\"--untracked-files\" mode is set to \"all\" (by config or command-line)\n\nNow, I imagine that addressing/improving these behaviors is very\nnon-trivial, but the impact could be reasonably limited if:\n - core.untrackedcache were defaulted to \"true\", at least under\nWindows, at least when the installer is asked to set\ncore.useBuiltinFSMonitor\n - The \"It took N.NN seconds to enumerate untracked files\" message\nwere to include a hint about core.untrackedcache, at least when the\n\"--untracked-files\" mode is set to \"normal\".\n\nFinal note: I personally would love to see \"core.useBuiltinFSMonitor\nactually makes things slower, when --untracked-files=all is specified\"\nbehavior be addressed, because common windows git integrations or\nfront-ends like Git Extensions or IntelliJ IDEA commonly use those\noptions, and therefore \"suffer\" a performance degradation on at least\nsome operations when core.useBuiltinFSMonitor is enabled.\n\nI don't know whether this is the right place to report Windows-centric\nconcerns, if not, my apologies.\n\nThanks,\nTao\n"},{"id":"427097","messageId":"nycvar.QRO.7.76.6.2106111122010.57@tvgsbejvaqbjf.bet","threadId":"55894","inReplyTo":"CAPMMpog7bNNPm3suZKu6OppHA+KDYgCfmaxW4HqTAr7_tTVAPQ@mail.gmail.com","subject":"Re: Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-06-11T09:49:18Z","receivedAt":"2021-06-11T09:49:31Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Tao,\n\nthank you for chiming in! It is good to see that more people are dabbling\nwith the built-in FSMonitor.\n\nOn Thu, 10 Jun 2021, Tao Klerks wrote:\n\n> With the new \"core.useBuiltinFSMonitor\" support in the Windows\n> installer, I think this subject is worth calling out explicitly (and\n> my apologies if I missed prior discussion):\n>\n> TL;DR:\n>  - I believe \"core.untrackedcache\" should be enabled by default in\n> Windows (and it does not appear to be)\n\nStolee indicated something similar that matches your observation:\nhttps://lore.kernel.org/git/af7a671c-fa32-6d9a-7d75-65582fdbcf24@gmail.com/\n\n\tInterestingly, the untracked cache extension makes a big\n\tdifference here. The performance of the overall behavior is much\n\tfaster if the untracked cache exists (when paired with the builtin\n\tFS Monitor; it doesn't make a significant difference when FS\n\tMonitor is disabled).\n\n>  - If a user enables \"core.useBuiltinFSMonitor\" (eg in the installer)\n> in the hopes of getting snappy \"git status\" on a repo with a large\n> deep working tree, they will be *unnecessarily* disappointed if\n> \"core.untrackedcache\" is not enabled\n\nYes.\n\nAnd. Unfortunately, there is an \"and\". I recently got a chance to work\nwith the Functional Tests of Scalar (\"an opinionated repository management\ntool\" on top of Git, see https://github.com/microsoft/scalar#readme for\nmore details). Essentially, you can think of that test suite as\nintegration tests for Git in a large-scale context. And there, I ran into\ntrouble with the untracked cache on Windows (where it really provides the\nmost benefit).\n\nThe gist of it is that _sometimes_, the mtime of a directory seems not to\nbe updated immediately after an item in it was modified/added/deleted. And\nthat mtime is precisely what the untracked cache depends on.\n\nThe funny thing is: while the output of `git status` will therefore at\nfirst fail to pick up on, say, a new untracked file, running `git status`\n_immediately_ afterwards _will succeed_ to see that untracked file. So\nthere is something fishy going on with updating things (it might even be a\nfoul interaction between the FSCache and the untracked cache, but I have\nno evidence to back that up or to disprove it).\n\nIt is one of my big TODOs to look into that. If you have any insights, or\ntime to investigate, I woud be really interested.\n\n>  - There is also a lingering \"problem\" with \"git status -uall\", with\n> both \"core.useBuiltinFSMonitor\" and \"core.fsmonitor\", but that seems\n> far less trivial to address\n\nInteresting. I guess the untracked cache might become too clunky with many\nuntracked files? Or is there something else going on?\n\n> Detail:\n>\n> I just started testing the new \"core.useBuiltinFSMonitor\" option in\n> the new installer, and it's amazing, thanks Ben, Alex, Johannes and\n> Kevin!\n\nNot to forget Jeff Hostetler, who essentially spent the past half year on\nit on his own.\n\n> However, when I first enabled it, I was getting slightly *worse* git\n> status times than without it... and those worse git status times were\n> accompanied by a message along the lines of:\n> ---\n> It took 5.88 seconds to enumerate untracked files. 'status -uno' may\n> speed it up, but you have to be careful not to forget to add new files\n> yourself (see 'git help status').\n> ---\n>\n> For context, this is in a repo with 200,000 or so files, within 40,000\n> folders (avg path depth 4 I think?), with a reasonably-intricate set\n> of .gitignore patterns. Obviously that's not \"your average user\", but\n> I would imagine it matches \"the target audience for\n> 'core.useBuiltinFSMonitor'\" pretty well.\n\nRight. I had a somewhat similar setup, with Git for Windows' SDK, which\nconsists of ~160k files in ~8k directories.\n\nMy `.gitignore` consists of only ~40 heavily commented lines (containing\nfive lines with wildcards), but I do have a `.git/info/exclude` that\ncontains a set of generated file/directory lists, i.e. without any\nwildcards. This `exclude` file is ~26k lines long.\n\nA cold-cache `git status` takes ~24sec, a warm-cache one ~10sec (with the\nbuilt-in FSMonitor daemon now active).\n\nMy guess is that the amount of work to match the untracked vs ignored\nfiles is dominating the entire operation, by a lot.\n\n> After a little head-scratching, I recalled an exchange with Johannes\n> from last year:\n> https://lore.kernel.org/git/CAPMMpohJicVeCaKsPvommYbGEH-D1V02TTMaiVTV8ux+9z9vkQ@mail.gmail.com/\n>\n> I never did understand the relevant code paths in much detail, but the\n> practical conclusions were:\n>  - Without \"core.untrackedcache\" enabled, git ends up iterating\n> through the entire path structure of the working tree *even if\n> \"core.fsmonitor\" (and now \"core.useBuiltinFSMonitor\") is enabled*,\n> looking for untracked files to report\n>  - Even with \"core.untrackedcache\" enabled, if \"core.fsmonitor\" (and\n> now \"core.useBuiltinFSMonitor\") is enabled, git iterates through the\n> entire path structure of the working tree *single-threaded* when the\n> \"--untracked-files\" mode is set to \"all\" (by config or command-line)\n>\n> Now, I imagine that addressing/improving these behaviors is very\n> non-trivial, but the impact could be reasonably limited if:\n>  - core.untrackedcache were defaulted to \"true\", at least under\n> Windows, at least when the installer is asked to set\n> core.useBuiltinFSMonitor\n\nAs soon as I can fix the flakiness of the untracked cache on Windows, I\nwill do that!\n\n>  - The \"It took N.NN seconds to enumerate untracked files\" message\n> were to include a hint about core.untrackedcache, at least when the\n> \"--untracked-files\" mode is set to \"normal\".\n>\n> Final note: I personally would love to see \"core.useBuiltinFSMonitor\n> actually makes things slower, when --untracked-files=all is specified\"\n> behavior be addressed,\n\nYes, we need to spend some quality time with some perf tools there.\n\n> because common windows git integrations or front-ends like Git\n> Extensions or IntelliJ IDEA commonly use those options, and therefore\n> \"suffer\" a performance degradation on at least some operations when\n> core.useBuiltinFSMonitor is enabled.\n>\n> I don't know whether this is the right place to report Windows-centric\n> concerns, if not, my apologies.\n\nI would not necessarily call them \"Windows-centric\", even if yes, at the\nmoment the built-in FSMonitor is most easily enabled on Windows (because I\nadded that experimental option in Git for Windows' installer, after\nintegrating the experimental feature).\n\nInstead, I consider this more the type of feedback concerning large\nworktrees, and what Git can do to support that use case better.\n\nIn particular the built-in FSMonitor, which already supports Windows and\nmacOS, and hopefully we will find volunteers to work on the Linux side\nsoon, too. In my mind, the built-in FSMonitor, the untracked cache, and\n`git maintenance` are _crucial_ tools to allow Git to scale up.\n\nSo: thank you for your wonderful feedback!\n\nCiao,\nDscho\n"},{"id":"428032","messageId":"CAPMMpogeWeQujG0UL80REOsaBJipxhQyOpBTuWD9U9_jg=FMMA@mail.gmail.com","threadId":"55894","inReplyTo":"nycvar.QRO.7.76.6.2106111122010.57@tvgsbejvaqbjf.bet","subject":"Re: Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-21T12:50:52Z","receivedAt":"2021-06-21T12:51:08Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi Johannes,\n\nThanks for the detailed reply!\n\nMy apologies for the subsequent delay - I've been trying to understand\nthe behavior so that I can describe it in more detail, and that\nrequired me to learn a little C along the way :)\n\nOn Fri, Jun 11, 2021 at 11:49 AM Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> _sometimes_, the mtime of a directory seems not to\n> be updated immediately after an item in it was modified/added/deleted. And\n> that mtime is precisely what the untracked cache depends on.\n>\n> The funny thing is: while the output of `git status` will therefore at\n> first fail to pick up on, say, a new untracked file, running `git status`\n> _immediately_ afterwards _will succeed_ to see that untracked file.\n\n(I have nothing to add here - I don't understand what\nsynchronous-acknowledgement or ordering guarantees we rely on to\ndetermine when we *expect* a change to be available to the fsmonitor,\nnor to I have any familiarity with the underlying APIs we use)\n\n> On Thu, 10 Jun 2021, Tao Klerks wrote:\n> >  - There is also a lingering \"problem\" with \"git status -uall\", with\n> > both \"core.useBuiltinFSMonitor\" and \"core.fsmonitor\", but that seems\n> > far less trivial to address\n>\n> Interesting. I guess the untracked cache might become too clunky with many\n> untracked files? Or is there something else going on?\n\nI think I understand this now, but I don't think I can explain it\nparticularly well/succinctly.\n\nI've attached a sort of \"truth table\" of performance data for a\nparticular repository, running *warm* git status calls (no cold-cache\ntesting at all) with various config and command-line options, and\nreporting the durations of various phases/processes captured using\nGIT_TRACE2_PERF=1.\n\nThe claimed \"problem\" with \"git status -uall\", when both\n\"core.useBuiltinFSMonitor\" and \"core.fsmonitor\" are enabled, only\nexists from the perspective of someone who's got core.fscache enabled\nall the time:\n* core.fscache works (in the Windows port only) by doing slightly more\nexpensive work up-front on first directory query within a\nrequest/process lifetime, and then intercepting subsequent filesystem\n\"queries\"/operations\n* The context within which most of this more-expensive work typically\noccurs, in a \"git status\" request, is an explicitly and intentionally\nmulti-threaded filesystem \"lstat\"-checking process in preload-index.c\n(always 20 threads, for a large repo)\n* There are two sets of filesystem-iteration in a simple/naive git\nstatus call with core.preloadindex enabled as by default - the\nlstat-checking multithreaded loops for files in the index, and the\nrecursive directory scanning for untracked files that happens later\n* That second (not-multithreaded) set of work, with fscache enabled,\ngets to reuse a bunch of cached fs data from the first one. A walk\nthat cost 7s without fscache now costs only 2.5s, for example.\n* With fsmonitor enabled (and warm), the first walk simply doesn't\nhappen, so fscache stops making any difference to that\nuntracked-file-searching directory walk; it goes back to taking 7s;\nevery directory is queried once in series, so fscache has no impact at\nall.\n* Because the preload-index lstat-querying loop is parallelized, with\nfscache the initial cache population happens fast-ish - the total time\ntaken for git status is eg 5s (2.5s of parallelized & accelerated\nlstat-querying and 2.5s of\nuntracked-folder-iterating-and-processing-from-fscache)\n* So, with fscache enabled, turning on fsmonitor actually makes \"git\nstatus\" *slower* - it changes the \"lstat + untracked\" time from \"2.5s\n+ 2.5s\" to \"0s + 7s\"\n* We can hide/mitigate that by enabling the untracked cache - but that\n\"fails out\" in all sorts of conditions specified in dir.c\nvalidate_untracked_cache(), including specifying \"-uall\"/\"-u\" to \"git\nstatus\n* -> it is only in the context of fscache being enabled, and already\nspeeding up the filesystem work, that fsmonitor can make things worse\nby making the first directory-querying loop happen in a non-parallel\narea of code, and thereby cancel fscache's impact.\n* -> fsmonitor never makes things worse on linux, since there is no\nfscache and so untracked folder iteration never benefits from any\nmultithreading\n* -> when the untracked cache does apply, fsmonitor can *help* it,\navoiding the need for any sort of directory walk at all, on the basis\nthat \"nothing in the filesystem appears to have changed\"\n\nBased on this understanding, it looks like there are at least three\npossible directions to be explored:\n\n1. Making the untracked cache less sensitive to configuration in\ndir.c#validate_untracked_cache(), at the cost of doing more work in\nthe cold cases & saving more data in the index file (specific\nuntracked files, and .git folder names or something)\n ** This would result in peak performance, with no\nfilesystem-iterating at all in the ideal case\n ** This would apply / add substantial value in Windows and Linux\n ** This is probably the most complex change - dir.c is not easy to\nunderstand/navigate\n\n2. Making the untracked directory-search happen in a multithreaded way\n ** This would raise the performance with \"-uall\" to approximately the\nsame as it was before fsmonitor was introduced on windows, and speed\nit up slightly on linux\n ** This change would probably not be worthwhile, - its impact would\nnot be huge except in very specific cases, and it would still\nintroduce non-trivial complexity\n\n3. Forcing preload-index to actually \"do its multithreaded work\", even\nwhen fsmonitor is there, if we know that the untracked cached cannot\nbe used and fsmonitor is enabled\n ** This would raise the performance with \"-uall\" to approximately the\nsame as it was before fsmonitor was introduced on windows\n ** This change would probably be pretty easy - the main challenge is\nhow to get untracked-cache-applicability information at preload-index\ntime, since these happen in completely different parts of the codebase\n\nI mocked up a local hack for the third option, and confirmed that\nforcing preload-index to ignore it does indeed bring \"git status\n-uall\"  performance back to the same level as before enabling\nfsmonitor.\n\n> > I just started testing the new \"core.useBuiltinFSMonitor\" option in\n> > the new installer, and it's amazing, thanks Ben, Alex, Johannes and\n> > Kevin!\n>\n> Not to forget Jeff Hostetler, who essentially spent the past half year on\n> it on his own.\n\nArgh, thank you for the correction, and thank you Jeff also for all\nyour work on this.\n\n> > For context, this is in a repo with 200,000 or so files, within 40,000\n> > folders (avg path depth 4 I think?), with a reasonably-intricate set\n> > of .gitignore patterns.\n\n> My `.gitignore` consists of only ~40 heavily commented lines (containing\n> five lines with wildcards), but I do have a `.git/info/exclude` that\n> contains a set of generated file/directory lists, i.e. without any\n> wildcards. This `exclude` file is ~26k lines long.\n>\n> My guess is that the amount of work to match the untracked vs ignored\n> files is dominating the entire operation, by a lot.\n\nIn my case, as per the \"truth table\" referenced above, there are two\nkinds of things in play:\n1. Filesystem operations are the main thing that matters in Windows\n2. Some smaller amount of overhead (0.5-1s in my case) is associated\nwith other work (pattern-matching etc) during untracked file\ndetection, even with untracked cache enabled. The only way to avoid\nthat work altogether is to have fsmonitor *and* untracked cache\nworking, so that untracked cache can \"trust\" the fsmonitor results to\navoid having to do any work at all. In this ideal situation fscache\nmakes no difference, as there is no fs iteration.\n\n> > I don't know whether this is the right place to report Windows-centric\n> > concerns, if not, my apologies.\n>\n> I would not necessarily call them \"Windows-centric\", even if yes, at the\n> moment the built-in FSMonitor is most easily enabled on Windows (because I\n> added that experimental option in Git for Windows' installer, after\n> integrating the experimental feature).\n\nRight - now that I understand the situation better, there are three\nspecific ways in which I consider these to be windows-centric\nconcerns:\n* In my experience / in my context at least, the \"naive\" impact of\nfile operation performance differences results in a more than 10X \"git\nstatus\" reduction in performance for large repos (over linux); various\noptional and/or windows-specific strategies significantly close that\ngap\n* core.fscache is a windows-specific strategy for dealing with this,\nand interacts with other features/strategies in potentially-surprising\nways\n* The built-in FSMonitor is still only *easily* available in Windows\n\nBut, to your point, most of the *solutions* need not be\nwindows-centric at all. The \"best\" one, making untracked cache a\nlittle more forgiving, would definitely have tangible performance\nbenefits on linux (and presumably OSX).\n\nThanks,\nTao\n\n\nInput: OS,Input: FSCache,Input: UntrackedCache,Input: FSMonitor,Input: -uall,In: PreloadIndex,In: Command,Timing: read index,Timing: query fsmonitor,Timing: preload lstat 207k files,Timing: refresh,Timing: traverse trees,Timing: Name-hash-init,Timing: untracked,Approx Total Timing\nLinux,(N/A),false,disabled,(irrelevant),true,export GIT_TRACE2_PERF=1 && git -c core.untrackedcache=false -c core.fsmonitor= -c core.preloadindex=true status,0.04,0,0.07,0,0,0.06,0.82,0.99\nLinux,(N/A),true,disabled,false,true,export GIT_TRACE2_PERF=1 && git -c core.untrackedcache=true -c core.fsmonitor= -c core.preloadindex=true status,0.05,0,0.07,0,0,0.06,0.32,0.5\nLinux,(N/A),true,disabled,true,true,export GIT_TRACE2_PERF=1 && git -c core.untrackedcache=true -c core.fsmonitor= -c core.preloadindex=true status -u,0.04,0,0.07,0,0,0.07,0.78,0.96\nLinux,(N/A),false,disabled,(irrelevant),false,export GIT_TRACE2_PERF=1 && git -c core.untrackedcache=false -c core.fsmonitor= -c core.preloadindex=false status,0.04,0,0,0.37,0,0.07,0.87,1.35\nLinux,(N/A),true,disabled,false,false,export GIT_TRACE2_PERF=1 && git -c core.untrackedcache=true -c core.fsmonitor= -c core.preloadindex=false status,0.05,0,0,0.39,0,0.06,0.31,0.81\nLinux,(N/A),true,disabled,true,false,export GIT_TRACE2_PERF=1 && git -c core.untrackedcache=true -c core.fsmonitor= -c core.preloadindex=false status -u,0.05,0,0,0.38,0,0.07,0.78,1.28\nLinux,(N/A),false,enabled,(irrelevant),(irrelevant),export GIT_TRACE2_PERF=1 && git -c core.untrackedcache=false -c core.fsmonitor=/home/tao/build-git/.git/hooks/fsmonitor-watchman.sample status,0.04,0.04,0,0.02,0,0.06,0.81,0.97\nLinux,(N/A),true,enabled,false,(irrelevant),export GIT_TRACE2_PERF=1 && git -c core.untrackedcache=true -c core.fsmonitor=/home/tao/build-git/.git/hooks/fsmonitor-watchman.sample status,0.04,0.04,0,0.02,0,0.07,0.26,0.43\nLinux,(N/A),true,enabled,true,(irrelevant),export GIT_TRACE2_PERF=1 && git -c core.untrackedcache=true -c core.fsmonitor=/home/tao/build-git/.git/hooks/fsmonitor-watchman.sample status -u,0.05,0.04,0,0.02,0,0.07,0.78,0.96\nWindows,false,false,disabled,(irrelevant),true,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=false -c core.untrackedcache=false -c core.useBuiltInFSMonitor=false -c core.preloadindex=true status,0.07,0,3.36,0,0.32,0.03,5.89,9.67\nWindows,false,true,disabled,false,true,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=false -c core.untrackedcache=true -c core.useBuiltInFSMonitor=false -c core.preloadindex=true status,0.05,0,3.1,0,0.32,0.03,2.46,5.96\nWindows,false,true,disabled,true,true,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=false -c core.untrackedcache=true -c core.useBuiltInFSMonitor=false -c core.preloadindex=true status -u,0.05,0,3.05,0,0.31,0.03,5.95,9.39\nWindows,false,false,disabled,(irrelevant),false,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=false -c core.untrackedcache=false -c core.useBuiltInFSMonitor=false -c core.preloadindex=false status,0.05,0,0,12.12,0.31,0.03,5.95,18.46\nWindows,false,true,disabled,false,false,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=false -c core.untrackedcache=true -c core.useBuiltInFSMonitor=false -c core.preloadindex=false status,0.05,0,0,12.17,0.31,0.03,2.26,14.82\nWindows,false,true,disabled,true,false,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=false -c core.untrackedcache=true -c core.useBuiltInFSMonitor=false -c core.preloadindex=false status -u,0.05,0,0,12.13,0.31,0.03,6.1,18.62\nWindows,false,false,enabled,(irrelevant),(irrelevant),export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=false -c core.untrackedcache=false -c core.useBuiltInFSMonitor=true status,0.04,0,0,0.02,0.33,0.03,6.43,6.85\nWindows,false,true,enabled,false,(irrelevant),export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=false -c core.untrackedcache=true -c core.useBuiltInFSMonitor=true status,0.05,0,0,0.01,0.33,0.03,0.48,0.9\nWindows,false,true,enabled,true,(irrelevant),export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=false -c core.untrackedcache=true -c core.useBuiltInFSMonitor=true status -u,0.05,0,0,0.01,0.32,0.03,6.19,6.6\nWindows,true,false,disabled,(irrelevant),true,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=true -c core.untrackedcache=false -c core.useBuiltInFSMonitor=false -c core.preloadindex=true status,0.05,0,1.8,0,0.31,0.03,2.1,4.29\nWindows,true,true,disabled,false,true,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=true -c core.untrackedcache=true -c core.useBuiltInFSMonitor=false -c core.preloadindex=true status,0.06,0,1.5,0.01,0.35,0.03,0.98,2.93\nWindows,true,true,disabled,true,true,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=true -c core.untrackedcache=true -c core.useBuiltInFSMonitor=false -c core.preloadindex=true status -u,0.05,0,1.68,0.01,0.42,0.03,2,4.19\nWindows,true,false,disabled,(irrelevant),false,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=true -c core.untrackedcache=false -c core.useBuiltInFSMonitor=false -c core.preloadindex=false status,0.04,0,0,3.34,0.33,0.03,2.09,5.83\nWindows,true,true,disabled,false,false,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=true -c core.untrackedcache=true -c core.useBuiltInFSMonitor=false -c core.preloadindex=false status,0.05,0,0,3.46,0.34,0.03,0.84,4.72\nWindows,true,true,disabled,true,false,export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=true -c core.untrackedcache=true -c core.useBuiltInFSMonitor=false -c core.preloadindex=false status -u,0.05,0,0,3.38,0.32,0.04,1.85,5.64\nWindows,true,false,enabled,(irrelevant),(irrelevant),export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=true -c core.untrackedcache=false -c core.useBuiltInFSMonitor=true status,0.04,0,0.01,0.01,0.33,0.03,4.9,5.32\nWindows,true,true,enabled,false,(irrelevant),export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=true -c core.untrackedcache=true -c core.useBuiltInFSMonitor=true status,0.05,0,0.01,0.01,0.34,0.03,0.5,0.94\nWindows,true,true,enabled,true,(irrelevant),export GIT_TRACE2_PERF=1 && time /usr/src/git/git.exe -c core.fscache=true -c core.untrackedcache=true -c core.useBuiltInFSMonitor=true status -u,0.06,0,0.01,0.01,0.31,0.03,5.14,5.56\n"},{"id":"428086","messageId":"81153d02-8e7a-be59-e709-e90cd5906f3a@jeffhostetler.com","threadId":"55894","inReplyTo":"CAPMMpogeWeQujG0UL80REOsaBJipxhQyOpBTuWD9U9_jg=FMMA@mail.gmail.com","subject":"Re: Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-06-21T18:41:36Z","receivedAt":"2021-06-21T18:41:40Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 6/21/21 8:50 AM, Tao Klerks wrote:\n> Hi Johannes,\n> \n> Thanks for the detailed reply!\n> \n> My apologies for the subsequent delay - I've been trying to understand\n> the behavior so that I can describe it in more detail, and that\n> required me to learn a little C along the way :)\n> \n> On Fri, Jun 11, 2021 at 11:49 AM Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n>> _sometimes_, the mtime of a directory seems not to\n>> be updated immediately after an item in it was modified/added/deleted. And\n>> that mtime is precisely what the untracked cache depends on.\n>>\n>> The funny thing is: while the output of `git status` will therefore at\n>> first fail to pick up on, say, a new untracked file, running `git status`\n>> _immediately_ afterwards _will succeed_ to see that untracked file.\n> \n> (I have nothing to add here - I don't understand what\n> synchronous-acknowledgement or ordering guarantees we rely on to\n> determine when we *expect* a change to be available to the fsmonitor,\n> nor to I have any familiarity with the underlying APIs we use)\n> \n>> On Thu, 10 Jun 2021, Tao Klerks wrote:\n>>>   - There is also a lingering \"problem\" with \"git status -uall\", with\n>>> both \"core.useBuiltinFSMonitor\" and \"core.fsmonitor\", but that seems\n>>> far less trivial to address\n>>\n>> Interesting. I guess the untracked cache might become too clunky with many\n>> untracked files? Or is there something else going on?\n> \n> I think I understand this now, but I don't think I can explain it\n> particularly well/succinctly.\n> \n> I've attached a sort of \"truth table\" of performance data for a\n> particular repository, running *warm* git status calls (no cold-cache\n> testing at all) with various config and command-line options, and\n> reporting the durations of various phases/processes captured using\n> GIT_TRACE2_PERF=1.\n> \n> The claimed \"problem\" with \"git status -uall\", when both\n> \"core.useBuiltinFSMonitor\" and \"core.fsmonitor\" are enabled, only\n> exists from the perspective of someone who's got core.fscache enabled\n> all the time:\n> * core.fscache works (in the Windows port only) by doing slightly more\n> expensive work up-front on first directory query within a\n> request/process lifetime, and then intercepting subsequent filesystem\n> \"queries\"/operations\n> * The context within which most of this more-expensive work typically\n> occurs, in a \"git status\" request, is an explicitly and intentionally\n> multi-threaded filesystem \"lstat\"-checking process in preload-index.c\n> (always 20 threads, for a large repo)\n> * There are two sets of filesystem-iteration in a simple/naive git\n> status call with core.preloadindex enabled as by default - the\n> lstat-checking multithreaded loops for files in the index, and the\n> recursive directory scanning for untracked files that happens later\n> * That second (not-multithreaded) set of work, with fscache enabled,\n> gets to reuse a bunch of cached fs data from the first one. A walk\n> that cost 7s without fscache now costs only 2.5s, for example.\n> * With fsmonitor enabled (and warm), the first walk simply doesn't\n> happen, so fscache stops making any difference to that\n> untracked-file-searching directory walk; it goes back to taking 7s;\n> every directory is queried once in series, so fscache has no impact at\n> all.\n> * Because the preload-index lstat-querying loop is parallelized, with\n> fscache the initial cache population happens fast-ish - the total time\n> taken for git status is eg 5s (2.5s of parallelized & accelerated\n> lstat-querying and 2.5s of\n> untracked-folder-iterating-and-processing-from-fscache)\n> * So, with fscache enabled, turning on fsmonitor actually makes \"git\n> status\" *slower* - it changes the \"lstat + untracked\" time from \"2.5s\n> + 2.5s\" to \"0s + 7s\"\n> * We can hide/mitigate that by enabling the untracked cache - but that\n> \"fails out\" in all sorts of conditions specified in dir.c\n> validate_untracked_cache(), including specifying \"-uall\"/\"-u\" to \"git\n> status\n> * -> it is only in the context of fscache being enabled, and already\n> speeding up the filesystem work, that fsmonitor can make things worse\n> by making the first directory-querying loop happen in a non-parallel\n> area of code, and thereby cancel fscache's impact.\n> * -> fsmonitor never makes things worse on linux, since there is no\n> fscache and so untracked folder iteration never benefits from any\n> multithreading\n> * -> when the untracked cache does apply, fsmonitor can *help* it,\n> avoiding the need for any sort of directory walk at all, on the basis\n> that \"nothing in the filesystem appears to have changed\"\n> \n> Based on this understanding, it looks like there are at least three\n> possible directions to be explored:\n> \n> 1. Making the untracked cache less sensitive to configuration in\n> dir.c#validate_untracked_cache(), at the cost of doing more work in\n> the cold cases & saving more data in the index file (specific\n> untracked files, and .git folder names or something)\n>   ** This would result in peak performance, with no\n> filesystem-iterating at all in the ideal case\n>   ** This would apply / add substantial value in Windows and Linux\n>   ** This is probably the most complex change - dir.c is not easy to\n> understand/navigate\n> \n> 2. Making the untracked directory-search happen in a multithreaded way\n>   ** This would raise the performance with \"-uall\" to approximately the\n> same as it was before fsmonitor was introduced on windows, and speed\n> it up slightly on linux\n>   ** This change would probably not be worthwhile, - its impact would\n> not be huge except in very specific cases, and it would still\n> introduce non-trivial complexity\n> \n> 3. Forcing preload-index to actually \"do its multithreaded work\", even\n> when fsmonitor is there, if we know that the untracked cached cannot\n> be used and fsmonitor is enabled\n>   ** This would raise the performance with \"-uall\" to approximately the\n> same as it was before fsmonitor was introduced on windows\n>   ** This change would probably be pretty easy - the main challenge is\n> how to get untracked-cache-applicability information at preload-index\n> time, since these happen in completely different parts of the codebase\n> \n> I mocked up a local hack for the third option, and confirmed that\n> forcing preload-index to ignore it does indeed bring \"git status\n> -uall\"  performance back to the same level as before enabling\n> fsmonitor.\n> \n>>> I just started testing the new \"core.useBuiltinFSMonitor\" option in\n>>> the new installer, and it's amazing, thanks Ben, Alex, Johannes and\n>>> Kevin!\n>>\n>> Not to forget Jeff Hostetler, who essentially spent the past half year on\n>> it on his own.\n> \n> Argh, thank you for the correction, and thank you Jeff also for all\n> your work on this.\n> \n>>> For context, this is in a repo with 200,000 or so files, within 40,000\n>>> folders (avg path depth 4 I think?), with a reasonably-intricate set\n>>> of .gitignore patterns.\n> \n>> My `.gitignore` consists of only ~40 heavily commented lines (containing\n>> five lines with wildcards), but I do have a `.git/info/exclude` that\n>> contains a set of generated file/directory lists, i.e. without any\n>> wildcards. This `exclude` file is ~26k lines long.\n>>\n>> My guess is that the amount of work to match the untracked vs ignored\n>> files is dominating the entire operation, by a lot.\n> \n> In my case, as per the \"truth table\" referenced above, there are two\n> kinds of things in play:\n> 1. Filesystem operations are the main thing that matters in Windows\n> 2. Some smaller amount of overhead (0.5-1s in my case) is associated\n> with other work (pattern-matching etc) during untracked file\n> detection, even with untracked cache enabled. The only way to avoid\n> that work altogether is to have fsmonitor *and* untracked cache\n> working, so that untracked cache can \"trust\" the fsmonitor results to\n> avoid having to do any work at all. In this ideal situation fscache\n> makes no difference, as there is no fs iteration.\n> \n>>> I don't know whether this is the right place to report Windows-centric\n>>> concerns, if not, my apologies.\n>>\n>> I would not necessarily call them \"Windows-centric\", even if yes, at the\n>> moment the built-in FSMonitor is most easily enabled on Windows (because I\n>> added that experimental option in Git for Windows' installer, after\n>> integrating the experimental feature).\n> \n> Right - now that I understand the situation better, there are three\n> specific ways in which I consider these to be windows-centric\n> concerns:\n> * In my experience / in my context at least, the \"naive\" impact of\n> file operation performance differences results in a more than 10X \"git\n> status\" reduction in performance for large repos (over linux); various\n> optional and/or windows-specific strategies significantly close that\n> gap\n> * core.fscache is a windows-specific strategy for dealing with this,\n> and interacts with other features/strategies in potentially-surprising\n> ways\n> * The built-in FSMonitor is still only *easily* available in Windows\n> \n> But, to your point, most of the *solutions* need not be\n> windows-centric at all. The \"best\" one, making untracked cache a\n> little more forgiving, would definitely have tangible performance\n> benefits on linux (and presumably OSX).\n> \n> Thanks,\n> Tao\n> \n\nNice summary.  Thanks!\n\nWe're currently looking at a problem that we believe is in the\nuntracked-cache code.  This is causing some of our Scalar tests\nto fail on Windows when the untracked-cache is turned on.  This\nis independent of whether FSMonitor or FSCache is turned on.\nWe're still tracking this down.\n\nAnd yes, the best possible solution is to turn on FSMonitor *and*\nthe untracked-cache, so that the \"untracked\" status code doesn't\nhave to do anything.  So I want to look at tracking down the above\nproblem before doing anything else.\n\nThanks\nJeff\n\n\n\n"},{"id":"428107","messageId":"CAPMMpogjr43111HfPzk23c98JBaiEF2YA_OTkQVG1a63zyeiOw@mail.gmail.com","threadId":"55894","inReplyTo":"81153d02-8e7a-be59-e709-e90cd5906f3a@jeffhostetler.com","subject":"Re: Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-21T20:52:02Z","receivedAt":"2021-06-21T20:52:18Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi Jeff, thanks for the update!\n\nOn Mon, Jun 21, 2021 at 8:41 PM Jeff Hostetler <git@jeffhostetler.com> wrote:\n> We're currently looking at a problem that we believe is in the\n> untracked-cache code.  This is causing some of our Scalar tests\n> to fail on Windows when the untracked-cache is turned on.\n\nFor what it's worth, I just discovered an untracked cache bug this\nevening, although I doubt it's related to the one you mention - it's\nnot very exciting:\n\nIf you disable untracked cache (and write an index file), and then\nenable untracked cache and run status with \"-uall\" (writing a new\nindex file), the untracked cache data written in the new index file is\nempty/invalid, and subsequent \"git status\" calls perform just the same\nas if untracked cache were disabled:\n----\n# write an index without untracked cache\ngit -c core.untrackedcache=false status\n\n# write another index with invalid/empty/bad untracked cache because\n\"-uall\" skipped its population/maintenance\ngit -c core.untrackedcache=true status -uall # expected to be slow\n\n# run regular \"git status\" (with untracked cache) any number of times,\nbut don't get the benefit (and you don't write a new index because\nnothing appears to have changed)\ngit -c core.untrackedcache=true status # unexpectedly slow\ngit -c core.untrackedcache=true status # unexpectedly slow\ngit -c core.untrackedcache=true status # unexpectedly slow\n---\n# TO FIX:\n# write a new index file without untracked cache\ngit -c core.untrackedcache=false status\n\n# run regular \"git status\" (with untracked cache), does work and\nwrites a new index file\ngit -c core.untrackedcache=true status # slow as expected\n\n# run regular \"git status\" (with untracked cache) any number of times,\nis fast as expected\ngit -c core.untrackedcache=true status # fast as expected\ngit -c core.untrackedcache=true status # fast as expected\ngit -c core.untrackedcache=true status # fast as expected\n----\n\nI suspect this issue has bit me in the past when attempting to\nunderstand untracked cache behavior; it can be *very* confusing, if\nyou're using tooling like \"git extensions\" that can, in the above\nflow, \"poison\" your untracked cache if it just happens to run a \"git\nstatus -uall\" in the background as you are testing.\n\n(I discovered this issue while trying to understand the weird &\nwonderful relationship between the repo-level untracked cache\nreference, the dir-level untracked cache reference, and the mechanisms\nthat initialize a new one at dir.c#new_untracked_cache() and write the\nrepo-level one (even if the dir-level reference was voided due to flag\nmismatch) at dir.c#write_untracked_extension())\n\nShould I be reporting this in some more \"official\" form somewhere? Is\nthere a bug DB?\n\nThanks,\nTao\n"},{"id":"428361","messageId":"CAPMMpojKMZmorKPgMASkHsvKcQTaaGyWURsgDS+KTGVeuBh1ww@mail.gmail.com","threadId":"55894","inReplyTo":"81153d02-8e7a-be59-e709-e90cd5906f3a@jeffhostetler.com","subject":"Re: Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-24T05:25:05Z","receivedAt":"2021-06-24T05:25:19Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi Jeff,\n\nOn Mon, Jun 21, 2021 at 8:41 PM Jeff Hostetler <git@jeffhostetler.com> wrote:\n> We're currently looking at a problem that we believe is in the\n> untracked-cache code.  This is causing some of our Scalar tests\n> to fail on Windows when the untracked-cache is turned on.  This\n> is independent of whether FSMonitor or FSCache is turned on.\n> We're still tracking this down.\n>\n> And yes, the best possible solution is to turn on FSMonitor *and*\n> the untracked-cache, so that the \"untracked\" status code doesn't\n> have to do anything.  So I want to look at tracking down the above\n> problem before doing anything else.\n\nI got a bit excited about a possible clean path forward to getting\n-uall to work well with untracked cache, and submitted a patch along\nthose lines, but rereading the above I should probably have been a\nlittle more patient.\n\nIs there anything \"we\" can do to see/understand the\nscalar-test-suite-error you describe above, or is this\nmicrosoft-internal?\n\nThanks\nTao\n"},{"id":"428372","messageId":"596386e9-f9bf-c465-ec5e-dcf4356b6194@jeffhostetler.com","threadId":"55894","inReplyTo":"CAPMMpojKMZmorKPgMASkHsvKcQTaaGyWURsgDS+KTGVeuBh1ww@mail.gmail.com","subject":"Re: Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-06-24T13:10:35Z","receivedAt":"2021-06-24T13:10:38Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 6/24/21 1:25 AM, Tao Klerks wrote:\n> Hi Jeff,\n> \n> On Mon, Jun 21, 2021 at 8:41 PM Jeff Hostetler <git@jeffhostetler.com> wrote:\n>> We're currently looking at a problem that we believe is in the\n>> untracked-cache code.  This is causing some of our Scalar tests\n>> to fail on Windows when the untracked-cache is turned on.  This\n>> is independent of whether FSMonitor or FSCache is turned on.\n>> We're still tracking this down.\n>>\n>> And yes, the best possible solution is to turn on FSMonitor *and*\n>> the untracked-cache, so that the \"untracked\" status code doesn't\n>> have to do anything.  So I want to look at tracking down the above\n>> problem before doing anything else.\n> \n> I got a bit excited about a possible clean path forward to getting\n> -uall to work well with untracked cache, and submitted a patch along\n> those lines, but rereading the above I should probably have been a\n> little more patient.\n> \n> Is there anything \"we\" can do to see/understand the\n> scalar-test-suite-error you describe above, or is this\n> microsoft-internal?\n> \n> Thanks\n> Tao\n> \n\nThanks for looking into this.  The untracked-cache code is pretty\ndense and having another set of eyes is good.  I apologize that I'm\nstill working thru the backlog from my vacation and haven't gotten\nto spend any \"quality\" time with the untracked-cache code yet, so\nI need to do some homework and study the questions/issues that you've\nfound so far. (Thanks again)\n\nAll of our work is done in the open, so yes you should be able to\nsee what we're doing and the errors that we're getting.\n\nThe source for our Scalar functional tests is in:\nhttps://github.com/microsoft/scalar\n\nMy test branch 'test-no-fscache' is in my personal development fork:\nhttps://github.com/jeffhostetler/git\n\nI have a PR against microsoft/git (a fork of git-for-windows\nwhich is a fork of core git) which turns off a bunch of things\nto try to isolate the failures:\nhttps://github.com/microsoft/git/pull/383\n\nOutput for a recent run can be seen here:\nhttps://github.com/microsoft/git/pull/383/checks?check_run_id=2884524819\n\nLet us know if you have questions.\nJeff\n"},{"id":"428396","messageId":"CAPMMpogek1N=5D8OVBLGXyWGgur4=-1Txever0P=86Lr7AEeKQ@mail.gmail.com","threadId":"55894","inReplyTo":"CAPMMpogjr43111HfPzk23c98JBaiEF2YA_OTkQVG1a63zyeiOw@mail.gmail.com","subject":"Re: Windows: core.useBuiltinFSMonitor without core.untrackedcache - performance hazard?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-24T18:51:55Z","receivedAt":"2021-06-24T18:52:08Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Fwiw, I've now submitted a patch(set) that I believe addresses this\nparticular issue correctly:\n\nhttps://lore.kernel.org/git/pull.986.git.1624559401.gitgitgadget@gmail.com/\n\nThanks,\nTao\n\nOn Mon, Jun 21, 2021 at 10:52 PM Tao Klerks <tao@klerks.biz> wrote:\n>\n> Hi Jeff, thanks for the update!\n>\n> On Mon, Jun 21, 2021 at 8:41 PM Jeff Hostetler <git@jeffhostetler.com> wrote:\n> > We're currently looking at a problem that we believe is in the\n> > untracked-cache code.  This is causing some of our Scalar tests\n> > to fail on Windows when the untracked-cache is turned on.\n>\n> For what it's worth, I just discovered an untracked cache bug this\n> evening, although I doubt it's related to the one you mention - it's\n> not very exciting:\n>\n> If you disable untracked cache (and write an index file), and then\n> enable untracked cache and run status with \"-uall\" (writing a new\n> index file), the untracked cache data written in the new index file is\n> empty/invalid, and subsequent \"git status\" calls perform just the same\n> as if untracked cache were disabled:\n> ----\n> # write an index without untracked cache\n> git -c core.untrackedcache=false status\n>\n> # write another index with invalid/empty/bad untracked cache because\n> \"-uall\" skipped its population/maintenance\n> git -c core.untrackedcache=true status -uall # expected to be slow\n>\n> # run regular \"git status\" (with untracked cache) any number of times,\n> but don't get the benefit (and you don't write a new index because\n> nothing appears to have changed)\n> git -c core.untrackedcache=true status # unexpectedly slow\n> git -c core.untrackedcache=true status # unexpectedly slow\n> git -c core.untrackedcache=true status # unexpectedly slow\n> ---\n> # TO FIX:\n> # write a new index file without untracked cache\n> git -c core.untrackedcache=false status\n>\n> # run regular \"git status\" (with untracked cache), does work and\n> writes a new index file\n> git -c core.untrackedcache=true status # slow as expected\n>\n> # run regular \"git status\" (with untracked cache) any number of times,\n> is fast as expected\n> git -c core.untrackedcache=true status # fast as expected\n> git -c core.untrackedcache=true status # fast as expected\n> git -c core.untrackedcache=true status # fast as expected\n> ----\n>\n> I suspect this issue has bit me in the past when attempting to\n> understand untracked cache behavior; it can be *very* confusing, if\n> you're using tooling like \"git extensions\" that can, in the above\n> flow, \"poison\" your untracked cache if it just happens to run a \"git\n> status -uall\" in the background as you are testing.\n>\n> (I discovered this issue while trying to understand the weird &\n> wonderful relationship between the repo-level untracked cache\n> reference, the dir-level untracked cache reference, and the mechanisms\n> that initialize a new one at dir.c#new_untracked_cache() and write the\n> repo-level one (even if the dir-level reference was voided due to flag\n> mismatch) at dir.c#write_untracked_extension())\n>\n> Should I be reporting this in some more \"official\" form somewhere? Is\n> there a bug DB?\n>\n> Thanks,\n> Tao\n"}]}