{"thread":{"id":"54273","subject":"Question about fsmonitor and --untracked-files=all","startedAt":"2020-09-22T11:35:58Z","lastAt":"2020-09-24T12:15:11Z","messageCount":3,"participants":["Tao Klerks","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"406123","messageId":"CAPMMpoj+UhKCW_k34-cGkiWFghOOu13GhPgA0V-y4ZpLVppuiA@mail.gmail.com","threadId":"54273","inReplyTo":null,"subject":"Question about fsmonitor and --untracked-files=all","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2020-09-22T11:35:42Z","receivedAt":"2020-09-22T11:35:58Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi folks,\n\nI've got a couple questions about the \"fsmonitor\" functionality,\nuntracked files, and multithreading.\n\nBackground:\n\nIn a repo with:\n * A couple hundred thousand tracked files, and a couple hundred\nthousand .gitignored files, across a few thousand directories\n * The --untracked-cache setting, tested and working\n * core.fsmonitor set up with watchman (with the sample integration\nscript from january)\n * Git version 2.27.0.windows.1\n\n\"git status\" takes about 2s\n\"git status --untracked-files=all\" takes about 20s\n\nWhen I turn off \"core.fsmonitor\", the numbers change to something like:\n\"git status\": 8s\n\"git status --untracked-files=all\": 9s\n\nUsing windows' \"procmon\" to observe git.exe's behavior from outside, I\nthink I've understood a couple things that surprise me:\n1. when you specify \"--untracked-files=all\", git scans the entire\nfolder tree regardless of the \"fsmonitor\" hook\n2. when you specify the \"fsmonitor\" hook, git does any\nfilesystem-scanning in a single-threaded fashion (as opposed to\nmulti-threaded without \"fsmonitor\" / normally)\n\nThese two things combine so that with \"fsmonitor\" set, normal\ncommand-line git status performance is great, but the performance in\ntools that eagerly look for untracked files (like \"Git Extensions\" on\nwindows) actually suffers - it takes twice as long to run the 'git -c\ndiff.ignoreSubModules=none status --porcelain=2 -z\n--untracked-files=all' command that this UI wants (and blocks on, when\nyou go to a commit dialog).\n\nQuestions:\n\n1. Is there a reason \"--untracked-files=all\" causes a full directory\ntree scan even with the \"fsmonitor\" hook active, or is this\naccidental?\n2. Assuming that the full directory tree scan is indeed necessary even\nwith \"fsmonitor\" (when requesting all untracked files), could it be\nmade multithreaded?\n\n(my apologies for the simplistic \"outside-in\" observations; I don't\nfeel qualified to attempt to understand the git source code)\n\nThanks for any help understanding the optimization opportunities here!\n\nTao Klerks\n"},{"id":"406195","messageId":"nycvar.QRO.7.76.6.2009231238560.5061@tvgsbejvaqbjf.bet","threadId":"54273","inReplyTo":"CAPMMpoj+UhKCW_k34-cGkiWFghOOu13GhPgA0V-y4ZpLVppuiA@mail.gmail.com","subject":"Re: Question about fsmonitor and --untracked-files=all","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-09-23T10:40:09Z","receivedAt":"2020-09-23T14:42:31Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Tao,\n\nOn Tue, 22 Sep 2020, Tao Klerks wrote:\n\n> I've got a couple questions about the \"fsmonitor\" functionality,\n> untracked files, and multithreading.\n>\n> Background:\n>\n> In a repo with:\n>  * A couple hundred thousand tracked files, and a couple hundred\n> thousand .gitignored files, across a few thousand directories\n>  * The --untracked-cache setting, tested and working\n>  * core.fsmonitor set up with watchman (with the sample integration\n> script from january)\n>  * Git version 2.27.0.windows.1\n>\n> \"git status\" takes about 2s\n> \"git status --untracked-files=all\" takes about 20s\n>\n> When I turn off \"core.fsmonitor\", the numbers change to something like:\n> \"git status\": 8s\n> \"git status --untracked-files=all\": 9s\n>\n> Using windows' \"procmon\" to observe git.exe's behavior from outside, I\n> think I've understood a couple things that surprise me:\n> 1. when you specify \"--untracked-files=all\", git scans the entire\n> folder tree regardless of the \"fsmonitor\" hook\n> 2. when you specify the \"fsmonitor\" hook, git does any\n> filesystem-scanning in a single-threaded fashion (as opposed to\n> multi-threaded without \"fsmonitor\" / normally)\n>\n> These two things combine so that with \"fsmonitor\" set, normal\n> command-line git status performance is great, but the performance in\n> tools that eagerly look for untracked files (like \"Git Extensions\" on\n> windows) actually suffers - it takes twice as long to run the 'git -c\n> diff.ignoreSubModules=none status --porcelain=2 -z\n> --untracked-files=all' command that this UI wants (and blocks on, when\n> you go to a commit dialog).\n>\n> Questions:\n>\n> 1. Is there a reason \"--untracked-files=all\" causes a full directory\n> tree scan even with the \"fsmonitor\" hook active, or is this\n> accidental?\n\nI have a hunch that this might be related to a performance hack we have in\nGit for Windows: did you enable FSCache perchance?\n\nIf so, I _suspect_ that turning it off would accelerate `git status\n--untracked-files=all`.\n\nCiao,\nJohannes\n\n> 2. Assuming that the full directory tree scan is indeed necessary even\n> with \"fsmonitor\" (when requesting all untracked files), could it be\n> made multithreaded?\n>\n> (my apologies for the simplistic \"outside-in\" observations; I don't\n> feel qualified to attempt to understand the git source code)\n>\n> Thanks for any help understanding the optimization opportunities here!\n>\n> Tao Klerks\n>\n"},{"id":"406254","messageId":"CAPMMpohJicVeCaKsPvommYbGEH-D1V02TTMaiVTV8ux+9z9vkQ@mail.gmail.com","threadId":"54273","inReplyTo":"nycvar.QRO.7.76.6.2009231238560.5061@tvgsbejvaqbjf.bet","subject":"Re: Question about fsmonitor and --untracked-files=all","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2020-09-24T12:14:56Z","receivedAt":"2020-09-24T12:15:11Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi Johannes,\n\nThanks for the tip - unfortunately, that doesn't seem to have worked /\nhad any positive effect.\n\nWith \"git config core.fscache false\", everything/anything takes longer\nexcept a simple \"git status\" with the fsmonitor enabled and the\nuntrackedCache enabled (in which case I guess nothing ends up needing\nthe filesystem). This combination (fsmonitor enabled, untrackedCache\nenabled, and running simple \"git status\") is the *only* combination\nthat I've found so far that doesn't force a directory scan - and\n*when* there is a directory scan (because of \"--untracked-files=all\",\nor because the fsmonitor is disabled, or because the untrackedCache is\ndisabled), then having fscache disabled makes things significantly\nworse/slower (20% slower to double the time, depending on the exact\ncombination).\n\nI tried to stumble my way around some of the source code, and I\nsuspect I've found at least one explanation: The untracked cache\nappears to be ignored when \"--untracked-files=all\" is specified, and\nthis appears to be intentional:\n* In wt-status.c#wt_status_collect_untracked(), the \"dir.flags\" are\nupdated to include \"DIR_SHOW_OTHER_DIRECTORIES\" when the\n\"SHOW_ALL_UNTRACKED_FILES\" arg is detected\n* In later logic nested in dir.c#validate_untracked_cache(), the\npresence of the \"DIR_SHOW_OTHER_DIRECTORIES\" flag causes the\nvalidation to fail and, up one level in read_directory(), this causes\nthe untracked structure to be discarded\n\nThe relevant comment in \"validate_untracked_cache()\" says \"See\ntreat_directory(), case index_nonexistent. Without this\n[DIR_SHOW_OTHER_DIRECTORIES] flag, we may need to also cache .git file\ncontent for the resolve_gitlink_ref() call, which we don't.\". I can't\nclaim to understand the comment, the relationship to gitlinks, etc :(\n\nDoes this look like something solvable? It looks like supporting the\nuntrackedCache even with \"--untracked-files=all\" would make a\n(potentially) large difference to git status performance in some\nworkflows with fsmonitor enabled.\n\n(all that said, I still haven't understood why the presence of the\nfsmonitor hook makes the difference, in terms of behavior, between\n*multi-threaded* directory tree scanning for all directory contents\n(without the fsmonitor), and *single-threaded* directory scanning for\nuntracked files specifically (with the fsmonitor))\n\nThanks for looking, any further thoughts will of course be most appreciated!\n\nTao Klerks\n\nOn Wed, Sep 23, 2020 at 4:42 PM Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n>\n> Hi Tao,\n>\n> On Tue, 22 Sep 2020, Tao Klerks wrote:\n>\n> > I've got a couple questions about the \"fsmonitor\" functionality,\n> > untracked files, and multithreading.\n> >\n> > Background:\n> >\n> > In a repo with:\n> >  * A couple hundred thousand tracked files, and a couple hundred\n> > thousand .gitignored files, across a few thousand directories\n> >  * The --untracked-cache setting, tested and working\n> >  * core.fsmonitor set up with watchman (with the sample integration\n> > script from january)\n> >  * Git version 2.27.0.windows.1\n> >\n> > \"git status\" takes about 2s\n> > \"git status --untracked-files=all\" takes about 20s\n> >\n> > When I turn off \"core.fsmonitor\", the numbers change to something like:\n> > \"git status\": 8s\n> > \"git status --untracked-files=all\": 9s\n> >\n> > Using windows' \"procmon\" to observe git.exe's behavior from outside, I\n> > think I've understood a couple things that surprise me:\n> > 1. when you specify \"--untracked-files=all\", git scans the entire\n> > folder tree regardless of the \"fsmonitor\" hook\n> > 2. when you specify the \"fsmonitor\" hook, git does any\n> > filesystem-scanning in a single-threaded fashion (as opposed to\n> > multi-threaded without \"fsmonitor\" / normally)\n> >\n> > These two things combine so that with \"fsmonitor\" set, normal\n> > command-line git status performance is great, but the performance in\n> > tools that eagerly look for untracked files (like \"Git Extensions\" on\n> > windows) actually suffers - it takes twice as long to run the 'git -c\n> > diff.ignoreSubModules=none status --porcelain=2 -z\n> > --untracked-files=all' command that this UI wants (and blocks on, when\n> > you go to a commit dialog).\n> >\n> > Questions:\n> >\n> > 1. Is there a reason \"--untracked-files=all\" causes a full directory\n> > tree scan even with the \"fsmonitor\" hook active, or is this\n> > accidental?\n>\n> I have a hunch that this might be related to a performance hack we have in\n> Git for Windows: did you enable FSCache perchance?\n>\n> If so, I _suspect_ that turning it off would accelerate `git status\n> --untracked-files=all`.\n>\n> Ciao,\n> Johannes\n>\n> > 2. Assuming that the full directory tree scan is indeed necessary even\n> > with \"fsmonitor\" (when requesting all untracked files), could it be\n> > made multithreaded?\n> >\n> > (my apologies for the simplistic \"outside-in\" observations; I don't\n> > feel qualified to attempt to understand the git source code)\n> >\n> > Thanks for any help understanding the optimization opportunities here!\n> >\n> > Tao Klerks\n> >\n"}]}