{"thread":{"id":"66771","subject":"[PATCH] fsmonitor: invalidate untracked cache after losing history","startedAt":"2026-11-04T15:37:59Z","lastAt":"2026-11-04T22:52:16Z","messageCount":3,"participants":["Derrick Stolee via GitGitGadget","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"557445","messageId":"pull.2295.git.1793806679163.gitgitgadget@gmail.com","threadId":"66771","inReplyTo":null,"subject":"[PATCH] fsmonitor: invalidate untracked cache after losing history","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-11-04T15:37:59Z","receivedAt":"2026-11-04T15:37:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nWhen fsmonitor fails or returns a trivial response, refresh_fsmonitor()\nclears CE_FSMONITOR_VALID and disables fsmonitor use for the untracked\ncache. Unfortunately, this may leave untracked cache entries marked\n'valid' so future commands may miss necessary updates.\n\nA command such as 'git update-index --refresh' can save the replacement\nfsmonitor token without scanning untracked directories. If the next\nquery returns no changes, fsmonitor use is enabled again and 'git status'\ntrusts directory contents cached before the loss of event history. New\nuntracked files can remain invisible.\n\nThis integration bug between fsmonitor and the untracked cache has\nexisted since the fsmonitor feature was introduced in 883e248b8a0\n(fsmonitor: teach git to optionally utilize a file system monitor to\nspeed up detecting new or changed files., 2017-09-22).\n\nTo fix the problem, react more completely on a failed fsmonitor token\nrequest. In this case, invalidate every untracked cache directory and\nmark the untracked extension as changed. To guarantee that the index is\nupdatede, set UNTRACKED_CHANGED so that the index is written even when\nno tracked entry changes. Keep the directory nodes intact to preserve\npointers held by callers.\n\nA new pair of tests validate this approach. They use the fsmonitor hook\nimplementation because it's easier to trigger these failure conditions,\nbut the fix applies to both the hook and builtin versions. The tests\nconfirm that the untracked cache is invalidated.\n\nPerformance tests in p7527-builtin-fsmonitor.sh demonstrate that this\nchange does not introduce any performance degradation, with one caveat.\nMy initial testing on Linux did not have its max_user_watches high\nenough, so the first test run _did_ show a regression for the cases\nwhere the untracked cache was enabled. This added 50-75% additional time\nto commands, but this is because it correctly did the extra work to\nvalidate the untracked cache when the fsmonitor failed to start.\n\nRecall that to extend the maximum user watches, one can run\n\n  sudo sysctl -w fs.inotify.max_user_watches=1048576\n\nWith this increased watch count, my peformance numbers equalized again,\nsince the fsmonitor could start successfully and did not invalidate the\nuntracked cache, as it would with normal operation.\n\nAssisted-by: GPT-6.1 Sol\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n    fsmonitor: invalidate untracked cache after losing history\n    \n    A user of an internal monorepo reported this behavior, including an\n    AI-assisted powershell script to prove how it worked. I used an agent to\n    assist in porting that to a Git test script and a code fix, but then\n    applied significant human editing and validation on top to get it up to\n    my standards.\n    \n    Thanks, -Stolee\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-2295%2Fderrickstolee%2Ffsmonitor-untracked-v2.56-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2295/derrickstolee/fsmonitor-untracked-v2.56-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/2295\n\n dir.c                       | 18 ++++++++++++++++++\n dir.h                       |  1 +\n fsmonitor.c                 |  4 +++-\n t/t7519-status-fsmonitor.sh | 38 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex d896e7be4b..99a2bd9fe2 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -3976,6 +3976,24 @@ static void invalidate_one_directory(struct untracked_cache *uc,\n \tucd->untracked_nr = 0;\n }\n \n+static void invalidate_directory_recursive(struct untracked_cache *uc,\n+\t\t\t\t\t  struct untracked_cache_dir *ucd)\n+{\n+\tinvalidate_one_directory(uc, ucd);\n+\tfor (size_t i = 0; i < ucd->dirs_nr; i++)\n+\t\tinvalidate_directory_recursive(uc, ucd->dirs[i]);\n+}\n+\n+void untracked_cache_invalidate_all(struct index_state *istate)\n+{\n+\tif (!istate->untracked || !istate->untracked->root)\n+\t\treturn;\n+\n+\tinvalidate_directory_recursive(istate->untracked,\n+\t\t\t\t       istate->untracked->root);\n+\tistate->cache_changed |= UNTRACKED_CHANGED;\n+}\n+\n /*\n  * Normally when an entry is added or removed from a directory,\n  * invalidating that directory is enough. No need to touch its\ndiff --git a/dir.h b/dir.h\nindex 83e0f648a8..d2de762da8 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -596,6 +596,7 @@ static inline int dir_path_match(struct index_state *istate,\n int cmp_dir_entry(const void *p1, const void *p2);\n int check_dir_entry_contains(const struct dir_entry *out, const struct dir_entry *in);\n \n+void untracked_cache_invalidate_all(struct index_state *);\n void untracked_cache_invalidate_path(struct index_state *, const char *, int safe_path);\n /*\n  * Invalidate the untracked-cache for this path, but first strip\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 107767527e..9c3079f64a 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -713,8 +713,10 @@ apply_results:\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-\t\tif (istate->untracked)\n+\t\tif (istate->untracked) {\n+\t\t\tuntracked_cache_invalidate_all(istate);\n \t\t\tistate->untracked->use_fsmonitor = 0;\n+\t\t}\n \t}\n \ttrace2_region_leave(\"fsmonitor\", \"apply_results\", istate->repo);\n \ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 93973ed25a..7d3ff4ec6b 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -394,6 +394,44 @@ test_expect_success UNTRACKED_CACHE 'ignore .git changes when invalidating UNTR'\n \ttest_cmp before after\n '\n \n+for response in trivial failure\n+do\n+\ttest_expect_success UNTRACKED_CACHE \\\n+\t\t\"invalidate untracked cache after fsmonitor response: $response\" '\n+\t\ttest_create_repo \"untracked-$response\" &&\n+\t\t(\n+\t\t\tcd \"untracked-$response\" &&\n+\t\t\tgit config core.untrackedCache true &&\n+\t\t\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\t\t\tgit config core.fsmonitorHookVersion 2 &&\n+\t\t\techo empty >.git/fsmonitor-response &&\n+\t\t\ttest_hook --setup fsmonitor-test <<-\\EOF &&\n+\t\t\tprintf \"token\\0\"\n+\t\t\tcase \"$(cat .git/fsmonitor-response)\" in\n+\t\t\ttrivial) printf \"/\\0\" ;;\n+\t\t\tfailure) exit 1 ;;\n+\t\t\tesac\n+\t\t\tEOF\n+\n+\t\t\tmkdir -p inc/deep &&\n+\t\t\tgit update-index --force-write-index &&\n+\t\t\tgit status --porcelain >../actual &&\n+\t\t\ttest_must_be_empty ../actual &&\n+\n+\t\t\t>inc/deep/new &&\n+\t\t\techo \"$response\" >.git/fsmonitor-response &&\n+\t\t\tgit update-index --refresh &&\n+\t\t\techo empty >.git/fsmonitor-response &&\n+\n+\t\t\ttest-tool dump-untracked-cache >../actual &&\n+\t\t\ttest_grep ! \" valid\" ../actual &&\n+\t\t\techo \"?? inc/\" >../expect &&\n+\t\t\tgit status --porcelain >../actual &&\n+\t\t\ttest_cmp ../expect ../actual\n+\t\t)\n+\t'\n+done\n+\n test_expect_success 'discard_index() also discards fsmonitor info' '\n \ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n \ttest_might_fail git update-index --refresh &&\n\nbase-commit: a018953688f1b10bddf91bff8747068f5f4746a4\n-- \ngitgitgadget\n\n"},{"id":"557501","messageId":"xmqq1p901gd7.fsf@gitster.g","threadId":"66771","inReplyTo":"pull.2295.git.1793806679163.gitgitgadget@gmail.com","subject":"Re: [PATCH] fsmonitor: invalidate untracked cache after losing history","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-04T20:39:00Z","receivedAt":"2026-11-04T20:39:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> +static void invalidate_directory_recursive(struct untracked_cache *uc,\n> +\t\t\t\t\t  struct untracked_cache_dir *ucd)\n> +{\n> +\tinvalidate_one_directory(uc, ucd);\n> +\tfor (size_t i = 0; i < ucd->dirs_nr; i++)\n> +\t\tinvalidate_directory_recursive(uc, ucd->dirs[i]);\n> +}\n> +\n> +void untracked_cache_invalidate_all(struct index_state *istate)\n> +{\n> +\tif (!istate->untracked || !istate->untracked->root)\n> +\t\treturn;\n> +\n> +\tinvalidate_directory_recursive(istate->untracked,\n> +\t\t\t\t       istate->untracked->root);\n> +\tistate->cache_changed |= UNTRACKED_CHANGED;\n> +}\n> +\n>  /*\n>   * Normally when an entry is added or removed from a directory,\n>   * invalidating that directory is enough. No need to touch its\n> diff --git a/dir.h b/dir.h\n> index 83e0f648a8..d2de762da8 100644\n> --- a/dir.h\n> +++ b/dir.h\n> @@ -596,6 +596,7 @@ static inline int dir_path_match(struct index_state *istate,\n>  int cmp_dir_entry(const void *p1, const void *p2);\n>  int check_dir_entry_contains(const struct dir_entry *out, const struct dir_entry *in);\n>  \n> +void untracked_cache_invalidate_all(struct index_state *);\n>  void untracked_cache_invalidate_path(struct index_state *, const char *, int safe_path);\n>  /*\n>   * Invalidate the untracked-cache for this path, but first strip\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 107767527e..9c3079f64a 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -713,8 +713,10 @@ apply_results:\n>  \t\tif (is_cache_changed)\n>  \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n>  \n> -\t\tif (istate->untracked)\n> +\t\tif (istate->untracked) {\n> +\t\t\tuntracked_cache_invalidate_all(istate);\n>  \t\t\tistate->untracked->use_fsmonitor = 0;\n> +\t\t}\n>  \t}\n>  \ttrace2_region_leave(\"fsmonitor\", \"apply_results\", istate->repo);\n\nThat is rather a blunt instrument, but being without acceleration is\nmuch better than being fast and incorrect.\n\nWill queue.  Thanks.\n\n"},{"id":"557511","messageId":"xmqqmrroyztr.fsf@gitster.g","threadId":"66771","inReplyTo":"pull.2295.git.1793806679163.gitgitgadget@gmail.com","subject":"Re: [PATCH] fsmonitor: invalidate untracked cache after losing history","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-04T22:52:16Z","receivedAt":"2026-11-04T22:52:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> When fsmonitor fails or returns a trivial response, refresh_fsmonitor()\n> clears CE_FSMONITOR_VALID and disables fsmonitor use for the untracked\n> cache. Unfortunately, this may leave untracked cache entries marked\n> 'valid' so future commands may miss necessary updates.\n>\n> A command such as 'git update-index --refresh' can save the replacement\n> fsmonitor token without scanning untracked directories. If the next\n> query returns no changes, fsmonitor use is enabled again and 'git status'\n> trusts directory contents cached before the loss of event history. New\n> untracked files can remain invisible.\n>\n> This integration bug between fsmonitor and the untracked cache has\n> existed since the fsmonitor feature was introduced in 883e248b8a0\n> (fsmonitor: teach git to optionally utilize a file system monitor to\n> speed up detecting new or changed files., 2017-09-22).\n>\n> To fix the problem, react more completely on a failed fsmonitor token\n> request. In this case, invalidate every untracked cache directory and\n> mark the untracked extension as changed. To guarantee that the index is\n> updatede, set UNTRACKED_CHANGED so that the index is written even when\n> no tracked entry changes. Keep the directory nodes intact to preserve\n> pointers held by callers.\n> ...\n>     fsmonitor: invalidate untracked cache after losing history\n>     \n>     A user of an internal monorepo reported this behavior, including an\n>     AI-assisted powershell script to prove how it worked. I used an agent to\n>     assist in porting that to a Git test script and a code fix, but then\n>     applied significant human editing and validation on top to get it up to\n>     my standards.\n>     \n>     Thanks, -Stolee\n\nWould this be a viable replacement for ik/fsmonitor-untracked-cache-trivial-response\n\n  https://lore.kernel.org/git/pull.2247.git.1791127155614.gitgitgadget@gmail.com/\n\nthat has been left unreviewed for a month?\n\n"}]}