threads / patch / 66460

patchfsmonitor: check the untracked cache after a trivial response

Subject: [PATCH] fsmonitor: check the untracked cache after a trivial response

## tl;dr

Both patches address stale untracked-cache entries after fsmonitor loses event history. Ilia's clears only the stale entries, which avoids a measurable slowdown. Read the story.

replies: 7people: 4as markdown or json

Ilia via GitGitGadget· Oct 4, 2026, 15:19 UTC · lore
From: Ilia K <ki.stfu@gmail.com>

When the monitor sends a trivial response ("/"), for example after the daemon restarted or lost events, refresh_fsmonitor() marks every index entry dirty, which is saved in the index. For the untracked cache it only clears the in-memory flag use_fsmonitor. The valid bits of the cached directories are saved unchanged, although nothing checks them anymore: valid_cached_dir() does not lstat() a valid directory while the monitor is trusted.

A command that does not scan for untracked files, like "git add" or "git checkout", therefore saves the new token next to stale valid bits, and every later "git status" trusts them. A directory removed in the meantime keeps its untracked files listed, or, when its own entry is invalid, makes every run print ``` warning: could not open directory 'sub/': No such file or directory ``` Check every cached directory against its stat data before the cache is saved next to such a token, as valid_cached_dir() does without a monitor, and invalidate the ones that changed or disappeared. Doing this at write time skips commands that do not write the index, and sparse index writes, which drop the fsmonitor extension anyway. The cost is one lstat() per cached directory, once per trivial response.

Signed-off-by: Ilia K <ki.stfu@gmail.com>
---
    fsmonitor: check the untracked cache after a trivial response
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2247%2Fk15tfu%2Funtracked-cache-trivial-response-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2247/k15tfu/untracked-cache-trivial-response-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2247
 dir.c                        | 56 ++++++++++++++++++++++++++++++++++++
 dir.h                        |  7 +++++
 read-cache.c                 | 16 +++++++++++
 t/t7519-status-fsmonitor.sh  | 55 +++++++++++++++++++++++++++++++++++
 t/t7527-builtin-fsmonitor.sh | 44 ++++++++++++++++++++++++++++
 5 files changed, 178 insertions(+)
Show changes to 5 files +178 −0

dir.c, dir.h, read-cache.c, t/t7519-status-fsmonitor.sh, t/t7527-builtin-fsmonitor.sh

diff --git a/dir.c b/dir.c
index d896e7be4b..fc41ca9694 100644
--- a/dir.c
+++ b/dir.c
@@ -4053,6 +4053,62 @@ void untracked_cache_invalidate_trimmed_path(struct index_state *istate,
 	}
 }
 
+static int invalidate_stale_dirs(struct untracked_cache *uc,
+				 struct untracked_cache_dir *ucd,
+				 struct index_state *istate,
+				 struct strbuf *path)
+{
+	struct stat st;
+	size_t len = path->len;
+	int nr_invalidated = 0;
+	unsigned int i;
+
+	if (ucd->valid &&
+	    (lstat(path->buf, &st) ||
+	     match_stat_data_racy(istate, &ucd->stat_data, &st))) {
+		invalidate_one_directory(uc, ucd);
+		nr_invalidated++;
+	}
+
+	for (i = 0; i < ucd->dirs_nr; i++) {
+		/* not written to the index, see write_one_dir() */
+		if (!ucd->dirs[i]->recurse)
+			continue;
+		strbuf_addch(path, '/');
+		strbuf_addstr(path, ucd->dirs[i]->name);
+		nr_invalidated += invalidate_stale_dirs(uc, ucd->dirs[i],
+							istate, path);
+		strbuf_setlen(path, len);
+	}
+
+	return nr_invalidated;
+}
+
+int untracked_cache_invalidate_stale_dirs(struct index_state *istate)
+{
+	struct strbuf path = STRBUF_INIT;
+	const char *worktree;
+	int nr_invalidated;
+
+	if (!istate->untracked || !istate->untracked->root)
+		return 0;
+
+	/*
+	 * The index is also read and written by commands that do not
+	 * run in the top-level directory of the worktree.
+	 */
+	worktree = repo_get_work_tree(istate->repo);
+	if (!worktree)
+		return 0;
+
+	strbuf_addstr(&path, worktree);
+	nr_invalidated = invalidate_stale_dirs(istate->untracked,
+					       istate->untracked->root,
+					       istate, &path);
+	strbuf_release(&path);
+	return nr_invalidated;
+}
+
 void untracked_cache_remove_from_index(struct index_state *istate,
 				       const char *path)
 {
diff --git a/dir.h b/dir.h
index 83e0f648a8..7af1562b44 100644
--- a/dir.h
+++ b/dir.h
@@ -604,6 +604,13 @@ void untracked_cache_invalidate_path(struct index_state *, const char *, int saf
 void untracked_cache_invalidate_trimmed_path(struct index_state *,
 					     const char *path,
 					     int safe_path);
+/*
+ * Invalidate every cached directory that no longer exists or whose
+ * stat data no longer matches the working tree. valid_cached_dir()
+ * skips this check while the file system monitor is trusted.
+ * Returns the number of invalidated directories.
+ */
+int untracked_cache_invalidate_stale_dirs(struct index_state *);
 void untracked_cache_remove_from_index(struct index_state *, const char *);
 void untracked_cache_add_to_index(struct index_state *, const char *);
 
diff --git a/read-cache.c b/read-cache.c
index c4cf08a3a3..00c9960c8e 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3034,6 +3034,22 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
 	    istate->untracked) {
 		strbuf_reset(&sb);
 
+		/*
+		 * The monitor could not say what changed (see the trivial
+		 * response in refresh_fsmonitor()), so nothing kept the
+		 * valid bits up to date. Check them before they are saved
+		 * next to the new token, which later commands trust even
+		 * when this command did not look for untracked files.
+		 */
+		if (write_extensions & WRITE_FSMONITOR_EXTENSION &&
+		    istate->fsmonitor_last_update &&
+		    !istate->untracked->use_fsmonitor) {
+			int nr = untracked_cache_invalidate_stale_dirs(istate);
+
+			trace2_data_intmax("index", istate->repo,
+					   "extension/untr/invalidated", nr);
+		}
+
 		write_untracked_extension(&sb, istate->untracked);
 		err = write_index_ext_header(f, eoie_c, CACHE_EXT_UNTRACKED,
 					     sb.len) < 0;
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index 93973ed25a..2e1d795a4b 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -336,6 +336,61 @@ do
 	done
 done
 
+# After a trivial response ("/") the monitor cannot vouch for the
+# untracked cache. Even a command that does not look for untracked
+# files must drop the stale entries, or the next "git status" trusts them.
+test_expect_success UNTRACKED_CACHE 'untracked cache is checked after a trivial response' '
+	test_when_finished "rm -rf trivial err" &&
+	git init trivial &&
+	(
+		cd trivial &&
+		mkdir -p dir/sub &&
+		echo tracked >dir/sub/tracked &&
+		git add dir &&
+		git commit -m initial &&
+		git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-none" &&
+		# Version 1 only, or the hook prints a version complaint
+		# on stderr at every query.
+		git config core.fsmonitorHookVersion 1 &&
+		git config core.untrackedCache true &&
+		# With "normal", invalidating one path also invalidates
+		# its parents, and the stale parent below is never seen.
+		git config status.showUntrackedFiles all &&
+		echo untracked >dir/sub/untracked &&
+		echo "?? dir/sub/untracked" >../expect &&
+		git status --porcelain >../actual &&
+		test_cmp ../expect ../actual &&
+		git status --porcelain >../actual &&
+		test_cmp ../expect ../actual &&
+
+		# The monitor misses the removal of dir/sub. "git add other"
+		# gets the trivial response and does not touch the entries
+		# of dir and dir/sub by itself.
+		rm -r dir/sub &&
+		echo other >other &&
+		git -c core.fsmonitor="$TEST_DIRECTORY/t7519/fsmonitor-all" \
+			add other &&
+		cat >../expect <<-\EOF &&
+		 D dir/sub/tracked
+		A  other
+		EOF
+		git status --porcelain >../actual 2>../err &&
+		test_must_be_empty ../err &&
+		test_cmp ../expect ../actual &&
+
+		# Invalidate the entry of dir/sub. The stale entry of dir then
+		# makes "git status" open the removed directory and warn.
+		git update-index --remove dir/sub/tracked &&
+		cat >../expect <<-\EOF &&
+		D  dir/sub/tracked
+		A  other
+		EOF
+		git status --porcelain >../actual 2>../err &&
+		test_must_be_empty ../err &&
+		test_cmp ../expect ../actual
+	)
+'
+
 # test that splitting the index doesn't interfere
 test_expect_success 'splitting the index results in the same state' '
 	write_integration_script &&
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 86195770e9..46b97ef784 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -1389,4 +1389,48 @@ test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '
 	test_grep -q " M dir1/dir2/dir4/FILE-4-A" "$PWD/file_case_wrong-try3.out"
 '
 
+# After a restart the daemon sends a trivial response ("/"), because it
+# cannot know what changed while it was down. Even a command that does
+# not look for untracked files must then drop the stale untracked cache
+# entries, or the next "git status" trusts them.
+test_expect_success UNTRACKED_CACHE 'untracked cache is checked after a trivial response' '
+	test_when_finished "stop_daemon_delete_repo test_trivial" &&
+
+	git init test_trivial &&
+	mkdir -p test_trivial/dir/sub &&
+	echo tracked >test_trivial/dir/sub/tracked &&
+	git -C test_trivial add dir &&
+	git -C test_trivial commit -m initial &&
+	git -C test_trivial config core.fsmonitor true &&
+	git -C test_trivial config core.untrackedCache true &&
+	echo untracked >test_trivial/dir/sub/untracked &&
+
+	# The first status starts the daemon and builds the untracked
+	# cache, the second one trusts it.
+	echo "?? dir/sub/untracked" >expect &&
+	git -C test_trivial status --porcelain >actual &&
+	test_cmp expect actual &&
+	git -C test_trivial status --porcelain >actual &&
+	test_cmp expect actual &&
+
+	# Remove dir/sub while no daemon is running. "git add" then
+	# starts a new daemon, receives its trivial response, and does
+	# not look for untracked files.
+	git -C test_trivial fsmonitor--daemon stop &&
+	rm -r test_trivial/dir/sub &&
+	echo other >test_trivial/other &&
+	GIT_TRACE2_EVENT="$PWD/trace_trivial" \
+		git -C test_trivial add other &&
+	have_t2_data_event fsm_client query/trivial-response <trace_trivial &&
+	git -C test_trivial fsmonitor--daemon status &&
+
+	cat >expect <<-\EOF &&
+	 D dir/sub/tracked
+	A  other
+	EOF
+	git -C test_trivial status --porcelain >actual 2>err &&
+	test_must_be_empty err &&
+	test_cmp expect actual
+'
+
 test_done

base-commit: c46c1e37724f0478939de636ab8ea5a89086d532
-- 
gitgitgadget
Derrick Stolee· Nov 5, 2026, 15:01 UTC · re: Ilia via GitGitGadget · lore

Re: [PATCH] fsmonitor: check the untracked cache after a trivial response

On 10/4/2026 11:19 AM, Ilia via GitGitGadget wrote:
Show 9 quoted lines
> From: Ilia K <ki.stfu@gmail.com>
> 
> When the monitor sends a trivial response ("/"), for example after
> the daemon restarted or lost events, refresh_fsmonitor() marks every
> index entry dirty, which is saved in the index. For the untracked
> cache it only clears the in-memory flag use_fsmonitor. The valid bits
> of the cached directories are saved unchanged, although nothing
> checks them anymore: valid_cached_dir() does not lstat() a valid
> directory while the monitor is trusted.

Sorry that I declared "mailing list bankruptcy" and didn't see your patch when I was preparing mine [1] for the same issue.

[1] https://lore.kernel.org/git/pull.2295.git.1793806679163.gitgitgadget@gmail.com/
...
Show 7 quoted lines
> Check every cached directory against its stat data before the cache
> is saved next to such a token, as valid_cached_dir() does without a
> monitor, and invalidate the ones that changed or disappeared. Doing
> this at write time skips commands that do not write the index, and
> sparse index writes, which drop the fsmonitor extension anyway.
> The cost is one lstat() per cached directory, once per trivial
> response.

This is the main difference between your patch and mine. My patch does a full clear of the untracked cache, but you do the more efficient thing to only clear the entries that don't match the filesystem info.

This is visible when running p7527-bulitin-fsmonitor.sh on Linux with a small setting in fs.inotify.max_user_watches, causing the builtin fsmonitor to always fail. My "clear everything" patch gets a 50-75% penalty on the untracked cache tests while yours has no measurable impact.

For this reason, I think your approach is better and worth the slight increase in code complexity.

Show 25 quoted lines
> +static int invalidate_stale_dirs(struct untracked_cache *uc,
> +				 struct untracked_cache_dir *ucd,
> +				 struct index_state *istate,
> +				 struct strbuf *path)
> +{
> +	struct stat st;
> +	size_t len = path->len;
> +	int nr_invalidated = 0;
> +	unsigned int i;
> +
> +	if (ucd->valid &&
> +	    (lstat(path->buf, &st) ||
> +	     match_stat_data_racy(istate, &ucd->stat_data, &st))) {
> +		invalidate_one_directory(uc, ucd);
> +		nr_invalidated++;
> +	}
> +
> +	for (i = 0; i < ucd->dirs_nr; i++) {
> +		/* not written to the index, see write_one_dir() */
> +		if (!ucd->dirs[i]->recurse)
> +			continue;
> +		strbuf_addch(path, '/');
> +		strbuf_addstr(path, ucd->dirs[i]->name);
> +		nr_invalidated += invalidate_stale_dirs(uc, ucd->dirs[i],
> +							istate, path);

I'm conditioned to worry about recursive calls without depth limits, but I suppose that we are limited by the untracked cache data structure itself, which was presumably created recursively without hitting stack limits.

We could make this iterative using a stack-based data structure, but that would be more complicated than this is worth, probably. These are not used in server environments and the .git/index file shouldn't be considered "unsafe" or else we have bigger safety problems.

Show 22 quoted lines
> +		strbuf_setlen(path, len);
> +	}
> +
> +	return nr_invalidated;
> +}
> +
> +int untracked_cache_invalidate_stale_dirs(struct index_state *istate)
> +{
> +	struct strbuf path = STRBUF_INIT;
> +	const char *worktree;
> +	int nr_invalidated;
> +
> +	if (!istate->untracked || !istate->untracked->root)
> +		return 0;
> +
> +	/*
> +	 * The index is also read and written by commands that do not
> +	 * run in the top-level directory of the worktree.
> +	 */
> +	worktree = repo_get_work_tree(istate->repo);
> +	if (!worktree)
> +		return 0;

I wonder if this is the right response if we fail to get a worktree. Maybe we should invalidate the entire untracked cache in this case, along with a warning that we couldn't evaluate the worktree? Or should we return -1 to signal an error?

Show 8 quoted lines
> +
> +	strbuf_addstr(&path, worktree);
> +	nr_invalidated = invalidate_stale_dirs(istate->untracked,
> +					       istate->untracked->root,
> +					       istate, &path);
> +	strbuf_release(&path);
> +	return nr_invalidated;
> +}
...
Show 7 quoted lines
> +/*
> + * Invalidate every cached directory that no longer exists or whose
> + * stat data no longer matches the working tree. valid_cached_dir()
> + * skips this check while the file system monitor is trusted.
> + * Returns the number of invalidated directories.
> + */
> +int untracked_cache_invalidate_stale_dirs(struct index_state *);
We could update this doc comment if we return negative on error.
Show 23 quoted lines
> diff --git a/read-cache.c b/read-cache.c
> index c4cf08a3a3..00c9960c8e 100644
> --- a/read-cache.c
> +++ b/read-cache.c
> @@ -3034,6 +3034,22 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
>  	    istate->untracked) {
>  		strbuf_reset(&sb);
>  
> +		/*
> +		 * The monitor could not say what changed (see the trivial
> +		 * response in refresh_fsmonitor()), so nothing kept the
> +		 * valid bits up to date. Check them before they are saved
> +		 * next to the new token, which later commands trust even
> +		 * when this command did not look for untracked files.
> +		 */
> +		if (write_extensions & WRITE_FSMONITOR_EXTENSION &&
> +		    istate->fsmonitor_last_update &&
> +		    !istate->untracked->use_fsmonitor) {
> +			int nr = untracked_cache_invalidate_stale_dirs(istate);
> +
> +			trace2_data_intmax("index", istate->repo,
> +					   "extension/untr/invalidated", nr);
> +		}

This surprised me, that you're computing this in do_write_index(). I suppose that this avoids doing this computation unless the index is written, but is it possible that between refresh_fsmonitor() and do_write_index() that the untracked cache is read and used to determine what this index update should be?

I used a coding agent to explore my concerns in this space, and it couldn't find an example where doing this in refresh_fsmonitor() would be better than its current placement.

> +# After a trivial response ("/") the monitor cannot vouch for the
> +# untracked cache. Even a command that does not look for untracked
> +# files must drop the stale entries, or the next "git status" trusts them.

nit: Generally, we don't need these comments because the tests should be self-documenting.

Show 34 quoted lines
> +test_expect_success UNTRACKED_CACHE 'untracked cache is checked after a trivial response' '
> +	test_when_finished "rm -rf trivial err" &&
> +	git init trivial &&
> +	(
> +		cd trivial &&
> +		mkdir -p dir/sub &&
> +		echo tracked >dir/sub/tracked &&
> +		git add dir &&
> +		git commit -m initial &&
> +		git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-none" &&
> +		# Version 1 only, or the hook prints a version complaint
> +		# on stderr at every query.
> +		git config core.fsmonitorHookVersion 1 &&
> +		git config core.untrackedCache true &&
> +		# With "normal", invalidating one path also invalidates
> +		# its parents, and the stale parent below is never seen.
> +		git config status.showUntrackedFiles all &&
> +		echo untracked >dir/sub/untracked &&
> +		echo "?? dir/sub/untracked" >../expect &&
> +		git status --porcelain >../actual &&
> +		test_cmp ../expect ../actual &&
> +		git status --porcelain >../actual &&
> +		test_cmp ../expect ../actual &&
> +
> +		# The monitor misses the removal of dir/sub. "git add other"
> +		# gets the trivial response and does not touch the entries
> +		# of dir and dir/sub by itself.
> +		rm -r dir/sub &&
> +		echo other >other &&
> +		git -c core.fsmonitor="$TEST_DIRECTORY/t7519/fsmonitor-all" \
> +			add other &&
> +		cat >../expect <<-\EOF &&
> +		 D dir/sub/tracked
> +		A  other

I initially thought that this difference in left-padding was a typo, but it is functional. We need a space before D and a space after A to signal the different states.

Show 11 quoted lines
> +		EOF
> +		git status --porcelain >../actual 2>../err &&
> +		test_must_be_empty ../err &&
> +		test_cmp ../expect ../actual &&
> +
> +		# Invalidate the entry of dir/sub. The stale entry of dir then
> +		# makes "git status" open the removed directory and warn.
> +		git update-index --remove dir/sub/tracked &&
> +		cat >../expect <<-\EOF &&
> +		D  dir/sub/tracked
> +		A  other
And of course now the deletion is staged. Good.
Show 6 quoted lines
> +		EOF
> +		git status --porcelain >../actual 2>../err &&
> +		test_must_be_empty ../err &&
> +		test_cmp ../expect ../actual
> +	)
> +'
> +# After a restart the daemon sends a trivial response ("/"), because it
> +# cannot know what changed while it was down. Even a command that does
> +# not look for untracked files must then drop the stale untracked cache
> +# entries, or the next "git status" trusts them.
same nit: comments shouldn't be necessary here.
Show 39 quoted lines
> +test_expect_success UNTRACKED_CACHE 'untracked cache is checked after a trivial response' '
> +	test_when_finished "stop_daemon_delete_repo test_trivial" &&
> +
> +	git init test_trivial &&
> +	mkdir -p test_trivial/dir/sub &&
> +	echo tracked >test_trivial/dir/sub/tracked &&
> +	git -C test_trivial add dir &&
> +	git -C test_trivial commit -m initial &&
> +	git -C test_trivial config core.fsmonitor true &&
> +	git -C test_trivial config core.untrackedCache true &&
> +	echo untracked >test_trivial/dir/sub/untracked &&
> +
> +	# The first status starts the daemon and builds the untracked
> +	# cache, the second one trusts it.
> +	echo "?? dir/sub/untracked" >expect &&
> +	git -C test_trivial status --porcelain >actual &&
> +	test_cmp expect actual &&
> +	git -C test_trivial status --porcelain >actual &&
> +	test_cmp expect actual &&
> +
> +	# Remove dir/sub while no daemon is running. "git add" then
> +	# starts a new daemon, receives its trivial response, and does
> +	# not look for untracked files.
> +	git -C test_trivial fsmonitor--daemon stop &&
> +	rm -r test_trivial/dir/sub &&
> +	echo other >test_trivial/other &&
> +	GIT_TRACE2_EVENT="$PWD/trace_trivial" \
> +		git -C test_trivial add other &&
> +	have_t2_data_event fsm_client query/trivial-response <trace_trivial &&
> +	git -C test_trivial fsmonitor--daemon status &&
> +
> +	cat >expect <<-\EOF &&
> +	 D dir/sub/tracked
> +	A  other
> +	EOF
> +	git -C test_trivial status --porcelain >actual 2>err &&
> +	test_must_be_empty err &&
> +	test_cmp expect actual
> +'

Overall, this patch appears to solve the same problem that pointed me in this direction. I don't see any issues worth re-rolling this patch, so it LGTM.

Thanks, -Stolee

Junio C Hamano· Nov 5, 2026, 18:24 UTC · re: Derrick Stolee · lore

Re: [PATCH] fsmonitor: check the untracked cache after a trivial response

Derrick Stolee <stolee@gmail.com> writes:
Show 17 quoted lines
>> +	for (i = 0; i < ucd->dirs_nr; i++) {
>> +		/* not written to the index, see write_one_dir() */
>> +		if (!ucd->dirs[i]->recurse)
>> +			continue;
>> +		strbuf_addch(path, '/');
>> +		strbuf_addstr(path, ucd->dirs[i]->name);
>> +		nr_invalidated += invalidate_stale_dirs(uc, ucd->dirs[i],
>> +							istate, path);
>
> I'm conditioned to worry about recursive calls without depth limits, but
> I suppose that we are limited by the untracked cache data structure itself,
> which was presumably created recursively without hitting stack limits.
>
> We could make this iterative using a stack-based data structure, but that
> would be more complicated than this is worth, probably. These are not used
> in server environments and the .git/index file shouldn't be considered
> "unsafe" or else we have bigger safety problems.
Good reasoning.  Thanks for spelling it out.
Show 12 quoted lines
>> +	/*
>> +	 * The index is also read and written by commands that do not
>> +	 * run in the top-level directory of the worktree.
>> +	 */
>> +	worktree = repo_get_work_tree(istate->repo);
>> +	if (!worktree)
>> +		return 0;
>
> I wonder if this is the right response if we fail to get a worktree. Maybe
> we should invalidate the entire untracked cache in this case, along with a
> warning that we couldn't evaluate the worktree? Or should we return -1 to
> signal an error?

It probably is safer to invalidate and continue. If there is truly not a worktree, untracked cache entries do not matter ;-)

Thanks for a review.
Ilia K· Nov 9, 2026, 18:59 UTC · re: Junio C Hamano · lore

Re: [PATCH] fsmonitor: check the untracked cache after a trivial response

On Thu, Nov 5, 2026 at 4:01 PM Derrick Stolee <stolee@gmail.com> wrote:
Show 28 quoted lines
>
> > +             strbuf_setlen(path, len);
> > +     }
> > +
> > +     return nr_invalidated;
> > +}
> > +
> > +int untracked_cache_invalidate_stale_dirs(struct index_state *istate)
> > +{
> > +     struct strbuf path = STRBUF_INIT;
> > +     const char *worktree;
> > +     int nr_invalidated;
> > +
> > +     if (!istate->untracked || !istate->untracked->root)
> > +             return 0;
> > +
> > +     /*
> > +      * The index is also read and written by commands that do not
> > +      * run in the top-level directory of the worktree.
> > +      */
> > +     worktree = repo_get_work_tree(istate->repo);
> > +     if (!worktree)
> > +             return 0;
>
> I wonder if this is the right response if we fail to get a worktree. Maybe
> we should invalidate the entire untracked cache in this case, along with a
> warning that we couldn't evaluate the worktree? Or should we return -1 to
> signal an error?
Fixed, thank you!
Show 29 quoted lines
> > diff --git a/read-cache.c b/read-cache.c
> > index c4cf08a3a3..00c9960c8e 100644
> > --- a/read-cache.c
> > +++ b/read-cache.c
> > @@ -3034,6 +3034,22 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
> >           istate->untracked) {
> >               strbuf_reset(&sb);
> >
> > +             /*
> > +              * The monitor could not say what changed (see the trivial
> > +              * response in refresh_fsmonitor()), so nothing kept the
> > +              * valid bits up to date. Check them before they are saved
> > +              * next to the new token, which later commands trust even
> > +              * when this command did not look for untracked files.
> > +              */
> > +             if (write_extensions & WRITE_FSMONITOR_EXTENSION &&
> > +                 istate->fsmonitor_last_update &&
> > +                 !istate->untracked->use_fsmonitor) {
> > +                     int nr = untracked_cache_invalidate_stale_dirs(istate);
> > +
> > +                     trace2_data_intmax("index", istate->repo,
> > +                                        "extension/untr/invalidated", nr);
> > +             }
>
> This surprised me, that you're computing this in do_write_index(). I suppose
> that this avoids doing this computation unless the index is written, but is
> it possible that between refresh_fsmonitor() and do_write_index() that the
> untracked cache is read and used to determine what this index update should
> be?

No, nothing between `refresh_fsmonitor()` and `do_write_index()` trusts the cache because use_fsmonitor stays 0 until the index is written.

Show 6 quoted lines
> > +# After a trivial response ("/") the monitor cannot vouch for the
> > +# untracked cache. Even a command that does not look for untracked
> > +# files must drop the stale entries, or the next "git status" trusts them.
>
> nit: Generally, we don't need these comments because the tests should be
> self-documenting.
Fixed, thank you!
Show 13 quoted lines
> > +             EOF
> > +             git status --porcelain >../actual 2>../err &&
> > +             test_must_be_empty ../err &&
> > +             test_cmp ../expect ../actual
> > +     )
> > +'
>
> > +# After a restart the daemon sends a trivial response ("/"), because it
> > +# cannot know what changed while it was down. Even a command that does
> > +# not look for untracked files must then drop the stale untracked cache
> > +# entries, or the next "git status" trusts them.
>
> same nit: comments shouldn't be necessary here.
Fixed, thank you!
--
- Ilia
Ilia via GitGitGadget· Nov 9, 2026, 19:14 UTC · re: Ilia via GitGitGadget · lore

[PATCH v2] fsmonitor: check the untracked cache after a trivial response

From: Ilia K <ki.stfu@gmail.com>

When the monitor sends a trivial response ("/"), for example after the daemon restarted or lost events, refresh_fsmonitor() marks every index entry dirty, which is saved in the index. For the untracked cache it only clears the in-memory flag use_fsmonitor. The valid bits of the cached directories are saved unchanged, although nothing checks them anymore: valid_cached_dir() does not lstat() a valid directory while the monitor is trusted.

A command that does not scan for untracked files, like "git add" or "git checkout", therefore saves the new token next to stale valid bits, and every later "git status" trusts them. A directory removed in the meantime keeps its untracked files listed, or, when its own entry is invalid, makes every run print ``` warning: could not open directory 'sub/': No such file or directory ``` Check every cached directory against its stat data before the cache is saved next to such a token, as valid_cached_dir() does without a monitor, and invalidate the ones that changed or disappeared. Within the command itself, valid_cached_dir() already checks every directory, because use_fsmonitor is cleared. Only what is saved needs the check. Doing this at write time skips commands that do not write the index, and sparse index writes, which drop the fsmonitor extension anyway. The cost is one lstat() per cached directory, once per trivial response.

Signed-off-by: Ilia K <ki.stfu@gmail.com>
---
    fsmonitor: check the untracked cache after a trivial response
    
    Changes since v1:
    
     * the whole untracked cache is dropped if !worktree
     * removed the comments as per nits
     * rebased against ToT
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2247%2Fk15tfu%2Funtracked-cache-trivial-response-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2247/k15tfu/untracked-cache-trivial-response-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2247
Range-diff vs v1:
 1:  84618b83fe ! 1:  d2cad8abe6 fsmonitor: check the untracked cache after a trivial response
     @@ Commit message
          ```
          Check every cached directory against its stat data before the cache
          is saved next to such a token, as valid_cached_dir() does without a
     -    monitor, and invalidate the ones that changed or disappeared. Doing
     -    this at write time skips commands that do not write the index, and
     -    sparse index writes, which drop the fsmonitor extension anyway.
     +    monitor, and invalidate the ones that changed or disappeared. Within
     +    the command itself, valid_cached_dir() already checks every directory,
     +    because use_fsmonitor is cleared. Only what is saved needs the check.
     +    Doing this at write time skips commands that do not write the index,
     +    and sparse index writes, which drop the fsmonitor extension anyway.
          The cost is one lstat() per cached directory, once per trivial
          response.
      
     @@ dir.c: void untracked_cache_invalidate_trimmed_path(struct index_state *istate,
      +	 * run in the top-level directory of the worktree.
      +	 */
      +	worktree = repo_get_work_tree(istate->repo);
     -+	if (!worktree)
     -+		return 0;
     ++	if (!worktree) {
     ++		invalidate_directory(istate->untracked,
     ++				     istate->untracked->root);
     ++		return 1;
     ++	}
      +
      +	strbuf_addstr(&path, worktree);
      +	nr_invalidated = invalidate_stale_dirs(istate->untracked,
     @@ t/t7519-status-fsmonitor.sh: do
       	done
       done
       
     -+# After a trivial response ("/") the monitor cannot vouch for the
     -+# untracked cache. Even a command that does not look for untracked
     -+# files must drop the stale entries, or the next "git status" trusts them.
      +test_expect_success UNTRACKED_CACHE 'untracked cache is checked after a trivial response' '
      +	test_when_finished "rm -rf trivial err" &&
      +	git init trivial &&
     @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'fsmonitor
       	test_grep -q " M dir1/dir2/dir4/FILE-4-A" "$PWD/file_case_wrong-try3.out"
       '
       
     -+# After a restart the daemon sends a trivial response ("/"), because it
     -+# cannot know what changed while it was down. Even a command that does
     -+# not look for untracked files must then drop the stale untracked cache
     -+# entries, or the next "git status" trusts them.
      +test_expect_success UNTRACKED_CACHE 'untracked cache is checked after a trivial response' '
      +	test_when_finished "stop_daemon_delete_repo test_trivial" &&
      +
 dir.c                        | 59 ++++++++++++++++++++++++++++++++++++
 dir.h                        |  7 +++++
 read-cache.c                 | 16 ++++++++++
 t/t7519-status-fsmonitor.sh  | 52 +++++++++++++++++++++++++++++++
 t/t7527-builtin-fsmonitor.sh | 40 ++++++++++++++++++++++++
 5 files changed, 174 insertions(+)
Show changes to 5 files +174 −0

dir.c, dir.h, read-cache.c, t/t7519-status-fsmonitor.sh, t/t7527-builtin-fsmonitor.sh

diff --git a/dir.c b/dir.c
index ac246ced1d..4f748800b1 100644
--- a/dir.c
+++ b/dir.c
@@ -4066,6 +4066,65 @@ void untracked_cache_invalidate_trimmed_path(struct index_state *istate,
 	}
 }
 
+static int invalidate_stale_dirs(struct untracked_cache *uc,
+				 struct untracked_cache_dir *ucd,
+				 struct index_state *istate,
+				 struct strbuf *path)
+{
+	struct stat st;
+	size_t len = path->len;
+	int nr_invalidated = 0;
+	unsigned int i;
+
+	if (ucd->valid &&
+	    (lstat(path->buf, &st) ||
+	     match_stat_data_racy(istate, &ucd->stat_data, &st))) {
+		invalidate_one_directory(uc, ucd);
+		nr_invalidated++;
+	}
+
+	for (i = 0; i < ucd->dirs_nr; i++) {
+		/* not written to the index, see write_one_dir() */
+		if (!ucd->dirs[i]->recurse)
+			continue;
+		strbuf_addch(path, '/');
+		strbuf_addstr(path, ucd->dirs[i]->name);
+		nr_invalidated += invalidate_stale_dirs(uc, ucd->dirs[i],
+							istate, path);
+		strbuf_setlen(path, len);
+	}
+
+	return nr_invalidated;
+}
+
+int untracked_cache_invalidate_stale_dirs(struct index_state *istate)
+{
+	struct strbuf path = STRBUF_INIT;
+	const char *worktree;
+	int nr_invalidated;
+
+	if (!istate->untracked || !istate->untracked->root)
+		return 0;
+
+	/*
+	 * The index is also read and written by commands that do not
+	 * run in the top-level directory of the worktree.
+	 */
+	worktree = repo_get_work_tree(istate->repo);
+	if (!worktree) {
+		invalidate_directory(istate->untracked,
+				     istate->untracked->root);
+		return 1;
+	}
+
+	strbuf_addstr(&path, worktree);
+	nr_invalidated = invalidate_stale_dirs(istate->untracked,
+					       istate->untracked->root,
+					       istate, &path);
+	strbuf_release(&path);
+	return nr_invalidated;
+}
+
 void untracked_cache_remove_from_index(struct index_state *istate,
 				       const char *path)
 {
diff --git a/dir.h b/dir.h
index 83e0f648a8..7af1562b44 100644
--- a/dir.h
+++ b/dir.h
@@ -604,6 +604,13 @@ void untracked_cache_invalidate_path(struct index_state *, const char *, int saf
 void untracked_cache_invalidate_trimmed_path(struct index_state *,
 					     const char *path,
 					     int safe_path);
+/*
+ * Invalidate every cached directory that no longer exists or whose
+ * stat data no longer matches the working tree. valid_cached_dir()
+ * skips this check while the file system monitor is trusted.
+ * Returns the number of invalidated directories.
+ */
+int untracked_cache_invalidate_stale_dirs(struct index_state *);
 void untracked_cache_remove_from_index(struct index_state *, const char *);
 void untracked_cache_add_to_index(struct index_state *, const char *);
 
diff --git a/read-cache.c b/read-cache.c
index 1d2b97c429..5d95ba7680 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3084,6 +3084,22 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
 	    istate->untracked) {
 		strbuf_reset(&sb);
 
+		/*
+		 * The monitor could not say what changed (see the trivial
+		 * response in refresh_fsmonitor()), so nothing kept the
+		 * valid bits up to date. Check them before they are saved
+		 * next to the new token, which later commands trust even
+		 * when this command did not look for untracked files.
+		 */
+		if (write_extensions & WRITE_FSMONITOR_EXTENSION &&
+		    istate->fsmonitor_last_update &&
+		    !istate->untracked->use_fsmonitor) {
+			int nr = untracked_cache_invalidate_stale_dirs(istate);
+
+			trace2_data_intmax("index", istate->repo,
+					   "extension/untr/invalidated", nr);
+		}
+
 		write_untracked_extension(&sb, istate->untracked);
 		err = write_index_ext_header(f, eoie_c, CACHE_EXT_UNTRACKED,
 					     sb.len) < 0;
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index 93973ed25a..27c3194ae0 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -336,6 +336,58 @@ do
 	done
 done
 
+test_expect_success UNTRACKED_CACHE 'untracked cache is checked after a trivial response' '
+	test_when_finished "rm -rf trivial err" &&
+	git init trivial &&
+	(
+		cd trivial &&
+		mkdir -p dir/sub &&
+		echo tracked >dir/sub/tracked &&
+		git add dir &&
+		git commit -m initial &&
+		git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-none" &&
+		# Version 1 only, or the hook prints a version complaint
+		# on stderr at every query.
+		git config core.fsmonitorHookVersion 1 &&
+		git config core.untrackedCache true &&
+		# With "normal", invalidating one path also invalidates
+		# its parents, and the stale parent below is never seen.
+		git config status.showUntrackedFiles all &&
+		echo untracked >dir/sub/untracked &&
+		echo "?? dir/sub/untracked" >../expect &&
+		git status --porcelain >../actual &&
+		test_cmp ../expect ../actual &&
+		git status --porcelain >../actual &&
+		test_cmp ../expect ../actual &&
+
+		# The monitor misses the removal of dir/sub. "git add other"
+		# gets the trivial response and does not touch the entries
+		# of dir and dir/sub by itself.
+		rm -r dir/sub &&
+		echo other >other &&
+		git -c core.fsmonitor="$TEST_DIRECTORY/t7519/fsmonitor-all" \
+			add other &&
+		cat >../expect <<-\EOF &&
+		 D dir/sub/tracked
+		A  other
+		EOF
+		git status --porcelain >../actual 2>../err &&
+		test_must_be_empty ../err &&
+		test_cmp ../expect ../actual &&
+
+		# Invalidate the entry of dir/sub. The stale entry of dir then
+		# makes "git status" open the removed directory and warn.
+		git update-index --remove dir/sub/tracked &&
+		cat >../expect <<-\EOF &&
+		D  dir/sub/tracked
+		A  other
+		EOF
+		git status --porcelain >../actual 2>../err &&
+		test_must_be_empty ../err &&
+		test_cmp ../expect ../actual
+	)
+'
+
 # test that splitting the index doesn't interfere
 test_expect_success 'splitting the index results in the same state' '
 	write_integration_script &&
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 86195770e9..f2770e9056 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -1389,4 +1389,44 @@ test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '
 	test_grep -q " M dir1/dir2/dir4/FILE-4-A" "$PWD/file_case_wrong-try3.out"
 '
 
+test_expect_success UNTRACKED_CACHE 'untracked cache is checked after a trivial response' '
+	test_when_finished "stop_daemon_delete_repo test_trivial" &&
+
+	git init test_trivial &&
+	mkdir -p test_trivial/dir/sub &&
+	echo tracked >test_trivial/dir/sub/tracked &&
+	git -C test_trivial add dir &&
+	git -C test_trivial commit -m initial &&
+	git -C test_trivial config core.fsmonitor true &&
+	git -C test_trivial config core.untrackedCache true &&
+	echo untracked >test_trivial/dir/sub/untracked &&
+
+	# The first status starts the daemon and builds the untracked
+	# cache, the second one trusts it.
+	echo "?? dir/sub/untracked" >expect &&
+	git -C test_trivial status --porcelain >actual &&
+	test_cmp expect actual &&
+	git -C test_trivial status --porcelain >actual &&
+	test_cmp expect actual &&
+
+	# Remove dir/sub while no daemon is running. "git add" then
+	# starts a new daemon, receives its trivial response, and does
+	# not look for untracked files.
+	git -C test_trivial fsmonitor--daemon stop &&
+	rm -r test_trivial/dir/sub &&
+	echo other >test_trivial/other &&
+	GIT_TRACE2_EVENT="$PWD/trace_trivial" \
+		git -C test_trivial add other &&
+	have_t2_data_event fsm_client query/trivial-response <trace_trivial &&
+	git -C test_trivial fsmonitor--daemon status &&
+
+	cat >expect <<-\EOF &&
+	 D dir/sub/tracked
+	A  other
+	EOF
+	git -C test_trivial status --porcelain >actual 2>err &&
+	test_must_be_empty err &&
+	test_cmp expect actual
+'
+
 test_done

base-commit: fb255012acbcbe81189cb8014da0280b09c66ecc
-- 
gitgitgadget
Derrick Stolee· Nov 9, 2026, 20:47 UTC · re: Ilia via GitGitGadget · lore

Re: [PATCH v2] fsmonitor: check the untracked cache after a trivial response

On 11/9/2026 2:14 PM, Ilia via GitGitGadget wrote:
>     Changes since v1:
>     
>      * the whole untracked cache is dropped if !worktree
>      * removed the comments as per nits
Thanks for these improvements. The v2 patch LGTM.
>      * rebased against ToT

Generally, you shouldn't do this as the maintainer will apply your patch in the same spot and handle most merge conflicts via rerere, but rebasing interrupts that. The main exception is when you rebase onto a specific topic or with regards to a large refactor that merged recently.

Thanks, -Stolee

Junio C Hamano· Nov 9, 2026, 20:58 UTC · re: Derrick Stolee · lore

Re: [PATCH v2] fsmonitor: check the untracked cache after a trivial response

Derrick Stolee <stolee@gmail.com> writes:
Show 15 quoted lines
> On 11/9/2026 2:14 PM, Ilia via GitGitGadget wrote:
>>     Changes since v1:
>>     
>>      * the whole untracked cache is dropped if !worktree
>>      * removed the comments as per nits
>
> Thanks for these improvements. The v2 patch LGTM.
>
>>      * rebased against ToT
>
> Generally, you shouldn't do this as the maintainer will apply
> your patch in the same spot and handle most merge conflicts
> via rerere, but rebasing interrupts that. The main exception
> is when you rebase onto a specific topic or with regards to a
> large refactor that merged recently.

Good suggestion. Worse yet, the topic is already in 'next' so a wholesale replacement is not very welcome.

Junio C Hamano· Nov 9, 2026, 21:34 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] fsmonitor: check the untracked cache after a trivial response

Junio C Hamano <gitster@pobox.com> writes:
Show 12 quoted lines
>> Thanks for these improvements. The v2 patch LGTM.
>>
>>>      * rebased against ToT
>>
>> Generally, you shouldn't do this as the maintainer will apply
>> your patch in the same spot and handle most merge conflicts
>> via rerere, but rebasing interrupts that. The main exception
>> is when you rebase onto a specific topic or with regards to a
>> large refactor that merged recently.
>
> Good suggestion.  Worse yet, the topic is already in 'next' so a
> wholesale replacement is not very welcome.

Well, I ended up reverting the merge to 'next', and queued this iteration (applied on exactly the same base as v1 was applied in my tree---the patch cleanly applied as if there were no rebase). I'll mark the topic again ready for 'next' after waiting for a few days to see if there are any more feedback.

Thanks, all.

← back to recent threads