git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 1/1] unpack-trees: skip stat on fsmonitor-valid files

From
Utsav Shah via GitGitGadget <gitgitgadget@gmail.com>
Date
Nov 20, 2019, 08:32 UTC
Message-ID
<ea7880f2d07ef65f8b41f6786ff789f7a0ff8fd7.1574238737.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.424.v4.git.1574238737.gitgitgadget@gmail.com>
From: Utsav Shah <utsav@dropbox.com>

The index might be aware that a file hasn't modified via fsmonitor, but unpack-trees did not pay attention to it and checked via ie_match_stat which can be inefficient on certain filesystems. This significantly slows down commands that run oneway_merge, like checkout and reset --hard.

This patch makes oneway_merge check whether a file is considered unchanged through fsmonitor and skips ie_match_stat on it. unpack-trees also now correctly copies over fsmonitor validity state from the source index. Finally, for correctness, we force a refresh of fsmonitor state in tweak_fsmonitor.

After this change, commands like stash (that use reset --hard internally) go from 8s or more to ~2s on a 250k file repository on a mac.

Changes since the last version are:
* The sanity checks around accessing the fsmonitor_dirty bitmap have
been moved to another patch, which is in message id [1]
* Unintended indentation changes in fsmonitor have been removed
* A comment explaining what untracked->use_fsmonitor means has been
re-added (it was dropped in the previous version)
* A few "helped-by" entries have been added to the patch
[1]: (xmqqzhh0d0ma.fsf@gitster-ct.c.googlers.com)
Helped-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Kevin Willford <Kevin.Willford@microsoft.com>
Signed-off-by: Utsav Shah <utsav@dropbox.com>
---
 fsmonitor.c                 | 23 +++++++++++++++++------
 t/t7519-status-fsmonitor.sh |  9 +++++++--
 unpack-trees.c              |  6 +++++-
 3 files changed, 29 insertions(+), 9 deletions(-)
diff --git a/fsmonitor.c b/fsmonitor.c
index 1f4aa1b150..0d270da80f 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -189,13 +189,26 @@ void refresh_fsmonitor(struct index_state *istate)
 		}
 		if (bol < query_result.len)
 			fsmonitor_refresh_callback(istate, buf + bol);
+
+		/* Now mark the untracked cache for fsmonitor usage */
+		if (istate->untracked)
+			istate->untracked->use_fsmonitor = 1;
 	} else {
+
+		/* We only want to run the post index changed hook if we've actually changed entries, so keep track
+		 * if we actually changed entries or not */
+		int is_cache_changed = 0;
 		/* Mark all entries invalid */
-		for (i = 0; i < istate->cache_nr; i++)
-			istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
+		for (i = 0; i < istate->cache_nr; i++) {
+			if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {
+				is_cache_changed = 1;
+				istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
+			}
+		}
 
 		/* If we're going to check every file, ensure we save the results */
-		istate->cache_changed |= FSMONITOR_CHANGED;
+		if (is_cache_changed)
+			istate->cache_changed |= FSMONITOR_CHANGED;
 
 		if (istate->untracked)
 			istate->untracked->use_fsmonitor = 0;
@@ -257,9 +270,7 @@ void tweak_fsmonitor(struct index_state *istate)
 				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
 			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
 
-			/* Now mark the untracked cache for fsmonitor usage */
-			if (istate->untracked)
-				istate->untracked->use_fsmonitor = 1;
+			refresh_fsmonitor(istate);
 		}
 
 		ewah_free(istate->fsmonitor_dirty);
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index d8df990972..69908b6a9b 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -106,6 +106,8 @@ EOF
 
 # test that "update-index --fsmonitor-valid" sets the fsmonitor valid bit
 test_expect_success 'update-index --fsmonitor-valid" sets the fsmonitor valid bit' '
+	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	EOF
 	git update-index --fsmonitor &&
 	git update-index --fsmonitor-valid dir1/modified &&
 	git update-index --fsmonitor-valid dir2/modified &&
@@ -164,6 +166,8 @@ EOF
 
 # test that newly added files are marked valid
 test_expect_success 'newly added files are marked valid' '
+	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	EOF
 	git add new &&
 	git add dir1/new &&
 	git add dir2/new &&
@@ -218,11 +222,12 @@ test_expect_success '*only* files returned by the integration script get flagged
 # Ensure commands that call refresh_index() to move the index back in time
 # properly invalidate the fsmonitor cache
 test_expect_success 'refresh_index() invalidates fsmonitor cache' '
-	write_script .git/hooks/fsmonitor-test<<-\EOF &&
-	EOF
 	clean_repo &&
 	dirty_repo &&
+	write_integration_script &&
 	git add . &&
+	write_script .git/hooks/fsmonitor-test<<-\EOF &&
+	EOF
 	git commit -m "to reset" &&
 	git reset HEAD~1 &&
 	git status >actual &&
diff --git a/unpack-trees.c b/unpack-trees.c
index 33ea7810d8..fc5ceb932c 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1504,6 +1504,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	o->merge_size = len;
 	mark_all_ce_unused(o->src_index);
 
+	if (o->src_index->fsmonitor_last_update)
+		o->result.fsmonitor_last_update = o->src_index->fsmonitor_last_update;
+
 	/*
 	 * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries
 	 */
@@ -2384,7 +2387,8 @@ int oneway_merge(const struct cache_entry * const *src,
 
 	if (old && same(old, a)) {
 		int update = 0;
-		if (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old)) {
+		if (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old) &&
+			!(old->ce_flags & CE_FSMONITOR_VALID)) {
 			struct stat st;
 			if (lstat(old->name, &st) ||
 			    ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE))

-- 
gitgitgadget
Previous: Utsav Shah via GitGitGadgetNext: Junio C Hamano
Message 25 of 26 in “unpack-trees: skip lstat on files based on fsmonitor”
  1. 0/1 unpack-trees: skip lstat on files based on fsmonitorUtsav Shah via GitGitGadget, Oct 25, 2019
  2. 1/1 unpack-trees: skip lstat based on fsmonitorUtsav Shah via GitGitGadget, Oct 25, 2019
  3. Junio C HamanoOct 28, 2019
  4. Utsav ShahOct 28, 2019
  5. Kevin WillfordOct 28, 2019
  6. Utsav ShahOct 29, 2019
  7. Kevin WillfordOct 29, 2019
  8. Utsav ShahOct 29, 2019
  9. Junio C HamanoOct 30, 2019
  10. Utsav ShahOct 30, 2019
  11. Junio C HamanoNov 4, 2019
  12. 0/1 unpack-trees: skip stat on fsmonitor-valid filesUtsav Shah via GitGitGadget, Nov 5, 2019
  13. 1/1 unpack-trees: skip stat on fsmonitor-valid filesUtsav Shah via GitGitGadget, Nov 5, 2019
  14. Kevin WillfordNov 5, 2019
  15. Utsav ShahNov 6, 2019
  16. Kevin WillfordNov 6, 2019
  17. 0/1 unpack-trees: skip stat on fsmonitor-valid filesUtsav Shah via GitGitGadget, Nov 6, 2019
  18. 1/1 unpack-trees: skip stat on fsmonitor-valid filesUtsav Shah via GitGitGadget, Nov 6, 2019
  19. Junio C HamanoNov 6, 2019
  20. Utsav ShahNov 6, 2019
  21. Utsav ShahNov 8, 2019
  22. Junio C HamanoNov 8, 2019
  23. Junio C HamanoNov 6, 2019
  24. 0/1 unpack-trees: skip stat on fsmonitor-valid filesUtsav Shah via GitGitGadget, Nov 20, 2019
  25. 1/1 unpack-trees: skip stat on fsmonitor-valid filesUtsav Shah via GitGitGadget, Nov 20, 2019
  26. Junio C HamanoNov 21, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.