git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v8 00/12] Fast git status via a file system watcher

From
Ben Peart <benpeart@microsoft.com>
Date
Sep 22, 2017, 16:35 UTC
Message-ID
<20170922163548.11288-1-benpeart@microsoft.com>
In-Reply-To
<20170919192744.19224-1-benpeart@microsoft.com>

Thanks again to everyone who provided feedback. There are several spelling, documentation, and code comment changes.

The only behavioral change from V7 is the removal of unnecessary uses of CE_MATCH_IGNORE_FSMONITOR. With a better understanding of *why* the CE_MATCH_IGNORE_* flags are used, it is now clear they are not required in most cases where CE_MATCH_IGNORE_FSMONITOR was being passed out of an abundance of caution.

Since the fsmonitor data can be trusted and is kept in sync with the working directory, the only remaining valid uses are those locations where we don't want to trigger an unneeded refresh_fsmonitor() call.

One is where preload_index() is doing a fast precompute of state for the bulk of the index entries but is not required for correctness as refresh_cache_ent() will ensure any "missed" by preload_index() are up-to-date if/when they are needed.

The second is in is_staging_gitmodules_ok() where we don't want to trigger a complete refresh just to check the .gitignore file.

The net result of this change will be that there are more cases where we will be able to use the cached index state and avoid unnecessary lstat() calls.

Interdiff between V7 and V8:
diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt
index 95231dbfcb..7c2f880a22 100644
--- a/Documentation/git-update-index.txt
+++ b/Documentation/git-update-index.txt
@@ -476,7 +476,7 @@ inform it as to what files have been modified. This enables git to avoid
 having to lstat() every file to find modified files.
 
 When used in conjunction with the untracked cache, it can further improve
-performance by avoiding the cost of scaning the entire working directory
+performance by avoiding the cost of scanning the entire working directory
 looking for new files.
 
 If you want to enable (or disable) this feature, it is easier to use
diff --git a/apply.c b/apply.c
index 9061cc5f15..71cbbd141c 100644
--- a/apply.c
+++ b/apply.c
@@ -3399,7 +3399,7 @@ static int verify_index_match(const struct cache_entry *ce, struct stat *st)
 			return -1;
 		return 0;
 	}
-	return ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);
+	return ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);
 }
 
 #define SUBMODULE_PATCH_WITHOUT_INDEX 1
diff --git a/cache.h b/cache.h
index eccab968bd..f1c903e1b6 100644
--- a/cache.h
+++ b/cache.h
@@ -682,7 +682,7 @@ extern void *read_blob_data_from_index(const struct index_state *, const char *,
 #define CE_MATCH_IGNORE_MISSING		0x08
 /* enable stat refresh */
 #define CE_MATCH_REFRESH		0x10
-/* do stat comparison even if CE_FSMONITOR_VALID is true */
+/* don't refresh_fsmonitor state or do stat comparison even if CE_FSMONITOR_VALID is true */
 #define CE_MATCH_IGNORE_FSMONITOR 0X20
 extern int ie_match_stat(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);
 extern int ie_modified(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);
diff --git a/entry.c b/entry.c
index 5e6794f9fc..3a7b667373 100644
--- a/entry.c
+++ b/entry.c
@@ -404,7 +404,7 @@ int checkout_entry(struct cache_entry *ce,
 
 	if (!check_path(path.buf, path.len, &st, state->base_dir_len)) {
 		const struct submodule *sub;
-		unsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);
+		unsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);
 		/*
 		 * Needs to be checked before !changed returns early,
 		 * as the possibly empty directory was not changed
diff --git a/fsmonitor.c b/fsmonitor.c
index b8b2d88fe1..7c1540c054 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -176,7 +176,7 @@ void refresh_fsmonitor(struct index_state *istate)
 			core_fsmonitor, query_success ? "success" : "failure");
 	}
 
-	/* a fsmonitor process can return '*' to indicate all entries are invalid */
+	/* a fsmonitor process can return '/' to indicate all entries are invalid */
 	if (query_success && query_result.buf[0] != '/') {
 		/* Mark all entries returned by the monitor as dirty */
 		buf = query_result.buf;
diff --git a/fsmonitor.h b/fsmonitor.h
index c2240b811a..8eb6163455 100644
--- a/fsmonitor.h
+++ b/fsmonitor.h
@@ -35,7 +35,9 @@ extern void tweak_fsmonitor(struct index_state *istate);
 extern void refresh_fsmonitor(struct index_state *istate);
 
 /*
- * Set the given cache entries CE_FSMONITOR_VALID bit.
+ * Set the given cache entries CE_FSMONITOR_VALID bit. This should be
+ * called any time the cache entry has been updated to reflect the
+ * current state of the file on disk.
  */
 static inline void mark_fsmonitor_valid(struct cache_entry *ce)
 {
@@ -46,8 +48,11 @@ static inline void mark_fsmonitor_valid(struct cache_entry *ce)
 }
 
 /*
- * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate any
- * corresponding untracked cache directory structures.
+ * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate
+ * any corresponding untracked cache directory structures. This should
+ * be called any time git creates or modifies a file that should
+ * trigger an lstat() or invalidate the untracked cache for the
+ * corresponding directory
  */
 static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)
 {
diff --git a/read-cache.c b/read-cache.c
index 53093dbebf..05c0a33fdd 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -641,7 +641,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,
 	int size, namelen, was_same;
 	mode_t st_mode = st->st_mode;
 	struct cache_entry *ce, *alias;
-	unsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR;
+	unsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;
 	int verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);
 	int pretend = flags & ADD_CACHE_PRETEND;
 	int intent_only = flags & ADD_CACHE_INTENT;
@@ -1356,7 +1356,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,
 	int first = 1;
 	int in_porcelain = (flags & REFRESH_IN_PORCELAIN);
 	unsigned int options = (CE_MATCH_REFRESH |
-				(really ? CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_FSMONITOR : 0) |
+				(really ? CE_MATCH_IGNORE_VALID : 0) |
 				(not_new ? CE_MATCH_IGNORE_MISSING : 0));
 	const char *modified_fmt;
 	const char *deleted_fmt;
diff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c
index 4e5ca8f397..bd1a857d52 100644
--- a/t/helper/test-drop-caches.c
+++ b/t/helper/test-drop-caches.c
@@ -82,6 +82,8 @@ static int cmd_dropcaches(void)
 	HANDLE hProcess = GetCurrentProcess();
 	HANDLE hToken;
 	HMODULE ntdll;
+	DWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG);
+	SYSTEM_MEMORY_LIST_COMMAND command;
 	int status;
 
 	if (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))
@@ -96,12 +98,12 @@ static int cmd_dropcaches(void)
 	if (!ntdll)
 		return error("Can't load ntdll.dll, wrong Windows version?");
 
-	DWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =
+	NtSetSystemInformation =
 		(DWORD(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, "NtSetSystemInformation");
 	if (!NtSetSystemInformation)
 		return error("Can't get function addresses, wrong Windows version?");
 
-	SYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;
+	command = MemoryPurgeStandbyList;
 	status = NtSetSystemInformation(
 		SystemMemoryListInformation,
 		&command,
diff --git a/unpack-trees.c b/unpack-trees.c
index f724a61ac0..1f5d371636 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1456,7 +1456,7 @@ static int verify_uptodate_1(const struct cache_entry *ce,
 		return 0;
 
 	if (!lstat(ce->name, &st)) {
-		int flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR;
+		int flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE;
 		unsigned changed = ie_match_stat(o->src_index, ce, &st, flags);
 
 		if (submodule_from_ce(ce)) {
@@ -1612,7 +1612,7 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le
 	const struct cache_entry *src;
 
 	src = index_file_exists(o->src_index, name, len, 1);
-	return src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);
+	return src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);
 }
 
 static int check_ok_to_remove(const char *name, int len, int dtype,
@@ -2136,7 +2136,7 @@ int oneway_merge(const struct cache_entry * const *src,
 		if (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old)) {
 			struct stat st;
 			if (lstat(old->name, &st) ||
-			    ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR))
+			    ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE))
 				update |= CE_UPDATE;
 		}
 		add_entry(o, old, update, 0);

Ben Peart (12):
  bswap: add 64 bit endianness helper get_be64
  preload-index: add override to enable testing preload-index
  update-index: add a new --force-write-index option
  fsmonitor: teach git to optionally utilize a file system monitor to
    speed up detecting new or changed files.
  fsmonitor: add documentation for the fsmonitor extension.
  ls-files: Add support in ls-files to display the fsmonitor valid bit
  update-index: add fsmonitor support to update-index
  fsmonitor: add a test tool to dump the index extension
  split-index: disable the fsmonitor extension when running the split
    index test
  fsmonitor: add test cases for fsmonitor extension
  fsmonitor: add a sample integration script for Watchman
  fsmonitor: add a performance test

 Documentation/config.txt                   |   7 +
 Documentation/git-ls-files.txt             |   7 +-
 Documentation/git-update-index.txt         |  45 +++++
 Documentation/githooks.txt                 |  28 +++
 Documentation/technical/index-format.txt   |  19 ++
 Makefile                                   |   3 +
 builtin/ls-files.c                         |   8 +-
 builtin/update-index.c                     |  38 +++-
 cache.h                                    |  10 +-
 compat/bswap.h                             |  22 +++
 config.c                                   |  14 ++
 config.h                                   |   1 +
 diff-lib.c                                 |   2 +
 dir.c                                      |  27 ++-
 dir.h                                      |   2 +
 entry.c                                    |   2 +
 environment.c                              |   1 +
 fsmonitor.c                                | 253 ++++++++++++++++++++++++
 fsmonitor.h                                |  66 +++++++
 preload-index.c                            |   8 +-
 read-cache.c                               |  45 ++++-
 submodule.c                                |   2 +-
 t/helper/.gitignore                        |   1 +
 t/helper/test-drop-caches.c                | 164 ++++++++++++++++
 t/helper/test-dump-fsmonitor.c             |  21 ++
 t/perf/p7519-fsmonitor.sh                  | 184 +++++++++++++++++
 t/t1700-split-index.sh                     |   1 +
 t/t7519-status-fsmonitor.sh                | 304 +++++++++++++++++++++++++++++
 t/t7519/fsmonitor-all                      |  24 +++
 t/t7519/fsmonitor-none                     |  22 +++
 t/t7519/fsmonitor-watchman                 | 140 +++++++++++++
 templates/hooks--fsmonitor-watchman.sample | 122 ++++++++++++
 unpack-trees.c                             |   2 +
 33 files changed, 1572 insertions(+), 23 deletions(-)
 create mode 100644 fsmonitor.c
 create mode 100644 fsmonitor.h
 create mode 100644 t/helper/test-drop-caches.c
 create mode 100644 t/helper/test-dump-fsmonitor.c
 create mode 100755 t/perf/p7519-fsmonitor.sh
 create mode 100755 t/t7519-status-fsmonitor.sh
 create mode 100755 t/t7519/fsmonitor-all
 create mode 100755 t/t7519/fsmonitor-none
 create mode 100755 t/t7519/fsmonitor-watchman
 create mode 100755 templates/hooks--fsmonitor-watchman.sample

-- 
2.14.1.549.g6ff7ed0467
Previous: Ben PeartNext: Ben Peart
Message 110 of 137 in “Fast git status via a file system watcher”
  1. 0/7 Fast git status via a file system watcherBen Peart, Jun 10, 2017
  2. 2/7 dir: make lookup_untracked() available outside of dir.cBen Peart, Jun 10, 2017
  3. 1/7 bswap: add 64 bit endianness helper get_be64Ben Peart, Jun 10, 2017
  4. 4/7 fsmonitor: add test cases for fsmonitor extensionBen Peart, Jun 10, 2017
  5. Christian CouderJun 27, 2017
  6. Ben PeartJul 7, 2017
  7. 3/7 fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.Ben Peart, Jun 10, 2017
  8. Christian CouderJun 27, 2017
  9. Ben PeartJul 3, 2017
  10. 6/7 fsmonitor: add a sample query-fsmonitor hook script for WatchmanBen Peart, Jun 10, 2017
  11. 7/7 fsmonitor: add a performance testBen Peart, Jun 10, 2017
  12. Ben PeartJun 10, 2017
  13. Junio C HamanoJun 12, 2017
  14. Ben PeartJun 14, 2017
  15. Junio C HamanoJun 14, 2017
  16. Ben PeartJul 7, 2017
  17. Junio C HamanoJul 7, 2017
  18. Ben PeartJul 7, 2017
  19. David TurnerJul 7, 2017
  20. Christian CouderJul 8, 2017
  21. 5/7 fsmonitor: add documentation for the fsmonitor extension.Ben Peart, Jun 10, 2017
  22. Christian CouderJun 28, 2017
  23. Ben PeartJul 10, 2017
  24. Ben PeartJul 10, 2017
  25. 00/12 Fast git status via a file system watcherBen Peart, Sep 15, 2017
  26. 01/12 bswap: add 64 bit endianness helper get_be64Ben Peart, Sep 15, 2017
  27. 04/12 fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.Ben Peart, Sep 15, 2017
  28. David TurnerSep 15, 2017
  29. Ben PeartSep 18, 2017
  30. David TurnerSep 18, 2017
  31. Ben PeartSep 18, 2017
  32. 05/12 fsmonitor: add documentation for the fsmonitor extension.Ben Peart, Sep 15, 2017
  33. David TurnerSep 15, 2017
  34. Ben PeartSep 18, 2017
  35. Junio C HamanoSep 17, 2017
  36. Ben PeartSep 18, 2017
  37. 07/12 update-index: add fsmonitor support to update-indexBen Peart, Sep 15, 2017
  38. 06/12 ls-files: Add support in ls-files to display the fsmonitor valid bitBen Peart, Sep 15, 2017
  39. David TurnerSep 15, 2017
  40. 10/12 fsmonitor: add test cases for fsmonitor extensionBen Peart, Sep 15, 2017
  41. David TurnerSep 15, 2017
  42. David TurnerSep 19, 2017
  43. Ben PeartSep 19, 2017
  44. Torsten BögershausenSep 16, 2017
  45. 1/1 test-lint: echo -e (or -E) is not portabletboegi@web.de, Sep 17, 2017
  46. Jonathan NiederSep 19, 2017
  47. Torsten BögershausenSep 20, 2017
  48. Junio C HamanoSep 22, 2017
  49. Ben PeartSep 18, 2017
  50. Junio C HamanoSep 17, 2017
  51. Ben PeartSep 18, 2017
  52. Jonathan NiederSep 19, 2017
  53. 12/12 fsmonitor: add a performance testBen Peart, Sep 15, 2017
  54. David TurnerSep 15, 2017
  55. Johannes SchindelinSep 18, 2017
  56. Ben PeartSep 18, 2017
  57. Johannes SchindelinSep 19, 2017
  58. 11/12 fsmonitor: add a sample integration script for WatchmanBen Peart, Sep 15, 2017
  59. 08/12 fsmonitor: add a test tool to dump the index extensionBen Peart, Sep 15, 2017
  60. Junio C HamanoSep 17, 2017
  61. Ben PeartSep 18, 2017
  62. Torsten BögershausenSep 18, 2017
  63. Ben PeartSep 18, 2017
  64. Torsten BögershausenSep 19, 2017
  65. Ben PeartSep 19, 2017
  66. 09/12 split-index: disable the fsmonitor extension when running the split index testBen Peart, Sep 15, 2017
  67. Jonathan NiederSep 19, 2017
  68. Ben PeartSep 20, 2017
  69. Jonathan NiederSep 20, 2017
  70. Ben PeartSep 21, 2017
  71. 03/12 update-index: add a new --force-write-index optionBen Peart, Sep 15, 2017
  72. 02/12 preload-index: add override to enable testing preload-indexBen Peart, Sep 15, 2017
  73. 00/12 Fast git status via a file system watcherBen Peart, Sep 19, 2017
  74. 01/12 bswap: add 64 bit endianness helper get_be64Ben Peart, Sep 19, 2017
  75. 02/12 preload-index: add override to enable testing preload-indexBen Peart, Sep 19, 2017
  76. Stefan BellerSep 20, 2017
  77. Ben PeartSep 21, 2017
  78. Stefan BellerSep 21, 2017
  79. 05/12 fsmonitor: add documentation for the fsmonitor extension.Ben Peart, Sep 19, 2017
  80. Martin ÅgrenSep 20, 2017
  81. Ben PeartSep 20, 2017
  82. Martin ÅgrenSep 20, 2017
  83. 04/12 fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.Ben Peart, Sep 19, 2017
  84. Junio C HamanoSep 20, 2017
  85. Ben PeartSep 20, 2017
  86. Junio C HamanoSep 21, 2017
  87. Ben PeartSep 21, 2017
  88. Ben PeartSep 21, 2017
  89. Junio C HamanoSep 22, 2017
  90. Junio C HamanoSep 20, 2017
  91. Ben PeartSep 20, 2017
  92. 08/12 fsmonitor: add a test tool to dump the index extensionBen Peart, Sep 19, 2017
  93. 09/12 split-index: disable the fsmonitor extension when running the split index testBen Peart, Sep 19, 2017
  94. 07/12 update-index: add fsmonitor support to update-indexBen Peart, Sep 19, 2017
  95. 10/12 fsmonitor: add test cases for fsmonitor extensionBen Peart, Sep 19, 2017
  96. 03/12 update-index: add a new --force-write-index optionBen Peart, Sep 19, 2017
  97. Junio C HamanoSep 20, 2017
  98. Ben PeartSep 20, 2017
  99. Junio C HamanoSep 21, 2017
  100. Ben PeartSep 21, 2017
  101. Junio C HamanoSep 21, 2017
  102. Junio C HamanoSep 21, 2017
  103. 12/12 fsmonitor: add a performance testBen Peart, Sep 19, 2017
  104. 11/12 fsmonitor: add a sample integration script for WatchmanBen Peart, Sep 19, 2017
  105. 06/12 ls-files: Add support in ls-files to display the fsmonitor valid bitBen Peart, Sep 19, 2017
  106. David TurnerSep 19, 2017
  107. Ben PeartSep 19, 2017
  108. David TurnerSep 19, 2017
  109. Ben PeartSep 19, 2017
  110. 00/12 Fast git status via a file system watcherBen Peart, Sep 22, 2017
  111. 02/12 preload-index: add override to enable testing preload-indexBen Peart, Sep 22, 2017
  112. 01/12 bswap: add 64 bit endianness helper get_be64Ben Peart, Sep 22, 2017
  113. Martin ÅgrenSep 22, 2017
  114. Ben PeartSep 23, 2017
  115. Jeff KingSep 24, 2017
  116. Junio C HamanoSep 24, 2017
  117. 03/12 update-index: add a new --force-write-index optionBen Peart, Sep 22, 2017
  118. 11/12 fsmonitor: add a sample integration script for WatchmanBen Peart, Sep 22, 2017
  119. 04/12 fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.Ben Peart, Sep 22, 2017
  120. 10/12 fsmonitor: add test cases for fsmonitor extensionBen Peart, Sep 22, 2017
  121. 09/12 split-index: disable the fsmonitor extension when running the split index testBen Peart, Sep 22, 2017
  122. 12/12 fsmonitor: add a performance testBen Peart, Sep 22, 2017
  123. 06/12 ls-files: Add support in ls-files to display the fsmonitor valid bitBen Peart, Sep 22, 2017
  124. 05/12 fsmonitor: add documentation for the fsmonitor extension.Ben Peart, Sep 22, 2017
  125. 07/12 update-index: add fsmonitor support to update-indexBen Peart, Sep 22, 2017
  126. 08/12 fsmonitor: add a test tool to dump the index extensionBen Peart, Sep 22, 2017
  127. Martin ÅgrenSep 22, 2017
  128. Ben PeartSep 23, 2017
  129. Junio C HamanoSep 24, 2017
  130. Junio C HamanoSep 29, 2017
  131. Ben PeartSep 29, 2017
  132. Junio C HamanoOct 1, 2017
  133. Ben PeartOct 3, 2017
  134. Junio C HamanoOct 4, 2017
  135. Alex VandiverOct 4, 2017
  136. Ben PeartOct 4, 2017
  137. Ben PeartOct 4, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.