git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 14/26] read-cache: put some limits on file watching

From
Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date
Feb 3, 2014, 04:29 UTC
Message-ID
<1391401754-15347-15-git-send-email-pclouds@gmail.com>
In-Reply-To
<1391401754-15347-1-git-send-email-pclouds@gmail.com>

watch_entries() is a lot of computation and could trigger a lot more lookups in file-watcher. Normally after the first set of watches are in place, we do not need to update often. Moreover if the number of entries is small, the overhead of file watcher may actually slow git down.

This patch only allows to update watches if the number of watchable files is over a limit (and there are new files added if this is not the first time). Measurements on Core i5-2520M and Linux 3.7.6, about 920 lstat() take 1ms. Somewhere between 2^16 and 2^17 lstat calls that it starts to take longer than 100ms. 2^16 is chosen at the minimum limit to start using file watcher.

Of course this is only sensible default for single-repo use case. Lower it when you need to work with many small repos.

Recently updated files are not considered watchable because they are likely to be updated again soon, not worth the ping-pong game with file watcher. The default limit 10min is just a random value. Recent limit is ignored if there are no watched files (e.g. a fresh clone, or after a bad hand shake with file watcher).

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/config.txt                 |  9 +++++++
 Documentation/technical/index-format.txt |  3 +++
 cache.h                                  |  1 +
 file-watcher-lib.c                       | 42 ++++++++++++++++++++++++++------
 read-cache.c                             | 11 ++++++---
 5 files changed, 56 insertions(+), 10 deletions(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 6ad653a..451c100 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1052,6 +1052,15 @@ filewatcher.timeout::
 	the file watcher to respond before giving up. Default value is
 	50. Setting to -1 makes Git wait forever.
 
+filewatcher.minfiles::
+	Start watching files if the number of watchable files are
+	above this limit. Default value is 65536.
+
+filewatcher.recentlimit::
+	Files that are last updated within filewatcher.recentlimit
+	seconds from now are not considered watchable. Default value
+	is 600 (5 minutes).
+
 fetch.recurseSubmodules::
 	This option can be either set to a boolean value or to 'on-demand'.
 	Setting it to a boolean changes the behavior of fetch and pull to
diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt
index 24fd0ae..7081e55 100644
--- a/Documentation/technical/index-format.txt
+++ b/Documentation/technical/index-format.txt
@@ -204,3 +204,6 @@ Git index format
 
   - A bit map of all entries in the index, n-th bit of m-th byte
     corresponds to CE_WATCHED of the <m * 8+ n>-th index entry.
+
+  - 1-byte, non-zero indicates the index should be scanned for new
+    watched entries.
diff --git a/cache.h b/cache.h
index b3ea574..10ff33e 100644
--- a/cache.h
+++ b/cache.h
@@ -279,6 +279,7 @@ struct index_state {
 	struct cache_tree *cache_tree;
 	struct cache_time timestamp;
 	unsigned name_hash_initialized : 1,
+		 update_watches : 1,
 		 initialized : 1;
 	struct hash_table name_hash;
 	struct hash_table dir_hash;
diff --git a/file-watcher-lib.c b/file-watcher-lib.c
index 791faae..d4949a5 100644
--- a/file-watcher-lib.c
+++ b/file-watcher-lib.c
@@ -5,6 +5,8 @@
 
 static char *watcher_path;
 static int WAIT_TIME = 50;	/* in ms */
+static int watch_lowerlimit = 65536;
+static int recent_limit = 600;
 
 static int connect_watcher(const char *path)
 {
@@ -22,12 +24,17 @@ static int connect_watcher(const char *path)
 
 static void reset_watches(struct index_state *istate, int disconnect)
 {
-	int i;
+	int i, changed = 0;
 	for (i = 0; i < istate->cache_nr; i++)
 		if (istate->cache[i]->ce_flags & CE_WATCHED) {
 			istate->cache[i]->ce_flags &= ~(CE_WATCHED | CE_VALID);
-			istate->cache_changed = 1;
+			changed = 1;
 		}
+	recent_limit = 0;
+	if (changed) {
+		istate->update_watches = 1;
+		istate->cache_changed = 1;
+	}
 	if (disconnect && istate->watcher > 0) {
 		close(istate->watcher);
 		istate->watcher = -1;
@@ -49,6 +56,14 @@ static int watcher_config(const char *var, const char *value, void *data)
 		WAIT_TIME = git_config_int(var, value);
 		return 0;
 	}
+	if (!strcmp(var, "filewatcher.minfiles")) {
+		watch_lowerlimit = git_config_int(var, value);
+		return 0;
+	}
+	if (!strcmp(var, "filewatcher.recentlimit")) {
+		recent_limit = git_config_int(var, value);
+		return 0;
+	}
 	return 0;
 }
 
@@ -63,12 +78,18 @@ void open_watcher(struct index_state *istate)
 	}
 
 	if (!read_config) {
+		int i;
 		/*
 		 * can't hook into git_default_config because
 		 * read_cache() may be called even before git_config()
 		 * call.
 		 */
 		git_config(watcher_config, NULL);
+		for (i = 0; i < istate->cache_nr; i++)
+			if (istate->cache[i]->ce_flags & CE_WATCHED)
+				break;
+		if (i == istate->cache_nr)
+			recent_limit = 0;
 		read_config = 1;
 	}
 
@@ -86,6 +107,7 @@ void open_watcher(struct index_state *istate)
 	    (msg = packet_read_line_timeout(istate->watcher, WAIT_TIME, NULL)) == NULL ||
 	    strcmp(msg, "ok")) {
 		reset_watches(istate, 0);
+		istate->update_watches = 1;
 		return;
 	}
 }
@@ -99,7 +121,7 @@ static int sort_by_date(const void *a_, const void *b_)
 	return seca - secb;
 }
 
-static inline int ce_watchable(struct cache_entry *ce)
+static inline int ce_watchable(struct cache_entry *ce, time_t now)
 {
 	return
 		!(ce->ce_flags & CE_WATCHED) &&
@@ -109,7 +131,8 @@ static inline int ce_watchable(struct cache_entry *ce)
 		 * obviously. S_IFLNK could be problematic because
 		 * inotify may follow symlinks without IN_DONT_FOLLOW
 		 */
-		S_ISREG(ce->ce_mode);
+		S_ISREG(ce->ce_mode) &&
+		(ce->ce_stat_data.sd_mtime.sec + recent_limit <= now);
 }
 
 static void send_watches(struct index_state *istate,
@@ -158,15 +181,20 @@ void watch_entries(struct index_state *istate)
 {
 	int i, nr;
 	struct cache_entry **sorted;
+	time_t now = time(NULL);
 
-	if (istate->watcher <= 0)
+	if (istate->watcher <= 0 || !istate->update_watches)
 		return;
+	istate->update_watches = 0;
+	istate->cache_changed = 1;
 	for (i = nr = 0; i < istate->cache_nr; i++)
-		if (ce_watchable(istate->cache[i]))
+		if (ce_watchable(istate->cache[i], now))
 			nr++;
+	if (nr < watch_lowerlimit)
+		return;
 	sorted = xmalloc(sizeof(*sorted) * nr);
 	for (i = nr = 0; i < istate->cache_nr; i++)
-		if (ce_watchable(istate->cache[i]))
+		if (ce_watchable(istate->cache[i], now))
 			sorted[nr++] = istate->cache[i];
 	qsort(sorted, nr, sizeof(*sorted), sort_by_date);
 	send_watches(istate, sorted, nr);
diff --git a/read-cache.c b/read-cache.c
index cb2188f..dc49858 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1009,6 +1009,7 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti
 			(istate->cache_nr - pos - 1) * sizeof(ce));
 	set_index_entry(istate, pos, ce);
 	istate->cache_changed = 1;
+	istate->update_watches = 1;
 	return 0;
 }
 
@@ -1295,13 +1296,14 @@ static void read_watch_extension(struct index_state *istate, uint8_t *data,
 				 unsigned long sz)
 {
 	int i;
-	if ((istate->cache_nr + 7) / 8 != sz) {
+	if ((istate->cache_nr + 7) / 8 + 1 != sz) {
 		error("invalid 'WATC' extension");
 		return;
 	}
 	for (i = 0; i < istate->cache_nr; i++)
 		if (data[i / 8] & (1 << (i % 8)))
 			istate->cache[i]->ce_flags |= CE_WATCHED;
+	istate->update_watches = data[sz - 1];
 }
 
 static int read_index_extension(struct index_state *istate,
@@ -1488,6 +1490,7 @@ int read_index_from(struct index_state *istate, const char *path)
 	istate->cache_alloc = alloc_nr(istate->cache_nr);
 	istate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));
 	istate->initialized = 1;
+	istate->update_watches = 1;
 
 	if (istate->version == 4)
 		previous_name = &previous_name_buf;
@@ -1896,8 +1899,9 @@ int write_index(struct index_state *istate, int newfd)
 		if (err)
 			return -1;
 	}
-	if (has_watches) {
-		int id, sz = (entries - removed + 7) / 8;
+	if (has_watches ||
+	    (istate->watcher != -1 && !istate->update_watches)) {
+		int id, sz = (entries - removed + 7) / 8 + 1;
 		uint8_t *data = xmalloc(sz);
 		memset(data, 0, sz);
 		for (i = 0, id = 0; i < entries && has_watches; i++) {
@@ -1910,6 +1914,7 @@ int write_index(struct index_state *istate, int newfd)
 			}
 			id++;
 		}
+		data[sz - 1] = istate->update_watches;
 		err = write_index_ext_header(&c, newfd, CACHE_EXT_WATCH, sz) < 0
 			|| ce_write(&c, newfd, data, sz) < 0;
 		free(data);

-- 
1.8.5.2.240.g8478abd
Previous: Nguyễn Thái Ngọc DuyNext: Nguyễn Thái Ngọc Duy
Message 51 of 72 in “inotify support”
  1. 0/6 inotify supportNguyễn Thái Ngọc Duy, Jan 12, 2014
  2. 1/6 read-cache: save trailing sha-1Nguyễn Thái Ngọc Duy, Jan 12, 2014
  3. 2/6 read-cache: new extension to mark what file is watchedNguyễn Thái Ngọc Duy, Jan 12, 2014
  4. Jonathan NiederJan 13, 2014
  5. Duy NguyenJan 14, 2014
  6. Duy NguyenJan 14, 2014
  7. 3/6 read-cache: connect to file watcherNguyễn Thái Ngọc Duy, Jan 12, 2014
  8. Jeff KingJan 15, 2014
  9. 4/6 read-cache: get "updated" path list from file watcherNguyễn Thái Ngọc Duy, Jan 12, 2014
  10. 5/6 read-cache: ask file watcher to watch filesNguyễn Thái Ngọc Duy, Jan 12, 2014
  11. 6/6 file-watcher: support inotifyNguyễn Thái Ngọc Duy, Jan 12, 2014
  12. 00/14 inotify supportNguyễn Thái Ngọc Duy, Jan 17, 2014
  13. 01/14 read-cache: save trailing sha-1Nguyễn Thái Ngọc Duy, Jan 17, 2014
  14. 02/14 read-cache: new extension to mark what file is watchedNguyễn Thái Ngọc Duy, Jan 17, 2014
  15. Thomas GummererJan 17, 2014
  16. Thomas RastJan 19, 2014
  17. Duy NguyenJan 20, 2014
  18. 03/14 read-cache: connect to file watcherNguyễn Thái Ngọc Duy, Jan 17, 2014
  19. Torsten BögershausenJan 17, 2014
  20. Duy NguyenJan 17, 2014
  21. 04/14 read-cache: ask file watcher to watch filesNguyễn Thái Ngọc Duy, Jan 17, 2014
  22. 05/14 read-cache: put some limits on file watchingNguyễn Thái Ngọc Duy, Jan 17, 2014
  23. Thomas RastJan 19, 2014
  24. Duy NguyenJan 20, 2014
  25. 06/14 read-cache: get modified file list from file watcherNguyễn Thái Ngọc Duy, Jan 17, 2014
  26. 07/14 read-cache: add config to start file watcher automaticallyNguyễn Thái Ngọc Duy, Jan 17, 2014
  27. 08/14 read-cache: add GIT_TEST_FORCE_WATCHER for testingNguyễn Thái Ngọc Duy, Jan 17, 2014
  28. Thomas RastJan 19, 2014
  29. Duy NguyenJan 20, 2014
  30. 09/14 file-watcher: add --shutdown and --log optionsNguyễn Thái Ngọc Duy, Jan 17, 2014
  31. 10/14 file-watcher: automatically quitNguyễn Thái Ngọc Duy, Jan 17, 2014
  32. 11/14 file-watcher: support inotifyNguyễn Thái Ngọc Duy, Jan 17, 2014
  33. Thomas RastJan 19, 2014
  34. Duy NguyenJan 20, 2014
  35. Thomas RastJan 20, 2014
  36. Duy NguyenJan 28, 2014
  37. 00/26 inotify supportNguyễn Thái Ngọc Duy, Feb 3, 2014
  38. 01/26 pkt-line.c: rename global variable buffer[] to something less genericNguyễn Thái Ngọc Duy, Feb 3, 2014
  39. 02/26 pkt-line.c: add packet_write_timeout()Nguyễn Thái Ngọc Duy, Feb 3, 2014
  40. 03/26 pkt-line.c: add packet_read_line_timeout()Nguyễn Thái Ngọc Duy, Feb 3, 2014
  41. 04/26 unix-socket: make unlink() optional in unix_stream_listen()Nguyễn Thái Ngọc Duy, Feb 3, 2014
  42. 05/26 Add git-file-watcher and basic connection handling logicNguyễn Thái Ngọc Duy, Feb 3, 2014
  43. 06/26 file-watcher: check socket directory permissionNguyễn Thái Ngọc Duy, Feb 3, 2014
  44. 07/26 file-watcher: remove socket on exitNguyễn Thái Ngọc Duy, Feb 3, 2014
  45. 08/26 file-watcher: add --detachNguyễn Thái Ngọc Duy, Feb 3, 2014
  46. 09/26 read-cache: save trailing sha-1Nguyễn Thái Ngọc Duy, Feb 3, 2014
  47. 10/26 read-cache: new flag CE_WATCHED to mark what file is watchedNguyễn Thái Ngọc Duy, Feb 3, 2014
  48. 11/26 Clear CE_WATCHED when set CE_VALID aloneNguyễn Thái Ngọc Duy, Feb 3, 2014
  49. 12/26 read-cache: basic hand shaking to the file watcherNguyễn Thái Ngọc Duy, Feb 3, 2014
  50. 13/26 read-cache: ask file watcher to watch filesNguyễn Thái Ngọc Duy, Feb 3, 2014
  51. 14/26 read-cache: put some limits on file watchingNguyễn Thái Ngọc Duy, Feb 3, 2014
  52. 15/26 read-cache: get changed file list from file watcherNguyễn Thái Ngọc Duy, Feb 3, 2014
  53. 16/26 git-compat-util.h: add inotify stubs on non-Linux platformsNguyễn Thái Ngọc Duy, Feb 3, 2014
  54. 17/26 file-watcher: inotify support, watching partNguyễn Thái Ngọc Duy, Feb 3, 2014
  55. 18/26 file-watcher: inotify support, notification partNguyễn Thái Ngọc Duy, Feb 3, 2014
  56. 19/26 Wrap CE_VALID test with ce_valid()Nguyễn Thái Ngọc Duy, Feb 3, 2014
  57. 20/26 read-cache: new variable to verify file-watcher resultsNguyễn Thái Ngọc Duy, Feb 3, 2014
  58. 21/26 Support running file watcher with the test suiteNguyễn Thái Ngọc Duy, Feb 3, 2014
  59. 22/26 file-watcher: quit if $WATCHER/socket is goneNguyễn Thái Ngọc Duy, Feb 3, 2014
  60. 23/26 file-watcher: tests for the daemonNguyễn Thái Ngọc Duy, Feb 3, 2014
  61. 24/26 ls-files: print CE_WATCHED as W (or "w" with CE_VALID)Nguyễn Thái Ngọc Duy, Feb 3, 2014
  62. 25/26 file-watcher: tests for the client sideNguyễn Thái Ngọc Duy, Feb 3, 2014
  63. 26/26 Disable file-watcher with system inotify on some testsNguyễn Thái Ngọc Duy, Feb 3, 2014
  64. Torsten BögershausenFeb 8, 2014
  65. Duy NguyenFeb 8, 2014
  66. Torsten BögershausenFeb 9, 2014
  67. Duy NguyenFeb 10, 2014
  68. Torsten BögershausenFeb 10, 2014
  69. Duy NguyenFeb 10, 2014
  70. Duy NguyenFeb 17, 2014
  71. Shawn PearceFeb 19, 2014
  72. Duy NguyenFeb 19, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.