{"thread":{"id":"45997","subject":"[PATCH v2 0/6] Fast git status via a file system watcher","startedAt":"2017-05-18T20:13:52Z","lastAt":"2017-05-31T14:10:36Z","messageCount":30,"participants":["Ben Peart","Junio C Hamano","Ævar Arnfjörð Bjarmason","Torsten Bögershausen","Christian Couder"],"isPatch":true,"patchVersion":2,"patchTotal":6},"messages":[{"id":"320202","messageId":"20170518201333.13088-1-benpeart@microsoft.com","threadId":"45997","inReplyTo":null,"subject":"[PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-18T20:13:27Z","receivedAt":"2017-05-18T20:13:52Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Changes from V1 include:\n\n - add 64 bit endianness helper get_be64 in compat/bswap.h\n - switch to using get_be helpers when reading index extension\n - fix leak of strbuf in refresh_by_fsmonitor\n - rename update_istate and clean up parameter list\n - make t/t7519-status-fsmonitor.sh executable\n - update test cases to ensure fsmonitor extension is being used\n - update commit messages and titles\n\n\nGoal\n~~~~\n�\nToday, git must check existing files to see if there have been changes\nand scan the working directory looking for new, untracked files. As the\nnumber of files and folders in the working directory increases, the time\nto perform these checks can become very expensive O(# files in working\ndirectory).\n\nGiven the number of new or modified files is typically a very small\npercentage of the total number of files, it would be much more\nperformant if git only had to check files and folders that potentially\nhad changes. This reduces the cost to O(# modified files).\n\nThis patch series makes it possible to optionally add a hook process\nthat can return the set of files that may have been changed since the\nrequested time. Git can then use this to limit its scan to only those\nfiles and folders that potentially have changes.\n\nDesign\n~~~~~~\n\nA new git hook (query-fsmonitor) must exist and be enabled \n(core.fsmonitor=true) that takes a time_t formatted as a string and\noutputs to stdout all files that have been modified since the requested\ntime.\n\nA new 'fsmonitor' index extension has been added to store the time the\nfsmonitor hook was last queried and a ewah bitmap of the current\n'fsmonitor-dirty' files. Unmarked entries are 'fsmonitor-clean', marked\nentries are 'fsmonitor-dirty.'\n\nAs needed, git will call the query-fsmonitor hook proc for the set of\nchanges since the index was last updated. Git then uses this set of\nfiles along with the list saved in the fsmonitor index extension to flag\nthe potentially dirty index and untracked cache entries. \n\nrefresh_index() and valid_cached_dir() are updated so that any entry not\nflagged as potentially dirty is not checked as it cannot have any\nchanges. This saves all the work of checking files and folders for\nchanges that are already known to be clean.\n\nIf git finds out some entries are 'fsmonitor-dirty', but are really\nunchanged (e.g. the file was changed, then reverted back), then Git will\nclear the marking in the extension. If git adds or updates an index\nentry, it is marked 'fsmonitor-dirty' to ensure it is checked for\nchanges.\n\nThe code is conservative so in case of any error (missing index\nextension, error from hook, etc) it falls back to normal logic of\nchecking everything.\n\nA sample hook is provided in query-fsmonitor.sample to integrate with\nthe cross platform Watchman file watching service\nhttps://facebook.github.io/watchman/\n\n\nPerformance\n~~~~~~~~~~~\n\nThe performance wins of this model are pretty dramatic. Each test was\nrun 3 times and averaged. \"Files\" is the number of files in the working\ndirectory. Tests were done with a cold file system cache as well as\nwith a warm file system cache on a HDD. SSD speeds were typically about\n10x faster than the HDD. Typical real world results would fall\nsomewhere between these extremes. \n\n*--------------------------------------------------------*\n| Repo on HDD | Cache | fsmonitor=false | fsmonitor=true |\n*--------------------------------------------------------*\n| 3K Files | Cold | 0.77s | 0.55s |\n+--------------------------------------------------------+\n| 100K Files | Cold | 38.76s | 2.17s |\n+--------------------------------------------------------+\n| 3M Files | Cold | 421.55s | 18.57s |\n+--------------------------------------------------------+\n| 3K Files | Warm | 0.05s | 0.24s |\n+--------------------------------------------------------+\n| 100K Files | Warm | 1.13s | 0.40s |\n+--------------------------------------------------------+\n| 3M Files | Warm | 59.33s | 4.19s |\n+--------------------------------------------------------+\n\nNote that with the smallest repo, warm times actually increase slightly\nas the overhead of calling the hook, watchman and perl outweighs the\nsavings of not scanning the working directory.\n\n\nCredits\n~~~~~~~\n\nIdea taken and code refactored from \nhttp://public-inbox.org/git/1466914464-10358-1-git-send-email-novalis@novalis.org/\n\nCurrent version as a fork of GFW on GitHub here: \nhttps://github.com/benpeart/git-for-windows/tree/fsmonitor\n\nBen Peart (6):\n bswap: add 64 bit endianness helper get_be64\n dir: make lookup_untracked() available outside of dir.c\n fsmonitor: teach git to optionally utilize a file system monitor to\n speed up detecting new or changed files.\n fsmonitor: add test cases for fsmonitor extension\n fsmonitor: add documentation for the fsmonitor extension.\n fsmonitor: add a sample query-fsmonitor hook script for Watchman\n\n Documentation/config.txt | 7 +\n Documentation/githooks.txt | 23 +++\n Documentation/technical/index-format.txt | 18 +++\n Makefile | 1 +\n builtin/update-index.c | 1 +\n cache.h | 5 +\n compat/bswap.h | 4 +\n config.c | 5 +\n dir.c | 15 +-\n dir.h | 5 +\n entry.c | 1 +\n environment.c | 1 +\n fsmonitor.c | 231 +++++++++++++++++++++++++++++++\n fsmonitor.h | 9 ++\n read-cache.c | 28 +++-\n t/t7519-status-fsmonitor.sh | 153 ++++++++++++++++++++\n templates/hooks--query-fsmonitor.sample | 27 ++++\n unpack-trees.c | 1 +\n 18 files changed, 532 insertions(+), 3 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n create mode 100755 t/t7519-status-fsmonitor.sh\n create mode 100644 templates/hooks--query-fsmonitor.sample\n\n-- \n2.13.0.windows.1.6.g4597375fc3\n\n"},{"id":"320203","messageId":"20170518201333.13088-2-benpeart@microsoft.com","threadId":"45997","inReplyTo":"20170518201333.13088-1-benpeart@microsoft.com","subject":"[PATCH v2 1/6] bswap: add 64 bit endianness helper get_be64","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-18T20:13:28Z","receivedAt":"2017-05-18T20:14:09Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a new get_be64 macro to enable 64 bit endian conversions on memory\nthat may or may not be aligned.\n\nSigned-off-by: Ben Peart \n---\n compat/bswap.h | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/compat/bswap.h b/compat/bswap.h\nindex d47c003544..f89fe7f4b5 100644\n--- a/compat/bswap.h\n+++ b/compat/bswap.h\n@@ -158,6 +158,7 @@ static inline uint64_t git_bswap64(uint64_t x)\n \n #define get_be16(p)\tntohs(*(unsigned short *)(p))\n #define get_be32(p)\tntohl(*(unsigned int *)(p))\n+#define get_be64(p)\tntohll(*(uint64_t *)(p))\n #define put_be32(p, v)\tdo { *(unsigned int *)(p) = htonl(v); } while (0)\n \n #else\n@@ -170,6 +171,9 @@ static inline uint64_t git_bswap64(uint64_t x)\n \t(*((unsigned char *)(p) + 1) << 16) | \\\n \t(*((unsigned char *)(p) + 2) << 8) | \\\n \t(*((unsigned char *)(p) + 3) << 0) )\n+#define get_be64(p)\t( \\\n+\t((uint64_t)get_be32((unsigned char *)(p) + 0) << 32) | \\\n+\t((uint64_t)get_be32((unsigned char *)(p) + 4) << 0)\n #define put_be32(p, v)\tdo { \\\n \tunsigned int __v = (v); \\\n \t*((unsigned char *)(p) + 0) = __v >> 24; \\\n-- \n2.13.0.windows.1.6.g4597375fc3\n\n"},{"id":"320204","messageId":"20170518201333.13088-3-benpeart@microsoft.com","threadId":"45997","inReplyTo":"20170518201333.13088-1-benpeart@microsoft.com","subject":"[PATCH v2 2/6] dir: make lookup_untracked() available outside of dir.c","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-18T20:13:29Z","receivedAt":"2017-05-18T20:14:13Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Remove the static qualifier from lookup_untracked() and make it\navailable to other modules by exporting it from dir.h. This will be\nused later when we need to find entries to mark 'fsmonitor dirty.'\n\nSigned-off-by: Ben Peart \n---\n dir.c | 2 +-\n dir.h | 3 +++\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex f451bfa48c..1b5558fdf9 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -660,7 +660,7 @@ static void trim_trailing_spaces(char *buf)\n *\n * If \"name\" has the trailing slash, it'll be excluded in the search.\n */\n-static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n \t\t\t\t\t\t struct untracked_cache_dir *dir,\n \t\t\t\t\t\t const char *name, int len)\n {\ndiff --git a/dir.h b/dir.h\nindex bf23a470af..9e387551bd 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -339,4 +339,7 @@ extern void connect_work_tree_and_git_dir(const char *work_tree, const char *git\n extern void relocate_gitdir(const char *path,\n \t\t\t const char *old_git_dir,\n \t\t\t const char *new_git_dir);\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+\t\t\t\t\t struct untracked_cache_dir *dir,\n+\t\t\t\t\t const char *name, int len);\n #endif\n-- \n2.13.0.windows.1.6.g4597375fc3\n\n"},{"id":"320205","messageId":"20170518201333.13088-4-benpeart@microsoft.com","threadId":"45997","inReplyTo":"20170518201333.13088-1-benpeart@microsoft.com","subject":"[PATCH v2 3/6] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-18T20:13:30Z","receivedAt":"2017-05-18T20:14:17Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"When the index is read from disk, the query-fsmonitor index extension is\nused to flag the last known potentially dirty index and untracked cach\nentries.\n\nIf git finds out some entries are 'fsmonitor-dirty', but are really\nunchanged (e.g. the file was changed, then reverted back), then Git will\nclear the marking in the extension. If git adds or updates an index\nentry, it is marked 'fsmonitor-dirty' to ensure it is checked for\nchanges in the working directory.\n\nBefore the 'fsmonitor-dirty' flags are used to limit the scope of the\nfiles to be checked, the query-fsmonitor hook proc is called with the\ntime the index was last updated. The hook proc returns the list of\nfiles changed since that last updated time and the list of\npotentially dirty entries is updated to reflect the current state.\n\nrefresh_index() and valid_cached_dir() are updated so that any entry not\nflagged as potentially dirty is not checked as it cannot have any\nchanges.\n\nSigned-off-by: Ben Peart \n---\n Makefile | 1 +\n builtin/update-index.c | 1 +\n cache.h | 5 ++\n config.c | 5 ++\n dir.c | 13 +++\n dir.h | 2 +\n entry.c | 1 +\n environment.c | 1 +\n fsmonitor.c | 231 +++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor.h | 9 ++\n read-cache.c | 28 +++++-\n unpack-trees.c | 1 +\n 12 files changed, 296 insertions(+), 2 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n\ndiff --git a/Makefile b/Makefile\nindex e35542e631..488a4466cc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -760,6 +760,7 @@ LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n+LIB_OBJS += fsmonitor.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex ebfc09faa0..32fd977b43 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -232,6 +232,7 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tactive_cache[pos]->ce_flags |= CE_FSMONITOR_DIRTY;\n \t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\ndiff --git a/cache.h b/cache.h\nindex 188811920c..36423c77cc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -201,6 +201,7 @@ struct cache_entry {\n #define CE_ADDED (1 << 19)\n \n #define CE_HASHED (1 << 20)\n+#define CE_FSMONITOR_DIRTY (1 << 21)\n #define CE_WT_REMOVE (1 << 22) /* remove in work directory */\n #define CE_CONFLICTED (1 << 23)\n \n@@ -324,6 +325,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n #define UNTRACKED_CHANGED\t(1 << 7)\n+#define FSMONITOR_CHANGED\t(1 << 8)\n \n struct split_index;\n struct untracked_cache;\n@@ -342,6 +344,8 @@ struct index_state {\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n \tstruct untracked_cache *untracked;\n+\ttime_t fsmonitor_last_update;\n+\tstruct ewah_bitmap *fsmonitor_dirty_bitmap;\n };\n \n extern struct index_state the_index;\n@@ -766,6 +770,7 @@ extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n extern int git_db_env, git_index_env, git_graft_env, git_common_dir_env;\n+extern int core_fsmonitor;\n \n /*\n * Include broken refs in all ref iterations, which will\ndiff --git a/config.c b/config.c\nindex bb4d735701..1a8108636d 100644\n--- a/config.c\n+++ b/config.c\n@@ -1232,6 +1232,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.fsmonitor\")) {\n+\t\tcore_fsmonitor = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/dir.c b/dir.c\nindex 1b5558fdf9..da428489e2 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1652,6 +1652,18 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n+\trefresh_by_fsmonitor(&the_index);\n+\tif (dir->untracked->use_fsmonitor) {\n+\t\t/*\n+\t\t * With fsmonitor, we can trust the untracked cache's\n+\t\t * valid field.\n+\t\t */\n+\t\tif (untracked->valid)\n+\t\t\tgoto skip_stat;\n+\t\telse\n+\t\t\tinvalidate_directory(dir->untracked, untracked);\n+\t}\n+\n \tif (stat(path->len ? path->buf : \".\", &st)) {\n \t\tinvalidate_directory(dir->untracked, untracked);\n \t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n@@ -1665,6 +1677,7 @@ static int valid_cached_dir(struct dir_struct *dir,\n \t\treturn 0;\n \t}\n \n+skip_stat:\n \tif (untracked->check_only != !!check_only) {\n \t\tinvalidate_directory(dir->untracked, untracked);\n \t\treturn 0;\ndiff --git a/dir.h b/dir.h\nindex 9e387551bd..ff6a00abcc 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -139,6 +139,8 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n+\t/* fsmonitor invalidation data */\n+\tunsigned int use_fsmonitor : 1;\n };\n \n struct dir_struct {\ndiff --git a/entry.c b/entry.c\nindex d2b512da90..c2d3c1079c 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -221,6 +221,7 @@ static int write_entry(struct cache_entry *ce,\n \t\t\tlstat(ce->name, &st);\n \t\tfill_stat_cache_info(ce, &st);\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 560408953c..1afabbae8c 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -64,6 +64,7 @@ int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\n enum hide_dotfiles_type hide_dotfiles = HIDE_DOTFILES_DOTGITONLY;\n enum log_refs_config log_all_ref_updates = LOG_REFS_UNSET;\n+int core_fsmonitor;\n \n #ifndef PROTECT_HFS_DEFAULT\n #define PROTECT_HFS_DEFAULT 0\ndiff --git a/fsmonitor.c b/fsmonitor.c\nnew file mode 100644\nindex 0000000000..6356dc795e\n--- /dev/null\n+++ b/fsmonitor.c\n@@ -0,0 +1,231 @@\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"ewah/ewok.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"fsmonitor.h\"\n+\n+static struct untracked_cache_dir *find_untracked_cache_dir(\n+\tstruct untracked_cache *uc, struct untracked_cache_dir *ucd,\n+\tconst char *name)\n+{\n+\tconst char *end;\n+\tstruct untracked_cache_dir *dir = ucd;\n+\n+\tif (!*name)\n+\t\treturn dir;\n+\n+\tend = strchr(name, '/');\n+\tif (end) {\n+\t\tdir = lookup_untracked(uc, ucd, name, end - name);\n+\t\tif (dir)\n+\t\t\treturn find_untracked_cache_dir(uc, dir, end + 1);\n+\t}\n+\n+\treturn dir;\n+}\n+\n+/* This function will be passed to ewah_each_bit() */\n+static void mark_no_fsmonitor(size_t pos, void *is)\n+{\n+\tstruct index_state *istate = is;\n+\tstruct untracked_cache_dir *dir;\n+\tstruct cache_entry *ce = istate->cache[pos];\n+\n+\tassert(pos < istate->cache_nr);\n+\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n+\n+\tif (!istate->untracked || !istate->untracked->root)\n+\t\treturn;\n+\n+\tdir = find_untracked_cache_dir(istate->untracked, istate->untracked->root, ce->name);\n+\tif (dir)\n+\t\tdir->valid = 0;\n+}\n+\n+int read_fsmonitor_extension(struct index_state *istate, const void *data,\n+\tunsigned long sz)\n+{\n+\tconst char *index = data;\n+\tuint32_t hdr_version;\n+\tuint32_t ewah_size;\n+\tint ret;\n+\n+\tif (sz < sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t))\n+\t\treturn error(\"corrupt fsmonitor extension (too short)\");\n+\n+\thdr_version = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\tif (hdr_version != 1)\n+\t\treturn error(\"bad fsmonitor version %d\", hdr_version);\n+\n+\tistate->fsmonitor_last_update = (time_t)get_be64(index);\n+\tindex += sizeof(uint64_t);\n+\n+\tewah_size = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\n+\tistate->fsmonitor_dirty_bitmap = ewah_new();\n+\tret = ewah_read_mmap(istate->fsmonitor_dirty_bitmap, index, ewah_size);\n+\tif (ret != ewah_size) {\n+\t\tewah_free(istate->fsmonitor_dirty_bitmap);\n+\t\tistate->fsmonitor_dirty_bitmap = NULL;\n+\t\treturn error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n+\t}\n+\n+\treturn 0;\n+}\n+\n+void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n+{\n+\tuint32_t hdr_version;\n+\tuint64_t tm;\n+\tstruct ewah_bitmap *bitmap;\n+\tint i;\n+\tuint32_t ewah_start;\n+\tuint32_t ewah_size = 0;\n+\tint fixup = 0;\n+\n+\thdr_version = htonl(1);\n+\tstrbuf_add(sb, &hdr_version, sizeof(uint32_t));\n+\n+\ttm = htonll((uint64_t)istate->fsmonitor_last_update);\n+\tstrbuf_add(sb, &tm, sizeof(uint64_t));\n+\tfixup = sb->len;\n+\tstrbuf_add(sb, &ewah_size, sizeof(uint32_t)); /* we'll fix this up later */\n+\n+\tewah_start = sb->len;\n+\tbitmap = ewah_new();\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_DIRTY)\n+\t\t\tewah_set(bitmap, i);\n+\tewah_serialize_strbuf(bitmap, sb);\n+\tewah_free(bitmap);\n+\n+\t/* fix up size field */\n+\tewah_size = htonl(sb->len - ewah_start);\n+\tmemcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t));\n+}\n+\n+static void mark_file_dirty(struct index_state *istate, const char *name)\n+{\n+\tstruct untracked_cache_dir *dir;\n+\tint pos;\n+\n+\t/* find it in the index and mark that entry as dirty */\n+\tpos = index_name_pos(istate, name, strlen(name));\n+\tif (pos >= 0)\n+\t\tistate->cache[pos]->ce_flags |= CE_FSMONITOR_DIRTY;\n+\n+\t/*\n+\t * Find the corresponding directory in the untracked cache\n+\t * and mark it as invalid\n+\t */\n+\tif (!istate->untracked || !istate->untracked->root)\n+\t\treturn;\n+\n+\tdir = find_untracked_cache_dir(istate->untracked, istate->untracked->root, name);\n+\tif (dir)\n+\t\tdir->valid = 0;\n+}\n+\n+/*\n+ * Call the query-fsmonitor hook passing the time of the last saved results.\n+ */\n+static int query_fsmonitor(time_t last_update, struct strbuf *query_result)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tchar date[64];\n+\tconst char *argv[3];\n+\n+\tif (!(argv[0] = find_hook(\"query-fsmonitor\")))\n+\t\treturn -1;\n+\n+\tsnprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update);\n+\targv[1] = date;\n+\targv[2] = NULL;\n+\tcp.argv = argv;\n+\tcp.out = -1;\n+\n+\treturn capture_command(&cp, query_result, 1024);\n+}\n+\n+void process_fsmonitor_extension(struct index_state *istate)\n+{\n+\tif (!istate->fsmonitor_dirty_bitmap)\n+\t\treturn;\n+\n+\tewah_each_bit(istate->fsmonitor_dirty_bitmap, mark_no_fsmonitor, istate);\n+\tewah_free(istate->fsmonitor_dirty_bitmap);\n+\tistate->fsmonitor_dirty_bitmap = NULL;\n+}\n+\n+void refresh_by_fsmonitor(struct index_state *istate)\n+{\n+\tstatic has_run_once = FALSE;\n+\tstruct strbuf query_result = STRBUF_INIT;\n+\tint query_success = 0;\n+\tsize_t bol = 0; /* beginning of line */\n+\ttime_t last_update;\n+\tchar *buf, *entry;\n+\tint i;\n+\n+\tif (!core_fsmonitor || has_run_once)\n+\t\treturn;\n+\thas_run_once = TRUE;\n+\n+\t/*\n+\t * This could be racy so save the date/time now and the hook\n+\t * should be inclusive to ensure we don't miss potential changes.\n+\t */\n+\tlast_update = time(NULL);\n+\n+\t/* If we have a last update time, call query-monitor for the set of changes since that time */\n+\tif (istate->fsmonitor_last_update) {\n+\t\tquery_success = !query_fsmonitor(istate->fsmonitor_last_update, &query_result);\n+\t}\n+\n+\tif (query_success) {\n+\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\tbuf = entry = query_result.buf;\n+\t\tfor (i = 0; i < query_result.len; i++) {\n+\t\t\tif (buf[i] != '\\0')\n+\t\t\t\tcontinue;\n+\t\t\tmark_file_dirty(istate, buf + bol);\n+\t\t\tbol = i + 1;\n+\t\t}\n+\t\tif (bol < query_result.len)\n+\t\t\tmark_file_dirty(istate, buf + bol);\n+\n+\t\t/* Mark all clean entries up-to-date */\n+\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\t\tif (ce_stage(ce) || (ce->ce_flags & CE_FSMONITOR_DIRTY))\n+\t\t\t\tcontinue;\n+\t\t\tce_mark_uptodate(ce);\n+\t\t}\n+\n+\t\t/*\n+\t\t * Now that we've marked the invalid entries in the\n+\t\t * untracked-cache itself, we can mark the untracked cache for\n+\t\t * fsmonitor usage.\n+\t\t */\n+\t\tif (istate->untracked) {\n+\t\t\tistate->untracked->use_fsmonitor = 1;\n+\t\t}\n+\t}\n+\telse {\n+\t\t/* if we can't update the cache, fall back to checking them all */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_DIRTY;\n+\n+\t\t/* mark the untracked cache as unusable for fsmonitor */\n+\t\tif (istate->untracked)\n+\t\t\tistate->untracked->use_fsmonitor = 0;\n+\t}\n+\tstrbuf_release(&query_result);\n+\n+\t/* Now that we've updated istate, save the last_update time */\n+\tistate->fsmonitor_last_update = last_update;\n+\tistate->cache_changed |= FSMONITOR_CHANGED;\n+}\ndiff --git a/fsmonitor.h b/fsmonitor.h\nnew file mode 100644\nindex 0000000000..57b061688f\n--- /dev/null\n+++ b/fsmonitor.h\n@@ -0,0 +1,9 @@\n+#ifndef FSMONITOR_H\n+#define FSMONITOR_H\n+\n+int read_fsmonitor_extension(struct index_state *istate, const void *data, unsigned long sz);\n+void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate);\n+void process_fsmonitor_extension(struct index_state *istate);\n+void refresh_by_fsmonitor(struct index_state *istate);\n+\n+#endif\ndiff --git a/read-cache.c b/read-cache.c\nindex 3339de8124..5b52f08db2 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -18,6 +18,7 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"utf8.h\"\n+#include \"fsmonitor.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -37,11 +38,12 @@\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n #define CACHE_EXT_UNTRACKED 0x554E5452\t /* \"UNTR\" */\n+#define CACHE_EXT_FSMONITOR 0x46534D4E\t /* \"FSMN\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -61,6 +63,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \tfree(old);\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n }\n \n@@ -777,6 +780,7 @@ int chmod_index_entry(struct index_state *istate, struct cache_entry *ce,\n \t}\n \tcache_tree_invalidate_path(istate, ce->name);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n \n \treturn 0;\n@@ -1344,6 +1348,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tconst char *added_fmt;\n \tconst char *unmerged_fmt;\n \n+\trefresh_by_fsmonitor(istate);\n+\n \tmodified_fmt = (in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\");\n \tdeleted_fmt = (in_porcelain ? \"D\\t%s\\n\" : \"%s: needs update\\n\");\n \ttypechange_fmt = (in_porcelain ? \"T\\t%s\\n\" : \"%s needs update\\n\");\n@@ -1380,8 +1386,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\tcontinue;\n \n \t\tnew = refresh_cache_ent(istate, ce, options, &cache_errno, &changed);\n-\t\tif (new == ce)\n+\t\tif (new == ce) {\n+\t\t\tce->ce_flags &= ~CE_FSMONITOR_DIRTY;\n \t\t\tcontinue;\n+\t\t}\n+\n \t\tif (!new) {\n \t\t\tconst char *fmt;\n \n@@ -1391,6 +1400,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\t\t */\n \t\t\t\tce->ce_flags &= ~CE_VALID;\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\t\t\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \t\t\t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t\t\t}\n \t\t\tif (quiet)\n@@ -1549,6 +1559,9 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_UNTRACKED:\n \t\tistate->untracked = read_untracked_extension(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_FSMONITOR:\n+\t\tread_fsmonitor_extension(istate, data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1721,6 +1734,7 @@ static void post_read_index_from(struct index_state *istate)\n \tcheck_ce_order(istate);\n \ttweak_untracked_cache(istate);\n \ttweak_split_index(istate);\n+\tprocess_fsmonitor_extension(istate);\n }\n \n /* remember to discard_cache() before reading a different cache! */\n@@ -2300,6 +2314,16 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->fsmonitor_last_update) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_fsmonitor_extension(&sb, istate);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_FSMONITOR, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st))\n \t\treturn -1;\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex aa15111fef..259e6960b9 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -412,6 +412,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n \tif (was_skip_worktree != ce_skip_worktree(ce)) {\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \n-- \n2.13.0.windows.1.6.g4597375fc3\n\n"},{"id":"320206","messageId":"20170518201333.13088-5-benpeart@microsoft.com","threadId":"45997","inReplyTo":"20170518201333.13088-1-benpeart@microsoft.com","subject":"[PATCH v2 4/6] fsmonitor: add test cases for fsmonitor extension","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-18T20:13:31Z","receivedAt":"2017-05-18T20:14:19Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add test cases that ensure status results are correct when using the new\nfsmonitor extension. Test untracked, modified, and new files by\nensuring the results are identical to when not using the extension.\n\nAdd a test to ensure updates to the index properly mark corresponding\nentries in the index extension as dirty so that the status is correct\nafter commands that modify the index but don't trigger changes in the\nworking directory.\n\nAdd a test that verifies that if the fsmonitor extension doesn't tell\ngit about a change, it doesn't discover it on its own. This ensures\ngit is honoring the extension and that we get the performance benefits\ndesired.\n\nAll test hooks output a marker file that is used to ensure the hook\nwas actually used to generate the test results.\n\nSigned-off-by: Ben Peart \n---\n t/t7519-status-fsmonitor.sh | 153 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 153 insertions(+)\n create mode 100755 t/t7519-status-fsmonitor.sh\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nnew file mode 100755\nindex 0000000000..d3cffc758f\n--- /dev/null\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -0,0 +1,153 @@\n+#!/bin/sh\n+\n+test_description='git status with file system watcher'\n+\n+. ./test-lib.sh\n+\n+clean_repo () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+\trm marker -f\n+}\n+\n+dirty_repo () {\n+\t: >untracked\n+\t: >dir1/untracked\n+\t: >dir2/untracked\n+\techo 1 >modified\n+\techo 2 >dir1/modified\n+\techo 3 >dir2/modified\n+\techo 4 >new\n+\techo 5 >dir1/new\n+\techo 6 >dir2/new\n+\tgit add new\n+\tgit add dir1/new\n+\tgit add dir2/new\n+}\n+\n+# The test query-fsmonitor hook proc will output a marker file we can use to\n+# ensure the hook was actually used to generate the correct results.\n+\n+test_expect_success 'setup' '\n+\tmkdir -p .git/hooks &&\n+\twrite_script .git/hooks/query-fsmonitor<<-\\EOF &&\n+\t: >marker\n+\tprintf \"untracked\\0\"\n+\tprintf \"dir1/untracked\\0\"\n+\tprintf \"dir2/untracked\\0\"\n+\tprintf \"modified\\0\"\n+\tprintf \"dir1/modified\\0\"\n+\tprintf \"dir2/modified\\0\"\n+\tprintf \"new\\0\"\"\n+\tprintf \"dir1/new\\0\"\n+\tprintf \"dir2/new\\0\"\n+\tEOF\n+\t: >tracked &&\n+\t: >modified &&\n+\tmkdir dir1 &&\n+\t: >dir1/tracked &&\n+\t: >dir1/modified &&\n+\tmkdir dir2 &&\n+\t: >dir2/tracked &&\n+\t: >dir2/modified &&\n+\tgit add . &&\n+\ttest_tick &&\n+\tgit commit -m initial &&\n+\tdirty_repo\n+'\n+\n+cat >.gitignore <<\\EOF\n+.gitignore\n+expect*\n+output*\n+marker*\n+EOF\n+\n+# Status is well tested elsewhere so we'll just ensure that the results are\n+# the same when using core.fsmonitor. First call after turning on the option\n+# does a complete scan so need to do two calls to ensure we test the new\n+# codepath.\n+\n+test_expect_success 'status with core.untrackedcache true' '\n+\tgit config core.fsmonitor true &&\n+\tgit config core.untrackedcache true &&\n+\tgit -c core.fsmonitor=false -c core.untrackedcache=true status >expect &&\n+\tclean_repo &&\n+\tgit status &&\n+\ttest_path_is_missing marker &&\n+\tdirty_repo &&\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\ttest_i18ncmp expect output\n+'\n+\n+\n+test_expect_success 'status with core.untrackedcache false' '\n+\tgit config core.fsmonitor true &&\n+\tgit config core.untrackedcache false &&\n+\tgit -c core.fsmonitor=false -c core.untrackedcache=false status >expect &&\n+\tclean_repo &&\n+\tgit status &&\n+\ttest_path_is_missing marker &&\n+\tdirty_repo &&\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\ttest_i18ncmp expect output\n+'\n+\n+# Ensure commands that call refresh_index() to move the index back in time\n+# properly invalidate the fsmonitor cache\n+\n+test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n+\tgit config core.fsmonitor true &&\n+\tgit config core.untrackedcache true &&\n+\tclean_repo &&\n+\tgit status &&\n+\ttest_path_is_missing marker &&\n+\tdirty_repo &&\n+\twrite_script .git/hooks/query-fsmonitor<<-\\EOF &&\n+\t:>marker\n+\tEOF\n+\tgit add . &&\n+\tgit commit -m \"to reset\" &&\n+\tgit status &&\n+\ttest_path_is_file marker &&\n+\tgit reset HEAD~1 &&\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\tgit -c core.fsmonitor=false status >expect &&\n+\ttest_i18ncmp expect output\n+'\n+\n+# Now make sure it's actually skipping the check for modified and untracked\n+# files unless it is told about them. Note, after \"git reset --hard HEAD\" no\n+# extensions exist other than 'TREE' so do a \"git status\" to get the extension\n+# written before testing the results.\n+\n+test_expect_success 'status doesnt detect unreported modifications' '\n+\tgit config core.fsmonitor true &&\n+\tgit config core.untrackedcache true &&\n+\twrite_script .git/hooks/query-fsmonitor<<-\\EOF &&\n+\t:>marker\n+\tEOF\n+\tclean_repo &&\n+\tgit status &&\n+\ttest_path_is_missing marker &&\n+\t: >untracked &&\n+\techo 2 >dir1/modified &&\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\ttest_i18ngrep ! \"Changes not staged for commit:\" output &&\n+\ttest_i18ngrep ! \"Untracked files:\" output &&\n+\twrite_script .git/hooks/query-fsmonitor<<-\\EOF &&\n+\t:>marker\n+\tprintf \"untracked%s\\0\"\n+\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\ttest_i18ngrep \"Changes not staged for commit:\" output &&\n+\ttest_i18ngrep \"Untracked files:\" output\n+'\n+\n+test_done\n-- \n2.13.0.windows.1.6.g4597375fc3\n\n"},{"id":"320207","messageId":"20170518201333.13088-7-benpeart@microsoft.com","threadId":"45997","inReplyTo":"20170518201333.13088-1-benpeart@microsoft.com","subject":"[PATCH v2 6/6] fsmonitor: add a sample query-fsmonitor hook script for Watchman","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-18T20:13:33Z","receivedAt":"2017-05-18T20:14:21Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This hook script integrates the new fsmonitor capabilities of git with\nthe cross platform Watchman file watching service. To use the script:\n\nDownload and install Watchman from https://facebook.github.io/watchman/\nand instruct Watchman to watch your working directory for changes\n('watchman watch-project /usr/src/git').\n\nRename the sample integration hook from query-fsmonitor.sample to\nquery-fsmonitor.\n\nConfigure git to use the extension ('git config core.fsmonitor true')\nand optionally turn on the untracked cache for optimal performance\n('git config core.untrackedcache true').\n\nSigned-off-by: Ben Peart \nSigned-off-by: Johannes Schindelin \n---\n templates/hooks--query-fsmonitor.sample | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n create mode 100644 templates/hooks--query-fsmonitor.sample\n\ndiff --git a/templates/hooks--query-fsmonitor.sample b/templates/hooks--query-fsmonitor.sample\nnew file mode 100644\nindex 0000000000..4bd22f21d8\n--- /dev/null\n+++ b/templates/hooks--query-fsmonitor.sample\n@@ -0,0 +1,27 @@\n+#!/bin/sh\n+#\n+# An example hook script to integrate Watchman\n+# (https://facebook.github.io/watchman/) with git to provide fast\n+# git status.\n+#\n+# The hook is passed a time_t formatted as a string and outputs to\n+# stdout all files that have been modified since the given time.\n+# Paths must be relative to the root of the working tree and\n+# separated by a single NUL.\n+#\n+# To enable this hook, rename this file to \"query-fsmonitor\"\n+\n+# Convert unix style paths to escaped Windows style paths\n+case \"$(uname -s)\" in\n+MINGW*|MSYS_NT*)\n+ GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,\\\\\\\\,g')\"\n+ ;;\n+*)\n+ GIT_WORK_TREE=\"$PWD\"\n+ ;;\n+esac\n+\n+# Query Watchman for all the changes since the requested time\n+echo \"[\\\"query\\\", \\\"$GIT_WORK_TREE\\\", {\\\"since\\\": $1, \\\"fields\\\":[\\\"name\\\"]}]\" | \\\n+watchman -j | \\\n+perl -e 'use JSON::PP; my $o = JSON::PP->new->utf8->decode(join(\"\", <>)); die \"Watchman: $o->{'error'}.\\nFalling back to scanning...\\n\" if defined($o->{\"error\"}); print(join(\"\\0\", @{$o->{\"files\"}}));'\n-- \n2.13.0.windows.1.6.g4597375fc3\n\n"},{"id":"320208","messageId":"20170518201333.13088-6-benpeart@microsoft.com","threadId":"45997","inReplyTo":"20170518201333.13088-1-benpeart@microsoft.com","subject":"[PATCH v2 5/6] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-18T20:13:32Z","receivedAt":"2017-05-18T20:14:24Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This includes the core.fsmonitor setting, the query-fsmonitor hook,\nand the fsmonitor index extension.\n\nSigned-off-by: Ben Peart \n---\n Documentation/config.txt | 7 +++++++\n Documentation/githooks.txt | 23 +++++++++++++++++++++++\n Documentation/technical/index-format.txt | 18 ++++++++++++++++++\n 3 files changed, 48 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 96e9cf8b73..4ffbf0d4c2 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -389,6 +389,13 @@ core.protectNTFS::\n \t8.3 \"short\" names.\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n+core.fsmonitor::\n+\tIf set to true, call the query-fsmonitor hook proc which will\n+\tidentify all files that may have had changes since the last\n+\trequest. This information is used to speed up operations like\n+\t'git commit' and 'git status' by limiting what git must scan to\n+\tdetect changes.\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 706091a569..f7b4b4a844 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -448,6 +448,29 @@ The commits are guaranteed to be listed in the order that they were\n processed by rebase.\n \n \n+[[query-fsmonitor]]\n+query-fsmonitor\n+~~~~~~~~~~~~\n+\n+This hook is invoked when the configuration option core.fsmonitor is\n+set and git needs to identify changed or untracked files. It takes\n+a single argument which is the time in elapsed seconds since midnight,\n+January 1, 1970.\n+\n+The hook should output to stdout the list of all files in the working\n+directory that may have changed since the requested time. The logic\n+should be inclusive so that it does not miss any potential changes.\n+The paths should be relative to the root of the working directory\n+and be separated by a single NUL.\n+\n+Git will limit what files it checks for changes as well as which\n+directories are checked for untracked files based on the path names\n+given.\n+\n+The exit status determines whether git will use the data from the\n+hook to limit its search. On error, it will fall back to verifying\n+all files and folders.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex ade0b0c445..b002d23c05 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -295,3 +295,21 @@ The remaining data of each directory block is grouped by type:\n in the previous ewah bitmap.\n \n - One NUL.\n+\n+== File System Monitor cache\n+\n+ The file system monitor cache tracks files for which the query-fsmonitor\n+ hook has told us about changes. The signature for this extension is\n+ { 'F', 'S', 'M', 'N' }.\n+\n+ The extension starts with\n+\n+ - 32-bit version number: the current supported version is 1.\n+\n+ - 64-bit time: the extension data reflects all changes through the given\n+\ttime which is stored as the seconds elapsed since midnight, January 1, 1970.\n+\n+ - 32-bit bitmap size: the size of the CE_FSMONITOR_DIRTY bitmap.\n+\n+ - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n+ is CE_FSMONITOR_DIRTY.\n-- \n2.13.0.windows.1.6.g4597375fc3\n\n"},{"id":"320276","messageId":"41d6939f-0095-cfc5-cde5-2ebdd74a09b2@gmail.com","threadId":"45997","inReplyTo":"20170518201333.13088-4-benpeart@microsoft.com","subject":"Re: [PATCH v2 3/6] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-19T15:33:24Z","receivedAt":"2017-05-19T15:33:34Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"After sending this, I noticed that using a different compiler generated \na couple of warnings I should fix. I'm assuming I'll need another \nre-roll but if not, here are the changes that will be squashed in.\n\nBen\n\n\n\ndiff --git a/dir.c b/dir.c\nindex da428489e2..37f1c580a5 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -16,6 +16,7 @@\n #include \"utf8.h\"\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsmonitor.h\"\n\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 6356dc795e..9f08e66db9 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -162,7 +162,7 @@ void process_fsmonitor_extension(struct index_state \n*istate)\n\n void refresh_by_fsmonitor(struct index_state *istate)\n {\n- static has_run_once = FALSE;\n+ static int has_run_once = FALSE;\n struct strbuf query_result = STRBUF_INIT;\n int query_success = 0;\n size_t bol = 0; /* beginning of line */\n\n\nOn 5/18/2017 4:13 PM, Ben Peart wrote:\n> When the index is read from disk, the query-fsmonitor index extension is\n> used to flag the last known potentially dirty index and untracked cach\n> entries.\n>\n> If git finds out some entries are 'fsmonitor-dirty', but are really\n> unchanged (e.g. the file was changed, then reverted back), then Git will\n> clear the marking in the extension. If git adds or updates an index\n> entry, it is marked 'fsmonitor-dirty' to ensure it is checked for\n> changes in the working directory.\n>\n> Before the 'fsmonitor-dirty' flags are used to limit the scope of the\n> files to be checked, the query-fsmonitor hook proc is called with the\n> time the index was last updated. The hook proc returns the list of\n> files changed since that last updated time and the list of\n> potentially dirty entries is updated to reflect the current state.\n>\n> refresh_index() and valid_cached_dir() are updated so that any entry not\n> flagged as potentially dirty is not checked as it cannot have any\n> changes.\n>\n> Signed-off-by: Ben Peart \n> ---\n> Makefile | 1 +\n> builtin/update-index.c | 1 +\n> cache.h | 5 ++\n> config.c | 5 ++\n> dir.c | 13 +++\n> dir.h | 2 +\n> entry.c | 1 +\n> environment.c | 1 +\n> fsmonitor.c | 231 +++++++++++++++++++++++++++++++++++++++++++++++++\n> fsmonitor.h | 9 ++\n> read-cache.c | 28 +++++-\n> unpack-trees.c | 1 +\n> 12 files changed, 296 insertions(+), 2 deletions(-)\n> create mode 100644 fsmonitor.c\n> create mode 100644 fsmonitor.h\n>\n> diff --git a/Makefile b/Makefile\n> index e35542e631..488a4466cc 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -760,6 +760,7 @@ LIB_OBJS += ewah/ewah_rlw.o\n> LIB_OBJS += exec_cmd.o\n> LIB_OBJS += fetch-pack.o\n> LIB_OBJS += fsck.o\n> +LIB_OBJS += fsmonitor.o\n> LIB_OBJS += gettext.o\n> LIB_OBJS += gpg-interface.o\n> LIB_OBJS += graph.o\n> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> index ebfc09faa0..32fd977b43 100644\n> --- a/builtin/update-index.c\n> +++ b/builtin/update-index.c\n> @@ -232,6 +232,7 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n> \t\telse\n> \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n> \t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n> +\t\tactive_cache[pos]->ce_flags |= CE_FSMONITOR_DIRTY;\n> \t\tcache_tree_invalidate_path(&the_index, path);\n> \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n> \t\treturn 0;\n> diff --git a/cache.h b/cache.h\n> index 188811920c..36423c77cc 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -201,6 +201,7 @@ struct cache_entry {\n> #define CE_ADDED (1 << 19)\n>\n> #define CE_HASHED (1 << 20)\n> +#define CE_FSMONITOR_DIRTY (1 << 21)\n> #define CE_WT_REMOVE (1 << 22) /* remove in work directory */\n> #define CE_CONFLICTED (1 << 23)\n>\n> @@ -324,6 +325,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n> #define CACHE_TREE_CHANGED\t(1 << 5)\n> #define SPLIT_INDEX_ORDERED\t(1 << 6)\n> #define UNTRACKED_CHANGED\t(1 << 7)\n> +#define FSMONITOR_CHANGED\t(1 << 8)\n>\n> struct split_index;\n> struct untracked_cache;\n> @@ -342,6 +344,8 @@ struct index_state {\n> \tstruct hashmap dir_hash;\n> \tunsigned char sha1[20];\n> \tstruct untracked_cache *untracked;\n> +\ttime_t fsmonitor_last_update;\n> +\tstruct ewah_bitmap *fsmonitor_dirty_bitmap;\n> };\n>\n> extern struct index_state the_index;\n> @@ -766,6 +770,7 @@ extern int precomposed_unicode;\n> extern int protect_hfs;\n> extern int protect_ntfs;\n> extern int git_db_env, git_index_env, git_graft_env, git_common_dir_env;\n> +extern int core_fsmonitor;\n>\n> /*\n> * Include broken refs in all ref iterations, which will\n> diff --git a/config.c b/config.c\n> index bb4d735701..1a8108636d 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1232,6 +1232,11 @@ static int git_default_core_config(const char *var, const char *value)\n> \t\treturn 0;\n> \t}\n>\n> +\tif (!strcmp(var, \"core.fsmonitor\")) {\n> +\t\tcore_fsmonitor = git_config_bool(var, value);\n> +\t\treturn 0;\n> +\t}\n> +\n> \t/* Add other config variables here and to Documentation/config.txt. */\n> \treturn 0;\n> }\n> diff --git a/dir.c b/dir.c\n> index 1b5558fdf9..da428489e2 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1652,6 +1652,18 @@ static int valid_cached_dir(struct dir_struct *dir,\n> \tif (!untracked)\n> \t\treturn 0;\n>\n> +\trefresh_by_fsmonitor(&the_index);\n> +\tif (dir->untracked->use_fsmonitor) {\n> +\t\t/*\n> +\t\t * With fsmonitor, we can trust the untracked cache's\n> +\t\t * valid field.\n> +\t\t */\n> +\t\tif (untracked->valid)\n> +\t\t\tgoto skip_stat;\n> +\t\telse\n> +\t\t\tinvalidate_directory(dir->untracked, untracked);\n> +\t}\n> +\n> \tif (stat(path->len ? path->buf : \".\", &st)) {\n> \t\tinvalidate_directory(dir->untracked, untracked);\n> \t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n> @@ -1665,6 +1677,7 @@ static int valid_cached_dir(struct dir_struct *dir,\n> \t\treturn 0;\n> \t}\n>\n> +skip_stat:\n> \tif (untracked->check_only != !!check_only) {\n> \t\tinvalidate_directory(dir->untracked, untracked);\n> \t\treturn 0;\n> diff --git a/dir.h b/dir.h\n> index 9e387551bd..ff6a00abcc 100644\n> --- a/dir.h\n> +++ b/dir.h\n> @@ -139,6 +139,8 @@ struct untracked_cache {\n> \tint gitignore_invalidated;\n> \tint dir_invalidated;\n> \tint dir_opened;\n> +\t/* fsmonitor invalidation data */\n> +\tunsigned int use_fsmonitor : 1;\n> };\n>\n> struct dir_struct {\n> diff --git a/entry.c b/entry.c\n> index d2b512da90..c2d3c1079c 100644\n> --- a/entry.c\n> +++ b/entry.c\n> @@ -221,6 +221,7 @@ static int write_entry(struct cache_entry *ce,\n> \t\t\tlstat(ce->name, &st);\n> \t\tfill_stat_cache_info(ce, &st);\n> \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n> +\t\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n> \t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n> \t}\n> \treturn 0;\n> diff --git a/environment.c b/environment.c\n> index 560408953c..1afabbae8c 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -64,6 +64,7 @@ int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n> unsigned long pack_size_limit_cfg;\n> enum hide_dotfiles_type hide_dotfiles = HIDE_DOTFILES_DOTGITONLY;\n> enum log_refs_config log_all_ref_updates = LOG_REFS_UNSET;\n> +int core_fsmonitor;\n>\n> #ifndef PROTECT_HFS_DEFAULT\n> #define PROTECT_HFS_DEFAULT 0\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> new file mode 100644\n> index 0000000000..6356dc795e\n> --- /dev/null\n> +++ b/fsmonitor.c\n> @@ -0,0 +1,231 @@\n> +#include \"cache.h\"\n> +#include \"dir.h\"\n> +#include \"ewah/ewok.h\"\n> +#include \"run-command.h\"\n> +#include \"strbuf.h\"\n> +#include \"fsmonitor.h\"\n> +\n> +static struct untracked_cache_dir *find_untracked_cache_dir(\n> +\tstruct untracked_cache *uc, struct untracked_cache_dir *ucd,\n> +\tconst char *name)\n> +{\n> +\tconst char *end;\n> +\tstruct untracked_cache_dir *dir = ucd;\n> +\n> +\tif (!*name)\n> +\t\treturn dir;\n> +\n> +\tend = strchr(name, '/');\n> +\tif (end) {\n> +\t\tdir = lookup_untracked(uc, ucd, name, end - name);\n> +\t\tif (dir)\n> +\t\t\treturn find_untracked_cache_dir(uc, dir, end + 1);\n> +\t}\n> +\n> +\treturn dir;\n> +}\n> +\n> +/* This function will be passed to ewah_each_bit() */\n> +static void mark_no_fsmonitor(size_t pos, void *is)\n> +{\n> +\tstruct index_state *istate = is;\n> +\tstruct untracked_cache_dir *dir;\n> +\tstruct cache_entry *ce = istate->cache[pos];\n> +\n> +\tassert(pos < istate->cache_nr);\n> +\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n> +\n> +\tif (!istate->untracked || !istate->untracked->root)\n> +\t\treturn;\n> +\n> +\tdir = find_untracked_cache_dir(istate->untracked, istate->untracked->root, ce->name);\n> +\tif (dir)\n> +\t\tdir->valid = 0;\n> +}\n> +\n> +int read_fsmonitor_extension(struct index_state *istate, const void *data,\n> +\tunsigned long sz)\n> +{\n> +\tconst char *index = data;\n> +\tuint32_t hdr_version;\n> +\tuint32_t ewah_size;\n> +\tint ret;\n> +\n> +\tif (sz < sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t))\n> +\t\treturn error(\"corrupt fsmonitor extension (too short)\");\n> +\n> +\thdr_version = get_be32(index);\n> +\tindex += sizeof(uint32_t);\n> +\tif (hdr_version != 1)\n> +\t\treturn error(\"bad fsmonitor version %d\", hdr_version);\n> +\n> +\tistate->fsmonitor_last_update = (time_t)get_be64(index);\n> +\tindex += sizeof(uint64_t);\n> +\n> +\tewah_size = get_be32(index);\n> +\tindex += sizeof(uint32_t);\n> +\n> +\tistate->fsmonitor_dirty_bitmap = ewah_new();\n> +\tret = ewah_read_mmap(istate->fsmonitor_dirty_bitmap, index, ewah_size);\n> +\tif (ret != ewah_size) {\n> +\t\tewah_free(istate->fsmonitor_dirty_bitmap);\n> +\t\tistate->fsmonitor_dirty_bitmap = NULL;\n> +\t\treturn error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n> +{\n> +\tuint32_t hdr_version;\n> +\tuint64_t tm;\n> +\tstruct ewah_bitmap *bitmap;\n> +\tint i;\n> +\tuint32_t ewah_start;\n> +\tuint32_t ewah_size = 0;\n> +\tint fixup = 0;\n> +\n> +\thdr_version = htonl(1);\n> +\tstrbuf_add(sb, &hdr_version, sizeof(uint32_t));\n> +\n> +\ttm = htonll((uint64_t)istate->fsmonitor_last_update);\n> +\tstrbuf_add(sb, &tm, sizeof(uint64_t));\n> +\tfixup = sb->len;\n> +\tstrbuf_add(sb, &ewah_size, sizeof(uint32_t)); /* we'll fix this up later */\n> +\n> +\tewah_start = sb->len;\n> +\tbitmap = ewah_new();\n> +\tfor (i = 0; i < istate->cache_nr; i++)\n> +\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_DIRTY)\n> +\t\t\tewah_set(bitmap, i);\n> +\tewah_serialize_strbuf(bitmap, sb);\n> +\tewah_free(bitmap);\n> +\n> +\t/* fix up size field */\n> +\tewah_size = htonl(sb->len - ewah_start);\n> +\tmemcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t));\n> +}\n> +\n> +static void mark_file_dirty(struct index_state *istate, const char *name)\n> +{\n> +\tstruct untracked_cache_dir *dir;\n> +\tint pos;\n> +\n> +\t/* find it in the index and mark that entry as dirty */\n> +\tpos = index_name_pos(istate, name, strlen(name));\n> +\tif (pos >= 0)\n> +\t\tistate->cache[pos]->ce_flags |= CE_FSMONITOR_DIRTY;\n> +\n> +\t/*\n> +\t * Find the corresponding directory in the untracked cache\n> +\t * and mark it as invalid\n> +\t */\n> +\tif (!istate->untracked || !istate->untracked->root)\n> +\t\treturn;\n> +\n> +\tdir = find_untracked_cache_dir(istate->untracked, istate->untracked->root, name);\n> +\tif (dir)\n> +\t\tdir->valid = 0;\n> +}\n> +\n> +/*\n> + * Call the query-fsmonitor hook passing the time of the last saved results.\n> + */\n> +static int query_fsmonitor(time_t last_update, struct strbuf *query_result)\n> +{\n> +\tstruct child_process cp = CHILD_PROCESS_INIT;\n> +\tchar date[64];\n> +\tconst char *argv[3];\n> +\n> +\tif (!(argv[0] = find_hook(\"query-fsmonitor\")))\n> +\t\treturn -1;\n> +\n> +\tsnprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update);\n> +\targv[1] = date;\n> +\targv[2] = NULL;\n> +\tcp.argv = argv;\n> +\tcp.out = -1;\n> +\n> +\treturn capture_command(&cp, query_result, 1024);\n> +}\n> +\n> +void process_fsmonitor_extension(struct index_state *istate)\n> +{\n> +\tif (!istate->fsmonitor_dirty_bitmap)\n> +\t\treturn;\n> +\n> +\tewah_each_bit(istate->fsmonitor_dirty_bitmap, mark_no_fsmonitor, istate);\n> +\tewah_free(istate->fsmonitor_dirty_bitmap);\n> +\tistate->fsmonitor_dirty_bitmap = NULL;\n> +}\n> +\n> +void refresh_by_fsmonitor(struct index_state *istate)\n> +{\n> +\tstatic has_run_once = FALSE;\n> +\tstruct strbuf query_result = STRBUF_INIT;\n> +\tint query_success = 0;\n> +\tsize_t bol = 0; /* beginning of line */\n> +\ttime_t last_update;\n> +\tchar *buf, *entry;\n> +\tint i;\n> +\n> +\tif (!core_fsmonitor || has_run_once)\n> +\t\treturn;\n> +\thas_run_once = TRUE;\n> +\n> +\t/*\n> +\t * This could be racy so save the date/time now and the hook\n> +\t * should be inclusive to ensure we don't miss potential changes.\n> +\t */\n> +\tlast_update = time(NULL);\n> +\n> +\t/* If we have a last update time, call query-monitor for the set of changes since that time */\n> +\tif (istate->fsmonitor_last_update) {\n> +\t\tquery_success = !query_fsmonitor(istate->fsmonitor_last_update, &query_result);\n> +\t}\n> +\n> +\tif (query_success) {\n> +\t\t/* Mark all entries returned by the monitor as dirty */\n> +\t\tbuf = entry = query_result.buf;\n> +\t\tfor (i = 0; i < query_result.len; i++) {\n> +\t\t\tif (buf[i] != '\\0')\n> +\t\t\t\tcontinue;\n> +\t\t\tmark_file_dirty(istate, buf + bol);\n> +\t\t\tbol = i + 1;\n> +\t\t}\n> +\t\tif (bol < query_result.len)\n> +\t\t\tmark_file_dirty(istate, buf + bol);\n> +\n> +\t\t/* Mark all clean entries up-to-date */\n> +\t\tfor (i = 0; i < istate->cache_nr; i++) {\n> +\t\t\tstruct cache_entry *ce = istate->cache[i];\n> +\t\t\tif (ce_stage(ce) || (ce->ce_flags & CE_FSMONITOR_DIRTY))\n> +\t\t\t\tcontinue;\n> +\t\t\tce_mark_uptodate(ce);\n> +\t\t}\n> +\n> +\t\t/*\n> +\t\t * Now that we've marked the invalid entries in the\n> +\t\t * untracked-cache itself, we can mark the untracked cache for\n> +\t\t * fsmonitor usage.\n> +\t\t */\n> +\t\tif (istate->untracked) {\n> +\t\t\tistate->untracked->use_fsmonitor = 1;\n> +\t\t}\n> +\t}\n> +\telse {\n> +\t\t/* if we can't update the cache, fall back to checking them all */\n> +\t\tfor (i = 0; i < istate->cache_nr; i++)\n> +\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_DIRTY;\n> +\n> +\t\t/* mark the untracked cache as unusable for fsmonitor */\n> +\t\tif (istate->untracked)\n> +\t\t\tistate->untracked->use_fsmonitor = 0;\n> +\t}\n> +\tstrbuf_release(&query_result);\n> +\n> +\t/* Now that we've updated istate, save the last_update time */\n> +\tistate->fsmonitor_last_update = last_update;\n> +\tistate->cache_changed |= FSMONITOR_CHANGED;\n> +}\n> diff --git a/fsmonitor.h b/fsmonitor.h\n> new file mode 100644\n> index 0000000000..57b061688f\n> --- /dev/null\n> +++ b/fsmonitor.h\n> @@ -0,0 +1,9 @@\n> +#ifndef FSMONITOR_H\n> +#define FSMONITOR_H\n> +\n> +int read_fsmonitor_extension(struct index_state *istate, const void *data, unsigned long sz);\n> +void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate);\n> +void process_fsmonitor_extension(struct index_state *istate);\n> +void refresh_by_fsmonitor(struct index_state *istate);\n> +\n> +#endif\n> diff --git a/read-cache.c b/read-cache.c\n> index 3339de8124..5b52f08db2 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -18,6 +18,7 @@\n> #include \"varint.h\"\n> #include \"split-index.h\"\n> #include \"utf8.h\"\n> +#include \"fsmonitor.h\"\n>\n> /* Mask for the name length in ce_flags in the on-disk index */\n>\n> @@ -37,11 +38,12 @@\n> #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n> #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n> #define CACHE_EXT_UNTRACKED 0x554E5452\t /* \"UNTR\" */\n> +#define CACHE_EXT_FSMONITOR 0x46534D4E\t /* \"FSMN\" */\n>\n> /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n> #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n> \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n> -\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n> +\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n>\n> struct index_state the_index;\n> static const char *alternate_index_output;\n> @@ -61,6 +63,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n> \tfree(old);\n> \tset_index_entry(istate, nr, ce);\n> \tce->ce_flags |= CE_UPDATE_IN_BASE;\n> +\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n> \tistate->cache_changed |= CE_ENTRY_CHANGED;\n> }\n>\n> @@ -777,6 +780,7 @@ int chmod_index_entry(struct index_state *istate, struct cache_entry *ce,\n> \t}\n> \tcache_tree_invalidate_path(istate, ce->name);\n> \tce->ce_flags |= CE_UPDATE_IN_BASE;\n> +\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n> \tistate->cache_changed |= CE_ENTRY_CHANGED;\n>\n> \treturn 0;\n> @@ -1344,6 +1348,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> \tconst char *added_fmt;\n> \tconst char *unmerged_fmt;\n>\n> +\trefresh_by_fsmonitor(istate);\n> +\n> \tmodified_fmt = (in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\");\n> \tdeleted_fmt = (in_porcelain ? \"D\\t%s\\n\" : \"%s: needs update\\n\");\n> \ttypechange_fmt = (in_porcelain ? \"T\\t%s\\n\" : \"%s needs update\\n\");\n> @@ -1380,8 +1386,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> \t\t\tcontinue;\n>\n> \t\tnew = refresh_cache_ent(istate, ce, options, &cache_errno, &changed);\n> -\t\tif (new == ce)\n> +\t\tif (new == ce) {\n> +\t\t\tce->ce_flags &= ~CE_FSMONITOR_DIRTY;\n> \t\t\tcontinue;\n> +\t\t}\n> +\n> \t\tif (!new) {\n> \t\t\tconst char *fmt;\n>\n> @@ -1391,6 +1400,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> \t\t\t\t */\n> \t\t\t\tce->ce_flags &= ~CE_VALID;\n> \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n> +\t\t\t\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n> \t\t\t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n> \t\t\t}\n> \t\t\tif (quiet)\n> @@ -1549,6 +1559,9 @@ static int read_index_extension(struct index_state *istate,\n> \tcase CACHE_EXT_UNTRACKED:\n> \t\tistate->untracked = read_untracked_extension(data, sz);\n> \t\tbreak;\n> +\tcase CACHE_EXT_FSMONITOR:\n> +\t\tread_fsmonitor_extension(istate, data, sz);\n> +\t\tbreak;\n> \tdefault:\n> \t\tif (*ext < 'A' || 'Z' < *ext)\n> \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n> @@ -1721,6 +1734,7 @@ static void post_read_index_from(struct index_state *istate)\n> \tcheck_ce_order(istate);\n> \ttweak_untracked_cache(istate);\n> \ttweak_split_index(istate);\n> +\tprocess_fsmonitor_extension(istate);\n> }\n>\n> /* remember to discard_cache() before reading a different cache! */\n> @@ -2300,6 +2314,16 @@ static int do_write_index(struct index_state *istate, int newfd,\n> \t\tif (err)\n> \t\t\treturn -1;\n> \t}\n> +\tif (!strip_extensions && istate->fsmonitor_last_update) {\n> +\t\tstruct strbuf sb = STRBUF_INIT;\n> +\n> +\t\twrite_fsmonitor_extension(&sb, istate);\n> +\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_FSMONITOR, sb.len) < 0\n> +\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n> +\t\tstrbuf_release(&sb);\n> +\t\tif (err)\n> +\t\t\treturn -1;\n> +\t}\n>\n> \tif (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st))\n> \t\treturn -1;\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index aa15111fef..259e6960b9 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -412,6 +412,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n> \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n> \tif (was_skip_worktree != ce_skip_worktree(ce)) {\n> \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n> +\t\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n> \t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n> \t}\n>\n>\n"},{"id":"320308","messageId":"xmqqtw4fstd7.fsf@gitster.mtv.corp.google.com","threadId":"45997","inReplyTo":"41d6939f-0095-cfc5-cde5-2ebdd74a09b2@gmail.com","subject":"Re: [PATCH v2 3/6] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-20T10:41:24Z","receivedAt":"2017-05-20T10:41:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> After sending this, I noticed that using a different compiler\n> generated a couple of warnings I should fix. I'm assuming I'll need\n> another re-roll but if not, here are the changes that will be squashed\n> in.\n>\n> Ben\n\nThanks, in addition, I am missing the definition of TRUE and FALSE.\n"},{"id":"320310","messageId":"xmqqlgprsr6p.fsf@gitster.mtv.corp.google.com","threadId":"45997","inReplyTo":"20170518201333.13088-6-benpeart@microsoft.com","subject":"Re: [PATCH v2 5/6] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-20T11:28:30Z","receivedAt":"2017-05-20T11:28:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\n> index 706091a569..f7b4b4a844 100644\n> --- a/Documentation/githooks.txt\n> +++ b/Documentation/githooks.txt\n> @@ -448,6 +448,29 @@ The commits are guaranteed to be listed in the order that they were\n> processed by rebase.\n> \n> \n> +[[query-fsmonitor]]\n> +query-fsmonitor\n> +~~~~~~~~~~~~\n\nThis underline is short by 3 '~' for the string it underlines.\n"},{"id":"320314","messageId":"CACBZZX5URAeA+=12ezW-oDGnkdAqvQqV7it=HBaYCKUdx0p_XA@mail.gmail.com","threadId":"45997","inReplyTo":"20170518201333.13088-6-benpeart@microsoft.com","subject":"Re: [PATCH v2 5/6] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-20T12:10:27Z","receivedAt":"2017-05-20T12:10:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Thu, May 18, 2017 at 10:13 PM, Ben Peart wrote:\n> This includes the core.fsmonitor setting, the query-fsmonitor hook,\n> and the fsmonitor index extension.\n>\n> Signed-off-by: Ben Peart \n> ---\n> Documentation/config.txt | 7 +++++++\n> Documentation/githooks.txt | 23 +++++++++++++++++++++++\n> Documentation/technical/index-format.txt | 18 ++++++++++++++++++\n> 3 files changed, 48 insertions(+)\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 96e9cf8b73..4ffbf0d4c2 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -389,6 +389,13 @@ core.protectNTFS::\n> 8.3 \"short\" names.\n> Defaults to `true` on Windows, and `false` elsewhere.\n>\n> +core.fsmonitor::\n> + If set to true, call the query-fsmonitor hook proc which will\n> + identify all files that may have had changes since the last\n> + request. This information is used to speed up operations like\n> + 'git commit' and 'git status' by limiting what git must scan to\n> + detect changes.\n> +\n> core.trustctime::\n> If false, the ctime differences between the index and the\n> working tree are ignored; useful when the inode change time\n> diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\n> index 706091a569..f7b4b4a844 100644\n> --- a/Documentation/githooks.txt\n> +++ b/Documentation/githooks.txt\n> @@ -448,6 +448,29 @@ The commits are guaranteed to be listed in the order that they were\n> processed by rebase.\n>\n>\n> +[[query-fsmonitor]]\n> +query-fsmonitor\n> +~~~~~~~~~~~~\n> +\n> +This hook is invoked when the configuration option core.fsmonitor is\n> +set and git needs to identify changed or untracked files. It takes\n> +a single argument which is the time in elapsed seconds since midnight,\n> +January 1, 1970.\n> +\n> +The hook should output to stdout the list of all files in the working\n> +directory that may have changed since the requested time. The logic\n> +should be inclusive so that it does not miss any potential changes.\n> +The paths should be relative to the root of the working directory\n> +and be separated by a single NUL.\n> +\n> +Git will limit what files it checks for changes as well as which\n> +directories are checked for untracked files based on the path names\n> +given.\n> +\n> +The exit status determines whether git will use the data from the\n> +hook to limit its search. On error, it will fall back to verifying\n> +all files and folders.\n> +\n> GIT\n> ---\n> Part of the linkgit:git[1] suite\n> diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\n> index ade0b0c445..b002d23c05 100644\n> --- a/Documentation/technical/index-format.txt\n> +++ b/Documentation/technical/index-format.txt\n> @@ -295,3 +295,21 @@ The remaining data of each directory block is grouped by type:\n> in the previous ewah bitmap.\n>\n> - One NUL.\n> +\n> +== File System Monitor cache\n> +\n> + The file system monitor cache tracks files for which the query-fsmonitor\n> + hook has told us about changes. The signature for this extension is\n> + { 'F', 'S', 'M', 'N' }.\n> +\n> + The extension starts with\n> +\n> + - 32-bit version number: the current supported version is 1.\n> +\n> + - 64-bit time: the extension data reflects all changes through the given\n> + time which is stored as the seconds elapsed since midnight, January 1, 1970.\n> +\n> + - 32-bit bitmap size: the size of the CE_FSMONITOR_DIRTY bitmap.\n> +\n> + - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n> + is CE_FSMONITOR_DIRTY.\n\nWe already have a uint64_t in one place in the codebase (getnanotime)\nwhich uses a 64 bit time for nanosecond accuracy, and numerous\nfilesystems already support nanosecond timestamps (ext4, that new\nApple thingy...).\n\nI don't know if any of the inotify/fsmonitor APIs support that yet,\nbut it seems inevitable that that'll be added if not, in some\npathological cases we can have a lot of files modified in 1 second, so\nusing nanosecond accuracy means there'll be a lot less data to\nconsider in some cases.\n\nIt does mean this'll only work until the year ~2500, but that seems\nlike an acceptable trade-off.\n"},{"id":"320317","messageId":"da2523fc-e6ef-ff26-0449-83f8d46bed84@web.de","threadId":"45997","inReplyTo":"20170518201333.13088-5-benpeart@microsoft.com","subject":"Re: [PATCH v2 4/6] fsmonitor: add test cases for fsmonitor extension","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-05-20T16:55:13Z","receivedAt":"2017-05-20T16:55:50Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"> diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\n> new file mode 100755\n> index 0000000000..d3cffc758f\n> --- /dev/null\n> +++ b/t/t7519-status-fsmonitor.sh\n> @@ -0,0 +1,153 @@\n> +#!/bin/sh\n> +\n> +test_description='git status with file system watcher'\n> +\n> +. ./test-lib.sh\n> +\n> +clean_repo () {\n> +\tgit reset --hard HEAD\n> +\tgit clean -fd\n> +\trm marker -f\n\nThis Works under Linux, but not e.g. Mac OS X. Should be\n\nrm -f marker\n\n> +}\n\n"},{"id":"320448","messageId":"5ab333a4-c3cd-1cb5-ba3e-6b08fa14c9e7@gmail.com","threadId":"45997","inReplyTo":"CACBZZX5URAeA+=12ezW-oDGnkdAqvQqV7it=HBaYCKUdx0p_XA@mail.gmail.com","subject":"Re: [PATCH v2 5/6] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-22T16:18:20Z","receivedAt":"2017-05-22T16:18:26Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 5/20/2017 8:10 AM, Ævar Arnfjörð Bjarmason wrote:\n>> +== File System Monitor cache\n>> +\n>> + The file system monitor cache tracks files for which the query-fsmonitor\n>> + hook has told us about changes. The signature for this extension is\n>> + { 'F', 'S', 'M', 'N' }.\n>> +\n>> + The extension starts with\n>> +\n>> + - 32-bit version number: the current supported version is 1.\n>> +\n>> + - 64-bit time: the extension data reflects all changes through the given\n>> + time which is stored as the seconds elapsed since midnight, January 1, 1970.\n>> +\n>> + - 32-bit bitmap size: the size of the CE_FSMONITOR_DIRTY bitmap.\n>> +\n>> + - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n>> + is CE_FSMONITOR_DIRTY.\n>\n> We already have a uint64_t in one place in the codebase (getnanotime)\n> which uses a 64 bit time for nanosecond accuracy, and numerous\n> filesystems already support nanosecond timestamps (ext4, that new\n> Apple thingy...).\n>\n> I don't know if any of the inotify/fsmonitor APIs support that yet,\n> but it seems inevitable that that'll be added if not, in some\n> pathological cases we can have a lot of files modified in 1 second, so\n> using nanosecond accuracy means there'll be a lot less data to\n> consider in some cases.\n>\n> It does mean this'll only work until the year ~2500, but that seems\n> like an acceptable trade-off.\n>\n\nI really don't think nano-second resolution is needed in this case for a \nfew reasons.\n\nThe number of files that can change within a given second is limited by \nthe IO throughput of the underlying device. Even assuming a very fast \ndevice and very small files and changes, this won't be that many files.\n\nWithout this patch, git would have scanned all those files every time. \nWith this patch, git will only scan those files a 2nd time that are \nmodified in the same second that it did the first scan *that came before \nthe first scan started* (the \"lots of files modified\" section in the 1 \nsecond timeline below).\n\n|------------------------- one second ---------------------|\n|-lots of files modified - git status - more file modified-|\n\nYes, some duplicate status checks can be made but its still a \nsignificant win in any reasonable scenario. Especially when you consider \nthat it is pretty unusual to do git status/add/commit calls in the \nmiddle of making lots of changes to files.\n\nIn addition, the backing file system monitor (Watchman) supports number \nof seconds since the unix epoch (unix time_t style). This means any \nsupport of nano seconds by git is academic until someone provides a file \nsystem watcher that does support nano second granularity.\n\nFinally, the fsmonitor index extension is versioned so that we can \nseamlessly upgrade to nano second resolution later if we desire.\n"},{"id":"320453","messageId":"CACBZZX6LENwuVuTyU-RetaXz8jZtUp1dv+gmQQ281sPx1czPnA@mail.gmail.com","threadId":"45997","inReplyTo":"5ab333a4-c3cd-1cb5-ba3e-6b08fa14c9e7@gmail.com","subject":"Re: [PATCH v2 5/6] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-22T17:28:12Z","receivedAt":"2017-05-22T17:28:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Mon, May 22, 2017 at 6:18 PM, Ben Peart wrote:\n> On 5/20/2017 8:10 AM, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> +== File System Monitor cache\n>>> +\n>>> + The file system monitor cache tracks files for which the\n>>> query-fsmonitor\n>>> + hook has told us about changes. The signature for this extension is\n>>> + { 'F', 'S', 'M', 'N' }.\n>>> +\n>>> + The extension starts with\n>>> +\n>>> + - 32-bit version number: the current supported version is 1.\n>>> +\n>>> + - 64-bit time: the extension data reflects all changes through the\n>>> given\n>>> + time which is stored as the seconds elapsed since midnight,\n>>> January 1, 1970.\n>>> +\n>>> + - 32-bit bitmap size: the size of the CE_FSMONITOR_DIRTY bitmap.\n>>> +\n>>> + - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n>>> + is CE_FSMONITOR_DIRTY.\n>>\n>>\n>> We already have a uint64_t in one place in the codebase (getnanotime)\n>> which uses a 64 bit time for nanosecond accuracy, and numerous\n>> filesystems already support nanosecond timestamps (ext4, that new\n>> Apple thingy...).\n>>\n>> I don't know if any of the inotify/fsmonitor APIs support that yet,\n>> but it seems inevitable that that'll be added if not, in some\n>> pathological cases we can have a lot of files modified in 1 second, so\n>> using nanosecond accuracy means there'll be a lot less data to\n>> consider in some cases.\n>>\n>> It does mean this'll only work until the year ~2500, but that seems\n>> like an acceptable trade-off.\n>>\n>\n> I really don't think nano-second resolution is needed in this case for a few\n> reasons.\n>\n> The number of files that can change within a given second is limited by the\n> IO throughput of the underlying device. Even assuming a very fast device and\n> very small files and changes, this won't be that many files.\n>\n> Without this patch, git would have scanned all those files every time. With\n> this patch, git will only scan those files a 2nd time that are modified in\n> the same second that it did the first scan *that came before the first scan\n> started* (the \"lots of files modified\" section in the 1 second timeline\n> below).\n>\n> |------------------------- one second ---------------------|\n> |-lots of files modified - git status - more file modified-|\n>\n> Yes, some duplicate status checks can be made but its still a significant\n> win in any reasonable scenario. Especially when you consider that it is\n> pretty unusual to do git status/add/commit calls in the middle of making\n> lots of changes to files.\n\nI understand that we get most of the wins either way.\n\nI'm just wondering why not make it nanosecond-resolution now from the\nbeginning since that's where major filesystems are heading already,\nand changing stuff like this can be a hassle once it's initially out\nthere, whereas just dividing by 10^9 for APIs that need seconds from\nthe beginning is easy & future-proof.\n\nThere are some uses of git where this would probably matter in practice.\n\nE.g. consider a git-annex backed fileserver using ext4, currently\ngit-annex comes with its own FS watcher as a daemon invoked via `git\nannex watch` to constantly add new files without killing performance\nvia a status/add in a loop, with this a daemon like that could just\nrun status/add in a loop, but on a busy repo the 1s interval size\nmight start to matter as you're constantly inspecting larger\nintervals.\n\nMore importantly though, I can't think of any case where having it in\nnanoseconds to begin with would do any harm.\n\n> In addition, the backing file system monitor (Watchman) supports number of\n> seconds since the unix epoch (unix time_t style). This means any support of\n> nano seconds by git is academic until someone provides a file system watcher\n> that does support nano second granularity.\n\nI haven't used watchman for anything non-trivial, but the\ndocumentation for the query API you seem to be using says you can\nrequest a {ctime,mtime}_ns field:\n\nhttps://github.com/facebook/watchman/blob/master/website/_docs/cmd.query.markdown#user-content-available-fields\n\nAnd isn't this the documentation for the \"since\" query you're using,\nsaying you can specify any arbitrary fs timing field, such as a _ns\ntime field:\n\nhttps://github.com/facebook/watchman/blob/master/website/_docs/expr.since.md\n\n?\n\n> Finally, the fsmonitor index extension is versioned so that we can\n> seamlessly upgrade to nano second resolution later if we desire.\n\nAside from my nanosecond bikeshedding, let's say we change the\nsemantics of any of this in the future: The index has the version, but\nthere's one argument passed to the hook without a version. Is the hook\nexpected to potentially be reading the version from the index to make\nsense of whether (in this case) the argument is a mtime or mtime_ns?\n\nWouldn't this make more sense than that on top, i.e. pass the version\nto the hook, untested (and probably whines about the sprintf\nformat...):\n\n$ git diff -U1\ndiff --git a/cache.h b/cache.h\nindex 6eafd34fff..3c63f179f8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -346,2 +346,3 @@ struct index_state {\n struct untracked_cache *untracked;\n+ uint32_t fsmonitor_version;\n time_t fsmonitor_last_update;\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex f5a9f818e8..7236b538ac 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -60,2 +60,4 @@ int read_fsmonitor_extension(struct index_state\n*istate, const void *data,\n return error(\"bad fsmonitor version %d\", hdr_version);\n+ else\n+ istate->fsmonitor_version = hdr_version;\n\n@@ -137,2 +139,3 @@ static int query_fsmonitor(time_t last_update,\nstruct strbuf *query_result)\n struct child_process cp = CHILD_PROCESS_INIT;\n+ char version[1];\n char date[64];\n@@ -143,2 +146,3 @@ static int query_fsmonitor(time_t last_update,\nstruct strbuf *query_result)\n\n+ snprintf(version, sizeof(version), \"%d\", istate->fsmonitor_version);\n snprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update)\n"},{"id":"320658","messageId":"CAP8UFD20gvTZqCOcpd1iozNDHOZR1sUKRwi062wL52FNHWNC3w@mail.gmail.com","threadId":"45997","inReplyTo":"20170518201333.13088-1-benpeart@microsoft.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-05-24T10:54:48Z","receivedAt":"2017-05-24T10:57:25Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"> Design\n> ~~~~~~\n>\n> A new git hook (query-fsmonitor) must exist and be enabled\n> (core.fsmonitor=true) that takes a time_t formatted as a string and\n> outputs to stdout all files that have been modified since the requested\n> time.\n\nIs there a reason why there is a new hook, instead of a\n\"core.fsmonitorquery\" config option to which you could pass whatever\ncommand line with options?\n\n> A new 'fsmonitor' index extension has been added to store the time the\n> fsmonitor hook was last queried and a ewah bitmap of the current\n> 'fsmonitor-dirty' files. Unmarked entries are 'fsmonitor-clean', marked\n> entries are 'fsmonitor-dirty.'\n>\n> As needed, git will call the query-fsmonitor hook proc for the set of\n> changes since the index was last updated. Git then uses this set of\n> files along with the list saved in the fsmonitor index extension to flag\n> the potentially dirty index and untracked cache entries.\n\nSo this can work only if \"core.untrackedCache\" is set to true?\n\nThanks for working on this,\nChristian.\n"},{"id":"320659","messageId":"CAP8UFD0Q6kpsOBWWtt=2OV0MZvKayBQxVk9H1EfTCOuUaB-onw@mail.gmail.com","threadId":"45997","inReplyTo":"20170518201333.13088-4-benpeart@microsoft.com","subject":"Re: [PATCH v2 3/6] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-05-24T12:30:00Z","receivedAt":"2017-05-24T12:31:30Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, May 18, 2017 at 10:13 PM, Ben Peart wrote:\n> When the index is read from disk, the query-fsmonitor index extension is\n> used to flag the last known potentially dirty index and untracked cach\n\ns/cach/cache/\n\n> entries.\n\n[...]\n\n> diff --git a/cache.h b/cache.h\n> index 188811920c..36423c77cc 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -201,6 +201,7 @@ struct cache_entry {\n> #define CE_ADDED (1 << 19)\n>\n> #define CE_HASHED (1 << 20)\n> +#define CE_FSMONITOR_DIRTY (1 << 21)\n\nIt looks like the (1 << 21) value was used before (as CE_UNHASHED) and\nwas removed in:\n\n419a597f64 (name-hash.c: remove cache entries instead of marking them\nCE_UNHASHED, 2013-11-14)\n\nI wondered if using this value again could make old and new versions\nof git interact badly, but it looks like these are in memory only\nflags, so it should be ok.\n\n> #define CE_WT_REMOVE (1 << 22) /* remove in work directory */\n> #define CE_CONFLICTED (1 << 23)\n>\n> struct split_index;\n> struct untracked_cache;\n> @@ -342,6 +344,8 @@ struct index_state {\n> struct hashmap dir_hash;\n> unsigned char sha1[20];\n> struct untracked_cache *untracked;\n> + time_t fsmonitor_last_update;\n> + struct ewah_bitmap *fsmonitor_dirty_bitmap;\n\nMaybe you could remove \"_bitmap\" at the end of the name.\n\n> };\n\n\n> diff --git a/dir.c b/dir.c\n> index 1b5558fdf9..da428489e2 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1652,6 +1652,18 @@ static int valid_cached_dir(struct dir_struct *dir,\n> if (!untracked)\n> return 0;\n>\n> + refresh_by_fsmonitor(&the_index);\n> + if (dir->untracked->use_fsmonitor) {\n> + /*\n> + * With fsmonitor, we can trust the untracked cache's\n> + * valid field.\n> + */\n> + if (untracked->valid)\n> + goto skip_stat;\n\nMaybe you could avoid this goto using a valid_cached_dir_after_stat()\nfunction that would do what is after the \"skip_stat:\" label below?\n\n> + else\n> + invalidate_directory(dir->untracked, untracked);\n> + }\n> +\n> if (stat(path->len ? path->buf : \".\", &st)) {\n> invalidate_directory(dir->untracked, untracked);\n> memset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n> @@ -1665,6 +1677,7 @@ static int valid_cached_dir(struct dir_struct *dir,\n> return 0;\n> }\n>\n> +skip_stat:\n> if (untracked->check_only != !!check_only) {\n> invalidate_directory(dir->untracked, untracked);\n> return 0;\n\n[...]\n\n> +void refresh_by_fsmonitor(struct index_state *istate)\n> +{\n> + static has_run_once = FALSE;\n> + struct strbuf query_result = STRBUF_INIT;\n> + int query_success = 0;\n> + size_t bol = 0; /* beginning of line */\n> + time_t last_update;\n> + char *buf, *entry;\n> + int i;\n> +\n> + if (!core_fsmonitor || has_run_once)\n> + return;\n> + has_run_once = TRUE;\n> +\n> + /*\n> + * This could be racy so save the date/time now and the hook\n> + * should be inclusive to ensure we don't miss potential changes.\n> + */\n> + last_update = time(NULL);\n> +\n> + /* If we have a last update time, call query-monitor for the set of changes since that time */\n> + if (istate->fsmonitor_last_update) {\n> + query_success = !query_fsmonitor(istate->fsmonitor_last_update, &query_result);\n> + }\n\nBraces can be removed.\n\n> + if (query_success) {\n> + /* Mark all entries returned by the monitor as dirty */\n> + buf = entry = query_result.buf;\n> + for (i = 0; i < query_result.len; i++) {\n> + if (buf[i] != '\\0')\n> + continue;\n> + mark_file_dirty(istate, buf + bol);\n> + bol = i + 1;\n> + }\n> + if (bol < query_result.len)\n> + mark_file_dirty(istate, buf + bol);\n> +\n> + /* Mark all clean entries up-to-date */\n> + for (i = 0; i < istate->cache_nr; i++) {\n> + struct cache_entry *ce = istate->cache[i];\n> + if (ce_stage(ce) || (ce->ce_flags & CE_FSMONITOR_DIRTY))\n> + continue;\n> + ce_mark_uptodate(ce);\n> + }\n> +\n> + /*\n> + * Now that we've marked the invalid entries in the\n> + * untracked-cache itself, we can mark the untracked cache for\n> + * fsmonitor usage.\n> + */\n> + if (istate->untracked) {\n> + istate->untracked->use_fsmonitor = 1;\n> + }\n\nBraces can be removed.\n\n> + }\n> + else {\n> + /* if we can't update the cache, fall back to checking them all */\n> + for (i = 0; i < istate->cache_nr; i++)\n> + istate->cache[i]->ce_flags |= CE_FSMONITOR_DIRTY;\n> +\n> + /* mark the untracked cache as unusable for fsmonitor */\n> + if (istate->untracked)\n> + istate->untracked->use_fsmonitor = 0;\n> + }\n> + strbuf_release(&query_result);\n> +\n> + /* Now that we've updated istate, save the last_update time */\n> + istate->fsmonitor_last_update = last_update;\n> + istate->cache_changed |= FSMONITOR_CHANGED;\n> +}\n"},{"id":"320660","messageId":"CAP8UFD340yndvkT8=F9zdzX9QtbuPspn8rfiK8q7f8c=kGeWQg@mail.gmail.com","threadId":"45997","inReplyTo":"20170518201333.13088-7-benpeart@microsoft.com","subject":"Re: [PATCH v2 6/6] fsmonitor: add a sample query-fsmonitor hook script for Watchman","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-05-24T13:12:54Z","receivedAt":"2017-05-24T13:13:00Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, May 18, 2017 at 10:13 PM, Ben Peart wrote:\n> This hook script integrates the new fsmonitor capabilities of git with\n> the cross platform Watchman file watching service. To use the script:\n>\n> Download and install Watchman from https://facebook.github.io/watchman/\n> and instruct Watchman to watch your working directory for changes\n> ('watchman watch-project /usr/src/git').\n>\n> Rename the sample integration hook from query-fsmonitor.sample to\n> query-fsmonitor.\n>\n> Configure git to use the extension ('git config core.fsmonitor true')\n> and optionally turn on the untracked cache for optimal performance\n> ('git config core.untrackedcache true').\n\nOk, it looks like the untracked cache can be used, but it could work without it.\n\n> Signed-off-by: Ben Peart \n> Signed-off-by: Johannes Schindelin \n> ---\n> templates/hooks--query-fsmonitor.sample | 27 +++++++++++++++++++++++++++\n> 1 file changed, 27 insertions(+)\n> create mode 100644 templates/hooks--query-fsmonitor.sample\n>\n> diff --git a/templates/hooks--query-fsmonitor.sample b/templates/hooks--query-fsmonitor.sample\n> new file mode 100644\n> index 0000000000..4bd22f21d8\n> --- /dev/null\n> +++ b/templates/hooks--query-fsmonitor.sample\n> @@ -0,0 +1,27 @@\n> +#!/bin/sh\n> +#\n> +# An example hook script to integrate Watchman\n> +# (https://facebook.github.io/watchman/) with git to provide fast\n> +# git status.\n> +#\n> +# The hook is passed a time_t formatted as a string and outputs to\n> +# stdout all files that have been modified since the given time.\n> +# Paths must be relative to the root of the working tree and\n> +# separated by a single NUL.\n> +#\n> +# To enable this hook, rename this file to \"query-fsmonitor\"\n> +\n> +# Convert unix style paths to escaped Windows style paths\n> +case \"$(uname -s)\" in\n> +MINGW*|MSYS_NT*)\n> + GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,\\\\\\\\,g')\"\n> + ;;\n> +*)\n> + GIT_WORK_TREE=\"$PWD\"\n> + ;;\n> +esac\n> +\n> +# Query Watchman for all the changes since the requested time\n> +echo \"[\\\"query\\\", \\\"$GIT_WORK_TREE\\\", {\\\"since\\\": $1, \\\"fields\\\":[\\\"name\\\"]}]\" | \\\n> +watchman -j | \\\n> +perl -e 'use JSON::PP; my $o = JSON::PP->new->utf8->decode(join(\"\", <>)); die \"Watchman: $o->{'error'}.\\nFalling back to scanning...\\n\" if defined($o->{\"error\"}); print(join(\"\\0\", @{$o->{\"files\"}}));'\n\nMaybe put the above small perl script on multiple lines for improved\nreadability.\n\nIs JSON::PP always available in Perl >= 5.8?\nWhat happens if watchman is not installed or not in the PATH?\nIt seems to me that the git process will not die, and will just work\nas if the hook does not exist, except that it will call the hook which\nwill probably output error messages.\n"},{"id":"320711","messageId":"52d8376d-63ad-14f8-fbd5-3540a4926e92@gmail.com","threadId":"45997","inReplyTo":"CACBZZX6LENwuVuTyU-RetaXz8jZtUp1dv+gmQQ281sPx1czPnA@mail.gmail.com","subject":"Re: [PATCH v2 5/6] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-25T13:49:20Z","receivedAt":"2017-05-25T13:49:31Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 5/22/2017 1:28 PM, Ævar Arnfjörð Bjarmason wrote:\n> On Mon, May 22, 2017 at 6:18 PM, Ben Peart wrote:\n>> On 5/20/2017 8:10 AM, Ævar Arnfjörð Bjarmason wrote:\n>>>>\n>>>> +== File System Monitor cache\n>>>> +\n>>>> + The file system monitor cache tracks files for which the\n>>>> query-fsmonitor\n>>>> + hook has told us about changes. The signature for this extension is\n>>>> + { 'F', 'S', 'M', 'N' }.\n>>>> +\n>>>> + The extension starts with\n>>>> +\n>>>> + - 32-bit version number: the current supported version is 1.\n>>>> +\n>>>> + - 64-bit time: the extension data reflects all changes through the\n>>>> given\n>>>> + time which is stored as the seconds elapsed since midnight,\n>>>> January 1, 1970.\n>>>> +\n>>>> + - 32-bit bitmap size: the size of the CE_FSMONITOR_DIRTY bitmap.\n>>>> +\n>>>> + - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n>>>> + is CE_FSMONITOR_DIRTY.\n>>>\n>>>\n>>> We already have a uint64_t in one place in the codebase (getnanotime)\n>>> which uses a 64 bit time for nanosecond accuracy, and numerous\n>>> filesystems already support nanosecond timestamps (ext4, that new\n>>> Apple thingy...).\n>>>\n>>> I don't know if any of the inotify/fsmonitor APIs support that yet,\n>>> but it seems inevitable that that'll be added if not, in some\n>>> pathological cases we can have a lot of files modified in 1 second, so\n>>> using nanosecond accuracy means there'll be a lot less data to\n>>> consider in some cases.\n>>>\n>>> It does mean this'll only work until the year ~2500, but that seems\n>>> like an acceptable trade-off.\n>>>\n>>\n>> I really don't think nano-second resolution is needed in this case for a few\n>> reasons.\n>>\n>> The number of files that can change within a given second is limited by the\n>> IO throughput of the underlying device. Even assuming a very fast device and\n>> very small files and changes, this won't be that many files.\n>>\n>> Without this patch, git would have scanned all those files every time. With\n>> this patch, git will only scan those files a 2nd time that are modified in\n>> the same second that it did the first scan *that came before the first scan\n>> started* (the \"lots of files modified\" section in the 1 second timeline\n>> below).\n>>\n>> |------------------------- one second ---------------------|\n>> |-lots of files modified - git status - more file modified-|\n>>\n>> Yes, some duplicate status checks can be made but its still a significant\n>> win in any reasonable scenario. Especially when you consider that it is\n>> pretty unusual to do git status/add/commit calls in the middle of making\n>> lots of changes to files.\n> \n> I understand that we get most of the wins either way.\n> \n> I'm just wondering why not make it nanosecond-resolution now from the\n> beginning since that's where major filesystems are heading already,\n> and changing stuff like this can be a hassle once it's initially out\n> there, whereas just dividing by 10^9 for APIs that need seconds from\n> the beginning is easy & future-proof.\n> \n> There are some uses of git where this would probably matter in practice.\n> \n> E.g. consider a git-annex backed fileserver using ext4, currently\n> git-annex comes with its own FS watcher as a daemon invoked via `git\n> annex watch` to constantly add new files without killing performance\n> via a status/add in a loop, with this a daemon like that could just\n> run status/add in a loop, but on a busy repo the 1s interval size\n> might start to matter as you're constantly inspecting larger\n> intervals.\n> \n> More importantly though, I can't think of any case where having it in\n> nanoseconds to begin with would do any harm.\n\nYou're right, it's not hard to support nano second resolution and it \ndoesn't do any harm. I switch the index format and interface as I don't \nexpect this code will still be running when the timer rolls over. \nSomeone long after me will have to fix it if it is. :)\n\n> \n>> In addition, the backing file system monitor (Watchman) supports number of\n>> seconds since the unix epoch (unix time_t style). This means any support of\n>> nano seconds by git is academic until someone provides a file system watcher\n>> that does support nano second granularity.\n> \n> I haven't used watchman for anything non-trivial, but the\n> documentation for the query API you seem to be using says you can\n> request a {ctime,mtime}_ns field:\n> \n> https://github.com/facebook/watchman/blob/master/website/_docs/cmd.query.markdown#user-content-available-fields\n> \n> And isn't this the documentation for the \"since\" query you're using,\n> saying you can specify any arbitrary fs timing field, such as a _ns\n> time field:\n> \n> https://github.com/facebook/watchman/blob/master/website/_docs/expr.since.md\n> \n> ?\n\nTo keep the index extension and hook interface generic, I have not \nadopted the Watchman specific clock format. This enables anyone to \nprovide a different file system watcher that can inter-operate as nano \nseconds since epoc is easy for anyone to support.\n\n> \n>> Finally, the fsmonitor index extension is versioned so that we can\n>> seamlessly upgrade to nano second resolution later if we desire.\n> \n> Aside from my nanosecond bikeshedding, let's say we change the\n> semantics of any of this in the future: The index has the version, but\n> there's one argument passed to the hook without a version. Is the hook\n> expected to potentially be reading the version from the index to make\n> sense of whether (in this case) the argument is a mtime or mtime_ns?\n> \n> Wouldn't this make more sense than that on top, i.e. pass the version\n> to the hook, untested (and probably whines about the sprintf\n> format...):\n\nIt's a good point that the index extension is versioned and the hook \ninterface is not. I've not seen versioning in any hook interface to \ndate but its never to late to start. I'll add a separate version to the \nhook interface as well so they can be updated independently if needed.\n\n> \n> $ git diff -U1\n> diff --git a/cache.h b/cache.h\n> index 6eafd34fff..3c63f179f8 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -346,2 +346,3 @@ struct index_state {\n> struct untracked_cache *untracked;\n> + uint32_t fsmonitor_version;\n> time_t fsmonitor_last_update;\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index f5a9f818e8..7236b538ac 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -60,2 +60,4 @@ int read_fsmonitor_extension(struct index_state\n> *istate, const void *data,\n> return error(\"bad fsmonitor version %d\", hdr_version);\n> + else\n> + istate->fsmonitor_version = hdr_version;\n> \n> @@ -137,2 +139,3 @@ static int query_fsmonitor(time_t last_update,\n> struct strbuf *query_result)\n> struct child_process cp = CHILD_PROCESS_INIT;\n> + char version[1];\n> char date[64];\n> @@ -143,2 +146,3 @@ static int query_fsmonitor(time_t last_update,\n> struct strbuf *query_result)\n> \n> + snprintf(version, sizeof(version), \"%d\", istate->fsmonitor_version);\n> snprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update)\n> \n"},{"id":"320712","messageId":"ab37d437-2a4e-b6ed-621f-5978083cd15b@gmail.com","threadId":"45997","inReplyTo":"CAP8UFD20gvTZqCOcpd1iozNDHOZR1sUKRwi062wL52FNHWNC3w@mail.gmail.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-25T13:55:35Z","receivedAt":"2017-05-25T13:55:43Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 5/24/2017 6:54 AM, Christian Couder wrote:\n>> Design\n>> ~~~~~~\n>>\n>> A new git hook (query-fsmonitor) must exist and be enabled\n>> (core.fsmonitor=true) that takes a time_t formatted as a string and\n>> outputs to stdout all files that have been modified since the requested\n>> time.\n> \n> Is there a reason why there is a new hook, instead of a\n> \"core.fsmonitorquery\" config option to which you could pass whatever\n> command line with options?\n\nA hook is a simple and well defined way to integrate git with another \nprocess. If there is some fixed set of arguments that need to be passed \nto a file system monitor (beyond the timestamp stored in the index \nextension), they can be encoded in the integration script like I've done \nin the Watchman integration sample hook.\n\n> \n>> A new 'fsmonitor' index extension has been added to store the time the\n>> fsmonitor hook was last queried and a ewah bitmap of the current\n>> 'fsmonitor-dirty' files. Unmarked entries are 'fsmonitor-clean', marked\n>> entries are 'fsmonitor-dirty.'\n>>\n>> As needed, git will call the query-fsmonitor hook proc for the set of\n>> changes since the index was last updated. Git then uses this set of\n>> files along with the list saved in the fsmonitor index extension to flag\n>> the potentially dirty index and untracked cache entries.\n> \n> So this can work only if \"core.untrackedCache\" is set to true?\n> \n\nThis works with core.untrackedCache set to true or false. If it is set \nto false, you get valid results, you just don't get the speed up when \nchecking for untracked files.\n\n> Thanks for working on this,\n> Christian.\n> \n"},{"id":"320770","messageId":"CACBZZX7AA5sbdqhY+W2v3HRSudkWv4cHM+EQUr-X34FGvLH2hw@mail.gmail.com","threadId":"45997","inReplyTo":"20170518201333.13088-7-benpeart@microsoft.com","subject":"Re: [PATCH v2 6/6] fsmonitor: add a sample query-fsmonitor hook script for Watchman","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-25T21:05:11Z","receivedAt":"2017-05-25T21:05:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Thu, May 18, 2017 at 10:13 PM, Ben Peart wrote:\n> This hook script integrates the new fsmonitor capabilities of git with\n> the cross platform Watchman file watching service. To use the script:\n>\n> Download and install Watchman from https://facebook.github.io/watchman/\n> and instruct Watchman to watch your working directory for changes\n> ('watchman watch-project /usr/src/git').\n>\n> Rename the sample integration hook from query-fsmonitor.sample to\n> query-fsmonitor.\n>\n> Configure git to use the extension ('git config core.fsmonitor true')\n> and optionally turn on the untracked cache for optimal performance\n> ('git config core.untrackedcache true').\n>\n> Signed-off-by: Ben Peart \n> Signed-off-by: Johannes Schindelin \n> ---\n> templates/hooks--query-fsmonitor.sample | 27 +++++++++++++++++++++++++++\n> 1 file changed, 27 insertions(+)\n> create mode 100644 templates/hooks--query-fsmonitor.sample\n>\n> diff --git a/templates/hooks--query-fsmonitor.sample b/templates/hooks--query-fsmonitor.sample\n> new file mode 100644\n> index 0000000000..4bd22f21d8\n> --- /dev/null\n> +++ b/templates/hooks--query-fsmonitor.sample\n> @@ -0,0 +1,27 @@\n> +#!/bin/sh\n> +#\n> +# An example hook script to integrate Watchman\n> +# (https://facebook.github.io/watchman/) with git to provide fast\n> +# git status.\n> +#\n> +# The hook is passed a time_t formatted as a string and outputs to\n> +# stdout all files that have been modified since the given time.\n> +# Paths must be relative to the root of the working tree and\n> +# separated by a single NUL.\n> +#\n> +# To enable this hook, rename this file to \"query-fsmonitor\"\n> +\n> +# Convert unix style paths to escaped Windows style paths\n> +case \"$(uname -s)\" in\n> +MINGW*|MSYS_NT*)\n> + GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,\\\\\\\\,g')\"\n> + ;;\n> +*)\n> + GIT_WORK_TREE=\"$PWD\"\n> + ;;\n> +esac\n> +\n> +# Query Watchman for all the changes since the requested time\n> +echo \"[\\\"query\\\", \\\"$GIT_WORK_TREE\\\", {\\\"since\\\": $1, \\\"fields\\\":[\\\"name\\\"]}]\" | \\\n> +watchman -j | \\\n> +perl -e 'use JSON::PP; my $o = JSON::PP->new->utf8->decode(join(\"\", <>)); die \"Watchman: $o->{'error'}.\\nFalling back to scanning...\\n\" if defined($o->{\"error\"}); print(join(\"\\0\", @{$o->{\"files\"}}));'\n\nI couldn't get watchman to work for me (built from source, keep\ngetting [1]), but I hacked up something you can hopefully test &\nsquash on top of this:\n\n # Query Watchman for all the changes since the requested time\n-echo \"[\\\"query\\\", \\\"$GIT_WORK_TREE\\\", {\\\"since\\\": $time_t,\n\\\"fields\\\":[\\\"name\\\"]}]\" | \\\n-watchman -j | \\\n-perl -e 'use JSON::PP; my $o = JSON::PP->new->utf8->decode(join(\"\",\n<>)); die \"Watchman: $o->{'error'}.\\nFalling back to scanning...\\n\" if\ndefined($o-\n+echo \"[\\\"query\\\", \\\"$GIT_WORK_TREE\\\", {\\\"since\\\": $time_t,\n\\\"fields\\\":[\\\"name\\\"]}]\" |\n+ watchman -j |\n+ perl -0666 -e '\n+ use strict;\n+ use warnings;\n+\n+ my $stdin = <>;\n+ die \"Watchman: The watchman command returned no\noutput, error above?\\n\" if $stdin eq \"\";\n+ die \"Watchman: Invalid input: $stdin\\n\" unless $stdin =~ /^\\{/;\n+\n+ my $json_pkg;\n+ eval {\n+ require JSON::XS;\n+ $json_pkg = \"JSON::XS\";\n+ 1;\n+ } or do {\n+ require JSON::PP;\n+ $json_pkg = \"JSON::PP\";\n+ };\n+\n+ my $o = $json_pkg->new->utf8->decode($stdin);\n+ die \"Watchman: $o->{error}.\\nFalling back to scanning...\\n\"\n+ if $o->{error};\n+\n+ local $, = \"\\0\";\n+ print @{$o->{files}};\n+ '\n\nRationale:\n\n * We use the much faster JSON::XS if it's installed.\n * We use strict/warnings\n * Micro optimization: Replace joining stdin with an equivalent -0666\n invocation. See \"perldoc perlrun\".\n * Micro optimization: No need to join up the possibly large list of\n files into one big string, just set $, to \\0 and stream out the\n array.\n * Error handling: My watchman is broken (so actually this isn't\n tested), it only spews to stderr and exits. Handle that by checking\n whether stdin is \"\".\n\nThose changes are available at\nhttps://github.com/avar/git/commits/avar/fsmonitor\n\n\n\n1. watchman: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version\n`CXXABI_1.3.11' not found (required by watchman)\n"},{"id":"320811","messageId":"CACBZZX4X9v5S6EQaX0tX71sK6c5kriaCTMoXqUx0Fg6WO9Wmiw@mail.gmail.com","threadId":"45997","inReplyTo":"CAP8UFD340yndvkT8=F9zdzX9QtbuPspn8rfiK8q7f8c=kGeWQg@mail.gmail.com","subject":"Re: [PATCH v2 6/6] fsmonitor: add a sample query-fsmonitor hook script for Watchman","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-26T09:47:43Z","receivedAt":"2017-05-26T09:48:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Wed, May 24, 2017 at 3:12 PM, Christian Couder\n wrote:\n> On Thu, May 18, 2017 at 10:13 PM, Ben Peart wrote:\n>> This hook script integrates the new fsmonitor capabilities of git with\n>> the cross platform Watchman file watching service. To use the script:\n>>\n>> Download and install Watchman from https://facebook.github.io/watchman/\n>> and instruct Watchman to watch your working directory for changes\n>> ('watchman watch-project /usr/src/git').\n>>\n>> Rename the sample integration hook from query-fsmonitor.sample to\n>> query-fsmonitor.\n>>\n>> Configure git to use the extension ('git config core.fsmonitor true')\n>> and optionally turn on the untracked cache for optimal performance\n>> ('git config core.untrackedcache true').\n>\n> Ok, it looks like the untracked cache can be used, but it could work without it.\n>\n>> Signed-off-by: Ben Peart \n>> Signed-off-by: Johannes Schindelin \n>> ---\n>> templates/hooks--query-fsmonitor.sample | 27 +++++++++++++++++++++++++++\n>> 1 file changed, 27 insertions(+)\n>> create mode 100644 templates/hooks--query-fsmonitor.sample\n>>\n>> diff --git a/templates/hooks--query-fsmonitor.sample b/templates/hooks--query-fsmonitor.sample\n>> new file mode 100644\n>> index 0000000000..4bd22f21d8\n>> --- /dev/null\n>> +++ b/templates/hooks--query-fsmonitor.sample\n>> @@ -0,0 +1,27 @@\n>> +#!/bin/sh\n>> +#\n>> +# An example hook script to integrate Watchman\n>> +# (https://facebook.github.io/watchman/) with git to provide fast\n>> +# git status.\n>> +#\n>> +# The hook is passed a time_t formatted as a string and outputs to\n>> +# stdout all files that have been modified since the given time.\n>> +# Paths must be relative to the root of the working tree and\n>> +# separated by a single NUL.\n>> +#\n>> +# To enable this hook, rename this file to \"query-fsmonitor\"\n>> +\n>> +# Convert unix style paths to escaped Windows style paths\n>> +case \"$(uname -s)\" in\n>> +MINGW*|MSYS_NT*)\n>> + GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,\\\\\\\\,g')\"\n>> + ;;\n>> +*)\n>> + GIT_WORK_TREE=\"$PWD\"\n>> + ;;\n>> +esac\n>> +\n>> +# Query Watchman for all the changes since the requested time\n>> +echo \"[\\\"query\\\", \\\"$GIT_WORK_TREE\\\", {\\\"since\\\": $1, \\\"fields\\\":[\\\"name\\\"]}]\" | \\\n>> +watchman -j | \\\n>> +perl -e 'use JSON::PP; my $o = JSON::PP->new->utf8->decode(join(\"\", <>)); die \"Watchman: $o->{'error'}.\\nFalling back to scanning...\\n\" if defined($o->{\"error\"}); print(join(\"\\0\", @{$o->{\"files\"}}));'\n>\n> Maybe put the above small perl script on multiple lines for improved\n> readability.\n>\n> Is JSON::PP always available in Perl >= 5.8?\n\nNo, it's shipped with perl as of 5.14.0, which came out in May 2011.\n\nI think depending on that is fine. FWIW we bumped the required core\ndependency (but you might still need to install modules) in 2010 in my\nd48b284183 (\"perl: bump the required Perl version to 5.8 from\n5.6.[21]\", 2010-09-24). Maybe we should be bumping that again...\n\n> What happens if watchman is not installed or not in the PATH?\n> It seems to me that the git process will not die, and will just work\n> as if the hook does not exist, except that it will call the hook which\n> will probably output error messages.\n\nI think a good solution to this is to just add \"set -euo pipefail\" to\nthat script.\n\nThen we'll print out on stderr that we couldn't find the watchman\ncommand. Right now (with my patch) it'll make its way to the perl\nprogram and get empty input.\n"},{"id":"320820","messageId":"1a55f46e-7642-587b-3547-7f7b406d2be8@gmail.com","threadId":"45997","inReplyTo":"CACBZZX4X9v5S6EQaX0tX71sK6c5kriaCTMoXqUx0Fg6WO9Wmiw@mail.gmail.com","subject":"Re: [PATCH v2 6/6] fsmonitor: add a sample query-fsmonitor hook script for Watchman","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-26T16:02:19Z","receivedAt":"2017-05-26T16:02:29Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 5/26/2017 5:47 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Wed, May 24, 2017 at 3:12 PM, Christian Couder\n> wrote:\n>> On Thu, May 18, 2017 at 10:13 PM, Ben Peart wrote:\n>>> This hook script integrates the new fsmonitor capabilities of git with\n>>> the cross platform Watchman file watching service. To use the script:\n>>>\n>>> Download and install Watchman from https://facebook.github.io/watchman/\n>>> and instruct Watchman to watch your working directory for changes\n>>> ('watchman watch-project /usr/src/git').\n>>>\n>>> Rename the sample integration hook from query-fsmonitor.sample to\n>>> query-fsmonitor.\n>>>\n>>> Configure git to use the extension ('git config core.fsmonitor true')\n>>> and optionally turn on the untracked cache for optimal performance\n>>> ('git config core.untrackedcache true').\n>>\n>> Ok, it looks like the untracked cache can be used, but it could work without it.\n>>\n>>> Signed-off-by: Ben Peart \n>>> Signed-off-by: Johannes Schindelin \n>>> ---\n>>> templates/hooks--query-fsmonitor.sample | 27 +++++++++++++++++++++++++++\n>>> 1 file changed, 27 insertions(+)\n>>> create mode 100644 templates/hooks--query-fsmonitor.sample\n>>>\n>>> diff --git a/templates/hooks--query-fsmonitor.sample b/templates/hooks--query-fsmonitor.sample\n>>> new file mode 100644\n>>> index 0000000000..4bd22f21d8\n>>> --- /dev/null\n>>> +++ b/templates/hooks--query-fsmonitor.sample\n>>> @@ -0,0 +1,27 @@\n>>> +#!/bin/sh\n>>> +#\n>>> +# An example hook script to integrate Watchman\n>>> +# (https://facebook.github.io/watchman/) with git to provide fast\n>>> +# git status.\n>>> +#\n>>> +# The hook is passed a time_t formatted as a string and outputs to\n>>> +# stdout all files that have been modified since the given time.\n>>> +# Paths must be relative to the root of the working tree and\n>>> +# separated by a single NUL.\n>>> +#\n>>> +# To enable this hook, rename this file to \"query-fsmonitor\"\n>>> +\n>>> +# Convert unix style paths to escaped Windows style paths\n>>> +case \"$(uname -s)\" in\n>>> +MINGW*|MSYS_NT*)\n>>> + GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,\\\\\\\\,g')\"\n>>> + ;;\n>>> +*)\n>>> + GIT_WORK_TREE=\"$PWD\"\n>>> + ;;\n>>> +esac\n>>> +\n>>> +# Query Watchman for all the changes since the requested time\n>>> +echo \"[\\\"query\\\", \\\"$GIT_WORK_TREE\\\", {\\\"since\\\": $1, \\\"fields\\\":[\\\"name\\\"]}]\" | \\\n>>> +watchman -j | \\\n>>> +perl -e 'use JSON::PP; my $o = JSON::PP->new->utf8->decode(join(\"\", <>)); die \"Watchman: $o->{'error'}.\\nFalling back to scanning...\\n\" if defined($o->{\"error\"}); print(join(\"\\0\", @{$o->{\"files\"}}));'\n>>\n>> Maybe put the above small perl script on multiple lines for improved\n>> readability.\n>>\n\nI'll pick up the suggestions from Ævar on the perl script. I believe \nthat fixes all the issues you have raised.\n\nI've also tested the various error cases of watchman not installed and \nwhen watchman returns an error and they are all properly handled by 1) \ngiving an error message and 2) falling back to the git codepath without \nfsmonitor so that the git command succeeds.\n\n>> Is JSON::PP always available in Perl >= 5.8?\n> \n> No, it's shipped with perl as of 5.14.0, which came out in May 2011.\n> \n> I think depending on that is fine. FWIW we bumped the required core\n> dependency (but you might still need to install modules) in 2010 in my\n> d48b284183 (\"perl: bump the required Perl version to 5.8 from\n> 5.6.[21]\", 2010-09-24). Maybe we should be bumping that again...\n> \n>> What happens if watchman is not installed or not in the PATH?\n>> It seems to me that the git process will not die, and will just work\n>> as if the hook does not exist, except that it will call the hook which\n>> will probably output error messages.\n> \n> I think a good solution to this is to just add \"set -euo pipefail\" to\n> that script.\n> \n> Then we'll print out on stderr that we couldn't find the watchman\n> command. Right now (with my patch) it'll make its way to the perl\n> program and get empty input.\n> \n\nWith or without \"set -euo pipefail\" the output if watchman is not \ninstalled is:\n\n.git/hooks/query-fsmonitor: line 37: watchman: command not found\nWatchman: command returned no output.\nFalling back to scanning...\nOn branch fsmonitor\nYour branch is up-to-date with 'benpeart/fsmonitor'.\n\n\nTo try this out on Mac, you can install the package maintained by \nMacPorts by running \"sudo port install watchman\"\n\nhttps://facebook.github.io/watchman/docs/install.html\n\n"},{"id":"320861","messageId":"CAP8UFD1L+caWyLS5KCX3QqToVuzii+0tX81=eGA=1_+L6O7ynA@mail.gmail.com","threadId":"45997","inReplyTo":"ab37d437-2a4e-b6ed-621f-5978083cd15b@gmail.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-05-27T06:57:16Z","receivedAt":"2017-05-27T06:57:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, May 25, 2017 at 3:55 PM, Ben Peart wrote:\n>\n>\n> On 5/24/2017 6:54 AM, Christian Couder wrote:\n>>>\n>>> Design\n>>> ~~~~~~\n>>>\n>>> A new git hook (query-fsmonitor) must exist and be enabled\n>>> (core.fsmonitor=true) that takes a time_t formatted as a string and\n>>> outputs to stdout all files that have been modified since the requested\n>>> time.\n>>\n>>\n>> Is there a reason why there is a new hook, instead of a\n>> \"core.fsmonitorquery\" config option to which you could pass whatever\n>> command line with options?\n>\n>\n> A hook is a simple and well defined way to integrate git with another\n> process. If there is some fixed set of arguments that need to be passed to\n> a file system monitor (beyond the timestamp stored in the index extension),\n> they can be encoded in the integration script like I've done in the Watchman\n> integration sample hook.\n\nYeah, they could be encoded in the integration script, but it could be\nbetter if it was possible to just configure a generic command line.\n\nFor example if the directory that should be watched for filesystem\nchanges could be passed as well as the time since the last changes,\nperhaps only a generic command line would be need.\n\nI am also wondering about sparse checkout, as we might want to pass\nall the directories we are interested in.\nHow is it supposed to work with sparse checkout?\n\n>>> A new 'fsmonitor' index extension has been added to store the time the\n>>> fsmonitor hook was last queried and a ewah bitmap of the current\n>>> 'fsmonitor-dirty' files. Unmarked entries are 'fsmonitor-clean', marked\n>>> entries are 'fsmonitor-dirty.'\n>>>\n>>> As needed, git will call the query-fsmonitor hook proc for the set of\n>>> changes since the index was last updated. Git then uses this set of\n>>> files along with the list saved in the fsmonitor index extension to flag\n>>> the potentially dirty index and untracked cache entries.\n>>\n>>\n>> So this can work only if \"core.untrackedCache\" is set to true?\n>>\n>\n> This works with core.untrackedCache set to true or false. If it is set to\n> false, you get valid results, you just don't get the speed up when checking\n> for untracked files.\n\nGreat!\n"},{"id":"321070","messageId":"8ac721a5-a90d-3d58-e9dd-a3eb2199d387@gmail.com","threadId":"45997","inReplyTo":"CAP8UFD1L+caWyLS5KCX3QqToVuzii+0tX81=eGA=1_+L6O7ynA@mail.gmail.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-30T18:05:52Z","receivedAt":"2017-05-30T18:06:07Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 5/27/2017 2:57 AM, Christian Couder wrote:\n> On Thu, May 25, 2017 at 3:55 PM, Ben Peart wrote:\n>>\n>>\n>> On 5/24/2017 6:54 AM, Christian Couder wrote:\n>>>>\n>>>> Design\n>>>> ~~~~~~\n>>>>\n>>>> A new git hook (query-fsmonitor) must exist and be enabled\n>>>> (core.fsmonitor=true) that takes a time_t formatted as a string and\n>>>> outputs to stdout all files that have been modified since the requested\n>>>> time.\n>>>\n>>>\n>>> Is there a reason why there is a new hook, instead of a\n>>> \"core.fsmonitorquery\" config option to which you could pass whatever\n>>> command line with options?\n>>\n>>\n>> A hook is a simple and well defined way to integrate git with another\n>> process. If there is some fixed set of arguments that need to be passed to\n>> a file system monitor (beyond the timestamp stored in the index extension),\n>> they can be encoded in the integration script like I've done in the Watchman\n>> integration sample hook.\n> \n> Yeah, they could be encoded in the integration script, but it could be\n> better if it was possible to just configure a generic command line.\n> \n> For example if the directory that should be watched for filesystem\n> changes could be passed as well as the time since the last changes,\n> perhaps only a generic command line would be need.\n\nMaybe I'm not understanding what you have in mind but I haven't found \nthis to be the case in the two integrations I've done with file system \nwatchers (one internal and Watchman). They require you download, \ninstall, and configure them by telling them about the folders you want \nmonitored. Then you can start querying them for changes and processing \nthe output to match what git expects. While the download and install \nsteps vary, having that query + process and return results wrapped up in \nan integration hook has worked well.\n\n> \n> I am also wondering about sparse checkout, as we might want to pass\n> all the directories we are interested in.\n> How is it supposed to work with sparse checkout?\n> \n\nThe fsmonitor code works well with or without a sparse-checkout. The \nfile system monitor is unaware of the sparse checkout so will notify git \nabout any change irrespective of whether git will eventually ignore it \nbecause the skip worktree bit is set.\n\n>>>> A new 'fsmonitor' index extension has been added to store the time the\n>>>> fsmonitor hook was last queried and a ewah bitmap of the current\n>>>> 'fsmonitor-dirty' files. Unmarked entries are 'fsmonitor-clean', marked\n>>>> entries are 'fsmonitor-dirty.'\n>>>>\n>>>> As needed, git will call the query-fsmonitor hook proc for the set of\n>>>> changes since the index was last updated. Git then uses this set of\n>>>> files along with the list saved in the fsmonitor index extension to flag\n>>>> the potentially dirty index and untracked cache entries.\n>>>\n>>>\n>>> So this can work only if \"core.untrackedCache\" is set to true?\n>>>\n>>\n>> This works with core.untrackedCache set to true or false. If it is set to\n>> false, you get valid results, you just don't get the speed up when checking\n>> for untracked files.\n> \n> Great!\n> \n"},{"id":"321077","messageId":"CAP8UFD0xrGrB+QFSV9XYq23w8-eN6JKWSJm3iTc4Sn9Kf44NUw@mail.gmail.com","threadId":"45997","inReplyTo":"8ac721a5-a90d-3d58-e9dd-a3eb2199d387@gmail.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-05-30T20:33:14Z","receivedAt":"2017-05-30T20:33:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, May 30, 2017 at 8:05 PM, Ben Peart wrote:\n>\n>\n> On 5/27/2017 2:57 AM, Christian Couder wrote:\n>>\n>> On Thu, May 25, 2017 at 3:55 PM, Ben Peart wrote:\n>>>\n>>>\n>>>\n>>> On 5/24/2017 6:54 AM, Christian Couder wrote:\n>>>>>\n>>>>>\n>>>>> Design\n>>>>> ~~~~~~\n>>>>>\n>>>>> A new git hook (query-fsmonitor) must exist and be enabled\n>>>>> (core.fsmonitor=true) that takes a time_t formatted as a string and\n>>>>> outputs to stdout all files that have been modified since the requested\n>>>>> time.\n>>>>\n>>>>\n>>>>\n>>>> Is there a reason why there is a new hook, instead of a\n>>>> \"core.fsmonitorquery\" config option to which you could pass whatever\n>>>> command line with options?\n>>>\n>>>\n>>>\n>>> A hook is a simple and well defined way to integrate git with another\n>>> process. If there is some fixed set of arguments that need to be passed\n>>> to\n>>> a file system monitor (beyond the timestamp stored in the index\n>>> extension),\n>>> they can be encoded in the integration script like I've done in the\n>>> Watchman\n>>> integration sample hook.\n>>\n>>\n>> Yeah, they could be encoded in the integration script, but it could be\n>> better if it was possible to just configure a generic command line.\n>>\n>> For example if the directory that should be watched for filesystem\n>> changes could be passed as well as the time since the last changes,\n>> perhaps only a generic command line would be need.\n>\n>\n> Maybe I'm not understanding what you have in mind but I haven't found this\n> to be the case in the two integrations I've done with file system watchers\n> (one internal and Watchman). They require you download, install, and\n> configure them by telling them about the folders you want monitored. Then\n> you can start querying them for changes and processing the output to match\n> what git expects. While the download and install steps vary, having that\n> query + process and return results wrapped up in an integration hook has\n> worked well.\n\nIt looks like one can also just ask watchman to monitor a directory with:\n\nwatchman watch /path/to/dir\n\nor:\n\necho '[\"watch\", \"/path/to/dir\"]' | watchman -j\n\nAlso for example on Linux people might want to use command line tools like:\n\nhttps://linux.die.net/man/1/inotifywait\n\nand you can pass the directories you want to be watched as arguments\nto this kind of tools.\n\nSo it would be nice, if we didn't require the user to configure\nanything and we could just configure the watching of what we need in\nthe hook (or a command configured using a config option). If the hook\n(or configured command) could be passed the directory by git, it could\nalso be generic.\n\n>> I am also wondering about sparse checkout, as we might want to pass\n>> all the directories we are interested in.\n>> How is it supposed to work with sparse checkout?\n>\n> The fsmonitor code works well with or without a sparse-checkout. The file\n> system monitor is unaware of the sparse checkout so will notify git about\n> any change irrespective of whether git will eventually ignore it because the\n> skip worktree bit is set.\n\nI was wondering if it could ease the job for the monitoring service\nand perhaps improve performance to just ask to watch the directories\nwe are interested in when using sparse checkout.\nOn Linux it looks like a separate inotify watch is created for every\nsubdirectory and there is maximum amount of inotify watches per user.\nThis can be increased by writing in\n/proc/sys/fs/inotify/max_user_watches, but it is not nice to have to\nask admins to increase this.\n"},{"id":"321093","messageId":"0d553422-5b4f-cfdd-961a-d007826b68cb@gmail.com","threadId":"45997","inReplyTo":"CAP8UFD0xrGrB+QFSV9XYq23w8-eN6JKWSJm3iTc4Sn9Kf44NUw@mail.gmail.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-30T23:11:25Z","receivedAt":"2017-05-30T23:11:33Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 5/30/2017 4:33 PM, Christian Couder wrote:\n> On Tue, May 30, 2017 at 8:05 PM, Ben Peart wrote:\n>>\n>>\n>> On 5/27/2017 2:57 AM, Christian Couder wrote:\n>>>\n>>> On Thu, May 25, 2017 at 3:55 PM, Ben Peart wrote:\n>>>>\n>>>>\n>>>>\n>>>> On 5/24/2017 6:54 AM, Christian Couder wrote:\n>>>>>>\n>>>>>>\n>>>>>> Design\n>>>>>> ~~~~~~\n>>>>>>\n>>>>>> A new git hook (query-fsmonitor) must exist and be enabled\n>>>>>> (core.fsmonitor=true) that takes a time_t formatted as a string and\n>>>>>> outputs to stdout all files that have been modified since the requested\n>>>>>> time.\n>>>>>\n>>>>>\n>>>>>\n>>>>> Is there a reason why there is a new hook, instead of a\n>>>>> \"core.fsmonitorquery\" config option to which you could pass whatever\n>>>>> command line with options?\n>>>>\n>>>>\n>>>>\n>>>> A hook is a simple and well defined way to integrate git with another\n>>>> process. If there is some fixed set of arguments that need to be passed\n>>>> to\n>>>> a file system monitor (beyond the timestamp stored in the index\n>>>> extension),\n>>>> they can be encoded in the integration script like I've done in the\n>>>> Watchman\n>>>> integration sample hook.\n>>>\n>>>\n>>> Yeah, they could be encoded in the integration script, but it could be\n>>> better if it was possible to just configure a generic command line.\n>>>\n>>> For example if the directory that should be watched for filesystem\n>>> changes could be passed as well as the time since the last changes,\n>>> perhaps only a generic command line would be need.\n>>\n>>\n>> Maybe I'm not understanding what you have in mind but I haven't found this\n>> to be the case in the two integrations I've done with file system watchers\n>> (one internal and Watchman). They require you download, install, and\n>> configure them by telling them about the folders you want monitored. Then\n>> you can start querying them for changes and processing the output to match\n>> what git expects. While the download and install steps vary, having that\n>> query + process and return results wrapped up in an integration hook has\n>> worked well.\n> \n> It looks like one can also just ask watchman to monitor a directory with:\n> \n> watchman watch /path/to/dir\n> \n> or:\n> \n> echo '[\"watch\", \"/path/to/dir\"]' | watchman -j\n> \n> Also for example on Linux people might want to use command line tools like:\n> \n> https://linux.die.net/man/1/inotifywait\n> \n> and you can pass the directories you want to be watched as arguments\n> to this kind of tools.\n> \n> So it would be nice, if we didn't require the user to configure\n> anything and we could just configure the watching of what we need in\n> the hook (or a command configured using a config option). If the hook\n> (or configured command) could be passed the directory by git, it could\n> also be generic.\n> \n\nOK, I think I understand what you're attempting to accomplish now. \nOften, Watchman (and other similar tools) are used to do much more than \nspeed up git (in fact, _all_ use cases today are not used for that since \nthis patch series hasn't been accepted yet :)). They trigger builds, \nrun verification tools, test passes, or other tasks.\n\nI'm afraid that attempting to have the user configure git to configure \nthe tool \"automatically\" is just adding an extra layer of complexity \nrather than making it simpler. I'll leave that to a future patch series \nto work out.\n\n>>> I am also wondering about sparse checkout, as we might want to pass\n>>> all the directories we are interested in.\n>>> How is it supposed to work with sparse checkout?\n>>\n>> The fsmonitor code works well with or without a sparse-checkout. The file\n>> system monitor is unaware of the sparse checkout so will notify git about\n>> any change irrespective of whether git will eventually ignore it because the\n>> skip worktree bit is set.\n> \n> I was wondering if it could ease the job for the monitoring service\n> and perhaps improve performance to just ask to watch the directories\n> we are interested in when using sparse checkout.\n> On Linux it looks like a separate inotify watch is created for every\n> subdirectory and there is maximum amount of inotify watches per user.\n> This can be increased by writing in\n> /proc/sys/fs/inotify/max_user_watches, but it is not nice to have to\n> ask admins to increase this.\n> \n\nHaving a single instance that watches the root of the working directory \nis the simplest model and minimizes use of system resources like inotify \nas there is only one needed per clone.\n\nIn addition, when the sparse-checkout file is modified, there is no need \nto try and automatically update the monitor by adding and removing \nfolders as necessary.\n\nFinally, if files or directories are excluded via sparse-checkout, they \nare removed from the working directory at checkout time so don't add any \nadditional overhead to the file system watcher anyway as they clearly \ncan't generate write events if they don't exist.\n"},{"id":"321117","messageId":"CAP8UFD3aZU1D+bbqDEV05QG06gySt9PPL7XD4Xz0TP84zM8M6A@mail.gmail.com","threadId":"45997","inReplyTo":"0d553422-5b4f-cfdd-961a-d007826b68cb@gmail.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-05-31T07:37:17Z","receivedAt":"2017-05-31T07:37:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":">>>> Yeah, they could be encoded in the integration script, but it could be\n>>>> better if it was possible to just configure a generic command line.\n>>>>\n>>>> For example if the directory that should be watched for filesystem\n>>>> changes could be passed as well as the time since the last changes,\n>>>> perhaps only a generic command line would be need.\n>>>\n>>> Maybe I'm not understanding what you have in mind but I haven't found\n>>> this\n>>> to be the case in the two integrations I've done with file system\n>>> watchers\n>>> (one internal and Watchman). They require you download, install, and\n>>> configure them by telling them about the folders you want monitored.\n>>> Then\n>>> you can start querying them for changes and processing the output to\n>>> match\n>>> what git expects. While the download and install steps vary, having that\n>>> query + process and return results wrapped up in an integration hook has\n>>> worked well.\n>>\n>> It looks like one can also just ask watchman to monitor a directory with:\n>>\n>> watchman watch /path/to/dir\n>>\n>> or:\n>>\n>> echo '[\"watch\", \"/path/to/dir\"]' | watchman -j\n>>\n>> Also for example on Linux people might want to use command line tools\n>> like:\n>>\n>> https://linux.die.net/man/1/inotifywait\n>>\n>> and you can pass the directories you want to be watched as arguments\n>> to this kind of tools.\n>>\n>> So it would be nice, if we didn't require the user to configure\n>> anything and we could just configure the watching of what we need in\n>> the hook (or a command configured using a config option). If the hook\n>> (or configured command) could be passed the directory by git, it could\n>> also be generic.\n>\n> OK, I think I understand what you're attempting to accomplish now. Often,\n> Watchman (and other similar tools) are used to do much more than speed up\n> git (in fact, _all_ use cases today are not used for that since this patch\n> series hasn't been accepted yet :)). They trigger builds, run verification\n> tools, test passes, or other tasks.\n\nYeah, but some people might only be interested in installing Watchman\nor similar tools to speed up \"git status\".\n\n> I'm afraid that attempting to have the user configure git to configure the\n> tool \"automatically\" is just adding an extra layer of complexity rather than\n> making it simpler.\n\nI think that for the user it makes things simpler, as the user\nwouldn't have to configure anything.\n\nFor example if the hook does something like the following :\n\n# Check that watchman is already watching the worktree\nif ! watchman watch-list | grep \"\\\"$GIT_WORK_TREE\\\"\"\nthen\n # Ask watchman to watch the worktree\n watchman watch \"$GIT_WORK_TREE\"\n exit 1\nfi\n\n# Query Watchman for all the changes since the requested time\necho \"[\\\"query\\\", \\\"$GIT_WORK_TREE\\\", {\\\"since\\\": $1,\n\\\"fields\\\":[\\\"name\\\"]}]\" | \\\nwatchman -j | \\\n...\n\nThen users might not need to configure Watchman in the first place,\nand if they move their repo they might not need to reconfigure\nWatchman.\n\n> I'll leave that to a future patch series to work out.\n\nYeah, the above improvement can be done later.\n\n>>>> I am also wondering about sparse checkout, as we might want to pass\n>>>> all the directories we are interested in.\n>>>> How is it supposed to work with sparse checkout?\n>>>\n>>>\n>>> The fsmonitor code works well with or without a sparse-checkout. The\n>>> file\n>>> system monitor is unaware of the sparse checkout so will notify git about\n>>> any change irrespective of whether git will eventually ignore it because\n>>> the\n>>> skip worktree bit is set.\n>>\n>> I was wondering if it could ease the job for the monitoring service\n>> and perhaps improve performance to just ask to watch the directories\n>> we are interested in when using sparse checkout.\n>> On Linux it looks like a separate inotify watch is created for every\n>> subdirectory and there is maximum amount of inotify watches per user.\n>> This can be increased by writing in\n>> /proc/sys/fs/inotify/max_user_watches, but it is not nice to have to\n>> ask admins to increase this.\n>\n> Having a single instance that watches the root of the working directory is\n> the simplest model and minimizes use of system resources like inotify as\n> there is only one needed per clone.\n\nFrom https://linux.die.net/man/1/inotifywait:\n\n-r, --recursive\n\nWatch all subdirectories of any directories passed as arguments.\nWatches will be set up recursively to an unlimited depth. Symbolic\nlinks are not traversed. Newly created subdirectories will also be\nwatched.\n\nWarning: If you use this option while watching the root directory of a\nlarge tree, it may take quite a while until all inotify watches are\nestablished, and events will not be received in this time. Also, since\none inotify watch will be established per subdirectory, it is possible\nthat the maximum amount of inotify watches per user will be reached.\nThe default maximum is 8192; it can be increased by writing to\n/proc/sys/fs/inotify/max_user_watches.\n\n> In addition, when the sparse-checkout file is modified, there is no need to\n> try and automatically update the monitor by adding and removing folders as\n> necessary.\n\nYeah, I agree it is simpler to not have to worry about the\nsparse-checkout file being modified.\n\n> Finally, if files or directories are excluded via sparse-checkout, they are\n> removed from the working directory at checkout time so don't add any\n> additional overhead to the file system watcher anyway as they clearly can't\n> generate write events if they don't exist.\n\nYeah, but if you configure sparse-checkout when you already have\nuntracked files in many directories, like .o files, these files and\nthe directories they are in are not removed when you perform a\ncheckout, so the directories are still watched by the file system\nwatcher.\n"},{"id":"321118","messageId":"CAP8UFD3R0dEsX7BtzYoSmvaKTZYsm9=bpQsu0jEk2aKm83z-2A@mail.gmail.com","threadId":"45997","inReplyTo":"ab37d437-2a4e-b6ed-621f-5978083cd15b@gmail.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-05-31T07:59:15Z","receivedAt":"2017-05-31T07:59:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, May 25, 2017 at 3:55 PM, Ben Peart wrote:\n>\n> On 5/24/2017 6:54 AM, Christian Couder wrote:\n>>>\n>>> A new git hook (query-fsmonitor) must exist and be enabled\n>>> (core.fsmonitor=true) that takes a time_t formatted as a string and\n>>> outputs to stdout all files that have been modified since the requested\n>>> time.\n>>\n>> Is there a reason why there is a new hook, instead of a\n>> \"core.fsmonitorquery\" config option to which you could pass whatever\n>> command line with options?\n>\n> A hook is a simple and well defined way to integrate git with another\n> process. If there is some fixed set of arguments that need to be passed to\n> a file system monitor (beyond the timestamp stored in the index extension),\n> they can be encoded in the integration script like I've done in the Watchman\n> integration sample hook.\n\nYeah, but a hook must also be called everytime git wants to\ncommunicate with the file system monitor. And we could perhaps get a\nspeed up if we could have only one long running process to communicate\nwith the file system monitor.\n"},{"id":"321127","messageId":"f2c21d3e-9892-bdcb-2686-341de87ee15d@gmail.com","threadId":"45997","inReplyTo":"CAP8UFD3R0dEsX7BtzYoSmvaKTZYsm9=bpQsu0jEk2aKm83z-2A@mail.gmail.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-05-31T13:37:33Z","receivedAt":"2017-05-31T13:37:40Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 5/31/2017 3:59 AM, Christian Couder wrote:\n> On Thu, May 25, 2017 at 3:55 PM, Ben Peart wrote:\n>>\n>> On 5/24/2017 6:54 AM, Christian Couder wrote:\n>>>>\n>>>> A new git hook (query-fsmonitor) must exist and be enabled\n>>>> (core.fsmonitor=true) that takes a time_t formatted as a string and\n>>>> outputs to stdout all files that have been modified since the requested\n>>>> time.\n>>>\n>>> Is there a reason why there is a new hook, instead of a\n>>> \"core.fsmonitorquery\" config option to which you could pass whatever\n>>> command line with options?\n>>\n>> A hook is a simple and well defined way to integrate git with another\n>> process. If there is some fixed set of arguments that need to be passed to\n>> a file system monitor (beyond the timestamp stored in the index extension),\n>> they can be encoded in the integration script like I've done in the Watchman\n>> integration sample hook.\n> \n> Yeah, but a hook must also be called everytime git wants to\n> communicate with the file system monitor. And we could perhaps get a\n> speed up if we could have only one long running process to communicate\n> with the file system monitor.\n> \n\nIn this particular case a long running background processes isn't \nhelpful because refresh_by_fsmonitor() already has logic to ensure the \nfile system monitor is only called once per git process.\n\nThe overhead of that one call isn't significant as demonstrated by the \nperformance numbers I sent out in the cover letter. Even with the cost \nof the hook and all the associated post-processing, this patch series \nstill results in a nice performance win (even on Windows where spawning \nprocesses is typically more expensive than on other platforms).\n\nI appreciate the close look at this patch series! Even when I'm pushing \nback, I'm glad someone is asking questions and making sure I've thought \nthrough things.\n"},{"id":"321129","messageId":"CACBZZX6x914+XHcxQzTUnjk6yjgr-QTi+1L6oE=_2hWjvXDbpw@mail.gmail.com","threadId":"45997","inReplyTo":"f2c21d3e-9892-bdcb-2686-341de87ee15d@gmail.com","subject":"Re: [PATCH v2 0/6] Fast git status via a file system watcher","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-31T14:10:03Z","receivedAt":"2017-05-31T14:10:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Wed, May 31, 2017 at 3:37 PM, Ben Peart wrote:\n> On 5/31/2017 3:59 AM, Christian Couder wrote:\n>>\n>> On Thu, May 25, 2017 at 3:55 PM, Ben Peart wrote:\n>>>\n>>>\n>>> On 5/24/2017 6:54 AM, Christian Couder wrote:\n>>>>>\n>>>>>\n>>>>> A new git hook (query-fsmonitor) must exist and be enabled\n>>>>> (core.fsmonitor=true) that takes a time_t formatted as a string and\n>>>>> outputs to stdout all files that have been modified since the requested\n>>>>> time.\n>>>>\n>>>>\n>>>> Is there a reason why there is a new hook, instead of a\n>>>> \"core.fsmonitorquery\" config option to which you could pass whatever\n>>>> command line with options?\n>>>\n>>>\n>>> A hook is a simple and well defined way to integrate git with another\n>>> process. If there is some fixed set of arguments that need to be passed\n>>> to\n>>> a file system monitor (beyond the timestamp stored in the index\n>>> extension),\n>>> they can be encoded in the integration script like I've done in the\n>>> Watchman\n>>> integration sample hook.\n>>\n>>\n>> Yeah, but a hook must also be called everytime git wants to\n>> communicate with the file system monitor. And we could perhaps get a\n>> speed up if we could have only one long running process to communicate\n>> with the file system monitor.\n>>\n>\n> In this particular case a long running background processes isn't helpful\n> because refresh_by_fsmonitor() already has logic to ensure the file system\n> monitor is only called once per git process.\n>\n> The overhead of that one call isn't significant as demonstrated by the\n> performance numbers I sent out in the cover letter. Even with the cost of\n> the hook and all the associated post-processing, this patch series still\n> results in a nice performance win (even on Windows where spawning processes\n> is typically more expensive than on other platforms).\n>\n> I appreciate the close look at this patch series! Even when I'm pushing\n> back, I'm glad someone is asking questions and making sure I've thought\n> through things.\n\nI think where this series is going for now makes perfect sense.\nLooking forward to the next version, need to get watchman compiling\nthough.\n\nI think what Christian may be getting at here (correct me if I'm\nwrong) is that David had a series 1-2 years ago that went a bit beyond\nwhat this is doing, i.e. it had a persistent index-helper daemon\ntalking to watchman, so \"git status\" would stream the index state over\nfrom this daemon, and it would consult watchman.\n\nThat's an interesting *additional* optimization, but a huge change on\ntop of this, so it makes perfect sense to start with something simpler\nas this series does, and my hunch from having tested/looked at both is\nthat just talking to watchman is going to be by far the biggest win.\n"}]}