{"thread":{"id":"46154","subject":"[PATCH v5 0/7] Fast git status via a file system watcher","startedAt":"2017-06-10T13:40:51Z","lastAt":"2017-10-04T12:48:47Z","messageCount":137,"participants":["Ben Peart","Junio C Hamano","Christian Couder","David Turner","Torsten Bögershausen","tboegi@web.de","Johannes Schindelin","Jonathan Nieder","Martin Ågren","Stefan Beller","Jeff King","Alex Vandiver"],"isPatch":true,"patchVersion":5,"patchTotal":7},"messages":[{"id":"321919","messageId":"20170610134026.104552-1-benpeart@microsoft.com","threadId":"46154","inReplyTo":null,"subject":"[PATCH v5 0/7] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-10T13:40:19Z","receivedAt":"2017-06-10T13:40:51Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Changes from V4 include:\n\nfsmonitor.c:\n\tOnly flag index dirty if fsmonitor extension is added or removed\n\tor if new cache or untracked cache entries are marked dirty\n\ndir.c:\n\tIf an untracked cache entry is flagged as fsmonitor_dirty,\n\tfall back to existing logic to stat the file and check for\n\tchanges instead of assuming it is dirty.\n\nhooks--query-fsmonitor.sample: \n\tOptimize query to exclude transitory files (files that were\n\tcreated and deleted since the last call).\n\t\n\tUpdate path mangling on Windows to match Watchman beta.\n\ntest-drop-caches.c:\n\tAdd perf helper to drop the disk cache on Windows.\n\np7519-fsmonitor.sh:\n\tAdd perf test for fsmonitor changes\n\nBen Peart (7):\n bswap: add 64 bit endianness helper get_be64\n dir: make lookup_untracked() available outside of dir.c\n fsmonitor: teach git to optionally utilize a file system monitor to\n speed up detecting new or changed files.\n fsmonitor: add test cases for fsmonitor extension\n fsmonitor: add documentation for the fsmonitor extension.\n fsmonitor: add a sample query-fsmonitor hook script for Watchman\n fsmonitor: add a performance test\n\n Documentation/config.txt | 7 +\n Documentation/githooks.txt | 23 +++\n Documentation/technical/index-format.txt | 19 +++\n Makefile | 2 +\n builtin/update-index.c | 1 +\n cache.h | 5 +\n compat/bswap.h | 4 +\n config.c | 4 +\n dir.c | 29 ++--\n dir.h | 5 +\n entry.c | 1 +\n environment.c | 1 +\n fsmonitor.c | 261 +++++++++++++++++++++++++++++++\n fsmonitor.h | 9 ++\n read-cache.c | 28 +++-\n t/helper/test-drop-caches.c | 107 +++++++++++++\n t/perf/p7519-fsmonitor.sh | 161 +++++++++++++++++++\n t/t7519-status-fsmonitor.sh | 173 ++++++++++++++++++++\n templates/hooks--query-fsmonitor.sample | 76 +++++++++\n unpack-trees.c | 1 +\n 20 files changed, 904 insertions(+), 13 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n create mode 100644 t/helper/test-drop-caches.c\n create mode 100755 t/perf/p7519-fsmonitor.sh\n create mode 100755 t/t7519-status-fsmonitor.sh\n create mode 100755 templates/hooks--query-fsmonitor.sample\n\n-- \n2.13.0\n\n"},{"id":"321920","messageId":"20170610134026.104552-3-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170610134026.104552-1-benpeart@microsoft.com","subject":"[PATCH v5 2/7] dir: make lookup_untracked() available outside of dir.c","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-10T13:40:21Z","receivedAt":"2017-06-10T13:40:52Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Remove the static qualifier from lookup_untracked() and make it\navailable to other modules by exporting it from dir.h. This will be\nused later when we need to find entries to mark 'fsmonitor dirty.'\n\nSigned-off-by: Ben Peart \n---\n dir.c | 2 +-\n dir.h | 3 +++\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 9efcf1eab6..5f531d9eed 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -666,7 +666,7 @@ static void trim_trailing_spaces(char *buf)\n *\n * If \"name\" has the trailing slash, it'll be excluded in the search.\n */\n-static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n \t\t\t\t\t\t struct untracked_cache_dir *dir,\n \t\t\t\t\t\t const char *name, int len)\n {\ndiff --git a/dir.h b/dir.h\nindex a89c13e27a..75655ebe35 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -354,4 +354,7 @@ extern void connect_work_tree_and_git_dir(const char *work_tree, const char *git\n extern void relocate_gitdir(const char *path,\n \t\t\t const char *old_git_dir,\n \t\t\t const char *new_git_dir);\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+\t\t\t\t\t struct untracked_cache_dir *dir,\n+\t\t\t\t\t const char *name, int len);\n #endif\n-- \n2.13.0\n\n"},{"id":"321921","messageId":"20170610134026.104552-2-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170610134026.104552-1-benpeart@microsoft.com","subject":"[PATCH v5 1/7] bswap: add 64 bit endianness helper get_be64","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-10T13:40:20Z","receivedAt":"2017-06-10T13:40:53Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a new get_be64 macro to enable 64 bit endian conversions on memory\nthat may or may not be aligned.\n\nSigned-off-by: Ben Peart \n---\n compat/bswap.h | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/compat/bswap.h b/compat/bswap.h\nindex d47c003544..f89fe7f4b5 100644\n--- a/compat/bswap.h\n+++ b/compat/bswap.h\n@@ -158,6 +158,7 @@ static inline uint64_t git_bswap64(uint64_t x)\n \n #define get_be16(p)\tntohs(*(unsigned short *)(p))\n #define get_be32(p)\tntohl(*(unsigned int *)(p))\n+#define get_be64(p)\tntohll(*(uint64_t *)(p))\n #define put_be32(p, v)\tdo { *(unsigned int *)(p) = htonl(v); } while (0)\n \n #else\n@@ -170,6 +171,9 @@ static inline uint64_t git_bswap64(uint64_t x)\n \t(*((unsigned char *)(p) + 1) << 16) | \\\n \t(*((unsigned char *)(p) + 2) << 8) | \\\n \t(*((unsigned char *)(p) + 3) << 0) )\n+#define get_be64(p)\t( \\\n+\t((uint64_t)get_be32((unsigned char *)(p) + 0) << 32) | \\\n+\t((uint64_t)get_be32((unsigned char *)(p) + 4) << 0)\n #define put_be32(p, v)\tdo { \\\n \tunsigned int __v = (v); \\\n \t*((unsigned char *)(p) + 0) = __v >> 24; \\\n-- \n2.13.0\n\n"},{"id":"321922","messageId":"20170610134026.104552-5-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170610134026.104552-1-benpeart@microsoft.com","subject":"[PATCH v5 4/7] fsmonitor: add test cases for fsmonitor extension","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-10T13:40:23Z","receivedAt":"2017-06-10T13:40:58Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add test cases that ensure status results are correct when using the new\nfsmonitor extension. Test untracked, modified, and new files by\nensuring the results are identical to when not using the extension.\n\nAdd a test to ensure updates to the index properly mark corresponding\nentries in the index extension as dirty so that the status is correct\nafter commands that modify the index but don't trigger changes in the\nworking directory.\n\nAdd a test that verifies that if the fsmonitor extension doesn't tell\ngit about a change, it doesn't discover it on its own. This ensures\ngit is honoring the extension and that we get the performance benefits\ndesired.\n\nAll test hooks output a marker file that is used to ensure the hook\nwas actually used to generate the test results.\n\nSigned-off-by: Ben Peart \n---\n t/t7519-status-fsmonitor.sh | 173 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 173 insertions(+)\n create mode 100755 t/t7519-status-fsmonitor.sh\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nnew file mode 100755\nindex 0000000000..458eabe6dc\n--- /dev/null\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -0,0 +1,173 @@\n+#!/bin/sh\n+\n+test_description='git status with file system watcher'\n+\n+. ./test-lib.sh\n+\n+clean_repo () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd &&\n+\trm -f marker\n+}\n+\n+dirty_repo () {\n+\t: >untracked &&\n+\t: >dir1/untracked &&\n+\t: >dir2/untracked &&\n+\techo 1 >modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\techo 4 >new &&\n+\techo 5 >dir1/new &&\n+\techo 6 >dir2/new &&\n+\tgit add new &&\n+\tgit add dir1/new &&\n+\tgit add dir2/new\n+}\n+\n+# The test query-fsmonitor hook proc will output a marker file we can use to\n+# ensure the hook was actually used to generate the correct results.\n+\n+# fsmonitor works correctly with or without the untracked cache\n+# but if it is available, we'll turn it on to ensure we test that\n+# codepath as well.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+if test_have_prereq UNTRACKED_CACHE; then\n+\tgit config core.untrackedcache true\n+else\n+\tgit config core.untrackedcache false\n+fi\n+\n+test_expect_success 'setup' '\n+\tmkdir -p .git/hooks &&\n+\t: >tracked &&\n+\t: >modified &&\n+\tmkdir dir1 &&\n+\t: >dir1/tracked &&\n+\t: >dir1/modified &&\n+\tmkdir dir2 &&\n+\t: >dir2/tracked &&\n+\t: >dir2/modified &&\n+\tgit add . &&\n+\ttest_tick &&\n+\tgit commit -m initial &&\n+\tgit config core.fsmonitor true &&\n+\tcat >.gitignore <<-\\EOF\n+\t.gitignore\n+\texpect*\n+\toutput*\n+\tmarker*\n+\tEOF\n+'\n+\n+# Ensure commands that call refresh_index() to move the index back in time\n+# properly invalidate the fsmonitor cache\n+\n+test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n+\tgit status &&\n+\ttest_path_is_missing marker &&\n+\tdirty_repo &&\n+\twrite_script .git/hooks/query-fsmonitor<<-\\EOF &&\n+\t:>marker\n+\tEOF\n+\tgit add . &&\n+\tgit commit -m \"to reset\" &&\n+\tgit status &&\n+\ttest_path_is_file marker &&\n+\tgit reset HEAD~1 &&\n+\trm -f marker &&\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\tgit -c core.fsmonitor=false status >expect &&\n+\ttest_i18ncmp expect output\n+'\n+\n+# Now make sure it's actually skipping the check for modified and untracked\n+# files unless it is told about them. Note, after \"git reset --hard HEAD\" no\n+# extensions exist other than 'TREE' so do a \"git status\" to get the extension\n+# written before testing the results.\n+\n+test_expect_success \"status doesn't detect unreported modifications\" '\n+\twrite_script .git/hooks/query-fsmonitor<<-\\EOF &&\n+\t:>marker\n+\tEOF\n+\tclean_repo &&\n+\tgit status &&\n+\ttest_path_is_missing marker &&\n+\t: >untracked &&\n+\techo 2 >dir1/modified &&\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\ttest_i18ngrep ! \"Changes not staged for commit:\" output &&\n+\ttest_i18ngrep ! \"Untracked files:\" output &&\n+\twrite_script .git/hooks/query-fsmonitor<<-\\EOF &&\n+\t:>marker\n+\tprintf \"untracked\\0\"\n+\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\trm -f marker &&\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\ttest_i18ngrep \"Changes not staged for commit:\" output &&\n+\ttest_i18ngrep \"Untracked files:\" output\n+'\n+\n+# Status is well tested elsewhere so we'll just ensure that the results are\n+# the same when using core.fsmonitor. First call after turning on the option\n+# does a complete scan so we need to do two calls to ensure we test the new\n+# codepath.\n+\n+test_expect_success 'status with core.untrackedcache false' '\n+\tgit config core.untrackedcache false &&\n+\twrite_script .git/hooks/query-fsmonitor<<-\\EOF &&\n+\tif [ $1 -ne 1 ]\n+\tthen\n+\t\techo -e \"Unsupported query-fsmonitor hook version.\\n\" >&2\n+\t\texit 1;\n+\tfi\n+\t: >marker\n+\tprintf \"untracked\\0\"\n+\tprintf \"dir1/untracked\\0\"\n+\tprintf \"dir2/untracked\\0\"\n+\tprintf \"modified\\0\"\n+\tprintf \"dir1/modified\\0\"\n+\tprintf \"dir2/modified\\0\"\n+\tprintf \"new\\0\"\"\n+\tprintf \"dir1/new\\0\"\n+\tprintf \"dir2/new\\0\"\n+\tEOF\n+\tclean_repo &&\n+\tdirty_repo &&\n+\tgit -c core.fsmonitor=false status >expect &&\n+\tclean_repo &&\n+\tgit status &&\n+\ttest_path_is_missing marker &&\n+\tdirty_repo &&\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\ttest_i18ncmp expect output\n+'\n+\n+if ! test_have_prereq UNTRACKED_CACHE; then\n+\tskip_all='This system does not support untracked cache'\n+\ttest_done\n+fi\n+\n+test_expect_success 'status with core.untrackedcache true' '\n+\tgit config core.untrackedcache true &&\n+\tgit -c core.fsmonitor=false status >expect &&\n+\tclean_repo &&\n+\tgit status &&\n+\ttest_path_is_missing marker &&\n+\tdirty_repo &&\n+\tgit status >output &&\n+\ttest_path_is_file marker &&\n+\ttest_i18ncmp expect output\n+'\n+\n+test_done\n-- \n2.13.0\n\n"},{"id":"321923","messageId":"20170610134026.104552-4-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170610134026.104552-1-benpeart@microsoft.com","subject":"[PATCH v5 3/7] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-10T13:40:22Z","receivedAt":"2017-06-10T13:41:02Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"When the index is read from disk, the query-fsmonitor index extension is\nused to flag the last known potentially dirty index and untracked cache\nentries.\n\nIf git finds out some entries are 'fsmonitor-dirty', but are really\nunchanged (e.g. the file was changed, then reverted back), then Git will\nclear the marking in the extension. If git adds or updates an index\nentry, it is marked 'fsmonitor-dirty' to ensure it is checked for\nchanges in the working directory.\n\nBefore the 'fsmonitor-dirty' flags are used to limit the scope of the\nfiles to be checked, the query-fsmonitor hook proc is called with the\ntime the index was last updated. The hook proc returns the list of\nfiles changed since that last updated time and the list of\npotentially dirty entries is updated to reflect the current state.\n\nrefresh_index() and valid_cached_dir() are updated so that any entry not\nflagged as potentially dirty is not checked as it cannot have any\nchanges.\n\nSigned-off-by: Ben Peart \n---\n Makefile | 1 +\n builtin/update-index.c | 1 +\n cache.h | 5 +\n config.c | 4 +\n dir.c | 27 +++--\n dir.h | 2 +\n entry.c | 1 +\n environment.c | 1 +\n fsmonitor.c | 261 +++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor.h | 9 ++\n read-cache.c | 28 +++++-\n unpack-trees.c | 1 +\n 12 files changed, 329 insertions(+), 12 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n\ndiff --git a/Makefile b/Makefile\nindex 7c621f7f76..992dd58801 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -763,6 +763,7 @@ LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n+LIB_OBJS += fsmonitor.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex ebfc09faa0..32fd977b43 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -232,6 +232,7 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tactive_cache[pos]->ce_flags |= CE_FSMONITOR_DIRTY;\n \t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\ndiff --git a/cache.h b/cache.h\nindex 4d92aae0e8..220efbfde7 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -201,6 +201,7 @@ struct cache_entry {\n #define CE_ADDED (1 << 19)\n \n #define CE_HASHED (1 << 20)\n+#define CE_FSMONITOR_DIRTY (1 << 21)\n #define CE_WT_REMOVE (1 << 22) /* remove in work directory */\n #define CE_CONFLICTED (1 << 23)\n \n@@ -324,6 +325,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n #define UNTRACKED_CHANGED\t(1 << 7)\n+#define FSMONITOR_CHANGED\t(1 << 8)\n \n struct split_index;\n struct untracked_cache;\n@@ -342,6 +344,8 @@ struct index_state {\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n \tstruct untracked_cache *untracked;\n+\tuint64_t fsmonitor_last_update;\n+\tstruct ewah_bitmap *fsmonitor_dirty;\n };\n \n extern struct index_state the_index;\n@@ -767,6 +771,7 @@ extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n extern int git_db_env, git_index_env, git_graft_env, git_common_dir_env;\n+extern int core_fsmonitor;\n \n /*\n * Include broken refs in all ref iterations, which will\ndiff --git a/config.c b/config.c\nindex 146cb3452a..c70f667640 100644\n--- a/config.c\n+++ b/config.c\n@@ -1245,6 +1245,10 @@ static int git_default_core_config(const char *var, const char *value)\n \t\t\thide_dotfiles = HIDE_DOTFILES_DOTGITONLY;\n \t\telse\n \t\t\thide_dotfiles = git_config_bool(var, value);\n+\t}\n+\n+\tif (!strcmp(var, \"core.fsmonitor\")) {\n+\t\tcore_fsmonitor = git_config_bool(var, value);\n \t\treturn 0;\n \t}\n \ndiff --git a/dir.c b/dir.c\nindex 5f531d9eed..1f4a43dc75 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -17,6 +17,7 @@\n #include \"utf8.h\"\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Tells read_directory_recursive how a file or directory should be treated.\n@@ -1680,17 +1681,23 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n-\tif (stat(path->len ? path->buf : \".\", &st)) {\n-\t\tinvalidate_directory(dir->untracked, untracked);\n-\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n-\t\treturn 0;\n-\t}\n-\tif (!untracked->valid ||\n-\t match_stat_data_racy(istate, &untracked->stat_data, &st)) {\n-\t\tif (untracked->valid)\n+\t/*\n+\t * With fsmonitor, we can trust the untracked cache's valid field.\n+\t */\n+\trefresh_by_fsmonitor(istate);\n+\tif (!(dir->untracked->use_fsmonitor && untracked->valid)) {\n+\t\tif (stat(path->len ? path->buf : \".\", &st)) {\n \t\t\tinvalidate_directory(dir->untracked, untracked);\n-\t\tfill_stat_data(&untracked->stat_data, &st);\n-\t\treturn 0;\n+\t\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (!untracked->valid ||\n+\t\t match_stat_data_racy(istate, &untracked->stat_data, &st)) {\n+\t\t\tif (untracked->valid)\n+\t\t\t\tinvalidate_directory(dir->untracked, untracked);\n+\t\t\tfill_stat_data(&untracked->stat_data, &st);\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n \tif (untracked->check_only != !!check_only) {\ndiff --git a/dir.h b/dir.h\nindex 75655ebe35..1a67cb4324 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -139,6 +139,8 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n+\t/* fsmonitor invalidation data */\n+\tunsigned int use_fsmonitor : 1;\n };\n \n struct dir_struct {\ndiff --git a/entry.c b/entry.c\nindex d6b263f78e..667ca5734b 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -222,6 +222,7 @@ static int write_entry(struct cache_entry *ce,\n \t\t\tlstat(ce->name, &st);\n \t\tfill_stat_cache_info(ce, &st);\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \treturn 0;\ndiff --git a/environment.c b/environment.c\nindex aa478e71de..ea63ee4604 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -64,6 +64,7 @@ int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\n enum hide_dotfiles_type hide_dotfiles = HIDE_DOTFILES_DOTGITONLY;\n enum log_refs_config log_all_ref_updates = LOG_REFS_UNSET;\n+int core_fsmonitor;\n \n #ifndef PROTECT_HFS_DEFAULT\n #define PROTECT_HFS_DEFAULT 0\ndiff --git a/fsmonitor.c b/fsmonitor.c\nnew file mode 100644\nindex 0000000000..c1f89abccd\n--- /dev/null\n+++ b/fsmonitor.c\n@@ -0,0 +1,261 @@\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"ewah/ewok.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"fsmonitor.h\"\n+\n+#define INDEX_EXTENSION_VERSION\t1\n+#define HOOK_INTERFACE_VERSION\t\t1\n+\n+int read_fsmonitor_extension(struct index_state *istate, const void *data,\n+\tunsigned long sz)\n+{\n+\tconst char *index = data;\n+\tuint32_t hdr_version;\n+\tuint32_t ewah_size;\n+\tint ret;\n+\n+\tif (sz < sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t))\n+\t\treturn error(\"corrupt fsmonitor extension (too short)\");\n+\n+\thdr_version = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\tif (hdr_version != INDEX_EXTENSION_VERSION)\n+\t\treturn error(\"bad fsmonitor version %d\", hdr_version);\n+\n+\tistate->fsmonitor_last_update = get_be64(index);\n+\tindex += sizeof(uint64_t);\n+\n+\tewah_size = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\n+\tistate->fsmonitor_dirty = ewah_new();\n+\tret = ewah_read_mmap(istate->fsmonitor_dirty, index, ewah_size);\n+\tif (ret != ewah_size) {\n+\t\tewah_free(istate->fsmonitor_dirty);\n+\t\tistate->fsmonitor_dirty = NULL;\n+\t\treturn error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n+\t}\n+\n+\treturn 0;\n+}\n+\n+void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n+{\n+\tuint32_t hdr_version;\n+\tuint64_t tm;\n+\tstruct ewah_bitmap *bitmap;\n+\tint i;\n+\tuint32_t ewah_start;\n+\tuint32_t ewah_size = 0;\n+\tint fixup = 0;\n+\n+\thdr_version = htonl(INDEX_EXTENSION_VERSION);\n+\tstrbuf_add(sb, &hdr_version, sizeof(uint32_t));\n+\n+\ttm = htonll((uint64_t)istate->fsmonitor_last_update);\n+\tstrbuf_add(sb, &tm, sizeof(uint64_t));\n+\tfixup = sb->len;\n+\tstrbuf_add(sb, &ewah_size, sizeof(uint32_t)); /* we'll fix this up later */\n+\n+\tewah_start = sb->len;\n+\tbitmap = ewah_new();\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_DIRTY)\n+\t\t\tewah_set(bitmap, i);\n+\tewah_serialize_strbuf(bitmap, sb);\n+\tewah_free(bitmap);\n+\n+\t/* fix up size field */\n+\tewah_size = htonl(sb->len - ewah_start);\n+\tmemcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t));\n+}\n+\n+static struct untracked_cache_dir *find_untracked_cache_dir(\n+\tstruct untracked_cache *uc, struct untracked_cache_dir *ucd,\n+\tconst char *name)\n+{\n+\tconst char *end;\n+\tstruct untracked_cache_dir *dir = ucd;\n+\n+\tif (!*name)\n+\t\treturn dir;\n+\n+\tend = strchr(name, '/');\n+\tif (end) {\n+\t\tdir = lookup_untracked(uc, ucd, name, end - name);\n+\t\tif (dir)\n+\t\t\treturn find_untracked_cache_dir(uc, dir, end + 1);\n+\t}\n+\n+\treturn dir;\n+}\n+\n+/* This function will be passed to ewah_each_bit() */\n+static void mark_fsmonitor_dirty(size_t pos, void *is)\n+{\n+\tstruct index_state *istate = is;\n+\tstruct untracked_cache_dir *dir;\n+\tstruct cache_entry *ce = istate->cache[pos];\n+\n+\tassert(pos < istate->cache_nr);\n+\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n+\n+\tif (!istate->untracked || !istate->untracked->root)\n+\t\treturn;\n+\n+\tdir = find_untracked_cache_dir(istate->untracked, istate->untracked->root, ce->name);\n+\tif (dir)\n+\t\tdir->valid = 0;\n+}\n+\n+void tweak_fsmonitor_extension(struct index_state *istate)\n+{\n+\tint val, fsmonitor = 0;\n+\n+\tif (!git_config_get_maybe_bool(\"core.fsmonitor\", &val))\n+\t\tfsmonitor = val;\n+\n+\tif (fsmonitor) {\n+\t\tif (!istate->fsmonitor_last_update)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\tif (istate->fsmonitor_dirty)\n+\t\t\tewah_each_bit(istate->fsmonitor_dirty, mark_fsmonitor_dirty, istate);\n+\t} else {\n+\t\tif (istate->fsmonitor_last_update)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\tistate->fsmonitor_last_update = 0;\n+\t}\n+\n+\tif (istate->fsmonitor_dirty) {\n+\t\tewah_free(istate->fsmonitor_dirty);\n+\t\tistate->fsmonitor_dirty = NULL;\n+\t}\n+}\n+\n+/*\n+ * Call the query-fsmonitor hook passing the time of the last saved results.\n+ */\n+static int query_fsmonitor(int version, uint64_t last_update, struct strbuf *query_result)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tchar ver[64];\n+\tchar date[64];\n+\tconst char *argv[4];\n+\n+\tif (!(argv[0] = find_hook(\"query-fsmonitor\")))\n+\t\treturn -1;\n+\n+\tsnprintf(ver, sizeof(version), \"%d\", version);\n+\tsnprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update);\n+\targv[1] = ver;\n+\targv[2] = date;\n+\targv[3] = NULL;\n+\tcp.argv = argv;\n+\tcp.out = -1;\n+\n+\treturn capture_command(&cp, query_result, 1024);\n+}\n+\n+static void mark_file_dirty(struct index_state *istate, const char *name)\n+{\n+\tstruct untracked_cache_dir *dir;\n+\tint pos;\n+\n+\t/* find it in the index and mark that entry as dirty */\n+\tpos = index_name_pos(istate, name, strlen(name));\n+\tif (pos >= 0) {\n+\t\tif (!(istate->cache[pos]->ce_flags & CE_FSMONITOR_DIRTY)) {\n+\t\t\tistate->cache[pos]->ce_flags |= CE_FSMONITOR_DIRTY;\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Find the corresponding directory in the untracked cache\n+\t * and mark it as invalid\n+\t */\n+\tif (!istate->untracked || !istate->untracked->root)\n+\t\treturn;\n+\n+\tdir = find_untracked_cache_dir(istate->untracked, istate->untracked->root, name);\n+\tif (dir) {\n+\t\tif (dir->valid) {\n+\t\t\tdir->valid = 0;\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\t}\n+\t}\n+}\n+\n+void refresh_by_fsmonitor(struct index_state *istate)\n+{\n+\tstatic int has_run_once = 0;\n+\tstruct strbuf query_result = STRBUF_INIT;\n+\tint query_success = 0;\n+\tsize_t bol = 0; /* beginning of line */\n+\tuint64_t last_update;\n+\tchar *buf, *entry;\n+\tint i;\n+\n+\tif (!core_fsmonitor || has_run_once)\n+\t\treturn;\n+\thas_run_once = 1;\n+\n+\t/*\n+\t * This could be racy so save the date/time now and the hook\n+\t * should be inclusive to ensure we don't miss potential changes.\n+\t */\n+\tlast_update = getnanotime();\n+\n+\t/*\n+\t * If we have a last update time, call query-monitor for the set of\n+\t * changes since that time.\n+\t */\n+\tif (istate->fsmonitor_last_update) {\n+\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION,\n+\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\ttrace_performance_since(last_update, \"query-fsmonitor\");\n+\t}\n+\n+\tif (query_success) {\n+\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\tbuf = entry = query_result.buf;\n+\t\tfor (i = 0; i < query_result.len; i++) {\n+\t\t\tif (buf[i] != '\\0')\n+\t\t\t\tcontinue;\n+\t\t\tmark_file_dirty(istate, buf + bol);\n+\t\t\tbol = i + 1;\n+\t\t}\n+\t\tif (bol < query_result.len)\n+\t\t\tmark_file_dirty(istate, buf + bol);\n+\n+\t\t/* Mark all clean entries up-to-date */\n+\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\t\tif (ce_stage(ce) || (ce->ce_flags & CE_FSMONITOR_DIRTY))\n+\t\t\t\tcontinue;\n+\t\t\tce_mark_uptodate(ce);\n+\t\t}\n+\n+\t\t/*\n+\t\t * Now that we've marked the invalid entries in the\n+\t\t * untracked-cache itself, we can mark the untracked cache for\n+\t\t * fsmonitor usage.\n+\t\t */\n+\t\tif (istate->untracked)\n+\t\t\tistate->untracked->use_fsmonitor = 1;\n+\t} else {\n+\t\t/* if we can't update the cache, fall back to checking them all */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_DIRTY;\n+\n+\t\t/* mark the untracked cache as unusable for fsmonitor */\n+\t\tif (istate->untracked)\n+\t\t\tistate->untracked->use_fsmonitor = 0;\n+\t}\n+\tstrbuf_release(&query_result);\n+\n+\t/* Now that we've updated istate, save the last_update time */\n+\tistate->fsmonitor_last_update = last_update;\n+}\ndiff --git a/fsmonitor.h b/fsmonitor.h\nnew file mode 100644\nindex 0000000000..9c1e2b480f\n--- /dev/null\n+++ b/fsmonitor.h\n@@ -0,0 +1,9 @@\n+#ifndef FSMONITOR_H\n+#define FSMONITOR_H\n+\n+int read_fsmonitor_extension(struct index_state *istate, const void *data, unsigned long sz);\n+void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate);\n+void tweak_fsmonitor_extension(struct index_state *istate);\n+void refresh_by_fsmonitor(struct index_state *istate);\n+\n+#endif\ndiff --git a/read-cache.c b/read-cache.c\nindex bc156a133e..68a1580c96 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -18,6 +18,7 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"utf8.h\"\n+#include \"fsmonitor.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -37,11 +38,12 @@\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n #define CACHE_EXT_UNTRACKED 0x554E5452\t /* \"UNTR\" */\n+#define CACHE_EXT_FSMONITOR 0x46534D4E\t /* \"FSMN\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -61,6 +63,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \tfree(old);\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n }\n \n@@ -777,6 +780,7 @@ int chmod_index_entry(struct index_state *istate, struct cache_entry *ce,\n \t}\n \tcache_tree_invalidate_path(istate, ce->name);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n \n \treturn 0;\n@@ -1344,6 +1348,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tconst char *added_fmt;\n \tconst char *unmerged_fmt;\n \n+\trefresh_by_fsmonitor(istate);\n+\n \tmodified_fmt = (in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\");\n \tdeleted_fmt = (in_porcelain ? \"D\\t%s\\n\" : \"%s: needs update\\n\");\n \ttypechange_fmt = (in_porcelain ? \"T\\t%s\\n\" : \"%s needs update\\n\");\n@@ -1380,8 +1386,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\tcontinue;\n \n \t\tnew = refresh_cache_ent(istate, ce, options, &cache_errno, &changed);\n-\t\tif (new == ce)\n+\t\tif (new == ce) {\n+\t\t\tce->ce_flags &= ~CE_FSMONITOR_DIRTY;\n \t\t\tcontinue;\n+\t\t}\n+\n \t\tif (!new) {\n \t\t\tconst char *fmt;\n \n@@ -1391,6 +1400,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\t\t */\n \t\t\t\tce->ce_flags &= ~CE_VALID;\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\t\t\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \t\t\t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t\t\t}\n \t\t\tif (quiet)\n@@ -1549,6 +1559,9 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_UNTRACKED:\n \t\tistate->untracked = read_untracked_extension(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_FSMONITOR:\n+\t\tread_fsmonitor_extension(istate, data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1721,6 +1734,7 @@ static void post_read_index_from(struct index_state *istate)\n \tcheck_ce_order(istate);\n \ttweak_untracked_cache(istate);\n \ttweak_split_index(istate);\n+\ttweak_fsmonitor_extension(istate);\n }\n \n /* remember to discard_cache() before reading a different cache! */\n@@ -2295,6 +2309,16 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->fsmonitor_last_update) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_fsmonitor_extension(&sb, istate);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_FSMONITOR, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1))\n \t\treturn -1;\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d38c37e38c..0ebc505b6c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -417,6 +417,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n \tif (was_skip_worktree != ce_skip_worktree(ce)) {\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tce->ce_flags |= CE_FSMONITOR_DIRTY;\n \t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \n-- \n2.13.0\n\n"},{"id":"321924","messageId":"20170610134026.104552-7-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170610134026.104552-1-benpeart@microsoft.com","subject":"[PATCH v5 6/7] fsmonitor: add a sample query-fsmonitor hook script for Watchman","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-10T13:40:25Z","receivedAt":"2017-06-10T13:41:10Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This hook script integrates the new fsmonitor capabilities of git with\nthe cross platform Watchman file watching service. To use the script:\n\nDownload and install Watchman from https://facebook.github.io/watchman/\nand instruct Watchman to watch your working directory for changes\n('watchman watch-project /usr/src/git').\n\nRename the sample integration hook from query-fsmonitor.sample to\nquery-fsmonitor.\n\nConfigure git to use the extension ('git config core.fsmonitor true')\nand optionally turn on the untracked cache for optimal performance\n('git config core.untrackedcache true').\n\nSigned-off-by: Ben Peart \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n templates/hooks--query-fsmonitor.sample | 76 +++++++++++++++++++++++++++++++++\n 1 file changed, 76 insertions(+)\n create mode 100755 templates/hooks--query-fsmonitor.sample\n\ndiff --git a/templates/hooks--query-fsmonitor.sample b/templates/hooks--query-fsmonitor.sample\nnew file mode 100755\nindex 0000000000..8d05b87a90\n--- /dev/null\n+++ b/templates/hooks--query-fsmonitor.sample\n@@ -0,0 +1,76 @@\n+#!/bin/sh\n+#\n+# An example hook script to integrate Watchman\n+# (https://facebook.github.io/watchman/) with git to provide fast\n+# git status.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+# To enable this hook, rename this file to \"query-fsmonitor\"\n+\n+# check the hook interface version\n+if [ \"$1\" -eq 1 ]\n+then\n+\t# convert nanoseconds to seconds\n+\ttime_t=$(($2/1000000000))\n+else\n+\techo -e \"Unsupported query-fsmonitor hook version.\\nFalling back to scanning...\\n\" >&2\n+\texit 1;\n+fi\n+\n+# Convert unix style paths to what Watchman expects\n+case \"$(uname -s)\" in\n+MINGW*|MSYS_NT*)\n+ GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,/,g')\"\n+ ;;\n+*)\n+ GIT_WORK_TREE=\"$PWD\"\n+ ;;\n+esac\n+\n+# In the query expression below we're asking for names of files that\n+# changed since $time_t but were not transient (ie created after\n+# $time_t but no longer exist).\n+#\n+# To accomplish this, we're using the \"since\" generator to use the\n+# recency index to select candidate nodes and \"fields\" to limit the\n+# output to file names only. Then we're using the \"expression\" term to\n+# further constrain the results.\n+#\n+# The category of transient files that we want to ignore will have a\n+# creation clock (cclock) newer than $time_t value and will also not\n+# currently exist.\n+\n+echo \"[\\\"query\\\", \\\"$GIT_WORK_TREE\\\", { \\\n+\t\\\"since\\\": $time_t, \\\n+\t\\\"fields\\\": [\\\"name\\\"], \\\n+\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time_t, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n+\t}]\" | \\\n+\twatchman -j |\n+\tperl -0666 -e '\n+\t\tuse strict;\n+\t\tuse warnings;\n+\n+\t\tmy $stdin = <>;\n+\t\tdie \"Watchman: command returned no output.\\nFalling back to scanning...\\n\" if $stdin eq \"\";\n+\t\tdie \"Watchman: command returned invalid output: $stdin\\nFalling back to scanning...\\n\" unless $stdin =~ /^\\{/;\n+\n+\t\tmy $json_pkg;\n+\t\teval {\n+\t\t\trequire JSON::XS;\n+\t\t\t$json_pkg = \"JSON::XS\";\n+\t\t\t1;\n+\t\t} or do {\n+\t\t\trequire JSON::PP;\n+\t\t\t$json_pkg = \"JSON::PP\";\n+\t\t};\n+\n+\t\tmy $o = $json_pkg->new->utf8->decode($stdin);\n+\t\tdie \"Watchman: $o->{error}.\\nFalling back to scanning...\\n\" if $o->{error};\n+\n+\t\tlocal $, = \"\\0\";\n+\t\tprint @{$o->{files}};\n+\t'\n-- \n2.13.0\n\n"},{"id":"321925","messageId":"20170610134026.104552-8-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170610134026.104552-1-benpeart@microsoft.com","subject":"[PATCH v5 7/7] fsmonitor: add a performance test","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-10T13:40:26Z","receivedAt":"2017-06-10T13:41:12Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a test utility (test-drop-caches) that enables dropping the file\nsystem cache on Windows.\n\nAdd a perf test (p7519-fsmonitor.sh) for fsmonitor.\n\nSigned-off-by: Ben Peart \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 1 +\n t/helper/test-drop-caches.c | 107 +++++++++++++++++++++++++++++\n t/perf/p7519-fsmonitor.sh | 161 ++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 269 insertions(+)\n create mode 100644 t/helper/test-drop-caches.c\n create mode 100755 t/perf/p7519-fsmonitor.sh\n\ndiff --git a/Makefile b/Makefile\nindex 992dd58801..893947839f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -648,6 +648,7 @@ TEST_PROGRAMS_NEED_X += test-subprocess\n TEST_PROGRAMS_NEED_X += test-svn-fe\n TEST_PROGRAMS_NEED_X += test-urlmatch-normalization\n TEST_PROGRAMS_NEED_X += test-wildmatch\n+TEST_PROGRAMS_NEED_X += test-drop-caches\n \n TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))\n \ndiff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\nnew file mode 100644\nindex 0000000000..80830d920b\n--- /dev/null\n+++ b/t/helper/test-drop-caches.c\n@@ -0,0 +1,107 @@\n+#include \"git-compat-util.h\"\n+#include \n+\n+typedef DWORD NTSTATUS;\n+\n+#ifdef GIT_WINDOWS_NATIVE\n+#include \n+\n+#define STATUS_SUCCESS\t\t\t(0x00000000L)\n+#define STATUS_PRIVILEGE_NOT_HELD\t(0xC0000061L)\n+\n+typedef enum _SYSTEM_INFORMATION_CLASS {\n+\tSystemMemoryListInformation = 80, // 80, q: SYSTEM_MEMORY_LIST_INFORMATION; s: SYSTEM_MEMORY_LIST_COMMAND (requires SeProfileSingleProcessPrivilege)\n+} SYSTEM_INFORMATION_CLASS;\n+\n+// private\n+typedef enum _SYSTEM_MEMORY_LIST_COMMAND\n+{\n+\tMemoryCaptureAccessedBits,\n+\tMemoryCaptureAndResetAccessedBits,\n+\tMemoryEmptyWorkingSets,\n+\tMemoryFlushModifiedList,\n+\tMemoryPurgeStandbyList,\n+\tMemoryPurgeLowPriorityStandbyList,\n+\tMemoryCommandMax\n+} SYSTEM_MEMORY_LIST_COMMAND;\n+\n+BOOL GetPrivilege(HANDLE TokenHandle, LPCSTR lpName, int flags)\n+{\n+\tBOOL bResult;\n+\tDWORD dwBufferLength;\n+\tLUID luid;\n+\tTOKEN_PRIVILEGES tpPreviousState;\n+\tTOKEN_PRIVILEGES tpNewState;\n+\n+\tdwBufferLength = 16;\n+\tbResult = LookupPrivilegeValueA(0, lpName, &luid);\n+\tif (bResult)\n+\t{\n+\t\ttpNewState.PrivilegeCount = 1;\n+\t\ttpNewState.Privileges[0].Luid = luid;\n+\t\ttpNewState.Privileges[0].Attributes = 0;\n+\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpNewState, (DWORD)((LPBYTE)&(tpNewState.Privileges[1]) - (LPBYTE)&tpNewState), &tpPreviousState, &dwBufferLength);\n+\t\tif (bResult)\n+\t\t{\n+\t\t\ttpPreviousState.PrivilegeCount = 1;\n+\t\t\ttpPreviousState.Privileges[0].Luid = luid;\n+\t\t\ttpPreviousState.Privileges[0].Attributes = flags != 0 ? 2 : 0;\n+\t\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpPreviousState, dwBufferLength, 0, 0);\n+\t\t}\n+\t}\n+\treturn bResult;\n+}\n+#endif\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tNTSTATUS status = 1;\n+#ifdef GIT_WINDOWS_NATIVE\n+\tHANDLE hProcess = GetCurrentProcess();\n+\tHANDLE hToken;\n+\tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n+\t{\n+\t\t_ftprintf(stderr, _T(\"Can't open current process token\\n\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (!GetPrivilege(hToken, \"SeProfileSingleProcessPrivilege\", 1))\n+\t{\n+\t\t_ftprintf(stderr, _T(\"Can't get SeProfileSingleProcessPrivilege\\n\"));\n+\t\treturn 1;\n+\t}\n+\n+\tCloseHandle(hToken);\n+\n+\tHMODULE ntdll = LoadLibrary(_T(\"ntdll.dll\"));\n+\tif (!ntdll)\n+\t{\n+\t\t_ftprintf(stderr, _T(\"Can't load ntdll.dll, wrong Windows version?\\n\"));\n+\t\treturn 1;\n+\t}\n+\n+\tNTSTATUS(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) = (NTSTATUS(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n+\tif (!NtSetSystemInformation)\n+\t{\n+\t\t_ftprintf(stderr, _T(\"Can't get function addresses, wrong Windows version?\\n\"));\n+\t\treturn 1;\n+\t}\n+\n+\tSYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;\n+\tstatus = NtSetSystemInformation(\n+\t\tSystemMemoryListInformation,\n+\t\t&command,\n+\t\tsizeof(SYSTEM_MEMORY_LIST_COMMAND)\n+\t);\n+\tif (status == STATUS_PRIVILEGE_NOT_HELD)\n+\t{\n+\t\t_ftprintf(stderr, _T(\"Insufficient privileges to execute the memory list command\"));\n+\t}\n+\telse if (status != STATUS_SUCCESS)\n+\t{\n+\t\t_ftprintf(stderr, _T(\"Unable to execute the memory list command %lX\"), status);\n+\t}\n+#endif\n+\n+\treturn status;\n+}\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nnew file mode 100755\nindex 0000000000..e41905cb9b\n--- /dev/null\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -0,0 +1,161 @@\n+#!/bin/sh\n+\n+test_description=\"Test core.fsmonitor\"\n+\n+. ./perf-lib.sh\n+\n+# This has to be run with GIT_PERF_REPEAT_COUNT=1 to generate valid results.\n+# Otherwise the caching that happens for the nth run will negate the validity\n+# of the comparisons.\n+if [ \"$GIT_PERF_REPEAT_COUNT\" -ne 1 ]\n+then\n+\techo \"warning: This test must be run with GIT_PERF_REPEAT_COUNT=1 to generate valid results.\" >&2\n+\techo \"warning: Setting GIT_PERF_REPEAT_COUNT=1\" >&2\n+\tGIT_PERF_REPEAT_COUNT=1\n+fi\n+\n+test_perf_large_repo\n+test_checkout_worktree\n+\n+# Convert unix style paths to what Watchman expects\n+case \"$(uname -s)\" in\n+MINGW*|MSYS_NT*)\n+ GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,/,g')\"\n+ ;;\n+*)\n+ GIT_WORK_TREE=\"$PWD\"\n+ ;;\n+esac\n+\n+# The big win for using fsmonitor is the elimination of the need to scan\n+# the working directory looking for changed files and untracked files. If\n+# the file information is all cached in RAM, the benefits are reduced.\n+\n+flush_disk_cache () {\n+\tcase \"$(uname -s)\" in\n+\tMINGW*|MSYS_NT*)\n+\t sync && test-drop-caches\n+\t ;;\n+\t*)\n+\t sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches\n+\t ;;\n+\tesac\n+\n+}\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_expect_success \"setup\" '\n+\t# Maybe set untrackedCache & splitIndex depending on the environment\n+\tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n+\tthen\n+\t\tgit config core.untrackedCache \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n+\telse\n+\t\tif test_have_prereq UNTRACKED_CACHE\n+\t\tthen\n+\t\t\tgit config core.untrackedCache true\n+\t\telse\n+\t\t\tgit config core.untrackedCache false\n+\t\tfi\n+\tfi &&\n+\n+\tif test -n \"$GIT_PERF_7519_SPLIT_INDEX\"\n+\tthen\n+\t\tgit config core.splitIndex \"$GIT_PERF_7519_SPLIT_INDEX\"\n+\tfi &&\n+\n+\t# Hook scaffolding\n+\tmkdir .git/hooks &&\n+\tcp ../../../templates/hooks--query-fsmonitor.sample .git/hooks/query-fsmonitor &&\n+\n+\t# have Watchman monitor the test folder\n+\twatchman watch \"$GIT_WORK_TREE\" &&\n+\twatchman watch-list | grep -q -F \"$GIT_WORK_TREE\"\n+'\n+\n+# Worst case without fsmonitor\n+test_expect_success \"clear fs cache\" '\n+\tgit config core.fsmonitor false &&\n+\tflush_disk_cache\n+'\n+test_perf \"status (fsmonitor=false, cold fs cache)\" '\n+\tgit status\n+'\n+\n+# Best case without fsmonitor\n+test_perf \"status (fsmonitor=false, warm fs cache)\" '\n+\tgit status\n+'\n+\n+# Let's see if -uno & -uall make any difference\n+test_expect_success \"clear fs cache\" '\n+\tflush_disk_cache\n+'\n+test_perf \"status -uno (fsmonitor=false, cold fs cache)\" '\n+\tgit status -uno\n+'\n+\n+test_expect_success \"clear fs cache\" '\n+\tflush_disk_cache\n+'\n+test_perf \"status -uall (fsmonitor=false, cold fs cache)\" '\n+\tgit status -uall\n+'\n+\n+# The first run with core.fsmonitor=true has to do a normal scan and write\n+# out the index extension.\n+test_expect_success \"populate extension\" '\n+\t# core.preloadIndex defeats the benefits of core.fsMonitor as it\n+\t# calls lstat for the index entries. Turn it off as _not_ doing\n+\t# the work is faster than doing the work across multiple threads.\n+\tgit config core.fsmonitor true &&\n+\tgit config core.preloadIndex false &&\n+\tgit status\n+'\n+\n+# Worst case with fsmonitor\n+test_expect_success \"shutdown fsmonitor, clear fs cache\" '\n+\twatchman shutdown-server &&\n+\tflush_disk_cache\n+'\n+test_perf \"status (fsmonitor=true, cold fs cache, cold fsmonitor)\" '\n+\tgit status\n+'\n+\n+# Best case with fsmonitor\n+test_perf \"status (fsmonitor=true, warm fs cache, warm fsmonitor)\" '\n+\tgit status\n+'\n+\n+# Best improved with fsmonitor (compare to worst case without fsmonitor)\n+test_expect_success \"clear fs cache\" '\n+\tflush_disk_cache\n+'\n+test_perf \"status (fsmonitor=true, cold fs cache, warm fsmonitor)\" '\n+\tgit status\n+'\n+\n+# Let's see if -uno & -uall make any difference\n+test_expect_success \"clear fs cache\" '\n+\tflush_disk_cache\n+'\n+test_perf \"status -uno (fsmonitor=true, cold fs cache)\" '\n+\tgit status -uno\n+'\n+\n+test_expect_success \"clear fs cache\" '\n+\tflush_disk_cache\n+'\n+test_perf \"status -uall (fsmonitor=true, cold fs cache)\" '\n+\tgit status -uall\n+'\n+\n+test_expect_success \"cleanup\" '\n+\twatchman watch-del \"$GIT_WORK_TREE\" &&\n+\twatchman shutdown-server\n+'\n+\n+test_done\n-- \n2.13.0\n\n"},{"id":"321926","messageId":"20170610134026.104552-6-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170610134026.104552-1-benpeart@microsoft.com","subject":"[PATCH v5 5/7] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-10T13:40:24Z","receivedAt":"2017-06-10T13:41:19Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This includes the core.fsmonitor setting, the query-fsmonitor hook,\nand the fsmonitor index extension.\n\nSigned-off-by: Ben Peart \n---\n Documentation/config.txt | 7 +++++++\n Documentation/githooks.txt | 23 +++++++++++++++++++++++\n Documentation/technical/index-format.txt | 19 +++++++++++++++++++\n 3 files changed, 49 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dd4beec39d..d883e3318c 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -410,6 +410,13 @@ core.protectNTFS::\n \t8.3 \"short\" names.\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n+core.fsmonitor::\n+\tIf set to true, call the query-fsmonitor hook proc which will\n+\tidentify all files that may have had changes since the last\n+\trequest. This information is used to speed up operations like\n+\t'git commit' and 'git status' by limiting what git must scan to\n+\tdetect changes.\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex b2514f4d44..219786b2da 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -456,6 +456,29 @@ non-zero status causes 'git send-email' to abort before sending any\n e-mails.\n \n \n+[[query-fsmonitor]]\n+query-fsmonitor\n+~~~~~~~~~~~~~~~\n+\n+This hook is invoked when the configuration option core.fsmonitor is\n+set and git needs to identify changed or untracked files. It takes\n+two arguments, a version (currently 1) and the time in elapsed\n+nanoseconds since midnight, January 1, 1970.\n+\n+The hook should output to stdout the list of all files in the working\n+directory that may have changed since the requested time. The logic\n+should be inclusive so that it does not miss any potential changes.\n+The paths should be relative to the root of the working directory\n+and be separated by a single NUL.\n+\n+Git will limit what files it checks for changes as well as which\n+directories are checked for untracked files based on the path names\n+given.\n+\n+The exit status determines whether git will use the data from the\n+hook to limit its search. On error, it will fall back to verifying\n+all files and folders.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex ade0b0c445..7aeeea6f94 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -295,3 +295,22 @@ The remaining data of each directory block is grouped by type:\n in the previous ewah bitmap.\n \n - One NUL.\n+\n+== File System Monitor cache\n+\n+ The file system monitor cache tracks files for which the query-fsmonitor\n+ hook has told us about changes. The signature for this extension is\n+ { 'F', 'S', 'M', 'N' }.\n+\n+ The extension starts with\n+\n+ - 32-bit version number: the current supported version is 1.\n+\n+ - 64-bit time: the extension data reflects all changes through the given\n+\ttime which is stored as the nanoseconds elapsed since midnight,\n+\tJanuary 1, 1970.\n+\n+ - 32-bit bitmap size: the size of the CE_FSMONITOR_DIRTY bitmap.\n+\n+ - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n+ is CE_FSMONITOR_DIRTY.\n-- \n2.13.0\n\n"},{"id":"321927","messageId":"5d0d07a8-c47c-e596-5ea2-4ca1e041f77a@gmail.com","threadId":"46154","inReplyTo":"20170610134026.104552-8-benpeart@microsoft.com","subject":"Re: [PATCH v5 7/7] fsmonitor: add a performance test","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-10T14:04:29Z","receivedAt":"2017-06-10T14:06:04Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Here are some perf test results for repos of various size generated with \nmany-files.sh.\n\nComparing cold fs cache times on a fast SSD running Windows:\n\n# files\t\tpreloadindex\tfsmonitor\treduction\n=========================================================\n10,000\t\t0.69\t\t0.46\t\t33%\n100,000\t\t3.75\t\t0.7\t\t81%\n1,000,000\t35.07\t\t3.24\t\t91%\n\n\n\n10,000 files\nGIT_PERF_REPEAT_COUNT=1 \nGIT_PERF_LARGE_REPO=/c/Repos/gen-many-files-3.10.9.git/ ./run \np7519-fsmonitor.sh\nTest this tree\n---------------------------------------------------------------------------------\n7519.3: status (fsmonitor=false, cold fs cache) \n0.69(0.03+0.06)\n7519.4: status (fsmonitor=false, warm fs cache) \n0.45(0.01+0.06)\n7519.6: status -uno (fsmonitor=false, cold fs cache) \n0.53(0.03+0.04)\n7519.8: status -uall (fsmonitor=false, cold fs cache) \n0.57(0.04+0.01)\n7519.11: status (fsmonitor=true, cold fs cache, cold fsmonitor) \n2.16(0.01+0.06)\n7519.12: status (fsmonitor=true, warm fs cache, warm fsmonitor) \n0.36(0.01+0.07)\n7519.14: status (fsmonitor=true, cold fs cache, warm fsmonitor) \n0.46(0.04+0.06)\n7519.16: status -uno (fsmonitor=true, cold fs cache) \n0.46(0.03+0.04)\n7519.18: status -uall (fsmonitor=true, cold fs cache) \n0.76(0.04+0.04)\n\n100,000 files\nGIT_PERF_REPEAT_COUNT=1 \nGIT_PERF_LARGE_REPO=/c/Repos/gen-many-files-4.10.9.git/ ./run \np7519-fsmonitor.sh\nTest this tree\n----------------------------------------------------------------------------------\n7519.3: status (fsmonitor=false, cold fs cache) \n3.75(0.01+0.04)\n7519.4: status (fsmonitor=false, warm fs cache) \n2.74(0.06+0.06)\n7519.6: status -uno (fsmonitor=false, cold fs cache) \n2.88(0.01+0.06)\n7519.8: status -uall (fsmonitor=false, cold fs cache) \n3.24(0.00+0.06)\n7519.11: status (fsmonitor=true, cold fs cache, cold fsmonitor) \n17.90(0.00+0.10)\n7519.12: status (fsmonitor=true, warm fs cache, warm fsmonitor) \n0.59(0.00+0.09)\n7519.14: status (fsmonitor=true, cold fs cache, warm fsmonitor) \n0.70(0.00+0.07)\n7519.16: status -uno (fsmonitor=true, cold fs cache) \n0.69(0.01+0.01)\n7519.18: status -uall (fsmonitor=true, cold fs cache) \n4.51(0.00+0.06)\n\n1,000,000 files\nGIT_PERF_REPEAT_COUNT=1 \nGIT_PERF_LARGE_REPO=/c/Repos/gen-many-files-5.10.9.git/ ./run \np7519-fsmonitor.sh\nTest this tree\n-----------------------------------------------------------------------------------\n7519.3: status (fsmonitor=false, cold fs cache) \n35.07(0.01+0.03)\n7519.4: status (fsmonitor=false, warm fs cache) \n26.58(0.03+0.07)\n7519.6: status -uno (fsmonitor=false, cold fs cache) \n26.46(0.03+0.06)\n7519.8: status -uall (fsmonitor=false, cold fs cache) \n31.55(0.01+0.03)\n7519.11: status (fsmonitor=true, cold fs cache, cold fsmonitor) \n193.15(0.01+0.04)\n7519.12: status (fsmonitor=true, warm fs cache, warm fsmonitor) \n3.03(0.01+0.07)\n7519.14: status (fsmonitor=true, cold fs cache, warm fsmonitor) \n3.24(0.01+0.04)\n7519.16: status -uno (fsmonitor=true, cold fs cache) \n2.99(0.03+0.03)\n7519.18: status -uall (fsmonitor=true, cold fs cache) \n35.07(0.03+0.07)\n\n\nOn 6/10/2017 9:40 AM, Ben Peart wrote:\n> Add a test utility (test-drop-caches) that enables dropping the file\n> system cache on Windows.\n> \n> Add a perf test (p7519-fsmonitor.sh) for fsmonitor.\n> \n> Signed-off-by: Ben Peart \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> Makefile | 1 +\n> t/helper/test-drop-caches.c | 107 +++++++++++++++++++++++++++++\n> t/perf/p7519-fsmonitor.sh | 161 ++++++++++++++++++++++++++++++++++++++++++++\n> 3 files changed, 269 insertions(+)\n> create mode 100644 t/helper/test-drop-caches.c\n> create mode 100755 t/perf/p7519-fsmonitor.sh\n> \n> diff --git a/Makefile b/Makefile\n> index 992dd58801..893947839f 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -648,6 +648,7 @@ TEST_PROGRAMS_NEED_X += test-subprocess\n> TEST_PROGRAMS_NEED_X += test-svn-fe\n> TEST_PROGRAMS_NEED_X += test-urlmatch-normalization\n> TEST_PROGRAMS_NEED_X += test-wildmatch\n> +TEST_PROGRAMS_NEED_X += test-drop-caches\n> \n> TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))\n> \n> diff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\n> new file mode 100644\n> index 0000000000..80830d920b\n> --- /dev/null\n> +++ b/t/helper/test-drop-caches.c\n> @@ -0,0 +1,107 @@\n> +#include \"git-compat-util.h\"\n> +#include \n> +\n> +typedef DWORD NTSTATUS;\n> +\n> +#ifdef GIT_WINDOWS_NATIVE\n> +#include \n> +\n> +#define STATUS_SUCCESS\t\t\t(0x00000000L)\n> +#define STATUS_PRIVILEGE_NOT_HELD\t(0xC0000061L)\n> +\n> +typedef enum _SYSTEM_INFORMATION_CLASS {\n> +\tSystemMemoryListInformation = 80, // 80, q: SYSTEM_MEMORY_LIST_INFORMATION; s: SYSTEM_MEMORY_LIST_COMMAND (requires SeProfileSingleProcessPrivilege)\n> +} SYSTEM_INFORMATION_CLASS;\n> +\n> +// private\n> +typedef enum _SYSTEM_MEMORY_LIST_COMMAND\n> +{\n> +\tMemoryCaptureAccessedBits,\n> +\tMemoryCaptureAndResetAccessedBits,\n> +\tMemoryEmptyWorkingSets,\n> +\tMemoryFlushModifiedList,\n> +\tMemoryPurgeStandbyList,\n> +\tMemoryPurgeLowPriorityStandbyList,\n> +\tMemoryCommandMax\n> +} SYSTEM_MEMORY_LIST_COMMAND;\n> +\n> +BOOL GetPrivilege(HANDLE TokenHandle, LPCSTR lpName, int flags)\n> +{\n> +\tBOOL bResult;\n> +\tDWORD dwBufferLength;\n> +\tLUID luid;\n> +\tTOKEN_PRIVILEGES tpPreviousState;\n> +\tTOKEN_PRIVILEGES tpNewState;\n> +\n> +\tdwBufferLength = 16;\n> +\tbResult = LookupPrivilegeValueA(0, lpName, &luid);\n> +\tif (bResult)\n> +\t{\n> +\t\ttpNewState.PrivilegeCount = 1;\n> +\t\ttpNewState.Privileges[0].Luid = luid;\n> +\t\ttpNewState.Privileges[0].Attributes = 0;\n> +\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpNewState, (DWORD)((LPBYTE)&(tpNewState.Privileges[1]) - (LPBYTE)&tpNewState), &tpPreviousState, &dwBufferLength);\n> +\t\tif (bResult)\n> +\t\t{\n> +\t\t\ttpPreviousState.PrivilegeCount = 1;\n> +\t\t\ttpPreviousState.Privileges[0].Luid = luid;\n> +\t\t\ttpPreviousState.Privileges[0].Attributes = flags != 0 ? 2 : 0;\n> +\t\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpPreviousState, dwBufferLength, 0, 0);\n> +\t\t}\n> +\t}\n> +\treturn bResult;\n> +}\n> +#endif\n> +\n> +int cmd_main(int argc, const char **argv)\n> +{\n> +\tNTSTATUS status = 1;\n> +#ifdef GIT_WINDOWS_NATIVE\n> +\tHANDLE hProcess = GetCurrentProcess();\n> +\tHANDLE hToken;\n> +\tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Can't open current process token\\n\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tif (!GetPrivilege(hToken, \"SeProfileSingleProcessPrivilege\", 1))\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Can't get SeProfileSingleProcessPrivilege\\n\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tCloseHandle(hToken);\n> +\n> +\tHMODULE ntdll = LoadLibrary(_T(\"ntdll.dll\"));\n> +\tif (!ntdll)\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Can't load ntdll.dll, wrong Windows version?\\n\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tNTSTATUS(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) = (NTSTATUS(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n> +\tif (!NtSetSystemInformation)\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Can't get function addresses, wrong Windows version?\\n\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tSYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;\n> +\tstatus = NtSetSystemInformation(\n> +\t\tSystemMemoryListInformation,\n> +\t\t&command,\n> +\t\tsizeof(SYSTEM_MEMORY_LIST_COMMAND)\n> +\t);\n> +\tif (status == STATUS_PRIVILEGE_NOT_HELD)\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Insufficient privileges to execute the memory list command\"));\n> +\t}\n> +\telse if (status != STATUS_SUCCESS)\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Unable to execute the memory list command %lX\"), status);\n> +\t}\n> +#endif\n> +\n> +\treturn status;\n> +}\n> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n> new file mode 100755\n> index 0000000000..e41905cb9b\n> --- /dev/null\n> +++ b/t/perf/p7519-fsmonitor.sh\n> @@ -0,0 +1,161 @@\n> +#!/bin/sh\n> +\n> +test_description=\"Test core.fsmonitor\"\n> +\n> +. ./perf-lib.sh\n> +\n> +# This has to be run with GIT_PERF_REPEAT_COUNT=1 to generate valid results.\n> +# Otherwise the caching that happens for the nth run will negate the validity\n> +# of the comparisons.\n> +if [ \"$GIT_PERF_REPEAT_COUNT\" -ne 1 ]\n> +then\n> +\techo \"warning: This test must be run with GIT_PERF_REPEAT_COUNT=1 to generate valid results.\" >&2\n> +\techo \"warning: Setting GIT_PERF_REPEAT_COUNT=1\" >&2\n> +\tGIT_PERF_REPEAT_COUNT=1\n> +fi\n> +\n> +test_perf_large_repo\n> +test_checkout_worktree\n> +\n> +# Convert unix style paths to what Watchman expects\n> +case \"$(uname -s)\" in\n> +MINGW*|MSYS_NT*)\n> + GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,/,g')\"\n> + ;;\n> +*)\n> + GIT_WORK_TREE=\"$PWD\"\n> + ;;\n> +esac\n> +\n> +# The big win for using fsmonitor is the elimination of the need to scan\n> +# the working directory looking for changed files and untracked files. If\n> +# the file information is all cached in RAM, the benefits are reduced.\n> +\n> +flush_disk_cache () {\n> +\tcase \"$(uname -s)\" in\n> +\tMINGW*|MSYS_NT*)\n> +\t sync && test-drop-caches\n> +\t ;;\n> +\t*)\n> +\t sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches\n> +\t ;;\n> +\tesac\n> +\n> +}\n> +\n> +test_lazy_prereq UNTRACKED_CACHE '\n> +\t{ git update-index --test-untracked-cache; ret=$?; } &&\n> +\ttest $ret -ne 1\n> +'\n> +\n> +test_expect_success \"setup\" '\n> +\t# Maybe set untrackedCache & splitIndex depending on the environment\n> +\tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n> +\tthen\n> +\t\tgit config core.untrackedCache \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n> +\telse\n> +\t\tif test_have_prereq UNTRACKED_CACHE\n> +\t\tthen\n> +\t\t\tgit config core.untrackedCache true\n> +\t\telse\n> +\t\t\tgit config core.untrackedCache false\n> +\t\tfi\n> +\tfi &&\n> +\n> +\tif test -n \"$GIT_PERF_7519_SPLIT_INDEX\"\n> +\tthen\n> +\t\tgit config core.splitIndex \"$GIT_PERF_7519_SPLIT_INDEX\"\n> +\tfi &&\n> +\n> +\t# Hook scaffolding\n> +\tmkdir .git/hooks &&\n> +\tcp ../../../templates/hooks--query-fsmonitor.sample .git/hooks/query-fsmonitor &&\n> +\n> +\t# have Watchman monitor the test folder\n> +\twatchman watch \"$GIT_WORK_TREE\" &&\n> +\twatchman watch-list | grep -q -F \"$GIT_WORK_TREE\"\n> +'\n> +\n> +# Worst case without fsmonitor\n> +test_expect_success \"clear fs cache\" '\n> +\tgit config core.fsmonitor false &&\n> +\tflush_disk_cache\n> +'\n> +test_perf \"status (fsmonitor=false, cold fs cache)\" '\n> +\tgit status\n> +'\n> +\n> +# Best case without fsmonitor\n> +test_perf \"status (fsmonitor=false, warm fs cache)\" '\n> +\tgit status\n> +'\n> +\n> +# Let's see if -uno & -uall make any difference\n> +test_expect_success \"clear fs cache\" '\n> +\tflush_disk_cache\n> +'\n> +test_perf \"status -uno (fsmonitor=false, cold fs cache)\" '\n> +\tgit status -uno\n> +'\n> +\n> +test_expect_success \"clear fs cache\" '\n> +\tflush_disk_cache\n> +'\n> +test_perf \"status -uall (fsmonitor=false, cold fs cache)\" '\n> +\tgit status -uall\n> +'\n> +\n> +# The first run with core.fsmonitor=true has to do a normal scan and write\n> +# out the index extension.\n> +test_expect_success \"populate extension\" '\n> +\t# core.preloadIndex defeats the benefits of core.fsMonitor as it\n> +\t# calls lstat for the index entries. Turn it off as _not_ doing\n> +\t# the work is faster than doing the work across multiple threads.\n> +\tgit config core.fsmonitor true &&\n> +\tgit config core.preloadIndex false &&\n> +\tgit status\n> +'\n> +\n> +# Worst case with fsmonitor\n> +test_expect_success \"shutdown fsmonitor, clear fs cache\" '\n> +\twatchman shutdown-server &&\n> +\tflush_disk_cache\n> +'\n> +test_perf \"status (fsmonitor=true, cold fs cache, cold fsmonitor)\" '\n> +\tgit status\n> +'\n> +\n> +# Best case with fsmonitor\n> +test_perf \"status (fsmonitor=true, warm fs cache, warm fsmonitor)\" '\n> +\tgit status\n> +'\n> +\n> +# Best improved with fsmonitor (compare to worst case without fsmonitor)\n> +test_expect_success \"clear fs cache\" '\n> +\tflush_disk_cache\n> +'\n> +test_perf \"status (fsmonitor=true, cold fs cache, warm fsmonitor)\" '\n> +\tgit status\n> +'\n> +\n> +# Let's see if -uno & -uall make any difference\n> +test_expect_success \"clear fs cache\" '\n> +\tflush_disk_cache\n> +'\n> +test_perf \"status -uno (fsmonitor=true, cold fs cache)\" '\n> +\tgit status -uno\n> +'\n> +\n> +test_expect_success \"clear fs cache\" '\n> +\tflush_disk_cache\n> +'\n> +test_perf \"status -uall (fsmonitor=true, cold fs cache)\" '\n> +\tgit status -uall\n> +'\n> +\n> +test_expect_success \"cleanup\" '\n> +\twatchman watch-del \"$GIT_WORK_TREE\" &&\n> +\twatchman shutdown-server\n> +'\n> +\n> +test_done\n> \n"},{"id":"322009","messageId":"xmqqo9tsn9qg.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170610134026.104552-8-benpeart@microsoft.com","subject":"Re: [PATCH v5 7/7] fsmonitor: add a performance test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-12T22:04:39Z","receivedAt":"2017-06-12T22:04:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> diff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\n> new file mode 100644\n> index 0000000000..80830d920b\n> --- /dev/null\n> +++ b/t/helper/test-drop-caches.c\n> @@ -0,0 +1,107 @@\n> +#include \"git-compat-util.h\"\n> +#include \n\nI thought compat-util should already include stdio?\n\n> +\n> +typedef DWORD NTSTATUS;\n\nIs this safe to have outside \"#ifdef GIT_WINDOWS_NATIVE\"?\n\n> +\n> +#ifdef GIT_WINDOWS_NATIVE\n> +#include \n> +\n> +#define STATUS_SUCCESS\t\t\t(0x00000000L)\n> +#define STATUS_PRIVILEGE_NOT_HELD\t(0xC0000061L)\n> +\n> +typedef enum _SYSTEM_INFORMATION_CLASS {\n> +\tSystemMemoryListInformation = 80, // 80, q: SYSTEM_MEMORY_LIST_INFORMATION; s: SYSTEM_MEMORY_LIST_COMMAND (requires SeProfileSingleProcessPrivilege)\n\nI would have said \"Please avoid // comment in this codebase unless\nwe know we only use the compilers that grok it\". This particular\none may be OK, as it is inside GIT_WINDOWS_NATIVE and I assume\neverybody on that platform uses recent GCC (or VStudio groks it I\nguess)?\n\n> +} SYSTEM_INFORMATION_CLASS;\n> +\n> +// private\n> +typedef enum _SYSTEM_MEMORY_LIST_COMMAND\n> +{\n\nStyle: '{' comes at the end of the previous line, with a single SP\nimmediately before it, unless it is the beginning of the function body.\n\nWhat you did for _SYSTEM_INFORMATION_CLASS above is correct.\n\n> +\tMemoryCaptureAccessedBits,\n> +\tMemoryCaptureAndResetAccessedBits,\n> +\tMemoryEmptyWorkingSets,\n> +\tMemoryFlushModifiedList,\n> +\tMemoryPurgeStandbyList,\n> +\tMemoryPurgeLowPriorityStandbyList,\n> +\tMemoryCommandMax\n\nStyle: avoid CamelCase.\n\n> +} SYSTEM_MEMORY_LIST_COMMAND;\n> +\n> +BOOL GetPrivilege(HANDLE TokenHandle, LPCSTR lpName, int flags)\n> +{\n> +\tBOOL bResult;\n> +\tDWORD dwBufferLength;\n> +\tLUID luid;\n> +\tTOKEN_PRIVILEGES tpPreviousState;\n> +\tTOKEN_PRIVILEGES tpNewState;\n> +\n> +\tdwBufferLength = 16;\n> +\tbResult = LookupPrivilegeValueA(0, lpName, &luid);\n> +\tif (bResult)\n> +\t{\n\nStyle: '{' comes at the end of the previous line, with a single SP\nimmediately before it, unless it is the beginning of the function body.\n\n> +\t\ttpNewState.PrivilegeCount = 1;\n> +\t\ttpNewState.Privileges[0].Luid = luid;\n> +\t\ttpNewState.Privileges[0].Attributes = 0;\n> +\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpNewState, (DWORD)((LPBYTE)&(tpNewState.Privileges[1]) - (LPBYTE)&tpNewState), &tpPreviousState, &dwBufferLength);\n> +\t\tif (bResult)\n> +\t\t{\n> +\t\t\ttpPreviousState.PrivilegeCount = 1;\n> +\t\t\ttpPreviousState.Privileges[0].Luid = luid;\n> +\t\t\ttpPreviousState.Privileges[0].Attributes = flags != 0 ? 2 : 0;\n> +\t\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpPreviousState, dwBufferLength, 0, 0);\n> +\t\t}\n> +\t}\n> +\treturn bResult;\n> +}\n> +#endif\n> +\n> +int cmd_main(int argc, const char **argv)\n> +{\n> +\tNTSTATUS status = 1;\n> +#ifdef GIT_WINDOWS_NATIVE\n> +\tHANDLE hProcess = GetCurrentProcess();\n> +\tHANDLE hToken;\n> +\tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Can't open current process token\\n\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tif (!GetPrivilege(hToken, \"SeProfileSingleProcessPrivilege\", 1))\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Can't get SeProfileSingleProcessPrivilege\\n\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tCloseHandle(hToken);\n> +\n> +\tHMODULE ntdll = LoadLibrary(_T(\"ntdll.dll\"));\n> +\tif (!ntdll)\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Can't load ntdll.dll, wrong Windows version?\\n\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tNTSTATUS(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) = (NTSTATUS(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n\nIs this \"decl-after-stmt\"? Avoid it.\n\n> +\tif (!NtSetSystemInformation)\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Can't get function addresses, wrong Windows version?\\n\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tSYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;\n> +\tstatus = NtSetSystemInformation(\n> +\t\tSystemMemoryListInformation,\n> +\t\t&command,\n> +\t\tsizeof(SYSTEM_MEMORY_LIST_COMMAND)\n> +\t);\n> +\tif (status == STATUS_PRIVILEGE_NOT_HELD)\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Insufficient privileges to execute the memory list command\"));\n> +\t}\n> +\telse if (status != STATUS_SUCCESS)\n> +\t{\n> +\t\t_ftprintf(stderr, _T(\"Unable to execute the memory list command %lX\"), status);\n> +\t}\n> +#endif\n> +\n> +\treturn status;\n> +}\n\nSo status is initialized to 1 and anybody without GIT_WINDOWS_NATIVE\nunconditionally get exit(1)?\n\nGiven that 'status' is the return value of this function that\nreturns 'int', I wonder if we need NTSTATUS type here.\n\nHaving said all that, I think you are using this ONLY on windows;\nperhaps it is better to drop #ifdef GIT_WINDOWS_NATIVE from all of\nthe above and arrange Makefile to build test-drop-cache only on that\nplatform, or something?\n\n\n> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n> new file mode 100755\n> index 0000000000..e41905cb9b\n> --- /dev/null\n> +++ b/t/perf/p7519-fsmonitor.sh\n> @@ -0,0 +1,161 @@\n> +#!/bin/sh\n> +\n> +test_description=\"Test core.fsmonitor\"\n> +\n> +. ./perf-lib.sh\n> +\n> +# This has to be run with GIT_PERF_REPEAT_COUNT=1 to generate valid results.\n> +# Otherwise the caching that happens for the nth run will negate the validity\n> +# of the comparisons.\n> +if [ \"$GIT_PERF_REPEAT_COUNT\" -ne 1 ]\n> +then\n\nStyle: \n\n\tif test \"$GIT_PERF_REPEAT_COUNT\" -ne 1\n\tthen\n\t\n> + ...\n> +test_expect_success \"setup\" '\n> +...\n> +\t# Hook scaffolding\n> +\tmkdir .git/hooks &&\n> +\tcp ../../../templates/hooks--query-fsmonitor.sample .git/hooks/query-fsmonitor &&\n\nDoes this assume $TRASH_DIRECTORY must be in $TEST_DIRECTORY/perf/?\nShouldn't t/perf/p[0-9][0-9][0-9][0-9]-*.sh scripts be capable of\nrunning with the --root= option? Perhaps take the copy\nrelative to $TEST_DIRECTORY?\n"},{"id":"322262","messageId":"9c1ed8d4-6bdb-e709-758d-4b010525e9e3@gmail.com","threadId":"46154","inReplyTo":"xmqqo9tsn9qg.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 7/7] fsmonitor: add a performance test","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-06-14T14:12:42Z","receivedAt":"2017-06-14T14:12:51Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 6/12/2017 6:04 PM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> diff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\n>> new file mode 100644\n>> index 0000000000..80830d920b\n>> --- /dev/null\n>> +++ b/t/helper/test-drop-caches.c\n>> @@ -0,0 +1,107 @@\n>> +#include \"git-compat-util.h\"\n>> +#include \n> \n> I thought compat-util should already include stdio?\n> \n>> +\n>> +typedef DWORD NTSTATUS;\n> \n> Is this safe to have outside \"#ifdef GIT_WINDOWS_NATIVE\"?\n\nI think it's safe but it isn't required. I remove it to be sure.\n\n> \n>> +\n>> +#ifdef GIT_WINDOWS_NATIVE\n>> +#include \n>> +\n>> +#define STATUS_SUCCESS\t\t\t(0x00000000L)\n>> +#define STATUS_PRIVILEGE_NOT_HELD\t(0xC0000061L)\n>> +\n>> +typedef enum _SYSTEM_INFORMATION_CLASS {\n>> +\tSystemMemoryListInformation = 80, // 80, q: SYSTEM_MEMORY_LIST_INFORMATION; s: SYSTEM_MEMORY_LIST_COMMAND (requires SeProfileSingleProcessPrivilege)\n> \n> I would have said \"Please avoid // comment in this codebase unless\n> we know we only use the compilers that grok it\". This particular\n> one may be OK, as it is inside GIT_WINDOWS_NATIVE and I assume\n> everybody on that platform uses recent GCC (or VStudio groks it I\n> guess)?\n> \n\nI'll remove them to be sure.\n\n>> +} SYSTEM_INFORMATION_CLASS;\n>> +\n>> +// private\n>> +typedef enum _SYSTEM_MEMORY_LIST_COMMAND\n>> +{\n> \n> Style: '{' comes at the end of the previous line, with a single SP\n> immediately before it, unless it is the beginning of the function body.\n> \n\nOld habits. :) Started writing Windows code and adopted their coding \nstyle without even thinking about it. I'll 'gitify' the style as much \nas possible. This should address the various style comments below.\n\n> What you did for _SYSTEM_INFORMATION_CLASS above is correct.\n> \n>> +\tMemoryCaptureAccessedBits,\n>> +\tMemoryCaptureAndResetAccessedBits,\n>> +\tMemoryEmptyWorkingSets,\n>> +\tMemoryFlushModifiedList,\n>> +\tMemoryPurgeStandbyList,\n>> +\tMemoryPurgeLowPriorityStandbyList,\n>> +\tMemoryCommandMax\n> \n> Style: avoid CamelCase.\n> \n> \n> So status is initialized to 1 and anybody without GIT_WINDOWS_NATIVE\n> unconditionally get exit(1)?\n> \n> Given that 'status' is the return value of this function that\n> returns 'int', I wonder if we need NTSTATUS type here.\n> \n> Having said all that, I think you are using this ONLY on windows;\n> perhaps it is better to drop #ifdef GIT_WINDOWS_NATIVE from all of\n> the above and arrange Makefile to build test-drop-cache only on that\n> platform, or something?\n> \n\nI didn't find any other examples of Windows only tools. I'll update the \n#ifdef to properly dump the file system cache on Linux as well and only \nerror out on other platforms.\n\n> \n>> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n>> new file mode 100755\n>> index 0000000000..e41905cb9b\n>> --- /dev/null\n>> +++ b/t/perf/p7519-fsmonitor.sh\n>> @@ -0,0 +1,161 @@\n>> +#!/bin/sh\n>> +\n>> +test_description=\"Test core.fsmonitor\"\n>> +\n>> +. ./perf-lib.sh\n>> +\n>> +# This has to be run with GIT_PERF_REPEAT_COUNT=1 to generate valid results.\n>> +# Otherwise the caching that happens for the nth run will negate the validity\n>> +# of the comparisons.\n>> +if [ \"$GIT_PERF_REPEAT_COUNT\" -ne 1 ]\n>> +then\n> \n> Style:\n> \n> \tif test \"$GIT_PERF_REPEAT_COUNT\" -ne 1\n> \tthen\n> \t\n>> + ...\n>> +test_expect_success \"setup\" '\n>> +...\n>> +\t# Hook scaffolding\n>> +\tmkdir .git/hooks &&\n>> +\tcp ../../../templates/hooks--query-fsmonitor.sample .git/hooks/query-fsmonitor &&\n> \n> Does this assume $TRASH_DIRECTORY must be in $TEST_DIRECTORY/perf/?\n> Shouldn't t/perf/p[0-9][0-9][0-9][0-9]-*.sh scripts be capable of\n> running with the --root= option? Perhaps take the copy\n> relative to $TEST_DIRECTORY?\n> \n\nI'll copy it relative to $TEST_DIRECTORY in the next iteration.\n\n"},{"id":"322281","messageId":"xmqqvany2z84.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"9c1ed8d4-6bdb-e709-758d-4b010525e9e3@gmail.com","subject":"Re: [PATCH v5 7/7] fsmonitor: add a performance test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-14T18:36:27Z","receivedAt":"2017-06-14T18:36:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n>> Having said all that, I think you are using this ONLY on windows;\n>> perhaps it is better to drop #ifdef GIT_WINDOWS_NATIVE from all of\n>> the above and arrange Makefile to build test-drop-cache only on that\n>> platform, or something?\n>\n> I didn't find any other examples of Windows only tools. I'll update\n> the #ifdef to properly dump the file system cache on Linux as well and\n> only error out on other platforms.\n\nIf this will become Windows-only, then I have no problem with\nplatform specfic typedef ;-) I have no problem with CamelCase,\neither, as that follows the local convention on the platform\n(similar to those in compat/* that are only for Windows).\n\nHaving said all that.\n\nAnother approach is to build this helper on all platforms, with\nsections protected by \"#ifdef LINUX\", \"#ifdef WINDOWS\", etc. That\nway, the platform detection and switching between running this\nprogram and echoing something into /proc filesystem performed in\np7519 can be removed (this test-helper program will be responsible\nto implement that logic instead). When p7519 wants to drop the\nfilesystem cache, regardless of the platform, it can call this\ntest-helper without having to know how the filesystem cache is\ndropped.\n\nI do not have strong preference either way, but I have a slight\nsuspicion that the \"another approach\" above may give us a better\nresult. For one thing, the test-helper can be reused in new perf\nscripts people will write in the future.\n\nThanks.\n"},{"id":"323339","messageId":"CAP8UFD1kL9i93HnnigYcB9U8u2dUijHj5djey5k=33VrFaWUBw@mail.gmail.com","threadId":"46154","inReplyTo":"20170610134026.104552-4-benpeart@microsoft.com","subject":"Re: [PATCH v5 3/7] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-06-27T15:43:07Z","receivedAt":"2017-06-27T15:43:13Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Jun 10, 2017 at 3:40 PM, Ben Peart wrote:\n\n> +int read_fsmonitor_extension(struct index_state *istate, const void *data,\n> + unsigned long sz)\n> +{\n> + const char *index = data;\n> + uint32_t hdr_version;\n> + uint32_t ewah_size;\n> + int ret;\n> +\n> + if (sz < sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t))\n> + return error(\"corrupt fsmonitor extension (too short)\");\n> +\n> + hdr_version = get_be32(index);\n\nHere we use get_be32(), ...\n\n> + index += sizeof(uint32_t);\n> + if (hdr_version != INDEX_EXTENSION_VERSION)\n> + return error(\"bad fsmonitor version %d\", hdr_version);\n> +\n> + istate->fsmonitor_last_update = get_be64(index);\n\n...get_be64(), ...\n\n> + index += sizeof(uint64_t);\n> +\n> + ewah_size = get_be32(index);\n\n... and get_be32 again, ...\n\n> + index += sizeof(uint32_t);\n> +\n> + istate->fsmonitor_dirty = ewah_new();\n> + ret = ewah_read_mmap(istate->fsmonitor_dirty, index, ewah_size);\n> + if (ret != ewah_size) {\n> + ewah_free(istate->fsmonitor_dirty);\n> + istate->fsmonitor_dirty = NULL;\n> + return error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n> +{\n> + uint32_t hdr_version;\n> + uint64_t tm;\n> + struct ewah_bitmap *bitmap;\n> + int i;\n> + uint32_t ewah_start;\n> + uint32_t ewah_size = 0;\n> + int fixup = 0;\n> +\n> + hdr_version = htonl(INDEX_EXTENSION_VERSION);\n\n... but here we use htonl() instead of put_be32(), ...\n\n> + strbuf_add(sb, &hdr_version, sizeof(uint32_t));\n> +\n> + tm = htonll((uint64_t)istate->fsmonitor_last_update);\n\n... htonll(), ...\n\n> + strbuf_add(sb, &tm, sizeof(uint64_t));\n> + fixup = sb->len;\n> + strbuf_add(sb, &ewah_size, sizeof(uint32_t)); /* we'll fix this up later */\n> +\n> + ewah_start = sb->len;\n> + bitmap = ewah_new();\n> + for (i = 0; i < istate->cache_nr; i++)\n> + if (istate->cache[i]->ce_flags & CE_FSMONITOR_DIRTY)\n> + ewah_set(bitmap, i);\n> + ewah_serialize_strbuf(bitmap, sb);\n> + ewah_free(bitmap);\n> +\n> + /* fix up size field */\n> + ewah_size = htonl(sb->len - ewah_start);\n\n... and htonl() again.\n\nIt would be more consistent (and perhaps more correct) to use\nput_beXX() functions, instead of the htonl() and htonll() functions.\n\n> + memcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t));\n> +}\n\n> +/*\n> + * Call the query-fsmonitor hook passing the time of the last saved results.\n> + */\n> +static int query_fsmonitor(int version, uint64_t last_update, struct strbuf *query_result)\n> +{\n> + struct child_process cp = CHILD_PROCESS_INIT;\n> + char ver[64];\n> + char date[64];\n> + const char *argv[4];\n> +\n> + if (!(argv[0] = find_hook(\"query-fsmonitor\")))\n> + return -1;\n> +\n> + snprintf(ver, sizeof(version), \"%d\", version);\n> + snprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update);\n> + argv[1] = ver;\n> + argv[2] = date;\n> + argv[3] = NULL;\n> + cp.argv = argv;\n\nMaybe it would be nicer using the argv_array_pushX() functions.\n\n> + cp.out = -1;\n> +\n> + return capture_command(&cp, query_result, 1024);\n> +}\n> +\n> +static void mark_file_dirty(struct index_state *istate, const char *name)\n> +{\n> + struct untracked_cache_dir *dir;\n> + int pos;\n> +\n> + /* find it in the index and mark that entry as dirty */\n> + pos = index_name_pos(istate, name, strlen(name));\n> + if (pos >= 0) {\n> + if (!(istate->cache[pos]->ce_flags & CE_FSMONITOR_DIRTY)) {\n> + istate->cache[pos]->ce_flags |= CE_FSMONITOR_DIRTY;\n> + istate->cache_changed |= FSMONITOR_CHANGED;\n> + }\n> + }\n> +\n> + /*\n> + * Find the corresponding directory in the untracked cache\n> + * and mark it as invalid\n> + */\n> + if (!istate->untracked || !istate->untracked->root)\n> + return;\n> +\n> + dir = find_untracked_cache_dir(istate->untracked, istate->untracked->root, name);\n> + if (dir) {\n> + if (dir->valid) {\n> + dir->valid = 0;\n> + istate->cache_changed |= FSMONITOR_CHANGED;\n> + }\n> + }\n\nThe code above is quite similar as what is in mark_fsmonitor_dirty(),\nso I wonder if a refactoring is possible.\n\n> +}\n> +\n> +void refresh_by_fsmonitor(struct index_state *istate)\n> +{\n> + static int has_run_once = 0;\n> + struct strbuf query_result = STRBUF_INIT;\n> + int query_success = 0;\n> + size_t bol = 0; /* beginning of line */\n> + uint64_t last_update;\n> + char *buf, *entry;\n> + int i;\n> +\n> + if (!core_fsmonitor || has_run_once)\n> + return;\n> + has_run_once = 1;\n> +\n> + /*\n> + * This could be racy so save the date/time now and the hook\n> + * should be inclusive to ensure we don't miss potential changes.\n> + */\n> + last_update = getnanotime();\n> +\n> + /*\n> + * If we have a last update time, call query-monitor for the set of\n> + * changes since that time.\n> + */\n> + if (istate->fsmonitor_last_update) {\n> + query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION,\n> + istate->fsmonitor_last_update, &query_result);\n> + trace_performance_since(last_update, \"query-fsmonitor\");\n> + }\n> +\n> + if (query_success) {\n> + /* Mark all entries returned by the monitor as dirty */\n> + buf = entry = query_result.buf;\n> + for (i = 0; i < query_result.len; i++) {\n> + if (buf[i] != '\\0')\n> + continue;\n> + mark_file_dirty(istate, buf + bol);\n\nIt looks like bol is always equal to i here...\n\n> + bol = i + 1;\n> + }\n> + if (bol < query_result.len)\n> + mark_file_dirty(istate, buf + bol);\n\n... and here too. As it is not used below, I wonder if you really need\nthe bol variable.\n\n> + /* Mark all clean entries up-to-date */\n> + for (i = 0; i < istate->cache_nr; i++) {\n> + struct cache_entry *ce = istate->cache[i];\n> + if (ce_stage(ce) || (ce->ce_flags & CE_FSMONITOR_DIRTY))\n> + continue;\n> + ce_mark_uptodate(ce);\n> + }\n"},{"id":"323342","messageId":"CAP8UFD3FtddFuaVY9qEyxbm6hiYhFR2TrZ32Cubp4GAEf9+vaQ@mail.gmail.com","threadId":"46154","inReplyTo":"20170610134026.104552-5-benpeart@microsoft.com","subject":"Re: [PATCH v5 4/7] fsmonitor: add test cases for fsmonitor extension","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-06-27T16:20:19Z","receivedAt":"2017-06-27T16:20:25Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Jun 10, 2017 at 3:40 PM, Ben Peart wrote:\n\n> +# fsmonitor works correctly with or without the untracked cache\n> +# but if it is available, we'll turn it on to ensure we test that\n> +# codepath as well.\n> +\n> +test_lazy_prereq UNTRACKED_CACHE '\n> + { git update-index --test-untracked-cache; ret=$?; } &&\n> + test $ret -ne 1\n> +'\n> +\n> +if test_have_prereq UNTRACKED_CACHE; then\n> + git config core.untrackedcache true\n> +else\n> + git config core.untrackedcache false\n> +fi\n\nI wonder if it would be better to just do something like:\n\n=====================\n\ntest_expect_success 'setup' '\n ....\n'\n\nuc_values=\"false\"\ntest_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n\nfor uc_val in $uc_values\ndo\n\n test_expect_success \"setup untracked cache to $uc_val\" '\n git config core.untrackedcache $uc_val\n '\n\n test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n ...\n '\n\n test_expect_success \"status doesn't detect unreported modifications\" '\n ...\n '\n\n...\n\ndone\n\n=====================\n"},{"id":"323416","messageId":"CAP8UFD3LA1WvjOZN9WgzMQhuTTXNX7yKgChgAJsLFHX0Ag17JA@mail.gmail.com","threadId":"46154","inReplyTo":"20170610134026.104552-1-benpeart@microsoft.com","subject":"Re: [PATCH v5 0/7] Fast git status via a file system watcher","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-06-28T05:11:24Z","receivedAt":"2017-06-28T05:11:30Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Jun 10, 2017 at 3:40 PM, Ben Peart wrote:\n> Changes from V4 include:\n...\n\nI took a look at this patch series except the last patch ([PATCH v5\n7/7] fsmonitor: add a performance test) as Junio reviewed it already,\nand had only a few comments on patches 3/7 and 4/7.\n\nI am still not convinced by the discussions following v2\n(http://public-inbox.org/git/20170518201333.13088-1-benpeart@microsoft.com/)\nabout using a hook instead of for example a \"core.fsmonitorcommand\".\n\nI think using a hook is not necessary and might not be a good match\nfor later optimizations. For example people might want to use a\nlibrary or some OS specific system calls to do what the hook does.\n\nAEvar previously reported some not so great performance numbers on\nsome big Booking.com boxes with a big monorepo and it seems that using\ntaskset for example to make sure that the hook is run on the same CPU\nimproves these numbers significantly. So avoiding to run a separate\nprocess can be important in some cases.\n"},{"id":"323784","messageId":"6b05b590-2fda-31e4-e07e-bda10f510cc4@gmail.com","threadId":"46154","inReplyTo":"CAP8UFD1kL9i93HnnigYcB9U8u2dUijHj5djey5k=33VrFaWUBw@mail.gmail.com","subject":"Re: [PATCH v5 3/7] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-03T21:25:32Z","receivedAt":"2017-07-03T21:25:41Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 6/27/2017 11:43 AM, Christian Couder wrote:\n> On Sat, Jun 10, 2017 at 3:40 PM, Ben Peart wrote:\n> \n>> +int read_fsmonitor_extension(struct index_state *istate, const void *data,\n>> + unsigned long sz)\n>> +{\n>> + const char *index = data;\n>> + uint32_t hdr_version;\n>> + uint32_t ewah_size;\n>> + int ret;\n>> +\n>> + if (sz < sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t))\n>> + return error(\"corrupt fsmonitor extension (too short)\");\n>> +\n>> + hdr_version = get_be32(index);\n> \n> Here we use get_be32(), ...\n> \n>> + index += sizeof(uint32_t);\n>> + if (hdr_version != INDEX_EXTENSION_VERSION)\n>> + return error(\"bad fsmonitor version %d\", hdr_version);\n>> +\n>> + istate->fsmonitor_last_update = get_be64(index);\n> \n> ...get_be64(), ...\n> \n>> + index += sizeof(uint64_t);\n>> +\n>> + ewah_size = get_be32(index);\n> \n> ... and get_be32 again, ...\n> \n>> + index += sizeof(uint32_t);\n>> +\n>> + istate->fsmonitor_dirty = ewah_new();\n>> + ret = ewah_read_mmap(istate->fsmonitor_dirty, index, ewah_size);\n>> + if (ret != ewah_size) {\n>> + ewah_free(istate->fsmonitor_dirty);\n>> + istate->fsmonitor_dirty = NULL;\n>> + return error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n>> + }\n>> +\n>> + return 0;\n>> +}\n>> +\n>> +void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n>> +{\n>> + uint32_t hdr_version;\n>> + uint64_t tm;\n>> + struct ewah_bitmap *bitmap;\n>> + int i;\n>> + uint32_t ewah_start;\n>> + uint32_t ewah_size = 0;\n>> + int fixup = 0;\n>> +\n>> + hdr_version = htonl(INDEX_EXTENSION_VERSION);\n> \n> ... but here we use htonl() instead of put_be32(), ...\n> \n>> + strbuf_add(sb, &hdr_version, sizeof(uint32_t));\n>> +\n>> + tm = htonll((uint64_t)istate->fsmonitor_last_update);\n> \n> ... htonll(), ...\n> \n>> + strbuf_add(sb, &tm, sizeof(uint64_t));\n>> + fixup = sb->len;\n>> + strbuf_add(sb, &ewah_size, sizeof(uint32_t)); /* we'll fix this up later */\n>> +\n>> + ewah_start = sb->len;\n>> + bitmap = ewah_new();\n>> + for (i = 0; i < istate->cache_nr; i++)\n>> + if (istate->cache[i]->ce_flags & CE_FSMONITOR_DIRTY)\n>> + ewah_set(bitmap, i);\n>> + ewah_serialize_strbuf(bitmap, sb);\n>> + ewah_free(bitmap);\n>> +\n>> + /* fix up size field */\n>> + ewah_size = htonl(sb->len - ewah_start);\n> \n> ... and htonl() again.\n> \n> It would be more consistent (and perhaps more correct) to use\n> put_beXX() functions, instead of the htonl() and htonll() functions.\n\nI agree that these are asymmetric. I followed the pattern used in the \nuntracked cache in which write_untracked_extension uses htonl and \nread_untracked_extension uses get_be32. I can change this to be more \nsymmetric.\n\n> \n>> + memcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t));\n>> +}\n> \n>> +/*\n>> + * Call the query-fsmonitor hook passing the time of the last saved results.\n>> + */\n>> +static int query_fsmonitor(int version, uint64_t last_update, struct strbuf *query_result)\n>> +{\n>> + struct child_process cp = CHILD_PROCESS_INIT;\n>> + char ver[64];\n>> + char date[64];\n>> + const char *argv[4];\n>> +\n>> + if (!(argv[0] = find_hook(\"query-fsmonitor\")))\n>> + return -1;\n>> +\n>> + snprintf(ver, sizeof(version), \"%d\", version);\n>> + snprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update);\n>> + argv[1] = ver;\n>> + argv[2] = date;\n>> + argv[3] = NULL;\n>> + cp.argv = argv;\n> \n> Maybe it would be nicer using the argv_array_pushX() functions.\n\nWhen the number of arguments is fixed and known, I prefer avoiding the \ndynamic allocations that come along with the argv_array_pushX() functions.\n\n> \n>> + cp.out = -1;\n>> +\n>> + return capture_command(&cp, query_result, 1024);\n>> +}\n>> +\n>> +static void mark_file_dirty(struct index_state *istate, const char *name)\n>> +{\n>> + struct untracked_cache_dir *dir;\n>> + int pos;\n>> +\n>> + /* find it in the index and mark that entry as dirty */\n>> + pos = index_name_pos(istate, name, strlen(name));\n>> + if (pos >= 0) {\n>> + if (!(istate->cache[pos]->ce_flags & CE_FSMONITOR_DIRTY)) {\n>> + istate->cache[pos]->ce_flags |= CE_FSMONITOR_DIRTY;\n>> + istate->cache_changed |= FSMONITOR_CHANGED;\n>> + }\n>> + }\n>> +\n>> + /*\n>> + * Find the corresponding directory in the untracked cache\n>> + * and mark it as invalid\n>> + */\n>> + if (!istate->untracked || !istate->untracked->root)\n>> + return;\n>> +\n>> + dir = find_untracked_cache_dir(istate->untracked, istate->untracked->root, name);\n>> + if (dir) {\n>> + if (dir->valid) {\n>> + dir->valid = 0;\n>> + istate->cache_changed |= FSMONITOR_CHANGED;\n>> + }\n>> + }\n> \n> The code above is quite similar as what is in mark_fsmonitor_dirty(),\n> so I wonder if a refactoring is possible.\n\nI've felt the same way and looked at how to refactor it better a number \nof times but never came up with a way that made it simpler, clearer and \neasier to maintain. I'm happy to review a patch if you can figure out \nsomething better. :)\n\n> \n>> +}\n>> +\n>> +void refresh_by_fsmonitor(struct index_state *istate)\n>> +{\n>> + static int has_run_once = 0;\n>> + struct strbuf query_result = STRBUF_INIT;\n>> + int query_success = 0;\n>> + size_t bol = 0; /* beginning of line */\n>> + uint64_t last_update;\n>> + char *buf, *entry;\n>> + int i;\n>> +\n>> + if (!core_fsmonitor || has_run_once)\n>> + return;\n>> + has_run_once = 1;\n>> +\n>> + /*\n>> + * This could be racy so save the date/time now and the hook\n>> + * should be inclusive to ensure we don't miss potential changes.\n>> + */\n>> + last_update = getnanotime();\n>> +\n>> + /*\n>> + * If we have a last update time, call query-monitor for the set of\n>> + * changes since that time.\n>> + */\n>> + if (istate->fsmonitor_last_update) {\n>> + query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION,\n>> + istate->fsmonitor_last_update, &query_result);\n>> + trace_performance_since(last_update, \"query-fsmonitor\");\n>> + }\n>> +\n>> + if (query_success) {\n>> + /* Mark all entries returned by the monitor as dirty */\n>> + buf = entry = query_result.buf;\n>> + for (i = 0; i < query_result.len; i++) {\n>> + if (buf[i] != '\\0')\n>> + continue;\n>> + mark_file_dirty(istate, buf + bol);\n> \n> It looks like bol is always equal to i here...\n> \n>> + bol = i + 1;\n>> + }\n>> + if (bol < query_result.len)\n>> + mark_file_dirty(istate, buf + bol);\n> \n> ... and here too. As it is not used below, I wonder if you really need\n> the bol variable.\n\nbol saves the position of the beginning of the current line while i \niterates through the remainder of the string looking for the NULL.\n\nHowever, the entry variable is no longer used so I will remove it.\n\n> \n>> + /* Mark all clean entries up-to-date */\n>> + for (i = 0; i < istate->cache_nr; i++) {\n>> + struct cache_entry *ce = istate->cache[i];\n>> + if (ce_stage(ce) || (ce->ce_flags & CE_FSMONITOR_DIRTY))\n>> + continue;\n>> + ce_mark_uptodate(ce);\n>> + }\n"},{"id":"323968","messageId":"7ec36d90-7fbc-c30f-e15e-f06d39e1f206@gmail.com","threadId":"46154","inReplyTo":"xmqqvany2z84.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 7/7] fsmonitor: add a performance test","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-07T18:14:20Z","receivedAt":"2017-07-07T18:14:29Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 6/14/2017 2:36 PM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>>> Having said all that, I think you are using this ONLY on windows;\n>>> perhaps it is better to drop #ifdef GIT_WINDOWS_NATIVE from all of\n>>> the above and arrange Makefile to build test-drop-cache only on that\n>>> platform, or something?\n>>\n>> I didn't find any other examples of Windows only tools. I'll update\n>> the #ifdef to properly dump the file system cache on Linux as well and\n>> only error out on other platforms.\n> \n> If this will become Windows-only, then I have no problem with\n> platform specfic typedef ;-) I have no problem with CamelCase,\n> either, as that follows the local convention on the platform\n> (similar to those in compat/* that are only for Windows).\n> \n> Having said all that.\n> \n> Another approach is to build this helper on all platforms, with\n> sections protected by \"#ifdef LINUX\", \"#ifdef WINDOWS\", etc. That\n> way, the platform detection and switching between running this\n> program and echoing something into /proc filesystem performed in\n> p7519 can be removed (this test-helper program will be responsible\n> to implement that logic instead). When p7519 wants to drop the\n> filesystem cache, regardless of the platform, it can call this\n> test-helper without having to know how the filesystem cache is\n> dropped.\n> \n\nI'll take a cut at doing this but it is obviously very platform \ndependent and I'm unfamiliar with platforms other than Windows.\n\nFor everything other than Windows, my implementation will be calling \n\"system\" for external utilities based on what I can find on the web. \nOh, and I have no way to test it other than on Windows so could use some \nreview/testing from others. :)\n\n> I do not have strong preference either way, but I have a slight\n> suspicion that the \"another approach\" above may give us a better\n> result. For one thing, the test-helper can be reused in new perf\n> scripts people will write in the future.\n> \n> Thanks.\n> \n"},{"id":"323970","messageId":"xmqqeftsayeg.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"7ec36d90-7fbc-c30f-e15e-f06d39e1f206@gmail.com","subject":"Re: [PATCH v5 7/7] fsmonitor: add a performance test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-07T18:35:03Z","receivedAt":"2017-07-07T18:35:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> On 6/14/2017 2:36 PM, Junio C Hamano wrote:\n>> Ben Peart writes:\n>>\n>>>> Having said all that, I think you are using this ONLY on windows;\n>>>> perhaps it is better to drop #ifdef GIT_WINDOWS_NATIVE from all of\n>>>> the above and arrange Makefile to build test-drop-cache only on that\n>>>> platform, or something?\n>>>\n>>> I didn't find any other examples of Windows only tools. I'll update\n>>> the #ifdef to properly dump the file system cache on Linux as well and\n>>> only error out on other platforms.\n>>\n>> If this will become Windows-only, then I have no problem with\n>> platform specfic typedef ;-) I have no problem with CamelCase,\n>> either, as that follows the local convention on the platform\n>> (similar to those in compat/* that are only for Windows).\n>>\n>> Having said all that.\n>>\n>> Another approach is to build this helper on all platforms, ...\n\n... and having said all that, I think it is perfectly fine to do\nsuch a clean-up long after the series gets more exposure to wider\naudiences as a follow-up patch. Let's get the primary part that\naffects people's everyday use of Git right and then worry about the\ntest details later.\n\nA quick show of hands to the list audiences. How many of you guys\nactually tried this series on 'pu' and checked to see its\nperformance (and correctness ;-) characteristics? \n\nDo you folks like it? Rather not have such complexity in the core\npart of the system? A good first step to start adding more\nperformance improvements? No opinion?\n\n\n"},{"id":"323971","messageId":"19e50acb-d4f0-b8f0-2966-0a33770296b1@gmail.com","threadId":"46154","inReplyTo":"CAP8UFD3FtddFuaVY9qEyxbm6hiYhFR2TrZ32Cubp4GAEf9+vaQ@mail.gmail.com","subject":"Re: [PATCH v5 4/7] fsmonitor: add test cases for fsmonitor extension","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-07T18:50:13Z","receivedAt":"2017-07-07T18:50:20Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 6/27/2017 12:20 PM, Christian Couder wrote:\n> On Sat, Jun 10, 2017 at 3:40 PM, Ben Peart wrote:\n> \n>> +# fsmonitor works correctly with or without the untracked cache\n>> +# but if it is available, we'll turn it on to ensure we test that\n>> +# codepath as well.\n>> +\n>> +test_lazy_prereq UNTRACKED_CACHE '\n>> + { git update-index --test-untracked-cache; ret=$?; } &&\n>> + test $ret -ne 1\n>> +'\n>> +\n>> +if test_have_prereq UNTRACKED_CACHE; then\n>> + git config core.untrackedcache true\n>> +else\n>> + git config core.untrackedcache false\n>> +fi\n> \n> I wonder if it would be better to just do something like:\n\nThat is a good idea; I'll add that around the tests that aren't \nexplicitly testing interop with and without the untracked cache.\n\nThanks!\n\n> \n> =====================\n> \n> test_expect_success 'setup' '\n> ....\n> '\n> \n> uc_values=\"false\"\n> test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n> \n> for uc_val in $uc_values\n> do\n> \n> test_expect_success \"setup untracked cache to $uc_val\" '\n> git config core.untrackedcache $uc_val\n> '\n> \n> test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n> ...\n> '\n> \n> test_expect_success \"status doesn't detect unreported modifications\" '\n> ...\n> '\n> \n> ...\n> \n> done\n> \n> =====================\n> \n"},{"id":"323972","messageId":"bade1166-e646-b05a-f65b-adb8da8ba0a7@gmail.com","threadId":"46154","inReplyTo":"xmqqeftsayeg.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 7/7] fsmonitor: add a performance test","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-07T19:07:31Z","receivedAt":"2017-07-07T19:07:38Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/7/2017 2:35 PM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> On 6/14/2017 2:36 PM, Junio C Hamano wrote:\n>>> Ben Peart writes:\n>>>\n>>>>> Having said all that, I think you are using this ONLY on windows;\n>>>>> perhaps it is better to drop #ifdef GIT_WINDOWS_NATIVE from all of\n>>>>> the above and arrange Makefile to build test-drop-cache only on that\n>>>>> platform, or something?\n>>>>\n>>>> I didn't find any other examples of Windows only tools. I'll update\n>>>> the #ifdef to properly dump the file system cache on Linux as well and\n>>>> only error out on other platforms.\n>>>\n>>> If this will become Windows-only, then I have no problem with\n>>> platform specfic typedef ;-) I have no problem with CamelCase,\n>>> either, as that follows the local convention on the platform\n>>> (similar to those in compat/* that are only for Windows).\n>>>\n>>> Having said all that.\n>>>\n>>> Another approach is to build this helper on all platforms, ...\n> \n> ... and having said all that, I think it is perfectly fine to do\n> such a clean-up long after the series gets more exposure to wider\n> audiences as a follow-up patch. Let's get the primary part that\n> affects people's everyday use of Git right and then worry about the\n> test details later.\n> \n> A quick show of hands to the list audiences. How many of you guys\n> actually tried this series on 'pu' and checked to see its\n> performance (and correctness ;-) characteristics?\n\nTLDR: the current version isn't correct.\n\nOne of the things I did was hack up the test script to enable running \nall the tests with fsmonitor enabled. I found a number of bugs in \nWatchman on Windows and have been working with Wez to get them fixed.\n\nJust last week Watchman got to the point where I could run the complete \ngit test suite. As a result, I found that fsmonitor is overly \naggressive in marking things with ce_mark_uptodate. Submodules are \ncurrently broken as are commands that pass \"--ignore-missing.\"\n\nI started reworking the logic to fix these bugs and realized I was \nduplicating the set of tests that already exist in preload_thread. I'm \ncurrently working on integrating the logic into preload_thread so that \nboth options can be used in combination and so I can avoid doing the \n(nearly identical) loop twice.\n\n> \n> Do you folks like it? Rather not have such complexity in the core\n> part of the system? A good first step to start adding more\n> performance improvements? No opinion?\n> \n> \n"},{"id":"323974","messageId":"5aeeffc9774249b08ee510d7fbd93d29@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"xmqqeftsayeg.fsf@gitster.mtv.corp.google.com","subject":"RE: [PATCH v5 7/7] fsmonitor: add a performance test","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-07-07T19:33:03Z","receivedAt":"2017-07-07T19:33:10Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"> -----Original Message-----\n> From: Junio C Hamano [mailto:jch2355@gmail.com] On Behalf Of Junio C\n> Hamano\n> Sent: Friday, July 7, 2017 2:35 PM\n> To: Ben Peart \n> Cc: git@vger.kernel.org; benpeart@microsoft.com; pclouds@gmail.com;\n> johannes.schindelin@gmx.de; David Turner ;\n> peff@peff.net; christian.couder@gmail.com; avarab@gmail.com\n> Subject: Re: [PATCH v5 7/7] fsmonitor: add a performance test\n> \n> Ben Peart writes:\n> \n> > On 6/14/2017 2:36 PM, Junio C Hamano wrote:\n> >> Ben Peart writes:\n> >>\n> >>>> Having said all that, I think you are using this ONLY on windows;\n> >>>> perhaps it is better to drop #ifdef GIT_WINDOWS_NATIVE from all of\n> >>>> the above and arrange Makefile to build test-drop-cache only on\n> >>>> that platform, or something?\n> >>>\n> >>> I didn't find any other examples of Windows only tools. I'll update\n> >>> the #ifdef to properly dump the file system cache on Linux as well\n> >>> and only error out on other platforms.\n> >>\n> >> If this will become Windows-only, then I have no problem with\n> >> platform specfic typedef ;-) I have no problem with CamelCase,\n> >> either, as that follows the local convention on the platform (similar\n> >> to those in compat/* that are only for Windows).\n> >>\n> >> Having said all that.\n> >>\n> >> Another approach is to build this helper on all platforms, ...\n> \n> ... and having said all that, I think it is perfectly fine to do such a clean-up long\n> after the series gets more exposure to wider audiences as a follow-up patch.\n> Let's get the primary part that affects people's everyday use of Git right and then\n> worry about the test details later.\n> \n> A quick show of hands to the list audiences. How many of you guys actually\n> tried this series on 'pu' and checked to see its performance (and correctness ;-)\n> characteristics?\n> \n> Do you folks like it? Rather not have such complexity in the core part of the\n> system? A good first step to start adding more performance improvements? No\n> opinion?\n\nI have not had the chance to test the latest version out yet. The idea, broadly, seems sound, but as Ben notes in a later mail, the details are important. Since he's going to re-roll with more interesting invalidation logic, I'll wait to try it again until a new version is available.\n"},{"id":"323985","messageId":"CAP8UFD1hEtiBhRWvrMq+uugkez74fsaHnZLfC=tyEnMNeLGW8g@mail.gmail.com","threadId":"46154","inReplyTo":"xmqqeftsayeg.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 7/7] fsmonitor: add a performance test","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-07-08T07:19:09Z","receivedAt":"2017-07-08T07:19:16Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Jul 7, 2017 at 8:35 PM, Junio C Hamano wrote:\n> Ben Peart writes:\n>\n>> On 6/14/2017 2:36 PM, Junio C Hamano wrote:\n>>> Ben Peart writes:\n>>>\n>>>>> Having said all that, I think you are using this ONLY on windows;\n>>>>> perhaps it is better to drop #ifdef GIT_WINDOWS_NATIVE from all of\n>>>>> the above and arrange Makefile to build test-drop-cache only on that\n>>>>> platform, or something?\n>>>>\n>>>> I didn't find any other examples of Windows only tools. I'll update\n>>>> the #ifdef to properly dump the file system cache on Linux as well and\n>>>> only error out on other platforms.\n>>>\n>>> If this will become Windows-only, then I have no problem with\n>>> platform specfic typedef ;-) I have no problem with CamelCase,\n>>> either, as that follows the local convention on the platform\n>>> (similar to those in compat/* that are only for Windows).\n>>>\n>>> Having said all that.\n>>>\n>>> Another approach is to build this helper on all platforms, ...\n>\n> ... and having said all that, I think it is perfectly fine to do\n> such a clean-up long after the series gets more exposure to wider\n> audiences as a follow-up patch. Let's get the primary part that\n> affects people's everyday use of Git right and then worry about the\n> test details later.\n>\n> A quick show of hands to the list audiences. How many of you guys\n> actually tried this series on 'pu' and checked to see its\n> performance (and correctness ;-) characteristics?\n\nAs you can guess from my previous replies to this thread (and the\nprevious version of this patch series), I lightly tried it and checked\nits performance for Booking.com.\n\n> Do you folks like it? Rather not have such complexity in the core\n> part of the system? A good first step to start adding more\n> performance improvements? No opinion?\n\nI already gave my opinion which I think is shared with Ævar. In short\nI don't think it should be a hook, as that limits the performance and\nis not necessary, but it is going in the right direction.\n"},{"id":"324064","messageId":"9b1af00c-2451-501c-5315-7ed2b81f3f18@gmail.com","threadId":"46154","inReplyTo":"CAP8UFD3LA1WvjOZN9WgzMQhuTTXNX7yKgChgAJsLFHX0Ag17JA@mail.gmail.com","subject":"Re: [PATCH v5 0/7] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-10T13:36:33Z","receivedAt":"2017-07-10T13:36:41Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 6/28/2017 1:11 AM, Christian Couder wrote:\n> On Sat, Jun 10, 2017 at 3:40 PM, Ben Peart wrote:\n>> Changes from V4 include:\n> ...\n> \n> I took a look at this patch series except the last patch ([PATCH v5\n> 7/7] fsmonitor: add a performance test) as Junio reviewed it already,\n> and had only a few comments on patches 3/7 and 4/7.\n> \n> I am still not convinced by the discussions following v2\n> (http://public-inbox.org/git/20170518201333.13088-1-benpeart@microsoft.com/)\n> about using a hook instead of for example a \"core.fsmonitorcommand\".\n> \n> I think using a hook is not necessary and might not be a good match\n> for later optimizations. For example people might want to use a\n> library or some OS specific system calls to do what the hook does.\n> \n> AEvar previously reported some not so great performance numbers on\n> some big Booking.com boxes with a big monorepo and it seems that using\n> taskset for example to make sure that the hook is run on the same CPU\n> improves these numbers significantly. So avoiding to run a separate\n> process can be important in some cases.\n> \n\nUsing a hook is the only pattern I've seen in git that provides a way to \nenable OS specific calls. I used a hook so that different file \nmonitoring services could be plugged in depending on the OS or tools \navailable. The Watchman integration script was mostly intended as a \nsample that could be used where Watchman is available and works well.\n\nI had not heard about the taskset issues you mention above. If there is \nsomething else that can be done to make this work better, please let me \nknow the details.\n\nIf I'm understanding you correctly, you are suggesting that someone \nshould be able to configure a setting (core.fsmonitorcommand) that gives \na custom command line that would be run instead of running the \nquery-fsmonitor hook.\n\nI'm not entirely sure how that should work. There are command line \noptions that need to be passed (currently the interface version as well \nas the current clock in nanoseconds). How would those passed when using \nthe custom command?\n\nIs it OK to just append them to the given command line? Does there need \nto be some substitution token to indicate where they should be inserted \n(ie \"mycustomcommand --custom --options %version% --more-options \n%timestamp%\"). Are there any other tokens that should be supported (ie \nPID or processor mask?).\n\nIs there a design pattern already used somewhere in git that I can \nfollow or is this all blazing a new trail?\n\nThanks for continuing to look into this. Feedback is good!\n\n\n"},{"id":"324071","messageId":"97342e22-c2ca-9c09-4db0-24cac014b689@gmail.com","threadId":"46154","inReplyTo":"9b1af00c-2451-501c-5315-7ed2b81f3f18@gmail.com","subject":"Re: [PATCH v5 0/7] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-10T14:40:15Z","receivedAt":"2017-07-10T14:40:25Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/10/2017 9:36 AM, Ben Peart wrote:\n> \n> \n> On 6/28/2017 1:11 AM, Christian Couder wrote:\n>> On Sat, Jun 10, 2017 at 3:40 PM, Ben Peart wrote:\n>>> Changes from V4 include:\n>> ...\n>>\n>> I took a look at this patch series except the last patch ([PATCH v5\n>> 7/7] fsmonitor: add a performance test) as Junio reviewed it already,\n>> and had only a few comments on patches 3/7 and 4/7.\n>>\n>> I am still not convinced by the discussions following v2\n>> (http://public-inbox.org/git/20170518201333.13088-1-benpeart@microsoft.com/) \n>>\n>> about using a hook instead of for example a \"core.fsmonitorcommand\".\n>>\n>> I think using a hook is not necessary and might not be a good match\n>> for later optimizations. For example people might want to use a\n>> library or some OS specific system calls to do what the hook does.\n>>\n>> AEvar previously reported some not so great performance numbers on\n>> some big Booking.com boxes with a big monorepo and it seems that using\n>> taskset for example to make sure that the hook is run on the same CPU\n>> improves these numbers significantly. So avoiding to run a separate\n>> process can be important in some cases.\n>>\n> \n> Using a hook is the only pattern I've seen in git that provides a way to \n> enable OS specific calls. I used a hook so that different file \n> monitoring services could be plugged in depending on the OS or tools \n> available. The Watchman integration script was mostly intended as a \n> sample that could be used where Watchman is available and works well.\n> \n> I had not heard about the taskset issues you mention above. If there is \n> something else that can be done to make this work better, please let me \n> know the details.\n> \n> If I'm understanding you correctly, you are suggesting that someone \n> should be able to configure a setting (core.fsmonitorcommand) that gives \n> a custom command line that would be run instead of running the \n> query-fsmonitor hook.\n> \n> I'm not entirely sure how that should work. There are command line \n> options that need to be passed (currently the interface version as well \n> as the current clock in nanoseconds). How would those passed when using \n> the custom command?\n> \n> Is it OK to just append them to the given command line? Does there need \n> to be some substitution token to indicate where they should be inserted \n> (ie \"mycustomcommand --custom --options %version% --more-options \n> %timestamp%\"). Are there any other tokens that should be supported (ie \n> PID or processor mask?).\n> \n> Is there a design pattern already used somewhere in git that I can \n> follow or is this all blazing a new trail?\n> \n\nMy co-worker reminded me about git difftool - is this what you had in mind?\n\n> Thanks for continuing to look into this. Feedback is good!\n> \n> \n"},{"id":"328144","messageId":"20170915192043.4516-1-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170610134026.104552-1-benpeart@microsoft.com","subject":"[PATCH v6 00/12] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:31Z","receivedAt":"2017-09-15T19:21:24Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This is a fairly significant rewrite since V5. The big changes include:\n\nMultiple functions including preload-index(), ie_match_stat(), and\nrefresh_cache_ent() have been updated to honor the CE_FSMONITOR_VALID bit\nfollowing the same pattern as skip_worktree and CE_VALID. As a result,\nperformance improvements apply to all git commands that would otherwise\nhave had to scan the entire working directory.\n\ncore.fsmonitor is now a registered command (instead of a hook) to\nprovide additional flexibility. It is called when needed to ensure the\nstate of the index is up-to-date.\n\nThe Watchman integration script is now entirely written in perl to\nminimize spawning additional helper commands. This along with the other\nchanges have helped reduce the overhead and made the extension applicable\nto more (ie smaller) repos.\n\nThere are additional opportunities for performance improvements but I\nwanted to get this version out there and then build on it as the\nfoundation. Some potential examples of future patches include:\n\n - call the integration script on a background thread so that it can\n execute in parallel.\n\n - optimize traverse trees by pruning out entire branches that do not\n contain any changes.\n\nOther optimizations likely exist where knowledge that files have not\nchanged can be used to short circuit some of the normal workflow.\n\nPerformance\n===========\n\nWith the various enhancements, performance has been improved especially\nfor smaller repos. The included perf test compares status times without\nfsmonitor to those with fsmonitor using the provided Watchman integration\nscript.\n\nDue to the overhead of calling out to Watchman, on very small repos\n(<10K files) the overhead exceeds the savings. Once repos hit 10K files\nthe savings kick in and for repos beyond that, the savings are dramatic.\n\nTest with 10,000 files this tree\n------------------------------------------------------------------------\n7519.2: status (fsmonitor=.git/hooks/fsmonitor-watchman) 0.35(0.03+0.04)\n7519.3: status -uno (fsmonitor=.git/hooks/fsmonitor-watchman) 0.37(0.00+0.09)\n7519.4: status -uall (fsmonitor=.git/hooks/fsmonitor-watchman) 0.43(0.03+0.06)\n7519.6: status (fsmonitor=) 0.45(0.00+0.07)\n7519.7: status -uno (fsmonitor=) 0.40(0.03+0.07)\n7519.8: status -uall (fsmonitor=) 0.44(0.04+0.04)\n\nTest with 100,000 files this tree\n------------------------------------------------------------------------\n7519.2: status (fsmonitor=.git/hooks/fsmonitor-watchman) 0.33(0.01+0.03)\n7519.3: status -uno (fsmonitor=.git/hooks/fsmonitor-watchman) 0.36(0.00+0.06)\n7519.4: status -uall (fsmonitor=.git/hooks/fsmonitor-watchman) 0.93(0.00+0.07)\n7519.6: status (fsmonitor=) 2.66(0.04+0.03)\n7519.7: status -uno (fsmonitor=) 2.44(0.01+0.06)\n7519.8: status -uall (fsmonitor=) 2.94(0.03+0.07)\n\nTest with 1,000,000 files this tree\n---------------------------------------------------------------------------------\n7519.2: status (fsmonitor=.git/hooks/fsmonitor-watchman) 1.45(0.00+0.06)\n7519.3: status -uno (fsmonitor=.git/hooks/fsmonitor-watchman) 0.88(0.01+0.04)\n7519.4: status -uall (fsmonitor=.git/hooks/fsmonitor-watchman) 6.14(0.03+0.04)\n7519.6: status (fsmonitor=) 25.91(0.04+0.06)\n7519.7: status -uno (fsmonitor=) 23.96(0.04+0.03)\n7519.8: status -uall (fsmonitor=) 28.81(0.00+0.07)\n\nNote: all numbers above are with a warm disk cache on a fast SSD, real\nworld performance numbers are often dramatically better as fsmonitor can\neliminate all the file IO to lstat every file and then traverse the\nworking directory looking for untracked files. For example, a cold\nstatus without fsmonitor on a HDD with 1M files takes 1m22.774s\n\n$ time git -c core.fsmonitor= status\nOn branch p0006-ballast\n\nIt took 2.09 seconds to enumerate untracked files. 'status -uno'\nmay speed it up, but you have to be careful not to forget to add\nnew files yourself (see 'git help status').\nnothing to commit, working tree clean\n\nreal 1m22.774s\nuser 0m0.000s\nsys 0m0.000s\n\n\nBen Peart (12):\n bswap: add 64 bit endianness helper get_be64\n preload-index: add override to enable testing preload-index\n update-index: add a new --force-write-index option\n fsmonitor: teach git to optionally utilize a file system monitor to\n speed up detecting new or changed files.\n fsmonitor: add documentation for the fsmonitor extension.\n ls-files: Add support in ls-files to display the fsmonitor valid bit\n update-index: add fsmonitor support to update-index\n fsmonitor: add a test tool to dump the index extension\n split-index: disable the fsmonitor extension when running the split\n index test\n fsmonitor: add test cases for fsmonitor extension\n fsmonitor: add a sample integration script for Watchman\n fsmonitor: add a performance test\n\n Documentation/config.txt | 6 +\n Documentation/githooks.txt | 23 +++\n Documentation/technical/index-format.txt | 19 +++\n Makefile | 3 +\n apply.c | 2 +-\n builtin/ls-files.c | 8 +-\n builtin/update-index.c | 26 ++-\n cache.h | 10 +-\n compat/bswap.h | 22 +++\n config.c | 14 ++\n config.h | 1 +\n diff-lib.c | 2 +\n dir.c | 27 +--\n dir.h | 2 +\n entry.c | 4 +-\n environment.c | 1 +\n fsmonitor.c | 253 ++++++++++++++++++++++++++++\n fsmonitor.h | 61 +++++++\n preload-index.c | 8 +-\n read-cache.c | 49 +++++-\n submodule.c | 2 +-\n t/helper/.gitignore | 1 +\n t/helper/test-drop-caches.c | 161 ++++++++++++++++++\n t/helper/test-dump-fsmonitor.c | 21 +++\n t/perf/p7519-fsmonitor.sh | 184 ++++++++++++++++++++\n t/t1700-split-index.sh | 1 +\n t/t7519-status-fsmonitor.sh | 259 +++++++++++++++++++++++++++++\n t/t7519/fsmonitor-all | 23 +++\n t/t7519/fsmonitor-none | 21 +++\n t/t7519/fsmonitor-watchman | 128 ++++++++++++++\n templates/hooks--fsmonitor-watchman.sample | 119 +++++++++++++\n unpack-trees.c | 8 +-\n 32 files changed, 1440 insertions(+), 29 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n create mode 100644 t/helper/test-drop-caches.c\n create mode 100644 t/helper/test-dump-fsmonitor.c\n create mode 100755 t/perf/p7519-fsmonitor.sh\n create mode 100755 t/t7519-status-fsmonitor.sh\n create mode 100755 t/t7519/fsmonitor-all\n create mode 100755 t/t7519/fsmonitor-none\n create mode 100755 t/t7519/fsmonitor-watchman\n create mode 100755 templates/hooks--fsmonitor-watchman.sample\n\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328145","messageId":"20170915192043.4516-2-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 01/12] bswap: add 64 bit endianness helper get_be64","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:32Z","receivedAt":"2017-09-15T19:21:27Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a new get_be64 macro to enable 64 bit endian conversions on memory\nthat may or may not be aligned.\n\nSigned-off-by: Ben Peart \n---\n compat/bswap.h | 22 ++++++++++++++++++++++\n 1 file changed, 22 insertions(+)\n\ndiff --git a/compat/bswap.h b/compat/bswap.h\nindex 7d063e9e40..6b22c46214 100644\n--- a/compat/bswap.h\n+++ b/compat/bswap.h\n@@ -158,7 +158,9 @@ static inline uint64_t git_bswap64(uint64_t x)\n \n #define get_be16(p)\tntohs(*(unsigned short *)(p))\n #define get_be32(p)\tntohl(*(unsigned int *)(p))\n+#define get_be64(p)\tntohll(*(uint64_t *)(p))\n #define put_be32(p, v)\tdo { *(unsigned int *)(p) = htonl(v); } while (0)\n+#define put_be64(p, v)\tdo { *(uint64_t *)(p) = htonll(v); } while (0)\n \n #else\n \n@@ -178,6 +180,13 @@ static inline uint32_t get_be32(const void *ptr)\n \t\t(uint32_t)p[3] << 0;\n }\n \n+static inline uint64_t get_be64(const void *ptr)\n+{\n+\tconst unsigned char *p = ptr;\n+\treturn\t(uint64_t)get_be32(p[0]) << 32 |\n+\t\t(uint64_t)get_be32(p[4]) << 0;\n+}\n+\n static inline void put_be32(void *ptr, uint32_t value)\n {\n \tunsigned char *p = ptr;\n@@ -187,4 +196,17 @@ static inline void put_be32(void *ptr, uint32_t value)\n \tp[3] = value >> 0;\n }\n \n+static inline void put_be64(void *ptr, uint64_t value)\n+{\n+\tunsigned char *p = ptr;\n+\tp[0] = value >> 56;\n+\tp[1] = value >> 48;\n+\tp[2] = value >> 40;\n+\tp[3] = value >> 32;\n+\tp[4] = value >> 24;\n+\tp[5] = value >> 16;\n+\tp[6] = value >> 8;\n+\tp[7] = value >> 0;\n+}\n+\n #endif\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328146","messageId":"20170915192043.4516-5-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:35Z","receivedAt":"2017-09-15T19:21:40Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"When the index is read from disk, the fsmonitor index extension is used\nto flag the last known potentially dirty index entries. The registered\ncore.fsmonitor command is called with the time the index was last\nupdated and returns the list of files changed since that time. This list\nis used to flag any additional dirty cache entries and untracked cache\ndirectories.\n\nWe can then use this valid state to speed up preload_index(),\nie_match_stat(), and refresh_cache_ent() as they do not need to lstat()\nfiles to detect potential changes for those entries marked\nCE_FSMONITOR_VALID.\n\nIn addition, if the untracked cache is turned on valid_cached_dir() can\nskip checking directories for new or changed files as fsmonitor will\ninvalidate the cache only for those directories that have been\nidentified as having potential changes.\n\nTo keep the CE_FSMONITOR_VALID state accurate during git operations;\nwhen git updates a cache entry to match the current state on disk,\nit will now set the CE_FSMONITOR_VALID bit.\n\nInversely, anytime git changes a cache entry, the CE_FSMONITOR_VALID bit\nis cleared and the corresponding untracked cache directory is marked\ninvalid.\n\nSigned-off-by: Ben Peart \n---\n Makefile | 1 +\n apply.c | 2 +-\n builtin/update-index.c | 2 +\n cache.h | 10 +-\n config.c | 14 +++\n config.h | 1 +\n diff-lib.c | 2 +\n dir.c | 27 ++++--\n dir.h | 2 +\n entry.c | 4 +-\n environment.c | 1 +\n fsmonitor.c | 253 +++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor.h | 61 ++++++++++++\n preload-index.c | 6 +-\n read-cache.c | 49 ++++++++--\n submodule.c | 2 +-\n unpack-trees.c | 8 +-\n 17 files changed, 419 insertions(+), 26 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n\ndiff --git a/Makefile b/Makefile\nindex f2bb7f2f63..9d6ec9c1e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -786,6 +786,7 @@ LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n+LIB_OBJS += fsmonitor.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/apply.c b/apply.c\nindex 71cbbd141c..9061cc5f15 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -3399,7 +3399,7 @@ static int verify_index_match(const struct cache_entry *ce, struct stat *st)\n \t\t\treturn -1;\n \t\treturn 0;\n \t}\n-\treturn ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n+\treturn ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n }\n \n #define SUBMODULE_PATCH_WITHOUT_INDEX 1\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex e1ca0759d5..6f39ee9274 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -16,6 +16,7 @@\n #include \"pathspec.h\"\n #include \"dir.h\"\n #include \"split-index.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Default to not allowing changes to the list of files. The\n@@ -233,6 +234,7 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(&the_index, active_cache[pos]);\n \t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\ndiff --git a/cache.h b/cache.h\nindex a916bc79e3..eccab968bd 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -203,6 +203,7 @@ struct cache_entry {\n #define CE_ADDED (1 << 19)\n \n #define CE_HASHED (1 << 20)\n+#define CE_FSMONITOR_VALID (1 << 21)\n #define CE_WT_REMOVE (1 << 22) /* remove in work directory */\n #define CE_CONFLICTED (1 << 23)\n \n@@ -326,6 +327,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n #define UNTRACKED_CHANGED\t(1 << 7)\n+#define FSMONITOR_CHANGED\t(1 << 8)\n \n struct split_index;\n struct untracked_cache;\n@@ -344,6 +346,7 @@ struct index_state {\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n \tstruct untracked_cache *untracked;\n+\tuint64_t fsmonitor_last_update;\n };\n \n extern struct index_state the_index;\n@@ -679,8 +682,10 @@ extern void *read_blob_data_from_index(const struct index_state *, const char *,\n #define CE_MATCH_IGNORE_MISSING\t\t0x08\n /* enable stat refresh */\n #define CE_MATCH_REFRESH\t\t0x10\n-extern int ie_match_stat(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n-extern int ie_modified(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n+/* do stat comparison even if CE_FSMONITOR_VALID is true */\n+#define CE_MATCH_IGNORE_FSMONITOR 0X20\n+extern int ie_match_stat(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n+extern int ie_modified(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n \n #define HASH_WRITE_OBJECT 1\n #define HASH_FORMAT_CHECK 2\n@@ -773,6 +778,7 @@ extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n+extern const char *core_fsmonitor;\n \n /*\n * Include broken refs in all ref iterations, which will\ndiff --git a/config.c b/config.c\nindex d0d8ce823a..ddda96e584 100644\n--- a/config.c\n+++ b/config.c\n@@ -2165,6 +2165,20 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n+int git_config_get_fsmonitor(void)\n+{\n+\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n+\t\tcore_fsmonitor = getenv(\"GIT_FSMONITOR_TEST\");\n+\n+\tif (core_fsmonitor && !*core_fsmonitor)\n+\t\tcore_fsmonitor = NULL;\n+\n+\tif (core_fsmonitor)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\ndiff --git a/config.h b/config.h\nindex 97471b8873..c9fcf691ba 100644\n--- a/config.h\n+++ b/config.h\n@@ -211,6 +211,7 @@ extern int git_config_get_pathname(const char *key, const char **dest);\n extern int git_config_get_untracked_cache(void);\n extern int git_config_get_split_index(void);\n extern int git_config_get_max_percent_split_change(void);\n+extern int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n extern int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 2a52b07954..23c6d03ca9 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"submodule.h\"\n #include \"dir.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * diff-files\n@@ -228,6 +229,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \n \t\tif (!changed && !dirty_submodule) {\n \t\t\tce_mark_uptodate(ce);\n+\t\t\tmark_fsmonitor_valid(ce);\n \t\t\tif (!DIFF_OPT_TST(&revs->diffopt, FIND_COPIES_HARDER))\n \t\t\t\tcontinue;\n \t\t}\ndiff --git a/dir.c b/dir.c\nindex 1c55dc3e36..ac9833daec 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -18,6 +18,7 @@\n #include \"utf8.h\"\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Tells read_directory_recursive how a file or directory should be treated.\n@@ -1688,17 +1689,23 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n-\tif (stat(path->len ? path->buf : \".\", &st)) {\n-\t\tinvalidate_directory(dir->untracked, untracked);\n-\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n-\t\treturn 0;\n-\t}\n-\tif (!untracked->valid ||\n-\t match_stat_data_racy(istate, &untracked->stat_data, &st)) {\n-\t\tif (untracked->valid)\n+\t/*\n+\t * With fsmonitor, we can trust the untracked cache's valid field.\n+\t */\n+\trefresh_fsmonitor(istate);\n+\tif (!(dir->untracked->use_fsmonitor && untracked->valid)) {\n+\t\tif (stat(path->len ? path->buf : \".\", &st)) {\n \t\t\tinvalidate_directory(dir->untracked, untracked);\n-\t\tfill_stat_data(&untracked->stat_data, &st);\n-\t\treturn 0;\n+\t\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (!untracked->valid ||\n+\t\t\tmatch_stat_data_racy(istate, &untracked->stat_data, &st)) {\n+\t\t\tif (untracked->valid)\n+\t\t\t\tinvalidate_directory(dir->untracked, untracked);\n+\t\t\tfill_stat_data(&untracked->stat_data, &st);\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n \tif (untracked->check_only != !!check_only) {\ndiff --git a/dir.h b/dir.h\nindex e3717055d1..fab8fc1561 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -139,6 +139,8 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n+\t/* fsmonitor invalidation data */\n+\tunsigned int use_fsmonitor : 1;\n };\n \n struct dir_struct {\ndiff --git a/entry.c b/entry.c\nindex cb291aa88b..5e6794f9fc 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -4,6 +4,7 @@\n #include \"streaming.h\"\n #include \"submodule.h\"\n #include \"progress.h\"\n+#include \"fsmonitor.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -357,6 +358,7 @@ static int write_entry(struct cache_entry *ce,\n \t\t\tlstat(ce->name, &st);\n \t\tfill_stat_cache_info(ce, &st);\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(state->istate, ce);\n \t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \treturn 0;\n@@ -402,7 +404,7 @@ int checkout_entry(struct cache_entry *ce,\n \n \tif (!check_path(path.buf, path.len, &st, state->base_dir_len)) {\n \t\tconst struct submodule *sub;\n-\t\tunsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n+\t\tunsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n \t\t/*\n \t\t * Needs to be checked before !changed returns early,\n \t\t * as the possibly empty directory was not changed\ndiff --git a/environment.c b/environment.c\nindex 3fd4b10845..d0b9fc64d4 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -76,6 +76,7 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 0\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n+const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor.c b/fsmonitor.c\nnew file mode 100644\nindex 0000000000..144294b8df\n--- /dev/null\n+++ b/fsmonitor.c\n@@ -0,0 +1,253 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"ewah/ewok.h\"\n+#include \"fsmonitor.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+\n+#define INDEX_EXTENSION_VERSION\t(1)\n+#define HOOK_INTERFACE_VERSION\t(1)\n+\n+struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR);\n+\n+static void fsmonitor_ewah_callback(size_t pos, void *is)\n+{\n+\tstruct index_state *istate = (struct index_state *)is;\n+\tstruct cache_entry *ce = istate->cache[pos];\n+\n+\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+}\n+\n+int read_fsmonitor_extension(struct index_state *istate, const void *data,\n+\tunsigned long sz)\n+{\n+\tconst char *index = data;\n+\tuint32_t hdr_version;\n+\tuint32_t ewah_size;\n+\tstruct ewah_bitmap *fsmonitor_dirty;\n+\tint i;\n+\tint ret;\n+\n+\tif (sz < sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t))\n+\t\treturn error(\"corrupt fsmonitor extension (too short)\");\n+\n+\thdr_version = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\tif (hdr_version != INDEX_EXTENSION_VERSION)\n+\t\treturn error(\"bad fsmonitor version %d\", hdr_version);\n+\n+\tistate->fsmonitor_last_update = get_be64(index);\n+\tindex += sizeof(uint64_t);\n+\n+\tewah_size = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\n+\tfsmonitor_dirty = ewah_new();\n+\tret = ewah_read_mmap(fsmonitor_dirty, index, ewah_size);\n+\tif (ret != ewah_size) {\n+\t\tewah_free(fsmonitor_dirty);\n+\t\treturn error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n+\t}\n+\n+\tif (git_config_get_fsmonitor()) {\n+\t\t/* Mark all entries valid */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n+\n+\t\t/* Mark all previously saved entries as dirty */\n+\t\tewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n+\t\tewah_free(fsmonitor_dirty);\n+\n+\t\t/* Now mark the untracked cache for fsmonitor usage */\n+\t\tif (istate->untracked)\n+\t\t\tistate->untracked->use_fsmonitor = 1;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"read fsmonitor extension successful\");\n+\treturn 0;\n+}\n+\n+void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n+{\n+\tuint32_t hdr_version;\n+\tuint64_t tm;\n+\tstruct ewah_bitmap *bitmap;\n+\tint i;\n+\tuint32_t ewah_start;\n+\tuint32_t ewah_size = 0;\n+\tint fixup = 0;\n+\n+\tput_be32(&hdr_version, INDEX_EXTENSION_VERSION);\n+\tstrbuf_add(sb, &hdr_version, sizeof(uint32_t));\n+\n+\tput_be64(&tm, istate->fsmonitor_last_update);\n+\tstrbuf_add(sb, &tm, sizeof(uint64_t));\n+\tfixup = sb->len;\n+\tstrbuf_add(sb, &ewah_size, sizeof(uint32_t)); /* we'll fix this up later */\n+\n+\tewah_start = sb->len;\n+\tbitmap = ewah_new();\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))\n+\t\t\tewah_set(bitmap, i);\n+\tewah_serialize_strbuf(bitmap, sb);\n+\tewah_free(bitmap);\n+\n+\t/* fix up size field */\n+\tput_be32(&ewah_size, sb->len - ewah_start);\n+\tmemcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t));\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"write fsmonitor extension successful\");\n+}\n+\n+/*\n+ * Call the query-fsmonitor hook passing the time of the last saved results.\n+ */\n+static int query_fsmonitor(int version, uint64_t last_update, struct strbuf *query_result)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tchar ver[64];\n+\tchar date[64];\n+\tconst char *argv[4];\n+\n+\tif (!(argv[0] = core_fsmonitor))\n+\t\treturn -1;\n+\n+\tsnprintf(ver, sizeof(version), \"%d\", version);\n+\tsnprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update);\n+\targv[1] = ver;\n+\targv[2] = date;\n+\targv[3] = NULL;\n+\tcp.argv = argv;\n+\tcp.use_shell = 1;\n+\n+\treturn capture_command(&cp, query_result, 1024);\n+}\n+\n+static void fsmonitor_refresh_callback(struct index_state *istate, const char *name)\n+{\n+\tint pos = index_name_pos(istate, name, strlen(name));\n+\n+\tif (pos >= 0) {\n+\t\tstruct cache_entry *ce = istate->cache[pos];\n+\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t}\n+\n+\t/*\n+\t * Mark the untracked cache dirty even if it wasn't found in the index\n+\t * as it could be a new untracked file.\n+\t */\n+\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n+\tuntracked_cache_invalidate_path(istate, name);\n+}\n+\n+void refresh_fsmonitor(struct index_state *istate)\n+{\n+\tstatic int has_run_once = 0;\n+\tstruct strbuf query_result = STRBUF_INIT;\n+\tint query_success = 0;\n+\tsize_t bol; /* beginning of line */\n+\tuint64_t last_update;\n+\tchar *buf;\n+\tint i;\n+\n+\tif (!core_fsmonitor || has_run_once)\n+\t\treturn;\n+\thas_run_once = 1;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\t/*\n+\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * should be inclusive to ensure we don't miss potential changes.\n+\t */\n+\tlast_update = getnanotime();\n+\n+\t/*\n+\t * If we have a last update time, call query_fsmonitor for the set of\n+\t * changes since that time, else assume everything is possibly dirty\n+\t * and check it all.\n+\t */\n+\tif (istate->fsmonitor_last_update) {\n+\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION,\n+\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n+\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n+\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t}\n+\n+\t/* a fsmonitor process can return '*' to indicate all entries are invalid */\n+\tif (query_success && query_result.buf[0] != '*') {\n+\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\tbuf = query_result.buf;\n+\t\tbol = 0;\n+\t\tfor (i = 0; i < query_result.len; i++) {\n+\t\t\tif (buf[i] != '\\0')\n+\t\t\t\tcontinue;\n+\t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tbol = i + 1;\n+\t\t}\n+\t\tif (bol < query_result.len)\n+\t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t} else {\n+\t\t/* Mark all entries invalid */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\n+\t\tif (istate->untracked)\n+\t\t\tistate->untracked->use_fsmonitor = 0;\n+\t}\n+\tstrbuf_release(&query_result);\n+\n+\t/* Now that we've updated istate, save the last_update time */\n+\tistate->fsmonitor_last_update = last_update;\n+}\n+\n+void add_fsmonitor(struct index_state *istate)\n+{\n+\tint i;\n+\n+\tif (!istate->fsmonitor_last_update) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"add fsmonitor\");\n+\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\tistate->fsmonitor_last_update = getnanotime();\n+\n+\t\t/* reset the fsmonitor state */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\n+\t\t/* reset the untracked cache */\n+\t\tif (istate->untracked) {\n+\t\t\tadd_untracked_cache(istate);\n+\t\t\tistate->untracked->use_fsmonitor = 1;\n+\t\t}\n+\n+\t\t/* Update the fsmonitor state */\n+\t\trefresh_fsmonitor(istate);\n+\t}\n+}\n+\n+void remove_fsmonitor(struct index_state *istate)\n+{\n+\tif (istate->fsmonitor_last_update) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"remove fsmonitor\");\n+\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\tistate->fsmonitor_last_update = 0;\n+\t}\n+}\n+\n+void tweak_fsmonitor(struct index_state *istate)\n+{\n+\tswitch (git_config_get_fsmonitor()) {\n+\tcase -1: /* keep: do nothing */\n+\t\tbreak;\n+\tcase 0: /* false */\n+\t\tremove_fsmonitor(istate);\n+\t\tbreak;\n+\tcase 1: /* true */\n+\t\tadd_fsmonitor(istate);\n+\t\tbreak;\n+\tdefault: /* unknown value: do nothing */\n+\t\tbreak;\n+\t}\n+}\ndiff --git a/fsmonitor.h b/fsmonitor.h\nnew file mode 100644\nindex 0000000000..dadbe90283\n--- /dev/null\n+++ b/fsmonitor.h\n@@ -0,0 +1,61 @@\n+#ifndef FSMONITOR_H\n+#define FSMONITOR_H\n+\n+extern struct trace_key trace_fsmonitor;\n+\n+/*\n+ * Read the the fsmonitor index extension and (if configured) restore the\n+ * CE_FSMONITOR_VALID state.\n+ */\n+extern int read_fsmonitor_extension(struct index_state *istate, const void *data, unsigned long sz);\n+\n+/*\n+ * Write the CE_FSMONITOR_VALID state into the fsmonitor index extension.\n+ */\n+extern void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate);\n+\n+/*\n+ * Add/remove the fsmonitor index extension\n+ */\n+extern void add_fsmonitor(struct index_state *istate);\n+extern void remove_fsmonitor(struct index_state *istate);\n+\n+/*\n+ * Add/remove the fsmonitor index extension as necessary based on the current\n+ * core.fsmonitor setting.\n+ */\n+extern void tweak_fsmonitor(struct index_state *istate);\n+\n+/*\n+ * Run the configured fsmonitor integration script and clear the\n+ * CE_FSMONITOR_VALID bit for any files returned as dirty. Also invalidate\n+ * any corresponding untracked cache directory structures. Optimized to only\n+ * run the first time it is called.\n+ */\n+extern void refresh_fsmonitor(struct index_state *istate);\n+\n+/*\n+ * Set the given cache entries CE_FSMONITOR_VALID bit.\n+ */\n+static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n+{\n+\tif (core_fsmonitor) {\n+\t\tce->ce_flags |= CE_FSMONITOR_VALID;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n+\t}\n+}\n+\n+/*\n+ * Clear the given cache entries CE_FSMONITOR_VALID bit and invalidate any\n+ * corresponding untracked cache directory structures.\n+ */\n+static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n+{\n+\tif (core_fsmonitor) {\n+\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tuntracked_cache_invalidate_path(istate, ce->name);\n+\t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\n+\t}\n+}\n+\n+#endif\ndiff --git a/preload-index.c b/preload-index.c\nindex 75564c497a..2a83255e4e 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -4,6 +4,7 @@\n #include \"cache.h\"\n #include \"pathspec.h\"\n #include \"dir.h\"\n+#include \"fsmonitor.h\"\n \n #ifdef NO_PTHREADS\n static void preload_index(struct index_state *index,\n@@ -55,15 +56,18 @@ static void *preload_thread(void *_data)\n \t\t\tcontinue;\n \t\tif (ce_skip_worktree(ce))\n \t\t\tcontinue;\n+\t\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n+\t\t\tcontinue;\n \t\tif (!ce_path_match(ce, &p->pathspec, NULL))\n \t\t\tcontinue;\n \t\tif (threaded_has_symlink_leading_path(&cache, ce->name, ce_namelen(ce)))\n \t\t\tcontinue;\n \t\tif (lstat(ce->name, &st))\n \t\t\tcontinue;\n-\t\tif (ie_match_stat(index, ce, &st, CE_MATCH_RACY_IS_DIRTY))\n+\t\tif (ie_match_stat(index, ce, &st, CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR))\n \t\t\tcontinue;\n \t\tce_mark_uptodate(ce);\n+\t\tmark_fsmonitor_valid(ce);\n \t} while (--nr > 0);\n \tcache_def_clear(&cache);\n \treturn NULL;\ndiff --git a/read-cache.c b/read-cache.c\nindex 40da87ea71..53093dbebf 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -19,6 +19,7 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"utf8.h\"\n+#include \"fsmonitor.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -38,11 +39,12 @@\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n #define CACHE_EXT_UNTRACKED 0x554E5452\t /* \"UNTR\" */\n+#define CACHE_EXT_FSMONITOR 0x46534D4E\t /* \"FSMN\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -62,6 +64,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \tfree(old);\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tmark_fsmonitor_invalid(istate, ce);\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n }\n \n@@ -150,8 +153,10 @@ void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n \tif (assume_unchanged)\n \t\tce->ce_flags |= CE_VALID;\n \n-\tif (S_ISREG(st->st_mode))\n+\tif (S_ISREG(st->st_mode)) {\n \t\tce_mark_uptodate(ce);\n+\t\tmark_fsmonitor_valid(ce);\n+\t}\n }\n \n static int ce_compare_data(const struct cache_entry *ce, struct stat *st)\n@@ -300,7 +305,7 @@ int match_stat_data_racy(const struct index_state *istate,\n \treturn match_stat_data(sd, st);\n }\n \n-int ie_match_stat(const struct index_state *istate,\n+int ie_match_stat(struct index_state *istate,\n \t\t const struct cache_entry *ce, struct stat *st,\n \t\t unsigned int options)\n {\n@@ -308,7 +313,10 @@ int ie_match_stat(const struct index_state *istate,\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n \tint assume_racy_is_modified = options & CE_MATCH_RACY_IS_DIRTY;\n+\tint ignore_fsmonitor = options & CE_MATCH_IGNORE_FSMONITOR;\n \n+\tif (!ignore_fsmonitor)\n+\t\trefresh_fsmonitor(istate);\n \t/*\n \t * If it's marked as always valid in the index, it's\n \t * valid whatever the checked-out copy says.\n@@ -319,6 +327,8 @@ int ie_match_stat(const struct index_state *istate,\n \t\treturn 0;\n \tif (!ignore_valid && (ce->ce_flags & CE_VALID))\n \t\treturn 0;\n+\tif (!ignore_fsmonitor && (ce->ce_flags & CE_FSMONITOR_VALID))\n+\t\treturn 0;\n \n \t/*\n \t * Intent-to-add entries have not been added, so the index entry\n@@ -356,7 +366,7 @@ int ie_match_stat(const struct index_state *istate,\n \treturn changed;\n }\n \n-int ie_modified(const struct index_state *istate,\n+int ie_modified(struct index_state *istate,\n \t\tconst struct cache_entry *ce,\n \t\tstruct stat *st, unsigned int options)\n {\n@@ -631,7 +641,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \tint size, namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias;\n-\tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n+\tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR;\n \tint verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);\n \tint pretend = flags & ADD_CACHE_PRETEND;\n \tint intent_only = flags & ADD_CACHE_INTENT;\n@@ -777,6 +787,7 @@ int chmod_index_entry(struct index_state *istate, struct cache_entry *ce,\n \t}\n \tcache_tree_invalidate_path(istate, ce->name);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tmark_fsmonitor_invalid(istate, ce);\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n \n \treturn 0;\n@@ -1228,10 +1239,13 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n \tint ignore_missing = options & CE_MATCH_IGNORE_MISSING;\n+\tint ignore_fsmonitor = options & CE_MATCH_IGNORE_FSMONITOR;\n \n \tif (!refresh || ce_uptodate(ce))\n \t\treturn ce;\n \n+\tif (!ignore_fsmonitor)\n+\t\trefresh_fsmonitor(istate);\n \t/*\n \t * CE_VALID or CE_SKIP_WORKTREE means the user promised us\n \t * that the change to the work tree does not matter and told\n@@ -1245,6 +1259,10 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\tce_mark_uptodate(ce);\n \t\treturn ce;\n \t}\n+\tif (!ignore_fsmonitor && (ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tce_mark_uptodate(ce);\n+\t\treturn ce;\n+\t}\n \n \tif (has_symlink_leading_path(ce->name, ce_namelen(ce))) {\n \t\tif (ignore_missing)\n@@ -1282,8 +1300,10 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\t\t * because CE_UPTODATE flag is in-core only;\n \t\t\t * we are not going to write this change out.\n \t\t\t */\n-\t\t\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\t\tif (!S_ISGITLINK(ce->ce_mode)) {\n \t\t\t\tce_mark_uptodate(ce);\n+\t\t\t\tmark_fsmonitor_valid(ce);\n+\t\t\t}\n \t\t\treturn ce;\n \t\t}\n \t}\n@@ -1336,7 +1356,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tint first = 1;\n \tint in_porcelain = (flags & REFRESH_IN_PORCELAIN);\n \tunsigned int options = (CE_MATCH_REFRESH |\n-\t\t\t\t(really ? CE_MATCH_IGNORE_VALID : 0) |\n+\t\t\t\t(really ? CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_FSMONITOR : 0) |\n \t\t\t\t(not_new ? CE_MATCH_IGNORE_MISSING : 0));\n \tconst char *modified_fmt;\n \tconst char *deleted_fmt;\n@@ -1391,6 +1411,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\t\t */\n \t\t\t\tce->ce_flags &= ~CE_VALID;\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\t\t\tmark_fsmonitor_invalid(istate, ce);\n \t\t\t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t\t\t}\n \t\t\tif (quiet)\n@@ -1550,6 +1571,9 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_UNTRACKED:\n \t\tistate->untracked = read_untracked_extension(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_FSMONITOR:\n+\t\tread_fsmonitor_extension(istate, data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1722,6 +1746,7 @@ static void post_read_index_from(struct index_state *istate)\n \tcheck_ce_order(istate);\n \ttweak_untracked_cache(istate);\n \ttweak_split_index(istate);\n+\ttweak_fsmonitor(istate);\n }\n \n /* remember to discard_cache() before reading a different cache! */\n@@ -2306,6 +2331,16 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->fsmonitor_last_update) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_fsmonitor_extension(&sb, istate);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_FSMONITOR, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1))\n \t\treturn -1;\ndiff --git a/submodule.c b/submodule.c\nindex 3cea8221e0..8a931a1aaa 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -62,7 +62,7 @@ int is_staging_gitmodules_ok(const struct index_state *istate)\n \tif ((pos >= 0) && (pos < istate->cache_nr)) {\n \t\tstruct stat st;\n \t\tif (lstat(GITMODULES_FILE, &st) == 0 &&\n-\t\t ce_match_stat(istate->cache[pos], &st, 0) & DATA_CHANGED)\n+\t\t ce_match_stat(istate->cache[pos], &st, CE_MATCH_IGNORE_FSMONITOR) & DATA_CHANGED)\n \t\t\treturn 0;\n \t}\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 71b70ccb12..f724a61ac0 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -408,6 +409,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n \tif (was_skip_worktree != ce_skip_worktree(ce)) {\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(istate, ce);\n \t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \n@@ -1454,7 +1456,7 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n \t\treturn 0;\n \n \tif (!lstat(ce->name, &st)) {\n-\t\tint flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE;\n+\t\tint flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR;\n \t\tunsigned changed = ie_match_stat(o->src_index, ce, &st, flags);\n \n \t\tif (submodule_from_ce(ce)) {\n@@ -1610,7 +1612,7 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le\n \tconst struct cache_entry *src;\n \n \tsrc = index_file_exists(o->src_index, name, len, 1);\n-\treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n+\treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n }\n \n static int check_ok_to_remove(const char *name, int len, int dtype,\n@@ -2134,7 +2136,7 @@ int oneway_merge(const struct cache_entry * const *src,\n \t\tif (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old)) {\n \t\t\tstruct stat st;\n \t\t\tif (lstat(old->name, &st) ||\n-\t\t\t ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE))\n+\t\t\t ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR))\n \t\t\t\tupdate |= CE_UPDATE;\n \t\t}\n \t\tadd_entry(o, old, update, 0);\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328147","messageId":"20170915192043.4516-6-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:36Z","receivedAt":"2017-09-15T19:21:43Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This includes the core.fsmonitor setting, the query-fsmonitor hook,\nand the fsmonitor index extension.\n\nSigned-off-by: Ben Peart \n---\n Documentation/config.txt | 6 ++++++\n Documentation/githooks.txt | 23 +++++++++++++++++++++++\n Documentation/technical/index-format.txt | 19 +++++++++++++++++++\n 3 files changed, 48 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dc4e3f58a2..c196007a27 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -413,6 +413,12 @@ core.protectNTFS::\n \t8.3 \"short\" names.\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n+core.fsmonitor::\n+\tIf set, the value of this variable is used as a command which\n+\twill identify all files that may have changed since the\n+\trequested date/time. This information is used to speed up git by\n+\tavoiding unnecessary processing of files that have not changed.\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 1bb4f92d4d..da82d64b0b 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -456,6 +456,29 @@ non-zero status causes 'git send-email' to abort before sending any\n e-mails.\n \n \n+[[fsmonitor-watchman]]\n+fsmonitor-watchman\n+~~~~~~~~~~~~~~~\n+\n+This hook is invoked when the configuration option core.fsmonitor is\n+set to .git/hooks/fsmonitor-watchman. It takes two arguments, a version\n+(currently 1) and the time in elapsed nanoseconds since midnight,\n+January 1, 1970.\n+\n+The hook should output to stdout the list of all files in the working\n+directory that may have changed since the requested time. The logic\n+should be inclusive so that it does not miss any potential changes.\n+The paths should be relative to the root of the working directory\n+and be separated by a single NUL.\n+\n+Git will limit what files it checks for changes as well as which\n+directories are checked for untracked files based on the path names\n+given.\n+\n+The exit status determines whether git will use the data from the\n+hook to limit its search. On error, it will fall back to verifying\n+all files and folders.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex ade0b0c445..db3572626b 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -295,3 +295,22 @@ The remaining data of each directory block is grouped by type:\n in the previous ewah bitmap.\n \n - One NUL.\n+\n+== File System Monitor cache\n+\n+ The file system monitor cache tracks files for which the core.fsmonitor\n+ hook has told us about changes. The signature for this extension is\n+ { 'F', 'S', 'M', 'N' }.\n+\n+ The extension starts with\n+\n+ - 32-bit version number: the current supported version is 1.\n+\n+ - 64-bit time: the extension data reflects all changes through the given\n+\ttime which is stored as the nanoseconds elapsed since midnight,\n+\tJanuary 1, 1970.\n+\n+ - 32-bit bitmap size: the size of the CE_FSMONITOR_VALID bitmap.\n+\n+ - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n+ is not CE_FSMONITOR_VALID.\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328148","messageId":"20170915192043.4516-8-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 07/12] update-index: add fsmonitor support to update-index","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:38Z","receivedAt":"2017-09-15T19:21:48Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add support in update-index to manually add/remove the fsmonitor\nextension via --fsmonitor/--no-fsmonitor flags\n\nSigned-off-by: Ben Peart \n---\n builtin/update-index.c | 19 +++++++++++++++++++\n 1 file changed, 19 insertions(+)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 6f39ee9274..b03afc1f3a 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -918,6 +918,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tint lock_error = 0;\n \tint split_index = -1;\n \tint force_write = 0;\n+\tint fsmonitor = -1;\n \tstruct lock_file *lock_file;\n \tstruct parse_opt_ctx_t ctx;\n \tstrbuf_getline_fn getline_fn;\n@@ -1011,6 +1012,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"enable untracked cache without testing the filesystem\"), UC_FORCE),\n \t\tOPT_SET_INT(0, \"force-write-index\", &force_write,\n \t\t\tN_(\"write out the index even if is not flagged as changed\"), 1),\n+\t\tOPT_BOOL(0, \"fsmonitor\", &fsmonitor,\n+\t\t\tN_(\"enable or disable file system monitor\")),\n \t\tOPT_END()\n \t};\n \n@@ -1152,6 +1155,22 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n \t}\n \n+\tif (fsmonitor > 0) {\n+\t\tif (git_config_get_fsmonitor() == 0)\n+\t\t\twarning(_(\"core.fsmonitor is unset; \"\n+\t\t\t\t\"set it if you really want to \"\n+\t\t\t\t\"enable fsmonitor\"));\n+\t\tadd_fsmonitor(&the_index);\n+\t\treport(_(\"fsmonitor enabled\"));\n+\t} else if (!fsmonitor) {\n+\t\tif (git_config_get_fsmonitor() == 1)\n+\t\t\twarning(_(\"core.fsmonitor is set; \"\n+\t\t\t\t\"remove it if you really want to \"\n+\t\t\t\t\"disable fsmonitor\"));\n+\t\tremove_fsmonitor(&the_index);\n+\t\treport(_(\"fsmonitor disabled\"));\n+\t}\n+\n \tif (active_cache_changed || force_write) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328149","messageId":"20170915192043.4516-7-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:37Z","receivedAt":"2017-09-15T19:21:50Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a new command line option (-f) to ls-files to have it use lowercase\nletters for 'fsmonitor valid' files\n\nSigned-off-by: Ben Peart \n---\n builtin/ls-files.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex e1339e6d17..313962a0c1 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -31,6 +31,7 @@ static int show_resolve_undo;\n static int show_modified;\n static int show_killed;\n static int show_valid_bit;\n+static int show_fsmonitor_bit;\n static int line_terminator = '\\n';\n static int debug_mode;\n static int show_eol;\n@@ -86,7 +87,8 @@ static const char *get_tag(const struct cache_entry *ce, const char *tag)\n {\n \tstatic char alttag[4];\n \n-\tif (tag && *tag && show_valid_bit && (ce->ce_flags & CE_VALID)) {\n+\tif (tag && *tag && ((show_valid_bit && (ce->ce_flags & CE_VALID)) ||\n+\t\t(show_fsmonitor_bit && (ce->ce_flags & CE_FSMONITOR_VALID)))) {\n \t\tmemcpy(alttag, tag, 3);\n \n \t\tif (isalpha(tag[0])) {\n@@ -515,6 +517,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\t\tN_(\"identify the file status with tags\")),\n \t\tOPT_BOOL('v', NULL, &show_valid_bit,\n \t\t\tN_(\"use lowercase letters for 'assume unchanged' files\")),\n+\t\tOPT_BOOL('f', NULL, &show_fsmonitor_bit,\n+\t\t\tN_(\"use lowercase letters for 'fsmonitor clean' files\")),\n \t\tOPT_BOOL('c', \"cached\", &show_cached,\n \t\t\tN_(\"show cached files in the output (default)\")),\n \t\tOPT_BOOL('d', \"deleted\", &show_deleted,\n@@ -584,7 +588,7 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tfor (i = 0; i < exclude_list.nr; i++) {\n \t\tadd_exclude(exclude_list.items[i].string, \"\", 0, el, --exclude_args);\n \t}\n-\tif (show_tag || show_valid_bit) {\n+\tif (show_tag || show_valid_bit || show_fsmonitor_bit) {\n \t\ttag_cached = \"H \";\n \t\ttag_unmerged = \"M \";\n \t\ttag_removed = \"R \";\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328150","messageId":"20170915192043.4516-11-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:41Z","receivedAt":"2017-09-15T19:21:52Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Test the ability to add/remove the fsmonitor index extension via\nupdate-index.\n\nTest that dirty files returned from the integration script are properly\nrepresented in the index extension and verify that ls-files correctly\nreports their state.\n\nTest that ensure status results are correct when using the new fsmonitor\nextension. Test untracked, modified, and new files by ensuring the\nresults are identical to when not using the extension.\n\nTest that if the fsmonitor extension doesn't tell git about a change, it\ndoesn't discover it on its own. This ensures git is honoring the\nextension and that we get the performance benefits desired.\n\nThree test integration scripts are provided:\n\nfsmonitor-all - marks all files as dirty\nfsmonitor-none - marks no files as dirty\nfsmonitor-watchman - integrates with Watchman with debug logging\n\nTo run tests in the test suite while utilizing fsmonitor:\n\nFirst copy t/t7519/fsmonitor-all to a location in your path and then set\nGIT_FORCE_PRELOAD_TEST=true and GIT_FSMONITOR_TEST=fsmonitor-all and run\nyour tests.\n\nNote: currently when using the test script fsmonitor-watchman on\nWindows, many tests fail due to a reported but not yet fixed bug in\nWatchman where it holds on to handles for directories and files which\nprevents the test directory from being cleaned up properly.\n\nSigned-off-by: Ben Peart \n---\n t/t7519-status-fsmonitor.sh | 259 ++++++++++++++++++++++++++++++++++++++++++++\n t/t7519/fsmonitor-all | 23 ++++\n t/t7519/fsmonitor-none | 21 ++++\n t/t7519/fsmonitor-watchman | 128 ++++++++++++++++++++++\n 4 files changed, 431 insertions(+)\n create mode 100755 t/t7519-status-fsmonitor.sh\n create mode 100755 t/t7519/fsmonitor-all\n create mode 100755 t/t7519/fsmonitor-none\n create mode 100755 t/t7519/fsmonitor-watchman\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nnew file mode 100755\nindex 0000000000..6aa1e4e924\n--- /dev/null\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -0,0 +1,259 @@\n+#!/bin/sh\n+\n+test_description='git status with file system watcher'\n+\n+. ./test-lib.sh\n+\n+#\n+# To run the entire git test suite using fsmonitor:\n+#\n+# copy t/t7519/fsmonitor-all to a location in your path and then set\n+# GIT_FSMONITOR_TEST=fsmonitor-all and run your tests.\n+#\n+\n+# Note, after \"git reset --hard HEAD\" no extensions exist other than 'TREE'\n+# \"git update-index --fsmonitor\" can be used to get the extension written\n+# before testing the results.\n+\n+clean_repo () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd\n+}\n+\n+dirty_repo () {\n+\t: >untracked &&\n+\t: >dir1/untracked &&\n+\t: >dir2/untracked &&\n+\techo 1 >modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\techo 4 >new &&\n+\techo 5 >dir1/new &&\n+\techo 6 >dir2/new\n+}\n+\n+write_integration_script() {\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF\n+\tif [ \"$#\" -ne 2 ]; then\n+\t\techo \"$0: exactly 2 arguments expected\"\n+\t\texit 2\n+\tfi\n+\tif [ \"$1\" != 1 ]; then\n+\t\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n+\t\texit 1\n+\tfi\n+\tprintf \"untracked\\0\"\n+\tprintf \"dir1/untracked\\0\"\n+\tprintf \"dir2/untracked\\0\"\n+\tprintf \"modified\\0\"\n+\tprintf \"dir1/modified\\0\"\n+\tprintf \"dir2/modified\\0\"\n+\tprintf \"new\\0\"\n+\tprintf \"dir1/new\\0\"\n+\tprintf \"dir2/new\\0\"\n+\tEOF\n+}\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_expect_success 'setup' '\n+\tmkdir -p .git/hooks &&\n+\t: >tracked &&\n+\t: >modified &&\n+\tmkdir dir1 &&\n+\t: >dir1/tracked &&\n+\t: >dir1/modified &&\n+\tmkdir dir2 &&\n+\t: >dir2/tracked &&\n+\t: >dir2/modified &&\n+\tgit -c core.fsmonitor= add . &&\n+\tgit -c core.fsmonitor= commit -m initial &&\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcat >.gitignore <<-\\EOF\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tmarker*\n+\tEOF\n+'\n+\n+# test that the fsmonitor extension is off by default\n+test_expect_success 'fsmonitor extension is off by default' '\n+\ttest-dump-fsmonitor >actual &&\n+\tgrep \"^no fsmonitor\" actual\n+'\n+\n+# test that \"update-index --fsmonitor\" adds the fsmonitor extension\n+test_expect_success 'update-index --fsmonitor\" adds the fsmonitor extension' '\n+\tgit update-index --fsmonitor &&\n+\ttest-dump-fsmonitor >actual &&\n+\tgrep \"^fsmonitor last update\" actual\n+'\n+\n+# test that \"update-index --no-fsmonitor\" removes the fsmonitor extension\n+test_expect_success 'update-index --no-fsmonitor\" removes the fsmonitor extension' '\n+\tgit update-index --no-fsmonitor &&\n+\ttest-dump-fsmonitor >actual &&\n+\tgrep \"^no fsmonitor\" actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\ttest_must_fail git update-index --refresh --force-write-index &&\n+\tgit ls-files -f >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# Ensure commands that call refresh_index() to move the index back in time\n+# properly invalidate the fsmonitor cache\n+test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\tEOF\n+\tclean_repo &&\n+\tdirty_repo &&\n+\tgit add . &&\n+\tgit commit -m \"to reset\" &&\n+\tgit reset HEAD~1 &&\n+\tgit status >actual &&\n+\tgit -c core.fsmonitor= status >expect &&\n+\ttest_i18ncmp expect actual\n+'\n+\n+# test fsmonitor with and without preloadIndex\n+preload_values=\"false true\"\n+for preload_val in $preload_values\n+do\n+\ttest_expect_success \"setup preloadIndex to $preload_val\" '\n+\t\tgit config core.preloadIndex $preload_val &&\n+\t\tif [ $preload_val -eq true ]\n+\t\tthen\n+\t\t\tGIT_FORCE_PRELOAD_TEST=$preload_val; export GIT_FORCE_PRELOAD_TEST\n+\t\telse\n+\t\t\tunset GIT_FORCE_PRELOAD_TEST\n+\t\tfi\n+\t'\n+\n+\t# test fsmonitor with and without the untracked cache (if available)\n+\tuc_values=\"false\"\n+\ttest_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\ttest_expect_success \"setup untracked cache to $uc_val\" '\n+\t\t\tgit config core.untrackedcache $uc_val\n+\t\t'\n+\n+\t\t# Status is well tested elsewhere so we'll just ensure that the results are\n+\t\t# the same when using core.fsmonitor.\n+\t\ttest_expect_success 'compare status with and without fsmonitor' '\n+\t\t\twrite_integration_script &&\n+\t\t\tclean_repo &&\n+\t\t\tdirty_repo &&\n+\t\t\tgit add new &&\n+\t\t\tgit add dir1/new &&\n+\t\t\tgit add dir2/new &&\n+\t\t\tgit status >actual &&\n+\t\t\tgit -c core.fsmonitor= status >expect &&\n+\t\t\ttest_i18ncmp expect actual\n+\t\t'\n+\n+\t\t# Make sure it's actually skipping the check for modified and untracked\n+\t\t# (if enabled) files unless it is told about them.\n+\t\ttest_expect_success \"status doesn't detect unreported modifications\" '\n+\t\t\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\t\t:>marker\n+\t\t\tEOF\n+\t\t\tclean_repo &&\n+\t\t\tgit status &&\n+\t\t\ttest_path_is_file marker &&\n+\t\t\tdirty_repo &&\n+\t\t\trm -f marker &&\n+\t\t\tgit status >actual &&\n+\t\t\ttest_path_is_file marker &&\n+\t\t\ttest_i18ngrep ! \"Changes not staged for commit:\" actual &&\n+\t\t\tif [ $uc_val -eq true ]; then test_i18ngrep ! \"Untracked files:\" actual; fi &&\n+\t\t\tif [ $uc_val -eq false ]; then test_i18ngrep \"Untracked files:\" actual; fi &&\n+\t\t\trm -f marker\n+\t\t'\n+\tdone\n+done\n+\n+test_done\ndiff --git a/t/t7519/fsmonitor-all b/t/t7519/fsmonitor-all\nnew file mode 100755\nindex 0000000000..a3870e431e\n--- /dev/null\n+++ b/t/t7519/fsmonitor-all\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+#\n+# An test hook script to integrate with git to test fsmonitor.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+#echo \"$0 $*\" >&2\n+\n+if [ \"$#\" -ne 2 ] ; then\n+\techo -e \"$0: exactly 2 arguments expected\\n\" >&2\n+\texit 2\n+fi\n+\n+if [ \"$1\" != 1 ]\n+then\n+\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n+\texit 1\n+fi\n+\n+echo \"*\"\n\\ No newline at end of file\ndiff --git a/t/t7519/fsmonitor-none b/t/t7519/fsmonitor-none\nnew file mode 100755\nindex 0000000000..c500bb0f26\n--- /dev/null\n+++ b/t/t7519/fsmonitor-none\n@@ -0,0 +1,21 @@\n+#!/bin/sh\n+#\n+# An test hook script to integrate with git to test fsmonitor.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+#echo \"$0 $*\" >&2\n+\n+if [ \"$#\" -ne 2 ] ; then\n+\techo -e \"$0: exactly 2 arguments expected\\n\" >&2\n+\texit 2\n+fi\n+\n+if [ \"$1\" != 1 ]\n+then\n+\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n+\texit 1\n+fi\ndiff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\nnew file mode 100755\nindex 0000000000..aaee5d1fe3\n--- /dev/null\n+++ b/t/t7519/fsmonitor-watchman\n@@ -0,0 +1,128 @@\n+#!/usr/bin/perl\n+\n+use strict;\n+use warnings;\n+use IPC::Open2;\n+\n+# An example hook script to integrate Watchman\n+# (https://facebook.github.io/watchman/) with git to speed up detecting\n+# new and modified files.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+# To enable this hook, rename this file to \"query-watchman\" and set\n+# 'git config core.fsmonitor .git/hooks/query-watchman'\n+#\n+my ($version, $time) = @ARGV;\n+print STDERR \"$0 $version $time\\n\";\n+\n+# Check the hook interface version\n+\n+if ($version == 1) {\n+\t# convert nanoseconds to seconds\n+\t$time = int $time / 1000000000;\n+} else {\n+\tdie \"Unsupported query-fsmonitor hook version '$version'.\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+}\n+\n+my $git_work_tree = $ENV{'PWD'};\n+\n+my $retry = 1;\n+\n+launch_watchman();\n+\n+sub launch_watchman {\n+\n+\t# Set input record separator\n+\tlocal $/ = 0666;\n+\n+\tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j')\n+\t or die \"open2() failed: $!\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+\n+\t# In the query expression below we're asking for names of files that\n+\t# changed since $time but were not transient (ie created after\n+\t# $time but no longer exist).\n+\t#\n+\t# To accomplish this, we're using the \"since\" generator to use the\n+\t# recency index to select candidate nodes and \"fields\" to limit the\n+\t# output to file names only. Then we're using the \"expression\" term to\n+\t# further constrain the results.\n+\t#\n+\t# The category of transient files that we want to ignore will have a\n+\t# creation clock (cclock) newer than $time_t value and will also not\n+\t# currently exist.\n+\n+\topen (my $fh, \">\", \".git/watchman-query.json\");\n+\tprint $fh \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n+\t\\\"since\\\": $time, \\\n+\t\\\"fields\\\": [\\\"name\\\"], \\\n+\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n+\t}]\";\n+\tclose $fh;\n+\n+\tprint CHLD_IN \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n+\t\\\"since\\\": $time, \\\n+\t\\\"fields\\\": [\\\"name\\\"], \\\n+\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n+\t}]\";\n+\n+\tmy $response = ;\n+\n+\topen ($fh, \">\", \".git/watchman-response.json\");\n+\tprint $fh $response;\n+\tclose $fh;\n+\n+\tdie \"Watchman: command returned no output.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $response eq \"\";\n+\tdie \"Watchman: command returned invalid output: $response\\n\" .\n+\t \"Falling back to scanning...\\n\" unless $response =~ /^\\{/;\n+\n+\tmy $json_pkg;\n+\teval {\n+\t\trequire JSON::XS;\n+\t\t$json_pkg = \"JSON::XS\";\n+\t\t1;\n+\t} or do {\n+\t\trequire JSON::PP;\n+\t\t$json_pkg = \"JSON::PP\";\n+\t};\n+\n+\tmy $o = $json_pkg->new->utf8->decode($response);\n+\n+\tif ($retry > 0 and $o->{error} and $o->{error} =~ m/unable to resolve root .* directory (.*) is not watched/) {\n+\t\tprint STDERR \"Adding '$git_work_tree' to watchman's watch list.\\n\";\n+\t\t$retry--;\n+\t\tqx/watchman watch \"$git_work_tree\"/;\n+\t\tdie \"Failed to make watchman watch '$git_work_tree'.\\n\" .\n+\t\t \"Falling back to scanning...\\n\" if $? != 0;\n+\t\t# return fast \"everything is dirty\" flag\"\n+\t\tprint \"*\\0\";\n+\t\topen ($fh, \">\", \".git/watchman-output.out\");\n+\t\tprint \"*\\0\";\n+\t\tclose $fh;\n+\n+\t\t# Watchman will always return all files on the first query so\n+\t\t# return the fast \"everything is dirty\" flag to git and do the\n+\t\t# Watchman query just to get it over with now so we won't pay\n+\t\t# the cost in git to look up each individual file.\n+\t\tprint \"*\\0\";\n+\t\teval { launch_watchman() };\n+\t\texit 0;\n+\t}\n+\n+\tdie \"Watchman: $o->{error}.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $o->{error};\n+\n+\tbinmode STDOUT, \":utf8\";\n+\tlocal $, = \"\\0\";\n+\tprint @{$o->{files}};\n+\n+\topen ($fh, \">\", \".git/watchman-output.out\");\n+\tprint $fh @{$o->{files}};\n+\tclose $fh;\n+}\n\\ No newline at end of file\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328151","messageId":"20170915192043.4516-13-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 12/12] fsmonitor: add a performance test","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:43Z","receivedAt":"2017-09-15T19:21:58Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a test utility (test-drop-caches) that flushes all changes to disk\nthen drops file system cache on Windows, Linux, and OSX.\n\nAdd a perf test (p7519-fsmonitor.sh) for fsmonitor.\n\nBy default, the performance test will utilize the Watchman file system\nmonitor if it is installed. If Watchman is not installed, it will use a\ndummy integration script that does not report any new or modified files.\nThe dummy script has very little overhead which provides optimistic results.\n\nThe performance test will also use the untracked cache feature if it is\navailable as fsmonitor uses it to speed up scanning for untracked files.\n\nThere are 3 environment variables that can be used to alter the default\nbehavior of the performance test:\n\nGIT_PERF_7519_UNTRACKED_CACHE: used to configure core.untrackedCache\nGIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\nGIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor\n\nThe big win for using fsmonitor is the elimination of the need to scan the\nworking directory looking for changed and untracked files. If the file\ninformation is all cached in RAM, the benefits are reduced.\n\nGIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests\n\nSigned-off-by: Ben Peart \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 1 +\n t/helper/.gitignore | 1 +\n t/helper/test-drop-caches.c | 161 ++++++++++++++++++++++++++++++++++++++\n t/perf/p7519-fsmonitor.sh | 184 ++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 347 insertions(+)\n create mode 100644 t/helper/test-drop-caches.c\n create mode 100755 t/perf/p7519-fsmonitor.sh\n\ndiff --git a/Makefile b/Makefile\nindex d970cd00e9..b2653ee64f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -638,6 +638,7 @@ TEST_PROGRAMS_NEED_X += test-ctype\n TEST_PROGRAMS_NEED_X += test-config\n TEST_PROGRAMS_NEED_X += test-date\n TEST_PROGRAMS_NEED_X += test-delta\n+TEST_PROGRAMS_NEED_X += test-drop-caches\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n TEST_PROGRAMS_NEED_X += test-dump-fsmonitor\n TEST_PROGRAMS_NEED_X += test-dump-split-index\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 721650256e..f9328eebdd 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -3,6 +3,7 @@\n /test-config\n /test-date\n /test-delta\n+/test-drop-caches\n /test-dump-cache-tree\n /test-dump-split-index\n /test-dump-untracked-cache\ndiff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\nnew file mode 100644\nindex 0000000000..717079865c\n--- /dev/null\n+++ b/t/helper/test-drop-caches.c\n@@ -0,0 +1,161 @@\n+#include \"git-compat-util.h\"\n+\n+#if defined(GIT_WINDOWS_NATIVE)\n+\n+int cmd_sync(void)\n+{\n+\tchar Buffer[MAX_PATH];\n+\tDWORD dwRet;\n+\tchar szVolumeAccessPath[] = \"\\\\\\\\.\\\\X:\";\n+\tHANDLE hVolWrite;\n+\tint success = 0;\n+\n+\tdwRet = GetCurrentDirectory(MAX_PATH, Buffer);\n+\tif ((0 == dwRet) || (dwRet > MAX_PATH))\n+\t\treturn error(\"Error getting current directory\");\n+\n+\tif ((Buffer[0] < 'A') || (Buffer[0] > 'Z'))\n+\t\treturn error(\"Invalid drive letter '%c'\", Buffer[0]);\n+\n+\tszVolumeAccessPath[4] = Buffer[0];\n+\thVolWrite = CreateFile(szVolumeAccessPath, GENERIC_READ | GENERIC_WRITE,\n+\t\tFILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);\n+\tif (INVALID_HANDLE_VALUE == hVolWrite)\n+\t\treturn error(\"Unable to open volume for writing, need admin access\");\n+\n+\tsuccess = FlushFileBuffers(hVolWrite);\n+\tif (!success)\n+\t\terror(\"Unable to flush volume\");\n+\n+\tCloseHandle(hVolWrite);\n+\n+\treturn !success;\n+}\n+\n+#define STATUS_SUCCESS\t\t\t(0x00000000L)\n+#define STATUS_PRIVILEGE_NOT_HELD\t(0xC0000061L)\n+\n+typedef enum _SYSTEM_INFORMATION_CLASS {\n+\tSystemMemoryListInformation = 80,\n+} SYSTEM_INFORMATION_CLASS;\n+\n+typedef enum _SYSTEM_MEMORY_LIST_COMMAND {\n+\tMemoryCaptureAccessedBits,\n+\tMemoryCaptureAndResetAccessedBits,\n+\tMemoryEmptyWorkingSets,\n+\tMemoryFlushModifiedList,\n+\tMemoryPurgeStandbyList,\n+\tMemoryPurgeLowPriorityStandbyList,\n+\tMemoryCommandMax\n+} SYSTEM_MEMORY_LIST_COMMAND;\n+\n+BOOL GetPrivilege(HANDLE TokenHandle, LPCSTR lpName, int flags)\n+{\n+\tBOOL bResult;\n+\tDWORD dwBufferLength;\n+\tLUID luid;\n+\tTOKEN_PRIVILEGES tpPreviousState;\n+\tTOKEN_PRIVILEGES tpNewState;\n+\n+\tdwBufferLength = 16;\n+\tbResult = LookupPrivilegeValueA(0, lpName, &luid);\n+\tif (bResult) {\n+\t\ttpNewState.PrivilegeCount = 1;\n+\t\ttpNewState.Privileges[0].Luid = luid;\n+\t\ttpNewState.Privileges[0].Attributes = 0;\n+\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpNewState,\n+\t\t\t(DWORD)((LPBYTE)&(tpNewState.Privileges[1]) - (LPBYTE)&tpNewState),\n+\t\t\t&tpPreviousState, &dwBufferLength);\n+\t\tif (bResult) {\n+\t\t\ttpPreviousState.PrivilegeCount = 1;\n+\t\t\ttpPreviousState.Privileges[0].Luid = luid;\n+\t\t\ttpPreviousState.Privileges[0].Attributes = flags != 0 ? 2 : 0;\n+\t\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpPreviousState,\n+\t\t\t\tdwBufferLength, 0, 0);\n+\t\t}\n+\t}\n+\treturn bResult;\n+}\n+\n+int cmd_dropcaches(void)\n+{\n+\tHANDLE hProcess = GetCurrentProcess();\n+\tHANDLE hToken;\n+\tint status;\n+\n+\tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n+\t\treturn error(\"Can't open current process token\");\n+\n+\tif (!GetPrivilege(hToken, \"SeProfileSingleProcessPrivilege\", 1))\n+\t\treturn error(\"Can't get SeProfileSingleProcessPrivilege\");\n+\n+\tCloseHandle(hToken);\n+\n+\tHMODULE ntdll = LoadLibrary(\"ntdll.dll\");\n+\tif (!ntdll)\n+\t\treturn error(\"Can't load ntdll.dll, wrong Windows version?\");\n+\n+\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =\n+\t\t(DWORD(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n+\tif (!NtSetSystemInformation)\n+\t\treturn error(\"Can't get function addresses, wrong Windows version?\");\n+\n+\tSYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;\n+\tstatus = NtSetSystemInformation(\n+\t\tSystemMemoryListInformation,\n+\t\t&command,\n+\t\tsizeof(SYSTEM_MEMORY_LIST_COMMAND)\n+\t);\n+\tif (status == STATUS_PRIVILEGE_NOT_HELD)\n+\t\terror(\"Insufficient privileges to purge the standby list, need admin access\");\n+\telse if (status != STATUS_SUCCESS)\n+\t\terror(\"Unable to execute the memory list command %d\", status);\n+\n+\tFreeLibrary(ntdll);\n+\n+\treturn status;\n+}\n+\n+#elif defined(__linux__)\n+\n+int cmd_sync(void)\n+{\n+\treturn system(\"sync\");\n+}\n+\n+int cmd_dropcaches(void)\n+{\n+\treturn system(\"echo 3 | sudo tee /proc/sys/vm/drop_caches\");\n+}\n+\n+#elif defined(__APPLE__)\n+\n+int cmd_sync(void)\n+{\n+\treturn system(\"sync\");\n+}\n+\n+int cmd_dropcaches(void)\n+{\n+\treturn system(\"sudo purge\");\n+}\n+\n+#else\n+\n+int cmd_sync(void)\n+{\n+\treturn 0;\n+}\n+\n+int cmd_dropcaches(void)\n+{\n+\treturn error(\"drop caches not implemented on this platform\");\n+}\n+\n+#endif\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tcmd_sync();\n+\treturn cmd_dropcaches();\n+}\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nnew file mode 100755\nindex 0000000000..1c5978d5c8\n--- /dev/null\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -0,0 +1,184 @@\n+#!/bin/sh\n+\n+test_description=\"Test core.fsmonitor\"\n+\n+. ./perf-lib.sh\n+\n+#\n+# Performance test for the fsmonitor feature which enables git to talk to a\n+# file system change monitor and avoid having to scan the working directory\n+# for new or modified files.\n+#\n+# By default, the performance test will utilize the Watchman file system\n+# monitor if it is installed. If Watchman is not installed, it will use a\n+# dummy integration script that does not report any new or modified files.\n+# The dummy script has very little overhead which provides optimistic results.\n+#\n+# The performance test will also use the untracked cache feature if it is\n+# available as fsmonitor uses it to speed up scanning for untracked files.\n+#\n+# There are 3 environment variables that can be used to alter the default\n+# behavior of the performance test:\n+#\n+# GIT_PERF_7519_UNTRACKED_CACHE: used to configure core.untrackedCache\n+# GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\n+# GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor\n+#\n+# The big win for using fsmonitor is the elimination of the need to scan the\n+# working directory looking for changed and untracked files. If the file\n+# information is all cached in RAM, the benefits are reduced.\n+#\n+# GIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests\n+#\n+\n+test_perf_large_repo\n+test_checkout_worktree\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_lazy_prereq WATCHMAN '\n+\t{ command -v watchman >/dev/null 2>&1; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+if test_have_prereq WATCHMAN\n+then\n+\t# Convert unix style paths to escaped Windows style paths for Watchman\n+\tcase \"$(uname -s)\" in\n+\tMSYS_NT*)\n+\t GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,/,g')\"\n+\t ;;\n+\t*)\n+\t GIT_WORK_TREE=\"$PWD\"\n+\t ;;\n+\tesac\n+fi\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"\n+then\n+\t# When using GIT_PERF_7519_DROP_CACHE, GIT_PERF_REPEAT_COUNT must be 1 to\n+\t# generate valid results. Otherwise the caching that happens for the nth\n+\t# run will negate the validity of the comparisons.\n+\tif test \"$GIT_PERF_REPEAT_COUNT\" -ne 1\n+\tthen\n+\t\techo \"warning: Setting GIT_PERF_REPEAT_COUNT=1\" >&2\n+\t\tGIT_PERF_REPEAT_COUNT=1\n+\tfi\n+fi\n+\n+test_expect_success \"setup for fsmonitor\" '\n+\t# set untrackedCache depending on the environment\n+\tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n+\tthen\n+\t\tgit config core.untrackedCache \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n+\telse\n+\t\tif test_have_prereq UNTRACKED_CACHE\n+\t\tthen\n+\t\t\tgit config core.untrackedCache true\n+\t\telse\n+\t\t\tgit config core.untrackedCache false\n+\t\tfi\n+\tfi &&\n+\n+\t# set core.splitindex depending on the environment\n+\tif test -n \"$GIT_PERF_7519_SPLIT_INDEX\"\n+\tthen\n+\t\tgit config core.splitIndex \"$GIT_PERF_7519_SPLIT_INDEX\"\n+\tfi &&\n+\n+\t# set INTEGRATION_SCRIPT depending on the environment\n+\tif test -n \"$GIT_PERF_7519_FSMONITOR\"\n+\tthen\n+\t\tINTEGRATION_SCRIPT=\"$GIT_PERF_7519_FSMONITOR\"\n+\telse\n+\t\t#\n+\t\t# Choose integration script based on existance of Watchman.\n+\t\t# If Watchman exists, watch the work tree and attempt a query.\n+\t\t# If everything succeeds, use Watchman integration script,\n+\t\t# else fall back to an empty integration script.\n+\t\t#\n+\t\tmkdir .git/hooks &&\n+\t\tif test_have_prereq WATCHMAN\n+\t\tthen\n+\t\t\tINTEGRATION_SCRIPT=\".git/hooks/fsmonitor-watchman\" &&\n+\t\t\tcp \"$TEST_DIRECTORY/../templates/hooks--fsmonitor-watchman.sample\" \"$INTEGRATION_SCRIPT\" &&\n+\t\t\twatchman watch \"$GIT_WORK_TREE\" &&\n+\t\t\twatchman watch-list | grep -q -F \"$GIT_WORK_TREE\"\n+\t\telse\n+\t\t\tINTEGRATION_SCRIPT=\".git/hooks/fsmonitor-empty\" &&\n+\t\t\twrite_script \"$INTEGRATION_SCRIPT\"<<-\\EOF\n+\t\t\tEOF\n+\t\tfi\n+\tfi &&\n+\n+\tgit config core.fsmonitor \"$INTEGRATION_SCRIPT\" &&\n+\tgit update-index --fsmonitor\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uno (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uno\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uall (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uall\n+'\n+\n+test_expect_success \"setup without fsmonitor\" '\n+\tunset INTEGRATION_SCRIPT &&\n+\tgit config --unset core.fsmonitor &&\n+\tgit update-index --no-fsmonitor\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uno (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uno\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uall (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uall\n+'\n+\n+if test_have_prereq WATCHMAN\n+then\n+\twatchman watch-del \"$GIT_WORK_TREE\" >/dev/null 2>&1 &&\n+\n+\t# Work around Watchman bug on Windows where it holds on to handles\n+\t# preventing the removal of the trash directory\n+\twatchman shutdown-server >/dev/null 2>&1\n+fi\n+\n+test_done\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328152","messageId":"20170915192043.4516-12-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 11/12] fsmonitor: add a sample integration script for Watchman","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:42Z","receivedAt":"2017-09-15T19:22:00Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This script integrates the new fsmonitor capabilities of git with the\ncross platform Watchman file watching service. To use the script:\n\nDownload and install Watchman from https://facebook.github.io/watchman/.\nRename the sample integration hook from fsmonitor-watchman.sample to\nfsmonitor-watchman. Configure git to use the extension:\n\ngit config core.fsmonitor .git/hooks/fsmonitor-watchman\n\nOptionally turn on the untracked cache for optimal performance.\n\nSigned-off-by: Ben Peart \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Christian Couder \n---\n templates/hooks--fsmonitor-watchman.sample | 119 +++++++++++++++++++++++++++++\n 1 file changed, 119 insertions(+)\n create mode 100755 templates/hooks--fsmonitor-watchman.sample\n\ndiff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample\nnew file mode 100755\nindex 0000000000..2779d7edf3\n--- /dev/null\n+++ b/templates/hooks--fsmonitor-watchman.sample\n@@ -0,0 +1,119 @@\n+#!/usr/bin/perl\n+\n+use strict;\n+use warnings;\n+use IPC::Open2;\n+\n+# An example hook script to integrate Watchman\n+# (https://facebook.github.io/watchman/) with git to speed up detecting\n+# new and modified files.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+# To enable this hook, rename this file to \"query-watchman\" and set\n+# 'git config core.fsmonitor .git/hooks/query-watchman'\n+#\n+my ($version, $time) = @ARGV;\n+\n+# Check the hook interface version\n+\n+if ($version == 1) {\n+\t# convert nanoseconds to seconds\n+\t$time = int $time / 1000000000;\n+} else {\n+\tdie \"Unsupported query-fsmonitor hook version '$version'.\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+}\n+\n+# Convert unix style paths to escaped Windows style paths when running\n+# in Windows command prompt\n+\n+my $system = `uname -s`;\n+$system =~ s/[\\r\\n]+//g;\n+my $git_work_tree;\n+\n+if ($system =~ m/^MSYS_NT/) {\n+\t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n+\t$git_work_tree =~ s/[\\r\\n]+//g;\n+\t$git_work_tree =~ s,\\\\,/,g;\n+} else {\n+\t$git_work_tree = $ENV{'PWD'};\n+}\n+\n+my $retry = 1;\n+\n+launch_watchman();\n+\n+sub launch_watchman {\n+\n+\t# Set input record separator\n+\tlocal $/ = 0666;\n+\n+\tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j')\n+\t or die \"open2() failed: $!\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+\n+\t# In the query expression below we're asking for names of files that\n+\t# changed since $time but were not transient (ie created after\n+\t# $time but no longer exist).\n+\t#\n+\t# To accomplish this, we're using the \"since\" generator to use the\n+\t# recency index to select candidate nodes and \"fields\" to limit the\n+\t# output to file names only. Then we're using the \"expression\" term to\n+\t# further constrain the results.\n+\t#\n+\t# The category of transient files that we want to ignore will have a\n+\t# creation clock (cclock) newer than $time_t value and will also not\n+\t# currently exist.\n+\n+\tprint CHLD_IN \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n+\t\\\"since\\\": $time, \\\n+\t\\\"fields\\\": [\\\"name\\\"], \\\n+\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n+\t}]\";\n+\n+\tmy $response = ;\n+\n+\tdie \"Watchman: command returned no output.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $response eq \"\";\n+\tdie \"Watchman: command returned invalid output: $response\\n\" .\n+\t \"Falling back to scanning...\\n\" unless $response =~ /^\\{/;\n+\n+\tmy $json_pkg;\n+\teval {\n+\t\trequire JSON::XS;\n+\t\t$json_pkg = \"JSON::XS\";\n+\t\t1;\n+\t} or do {\n+\t\trequire JSON::PP;\n+\t\t$json_pkg = \"JSON::PP\";\n+\t};\n+\n+\tmy $o = $json_pkg->new->utf8->decode($response);\n+\n+\tif ($retry > 0 and $o->{error} and $o->{error} =~ m/unable to resolve root .* directory (.*) is not watched/) {\n+\t\tprint STDERR \"Adding '$git_work_tree' to watchman's watch list.\\n\";\n+\t\t$retry--;\n+\t\tqx/watchman watch \"$git_work_tree\"/;\n+\t\tdie \"Failed to make watchman watch '$git_work_tree'.\\n\" .\n+\t\t \"Falling back to scanning...\\n\" if $? != 0;\n+\n+\t\t# Watchman will always return all files on the first query so\n+\t\t# return the fast \"everything is dirty\" flag to git and do the\n+\t\t# Watchman query just to get it over with now so we won't pay\n+\t\t# the cost in git to look up each individual file.\n+\t\tprint \"*\\0\";\n+\t\teval { launch_watchman() };\n+\t\texit 0;\n+\t}\n+\n+\tdie \"Watchman: $o->{error}.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $o->{error};\n+\n+\tbinmode STDOUT, \":utf8\";\n+\tlocal $, = \"\\0\";\n+\tprint @{$o->{files}};\n+}\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328153","messageId":"20170915192043.4516-9-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:39Z","receivedAt":"2017-09-15T19:22:04Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a test utility (test-dump-fsmonitor) that will dump the fsmonitor\nindex extension.\n\nSigned-off-by: Ben Peart \n---\n Makefile | 1 +\n t/helper/test-dump-fsmonitor.c | 21 +++++++++++++++++++++\n 2 files changed, 22 insertions(+)\n create mode 100644 t/helper/test-dump-fsmonitor.c\n\ndiff --git a/Makefile b/Makefile\nindex 9d6ec9c1e9..d970cd00e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -639,6 +639,7 @@ TEST_PROGRAMS_NEED_X += test-config\n TEST_PROGRAMS_NEED_X += test-date\n TEST_PROGRAMS_NEED_X += test-delta\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n+TEST_PROGRAMS_NEED_X += test-dump-fsmonitor\n TEST_PROGRAMS_NEED_X += test-dump-split-index\n TEST_PROGRAMS_NEED_X += test-dump-untracked-cache\n TEST_PROGRAMS_NEED_X += test-fake-ssh\ndiff --git a/t/helper/test-dump-fsmonitor.c b/t/helper/test-dump-fsmonitor.c\nnew file mode 100644\nindex 0000000000..482d749bb9\n--- /dev/null\n+++ b/t/helper/test-dump-fsmonitor.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+\n+int cmd_main(int ac, const char **av)\n+{\n+\tstruct index_state *istate = &the_index;\n+\tint i;\n+\n+\tsetup_git_directory();\n+\tif (do_read_index(istate, get_index_file(), 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update) {\n+\t\tprintf(\"no fsmonitor\\n\");\n+\t\treturn 0;\n+\t}\n+\tprintf(\"fsmonitor last update %\"PRIuMAX\"\\n\", istate->fsmonitor_last_update);\n+\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tprintf((istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) ? \"+\" : \"-\");\n+\n+\treturn 0;\n+}\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328154","messageId":"20170915192043.4516-10-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:40Z","receivedAt":"2017-09-15T19:22:09Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"The split index test t1700-split-index.sh has hard coded SHA values for\nthe index. Currently it supports index V4 and V3 but assumes there are\nno index extensions loaded.\n\nWhen manually forcing the fsmonitor extension to be turned on when\nrunning the test suite, the SHA values no longer match which causes the\ntest to fail.\n\nThe potential matrix of index extensions and index versions can is quite\nlarge so instead disable the extension before attempting to run the test.\n\nSigned-off-by: Ben Peart \n---\n t/t1700-split-index.sh | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex 22f69a410b..af9b847761 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -6,6 +6,7 @@ test_description='split index mode tests'\n \n # We need total control of index splitting here\n sane_unset GIT_TEST_SPLIT_INDEX\n+sane_unset GIT_FSMONITOR_TEST\n \n test_expect_success 'enable split index' '\n \tgit config splitIndex.maxPercentChange 100 &&\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328155","messageId":"20170915192043.4516-4-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 03/12] update-index: add a new --force-write-index option","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:34Z","receivedAt":"2017-09-15T19:22:15Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"At times, it makes sense to avoid the cost of writing out the index\nwhen the only changes can easily be recomputed on demand. This causes\nproblems when trying to write test cases to verify that state as they\ncan't guarantee the state has been persisted to disk.\n\nAdd a new option (--force-write-index) to update-index that will\nensure the index is written out even if the cache_changed flag is not\nset.\n\nSigned-off-by: Ben Peart \n---\n builtin/update-index.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex d562f2ec69..e1ca0759d5 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -915,6 +915,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tstruct refresh_params refresh_args = {0, &has_errors};\n \tint lock_error = 0;\n \tint split_index = -1;\n+\tint force_write = 0;\n \tstruct lock_file *lock_file;\n \tstruct parse_opt_ctx_t ctx;\n \tstrbuf_getline_fn getline_fn;\n@@ -1006,6 +1007,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"test if the filesystem supports untracked cache\"), UC_TEST),\n \t\tOPT_SET_INT(0, \"force-untracked-cache\", &untracked_cache,\n \t\t\t N_(\"enable untracked cache without testing the filesystem\"), UC_FORCE),\n+\t\tOPT_SET_INT(0, \"force-write-index\", &force_write,\n+\t\t\tN_(\"write out the index even if is not flagged as changed\"), 1),\n \t\tOPT_END()\n \t};\n \n@@ -1147,7 +1150,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n \t}\n \n-\tif (active_cache_changed) {\n+\tif (active_cache_changed || force_write) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n \t\t\t\texit(128);\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328156","messageId":"20170915192043.4516-3-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v6 02/12] preload-index: add override to enable testing preload-index","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-15T19:20:33Z","receivedAt":"2017-09-15T19:22:34Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Preload index doesn't run unless it has a minimum number of 1000 files.\nTo enable running tests with fewer files, add an environment variable\n(GIT_FORCE_PRELOAD_TEST) which will override that minimum and set it to 2.\n\nSigned-off-by: Ben Peart \n---\n preload-index.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/preload-index.c b/preload-index.c\nindex 70a4c80878..75564c497a 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -79,6 +79,8 @@ static void preload_index(struct index_state *index,\n \t\treturn;\n \n \tthreads = index->cache_nr / THREAD_COST;\n+\tif ((index->cache_nr > 1) && (threads < 2) && getenv(\"GIT_FORCE_PRELOAD_TEST\"))\n+\t\tthreads = 2;\n \tif (threads < 2)\n \t\treturn;\n \tif (threads > MAX_PARALLEL)\n-- \n2.14.1.548.ge54b1befee.dirty\n\n"},{"id":"328160","messageId":"297da9019d8c44faa89434d699623408@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"20170915192043.4516-6-benpeart@microsoft.com","subject":"RE: [PATCH v6 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-09-15T19:43:31Z","receivedAt":"2017-09-15T19:51:05Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"\n\n> -----Original Message-----\n> From: Ben Peart [mailto:benpeart@microsoft.com]\n> Sent: Friday, September 15, 2017 3:21 PM\n> To: benpeart@microsoft.com\n> Cc: David Turner ; avarab@gmail.com;\n> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> Subject: [PATCH v6 05/12] fsmonitor: add documentation for the fsmonitor\n> extension.\n> \n> This includes the core.fsmonitor setting, the query-fsmonitor hook, and the\n> fsmonitor index extension.\n> \n> Signed-off-by: Ben Peart \n> ---\n> Documentation/config.txt | 6 ++++++\n> Documentation/githooks.txt | 23 +++++++++++++++++++++++\n> Documentation/technical/index-format.txt | 19 +++++++++++++++++++\n> 3 files changed, 48 insertions(+)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt index\n> dc4e3f58a2..c196007a27 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -413,6 +413,12 @@ core.protectNTFS::\n> \t8.3 \"short\" names.\n> \tDefaults to `true` on Windows, and `false` elsewhere.\n> \n> +core.fsmonitor::\n> +\tIf set, the value of this variable is used as a command which\n> +\twill identify all files that may have changed since the\n> +\trequested date/time. This information is used to speed up git by\n> +\tavoiding unnecessary processing of files that have not changed.\n\nI'm confused here. You have a file called \"fsmonitor-watchman\", which seems to discuss the protocol for core.fsmonitor scripts in general, and you have this documentation, which does not link to that file. Can you clarify this? \n\n\n\n> +The hook should output to stdout the list of all files in the working\n> +directory that may have changed since the requested time. The logic\n> +should be inclusive so that it does not miss any potential changes.\n\n+\"It is OK to include files which have not actually changed. Newly-created and deleted files should also be included. When files are renamed, both the old and the new name should be included.\"\n\nAlso, please discuss case sensitivity issues (e.g. on OS X). \n\n> +The paths should be relative to the root of the working directory and\n> +be separated by a single NUL.\n\n\n\n> + - 32-bit version number: the current supported version is 1.\n> +\n> + - 64-bit time: the extension data reflects all changes through the given\n> +\ttime which is stored as the nanoseconds elapsed since midnight,\n> +\tJanuary 1, 1970.\n\nNit: Please specify signed or unsigned for these. (I expect to be getting out of \ncryosleep around 2262, and I want to know if my old git repos will keep working...)\n\n> + - 32-bit bitmap size: the size of the CE_FSMONITOR_VALID bitmap.\n> +\n> + - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n> + is not CE_FSMONITOR_VALID.\n\n"},{"id":"328163","messageId":"733a41051e084ba4b1f5c404ac8252e1@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"20170915192043.4516-7-benpeart@microsoft.com","subject":"RE: [PATCH v6 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-09-15T20:34:05Z","receivedAt":"2017-09-15T20:34:11Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"> -----Original Message-----\n> From: Ben Peart [mailto:benpeart@microsoft.com]\n> Sent: Friday, September 15, 2017 3:21 PM\n> To: benpeart@microsoft.com\n> Cc: David Turner ; avarab@gmail.com;\n> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> Subject: [PATCH v6 06/12] ls-files: Add support in ls-files to display the fsmonitor\n> valid bit\n> \n> Add a new command line option (-f) to ls-files to have it use lowercase letters\n> for 'fsmonitor valid' files\n\nDocument in man page, please.\n"},{"id":"328173","messageId":"850c2ad20acc4c14be87a767af851b19@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"20170915192043.4516-5-benpeart@microsoft.com","subject":"RE: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-09-15T21:35:24Z","receivedAt":"2017-09-15T21:35:30Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"> -----Original Message-----\n> From: Ben Peart [mailto:benpeart@microsoft.com]\n> Sent: Friday, September 15, 2017 3:21 PM\n> To: benpeart@microsoft.com\n> Cc: David Turner ; avarab@gmail.com;\n> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> Subject: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file system\n> monitor to speed up detecting new or changed files.\n \n> +int git_config_get_fsmonitor(void)\n> +{\n> +\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n> +\t\tcore_fsmonitor = getenv(\"GIT_FSMONITOR_TEST\");\n> +\n> +\tif (core_fsmonitor && !*core_fsmonitor)\n> +\t\tcore_fsmonitor = NULL;\n> +\n> +\tif (core_fsmonitor)\n> +\t\treturn 1;\n> +\n> +\treturn 0;\n> +}\n\nThis functions return values are backwards relative to the rest of the git_config_* functions.\n\n[snip]\n\n+>\t/*\n+>\t * With fsmonitor, we can trust the untracked cache's valid field.\n+>\t */\n\n[snip]\n\n> +int read_fsmonitor_extension(struct index_state *istate, const void *data,\n> +\tunsigned long sz)\n> +{\n\nIf git_config_get_fsmonitor returns 0, fsmonitor_dirty will leak.\n\n[snip]\n\n> +\t/* a fsmonitor process can return '*' to indicate all entries are invalid */\n\nThat's not documented in your documentation. Also, I'm not sure I like it: what \nif I have a file whose name starts with '*'? Yeah, that would be silly, but this indicates the need \nfor the protocol to have some sort of signaling mechanism that's out-of-band Maybe \nhave some key\\0value\\0 pairs and then \\0\\0 and then the list of files? Or, if you want to keep\nit really simple, allow an entry of '/' (which is an invalid filename) to mean 'all'.\n\n> +void add_fsmonitor(struct index_state *istate) {\n> +\tint i;\n> +\n> +\tif (!istate->fsmonitor_last_update) {\n[snip]\n> +\t\t/* reset the untracked cache */\n\nIs this really necessary? Shouldn't the untracked cache be in a correct state already? \n\n> +/*\n> + * Clear the given cache entries CE_FSMONITOR_VALID bit and invalidate\n\nNit: \"s/entries/entry's/\".\n \n\n"},{"id":"328177","messageId":"24c24fbeef8849f290df518dc2e4f9d7@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"20170915192043.4516-13-benpeart@microsoft.com","subject":"RE: [PATCH v6 12/12] fsmonitor: add a performance test","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-09-15T21:56:33Z","receivedAt":"2017-09-15T21:57:06Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"\n> -----Original Message-----\n> +\t\t# Choose integration script based on existance of Watchman.\n\nSpelling: existence\n\n\n"},{"id":"328178","messageId":"dc1506689d3d4639b56f0489e4219054@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"20170915192043.4516-11-benpeart@microsoft.com","subject":"RE: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-09-15T22:00:03Z","receivedAt":"2017-09-15T22:00:09Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"> -----Original Message-----\n> +dirty_repo () {\n> +\t: >untracked &&\n> +\t: >dir1/untracked &&\n> +\t: >dir2/untracked &&\n> +\techo 1 >modified &&\n> +\techo 2 >dir1/modified &&\n> +\techo 3 >dir2/modified &&\n> +\techo 4 >new &&\n> +\techo 5 >dir1/new &&\n> +\techo 6 >dir2/new\n\nIf I add an untracked file named dir3/untracked to dirty_repo\n (and write_integration_script), then \"status doesn't detect \nunreported modifications\", below, fails. Did I do something \nwrong, or does this turn up a bug?\n\n> +\ttest_expect_success \"setup preloadIndex to $preload_val\" '\n> +\t\tgit config core.preloadIndex $preload_val &&\n> +\t\tif [ $preload_val -eq true ]\n\n\"-eq\" is for numeric equality in POSIX shell. So this works if your \n/bin/sh is bash but not if it's e.g. dash. This happens twice more \nbelow. Use \"=\" instead.\n\n\n"},{"id":"328234","messageId":"ce17a042-91e4-bf84-d9e6-36a9e99f3644@web.de","threadId":"46154","inReplyTo":"20170915192043.4516-11-benpeart@microsoft.com","subject":"Re: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-09-16T15:27:20Z","receivedAt":"2017-09-16T15:27:54Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2017-09-15 21:20, Ben Peart wrote:\n> +if [ \"$1\" != 1 ]\n> +then\n> +\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n> +\texit 1\n> +fi\n\nThe echo -e not portable\n(It was detected by a tighter version of the lint script,\n which I have here, but not yet send to the list :-(\n\nThis will do:\necho \"Unsupported core.fsmonitor hook version.\" >&2\n"},{"id":"328244","messageId":"xmqqy3peszho.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170915192043.4516-11-benpeart@microsoft.com","subject":"Re: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-17T04:47:15Z","receivedAt":"2017-09-17T04:47:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> +write_integration_script() {\n> +\twrite_script .git/hooks/fsmonitor-test<<-\\EOF\n> +\tif [ \"$#\" -ne 2 ]; then\n> +\t\techo \"$0: exactly 2 arguments expected\"\n> +\t\texit 2\n> +\tfi\n> +\tif [ \"$1\" != 1 ]; then\n> +\t\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n> +\t\texit 1\n> +\tfi\n\nIn addition to \"echo -e\" thing pointed out earlier, these look\nsomewhat unusual in our shell scripts, relative to what\nDocumentation/CodingGuidelines tells us to do:\n\n - We prefer a space between the function name and the parentheses,\n and no space inside the parentheses. The opening \"{\" should also\n be on the same line.\n\n\t(incorrect)\n\tmy_function(){\n\t\t...\n\n\t(correct)\n\tmy_function () {\n\t\t...\n\n - We prefer \"test\" over \"[ ... ]\".\n\n - Do not write control structures on a single line with semicolon.\n \"then\" should be on the next line for if statements, and \"do\"\n should be on the next line for \"while\" and \"for\".\n\n\t(incorrect)\n\tif test -f hello; then\n\t\tdo this\n\tfi\n\n\t(correct)\n\tif test -f hello\n\tthen\n\t\tdo this\n\tfi\n\n> diff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\n> new file mode 100755\n> index 0000000000..aaee5d1fe3\n> --- /dev/null\n> +++ b/t/t7519/fsmonitor-watchman\n> @@ -0,0 +1,128 @@\n> +#!/usr/bin/perl\n> +\n> +use strict;\n> +use warnings;\n> +use IPC::Open2;\n> + ...\n> +\topen (my $fh, \">\", \".git/watchman-query.json\");\n> +\tprint $fh \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n> +\t\\\"since\\\": $time, \\\n> +\t\\\"fields\\\": [\\\"name\\\"], \\\n> +\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n> +\t}]\";\n> +\tclose $fh;\n> +\n> +\tprint CHLD_IN \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n> +\t\\\"since\\\": $time, \\\n> +\t\\\"fields\\\": [\\\"name\\\"], \\\n> +\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n> +\t}]\";\n\nThis look painful to read, write and maintain. IIRC, Perl supports\nthe < +}\n> \\ No newline at end of file\n\nOops.\n\nThanks.\n"},{"id":"328248","messageId":"20170917054318.28560-1-tboegi@web.de","threadId":"46154","inReplyTo":"ce17a042-91e4-bf84-d9e6-36a9e99f3644@web.de","subject":"[PATCH v1 1/1] test-lint: echo -e (or -E) is not portable","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-09-17T05:43:18Z","receivedAt":"2017-09-17T05:43:29Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Torsten Bögershausen \n\nSome implementations of `echo` support the '-e' option to enable\nbackslash interpretation of the following string.\nAs an addition, they support '-E' to turn it off.\n\nHowever, none of these are portable, POSIX doesn't even mention them,\nand many implementations don't support them.\n\nA check for '-n' is already done in check-non-portable-shell.pl,\nextend it to cover '-n', '-e' or '-E-'\n\nSigned-off-by: Torsten Bögershausen \n---\n t/check-non-portable-shell.pl | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/check-non-portable-shell.pl b/t/check-non-portable-shell.pl\nindex b170cbc045..03dc9d2852 100755\n--- a/t/check-non-portable-shell.pl\n+++ b/t/check-non-portable-shell.pl\n@@ -17,7 +17,7 @@ sub err {\n while (<>) {\n \tchomp;\n \t/\\bsed\\s+-i/ and err 'sed -i is not portable';\n-\t/\\becho\\s+-n/ and err 'echo -n is not portable (please use printf)';\n+\t/\\becho\\s+-[neE]/ and err 'echo with option is not portable (please use printf)';\n \t/^\\s*declare\\s+/ and err 'arrays/declare not portable';\n \t/^\\s*[^#]\\s*which\\s/ and err 'which is not portable (please use type)';\n \t/\\btest\\s+[^=]*==/ and err '\"test a == b\" is not portable (please use =)';\n-- \n2.14.1.145.gb3622a4ee9\n\n"},{"id":"328249","messageId":"xmqqtw01u50j.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170915192043.4516-9-benpeart@microsoft.com","subject":"Re: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-17T08:02:36Z","receivedAt":"2017-09-17T08:02:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> diff --git a/t/helper/test-dump-fsmonitor.c b/t/helper/test-dump-fsmonitor.c\n> new file mode 100644\n> index 0000000000..482d749bb9\n> --- /dev/null\n> +++ b/t/helper/test-dump-fsmonitor.c\n> @@ -0,0 +1,21 @@\n> +#include \"cache.h\"\n> +\n> +int cmd_main(int ac, const char **av)\n> +{\n> +\tstruct index_state *istate = &the_index;\n> +\tint i;\n> +\n> +\tsetup_git_directory();\n> +\tif (do_read_index(istate, get_index_file(), 0) < 0)\n> +\t\tdie(\"unable to read index file\");\n> +\tif (!istate->fsmonitor_last_update) {\n> +\t\tprintf(\"no fsmonitor\\n\");\n> +\t\treturn 0;\n> +\t}\n> +\tprintf(\"fsmonitor last update %\"PRIuMAX\"\\n\", istate->fsmonitor_last_update);\n\nAfter pushing this out and had Travis complain, I queued a squash on\ntop of this to cast the argument to (uintmax_t), like you did in an\nearlier step (I think it was [PATCH 04/12]).\n\n"},{"id":"328250","messageId":"xmqqpoapu4yp.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170915192043.4516-6-benpeart@microsoft.com","subject":"Re: [PATCH v6 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-17T08:03:42Z","receivedAt":"2017-09-17T08:03:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> +[[fsmonitor-watchman]]\n> +fsmonitor-watchman\n> +~~~~~~~~~~~~~~~\n\nI've queued a mini squash on top to make sure the ~~~~ line aligns\nwith the length of the string above it by adding three ~'s here.\n"},{"id":"328286","messageId":"3e0d003e-0643-0359-35fd-a5ecf9b751c3@gmail.com","threadId":"46154","inReplyTo":"850c2ad20acc4c14be87a767af851b19@exmbdft7.ad.twosigma.com","subject":"Re: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-18T13:07:23Z","receivedAt":"2017-09-18T13:07:32Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Thanks for taking the time to review/provide feedback!\n\nOn 9/15/2017 5:35 PM, David Turner wrote:\n>> -----Original Message-----\n>> From: Ben Peart [mailto:benpeart@microsoft.com]\n>> Sent: Friday, September 15, 2017 3:21 PM\n>> To: benpeart@microsoft.com\n>> Cc: David Turner ; avarab@gmail.com;\n>> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n>> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n>> Subject: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file system\n>> monitor to speed up detecting new or changed files.\n> \n>> +int git_config_get_fsmonitor(void)\n>> +{\n>> +\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n>> +\t\tcore_fsmonitor = getenv(\"GIT_FSMONITOR_TEST\");\n>> +\n>> +\tif (core_fsmonitor && !*core_fsmonitor)\n>> +\t\tcore_fsmonitor = NULL;\n>> +\n>> +\tif (core_fsmonitor)\n>> +\t\treturn 1;\n>> +\n>> +\treturn 0;\n>> +}\n> \n> This functions return values are backwards relative to the rest of the git_config_* functions.\n\nI'm confused. If core.fsmonitor is configured, it returns 1. If it is \nnot configured, it returns 0. I don't make use of the -1 /* default \nvalue */ option as I didn't see any use/value in this case. What is \nbackwards?\n\n> \n> [snip]\n> \n> +>\t/*\n> +>\t * With fsmonitor, we can trust the untracked cache's valid field.\n> +>\t */\n> \n\nDid you intend to make a comment here?\n\n> [snip]\n> \n>> +int read_fsmonitor_extension(struct index_state *istate, const void *data,\n>> +\tunsigned long sz)\n>> +{\n> \n> If git_config_get_fsmonitor returns 0, fsmonitor_dirty will leak.\n> \n\nGood catch! Thank you.\n\n> [snip]\n> \n>> +\t/* a fsmonitor process can return '*' to indicate all entries are invalid */\n> \n> That's not documented in your documentation. Also, I'm not sure I like it: what\n> if I have a file whose name starts with '*'? Yeah, that would be silly, but this indicates the need\n> for the protocol to have some sort of signaling mechanism that's out-of-band Maybe\n> have some key\\0value\\0 pairs and then \\0\\0 and then the list of files? Or, if you want to keep\n> it really simple, allow an entry of '/' (which is an invalid filename) to mean 'all'.\n> \n\nYea, this was an optimization I added late in the game to get around an \nissue in Watchman where it returns the name of every file the first time \nyou query it (rather than the set of files that have actually changed \nsince the requested time).\n\nI didn't realize the wild card '*' was a valid character for a filename. \n I'll switch to '/' as you suggest as I don't want to complicate things \nunnecessarily to handle this relatively rare optimization. I'll also \nget it documented properly. Thanks!\n\n>> +void add_fsmonitor(struct index_state *istate) {\n>> +\tint i;\n>> +\n>> +\tif (!istate->fsmonitor_last_update) {\n> [snip]\n>> +\t\t/* reset the untracked cache */\n> \n> Is this really necessary? Shouldn't the untracked cache be in a correct state already?\n> \n\nWhen fsmonitor is not turned on, I'm not explicitly invalidating \nuntracked cache directory entries as git makes changes to files. While I \ndoubt the sequence happens of 1) git making changes to files, *then* 2) \nturning on fsmonitor - I thought it better safe than sorry to assume \nthat pattern won't ever happen in the future. Especially since turning \non the extension is rare and the cost is low.\n\n>> +/*\n>> + * Clear the given cache entries CE_FSMONITOR_VALID bit and invalidate\n> \n> Nit: \"s/entries/entry's/\".\n> \n> \n"},{"id":"328287","messageId":"532a74ca-804d-93f3-9eef-ceffc9f49623@gmail.com","threadId":"46154","inReplyTo":"297da9019d8c44faa89434d699623408@exmbdft7.ad.twosigma.com","subject":"Re: [PATCH v6 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-18T13:27:35Z","receivedAt":"2017-09-18T13:27:44Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/15/2017 3:43 PM, David Turner wrote:\n> \n> \n>> -----Original Message-----\n>> From: Ben Peart [mailto:benpeart@microsoft.com]\n>> Sent: Friday, September 15, 2017 3:21 PM\n>> To: benpeart@microsoft.com\n>> Cc: David Turner ; avarab@gmail.com;\n>> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n>> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n>> Subject: [PATCH v6 05/12] fsmonitor: add documentation for the fsmonitor\n>> extension.\n>>\n>> This includes the core.fsmonitor setting, the query-fsmonitor hook, and the\n>> fsmonitor index extension.\n>>\n>> Signed-off-by: Ben Peart \n>> ---\n>> Documentation/config.txt | 6 ++++++\n>> Documentation/githooks.txt | 23 +++++++++++++++++++++++\n>> Documentation/technical/index-format.txt | 19 +++++++++++++++++++\n>> 3 files changed, 48 insertions(+)\n>>\n>> diff --git a/Documentation/config.txt b/Documentation/config.txt index\n>> dc4e3f58a2..c196007a27 100644\n>> --- a/Documentation/config.txt\n>> +++ b/Documentation/config.txt\n>> @@ -413,6 +413,12 @@ core.protectNTFS::\n>> \t8.3 \"short\" names.\n>> \tDefaults to `true` on Windows, and `false` elsewhere.\n>>\n>> +core.fsmonitor::\n>> +\tIf set, the value of this variable is used as a command which\n>> +\twill identify all files that may have changed since the\n>> +\trequested date/time. This information is used to speed up git by\n>> +\tavoiding unnecessary processing of files that have not changed.\n> \n> I'm confused here. You have a file called \"fsmonitor-watchman\", which seems to discuss the protocol for core.fsmonitor scripts in general, and you have this documentation, which does not link to that file. Can you clarify this?\n\nI'll add the missing link to the documentation in githooks.txt. The \ndocumentation should be enough for someone to develop another \nintegration script.\n\nThe fsmonitor-watchman script allows people to easily use this patch \nseries with the existing Watchman monitor but it can certainly also be \nused as a sample for how to integrate with another file system monitor.\n\n> \n> \n> \n>> +The hook should output to stdout the list of all files in the working\n>> +directory that may have changed since the requested time. The logic\n>> +should be inclusive so that it does not miss any potential changes.\n> \n> +\"It is OK to include files which have not actually changed. Newly-created and deleted files should also be included. When files are renamed, both the old and the new name should be included.\"\n> \n> Also, please discuss case sensitivity issues (e.g. on OS X).\n> \n>> +The paths should be relative to the root of the working directory and\n>> +be separated by a single NUL.\n> \n> \n> \n>> + - 32-bit version number: the current supported version is 1.\n>> +\n>> + - 64-bit time: the extension data reflects all changes through the given\n>> +\ttime which is stored as the nanoseconds elapsed since midnight,\n>> +\tJanuary 1, 1970.\n> \n> Nit: Please specify signed or unsigned for these. (I expect to be getting out of\n> cryosleep around 2262, and I want to know if my old git repos will keep working...)\n> \n\nWhile I'm not opposed to specifying unsigned, I did notice that the only \nplace signed/unsigned is specified today is in \"index entry.\" Everywhere \nelse doesn't specify so I left it off for consistency. I've not seen \nnegative version numbers nor negative time so am not entirely sure it is \nnecessary to clarify. :)\n\n>> + - 32-bit bitmap size: the size of the CE_FSMONITOR_VALID bitmap.\n>> +\n>> + - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n>> + is not CE_FSMONITOR_VALID.\n> \n"},{"id":"328288","messageId":"6edb012e-0553-7356-3f27-11479e4326b8@gmail.com","threadId":"46154","inReplyTo":"xmqqpoapu4yp.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-18T13:29:49Z","receivedAt":"2017-09-18T13:29:56Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/17/2017 4:03 AM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> +[[fsmonitor-watchman]]\n>> +fsmonitor-watchman\n>> +~~~~~~~~~~~~~~~\n> \n> I've queued a mini squash on top to make sure the ~~~~ line aligns\n> with the length of the string above it by adding three ~'s here.\n> \n\nThanks, I'll do the same assuming there will be another re-roll.\n"},{"id":"328289","messageId":"1f095732aec2469f86a7018aa9f1f8ec@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"3e0d003e-0643-0359-35fd-a5ecf9b751c3@gmail.com","subject":"RE: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-09-18T13:32:36Z","receivedAt":"2017-09-18T13:32:44Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"> -----Original Message-----\n> From: Ben Peart [mailto:peartben@gmail.com]\n> Sent: Monday, September 18, 2017 9:07 AM\n> To: David Turner ; 'Ben Peart'\n> \n> Cc: avarab@gmail.com; christian.couder@gmail.com; git@vger.kernel.org;\n> gitster@pobox.com; johannes.schindelin@gmx.de; pclouds@gmail.com;\n> peff@peff.net\n> Subject: Re: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file\n> system monitor to speed up detecting new or changed files.\n> \n> Thanks for taking the time to review/provide feedback!\n> \n> On 9/15/2017 5:35 PM, David Turner wrote:\n> >> -----Original Message-----\n> >> From: Ben Peart [mailto:benpeart@microsoft.com]\n> >> Sent: Friday, September 15, 2017 3:21 PM\n> >> To: benpeart@microsoft.com\n> >> Cc: David Turner ; avarab@gmail.com;\n> >> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n> >> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> >> Subject: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize\n> >> a file system monitor to speed up detecting new or changed files.\n> >\n> >> +int git_config_get_fsmonitor(void)\n> >> +{\n> >> +\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n> >> +\t\tcore_fsmonitor = getenv(\"GIT_FSMONITOR_TEST\");\n> >> +\n> >> +\tif (core_fsmonitor && !*core_fsmonitor)\n> >> +\t\tcore_fsmonitor = NULL;\n> >> +\n> >> +\tif (core_fsmonitor)\n> >> +\t\treturn 1;\n> >> +\n> >> +\treturn 0;\n> >> +}\n> >\n> > This functions return values are backwards relative to the rest of the\n> git_config_* functions.\n> \n> I'm confused. If core.fsmonitor is configured, it returns 1. If it is not\n> configured, it returns 0. I don't make use of the -1 /* default value */ option\n> as I didn't see any use/value in this case. What is backwards?\n\nThe other git_config_* functions return 1 for error and 0 for success.\n\n> > [snip]\n> >\n> > +>\t/*\n> > +>\t * With fsmonitor, we can trust the untracked cache's valid field.\n> > +>\t */\n> >\n> \n> Did you intend to make a comment here?\n\nSorry. I was going to make a comment that I didn't see how that could work \nsince we weren't touching the untracked cache here, but then I saw the bit \nfurther down. I'm still not sure it works (see comment on 10/12), but at\nleast it could in theory work.\n \n\n"},{"id":"328291","messageId":"5438765b-3119-ef15-8635-932b8cba08d3@gmail.com","threadId":"46154","inReplyTo":"xmqqtw01u50j.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-18T13:38:11Z","receivedAt":"2017-09-18T13:38:21Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/17/2017 4:02 AM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> diff --git a/t/helper/test-dump-fsmonitor.c b/t/helper/test-dump-fsmonitor.c\n>> new file mode 100644\n>> index 0000000000..482d749bb9\n>> --- /dev/null\n>> +++ b/t/helper/test-dump-fsmonitor.c\n>> @@ -0,0 +1,21 @@\n>> +#include \"cache.h\"\n>> +\n>> +int cmd_main(int ac, const char **av)\n>> +{\n>> +\tstruct index_state *istate = &the_index;\n>> +\tint i;\n>> +\n>> +\tsetup_git_directory();\n>> +\tif (do_read_index(istate, get_index_file(), 0) < 0)\n>> +\t\tdie(\"unable to read index file\");\n>> +\tif (!istate->fsmonitor_last_update) {\n>> +\t\tprintf(\"no fsmonitor\\n\");\n>> +\t\treturn 0;\n>> +\t}\n>> +\tprintf(\"fsmonitor last update %\"PRIuMAX\"\\n\", istate->fsmonitor_last_update);\n> \n> After pushing this out and had Travis complain, I queued a squash on\n> top of this to cast the argument to (uintmax_t), like you did in an\n> earlier step (I think it was [PATCH 04/12]).\n> \n\nThanks. I'll update this to cast it as (uint64_t) as that is what \nget/put_be64 use. As far as I can tell they both map to the same thing \n(unsigned long long) so there isn't functional difference.\n"},{"id":"328292","messageId":"36e55efe-b5d0-b7f3-50d2-600771235327@gmail.com","threadId":"46154","inReplyTo":"1f095732aec2469f86a7018aa9f1f8ec@exmbdft7.ad.twosigma.com","subject":"Re: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-18T13:49:57Z","receivedAt":"2017-09-18T13:50:07Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/18/2017 9:32 AM, David Turner wrote:\n>> -----Original Message-----\n>> From: Ben Peart [mailto:peartben@gmail.com]\n>> Sent: Monday, September 18, 2017 9:07 AM\n>> To: David Turner ; 'Ben Peart'\n>> \n>> Cc: avarab@gmail.com; christian.couder@gmail.com; git@vger.kernel.org;\n>> gitster@pobox.com; johannes.schindelin@gmx.de; pclouds@gmail.com;\n>> peff@peff.net\n>> Subject: Re: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file\n>> system monitor to speed up detecting new or changed files.\n>>\n>> Thanks for taking the time to review/provide feedback!\n>>\n>> On 9/15/2017 5:35 PM, David Turner wrote:\n>>>> -----Original Message-----\n>>>> From: Ben Peart [mailto:benpeart@microsoft.com]\n>>>> Sent: Friday, September 15, 2017 3:21 PM\n>>>> To: benpeart@microsoft.com\n>>>> Cc: David Turner ; avarab@gmail.com;\n>>>> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n>>>> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n>>>> Subject: [PATCH v6 04/12] fsmonitor: teach git to optionally utilize\n>>>> a file system monitor to speed up detecting new or changed files.\n>>>\n>>>> +int git_config_get_fsmonitor(void)\n>>>> +{\n>>>> +\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n>>>> +\t\tcore_fsmonitor = getenv(\"GIT_FSMONITOR_TEST\");\n>>>> +\n>>>> +\tif (core_fsmonitor && !*core_fsmonitor)\n>>>> +\t\tcore_fsmonitor = NULL;\n>>>> +\n>>>> +\tif (core_fsmonitor)\n>>>> +\t\treturn 1;\n>>>> +\n>>>> +\treturn 0;\n>>>> +}\n>>>\n>>> This functions return values are backwards relative to the rest of the\n>> git_config_* functions.\n>>\n>> I'm confused. If core.fsmonitor is configured, it returns 1. If it is not\n>> configured, it returns 0. I don't make use of the -1 /* default value */ option\n>> as I didn't see any use/value in this case. What is backwards?\n> \n> The other git_config_* functions return 1 for error and 0 for success.\n\nHmm, I followed the model (ie copy/paste :)) used by the tracked cache. \nIf you look at how it uses, the return value, it is 0 == false == remove \nthe extension, 1 == true == add the extension. I'm doing the same with \nfsmonitor.\n\nstatic void tweak_untracked_cache(struct index_state *istate)\n{\n\tswitch (git_config_get_untracked_cache()) {\n\tcase -1: /* keep: do nothing */\n\t\tbreak;\n\tcase 0: /* false */\n\t\tremove_untracked_cache(istate);\n\t\tbreak;\n\tcase 1: /* true */\n\t\tadd_untracked_cache(istate);\n\t\tbreak;\n\tdefault: /* unknown value: do nothing */\n\t\tbreak;\n\t}\n}\n\nvoid tweak_fsmonitor(struct index_state *istate)\n{\n\tswitch (git_config_get_fsmonitor()) {\n\tcase -1: /* keep: do nothing */\n\t\tbreak;\n\tcase 0: /* false */\n\t\tremove_fsmonitor(istate);\n\t\tbreak;\n\tcase 1: /* true */\n\t\tadd_fsmonitor(istate);\n\t\tbreak;\n\tdefault: /* unknown value: do nothing */\n\t\tbreak;\n\t}\n}\n\n> \n>>> [snip]\n>>>\n>>> +>\t/*\n>>> +>\t * With fsmonitor, we can trust the untracked cache's valid field.\n>>> +>\t */\n>>>\n>>\n>> Did you intend to make a comment here?\n> \n> Sorry. I was going to make a comment that I didn't see how that could work\n> since we weren't touching the untracked cache here, but then I saw the bit\n> further down. I'm still not sure it works (see comment on 10/12), but at\n> least it could in theory work.\n> \n> \n\nThe logic here assumes that when the index is written out, it is valid \nincluding the untracked cache and the CE_FSMONITOR_VALID bits. \nTherefore it should still all be valid as of the time the fsmonitor was \nqueried and the index saved.\n\nAnother way of thinking about this is that the fsmonitor extension is \nsimply adding another persisted bit on each index entry. It just gets \npersisted/restored differently than the other persisted bits.\n\nObviously, before we can use it assuming it reflects the *current* state \nof the working directory, we have to refresh the bits via the refresh logic.\n"},{"id":"328293","messageId":"61c14a6e-99e0-8b6e-d4ab-4a2547fba04a@gmail.com","threadId":"46154","inReplyTo":"ce17a042-91e4-bf84-d9e6-36a9e99f3644@web.de","subject":"Re: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-18T14:06:41Z","receivedAt":"2017-09-18T14:06:49Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/16/2017 11:27 AM, Torsten Bögershausen wrote:\n> On 2017-09-15 21:20, Ben Peart wrote:\n>> +if [ \"$1\" != 1 ]\n>> +then\n>> +\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n>> +\texit 1\n>> +fi\n> \n> The echo -e not portable\n> (It was detected by a tighter version of the lint script,\n> which I have here, but not yet send to the list :-(\n> \n> This will do:\n> echo \"Unsupported core.fsmonitor hook version.\" >&2\n> \n\nThanks, I'll fix these and the ones in the t/t7519 directory.\n"},{"id":"328295","messageId":"alpine.DEB.2.21.1.1709181615040.219280@virtualbox","threadId":"46154","inReplyTo":"20170915192043.4516-13-benpeart@microsoft.com","subject":"Re: [PATCH v6 12/12] fsmonitor: add a performance test","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-09-18T14:24:42Z","receivedAt":"2017-09-18T14:25:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ben,\n\nsorry for not catching this earlier:\n\nOn Fri, 15 Sep 2017, Ben Peart wrote:\n\n> [...]\n> +\n> +int cmd_dropcaches(void)\n> +{\n> +\tHANDLE hProcess = GetCurrentProcess();\n> +\tHANDLE hToken;\n> +\tint status;\n> +\n> +\tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n> +\t\treturn error(\"Can't open current process token\");\n> +\n> +\tif (!GetPrivilege(hToken, \"SeProfileSingleProcessPrivilege\", 1))\n> +\t\treturn error(\"Can't get SeProfileSingleProcessPrivilege\");\n> +\n> +\tCloseHandle(hToken);\n> +\n> +\tHMODULE ntdll = LoadLibrary(\"ntdll.dll\");\n\nGit's source code still tries to abide by C90, and for simplicity's sake,\nthis extends to the Windows-specific part. Therefore, the `ntdll` variable\nneeds to be declared at the beginning of the function (I do agree that it\nmakes for better code to reduce the scope of variables, but C90 simply did\nnot allow variables to be declared in the middle of functions).\n\nI wanted to send a patch address this in the obvious way, but then I\nencountered these lines:\n\n> +\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =\n> +\t\t(DWORD(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n> +\tif (!NtSetSystemInformation)\n> +\t\treturn error(\"Can't get function addresses, wrong Windows version?\");\n\nIt turns out that we have seen this plenty of times in Git for Windows'\nfork, so much so that we came up with a nice helper to make this all a bit\nmore robust and a bit more obvious, too: the DECLARE_PROC_ADDR and\nINIT_PROC_ADDR helpers in compat/win32/lazyload.h.\n\nMaybe this would be the perfect excuse to integrate this patch into\nupstream Git? This would be the patch (you can also cherry-pick it from\n25c4dc3a73352e72e995594cf1b4afa46e93d040 in https://github.com/dscho/git):\n\n-- snip --\nFrom 25c4dc3a73352e72e995594cf1b4afa46e93d040 Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin \nDate: Tue, 10 Jan 2017 23:14:20 +0100\nSubject: [PATCH] Win32: simplify loading of DLL functions\n\nDynamic loading of DLL functions is duplicated in several places in Git\nfor Windows' source code.\n\nThis patch adds a pair of macros to simplify the process: the\nDECLARE_PROC_ADDR(, , ,\n......) macro to be used at the beginning of a\ncode block, and the INIT_PROC_ADDR() macro to call before\nusing the declared function. The return value of the INIT_PROC_ADDR()\ncall has to be checked; If it is NULL, the function was not found in the\nspecified DLL.\n\nExample:\n\n DECLARE_PROC_ADDR(kernel32.dll, BOOL, CreateHardLinkW,\n LPCWSTR, LPCWSTR, LPSECURITY_ATTRIBUTES);\n\n if (!INIT_PROC_ADDR(CreateHardLinkW))\n return error(\"Could not find CreateHardLinkW() function\";\n\n\tif (!CreateHardLinkW(source, target, NULL))\n\t\treturn error(\"could not create hardlink from %S to %S\",\n\t\t\t source, target);\n\treturn 0;\n\nSigned-off-by: Karsten Blees \nSigned-off-by: Johannes Schindelin \n---\n compat/win32/lazyload.h | 44 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 44 insertions(+)\n create mode 100644 compat/win32/lazyload.h\n\ndiff --git a/compat/win32/lazyload.h b/compat/win32/lazyload.h\nnew file mode 100644\nindex 00000000000..91c10dad2fb\n--- /dev/null\n+++ b/compat/win32/lazyload.h\n@@ -0,0 +1,44 @@\n+#ifndef LAZYLOAD_H\n+#define LAZYLOAD_H\n+\n+/* simplify loading of DLL functions */\n+\n+struct proc_addr {\n+\tconst char *const dll;\n+\tconst char *const function;\n+\tFARPROC pfunction;\n+\tunsigned initialized : 1;\n+};\n+\n+/* Declares a function to be loaded dynamically from a DLL. */\n+#define DECLARE_PROC_ADDR(dll, rettype, function, ...) \\\n+\tstatic struct proc_addr proc_addr_##function = \\\n+\t{ #dll, #function, NULL, 0 }; \\\n+\tstatic rettype (WINAPI *function)(__VA_ARGS__)\n+\n+/*\n+ * Loads a function from a DLL (once-only).\n+ * Returns non-NULL function pointer on success.\n+ * Returns NULL + errno == ENOSYS on failure.\n+ */\n+#define INIT_PROC_ADDR(function) \\\n+\t(function = get_proc_addr(&proc_addr_##function))\n+\n+static inline void *get_proc_addr(struct proc_addr *proc)\n+{\n+\t/* only do this once */\n+\tif (!proc->initialized) {\n+\t\tHANDLE hnd;\n+\t\tproc->initialized = 1;\n+\t\thnd = LoadLibraryExA(proc->dll, NULL,\n+\t\t\t\t LOAD_LIBRARY_SEARCH_SYSTEM32);\n+\t\tif (hnd)\n+\t\t\tproc->pfunction = GetProcAddress(hnd, proc->function);\n+\t}\n+\t/* set ENOSYS if DLL or function was not found */\n+\tif (!proc->pfunction)\n+\t\terrno = ENOSYS;\n+\treturn proc->pfunction;\n+}\n+\n+#endif\n-- snap --\n\nWith this patch, this fixup to your patch would make things compile (you\ncan also cherry-pick d05996fb61027512b8ab31a36c4a7a677dea11bb from my\nfork):\n\n-- snipsnap --\nFrom d05996fb61027512b8ab31a36c4a7a677dea11bb Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin \nDate: Mon, 18 Sep 2017 14:56:40 +0200\nSubject: [PATCH] fixup! fsmonitor: add a performance test\n\n---\n t/helper/test-drop-caches.c | 16 ++++++----------\n 1 file changed, 6 insertions(+), 10 deletions(-)\n\ndiff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\nindex 717079865cb..b27358528f7 100644\n--- a/t/helper/test-drop-caches.c\n+++ b/t/helper/test-drop-caches.c\n@@ -1,6 +1,7 @@\n #include \"git-compat-util.h\"\n \n #if defined(GIT_WINDOWS_NATIVE)\n+#include \"compat/win32/lazyload.h\"\n \n int cmd_sync(void)\n {\n@@ -82,6 +83,9 @@ int cmd_dropcaches(void)\n \tHANDLE hProcess = GetCurrentProcess();\n \tHANDLE hToken;\n \tint status;\n+\tSYSTEM_MEMORY_LIST_COMMAND command;\n+\tDECLARE_PROC_ADDR(ntll,\n+\t\t\t DWORD, NtSetSystemInformation, INT, PVOID, ULONG);\n \n \tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n \t\treturn error(\"Can't open current process token\");\n@@ -91,16 +95,10 @@ int cmd_dropcaches(void)\n \n \tCloseHandle(hToken);\n \n-\tHMODULE ntdll = LoadLibrary(\"ntdll.dll\");\n-\tif (!ntdll)\n-\t\treturn error(\"Can't load ntdll.dll, wrong Windows\n \t\tversion?\");\n-\n-\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =\n-\t\t(DWORD(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll,\n \t\t\"NtSetSystemInformation\");\n-\tif (!NtSetSystemInformation)\n+\tif (!INIT_PROC_ADDR(NtSetSystemInformation))\n \t\treturn error(\"Can't get function addresses, wrong Windows version?\");\n \n-\tSYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;\n+\tcommand = MemoryPurgeStandbyList;\n \tstatus = NtSetSystemInformation(\n \t\tSystemMemoryListInformation,\n \t\t&command,\n@@ -111,8 +109,6 @@ int cmd_dropcaches(void)\n \telse if (status != STATUS_SUCCESS)\n \t\terror(\"Unable to execute the memory list command %d\", status);\n \n-\tFreeLibrary(ntdll);\n-\n \treturn status;\n }\n \n-- \n2.14.1.windows.1.510.g0cb6d35d23\n"},{"id":"328297","messageId":"2f3cdc85-f051-c0ae-b9db-fd13cac78aed@gmail.com","threadId":"46154","inReplyTo":"xmqqy3peszho.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-18T15:25:17Z","receivedAt":"2017-09-18T15:25:26Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/17/2017 12:47 AM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> +write_integration_script() {\n>> +\twrite_script .git/hooks/fsmonitor-test<<-\\EOF\n>> +\tif [ \"$#\" -ne 2 ]; then\n>> +\t\techo \"$0: exactly 2 arguments expected\"\n>> +\t\texit 2\n>> +\tfi\n>> +\tif [ \"$1\" != 1 ]; then\n>> +\t\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n>> +\t\texit 1\n>> +\tfi\n> \n> In addition to \"echo -e\" thing pointed out earlier, these look\n> somewhat unusual in our shell scripts, relative to what\n> Documentation/CodingGuidelines tells us to do:\n\nI'm happy to make these changes. I understand the difficulty of \ncreating a consistent coding style especially after the fact.\n\n\n\nCopy/paste is usually a developers best friend as it allows you to avoid \na lot of errors by reusing existing tested code. One of the times it \nbackfires is when that code doesn't match the current desired coding style.\n\nI only point these out to help lend some additional impetus to the \neffort to formalize the coding style and to provide tooling to handle \nwhat should mostly be a mechanical process. IMO, the goal should be to \nsave the maintainer and contributors the cost of having to write up and \nrespond to formatting feedback. :)\n\nSome stats on these same coding style errors in the current bash scripts:\n\n298 instances of \"[a-z]\\(\\).*\\{\" ie \"function_name() {\" (no space)\n140 instances of \"if \\[ .* \\]\" (not using the preferred \"test\")\n293 instances of \"if .*; then\"\n\nWouldn't it be great not to have to write up style feedback for when \nthese all get copy/pasted into new scripts? :)\n\n\n\n> \n> - We prefer a space between the function name and the parentheses,\n> and no space inside the parentheses. The opening \"{\" should also\n> be on the same line.\n> \n> \t(incorrect)\n> \tmy_function(){\n> \t\t...\n> \n> \t(correct)\n> \tmy_function () {\n> \t\t...\n> \n> - We prefer \"test\" over \"[ ... ]\".\n> \n> - Do not write control structures on a single line with semicolon.\n> \"then\" should be on the next line for if statements, and \"do\"\n> should be on the next line for \"while\" and \"for\".\n> \n> \t(incorrect)\n> \tif test -f hello; then\n> \t\tdo this\n> \tfi\n> \n> \t(correct)\n> \tif test -f hello\n> \tthen\n> \t\tdo this\n> \tfi\n> \n>> diff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\n>> new file mode 100755\n>> index 0000000000..aaee5d1fe3\n>> --- /dev/null\n>> +++ b/t/t7519/fsmonitor-watchman\n>> @@ -0,0 +1,128 @@\n>> +#!/usr/bin/perl\n>> +\n>> +use strict;\n>> +use warnings;\n>> +use IPC::Open2;\n>> + ...\n>> +\topen (my $fh, \">\", \".git/watchman-query.json\");\n>> +\tprint $fh \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n>> +\t\\\"since\\\": $time, \\\n>> +\t\\\"fields\\\": [\\\"name\\\"], \\\n>> +\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n>> +\t}]\";\n>> +\tclose $fh;\n>> +\n>> +\tprint CHLD_IN \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n>> +\t\\\"since\\\": $time, \\\n>> +\t\\\"fields\\\": [\\\"name\\\"], \\\n>> +\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n>> +\t}]\";\n> \n> This look painful to read, write and maintain. IIRC, Perl supports\n> the < these \"print\" we see above easier?\n> \n\nI agree! I'm definitely *not* a perl developer so was unaware of this \nconstruct. A few minutes with stack overflow and now I can clean this up.\n\n>> +}\n>> \\ No newline at end of file\n> \n> Oops.\n> \n> Thanks.\n> \n"},{"id":"328298","messageId":"6f55efc4-96b2-7c53-7d54-ff5801697550@web.de","threadId":"46154","inReplyTo":"5438765b-3119-ef15-8635-932b8cba08d3@gmail.com","subject":"Re: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-09-18T15:43:13Z","receivedAt":"2017-09-18T15:43:41Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2017-09-18 15:38, Ben Peart wrote:\n> \n> \n> On 9/17/2017 4:02 AM, Junio C Hamano wrote:\n>> Ben Peart writes:\n>>\n>>> diff --git a/t/helper/test-dump-fsmonitor.c b/t/helper/test-dump-fsmonitor.c\n>>> new file mode 100644\n>>> index 0000000000..482d749bb9\n>>> --- /dev/null\n>>> +++ b/t/helper/test-dump-fsmonitor.c\n>>> @@ -0,0 +1,21 @@\n>>> +#include \"cache.h\"\n>>> +\n>>> +int cmd_main(int ac, const char **av)\n>>> +{\n>>> +    struct index_state *istate = &the_index;\n>>> +    int i;\n>>> +\n>>> +    setup_git_directory();\n>>> +    if (do_read_index(istate, get_index_file(), 0) < 0)\n>>> +        die(\"unable to read index file\");\n>>> +    if (!istate->fsmonitor_last_update) {\n>>> +        printf(\"no fsmonitor\\n\");\n>>> +        return 0;\n>>> +    }\n>>> +    printf(\"fsmonitor last update %\"PRIuMAX\"\\n\",\n>>> istate->fsmonitor_last_update);\n>>\n>> After pushing this out and had Travis complain, I queued a squash on\n>> top of this to cast the argument to (uintmax_t), like you did in an\n>> earlier step (I think it was [PATCH 04/12]).\n>>\n> \n> Thanks. I'll update this to cast it as (uint64_t) as that is what get/put_be64\n> use.  As far as I can tell they both map to the same thing (unsigned long long)\n> so there isn't functional difference.\n(Just to double-check): This is the way to print \"PRIuMAX\" correctly\n (on all platforms):\n\nprintf(\"fsmonitor last update %\"PRIuMAX\"\\n\",\n (uintmax_t)istate->fsmonitor_last_update);\n\n\n"},{"id":"328302","messageId":"MWHPR21MB0478E4671130AD9E19B77460F4630@MWHPR21MB0478.namprd21.prod.outlook.com","threadId":"46154","inReplyTo":"6f55efc4-96b2-7c53-7d54-ff5801697550@web.de","subject":"RE: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2017-09-18T16:28:51Z","receivedAt":"2017-09-18T16:29:00Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"> -----Original Message-----\n> From: Torsten Bögershausen [mailto:tboegi@web.de]\n> Sent: Monday, September 18, 2017 11:43 AM\n> To: Ben Peart ; Junio C Hamano\n> ; Ben Peart \n> Cc: David.Turner@twosigma.com; avarab@gmail.com;\n> christian.couder@gmail.com; git@vger.kernel.org;\n> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> Subject: Re: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index\n> extension\n> \n> On 2017-09-18 15:38, Ben Peart wrote:\n> >\n> >\n> > On 9/17/2017 4:02 AM, Junio C Hamano wrote:\n> >> Ben Peart writes:\n> >>\n> >>> diff --git a/t/helper/test-dump-fsmonitor.c\n> >>> b/t/helper/test-dump-fsmonitor.c new file mode 100644 index\n> >>> 0000000000..482d749bb9\n> >>> --- /dev/null\n> >>> +++ b/t/helper/test-dump-fsmonitor.c\n> >>> @@ -0,0 +1,21 @@\n> >>> +#include \"cache.h\"\n> >>> +\n> >>> +int cmd_main(int ac, const char **av) {\n> >>> +    struct index_state *istate = &the_index;\n> >>> +    int i;\n> >>> +\n> >>> +    setup_git_directory();\n> >>> +    if (do_read_index(istate, get_index_file(), 0) < 0)\n> >>> +        die(\"unable to read index file\");\n> >>> +    if (!istate->fsmonitor_last_update) {\n> >>> +        printf(\"no fsmonitor\\n\");\n> >>> +        return 0;\n> >>> +    }\n> >>> +    printf(\"fsmonitor last update %\"PRIuMAX\"\\n\",\n> >>> istate->fsmonitor_last_update);\n> >>\n> >> After pushing this out and had Travis complain, I queued a squash on\n> >> top of this to cast the argument to (uintmax_t), like you did in an\n> >> earlier step (I think it was [PATCH 04/12]).\n> >>\n> >\n> > Thanks. I'll update this to cast it as (uint64_t) as that is what\n> > get/put_be64 use.  As far as I can tell they both map to the same\n> > thing (unsigned long long) so there isn't functional difference.\n> (Just to double-check): This is the way to print \"PRIuMAX\" correctly (on all\n> platforms):\n> \n> printf(\"fsmonitor last update %\"PRIuMAX\"\\n\", (uintmax_t)istate-\n> >fsmonitor_last_update);\n> \n\nShould I just make the variable type itself uintmax_t and then just skip\nthe cast altogether? I went with uint64_t because that is what\ngetnanotime returned.\n\n"},{"id":"328315","messageId":"MWHPR21MB04789C11E37F58FF9BC11C51F4630@MWHPR21MB0478.namprd21.prod.outlook.com","threadId":"46154","inReplyTo":"alpine.DEB.2.21.1.1709181615040.219280@virtualbox","subject":"RE: [PATCH v6 12/12] fsmonitor: add a performance test","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2017-09-18T18:19:54Z","receivedAt":"2017-09-18T18:20:02Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n> -----Original Message-----\n> From: Johannes Schindelin [mailto:Johannes.Schindelin@gmx.de]\n> Sent: Monday, September 18, 2017 10:25 AM\n> To: Ben Peart \n> Cc: David.Turner@twosigma.com; avarab@gmail.com;\n> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n> pclouds@gmail.com; peff@peff.net\n> Subject: Re: [PATCH v6 12/12] fsmonitor: add a performance test\n> \n> Hi Ben,\n> \n> sorry for not catching this earlier:\n> \n> On Fri, 15 Sep 2017, Ben Peart wrote:\n> \n> > [...]\n> > +\n> > +int cmd_dropcaches(void)\n> > +{\n> > +\tHANDLE hProcess = GetCurrentProcess();\n> > +\tHANDLE hToken;\n> > +\tint status;\n> > +\n> > +\tif (!OpenProcessToken(hProcess, TOKEN_QUERY |\n> TOKEN_ADJUST_PRIVILEGES, &hToken))\n> > +\t\treturn error(\"Can't open current process token\");\n> > +\n> > +\tif (!GetPrivilege(hToken, \"SeProfileSingleProcessPrivilege\", 1))\n> > +\t\treturn error(\"Can't get SeProfileSingleProcessPrivilege\");\n> > +\n> > +\tCloseHandle(hToken);\n> > +\n> > +\tHMODULE ntdll = LoadLibrary(\"ntdll.dll\");\n> \n\nThanks Johannes, I'll fix that.\n\n> Git's source code still tries to abide by C90, and for simplicity's sake, this\n> extends to the Windows-specific part. Therefore, the `ntdll` variable needs to\n> be declared at the beginning of the function (I do agree that it makes for\n> better code to reduce the scope of variables, but C90 simply did not allow\n> variables to be declared in the middle of functions).\n> \n> I wanted to send a patch address this in the obvious way, but then I\n> encountered these lines:\n> \n> > +\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =\n> > +\t\t(DWORD(WINAPI *)(INT, PVOID,\n> ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n> > +\tif (!NtSetSystemInformation)\n> > +\t\treturn error(\"Can't get function addresses, wrong Windows\n> > +version?\");\n> \n> It turns out that we have seen this plenty of times in Git for Windows'\n> fork, so much so that we came up with a nice helper to make this all a bit\n> more robust and a bit more obvious, too: the DECLARE_PROC_ADDR and\n> INIT_PROC_ADDR helpers in compat/win32/lazyload.h.\n> \n> Maybe this would be the perfect excuse to integrate this patch into upstream\n> Git? \n\nThis patch is pretty hefty already. How about you push this capability\nupstream and I take advantage of it in a later patch. :)\n\nThis would be the patch (you can also cherry-pick it from\n> 25c4dc3a73352e72e995594cf1b4afa46e93d040 in\n> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.\n> com%2Fdscho%2Fgit&data=02%7C01%7CBen.Peart%40microsoft.com%7C96\n> 4027bdc1f34a033c1f08d4fea1056e%7C72f988bf86f141af91ab2d7cd011db47\n> %7C1%7C0%7C636413414914282865&sdata=jyvu6G7myRY9UA1XxWx2tDZ%\n> 2BWsIWqLTRMT8WfzEGe5g%3D&reserved=0):\n> \n> -- snip --\n> From 25c4dc3a73352e72e995594cf1b4afa46e93d040 Mon Sep 17 00:00:00\n> 2001\n> From: Johannes Schindelin \n> Date: Tue, 10 Jan 2017 23:14:20 +0100\n> Subject: [PATCH] Win32: simplify loading of DLL functions\n> \n> Dynamic loading of DLL functions is duplicated in several places in Git for\n> Windows' source code.\n> \n> This patch adds a pair of macros to simplify the process: the\n> DECLARE_PROC_ADDR(, , ,\n> ......) macro to be used at the beginning of a\n> code block, and the INIT_PROC_ADDR() macro to call before\n> using the declared function. The return value of the INIT_PROC_ADDR() call\n> has to be checked; If it is NULL, the function was not found in the specified\n> DLL.\n> \n> Example:\n> \n> DECLARE_PROC_ADDR(kernel32.dll, BOOL, CreateHardLinkW,\n> LPCWSTR, LPCWSTR, LPSECURITY_ATTRIBUTES);\n> \n> if (!INIT_PROC_ADDR(CreateHardLinkW))\n> return error(\"Could not find CreateHardLinkW() function\";\n> \n> \tif (!CreateHardLinkW(source, target, NULL))\n> \t\treturn error(\"could not create hardlink from %S to %S\",\n> \t\t\t source, target);\n> \treturn 0;\n> \n> Signed-off-by: Karsten Blees \n> Signed-off-by: Johannes Schindelin \n> ---\n> compat/win32/lazyload.h | 44\n> ++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 44 insertions(+)\n> create mode 100644 compat/win32/lazyload.h\n> \n> diff --git a/compat/win32/lazyload.h b/compat/win32/lazyload.h new file\n> mode 100644 index 00000000000..91c10dad2fb\n> --- /dev/null\n> +++ b/compat/win32/lazyload.h\n> @@ -0,0 +1,44 @@\n> +#ifndef LAZYLOAD_H\n> +#define LAZYLOAD_H\n> +\n> +/* simplify loading of DLL functions */\n> +\n> +struct proc_addr {\n> +\tconst char *const dll;\n> +\tconst char *const function;\n> +\tFARPROC pfunction;\n> +\tunsigned initialized : 1;\n> +};\n> +\n> +/* Declares a function to be loaded dynamically from a DLL. */ #define\n> +DECLARE_PROC_ADDR(dll, rettype, function, ...) \\\n> +\tstatic struct proc_addr proc_addr_##function = \\\n> +\t{ #dll, #function, NULL, 0 }; \\\n> +\tstatic rettype (WINAPI *function)(__VA_ARGS__)\n> +\n> +/*\n> + * Loads a function from a DLL (once-only).\n> + * Returns non-NULL function pointer on success.\n> + * Returns NULL + errno == ENOSYS on failure.\n> + */\n> +#define INIT_PROC_ADDR(function) \\\n> +\t(function = get_proc_addr(&proc_addr_##function))\n> +\n> +static inline void *get_proc_addr(struct proc_addr *proc) {\n> +\t/* only do this once */\n> +\tif (!proc->initialized) {\n> +\t\tHANDLE hnd;\n> +\t\tproc->initialized = 1;\n> +\t\thnd = LoadLibraryExA(proc->dll, NULL,\n> +\t\t\t\t LOAD_LIBRARY_SEARCH_SYSTEM32);\n> +\t\tif (hnd)\n> +\t\t\tproc->pfunction = GetProcAddress(hnd, proc-\n> >function);\n> +\t}\n> +\t/* set ENOSYS if DLL or function was not found */\n> +\tif (!proc->pfunction)\n> +\t\terrno = ENOSYS;\n> +\treturn proc->pfunction;\n> +}\n> +\n> +#endif\n> -- snap --\n> \n> With this patch, this fixup to your patch would make things compile (you can\n> also cherry-pick d05996fb61027512b8ab31a36c4a7a677dea11bb from my\n> fork):\n> \n> -- snipsnap --\n> From d05996fb61027512b8ab31a36c4a7a677dea11bb Mon Sep 17 00:00:00\n> 2001\n> From: Johannes Schindelin \n> Date: Mon, 18 Sep 2017 14:56:40 +0200\n> Subject: [PATCH] fixup! fsmonitor: add a performance test\n> \n> ---\n> t/helper/test-drop-caches.c | 16 ++++++----------\n> 1 file changed, 6 insertions(+), 10 deletions(-)\n> \n> diff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c index\n> 717079865cb..b27358528f7 100644\n> --- a/t/helper/test-drop-caches.c\n> +++ b/t/helper/test-drop-caches.c\n> @@ -1,6 +1,7 @@\n> #include \"git-compat-util.h\"\n> \n> #if defined(GIT_WINDOWS_NATIVE)\n> +#include \"compat/win32/lazyload.h\"\n> \n> int cmd_sync(void)\n> {\n> @@ -82,6 +83,9 @@ int cmd_dropcaches(void)\n> \tHANDLE hProcess = GetCurrentProcess();\n> \tHANDLE hToken;\n> \tint status;\n> +\tSYSTEM_MEMORY_LIST_COMMAND command;\n> +\tDECLARE_PROC_ADDR(ntll,\n> +\t\t\t DWORD, NtSetSystemInformation, INT, PVOID,\n> ULONG);\n> \n> \tif (!OpenProcessToken(hProcess, TOKEN_QUERY |\n> TOKEN_ADJUST_PRIVILEGES, &hToken))\n> \t\treturn error(\"Can't open current process token\"); @@ -91,16\n> +95,10 @@ int cmd_dropcaches(void)\n> \n> \tCloseHandle(hToken);\n> \n> -\tHMODULE ntdll = LoadLibrary(\"ntdll.dll\");\n> -\tif (!ntdll)\n> -\t\treturn error(\"Can't load ntdll.dll, wrong Windows\n> \t\tversion?\");\n> -\n> -\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =\n> -\t\t(DWORD(WINAPI *)(INT, PVOID,\n> ULONG))GetProcAddress(ntdll,\n> \t\t\"NtSetSystemInformation\");\n> -\tif (!NtSetSystemInformation)\n> +\tif (!INIT_PROC_ADDR(NtSetSystemInformation))\n> \t\treturn error(\"Can't get function addresses, wrong Windows\n> version?\");\n> \n> -\tSYSTEM_MEMORY_LIST_COMMAND command =\n> MemoryPurgeStandbyList;\n> +\tcommand = MemoryPurgeStandbyList;\n> \tstatus = NtSetSystemInformation(\n> \t\tSystemMemoryListInformation,\n> \t\t&command,\n> @@ -111,8 +109,6 @@ int cmd_dropcaches(void)\n> \telse if (status != STATUS_SUCCESS)\n> \t\terror(\"Unable to execute the memory list command %d\",\n> status);\n> \n> -\tFreeLibrary(ntdll);\n> -\n> \treturn status;\n> }\n> \n> --\n> 2.14.1.windows.1.510.g0cb6d35d23\n"},{"id":"328386","messageId":"20170919141626.GA28437@tor.lan","threadId":"46154","inReplyTo":"MWHPR21MB0478E4671130AD9E19B77460F4630@MWHPR21MB0478.namprd21.prod.outlook.com","subject":"Re: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-09-19T14:16:26Z","receivedAt":"2017-09-19T14:16:54Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"> \n> Should I just make the variable type itself uintmax_t and then just skip\n> the cast altogether? I went with uint64_t because that is what\n> getnanotime returned.\n> \n\nThat is a bit of taste question (or answer)\n\nTypically you declare the variables in the type you need,\nand this is uint64_t.\n\nLet's step back a bit:\nTo print e.g a variable of type uint32_t, you use PRIu32 in the format\nstring, like this:\n\nfprintf(stderr, \"Total %\"PRIu32\" (delta %\"PRIu32\"),\",....\n\nIn theory (it is in the later specs, and it exists on many platforms),\nthere is a PRIu64 as well.\n\nWe don't seem to use it in Git, probably because uintmax_t is (more)\nportable and understood by all platforms which support Git.\n(And beside that, on most platforms uintmax_t is 64 bit).\n\nSo my suggestion would be to keep uint64_t and cast the variable into uintmax_t\nwhenever it is printed.\n\n\n"},{"id":"328389","messageId":"alpine.DEB.2.21.1.1709191726430.219280@virtualbox","threadId":"46154","inReplyTo":"MWHPR21MB04789C11E37F58FF9BC11C51F4630@MWHPR21MB0478.namprd21.prod.outlook.com","subject":"RE: [PATCH v6 12/12] fsmonitor: add a performance test","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-09-19T15:28:52Z","receivedAt":"2017-09-19T15:29:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ben,\n\nOn Mon, 18 Sep 2017, Ben Peart wrote:\n\n> > From: Johannes Schindelin [mailto:Johannes.Schindelin@gmx.de]\n> > On Fri, 15 Sep 2017, Ben Peart wrote:\n> > \n> > > +\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =\n> > > +\t\t(DWORD(WINAPI *)(INT, PVOID,\n> > ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n> > > +\tif (!NtSetSystemInformation)\n> > > +\t\treturn error(\"Can't get function addresses, wrong Windows\n> > > +version?\");\n> > \n> > It turns out that we have seen this plenty of times in Git for Windows'\n> > fork, so much so that we came up with a nice helper to make this all a bit\n> > more robust and a bit more obvious, too: the DECLARE_PROC_ADDR and\n> > INIT_PROC_ADDR helpers in compat/win32/lazyload.h.\n> > \n> > Maybe this would be the perfect excuse to integrate this patch into upstream\n> > Git? \n> \n> This patch is pretty hefty already. How about you push this capability\n> upstream and I take advantage of it in a later patch. :)\n\nDeal:\nhttps://public-inbox.org/git/f5a3add27206df3e7f39efeac8a3c3b47f2b79f2.1505834586.git.johannes.schindelin@gmx.de\n\nCiao,\nJohannes\n"},{"id":"328390","messageId":"MWHPR21MB0478181AE0B64901DA2C07CDF4600@MWHPR21MB0478.namprd21.prod.outlook.com","threadId":"46154","inReplyTo":"20170919141626.GA28437@tor.lan","subject":"RE: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2017-09-19T15:36:33Z","receivedAt":"2017-09-19T15:36:42Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"> -----Original Message-----\n> From: Torsten Bögershausen [mailto:tboegi@web.de]\n> Sent: Tuesday, September 19, 2017 10:16 AM\n> To: Ben Peart \n> Cc: Ben Peart ; Junio C Hamano\n> ; David.Turner@twosigma.com; avarab@gmail.com;\n> christian.couder@gmail.com; git@vger.kernel.org;\n> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> Subject: Re: [PATCH v6 08/12] fsmonitor: add a test tool to dump the index\n> extension\n> \n> >\n> > Should I just make the variable type itself uintmax_t and then just\n> > skip the cast altogether? I went with uint64_t because that is what\n> > getnanotime returned.\n> >\n> \n> That is a bit of taste question (or answer)\n> \n> Typically you declare the variables in the type you need, and this is uint64_t.\n> \n> Let's step back a bit:\n> To print e.g a variable of type uint32_t, you use PRIu32 in the format string,\n> like this:\n> \n> fprintf(stderr, \"Total %\"PRIu32\" (delta %\"PRIu32\"),\",....\n> \n> In theory (it is in the later specs, and it exists on many platforms), there is a\n> PRIu64 as well.\n> \n> We don't seem to use it in Git, probably because uintmax_t is (more)\n> portable and understood by all platforms which support Git.\n> (And beside that, on most platforms uintmax_t is 64 bit).\n> \n> So my suggestion would be to keep uint64_t and cast the variable into\n> uintmax_t whenever it is printed.\n> \n\nGreat! That is the way I have it.\n"},{"id":"328400","messageId":"20170919192744.19224-1-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170915192043.4516-1-benpeart@microsoft.com","subject":"[PATCH v7 00/12] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:32Z","receivedAt":"2017-09-19T19:28:29Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Subject: Fast git status via a file system watcher\n\nThanks to everyone who provided feedback. There are lots of minor style\nchanges, documentation updates and a fixed leak.\n\nThe only functional change is the addition of support to set/clear the\nfsmonitor valid bit via 'git update-index --[no-]fsmonitor-valid' with\nthe associated documentation and tests.\n\nInterdiff between V6 and V7:\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex c196007a27..db52645cb4 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -418,6 +418,7 @@ core.fsmonitor::\n \twill identify all files that may have changed since the\n \trequested date/time. This information is used to speed up git by\n \tavoiding unnecessary processing of files that have not changed.\n+\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex d153c17e06..3ac3e3a77d 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -9,7 +9,7 @@ git-ls-files - Show information about files in the index and the working tree\n SYNOPSIS\n --------\n [verse]\n-'git ls-files' [-z] [-t] [-v]\n+'git ls-files' [-z] [-t] [-v] [-f]\n \t\t(--[cached|deleted|others|ignored|stage|unmerged|killed|modified])*\n \t\t(-[c|d|o|i|s|u|k|m])*\n \t\t[--eol]\n@@ -133,6 +133,11 @@ a space) at the start of each line:\n \tthat are marked as 'assume unchanged' (see\n \tlinkgit:git-update-index[1]).\n \n+-f::\n+\tSimilar to `-t`, but use lowercase letters for files\n+\tthat are marked as 'fsmonitor valid' (see\n+\tlinkgit:git-update-index[1]).\n+\n --full-name::\n \tWhen run from a subdirectory, the command usually\n \toutputs paths relative to the current directory. This\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex e19eba62cd..95231dbfcb 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -16,9 +16,11 @@ SYNOPSIS\n \t [--chmod=(+|-)x]\n \t [--[no-]assume-unchanged]\n \t [--[no-]skip-worktree]\n+\t [--[no-]fsmonitor-valid]\n \t [--ignore-submodules]\n \t [--[no-]split-index]\n \t [--[no-|test-|force-]untracked-cache]\n+\t [--[no-]fsmonitor]\n \t [--really-refresh] [--unresolve] [--again | -g]\n \t [--info-only] [--index-info]\n \t [-z] [--stdin] [--index-version ]\n@@ -111,6 +113,12 @@ you will need to handle the situation manually.\n \tset and unset the \"skip-worktree\" bit for the paths. See\n \tsection \"Skip-worktree bit\" below for more information.\n \n+--[no-]fsmonitor-valid::\n+\tWhen one of these flags is specified, the object name recorded\n+\tfor the paths are not updated. Instead, these options\n+\tset and unset the \"fsmonitor valid\" bit for the paths. See\n+\tsection \"File System Monitor\" below for more information.\n+\n -g::\n --again::\n \tRuns 'git update-index' itself on the paths whose index\n@@ -201,6 +209,15 @@ will remove the intended effect of the option.\n \t`--untracked-cache` used to imply `--test-untracked-cache` but\n \tthis option would enable the extension unconditionally.\n \n+--fsmonitor::\n+--no-fsmonitor::\n+\tEnable or disable files system monitor feature. These options\n+\ttake effect whatever the value of the `core.fsmonitor`\n+\tconfiguration variable (see linkgit:git-config[1]). But a warning\n+\tis emitted when the change goes against the configured value, as\n+\tthe configured value will take effect next time the index is\n+\tread and this will remove the intended effect of the option.\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \n@@ -447,6 +464,34 @@ command reads the index; while when `--[no-|force-]untracked-cache`\n are used, the untracked cache is immediately added to or removed from\n the index.\n \n+File System Monitor\n+-------------------\n+\n+This feature is intended to speed up git operations for repos that have\n+large working directories.\n+\n+It enables git to work together with a file system monitor (see the\n+\"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n+inform it as to what files have been modified. This enables git to avoid\n+having to lstat() every file to find modified files.\n+\n+When used in conjunction with the untracked cache, it can further improve\n+performance by avoiding the cost of scaning the entire working directory\n+looking for new files.\n+\n+If you want to enable (or disable) this feature, it is easier to use\n+the `core.fsmonitor` configuration variable (see\n+linkgit:git-config[1]) than using the `--fsmonitor` option to\n+`git update-index` in each repository, especially if you want to do so\n+across all repositories you use, because you can set the configuration\n+variable to `true` (or `false`) in your `$HOME/.gitconfig` just once\n+and have it affect all repositories you touch.\n+\n+When the `core.fsmonitor` configuration variable is changed, the\n+file system monitor is added to or removed from the index the next time\n+a command reads the index. When `--[no-]fsmonitor` are used, the file\n+system monitor is immediately added to or removed from the index.\n+\n Configuration\n -------------\n \ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex da82d64b0b..ae60559cd9 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -455,10 +455,8 @@ the name of the file that holds the e-mail to be sent. Exiting with a\n non-zero status causes 'git send-email' to abort before sending any\n e-mails.\n \n-\n-[[fsmonitor-watchman]]\n fsmonitor-watchman\n-~~~~~~~~~~~~~~~\n+~~~~~~~~~~~~~~~~~~\n \n This hook is invoked when the configuration option core.fsmonitor is\n set to .git/hooks/fsmonitor-watchman. It takes two arguments, a version\n@@ -471,10 +469,17 @@ should be inclusive so that it does not miss any potential changes.\n The paths should be relative to the root of the working directory\n and be separated by a single NUL.\n \n+It is OK to include files which have not actually changed. All changes\n+including newly-created and deleted files should be included. When\n+files are renamed, both the old and the new name should be included.\n+\n Git will limit what files it checks for changes as well as which\n directories are checked for untracked files based on the path names\n given.\n \n+An optimized way to tell git \"all files have changed\" is to return\n+the filename '/'.\n+\n The exit status determines whether git will use the data from the\n hook to limit its search. On error, it will fall back to verifying\n all files and folders.\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex b03afc1f3a..41618db098 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -33,6 +33,7 @@ static int force_remove;\n static int verbose;\n static int mark_valid_only;\n static int mark_skip_worktree_only;\n+static int mark_fsmonitor_only;\n #define MARK_FLAG 1\n #define UNMARK_FLAG 2\n static struct strbuf mtime_dir = STRBUF_INIT;\n@@ -229,12 +230,12 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \tint namelen = strlen(path);\n \tint pos = cache_name_pos(path, namelen);\n \tif (0 <= pos) {\n+\t\tmark_fsmonitor_invalid(&the_index, active_cache[pos]);\n \t\tif (mark)\n \t\t\tactive_cache[pos]->ce_flags |= flag;\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\tmark_fsmonitor_invalid(&the_index, active_cache[pos]);\n \t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\n@@ -460,6 +461,11 @@ static void update_one(const char *path)\n \t\t\tdie(\"Unable to mark file %s\", path);\n \t\treturn;\n \t}\n+\tif (mark_fsmonitor_only) {\n+\t\tif (mark_ce_flags(path, CE_FSMONITOR_VALID, mark_fsmonitor_only == MARK_FLAG))\n+\t\t\tdie(\"Unable to mark file %s\", path);\n+\t\treturn;\n+\t}\n \n \tif (force_remove) {\n \t\tif (remove_file_from_cache(path))\n@@ -1014,6 +1020,12 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"write out the index even if is not flagged as changed\"), 1),\n \t\tOPT_BOOL(0, \"fsmonitor\", &fsmonitor,\n \t\t\tN_(\"enable or disable file system monitor\")),\n+\t\t{OPTION_SET_INT, 0, \"fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n+\t\t\tN_(\"mark files as fsmonitor valid\"),\n+\t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, MARK_FLAG},\n+\t\t{OPTION_SET_INT, 0, \"no-fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n+\t\t\tN_(\"clear fsmonitor valid bit\"),\n+\t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, UNMARK_FLAG},\n \t\tOPT_END()\n \t};\n \ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 144294b8df..b8b2d88fe1 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -57,12 +57,12 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,\n \n \t\t/* Mark all previously saved entries as dirty */\n \t\tewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n-\t\tewah_free(fsmonitor_dirty);\n \n \t\t/* Now mark the untracked cache for fsmonitor usage */\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t}\n+\tewah_free(fsmonitor_dirty);\n \n \ttrace_printf_key(&trace_fsmonitor, \"read fsmonitor extension successful\");\n \treturn 0;\n@@ -177,7 +177,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t}\n \n \t/* a fsmonitor process can return '*' to indicate all entries are invalid */\n-\tif (query_success && query_result.buf[0] != '*') {\n+\tif (query_success && query_result.buf[0] != '/') {\n \t\t/* Mark all entries returned by the monitor as dirty */\n \t\tbuf = query_result.buf;\n \t\tbol = 0;\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex dadbe90283..c2240b811a 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -46,7 +46,7 @@ static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n }\n \n /*\n- * Clear the given cache entries CE_FSMONITOR_VALID bit and invalidate any\n+ * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate any\n * corresponding untracked cache directory structures.\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\ndiff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\nindex 717079865c..4e5ca8f397 100644\n--- a/t/helper/test-drop-caches.c\n+++ b/t/helper/test-drop-caches.c\n@@ -2,7 +2,7 @@\n \n #if defined(GIT_WINDOWS_NATIVE)\n \n-int cmd_sync(void)\n+static int cmd_sync(void)\n {\n \tchar Buffer[MAX_PATH];\n \tDWORD dwRet;\n@@ -49,7 +49,7 @@ typedef enum _SYSTEM_MEMORY_LIST_COMMAND {\n \tMemoryCommandMax\n } SYSTEM_MEMORY_LIST_COMMAND;\n \n-BOOL GetPrivilege(HANDLE TokenHandle, LPCSTR lpName, int flags)\n+static BOOL GetPrivilege(HANDLE TokenHandle, LPCSTR lpName, int flags)\n {\n \tBOOL bResult;\n \tDWORD dwBufferLength;\n@@ -77,10 +77,11 @@ BOOL GetPrivilege(HANDLE TokenHandle, LPCSTR lpName, int flags)\n \treturn bResult;\n }\n \n-int cmd_dropcaches(void)\n+static int cmd_dropcaches(void)\n {\n \tHANDLE hProcess = GetCurrentProcess();\n \tHANDLE hToken;\n+\tHMODULE ntdll;\n \tint status;\n \n \tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n@@ -91,7 +92,7 @@ int cmd_dropcaches(void)\n \n \tCloseHandle(hToken);\n \n-\tHMODULE ntdll = LoadLibrary(\"ntdll.dll\");\n+\tntdll = LoadLibrary(\"ntdll.dll\");\n \tif (!ntdll)\n \t\treturn error(\"Can't load ntdll.dll, wrong Windows version?\");\n \n@@ -118,36 +119,36 @@ int cmd_dropcaches(void)\n \n #elif defined(__linux__)\n \n-int cmd_sync(void)\n+static int cmd_sync(void)\n {\n \treturn system(\"sync\");\n }\n \n-int cmd_dropcaches(void)\n+static int cmd_dropcaches(void)\n {\n \treturn system(\"echo 3 | sudo tee /proc/sys/vm/drop_caches\");\n }\n \n #elif defined(__APPLE__)\n \n-int cmd_sync(void)\n+static int cmd_sync(void)\n {\n \treturn system(\"sync\");\n }\n \n-int cmd_dropcaches(void)\n+static int cmd_dropcaches(void)\n {\n \treturn system(\"sudo purge\");\n }\n \n #else\n \n-int cmd_sync(void)\n+static int cmd_sync(void)\n {\n \treturn 0;\n }\n \n-int cmd_dropcaches(void)\n+static int cmd_dropcaches(void)\n {\n \treturn error(\"drop caches not implemented on this platform\");\n }\ndiff --git a/t/helper/test-dump-fsmonitor.c b/t/helper/test-dump-fsmonitor.c\nindex 482d749bb9..ad452707e8 100644\n--- a/t/helper/test-dump-fsmonitor.c\n+++ b/t/helper/test-dump-fsmonitor.c\n@@ -12,7 +12,7 @@ int cmd_main(int ac, const char **av)\n \t\tprintf(\"no fsmonitor\\n\");\n \t\treturn 0;\n \t}\n-\tprintf(\"fsmonitor last update %\"PRIuMAX\"\\n\", istate->fsmonitor_last_update);\n+\tprintf(\"fsmonitor last update %\"PRIuMAX\"\\n\", (uintmax_t)istate->fsmonitor_last_update);\n \n \tfor (i = 0; i < istate->cache_nr; i++)\n \t\tprintf((istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) ? \"+\" : \"-\");\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 1c5978d5c8..16d1bf72e5 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -95,7 +95,7 @@ test_expect_success \"setup for fsmonitor\" '\n \t\tINTEGRATION_SCRIPT=\"$GIT_PERF_7519_FSMONITOR\"\n \telse\n \t\t#\n-\t\t# Choose integration script based on existance of Watchman.\n+\t\t# Choose integration script based on existence of Watchman.\n \t\t# If Watchman exists, watch the work tree and attempt a query.\n \t\t# If everything succeeds, use Watchman integration script,\n \t\t# else fall back to an empty integration script.\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 6aa1e4e924..c6df85af5e 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -32,14 +32,16 @@ dirty_repo () {\n \techo 6 >dir2/new\n }\n \n-write_integration_script() {\n+write_integration_script () {\n \twrite_script .git/hooks/fsmonitor-test<<-\\EOF\n-\tif [ \"$#\" -ne 2 ]; then\n+\tif test \"$#\" -ne 2\n+\tthen\n \t\techo \"$0: exactly 2 arguments expected\"\n \t\texit 2\n \tfi\n-\tif [ \"$1\" != 1 ]; then\n-\t\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n+\tif test \"$1\" != 1\n+\tthen\n+\t\techo \"Unsupported core.fsmonitor hook version.\" >&2\n \t\texit 1\n \tfi\n \tprintf \"untracked\\0\"\n@@ -100,6 +102,43 @@ test_expect_success 'update-index --no-fsmonitor\" removes the fsmonitor extensio\n \tgrep \"^no fsmonitor\" actual\n '\n \n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n cat >expect <actual &&\n \t\t\ttest_path_is_file marker &&\n \t\t\ttest_i18ngrep ! \"Changes not staged for commit:\" actual &&\n-\t\t\tif [ $uc_val -eq true ]; then test_i18ngrep ! \"Untracked files:\" actual; fi &&\n-\t\t\tif [ $uc_val -eq false ]; then test_i18ngrep \"Untracked files:\" actual; fi &&\n+\t\t\tif test $uc_val = true\n+\t\t\tthen\n+\t\t\t\ttest_i18ngrep ! \"Untracked files:\" actual\n+\t\t\tfi &&\n+\t\t\tif test $uc_val = false\n+\t\t\tthen\n+\t\t\t\ttest_i18ngrep \"Untracked files:\" actual\n+\t\t\tfi &&\n \t\t\trm -f marker\n \t\t'\n \tdone\ndiff --git a/t/t7519/fsmonitor-all b/t/t7519/fsmonitor-all\nindex a3870e431e..691bc94dc2 100755\n--- a/t/t7519/fsmonitor-all\n+++ b/t/t7519/fsmonitor-all\n@@ -9,15 +9,16 @@\n #\n #echo \"$0 $*\" >&2\n \n-if [ \"$#\" -ne 2 ] ; then\n-\techo -e \"$0: exactly 2 arguments expected\\n\" >&2\n+if test \"$#\" -ne 2\n+then\n+\techo \"$0: exactly 2 arguments expected\" >&2\n \texit 2\n fi\n \n-if [ \"$1\" != 1 ]\n+if test \"$1\" != 1\n then\n-\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n+\techo \"Unsupported core.fsmonitor hook version.\" >&2\n \texit 1\n fi\n \n-echo \"*\"\n\\ No newline at end of file\n+echo \"/\"\ndiff --git a/t/t7519/fsmonitor-none b/t/t7519/fsmonitor-none\nindex c500bb0f26..ed9cf5a6a9 100755\n--- a/t/t7519/fsmonitor-none\n+++ b/t/t7519/fsmonitor-none\n@@ -9,13 +9,14 @@\n #\n #echo \"$0 $*\" >&2\n \n-if [ \"$#\" -ne 2 ] ; then\n-\techo -e \"$0: exactly 2 arguments expected\\n\" >&2\n+if test \"$#\" -ne 2\n+then\n+\techo \"$0: exactly 2 arguments expected\" >&2\n \texit 2\n fi\n \n-if [ \"$1\" != 1 ]\n+if test \"$1\" != 1\n then\n-\techo -e \"Unsupported core.fsmonitor hook version.\\n\" >&2\n+\techo \"Unsupported core.fsmonitor hook version.\" >&2\n \texit 1\n fi\ndiff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\nindex aaee5d1fe3..7ceb32dc18 100755\n--- a/t/t7519/fsmonitor-watchman\n+++ b/t/t7519/fsmonitor-watchman\n@@ -17,7 +17,7 @@ use IPC::Open2;\n # 'git config core.fsmonitor .git/hooks/query-watchman'\n #\n my ($version, $time) = @ARGV;\n-print STDERR \"$0 $version $time\\n\";\n+#print STDERR \"$0 $version $time\\n\";\n \n # Check the hook interface version\n \n@@ -29,7 +29,20 @@ if ($version == 1) {\n \t \"Falling back to scanning...\\n\";\n }\n \n-my $git_work_tree = $ENV{'PWD'};\n+# Convert unix style paths to escaped Windows style paths when running\n+# in Windows command prompt\n+\n+my $system = `uname -s`;\n+$system =~ s/[\\r\\n]+//g;\n+my $git_work_tree;\n+\n+if ($system =~ m/^MSYS_NT/) {\n+\t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n+\t$git_work_tree =~ s/[\\r\\n]+//g;\n+\t$git_work_tree =~ s,\\\\,/,g;\n+} else {\n+\t$git_work_tree = $ENV{'PWD'};\n+}\n \n my $retry = 1;\n \n@@ -57,20 +70,19 @@ sub launch_watchman {\n \t# creation clock (cclock) newer than $time_t value and will also not\n \t# currently exist.\n \n+\tmy $query = <<\"\tEND\";\n+\t\t[\"query\", \"$git_work_tree\", {\n+\t\t\t\"since\": $time,\n+\t\t\t\"fields\": [\"name\"],\n+\t\t\t\"expression\": [\"not\", [\"allof\", [\"since\", $time, \"cclock\"], [\"not\", \"exists\"]]]\n+\t\t}]\n+\tEND\n+\t\n \topen (my $fh, \">\", \".git/watchman-query.json\");\n-\tprint $fh \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n-\t\\\"since\\\": $time, \\\n-\t\\\"fields\\\": [\\\"name\\\"], \\\n-\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n-\t}]\";\n+\tprint $fh $query;\n \tclose $fh;\n \n-\tprint CHLD_IN \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n-\t\\\"since\\\": $time, \\\n-\t\\\"fields\\\": [\\\"name\\\"], \\\n-\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n-\t}]\";\n-\n+\tprint CHLD_IN $query;\n \tmy $response = ;\n \n \topen ($fh, \">\", \".git/watchman-response.json\");\n@@ -100,17 +112,17 @@ sub launch_watchman {\n \t\tqx/watchman watch \"$git_work_tree\"/;\n \t\tdie \"Failed to make watchman watch '$git_work_tree'.\\n\" .\n \t\t \"Falling back to scanning...\\n\" if $? != 0;\n-\t\t# return fast \"everything is dirty\" flag\"\n-\t\tprint \"*\\0\";\n-\t\topen ($fh, \">\", \".git/watchman-output.out\");\n-\t\tprint \"*\\0\";\n-\t\tclose $fh;\n \n \t\t# Watchman will always return all files on the first query so\n \t\t# return the fast \"everything is dirty\" flag to git and do the\n \t\t# Watchman query just to get it over with now so we won't pay\n \t\t# the cost in git to look up each individual file.\n-\t\tprint \"*\\0\";\n+\n+\t\topen ($fh, \">\", \".git/watchman-output.out\");\n+\t\tprint \"/\\0\";\n+\t\tclose $fh;\n+\n+\t\tprint \"/\\0\";\n \t\teval { launch_watchman() };\n \t\texit 0;\n \t}\n@@ -118,11 +130,11 @@ sub launch_watchman {\n \tdie \"Watchman: $o->{error}.\\n\" .\n \t \"Falling back to scanning...\\n\" if $o->{error};\n \n-\tbinmode STDOUT, \":utf8\";\n-\tlocal $, = \"\\0\";\n-\tprint @{$o->{files}};\n-\n \topen ($fh, \">\", \".git/watchman-output.out\");\n \tprint $fh @{$o->{files}};\n \tclose $fh;\n-}\n\\ No newline at end of file\n+\n+\tbinmode STDOUT, \":utf8\";\n+\tlocal $, = \"\\0\";\n+\tprint @{$o->{files}};\n+}\ndiff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample\nindex 2779d7edf3..870a59d237 100755\n--- a/templates/hooks--fsmonitor-watchman.sample\n+++ b/templates/hooks--fsmonitor-watchman.sample\n@@ -69,12 +69,15 @@ sub launch_watchman {\n \t# creation clock (cclock) newer than $time_t value and will also not\n \t# currently exist.\n \n-\tprint CHLD_IN \"[\\\"query\\\", \\\"$git_work_tree\\\", { \\\n-\t\\\"since\\\": $time, \\\n-\t\\\"fields\\\": [\\\"name\\\"], \\\n-\t\\\"expression\\\": [\\\"not\\\", [\\\"allof\\\", [\\\"since\\\", $time, \\\"cclock\\\"], [\\\"not\\\", \\\"exists\\\"]]] \\\n-\t}]\";\n-\n+\tmy $query = <<\"\tEND\";\n+\t\t[\"query\", \"$git_work_tree\", {\n+\t\t\t\"since\": $time,\n+\t\t\t\"fields\": [\"name\"],\n+\t\t\t\"expression\": [\"not\", [\"allof\", [\"since\", $time, \"cclock\"], [\"not\", \"exists\"]]]\n+\t\t}]\n+\tEND\n+\n+\tprint CHLD_IN $query;\n \tmy $response = ;\n \n \tdie \"Watchman: command returned no output.\\n\" .\n@@ -105,7 +108,7 @@ sub launch_watchman {\n \t\t# return the fast \"everything is dirty\" flag to git and do the\n \t\t# Watchman query just to get it over with now so we won't pay\n \t\t# the cost in git to look up each individual file.\n-\t\tprint \"*\\0\";\n+\t\tprint \"/\\0\";\n \t\teval { launch_watchman() };\n \t\texit 0;\n \t}\n\nBen Peart (12):\n bswap: add 64 bit endianness helper get_be64\n preload-index: add override to enable testing preload-index\n update-index: add a new --force-write-index option\n fsmonitor: teach git to optionally utilize a file system monitor to\n speed up detecting new or changed files.\n fsmonitor: add documentation for the fsmonitor extension.\n ls-files: Add support in ls-files to display the fsmonitor valid bit\n update-index: add fsmonitor support to update-index\n fsmonitor: add a test tool to dump the index extension\n split-index: disable the fsmonitor extension when running the split\n index test\n fsmonitor: add test cases for fsmonitor extension\n fsmonitor: add a sample integration script for Watchman\n fsmonitor: add a performance test\n\n Documentation/config.txt | 7 +\n Documentation/git-ls-files.txt | 7 +-\n Documentation/git-update-index.txt | 45 +++++\n Documentation/githooks.txt | 28 +++\n Documentation/technical/index-format.txt | 19 ++\n Makefile | 3 +\n apply.c | 2 +-\n builtin/ls-files.c | 8 +-\n builtin/update-index.c | 38 +++-\n cache.h | 10 +-\n compat/bswap.h | 22 +++\n config.c | 14 ++\n config.h | 1 +\n diff-lib.c | 2 +\n dir.c | 27 ++-\n dir.h | 2 +\n entry.c | 4 +-\n environment.c | 1 +\n fsmonitor.c | 253 ++++++++++++++++++++++++\n fsmonitor.h | 61 ++++++\n preload-index.c | 8 +-\n read-cache.c | 49 ++++-\n submodule.c | 2 +-\n t/helper/.gitignore | 1 +\n t/helper/test-drop-caches.c | 162 +++++++++++++++\n t/helper/test-dump-fsmonitor.c | 21 ++\n t/perf/p7519-fsmonitor.sh | 184 +++++++++++++++++\n t/t1700-split-index.sh | 1 +\n t/t7519-status-fsmonitor.sh | 304 +++++++++++++++++++++++++++++\n t/t7519/fsmonitor-all | 24 +++\n t/t7519/fsmonitor-none | 22 +++\n t/t7519/fsmonitor-watchman | 140 +++++++++++++\n templates/hooks--fsmonitor-watchman.sample | 122 ++++++++++++\n unpack-trees.c | 8 +-\n 34 files changed, 1572 insertions(+), 30 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n create mode 100644 t/helper/test-drop-caches.c\n create mode 100644 t/helper/test-dump-fsmonitor.c\n create mode 100755 t/perf/p7519-fsmonitor.sh\n create mode 100755 t/t7519-status-fsmonitor.sh\n create mode 100755 t/t7519/fsmonitor-all\n create mode 100755 t/t7519/fsmonitor-none\n create mode 100755 t/t7519/fsmonitor-watchman\n create mode 100755 templates/hooks--fsmonitor-watchman.sample\n\n-- \n2.14.1.windows.1\n\n"},{"id":"328401","messageId":"20170919192744.19224-2-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 01/12] bswap: add 64 bit endianness helper get_be64","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:33Z","receivedAt":"2017-09-19T19:28:34Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a new get_be64 macro to enable 64 bit endian conversions on memory\nthat may or may not be aligned.\n\nSigned-off-by: Ben Peart \n---\n compat/bswap.h | 22 ++++++++++++++++++++++\n 1 file changed, 22 insertions(+)\n\ndiff --git a/compat/bswap.h b/compat/bswap.h\nindex 7d063e9e40..6b22c46214 100644\n--- a/compat/bswap.h\n+++ b/compat/bswap.h\n@@ -158,7 +158,9 @@ static inline uint64_t git_bswap64(uint64_t x)\n \n #define get_be16(p)\tntohs(*(unsigned short *)(p))\n #define get_be32(p)\tntohl(*(unsigned int *)(p))\n+#define get_be64(p)\tntohll(*(uint64_t *)(p))\n #define put_be32(p, v)\tdo { *(unsigned int *)(p) = htonl(v); } while (0)\n+#define put_be64(p, v)\tdo { *(uint64_t *)(p) = htonll(v); } while (0)\n \n #else\n \n@@ -178,6 +180,13 @@ static inline uint32_t get_be32(const void *ptr)\n \t\t(uint32_t)p[3] << 0;\n }\n \n+static inline uint64_t get_be64(const void *ptr)\n+{\n+\tconst unsigned char *p = ptr;\n+\treturn\t(uint64_t)get_be32(p[0]) << 32 |\n+\t\t(uint64_t)get_be32(p[4]) << 0;\n+}\n+\n static inline void put_be32(void *ptr, uint32_t value)\n {\n \tunsigned char *p = ptr;\n@@ -187,4 +196,17 @@ static inline void put_be32(void *ptr, uint32_t value)\n \tp[3] = value >> 0;\n }\n \n+static inline void put_be64(void *ptr, uint64_t value)\n+{\n+\tunsigned char *p = ptr;\n+\tp[0] = value >> 56;\n+\tp[1] = value >> 48;\n+\tp[2] = value >> 40;\n+\tp[3] = value >> 32;\n+\tp[4] = value >> 24;\n+\tp[5] = value >> 16;\n+\tp[6] = value >> 8;\n+\tp[7] = value >> 0;\n+}\n+\n #endif\n-- \n2.14.1.windows.1\n\n"},{"id":"328402","messageId":"20170919192744.19224-3-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 02/12] preload-index: add override to enable testing preload-index","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:34Z","receivedAt":"2017-09-19T19:28:41Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Preload index doesn't run unless it has a minimum number of 1000 files.\nTo enable running tests with fewer files, add an environment variable\n(GIT_FORCE_PRELOAD_TEST) which will override that minimum and set it to 2.\n\nSigned-off-by: Ben Peart \n---\n preload-index.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/preload-index.c b/preload-index.c\nindex 70a4c80878..75564c497a 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -79,6 +79,8 @@ static void preload_index(struct index_state *index,\n \t\treturn;\n \n \tthreads = index->cache_nr / THREAD_COST;\n+\tif ((index->cache_nr > 1) && (threads < 2) && getenv(\"GIT_FORCE_PRELOAD_TEST\"))\n+\t\tthreads = 2;\n \tif (threads < 2)\n \t\treturn;\n \tif (threads > MAX_PARALLEL)\n-- \n2.14.1.windows.1\n\n"},{"id":"328403","messageId":"20170919192744.19224-6-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:37Z","receivedAt":"2017-09-19T19:28:44Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This includes the core.fsmonitor setting, the query-fsmonitor hook,\nand the fsmonitor index extension.\n\nSigned-off-by: Ben Peart \n---\n Documentation/config.txt | 7 +++++\n Documentation/git-ls-files.txt | 7 ++++-\n Documentation/git-update-index.txt | 45 ++++++++++++++++++++++++++++++++\n Documentation/githooks.txt | 28 ++++++++++++++++++++\n Documentation/technical/index-format.txt | 19 ++++++++++++++\n 5 files changed, 105 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dc4e3f58a2..db52645cb4 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -413,6 +413,13 @@ core.protectNTFS::\n \t8.3 \"short\" names.\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n+core.fsmonitor::\n+\tIf set, the value of this variable is used as a command which\n+\twill identify all files that may have changed since the\n+\trequested date/time. This information is used to speed up git by\n+\tavoiding unnecessary processing of files that have not changed.\n+\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex d153c17e06..3ac3e3a77d 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -9,7 +9,7 @@ git-ls-files - Show information about files in the index and the working tree\n SYNOPSIS\n --------\n [verse]\n-'git ls-files' [-z] [-t] [-v]\n+'git ls-files' [-z] [-t] [-v] [-f]\n \t\t(--[cached|deleted|others|ignored|stage|unmerged|killed|modified])*\n \t\t(-[c|d|o|i|s|u|k|m])*\n \t\t[--eol]\n@@ -133,6 +133,11 @@ a space) at the start of each line:\n \tthat are marked as 'assume unchanged' (see\n \tlinkgit:git-update-index[1]).\n \n+-f::\n+\tSimilar to `-t`, but use lowercase letters for files\n+\tthat are marked as 'fsmonitor valid' (see\n+\tlinkgit:git-update-index[1]).\n+\n --full-name::\n \tWhen run from a subdirectory, the command usually\n \toutputs paths relative to the current directory. This\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex e19eba62cd..95231dbfcb 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -16,9 +16,11 @@ SYNOPSIS\n \t [--chmod=(+|-)x]\n \t [--[no-]assume-unchanged]\n \t [--[no-]skip-worktree]\n+\t [--[no-]fsmonitor-valid]\n \t [--ignore-submodules]\n \t [--[no-]split-index]\n \t [--[no-|test-|force-]untracked-cache]\n+\t [--[no-]fsmonitor]\n \t [--really-refresh] [--unresolve] [--again | -g]\n \t [--info-only] [--index-info]\n \t [-z] [--stdin] [--index-version ]\n@@ -111,6 +113,12 @@ you will need to handle the situation manually.\n \tset and unset the \"skip-worktree\" bit for the paths. See\n \tsection \"Skip-worktree bit\" below for more information.\n \n+--[no-]fsmonitor-valid::\n+\tWhen one of these flags is specified, the object name recorded\n+\tfor the paths are not updated. Instead, these options\n+\tset and unset the \"fsmonitor valid\" bit for the paths. See\n+\tsection \"File System Monitor\" below for more information.\n+\n -g::\n --again::\n \tRuns 'git update-index' itself on the paths whose index\n@@ -201,6 +209,15 @@ will remove the intended effect of the option.\n \t`--untracked-cache` used to imply `--test-untracked-cache` but\n \tthis option would enable the extension unconditionally.\n \n+--fsmonitor::\n+--no-fsmonitor::\n+\tEnable or disable files system monitor feature. These options\n+\ttake effect whatever the value of the `core.fsmonitor`\n+\tconfiguration variable (see linkgit:git-config[1]). But a warning\n+\tis emitted when the change goes against the configured value, as\n+\tthe configured value will take effect next time the index is\n+\tread and this will remove the intended effect of the option.\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \n@@ -447,6 +464,34 @@ command reads the index; while when `--[no-|force-]untracked-cache`\n are used, the untracked cache is immediately added to or removed from\n the index.\n \n+File System Monitor\n+-------------------\n+\n+This feature is intended to speed up git operations for repos that have\n+large working directories.\n+\n+It enables git to work together with a file system monitor (see the\n+\"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n+inform it as to what files have been modified. This enables git to avoid\n+having to lstat() every file to find modified files.\n+\n+When used in conjunction with the untracked cache, it can further improve\n+performance by avoiding the cost of scaning the entire working directory\n+looking for new files.\n+\n+If you want to enable (or disable) this feature, it is easier to use\n+the `core.fsmonitor` configuration variable (see\n+linkgit:git-config[1]) than using the `--fsmonitor` option to\n+`git update-index` in each repository, especially if you want to do so\n+across all repositories you use, because you can set the configuration\n+variable to `true` (or `false`) in your `$HOME/.gitconfig` just once\n+and have it affect all repositories you touch.\n+\n+When the `core.fsmonitor` configuration variable is changed, the\n+file system monitor is added to or removed from the index the next time\n+a command reads the index. When `--[no-]fsmonitor` are used, the file\n+system monitor is immediately added to or removed from the index.\n+\n Configuration\n -------------\n \ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 1bb4f92d4d..ae60559cd9 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -455,6 +455,34 @@ the name of the file that holds the e-mail to be sent. Exiting with a\n non-zero status causes 'git send-email' to abort before sending any\n e-mails.\n \n+fsmonitor-watchman\n+~~~~~~~~~~~~~~~~~~\n+\n+This hook is invoked when the configuration option core.fsmonitor is\n+set to .git/hooks/fsmonitor-watchman. It takes two arguments, a version\n+(currently 1) and the time in elapsed nanoseconds since midnight,\n+January 1, 1970.\n+\n+The hook should output to stdout the list of all files in the working\n+directory that may have changed since the requested time. The logic\n+should be inclusive so that it does not miss any potential changes.\n+The paths should be relative to the root of the working directory\n+and be separated by a single NUL.\n+\n+It is OK to include files which have not actually changed. All changes\n+including newly-created and deleted files should be included. When\n+files are renamed, both the old and the new name should be included.\n+\n+Git will limit what files it checks for changes as well as which\n+directories are checked for untracked files based on the path names\n+given.\n+\n+An optimized way to tell git \"all files have changed\" is to return\n+the filename '/'.\n+\n+The exit status determines whether git will use the data from the\n+hook to limit its search. On error, it will fall back to verifying\n+all files and folders.\n \n GIT\n ---\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex ade0b0c445..db3572626b 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -295,3 +295,22 @@ The remaining data of each directory block is grouped by type:\n in the previous ewah bitmap.\n \n - One NUL.\n+\n+== File System Monitor cache\n+\n+ The file system monitor cache tracks files for which the core.fsmonitor\n+ hook has told us about changes. The signature for this extension is\n+ { 'F', 'S', 'M', 'N' }.\n+\n+ The extension starts with\n+\n+ - 32-bit version number: the current supported version is 1.\n+\n+ - 64-bit time: the extension data reflects all changes through the given\n+\ttime which is stored as the nanoseconds elapsed since midnight,\n+\tJanuary 1, 1970.\n+\n+ - 32-bit bitmap size: the size of the CE_FSMONITOR_VALID bitmap.\n+\n+ - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n+ is not CE_FSMONITOR_VALID.\n-- \n2.14.1.windows.1\n\n"},{"id":"328404","messageId":"20170919192744.19224-5-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:36Z","receivedAt":"2017-09-19T19:28:47Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"When the index is read from disk, the fsmonitor index extension is used\nto flag the last known potentially dirty index entries. The registered\ncore.fsmonitor command is called with the time the index was last\nupdated and returns the list of files changed since that time. This list\nis used to flag any additional dirty cache entries and untracked cache\ndirectories.\n\nWe can then use this valid state to speed up preload_index(),\nie_match_stat(), and refresh_cache_ent() as they do not need to lstat()\nfiles to detect potential changes for those entries marked\nCE_FSMONITOR_VALID.\n\nIn addition, if the untracked cache is turned on valid_cached_dir() can\nskip checking directories for new or changed files as fsmonitor will\ninvalidate the cache only for those directories that have been\nidentified as having potential changes.\n\nTo keep the CE_FSMONITOR_VALID state accurate during git operations;\nwhen git updates a cache entry to match the current state on disk,\nit will now set the CE_FSMONITOR_VALID bit.\n\nInversely, anytime git changes a cache entry, the CE_FSMONITOR_VALID bit\nis cleared and the corresponding untracked cache directory is marked\ninvalid.\n\nSigned-off-by: Ben Peart \n---\n Makefile | 1 +\n apply.c | 2 +-\n builtin/update-index.c | 2 +\n cache.h | 10 +-\n config.c | 14 +++\n config.h | 1 +\n diff-lib.c | 2 +\n dir.c | 27 ++++--\n dir.h | 2 +\n entry.c | 4 +-\n environment.c | 1 +\n fsmonitor.c | 253 +++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor.h | 61 ++++++++++++\n preload-index.c | 6 +-\n read-cache.c | 49 ++++++++--\n submodule.c | 2 +-\n unpack-trees.c | 8 +-\n 17 files changed, 419 insertions(+), 26 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n\ndiff --git a/Makefile b/Makefile\nindex f2bb7f2f63..9d6ec9c1e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -786,6 +786,7 @@ LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n+LIB_OBJS += fsmonitor.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/apply.c b/apply.c\nindex 71cbbd141c..9061cc5f15 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -3399,7 +3399,7 @@ static int verify_index_match(const struct cache_entry *ce, struct stat *st)\n \t\t\treturn -1;\n \t\treturn 0;\n \t}\n-\treturn ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n+\treturn ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n }\n \n #define SUBMODULE_PATCH_WITHOUT_INDEX 1\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex e1ca0759d5..6f39ee9274 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -16,6 +16,7 @@\n #include \"pathspec.h\"\n #include \"dir.h\"\n #include \"split-index.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Default to not allowing changes to the list of files. The\n@@ -233,6 +234,7 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(&the_index, active_cache[pos]);\n \t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\ndiff --git a/cache.h b/cache.h\nindex a916bc79e3..eccab968bd 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -203,6 +203,7 @@ struct cache_entry {\n #define CE_ADDED (1 << 19)\n \n #define CE_HASHED (1 << 20)\n+#define CE_FSMONITOR_VALID (1 << 21)\n #define CE_WT_REMOVE (1 << 22) /* remove in work directory */\n #define CE_CONFLICTED (1 << 23)\n \n@@ -326,6 +327,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n #define UNTRACKED_CHANGED\t(1 << 7)\n+#define FSMONITOR_CHANGED\t(1 << 8)\n \n struct split_index;\n struct untracked_cache;\n@@ -344,6 +346,7 @@ struct index_state {\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n \tstruct untracked_cache *untracked;\n+\tuint64_t fsmonitor_last_update;\n };\n \n extern struct index_state the_index;\n@@ -679,8 +682,10 @@ extern void *read_blob_data_from_index(const struct index_state *, const char *,\n #define CE_MATCH_IGNORE_MISSING\t\t0x08\n /* enable stat refresh */\n #define CE_MATCH_REFRESH\t\t0x10\n-extern int ie_match_stat(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n-extern int ie_modified(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n+/* do stat comparison even if CE_FSMONITOR_VALID is true */\n+#define CE_MATCH_IGNORE_FSMONITOR 0X20\n+extern int ie_match_stat(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n+extern int ie_modified(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n \n #define HASH_WRITE_OBJECT 1\n #define HASH_FORMAT_CHECK 2\n@@ -773,6 +778,7 @@ extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n+extern const char *core_fsmonitor;\n \n /*\n * Include broken refs in all ref iterations, which will\ndiff --git a/config.c b/config.c\nindex d0d8ce823a..ddda96e584 100644\n--- a/config.c\n+++ b/config.c\n@@ -2165,6 +2165,20 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n+int git_config_get_fsmonitor(void)\n+{\n+\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n+\t\tcore_fsmonitor = getenv(\"GIT_FSMONITOR_TEST\");\n+\n+\tif (core_fsmonitor && !*core_fsmonitor)\n+\t\tcore_fsmonitor = NULL;\n+\n+\tif (core_fsmonitor)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\ndiff --git a/config.h b/config.h\nindex 97471b8873..c9fcf691ba 100644\n--- a/config.h\n+++ b/config.h\n@@ -211,6 +211,7 @@ extern int git_config_get_pathname(const char *key, const char **dest);\n extern int git_config_get_untracked_cache(void);\n extern int git_config_get_split_index(void);\n extern int git_config_get_max_percent_split_change(void);\n+extern int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n extern int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 2a52b07954..23c6d03ca9 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"submodule.h\"\n #include \"dir.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * diff-files\n@@ -228,6 +229,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \n \t\tif (!changed && !dirty_submodule) {\n \t\t\tce_mark_uptodate(ce);\n+\t\t\tmark_fsmonitor_valid(ce);\n \t\t\tif (!DIFF_OPT_TST(&revs->diffopt, FIND_COPIES_HARDER))\n \t\t\t\tcontinue;\n \t\t}\ndiff --git a/dir.c b/dir.c\nindex 1c55dc3e36..ac9833daec 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -18,6 +18,7 @@\n #include \"utf8.h\"\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Tells read_directory_recursive how a file or directory should be treated.\n@@ -1688,17 +1689,23 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n-\tif (stat(path->len ? path->buf : \".\", &st)) {\n-\t\tinvalidate_directory(dir->untracked, untracked);\n-\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n-\t\treturn 0;\n-\t}\n-\tif (!untracked->valid ||\n-\t match_stat_data_racy(istate, &untracked->stat_data, &st)) {\n-\t\tif (untracked->valid)\n+\t/*\n+\t * With fsmonitor, we can trust the untracked cache's valid field.\n+\t */\n+\trefresh_fsmonitor(istate);\n+\tif (!(dir->untracked->use_fsmonitor && untracked->valid)) {\n+\t\tif (stat(path->len ? path->buf : \".\", &st)) {\n \t\t\tinvalidate_directory(dir->untracked, untracked);\n-\t\tfill_stat_data(&untracked->stat_data, &st);\n-\t\treturn 0;\n+\t\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (!untracked->valid ||\n+\t\t\tmatch_stat_data_racy(istate, &untracked->stat_data, &st)) {\n+\t\t\tif (untracked->valid)\n+\t\t\t\tinvalidate_directory(dir->untracked, untracked);\n+\t\t\tfill_stat_data(&untracked->stat_data, &st);\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n \tif (untracked->check_only != !!check_only) {\ndiff --git a/dir.h b/dir.h\nindex e3717055d1..fab8fc1561 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -139,6 +139,8 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n+\t/* fsmonitor invalidation data */\n+\tunsigned int use_fsmonitor : 1;\n };\n \n struct dir_struct {\ndiff --git a/entry.c b/entry.c\nindex cb291aa88b..5e6794f9fc 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -4,6 +4,7 @@\n #include \"streaming.h\"\n #include \"submodule.h\"\n #include \"progress.h\"\n+#include \"fsmonitor.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -357,6 +358,7 @@ static int write_entry(struct cache_entry *ce,\n \t\t\tlstat(ce->name, &st);\n \t\tfill_stat_cache_info(ce, &st);\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(state->istate, ce);\n \t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \treturn 0;\n@@ -402,7 +404,7 @@ int checkout_entry(struct cache_entry *ce,\n \n \tif (!check_path(path.buf, path.len, &st, state->base_dir_len)) {\n \t\tconst struct submodule *sub;\n-\t\tunsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n+\t\tunsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n \t\t/*\n \t\t * Needs to be checked before !changed returns early,\n \t\t * as the possibly empty directory was not changed\ndiff --git a/environment.c b/environment.c\nindex 3fd4b10845..d0b9fc64d4 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -76,6 +76,7 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 0\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n+const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor.c b/fsmonitor.c\nnew file mode 100644\nindex 0000000000..b8b2d88fe1\n--- /dev/null\n+++ b/fsmonitor.c\n@@ -0,0 +1,253 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"ewah/ewok.h\"\n+#include \"fsmonitor.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+\n+#define INDEX_EXTENSION_VERSION\t(1)\n+#define HOOK_INTERFACE_VERSION\t(1)\n+\n+struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR);\n+\n+static void fsmonitor_ewah_callback(size_t pos, void *is)\n+{\n+\tstruct index_state *istate = (struct index_state *)is;\n+\tstruct cache_entry *ce = istate->cache[pos];\n+\n+\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+}\n+\n+int read_fsmonitor_extension(struct index_state *istate, const void *data,\n+\tunsigned long sz)\n+{\n+\tconst char *index = data;\n+\tuint32_t hdr_version;\n+\tuint32_t ewah_size;\n+\tstruct ewah_bitmap *fsmonitor_dirty;\n+\tint i;\n+\tint ret;\n+\n+\tif (sz < sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t))\n+\t\treturn error(\"corrupt fsmonitor extension (too short)\");\n+\n+\thdr_version = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\tif (hdr_version != INDEX_EXTENSION_VERSION)\n+\t\treturn error(\"bad fsmonitor version %d\", hdr_version);\n+\n+\tistate->fsmonitor_last_update = get_be64(index);\n+\tindex += sizeof(uint64_t);\n+\n+\tewah_size = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\n+\tfsmonitor_dirty = ewah_new();\n+\tret = ewah_read_mmap(fsmonitor_dirty, index, ewah_size);\n+\tif (ret != ewah_size) {\n+\t\tewah_free(fsmonitor_dirty);\n+\t\treturn error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n+\t}\n+\n+\tif (git_config_get_fsmonitor()) {\n+\t\t/* Mark all entries valid */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n+\n+\t\t/* Mark all previously saved entries as dirty */\n+\t\tewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n+\n+\t\t/* Now mark the untracked cache for fsmonitor usage */\n+\t\tif (istate->untracked)\n+\t\t\tistate->untracked->use_fsmonitor = 1;\n+\t}\n+\tewah_free(fsmonitor_dirty);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"read fsmonitor extension successful\");\n+\treturn 0;\n+}\n+\n+void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n+{\n+\tuint32_t hdr_version;\n+\tuint64_t tm;\n+\tstruct ewah_bitmap *bitmap;\n+\tint i;\n+\tuint32_t ewah_start;\n+\tuint32_t ewah_size = 0;\n+\tint fixup = 0;\n+\n+\tput_be32(&hdr_version, INDEX_EXTENSION_VERSION);\n+\tstrbuf_add(sb, &hdr_version, sizeof(uint32_t));\n+\n+\tput_be64(&tm, istate->fsmonitor_last_update);\n+\tstrbuf_add(sb, &tm, sizeof(uint64_t));\n+\tfixup = sb->len;\n+\tstrbuf_add(sb, &ewah_size, sizeof(uint32_t)); /* we'll fix this up later */\n+\n+\tewah_start = sb->len;\n+\tbitmap = ewah_new();\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))\n+\t\t\tewah_set(bitmap, i);\n+\tewah_serialize_strbuf(bitmap, sb);\n+\tewah_free(bitmap);\n+\n+\t/* fix up size field */\n+\tput_be32(&ewah_size, sb->len - ewah_start);\n+\tmemcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t));\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"write fsmonitor extension successful\");\n+}\n+\n+/*\n+ * Call the query-fsmonitor hook passing the time of the last saved results.\n+ */\n+static int query_fsmonitor(int version, uint64_t last_update, struct strbuf *query_result)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tchar ver[64];\n+\tchar date[64];\n+\tconst char *argv[4];\n+\n+\tif (!(argv[0] = core_fsmonitor))\n+\t\treturn -1;\n+\n+\tsnprintf(ver, sizeof(version), \"%d\", version);\n+\tsnprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update);\n+\targv[1] = ver;\n+\targv[2] = date;\n+\targv[3] = NULL;\n+\tcp.argv = argv;\n+\tcp.use_shell = 1;\n+\n+\treturn capture_command(&cp, query_result, 1024);\n+}\n+\n+static void fsmonitor_refresh_callback(struct index_state *istate, const char *name)\n+{\n+\tint pos = index_name_pos(istate, name, strlen(name));\n+\n+\tif (pos >= 0) {\n+\t\tstruct cache_entry *ce = istate->cache[pos];\n+\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t}\n+\n+\t/*\n+\t * Mark the untracked cache dirty even if it wasn't found in the index\n+\t * as it could be a new untracked file.\n+\t */\n+\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n+\tuntracked_cache_invalidate_path(istate, name);\n+}\n+\n+void refresh_fsmonitor(struct index_state *istate)\n+{\n+\tstatic int has_run_once = 0;\n+\tstruct strbuf query_result = STRBUF_INIT;\n+\tint query_success = 0;\n+\tsize_t bol; /* beginning of line */\n+\tuint64_t last_update;\n+\tchar *buf;\n+\tint i;\n+\n+\tif (!core_fsmonitor || has_run_once)\n+\t\treturn;\n+\thas_run_once = 1;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\t/*\n+\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * should be inclusive to ensure we don't miss potential changes.\n+\t */\n+\tlast_update = getnanotime();\n+\n+\t/*\n+\t * If we have a last update time, call query_fsmonitor for the set of\n+\t * changes since that time, else assume everything is possibly dirty\n+\t * and check it all.\n+\t */\n+\tif (istate->fsmonitor_last_update) {\n+\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION,\n+\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n+\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n+\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t}\n+\n+\t/* a fsmonitor process can return '*' to indicate all entries are invalid */\n+\tif (query_success && query_result.buf[0] != '/') {\n+\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\tbuf = query_result.buf;\n+\t\tbol = 0;\n+\t\tfor (i = 0; i < query_result.len; i++) {\n+\t\t\tif (buf[i] != '\\0')\n+\t\t\t\tcontinue;\n+\t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tbol = i + 1;\n+\t\t}\n+\t\tif (bol < query_result.len)\n+\t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t} else {\n+\t\t/* Mark all entries invalid */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\n+\t\tif (istate->untracked)\n+\t\t\tistate->untracked->use_fsmonitor = 0;\n+\t}\n+\tstrbuf_release(&query_result);\n+\n+\t/* Now that we've updated istate, save the last_update time */\n+\tistate->fsmonitor_last_update = last_update;\n+}\n+\n+void add_fsmonitor(struct index_state *istate)\n+{\n+\tint i;\n+\n+\tif (!istate->fsmonitor_last_update) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"add fsmonitor\");\n+\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\tistate->fsmonitor_last_update = getnanotime();\n+\n+\t\t/* reset the fsmonitor state */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\n+\t\t/* reset the untracked cache */\n+\t\tif (istate->untracked) {\n+\t\t\tadd_untracked_cache(istate);\n+\t\t\tistate->untracked->use_fsmonitor = 1;\n+\t\t}\n+\n+\t\t/* Update the fsmonitor state */\n+\t\trefresh_fsmonitor(istate);\n+\t}\n+}\n+\n+void remove_fsmonitor(struct index_state *istate)\n+{\n+\tif (istate->fsmonitor_last_update) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"remove fsmonitor\");\n+\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\tistate->fsmonitor_last_update = 0;\n+\t}\n+}\n+\n+void tweak_fsmonitor(struct index_state *istate)\n+{\n+\tswitch (git_config_get_fsmonitor()) {\n+\tcase -1: /* keep: do nothing */\n+\t\tbreak;\n+\tcase 0: /* false */\n+\t\tremove_fsmonitor(istate);\n+\t\tbreak;\n+\tcase 1: /* true */\n+\t\tadd_fsmonitor(istate);\n+\t\tbreak;\n+\tdefault: /* unknown value: do nothing */\n+\t\tbreak;\n+\t}\n+}\ndiff --git a/fsmonitor.h b/fsmonitor.h\nnew file mode 100644\nindex 0000000000..c2240b811a\n--- /dev/null\n+++ b/fsmonitor.h\n@@ -0,0 +1,61 @@\n+#ifndef FSMONITOR_H\n+#define FSMONITOR_H\n+\n+extern struct trace_key trace_fsmonitor;\n+\n+/*\n+ * Read the the fsmonitor index extension and (if configured) restore the\n+ * CE_FSMONITOR_VALID state.\n+ */\n+extern int read_fsmonitor_extension(struct index_state *istate, const void *data, unsigned long sz);\n+\n+/*\n+ * Write the CE_FSMONITOR_VALID state into the fsmonitor index extension.\n+ */\n+extern void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate);\n+\n+/*\n+ * Add/remove the fsmonitor index extension\n+ */\n+extern void add_fsmonitor(struct index_state *istate);\n+extern void remove_fsmonitor(struct index_state *istate);\n+\n+/*\n+ * Add/remove the fsmonitor index extension as necessary based on the current\n+ * core.fsmonitor setting.\n+ */\n+extern void tweak_fsmonitor(struct index_state *istate);\n+\n+/*\n+ * Run the configured fsmonitor integration script and clear the\n+ * CE_FSMONITOR_VALID bit for any files returned as dirty. Also invalidate\n+ * any corresponding untracked cache directory structures. Optimized to only\n+ * run the first time it is called.\n+ */\n+extern void refresh_fsmonitor(struct index_state *istate);\n+\n+/*\n+ * Set the given cache entries CE_FSMONITOR_VALID bit.\n+ */\n+static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n+{\n+\tif (core_fsmonitor) {\n+\t\tce->ce_flags |= CE_FSMONITOR_VALID;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n+\t}\n+}\n+\n+/*\n+ * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate any\n+ * corresponding untracked cache directory structures.\n+ */\n+static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n+{\n+\tif (core_fsmonitor) {\n+\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tuntracked_cache_invalidate_path(istate, ce->name);\n+\t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\n+\t}\n+}\n+\n+#endif\ndiff --git a/preload-index.c b/preload-index.c\nindex 75564c497a..2a83255e4e 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -4,6 +4,7 @@\n #include \"cache.h\"\n #include \"pathspec.h\"\n #include \"dir.h\"\n+#include \"fsmonitor.h\"\n \n #ifdef NO_PTHREADS\n static void preload_index(struct index_state *index,\n@@ -55,15 +56,18 @@ static void *preload_thread(void *_data)\n \t\t\tcontinue;\n \t\tif (ce_skip_worktree(ce))\n \t\t\tcontinue;\n+\t\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n+\t\t\tcontinue;\n \t\tif (!ce_path_match(ce, &p->pathspec, NULL))\n \t\t\tcontinue;\n \t\tif (threaded_has_symlink_leading_path(&cache, ce->name, ce_namelen(ce)))\n \t\t\tcontinue;\n \t\tif (lstat(ce->name, &st))\n \t\t\tcontinue;\n-\t\tif (ie_match_stat(index, ce, &st, CE_MATCH_RACY_IS_DIRTY))\n+\t\tif (ie_match_stat(index, ce, &st, CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR))\n \t\t\tcontinue;\n \t\tce_mark_uptodate(ce);\n+\t\tmark_fsmonitor_valid(ce);\n \t} while (--nr > 0);\n \tcache_def_clear(&cache);\n \treturn NULL;\ndiff --git a/read-cache.c b/read-cache.c\nindex 40da87ea71..53093dbebf 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -19,6 +19,7 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"utf8.h\"\n+#include \"fsmonitor.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -38,11 +39,12 @@\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n #define CACHE_EXT_UNTRACKED 0x554E5452\t /* \"UNTR\" */\n+#define CACHE_EXT_FSMONITOR 0x46534D4E\t /* \"FSMN\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -62,6 +64,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \tfree(old);\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tmark_fsmonitor_invalid(istate, ce);\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n }\n \n@@ -150,8 +153,10 @@ void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n \tif (assume_unchanged)\n \t\tce->ce_flags |= CE_VALID;\n \n-\tif (S_ISREG(st->st_mode))\n+\tif (S_ISREG(st->st_mode)) {\n \t\tce_mark_uptodate(ce);\n+\t\tmark_fsmonitor_valid(ce);\n+\t}\n }\n \n static int ce_compare_data(const struct cache_entry *ce, struct stat *st)\n@@ -300,7 +305,7 @@ int match_stat_data_racy(const struct index_state *istate,\n \treturn match_stat_data(sd, st);\n }\n \n-int ie_match_stat(const struct index_state *istate,\n+int ie_match_stat(struct index_state *istate,\n \t\t const struct cache_entry *ce, struct stat *st,\n \t\t unsigned int options)\n {\n@@ -308,7 +313,10 @@ int ie_match_stat(const struct index_state *istate,\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n \tint assume_racy_is_modified = options & CE_MATCH_RACY_IS_DIRTY;\n+\tint ignore_fsmonitor = options & CE_MATCH_IGNORE_FSMONITOR;\n \n+\tif (!ignore_fsmonitor)\n+\t\trefresh_fsmonitor(istate);\n \t/*\n \t * If it's marked as always valid in the index, it's\n \t * valid whatever the checked-out copy says.\n@@ -319,6 +327,8 @@ int ie_match_stat(const struct index_state *istate,\n \t\treturn 0;\n \tif (!ignore_valid && (ce->ce_flags & CE_VALID))\n \t\treturn 0;\n+\tif (!ignore_fsmonitor && (ce->ce_flags & CE_FSMONITOR_VALID))\n+\t\treturn 0;\n \n \t/*\n \t * Intent-to-add entries have not been added, so the index entry\n@@ -356,7 +366,7 @@ int ie_match_stat(const struct index_state *istate,\n \treturn changed;\n }\n \n-int ie_modified(const struct index_state *istate,\n+int ie_modified(struct index_state *istate,\n \t\tconst struct cache_entry *ce,\n \t\tstruct stat *st, unsigned int options)\n {\n@@ -631,7 +641,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \tint size, namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias;\n-\tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n+\tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR;\n \tint verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);\n \tint pretend = flags & ADD_CACHE_PRETEND;\n \tint intent_only = flags & ADD_CACHE_INTENT;\n@@ -777,6 +787,7 @@ int chmod_index_entry(struct index_state *istate, struct cache_entry *ce,\n \t}\n \tcache_tree_invalidate_path(istate, ce->name);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tmark_fsmonitor_invalid(istate, ce);\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n \n \treturn 0;\n@@ -1228,10 +1239,13 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n \tint ignore_missing = options & CE_MATCH_IGNORE_MISSING;\n+\tint ignore_fsmonitor = options & CE_MATCH_IGNORE_FSMONITOR;\n \n \tif (!refresh || ce_uptodate(ce))\n \t\treturn ce;\n \n+\tif (!ignore_fsmonitor)\n+\t\trefresh_fsmonitor(istate);\n \t/*\n \t * CE_VALID or CE_SKIP_WORKTREE means the user promised us\n \t * that the change to the work tree does not matter and told\n@@ -1245,6 +1259,10 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\tce_mark_uptodate(ce);\n \t\treturn ce;\n \t}\n+\tif (!ignore_fsmonitor && (ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tce_mark_uptodate(ce);\n+\t\treturn ce;\n+\t}\n \n \tif (has_symlink_leading_path(ce->name, ce_namelen(ce))) {\n \t\tif (ignore_missing)\n@@ -1282,8 +1300,10 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\t\t * because CE_UPTODATE flag is in-core only;\n \t\t\t * we are not going to write this change out.\n \t\t\t */\n-\t\t\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\t\tif (!S_ISGITLINK(ce->ce_mode)) {\n \t\t\t\tce_mark_uptodate(ce);\n+\t\t\t\tmark_fsmonitor_valid(ce);\n+\t\t\t}\n \t\t\treturn ce;\n \t\t}\n \t}\n@@ -1336,7 +1356,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tint first = 1;\n \tint in_porcelain = (flags & REFRESH_IN_PORCELAIN);\n \tunsigned int options = (CE_MATCH_REFRESH |\n-\t\t\t\t(really ? CE_MATCH_IGNORE_VALID : 0) |\n+\t\t\t\t(really ? CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_FSMONITOR : 0) |\n \t\t\t\t(not_new ? CE_MATCH_IGNORE_MISSING : 0));\n \tconst char *modified_fmt;\n \tconst char *deleted_fmt;\n@@ -1391,6 +1411,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\t\t */\n \t\t\t\tce->ce_flags &= ~CE_VALID;\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\t\t\tmark_fsmonitor_invalid(istate, ce);\n \t\t\t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t\t\t}\n \t\t\tif (quiet)\n@@ -1550,6 +1571,9 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_UNTRACKED:\n \t\tistate->untracked = read_untracked_extension(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_FSMONITOR:\n+\t\tread_fsmonitor_extension(istate, data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1722,6 +1746,7 @@ static void post_read_index_from(struct index_state *istate)\n \tcheck_ce_order(istate);\n \ttweak_untracked_cache(istate);\n \ttweak_split_index(istate);\n+\ttweak_fsmonitor(istate);\n }\n \n /* remember to discard_cache() before reading a different cache! */\n@@ -2306,6 +2331,16 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->fsmonitor_last_update) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_fsmonitor_extension(&sb, istate);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_FSMONITOR, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1))\n \t\treturn -1;\ndiff --git a/submodule.c b/submodule.c\nindex 3cea8221e0..8a931a1aaa 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -62,7 +62,7 @@ int is_staging_gitmodules_ok(const struct index_state *istate)\n \tif ((pos >= 0) && (pos < istate->cache_nr)) {\n \t\tstruct stat st;\n \t\tif (lstat(GITMODULES_FILE, &st) == 0 &&\n-\t\t ce_match_stat(istate->cache[pos], &st, 0) & DATA_CHANGED)\n+\t\t ce_match_stat(istate->cache[pos], &st, CE_MATCH_IGNORE_FSMONITOR) & DATA_CHANGED)\n \t\t\treturn 0;\n \t}\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 71b70ccb12..f724a61ac0 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -408,6 +409,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n \tif (was_skip_worktree != ce_skip_worktree(ce)) {\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(istate, ce);\n \t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \n@@ -1454,7 +1456,7 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n \t\treturn 0;\n \n \tif (!lstat(ce->name, &st)) {\n-\t\tint flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE;\n+\t\tint flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR;\n \t\tunsigned changed = ie_match_stat(o->src_index, ce, &st, flags);\n \n \t\tif (submodule_from_ce(ce)) {\n@@ -1610,7 +1612,7 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le\n \tconst struct cache_entry *src;\n \n \tsrc = index_file_exists(o->src_index, name, len, 1);\n-\treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n+\treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n }\n \n static int check_ok_to_remove(const char *name, int len, int dtype,\n@@ -2134,7 +2136,7 @@ int oneway_merge(const struct cache_entry * const *src,\n \t\tif (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old)) {\n \t\t\tstruct stat st;\n \t\t\tif (lstat(old->name, &st) ||\n-\t\t\t ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE))\n+\t\t\t ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR))\n \t\t\t\tupdate |= CE_UPDATE;\n \t\t}\n \t\tadd_entry(o, old, update, 0);\n-- \n2.14.1.windows.1\n\n"},{"id":"328405","messageId":"20170919192744.19224-9-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:40Z","receivedAt":"2017-09-19T19:28:51Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a test utility (test-dump-fsmonitor) that will dump the fsmonitor\nindex extension.\n\nSigned-off-by: Ben Peart \n---\n Makefile | 1 +\n t/helper/test-dump-fsmonitor.c | 21 +++++++++++++++++++++\n 2 files changed, 22 insertions(+)\n create mode 100644 t/helper/test-dump-fsmonitor.c\n\ndiff --git a/Makefile b/Makefile\nindex 9d6ec9c1e9..d970cd00e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -639,6 +639,7 @@ TEST_PROGRAMS_NEED_X += test-config\n TEST_PROGRAMS_NEED_X += test-date\n TEST_PROGRAMS_NEED_X += test-delta\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n+TEST_PROGRAMS_NEED_X += test-dump-fsmonitor\n TEST_PROGRAMS_NEED_X += test-dump-split-index\n TEST_PROGRAMS_NEED_X += test-dump-untracked-cache\n TEST_PROGRAMS_NEED_X += test-fake-ssh\ndiff --git a/t/helper/test-dump-fsmonitor.c b/t/helper/test-dump-fsmonitor.c\nnew file mode 100644\nindex 0000000000..ad452707e8\n--- /dev/null\n+++ b/t/helper/test-dump-fsmonitor.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+\n+int cmd_main(int ac, const char **av)\n+{\n+\tstruct index_state *istate = &the_index;\n+\tint i;\n+\n+\tsetup_git_directory();\n+\tif (do_read_index(istate, get_index_file(), 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update) {\n+\t\tprintf(\"no fsmonitor\\n\");\n+\t\treturn 0;\n+\t}\n+\tprintf(\"fsmonitor last update %\"PRIuMAX\"\\n\", (uintmax_t)istate->fsmonitor_last_update);\n+\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tprintf((istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) ? \"+\" : \"-\");\n+\n+\treturn 0;\n+}\n-- \n2.14.1.windows.1\n\n"},{"id":"328406","messageId":"20170919192744.19224-10-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 09/12] split-index: disable the fsmonitor extension when running the split index test","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:41Z","receivedAt":"2017-09-19T19:28:53Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"The split index test t1700-split-index.sh has hard coded SHA values for\nthe index. Currently it supports index V4 and V3 but assumes there are\nno index extensions loaded.\n\nWhen manually forcing the fsmonitor extension to be turned on when\nrunning the test suite, the SHA values no longer match which causes the\ntest to fail.\n\nThe potential matrix of index extensions and index versions can is quite\nlarge so instead disable the extension before attempting to run the test.\n\nSigned-off-by: Ben Peart \n---\n t/t1700-split-index.sh | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex 22f69a410b..af9b847761 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -6,6 +6,7 @@ test_description='split index mode tests'\n \n # We need total control of index splitting here\n sane_unset GIT_TEST_SPLIT_INDEX\n+sane_unset GIT_FSMONITOR_TEST\n \n test_expect_success 'enable split index' '\n \tgit config splitIndex.maxPercentChange 100 &&\n-- \n2.14.1.windows.1\n\n"},{"id":"328407","messageId":"20170919192744.19224-8-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 07/12] update-index: add fsmonitor support to update-index","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:39Z","receivedAt":"2017-09-19T19:29:00Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add support in update-index to manually add/remove the fsmonitor\nextension via --[no-]fsmonitor flags.\n\nAdd support in update-index to manually set/clear the fsmonitor\nvalid bit via --[no-]fsmonitor-valid flags.\n\nSigned-off-by: Ben Peart \n---\n builtin/update-index.c | 33 ++++++++++++++++++++++++++++++++-\n 1 file changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 6f39ee9274..41618db098 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -33,6 +33,7 @@ static int force_remove;\n static int verbose;\n static int mark_valid_only;\n static int mark_skip_worktree_only;\n+static int mark_fsmonitor_only;\n #define MARK_FLAG 1\n #define UNMARK_FLAG 2\n static struct strbuf mtime_dir = STRBUF_INIT;\n@@ -229,12 +230,12 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \tint namelen = strlen(path);\n \tint pos = cache_name_pos(path, namelen);\n \tif (0 <= pos) {\n+\t\tmark_fsmonitor_invalid(&the_index, active_cache[pos]);\n \t\tif (mark)\n \t\t\tactive_cache[pos]->ce_flags |= flag;\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\tmark_fsmonitor_invalid(&the_index, active_cache[pos]);\n \t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\n@@ -460,6 +461,11 @@ static void update_one(const char *path)\n \t\t\tdie(\"Unable to mark file %s\", path);\n \t\treturn;\n \t}\n+\tif (mark_fsmonitor_only) {\n+\t\tif (mark_ce_flags(path, CE_FSMONITOR_VALID, mark_fsmonitor_only == MARK_FLAG))\n+\t\t\tdie(\"Unable to mark file %s\", path);\n+\t\treturn;\n+\t}\n \n \tif (force_remove) {\n \t\tif (remove_file_from_cache(path))\n@@ -918,6 +924,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tint lock_error = 0;\n \tint split_index = -1;\n \tint force_write = 0;\n+\tint fsmonitor = -1;\n \tstruct lock_file *lock_file;\n \tstruct parse_opt_ctx_t ctx;\n \tstrbuf_getline_fn getline_fn;\n@@ -1011,6 +1018,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"enable untracked cache without testing the filesystem\"), UC_FORCE),\n \t\tOPT_SET_INT(0, \"force-write-index\", &force_write,\n \t\t\tN_(\"write out the index even if is not flagged as changed\"), 1),\n+\t\tOPT_BOOL(0, \"fsmonitor\", &fsmonitor,\n+\t\t\tN_(\"enable or disable file system monitor\")),\n+\t\t{OPTION_SET_INT, 0, \"fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n+\t\t\tN_(\"mark files as fsmonitor valid\"),\n+\t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, MARK_FLAG},\n+\t\t{OPTION_SET_INT, 0, \"no-fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n+\t\t\tN_(\"clear fsmonitor valid bit\"),\n+\t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, UNMARK_FLAG},\n \t\tOPT_END()\n \t};\n \n@@ -1152,6 +1167,22 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n \t}\n \n+\tif (fsmonitor > 0) {\n+\t\tif (git_config_get_fsmonitor() == 0)\n+\t\t\twarning(_(\"core.fsmonitor is unset; \"\n+\t\t\t\t\"set it if you really want to \"\n+\t\t\t\t\"enable fsmonitor\"));\n+\t\tadd_fsmonitor(&the_index);\n+\t\treport(_(\"fsmonitor enabled\"));\n+\t} else if (!fsmonitor) {\n+\t\tif (git_config_get_fsmonitor() == 1)\n+\t\t\twarning(_(\"core.fsmonitor is set; \"\n+\t\t\t\t\"remove it if you really want to \"\n+\t\t\t\t\"disable fsmonitor\"));\n+\t\tremove_fsmonitor(&the_index);\n+\t\treport(_(\"fsmonitor disabled\"));\n+\t}\n+\n \tif (active_cache_changed || force_write) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n-- \n2.14.1.windows.1\n\n"},{"id":"328408","messageId":"20170919192744.19224-11-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:42Z","receivedAt":"2017-09-19T19:29:02Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Test the ability to add/remove the fsmonitor index extension via\nupdate-index.\n\nTest that dirty files returned from the integration script are properly\nrepresented in the index extension and verify that ls-files correctly\nreports their state.\n\nTest that ensure status results are correct when using the new fsmonitor\nextension. Test untracked, modified, and new files by ensuring the\nresults are identical to when not using the extension.\n\nTest that if the fsmonitor extension doesn't tell git about a change, it\ndoesn't discover it on its own. This ensures git is honoring the\nextension and that we get the performance benefits desired.\n\nThree test integration scripts are provided:\n\nfsmonitor-all - marks all files as dirty\nfsmonitor-none - marks no files as dirty\nfsmonitor-watchman - integrates with Watchman with debug logging\n\nTo run tests in the test suite while utilizing fsmonitor:\n\nFirst copy t/t7519/fsmonitor-all to a location in your path and then set\nGIT_FORCE_PRELOAD_TEST=true and GIT_FSMONITOR_TEST=fsmonitor-all and run\nyour tests.\n\nNote: currently when using the test script fsmonitor-watchman on\nWindows, many tests fail due to a reported but not yet fixed bug in\nWatchman where it holds on to handles for directories and files which\nprevents the test directory from being cleaned up properly.\n\nSigned-off-by: Ben Peart \n---\n t/t7519-status-fsmonitor.sh | 304 ++++++++++++++++++++++++++++++++++++++++++++\n t/t7519/fsmonitor-all | 24 ++++\n t/t7519/fsmonitor-none | 22 ++++\n t/t7519/fsmonitor-watchman | 140 ++++++++++++++++++++\n 4 files changed, 490 insertions(+)\n create mode 100755 t/t7519-status-fsmonitor.sh\n create mode 100755 t/t7519/fsmonitor-all\n create mode 100755 t/t7519/fsmonitor-none\n create mode 100755 t/t7519/fsmonitor-watchman\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nnew file mode 100755\nindex 0000000000..c6df85af5e\n--- /dev/null\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -0,0 +1,304 @@\n+#!/bin/sh\n+\n+test_description='git status with file system watcher'\n+\n+. ./test-lib.sh\n+\n+#\n+# To run the entire git test suite using fsmonitor:\n+#\n+# copy t/t7519/fsmonitor-all to a location in your path and then set\n+# GIT_FSMONITOR_TEST=fsmonitor-all and run your tests.\n+#\n+\n+# Note, after \"git reset --hard HEAD\" no extensions exist other than 'TREE'\n+# \"git update-index --fsmonitor\" can be used to get the extension written\n+# before testing the results.\n+\n+clean_repo () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd\n+}\n+\n+dirty_repo () {\n+\t: >untracked &&\n+\t: >dir1/untracked &&\n+\t: >dir2/untracked &&\n+\techo 1 >modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\techo 4 >new &&\n+\techo 5 >dir1/new &&\n+\techo 6 >dir2/new\n+}\n+\n+write_integration_script () {\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF\n+\tif test \"$#\" -ne 2\n+\tthen\n+\t\techo \"$0: exactly 2 arguments expected\"\n+\t\texit 2\n+\tfi\n+\tif test \"$1\" != 1\n+\tthen\n+\t\techo \"Unsupported core.fsmonitor hook version.\" >&2\n+\t\texit 1\n+\tfi\n+\tprintf \"untracked\\0\"\n+\tprintf \"dir1/untracked\\0\"\n+\tprintf \"dir2/untracked\\0\"\n+\tprintf \"modified\\0\"\n+\tprintf \"dir1/modified\\0\"\n+\tprintf \"dir2/modified\\0\"\n+\tprintf \"new\\0\"\n+\tprintf \"dir1/new\\0\"\n+\tprintf \"dir2/new\\0\"\n+\tEOF\n+}\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_expect_success 'setup' '\n+\tmkdir -p .git/hooks &&\n+\t: >tracked &&\n+\t: >modified &&\n+\tmkdir dir1 &&\n+\t: >dir1/tracked &&\n+\t: >dir1/modified &&\n+\tmkdir dir2 &&\n+\t: >dir2/tracked &&\n+\t: >dir2/modified &&\n+\tgit -c core.fsmonitor= add . &&\n+\tgit -c core.fsmonitor= commit -m initial &&\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcat >.gitignore <<-\\EOF\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tmarker*\n+\tEOF\n+'\n+\n+# test that the fsmonitor extension is off by default\n+test_expect_success 'fsmonitor extension is off by default' '\n+\ttest-dump-fsmonitor >actual &&\n+\tgrep \"^no fsmonitor\" actual\n+'\n+\n+# test that \"update-index --fsmonitor\" adds the fsmonitor extension\n+test_expect_success 'update-index --fsmonitor\" adds the fsmonitor extension' '\n+\tgit update-index --fsmonitor &&\n+\ttest-dump-fsmonitor >actual &&\n+\tgrep \"^fsmonitor last update\" actual\n+'\n+\n+# test that \"update-index --no-fsmonitor\" removes the fsmonitor extension\n+test_expect_success 'update-index --no-fsmonitor\" removes the fsmonitor extension' '\n+\tgit update-index --no-fsmonitor &&\n+\ttest-dump-fsmonitor >actual &&\n+\tgrep \"^no fsmonitor\" actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\ttest_must_fail git update-index --refresh --force-write-index &&\n+\tgit ls-files -f >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# Ensure commands that call refresh_index() to move the index back in time\n+# properly invalidate the fsmonitor cache\n+test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\tEOF\n+\tclean_repo &&\n+\tdirty_repo &&\n+\tgit add . &&\n+\tgit commit -m \"to reset\" &&\n+\tgit reset HEAD~1 &&\n+\tgit status >actual &&\n+\tgit -c core.fsmonitor= status >expect &&\n+\ttest_i18ncmp expect actual\n+'\n+\n+# test fsmonitor with and without preloadIndex\n+preload_values=\"false true\"\n+for preload_val in $preload_values\n+do\n+\ttest_expect_success \"setup preloadIndex to $preload_val\" '\n+\t\tgit config core.preloadIndex $preload_val &&\n+\t\tif test $preload_val = true\n+\t\tthen\n+\t\t\tGIT_FORCE_PRELOAD_TEST=$preload_val; export GIT_FORCE_PRELOAD_TEST\n+\t\telse\n+\t\t\tunset GIT_FORCE_PRELOAD_TEST\n+\t\tfi\n+\t'\n+\n+\t# test fsmonitor with and without the untracked cache (if available)\n+\tuc_values=\"false\"\n+\ttest_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\ttest_expect_success \"setup untracked cache to $uc_val\" '\n+\t\t\tgit config core.untrackedcache $uc_val\n+\t\t'\n+\n+\t\t# Status is well tested elsewhere so we'll just ensure that the results are\n+\t\t# the same when using core.fsmonitor.\n+\t\ttest_expect_success 'compare status with and without fsmonitor' '\n+\t\t\twrite_integration_script &&\n+\t\t\tclean_repo &&\n+\t\t\tdirty_repo &&\n+\t\t\tgit add new &&\n+\t\t\tgit add dir1/new &&\n+\t\t\tgit add dir2/new &&\n+\t\t\tgit status >actual &&\n+\t\t\tgit -c core.fsmonitor= status >expect &&\n+\t\t\ttest_i18ncmp expect actual\n+\t\t'\n+\n+\t\t# Make sure it's actually skipping the check for modified and untracked\n+\t\t# (if enabled) files unless it is told about them.\n+\t\ttest_expect_success \"status doesn't detect unreported modifications\" '\n+\t\t\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\t\t:>marker\n+\t\t\tEOF\n+\t\t\tclean_repo &&\n+\t\t\tgit status &&\n+\t\t\ttest_path_is_file marker &&\n+\t\t\tdirty_repo &&\n+\t\t\trm -f marker &&\n+\t\t\tgit status >actual &&\n+\t\t\ttest_path_is_file marker &&\n+\t\t\ttest_i18ngrep ! \"Changes not staged for commit:\" actual &&\n+\t\t\tif test $uc_val = true\n+\t\t\tthen\n+\t\t\t\ttest_i18ngrep ! \"Untracked files:\" actual\n+\t\t\tfi &&\n+\t\t\tif test $uc_val = false\n+\t\t\tthen\n+\t\t\t\ttest_i18ngrep \"Untracked files:\" actual\n+\t\t\tfi &&\n+\t\t\trm -f marker\n+\t\t'\n+\tdone\n+done\n+\n+test_done\ndiff --git a/t/t7519/fsmonitor-all b/t/t7519/fsmonitor-all\nnew file mode 100755\nindex 0000000000..691bc94dc2\n--- /dev/null\n+++ b/t/t7519/fsmonitor-all\n@@ -0,0 +1,24 @@\n+#!/bin/sh\n+#\n+# An test hook script to integrate with git to test fsmonitor.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+#echo \"$0 $*\" >&2\n+\n+if test \"$#\" -ne 2\n+then\n+\techo \"$0: exactly 2 arguments expected\" >&2\n+\texit 2\n+fi\n+\n+if test \"$1\" != 1\n+then\n+\techo \"Unsupported core.fsmonitor hook version.\" >&2\n+\texit 1\n+fi\n+\n+echo \"/\"\ndiff --git a/t/t7519/fsmonitor-none b/t/t7519/fsmonitor-none\nnew file mode 100755\nindex 0000000000..ed9cf5a6a9\n--- /dev/null\n+++ b/t/t7519/fsmonitor-none\n@@ -0,0 +1,22 @@\n+#!/bin/sh\n+#\n+# An test hook script to integrate with git to test fsmonitor.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+#echo \"$0 $*\" >&2\n+\n+if test \"$#\" -ne 2\n+then\n+\techo \"$0: exactly 2 arguments expected\" >&2\n+\texit 2\n+fi\n+\n+if test \"$1\" != 1\n+then\n+\techo \"Unsupported core.fsmonitor hook version.\" >&2\n+\texit 1\n+fi\ndiff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\nnew file mode 100755\nindex 0000000000..7ceb32dc18\n--- /dev/null\n+++ b/t/t7519/fsmonitor-watchman\n@@ -0,0 +1,140 @@\n+#!/usr/bin/perl\n+\n+use strict;\n+use warnings;\n+use IPC::Open2;\n+\n+# An example hook script to integrate Watchman\n+# (https://facebook.github.io/watchman/) with git to speed up detecting\n+# new and modified files.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+# To enable this hook, rename this file to \"query-watchman\" and set\n+# 'git config core.fsmonitor .git/hooks/query-watchman'\n+#\n+my ($version, $time) = @ARGV;\n+#print STDERR \"$0 $version $time\\n\";\n+\n+# Check the hook interface version\n+\n+if ($version == 1) {\n+\t# convert nanoseconds to seconds\n+\t$time = int $time / 1000000000;\n+} else {\n+\tdie \"Unsupported query-fsmonitor hook version '$version'.\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+}\n+\n+# Convert unix style paths to escaped Windows style paths when running\n+# in Windows command prompt\n+\n+my $system = `uname -s`;\n+$system =~ s/[\\r\\n]+//g;\n+my $git_work_tree;\n+\n+if ($system =~ m/^MSYS_NT/) {\n+\t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n+\t$git_work_tree =~ s/[\\r\\n]+//g;\n+\t$git_work_tree =~ s,\\\\,/,g;\n+} else {\n+\t$git_work_tree = $ENV{'PWD'};\n+}\n+\n+my $retry = 1;\n+\n+launch_watchman();\n+\n+sub launch_watchman {\n+\n+\t# Set input record separator\n+\tlocal $/ = 0666;\n+\n+\tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j')\n+\t or die \"open2() failed: $!\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+\n+\t# In the query expression below we're asking for names of files that\n+\t# changed since $time but were not transient (ie created after\n+\t# $time but no longer exist).\n+\t#\n+\t# To accomplish this, we're using the \"since\" generator to use the\n+\t# recency index to select candidate nodes and \"fields\" to limit the\n+\t# output to file names only. Then we're using the \"expression\" term to\n+\t# further constrain the results.\n+\t#\n+\t# The category of transient files that we want to ignore will have a\n+\t# creation clock (cclock) newer than $time_t value and will also not\n+\t# currently exist.\n+\n+\tmy $query = <<\"\tEND\";\n+\t\t[\"query\", \"$git_work_tree\", {\n+\t\t\t\"since\": $time,\n+\t\t\t\"fields\": [\"name\"],\n+\t\t\t\"expression\": [\"not\", [\"allof\", [\"since\", $time, \"cclock\"], [\"not\", \"exists\"]]]\n+\t\t}]\n+\tEND\n+\t\n+\topen (my $fh, \">\", \".git/watchman-query.json\");\n+\tprint $fh $query;\n+\tclose $fh;\n+\n+\tprint CHLD_IN $query;\n+\tmy $response = ;\n+\n+\topen ($fh, \">\", \".git/watchman-response.json\");\n+\tprint $fh $response;\n+\tclose $fh;\n+\n+\tdie \"Watchman: command returned no output.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $response eq \"\";\n+\tdie \"Watchman: command returned invalid output: $response\\n\" .\n+\t \"Falling back to scanning...\\n\" unless $response =~ /^\\{/;\n+\n+\tmy $json_pkg;\n+\teval {\n+\t\trequire JSON::XS;\n+\t\t$json_pkg = \"JSON::XS\";\n+\t\t1;\n+\t} or do {\n+\t\trequire JSON::PP;\n+\t\t$json_pkg = \"JSON::PP\";\n+\t};\n+\n+\tmy $o = $json_pkg->new->utf8->decode($response);\n+\n+\tif ($retry > 0 and $o->{error} and $o->{error} =~ m/unable to resolve root .* directory (.*) is not watched/) {\n+\t\tprint STDERR \"Adding '$git_work_tree' to watchman's watch list.\\n\";\n+\t\t$retry--;\n+\t\tqx/watchman watch \"$git_work_tree\"/;\n+\t\tdie \"Failed to make watchman watch '$git_work_tree'.\\n\" .\n+\t\t \"Falling back to scanning...\\n\" if $? != 0;\n+\n+\t\t# Watchman will always return all files on the first query so\n+\t\t# return the fast \"everything is dirty\" flag to git and do the\n+\t\t# Watchman query just to get it over with now so we won't pay\n+\t\t# the cost in git to look up each individual file.\n+\n+\t\topen ($fh, \">\", \".git/watchman-output.out\");\n+\t\tprint \"/\\0\";\n+\t\tclose $fh;\n+\n+\t\tprint \"/\\0\";\n+\t\teval { launch_watchman() };\n+\t\texit 0;\n+\t}\n+\n+\tdie \"Watchman: $o->{error}.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $o->{error};\n+\n+\topen ($fh, \">\", \".git/watchman-output.out\");\n+\tprint $fh @{$o->{files}};\n+\tclose $fh;\n+\n+\tbinmode STDOUT, \":utf8\";\n+\tlocal $, = \"\\0\";\n+\tprint @{$o->{files}};\n+}\n-- \n2.14.1.windows.1\n\n"},{"id":"328409","messageId":"20170919192744.19224-4-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 03/12] update-index: add a new --force-write-index option","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:35Z","receivedAt":"2017-09-19T19:29:04Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"At times, it makes sense to avoid the cost of writing out the index\nwhen the only changes can easily be recomputed on demand. This causes\nproblems when trying to write test cases to verify that state as they\ncan't guarantee the state has been persisted to disk.\n\nAdd a new option (--force-write-index) to update-index that will\nensure the index is written out even if the cache_changed flag is not\nset.\n\nSigned-off-by: Ben Peart \n---\n builtin/update-index.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex d562f2ec69..e1ca0759d5 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -915,6 +915,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tstruct refresh_params refresh_args = {0, &has_errors};\n \tint lock_error = 0;\n \tint split_index = -1;\n+\tint force_write = 0;\n \tstruct lock_file *lock_file;\n \tstruct parse_opt_ctx_t ctx;\n \tstrbuf_getline_fn getline_fn;\n@@ -1006,6 +1007,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"test if the filesystem supports untracked cache\"), UC_TEST),\n \t\tOPT_SET_INT(0, \"force-untracked-cache\", &untracked_cache,\n \t\t\t N_(\"enable untracked cache without testing the filesystem\"), UC_FORCE),\n+\t\tOPT_SET_INT(0, \"force-write-index\", &force_write,\n+\t\t\tN_(\"write out the index even if is not flagged as changed\"), 1),\n \t\tOPT_END()\n \t};\n \n@@ -1147,7 +1150,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n \t}\n \n-\tif (active_cache_changed) {\n+\tif (active_cache_changed || force_write) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n \t\t\t\texit(128);\n-- \n2.14.1.windows.1\n\n"},{"id":"328410","messageId":"20170919192744.19224-13-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 12/12] fsmonitor: add a performance test","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:44Z","receivedAt":"2017-09-19T19:29:07Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a test utility (test-drop-caches) that flushes all changes to disk\nthen drops file system cache on Windows, Linux, and OSX.\n\nAdd a perf test (p7519-fsmonitor.sh) for fsmonitor.\n\nBy default, the performance test will utilize the Watchman file system\nmonitor if it is installed. If Watchman is not installed, it will use a\ndummy integration script that does not report any new or modified files.\nThe dummy script has very little overhead which provides optimistic results.\n\nThe performance test will also use the untracked cache feature if it is\navailable as fsmonitor uses it to speed up scanning for untracked files.\n\nThere are 3 environment variables that can be used to alter the default\nbehavior of the performance test:\n\nGIT_PERF_7519_UNTRACKED_CACHE: used to configure core.untrackedCache\nGIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\nGIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor\n\nThe big win for using fsmonitor is the elimination of the need to scan the\nworking directory looking for changed and untracked files. If the file\ninformation is all cached in RAM, the benefits are reduced.\n\nGIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests\n\nSigned-off-by: Ben Peart \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 1 +\n t/helper/.gitignore | 1 +\n t/helper/test-drop-caches.c | 162 ++++++++++++++++++++++++++++++++++++++\n t/perf/p7519-fsmonitor.sh | 184 ++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 348 insertions(+)\n create mode 100644 t/helper/test-drop-caches.c\n create mode 100755 t/perf/p7519-fsmonitor.sh\n\ndiff --git a/Makefile b/Makefile\nindex d970cd00e9..b2653ee64f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -638,6 +638,7 @@ TEST_PROGRAMS_NEED_X += test-ctype\n TEST_PROGRAMS_NEED_X += test-config\n TEST_PROGRAMS_NEED_X += test-date\n TEST_PROGRAMS_NEED_X += test-delta\n+TEST_PROGRAMS_NEED_X += test-drop-caches\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n TEST_PROGRAMS_NEED_X += test-dump-fsmonitor\n TEST_PROGRAMS_NEED_X += test-dump-split-index\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 721650256e..f9328eebdd 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -3,6 +3,7 @@\n /test-config\n /test-date\n /test-delta\n+/test-drop-caches\n /test-dump-cache-tree\n /test-dump-split-index\n /test-dump-untracked-cache\ndiff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\nnew file mode 100644\nindex 0000000000..4e5ca8f397\n--- /dev/null\n+++ b/t/helper/test-drop-caches.c\n@@ -0,0 +1,162 @@\n+#include \"git-compat-util.h\"\n+\n+#if defined(GIT_WINDOWS_NATIVE)\n+\n+static int cmd_sync(void)\n+{\n+\tchar Buffer[MAX_PATH];\n+\tDWORD dwRet;\n+\tchar szVolumeAccessPath[] = \"\\\\\\\\.\\\\X:\";\n+\tHANDLE hVolWrite;\n+\tint success = 0;\n+\n+\tdwRet = GetCurrentDirectory(MAX_PATH, Buffer);\n+\tif ((0 == dwRet) || (dwRet > MAX_PATH))\n+\t\treturn error(\"Error getting current directory\");\n+\n+\tif ((Buffer[0] < 'A') || (Buffer[0] > 'Z'))\n+\t\treturn error(\"Invalid drive letter '%c'\", Buffer[0]);\n+\n+\tszVolumeAccessPath[4] = Buffer[0];\n+\thVolWrite = CreateFile(szVolumeAccessPath, GENERIC_READ | GENERIC_WRITE,\n+\t\tFILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);\n+\tif (INVALID_HANDLE_VALUE == hVolWrite)\n+\t\treturn error(\"Unable to open volume for writing, need admin access\");\n+\n+\tsuccess = FlushFileBuffers(hVolWrite);\n+\tif (!success)\n+\t\terror(\"Unable to flush volume\");\n+\n+\tCloseHandle(hVolWrite);\n+\n+\treturn !success;\n+}\n+\n+#define STATUS_SUCCESS\t\t\t(0x00000000L)\n+#define STATUS_PRIVILEGE_NOT_HELD\t(0xC0000061L)\n+\n+typedef enum _SYSTEM_INFORMATION_CLASS {\n+\tSystemMemoryListInformation = 80,\n+} SYSTEM_INFORMATION_CLASS;\n+\n+typedef enum _SYSTEM_MEMORY_LIST_COMMAND {\n+\tMemoryCaptureAccessedBits,\n+\tMemoryCaptureAndResetAccessedBits,\n+\tMemoryEmptyWorkingSets,\n+\tMemoryFlushModifiedList,\n+\tMemoryPurgeStandbyList,\n+\tMemoryPurgeLowPriorityStandbyList,\n+\tMemoryCommandMax\n+} SYSTEM_MEMORY_LIST_COMMAND;\n+\n+static BOOL GetPrivilege(HANDLE TokenHandle, LPCSTR lpName, int flags)\n+{\n+\tBOOL bResult;\n+\tDWORD dwBufferLength;\n+\tLUID luid;\n+\tTOKEN_PRIVILEGES tpPreviousState;\n+\tTOKEN_PRIVILEGES tpNewState;\n+\n+\tdwBufferLength = 16;\n+\tbResult = LookupPrivilegeValueA(0, lpName, &luid);\n+\tif (bResult) {\n+\t\ttpNewState.PrivilegeCount = 1;\n+\t\ttpNewState.Privileges[0].Luid = luid;\n+\t\ttpNewState.Privileges[0].Attributes = 0;\n+\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpNewState,\n+\t\t\t(DWORD)((LPBYTE)&(tpNewState.Privileges[1]) - (LPBYTE)&tpNewState),\n+\t\t\t&tpPreviousState, &dwBufferLength);\n+\t\tif (bResult) {\n+\t\t\ttpPreviousState.PrivilegeCount = 1;\n+\t\t\ttpPreviousState.Privileges[0].Luid = luid;\n+\t\t\ttpPreviousState.Privileges[0].Attributes = flags != 0 ? 2 : 0;\n+\t\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpPreviousState,\n+\t\t\t\tdwBufferLength, 0, 0);\n+\t\t}\n+\t}\n+\treturn bResult;\n+}\n+\n+static int cmd_dropcaches(void)\n+{\n+\tHANDLE hProcess = GetCurrentProcess();\n+\tHANDLE hToken;\n+\tHMODULE ntdll;\n+\tint status;\n+\n+\tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n+\t\treturn error(\"Can't open current process token\");\n+\n+\tif (!GetPrivilege(hToken, \"SeProfileSingleProcessPrivilege\", 1))\n+\t\treturn error(\"Can't get SeProfileSingleProcessPrivilege\");\n+\n+\tCloseHandle(hToken);\n+\n+\tntdll = LoadLibrary(\"ntdll.dll\");\n+\tif (!ntdll)\n+\t\treturn error(\"Can't load ntdll.dll, wrong Windows version?\");\n+\n+\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =\n+\t\t(DWORD(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n+\tif (!NtSetSystemInformation)\n+\t\treturn error(\"Can't get function addresses, wrong Windows version?\");\n+\n+\tSYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;\n+\tstatus = NtSetSystemInformation(\n+\t\tSystemMemoryListInformation,\n+\t\t&command,\n+\t\tsizeof(SYSTEM_MEMORY_LIST_COMMAND)\n+\t);\n+\tif (status == STATUS_PRIVILEGE_NOT_HELD)\n+\t\terror(\"Insufficient privileges to purge the standby list, need admin access\");\n+\telse if (status != STATUS_SUCCESS)\n+\t\terror(\"Unable to execute the memory list command %d\", status);\n+\n+\tFreeLibrary(ntdll);\n+\n+\treturn status;\n+}\n+\n+#elif defined(__linux__)\n+\n+static int cmd_sync(void)\n+{\n+\treturn system(\"sync\");\n+}\n+\n+static int cmd_dropcaches(void)\n+{\n+\treturn system(\"echo 3 | sudo tee /proc/sys/vm/drop_caches\");\n+}\n+\n+#elif defined(__APPLE__)\n+\n+static int cmd_sync(void)\n+{\n+\treturn system(\"sync\");\n+}\n+\n+static int cmd_dropcaches(void)\n+{\n+\treturn system(\"sudo purge\");\n+}\n+\n+#else\n+\n+static int cmd_sync(void)\n+{\n+\treturn 0;\n+}\n+\n+static int cmd_dropcaches(void)\n+{\n+\treturn error(\"drop caches not implemented on this platform\");\n+}\n+\n+#endif\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tcmd_sync();\n+\treturn cmd_dropcaches();\n+}\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nnew file mode 100755\nindex 0000000000..16d1bf72e5\n--- /dev/null\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -0,0 +1,184 @@\n+#!/bin/sh\n+\n+test_description=\"Test core.fsmonitor\"\n+\n+. ./perf-lib.sh\n+\n+#\n+# Performance test for the fsmonitor feature which enables git to talk to a\n+# file system change monitor and avoid having to scan the working directory\n+# for new or modified files.\n+#\n+# By default, the performance test will utilize the Watchman file system\n+# monitor if it is installed. If Watchman is not installed, it will use a\n+# dummy integration script that does not report any new or modified files.\n+# The dummy script has very little overhead which provides optimistic results.\n+#\n+# The performance test will also use the untracked cache feature if it is\n+# available as fsmonitor uses it to speed up scanning for untracked files.\n+#\n+# There are 3 environment variables that can be used to alter the default\n+# behavior of the performance test:\n+#\n+# GIT_PERF_7519_UNTRACKED_CACHE: used to configure core.untrackedCache\n+# GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\n+# GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor\n+#\n+# The big win for using fsmonitor is the elimination of the need to scan the\n+# working directory looking for changed and untracked files. If the file\n+# information is all cached in RAM, the benefits are reduced.\n+#\n+# GIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests\n+#\n+\n+test_perf_large_repo\n+test_checkout_worktree\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_lazy_prereq WATCHMAN '\n+\t{ command -v watchman >/dev/null 2>&1; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+if test_have_prereq WATCHMAN\n+then\n+\t# Convert unix style paths to escaped Windows style paths for Watchman\n+\tcase \"$(uname -s)\" in\n+\tMSYS_NT*)\n+\t GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,/,g')\"\n+\t ;;\n+\t*)\n+\t GIT_WORK_TREE=\"$PWD\"\n+\t ;;\n+\tesac\n+fi\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"\n+then\n+\t# When using GIT_PERF_7519_DROP_CACHE, GIT_PERF_REPEAT_COUNT must be 1 to\n+\t# generate valid results. Otherwise the caching that happens for the nth\n+\t# run will negate the validity of the comparisons.\n+\tif test \"$GIT_PERF_REPEAT_COUNT\" -ne 1\n+\tthen\n+\t\techo \"warning: Setting GIT_PERF_REPEAT_COUNT=1\" >&2\n+\t\tGIT_PERF_REPEAT_COUNT=1\n+\tfi\n+fi\n+\n+test_expect_success \"setup for fsmonitor\" '\n+\t# set untrackedCache depending on the environment\n+\tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n+\tthen\n+\t\tgit config core.untrackedCache \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n+\telse\n+\t\tif test_have_prereq UNTRACKED_CACHE\n+\t\tthen\n+\t\t\tgit config core.untrackedCache true\n+\t\telse\n+\t\t\tgit config core.untrackedCache false\n+\t\tfi\n+\tfi &&\n+\n+\t# set core.splitindex depending on the environment\n+\tif test -n \"$GIT_PERF_7519_SPLIT_INDEX\"\n+\tthen\n+\t\tgit config core.splitIndex \"$GIT_PERF_7519_SPLIT_INDEX\"\n+\tfi &&\n+\n+\t# set INTEGRATION_SCRIPT depending on the environment\n+\tif test -n \"$GIT_PERF_7519_FSMONITOR\"\n+\tthen\n+\t\tINTEGRATION_SCRIPT=\"$GIT_PERF_7519_FSMONITOR\"\n+\telse\n+\t\t#\n+\t\t# Choose integration script based on existence of Watchman.\n+\t\t# If Watchman exists, watch the work tree and attempt a query.\n+\t\t# If everything succeeds, use Watchman integration script,\n+\t\t# else fall back to an empty integration script.\n+\t\t#\n+\t\tmkdir .git/hooks &&\n+\t\tif test_have_prereq WATCHMAN\n+\t\tthen\n+\t\t\tINTEGRATION_SCRIPT=\".git/hooks/fsmonitor-watchman\" &&\n+\t\t\tcp \"$TEST_DIRECTORY/../templates/hooks--fsmonitor-watchman.sample\" \"$INTEGRATION_SCRIPT\" &&\n+\t\t\twatchman watch \"$GIT_WORK_TREE\" &&\n+\t\t\twatchman watch-list | grep -q -F \"$GIT_WORK_TREE\"\n+\t\telse\n+\t\t\tINTEGRATION_SCRIPT=\".git/hooks/fsmonitor-empty\" &&\n+\t\t\twrite_script \"$INTEGRATION_SCRIPT\"<<-\\EOF\n+\t\t\tEOF\n+\t\tfi\n+\tfi &&\n+\n+\tgit config core.fsmonitor \"$INTEGRATION_SCRIPT\" &&\n+\tgit update-index --fsmonitor\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uno (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uno\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uall (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uall\n+'\n+\n+test_expect_success \"setup without fsmonitor\" '\n+\tunset INTEGRATION_SCRIPT &&\n+\tgit config --unset core.fsmonitor &&\n+\tgit update-index --no-fsmonitor\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uno (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uno\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uall (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uall\n+'\n+\n+if test_have_prereq WATCHMAN\n+then\n+\twatchman watch-del \"$GIT_WORK_TREE\" >/dev/null 2>&1 &&\n+\n+\t# Work around Watchman bug on Windows where it holds on to handles\n+\t# preventing the removal of the trash directory\n+\twatchman shutdown-server >/dev/null 2>&1\n+fi\n+\n+test_done\n-- \n2.14.1.windows.1\n\n"},{"id":"328411","messageId":"20170919192744.19224-12-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 11/12] fsmonitor: add a sample integration script for Watchman","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:43Z","receivedAt":"2017-09-19T19:29:09Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This script integrates the new fsmonitor capabilities of git with the\ncross platform Watchman file watching service. To use the script:\n\nDownload and install Watchman from https://facebook.github.io/watchman/.\nRename the sample integration hook from fsmonitor-watchman.sample to\nfsmonitor-watchman. Configure git to use the extension:\n\ngit config core.fsmonitor .git/hooks/fsmonitor-watchman\n\nOptionally turn on the untracked cache for optimal performance.\n\nSigned-off-by: Ben Peart \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Christian Couder \n---\n templates/hooks--fsmonitor-watchman.sample | 122 +++++++++++++++++++++++++++++\n 1 file changed, 122 insertions(+)\n create mode 100755 templates/hooks--fsmonitor-watchman.sample\n\ndiff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample\nnew file mode 100755\nindex 0000000000..870a59d237\n--- /dev/null\n+++ b/templates/hooks--fsmonitor-watchman.sample\n@@ -0,0 +1,122 @@\n+#!/usr/bin/perl\n+\n+use strict;\n+use warnings;\n+use IPC::Open2;\n+\n+# An example hook script to integrate Watchman\n+# (https://facebook.github.io/watchman/) with git to speed up detecting\n+# new and modified files.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+# To enable this hook, rename this file to \"query-watchman\" and set\n+# 'git config core.fsmonitor .git/hooks/query-watchman'\n+#\n+my ($version, $time) = @ARGV;\n+\n+# Check the hook interface version\n+\n+if ($version == 1) {\n+\t# convert nanoseconds to seconds\n+\t$time = int $time / 1000000000;\n+} else {\n+\tdie \"Unsupported query-fsmonitor hook version '$version'.\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+}\n+\n+# Convert unix style paths to escaped Windows style paths when running\n+# in Windows command prompt\n+\n+my $system = `uname -s`;\n+$system =~ s/[\\r\\n]+//g;\n+my $git_work_tree;\n+\n+if ($system =~ m/^MSYS_NT/) {\n+\t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n+\t$git_work_tree =~ s/[\\r\\n]+//g;\n+\t$git_work_tree =~ s,\\\\,/,g;\n+} else {\n+\t$git_work_tree = $ENV{'PWD'};\n+}\n+\n+my $retry = 1;\n+\n+launch_watchman();\n+\n+sub launch_watchman {\n+\n+\t# Set input record separator\n+\tlocal $/ = 0666;\n+\n+\tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j')\n+\t or die \"open2() failed: $!\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+\n+\t# In the query expression below we're asking for names of files that\n+\t# changed since $time but were not transient (ie created after\n+\t# $time but no longer exist).\n+\t#\n+\t# To accomplish this, we're using the \"since\" generator to use the\n+\t# recency index to select candidate nodes and \"fields\" to limit the\n+\t# output to file names only. Then we're using the \"expression\" term to\n+\t# further constrain the results.\n+\t#\n+\t# The category of transient files that we want to ignore will have a\n+\t# creation clock (cclock) newer than $time_t value and will also not\n+\t# currently exist.\n+\n+\tmy $query = <<\"\tEND\";\n+\t\t[\"query\", \"$git_work_tree\", {\n+\t\t\t\"since\": $time,\n+\t\t\t\"fields\": [\"name\"],\n+\t\t\t\"expression\": [\"not\", [\"allof\", [\"since\", $time, \"cclock\"], [\"not\", \"exists\"]]]\n+\t\t}]\n+\tEND\n+\n+\tprint CHLD_IN $query;\n+\tmy $response = ;\n+\n+\tdie \"Watchman: command returned no output.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $response eq \"\";\n+\tdie \"Watchman: command returned invalid output: $response\\n\" .\n+\t \"Falling back to scanning...\\n\" unless $response =~ /^\\{/;\n+\n+\tmy $json_pkg;\n+\teval {\n+\t\trequire JSON::XS;\n+\t\t$json_pkg = \"JSON::XS\";\n+\t\t1;\n+\t} or do {\n+\t\trequire JSON::PP;\n+\t\t$json_pkg = \"JSON::PP\";\n+\t};\n+\n+\tmy $o = $json_pkg->new->utf8->decode($response);\n+\n+\tif ($retry > 0 and $o->{error} and $o->{error} =~ m/unable to resolve root .* directory (.*) is not watched/) {\n+\t\tprint STDERR \"Adding '$git_work_tree' to watchman's watch list.\\n\";\n+\t\t$retry--;\n+\t\tqx/watchman watch \"$git_work_tree\"/;\n+\t\tdie \"Failed to make watchman watch '$git_work_tree'.\\n\" .\n+\t\t \"Falling back to scanning...\\n\" if $? != 0;\n+\n+\t\t# Watchman will always return all files on the first query so\n+\t\t# return the fast \"everything is dirty\" flag to git and do the\n+\t\t# Watchman query just to get it over with now so we won't pay\n+\t\t# the cost in git to look up each individual file.\n+\t\tprint \"/\\0\";\n+\t\teval { launch_watchman() };\n+\t\texit 0;\n+\t}\n+\n+\tdie \"Watchman: $o->{error}.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $o->{error};\n+\n+\tbinmode STDOUT, \":utf8\";\n+\tlocal $, = \"\\0\";\n+\tprint @{$o->{files}};\n+}\n-- \n2.14.1.windows.1\n\n"},{"id":"328412","messageId":"20170919192744.19224-7-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v7 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-19T19:27:38Z","receivedAt":"2017-09-19T19:29:12Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a new command line option (-f) to ls-files to have it use lowercase\nletters for 'fsmonitor valid' files\n\nSigned-off-by: Ben Peart \n---\n builtin/ls-files.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex e1339e6d17..313962a0c1 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -31,6 +31,7 @@ static int show_resolve_undo;\n static int show_modified;\n static int show_killed;\n static int show_valid_bit;\n+static int show_fsmonitor_bit;\n static int line_terminator = '\\n';\n static int debug_mode;\n static int show_eol;\n@@ -86,7 +87,8 @@ static const char *get_tag(const struct cache_entry *ce, const char *tag)\n {\n \tstatic char alttag[4];\n \n-\tif (tag && *tag && show_valid_bit && (ce->ce_flags & CE_VALID)) {\n+\tif (tag && *tag && ((show_valid_bit && (ce->ce_flags & CE_VALID)) ||\n+\t\t(show_fsmonitor_bit && (ce->ce_flags & CE_FSMONITOR_VALID)))) {\n \t\tmemcpy(alttag, tag, 3);\n \n \t\tif (isalpha(tag[0])) {\n@@ -515,6 +517,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\t\tN_(\"identify the file status with tags\")),\n \t\tOPT_BOOL('v', NULL, &show_valid_bit,\n \t\t\tN_(\"use lowercase letters for 'assume unchanged' files\")),\n+\t\tOPT_BOOL('f', NULL, &show_fsmonitor_bit,\n+\t\t\tN_(\"use lowercase letters for 'fsmonitor clean' files\")),\n \t\tOPT_BOOL('c', \"cached\", &show_cached,\n \t\t\tN_(\"show cached files in the output (default)\")),\n \t\tOPT_BOOL('d', \"deleted\", &show_deleted,\n@@ -584,7 +588,7 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tfor (i = 0; i < exclude_list.nr; i++) {\n \t\tadd_exclude(exclude_list.items[i].string, \"\", 0, el, --exclude_args);\n \t}\n-\tif (show_tag || show_valid_bit) {\n+\tif (show_tag || show_valid_bit || show_fsmonitor_bit) {\n \t\ttag_cached = \"H \";\n \t\ttag_unmerged = \"M \";\n \t\ttag_removed = \"R \";\n-- \n2.14.1.windows.1\n\n"},{"id":"328413","messageId":"2f3593f790f5423e853f3bcb0a601ef9@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"dc1506689d3d4639b56f0489e4219054@exmbdft7.ad.twosigma.com","subject":"RE: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-09-19T19:32:44Z","receivedAt":"2017-09-19T19:32:50Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"I think my comment here might have gotten lost, and I don't want it to because it's something I'm really worried about:\n\n> -----Original Message-----\n> From: David Turner\n> Sent: Friday, September 15, 2017 6:00 PM\n> To: 'Ben Peart' \n> Cc: avarab@gmail.com; christian.couder@gmail.com; git@vger.kernel.org;\n> gitster@pobox.com; johannes.schindelin@gmx.de; pclouds@gmail.com;\n> peff@peff.net\n> Subject: RE: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor\n> extension\n> \n> > -----Original Message-----\n> > +dirty_repo () {\n> > +\t: >untracked &&\n> > +\t: >dir1/untracked &&\n> > +\t: >dir2/untracked &&\n> > +\techo 1 >modified &&\n> > +\techo 2 >dir1/modified &&\n> > +\techo 3 >dir2/modified &&\n> > +\techo 4 >new &&\n> > +\techo 5 >dir1/new &&\n> > +\techo 6 >dir2/new\n> \n> If I add an untracked file named dir3/untracked to dirty_repo (and\n> write_integration_script), then \"status doesn't detect unreported\n> modifications\", below, fails. Did I do something wrong, or does this turn up a\n> bug?\n\n\n"},{"id":"328417","messageId":"37b2e5b4ca1d490b9f50d430b3b420f0@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"20170919192744.19224-7-benpeart@microsoft.com","subject":"RE: [PATCH v7 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-09-19T19:46:17Z","receivedAt":"2017-09-19T19:46:23Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"> -----Original Message-----\n> From: Ben Peart [mailto:benpeart@microsoft.com]\n> Sent: Tuesday, September 19, 2017 3:28 PM\n> To: benpeart@microsoft.com\n> Cc: David Turner ; avarab@gmail.com;\n> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> Subject: [PATCH v7 06/12] ls-files: Add support in ls-files to display the\n> fsmonitor valid bit\n> \n> Add a new command line option (-f) to ls-files to have it use lowercase\n> letters for 'fsmonitor valid' files\n> \n> Signed-off-by: Ben Peart \n> ---\n> builtin/ls-files.c | 8 ++++++--\n> 1 file changed, 6 insertions(+), 2 deletions(-)\n\nThis is still missing the corresponding documentation patch. \n\nI can see from replies that at least some of my messages got through. In total, I sent messages about:\n04/12 (I see replies)\n05/12 (I see replies)\n06/12 (no reply, issue not fixed)\n10/12 (no reply, haven't checked whether same issue but I assume same issue since the new case I mentioned isn't added)\n12/12 (no reply, typo fixed -- no reply required)\n\n"},{"id":"328420","messageId":"84981984-02c1-f322-a617-57dfe1d87ad2@gmail.com","threadId":"46154","inReplyTo":"2f3593f790f5423e853f3bcb0a601ef9@exmbdft7.ad.twosigma.com","subject":"Re: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-19T20:30:37Z","receivedAt":"2017-09-19T20:30:46Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/19/2017 3:32 PM, David Turner wrote:\n> I think my comment here might have gotten lost, and I don't want it to because it's something I'm really worried about:\n> \n\nYou have to update the test completely to ensure it passes. If you run \nthe test with the '-v' option you will see the cause of the failure:\n\nt7519-status-fsmonitor.sh: line 27: dir3/untracked: No such file or \ndirectory\n\nTo fix this, you will also need to update the 'setup' test to create the \ndirectory for the new untracked file to get created into. Then you will \nneed to drop at least one file in it so that the directory is preserved \nthrough the 'git reset --hard' Then you have to update the several 'cat \n >expect' blocks to expect the new file.\n\nIn addition, the ability to avoid scanning for untracked files relies on \nthe untracked cache. If you don't have another file that git is aware \nof in that directory then there won't be a cache entry and git will do \nthe required scan and detect the new untracked file (by design).\n\nHere is a patch to the test that updates it to meet all these \nrequirements. I hope this helps.\n\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex c6df85af5e..29ae4e284f 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -24,12 +24,14 @@ dirty_repo () {\n \t: >untracked &&\n \t: >dir1/untracked &&\n \t: >dir2/untracked &&\n+\t: >dir3/untracked &&\n \techo 1 >modified &&\n \techo 2 >dir1/modified &&\n \techo 3 >dir2/modified &&\n \techo 4 >new &&\n \techo 5 >dir1/new &&\n \techo 6 >dir2/new\n+\techo 7 >dir3/new\n }\n\n write_integration_script () {\n@@ -47,12 +49,14 @@ write_integration_script () {\n \tprintf \"untracked\\0\"\n \tprintf \"dir1/untracked\\0\"\n \tprintf \"dir2/untracked\\0\"\n+\tprintf \"dir3/untracked\\0\"\n \tprintf \"modified\\0\"\n \tprintf \"dir1/modified\\0\"\n \tprintf \"dir2/modified\\0\"\n \tprintf \"new\\0\"\n \tprintf \"dir1/new\\0\"\n \tprintf \"dir2/new\\0\"\n+\tprintf \"dir3/new\\0\"\n \tEOF\n }\n\n@@ -71,6 +75,8 @@ test_expect_success 'setup' '\n \tmkdir dir2 &&\n \t: >dir2/tracked &&\n \t: >dir2/modified &&\n+\tmkdir dir3 &&\n+\t: >dir3/tracked &&\n \tgit -c core.fsmonitor= add . &&\n \tgit -c core.fsmonitor= commit -m initial &&\n \tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n@@ -107,6 +113,7 @@ h dir1/modified\n H dir1/tracked\n h dir2/modified\n H dir2/tracked\n+H dir3/tracked\n h modified\n H tracked\n EOF\n@@ -126,6 +133,7 @@ H dir1/modified\n H dir1/tracked\n H dir2/modified\n H dir2/tracked\n+H dir3/tracked\n H modified\n H tracked\n EOF\n@@ -144,6 +152,7 @@ H dir1/modified\n H dir1/tracked\n H dir2/modified\n H dir2/tracked\n+H dir3/tracked\n H modified\n H tracked\n EOF\n@@ -164,6 +173,8 @@ H dir1/tracked\n H dir2/modified\n h dir2/new\n H dir2/tracked\n+h dir3/new\n+H dir3/tracked\n H modified\n h new\n H tracked\n@@ -174,6 +185,7 @@ test_expect_success 'newly added files are marked \nvalid' '\n \tgit add new &&\n \tgit add dir1/new &&\n \tgit add dir2/new &&\n+\tgit add dir3/new &&\n \tgit ls-files -f >actual &&\n \ttest_cmp expect actual\n '\n@@ -185,6 +197,8 @@ h dir1/tracked\n H dir2/modified\n h dir2/new\n h dir2/tracked\n+h dir3/new\n+h dir3/tracked\n H modified\n h new\n h tracked\n@@ -203,6 +217,7 @@ H dir1/modified\n h dir1/tracked\n h dir2/modified\n h dir2/tracked\n+h dir3/tracked\n h modified\n h tracked\n EOF\n@@ -269,6 +284,7 @@ do\n \t\t\tgit add new &&\n \t\t\tgit add dir1/new &&\n \t\t\tgit add dir2/new &&\n+\t\t\tgit add dir3/new &&\n \t\t\tgit status >actual &&\n \t\t\tgit -c core.fsmonitor= status >expect &&\n \t\t\ttest_i18ncmp expect actual\n\n\n\n>> -----Original Message-----\n>> From: David Turner\n>> Sent: Friday, September 15, 2017 6:00 PM\n>> To: 'Ben Peart' \n>> Cc: avarab@gmail.com; christian.couder@gmail.com; git@vger.kernel.org;\n>> gitster@pobox.com; johannes.schindelin@gmx.de; pclouds@gmail.com;\n>> peff@peff.net\n>> Subject: RE: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor\n>> extension\n>>\n>>> -----Original Message-----\n>>> +dirty_repo () {\n>>> +\t: >untracked &&\n>>> +\t: >dir1/untracked &&\n>>> +\t: >dir2/untracked &&\n>>> +\techo 1 >modified &&\n>>> +\techo 2 >dir1/modified &&\n>>> +\techo 3 >dir2/modified &&\n>>> +\techo 4 >new &&\n>>> +\techo 5 >dir1/new &&\n>>> +\techo 6 >dir2/new\n>>\n>> If I add an untracked file named dir3/untracked to dirty_repo (and\n>> write_integration_script), then \"status doesn't detect unreported\n>> modifications\", below, fails. Did I do something wrong, or does this turn up a\n>> bug?\n> \n> \n"},{"id":"328421","messageId":"20170919203408.GE75068@aiede.mtv.corp.google.com","threadId":"46154","inReplyTo":"2f3cdc85-f051-c0ae-b9db-fd13cac78aed@gmail.com","subject":"Re: [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-09-19T20:34:08Z","receivedAt":"2017-09-19T20:34:16Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ben Peart wrote:\n\n> Some stats on these same coding style errors in the current bash scripts:\n>\n> 298 instances of \"[a-z]\\(\\).*\\{\" ie \"function_name() {\" (no space)\n> 140 instances of \"if \\[ .* \\]\" (not using the preferred \"test\")\n> 293 instances of \"if .*; then\"\n>\n> Wouldn't it be great not to have to write up style feedback for when\n> these all get copy/pasted into new scripts?\n\nAgreed. Care to write patches for these? :) (I think three patches,\none for each issue, would do the trick.)\n\nThanks,\nJonathan\n"},{"id":"328422","messageId":"20170919203714.GF75068@aiede.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170917054318.28560-1-tboegi@web.de","subject":"Re: [PATCH v1 1/1] test-lint: echo -e (or -E) is not portable","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-09-19T20:37:14Z","receivedAt":"2017-09-19T20:37:22Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Torsten Bögershausen wrote:\n\n> Some implementations of `echo` support the '-e' option to enable\n> backslash interpretation of the following string.\n> As an addition, they support '-E' to turn it off.\n\nnit: please wrap the commit message to a consistent line width.\n\n> However, none of these are portable, POSIX doesn't even mention them,\n> and many implementations don't support them.\n>\n> A check for '-n' is already done in check-non-portable-shell.pl,\n> extend it to cover '-n', '-e' or '-E-'\n>\n> Signed-off-by: Torsten Bögershausen \n> ---\n> t/check-non-portable-shell.pl | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n\nAn excellent change. Thanks for noticing and fixing this.\n\nReviewed-by: Jonathan Nieder \n"},{"id":"328423","messageId":"20170919204354.GG75068@aiede.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170915192043.4516-10-benpeart@microsoft.com","subject":"Re: [PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-09-19T20:43:54Z","receivedAt":"2017-09-19T20:44:02Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBen Peart wrote:\n\n> The split index test t1700-split-index.sh has hard coded SHA values for\n> the index. Currently it supports index V4 and V3 but assumes there are\n> no index extensions loaded.\n>\n> When manually forcing the fsmonitor extension to be turned on when\n> running the test suite, the SHA values no longer match which causes the\n> test to fail.\n>\n> The potential matrix of index extensions and index versions can is quite\n> large so instead disable the extension before attempting to run the test.\n\nThanks for finding and diagnosing this problem.\n\nThis feels to me like the wrong fix. Wouldn't it be better for the\ntest not to depend on the precise object ids? See the \"Tips for\nWriting Tests\" section in t/README:\n\n\t And\n\tsuch drastic changes to the core GIT that even changes these\n\totherwise supposedly stable object IDs should be accompanied by\n\tan update to t0000-basic.sh.\n\n\tHowever, other tests that simply rely on basic parts of the core\n\tGIT working properly should not have that level of intimate\n\tknowledge of the core GIT internals. If all the test scripts\n\thardcoded the object IDs like t0000-basic.sh does, that defeats\n\tthe purpose of t0000-basic.sh, which is to isolate that level of\n\tvalidation in one place. Your test also ends up needing\n\tupdating when such a change to the internal happens, so do _not_\n\tdo it and leave the low level of validation to t0000-basic.sh.\n\nWorse, t1700-split-index.sh doesn't explain where the object ids it\nuses comes from so it is not even obvious to a casual reader like me\nhow to fix it.\n\nSee t/diff-lib.sh for some examples of one way to avoid depending on\nthe object id computation. Another way that is often preferable is to\ncome up with commands to compute the expected hash values, like\n$(git rev-parse HEAD^{tree}), and use those instead of hard-coded\nvalues.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"328424","messageId":"f1213833-be5e-4850-0ffa-d063e2d1deb8@gmail.com","threadId":"46154","inReplyTo":"37b2e5b4ca1d490b9f50d430b3b420f0@exmbdft7.ad.twosigma.com","subject":"Re: [PATCH v7 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-19T20:44:30Z","receivedAt":"2017-09-19T20:44:39Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/19/2017 3:46 PM, David Turner wrote:\n>> -----Original Message-----\n>> From: Ben Peart [mailto:benpeart@microsoft.com]\n>> Sent: Tuesday, September 19, 2017 3:28 PM\n>> To: benpeart@microsoft.com\n>> Cc: David Turner ; avarab@gmail.com;\n>> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n>> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n>> Subject: [PATCH v7 06/12] ls-files: Add support in ls-files to display the\n>> fsmonitor valid bit\n>>\n>> Add a new command line option (-f) to ls-files to have it use lowercase\n>> letters for 'fsmonitor valid' files\n>>\n>> Signed-off-by: Ben Peart \n>> ---\n>> builtin/ls-files.c | 8 ++++++--\n>> 1 file changed, 6 insertions(+), 2 deletions(-)\n> \n> This is still missing the corresponding documentation patch.\n\nSorry for the confusion.\n\nThe documentation is all in a patch together as they all have links to \neach other. You can find it here:\n\nhttps://public-inbox.org/git/20170919192744.19224-6-benpeart@microsoft.com/T/#u\n\n> \n> I can see from replies that at least some of my messages got through. In total, I sent messages about:\n> 04/12 (I see replies)\n> 05/12 (I see replies)\n> 06/12 (no reply, issue not fixed)\n\nThe documentation is all in a patch together as they all have links to \neach other. You can find it here:\n\nhttps://public-inbox.org/git/20170919192744.19224-6-benpeart@microsoft.com/T/#u\n\n> 10/12 (no reply, haven't checked whether same issue but I assume same issue since the new case I mentioned isn't added)\n\nIt wasn't a bug so I didn't \"fix\" it. I just sent an explanation and \npatch demonstrating why. You can find it here:\n\nhttps://public-inbox.org/git/84981984-02c1-f322-a617-57dfe1d87ad2@gmail.com/T/#u\n\n> 12/12 (no reply, typo fixed -- no reply required)\n> \n\nHope this helps.\n"},{"id":"328427","messageId":"34effd8a881e497d8f5bdd56acc2285d@exmbdft7.ad.twosigma.com","threadId":"46154","inReplyTo":"f1213833-be5e-4850-0ffa-d063e2d1deb8@gmail.com","subject":"RE: [PATCH v7 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit","fromName":"David Turner","fromEmail":"david.turner@twosigma.com","sentAt":"2017-09-19T21:27:01Z","receivedAt":"2017-09-19T21:27:07Z","isPatch":true,"sender":{"key":"david.turner@twosigma.com","avatar":null},"body":"\n> -----Original Message-----\n> From: Ben Peart [mailto:peartben@gmail.com]\n> Sent: Tuesday, September 19, 2017 4:45 PM\n> To: David Turner ; 'Ben Peart'\n> \n> Cc: avarab@gmail.com; christian.couder@gmail.com; git@vger.kernel.org;\n> gitster@pobox.com; johannes.schindelin@gmx.de; pclouds@gmail.com;\n> peff@peff.net\n> Subject: Re: [PATCH v7 06/12] ls-files: Add support in ls-files to display the\n> fsmonitor valid bit\n> \n> \n> \n> On 9/19/2017 3:46 PM, David Turner wrote:\n> >> -----Original Message-----\n> >> From: Ben Peart [mailto:benpeart@microsoft.com]\n> >> Sent: Tuesday, September 19, 2017 3:28 PM\n> >> To: benpeart@microsoft.com\n> >> Cc: David Turner ; avarab@gmail.com;\n> >> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n> >> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> >> Subject: [PATCH v7 06/12] ls-files: Add support in ls-files to\n> >> display the fsmonitor valid bit\n> >>\n> >> Add a new command line option (-f) to ls-files to have it use\n> >> lowercase letters for 'fsmonitor valid' files\n> >>\n> >> Signed-off-by: Ben Peart \n> >> ---\n> >> builtin/ls-files.c | 8 ++++++--\n> >> 1 file changed, 6 insertions(+), 2 deletions(-)\n> >\n> > This is still missing the corresponding documentation patch.\n> \n> Sorry for the confusion.\n\nThanks for following up.\n\n> > 10/12 (no reply, haven't checked whether same issue but I assume same\n> > issue since the new case I mentioned isn't added)\n> \n> It wasn't a bug so I didn't \"fix\" it. I just sent an explanation and patch\n> demonstrating why. You can find it here:\n\nI was concerned about the case of an untracked file inside a directory \nthat contains no tracked files. Your patch in this mail treats dir3 just \nlike dir1 and dir2. I think you ought to test the case of a dir with no \ntracked files.\n\nAfter more careful checking, it looks like this case does work, but it's \nstill worth testing.\n\n"},{"id":"328431","messageId":"MWHPR21MB0478B3F5367538567A5C20EEF4600@MWHPR21MB0478.namprd21.prod.outlook.com","threadId":"46154","inReplyTo":"34effd8a881e497d8f5bdd56acc2285d@exmbdft7.ad.twosigma.com","subject":"RE: [PATCH v7 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2017-09-19T22:44:19Z","receivedAt":"2017-09-19T22:44:28Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"> -----Original Message-----\n> From: David Turner [mailto:David.Turner@twosigma.com]\n> Sent: Tuesday, September 19, 2017 5:27 PM\n> To: 'Ben Peart' ; Ben Peart\n> \n> Cc: avarab@gmail.com; christian.couder@gmail.com; git@vger.kernel.org;\n> gitster@pobox.com; johannes.schindelin@gmx.de; pclouds@gmail.com;\n> peff@peff.net\n> Subject: RE: [PATCH v7 06/12] ls-files: Add support in ls-files to display the\n> fsmonitor valid bit\n> \n> \n> > -----Original Message-----\n> > From: Ben Peart [mailto:peartben@gmail.com]\n> > Sent: Tuesday, September 19, 2017 4:45 PM\n> > To: David Turner ; 'Ben Peart'\n> > \n> > Cc: avarab@gmail.com; christian.couder@gmail.com; git@vger.kernel.org;\n> > gitster@pobox.com; johannes.schindelin@gmx.de; pclouds@gmail.com;\n> > peff@peff.net\n> > Subject: Re: [PATCH v7 06/12] ls-files: Add support in ls-files to\n> > display the fsmonitor valid bit\n> >\n> >\n> >\n> > On 9/19/2017 3:46 PM, David Turner wrote:\n> > >> -----Original Message-----\n> > >> From: Ben Peart [mailto:benpeart@microsoft.com]\n> > >> Sent: Tuesday, September 19, 2017 3:28 PM\n> > >> To: benpeart@microsoft.com\n> > >> Cc: David Turner ; avarab@gmail.com;\n> > >> christian.couder@gmail.com; git@vger.kernel.org; gitster@pobox.com;\n> > >> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> > >> Subject: [PATCH v7 06/12] ls-files: Add support in ls-files to\n> > >> display the fsmonitor valid bit\n> > >>\n> > >> Add a new command line option (-f) to ls-files to have it use\n> > >> lowercase letters for 'fsmonitor valid' files\n> > >>\n> > >> Signed-off-by: Ben Peart \n> > >> ---\n> > >> builtin/ls-files.c | 8 ++++++--\n> > >> 1 file changed, 6 insertions(+), 2 deletions(-)\n> > >\n> > > This is still missing the corresponding documentation patch.\n> >\n> > Sorry for the confusion.\n> \n> Thanks for following up.\n> \n> > > 10/12 (no reply, haven't checked whether same issue but I assume\n> > > same issue since the new case I mentioned isn't added)\n> >\n> > It wasn't a bug so I didn't \"fix\" it. I just sent an explanation and\n> > patch demonstrating why. You can find it here:\n> \n> I was concerned about the case of an untracked file inside a directory that\n> contains no tracked files. Your patch in this mail treats dir3 just like dir1 and\n> dir2. I think you ought to test the case of a dir with no tracked files.\n> \n\nIn the case where there is an untracked file inside a directory that contains no tracked files, git will (as shown by the \"failing\" test) actually find the untracked file. This is the correct/expected behavior. The test failure is just indicating that the optimization of not searching that directory for untracked files was not able to occur (because there was no entry in the untracked cache for that directory).\n\n> After more careful checking, it looks like this case does work, but it's still\n> worth testing.\n\n"},{"id":"328438","messageId":"xmqqr2v2p0gn.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170919192744.19224-5-benpeart@microsoft.com","subject":"Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-20T02:28:56Z","receivedAt":"2017-09-20T02:29:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> +/* do stat comparison even if CE_FSMONITOR_VALID is true */\n> +#define CE_MATCH_IGNORE_FSMONITOR 0X20\n\nHmm, when should a programmer use this flag?\n\n> +int git_config_get_fsmonitor(void)\n> +{\n> +\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n> +\t\tcore_fsmonitor = getenv(\"GIT_FSMONITOR_TEST\");\n\nWill the environment be part of the published API, or is it a\nremnant of a useful tool for debugging while developing the feature?\n\nIf it is the former (and I'd say why not, even though \"git -c\ncore.fsmontor=...\" may be easy enough), we might want to rename it\nto replace _TEST with _PROGRAM or something and document it.\n\n> diff --git a/diff-lib.c b/diff-lib.c\n> index 2a52b07954..23c6d03ca9 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -12,6 +12,7 @@\n> #include \"refs.h\"\n> #include \"submodule.h\"\n> #include \"dir.h\"\n> +#include \"fsmonitor.h\"\n> \n> /*\n> * diff-files\n> @@ -228,6 +229,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \n> \t\tif (!changed && !dirty_submodule) {\n> \t\t\tce_mark_uptodate(ce);\n> +\t\t\tmark_fsmonitor_valid(ce);\n\nI notice all calls to mark_fsmonitor_valid(ce) always follow a call\nto ce_mark_uptodate(ce). Should the call to the former be made as\npart of the latter instead? Are there cases where we want to call\nce_mark_uptodate(ce) to mark the ce up-to-date, yet we do not want\nto call mark_fsmonitor_valid(ce)? How does a programmer tell when\ns/he wants to call ce_mark_uptodate(ce) if s/he also should call\nmark_fsmonitor_valid(ce)?\n\nTogether with \"when to pass IGNORE_FSMONITOR\" question, is there a\nsummary that guides new programmers to answer these questions in the\nnew documentation?\n\n> diff --git a/dir.h b/dir.h\n> index e3717055d1..fab8fc1561 100644\n> --- a/dir.h\n> +++ b/dir.h\n> @@ -139,6 +139,8 @@ struct untracked_cache {\n> \tint gitignore_invalidated;\n> \tint dir_invalidated;\n> \tint dir_opened;\n> +\t/* fsmonitor invalidation data */\n> +\tunsigned int use_fsmonitor : 1;\n\nThis makes it look like we will add a bit more fields in later\npatches that are about \"invalidation\" around fsmonitor, but it\nappears that such an addition never happens til the end of the\nseries. And use_fsmonitor boolean does not seem to be what the\ncomment says---it just tells us if fsmonitor is in use in the\noperation of the untracked cache, no?\n\n> diff --git a/entry.c b/entry.c\n> index cb291aa88b..5e6794f9fc 100644\n> --- a/entry.c\n> +++ b/entry.c\n> @@ -4,6 +4,7 @@\n> #include \"streaming.h\"\n> #include \"submodule.h\"\n> #include \"progress.h\"\n> +#include \"fsmonitor.h\"\n> \n> static void create_directories(const char *path, int path_len,\n> \t\t\t const struct checkout *state)\n> @@ -357,6 +358,7 @@ static int write_entry(struct cache_entry *ce,\n> \t\t\tlstat(ce->name, &st);\n> \t\tfill_stat_cache_info(ce, &st);\n> \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n> +\t\tmark_fsmonitor_invalid(state->istate, ce);\n> \t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n\nSimilar to \"how does mark_fsmonitor_valid() relate to\nce_mark_uptodate()?\" question earlier, mark_fsmonitor_invalid()\nseems to often appear in pairs with the update to cache_changed.\nAre there cases where we need CE_ENTRY_CHANGED bit added to the\nindex state yet we do not want to call mark_fsmonitor_invalid()? I\nam wondering if there should be a single helper function that lets\ncallers say \"I changed this ce in this istate this way. Please\nupdate CE_VALID, CE_UPDATE_IN_BASE and CE_FSMONITOR_VALID\naccordingly\".\n\nThat \"changed _this way_\" is deliberately vague in my question\nabove, because it is not yet clear to me when mark-valid and\nmark-invalid should and should not be called from the series.\n\n> +\t/* a fsmonitor process can return '*' to indicate all entries are invalid */\n> +\tif (query_success && query_result.buf[0] != '/') {\n> +\t\t/* Mark all entries returned by the monitor as dirty */\n\nThe comment talks about '*' and code checks if it is not '/'? The\nelse clause of this if/else handles the \"invalidate all\" case, so\nshouldn't we be comparing with '*' instead here?\n\nAh, fsmonitor-watchman section of the doc tells us to write the hook\nin a way to return slash, so the comment above the code is stale and\nthe comparison with '/' is what we want, I guess.\n"},{"id":"328452","messageId":"xmqq7ewtor9u.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170919192744.19224-4-benpeart@microsoft.com","subject":"Re: [PATCH v7 03/12] update-index: add a new --force-write-index option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-20T05:47:25Z","receivedAt":"2017-09-20T05:47:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> +\t\tOPT_SET_INT(0, \"force-write-index\", &force_write,\n> +\t\t\tN_(\"write out the index even if is not flagged as changed\"), 1),\n\nHmph. The only time this makes difference is when the code forgets\nto mark active_cache_changed even when it actually made a change to\nthe index, no? I do understand the wish to be able to observe what\n_would_ be written if such a bug did not exist in order to debug the\nother aspects of the change in this series, but at the same time I\nfear that we may end up sweeping the problem under the rug by\nrunning the tests with this option.\n\n> \t\tOPT_END()\n> \t};\n> \n> @@ -1147,7 +1150,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n> \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n> \t}\n> \n> -\tif (active_cache_changed) {\n> +\tif (active_cache_changed || force_write) {\n> \t\tif (newfd < 0) {\n> \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n> \t\t\t\texit(128);\n"},{"id":"328453","messageId":"xmqq377hopma.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170919192744.19224-5-benpeart@microsoft.com","subject":"Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-20T06:23:09Z","receivedAt":"2017-09-20T06:23:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> @@ -344,6 +346,7 @@ struct index_state {\n> \tstruct hashmap dir_hash;\n> \tunsigned char sha1[20];\n> \tstruct untracked_cache *untracked;\n> +\tuint64_t fsmonitor_last_update;\n\nThis field being zero has more significance than just \"we haven't\ngot any update yet\", right? The way I am reading the code is that\nsetting it 0 is a way to signal that fsmon has been inactivated. It\nalso made me wonder if add_fsmonitor() that silently returns without\ndoing anything when this field is already non-zero is a bug (in\nother words, I couldn't tell what the right answer would be to a\nquestion \"shouldn't the caller be avoiding duplicate calls?\").\n\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> new file mode 100644\n> index 0000000000..b8b2d88fe1\n> --- /dev/null\n> +++ b/fsmonitor.c\n> ...\n\nThis part was a pleasant read.\n\nThanks.\n"},{"id":"328460","messageId":"CAN0heSqFsm0BSUzufXjCKNv=JpRpUQzFA9GkKfA6nd_hPhW_qg@mail.gmail.com","threadId":"46154","inReplyTo":"20170919192744.19224-6-benpeart@microsoft.com","subject":"Re: [PATCH v7 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2017-09-20T10:00:04Z","receivedAt":"2017-09-20T10:00:12Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 19 September 2017 at 21:27, Ben Peart wrote:\n> diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\n> index e19eba62cd..95231dbfcb 100644\n> --- a/Documentation/git-update-index.txt\n> +++ b/Documentation/git-update-index.txt\n> @@ -16,9 +16,11 @@ SYNOPSIS\n> [--chmod=(+|-)x]\n> [--[no-]assume-unchanged]\n> [--[no-]skip-worktree]\n> + [--[no-]fsmonitor-valid]\n> [--ignore-submodules]\n> [--[no-]split-index]\n> [--[no-|test-|force-]untracked-cache]\n> + [--[no-]fsmonitor]\n> [--really-refresh] [--unresolve] [--again | -g]\n> [--info-only] [--index-info]\n> [-z] [--stdin] [--index-version ]\n> @@ -111,6 +113,12 @@ you will need to handle the situation manually.\n> set and unset the \"skip-worktree\" bit for the paths. See\n> section \"Skip-worktree bit\" below for more information.\n>\n> +--[no-]fsmonitor-valid::\n> + When one of these flags is specified, the object name recorded\n> + for the paths are not updated. Instead, these options\n> + set and unset the \"fsmonitor valid\" bit for the paths. See\n> + section \"File System Monitor\" below for more information.\n> +\n\nSo --no-foo does not undo --foo, but there are three values: --foo,\n--no-foo and . I find that unintuitive, but maybe it's\njust me. Maybe there are other such options in the codebase already. How\nabout --fsmonitor-valid=set, --fsmonitor-valid=unset, and\n--no-fsmonitor-valid (which would be the default, and which would forget\nany earlier --fsmonitor-valid=...)?\n\n> -g::\n> --again::\n> Runs 'git update-index' itself on the paths whose index\n> @@ -201,6 +209,15 @@ will remove the intended effect of the option.\n> `--untracked-cache` used to imply `--test-untracked-cache` but\n> this option would enable the extension unconditionally.\n>\n> +--fsmonitor::\n> +--no-fsmonitor::\n\nMaybe \"--[no-]fsmonitor\" for symmetry with how you've done it above and\nlater.\n\n> +When used in conjunction with the untracked cache, it can further improve\n> +performance by avoiding the cost of scaning the entire working directory\n> +looking for new files.\n\ns/scaning/scanning/\n\n> +If you want to enable (or disable) this feature, it is easier to use\n> +the `core.fsmonitor` configuration variable (see\n> +linkgit:git-config[1]) than using the `--fsmonitor` option to\n> +`git update-index` in each repository, especially if you want to do so\n> +across all repositories you use, because you can set the configuration\n> +variable to `true` (or `false`) in your `$HOME/.gitconfig` just once\n> +and have it affect all repositories you touch.\n\nThis is a mouthful. Maybe you could split it a little, perhaps like so:\n\n If you want to enable (or disable) this feature, you will probably\n want to use the `core.fsmonitor` configuration variable (see\n linkgit:git-config[1]). By setting it to `true` (or `false`) in your\n `$HOME/.gitconfig`, it will affect all repositories you touch. For a\n more fine-grained control, you can set it per repository, or use the\n `--fsmonitor` option with `git update-index` in each repository.\n\nThe part about $HOME/.gitconfig vs per-repo config is perhaps generic\nenough that it doesn't belong here. So it'd only be about config vs.\noption. Where to place the config item and what implications that has is\narguably orthogonal to knowing that the option exists and what it does.\n\nMartin\n"},{"id":"328466","messageId":"20170920134952.GA1898@tor.lan","threadId":"46154","inReplyTo":"20170919203714.GF75068@aiede.mtv.corp.google.com","subject":"Re: [PATCH v1 1/1] test-lint: echo -e (or -E) is not portable","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-09-20T13:49:52Z","receivedAt":"2017-09-20T13:50:04Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Tue, Sep 19, 2017 at 01:37:14PM -0700, Jonathan Nieder wrote:\n> Torsten Bögershausen wrote:\n> \n> > Some implementations of `echo` support the '-e' option to enable\n> > backslash interpretation of the following string.\n> > As an addition, they support '-E' to turn it off.\n> \n> nit: please wrap the commit message to a consistent line width.\n> \n> > However, none of these are portable, POSIX doesn't even mention them,\n> > and many implementations don't support them.\n> >\n> > A check for '-n' is already done in check-non-portable-shell.pl,\n> > extend it to cover '-n', '-e' or '-E-'\n> >\n> > Signed-off-by: Torsten Bögershausen \n> > ---\n> > t/check-non-portable-shell.pl | 2 +-\n> > 1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> An excellent change. Thanks for noticing and fixing this.\n> \n> Reviewed-by: Jonathan Nieder \n\nThanks for the review.\nJunio, if you wouldn't mind to squash that in, \nanother fix is needed as well(trailing '-' after '-E') :\n\ns/'-n', '-e' or '-E-'/'-n', '-e' or '-E'\n ^\n \n\n"},{"id":"328468","messageId":"682237f1-4747-5712-e95b-285379eb1b69@gmail.com","threadId":"46154","inReplyTo":"xmqq7ewtor9u.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v7 03/12] update-index: add a new --force-write-index option","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-20T14:58:11Z","receivedAt":"2017-09-20T14:58:19Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/20/2017 1:47 AM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> +\t\tOPT_SET_INT(0, \"force-write-index\", &force_write,\n>> +\t\t\tN_(\"write out the index even if is not flagged as changed\"), 1),\n> \n> Hmph. The only time this makes difference is when the code forgets\n> to mark active_cache_changed even when it actually made a change to\n> the index, no? I do understand the wish to be able to observe what\n> _would_ be written if such a bug did not exist in order to debug the\n> other aspects of the change in this series, but at the same time I\n> fear that we may end up sweeping the problem under the rug by\n> running the tests with this option.\n> \n\nThis is to enable a performance optimization I discovered while perf \ntesting the patch series. It enables us to do a lazy index write for \nfsmonitor detected changes but still always generate correct results.\n\nLets see how my ascii art skills do at describing this:\n\n1) Index marked dirty on every fsmonitor change:\nA---x---B---y---C\n\n2) Index *not* marked dirty on fsmonitor changes:\nA---x---B---x,y---C\n\nAssume the index is written and up-to-date at point A.\n\nIn scenario #1 above, the index is marked fsmonitor dirty every time the \nfsmonitor detects a file that has been modified. At point B, the \nfsmonitor integration script returns that file 'x' has been modified \nsince A, the index is marked dirty and then written to disk with a \nlast_update time of B. At point C, the script returns 'y' as the \nchanges since point B, the index is marked dirty and written to disk again.\n\nIn scenario #2, the index is *not* marked fsmonitor dirty when changed \nare detected. At point B, the script returns 'x' but the index is not \nflagged dirty nor written to disk. At point C, the script will return \n'x' and 'y' (since both have been changed since time 'A') and again the \nindex is not marked dirty nor written to disk.\n\nCorrect results are generated in both scenarios but in scenario 2, there \nwere 2 fewer index writes. In short, the changed files were accumulated \nas the cost of processing 2 files at point C (vs 1) has no measurable \ndifference in perf but the savings of two unnecessary index writes is \nsignificant (especially when the index gets large).\n\nThere is no real concern about accumulating too many changes as 1) the \nprocessing cost for additional modified files is fairly trivial and 2) \nthe index ends up getting written out pretty frequently anyway as files \nare added/removed/staged/etc which updates the fsmonitor_last_update time.\n\nThe challenge came when it was time to test that the changes to the \nindex were correct. Since they are lazily written by default, I needed \na way to force the write so that I could verify the index on disk was \ncorrect. Hence, this patch.\n\n\n>> \t\tOPT_END()\n>> \t};\n>> \n>> @@ -1147,7 +1150,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n>> \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n>> \t}\n>> \n>> -\tif (active_cache_changed) {\n>> +\tif (active_cache_changed || force_write) {\n>> \t\tif (newfd < 0) {\n>> \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n>> \t\t\t\texit(128);\n"},{"id":"328469","messageId":"3311de8b-f9df-07e0-6c5d-7f491e9bcaa8@gmail.com","threadId":"46154","inReplyTo":"xmqqr2v2p0gn.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-20T16:19:29Z","receivedAt":"2017-09-20T16:19:39Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/19/2017 10:28 PM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> +/* do stat comparison even if CE_FSMONITOR_VALID is true */\n>> +#define CE_MATCH_IGNORE_FSMONITOR 0X20\n> \n> Hmm, when should a programmer use this flag?\n> \n\nPretty much the same places you would also use CE_MATCH_IGNORE_VALID and \nCE_MATCH_IGNORE_SKIP_WORKTREE which serve the same role for those \nfeatures. That is generally when you are about to overwrite data so \nwant to be *really* sure you have what you think you have.\n\nThe other place I used it was in preload_index(). In that case, I didn't \nwant to trigger the call to refresh_fsmonitor() as preload_index() is \nabout trying to do a fast precompute of state for the bulk of the index \nentries but is not required for correctness as refresh_cache_ent() will \nensure any \"missed\" by preload_index() are up-to-date if/when that is \nneeded.\n\n>> +int git_config_get_fsmonitor(void)\n>> +{\n>> +\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n>> +\t\tcore_fsmonitor = getenv(\"GIT_FSMONITOR_TEST\");\n> \n> Will the environment be part of the published API, or is it a\n> remnant of a useful tool for debugging while developing the feature?\n> \n> If it is the former (and I'd say why not, even though \"git -c\n> core.fsmontor=...\" may be easy enough), we might want to rename it\n> to replace _TEST with _PROGRAM or something and document it.\n> \n\nThis was added this to facilitate testing. That is why it has the magic \nnaming of \"GIT_***_TEST\" which is the only way I found to ensure that \nenv variable gets passed to tests. Its use is discussed in patch 10 \nwhich contains the tests.\n\n>> diff --git a/diff-lib.c b/diff-lib.c\n>> index 2a52b07954..23c6d03ca9 100644\n>> --- a/diff-lib.c\n>> +++ b/diff-lib.c\n>> @@ -12,6 +12,7 @@\n>> #include \"refs.h\"\n>> #include \"submodule.h\"\n>> #include \"dir.h\"\n>> +#include \"fsmonitor.h\"\n>> \n>> /*\n>> * diff-files\n>> @@ -228,6 +229,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n>> \n>> \t\tif (!changed && !dirty_submodule) {\n>> \t\t\tce_mark_uptodate(ce);\n>> +\t\t\tmark_fsmonitor_valid(ce);\n> \n> I notice all calls to mark_fsmonitor_valid(ce) always follow a call\n> to ce_mark_uptodate(ce). Should the call to the former be made as\n> part of the latter instead? Are there cases where we want to call\n> ce_mark_uptodate(ce) to mark the ce up-to-date, yet we do not want\n> to call mark_fsmonitor_valid(ce)? How does a programmer tell when\n> s/he wants to call ce_mark_uptodate(ce) if s/he also should call\n> mark_fsmonitor_valid(ce)?\n\nmark_fsmonitor_valid(ce) is the way to indicate that cache entries that \nwere once fsmonitor dirty are now properly reflected in the index so can \ncome off the \"dirty\" list. It can't really be combined with \nce_mark_uptodate(ce) as that would prevent the CE_MATCH_IGNORE_FSMONITOR \nlogic:\n\n\tif (!ignore_skip_worktree && ce_skip_worktree(ce)) {\n\t\tce_mark_uptodate(ce);\n\t\treturn ce;\n\t}\n\tif (!ignore_valid && (ce->ce_flags & CE_VALID)) {\n\t\tce_mark_uptodate(ce);\n\t\treturn ce;\n\t}\n\tif (!ignore_fsmonitor && (ce->ce_flags & CE_FSMONITOR_VALID)) {\n\t\tce_mark_uptodate(ce);\n\t\treturn ce;\n\t}\n\nIn addition, fsmonitor is an optional feature and so the \nmark_fsmonitor_valid(ce) call should only happen when the feature is \nturned on. I tried to keep it as simple as possible by making that test \nand set logic part of the function but still be performant by making the \nfunction inline.\n\n> \n> Together with \"when to pass IGNORE_FSMONITOR\" question, is there a\n> summary that guides new programmers to answer these questions in the\n> new documentation?\n> \n\nOnly the discussion in this mail thread. I could add something to the \nfunction header in fsmonitor.h if that would help. How about something \nlike:\n\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex c2240b811a..03bf3efe61 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -34,9 +34,11 @@ extern void tweak_fsmonitor(struct index_state *istate);\n */\n extern void refresh_fsmonitor(struct index_state *istate);\n\n-/*\n- * Set the given cache entries CE_FSMONITOR_VALID bit.\n- */\n+/*\n+ * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n+ * called any time the cache entry has been updated to reflect the\n+ * current state of the file on disk.\n+ */\n static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n {\n if (core_fsmonitor) {\n@@ -46,8 +48,11 @@ static inline void mark_fsmonitor_valid(struct \ncache_entry *ce)\n }\n\n /*\n- * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate any\n- * corresponding untracked cache directory structures.\n+ * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate\n+ * any corresponding untracked cache directory structures. This should\n+ * be called any time git creates or modifies a file that should\n+ * trigger an lstat() or invalidate the untracked cache for the\n+ * corresponding directory\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, \nstruct cache_entry *ce)\n {\n\n\n>> diff --git a/dir.h b/dir.h\n>> index e3717055d1..fab8fc1561 100644\n>> --- a/dir.h\n>> +++ b/dir.h\n>> @@ -139,6 +139,8 @@ struct untracked_cache {\n>> \tint gitignore_invalidated;\n>> \tint dir_invalidated;\n>> \tint dir_opened;\n>> +\t/* fsmonitor invalidation data */\n>> +\tunsigned int use_fsmonitor : 1;\n> \n> This makes it look like we will add a bit more fields in later\n> patches that are about \"invalidation\" around fsmonitor, but it\n> appears that such an addition never happens til the end of the\n> series. And use_fsmonitor boolean does not seem to be what the\n> comment says---it just tells us if fsmonitor is in use in the\n> operation of the untracked cache, no?\n\nI don't have any more planned bit fields. I only needed a single bit so \nused a bit field just in case someone else comes by later and needs \nanother bit. If you aren't worried about that, we can just make this an \nint.\n\nCorrect. The bit just indicates whether fsmonitor has been used to \nensure the cache is current or if it needs to be checked. I have \ncomments to this effect where the flag is used in the code but could \nmove/duplicate them into the header if wished. For example:\n\n/* With fsmonitor, we can trust the untracked cache's valid field. */\n\nand\n\n/* Now mark the untracked cache for fsmonitor usage */\n\n> \n>> diff --git a/entry.c b/entry.c\n>> index cb291aa88b..5e6794f9fc 100644\n>> --- a/entry.c\n>> +++ b/entry.c\n>> @@ -4,6 +4,7 @@\n>> #include \"streaming.h\"\n>> #include \"submodule.h\"\n>> #include \"progress.h\"\n>> +#include \"fsmonitor.h\"\n>> \n>> static void create_directories(const char *path, int path_len,\n>> \t\t\t const struct checkout *state)\n>> @@ -357,6 +358,7 @@ static int write_entry(struct cache_entry *ce,\n>> \t\t\tlstat(ce->name, &st);\n>> \t\tfill_stat_cache_info(ce, &st);\n>> \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n>> +\t\tmark_fsmonitor_invalid(state->istate, ce);\n>> \t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n> \n> Similar to \"how does mark_fsmonitor_valid() relate to\n> ce_mark_uptodate()?\" question earlier, mark_fsmonitor_invalid()\n> seems to often appear in pairs with the update to cache_changed.\n> Are there cases where we need CE_ENTRY_CHANGED bit added to the\n> index state yet we do not want to call mark_fsmonitor_invalid()? I\n> am wondering if there should be a single helper function that lets\n> callers say \"I changed this ce in this istate this way. Please\n> update CE_VALID, CE_UPDATE_IN_BASE and CE_FSMONITOR_VALID\n> accordingly\".\n> \n> That \"changed _this way_\" is deliberately vague in my question\n> above, because it is not yet clear to me when mark-valid and\n> mark-invalid should and should not be called from the series.\n> \n\nPlease let me know if my comment/patch above does not address this \nconcern sufficiently.\n\n>> +\t/* a fsmonitor process can return '*' to indicate all entries are invalid */\n>> +\tif (query_success && query_result.buf[0] != '/') {\n>> +\t\t/* Mark all entries returned by the monitor as dirty */\n> \n> The comment talks about '*' and code checks if it is not '/'? The\n> else clause of this if/else handles the \"invalidate all\" case, so\n> shouldn't we be comparing with '*' instead here?\n> \n> Ah, fsmonitor-watchman section of the doc tells us to write the hook\n> in a way to return slash, so the comment above the code is stale and\n> the comparison with '/' is what we want, I guess.\n> \n\nCorrect. Sorry about missing that. Here is a patch that can be \nsquashed in.\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex b8b2d88fe1..7c1540c054 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -176,7 +176,7 @@ void refresh_fsmonitor(struct index_state *istate)\n core_fsmonitor, query_success ? \"success\" : \n\"failure\");\n }\n\n- /* a fsmonitor process can return '*' to indicate all entries \nare invalid */\n+ /* a fsmonitor process can return '/' to indicate all entries \nare invalid */\n if (query_success && query_result.buf[0] != '/') {\n /* Mark all entries returned by the monitor as dirty */\n buf = query_result.buf;\n\n"},{"id":"328471","messageId":"53d05ca7-381b-255c-1362-ca1888d626fe@gmail.com","threadId":"46154","inReplyTo":"xmqq377hopma.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-20T16:29:37Z","receivedAt":"2017-09-20T16:29:47Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/20/2017 2:23 AM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> @@ -344,6 +346,7 @@ struct index_state {\n>> \tstruct hashmap dir_hash;\n>> \tunsigned char sha1[20];\n>> \tstruct untracked_cache *untracked;\n>> +\tuint64_t fsmonitor_last_update;\n> \n> This field being zero has more significance than just \"we haven't\n> got any update yet\", right? The way I am reading the code is that\n> setting it 0 is a way to signal that fsmon has been inactivated. It\n> also made me wonder if add_fsmonitor() that silently returns without\n> doing anything when this field is already non-zero is a bug (in\n> other words, I couldn't tell what the right answer would be to a\n> question \"shouldn't the caller be avoiding duplicate calls?\").\n> \n\nCorrect again. For better (and sometimes for worse) I followed the \npattern set by the untracked cache. If you compare them, you will \nnotice striking similarities. :)\n\n>> diff --git a/fsmonitor.c b/fsmonitor.c\n>> new file mode 100644\n>> index 0000000000..b8b2d88fe1\n>> --- /dev/null\n>> +++ b/fsmonitor.c\n>> ...\n> \n> This part was a pleasant read.\n> > Thanks.\n> \n\nThank you for your careful review. I appreciate having another set of \neyes taking a close look especially as I see this as a big first step \ntowards making many git operations O(# changed files) instead of O(# \nsize of working directory). Seeing status times drop from 1m22s to 1.45s \nis a huge perf win - but only if it is correct!\n"},{"id":"328473","messageId":"9cd0c22e-d044-b059-39d5-40b401b18b3e@gmail.com","threadId":"46154","inReplyTo":"CAN0heSqFsm0BSUzufXjCKNv=JpRpUQzFA9GkKfA6nd_hPhW_qg@mail.gmail.com","subject":"Re: [PATCH v7 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-20T17:02:20Z","receivedAt":"2017-09-20T17:02:27Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Thanks for the review. I'm not an English major so appreciate the \nfeedback on my attempts to document the feature.\n\nOn 9/20/2017 6:00 AM, Martin Ågren wrote:\n> On 19 September 2017 at 21:27, Ben Peart wrote:\n>> diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\n>> index e19eba62cd..95231dbfcb 100644\n>> --- a/Documentation/git-update-index.txt\n>> +++ b/Documentation/git-update-index.txt\n>> @@ -16,9 +16,11 @@ SYNOPSIS\n>> [--chmod=(+|-)x]\n>> [--[no-]assume-unchanged]\n>> [--[no-]skip-worktree]\n>> + [--[no-]fsmonitor-valid]\n>> [--ignore-submodules]\n>> [--[no-]split-index]\n>> [--[no-|test-|force-]untracked-cache]\n>> + [--[no-]fsmonitor]\n>> [--really-refresh] [--unresolve] [--again | -g]\n>> [--info-only] [--index-info]\n>> [-z] [--stdin] [--index-version ]\n>> @@ -111,6 +113,12 @@ you will need to handle the situation manually.\n>> set and unset the \"skip-worktree\" bit for the paths. See\n>> section \"Skip-worktree bit\" below for more information.\n>>\n>> +--[no-]fsmonitor-valid::\n>> + When one of these flags is specified, the object name recorded\n>> + for the paths are not updated. Instead, these options\n>> + set and unset the \"fsmonitor valid\" bit for the paths. See\n>> + section \"File System Monitor\" below for more information.\n>> +\n> \n> So --no-foo does not undo --foo, but there are three values: --foo,\n> --no-foo and . I find that unintuitive, but maybe it's\n> just me. Maybe there are other such options in the codebase already. \n\nI understand the unintuitive comment but the other such options in the \ncode base are just above the fsmonitor options as it is modeled on how \n'assume-unchanged' and 'skip-worktree' work. Consistency is certainly \nhelps the intuitiveness as once you have learned the model, it applies \nin other places.\n\nHow\n> about --fsmonitor-valid=set, --fsmonitor-valid=unset, and\n> --no-fsmonitor-valid (which would be the default, and which would forget\n> any earlier --fsmonitor-valid=...)?\n> \n>> -g::\n>> --again::\n>> Runs 'git update-index' itself on the paths whose index\n>> @@ -201,6 +209,15 @@ will remove the intended effect of the option.\n>> `--untracked-cache` used to imply `--test-untracked-cache` but\n>> this option would enable the extension unconditionally.\n>>\n>> +--fsmonitor::\n>> +--no-fsmonitor::\n> \n> Maybe \"--[no-]fsmonitor\" for symmetry with how you've done it above and\n> later.\n> \n\nFor better and for worse, I choose to be consistent with how the options \nwork (especially the untracked-cache option immediately above). This is \none weakness of reviewing patches via email - you don't see the patch in \ncontext with everything around it.\n\n>> +When used in conjunction with the untracked cache, it can further improve\n>> +performance by avoiding the cost of scaning the entire working directory\n>> +looking for new files.\n> \n> s/scaning/scanning/\n> \n\nThanks!\n\ndiff --git a/Documentation/git-update-index.txt \nb/Documentation/git-update-index.txt\nindex 95231dbfcb..7c2f880a22 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -476,7 +476,7 @@ inform it as to what files have been modified. This \nenables git to avoid\n having to lstat() every file to find modified files.\n\n When used in conjunction with the untracked cache, it can further improve\n-performance by avoiding the cost of scaning the entire working directory\n+performance by avoiding the cost of scanning the entire working directory\n looking for new files.\n\n If you want to enable (or disable) this feature, it is easier to use\n\n\n>> +If you want to enable (or disable) this feature, it is easier to use\n>> +the `core.fsmonitor` configuration variable (see\n>> +linkgit:git-config[1]) than using the `--fsmonitor` option to\n>> +`git update-index` in each repository, especially if you want to do so\n>> +across all repositories you use, because you can set the configuration\n>> +variable to `true` (or `false`) in your `$HOME/.gitconfig` just once\n>> +and have it affect all repositories you touch.\n> \n> This is a mouthful. Maybe you could split it a little, perhaps like so:\n> \n> If you want to enable (or disable) this feature, you will probably\n> want to use the `core.fsmonitor` configuration variable (see\n> linkgit:git-config[1]). By setting it to `true` (or `false`) in your\n> `$HOME/.gitconfig`, it will affect all repositories you touch. For a\n> more fine-grained control, you can set it per repository, or use the\n> `--fsmonitor` option with `git update-index` in each repository.\n> \n\nI'm going to sound like a broken record here. :) The description favored \nconsistency with the untracked cache feature immediate above this entry. \n It is literally a copy/paste/edit.\n\nThis is based on the assumption that the text had already been reviewed \nand found to be acceptable. This also means if you have figured it out \nfor one option, when you read the next, you're understanding can carry \nforward speeding up your comprehension.\n\n> The part about $HOME/.gitconfig vs per-repo config is perhaps generic\n> enough that it doesn't belong here. So it'd only be about config vs.\n> option. Where to place the config item and what implications that has is\n> arguably orthogonal to knowing that the option exists and what it does.\n> \n> Martin\n> \n"},{"id":"328474","messageId":"43c9d3b2-2895-adcb-6f77-b6967aacf9c8@gmail.com","threadId":"46154","inReplyTo":"20170919204354.GG75068@aiede.mtv.corp.google.com","subject":"Re: [PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-20T17:11:19Z","receivedAt":"2017-09-20T17:11:26Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/19/2017 4:43 PM, Jonathan Nieder wrote:\n> Hi,\n> \n> Ben Peart wrote:\n> \n>> The split index test t1700-split-index.sh has hard coded SHA values for\n>> the index. Currently it supports index V4 and V3 but assumes there are\n>> no index extensions loaded.\n>>\n>> When manually forcing the fsmonitor extension to be turned on when\n>> running the test suite, the SHA values no longer match which causes the\n>> test to fail.\n>>\n>> The potential matrix of index extensions and index versions can is quite\n>> large so instead disable the extension before attempting to run the test.\n> \n> Thanks for finding and diagnosing this problem.\n> \n> This feels to me like the wrong fix. Wouldn't it be better for the\n> test not to depend on the precise object ids? See the \"Tips for\n> Writing Tests\" section in t/README:\n> \n\nI completely agree that a better fix would be to rewrite the test to not \nhard code the SHA values. I'm sure this will come to bite us again as \nwe discuss the migration to a different SHA algorithm.\n\nThat said, I think fixing this correctly is outside the scope of this \npatch series. It has been written this way since it was created back in \n2014 (and patched in 2015 to hard code the V4 index SHA).\n\nIf desired, this patch can simply be dropped from the series entirely as \nI doubt anyone other than me will attempt to run it with the fsmonitor \nextension turned on.\n\n> \t And\n> \tsuch drastic changes to the core GIT that even changes these\n> \totherwise supposedly stable object IDs should be accompanied by\n> \tan update to t0000-basic.sh.\n> \n> \tHowever, other tests that simply rely on basic parts of the core\n> \tGIT working properly should not have that level of intimate\n> \tknowledge of the core GIT internals. If all the test scripts\n> \thardcoded the object IDs like t0000-basic.sh does, that defeats\n> \tthe purpose of t0000-basic.sh, which is to isolate that level of\n> \tvalidation in one place. Your test also ends up needing\n> \tupdating when such a change to the internal happens, so do _not_\n> \tdo it and leave the low level of validation to t0000-basic.sh.\n> \n> Worse, t1700-split-index.sh doesn't explain where the object ids it\n> uses comes from so it is not even obvious to a casual reader like me\n> how to fix it.\n> \n> See t/diff-lib.sh for some examples of one way to avoid depending on\n> the object id computation. Another way that is often preferable is to\n> come up with commands to compute the expected hash values, like\n> $(git rev-parse HEAD^{tree}), and use those instead of hard-coded\n> values.\n> \n> Thanks and hope that helps,\n> Jonathan\n> \n"},{"id":"328475","messageId":"CAN0heSoSLJox6xzVwOibjgaa7tj=L3Pm+0XxVr6LYBp3+t1x6w@mail.gmail.com","threadId":"46154","inReplyTo":"9cd0c22e-d044-b059-39d5-40b401b18b3e@gmail.com","subject":"Re: [PATCH v7 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2017-09-20T17:11:34Z","receivedAt":"2017-09-20T17:11:40Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 20 September 2017 at 19:02, Ben Peart wrote:\n>>> +--[no-]fsmonitor-valid::\n>>> + When one of these flags is specified, the object name recorded\n>>> + for the paths are not updated. Instead, these options\n>>> + set and unset the \"fsmonitor valid\" bit for the paths. See\n>>> + section \"File System Monitor\" below for more information.\n>>> +\n>>\n>>\n>> So --no-foo does not undo --foo, but there are three values: --foo,\n>> --no-foo and . I find that unintuitive, but maybe it's\n>> just me. Maybe there are other such options in the codebase already.\n>\n>\n> I understand the unintuitive comment but the other such options in the code\n> base are just above the fsmonitor options as it is modeled on how\n> 'assume-unchanged' and 'skip-worktree' work. Consistency is certainly helps\n> the intuitiveness as once you have learned the model, it applies in other\n> places.\n>\n[...]\n>\n> For better and for worse, I choose to be consistent with how the options\n> work (especially the untracked-cache option immediately above). This is one\n> weakness of reviewing patches via email - you don't see the patch in context\n> with everything around it.\n>\n[...]\n>\n> I'm going to sound like a broken record here. :) The description favored\n> consistency with the untracked cache feature immediate above this entry. It\n> is literally a copy/paste/edit.\n\nOh. Well, that's what I get for \"reviewing\" by e-mail. You are indeed\nfollowing the current style very well! Sorry for the noise.\n\nMartin\n"},{"id":"328477","messageId":"20170920174613.GA27425@aiede.mtv.corp.google.com","threadId":"46154","inReplyTo":"43c9d3b2-2895-adcb-6f77-b6967aacf9c8@gmail.com","subject":"Re: [PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-09-20T17:46:13Z","receivedAt":"2017-09-20T17:46:21Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBen Peart wrote:\n> On 9/19/2017 4:43 PM, Jonathan Nieder wrote:\n\n>> This feels to me like the wrong fix. Wouldn't it be better for the\n>> test not to depend on the precise object ids? See the \"Tips for\n>> Writing Tests\" section in t/README:\n>\n> I completely agree that a better fix would be to rewrite the test to\n> not hard code the SHA values. I'm sure this will come to bite us\n> again as we discuss the migration to a different SHA algorithm.\n\nnit: the kind of change I'm proposing does not entail a full rewrite. :)\n\nThe SHA migration aspect is true, but that's actually the least of my\nworries. I intend to introduce a SHA1 test prereq that crazy tests\nwhich want to depend on the hash function can declare a dependency on.\n\nMy actual worry is that tests hard-coding object ids are (1) hard to\nunderstand, as illustrated by my having no clue what these particular\nobject ids refer to and (2) very brittle, since an object id changes\nwhenever a timestamp or any of the history leading to an object\nchanges. They create a trap for anyone wanting to change the test\nlater. They are basically change detector tests, which is generally\naccepted to be a bad practice.\n\n> That said, I think fixing this correctly is outside the scope of\n> this patch series. It has been written this way since it was\n> created back in 2014 (and patched in 2015 to hard code the V4 index\n> SHA).\n\nFair enough.\n\n> If desired, this patch can simply be dropped from the series\n> entirely as I doubt anyone other than me will attempt to run it with\n> the fsmonitor extension turned on.\n\n*shrug*\n\nMy motivations in the context of the review were:\n\n * now that we noticed the problem, we have an opportunity to fix it!\n (i.e. a fix would not have to be part of this series and would not\n necessarily have to be written by you)\n\n * if we include this non-fix, the commit message really needs to say\n something about it. Otherwise people are likely to cargo-cult it\n in other contexts and make the problem worse.\n\nThanks,\nJonathan\n"},{"id":"328507","messageId":"CAGZ79kaJWDWGgqRGTdgc+RjVxAb52CpUCk2DCotSksni4cH1Ww@mail.gmail.com","threadId":"46154","inReplyTo":"20170919192744.19224-3-benpeart@microsoft.com","subject":"Re: [PATCH v7 02/12] preload-index: add override to enable testing preload-index","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-09-20T22:06:54Z","receivedAt":"2017-09-20T22:07:00Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Sep 19, 2017 at 12:27 PM, Ben Peart wrote:\n> Preload index doesn't run unless it has a minimum number of 1000 files.\n> To enable running tests with fewer files, add an environment variable\n> (GIT_FORCE_PRELOAD_TEST) which will override that minimum and set it to 2.\n\n'it' being the number of threads ('it' was not mentioned before,\nso reading the commit message confused me initially)\n\n>\n> Signed-off-by: Ben Peart \n> ---\n> preload-index.c | 2 ++\n> 1 file changed, 2 insertions(+)\n>\n> diff --git a/preload-index.c b/preload-index.c\n> index 70a4c80878..75564c497a 100644\n> --- a/preload-index.c\n> +++ b/preload-index.c\n> @@ -79,6 +79,8 @@ static void preload_index(struct index_state *index,\n> return;\n>\n> threads = index->cache_nr / THREAD_COST;\n> + if ((index->cache_nr > 1) && (threads < 2) && getenv(\"GIT_FORCE_PRELOAD_TEST\"))\n> + threads = 2;\n\nAdding these lines is just a bandaid to trick\n\n> if (threads < 2)\n> return;\n\nto not return early as the commit message does not discuss why\nwe set it to 2.\n\nDo we need threads at all for these tests, or would a patch like\n\n- if (threads < 2)\n+ if (threads < 2 && !GIT_FORCE_PRELOAD_TEST)\n return;\n\nwork as well?\n\nThat way tests can use any number of threads, though\nthey currently have no way of overriding the heuristic, yet.\n\nWith this alternative patch, it sounds to me as if the\nissues kept more orthogonal. (specifically \"Do we use preload?\"\nand \"How many threads?\". One could imagine that we later\nwant to introduce GIT_PRELOAD_THREADS for $reasons\nand that would go over well in combination with\nGIT_FORCE_PRELOAD_TEST)\n\nIt seems to be only an internal test variable, such that we do\nnot need documentation. (Is that worth mentioning in the\ncommit message?)\n\nThe test to make use of this new variable is found\nin another patch I presume?\n\nStefan\n"},{"id":"328510","messageId":"0bc0b797-fb82-2300-6df1-dc5380e64c85@gmail.com","threadId":"46154","inReplyTo":"CAGZ79kaJWDWGgqRGTdgc+RjVxAb52CpUCk2DCotSksni4cH1Ww@mail.gmail.com","subject":"Re: [PATCH v7 02/12] preload-index: add override to enable testing preload-index","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-21T00:02:16Z","receivedAt":"2017-09-21T00:02:25Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/20/2017 6:06 PM, Stefan Beller wrote:\n> On Tue, Sep 19, 2017 at 12:27 PM, Ben Peart wrote:\n>> Preload index doesn't run unless it has a minimum number of 1000 files.\n>> To enable running tests with fewer files, add an environment variable\n>> (GIT_FORCE_PRELOAD_TEST) which will override that minimum and set it to 2.\n> \n> 'it' being the number of threads ('it' was not mentioned before,\n> so reading the commit message confused me initially)\n> \n>>\n>> Signed-off-by: Ben Peart \n>> ---\n>> preload-index.c | 2 ++\n>> 1 file changed, 2 insertions(+)\n>>\n>> diff --git a/preload-index.c b/preload-index.c\n>> index 70a4c80878..75564c497a 100644\n>> --- a/preload-index.c\n>> +++ b/preload-index.c\n>> @@ -79,6 +79,8 @@ static void preload_index(struct index_state *index,\n>> return;\n>>\n>> threads = index->cache_nr / THREAD_COST;\n>> + if ((index->cache_nr > 1) && (threads < 2) && getenv(\"GIT_FORCE_PRELOAD_TEST\"))\n>> + threads = 2;\n> \n> Adding these lines is just a bandaid to trick\n> \n>> if (threads < 2)\n>> return;\n> \n> to not return early as the commit message does not discuss why\n> we set it to 2.\n> \n\nTo execute the preload code path, we need a minimum of 2 cache entries \nand 2 threads so that each thread actually has work to do. Otherwise \nthe logic below that divides the work up would need to be updated as \nwell. The additional complexity didn't seem worth it just to enable the \ncode path to execute with a single thread on a single cache entry.\n\n> Do we need threads at all for these tests, or would a patch like\n> \n> - if (threads < 2)\n> + if (threads < 2 && !GIT_FORCE_PRELOAD_TEST)\n> return;\n> \n> work as well?\n\nThat would require a larger patch that would update the work division \nand thread creation logic for little to no gain.\n\n> \n> That way tests can use any number of threads, though\n> they currently have no way of overriding the heuristic, yet.\n> \n> With this alternative patch, it sounds to me as if the\n> issues kept more orthogonal. (specifically \"Do we use preload?\"\n> and \"How many threads?\". One could imagine that we later\n> want to introduce GIT_PRELOAD_THREADS for $reasons\n> and that would go over well in combination with\n> GIT_FORCE_PRELOAD_TEST)\n> \n\nI'm not sure why someone would want to test varying numbers of threads \nas that isn't a case that can ever actually happen with the existing \ncode/feature.\n\nI was just enabling testing of the code path with fewer than 1000 files \nas when I made my changes, I discovered that 1) there were no test cases \nfor the core.preloadindex feature at all and 2) there was no way to run \nthe existing tests with core.preloadindex as they don't have the minimum \nnumber of files. This patch allows you to run the existing test suite \nwith preload_index turned on for any test case that has 2 or more files \n(it passes by the way :)).\n\n> It seems to be only an internal test variable, such that we do\n> not need documentation. (Is that worth mentioning in the\n> commit message?)\n\nCorrect. That is the reason I used the magic GIT_***_TEST naming \npattern as that is the only way to ensure the environment variable is \npreserved when running the tests.\n\n> \n> The test to make use of this new variable is found\n> in another patch I presume?\n> \n\nThere are no new tests that take advantage of this new environment \nvariable. Instead you can run the entire git test suite with it by running:\n\nGIT_FORCE_PRELOAD_TEXT=1 prove -j12 --state=all ./t[0-9]*.sh\n\nI can add that to the commit message to make it more obvious if desired.\n\n> Stefan\n> \n"},{"id":"328511","messageId":"ee8157e0-1f82-2384-3742-1dd00ea4b58c@gmail.com","threadId":"46154","inReplyTo":"20170920174613.GA27425@aiede.mtv.corp.google.com","subject":"Re: [PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-21T00:05:59Z","receivedAt":"2017-09-21T00:06:07Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/20/2017 1:46 PM, Jonathan Nieder wrote:\n> Hi,\n> \n> Ben Peart wrote:\n>> On 9/19/2017 4:43 PM, Jonathan Nieder wrote:\n> \n>>> This feels to me like the wrong fix. Wouldn't it be better for the\n>>> test not to depend on the precise object ids? See the \"Tips for\n>>> Writing Tests\" section in t/README:\n>>\n>> I completely agree that a better fix would be to rewrite the test to\n>> not hard code the SHA values. I'm sure this will come to bite us\n>> again as we discuss the migration to a different SHA algorithm.\n> \n> nit: the kind of change I'm proposing does not entail a full rewrite. :)\n> \n> The SHA migration aspect is true, but that's actually the least of my\n> worries. I intend to introduce a SHA1 test prereq that crazy tests\n> which want to depend on the hash function can declare a dependency on.\n> \n> My actual worry is that tests hard-coding object ids are (1) hard to\n> understand, as illustrated by my having no clue what these particular\n> object ids refer to and (2) very brittle, since an object id changes\n> whenever a timestamp or any of the history leading to an object\n> changes. They create a trap for anyone wanting to change the test\n> later. They are basically change detector tests, which is generally\n> accepted to be a bad practice.\n> \n>> That said, I think fixing this correctly is outside the scope of\n>> this patch series. It has been written this way since it was\n>> created back in 2014 (and patched in 2015 to hard code the V4 index\n>> SHA).\n> \n> Fair enough.\n> \n>> If desired, this patch can simply be dropped from the series\n>> entirely as I doubt anyone other than me will attempt to run it with\n>> the fsmonitor extension turned on.\n> \n> *shrug*\n> \n> My motivations in the context of the review were:\n> \n> * now that we noticed the problem, we have an opportunity to fix it!\n> (i.e. a fix would not have to be part of this series and would not\n> necessarily have to be written by you)\n> \n> * if we include this non-fix, the commit message really needs to say\n> something about it. Otherwise people are likely to cargo-cult it\n> in other contexts and make the problem worse.\n> \n\nI'll update the commit message to indicate this is a temporary \nworkaround until the underlying hard coded SHA issue is fixed.\n\n> Thanks,\n> Jonathan\n> \n"},{"id":"328514","messageId":"CAGZ79kbW1GnaSz8qhCgV6BP23CWiYAzq_WbDVv_Yf1GN_3u=wQ@mail.gmail.com","threadId":"46154","inReplyTo":"0bc0b797-fb82-2300-6df1-dc5380e64c85@gmail.com","subject":"Re: [PATCH v7 02/12] preload-index: add override to enable testing preload-index","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-09-21T00:44:21Z","receivedAt":"2017-09-21T00:44:28Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> To execute the preload code path, we need a minimum of 2 cache entries and 2\n> threads so that each thread actually has work to do. Otherwise the logic\n> below that divides the work up would need to be updated as well. The\n> additional complexity didn't seem worth it just to enable the code path to\n> execute with a single thread on a single cache entry.\n\n[snip]\n\n>\n> That would require a larger patch that would update the work division and\n> thread creation logic for little to no gain.\n\nOh I was not aware of the additional needed refactoring. (I assumed that\nwould *just work*, experience should have told me to have a look first).\n\n>\n> There are no new tests that take advantage of this new environment variable.\n> Instead you can run the entire git test suite with it by running:\n>\n> GIT_FORCE_PRELOAD_TEXT=1 prove -j12 --state=all ./t[0-9]*.sh\n>\n> I can add that to the commit message to make it more obvious if desired.\n\nit would have helped me being less confused as there was an \"obvious\"\neasier and more correct alternative, which you just explained\nis neither easier nor more correct as we probably do not desire that\nanyway.\n\nThanks,\nStefan\n"},{"id":"328522","messageId":"xmqq8th8n7rf.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"682237f1-4747-5712-e95b-285379eb1b69@gmail.com","subject":"Re: [PATCH v7 03/12] update-index: add a new --force-write-index option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-21T01:46:28Z","receivedAt":"2017-09-21T01:46:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> Lets see how my ascii art skills do at describing this:\n\nYour ascii art is fine. If you said upfront that the capital\nletters signify points in time, lower letters are file-touching\nevents, and time flows from left to right, it would have been\nperfect ;-)\n\n> There is no real concern about accumulating too many changes as 1) the\n> processing cost for additional modified files is fairly trivial and 2)\n> the index ends up getting written out pretty frequently anyway as\n> files are added/removed/staged/etc which updates the\n> fsmonitor_last_update time.\n\nI still see some impedance mismatch here. The optimization\ndescribed is valid but the code to do the optimization would avoid\nwriting the index file out when the in-core index is dirty only\nbecause the status reported by fsmonitor changed--if there were\nother changes (e.g. the code added a new index entry), even with the\noptimization, you would still want to write the index out, right?\n\nWith or without the need for forced flush to help debugging, would\nthat suggest that you need two bits now, instead of just a single\n'active-cache-changed' bit?\n\nBy keeping track of that new bit that tells us if we have\nfsmonitor-only changes that we _could_ flush, this patch can further\nreduce the need for forced flushing (i.e. if we know we didn't get\nfsmonitor induced dirtyness, force_write can still be no-op), no?\n\n>\n> The challenge came when it was time to test that the changes to the\n> index were correct. Since they are lazily written by default, I\n> needed a way to force the write so that I could verify the index on\n> disk was correct. Hence, this patch.\n>\n>\n>>> \t\tOPT_END()\n>>> \t};\n>>> @@ -1147,7 +1150,7 @@ int cmd_update_index(int argc, const char\n>>> **argv, const char *prefix)\n>>> \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n>>> \t}\n>>> -\tif (active_cache_changed) {\n>>> +\tif (active_cache_changed || force_write) {\n>>> \t\tif (newfd < 0) {\n>>> \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n>>> \t\t\t\texit(128);\n"},{"id":"328523","messageId":"xmqq377gn74p.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"3311de8b-f9df-07e0-6c5d-7f491e9bcaa8@gmail.com","subject":"Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-21T02:00:06Z","receivedAt":"2017-09-21T02:00:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> Pretty much the same places you would also use CE_MATCH_IGNORE_VALID\n> and CE_MATCH_IGNORE_SKIP_WORKTREE which serve the same role for those\n> features. That is generally when you are about to overwrite data so\n> want to be *really* sure you have what you think you have.\n\nNow that makes me worried gravely. \n\nIGNORE_VALID is ignored in these places because we have been burned\nby end-users lying to us. IGNORE_SKIP_WORKTREE must be ignored\nbecause we know that the working tree state does not match the\n\"reality\" the index wants to have. The fact that the code treats\nthe status reported and kept up to date by fsmonitor the same way as\nthese two implies that it is merely advisory and cannot be trusted?\nIs that the reason why we tell the codepath with IGNORE_FSMONITOR to\nignore the state fsmonitor reported and check the state ourselves?\n\nOh, wait...\n\n\n> The other place I used it was in preload_index(). In that case, I\n> didn't want to trigger the call to refresh_fsmonitor() as\n> preload_index() is about trying to do a fast precompute of state for\n> the bulk of the index entries but is not required for correctness as\n> refresh_cache_ent() will ensure any \"missed\" by preload_index() are\n> up-to-date if/when that is needed.\n\nThat is a very valid design decision. So IGNORE_FSMONITOR is,\nunlike IGNORE_VALID and IGNORE_SKIP_WORKTREE, to tell us \"do not\nbother asking fsmonitor to refresh the state of this entry--it is OK\nfor us to use a slightly stale information\"? That would make sense\nas an optimization, but that does not mesh well with the previous\n\"we need to be really really sure\" usecase. That one wants \"we do\nnot trust fsmonitor, so do not bother asking to refresh; we will do\nso ourselves\", which would not help the \"we can use slightly stale\none and that is OK\" usecase.\n\nPuzzled...\n"},{"id":"328524","messageId":"28891f82-32e5-dc46-961c-a44e86e84989@gmail.com","threadId":"46154","inReplyTo":"xmqq8th8n7rf.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v7 03/12] update-index: add a new --force-write-index option","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-21T02:06:00Z","receivedAt":"2017-09-21T02:06:08Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/20/2017 9:46 PM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> Lets see how my ascii art skills do at describing this:\n> \n> Your ascii art is fine. If you said upfront that the capital\n> letters signify points in time, lower letters are file-touching\n> events, and time flows from left to right, it would have been\n> perfect ;-)\n> \n\nRats, so close and yet... ;-)\n\n>> There is no real concern about accumulating too many changes as 1) the\n>> processing cost for additional modified files is fairly trivial and 2)\n>> the index ends up getting written out pretty frequently anyway as\n>> files are added/removed/staged/etc which updates the\n>> fsmonitor_last_update time.\n> \n> I still see some impedance mismatch here. The optimization\n> described is valid but the code to do the optimization would avoid\n> writing the index file out when the in-core index is dirty only\n> because the status reported by fsmonitor changed--if there were\n> other changes (e.g. the code added a new index entry), even with the\n> optimization, you would still want to write the index out, right?\n> \n\nYes, that is exactly how it works. FSMONITOR_CHANGED is only set when \nthe fsmonitor index extension is added or removed but all the other \nlogic to flag the index dirty (CE_ENTRY_CHANGED/REMOVED/ADDED, \nUNTRACKED_CHANGED, etc) still exists and will still trigger the index to \nbe written out as it always has. It's not to say some of these other \nchanges could not do the same optimization (I haven't looked) if \nrecomputing is cheaper than writing out the index.\n\n> With or without the need for forced flush to help debugging, would\n> that suggest that you need two bits now, instead of just a single\n> 'active-cache-changed' bit?\n> \n> By keeping track of that new bit that tells us if we have\n> fsmonitor-only changes that we _could_ flush, this patch can further\n> reduce the need for forced flushing (i.e. if we know we didn't get\n> fsmonitor induced dirtyness, force_write can still be no-op), no?\n> \n\nYes, I suppose we _could_ add a 2nd bit (and then add the logic to set \nthat bit every time a fsmonitor change was made) but I don't see that it \nreally buys us anything useful. The force write flag in update-index is \noff by default and the only scenario we have that someone would set it \nis for test cases where the perf of writing out the index when it is not \nneeded just doesn't matter.\n\n>>\n>> The challenge came when it was time to test that the changes to the\n>> index were correct. Since they are lazily written by default, I\n>> needed a way to force the write so that I could verify the index on\n>> disk was correct. Hence, this patch.\n>>\n>>\n>>>> \t\tOPT_END()\n>>>> \t};\n>>>> @@ -1147,7 +1150,7 @@ int cmd_update_index(int argc, const char\n>>>> **argv, const char *prefix)\n>>>> \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n>>>> \t}\n>>>> -\tif (active_cache_changed) {\n>>>> +\tif (active_cache_changed || force_write) {\n>>>> \t\tif (newfd < 0) {\n>>>> \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n>>>> \t\t\t\texit(128);\n"},{"id":"328525","messageId":"xmqqy3p8lrpp.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"28891f82-32e5-dc46-961c-a44e86e84989@gmail.com","subject":"Re: [PATCH v7 03/12] update-index: add a new --force-write-index option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-21T02:18:26Z","receivedAt":"2017-09-21T02:18:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> On 9/20/2017 9:46 PM, Junio C Hamano wrote:\n>> Ben Peart writes:\n>>\n>>> Lets see how my ascii art skills do at describing this:\n>>\n>> Your ascii art is fine. If you said upfront that the capital\n>> letters signify points in time, lower letters are file-touching\n>> events, and time flows from left to right, it would have been\n>> perfect ;-)\n>\n> Rats, so close and yet... ;-)\n\nNah, sorry for forgetting to add \"... but I could guess that was the\ncase after reading a few paragraphs, at which point I rewound and\nstarted reading from the beginning, and it was crystal clear.\"\n\n> Yes, I suppose we _could_ add a 2nd bit (and then add the logic to set\n> that bit every time a fsmonitor change was made) but I don't see that\n> it really buys us anything useful. The force write flag in\n> update-index is off by default and the only scenario we have that\n> someone would set it is for test cases where the perf of writing out\n> the index when it is not needed just doesn't matter.\n\nI tend to agree now. \n\nMy reaction primarily came from that I couldn't quite tell what the\nIGNORE_* bit was ment to do and assumed that it meant pretty much\nthe same thing as existing ones like \"valid bit is untrustworthy, so\ndo not pay attention to it\". It turns out that this one has quite a\ndifferent meaning, that is not connected to how much we should trust\nstate maintained by the fsmonitor, which force me off-track.\n\nThanks.\n"},{"id":"328526","messageId":"a4ab4766-0367-ff18-a3a9-e48ed49ccd80@gmail.com","threadId":"46154","inReplyTo":"xmqq377gn74p.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-21T02:24:58Z","receivedAt":"2017-09-21T02:25:06Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/20/2017 10:00 PM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> Pretty much the same places you would also use CE_MATCH_IGNORE_VALID\n>> and CE_MATCH_IGNORE_SKIP_WORKTREE which serve the same role for those\n>> features. That is generally when you are about to overwrite data so\n>> want to be *really* sure you have what you think you have.\n> \n> Now that makes me worried gravely.\n> \n> IGNORE_VALID is ignored in these places because we have been burned\n> by end-users lying to us. IGNORE_SKIP_WORKTREE must be ignored\n> because we know that the working tree state does not match the\n> \"reality\" the index wants to have. The fact that the code treats\n> the status reported and kept up to date by fsmonitor the same way as\n> these two implies that it is merely advisory and cannot be trusted?\n> Is that the reason why we tell the codepath with IGNORE_FSMONITOR to\n> ignore the state fsmonitor reported and check the state ourselves?\n> \n\nSorry for causing unnecessary worry. The fsmonitor data can be trusted \n(as much as you can trust that Watchman or your file system monitor is \nnot buggy). I wasn't 100% sure *why* these places passed the various \nIGNORE_VALID and IGNORE_SKIP_WORKTREE flags. When I looked at them, \nthat lack of trust seemed to be the reason.\n\nAdding IGNORE_FSMONITOR in those same places was simply an abundance of \ncaution on my part. The only down side of passing the flag for \nfsmonitor is that we will end up calling lstat() on a file where we \ntechnically didn't need too. That seemed safer than potentially missing \na change if I had misunderstood the code.\n\nI'd much rather return correct results (and fall back to the old \nperformance) than potentially be incorrect. I followed that same \nprincipal in the entire design of fsmonitor - if anything doesn't look \nright, fall back to the old code path just in case...\n\n> Oh, wait...\n> \n> \n>> The other place I used it was in preload_index(). In that case, I\n>> didn't want to trigger the call to refresh_fsmonitor() as\n>> preload_index() is about trying to do a fast precompute of state for\n>> the bulk of the index entries but is not required for correctness as\n>> refresh_cache_ent() will ensure any \"missed\" by preload_index() are\n>> up-to-date if/when that is needed.\n> \n> That is a very valid design decision. So IGNORE_FSMONITOR is,\n> unlike IGNORE_VALID and IGNORE_SKIP_WORKTREE, to tell us \"do not\n> bother asking fsmonitor to refresh the state of this entry--it is OK\n> for us to use a slightly stale information\"? That would make sense\n> as an optimization, but that does not mesh well with the previous\n> \"we need to be really really sure\" usecase. That one wants \"we do\n> not trust fsmonitor, so do not bother asking to refresh; we will do\n> so ourselves\", which would not help the \"we can use slightly stale\n> one and that is OK\" usecase.\n> \n> Puzzled...\n> \n"},{"id":"328527","messageId":"xmqqr2v0lr2e.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"xmqqy3p8lrpp.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v7 03/12] update-index: add a new --force-write-index option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-21T02:32:25Z","receivedAt":"2017-09-21T02:32:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Ben Peart writes:\n>\n>> On 9/20/2017 9:46 PM, Junio C Hamano wrote:\n>>> Ben Peart writes:\n>>>\n>>>> Lets see how my ascii art skills do at describing this:\n>>>\n>>> Your ascii art is fine. If you said upfront that the capital\n>>> letters signify points in time, lower letters are file-touching\n>>> events, and time flows from left to right, it would have been\n>>> perfect ;-)\n>>\n>> Rats, so close and yet... ;-)\n>\n> Nah, sorry for forgetting to add \"... but I could guess that was the\n> case after reading a few paragraphs, at which point I rewound and\n> started reading from the beginning, and it was crystal clear.\"\n>\n>> Yes, I suppose we _could_ add a 2nd bit (and then add the logic to set\n>> that bit every time a fsmonitor change was made) but I don't see that\n>> it really buys us anything useful. The force write flag in\n>> update-index is off by default and the only scenario we have that\n>> someone would set it is for test cases where the perf of writing out\n>> the index when it is not needed just doesn't matter.\n>\n> I tend to agree now. \n>\n> My reaction primarily came from ...\n\noops. please ignore the last paragraph, or transplant it to the\nother thread X-<.\n"},{"id":"328568","messageId":"f50825a4-fa15-9f28-a079-853e78ee8e2e@gmail.com","threadId":"46154","inReplyTo":"a4ab4766-0367-ff18-a3a9-e48ed49ccd80@gmail.com","subject":"Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-09-21T14:35:33Z","receivedAt":"2017-09-21T14:35:43Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/20/2017 10:24 PM, Ben Peart wrote:\n> \n> \n> On 9/20/2017 10:00 PM, Junio C Hamano wrote:\n>> Ben Peart writes:\n>>\n>>> Pretty much the same places you would also use CE_MATCH_IGNORE_VALID\n>>> and CE_MATCH_IGNORE_SKIP_WORKTREE which serve the same role for those\n>>> features.  That is generally when you are about to overwrite data so\n>>> want to be *really* sure you have what you think you have.\n>>\n>> Now that makes me worried gravely.\n>>\n>> IGNORE_VALID is ignored in these places because we have been burned\n>> by end-users lying to us.  IGNORE_SKIP_WORKTREE must be ignored\n>> because we know that the working tree state does not match the\n>> \"reality\" the index wants to have.  The fact that the code treats\n>> the status reported and kept up to date by fsmonitor the same way as\n>> these two implies that it is merely advisory and cannot be trusted?\n>> Is that the reason why we tell the codepath with IGNORE_FSMONITOR to\n>> ignore the state fsmonitor reported and check the state ourselves?\n>>\n> \n> Sorry for causing unnecessary worry.  The fsmonitor data can be trusted \n> (as much as you can trust that Watchman or your file system monitor is \n> not buggy).  I wasn't 100% sure *why* these places passed the various \n> IGNORE_VALID and IGNORE_SKIP_WORKTREE flags.  When I looked at them, \n> that lack of trust seemed to be the reason.\n> \n> Adding IGNORE_FSMONITOR in those same places was simply an abundance of \n> caution on my part.  The only down side of passing the flag for \n> fsmonitor is that we will end up calling lstat() on a file where we \n> technically didn't need too.  That seemed safer than potentially missing \n> a change if I had misunderstood the code.\n> \n> I'd much rather return correct results (and fall back to the old \n> performance) than potentially be incorrect.  I followed that same \n> principal in the entire design of fsmonitor - if anything doesn't look \n> right, fall back to the old code path just in case...\n> \n\nI spent some time with git blame/show trying to figure out the *why* for \nall the places CE_MATCH_IGNORE_* are passed without gaining a lot of \nadditional understanding. Based on your description above of why these \nexist, I believe there are very few places we actually need to pass \nCE_MATCH_IGNORE_FSMONITOR and that I was being overly cautious.\n\nHere is a patch that removes the unnecessary CE_MATCH_IGNORE_FSMONITOR \ninstances. While the test suite passes with this change, I'm not 100% \nconfident that we actually have test cases that would have detected all \nthe places that we needed the CE_MATCH_IGNORE_* flags.\n\nIf this seems like a reasonable additional optimization to make, I can \nroll it into the next iteration of the patch series as I have some \nspelling, documentation changes and other tweaks as a result of all the \nfeedback.\n\n\n From 6ff7ed0467fd736dca73efe62391bb3ee9b4e771 Mon Sep 17 00:00:00 2001\nFrom: Ben Peart \nDate: Thu, 21 Sep 2017 09:09:42 -0400\nSubject: [PATCH] fsmonitor: remove unnecessary uses of\n CE_MATCH_IGNORE_FSMONITOR\n\nWith a better understanding of *why* the CE_MATCH_IGNORE_* flags are\nused, it is now more clear they are not required in most cases where\nCE_MATCH_IGNORE_FSMONITOR was being passed out of an abundance of\ncaution.\n\nSince the fsmonitor data can be trusted and is kept in sync with the\nworking directory, the only remaining valid uses are those locations\nwhere we don't want to trigger an unneeded refresh_fsmonitor() call.\n\nOne is where preload_index() is doing a fast precompute of state for\nthe bulk of the index entries but is not required for correctness as\nrefresh_cache_ent() will ensure any \"missed\" by preload_index() are\nup-to-date if/when they are needed.\n\nThe second is in is_staging_gitmodules_ok() where we don't want to\ntrigger a complete refresh just to check the .gitignore file.\n\nThe net result of this change will be that there are more cases where\nwe will be able to use the cached index state and avoid unnecessary\nlstat() calls.\n\nSigned-off-by: Ben Peart \n---\n apply.c | 2 +-\n entry.c | 2 +-\n read-cache.c | 4 ++--\n unpack-trees.c | 6 +++---\n 4 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 9061cc5f15..71cbbd141c 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -3399,7 +3399,7 @@ static int verify_index_match(const struct \ncache_entry *ce, struct stat *st)\n \t\t\treturn -1;\n \t\treturn 0;\n \t}\n-\treturn ce_match_stat(ce, st, \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n+\treturn ce_match_stat(ce, st, \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n }\n\n #define SUBMODULE_PATCH_WITHOUT_INDEX 1\ndiff --git a/entry.c b/entry.c\nindex 5e6794f9fc..3a7b667373 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -404,7 +404,7 @@ int checkout_entry(struct cache_entry *ce,\n\n \tif (!check_path(path.buf, path.len, &st, state->base_dir_len)) {\n \t\tconst struct submodule *sub;\n-\t\tunsigned changed = ce_match_stat(ce, &st, \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n+\t\tunsigned changed = ce_match_stat(ce, &st, \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n \t\t/*\n \t\t * Needs to be checked before !changed returns early,\n \t\t * as the possibly empty directory was not changed\ndiff --git a/read-cache.c b/read-cache.c\nindex 53093dbebf..05c0a33fdd 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -641,7 +641,7 @@ int add_to_index(struct index_state *istate, const \nchar *path, struct stat *st,\n \tint size, namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias;\n-\tunsigned ce_option = \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR;\n+\tunsigned ce_option = \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n \tint verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);\n \tint pretend = flags & ADD_CACHE_PRETEND;\n \tint intent_only = flags & ADD_CACHE_INTENT;\n@@ -1356,7 +1356,7 @@ int refresh_index(struct index_state *istate, \nunsigned int flags,\n \tint first = 1;\n \tint in_porcelain = (flags & REFRESH_IN_PORCELAIN);\n \tunsigned int options = (CE_MATCH_REFRESH |\n-\t\t\t\t(really ? CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_FSMONITOR : 0) |\n+\t\t\t\t(really ? CE_MATCH_IGNORE_VALID : 0) |\n \t\t\t\t(not_new ? CE_MATCH_IGNORE_MISSING : 0));\n \tconst char *modified_fmt;\n \tconst char *deleted_fmt;\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f724a61ac0..1f5d371636 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1456,7 +1456,7 @@ static int verify_uptodate_1(const struct \ncache_entry *ce,\n \t\treturn 0;\n\n \tif (!lstat(ce->name, &st)) {\n-\t\tint flags = \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR;\n+\t\tint flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE;\n \t\tunsigned changed = ie_match_stat(o->src_index, ce, &st, flags);\n\n \t\tif (submodule_from_ce(ce)) {\n@@ -1612,7 +1612,7 @@ static int icase_exists(struct \nunpack_trees_options *o, const char *name, int le\n \tconst struct cache_entry *src;\n\n \tsrc = index_file_exists(o->src_index, name, len, 1);\n-\treturn src && !ie_match_stat(o->src_index, src, st, \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n+\treturn src && !ie_match_stat(o->src_index, src, st, \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n }\n\n static int check_ok_to_remove(const char *name, int len, int dtype,\n@@ -2136,7 +2136,7 @@ int oneway_merge(const struct cache_entry * const \n*src,\n \t\tif (o->reset && o->update && !ce_uptodate(old) && \n!ce_skip_worktree(old)) {\n \t\t\tstruct stat st;\n \t\t\tif (lstat(old->name, &st) ||\n-\t\t\t ie_match_stat(o->src_index, old, &st, \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR))\n+\t\t\t ie_match_stat(o->src_index, old, &st, \nCE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE))\n \t\t\t\tupdate |= CE_UPDATE;\n \t\t}\n \t\tadd_entry(o, old, update, 0);\n-- \n2.14.1.548.g237ef02b2b.dirty\n\n\n\n>> Oh, wait...\n>>\n>>\n>>> The other place I used it was in preload_index(). In that case, I\n>>> didn't want to trigger the call to refresh_fsmonitor() as\n>>> preload_index() is about trying to do a fast precompute of state for\n>>> the bulk of the index entries but is not required for correctness as\n>>> refresh_cache_ent() will ensure any \"missed\" by preload_index() are\n>>> up-to-date if/when that is needed.\n>>\n>> That is a very valid design decision.  So IGNORE_FSMONITOR is,\n>> unlike IGNORE_VALID and IGNORE_SKIP_WORKTREE, to tell us \"do not\n>> bother asking fsmonitor to refresh the state of this entry--it is OK\n>> for us to use a slightly stale information\"?  That would make sense\n>> as an optimization, but that does not mesh well with the previous\n>> \"we need to be really really sure\" usecase.  That one wants \"we do\n>> not trust fsmonitor, so do not bother asking to refresh; we will do\n>> so ourselves\", which would not help the \"we can use slightly stale\n>> one and that is OK\" usecase.\n>>\n>> Puzzled...\n>>\n"},{"id":"328604","messageId":"xmqqlgl7mtoe.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"f50825a4-fa15-9f28-a079-853e78ee8e2e@gmail.com","subject":"Re: [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-22T01:02:57Z","receivedAt":"2017-09-22T01:03:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> Since the fsmonitor data can be trusted and is kept in sync with the\n> working directory, the only remaining valid uses are those locations\n> where we don't want to trigger an unneeded refresh_fsmonitor() call.\n\nNow that is a lot more assuring ;-) And the exceptions below also\nmake sense to me.\n\nThanks for thinking this through.\n"},{"id":"328605","messageId":"xmqqh8vvmtmd.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170920134952.GA1898@tor.lan","subject":"Re: [PATCH v1 1/1] test-lint: echo -e (or -E) is not portable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-22T01:04:10Z","receivedAt":"2017-09-22T01:04:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n> Junio, if you wouldn't mind to squash that in, \n> another fix is needed as well(trailing '-' after '-E') :\n>\n> s/'-n', '-e' or '-E-'/'-n', '-e' or '-E'\n\nYup. Thanks all.\n\n"},{"id":"328648","messageId":"20170922163548.11288-1-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170919192744.19224-1-benpeart@microsoft.com","subject":"[PATCH v8 00/12] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:36Z","receivedAt":"2017-09-22T16:36:18Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Thanks again to everyone who provided feedback. There are several\nspelling, documentation, and code comment changes.\n\nThe only behavioral change from V7 is the removal of unnecessary uses of\nCE_MATCH_IGNORE_FSMONITOR. With a better understanding of *why* the\nCE_MATCH_IGNORE_* flags are used, it is now clear they are not required\nin most cases where CE_MATCH_IGNORE_FSMONITOR was being passed out of an\nabundance of caution.\n\nSince the fsmonitor data can be trusted and is kept in sync with the\nworking directory, the only remaining valid uses are those locations\nwhere we don't want to trigger an unneeded refresh_fsmonitor() call.\n\nOne is where preload_index() is doing a fast precompute of state for\nthe bulk of the index entries but is not required for correctness as\nrefresh_cache_ent() will ensure any \"missed\" by preload_index() are\nup-to-date if/when they are needed.\n\nThe second is in is_staging_gitmodules_ok() where we don't want to\ntrigger a complete refresh just to check the .gitignore file.\n\nThe net result of this change will be that there are more cases where\nwe will be able to use the cached index state and avoid unnecessary\nlstat() calls.\n\nInterdiff between V7 and V8:\n\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 95231dbfcb..7c2f880a22 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -476,7 +476,7 @@ inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n \n When used in conjunction with the untracked cache, it can further improve\n-performance by avoiding the cost of scaning the entire working directory\n+performance by avoiding the cost of scanning the entire working directory\n looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\ndiff --git a/apply.c b/apply.c\nindex 9061cc5f15..71cbbd141c 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -3399,7 +3399,7 @@ static int verify_index_match(const struct cache_entry *ce, struct stat *st)\n \t\t\treturn -1;\n \t\treturn 0;\n \t}\n-\treturn ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n+\treturn ce_match_stat(ce, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n }\n \n #define SUBMODULE_PATCH_WITHOUT_INDEX 1\ndiff --git a/cache.h b/cache.h\nindex eccab968bd..f1c903e1b6 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -682,7 +682,7 @@ extern void *read_blob_data_from_index(const struct index_state *, const char *,\n #define CE_MATCH_IGNORE_MISSING\t\t0x08\n /* enable stat refresh */\n #define CE_MATCH_REFRESH\t\t0x10\n-/* do stat comparison even if CE_FSMONITOR_VALID is true */\n+/* don't refresh_fsmonitor state or do stat comparison even if CE_FSMONITOR_VALID is true */\n #define CE_MATCH_IGNORE_FSMONITOR 0X20\n extern int ie_match_stat(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n extern int ie_modified(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\ndiff --git a/entry.c b/entry.c\nindex 5e6794f9fc..3a7b667373 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -404,7 +404,7 @@ int checkout_entry(struct cache_entry *ce,\n \n \tif (!check_path(path.buf, path.len, &st, state->base_dir_len)) {\n \t\tconst struct submodule *sub;\n-\t\tunsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n+\t\tunsigned changed = ce_match_stat(ce, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n \t\t/*\n \t\t * Needs to be checked before !changed returns early,\n \t\t * as the possibly empty directory was not changed\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex b8b2d88fe1..7c1540c054 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -176,7 +176,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '*' to indicate all entries are invalid */\n+\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n \tif (query_success && query_result.buf[0] != '/') {\n \t\t/* Mark all entries returned by the monitor as dirty */\n \t\tbuf = query_result.buf;\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex c2240b811a..8eb6163455 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -35,7 +35,9 @@ extern void tweak_fsmonitor(struct index_state *istate);\n extern void refresh_fsmonitor(struct index_state *istate);\n \n /*\n- * Set the given cache entries CE_FSMONITOR_VALID bit.\n+ * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n+ * called any time the cache entry has been updated to reflect the\n+ * current state of the file on disk.\n */\n static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n {\n@@ -46,8 +48,11 @@ static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n }\n \n /*\n- * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate any\n- * corresponding untracked cache directory structures.\n+ * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate\n+ * any corresponding untracked cache directory structures. This should\n+ * be called any time git creates or modifies a file that should\n+ * trigger an lstat() or invalidate the untracked cache for the\n+ * corresponding directory\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\ndiff --git a/read-cache.c b/read-cache.c\nindex 53093dbebf..05c0a33fdd 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -641,7 +641,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \tint size, namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias;\n-\tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR;\n+\tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n \tint verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);\n \tint pretend = flags & ADD_CACHE_PRETEND;\n \tint intent_only = flags & ADD_CACHE_INTENT;\n@@ -1356,7 +1356,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tint first = 1;\n \tint in_porcelain = (flags & REFRESH_IN_PORCELAIN);\n \tunsigned int options = (CE_MATCH_REFRESH |\n-\t\t\t\t(really ? CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_FSMONITOR : 0) |\n+\t\t\t\t(really ? CE_MATCH_IGNORE_VALID : 0) |\n \t\t\t\t(not_new ? CE_MATCH_IGNORE_MISSING : 0));\n \tconst char *modified_fmt;\n \tconst char *deleted_fmt;\ndiff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\nindex 4e5ca8f397..bd1a857d52 100644\n--- a/t/helper/test-drop-caches.c\n+++ b/t/helper/test-drop-caches.c\n@@ -82,6 +82,8 @@ static int cmd_dropcaches(void)\n \tHANDLE hProcess = GetCurrentProcess();\n \tHANDLE hToken;\n \tHMODULE ntdll;\n+\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG);\n+\tSYSTEM_MEMORY_LIST_COMMAND command;\n \tint status;\n \n \tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n@@ -96,12 +98,12 @@ static int cmd_dropcaches(void)\n \tif (!ntdll)\n \t\treturn error(\"Can't load ntdll.dll, wrong Windows version?\");\n \n-\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG) =\n+\tNtSetSystemInformation =\n \t\t(DWORD(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n \tif (!NtSetSystemInformation)\n \t\treturn error(\"Can't get function addresses, wrong Windows version?\");\n \n-\tSYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;\n+\tcommand = MemoryPurgeStandbyList;\n \tstatus = NtSetSystemInformation(\n \t\tSystemMemoryListInformation,\n \t\t&command,\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f724a61ac0..1f5d371636 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1456,7 +1456,7 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n \t\treturn 0;\n \n \tif (!lstat(ce->name, &st)) {\n-\t\tint flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR;\n+\t\tint flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE;\n \t\tunsigned changed = ie_match_stat(o->src_index, ce, &st, flags);\n \n \t\tif (submodule_from_ce(ce)) {\n@@ -1612,7 +1612,7 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le\n \tconst struct cache_entry *src;\n \n \tsrc = index_file_exists(o->src_index, name, len, 1);\n-\treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR);\n+\treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n }\n \n static int check_ok_to_remove(const char *name, int len, int dtype,\n@@ -2136,7 +2136,7 @@ int oneway_merge(const struct cache_entry * const *src,\n \t\tif (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old)) {\n \t\t\tstruct stat st;\n \t\t\tif (lstat(old->name, &st) ||\n-\t\t\t ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_IGNORE_FSMONITOR))\n+\t\t\t ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE))\n \t\t\t\tupdate |= CE_UPDATE;\n \t\t}\n \t\tadd_entry(o, old, update, 0);\n\nBen Peart (12):\n bswap: add 64 bit endianness helper get_be64\n preload-index: add override to enable testing preload-index\n update-index: add a new --force-write-index option\n fsmonitor: teach git to optionally utilize a file system monitor to\n speed up detecting new or changed files.\n fsmonitor: add documentation for the fsmonitor extension.\n ls-files: Add support in ls-files to display the fsmonitor valid bit\n update-index: add fsmonitor support to update-index\n fsmonitor: add a test tool to dump the index extension\n split-index: disable the fsmonitor extension when running the split\n index test\n fsmonitor: add test cases for fsmonitor extension\n fsmonitor: add a sample integration script for Watchman\n fsmonitor: add a performance test\n\n Documentation/config.txt | 7 +\n Documentation/git-ls-files.txt | 7 +-\n Documentation/git-update-index.txt | 45 +++++\n Documentation/githooks.txt | 28 +++\n Documentation/technical/index-format.txt | 19 ++\n Makefile | 3 +\n builtin/ls-files.c | 8 +-\n builtin/update-index.c | 38 +++-\n cache.h | 10 +-\n compat/bswap.h | 22 +++\n config.c | 14 ++\n config.h | 1 +\n diff-lib.c | 2 +\n dir.c | 27 ++-\n dir.h | 2 +\n entry.c | 2 +\n environment.c | 1 +\n fsmonitor.c | 253 ++++++++++++++++++++++++\n fsmonitor.h | 66 +++++++\n preload-index.c | 8 +-\n read-cache.c | 45 ++++-\n submodule.c | 2 +-\n t/helper/.gitignore | 1 +\n t/helper/test-drop-caches.c | 164 ++++++++++++++++\n t/helper/test-dump-fsmonitor.c | 21 ++\n t/perf/p7519-fsmonitor.sh | 184 +++++++++++++++++\n t/t1700-split-index.sh | 1 +\n t/t7519-status-fsmonitor.sh | 304 +++++++++++++++++++++++++++++\n t/t7519/fsmonitor-all | 24 +++\n t/t7519/fsmonitor-none | 22 +++\n t/t7519/fsmonitor-watchman | 140 +++++++++++++\n templates/hooks--fsmonitor-watchman.sample | 122 ++++++++++++\n unpack-trees.c | 2 +\n 33 files changed, 1572 insertions(+), 23 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n create mode 100644 t/helper/test-drop-caches.c\n create mode 100644 t/helper/test-dump-fsmonitor.c\n create mode 100755 t/perf/p7519-fsmonitor.sh\n create mode 100755 t/t7519-status-fsmonitor.sh\n create mode 100755 t/t7519/fsmonitor-all\n create mode 100755 t/t7519/fsmonitor-none\n create mode 100755 t/t7519/fsmonitor-watchman\n create mode 100755 templates/hooks--fsmonitor-watchman.sample\n\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328649","messageId":"20170922163548.11288-3-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 02/12] preload-index: add override to enable testing preload-index","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:38Z","receivedAt":"2017-09-22T16:36:22Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"By default, the preload index code path doesn't run unless there is a\nminimum of 1000 files. To enable running the test suite and having it\nexecute the preload-index path, add an environment variable\n(GIT_FORCE_PRELOAD_TEST) which will override that minimum and set it to 2.\n\nThis enables you run existing tests and have the core.preloadindex code\npath execute as long as the test has at least 2 files by setting\nGIT_FORCE_PRELOAD_TEXT=1 before running the test.\n\nSigned-off-by: Ben Peart \n---\n preload-index.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/preload-index.c b/preload-index.c\nindex 70a4c80878..75564c497a 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -79,6 +79,8 @@ static void preload_index(struct index_state *index,\n \t\treturn;\n \n \tthreads = index->cache_nr / THREAD_COST;\n+\tif ((index->cache_nr > 1) && (threads < 2) && getenv(\"GIT_FORCE_PRELOAD_TEST\"))\n+\t\tthreads = 2;\n \tif (threads < 2)\n \t\treturn;\n \tif (threads > MAX_PARALLEL)\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328650","messageId":"20170922163548.11288-2-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:37Z","receivedAt":"2017-09-22T16:36:24Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a new get_be64 macro to enable 64 bit endian conversions on memory\nthat may or may not be aligned.\n\nSigned-off-by: Ben Peart \n---\n compat/bswap.h | 22 ++++++++++++++++++++++\n 1 file changed, 22 insertions(+)\n\ndiff --git a/compat/bswap.h b/compat/bswap.h\nindex 7d063e9e40..6b22c46214 100644\n--- a/compat/bswap.h\n+++ b/compat/bswap.h\n@@ -158,7 +158,9 @@ static inline uint64_t git_bswap64(uint64_t x)\n \n #define get_be16(p)\tntohs(*(unsigned short *)(p))\n #define get_be32(p)\tntohl(*(unsigned int *)(p))\n+#define get_be64(p)\tntohll(*(uint64_t *)(p))\n #define put_be32(p, v)\tdo { *(unsigned int *)(p) = htonl(v); } while (0)\n+#define put_be64(p, v)\tdo { *(uint64_t *)(p) = htonll(v); } while (0)\n \n #else\n \n@@ -178,6 +180,13 @@ static inline uint32_t get_be32(const void *ptr)\n \t\t(uint32_t)p[3] << 0;\n }\n \n+static inline uint64_t get_be64(const void *ptr)\n+{\n+\tconst unsigned char *p = ptr;\n+\treturn\t(uint64_t)get_be32(p[0]) << 32 |\n+\t\t(uint64_t)get_be32(p[4]) << 0;\n+}\n+\n static inline void put_be32(void *ptr, uint32_t value)\n {\n \tunsigned char *p = ptr;\n@@ -187,4 +196,17 @@ static inline void put_be32(void *ptr, uint32_t value)\n \tp[3] = value >> 0;\n }\n \n+static inline void put_be64(void *ptr, uint64_t value)\n+{\n+\tunsigned char *p = ptr;\n+\tp[0] = value >> 56;\n+\tp[1] = value >> 48;\n+\tp[2] = value >> 40;\n+\tp[3] = value >> 32;\n+\tp[4] = value >> 24;\n+\tp[5] = value >> 16;\n+\tp[6] = value >> 8;\n+\tp[7] = value >> 0;\n+}\n+\n #endif\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328651","messageId":"20170922163548.11288-4-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 03/12] update-index: add a new --force-write-index option","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:39Z","receivedAt":"2017-09-22T16:36:34Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"At times, it makes sense to avoid the cost of writing out the index\nwhen the only changes can easily be recomputed on demand. This causes\nproblems when trying to write test cases to verify that state as they\ncan't guarantee the state has been persisted to disk.\n\nAdd a new option (--force-write-index) to update-index that will\nensure the index is written out even if the cache_changed flag is not\nset.\n\nSigned-off-by: Ben Peart \n---\n builtin/update-index.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex d562f2ec69..e1ca0759d5 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -915,6 +915,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tstruct refresh_params refresh_args = {0, &has_errors};\n \tint lock_error = 0;\n \tint split_index = -1;\n+\tint force_write = 0;\n \tstruct lock_file *lock_file;\n \tstruct parse_opt_ctx_t ctx;\n \tstrbuf_getline_fn getline_fn;\n@@ -1006,6 +1007,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"test if the filesystem supports untracked cache\"), UC_TEST),\n \t\tOPT_SET_INT(0, \"force-untracked-cache\", &untracked_cache,\n \t\t\t N_(\"enable untracked cache without testing the filesystem\"), UC_FORCE),\n+\t\tOPT_SET_INT(0, \"force-write-index\", &force_write,\n+\t\t\tN_(\"write out the index even if is not flagged as changed\"), 1),\n \t\tOPT_END()\n \t};\n \n@@ -1147,7 +1150,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n \t}\n \n-\tif (active_cache_changed) {\n+\tif (active_cache_changed || force_write) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n \t\t\t\texit(128);\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328652","messageId":"20170922163548.11288-12-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 11/12] fsmonitor: add a sample integration script for Watchman","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:47Z","receivedAt":"2017-09-22T16:36:38Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This script integrates the new fsmonitor capabilities of git with the\ncross platform Watchman file watching service. To use the script:\n\nDownload and install Watchman from https://facebook.github.io/watchman/.\nRename the sample integration hook from fsmonitor-watchman.sample to\nfsmonitor-watchman. Configure git to use the extension:\n\ngit config core.fsmonitor .git/hooks/fsmonitor-watchman\n\nOptionally turn on the untracked cache for optimal performance.\n\nSigned-off-by: Ben Peart \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Christian Couder \n---\n templates/hooks--fsmonitor-watchman.sample | 122 +++++++++++++++++++++++++++++\n 1 file changed, 122 insertions(+)\n create mode 100755 templates/hooks--fsmonitor-watchman.sample\n\ndiff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample\nnew file mode 100755\nindex 0000000000..870a59d237\n--- /dev/null\n+++ b/templates/hooks--fsmonitor-watchman.sample\n@@ -0,0 +1,122 @@\n+#!/usr/bin/perl\n+\n+use strict;\n+use warnings;\n+use IPC::Open2;\n+\n+# An example hook script to integrate Watchman\n+# (https://facebook.github.io/watchman/) with git to speed up detecting\n+# new and modified files.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+# To enable this hook, rename this file to \"query-watchman\" and set\n+# 'git config core.fsmonitor .git/hooks/query-watchman'\n+#\n+my ($version, $time) = @ARGV;\n+\n+# Check the hook interface version\n+\n+if ($version == 1) {\n+\t# convert nanoseconds to seconds\n+\t$time = int $time / 1000000000;\n+} else {\n+\tdie \"Unsupported query-fsmonitor hook version '$version'.\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+}\n+\n+# Convert unix style paths to escaped Windows style paths when running\n+# in Windows command prompt\n+\n+my $system = `uname -s`;\n+$system =~ s/[\\r\\n]+//g;\n+my $git_work_tree;\n+\n+if ($system =~ m/^MSYS_NT/) {\n+\t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n+\t$git_work_tree =~ s/[\\r\\n]+//g;\n+\t$git_work_tree =~ s,\\\\,/,g;\n+} else {\n+\t$git_work_tree = $ENV{'PWD'};\n+}\n+\n+my $retry = 1;\n+\n+launch_watchman();\n+\n+sub launch_watchman {\n+\n+\t# Set input record separator\n+\tlocal $/ = 0666;\n+\n+\tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j')\n+\t or die \"open2() failed: $!\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+\n+\t# In the query expression below we're asking for names of files that\n+\t# changed since $time but were not transient (ie created after\n+\t# $time but no longer exist).\n+\t#\n+\t# To accomplish this, we're using the \"since\" generator to use the\n+\t# recency index to select candidate nodes and \"fields\" to limit the\n+\t# output to file names only. Then we're using the \"expression\" term to\n+\t# further constrain the results.\n+\t#\n+\t# The category of transient files that we want to ignore will have a\n+\t# creation clock (cclock) newer than $time_t value and will also not\n+\t# currently exist.\n+\n+\tmy $query = <<\"\tEND\";\n+\t\t[\"query\", \"$git_work_tree\", {\n+\t\t\t\"since\": $time,\n+\t\t\t\"fields\": [\"name\"],\n+\t\t\t\"expression\": [\"not\", [\"allof\", [\"since\", $time, \"cclock\"], [\"not\", \"exists\"]]]\n+\t\t}]\n+\tEND\n+\n+\tprint CHLD_IN $query;\n+\tmy $response = ;\n+\n+\tdie \"Watchman: command returned no output.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $response eq \"\";\n+\tdie \"Watchman: command returned invalid output: $response\\n\" .\n+\t \"Falling back to scanning...\\n\" unless $response =~ /^\\{/;\n+\n+\tmy $json_pkg;\n+\teval {\n+\t\trequire JSON::XS;\n+\t\t$json_pkg = \"JSON::XS\";\n+\t\t1;\n+\t} or do {\n+\t\trequire JSON::PP;\n+\t\t$json_pkg = \"JSON::PP\";\n+\t};\n+\n+\tmy $o = $json_pkg->new->utf8->decode($response);\n+\n+\tif ($retry > 0 and $o->{error} and $o->{error} =~ m/unable to resolve root .* directory (.*) is not watched/) {\n+\t\tprint STDERR \"Adding '$git_work_tree' to watchman's watch list.\\n\";\n+\t\t$retry--;\n+\t\tqx/watchman watch \"$git_work_tree\"/;\n+\t\tdie \"Failed to make watchman watch '$git_work_tree'.\\n\" .\n+\t\t \"Falling back to scanning...\\n\" if $? != 0;\n+\n+\t\t# Watchman will always return all files on the first query so\n+\t\t# return the fast \"everything is dirty\" flag to git and do the\n+\t\t# Watchman query just to get it over with now so we won't pay\n+\t\t# the cost in git to look up each individual file.\n+\t\tprint \"/\\0\";\n+\t\teval { launch_watchman() };\n+\t\texit 0;\n+\t}\n+\n+\tdie \"Watchman: $o->{error}.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $o->{error};\n+\n+\tbinmode STDOUT, \":utf8\";\n+\tlocal $, = \"\\0\";\n+\tprint @{$o->{files}};\n+}\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328653","messageId":"20170922163548.11288-5-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:40Z","receivedAt":"2017-09-22T16:36:42Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"When the index is read from disk, the fsmonitor index extension is used\nto flag the last known potentially dirty index entries. The registered\ncore.fsmonitor command is called with the time the index was last\nupdated and returns the list of files changed since that time. This list\nis used to flag any additional dirty cache entries and untracked cache\ndirectories.\n\nWe can then use this valid state to speed up preload_index(),\nie_match_stat(), and refresh_cache_ent() as they do not need to lstat()\nfiles to detect potential changes for those entries marked\nCE_FSMONITOR_VALID.\n\nIn addition, if the untracked cache is turned on valid_cached_dir() can\nskip checking directories for new or changed files as fsmonitor will\ninvalidate the cache only for those directories that have been\nidentified as having potential changes.\n\nTo keep the CE_FSMONITOR_VALID state accurate during git operations;\nwhen git updates a cache entry to match the current state on disk,\nit will now set the CE_FSMONITOR_VALID bit.\n\nInversely, anytime git changes a cache entry, the CE_FSMONITOR_VALID bit\nis cleared and the corresponding untracked cache directory is marked\ninvalid.\n\nSigned-off-by: Ben Peart \n---\n Makefile | 1 +\n builtin/update-index.c | 2 +\n cache.h | 10 +-\n config.c | 14 +++\n config.h | 1 +\n diff-lib.c | 2 +\n dir.c | 27 ++++--\n dir.h | 2 +\n entry.c | 2 +\n environment.c | 1 +\n fsmonitor.c | 253 +++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor.h | 66 +++++++++++++\n preload-index.c | 6 +-\n read-cache.c | 45 ++++++++-\n submodule.c | 2 +-\n unpack-trees.c | 2 +\n 16 files changed, 417 insertions(+), 19 deletions(-)\n create mode 100644 fsmonitor.c\n create mode 100644 fsmonitor.h\n\ndiff --git a/Makefile b/Makefile\nindex f2bb7f2f63..9d6ec9c1e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -786,6 +786,7 @@ LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n+LIB_OBJS += fsmonitor.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex e1ca0759d5..6f39ee9274 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -16,6 +16,7 @@\n #include \"pathspec.h\"\n #include \"dir.h\"\n #include \"split-index.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Default to not allowing changes to the list of files. The\n@@ -233,6 +234,7 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(&the_index, active_cache[pos]);\n \t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\ndiff --git a/cache.h b/cache.h\nindex a916bc79e3..f1c903e1b6 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -203,6 +203,7 @@ struct cache_entry {\n #define CE_ADDED (1 << 19)\n \n #define CE_HASHED (1 << 20)\n+#define CE_FSMONITOR_VALID (1 << 21)\n #define CE_WT_REMOVE (1 << 22) /* remove in work directory */\n #define CE_CONFLICTED (1 << 23)\n \n@@ -326,6 +327,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n #define UNTRACKED_CHANGED\t(1 << 7)\n+#define FSMONITOR_CHANGED\t(1 << 8)\n \n struct split_index;\n struct untracked_cache;\n@@ -344,6 +346,7 @@ struct index_state {\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n \tstruct untracked_cache *untracked;\n+\tuint64_t fsmonitor_last_update;\n };\n \n extern struct index_state the_index;\n@@ -679,8 +682,10 @@ extern void *read_blob_data_from_index(const struct index_state *, const char *,\n #define CE_MATCH_IGNORE_MISSING\t\t0x08\n /* enable stat refresh */\n #define CE_MATCH_REFRESH\t\t0x10\n-extern int ie_match_stat(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n-extern int ie_modified(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n+/* don't refresh_fsmonitor state or do stat comparison even if CE_FSMONITOR_VALID is true */\n+#define CE_MATCH_IGNORE_FSMONITOR 0X20\n+extern int ie_match_stat(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n+extern int ie_modified(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n \n #define HASH_WRITE_OBJECT 1\n #define HASH_FORMAT_CHECK 2\n@@ -773,6 +778,7 @@ extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n+extern const char *core_fsmonitor;\n \n /*\n * Include broken refs in all ref iterations, which will\ndiff --git a/config.c b/config.c\nindex d0d8ce823a..ddda96e584 100644\n--- a/config.c\n+++ b/config.c\n@@ -2165,6 +2165,20 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n+int git_config_get_fsmonitor(void)\n+{\n+\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n+\t\tcore_fsmonitor = getenv(\"GIT_FSMONITOR_TEST\");\n+\n+\tif (core_fsmonitor && !*core_fsmonitor)\n+\t\tcore_fsmonitor = NULL;\n+\n+\tif (core_fsmonitor)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\ndiff --git a/config.h b/config.h\nindex 97471b8873..c9fcf691ba 100644\n--- a/config.h\n+++ b/config.h\n@@ -211,6 +211,7 @@ extern int git_config_get_pathname(const char *key, const char **dest);\n extern int git_config_get_untracked_cache(void);\n extern int git_config_get_split_index(void);\n extern int git_config_get_max_percent_split_change(void);\n+extern int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n extern int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 2a52b07954..23c6d03ca9 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"submodule.h\"\n #include \"dir.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * diff-files\n@@ -228,6 +229,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \n \t\tif (!changed && !dirty_submodule) {\n \t\t\tce_mark_uptodate(ce);\n+\t\t\tmark_fsmonitor_valid(ce);\n \t\t\tif (!DIFF_OPT_TST(&revs->diffopt, FIND_COPIES_HARDER))\n \t\t\t\tcontinue;\n \t\t}\ndiff --git a/dir.c b/dir.c\nindex 1c55dc3e36..ac9833daec 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -18,6 +18,7 @@\n #include \"utf8.h\"\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Tells read_directory_recursive how a file or directory should be treated.\n@@ -1688,17 +1689,23 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n-\tif (stat(path->len ? path->buf : \".\", &st)) {\n-\t\tinvalidate_directory(dir->untracked, untracked);\n-\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n-\t\treturn 0;\n-\t}\n-\tif (!untracked->valid ||\n-\t match_stat_data_racy(istate, &untracked->stat_data, &st)) {\n-\t\tif (untracked->valid)\n+\t/*\n+\t * With fsmonitor, we can trust the untracked cache's valid field.\n+\t */\n+\trefresh_fsmonitor(istate);\n+\tif (!(dir->untracked->use_fsmonitor && untracked->valid)) {\n+\t\tif (stat(path->len ? path->buf : \".\", &st)) {\n \t\t\tinvalidate_directory(dir->untracked, untracked);\n-\t\tfill_stat_data(&untracked->stat_data, &st);\n-\t\treturn 0;\n+\t\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (!untracked->valid ||\n+\t\t\tmatch_stat_data_racy(istate, &untracked->stat_data, &st)) {\n+\t\t\tif (untracked->valid)\n+\t\t\t\tinvalidate_directory(dir->untracked, untracked);\n+\t\t\tfill_stat_data(&untracked->stat_data, &st);\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n \tif (untracked->check_only != !!check_only) {\ndiff --git a/dir.h b/dir.h\nindex e3717055d1..fab8fc1561 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -139,6 +139,8 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n+\t/* fsmonitor invalidation data */\n+\tunsigned int use_fsmonitor : 1;\n };\n \n struct dir_struct {\ndiff --git a/entry.c b/entry.c\nindex cb291aa88b..3a7b667373 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -4,6 +4,7 @@\n #include \"streaming.h\"\n #include \"submodule.h\"\n #include \"progress.h\"\n+#include \"fsmonitor.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -357,6 +358,7 @@ static int write_entry(struct cache_entry *ce,\n \t\t\tlstat(ce->name, &st);\n \t\tfill_stat_cache_info(ce, &st);\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(state->istate, ce);\n \t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 3fd4b10845..d0b9fc64d4 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -76,6 +76,7 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 0\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n+const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor.c b/fsmonitor.c\nnew file mode 100644\nindex 0000000000..7c1540c054\n--- /dev/null\n+++ b/fsmonitor.c\n@@ -0,0 +1,253 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"ewah/ewok.h\"\n+#include \"fsmonitor.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+\n+#define INDEX_EXTENSION_VERSION\t(1)\n+#define HOOK_INTERFACE_VERSION\t(1)\n+\n+struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR);\n+\n+static void fsmonitor_ewah_callback(size_t pos, void *is)\n+{\n+\tstruct index_state *istate = (struct index_state *)is;\n+\tstruct cache_entry *ce = istate->cache[pos];\n+\n+\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+}\n+\n+int read_fsmonitor_extension(struct index_state *istate, const void *data,\n+\tunsigned long sz)\n+{\n+\tconst char *index = data;\n+\tuint32_t hdr_version;\n+\tuint32_t ewah_size;\n+\tstruct ewah_bitmap *fsmonitor_dirty;\n+\tint i;\n+\tint ret;\n+\n+\tif (sz < sizeof(uint32_t) + sizeof(uint64_t) + sizeof(uint32_t))\n+\t\treturn error(\"corrupt fsmonitor extension (too short)\");\n+\n+\thdr_version = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\tif (hdr_version != INDEX_EXTENSION_VERSION)\n+\t\treturn error(\"bad fsmonitor version %d\", hdr_version);\n+\n+\tistate->fsmonitor_last_update = get_be64(index);\n+\tindex += sizeof(uint64_t);\n+\n+\tewah_size = get_be32(index);\n+\tindex += sizeof(uint32_t);\n+\n+\tfsmonitor_dirty = ewah_new();\n+\tret = ewah_read_mmap(fsmonitor_dirty, index, ewah_size);\n+\tif (ret != ewah_size) {\n+\t\tewah_free(fsmonitor_dirty);\n+\t\treturn error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n+\t}\n+\n+\tif (git_config_get_fsmonitor()) {\n+\t\t/* Mark all entries valid */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n+\n+\t\t/* Mark all previously saved entries as dirty */\n+\t\tewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n+\n+\t\t/* Now mark the untracked cache for fsmonitor usage */\n+\t\tif (istate->untracked)\n+\t\t\tistate->untracked->use_fsmonitor = 1;\n+\t}\n+\tewah_free(fsmonitor_dirty);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"read fsmonitor extension successful\");\n+\treturn 0;\n+}\n+\n+void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n+{\n+\tuint32_t hdr_version;\n+\tuint64_t tm;\n+\tstruct ewah_bitmap *bitmap;\n+\tint i;\n+\tuint32_t ewah_start;\n+\tuint32_t ewah_size = 0;\n+\tint fixup = 0;\n+\n+\tput_be32(&hdr_version, INDEX_EXTENSION_VERSION);\n+\tstrbuf_add(sb, &hdr_version, sizeof(uint32_t));\n+\n+\tput_be64(&tm, istate->fsmonitor_last_update);\n+\tstrbuf_add(sb, &tm, sizeof(uint64_t));\n+\tfixup = sb->len;\n+\tstrbuf_add(sb, &ewah_size, sizeof(uint32_t)); /* we'll fix this up later */\n+\n+\tewah_start = sb->len;\n+\tbitmap = ewah_new();\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))\n+\t\t\tewah_set(bitmap, i);\n+\tewah_serialize_strbuf(bitmap, sb);\n+\tewah_free(bitmap);\n+\n+\t/* fix up size field */\n+\tput_be32(&ewah_size, sb->len - ewah_start);\n+\tmemcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t));\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"write fsmonitor extension successful\");\n+}\n+\n+/*\n+ * Call the query-fsmonitor hook passing the time of the last saved results.\n+ */\n+static int query_fsmonitor(int version, uint64_t last_update, struct strbuf *query_result)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tchar ver[64];\n+\tchar date[64];\n+\tconst char *argv[4];\n+\n+\tif (!(argv[0] = core_fsmonitor))\n+\t\treturn -1;\n+\n+\tsnprintf(ver, sizeof(version), \"%d\", version);\n+\tsnprintf(date, sizeof(date), \"%\" PRIuMAX, (uintmax_t)last_update);\n+\targv[1] = ver;\n+\targv[2] = date;\n+\targv[3] = NULL;\n+\tcp.argv = argv;\n+\tcp.use_shell = 1;\n+\n+\treturn capture_command(&cp, query_result, 1024);\n+}\n+\n+static void fsmonitor_refresh_callback(struct index_state *istate, const char *name)\n+{\n+\tint pos = index_name_pos(istate, name, strlen(name));\n+\n+\tif (pos >= 0) {\n+\t\tstruct cache_entry *ce = istate->cache[pos];\n+\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t}\n+\n+\t/*\n+\t * Mark the untracked cache dirty even if it wasn't found in the index\n+\t * as it could be a new untracked file.\n+\t */\n+\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n+\tuntracked_cache_invalidate_path(istate, name);\n+}\n+\n+void refresh_fsmonitor(struct index_state *istate)\n+{\n+\tstatic int has_run_once = 0;\n+\tstruct strbuf query_result = STRBUF_INIT;\n+\tint query_success = 0;\n+\tsize_t bol; /* beginning of line */\n+\tuint64_t last_update;\n+\tchar *buf;\n+\tint i;\n+\n+\tif (!core_fsmonitor || has_run_once)\n+\t\treturn;\n+\thas_run_once = 1;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\t/*\n+\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * should be inclusive to ensure we don't miss potential changes.\n+\t */\n+\tlast_update = getnanotime();\n+\n+\t/*\n+\t * If we have a last update time, call query_fsmonitor for the set of\n+\t * changes since that time, else assume everything is possibly dirty\n+\t * and check it all.\n+\t */\n+\tif (istate->fsmonitor_last_update) {\n+\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION,\n+\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n+\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n+\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t}\n+\n+\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n+\tif (query_success && query_result.buf[0] != '/') {\n+\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\tbuf = query_result.buf;\n+\t\tbol = 0;\n+\t\tfor (i = 0; i < query_result.len; i++) {\n+\t\t\tif (buf[i] != '\\0')\n+\t\t\t\tcontinue;\n+\t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tbol = i + 1;\n+\t\t}\n+\t\tif (bol < query_result.len)\n+\t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t} else {\n+\t\t/* Mark all entries invalid */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\n+\t\tif (istate->untracked)\n+\t\t\tistate->untracked->use_fsmonitor = 0;\n+\t}\n+\tstrbuf_release(&query_result);\n+\n+\t/* Now that we've updated istate, save the last_update time */\n+\tistate->fsmonitor_last_update = last_update;\n+}\n+\n+void add_fsmonitor(struct index_state *istate)\n+{\n+\tint i;\n+\n+\tif (!istate->fsmonitor_last_update) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"add fsmonitor\");\n+\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\tistate->fsmonitor_last_update = getnanotime();\n+\n+\t\t/* reset the fsmonitor state */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\n+\t\t/* reset the untracked cache */\n+\t\tif (istate->untracked) {\n+\t\t\tadd_untracked_cache(istate);\n+\t\t\tistate->untracked->use_fsmonitor = 1;\n+\t\t}\n+\n+\t\t/* Update the fsmonitor state */\n+\t\trefresh_fsmonitor(istate);\n+\t}\n+}\n+\n+void remove_fsmonitor(struct index_state *istate)\n+{\n+\tif (istate->fsmonitor_last_update) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"remove fsmonitor\");\n+\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\t\tistate->fsmonitor_last_update = 0;\n+\t}\n+}\n+\n+void tweak_fsmonitor(struct index_state *istate)\n+{\n+\tswitch (git_config_get_fsmonitor()) {\n+\tcase -1: /* keep: do nothing */\n+\t\tbreak;\n+\tcase 0: /* false */\n+\t\tremove_fsmonitor(istate);\n+\t\tbreak;\n+\tcase 1: /* true */\n+\t\tadd_fsmonitor(istate);\n+\t\tbreak;\n+\tdefault: /* unknown value: do nothing */\n+\t\tbreak;\n+\t}\n+}\ndiff --git a/fsmonitor.h b/fsmonitor.h\nnew file mode 100644\nindex 0000000000..8eb6163455\n--- /dev/null\n+++ b/fsmonitor.h\n@@ -0,0 +1,66 @@\n+#ifndef FSMONITOR_H\n+#define FSMONITOR_H\n+\n+extern struct trace_key trace_fsmonitor;\n+\n+/*\n+ * Read the the fsmonitor index extension and (if configured) restore the\n+ * CE_FSMONITOR_VALID state.\n+ */\n+extern int read_fsmonitor_extension(struct index_state *istate, const void *data, unsigned long sz);\n+\n+/*\n+ * Write the CE_FSMONITOR_VALID state into the fsmonitor index extension.\n+ */\n+extern void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate);\n+\n+/*\n+ * Add/remove the fsmonitor index extension\n+ */\n+extern void add_fsmonitor(struct index_state *istate);\n+extern void remove_fsmonitor(struct index_state *istate);\n+\n+/*\n+ * Add/remove the fsmonitor index extension as necessary based on the current\n+ * core.fsmonitor setting.\n+ */\n+extern void tweak_fsmonitor(struct index_state *istate);\n+\n+/*\n+ * Run the configured fsmonitor integration script and clear the\n+ * CE_FSMONITOR_VALID bit for any files returned as dirty. Also invalidate\n+ * any corresponding untracked cache directory structures. Optimized to only\n+ * run the first time it is called.\n+ */\n+extern void refresh_fsmonitor(struct index_state *istate);\n+\n+/*\n+ * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n+ * called any time the cache entry has been updated to reflect the\n+ * current state of the file on disk.\n+ */\n+static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n+{\n+\tif (core_fsmonitor) {\n+\t\tce->ce_flags |= CE_FSMONITOR_VALID;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n+\t}\n+}\n+\n+/*\n+ * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate\n+ * any corresponding untracked cache directory structures. This should\n+ * be called any time git creates or modifies a file that should\n+ * trigger an lstat() or invalidate the untracked cache for the\n+ * corresponding directory\n+ */\n+static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n+{\n+\tif (core_fsmonitor) {\n+\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tuntracked_cache_invalidate_path(istate, ce->name);\n+\t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\n+\t}\n+}\n+\n+#endif\ndiff --git a/preload-index.c b/preload-index.c\nindex 75564c497a..2a83255e4e 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -4,6 +4,7 @@\n #include \"cache.h\"\n #include \"pathspec.h\"\n #include \"dir.h\"\n+#include \"fsmonitor.h\"\n \n #ifdef NO_PTHREADS\n static void preload_index(struct index_state *index,\n@@ -55,15 +56,18 @@ static void *preload_thread(void *_data)\n \t\t\tcontinue;\n \t\tif (ce_skip_worktree(ce))\n \t\t\tcontinue;\n+\t\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n+\t\t\tcontinue;\n \t\tif (!ce_path_match(ce, &p->pathspec, NULL))\n \t\t\tcontinue;\n \t\tif (threaded_has_symlink_leading_path(&cache, ce->name, ce_namelen(ce)))\n \t\t\tcontinue;\n \t\tif (lstat(ce->name, &st))\n \t\t\tcontinue;\n-\t\tif (ie_match_stat(index, ce, &st, CE_MATCH_RACY_IS_DIRTY))\n+\t\tif (ie_match_stat(index, ce, &st, CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR))\n \t\t\tcontinue;\n \t\tce_mark_uptodate(ce);\n+\t\tmark_fsmonitor_valid(ce);\n \t} while (--nr > 0);\n \tcache_def_clear(&cache);\n \treturn NULL;\ndiff --git a/read-cache.c b/read-cache.c\nindex 40da87ea71..05c0a33fdd 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -19,6 +19,7 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"utf8.h\"\n+#include \"fsmonitor.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -38,11 +39,12 @@\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n #define CACHE_EXT_UNTRACKED 0x554E5452\t /* \"UNTR\" */\n+#define CACHE_EXT_FSMONITOR 0x46534D4E\t /* \"FSMN\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -62,6 +64,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \tfree(old);\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tmark_fsmonitor_invalid(istate, ce);\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n }\n \n@@ -150,8 +153,10 @@ void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n \tif (assume_unchanged)\n \t\tce->ce_flags |= CE_VALID;\n \n-\tif (S_ISREG(st->st_mode))\n+\tif (S_ISREG(st->st_mode)) {\n \t\tce_mark_uptodate(ce);\n+\t\tmark_fsmonitor_valid(ce);\n+\t}\n }\n \n static int ce_compare_data(const struct cache_entry *ce, struct stat *st)\n@@ -300,7 +305,7 @@ int match_stat_data_racy(const struct index_state *istate,\n \treturn match_stat_data(sd, st);\n }\n \n-int ie_match_stat(const struct index_state *istate,\n+int ie_match_stat(struct index_state *istate,\n \t\t const struct cache_entry *ce, struct stat *st,\n \t\t unsigned int options)\n {\n@@ -308,7 +313,10 @@ int ie_match_stat(const struct index_state *istate,\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n \tint assume_racy_is_modified = options & CE_MATCH_RACY_IS_DIRTY;\n+\tint ignore_fsmonitor = options & CE_MATCH_IGNORE_FSMONITOR;\n \n+\tif (!ignore_fsmonitor)\n+\t\trefresh_fsmonitor(istate);\n \t/*\n \t * If it's marked as always valid in the index, it's\n \t * valid whatever the checked-out copy says.\n@@ -319,6 +327,8 @@ int ie_match_stat(const struct index_state *istate,\n \t\treturn 0;\n \tif (!ignore_valid && (ce->ce_flags & CE_VALID))\n \t\treturn 0;\n+\tif (!ignore_fsmonitor && (ce->ce_flags & CE_FSMONITOR_VALID))\n+\t\treturn 0;\n \n \t/*\n \t * Intent-to-add entries have not been added, so the index entry\n@@ -356,7 +366,7 @@ int ie_match_stat(const struct index_state *istate,\n \treturn changed;\n }\n \n-int ie_modified(const struct index_state *istate,\n+int ie_modified(struct index_state *istate,\n \t\tconst struct cache_entry *ce,\n \t\tstruct stat *st, unsigned int options)\n {\n@@ -777,6 +787,7 @@ int chmod_index_entry(struct index_state *istate, struct cache_entry *ce,\n \t}\n \tcache_tree_invalidate_path(istate, ce->name);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\tmark_fsmonitor_invalid(istate, ce);\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n \n \treturn 0;\n@@ -1228,10 +1239,13 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n \tint ignore_missing = options & CE_MATCH_IGNORE_MISSING;\n+\tint ignore_fsmonitor = options & CE_MATCH_IGNORE_FSMONITOR;\n \n \tif (!refresh || ce_uptodate(ce))\n \t\treturn ce;\n \n+\tif (!ignore_fsmonitor)\n+\t\trefresh_fsmonitor(istate);\n \t/*\n \t * CE_VALID or CE_SKIP_WORKTREE means the user promised us\n \t * that the change to the work tree does not matter and told\n@@ -1245,6 +1259,10 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\tce_mark_uptodate(ce);\n \t\treturn ce;\n \t}\n+\tif (!ignore_fsmonitor && (ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tce_mark_uptodate(ce);\n+\t\treturn ce;\n+\t}\n \n \tif (has_symlink_leading_path(ce->name, ce_namelen(ce))) {\n \t\tif (ignore_missing)\n@@ -1282,8 +1300,10 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\t\t * because CE_UPTODATE flag is in-core only;\n \t\t\t * we are not going to write this change out.\n \t\t\t */\n-\t\t\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\t\tif (!S_ISGITLINK(ce->ce_mode)) {\n \t\t\t\tce_mark_uptodate(ce);\n+\t\t\t\tmark_fsmonitor_valid(ce);\n+\t\t\t}\n \t\t\treturn ce;\n \t\t}\n \t}\n@@ -1391,6 +1411,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\t\t */\n \t\t\t\tce->ce_flags &= ~CE_VALID;\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\t\t\tmark_fsmonitor_invalid(istate, ce);\n \t\t\t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t\t\t}\n \t\t\tif (quiet)\n@@ -1550,6 +1571,9 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_UNTRACKED:\n \t\tistate->untracked = read_untracked_extension(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_FSMONITOR:\n+\t\tread_fsmonitor_extension(istate, data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1722,6 +1746,7 @@ static void post_read_index_from(struct index_state *istate)\n \tcheck_ce_order(istate);\n \ttweak_untracked_cache(istate);\n \ttweak_split_index(istate);\n+\ttweak_fsmonitor(istate);\n }\n \n /* remember to discard_cache() before reading a different cache! */\n@@ -2306,6 +2331,16 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->fsmonitor_last_update) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_fsmonitor_extension(&sb, istate);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_FSMONITOR, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1))\n \t\treturn -1;\ndiff --git a/submodule.c b/submodule.c\nindex 3cea8221e0..8a931a1aaa 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -62,7 +62,7 @@ int is_staging_gitmodules_ok(const struct index_state *istate)\n \tif ((pos >= 0) && (pos < istate->cache_nr)) {\n \t\tstruct stat st;\n \t\tif (lstat(GITMODULES_FILE, &st) == 0 &&\n-\t\t ce_match_stat(istate->cache[pos], &st, 0) & DATA_CHANGED)\n+\t\t ce_match_stat(istate->cache[pos], &st, CE_MATCH_IGNORE_FSMONITOR) & DATA_CHANGED)\n \t\t\treturn 0;\n \t}\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 71b70ccb12..1f5d371636 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -408,6 +409,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n \tif (was_skip_worktree != ce_skip_worktree(ce)) {\n \t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(istate, ce);\n \t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328654","messageId":"20170922163548.11288-11-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 10/12] fsmonitor: add test cases for fsmonitor extension","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:46Z","receivedAt":"2017-09-22T16:36:48Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Test the ability to add/remove the fsmonitor index extension via\nupdate-index.\n\nTest that dirty files returned from the integration script are properly\nrepresented in the index extension and verify that ls-files correctly\nreports their state.\n\nTest that ensure status results are correct when using the new fsmonitor\nextension. Test untracked, modified, and new files by ensuring the\nresults are identical to when not using the extension.\n\nTest that if the fsmonitor extension doesn't tell git about a change, it\ndoesn't discover it on its own. This ensures git is honoring the\nextension and that we get the performance benefits desired.\n\nThree test integration scripts are provided:\n\nfsmonitor-all - marks all files as dirty\nfsmonitor-none - marks no files as dirty\nfsmonitor-watchman - integrates with Watchman with debug logging\n\nTo run tests in the test suite while utilizing fsmonitor:\n\nFirst copy t/t7519/fsmonitor-all to a location in your path and then set\nGIT_FORCE_PRELOAD_TEST=true and GIT_FSMONITOR_TEST=fsmonitor-all and run\nyour tests.\n\nNote: currently when using the test script fsmonitor-watchman on\nWindows, many tests fail due to a reported but not yet fixed bug in\nWatchman where it holds on to handles for directories and files which\nprevents the test directory from being cleaned up properly.\n\nSigned-off-by: Ben Peart \n---\n t/t7519-status-fsmonitor.sh | 304 ++++++++++++++++++++++++++++++++++++++++++++\n t/t7519/fsmonitor-all | 24 ++++\n t/t7519/fsmonitor-none | 22 ++++\n t/t7519/fsmonitor-watchman | 140 ++++++++++++++++++++\n 4 files changed, 490 insertions(+)\n create mode 100755 t/t7519-status-fsmonitor.sh\n create mode 100755 t/t7519/fsmonitor-all\n create mode 100755 t/t7519/fsmonitor-none\n create mode 100755 t/t7519/fsmonitor-watchman\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nnew file mode 100755\nindex 0000000000..c6df85af5e\n--- /dev/null\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -0,0 +1,304 @@\n+#!/bin/sh\n+\n+test_description='git status with file system watcher'\n+\n+. ./test-lib.sh\n+\n+#\n+# To run the entire git test suite using fsmonitor:\n+#\n+# copy t/t7519/fsmonitor-all to a location in your path and then set\n+# GIT_FSMONITOR_TEST=fsmonitor-all and run your tests.\n+#\n+\n+# Note, after \"git reset --hard HEAD\" no extensions exist other than 'TREE'\n+# \"git update-index --fsmonitor\" can be used to get the extension written\n+# before testing the results.\n+\n+clean_repo () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd\n+}\n+\n+dirty_repo () {\n+\t: >untracked &&\n+\t: >dir1/untracked &&\n+\t: >dir2/untracked &&\n+\techo 1 >modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\techo 4 >new &&\n+\techo 5 >dir1/new &&\n+\techo 6 >dir2/new\n+}\n+\n+write_integration_script () {\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF\n+\tif test \"$#\" -ne 2\n+\tthen\n+\t\techo \"$0: exactly 2 arguments expected\"\n+\t\texit 2\n+\tfi\n+\tif test \"$1\" != 1\n+\tthen\n+\t\techo \"Unsupported core.fsmonitor hook version.\" >&2\n+\t\texit 1\n+\tfi\n+\tprintf \"untracked\\0\"\n+\tprintf \"dir1/untracked\\0\"\n+\tprintf \"dir2/untracked\\0\"\n+\tprintf \"modified\\0\"\n+\tprintf \"dir1/modified\\0\"\n+\tprintf \"dir2/modified\\0\"\n+\tprintf \"new\\0\"\n+\tprintf \"dir1/new\\0\"\n+\tprintf \"dir2/new\\0\"\n+\tEOF\n+}\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_expect_success 'setup' '\n+\tmkdir -p .git/hooks &&\n+\t: >tracked &&\n+\t: >modified &&\n+\tmkdir dir1 &&\n+\t: >dir1/tracked &&\n+\t: >dir1/modified &&\n+\tmkdir dir2 &&\n+\t: >dir2/tracked &&\n+\t: >dir2/modified &&\n+\tgit -c core.fsmonitor= add . &&\n+\tgit -c core.fsmonitor= commit -m initial &&\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcat >.gitignore <<-\\EOF\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tmarker*\n+\tEOF\n+'\n+\n+# test that the fsmonitor extension is off by default\n+test_expect_success 'fsmonitor extension is off by default' '\n+\ttest-dump-fsmonitor >actual &&\n+\tgrep \"^no fsmonitor\" actual\n+'\n+\n+# test that \"update-index --fsmonitor\" adds the fsmonitor extension\n+test_expect_success 'update-index --fsmonitor\" adds the fsmonitor extension' '\n+\tgit update-index --fsmonitor &&\n+\ttest-dump-fsmonitor >actual &&\n+\tgrep \"^fsmonitor last update\" actual\n+'\n+\n+# test that \"update-index --no-fsmonitor\" removes the fsmonitor extension\n+test_expect_success 'update-index --no-fsmonitor\" removes the fsmonitor extension' '\n+\tgit update-index --no-fsmonitor &&\n+\ttest-dump-fsmonitor >actual &&\n+\tgrep \"^no fsmonitor\" actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+cat >expect <modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\ttest_must_fail git update-index --refresh --force-write-index &&\n+\tgit ls-files -f >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# Ensure commands that call refresh_index() to move the index back in time\n+# properly invalidate the fsmonitor cache\n+test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\tEOF\n+\tclean_repo &&\n+\tdirty_repo &&\n+\tgit add . &&\n+\tgit commit -m \"to reset\" &&\n+\tgit reset HEAD~1 &&\n+\tgit status >actual &&\n+\tgit -c core.fsmonitor= status >expect &&\n+\ttest_i18ncmp expect actual\n+'\n+\n+# test fsmonitor with and without preloadIndex\n+preload_values=\"false true\"\n+for preload_val in $preload_values\n+do\n+\ttest_expect_success \"setup preloadIndex to $preload_val\" '\n+\t\tgit config core.preloadIndex $preload_val &&\n+\t\tif test $preload_val = true\n+\t\tthen\n+\t\t\tGIT_FORCE_PRELOAD_TEST=$preload_val; export GIT_FORCE_PRELOAD_TEST\n+\t\telse\n+\t\t\tunset GIT_FORCE_PRELOAD_TEST\n+\t\tfi\n+\t'\n+\n+\t# test fsmonitor with and without the untracked cache (if available)\n+\tuc_values=\"false\"\n+\ttest_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\ttest_expect_success \"setup untracked cache to $uc_val\" '\n+\t\t\tgit config core.untrackedcache $uc_val\n+\t\t'\n+\n+\t\t# Status is well tested elsewhere so we'll just ensure that the results are\n+\t\t# the same when using core.fsmonitor.\n+\t\ttest_expect_success 'compare status with and without fsmonitor' '\n+\t\t\twrite_integration_script &&\n+\t\t\tclean_repo &&\n+\t\t\tdirty_repo &&\n+\t\t\tgit add new &&\n+\t\t\tgit add dir1/new &&\n+\t\t\tgit add dir2/new &&\n+\t\t\tgit status >actual &&\n+\t\t\tgit -c core.fsmonitor= status >expect &&\n+\t\t\ttest_i18ncmp expect actual\n+\t\t'\n+\n+\t\t# Make sure it's actually skipping the check for modified and untracked\n+\t\t# (if enabled) files unless it is told about them.\n+\t\ttest_expect_success \"status doesn't detect unreported modifications\" '\n+\t\t\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\t\t:>marker\n+\t\t\tEOF\n+\t\t\tclean_repo &&\n+\t\t\tgit status &&\n+\t\t\ttest_path_is_file marker &&\n+\t\t\tdirty_repo &&\n+\t\t\trm -f marker &&\n+\t\t\tgit status >actual &&\n+\t\t\ttest_path_is_file marker &&\n+\t\t\ttest_i18ngrep ! \"Changes not staged for commit:\" actual &&\n+\t\t\tif test $uc_val = true\n+\t\t\tthen\n+\t\t\t\ttest_i18ngrep ! \"Untracked files:\" actual\n+\t\t\tfi &&\n+\t\t\tif test $uc_val = false\n+\t\t\tthen\n+\t\t\t\ttest_i18ngrep \"Untracked files:\" actual\n+\t\t\tfi &&\n+\t\t\trm -f marker\n+\t\t'\n+\tdone\n+done\n+\n+test_done\ndiff --git a/t/t7519/fsmonitor-all b/t/t7519/fsmonitor-all\nnew file mode 100755\nindex 0000000000..691bc94dc2\n--- /dev/null\n+++ b/t/t7519/fsmonitor-all\n@@ -0,0 +1,24 @@\n+#!/bin/sh\n+#\n+# An test hook script to integrate with git to test fsmonitor.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+#echo \"$0 $*\" >&2\n+\n+if test \"$#\" -ne 2\n+then\n+\techo \"$0: exactly 2 arguments expected\" >&2\n+\texit 2\n+fi\n+\n+if test \"$1\" != 1\n+then\n+\techo \"Unsupported core.fsmonitor hook version.\" >&2\n+\texit 1\n+fi\n+\n+echo \"/\"\ndiff --git a/t/t7519/fsmonitor-none b/t/t7519/fsmonitor-none\nnew file mode 100755\nindex 0000000000..ed9cf5a6a9\n--- /dev/null\n+++ b/t/t7519/fsmonitor-none\n@@ -0,0 +1,22 @@\n+#!/bin/sh\n+#\n+# An test hook script to integrate with git to test fsmonitor.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+#echo \"$0 $*\" >&2\n+\n+if test \"$#\" -ne 2\n+then\n+\techo \"$0: exactly 2 arguments expected\" >&2\n+\texit 2\n+fi\n+\n+if test \"$1\" != 1\n+then\n+\techo \"Unsupported core.fsmonitor hook version.\" >&2\n+\texit 1\n+fi\ndiff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\nnew file mode 100755\nindex 0000000000..7ceb32dc18\n--- /dev/null\n+++ b/t/t7519/fsmonitor-watchman\n@@ -0,0 +1,140 @@\n+#!/usr/bin/perl\n+\n+use strict;\n+use warnings;\n+use IPC::Open2;\n+\n+# An example hook script to integrate Watchman\n+# (https://facebook.github.io/watchman/) with git to speed up detecting\n+# new and modified files.\n+#\n+# The hook is passed a version (currently 1) and a time in nanoseconds\n+# formatted as a string and outputs to stdout all files that have been\n+# modified since the given time. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+# To enable this hook, rename this file to \"query-watchman\" and set\n+# 'git config core.fsmonitor .git/hooks/query-watchman'\n+#\n+my ($version, $time) = @ARGV;\n+#print STDERR \"$0 $version $time\\n\";\n+\n+# Check the hook interface version\n+\n+if ($version == 1) {\n+\t# convert nanoseconds to seconds\n+\t$time = int $time / 1000000000;\n+} else {\n+\tdie \"Unsupported query-fsmonitor hook version '$version'.\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+}\n+\n+# Convert unix style paths to escaped Windows style paths when running\n+# in Windows command prompt\n+\n+my $system = `uname -s`;\n+$system =~ s/[\\r\\n]+//g;\n+my $git_work_tree;\n+\n+if ($system =~ m/^MSYS_NT/) {\n+\t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n+\t$git_work_tree =~ s/[\\r\\n]+//g;\n+\t$git_work_tree =~ s,\\\\,/,g;\n+} else {\n+\t$git_work_tree = $ENV{'PWD'};\n+}\n+\n+my $retry = 1;\n+\n+launch_watchman();\n+\n+sub launch_watchman {\n+\n+\t# Set input record separator\n+\tlocal $/ = 0666;\n+\n+\tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j')\n+\t or die \"open2() failed: $!\\n\" .\n+\t \"Falling back to scanning...\\n\";\n+\n+\t# In the query expression below we're asking for names of files that\n+\t# changed since $time but were not transient (ie created after\n+\t# $time but no longer exist).\n+\t#\n+\t# To accomplish this, we're using the \"since\" generator to use the\n+\t# recency index to select candidate nodes and \"fields\" to limit the\n+\t# output to file names only. Then we're using the \"expression\" term to\n+\t# further constrain the results.\n+\t#\n+\t# The category of transient files that we want to ignore will have a\n+\t# creation clock (cclock) newer than $time_t value and will also not\n+\t# currently exist.\n+\n+\tmy $query = <<\"\tEND\";\n+\t\t[\"query\", \"$git_work_tree\", {\n+\t\t\t\"since\": $time,\n+\t\t\t\"fields\": [\"name\"],\n+\t\t\t\"expression\": [\"not\", [\"allof\", [\"since\", $time, \"cclock\"], [\"not\", \"exists\"]]]\n+\t\t}]\n+\tEND\n+\t\n+\topen (my $fh, \">\", \".git/watchman-query.json\");\n+\tprint $fh $query;\n+\tclose $fh;\n+\n+\tprint CHLD_IN $query;\n+\tmy $response = ;\n+\n+\topen ($fh, \">\", \".git/watchman-response.json\");\n+\tprint $fh $response;\n+\tclose $fh;\n+\n+\tdie \"Watchman: command returned no output.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $response eq \"\";\n+\tdie \"Watchman: command returned invalid output: $response\\n\" .\n+\t \"Falling back to scanning...\\n\" unless $response =~ /^\\{/;\n+\n+\tmy $json_pkg;\n+\teval {\n+\t\trequire JSON::XS;\n+\t\t$json_pkg = \"JSON::XS\";\n+\t\t1;\n+\t} or do {\n+\t\trequire JSON::PP;\n+\t\t$json_pkg = \"JSON::PP\";\n+\t};\n+\n+\tmy $o = $json_pkg->new->utf8->decode($response);\n+\n+\tif ($retry > 0 and $o->{error} and $o->{error} =~ m/unable to resolve root .* directory (.*) is not watched/) {\n+\t\tprint STDERR \"Adding '$git_work_tree' to watchman's watch list.\\n\";\n+\t\t$retry--;\n+\t\tqx/watchman watch \"$git_work_tree\"/;\n+\t\tdie \"Failed to make watchman watch '$git_work_tree'.\\n\" .\n+\t\t \"Falling back to scanning...\\n\" if $? != 0;\n+\n+\t\t# Watchman will always return all files on the first query so\n+\t\t# return the fast \"everything is dirty\" flag to git and do the\n+\t\t# Watchman query just to get it over with now so we won't pay\n+\t\t# the cost in git to look up each individual file.\n+\n+\t\topen ($fh, \">\", \".git/watchman-output.out\");\n+\t\tprint \"/\\0\";\n+\t\tclose $fh;\n+\n+\t\tprint \"/\\0\";\n+\t\teval { launch_watchman() };\n+\t\texit 0;\n+\t}\n+\n+\tdie \"Watchman: $o->{error}.\\n\" .\n+\t \"Falling back to scanning...\\n\" if $o->{error};\n+\n+\topen ($fh, \">\", \".git/watchman-output.out\");\n+\tprint $fh @{$o->{files}};\n+\tclose $fh;\n+\n+\tbinmode STDOUT, \":utf8\";\n+\tlocal $, = \"\\0\";\n+\tprint @{$o->{files}};\n+}\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328655","messageId":"20170922163548.11288-10-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 09/12] split-index: disable the fsmonitor extension when running the split index test","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:45Z","receivedAt":"2017-09-22T16:36:50Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"The split index test t1700-split-index.sh has hard coded SHA values for\nthe index. Currently it supports index V4 and V3 but assumes there are\nno index extensions loaded.\n\nWhen manually forcing the fsmonitor extension to be turned on when\nrunning the test suite, the SHA values no longer match which causes the\ntest to fail.\n\nThe potential matrix of index extensions and index versions can is quite\nlarge so instead temporarily disable the extension before attempting to\nrun the test until the underlying problem of hard coded SHA values is fixed.\n\nSigned-off-by: Ben Peart \n---\n t/t1700-split-index.sh | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex 22f69a410b..af9b847761 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -6,6 +6,7 @@ test_description='split index mode tests'\n \n # We need total control of index splitting here\n sane_unset GIT_TEST_SPLIT_INDEX\n+sane_unset GIT_FSMONITOR_TEST\n \n test_expect_success 'enable split index' '\n \tgit config splitIndex.maxPercentChange 100 &&\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328656","messageId":"20170922163548.11288-13-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 12/12] fsmonitor: add a performance test","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:48Z","receivedAt":"2017-09-22T16:36:54Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a test utility (test-drop-caches) that flushes all changes to disk\nthen drops file system cache on Windows, Linux, and OSX.\n\nAdd a perf test (p7519-fsmonitor.sh) for fsmonitor.\n\nBy default, the performance test will utilize the Watchman file system\nmonitor if it is installed. If Watchman is not installed, it will use a\ndummy integration script that does not report any new or modified files.\nThe dummy script has very little overhead which provides optimistic results.\n\nThe performance test will also use the untracked cache feature if it is\navailable as fsmonitor uses it to speed up scanning for untracked files.\n\nThere are 4 environment variables that can be used to alter the default\nbehavior of the performance test:\n\nGIT_PERF_7519_UNTRACKED_CACHE: used to configure core.untrackedCache\nGIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\nGIT_PERF_7519_FSMONITOR: used to configure core.fsmonitor\nGIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests\n\nThe big win for using fsmonitor is the elimination of the need to scan the\nworking directory looking for changed and untracked files. If the file\ninformation is all cached in RAM, the benefits are reduced.\n\nSigned-off-by: Ben Peart \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 1 +\n t/helper/.gitignore | 1 +\n t/helper/test-drop-caches.c | 164 +++++++++++++++++++++++++++++++++++++++\n t/perf/p7519-fsmonitor.sh | 184 ++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 350 insertions(+)\n create mode 100644 t/helper/test-drop-caches.c\n create mode 100755 t/perf/p7519-fsmonitor.sh\n\ndiff --git a/Makefile b/Makefile\nindex d970cd00e9..b2653ee64f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -638,6 +638,7 @@ TEST_PROGRAMS_NEED_X += test-ctype\n TEST_PROGRAMS_NEED_X += test-config\n TEST_PROGRAMS_NEED_X += test-date\n TEST_PROGRAMS_NEED_X += test-delta\n+TEST_PROGRAMS_NEED_X += test-drop-caches\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n TEST_PROGRAMS_NEED_X += test-dump-fsmonitor\n TEST_PROGRAMS_NEED_X += test-dump-split-index\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 721650256e..f9328eebdd 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -3,6 +3,7 @@\n /test-config\n /test-date\n /test-delta\n+/test-drop-caches\n /test-dump-cache-tree\n /test-dump-split-index\n /test-dump-untracked-cache\ndiff --git a/t/helper/test-drop-caches.c b/t/helper/test-drop-caches.c\nnew file mode 100644\nindex 0000000000..bd1a857d52\n--- /dev/null\n+++ b/t/helper/test-drop-caches.c\n@@ -0,0 +1,164 @@\n+#include \"git-compat-util.h\"\n+\n+#if defined(GIT_WINDOWS_NATIVE)\n+\n+static int cmd_sync(void)\n+{\n+\tchar Buffer[MAX_PATH];\n+\tDWORD dwRet;\n+\tchar szVolumeAccessPath[] = \"\\\\\\\\.\\\\X:\";\n+\tHANDLE hVolWrite;\n+\tint success = 0;\n+\n+\tdwRet = GetCurrentDirectory(MAX_PATH, Buffer);\n+\tif ((0 == dwRet) || (dwRet > MAX_PATH))\n+\t\treturn error(\"Error getting current directory\");\n+\n+\tif ((Buffer[0] < 'A') || (Buffer[0] > 'Z'))\n+\t\treturn error(\"Invalid drive letter '%c'\", Buffer[0]);\n+\n+\tszVolumeAccessPath[4] = Buffer[0];\n+\thVolWrite = CreateFile(szVolumeAccessPath, GENERIC_READ | GENERIC_WRITE,\n+\t\tFILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);\n+\tif (INVALID_HANDLE_VALUE == hVolWrite)\n+\t\treturn error(\"Unable to open volume for writing, need admin access\");\n+\n+\tsuccess = FlushFileBuffers(hVolWrite);\n+\tif (!success)\n+\t\terror(\"Unable to flush volume\");\n+\n+\tCloseHandle(hVolWrite);\n+\n+\treturn !success;\n+}\n+\n+#define STATUS_SUCCESS\t\t\t(0x00000000L)\n+#define STATUS_PRIVILEGE_NOT_HELD\t(0xC0000061L)\n+\n+typedef enum _SYSTEM_INFORMATION_CLASS {\n+\tSystemMemoryListInformation = 80,\n+} SYSTEM_INFORMATION_CLASS;\n+\n+typedef enum _SYSTEM_MEMORY_LIST_COMMAND {\n+\tMemoryCaptureAccessedBits,\n+\tMemoryCaptureAndResetAccessedBits,\n+\tMemoryEmptyWorkingSets,\n+\tMemoryFlushModifiedList,\n+\tMemoryPurgeStandbyList,\n+\tMemoryPurgeLowPriorityStandbyList,\n+\tMemoryCommandMax\n+} SYSTEM_MEMORY_LIST_COMMAND;\n+\n+static BOOL GetPrivilege(HANDLE TokenHandle, LPCSTR lpName, int flags)\n+{\n+\tBOOL bResult;\n+\tDWORD dwBufferLength;\n+\tLUID luid;\n+\tTOKEN_PRIVILEGES tpPreviousState;\n+\tTOKEN_PRIVILEGES tpNewState;\n+\n+\tdwBufferLength = 16;\n+\tbResult = LookupPrivilegeValueA(0, lpName, &luid);\n+\tif (bResult) {\n+\t\ttpNewState.PrivilegeCount = 1;\n+\t\ttpNewState.Privileges[0].Luid = luid;\n+\t\ttpNewState.Privileges[0].Attributes = 0;\n+\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpNewState,\n+\t\t\t(DWORD)((LPBYTE)&(tpNewState.Privileges[1]) - (LPBYTE)&tpNewState),\n+\t\t\t&tpPreviousState, &dwBufferLength);\n+\t\tif (bResult) {\n+\t\t\ttpPreviousState.PrivilegeCount = 1;\n+\t\t\ttpPreviousState.Privileges[0].Luid = luid;\n+\t\t\ttpPreviousState.Privileges[0].Attributes = flags != 0 ? 2 : 0;\n+\t\t\tbResult = AdjustTokenPrivileges(TokenHandle, 0, &tpPreviousState,\n+\t\t\t\tdwBufferLength, 0, 0);\n+\t\t}\n+\t}\n+\treturn bResult;\n+}\n+\n+static int cmd_dropcaches(void)\n+{\n+\tHANDLE hProcess = GetCurrentProcess();\n+\tHANDLE hToken;\n+\tHMODULE ntdll;\n+\tDWORD(WINAPI *NtSetSystemInformation)(INT, PVOID, ULONG);\n+\tSYSTEM_MEMORY_LIST_COMMAND command;\n+\tint status;\n+\n+\tif (!OpenProcessToken(hProcess, TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))\n+\t\treturn error(\"Can't open current process token\");\n+\n+\tif (!GetPrivilege(hToken, \"SeProfileSingleProcessPrivilege\", 1))\n+\t\treturn error(\"Can't get SeProfileSingleProcessPrivilege\");\n+\n+\tCloseHandle(hToken);\n+\n+\tntdll = LoadLibrary(\"ntdll.dll\");\n+\tif (!ntdll)\n+\t\treturn error(\"Can't load ntdll.dll, wrong Windows version?\");\n+\n+\tNtSetSystemInformation =\n+\t\t(DWORD(WINAPI *)(INT, PVOID, ULONG))GetProcAddress(ntdll, \"NtSetSystemInformation\");\n+\tif (!NtSetSystemInformation)\n+\t\treturn error(\"Can't get function addresses, wrong Windows version?\");\n+\n+\tcommand = MemoryPurgeStandbyList;\n+\tstatus = NtSetSystemInformation(\n+\t\tSystemMemoryListInformation,\n+\t\t&command,\n+\t\tsizeof(SYSTEM_MEMORY_LIST_COMMAND)\n+\t);\n+\tif (status == STATUS_PRIVILEGE_NOT_HELD)\n+\t\terror(\"Insufficient privileges to purge the standby list, need admin access\");\n+\telse if (status != STATUS_SUCCESS)\n+\t\terror(\"Unable to execute the memory list command %d\", status);\n+\n+\tFreeLibrary(ntdll);\n+\n+\treturn status;\n+}\n+\n+#elif defined(__linux__)\n+\n+static int cmd_sync(void)\n+{\n+\treturn system(\"sync\");\n+}\n+\n+static int cmd_dropcaches(void)\n+{\n+\treturn system(\"echo 3 | sudo tee /proc/sys/vm/drop_caches\");\n+}\n+\n+#elif defined(__APPLE__)\n+\n+static int cmd_sync(void)\n+{\n+\treturn system(\"sync\");\n+}\n+\n+static int cmd_dropcaches(void)\n+{\n+\treturn system(\"sudo purge\");\n+}\n+\n+#else\n+\n+static int cmd_sync(void)\n+{\n+\treturn 0;\n+}\n+\n+static int cmd_dropcaches(void)\n+{\n+\treturn error(\"drop caches not implemented on this platform\");\n+}\n+\n+#endif\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tcmd_sync();\n+\treturn cmd_dropcaches();\n+}\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nnew file mode 100755\nindex 0000000000..16d1bf72e5\n--- /dev/null\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -0,0 +1,184 @@\n+#!/bin/sh\n+\n+test_description=\"Test core.fsmonitor\"\n+\n+. ./perf-lib.sh\n+\n+#\n+# Performance test for the fsmonitor feature which enables git to talk to a\n+# file system change monitor and avoid having to scan the working directory\n+# for new or modified files.\n+#\n+# By default, the performance test will utilize the Watchman file system\n+# monitor if it is installed. If Watchman is not installed, it will use a\n+# dummy integration script that does not report any new or modified files.\n+# The dummy script has very little overhead which provides optimistic results.\n+#\n+# The performance test will also use the untracked cache feature if it is\n+# available as fsmonitor uses it to speed up scanning for untracked files.\n+#\n+# There are 3 environment variables that can be used to alter the default\n+# behavior of the performance test:\n+#\n+# GIT_PERF_7519_UNTRACKED_CACHE: used to configure core.untrackedCache\n+# GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\n+# GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor\n+#\n+# The big win for using fsmonitor is the elimination of the need to scan the\n+# working directory looking for changed and untracked files. If the file\n+# information is all cached in RAM, the benefits are reduced.\n+#\n+# GIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests\n+#\n+\n+test_perf_large_repo\n+test_checkout_worktree\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_lazy_prereq WATCHMAN '\n+\t{ command -v watchman >/dev/null 2>&1; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+if test_have_prereq WATCHMAN\n+then\n+\t# Convert unix style paths to escaped Windows style paths for Watchman\n+\tcase \"$(uname -s)\" in\n+\tMSYS_NT*)\n+\t GIT_WORK_TREE=\"$(cygpath -aw \"$PWD\" | sed 's,\\\\,/,g')\"\n+\t ;;\n+\t*)\n+\t GIT_WORK_TREE=\"$PWD\"\n+\t ;;\n+\tesac\n+fi\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"\n+then\n+\t# When using GIT_PERF_7519_DROP_CACHE, GIT_PERF_REPEAT_COUNT must be 1 to\n+\t# generate valid results. Otherwise the caching that happens for the nth\n+\t# run will negate the validity of the comparisons.\n+\tif test \"$GIT_PERF_REPEAT_COUNT\" -ne 1\n+\tthen\n+\t\techo \"warning: Setting GIT_PERF_REPEAT_COUNT=1\" >&2\n+\t\tGIT_PERF_REPEAT_COUNT=1\n+\tfi\n+fi\n+\n+test_expect_success \"setup for fsmonitor\" '\n+\t# set untrackedCache depending on the environment\n+\tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n+\tthen\n+\t\tgit config core.untrackedCache \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n+\telse\n+\t\tif test_have_prereq UNTRACKED_CACHE\n+\t\tthen\n+\t\t\tgit config core.untrackedCache true\n+\t\telse\n+\t\t\tgit config core.untrackedCache false\n+\t\tfi\n+\tfi &&\n+\n+\t# set core.splitindex depending on the environment\n+\tif test -n \"$GIT_PERF_7519_SPLIT_INDEX\"\n+\tthen\n+\t\tgit config core.splitIndex \"$GIT_PERF_7519_SPLIT_INDEX\"\n+\tfi &&\n+\n+\t# set INTEGRATION_SCRIPT depending on the environment\n+\tif test -n \"$GIT_PERF_7519_FSMONITOR\"\n+\tthen\n+\t\tINTEGRATION_SCRIPT=\"$GIT_PERF_7519_FSMONITOR\"\n+\telse\n+\t\t#\n+\t\t# Choose integration script based on existence of Watchman.\n+\t\t# If Watchman exists, watch the work tree and attempt a query.\n+\t\t# If everything succeeds, use Watchman integration script,\n+\t\t# else fall back to an empty integration script.\n+\t\t#\n+\t\tmkdir .git/hooks &&\n+\t\tif test_have_prereq WATCHMAN\n+\t\tthen\n+\t\t\tINTEGRATION_SCRIPT=\".git/hooks/fsmonitor-watchman\" &&\n+\t\t\tcp \"$TEST_DIRECTORY/../templates/hooks--fsmonitor-watchman.sample\" \"$INTEGRATION_SCRIPT\" &&\n+\t\t\twatchman watch \"$GIT_WORK_TREE\" &&\n+\t\t\twatchman watch-list | grep -q -F \"$GIT_WORK_TREE\"\n+\t\telse\n+\t\t\tINTEGRATION_SCRIPT=\".git/hooks/fsmonitor-empty\" &&\n+\t\t\twrite_script \"$INTEGRATION_SCRIPT\"<<-\\EOF\n+\t\t\tEOF\n+\t\tfi\n+\tfi &&\n+\n+\tgit config core.fsmonitor \"$INTEGRATION_SCRIPT\" &&\n+\tgit update-index --fsmonitor\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uno (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uno\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uall (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uall\n+'\n+\n+test_expect_success \"setup without fsmonitor\" '\n+\tunset INTEGRATION_SCRIPT &&\n+\tgit config --unset core.fsmonitor &&\n+\tgit update-index --no-fsmonitor\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uno (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uno\n+'\n+\n+if test -n \"$GIT_PERF_7519_DROP_CACHE\"; then\n+\ttest-drop-caches\n+fi\n+\n+test_perf \"status -uall (fsmonitor=$INTEGRATION_SCRIPT)\" '\n+\tgit status -uall\n+'\n+\n+if test_have_prereq WATCHMAN\n+then\n+\twatchman watch-del \"$GIT_WORK_TREE\" >/dev/null 2>&1 &&\n+\n+\t# Work around Watchman bug on Windows where it holds on to handles\n+\t# preventing the removal of the trash directory\n+\twatchman shutdown-server >/dev/null 2>&1\n+fi\n+\n+test_done\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328657","messageId":"20170922163548.11288-7-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:42Z","receivedAt":"2017-09-22T16:36:57Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a new command line option (-f) to ls-files to have it use lowercase\nletters for 'fsmonitor valid' files\n\nSigned-off-by: Ben Peart \n---\n builtin/ls-files.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex e1339e6d17..313962a0c1 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -31,6 +31,7 @@ static int show_resolve_undo;\n static int show_modified;\n static int show_killed;\n static int show_valid_bit;\n+static int show_fsmonitor_bit;\n static int line_terminator = '\\n';\n static int debug_mode;\n static int show_eol;\n@@ -86,7 +87,8 @@ static const char *get_tag(const struct cache_entry *ce, const char *tag)\n {\n \tstatic char alttag[4];\n \n-\tif (tag && *tag && show_valid_bit && (ce->ce_flags & CE_VALID)) {\n+\tif (tag && *tag && ((show_valid_bit && (ce->ce_flags & CE_VALID)) ||\n+\t\t(show_fsmonitor_bit && (ce->ce_flags & CE_FSMONITOR_VALID)))) {\n \t\tmemcpy(alttag, tag, 3);\n \n \t\tif (isalpha(tag[0])) {\n@@ -515,6 +517,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\t\tN_(\"identify the file status with tags\")),\n \t\tOPT_BOOL('v', NULL, &show_valid_bit,\n \t\t\tN_(\"use lowercase letters for 'assume unchanged' files\")),\n+\t\tOPT_BOOL('f', NULL, &show_fsmonitor_bit,\n+\t\t\tN_(\"use lowercase letters for 'fsmonitor clean' files\")),\n \t\tOPT_BOOL('c', \"cached\", &show_cached,\n \t\t\tN_(\"show cached files in the output (default)\")),\n \t\tOPT_BOOL('d', \"deleted\", &show_deleted,\n@@ -584,7 +588,7 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tfor (i = 0; i < exclude_list.nr; i++) {\n \t\tadd_exclude(exclude_list.items[i].string, \"\", 0, el, --exclude_args);\n \t}\n-\tif (show_tag || show_valid_bit) {\n+\tif (show_tag || show_valid_bit || show_fsmonitor_bit) {\n \t\ttag_cached = \"H \";\n \t\ttag_unmerged = \"M \";\n \t\ttag_removed = \"R \";\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328658","messageId":"20170922163548.11288-6-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 05/12] fsmonitor: add documentation for the fsmonitor extension.","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:41Z","receivedAt":"2017-09-22T16:36:58Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This includes the core.fsmonitor setting, the fsmonitor integration hook,\nand the fsmonitor index extension.\n\nAlso add documentation for the new fsmonitor options to ls-files and\nupdate-index.\n\nSigned-off-by: Ben Peart \n---\n Documentation/config.txt | 7 +++++\n Documentation/git-ls-files.txt | 7 ++++-\n Documentation/git-update-index.txt | 45 ++++++++++++++++++++++++++++++++\n Documentation/githooks.txt | 28 ++++++++++++++++++++\n Documentation/technical/index-format.txt | 19 ++++++++++++++\n 5 files changed, 105 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dc4e3f58a2..db52645cb4 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -413,6 +413,13 @@ core.protectNTFS::\n \t8.3 \"short\" names.\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n+core.fsmonitor::\n+\tIf set, the value of this variable is used as a command which\n+\twill identify all files that may have changed since the\n+\trequested date/time. This information is used to speed up git by\n+\tavoiding unnecessary processing of files that have not changed.\n+\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex d153c17e06..3ac3e3a77d 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -9,7 +9,7 @@ git-ls-files - Show information about files in the index and the working tree\n SYNOPSIS\n --------\n [verse]\n-'git ls-files' [-z] [-t] [-v]\n+'git ls-files' [-z] [-t] [-v] [-f]\n \t\t(--[cached|deleted|others|ignored|stage|unmerged|killed|modified])*\n \t\t(-[c|d|o|i|s|u|k|m])*\n \t\t[--eol]\n@@ -133,6 +133,11 @@ a space) at the start of each line:\n \tthat are marked as 'assume unchanged' (see\n \tlinkgit:git-update-index[1]).\n \n+-f::\n+\tSimilar to `-t`, but use lowercase letters for files\n+\tthat are marked as 'fsmonitor valid' (see\n+\tlinkgit:git-update-index[1]).\n+\n --full-name::\n \tWhen run from a subdirectory, the command usually\n \toutputs paths relative to the current directory. This\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex e19eba62cd..7c2f880a22 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -16,9 +16,11 @@ SYNOPSIS\n \t [--chmod=(+|-)x]\n \t [--[no-]assume-unchanged]\n \t [--[no-]skip-worktree]\n+\t [--[no-]fsmonitor-valid]\n \t [--ignore-submodules]\n \t [--[no-]split-index]\n \t [--[no-|test-|force-]untracked-cache]\n+\t [--[no-]fsmonitor]\n \t [--really-refresh] [--unresolve] [--again | -g]\n \t [--info-only] [--index-info]\n \t [-z] [--stdin] [--index-version ]\n@@ -111,6 +113,12 @@ you will need to handle the situation manually.\n \tset and unset the \"skip-worktree\" bit for the paths. See\n \tsection \"Skip-worktree bit\" below for more information.\n \n+--[no-]fsmonitor-valid::\n+\tWhen one of these flags is specified, the object name recorded\n+\tfor the paths are not updated. Instead, these options\n+\tset and unset the \"fsmonitor valid\" bit for the paths. See\n+\tsection \"File System Monitor\" below for more information.\n+\n -g::\n --again::\n \tRuns 'git update-index' itself on the paths whose index\n@@ -201,6 +209,15 @@ will remove the intended effect of the option.\n \t`--untracked-cache` used to imply `--test-untracked-cache` but\n \tthis option would enable the extension unconditionally.\n \n+--fsmonitor::\n+--no-fsmonitor::\n+\tEnable or disable files system monitor feature. These options\n+\ttake effect whatever the value of the `core.fsmonitor`\n+\tconfiguration variable (see linkgit:git-config[1]). But a warning\n+\tis emitted when the change goes against the configured value, as\n+\tthe configured value will take effect next time the index is\n+\tread and this will remove the intended effect of the option.\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \n@@ -447,6 +464,34 @@ command reads the index; while when `--[no-|force-]untracked-cache`\n are used, the untracked cache is immediately added to or removed from\n the index.\n \n+File System Monitor\n+-------------------\n+\n+This feature is intended to speed up git operations for repos that have\n+large working directories.\n+\n+It enables git to work together with a file system monitor (see the\n+\"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n+inform it as to what files have been modified. This enables git to avoid\n+having to lstat() every file to find modified files.\n+\n+When used in conjunction with the untracked cache, it can further improve\n+performance by avoiding the cost of scanning the entire working directory\n+looking for new files.\n+\n+If you want to enable (or disable) this feature, it is easier to use\n+the `core.fsmonitor` configuration variable (see\n+linkgit:git-config[1]) than using the `--fsmonitor` option to\n+`git update-index` in each repository, especially if you want to do so\n+across all repositories you use, because you can set the configuration\n+variable to `true` (or `false`) in your `$HOME/.gitconfig` just once\n+and have it affect all repositories you touch.\n+\n+When the `core.fsmonitor` configuration variable is changed, the\n+file system monitor is added to or removed from the index the next time\n+a command reads the index. When `--[no-]fsmonitor` are used, the file\n+system monitor is immediately added to or removed from the index.\n+\n Configuration\n -------------\n \ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 1bb4f92d4d..ae60559cd9 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -455,6 +455,34 @@ the name of the file that holds the e-mail to be sent. Exiting with a\n non-zero status causes 'git send-email' to abort before sending any\n e-mails.\n \n+fsmonitor-watchman\n+~~~~~~~~~~~~~~~~~~\n+\n+This hook is invoked when the configuration option core.fsmonitor is\n+set to .git/hooks/fsmonitor-watchman. It takes two arguments, a version\n+(currently 1) and the time in elapsed nanoseconds since midnight,\n+January 1, 1970.\n+\n+The hook should output to stdout the list of all files in the working\n+directory that may have changed since the requested time. The logic\n+should be inclusive so that it does not miss any potential changes.\n+The paths should be relative to the root of the working directory\n+and be separated by a single NUL.\n+\n+It is OK to include files which have not actually changed. All changes\n+including newly-created and deleted files should be included. When\n+files are renamed, both the old and the new name should be included.\n+\n+Git will limit what files it checks for changes as well as which\n+directories are checked for untracked files based on the path names\n+given.\n+\n+An optimized way to tell git \"all files have changed\" is to return\n+the filename '/'.\n+\n+The exit status determines whether git will use the data from the\n+hook to limit its search. On error, it will fall back to verifying\n+all files and folders.\n \n GIT\n ---\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex ade0b0c445..db3572626b 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -295,3 +295,22 @@ The remaining data of each directory block is grouped by type:\n in the previous ewah bitmap.\n \n - One NUL.\n+\n+== File System Monitor cache\n+\n+ The file system monitor cache tracks files for which the core.fsmonitor\n+ hook has told us about changes. The signature for this extension is\n+ { 'F', 'S', 'M', 'N' }.\n+\n+ The extension starts with\n+\n+ - 32-bit version number: the current supported version is 1.\n+\n+ - 64-bit time: the extension data reflects all changes through the given\n+\ttime which is stored as the nanoseconds elapsed since midnight,\n+\tJanuary 1, 1970.\n+\n+ - 32-bit bitmap size: the size of the CE_FSMONITOR_VALID bitmap.\n+\n+ - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n+ is not CE_FSMONITOR_VALID.\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328659","messageId":"20170922163548.11288-8-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 07/12] update-index: add fsmonitor support to update-index","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:43Z","receivedAt":"2017-09-22T16:37:43Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add support in update-index to manually add/remove the fsmonitor\nextension via --[no-]fsmonitor flags.\n\nAdd support in update-index to manually set/clear the fsmonitor\nvalid bit via --[no-]fsmonitor-valid flags.\n\nSigned-off-by: Ben Peart \n---\n builtin/update-index.c | 33 ++++++++++++++++++++++++++++++++-\n 1 file changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 6f39ee9274..41618db098 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -33,6 +33,7 @@ static int force_remove;\n static int verbose;\n static int mark_valid_only;\n static int mark_skip_worktree_only;\n+static int mark_fsmonitor_only;\n #define MARK_FLAG 1\n #define UNMARK_FLAG 2\n static struct strbuf mtime_dir = STRBUF_INIT;\n@@ -229,12 +230,12 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \tint namelen = strlen(path);\n \tint pos = cache_name_pos(path, namelen);\n \tif (0 <= pos) {\n+\t\tmark_fsmonitor_invalid(&the_index, active_cache[pos]);\n \t\tif (mark)\n \t\t\tactive_cache[pos]->ce_flags |= flag;\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\tmark_fsmonitor_invalid(&the_index, active_cache[pos]);\n \t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\n@@ -460,6 +461,11 @@ static void update_one(const char *path)\n \t\t\tdie(\"Unable to mark file %s\", path);\n \t\treturn;\n \t}\n+\tif (mark_fsmonitor_only) {\n+\t\tif (mark_ce_flags(path, CE_FSMONITOR_VALID, mark_fsmonitor_only == MARK_FLAG))\n+\t\t\tdie(\"Unable to mark file %s\", path);\n+\t\treturn;\n+\t}\n \n \tif (force_remove) {\n \t\tif (remove_file_from_cache(path))\n@@ -918,6 +924,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tint lock_error = 0;\n \tint split_index = -1;\n \tint force_write = 0;\n+\tint fsmonitor = -1;\n \tstruct lock_file *lock_file;\n \tstruct parse_opt_ctx_t ctx;\n \tstrbuf_getline_fn getline_fn;\n@@ -1011,6 +1018,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"enable untracked cache without testing the filesystem\"), UC_FORCE),\n \t\tOPT_SET_INT(0, \"force-write-index\", &force_write,\n \t\t\tN_(\"write out the index even if is not flagged as changed\"), 1),\n+\t\tOPT_BOOL(0, \"fsmonitor\", &fsmonitor,\n+\t\t\tN_(\"enable or disable file system monitor\")),\n+\t\t{OPTION_SET_INT, 0, \"fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n+\t\t\tN_(\"mark files as fsmonitor valid\"),\n+\t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, MARK_FLAG},\n+\t\t{OPTION_SET_INT, 0, \"no-fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n+\t\t\tN_(\"clear fsmonitor valid bit\"),\n+\t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, UNMARK_FLAG},\n \t\tOPT_END()\n \t};\n \n@@ -1152,6 +1167,22 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tdie(\"BUG: bad untracked_cache value: %d\", untracked_cache);\n \t}\n \n+\tif (fsmonitor > 0) {\n+\t\tif (git_config_get_fsmonitor() == 0)\n+\t\t\twarning(_(\"core.fsmonitor is unset; \"\n+\t\t\t\t\"set it if you really want to \"\n+\t\t\t\t\"enable fsmonitor\"));\n+\t\tadd_fsmonitor(&the_index);\n+\t\treport(_(\"fsmonitor enabled\"));\n+\t} else if (!fsmonitor) {\n+\t\tif (git_config_get_fsmonitor() == 1)\n+\t\t\twarning(_(\"core.fsmonitor is set; \"\n+\t\t\t\t\"remove it if you really want to \"\n+\t\t\t\t\"disable fsmonitor\"));\n+\t\tremove_fsmonitor(&the_index);\n+\t\treport(_(\"fsmonitor disabled\"));\n+\t}\n+\n \tif (active_cache_changed || force_write) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328660","messageId":"20170922163548.11288-9-benpeart@microsoft.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"[PATCH v8 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Ben Peart","fromEmail":"benpeart@microsoft.com","sentAt":"2017-09-22T16:35:44Z","receivedAt":"2017-09-22T16:37:45Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Add a test utility (test-dump-fsmonitor) that will dump the fsmonitor\nindex extension.\n\nSigned-off-by: Ben Peart \n---\n Makefile | 1 +\n t/helper/test-dump-fsmonitor.c | 21 +++++++++++++++++++++\n 2 files changed, 22 insertions(+)\n create mode 100644 t/helper/test-dump-fsmonitor.c\n\ndiff --git a/Makefile b/Makefile\nindex 9d6ec9c1e9..d970cd00e9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -639,6 +639,7 @@ TEST_PROGRAMS_NEED_X += test-config\n TEST_PROGRAMS_NEED_X += test-date\n TEST_PROGRAMS_NEED_X += test-delta\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n+TEST_PROGRAMS_NEED_X += test-dump-fsmonitor\n TEST_PROGRAMS_NEED_X += test-dump-split-index\n TEST_PROGRAMS_NEED_X += test-dump-untracked-cache\n TEST_PROGRAMS_NEED_X += test-fake-ssh\ndiff --git a/t/helper/test-dump-fsmonitor.c b/t/helper/test-dump-fsmonitor.c\nnew file mode 100644\nindex 0000000000..ad452707e8\n--- /dev/null\n+++ b/t/helper/test-dump-fsmonitor.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+\n+int cmd_main(int ac, const char **av)\n+{\n+\tstruct index_state *istate = &the_index;\n+\tint i;\n+\n+\tsetup_git_directory();\n+\tif (do_read_index(istate, get_index_file(), 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update) {\n+\t\tprintf(\"no fsmonitor\\n\");\n+\t\treturn 0;\n+\t}\n+\tprintf(\"fsmonitor last update %\"PRIuMAX\"\\n\", (uintmax_t)istate->fsmonitor_last_update);\n+\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tprintf((istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) ? \"+\" : \"-\");\n+\n+\treturn 0;\n+}\n-- \n2.14.1.549.g6ff7ed0467\n\n"},{"id":"328706","messageId":"20170922233707.25414-1-martin.agren@gmail.com","threadId":"46154","inReplyTo":"20170922163548.11288-2-benpeart@microsoft.com","subject":"Re: [PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2017-09-22T23:37:06Z","receivedAt":"2017-09-22T23:37:32Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 22 September 2017 at 18:35, Ben Peart wrote:\n> Add a new get_be64 macro to enable 64 bit endian conversions on memory\n> that may or may not be aligned.\n\nI needed this to compile and pass the tests with NO_UNALIGNED_LOADS.\n\nMartin\n\ndiff --git a/compat/bswap.h b/compat/bswap.h\nindex 6b22c4621..9dc79bdf5 100644\n--- a/compat/bswap.h\n+++ b/compat/bswap.h\n@@ -183,8 +183,8 @@ static inline uint32_t get_be32(const void *ptr)\n static inline uint64_t get_be64(const void *ptr)\n {\n \tconst unsigned char *p = ptr;\n-\treturn\t(uint64_t)get_be32(p[0]) << 32 |\n-\t\t(uint64_t)get_be32(p[4]) << 0;\n+\treturn\t(uint64_t)get_be32(p + 0) << 32 |\n+\t\t(uint64_t)get_be32(p + 4) << 0;\n }\n \n static inline void put_be32(void *ptr, uint32_t value)\n-- \n2.14.1.727.g9ddaf86\n\n"},{"id":"328707","messageId":"20170922233707.25414-2-martin.agren@gmail.com","threadId":"46154","inReplyTo":"20170922163548.11288-9-benpeart@microsoft.com","subject":"Re: [PATCH v8 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2017-09-22T23:37:07Z","receivedAt":"2017-09-22T23:37:41Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 22 September 2017 at 18:35, Ben Peart wrote:\n> Add a test utility (test-dump-fsmonitor) that will dump the fsmonitor\n> index extension.\n>\n> Signed-off-by: Ben Peart \n> ---\n> Makefile | 1 +\n> t/helper/test-dump-fsmonitor.c | 21 +++++++++++++++++++++\n> 2 files changed, 22 insertions(+)\n> create mode 100644 t/helper/test-dump-fsmonitor.c\n\nYou forget to add the new binary to .gitignore. (In patch 12/12, you\nintroduce test-drop-caches, which you _do_ add to .gitignore.)\n\nMartin\n\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 6f07de62d..0fe2e0440 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -6,6 +6,7 @@\n /test-delta\n /test-drop-caches\n /test-dump-cache-tree\n+/test-dump-fsmonitor\n /test-dump-split-index\n /test-dump-untracked-cache\n /test-fake-ssh\n-- \n2.14.1.727.g9ddaf86\n\n"},{"id":"328748","messageId":"MWHPR21MB047887A575D5DFFB213092A1F4640@MWHPR21MB0478.namprd21.prod.outlook.com","threadId":"46154","inReplyTo":"20170922233707.25414-1-martin.agren@gmail.com","subject":"RE: [PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2017-09-23T23:31:50Z","receivedAt":"2017-09-23T23:31:58Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nThanks,\n\nBen\n\n> -----Original Message-----\n> From: Martin Ågren [mailto:martin.agren@gmail.com]\n> Sent: Friday, September 22, 2017 7:37 PM\n> To: Ben Peart \n> Cc: David Turner ; Ævar Arnfjörð Bjarmason\n> ; Christian Couder ;\n> git@vger.kernel.org; Junio C Hamano ; Johannes\n> Schindelin ; Nguyễn Thái Ngọc Duy\n> ; Jeff King \n> Subject: Re: [PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64\n> \n> On 22 September 2017 at 18:35, Ben Peart \n> wrote:\n> > Add a new get_be64 macro to enable 64 bit endian conversions on memory\n> > that may or may not be aligned.\n> \n> I needed this to compile and pass the tests with NO_UNALIGNED_LOADS.\n> \n> Martin\n> \n> diff --git a/compat/bswap.h b/compat/bswap.h index 6b22c4621..9dc79bdf5\n> 100644\n> --- a/compat/bswap.h\n> +++ b/compat/bswap.h\n> @@ -183,8 +183,8 @@ static inline uint32_t get_be32(const void *ptr) static\n> inline uint64_t get_be64(const void *ptr) {\n> \tconst unsigned char *p = ptr;\n> -\treturn\t(uint64_t)get_be32(p[0]) << 32 |\n> -\t\t(uint64_t)get_be32(p[4]) << 0;\n> +\treturn\t(uint64_t)get_be32(p + 0) << 32 |\n> +\t\t(uint64_t)get_be32(p + 4) << 0;\n\nThis is surprising. Every other function in the file uses the p[x] syntax. Just for\nconsistency, is there a way to stick to that syntax but still make it work correctly?\nIs there a typecast that can make it work?\n\n> }\n> \n> static inline void put_be32(void *ptr, uint32_t value)\n> --\n> 2.14.1.727.g9ddaf86\n\n"},{"id":"328749","messageId":"MWHPR21MB0478265059A957CFB8785CEFF4640@MWHPR21MB0478.namprd21.prod.outlook.com","threadId":"46154","inReplyTo":"20170922233707.25414-2-martin.agren@gmail.com","subject":"RE: [PATCH v8 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2017-09-23T23:33:00Z","receivedAt":"2017-09-23T23:33:19Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"> -----Original Message-----\n> From: Martin Ågren [mailto:martin.agren@gmail.com]\n> Sent: Friday, September 22, 2017 7:37 PM\n> To: Ben Peart \n> Cc: David Turner ; Ævar Arnfjörð Bjarmason\n> ; Christian Couder ;\n> git@vger.kernel.org; Junio C Hamano ; Johannes\n> Schindelin ; Nguyễn Thái Ngọc Duy\n> ; Jeff King \n> Subject: Re: [PATCH v8 08/12] fsmonitor: add a test tool to dump the index\n> extension\n> \n> On 22 September 2017 at 18:35, Ben Peart \n> wrote:\n> > Add a test utility (test-dump-fsmonitor) that will dump the fsmonitor\n> > index extension.\n> >\n> > Signed-off-by: Ben Peart \n> > ---\n> > Makefile | 1 +\n> > t/helper/test-dump-fsmonitor.c | 21 +++++++++++++++++++++\n> > 2 files changed, 22 insertions(+)\n> > create mode 100644 t/helper/test-dump-fsmonitor.c\n> \n> You forget to add the new binary to .gitignore. (In patch 12/12, you introduce\n> test-drop-caches, which you _do_ add to .gitignore.)\n> \n\nOops. Thanks! Hopefully Junio can squash this in...\n\n> Martin\n> \n> diff --git a/t/helper/.gitignore b/t/helper/.gitignore index\n> 6f07de62d..0fe2e0440 100644\n> --- a/t/helper/.gitignore\n> +++ b/t/helper/.gitignore\n> @@ -6,6 +6,7 @@\n> /test-delta\n> /test-drop-caches\n> /test-dump-cache-tree\n> +/test-dump-fsmonitor\n> /test-dump-split-index\n> /test-dump-untracked-cache\n> /test-fake-ssh\n> --\n> 2.14.1.727.g9ddaf86\n\n"},{"id":"328758","messageId":"20170924035127.cwh54bjbbisl2dqf@sigill.intra.peff.net","threadId":"46154","inReplyTo":"MWHPR21MB047887A575D5DFFB213092A1F4640@MWHPR21MB0478.namprd21.prod.outlook.com","subject":"Re: [PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-09-24T03:51:27Z","receivedAt":"2017-09-24T03:51:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Sep 23, 2017 at 11:31:50PM +0000, Ben Peart wrote:\n\n> > diff --git a/compat/bswap.h b/compat/bswap.h index 6b22c4621..9dc79bdf5\n> > 100644\n> > --- a/compat/bswap.h\n> > +++ b/compat/bswap.h\n> > @@ -183,8 +183,8 @@ static inline uint32_t get_be32(const void *ptr) static\n> > inline uint64_t get_be64(const void *ptr) {\n> > \tconst unsigned char *p = ptr;\n> > -\treturn\t(uint64_t)get_be32(p[0]) << 32 |\n> > -\t\t(uint64_t)get_be32(p[4]) << 0;\n> > +\treturn\t(uint64_t)get_be32(p + 0) << 32 |\n> > +\t\t(uint64_t)get_be32(p + 4) << 0;\n> \n> This is surprising. Every other function in the file uses the p[x] syntax. Just for\n> consistency, is there a way to stick to that syntax but still make it work correctly?\n> Is there a typecast that can make it work?\n\nThe other ones are accessing the byte values directly, but since you are\nbuilding on get_be32 here, you have to pass the pointer. So:\n\n return (uint64_t)get_be32(&p[0]) << 32 |\n (uint64_t)get_be32(&p[4]) << 0;\n\nwould work. Or of course you could just spell it out like the others:\n\n return (uint64_t)p[0] << 56 |\n (uint64_t)p[1] << 48 |\n (uint64_t)p[2] << 40 |\n\t (uint64_t)p[3] << 32 |\n (uint64_t)p[4] << 24 |\n (uint64_t)p[5] << 16 |\n (uint64_t)p[6] << 8 |\n (uint64_t)p[7] << 0;\n\nI suspect compilers would end up with the same output either way (on\nx86, \"gcc -O2\" actually turns the whole thing into a bswap instruction).\n\n-Peff\n"},{"id":"328759","messageId":"xmqqwp4ohhym.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"MWHPR21MB0478265059A957CFB8785CEFF4640@MWHPR21MB0478.namprd21.prod.outlook.com","subject":"Re: [PATCH v8 08/12] fsmonitor: add a test tool to dump the index extension","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-24T03:51:45Z","receivedAt":"2017-09-24T03:51:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n>> You forget to add the new binary to .gitignore. (In patch 12/12, you introduce\n>> test-drop-caches, which you _do_ add to .gitignore.)\n>> \n>\n> Oops. Thanks! Hopefully Junio can squash this in...\n\nOK, will do.\n"},{"id":"328760","messageId":"xmqqshfchhx1.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"MWHPR21MB047887A575D5DFFB213092A1F4640@MWHPR21MB0478.namprd21.prod.outlook.com","subject":"Re: [PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-24T03:52:42Z","receivedAt":"2017-09-24T03:52:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n>> @@ -183,8 +183,8 @@ static inline uint32_t get_be32(const void *ptr) static\n>> inline uint64_t get_be64(const void *ptr) {\n>> \tconst unsigned char *p = ptr;\n>> -\treturn\t(uint64_t)get_be32(p[0]) << 32 |\n>> -\t\t(uint64_t)get_be32(p[4]) << 0;\n>> +\treturn\t(uint64_t)get_be32(p + 0) << 32 |\n>> +\t\t(uint64_t)get_be32(p + 4) << 0;\n>\n> This is surprising. Every other function in the file uses the p[x] syntax. Just for\n> consistency, is there a way to stick to that syntax but still make it work correctly?\n> Is there a typecast that can make it work?\n\nI'll do \"get_be32(&p[0])\" etc. while queueing for now.\n\nThanks, both of you.\n"},{"id":"329140","messageId":"xmqq7ewiz1mz.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"20170922163548.11288-1-benpeart@microsoft.com","subject":"Re: [PATCH v8 00/12] Fast git status via a file system watcher","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-29T02:20:36Z","receivedAt":"2017-09-29T02:20:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> The only behavioral change from V7 is the removal of unnecessary uses of\n> CE_MATCH_IGNORE_FSMONITOR. With a better understanding of *why* the\n> CE_MATCH_IGNORE_* flags are used, it is now clear they are not required\n> in most cases where CE_MATCH_IGNORE_FSMONITOR was being passed out of an\n> abundance of caution.\n\nThe reviews and updates after this round was posted were to\n\n * 01/12 had an obvious pointer-vs-pointee thinko, which I think I\n have locally fixed;\n\n * 08/12 forgot to add a new test executable to .gitignore file,\n which I think I have locally fixed, too.\n\nAny other review comments and suggestions for improvements?\nOtherwise I am tempted to declare victory and merge this to 'next'\nsoonish.\n\nFor reference, here is the interdiff between what was posted as v8\nand what I have on 'pu'.\n\nThanks.\n\n compat/bswap.h | 4 ++--\n t/helper/.gitignore | 1 +\n 2 files changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git b/compat/bswap.h a/compat/bswap.h\nindex 6b22c46214..5078ce5ecc 100644\n--- b/compat/bswap.h\n+++ a/compat/bswap.h\n@@ -183,8 +183,8 @@ static inline uint32_t get_be32(const void *ptr)\n static inline uint64_t get_be64(const void *ptr)\n {\n \tconst unsigned char *p = ptr;\n-\treturn\t(uint64_t)get_be32(p[0]) << 32 |\n-\t\t(uint64_t)get_be32(p[4]) << 0;\n+\treturn\t(uint64_t)get_be32(&p[0]) << 32 |\n+\t\t(uint64_t)get_be32(&p[4]) << 0;\n }\n \n static inline void put_be32(void *ptr, uint32_t value)\ndiff --git b/t/helper/.gitignore a/t/helper/.gitignore\nindex f9328eebdd..87a648a7cf 100644\n--- b/t/helper/.gitignore\n+++ a/t/helper/.gitignore\n@@ -5,6 +5,7 @@\n /test-delta\n /test-drop-caches\n /test-dump-cache-tree\n+/test-dump-fsmonitor\n /test-dump-split-index\n /test-dump-untracked-cache\n /test-fake-ssh\n"},{"id":"329174","messageId":"MWHPR21MB0478B8D1C5442B2FCE1ED465F47E0@MWHPR21MB0478.namprd21.prod.outlook.com","threadId":"46154","inReplyTo":"xmqq7ewiz1mz.fsf@gitster.mtv.corp.google.com","subject":"RE: [PATCH v8 00/12] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2017-09-29T12:07:46Z","receivedAt":"2017-09-29T12:07:59Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"> -----Original Message-----\n> From: Junio C Hamano [mailto:gitster@pobox.com]\n> Sent: Thursday, September 28, 2017 10:21 PM\n> To: Ben Peart \n> Cc: David.Turner@twosigma.com; avarab@gmail.com;\n> christian.couder@gmail.com; git@vger.kernel.org;\n> johannes.schindelin@gmx.de; pclouds@gmail.com; peff@peff.net\n> Subject: Re: [PATCH v8 00/12] Fast git status via a file system watcher\n> \n> Ben Peart writes:\n> \n> > The only behavioral change from V7 is the removal of unnecessary uses\n> > of CE_MATCH_IGNORE_FSMONITOR. With a better understanding of\n> *why*\n> > the\n> > CE_MATCH_IGNORE_* flags are used, it is now clear they are not\n> > required in most cases where CE_MATCH_IGNORE_FSMONITOR was being\n> > passed out of an abundance of caution.\n> \n> The reviews and updates after this round was posted were to\n> \n> * 01/12 had an obvious pointer-vs-pointee thinko, which I think I\n> have locally fixed;\n> \n> * 08/12 forgot to add a new test executable to .gitignore file,\n> which I think I have locally fixed, too.\n> \n> Any other review comments and suggestions for improvements?\n> Otherwise I am tempted to declare victory and merge this to 'next'\n> soonish.\n> \n> For reference, here is the interdiff between what was posted as v8 and what\n> I have on 'pu'.\n\nI had accumulated the same set of changes with one addition of removing\na duplicate \"the\" from a comment in the fsmonitor.h file:\n\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 8eb6163455..0de644e01a 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -4,7 +4,7 @@\n extern struct trace_key trace_fsmonitor;\n \n /*\n- * Read the the fsmonitor index extension and (if configured) restore the\n+ * Read the fsmonitor index extension and (if configured) restore the\n * CE_FSMONITOR_VALID state.\n */\n extern int read_fsmonitor_extension(struct index_state *istate, const void *data, unsigned long sz); \n\n> \n> Thanks.\n> \n> compat/bswap.h | 4 ++--\n> t/helper/.gitignore | 1 +\n> 2 files changed, 3 insertions(+), 2 deletions(-)\n> \n> diff --git b/compat/bswap.h a/compat/bswap.h index\n> 6b22c46214..5078ce5ecc 100644\n> --- b/compat/bswap.h\n> +++ a/compat/bswap.h\n> @@ -183,8 +183,8 @@ static inline uint32_t get_be32(const void *ptr) static\n> inline uint64_t get_be64(const void *ptr) {\n> \tconst unsigned char *p = ptr;\n> -\treturn\t(uint64_t)get_be32(p[0]) << 32 |\n> -\t\t(uint64_t)get_be32(p[4]) << 0;\n> +\treturn\t(uint64_t)get_be32(&p[0]) << 32 |\n> +\t\t(uint64_t)get_be32(&p[4]) << 0;\n> }\n> \n> static inline void put_be32(void *ptr, uint32_t value) diff --git\n> b/t/helper/.gitignore a/t/helper/.gitignore index f9328eebdd..87a648a7cf\n> 100644\n> --- b/t/helper/.gitignore\n> +++ a/t/helper/.gitignore\n> @@ -5,6 +5,7 @@\n> /test-delta\n> /test-drop-caches\n> /test-dump-cache-tree\n> +/test-dump-fsmonitor\n> /test-dump-split-index\n> /test-dump-untracked-cache\n> /test-fake-ssh\n"},{"id":"329274","messageId":"xmqqk20fuvg7.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"MWHPR21MB0478B8D1C5442B2FCE1ED465F47E0@MWHPR21MB0478.namprd21.prod.outlook.com","subject":"Re: [PATCH v8 00/12] Fast git status via a file system watcher","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-01T08:24:40Z","receivedAt":"2017-10-01T08:24:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> I had accumulated the same set of changes with one addition of removing\n> a duplicate \"the\" from a comment in the fsmonitor.h file:\n>\n> diff --git a/fsmonitor.h b/fsmonitor.h\n> index 8eb6163455..0de644e01a 100644\n> --- a/fsmonitor.h\n> +++ b/fsmonitor.h\n> @@ -4,7 +4,7 @@\n> extern struct trace_key trace_fsmonitor;\n> \n> /*\n> - * Read the the fsmonitor index extension and (if configured) restore the\n> + * Read the fsmonitor index extension and (if configured) restore the\n> * CE_FSMONITOR_VALID state.\n> */\n> extern int read_fsmonitor_extension(struct index_state *istate, const void *data, unsigned long sz); \n>\n>> \n>> Thanks.\n\nOK, now my copy has the same, so we are in sync. Unless there is no\nmore comment that benefits from a reroll of the series, let's run\nwith this version for now and merge it to 'next'. Further updates\ncan be done incrementally on top.\n\nThanks.\n"},{"id":"329587","messageId":"fd972756-4ad3-4b96-23d0-e2c9e59189d1@gmail.com","threadId":"46154","inReplyTo":"xmqqk20fuvg7.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/12] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-10-03T19:48:06Z","receivedAt":"2017-10-03T19:48:19Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 10/1/2017 4:24 AM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> I had accumulated the same set of changes with one addition of removing\n>> a duplicate \"the\" from a comment in the fsmonitor.h file:\n>>\n>> diff --git a/fsmonitor.h b/fsmonitor.h\n>> index 8eb6163455..0de644e01a 100644\n>> --- a/fsmonitor.h\n>> +++ b/fsmonitor.h\n>> @@ -4,7 +4,7 @@\n>> extern struct trace_key trace_fsmonitor;\n>> \n>> /*\n>> - * Read the the fsmonitor index extension and (if configured) restore the\n>> + * Read the fsmonitor index extension and (if configured) restore the\n>> * CE_FSMONITOR_VALID state.\n>> */\n>> extern int read_fsmonitor_extension(struct index_state *istate, const void *data, unsigned long sz);\n>>\n>>>\n>>> Thanks.\n> \n> OK, now my copy has the same, so we are in sync. Unless there is no\n> more comment that benefits from a reroll of the series, let's run\n> with this version for now and merge it to 'next'. Further updates\n> can be done incrementally on top.\n> \n> Thanks.\n> \n\nWell, rats. I found one more issue that applies to two of the commits. \nCan you squash this in as well or do you want it in some other form?\n\n\ndiff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\nindex 7ceb32dc18..cca3d71e90 100755\n--- a/t/t7519/fsmonitor-watchman\n+++ b/t/t7519/fsmonitor-watchman\n@@ -36,7 +36,7 @@ my $system = `uname -s`;\n $system =~ s/[\\r\\n]+//g;\n my $git_work_tree;\n\n-if ($system =~ m/^MSYS_NT/) {\n+if ($system =~ m/^MSYS_NT/ || $system =~ m/^MINGW/) {\n $git_work_tree = `cygpath -aw \"\\$PWD\"`;\n $git_work_tree =~ s/[\\r\\n]+//g;\n $git_work_tree =~ s,\\\\,/,g;\ndiff --git a/templates/hooks--fsmonitor-watchman.sample \nb/templates/hooks--fsmonitor-watchman.sample\nindex 870a59d237..c68038ef00 100755\n--- a/templates/hooks--fsmonitor-watchman.sample\n+++ b/templates/hooks--fsmonitor-watchman.sample\n@@ -35,7 +35,7 @@ my $system = `uname -s`;\n $system =~ s/[\\r\\n]+//g;\n my $git_work_tree;\n\n-if ($system =~ m/^MSYS_NT/) {\n+if ($system =~ m/^MSYS_NT/ || $system =~ m/^MINGW/) {\n $git_work_tree = `cygpath -aw \"\\$PWD\"`;\n $git_work_tree =~ s/[\\r\\n]+//g;\n $git_work_tree =~ s,\\\\,/,g;\n\n"},{"id":"329632","messageId":"xmqqzi97fytt.fsf@gitster.mtv.corp.google.com","threadId":"46154","inReplyTo":"fd972756-4ad3-4b96-23d0-e2c9e59189d1@gmail.com","subject":"Re: [PATCH v8 00/12] Fast git status via a file system watcher","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-04T02:09:50Z","receivedAt":"2017-10-04T02:09:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> Well, rats. I found one more issue that applies to two of the\n> commits. Can you squash this in as well or do you want it in some\n> other form?\n\nRats indeed. Let's go incremental as promised, perhaps like this\n(but please supply a better description if you have one).\n\n-- >8 --\nFrom: Ben Peart \nSubject: fsmonitor: MINGW support for watchman integration\n\nInstead of just taking $ENV{'PWD'}, use the same logic that converts\nPWD to $git_work_tree on MSYS_NT in the watchman integration hook\nscript also on MINGW.\n\nSigned-off-by: Ben Peart \nSigned-off-by: Junio C Hamano \n---\n t/t7519/fsmonitor-watchman | 2 +-\n templates/hooks--fsmonitor-watchman.sample | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\nindex 7ceb32dc18..cca3d71e90 100755\n--- a/t/t7519/fsmonitor-watchman\n+++ b/t/t7519/fsmonitor-watchman\n@@ -36,7 +36,7 @@ my $system = `uname -s`;\n $system =~ s/[\\r\\n]+//g;\n my $git_work_tree;\n \n-if ($system =~ m/^MSYS_NT/) {\n+if ($system =~ m/^MSYS_NT/ || $system =~ m/^MINGW/) {\n \t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n \t$git_work_tree =~ s/[\\r\\n]+//g;\n \t$git_work_tree =~ s,\\\\,/,g;\ndiff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample\nindex 870a59d237..c68038ef00 100755\n--- a/templates/hooks--fsmonitor-watchman.sample\n+++ b/templates/hooks--fsmonitor-watchman.sample\n@@ -35,7 +35,7 @@ my $system = `uname -s`;\n $system =~ s/[\\r\\n]+//g;\n my $git_work_tree;\n \n-if ($system =~ m/^MSYS_NT/) {\n+if ($system =~ m/^MSYS_NT/ || $system =~ m/^MINGW/) {\n \t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n \t$git_work_tree =~ s/[\\r\\n]+//g;\n \t$git_work_tree =~ s,\\\\,/,g;\n\n"},{"id":"329659","messageId":"alpine.DEB.2.10.1710032325310.28521@alexmv-linux","threadId":"46154","inReplyTo":"xmqqzi97fytt.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/12] Fast git status via a file system watcher","fromName":"Alex Vandiver","fromEmail":"alexmv@dropbox.com","sentAt":"2017-10-04T06:38:39Z","receivedAt":"2017-10-04T06:40:02Z","isPatch":true,"sender":{"key":"alexmv@dropbox.com","avatar":null},"body":"On Wed, 4 Oct 2017, Junio C Hamano wrote:\n> Rats indeed. Let's go incremental as promised, perhaps like this\n> (but please supply a better description if you have one).\n\nI think you'll also want the following squashed into 5c8cdcfd8 and\ndef437671:\n\n-- >8 --\nFrom 445d45027bb5b7823338cf111910d2884af6318b Mon Sep 17 00:00:00 2001\nFrom: Alex Vandiver \nDate: Tue, 3 Oct 2017 23:27:46 -0700\nSubject: [PATCH] fsmonitor: Read entirety of watchman output\n\nIn perl, setting $/ sets the string that is used as the \"record\nseparator,\" which sets the boundary that the `<>` construct reads to.\nSetting `local $/ = 0666;` evaluates the octal, getting 438, and\nstringifies it. Thus, the later read from `` stops as soon\nas it encounters the string \"438\" in the watchman output, yielding\ninvalid JSON; repositories containing filenames with SHA1 hashes are\nable to trip this easily.\n\nSet `$/` to undefined, thus slurping all output from watchman. Also\nclose STDIN which is provided to watchman, to better guarantee that we\ncannot deadlock with watchman while both attempting to read.\n\nSigned-off-by: Alex Vandiver \n---\n t/t7519/fsmonitor-watchman | 6 ++----\n templates/hooks--fsmonitor-watchman.sample | 6 ++----\n 2 files changed, 4 insertions(+), 8 deletions(-)\n\ndiff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\nindex 7ceb32dc1..7d6aef635 100755\n--- a/t/t7519/fsmonitor-watchman\n+++ b/t/t7519/fsmonitor-watchman\n@@ -50,9 +50,6 @@ launch_watchman();\n \n sub launch_watchman {\n \n-\t# Set input record separator\n-\tlocal $/ = 0666;\n-\n \tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j')\n \t or die \"open2() failed: $!\\n\" .\n \t \"Falling back to scanning...\\n\";\n@@ -83,7 +80,8 @@ sub launch_watchman {\n \tclose $fh;\n \n \tprint CHLD_IN $query;\n-\tmy $response = ;\n+\tclose CHLD_IN;\n+\tmy $response = do {local $/; };\n \n \topen ($fh, \">\", \".git/watchman-response.json\");\n \tprint $fh $response;\ndiff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample\nindex 870a59d23..1b8ed173e 100755\n--- a/templates/hooks--fsmonitor-watchman.sample\n+++ b/templates/hooks--fsmonitor-watchman.sample\n@@ -49,9 +49,6 @@ launch_watchman();\n \n sub launch_watchman {\n \n-\t# Set input record separator\n-\tlocal $/ = 0666;\n-\n \tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j')\n \t or die \"open2() failed: $!\\n\" .\n \t \"Falling back to scanning...\\n\";\n@@ -78,7 +75,8 @@ sub launch_watchman {\n \tEND\n \n \tprint CHLD_IN $query;\n-\tmy $response = ;\n+\tclose CHLD_IN;\n+\tmy $response = do {local $/; };\n \n \tdie \"Watchman: command returned no output.\\n\" .\n \t \"Falling back to scanning...\\n\" if $response eq \"\";\n-- \n2.14.2.959.g6663358d3\n"},{"id":"329680","messageId":"a7e5d944-7ff0-9c80-17db-7802607d4087@gmail.com","threadId":"46154","inReplyTo":"xmqqzi97fytt.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v8 00/12] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-10-04T12:27:47Z","receivedAt":"2017-10-04T12:27:55Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 10/3/2017 10:09 PM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> Well, rats. I found one more issue that applies to two of the\n>> commits. Can you squash this in as well or do you want it in some\n>> other form?\n> \n> Rats indeed. Let's go incremental as promised, perhaps like this\n> (but please supply a better description if you have one).\n\nThank you. Looks good.\n\n> \n> -- >8 --\n> From: Ben Peart \n> Subject: fsmonitor: MINGW support for watchman integration\n> \n> Instead of just taking $ENV{'PWD'}, use the same logic that converts\n> PWD to $git_work_tree on MSYS_NT in the watchman integration hook\n> script also on MINGW.\n> \n> Signed-off-by: Ben Peart \n> Signed-off-by: Junio C Hamano \n> ---\n> t/t7519/fsmonitor-watchman | 2 +-\n> templates/hooks--fsmonitor-watchman.sample | 2 +-\n> 2 files changed, 2 insertions(+), 2 deletions(-)\n> \n> diff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\n> index 7ceb32dc18..cca3d71e90 100755\n> --- a/t/t7519/fsmonitor-watchman\n> +++ b/t/t7519/fsmonitor-watchman\n> @@ -36,7 +36,7 @@ my $system = `uname -s`;\n> $system =~ s/[\\r\\n]+//g;\n> my $git_work_tree;\n> \n> -if ($system =~ m/^MSYS_NT/) {\n> +if ($system =~ m/^MSYS_NT/ || $system =~ m/^MINGW/) {\n> \t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n> \t$git_work_tree =~ s/[\\r\\n]+//g;\n> \t$git_work_tree =~ s,\\\\,/,g;\n> diff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample\n> index 870a59d237..c68038ef00 100755\n> --- a/templates/hooks--fsmonitor-watchman.sample\n> +++ b/templates/hooks--fsmonitor-watchman.sample\n> @@ -35,7 +35,7 @@ my $system = `uname -s`;\n> $system =~ s/[\\r\\n]+//g;\n> my $git_work_tree;\n> \n> -if ($system =~ m/^MSYS_NT/) {\n> +if ($system =~ m/^MSYS_NT/ || $system =~ m/^MINGW/) {\n> \t$git_work_tree = `cygpath -aw \"\\$PWD\"`;\n> \t$git_work_tree =~ s/[\\r\\n]+//g;\n> \t$git_work_tree =~ s,\\\\,/,g;\n> \n"},{"id":"329682","messageId":"583ff3fa-425c-6eb9-ddcb-8b0049d422ea@gmail.com","threadId":"46154","inReplyTo":"alpine.DEB.2.10.1710032325310.28521@alexmv-linux","subject":"Re: [PATCH v8 00/12] Fast git status via a file system watcher","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-10-04T12:48:39Z","receivedAt":"2017-10-04T12:48:47Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 10/4/2017 2:38 AM, Alex Vandiver wrote:\n> On Wed, 4 Oct 2017, Junio C Hamano wrote:\n>> Rats indeed. Let's go incremental as promised, perhaps like this\n>> (but please supply a better description if you have one).\n> \n> I think you'll also want the following squashed into 5c8cdcfd8 and\n> def437671:\n> \n> -- >8 --\n> From 445d45027bb5b7823338cf111910d2884af6318b Mon Sep 17 00:00:00 2001\n> From: Alex Vandiver \n> Date: Tue, 3 Oct 2017 23:27:46 -0700\n> Subject: [PATCH] fsmonitor: Read entirety of watchman output\n> \n> In perl, setting $/ sets the string that is used as the \"record\n> separator,\" which sets the boundary that the `<>` construct reads to.\n> Setting `local $/ = 0666;` evaluates the octal, getting 438, and\n> stringifies it. Thus, the later read from `` stops as soon\n> as it encounters the string \"438\" in the watchman output, yielding\n> invalid JSON; repositories containing filenames with SHA1 hashes are\n> able to trip this easily.\n> \n> Set `$/` to undefined, thus slurping all output from watchman. Also\n> close STDIN which is provided to watchman, to better guarantee that we\n> cannot deadlock with watchman while both attempting to read.\n> \n\nThank you! I'm a perl neophyte so have to rely on others when it comes \nto these types of perl issues. I tried out your fixes and they appear \nto work well.\n\nWhile testing them, I discovered that your fix of `local $/ = 0666;` \nexposed an existing issue in the test version of the integration script. \n The fix for that is within my perl capabilities and is fixed with the \nfollowing patch:\n\n-- >8 --\n From 3e3b983a4208a62d166c233a7de3bf045322f6c7 Mon Sep 17 00:00:00 2001\nFrom: Ben Peart \nDate: Wed, 4 Oct 2017 08:33:39 -0400\nSubject: [PATCH] fsmonitor: preserve utf8 filenames in \nfsmonitor-watchman log\n\nUpdate the test fsmonitor-watchman integration script to properly\npreserve utf8 filenames when outputting the .git/watchman-output.out log\nfile.\n\nSigned-off-by: Ben Peart \n---\n t/t7519/fsmonitor-watchman | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/t/t7519/fsmonitor-watchman b/t/t7519/fsmonitor-watchman\nindex 51330f8b3d..a3e30bf54f 100755\n--- a/t/t7519/fsmonitor-watchman\n+++ b/t/t7519/fsmonitor-watchman\n@@ -129,6 +129,7 @@ sub launch_watchman {\n \t \"Falling back to scanning...\\n\" if $o->{error};\n\n \topen ($fh, \">\", \".git/watchman-output.out\");\n+\tbinmode $fh, \":utf8\";\n \tprint $fh @{$o->{files}};\n \tclose $fh;\n\n-- \n2.14.1.windows.1.1034.g0776750557\n\n"}]}