{"thread":{"id":"48263","subject":"[PATCH v1 0/2] fsexcludes: Add programmatic way to exclude files","startedAt":"2018-04-10T21:04:36Z","lastAt":"2018-04-18T21:26:03Z","messageCount":17,"participants":["Ben Peart","Martin Ågren","Junio C Hamano","Duy Nguyen"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"344393","messageId":"20180410210408.13788-1-benpeart@microsoft.com","threadId":"48263","inReplyTo":null,"subject":"[PATCH v1 0/2] fsexcludes: Add programmatic way to exclude files","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2018-04-10T21:04:23Z","receivedAt":"2018-04-10T21:04:36Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"In git repos with large working directories an external file system monitor\n(like fsmonitor or gvfs) can track what files in the working directory have been\nmodified.  This information can be used to speed up git operations that scale\nbased on the size of the working directory so that they become O(# of modified\nfiles) vs O(# of files in the working directory).\n\nThe fsmonitor patch series added logic to limit what files git had to stat() to\nthe set of modified files provided by the fsmonitor hook proc.  It also used the\nuntracked cache (if enabled) to limit the files/folders git had to scan looking\nfor new/untracked files.  GVFS is another external file system model that also\nspeeds up git working directory based operations that has been using a different\nmechanism (programmatically generating an excludes file) to enable git to be\nO(# of modified files).\n\nThis patch series will introduce a new way to limit git�s traversal of the\nworking directory that does not require the untracked cache (fsmonitor) or using\nthe excludes feature (GVFS).  It does this by enhancing the existing excludes\nlogic in dir.c to support a new �File System Excludes� or fsexcludes API that is\nbetter tuned to these programmatic applications.\n\nBase Ref: master\nWeb-Diff: https://github.com/benpeart/git/commit/2ccbcd6360\nCheckout: git fetch https://github.com/benpeart/git fsexcludes-v1 && git checkout 2ccbcd6360\n\nBen Peart (2):\n  fsexcludes: add a programmatic way to exclude files from git's working\n    directory traversal logic\n  fsmonitor: switch to use new fsexcludes logic and remove unused\n    untracked cache based logic\n\n Makefile                    |   1 +\n dir.c                       |  33 ++++--\n dir.h                       |   2 -\n fsexcludes.c                | 210 ++++++++++++++++++++++++++++++++++++\n fsexcludes.h                |  27 +++++\n fsmonitor.c                 |  21 +---\n fsmonitor.h                 |  10 +-\n t/t7519-status-fsmonitor.sh |  14 +--\n 8 files changed, 270 insertions(+), 48 deletions(-)\n create mode 100644 fsexcludes.c\n create mode 100644 fsexcludes.h\n\n\nbase-commit: 0b0cc9f86731f894cff8dd25299a9b38c254569e\n-- \n2.17.0.windows.1\n\n\n"},{"id":"344394","messageId":"20180410210408.13788-3-benpeart@microsoft.com","threadId":"48263","inReplyTo":"20180410210408.13788-1-benpeart@microsoft.com","subject":"[PATCH v1 2/2] fsmonitor: switch to use new fsexcludes logic and remove unused untracked cache based logic","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2018-04-10T21:04:25Z","receivedAt":"2018-04-10T21:04:37Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Update fsmonitor to utilize the new fsexcludes based logic for excluding paths\nthat do not need to be scaned for new or modified files.  Remove the old logic\nin dir.c that utilized the untracked cache (if enabled) to accomplish the same\ngoal.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n dir.c                       | 33 ++++++++++++++++++++++++---------\n dir.h                       |  2 --\n fsmonitor.c                 | 21 ++-------------------\n fsmonitor.h                 | 10 +++-------\n t/t7519-status-fsmonitor.sh | 14 +++-----------\n 5 files changed, 32 insertions(+), 48 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 63a917be45..28c2c83f76 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -18,7 +18,7 @@\n #include \"utf8.h\"\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n-#include \"fsmonitor.h\"\n+#include \"fsexcludes.h\"\n \n /*\n  * Tells read_directory_recursive how a file or directory should be treated.\n@@ -1102,6 +1102,12 @@ int is_excluded_from_list(const char *pathname,\n \t\t\t  struct exclude_list *el, struct index_state *istate)\n {\n \tstruct exclude *exclude;\n+\n+\tif (*dtype == DT_UNKNOWN)\n+\t\t*dtype = get_dtype(NULL, istate, pathname, pathlen);\n+\tif (fsexcludes_is_excluded_from(istate, pathname, pathlen, *dtype) > 0)\n+\t\treturn 1;\n+\n \texclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n \t\t\t\t\t\t  dtype, el, istate);\n \tif (exclude)\n@@ -1317,8 +1323,15 @@ struct exclude *last_exclude_matching(struct dir_struct *dir,\n int is_excluded(struct dir_struct *dir, struct index_state *istate,\n \t\tconst char *pathname, int *dtype_p)\n {\n-\tstruct exclude *exclude =\n-\t\tlast_exclude_matching(dir, istate, pathname, dtype_p);\n+\tstruct exclude *exclude;\n+\tint pathlen = strlen(pathname);\n+\n+\tif (*dtype_p == DT_UNKNOWN)\n+\t\t*dtype_p = get_dtype(NULL, istate, pathname, pathlen);\n+\tif (fsexcludes_is_excluded_from(istate, pathname, pathlen, *dtype_p) > 0)\n+\t\treturn 1;\n+\n+\texclude = last_exclude_matching(dir, istate, pathname, dtype_p);\n \tif (exclude)\n \t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n \treturn 0;\n@@ -1671,6 +1684,9 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \tif (dtype != DT_DIR && has_path_in_index)\n \t\treturn path_none;\n \n+\tif (fsexcludes_is_excluded_from(istate, path->buf, path->len, dtype) > 0)\n+\t\treturn path_excluded;\n+\n \t/*\n \t * When we are looking at a directory P in the working tree,\n \t * there are three cases:\n@@ -1810,12 +1826,9 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n-\t/*\n-\t * With fsmonitor, we can trust the untracked cache's valid field.\n-\t */\n-\trefresh_fsmonitor(istate);\n-\tif (!(dir->untracked->use_fsmonitor && untracked->valid)) {\n-\t\tif (lstat(path->len ? path->buf : \".\", &st)) {\n+\tif (!untracked->valid) {\n+\t\tif (stat(path->len ? path->buf : \".\", &st)) {\n+\t\t\tinvalidate_directory(dir->untracked, untracked);\n \t\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n \t\t\treturn 0;\n \t\t}\n@@ -2011,6 +2024,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t/* add the path to the appropriate result list */\n \t\tswitch (state) {\n \t\tcase path_excluded:\n+\t\t\tif (fsexcludes_is_excluded_from(istate, path.buf, path.len, DTYPE(cdir.de)) > 0)\n+\t\t\t\tbreak;\n \t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n \t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n \t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\ndiff --git a/dir.h b/dir.h\nindex b0758b82a2..e67ccfbb29 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -139,8 +139,6 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n-\t/* fsmonitor invalidation data */\n-\tunsigned int use_fsmonitor : 1;\n };\n \n struct dir_struct {\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 6d7bcd5d0e..dd67eef851 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsexcludes.h\"\n #include \"fsmonitor.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n@@ -125,12 +126,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, const char *n\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t}\n \n-\t/*\n-\t * Mark the untracked cache dirty even if it wasn't found in the index\n-\t * as it could be a new untracked file.\n-\t */\n \ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n-\tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n void refresh_fsmonitor(struct index_state *istate)\n@@ -184,11 +180,8 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t/* Mark all entries invalid */\n \t\tfor (i = 0; i < istate->cache_nr; i++)\n \t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\n-\t\tif (istate->untracked)\n-\t\t\tistate->untracked->use_fsmonitor = 0;\n \t}\n-\tstrbuf_release(&query_result);\n+\tfsexcludes_init(&query_result);\n \n \t/* Now that we've updated istate, save the last_update time */\n \tistate->fsmonitor_last_update = last_update;\n@@ -207,12 +200,6 @@ void add_fsmonitor(struct index_state *istate)\n \t\tfor (i = 0; i < istate->cache_nr; i++)\n \t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \n-\t\t/* reset the untracked cache */\n-\t\tif (istate->untracked) {\n-\t\t\tadd_untracked_cache(istate);\n-\t\t\tistate->untracked->use_fsmonitor = 1;\n-\t\t}\n-\n \t\t/* Update the fsmonitor state */\n \t\trefresh_fsmonitor(istate);\n \t}\n@@ -241,10 +228,6 @@ void tweak_fsmonitor(struct index_state *istate)\n \n \t\t\t/* Mark all previously saved entries as dirty */\n \t\t\tewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n-\n-\t\t\t/* Now mark the untracked cache for fsmonitor usage */\n-\t\t\tif (istate->untracked)\n-\t\t\t\tistate->untracked->use_fsmonitor = 1;\n \t\t}\n \n \t\tewah_free(istate->fsmonitor_dirty);\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 65f3743636..f7adfc1f7c 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -35,8 +35,7 @@ extern void tweak_fsmonitor(struct index_state *istate);\n \n /*\n  * Run the configured fsmonitor integration script and clear the\n- * CE_FSMONITOR_VALID bit for any files returned as dirty.  Also invalidate\n- * any corresponding untracked cache directory structures. Optimized to only\n+ * CE_FSMONITOR_VALID bit for any files returned as dirty. Optimized to only\n  * run the first time it is called.\n  */\n extern void refresh_fsmonitor(struct index_state *istate);\n@@ -55,17 +54,14 @@ static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n }\n \n /*\n- * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate\n- * any corresponding untracked cache directory structures. This should\n+ * Clear the given cache entry's CE_FSMONITOR_VALID bit. This should\n  * be called any time git creates or modifies a file that should\n- * trigger an lstat() or invalidate the untracked cache for the\n- * corresponding directory\n+ * trigger an lstat() for the corresponding directory\n  */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n \tif (core_fsmonitor) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\n \t}\n }\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 756beb0d8e..d6a1da5a0a 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -225,8 +225,7 @@ test_expect_success '*only* files returned by the integration script get flagged\n # Ensure commands that call refresh_index() to move the index back in time\n # properly invalidate the fsmonitor cache\n test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n-\tEOF\n+\twrite_integration_script &&\n \tclean_repo &&\n \tdirty_repo &&\n \tgit add . &&\n@@ -275,7 +274,7 @@ do\n \t\t'\n \n \t\t# Make sure it's actually skipping the check for modified and untracked\n-\t\t# (if enabled) files unless it is told about them.\n+\t\t# files unless it is told about them.\n \t\ttest_expect_success \"status doesn't detect unreported modifications\" '\n \t\t\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n \t\t\t:>marker\n@@ -288,14 +287,7 @@ do\n \t\t\tgit status >actual &&\n \t\t\ttest_path_is_file marker &&\n \t\t\ttest_i18ngrep ! \"Changes not staged for commit:\" actual &&\n-\t\t\tif test $uc_val = true\n-\t\t\tthen\n-\t\t\t\ttest_i18ngrep ! \"Untracked files:\" actual\n-\t\t\tfi &&\n-\t\t\tif test $uc_val = false\n-\t\t\tthen\n-\t\t\t\ttest_i18ngrep \"Untracked files:\" actual\n-\t\t\tfi &&\n+\t\t\ttest_i18ngrep ! \"Untracked files:\" actual &&\n \t\t\trm -f marker\n \t\t'\n \tdone\n-- \n2.17.0.windows.1\n\n"},{"id":"344395","messageId":"20180410210408.13788-2-benpeart@microsoft.com","threadId":"48263","inReplyTo":"20180410210408.13788-1-benpeart@microsoft.com","subject":"[PATCH v1 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2018-04-10T21:04:24Z","receivedAt":"2018-04-10T21:04:43Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"The File System Excludes module is a new programmatic way to exclude files and\nfolders from git's traversal of the working directory.  fsexcludes_init() should\nbe called with a string buffer that contains a NUL separated list of path names\nof the files and/or directories that should be included.  Any path not listed\nwill be excluded. The paths should be relative to the root of the working\ndirectory and be separated by a single NUL.\n\nThe excludes logic in dir.c has been updated to honor the results of\nfsexcludes_is_excluded_from().  If fsexcludes does not exclude the file, the\nnormal excludes logic is also checked as it could further reduce the set of\nfiles that should be included.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n Makefile     |   1 +\n fsexcludes.c | 210 +++++++++++++++++++++++++++++++++++++++++++++++++++\n fsexcludes.h |  27 +++++++\n 3 files changed, 238 insertions(+)\n create mode 100644 fsexcludes.c\n create mode 100644 fsexcludes.h\n\ndiff --git a/Makefile b/Makefile\nindex 96f6138f63..c102d2f75a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -819,6 +819,7 @@ LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-object.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n+LIB_OBJS += fsexcludes.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\ndiff --git a/fsexcludes.c b/fsexcludes.c\nnew file mode 100644\nindex 0000000000..07bfe376a0\n--- /dev/null\n+++ b/fsexcludes.c\n@@ -0,0 +1,210 @@\n+#include \"cache.h\"\n+#include \"fsexcludes.h\"\n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n+static int fsexcludes_initialized = 0;\n+static struct strbuf fsexcludes_data = STRBUF_INIT;\n+static struct hashmap fsexcludes_hashmap;\n+static struct hashmap parent_directory_hashmap;\n+\n+struct fsexcludes {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tconst char *pattern;\n+\tint patternlen;\n+};\n+\n+static unsigned int(*fsexcludeshash)(const void *buf, size_t len);\n+static int(*fsexcludescmp)(const char *a, const char *b, size_t len);\n+\n+static int fsexcludes_hashmap_cmp(const void *unused_cmp_data,\n+\tconst void *a, const void *b, const void *key)\n+{\n+\tconst struct fsexcludes *fse1 = a;\n+\tconst struct fsexcludes *fse2 = b;\n+\n+\treturn fsexcludescmp(fse1->pattern, fse2->pattern, fse1->patternlen);\n+}\n+\n+static int check_fsexcludes_hashmap(struct hashmap *map, const char *pattern, int patternlen)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct fsexcludes fse;\n+\tchar *slash;\n+\n+\t/* Check straight mapping */\n+\tstrbuf_reset(&sb);\n+\tstrbuf_add(&sb, pattern, patternlen);\n+\tfse.pattern = sb.buf;\n+\tfse.patternlen = sb.len;\n+\thashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n+\tif (hashmap_get(map, &fse, NULL)) {\n+\t\tstrbuf_release(&sb);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * Check to see if it matches a directory or any path\n+\t * underneath it.  In other words, 'a/b/foo.txt' will match\n+\t * '/', 'a/', and 'a/b/'.\n+\t */\n+\tslash = strchr(sb.buf, '/');\n+\twhile (slash) {\n+\t\tfse.pattern = sb.buf;\n+\t\tfse.patternlen = slash - sb.buf + 1;\n+\t\thashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n+\t\tif (hashmap_get(map, &fse, NULL)) {\n+\t\t\tstrbuf_release(&sb);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tslash = strchr(slash + 1, '/');\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn 1;\n+}\n+\n+static void fsexcludes_hashmap_add(struct hashmap *map, const char *pattern, const int patternlen)\n+{\n+\tstruct fsexcludes *fse;\n+\n+\tfse = xmalloc(sizeof(struct fsexcludes));\n+\tfse->pattern = pattern;\n+\tfse->patternlen = patternlen;\n+\thashmap_entry_init(fse, fsexcludeshash(fse->pattern, fse->patternlen));\n+\thashmap_add(map, fse);\n+}\n+\n+static void initialize_fsexcludes_hashmap(struct hashmap *map, struct strbuf *fsexcludes_data)\n+{\n+\tchar *buf, *entry;\n+\tsize_t len;\n+\tint i;\n+\n+\t/*\n+\t * Build a hashmap of the fsexcludes data we can use to look\n+\t * for cache entry matches quickly\n+\t */\n+\tfsexcludeshash = ignore_case ? memihash : memhash;\n+\tfsexcludescmp = ignore_case ? strncasecmp : strncmp;\n+\thashmap_init(map, fsexcludes_hashmap_cmp, NULL, 0);\n+\n+\tentry = buf = fsexcludes_data->buf;\n+\tlen = fsexcludes_data->len;\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (buf[i] == '\\0') {\n+\t\t\tfsexcludes_hashmap_add(map, entry, buf + i - entry);\n+\t\t\tentry = buf + i + 1;\n+\t\t}\n+\t}\n+}\n+\n+static void parent_directory_hashmap_add(struct hashmap *map, const char *pattern, const int patternlen)\n+{\n+\tchar *slash;\n+\tstruct fsexcludes *fse;\n+\n+\t/*\n+\t * Add any directories leading up to the file as the excludes logic\n+\t * needs to match directories leading up to the files as well. Detect\n+\t * and prevent unnecessary duplicate entries which will be common.\n+\t */\n+\tif (patternlen > 1) {\n+\t\tslash = strchr(pattern + 1, '/');\n+\t\twhile (slash) {\n+\t\t\tfse = xmalloc(sizeof(struct fsexcludes));\n+\t\t\tfse->pattern = pattern;\n+\t\t\tfse->patternlen = slash - pattern + 1;\n+\t\t\thashmap_entry_init(fse, fsexcludeshash(fse->pattern, fse->patternlen));\n+\t\t\tif (hashmap_get(map, fse, NULL))\n+\t\t\t\tfree(fse);\n+\t\t\telse\n+\t\t\t\thashmap_add(map, fse);\n+\t\t\tslash = strchr(slash + 1, '/');\n+\t\t}\n+\t}\n+}\n+\n+static void initialize_parent_directory_hashmap(struct hashmap *map, struct strbuf *vfs_data)\n+{\n+\tchar *buf, *entry;\n+\tsize_t len;\n+\tint i;\n+\n+\t/*\n+\t * Build a hashmap of the parent directories contained in the virtual\n+\t * file system data we can use to look for matches quickly\n+\t */\n+\tfsexcludeshash = ignore_case ? memihash : memhash;\n+\tfsexcludescmp = ignore_case ? strncasecmp : strncmp;\n+\thashmap_init(map, fsexcludes_hashmap_cmp, NULL, 0);\n+\n+\tentry = buf = vfs_data->buf;\n+\tlen = vfs_data->len;\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (buf[i] == '\\0') {\n+\t\t\tparent_directory_hashmap_add(map, entry, buf + i - entry);\n+\t\t\tentry = buf + i + 1;\n+\t\t}\n+\t}\n+}\n+\n+static int check_directory_hashmap(struct hashmap *map, const char *pathname, int pathlen)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct fsexcludes fse;\n+\n+\t/* Check for directory */\n+\tstrbuf_reset(&sb);\n+\tstrbuf_add(&sb, pathname, pathlen);\n+\tstrbuf_addch(&sb, '/');\n+\tfse.pattern = sb.buf;\n+\tfse.patternlen = sb.len;\n+\thashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n+\tif (hashmap_get(map, &fse, NULL)) {\n+\t\tstrbuf_release(&sb);\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn 1;\n+}\n+\n+/*\n+ * Return 1 for exclude, 0 for include and -1 for undecided.\n+ */\n+int fsexcludes_is_excluded_from(struct index_state *istate,\n+\tconst char *pathname, int pathlen, int dtype)\n+{\n+\tif (!fsexcludes_initialized)\n+\t\treturn -1;\n+\n+\tif (dtype == DT_REG) {\n+\t\t/* lazily init the hashmap */\n+\t\tif (!fsexcludes_hashmap.cmpfn_data)\n+\t\t\tinitialize_fsexcludes_hashmap(&fsexcludes_hashmap, &fsexcludes_data);\n+\n+\t\treturn check_fsexcludes_hashmap(&fsexcludes_hashmap, pathname, pathlen);\n+\t}\n+\n+\tif (dtype == DT_DIR || dtype == DT_LNK) {\n+\t\t/* lazily init the hashmap */\n+\t\tif (!parent_directory_hashmap.cmpfn_data)\n+\t\t\tinitialize_parent_directory_hashmap(&parent_directory_hashmap, &fsexcludes_data);\n+\n+\t\treturn check_directory_hashmap(&parent_directory_hashmap, pathname, pathlen);\n+\t}\n+\n+\treturn -1;\n+}\n+\n+void fsexcludes_init(struct strbuf *sb) {\n+\tfsexcludes_initialized = 1;\n+\tfsexcludes_data = *sb;\n+}\n+\n+void fsexcludes_free() {\n+\tstrbuf_release(&fsexcludes_data);\n+\thashmap_free(&fsexcludes_hashmap, 1);\n+\thashmap_free(&parent_directory_hashmap, 1);\n+}\ndiff --git a/fsexcludes.h b/fsexcludes.h\nnew file mode 100644\nindex 0000000000..1c4101343c\n--- /dev/null\n+++ b/fsexcludes.h\n@@ -0,0 +1,27 @@\n+#ifndef FSEXCLUDES_H\n+#define FSEXCLUDES_H\n+\n+/*\n+ * The file system excludes functions provides a way to programatically limit\n+ * where git will scan for untracked files.  This is used to speed up the\n+ * scan by avoiding scanning parts of the work directory that do not have\n+ * any new files.\n+ *\n+ */\n+\n+/*\n+ * sb should contain a NUL separated list of path names of the files\n+ * and/or directories that should be checked.  Any path not listed will\n+ * be excluded from the scan.\n+ */\n+void fsexcludes_init(struct strbuf *sb);\n+void fsexcludes_free();\n+\n+/*\n+ * Return 1 for exclude, 0 for include and -1 for undecided.\n+ */\n+int fsexcludes_is_excluded_from(struct index_state *istate,\n+\tconst char *pathname, int pathlen, int dtype_p);\n+\n+\n+#endif\n-- \n2.17.0.windows.1\n\n"},{"id":"344414","messageId":"CAN0heSpKzG93OcAAAoHQxURVGsHFWz6j494C+3bezHLTOovQHA@mail.gmail.com","threadId":"48263","inReplyTo":"20180410210408.13788-2-benpeart@microsoft.com","subject":"Re: [PATCH v1 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-04-10T22:09:31Z","receivedAt":"2018-04-10T22:09:36Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 10 April 2018 at 23:04, Ben Peart <Ben.Peart@microsoft.com> wrote:\n> The File System Excludes module is a new programmatic way to exclude files and\n> folders from git's traversal of the working directory.  fsexcludes_init() should\n> be called with a string buffer that contains a NUL separated list of path names\n> of the files and/or directories that should be included.  Any path not listed\n> will be excluded. The paths should be relative to the root of the working\n> directory and be separated by a single NUL.\n>\n> The excludes logic in dir.c has been updated to honor the results of\n> fsexcludes_is_excluded_from().  If fsexcludes does not exclude the file, the\n> normal excludes logic is also checked as it could further reduce the set of\n> files that should be included.\n\nHere you mention a change in dir.c...\n\n>  Makefile     |   1 +\n>  fsexcludes.c | 210 +++++++++++++++++++++++++++++++++++++++++++++++++++\n>  fsexcludes.h |  27 +++++++\n>  3 files changed, 238 insertions(+)\n\n... but this patch does not seem to touch dir.c at all.\n\n> +static int check_fsexcludes_hashmap(struct hashmap *map, const char *pattern, int patternlen)\n> +{\n> +       struct strbuf sb = STRBUF_INIT;\n> +       struct fsexcludes fse;\n> +       char *slash;\n> +\n> +       /* Check straight mapping */\n> +       strbuf_reset(&sb);\n\nYou could drop this strbuf_reset(). Or did you intend to use a static\nstruct strbuf?\n\n> +       /*\n> +        * Check to see if it matches a directory or any path\n> +        * underneath it.  In other words, 'a/b/foo.txt' will match\n> +        * '/', 'a/', and 'a/b/'.\n> +        */\n> +       slash = strchr(sb.buf, '/');\n> +       while (slash) {\n> +               fse.pattern = sb.buf;\n> +               fse.patternlen = slash - sb.buf + 1;\n> +               hashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n> +               if (hashmap_get(map, &fse, NULL)) {\n> +                       strbuf_release(&sb);\n> +                       return 0;\n> +               }\n> +               slash = strchr(slash + 1, '/');\n> +       }\n\nMaybe a for-loop would make this slightly more obvious:\n\nfor (slash = strchr(sb.buf, '/'); slash; slash = strchr(slash + 1, '/'))\n\nOn second thought, maybe not.\n\n> +       entry = buf = fsexcludes_data->buf;\n> +       len = fsexcludes_data->len;\n> +       for (i = 0; i < len; i++) {\n> +               if (buf[i] == '\\0') {\n> +                       fsexcludes_hashmap_add(map, entry, buf + i - entry);\n> +                       entry = buf + i + 1;\n> +               }\n> +       }\n> +}\n\nVery minor: I would have found \"buf - entry + i\" clearer here and later,\nbut I'm sure you'll find someone of the opposing opinion (e.g.,\nyourself). ;-)\n\n> +static int check_directory_hashmap(struct hashmap *map, const char *pathname, int pathlen)\n> +{\n> +       struct strbuf sb = STRBUF_INIT;\n> +       struct fsexcludes fse;\n> +\n> +       /* Check for directory */\n> +       strbuf_reset(&sb);\n\nSame comment as above about this spurious reset.\n\n> +       if (hashmap_get(map, &fse, NULL)) {\n> +               strbuf_release(&sb);\n> +               return 0;\n> +       }\n> +\n> +       strbuf_release(&sb);\n> +       return 1;\n> +}\n> +\n> +/*\n> + * Return 1 for exclude, 0 for include and -1 for undecided.\n> + */\n> +int fsexcludes_is_excluded_from(struct index_state *istate,\n> +       const char *pathname, int pathlen, int dtype)\n> +{\n\nWill we at some point regret not being able to \"return negative on\nerror\"? I guess that would be \"-2\" or \"negative other than -1\".\n\n> +void fsexcludes_init(struct strbuf *sb) {\n> +       fsexcludes_initialized = 1;\n> +       fsexcludes_data = *sb;\n> +}\n\nGrabbing the strbuf's members looks a bit odd. Is this\nperformance-sensitive enough that you do not want to make a copy? If a\ncaller releases its strbuf, which would normally be a good thing to do,\nwe may be in big trouble later. (Not only may .buf be stale, .len may\nindicate we actually have something to read.)\n\nI can understand that you do not want to pass a pointer+len, and that it\nis not enough to pass sb.buf, since the string may contain nuls.\n\nMaybe detach the original strbuf? That way, if a caller releases its\nbuffer, that is a no-op. A caller which goes on to use its buffer should\nfail quickly and obviously. Right now, an incorrect caller would\nprobably fail more subtly and less reproducibly.\n\nIn any case, maybe document this in the .h-file?\n\nMartin\n"},{"id":"344444","messageId":"xmqqr2nm44b0.fsf@gitster-ct.c.googlers.com","threadId":"48263","inReplyTo":"20180410210408.13788-2-benpeart@microsoft.com","subject":"Re: [PATCH v1 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-11T06:58:59Z","receivedAt":"2018-04-11T06:59:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart <Ben.Peart@microsoft.com> writes:\n\n> +void fsexcludes_free() {\n\nWrite this line like so:\n\n        void fsexcludes_free(void)\n        {\n\n> +void fsexcludes_free();\n\nvoid fsexcludes_free(void);\n"},{"id":"344476","messageId":"082cda6c-eb9f-c853-c09b-e0d3bc04cb9c@gmail.com","threadId":"48263","inReplyTo":"CAN0heSpKzG93OcAAAoHQxURVGsHFWz6j494C+3bezHLTOovQHA@mail.gmail.com","subject":"Re: [PATCH v1 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-04-11T19:56:35Z","receivedAt":"2018-04-11T19:56:44Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 4/10/2018 6:09 PM, Martin Ågren wrote:\n> On 10 April 2018 at 23:04, Ben Peart <Ben.Peart@microsoft.com> wrote:\n>> The File System Excludes module is a new programmatic way to exclude files and\n>> folders from git's traversal of the working directory.  fsexcludes_init() should\n>> be called with a string buffer that contains a NUL separated list of path names\n>> of the files and/or directories that should be included.  Any path not listed\n>> will be excluded. The paths should be relative to the root of the working\n>> directory and be separated by a single NUL.\n>>\n>> The excludes logic in dir.c has been updated to honor the results of\n>> fsexcludes_is_excluded_from().  If fsexcludes does not exclude the file, the\n>> normal excludes logic is also checked as it could further reduce the set of\n>> files that should be included.\n> \n> Here you mention a change in dir.c...\n> \n>>   Makefile     |   1 +\n>>   fsexcludes.c | 210 +++++++++++++++++++++++++++++++++++++++++++++++++++\n>>   fsexcludes.h |  27 +++++++\n>>   3 files changed, 238 insertions(+)\n> \n> ... but this patch does not seem to touch dir.c at all.\n> \n\nOops! Fixed in V2.\n\n>> +static int check_fsexcludes_hashmap(struct hashmap *map, const char *pattern, int patternlen)\n>> +{\n>> +       struct strbuf sb = STRBUF_INIT;\n>> +       struct fsexcludes fse;\n>> +       char *slash;\n>> +\n>> +       /* Check straight mapping */\n>> +       strbuf_reset(&sb);\n> \n> You could drop this strbuf_reset(). Or did you intend to use a static\n> struct strbuf?\n> \n\nGood point, fixed in V2.\n\n>> +       /*\n>> +        * Check to see if it matches a directory or any path\n>> +        * underneath it.  In other words, 'a/b/foo.txt' will match\n>> +        * '/', 'a/', and 'a/b/'.\n>> +        */\n>> +       slash = strchr(sb.buf, '/');\n>> +       while (slash) {\n>> +               fse.pattern = sb.buf;\n>> +               fse.patternlen = slash - sb.buf + 1;\n>> +               hashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n>> +               if (hashmap_get(map, &fse, NULL)) {\n>> +                       strbuf_release(&sb);\n>> +                       return 0;\n>> +               }\n>> +               slash = strchr(slash + 1, '/');\n>> +       }\n> \n> Maybe a for-loop would make this slightly more obvious:\n> \n> for (slash = strchr(sb.buf, '/'); slash; slash = strchr(slash + 1, '/'))\n> \n> On second thought, maybe not.\n> \n>> +       entry = buf = fsexcludes_data->buf;\n>> +       len = fsexcludes_data->len;\n>> +       for (i = 0; i < len; i++) {\n>> +               if (buf[i] == '\\0') {\n>> +                       fsexcludes_hashmap_add(map, entry, buf + i - entry);\n>> +                       entry = buf + i + 1;\n>> +               }\n>> +       }\n>> +}\n> \n> Very minor: I would have found \"buf - entry + i\" clearer here and later,\n> but I'm sure you'll find someone of the opposing opinion (e.g.,\n> yourself). ;-)\n> \n>> +static int check_directory_hashmap(struct hashmap *map, const char *pathname, int pathlen)\n>> +{\n>> +       struct strbuf sb = STRBUF_INIT;\n>> +       struct fsexcludes fse;\n>> +\n>> +       /* Check for directory */\n>> +       strbuf_reset(&sb);\n> \n> Same comment as above about this spurious reset.\n\nGood point, fixed in V2.\n\n> \n>> +       if (hashmap_get(map, &fse, NULL)) {\n>> +               strbuf_release(&sb);\n>> +               return 0;\n>> +       }\n>> +\n>> +       strbuf_release(&sb);\n>> +       return 1;\n>> +}\n>> +\n>> +/*\n>> + * Return 1 for exclude, 0 for include and -1 for undecided.\n>> + */\n>> +int fsexcludes_is_excluded_from(struct index_state *istate,\n>> +       const char *pathname, int pathlen, int dtype)\n>> +{\n> \n> Will we at some point regret not being able to \"return negative on\n> error\"? I guess that would be \"-2\" or \"negative other than -1\".\n> \n\nThis function is modeled after the other is_excluded_from* functions in \ndir.c so that the return value can be handled the same way.  I don't \nanticipate any need for change but you're right, we could return some \nother \"negative other than -1\" if it was ever needed.\n\n>> +void fsexcludes_init(struct strbuf *sb) {\n>> +       fsexcludes_initialized = 1;\n>> +       fsexcludes_data = *sb;\n>> +}\n> \n> Grabbing the strbuf's members looks a bit odd. Is this\n> performance-sensitive enough that you do not want to make a copy? If a\n> caller releases its strbuf, which would normally be a good thing to do,\n> we may be in big trouble later. (Not only may .buf be stale, .len may\n> indicate we actually have something to read.)\n> \n> I can understand that you do not want to pass a pointer+len, and that it\n> is not enough to pass sb.buf, since the string may contain nuls.\n> \n> Maybe detach the original strbuf? That way, if a caller releases its\n> buffer, that is a no-op. A caller which goes on to use its buffer should\n> fail quickly and obviously. Right now, an incorrect caller would\n> probably fail more subtly and less reproducibly.\n> \n> In any case, maybe document this in the .h-file?\n\nGreat suggestion!  I was looking for a better way to ensure the buffer \nownership transfer was robust.  I'll do both strbuf_detach() and update \nthe header file.  Thank you.\n\n> \n> Martin\n> \n"},{"id":"344478","messageId":"20180411200007.8612-1-benpeart@microsoft.com","threadId":"48263","inReplyTo":"20180410210408.13788-1-benpeart@microsoft.com","subject":"[PATCH v2 0/2] fsexcludes: Add programmatic way to exclude files","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2018-04-11T20:01:24Z","receivedAt":"2018-04-11T20:01:33Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Updated to incorporate feedback from V1.\n\nI'd really like a close review of the changes in dir.c where I added the calls\nto fsexcludes_is_excluded_from().  While they work and pass all the git tests\nas well as our internal functional tests, I'd like to be sure I haven't missed\nanything.\n\nBase Ref: master\nWeb-Diff: https://github.com/benpeart/git/commit/08442c209d\nCheckout: git fetch https://github.com/benpeart/git fsexcludes-v2 && git checkout 08442c209d\n\n\n### Interdiff (v1..v2):\n\ndiff --git a/fsexcludes.c b/fsexcludes.c\nindex 07bfe376a0..0ef57f107b 100644\n--- a/fsexcludes.c\n+++ b/fsexcludes.c\n@@ -33,7 +33,6 @@ static int check_fsexcludes_hashmap(struct hashmap *map, const char *pattern, in\n \tchar *slash;\n \n \t/* Check straight mapping */\n-\tstrbuf_reset(&sb);\n \tstrbuf_add(&sb, pattern, patternlen);\n \tfse.pattern = sb.buf;\n \tfse.patternlen = sb.len;\n@@ -155,7 +154,6 @@ static int check_directory_hashmap(struct hashmap *map, const char *pathname, in\n \tstruct fsexcludes fse;\n \n \t/* Check for directory */\n-\tstrbuf_reset(&sb);\n \tstrbuf_add(&sb, pathname, pathlen);\n \tstrbuf_addch(&sb, '/');\n \tfse.pattern = sb.buf;\n@@ -198,13 +196,16 @@ int fsexcludes_is_excluded_from(struct index_state *istate,\n \treturn -1;\n }\n \n-void fsexcludes_init(struct strbuf *sb) {\n+void fsexcludes_init(struct strbuf *sb)\n+{\n \tfsexcludes_initialized = 1;\n \tfsexcludes_data = *sb;\n+\tstrbuf_detach(sb, NULL);\n }\n \n-void fsexcludes_free() {\n+void fsexcludes_free(void) {\n \tstrbuf_release(&fsexcludes_data);\n \thashmap_free(&fsexcludes_hashmap, 1);\n \thashmap_free(&parent_directory_hashmap, 1);\n+\tfsexcludes_initialized = 0;\n }\ndiff --git a/fsexcludes.h b/fsexcludes.h\nindex 1c4101343c..10246daa02 100644\n--- a/fsexcludes.h\n+++ b/fsexcludes.h\n@@ -6,16 +6,18 @@\n  * where git will scan for untracked files.  This is used to speed up the\n  * scan by avoiding scanning parts of the work directory that do not have\n  * any new files.\n- *\n  */\n \n /*\n  * sb should contain a NUL separated list of path names of the files\n  * and/or directories that should be checked.  Any path not listed will\n  * be excluded from the scan.\n+ *\n+ * NOTE: fsexcludes_init() will take ownership of the storage passed in\n+ * sb and will reset sb to `STRBUF_INIT`\n  */\n void fsexcludes_init(struct strbuf *sb);\n-void fsexcludes_free();\n+void fsexcludes_free(void);\n \n /*\n  * Return 1 for exclude, 0 for include and -1 for undecided.\n\n\n### Patches\n\nBen Peart (2):\n  fsexcludes: add a programmatic way to exclude files from git's working\n    directory traversal logic\n  fsmonitor: switch to use new fsexcludes logic and remove unused\n    untracked cache based logic\n\n Makefile                    |   1 +\n dir.c                       |  33 ++++--\n dir.h                       |   2 -\n fsexcludes.c                | 211 ++++++++++++++++++++++++++++++++++++\n fsexcludes.h                |  29 +++++\n fsmonitor.c                 |  21 +---\n fsmonitor.h                 |  10 +-\n t/t7519-status-fsmonitor.sh |  14 +--\n 8 files changed, 273 insertions(+), 48 deletions(-)\n create mode 100644 fsexcludes.c\n create mode 100644 fsexcludes.h\n\n\nbase-commit: 0b0cc9f86731f894cff8dd25299a9b38c254569e\n-- \n2.17.0.windows.1\n\n\n"},{"id":"344480","messageId":"20180411200007.8612-2-benpeart@microsoft.com","threadId":"48263","inReplyTo":"20180411200007.8612-1-benpeart@microsoft.com","subject":"[PATCH v2 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2018-04-11T20:01:34Z","receivedAt":"2018-04-11T20:01:43Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"The File System Excludes module is a new programmatic way to exclude files and\nfolders from git's traversal of the working directory.  fsexcludes_init() should\nbe called with a string buffer that contains a NUL separated list of path names\nof the files and/or directories that should be included.  Any path not listed\nwill be excluded. The paths should be relative to the root of the working\ndirectory and be separated by a single NUL.\n\nThe excludes logic in dir.c has been updated to honor the results of\nfsexcludes_is_excluded_from().  If fsexcludes does not exclude the file, the\nnormal excludes logic is also checked as it could further reduce the set of\nfiles that should be included.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n Makefile     |   1 +\n dir.c        |  23 +++++-\n fsexcludes.c | 211 +++++++++++++++++++++++++++++++++++++++++++++++++++\n fsexcludes.h |  29 +++++++\n 4 files changed, 262 insertions(+), 2 deletions(-)\n create mode 100644 fsexcludes.c\n create mode 100644 fsexcludes.h\n\ndiff --git a/Makefile b/Makefile\nindex 96f6138f63..c102d2f75a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -819,6 +819,7 @@ LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-object.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n+LIB_OBJS += fsexcludes.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\ndiff --git a/dir.c b/dir.c\nindex 63a917be45..1aa639b9f4 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -18,6 +18,7 @@\n #include \"utf8.h\"\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsexcludes.h\"\n #include \"fsmonitor.h\"\n \n /*\n@@ -1102,6 +1103,12 @@ int is_excluded_from_list(const char *pathname,\n \t\t\t  struct exclude_list *el, struct index_state *istate)\n {\n \tstruct exclude *exclude;\n+\n+\tif (*dtype == DT_UNKNOWN)\n+\t\t*dtype = get_dtype(NULL, istate, pathname, pathlen);\n+\tif (fsexcludes_is_excluded_from(istate, pathname, pathlen, *dtype) > 0)\n+\t\treturn 1;\n+\n \texclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n \t\t\t\t\t\t  dtype, el, istate);\n \tif (exclude)\n@@ -1317,8 +1324,15 @@ struct exclude *last_exclude_matching(struct dir_struct *dir,\n int is_excluded(struct dir_struct *dir, struct index_state *istate,\n \t\tconst char *pathname, int *dtype_p)\n {\n-\tstruct exclude *exclude =\n-\t\tlast_exclude_matching(dir, istate, pathname, dtype_p);\n+\tstruct exclude *exclude;\n+\tint pathlen = strlen(pathname);\n+\n+\tif (*dtype_p == DT_UNKNOWN)\n+\t\t*dtype_p = get_dtype(NULL, istate, pathname, pathlen);\n+\tif (fsexcludes_is_excluded_from(istate, pathname, pathlen, *dtype_p) > 0)\n+\t\treturn 1;\n+\n+\texclude = last_exclude_matching(dir, istate, pathname, dtype_p);\n \tif (exclude)\n \t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n \treturn 0;\n@@ -1671,6 +1685,9 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \tif (dtype != DT_DIR && has_path_in_index)\n \t\treturn path_none;\n \n+\tif (fsexcludes_is_excluded_from(istate, path->buf, path->len, dtype) > 0)\n+\t\treturn path_excluded;\n+\n \t/*\n \t * When we are looking at a directory P in the working tree,\n \t * there are three cases:\n@@ -2011,6 +2028,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t/* add the path to the appropriate result list */\n \t\tswitch (state) {\n \t\tcase path_excluded:\n+\t\t\tif (fsexcludes_is_excluded_from(istate, path.buf, path.len, DTYPE(cdir.de)) > 0)\n+\t\t\t\tbreak;\n \t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n \t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n \t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\ndiff --git a/fsexcludes.c b/fsexcludes.c\nnew file mode 100644\nindex 0000000000..0ef57f107b\n--- /dev/null\n+++ b/fsexcludes.c\n@@ -0,0 +1,211 @@\n+#include \"cache.h\"\n+#include \"fsexcludes.h\"\n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n+static int fsexcludes_initialized = 0;\n+static struct strbuf fsexcludes_data = STRBUF_INIT;\n+static struct hashmap fsexcludes_hashmap;\n+static struct hashmap parent_directory_hashmap;\n+\n+struct fsexcludes {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tconst char *pattern;\n+\tint patternlen;\n+};\n+\n+static unsigned int(*fsexcludeshash)(const void *buf, size_t len);\n+static int(*fsexcludescmp)(const char *a, const char *b, size_t len);\n+\n+static int fsexcludes_hashmap_cmp(const void *unused_cmp_data,\n+\tconst void *a, const void *b, const void *key)\n+{\n+\tconst struct fsexcludes *fse1 = a;\n+\tconst struct fsexcludes *fse2 = b;\n+\n+\treturn fsexcludescmp(fse1->pattern, fse2->pattern, fse1->patternlen);\n+}\n+\n+static int check_fsexcludes_hashmap(struct hashmap *map, const char *pattern, int patternlen)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct fsexcludes fse;\n+\tchar *slash;\n+\n+\t/* Check straight mapping */\n+\tstrbuf_add(&sb, pattern, patternlen);\n+\tfse.pattern = sb.buf;\n+\tfse.patternlen = sb.len;\n+\thashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n+\tif (hashmap_get(map, &fse, NULL)) {\n+\t\tstrbuf_release(&sb);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * Check to see if it matches a directory or any path\n+\t * underneath it.  In other words, 'a/b/foo.txt' will match\n+\t * '/', 'a/', and 'a/b/'.\n+\t */\n+\tslash = strchr(sb.buf, '/');\n+\twhile (slash) {\n+\t\tfse.pattern = sb.buf;\n+\t\tfse.patternlen = slash - sb.buf + 1;\n+\t\thashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n+\t\tif (hashmap_get(map, &fse, NULL)) {\n+\t\t\tstrbuf_release(&sb);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tslash = strchr(slash + 1, '/');\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn 1;\n+}\n+\n+static void fsexcludes_hashmap_add(struct hashmap *map, const char *pattern, const int patternlen)\n+{\n+\tstruct fsexcludes *fse;\n+\n+\tfse = xmalloc(sizeof(struct fsexcludes));\n+\tfse->pattern = pattern;\n+\tfse->patternlen = patternlen;\n+\thashmap_entry_init(fse, fsexcludeshash(fse->pattern, fse->patternlen));\n+\thashmap_add(map, fse);\n+}\n+\n+static void initialize_fsexcludes_hashmap(struct hashmap *map, struct strbuf *fsexcludes_data)\n+{\n+\tchar *buf, *entry;\n+\tsize_t len;\n+\tint i;\n+\n+\t/*\n+\t * Build a hashmap of the fsexcludes data we can use to look\n+\t * for cache entry matches quickly\n+\t */\n+\tfsexcludeshash = ignore_case ? memihash : memhash;\n+\tfsexcludescmp = ignore_case ? strncasecmp : strncmp;\n+\thashmap_init(map, fsexcludes_hashmap_cmp, NULL, 0);\n+\n+\tentry = buf = fsexcludes_data->buf;\n+\tlen = fsexcludes_data->len;\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (buf[i] == '\\0') {\n+\t\t\tfsexcludes_hashmap_add(map, entry, buf + i - entry);\n+\t\t\tentry = buf + i + 1;\n+\t\t}\n+\t}\n+}\n+\n+static void parent_directory_hashmap_add(struct hashmap *map, const char *pattern, const int patternlen)\n+{\n+\tchar *slash;\n+\tstruct fsexcludes *fse;\n+\n+\t/*\n+\t * Add any directories leading up to the file as the excludes logic\n+\t * needs to match directories leading up to the files as well. Detect\n+\t * and prevent unnecessary duplicate entries which will be common.\n+\t */\n+\tif (patternlen > 1) {\n+\t\tslash = strchr(pattern + 1, '/');\n+\t\twhile (slash) {\n+\t\t\tfse = xmalloc(sizeof(struct fsexcludes));\n+\t\t\tfse->pattern = pattern;\n+\t\t\tfse->patternlen = slash - pattern + 1;\n+\t\t\thashmap_entry_init(fse, fsexcludeshash(fse->pattern, fse->patternlen));\n+\t\t\tif (hashmap_get(map, fse, NULL))\n+\t\t\t\tfree(fse);\n+\t\t\telse\n+\t\t\t\thashmap_add(map, fse);\n+\t\t\tslash = strchr(slash + 1, '/');\n+\t\t}\n+\t}\n+}\n+\n+static void initialize_parent_directory_hashmap(struct hashmap *map, struct strbuf *vfs_data)\n+{\n+\tchar *buf, *entry;\n+\tsize_t len;\n+\tint i;\n+\n+\t/*\n+\t * Build a hashmap of the parent directories contained in the virtual\n+\t * file system data we can use to look for matches quickly\n+\t */\n+\tfsexcludeshash = ignore_case ? memihash : memhash;\n+\tfsexcludescmp = ignore_case ? strncasecmp : strncmp;\n+\thashmap_init(map, fsexcludes_hashmap_cmp, NULL, 0);\n+\n+\tentry = buf = vfs_data->buf;\n+\tlen = vfs_data->len;\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (buf[i] == '\\0') {\n+\t\t\tparent_directory_hashmap_add(map, entry, buf + i - entry);\n+\t\t\tentry = buf + i + 1;\n+\t\t}\n+\t}\n+}\n+\n+static int check_directory_hashmap(struct hashmap *map, const char *pathname, int pathlen)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct fsexcludes fse;\n+\n+\t/* Check for directory */\n+\tstrbuf_add(&sb, pathname, pathlen);\n+\tstrbuf_addch(&sb, '/');\n+\tfse.pattern = sb.buf;\n+\tfse.patternlen = sb.len;\n+\thashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n+\tif (hashmap_get(map, &fse, NULL)) {\n+\t\tstrbuf_release(&sb);\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn 1;\n+}\n+\n+/*\n+ * Return 1 for exclude, 0 for include and -1 for undecided.\n+ */\n+int fsexcludes_is_excluded_from(struct index_state *istate,\n+\tconst char *pathname, int pathlen, int dtype)\n+{\n+\tif (!fsexcludes_initialized)\n+\t\treturn -1;\n+\n+\tif (dtype == DT_REG) {\n+\t\t/* lazily init the hashmap */\n+\t\tif (!fsexcludes_hashmap.cmpfn_data)\n+\t\t\tinitialize_fsexcludes_hashmap(&fsexcludes_hashmap, &fsexcludes_data);\n+\n+\t\treturn check_fsexcludes_hashmap(&fsexcludes_hashmap, pathname, pathlen);\n+\t}\n+\n+\tif (dtype == DT_DIR || dtype == DT_LNK) {\n+\t\t/* lazily init the hashmap */\n+\t\tif (!parent_directory_hashmap.cmpfn_data)\n+\t\t\tinitialize_parent_directory_hashmap(&parent_directory_hashmap, &fsexcludes_data);\n+\n+\t\treturn check_directory_hashmap(&parent_directory_hashmap, pathname, pathlen);\n+\t}\n+\n+\treturn -1;\n+}\n+\n+void fsexcludes_init(struct strbuf *sb)\n+{\n+\tfsexcludes_initialized = 1;\n+\tfsexcludes_data = *sb;\n+\tstrbuf_detach(sb, NULL);\n+}\n+\n+void fsexcludes_free(void) {\n+\tstrbuf_release(&fsexcludes_data);\n+\thashmap_free(&fsexcludes_hashmap, 1);\n+\thashmap_free(&parent_directory_hashmap, 1);\n+\tfsexcludes_initialized = 0;\n+}\ndiff --git a/fsexcludes.h b/fsexcludes.h\nnew file mode 100644\nindex 0000000000..10246daa02\n--- /dev/null\n+++ b/fsexcludes.h\n@@ -0,0 +1,29 @@\n+#ifndef FSEXCLUDES_H\n+#define FSEXCLUDES_H\n+\n+/*\n+ * The file system excludes functions provides a way to programatically limit\n+ * where git will scan for untracked files.  This is used to speed up the\n+ * scan by avoiding scanning parts of the work directory that do not have\n+ * any new files.\n+ */\n+\n+/*\n+ * sb should contain a NUL separated list of path names of the files\n+ * and/or directories that should be checked.  Any path not listed will\n+ * be excluded from the scan.\n+ *\n+ * NOTE: fsexcludes_init() will take ownership of the storage passed in\n+ * sb and will reset sb to `STRBUF_INIT`\n+ */\n+void fsexcludes_init(struct strbuf *sb);\n+void fsexcludes_free(void);\n+\n+/*\n+ * Return 1 for exclude, 0 for include and -1 for undecided.\n+ */\n+int fsexcludes_is_excluded_from(struct index_state *istate,\n+\tconst char *pathname, int pathlen, int dtype_p);\n+\n+\n+#endif\n-- \n2.17.0.windows.1\n\n"},{"id":"344481","messageId":"20180411200007.8612-3-benpeart@microsoft.com","threadId":"48263","inReplyTo":"20180411200007.8612-1-benpeart@microsoft.com","subject":"[PATCH v2 2/2] fsmonitor: switch to use new fsexcludes logic and remove unused untracked cache based logic","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2018-04-11T20:01:41Z","receivedAt":"2018-04-11T20:01:51Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Update fsmonitor to utilize the new fsexcludes based logic for excluding paths\nthat do not need to be scaned for new or modified files.  Remove the old logic\nin dir.c that utilized the untracked cache (if enabled) to accomplish the same\ngoal.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n dir.c                       | 10 +++-------\n dir.h                       |  2 --\n fsmonitor.c                 | 21 ++-------------------\n fsmonitor.h                 | 10 +++-------\n t/t7519-status-fsmonitor.sh | 14 +++-----------\n 5 files changed, 11 insertions(+), 46 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 1aa639b9f4..28c2c83f76 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -19,7 +19,6 @@\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n #include \"fsexcludes.h\"\n-#include \"fsmonitor.h\"\n \n /*\n  * Tells read_directory_recursive how a file or directory should be treated.\n@@ -1827,12 +1826,9 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n-\t/*\n-\t * With fsmonitor, we can trust the untracked cache's valid field.\n-\t */\n-\trefresh_fsmonitor(istate);\n-\tif (!(dir->untracked->use_fsmonitor && untracked->valid)) {\n-\t\tif (lstat(path->len ? path->buf : \".\", &st)) {\n+\tif (!untracked->valid) {\n+\t\tif (stat(path->len ? path->buf : \".\", &st)) {\n+\t\t\tinvalidate_directory(dir->untracked, untracked);\n \t\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n \t\t\treturn 0;\n \t\t}\ndiff --git a/dir.h b/dir.h\nindex b0758b82a2..e67ccfbb29 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -139,8 +139,6 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n-\t/* fsmonitor invalidation data */\n-\tunsigned int use_fsmonitor : 1;\n };\n \n struct dir_struct {\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 6d7bcd5d0e..dd67eef851 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsexcludes.h\"\n #include \"fsmonitor.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n@@ -125,12 +126,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, const char *n\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t}\n \n-\t/*\n-\t * Mark the untracked cache dirty even if it wasn't found in the index\n-\t * as it could be a new untracked file.\n-\t */\n \ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n-\tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n void refresh_fsmonitor(struct index_state *istate)\n@@ -184,11 +180,8 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t/* Mark all entries invalid */\n \t\tfor (i = 0; i < istate->cache_nr; i++)\n \t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\n-\t\tif (istate->untracked)\n-\t\t\tistate->untracked->use_fsmonitor = 0;\n \t}\n-\tstrbuf_release(&query_result);\n+\tfsexcludes_init(&query_result);\n \n \t/* Now that we've updated istate, save the last_update time */\n \tistate->fsmonitor_last_update = last_update;\n@@ -207,12 +200,6 @@ void add_fsmonitor(struct index_state *istate)\n \t\tfor (i = 0; i < istate->cache_nr; i++)\n \t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \n-\t\t/* reset the untracked cache */\n-\t\tif (istate->untracked) {\n-\t\t\tadd_untracked_cache(istate);\n-\t\t\tistate->untracked->use_fsmonitor = 1;\n-\t\t}\n-\n \t\t/* Update the fsmonitor state */\n \t\trefresh_fsmonitor(istate);\n \t}\n@@ -241,10 +228,6 @@ void tweak_fsmonitor(struct index_state *istate)\n \n \t\t\t/* Mark all previously saved entries as dirty */\n \t\t\tewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n-\n-\t\t\t/* Now mark the untracked cache for fsmonitor usage */\n-\t\t\tif (istate->untracked)\n-\t\t\t\tistate->untracked->use_fsmonitor = 1;\n \t\t}\n \n \t\tewah_free(istate->fsmonitor_dirty);\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 65f3743636..f7adfc1f7c 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -35,8 +35,7 @@ extern void tweak_fsmonitor(struct index_state *istate);\n \n /*\n  * Run the configured fsmonitor integration script and clear the\n- * CE_FSMONITOR_VALID bit for any files returned as dirty.  Also invalidate\n- * any corresponding untracked cache directory structures. Optimized to only\n+ * CE_FSMONITOR_VALID bit for any files returned as dirty. Optimized to only\n  * run the first time it is called.\n  */\n extern void refresh_fsmonitor(struct index_state *istate);\n@@ -55,17 +54,14 @@ static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n }\n \n /*\n- * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate\n- * any corresponding untracked cache directory structures. This should\n+ * Clear the given cache entry's CE_FSMONITOR_VALID bit. This should\n  * be called any time git creates or modifies a file that should\n- * trigger an lstat() or invalidate the untracked cache for the\n- * corresponding directory\n+ * trigger an lstat() for the corresponding directory\n  */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n \tif (core_fsmonitor) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\n \t}\n }\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 756beb0d8e..d6a1da5a0a 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -225,8 +225,7 @@ test_expect_success '*only* files returned by the integration script get flagged\n # Ensure commands that call refresh_index() to move the index back in time\n # properly invalidate the fsmonitor cache\n test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n-\tEOF\n+\twrite_integration_script &&\n \tclean_repo &&\n \tdirty_repo &&\n \tgit add . &&\n@@ -275,7 +274,7 @@ do\n \t\t'\n \n \t\t# Make sure it's actually skipping the check for modified and untracked\n-\t\t# (if enabled) files unless it is told about them.\n+\t\t# files unless it is told about them.\n \t\ttest_expect_success \"status doesn't detect unreported modifications\" '\n \t\t\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n \t\t\t:>marker\n@@ -288,14 +287,7 @@ do\n \t\t\tgit status >actual &&\n \t\t\ttest_path_is_file marker &&\n \t\t\ttest_i18ngrep ! \"Changes not staged for commit:\" actual &&\n-\t\t\tif test $uc_val = true\n-\t\t\tthen\n-\t\t\t\ttest_i18ngrep ! \"Untracked files:\" actual\n-\t\t\tfi &&\n-\t\t\tif test $uc_val = false\n-\t\t\tthen\n-\t\t\t\ttest_i18ngrep \"Untracked files:\" actual\n-\t\t\tfi &&\n+\t\t\ttest_i18ngrep ! \"Untracked files:\" actual &&\n \t\t\trm -f marker\n \t\t'\n \tdone\n-- \n2.17.0.windows.1\n\n"},{"id":"344510","messageId":"xmqqsh812td0.fsf@gitster-ct.c.googlers.com","threadId":"48263","inReplyTo":"20180411200007.8612-2-benpeart@microsoft.com","subject":"Re: [PATCH v2 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-11T23:52:59Z","receivedAt":"2018-04-11T23:53:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\nI haven't studied and thought about the motivation behind these two\npatches, but one thing I noticed...\n\nBen Peart <Ben.Peart@microsoft.com> writes:\n\n> diff --git a/dir.c b/dir.c\n> index 63a917be45..1aa639b9f4 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1102,6 +1103,12 @@ int is_excluded_from_list(const char *pathname,\n>  \t\t\t  struct exclude_list *el, struct index_state *istate)\n>  {\n>  \tstruct exclude *exclude;\n> +\n> +\tif (*dtype == DT_UNKNOWN)\n> +\t\t*dtype = get_dtype(NULL, istate, pathname, pathlen);\n> +\tif (fsexcludes_is_excluded_from(istate, pathname, pathlen, *dtype) > 0)\n> +\t\treturn 1;\n> +\n>  \texclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n>  \t\t\t\t\t\t  dtype, el, istate);\n>  \tif (exclude)\n> @@ -1317,8 +1324,15 @@ struct exclude *last_exclude_matching(struct dir_struct *dir,\n>  int is_excluded(struct dir_struct *dir, struct index_state *istate,\n>  \t\tconst char *pathname, int *dtype_p)\n>  {\n> -\tstruct exclude *exclude =\n> -\t\tlast_exclude_matching(dir, istate, pathname, dtype_p);\n> +\tstruct exclude *exclude;\n> +\tint pathlen = strlen(pathname);\n> +\n> +\tif (*dtype_p == DT_UNKNOWN)\n> +\t\t*dtype_p = get_dtype(NULL, istate, pathname, pathlen);\n> +\tif (fsexcludes_is_excluded_from(istate, pathname, pathlen, *dtype_p) > 0)\n> +\t\treturn 1;\n> +\n> +\texclude = last_exclude_matching(dir, istate, pathname, dtype_p);\n>  \tif (exclude)\n>  \t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n>  \treturn 0;\n\nA piece of impression I am getting from the above two hunks is that\nthe fsexcludes_is_excluded_from() function requires a real dtype in\nits last parameter (i.e. DT_UNKNOWN is not acceptable).\n\n> @@ -1671,6 +1685,9 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n>  \tif (dtype != DT_DIR && has_path_in_index)\n>  \t\treturn path_none;\n>  \n> +\tif (fsexcludes_is_excluded_from(istate, path->buf, path->len, dtype) > 0)\n> +\t\treturn path_excluded;\n> +\n\nAnd this hunk reinforces that impression (we are comparing dtype\nwith DT_DIR, so we know we cannot be passing DT_UNKNOWN to it).\n\n> @@ -2011,6 +2028,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n>  \t\t/* add the path to the appropriate result list */\n>  \t\tswitch (state) {\n>  \t\tcase path_excluded:\n> +\t\t\tif (fsexcludes_is_excluded_from(istate, path.buf, path.len, DTYPE(cdir.de)) > 0)\n> +\t\t\t\tbreak;\n\nThen the use of DTYPE() looks a bit odd here.  On\nNO_D_TYPE_IN_DIRENT platforms, we would get DT_UNKNOWN out of it and\nthen end up passing DT_UNKNOWN to the function.\n"},{"id":"344602","messageId":"0dc3c045-8851-cbed-6756-8dfda5e25868@gmail.com","threadId":"48263","inReplyTo":"xmqqsh812td0.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-04-13T11:53:24Z","receivedAt":"2018-04-13T11:53:33Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 4/11/2018 7:52 PM, Junio C Hamano wrote:\n>> @@ -2011,6 +2028,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n>>   \t\t/* add the path to the appropriate result list */\n>>   \t\tswitch (state) {\n>>   \t\tcase path_excluded:\n>> +\t\t\tif (fsexcludes_is_excluded_from(istate, path.buf, path.len, DTYPE(cdir.de)) > 0)\n>> +\t\t\t\tbreak;\n> \n> Then the use of DTYPE() looks a bit odd here.  On\n> NO_D_TYPE_IN_DIRENT platforms, we would get DT_UNKNOWN out of it and\n> then end up passing DT_UNKNOWN to the function.\n> \n\nGood catch.  I was trying to optimize this path and didn't realize the \nplatform implications of using DTYPE().  I'll update it to match the others.\n"},{"id":"344605","messageId":"20180413122218.1756-1-benpeart@microsoft.com","threadId":"48263","inReplyTo":"20180410210408.13788-1-benpeart@microsoft.com","subject":"[PATCH v3 0/2] fsexcludes: Add programmatic way to exclude files","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2018-04-13T12:22:42Z","receivedAt":"2018-04-13T12:22:49Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Only minor changes from V2: \n\nSwitched to using get_dtype() instead of DTYPE() for platform independence.\nCleaned up reverting of fsmonitor code in the untracked cache.\n\nBase Ref: master\nWeb-Diff: https://github.com/benpeart/git/commit/709470f33f\nCheckout: git fetch https://github.com/benpeart/git fsexcludes-v3 && git checkout 709470f33f\n\n\n\n### Patches\n\nBen Peart (2):\n  fsexcludes: add a programmatic way to exclude files from git's working\n    directory traversal logic\n  fsmonitor: switch to use new fsexcludes logic and remove unused\n    untracked cache based logic\n\n Makefile                    |   1 +\n dir.c                       |  47 +++++---\n dir.h                       |   2 -\n fsexcludes.c                | 211 ++++++++++++++++++++++++++++++++++++\n fsexcludes.h                |  29 +++++\n fsmonitor.c                 |  21 +---\n fsmonitor.h                 |  10 +-\n t/t7519-status-fsmonitor.sh |  14 +--\n 8 files changed, 279 insertions(+), 56 deletions(-)\n create mode 100644 fsexcludes.c\n create mode 100644 fsexcludes.h\n\n\nbase-commit: fe0a9eaf31dd0c349ae4308498c33a5c3794b293\n-- \n2.17.0.windows.1\n\n\n"},{"id":"344606","messageId":"20180413122218.1756-2-benpeart@microsoft.com","threadId":"48263","inReplyTo":"20180413122218.1756-1-benpeart@microsoft.com","subject":"[PATCH v3 1/2] fsexcludes: add a programmatic way to exclude files from git's working directory traversal logic","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2018-04-13T12:22:52Z","receivedAt":"2018-04-13T12:23:00Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"The File System Excludes module is a new programmatic way to exclude files and\nfolders from git's traversal of the working directory.  fsexcludes_init() should\nbe called with a string buffer that contains a NUL separated list of path names\nof the files and/or directories that should be included.  Any path not listed\nwill be excluded. The paths should be relative to the root of the working\ndirectory and be separated by a single NUL.\n\nThe excludes logic in dir.c has been updated to honor the results of\nfsexcludes_is_excluded_from().  If fsexcludes does not exclude the file, the\nnormal excludes logic is also checked as it could further reduce the set of\nfiles that should be included.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n Makefile     |   1 +\n dir.c        |  24 +++++-\n fsexcludes.c | 211 +++++++++++++++++++++++++++++++++++++++++++++++++++\n fsexcludes.h |  29 +++++++\n 4 files changed, 263 insertions(+), 2 deletions(-)\n create mode 100644 fsexcludes.c\n create mode 100644 fsexcludes.h\n\ndiff --git a/Makefile b/Makefile\nindex f181687250..a4f1471272 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -822,6 +822,7 @@ LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-object.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n+LIB_OBJS += fsexcludes.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\ndiff --git a/dir.c b/dir.c\nindex 63a917be45..47a073efe1 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -18,6 +18,7 @@\n #include \"utf8.h\"\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsexcludes.h\"\n #include \"fsmonitor.h\"\n \n /*\n@@ -1102,6 +1103,12 @@ int is_excluded_from_list(const char *pathname,\n \t\t\t  struct exclude_list *el, struct index_state *istate)\n {\n \tstruct exclude *exclude;\n+\n+\tif (*dtype == DT_UNKNOWN)\n+\t\t*dtype = get_dtype(NULL, istate, pathname, pathlen);\n+\tif (fsexcludes_is_excluded_from(istate, pathname, pathlen, *dtype) > 0)\n+\t\treturn 1;\n+\n \texclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n \t\t\t\t\t\t  dtype, el, istate);\n \tif (exclude)\n@@ -1317,8 +1324,15 @@ struct exclude *last_exclude_matching(struct dir_struct *dir,\n int is_excluded(struct dir_struct *dir, struct index_state *istate,\n \t\tconst char *pathname, int *dtype_p)\n {\n-\tstruct exclude *exclude =\n-\t\tlast_exclude_matching(dir, istate, pathname, dtype_p);\n+\tstruct exclude *exclude;\n+\tint pathlen = strlen(pathname);\n+\n+\tif (*dtype_p == DT_UNKNOWN)\n+\t\t*dtype_p = get_dtype(NULL, istate, pathname, pathlen);\n+\tif (fsexcludes_is_excluded_from(istate, pathname, pathlen, *dtype_p) > 0)\n+\t\treturn 1;\n+\n+\texclude = last_exclude_matching(dir, istate, pathname, dtype_p);\n \tif (exclude)\n \t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n \treturn 0;\n@@ -1671,6 +1685,9 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \tif (dtype != DT_DIR && has_path_in_index)\n \t\treturn path_none;\n \n+\tif (fsexcludes_is_excluded_from(istate, path->buf, path->len, dtype) > 0)\n+\t\treturn path_excluded;\n+\n \t/*\n \t * When we are looking at a directory P in the working tree,\n \t * there are three cases:\n@@ -2011,6 +2028,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t/* add the path to the appropriate result list */\n \t\tswitch (state) {\n \t\tcase path_excluded:\n+\t\t\tif (fsexcludes_is_excluded_from(istate, path.buf, path.len,\n+\t\t\t\t\tget_dtype(cdir.de, istate, path.buf, path.len)) > 0)\n+\t\t\t\tbreak;\n \t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n \t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n \t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\ndiff --git a/fsexcludes.c b/fsexcludes.c\nnew file mode 100644\nindex 0000000000..0ef57f107b\n--- /dev/null\n+++ b/fsexcludes.c\n@@ -0,0 +1,211 @@\n+#include \"cache.h\"\n+#include \"fsexcludes.h\"\n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n+static int fsexcludes_initialized = 0;\n+static struct strbuf fsexcludes_data = STRBUF_INIT;\n+static struct hashmap fsexcludes_hashmap;\n+static struct hashmap parent_directory_hashmap;\n+\n+struct fsexcludes {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tconst char *pattern;\n+\tint patternlen;\n+};\n+\n+static unsigned int(*fsexcludeshash)(const void *buf, size_t len);\n+static int(*fsexcludescmp)(const char *a, const char *b, size_t len);\n+\n+static int fsexcludes_hashmap_cmp(const void *unused_cmp_data,\n+\tconst void *a, const void *b, const void *key)\n+{\n+\tconst struct fsexcludes *fse1 = a;\n+\tconst struct fsexcludes *fse2 = b;\n+\n+\treturn fsexcludescmp(fse1->pattern, fse2->pattern, fse1->patternlen);\n+}\n+\n+static int check_fsexcludes_hashmap(struct hashmap *map, const char *pattern, int patternlen)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct fsexcludes fse;\n+\tchar *slash;\n+\n+\t/* Check straight mapping */\n+\tstrbuf_add(&sb, pattern, patternlen);\n+\tfse.pattern = sb.buf;\n+\tfse.patternlen = sb.len;\n+\thashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n+\tif (hashmap_get(map, &fse, NULL)) {\n+\t\tstrbuf_release(&sb);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * Check to see if it matches a directory or any path\n+\t * underneath it.  In other words, 'a/b/foo.txt' will match\n+\t * '/', 'a/', and 'a/b/'.\n+\t */\n+\tslash = strchr(sb.buf, '/');\n+\twhile (slash) {\n+\t\tfse.pattern = sb.buf;\n+\t\tfse.patternlen = slash - sb.buf + 1;\n+\t\thashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n+\t\tif (hashmap_get(map, &fse, NULL)) {\n+\t\t\tstrbuf_release(&sb);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tslash = strchr(slash + 1, '/');\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn 1;\n+}\n+\n+static void fsexcludes_hashmap_add(struct hashmap *map, const char *pattern, const int patternlen)\n+{\n+\tstruct fsexcludes *fse;\n+\n+\tfse = xmalloc(sizeof(struct fsexcludes));\n+\tfse->pattern = pattern;\n+\tfse->patternlen = patternlen;\n+\thashmap_entry_init(fse, fsexcludeshash(fse->pattern, fse->patternlen));\n+\thashmap_add(map, fse);\n+}\n+\n+static void initialize_fsexcludes_hashmap(struct hashmap *map, struct strbuf *fsexcludes_data)\n+{\n+\tchar *buf, *entry;\n+\tsize_t len;\n+\tint i;\n+\n+\t/*\n+\t * Build a hashmap of the fsexcludes data we can use to look\n+\t * for cache entry matches quickly\n+\t */\n+\tfsexcludeshash = ignore_case ? memihash : memhash;\n+\tfsexcludescmp = ignore_case ? strncasecmp : strncmp;\n+\thashmap_init(map, fsexcludes_hashmap_cmp, NULL, 0);\n+\n+\tentry = buf = fsexcludes_data->buf;\n+\tlen = fsexcludes_data->len;\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (buf[i] == '\\0') {\n+\t\t\tfsexcludes_hashmap_add(map, entry, buf + i - entry);\n+\t\t\tentry = buf + i + 1;\n+\t\t}\n+\t}\n+}\n+\n+static void parent_directory_hashmap_add(struct hashmap *map, const char *pattern, const int patternlen)\n+{\n+\tchar *slash;\n+\tstruct fsexcludes *fse;\n+\n+\t/*\n+\t * Add any directories leading up to the file as the excludes logic\n+\t * needs to match directories leading up to the files as well. Detect\n+\t * and prevent unnecessary duplicate entries which will be common.\n+\t */\n+\tif (patternlen > 1) {\n+\t\tslash = strchr(pattern + 1, '/');\n+\t\twhile (slash) {\n+\t\t\tfse = xmalloc(sizeof(struct fsexcludes));\n+\t\t\tfse->pattern = pattern;\n+\t\t\tfse->patternlen = slash - pattern + 1;\n+\t\t\thashmap_entry_init(fse, fsexcludeshash(fse->pattern, fse->patternlen));\n+\t\t\tif (hashmap_get(map, fse, NULL))\n+\t\t\t\tfree(fse);\n+\t\t\telse\n+\t\t\t\thashmap_add(map, fse);\n+\t\t\tslash = strchr(slash + 1, '/');\n+\t\t}\n+\t}\n+}\n+\n+static void initialize_parent_directory_hashmap(struct hashmap *map, struct strbuf *vfs_data)\n+{\n+\tchar *buf, *entry;\n+\tsize_t len;\n+\tint i;\n+\n+\t/*\n+\t * Build a hashmap of the parent directories contained in the virtual\n+\t * file system data we can use to look for matches quickly\n+\t */\n+\tfsexcludeshash = ignore_case ? memihash : memhash;\n+\tfsexcludescmp = ignore_case ? strncasecmp : strncmp;\n+\thashmap_init(map, fsexcludes_hashmap_cmp, NULL, 0);\n+\n+\tentry = buf = vfs_data->buf;\n+\tlen = vfs_data->len;\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (buf[i] == '\\0') {\n+\t\t\tparent_directory_hashmap_add(map, entry, buf + i - entry);\n+\t\t\tentry = buf + i + 1;\n+\t\t}\n+\t}\n+}\n+\n+static int check_directory_hashmap(struct hashmap *map, const char *pathname, int pathlen)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct fsexcludes fse;\n+\n+\t/* Check for directory */\n+\tstrbuf_add(&sb, pathname, pathlen);\n+\tstrbuf_addch(&sb, '/');\n+\tfse.pattern = sb.buf;\n+\tfse.patternlen = sb.len;\n+\thashmap_entry_init(&fse, fsexcludeshash(fse.pattern, fse.patternlen));\n+\tif (hashmap_get(map, &fse, NULL)) {\n+\t\tstrbuf_release(&sb);\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn 1;\n+}\n+\n+/*\n+ * Return 1 for exclude, 0 for include and -1 for undecided.\n+ */\n+int fsexcludes_is_excluded_from(struct index_state *istate,\n+\tconst char *pathname, int pathlen, int dtype)\n+{\n+\tif (!fsexcludes_initialized)\n+\t\treturn -1;\n+\n+\tif (dtype == DT_REG) {\n+\t\t/* lazily init the hashmap */\n+\t\tif (!fsexcludes_hashmap.cmpfn_data)\n+\t\t\tinitialize_fsexcludes_hashmap(&fsexcludes_hashmap, &fsexcludes_data);\n+\n+\t\treturn check_fsexcludes_hashmap(&fsexcludes_hashmap, pathname, pathlen);\n+\t}\n+\n+\tif (dtype == DT_DIR || dtype == DT_LNK) {\n+\t\t/* lazily init the hashmap */\n+\t\tif (!parent_directory_hashmap.cmpfn_data)\n+\t\t\tinitialize_parent_directory_hashmap(&parent_directory_hashmap, &fsexcludes_data);\n+\n+\t\treturn check_directory_hashmap(&parent_directory_hashmap, pathname, pathlen);\n+\t}\n+\n+\treturn -1;\n+}\n+\n+void fsexcludes_init(struct strbuf *sb)\n+{\n+\tfsexcludes_initialized = 1;\n+\tfsexcludes_data = *sb;\n+\tstrbuf_detach(sb, NULL);\n+}\n+\n+void fsexcludes_free(void) {\n+\tstrbuf_release(&fsexcludes_data);\n+\thashmap_free(&fsexcludes_hashmap, 1);\n+\thashmap_free(&parent_directory_hashmap, 1);\n+\tfsexcludes_initialized = 0;\n+}\ndiff --git a/fsexcludes.h b/fsexcludes.h\nnew file mode 100644\nindex 0000000000..10246daa02\n--- /dev/null\n+++ b/fsexcludes.h\n@@ -0,0 +1,29 @@\n+#ifndef FSEXCLUDES_H\n+#define FSEXCLUDES_H\n+\n+/*\n+ * The file system excludes functions provides a way to programatically limit\n+ * where git will scan for untracked files.  This is used to speed up the\n+ * scan by avoiding scanning parts of the work directory that do not have\n+ * any new files.\n+ */\n+\n+/*\n+ * sb should contain a NUL separated list of path names of the files\n+ * and/or directories that should be checked.  Any path not listed will\n+ * be excluded from the scan.\n+ *\n+ * NOTE: fsexcludes_init() will take ownership of the storage passed in\n+ * sb and will reset sb to `STRBUF_INIT`\n+ */\n+void fsexcludes_init(struct strbuf *sb);\n+void fsexcludes_free(void);\n+\n+/*\n+ * Return 1 for exclude, 0 for include and -1 for undecided.\n+ */\n+int fsexcludes_is_excluded_from(struct index_state *istate,\n+\tconst char *pathname, int pathlen, int dtype_p);\n+\n+\n+#endif\n-- \n2.17.0.windows.1\n\n"},{"id":"344607","messageId":"20180413122218.1756-3-benpeart@microsoft.com","threadId":"48263","inReplyTo":"20180413122218.1756-1-benpeart@microsoft.com","subject":"[PATCH v3 2/2] fsmonitor: switch to use new fsexcludes logic and remove unused untracked cache based logic","fromName":"Ben Peart","fromEmail":"ben.peart@microsoft.com","sentAt":"2018-04-13T12:22:58Z","receivedAt":"2018-04-13T12:23:05Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Update fsmonitor to utilize the new fsexcludes based logic for excluding paths\nthat do not need to be scaned for new or modified files.  Remove the old logic\nin dir.c that utilized the untracked cache (if enabled) to accomplish the same\ngoal.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n dir.c                       | 23 ++++++++---------------\n dir.h                       |  2 --\n fsmonitor.c                 | 21 ++-------------------\n fsmonitor.h                 | 10 +++-------\n t/t7519-status-fsmonitor.sh | 14 +++-----------\n 5 files changed, 16 insertions(+), 54 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 47a073efe1..b1859f4311 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -19,7 +19,6 @@\n #include \"varint.h\"\n #include \"ewah/ewok.h\"\n #include \"fsexcludes.h\"\n-#include \"fsmonitor.h\"\n \n /*\n  * Tells read_directory_recursive how a file or directory should be treated.\n@@ -1827,20 +1826,14 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n-\t/*\n-\t * With fsmonitor, we can trust the untracked cache's valid field.\n-\t */\n-\trefresh_fsmonitor(istate);\n-\tif (!(dir->untracked->use_fsmonitor && untracked->valid)) {\n-\t\tif (lstat(path->len ? path->buf : \".\", &st)) {\n-\t\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n-\t\t\treturn 0;\n-\t\t}\n-\t\tif (!untracked->valid ||\n-\t\t\tmatch_stat_data_racy(istate, &untracked->stat_data, &st)) {\n-\t\t\tfill_stat_data(&untracked->stat_data, &st);\n-\t\t\treturn 0;\n-\t\t}\n+\tif (stat(path->len ? path->buf : \".\", &st)) {\n+\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n+\t\treturn 0;\n+\t}\n+\tif (!untracked->valid ||\n+\t    match_stat_data_racy(istate, &untracked->stat_data, &st)) {\n+\t\tfill_stat_data(&untracked->stat_data, &st);\n+\t\treturn 0;\n \t}\n \n \tif (untracked->check_only != !!check_only)\ndiff --git a/dir.h b/dir.h\nindex b0758b82a2..e67ccfbb29 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -139,8 +139,6 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n-\t/* fsmonitor invalidation data */\n-\tunsigned int use_fsmonitor : 1;\n };\n \n struct dir_struct {\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 6d7bcd5d0e..dd67eef851 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n+#include \"fsexcludes.h\"\n #include \"fsmonitor.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n@@ -125,12 +126,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, const char *n\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t}\n \n-\t/*\n-\t * Mark the untracked cache dirty even if it wasn't found in the index\n-\t * as it could be a new untracked file.\n-\t */\n \ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n-\tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n void refresh_fsmonitor(struct index_state *istate)\n@@ -184,11 +180,8 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t/* Mark all entries invalid */\n \t\tfor (i = 0; i < istate->cache_nr; i++)\n \t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\n-\t\tif (istate->untracked)\n-\t\t\tistate->untracked->use_fsmonitor = 0;\n \t}\n-\tstrbuf_release(&query_result);\n+\tfsexcludes_init(&query_result);\n \n \t/* Now that we've updated istate, save the last_update time */\n \tistate->fsmonitor_last_update = last_update;\n@@ -207,12 +200,6 @@ void add_fsmonitor(struct index_state *istate)\n \t\tfor (i = 0; i < istate->cache_nr; i++)\n \t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \n-\t\t/* reset the untracked cache */\n-\t\tif (istate->untracked) {\n-\t\t\tadd_untracked_cache(istate);\n-\t\t\tistate->untracked->use_fsmonitor = 1;\n-\t\t}\n-\n \t\t/* Update the fsmonitor state */\n \t\trefresh_fsmonitor(istate);\n \t}\n@@ -241,10 +228,6 @@ void tweak_fsmonitor(struct index_state *istate)\n \n \t\t\t/* Mark all previously saved entries as dirty */\n \t\t\tewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n-\n-\t\t\t/* Now mark the untracked cache for fsmonitor usage */\n-\t\t\tif (istate->untracked)\n-\t\t\t\tistate->untracked->use_fsmonitor = 1;\n \t\t}\n \n \t\tewah_free(istate->fsmonitor_dirty);\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 65f3743636..f7adfc1f7c 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -35,8 +35,7 @@ extern void tweak_fsmonitor(struct index_state *istate);\n \n /*\n  * Run the configured fsmonitor integration script and clear the\n- * CE_FSMONITOR_VALID bit for any files returned as dirty.  Also invalidate\n- * any corresponding untracked cache directory structures. Optimized to only\n+ * CE_FSMONITOR_VALID bit for any files returned as dirty. Optimized to only\n  * run the first time it is called.\n  */\n extern void refresh_fsmonitor(struct index_state *istate);\n@@ -55,17 +54,14 @@ static inline void mark_fsmonitor_valid(struct cache_entry *ce)\n }\n \n /*\n- * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate\n- * any corresponding untracked cache directory structures. This should\n+ * Clear the given cache entry's CE_FSMONITOR_VALID bit. This should\n  * be called any time git creates or modifies a file that should\n- * trigger an lstat() or invalidate the untracked cache for the\n- * corresponding directory\n+ * trigger an lstat() for the corresponding directory\n  */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n \tif (core_fsmonitor) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\n \t}\n }\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 756beb0d8e..d6a1da5a0a 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -225,8 +225,7 @@ test_expect_success '*only* files returned by the integration script get flagged\n # Ensure commands that call refresh_index() to move the index back in time\n # properly invalidate the fsmonitor cache\n test_expect_success 'refresh_index() invalidates fsmonitor cache' '\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n-\tEOF\n+\twrite_integration_script &&\n \tclean_repo &&\n \tdirty_repo &&\n \tgit add . &&\n@@ -275,7 +274,7 @@ do\n \t\t'\n \n \t\t# Make sure it's actually skipping the check for modified and untracked\n-\t\t# (if enabled) files unless it is told about them.\n+\t\t# files unless it is told about them.\n \t\ttest_expect_success \"status doesn't detect unreported modifications\" '\n \t\t\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n \t\t\t:>marker\n@@ -288,14 +287,7 @@ do\n \t\t\tgit status >actual &&\n \t\t\ttest_path_is_file marker &&\n \t\t\ttest_i18ngrep ! \"Changes not staged for commit:\" actual &&\n-\t\t\tif test $uc_val = true\n-\t\t\tthen\n-\t\t\t\ttest_i18ngrep ! \"Untracked files:\" actual\n-\t\t\tfi &&\n-\t\t\tif test $uc_val = false\n-\t\t\tthen\n-\t\t\t\ttest_i18ngrep \"Untracked files:\" actual\n-\t\t\tfi &&\n+\t\t\ttest_i18ngrep ! \"Untracked files:\" actual &&\n \t\t\trm -f marker\n \t\t'\n \tdone\n-- \n2.17.0.windows.1\n\n"},{"id":"344688","messageId":"CACsJy8B+fTfUiGA-cFE5QEipa_4pSfC4_GhUSJvCKL82G21xWw@mail.gmail.com","threadId":"48263","inReplyTo":"20180410210408.13788-1-benpeart@microsoft.com","subject":"Re: [PATCH v1 0/2] fsexcludes: Add programmatic way to exclude files","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-04-14T15:59:30Z","receivedAt":"2018-04-14T16:00:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Apr 10, 2018 at 11:04 PM, Ben Peart <Ben.Peart@microsoft.com> wrote:\n> In git repos with large working directories an external file system monitor\n> (like fsmonitor or gvfs) can track what files in the working directory have been\n> modified.  This information can be used to speed up git operations that scale\n> based on the size of the working directory so that they become O(# of modified\n> files) vs O(# of files in the working directory).\n>\n> The fsmonitor patch series added logic to limit what files git had to stat() to\n> the set of modified files provided by the fsmonitor hook proc.  It also used the\n> untracked cache (if enabled) to limit the files/folders git had to scan looking\n> for new/untracked files.  GVFS is another external file system model that also\n> speeds up git working directory based operations that has been using a different\n> mechanism (programmatically generating an excludes file) to enable git to be\n> O(# of modified files).\n>\n> This patch series will introduce a new way to limit git�s traversal of the\n> working directory that does not require the untracked cache (fsmonitor) or using\n> the excludes feature (GVFS).  It does this by enhancing the existing excludes\n> logic in dir.c to support a new �File System Excludes� or fsexcludes API that is\n> better tuned to these programmatic applications.\n\nI have not had a chance to really look at the patches yet but I think\nthese three paragraphs should somehow be included in the commit\ndescription of 1/2 (or spread out between 1/2 and 2/2). 1/2\ndescription for example briefly talks about how to use the new thing,\nbut not really tell what it's for, why you need to add it.\n-- \nDuy\n"},{"id":"344991","messageId":"0de30972-b0a2-67e8-7cff-c19daf9ece8b@gmail.com","threadId":"48263","inReplyTo":"20180413122218.1756-1-benpeart@microsoft.com","subject":"Re: [PATCH v3 0/2] fsexcludes: Add programmatic way to exclude files","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-04-18T15:31:15Z","receivedAt":"2018-04-18T15:31:22Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"I found a bug with how this patch series deals with untracked files. \nI'm going to retract this patch until I have time to create a new test \ncase to demonstrate the bug and come up with a good fix.\n\nBen\n"},{"id":"345023","messageId":"xmqqin8ogqaj.fsf@gitster-ct.c.googlers.com","threadId":"48263","inReplyTo":"0de30972-b0a2-67e8-7cff-c19daf9ece8b@gmail.com","subject":"Re: [PATCH v3 0/2] fsexcludes: Add programmatic way to exclude files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-18T21:25:56Z","receivedAt":"2018-04-18T21:26:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart <peartben@gmail.com> writes:\n\n> I found a bug with how this patch series deals with untracked\n> files. I'm going to retract this patch until I have time to create a\n> new test case to demonstrate the bug and come up with a good fix.\n>\n> Ben\n\nThanks.\n"}]}