{"thread":{"id":"36571","subject":"Watchman support for git","startedAt":"2014-05-02T23:14:08Z","lastAt":"2014-05-19T10:10:38Z","messageCount":43,"participants":["dturner@twopensource.com","Felipe Contreras","Duy Nguyen","David Turner","Junio C Hamano","Jeff King","Sebastian Schuberth","David Lang"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"240594","messageId":"1399072451-15561-1-git-send-email-dturner@twopensource.com","threadId":"36571","inReplyTo":null,"subject":"Watchman support for git","fromName":"","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-02T23:14:08Z","receivedAt":"2014-05-02T23:14:08Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"The most sigificant patch uses Facebook's watchman daemon[1] to monitor\nthe repository work tree for changes.  This makes allows git status\nto avoid traversing the entire work tree to find changes.\n\nThis change requires libwatchman[2], a client library that I wrote for\nwatchman.\n\nWhile making the watchman change, I also made a change to the index\nformat (contributed here in a separate patch).  Index integrity\nchecking uses the same SHA1 algorithm as the rest of git; this is\nactually relatively slow.  It's not a huge part of run-time, but since\nI wanted to do the same checking for the Watchman filesystem cache\n(which is about twice as large as the index), I decided to optimize it\nanyway.  I switched to VMAC.  VMAC is supposed to be a MAC, but\nthere's no reason it can't be used with a fixed key as a simple\nintegrity check.  VMAC is roughly five times faster than SHA1 on my\nmachine; This adds up to a 5% overal speed improvement on git status\n(depending on the structure of your repo, and about 15% on git diff\n--cached with no cached changes).\n\nThe index format change might be less important with the split index;\nI haven't investigated that since at the time I wrote these patches,\nit didn't exist.\n\nSome numbers follow.  They are on my laptop, which has 4x i5-2520M\nprocessors, 8GB of RAM, and a solid state disk.  They're all tested\nwith a hot cache.\n\nTest repository 1: Linux\n\nLinux is about 45k files in 3k directories.  The average length of a\nfilename is about 32 bytes.\n\nGit status timing:\nno watchman: 125ms\nwatchman: 90ms\n\nTest repository 2: Superscience\n\nMy second test repository (which is a semi-synthetic repo generated\nfrom various Twitter internal repos) is somewhat larger than this, and\ngets a correspondingly larger improvement.  It is about 65k files in\n20k directories; the average length of a filename is 67 bytes.\n\nGit status timing:\nno watchman, index version 4: 370 ms\nno watchman, index version 5: 365 ms\nwatchman, index version 4: 170 ms\nwatchman, index version 5: 165 ms\n\n\n[1] https://github.com/facebook/watchman\n[2] https://github.com/twitter/libwatchman\n"},{"id":"240595","messageId":"1399072451-15561-2-git-send-email-dturner@twopensource.com","threadId":"36571","inReplyTo":"1399072451-15561-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 1/3] After chdir to run grep, return to old directory","fromName":"","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-02T23:14:09Z","receivedAt":"2014-05-02T23:14:09Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner <dturner@twitter.com>\n\nSigned-off-by: David Turner <dturner@twitter.com>\n---\n builtin/grep.c | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 69ac2d8..e9fe040 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -355,15 +355,25 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n {\n \tstruct string_list *path_list = opt->output_priv;\n \tconst char **argv = xmalloc(sizeof(const char *) * (path_list->nr + 1));\n+\tstatic char old_directory[PATH_MAX+1];\n \tint i, status;\n \n \tfor (i = 0; i < path_list->nr; i++)\n \t\targv[i] = path_list->items[i].string;\n \targv[path_list->nr] = NULL;\n \n-\tif (prefix && chdir(prefix))\n-\t\tdie(_(\"Failed to chdir: %s\"), prefix);\n+\n+\tif (prefix) {\n+\t\tif (!getcwd(old_directory, PATH_MAX+1))\n+\t\t\tdie(_(\"Failed to get cwd: %s\"), prefix);\n+\t\tif (chdir(prefix))\n+\t\t\tdie(_(\"Failed to chdir: %s\"), prefix);\n+\t}\n \tstatus = run_command_v_opt(argv, RUN_USING_SHELL);\n+\tif (prefix)\n+\t\tif (chdir(old_directory))\n+\t\t\tdie(_(\"Failed to chdir: %s\"), old_directory);\n+\n \tif (status)\n \t\texit(status);\n \tfree(argv);\n-- \n2.0.0.rc0.31.g69c1a2d\n"},{"id":"240596","messageId":"1399072451-15561-4-git-send-email-dturner@twopensource.com","threadId":"36571","inReplyTo":"1399072451-15561-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 3/3] Watchman support","fromName":"","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-02T23:14:11Z","receivedAt":"2014-05-02T23:14:11Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner <dturner@twitter.com>\n\nAdd support for filesystem view caching and updating via Facebook's\nWatchman daemon.\n\nSigned-off-by: David Turner <dturner@twitter.com>\n---\n Documentation/watchman.txt       |  27 ++\n Makefile                         |   9 +\n cache.h                          |   4 +\n config.c                         |  10 +\n configure.ac                     |   8 +\n diff-lib.c                       |  70 +++--\n dir.c                            | 210 +++++++++----\n dir.h                            |  18 +-\n environment.c                    |  18 +-\n fs_cache.c                       | 645 +++++++++++++++++++++++++++++++++++++++\n fs_cache.h                       | 138 +++++++++\n hash-io.c                        |   4 +-\n read-cache.c                     |  61 +++-\n t/t1012-read-tree-df.sh          |   2 +-\n t/t2201-add-update-typechange.sh |   5 +-\n t/t2204-add-ignored.sh           |   6 +-\n t/t6001-rev-list-graft.sh        |   2 +-\n t/t7900-watchman.sh              | 249 +++++++++++++++\n t/test-lib.sh                    |   1 +\n unpack-trees.c                   |   2 +-\n watchman-support.c               | 640 ++++++++++++++++++++++++++++++++++++++\n watchman-support.h               |  10 +\n 22 files changed, 2045 insertions(+), 94 deletions(-)\n create mode 100644 Documentation/watchman.txt\n create mode 100644 fs_cache.c\n create mode 100644 fs_cache.h\n create mode 100755 t/t7900-watchman.sh\n create mode 100644 watchman-support.c\n create mode 100644 watchman-support.h\n\ndiff --git a/Documentation/watchman.txt b/Documentation/watchman.txt\nnew file mode 100644\nindex 0000000..b849e98\n--- /dev/null\n+++ b/Documentation/watchman.txt\n@@ -0,0 +1,27 @@\n+How git uses watchman\n+---------------------\n+\n+Git status (and some other commands) have to determine which files\n+have changed between the working copy and the index.  Ordinarily, this\n+requires checking every file in the working directory.  But if you\n+have watchman (https://github.com/facebook/watchman) installed, git\n+can cache the state of the working directory and use watchman to track\n+file changes, making commands like git status faster.\n+\n+set core.usewatchman = true to use watchman.\n+You can also set\n+core.watchmansynctimeout = [number of milliseconds]\n+to change watchman's sync timeout; see the watchman docs for details\n+on this.  You should only change this if you see watchman timeout\n+error messages.\n+\n+Internals\n+---------\n+\n+The filesystem cache stores information about every file in the\n+working tree.  In almost every case where git calls lstat or\n+opendir/readdir, the modified file cache can be consulted instead.\n+\n+The file system cache is stored on disk in .git/fs_cache.  It is\n+stored very similarly to the index, except without path prefix\n+compression.\ndiff --git a/Makefile b/Makefile\nindex 4cd642d..0c10a28 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -667,6 +667,7 @@ LIB_H += ewah/ewok.h\n LIB_H += ewah/ewok_rlw.h\n LIB_H += fetch-pack.h\n LIB_H += fmt-merge-msg.h\n+LIB_H += fs_cache.h\n LIB_H += fsck.h\n LIB_H += gettext.h\n LIB_H += git-compat-util.h\n@@ -808,6 +809,7 @@ LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-pack.o\n+LIB_OBJS += fs_cache.o\n LIB_OBJS += fsck.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n@@ -1445,6 +1447,13 @@ ifdef RUNTIME_PREFIX\n \tCOMPAT_CFLAGS += -DRUNTIME_PREFIX\n endif\n \n+ifdef USE_WATCHMAN\n+\tLIB_H += watchman-support.h\n+\tLIB_OBJS += watchman-support.o\n+\tEXTLIBS += $(WATCHMAN_LIBS)\n+\tBASIC_CFLAGS += -DUSE_WATCHMAN\n+endif\n+\n ifdef NO_PTHREADS\n \tBASIC_CFLAGS += -DNO_PTHREADS\n else\ndiff --git a/cache.h b/cache.h\nindex c5917f4..023f9c9 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -272,6 +272,7 @@ struct index_state {\n \tstruct cache_entry **cache;\n \tunsigned int version;\n \tunsigned int cache_nr, cache_alloc, cache_changed;\n+\tstruct fs_cache *fs_cache;\n \tstruct string_list *resolve_undo;\n \tstruct cache_tree *cache_tree;\n \tstruct cache_time timestamp;\n@@ -352,6 +353,7 @@ static inline enum object_type object_type(unsigned int mode)\n #define DEFAULT_GIT_DIR_ENVIRONMENT \".git\"\n #define DB_ENVIRONMENT \"GIT_OBJECT_DIRECTORY\"\n #define INDEX_ENVIRONMENT \"GIT_INDEX_FILE\"\n+#define FS_CACHE_ENVIRONMENT \"GIT_FS_CACHE_FILE\"\n #define GRAFT_ENVIRONMENT \"GIT_GRAFT_FILE\"\n #define GIT_SHALLOW_FILE_ENVIRONMENT \"GIT_SHALLOW_FILE\"\n #define TEMPLATE_DIR_ENVIRONMENT \"GIT_TEMPLATE_DIR\"\n@@ -594,6 +596,8 @@ extern int check_replace_refs;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n+extern int core_use_watchman;\n+extern int core_watchman_sync_timeout;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n \ndiff --git a/config.c b/config.c\nindex a30cb5c..6bbdac4 100644\n--- a/config.c\n+++ b/config.c\n@@ -854,6 +854,16 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.usewatchman\")) {\n+\t\tcore_use_watchman = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(var, \"core.watchmansynctimeout\")) {\n+\t\tcore_watchman_sync_timeout = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.createobject\")) {\n \t\tif (!strcmp(value, \"rename\"))\n \t\t\tobject_creation_mode = OBJECT_CREATION_USES_RENAMES;\ndiff --git a/configure.ac b/configure.ac\nindex b711254..76fcb83 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -962,6 +962,14 @@ GIT_CONF_SUBST([NO_INITGROUPS])\n #\n # Define NO_ICONV if your libc does not properly support iconv.\n \n+# Check for watchman client library\n+\n+AC_CHECK_LIB([watchman], [watchman_connect],\n+       [WATCHMAN_LIBS=\"-lwatchman\"\n+\tUSE_WATCHMAN=YesPlease],\n+       [USE_WATCHMAN=])\n+GIT_CONF_SUBST([WATCHMAN_LIBS])\n+GIT_CONF_SUBST([USE_WATCHMAN])\n \n ## Other checks.\n # Define USE_PIC if you need the main git objects to be built with -fPIC\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 0448729..bbeca00 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -2,6 +2,7 @@\n  * Copyright (C) 2005 Junio C Hamano\n  */\n #include \"cache.h\"\n+#include \"fs_cache.h\"\n #include \"quote.h\"\n #include \"commit.h\"\n #include \"diff.h\"\n@@ -17,24 +18,8 @@\n  * diff-files\n  */\n \n-/*\n- * Has the work tree entity been removed?\n- *\n- * Return 1 if it was removed from the work tree, 0 if an entity to be\n- * compared with the cache entry ce still exists (the latter includes\n- * the case where a directory that is not a submodule repository\n- * exists for ce that is a submodule -- it is a submodule that is not\n- * checked out).  Return negative for an error.\n- */\n-static int check_removed(const struct cache_entry *ce, struct stat *st)\n+static int check_gitlink(const struct cache_entry *ce, struct stat *st)\n {\n-\tif (lstat(ce->name, st) < 0) {\n-\t\tif (errno != ENOENT && errno != ENOTDIR)\n-\t\t\treturn -1;\n-\t\treturn 1;\n-\t}\n-\tif (has_symlink_leading_path(ce->name, ce_namelen(ce)))\n-\t\treturn 1;\n \tif (S_ISDIR(st->st_mode)) {\n \t\tunsigned char sub[20];\n \n@@ -56,6 +41,57 @@ static int check_removed(const struct cache_entry *ce, struct stat *st)\n \treturn 0;\n }\n \n+static int fs_cache_check_removed(const struct fs_cache *fs_cache, const struct cache_entry *ce, struct stat *st)\n+{\n+\tstruct fsc_entry *fe;\n+\n+\tfe = fs_cache_file_exists(fs_cache, ce->name, ce_namelen(ce));\n+\tif (!fe) {\n+\t\treturn 1;\n+\t}\n+\tif (fe_deleted(fe)) {\n+\t\treturn 1;\n+\t}\n+\n+\tfe_to_stat(fe, st);\n+\n+/* This should not be necessary, because the fs_cache does not store\n+   children of symlinked directories.  TODO: ensure this\n+\tif (has_symlink_leading_path(ce->name, ce_namelen(ce)))\n+\t\treturn 1;\n+*/\n+\tif (check_gitlink(ce, st))\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Has the work tree entity been removed?\n+ *\n+ * Return 1 if it was removed from the work tree, 0 if an entity to be\n+ * compared with the cache entry ce still exists (the latter includes\n+ * the case where a directory that is not a submodule repository\n+ * exists for ce that is a submodule -- it is a submodule that is not\n+ * checked out).  Return negative for an error.\n+ */\n+static int check_removed(const struct cache_entry *ce, struct stat *st)\n+{\n+\tif (the_index.fs_cache)\n+\t\treturn fs_cache_check_removed(the_index.fs_cache, ce, st);\n+\n+\tif (lstat(ce->name, st) < 0) {\n+\t\tif (errno != ENOENT && errno != ENOTDIR)\n+\t\t\treturn -1;\n+\t\treturn 1;\n+\t}\n+\tif (has_symlink_leading_path(ce->name, ce_namelen(ce)))\n+\t\treturn 1;\n+\tif (check_gitlink(ce, st))\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n /*\n  * Has a file changed or has a submodule new commits or a dirty work tree?\n  *\ndiff --git a/dir.c b/dir.c\nindex eb6f581..41258ce 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -8,6 +8,7 @@\n  *\t\t Junio Hamano, 2005-2006\n  */\n #include \"cache.h\"\n+#include \"fs_cache.h\"\n #include \"dir.h\"\n #include \"refs.h\"\n #include \"wildmatch.h\"\n@@ -33,8 +34,8 @@ enum path_treatment {\n \n static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tconst char *path, int len,\n-\tint check_only, const struct path_simplify *simplify);\n-static int get_dtype(struct dirent *de, const char *path, int len);\n+\tint check_only, const struct path_simplify *simplify,\n+\tstruct fsc_entry *fe);\n \n /* helper string functions with support for the ignore_case flag */\n int strcmp_icase(const char *a, const char *b)\n@@ -179,7 +180,8 @@ int fill_directory(struct dir_struct *dir, const struct pathspec *pathspec)\n \tlen = common_prefix_len(pathspec);\n \n \t/* Read the directory and prune it */\n-\tread_directory(dir, pathspec->nr ? pathspec->_raw[0] : \"\", len, pathspec);\n+\tread_directory(&the_index, dir, pathspec->nr ? pathspec->_raw[0] : \"\",\n+\t\t       len, pathspec);\n \treturn len;\n }\n \n@@ -536,7 +538,7 @@ int add_excludes_from_file_to_list(const char *fname,\n \tsize_t size = 0;\n \tchar *buf, *entry;\n \n-\tfd = open(fname, O_RDONLY);\n+\tfd = fs_cache_open(the_index.fs_cache, fname, O_RDONLY);\n \tif (fd < 0 || fstat(fd, &st) < 0) {\n \t\tif (errno != ENOENT)\n \t\t\twarn_on_inaccessible(fname);\n@@ -597,6 +599,8 @@ struct exclude_list *add_exclude_list(struct dir_struct *dir,\n \tel = &group->el[group->nr++];\n \tmemset(el, 0, sizeof(*el));\n \tel->src = src;\n+\tif (group_type == EXC_FILE)\n+\t\tdir->flags &= ~DIR_STD_EXCLUDES;\n \treturn el;\n }\n \n@@ -609,6 +613,7 @@ void add_excludes_from_file(struct dir_struct *dir, const char *fname)\n \tel = add_exclude_list(dir, EXC_FILE, fname);\n \tif (add_excludes_from_file_to_list(fname, \"\", 0, el, 0) < 0)\n \t\tdie(\"cannot use %s as an exclude file\", fname);\n+\tdir->flags &= ~DIR_STD_EXCLUDES;\n }\n \n int match_basename(const char *basename, int basenamelen,\n@@ -763,7 +768,9 @@ static struct exclude *last_exclude_matching_from_lists(struct dir_struct *dir,\n \tint i, j;\n \tstruct exclude_list_group *group;\n \tstruct exclude *exclude;\n-\tfor (i = EXC_CMDL; i <= EXC_FILE; i++) {\n+\tint last = dir->flags & DIR_EXCLUDE_CMDL_ONLY ? EXC_CMDL : EXC_FILE;\n+\n+\tfor (i = EXC_CMDL; i <= last; i++) {\n \t\tgroup = &dir->exclude_list_group[i];\n \t\tfor (j = group->nr - 1; j >= 0; j--) {\n \t\t\texclude = last_exclude_matching_from_list(\n@@ -852,6 +859,7 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \n \t\t/* Try to read per-directory file unless path is too long */\n \t\tif (dir->exclude_per_dir &&\n+\t\t    !(dir->flags & DIR_EXCLUDE_CMDL_ONLY) &&\n \t\t    stk->baselen + strlen(dir->exclude_per_dir) < PATH_MAX) {\n \t\t\tstrcpy(dir->basebuf + stk->baselen,\n \t\t\t\t\tdir->exclude_per_dir);\n@@ -910,6 +918,17 @@ int is_excluded(struct dir_struct *dir, const char *pathname, int *dtype_p)\n \treturn 0;\n }\n \n+static int fs_cache_is_excluded(struct dir_struct *dir, const char *pathname, int *dtype_p, struct fsc_entry *fe)\n+{\n+\tstruct exclude *exclude;\n+\texclude = last_exclude_matching(dir, pathname, dtype_p);\n+\tif (exclude)\n+\t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n+\tif (dir->flags & DIR_STD_EXCLUDES && fe)\n+\t\treturn fe_excluded(fe);\n+\treturn 0;\n+}\n+\n static struct dir_entry *dir_entry_new(const char *pathname, int len)\n {\n \tstruct dir_entry *ent;\n@@ -1047,7 +1066,7 @@ static enum exist_status directory_exists_in_index(const char *dirname, int len)\n  */\n static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int exclude,\n-\tconst struct path_simplify *simplify)\n+\tconst struct path_simplify *simplify, struct fsc_entry *fe)\n {\n \t/* The \"len-1\" is to strip the final '/' */\n \tswitch (directory_exists_in_index(dirname, len-1)) {\n@@ -1073,7 +1092,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n \t\treturn exclude ? path_excluded : path_untracked;\n \n-\treturn read_directory_recursive(dir, dirname, len, 1, simplify);\n+\treturn read_directory_recursive(dir, dirname, len, 1, simplify, fe);\n }\n \n /*\n@@ -1167,7 +1186,7 @@ static int get_index_dtype(const char *path, int len)\n \treturn DT_UNKNOWN;\n }\n \n-static int get_dtype(struct dirent *de, const char *path, int len)\n+int get_dtype(struct dirent *de, const char *path, int len)\n {\n \tint dtype = de ? DTYPE(de) : DT_UNKNOWN;\n \tstruct stat st;\n@@ -1191,7 +1210,8 @@ static int get_dtype(struct dirent *de, const char *path, int len)\n static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t\t\t\t\t  struct strbuf *path,\n \t\t\t\t\t  const struct path_simplify *simplify,\n-\t\t\t\t\t  int dtype, struct dirent *de)\n+\t\t\t\t\t  int dtype, struct dirent *de,\n+\t\t\t\t\t  struct fsc_entry *fe)\n {\n \tint exclude;\n \tint has_path_in_index = !!cache_file_exists(path->buf, path->len, ignore_case);\n@@ -1227,7 +1247,7 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t    (directory_exists_in_index(path->buf, path->len) == index_nonexistent))\n \t\treturn path_none;\n \n-\texclude = is_excluded(dir, path->buf, &dtype);\n+\texclude = fs_cache_is_excluded(dir, path->buf, &dtype, fe);\n \n \t/*\n \t * Excluded? If we don't explicitly want to show\n@@ -1242,7 +1262,7 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \tcase DT_DIR:\n \t\tstrbuf_addch(path, '/');\n \t\treturn treat_directory(dir, path->buf, path->len, exclude,\n-\t\t\tsimplify);\n+\t\t\tsimplify, fe);\n \tcase DT_REG:\n \tcase DT_LNK:\n \t\treturn exclude ? path_excluded : path_untracked;\n@@ -1253,7 +1273,8 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t      struct dirent *de,\n \t\t\t\t      struct strbuf *path,\n \t\t\t\t      int baselen,\n-\t\t\t\t      const struct path_simplify *simplify)\n+\t\t\t\t      const struct path_simplify *simplify,\n+\t\t\t\t      struct fsc_entry *fe)\n {\n \tint dtype;\n \n@@ -1265,7 +1286,59 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\treturn path_none;\n \n \tdtype = DTYPE(de);\n-\treturn treat_one_path(dir, path, simplify, dtype, de);\n+\treturn treat_one_path(dir, path, simplify, dtype, de, fe);\n+}\n+\n+static int handle(struct dir_struct *dir, const char *base, int baselen,\n+\t\t  int check_only, const struct path_simplify *simplify,\n+\t\t  struct dirent *de, enum path_treatment *dir_state,\n+\t\t  struct strbuf *path, struct fsc_entry *fe)\n+{\n+\tenum path_treatment state, subdir_state;\n+\n+\t/* check how the file or directory should be treated */\n+\tstate = treat_path(dir, de, path, baselen, simplify, fe);\n+\n+\tif (state > *dir_state)\n+\t\t*dir_state = state;\n+\n+\t/* recurse into subdir if instructed by treat_path */\n+\tif (state == path_recurse) {\n+\t\tsubdir_state = read_directory_recursive(dir, path->buf,\n+\t\t\tpath->len, check_only, simplify, fe);\n+\t\tif (subdir_state > *dir_state)\n+\t\t\t*dir_state = subdir_state;\n+\t}\n+\n+\tif (check_only) {\n+\t\t/* abort early if maximum state has been reached */\n+\t\tif (*dir_state == path_untracked)\n+\t\t\treturn 1;\n+\t\t/* skip the dir_add_* part */\n+\t\treturn 0;\n+\t}\n+\n+\t/* add the path to the appropriate result list */\n+\tswitch (state) {\n+\tcase path_excluded:\n+\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\tdir_add_name(dir, path->buf, path->len);\n+\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n+\t\t\t ((dir->flags & DIR_COLLECT_IGNORED) &&\n+\t\t\t  exclude_matches_pathspec(path->buf, path->len,\n+\t\t\t\tsimplify)))\n+\t\t\tdir_add_ignored(dir, path->buf, path->len);\n+\t\tbreak;\n+\n+\tcase path_untracked:\n+\t\tif (!(dir->flags & DIR_SHOW_IGNORED))\n+\t\t\tdir_add_name(dir, path->buf, path->len);\n+\t\tbreak;\n+\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn 0;\n }\n \n /*\n@@ -1282,63 +1355,44 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t    const char *base, int baselen,\n \t\t\t\t    int check_only,\n-\t\t\t\t    const struct path_simplify *simplify)\n+\t\t\t\t    const struct path_simplify *simplify,\n+\t\t\t\t    struct fsc_entry *parent)\n {\n \tDIR *fdir;\n-\tenum path_treatment state, subdir_state, dir_state = path_none;\n-\tstruct dirent *de;\n+\tenum path_treatment dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n \n \tstrbuf_add(&path, base, baselen);\n \n-\tfdir = opendir(path.len ? path.buf : \".\");\n-\tif (!fdir)\n-\t\tgoto out;\n-\n-\twhile ((de = readdir(fdir)) != NULL) {\n-\t\t/* check how the file or directory should be treated */\n-\t\tstate = treat_path(dir, de, &path, baselen, simplify);\n-\t\tif (state > dir_state)\n-\t\t\tdir_state = state;\n-\n-\t\t/* recurse into subdir if instructed by treat_path */\n-\t\tif (state == path_recurse) {\n-\t\t\tsubdir_state = read_directory_recursive(dir, path.buf,\n-\t\t\t\tpath.len, check_only, simplify);\n-\t\t\tif (subdir_state > dir_state)\n-\t\t\t\tdir_state = subdir_state;\n-\t\t}\n+\tif (the_index.fs_cache) {\n+\t\tstruct fsc_entry *fe;\n \n-\t\tif (check_only) {\n-\t\t\t/* abort early if maximum state has been reached */\n-\t\t\tif (dir_state == path_untracked)\n+\t\tif (!parent)\n+\t\t\tgoto out;\n+\n+\t\tfor (fe = parent->first_child; fe; fe = fe->next_sibling) {\n+\t\t\tstruct dirent de;\n+\t\t\tif (fe_deleted(fe))\n+\t\t\t\tcontinue;\n+\t\t\tde.d_ino = -1;\n+\t\t\tde.d_type = fe_dtype(fe);\n+\t\t\tstrcpy(de.d_name, basename(fe->path));\n+\t\t\tif (handle(dir, base, baselen, check_only, simplify, &de, &dir_state, &path, fe))\n \t\t\t\tbreak;\n-\t\t\t/* skip the dir_add_* part */\n-\t\t\tcontinue;\n \t\t}\n+\t} else {\n+\t\tstruct dirent *de;\n+\t\tfdir = opendir(path.len ? path.buf : \".\");\n+\t\tif (!fdir)\n+\t\t\tgoto out;\n \n-\t\t/* add the path to the appropriate result list */\n-\t\tswitch (state) {\n-\t\tcase path_excluded:\n-\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n-\t\t\t\tdir_add_name(dir, path.buf, path.len);\n-\t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t\t((dir->flags & DIR_COLLECT_IGNORED) &&\n-\t\t\t\texclude_matches_pathspec(path.buf, path.len,\n-\t\t\t\t\tsimplify)))\n-\t\t\t\tdir_add_ignored(dir, path.buf, path.len);\n-\t\t\tbreak;\n-\n-\t\tcase path_untracked:\n-\t\t\tif (!(dir->flags & DIR_SHOW_IGNORED))\n-\t\t\t\tdir_add_name(dir, path.buf, path.len);\n-\t\t\tbreak;\n-\n-\t\tdefault:\n-\t\t\tbreak;\n+\t\twhile ((de = readdir(fdir)) != NULL) {\n+\t\t\tif (handle(dir, base, baselen, check_only, simplify, de, &dir_state, &path, NULL))\n+\t\t\t\tbreak;\n \t\t}\n+\t\tclosedir(fdir);\n \t}\n-\tclosedir(fdir);\n+\n  out:\n \tstrbuf_release(&path);\n \n@@ -1383,7 +1437,8 @@ static void free_simplify(struct path_simplify *simplify)\n \n static int treat_leading_path(struct dir_struct *dir,\n \t\t\t      const char *path, int len,\n-\t\t\t      const struct path_simplify *simplify)\n+\t\t\t      const struct path_simplify *simplify,\n+\t\t\t      struct fsc_entry *fe)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint baselen, rc = 0;\n@@ -1410,7 +1465,7 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tif (simplify_away(sb.buf, sb.len, simplify))\n \t\t\tbreak;\n \t\tif (treat_one_path(dir, &sb, simplify,\n-\t\t\t\t   DT_DIR, NULL) == path_none)\n+\t\t\t\t   DT_DIR, NULL, fe) == path_none)\n \t\t\tbreak; /* do not recurse into it */\n \t\tif (len <= baselen) {\n \t\t\trc = 1;\n@@ -1422,9 +1477,13 @@ static int treat_leading_path(struct dir_struct *dir,\n \treturn rc;\n }\n \n-int read_directory(struct dir_struct *dir, const char *path, int len, const struct pathspec *pathspec)\n+\n+int read_directory(struct index_state *istate, struct dir_struct *dir,\n+\t\t   const char *path, int len, const struct pathspec *pathspec)\n {\n \tstruct path_simplify *simplify;\n+\tint saved_flags = dir->flags;\n+\tstruct fsc_entry *fe = NULL;\n \n \t/*\n \t * Check out create_simplify()\n@@ -1448,11 +1507,32 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const stru\n \t * create_simplify().\n \t */\n \tsimplify = create_simplify(pathspec ? pathspec->_raw : NULL);\n-\tif (!len || treat_leading_path(dir, path, len, simplify))\n-\t\tread_directory_recursive(dir, path, len, 0, simplify);\n+\n+\t/*\n+\t  Check for standard excludes.\n+\t  Standard excludes means: exclude_per_dir\n+\t */\n+\tif (!dir->exclude_per_dir || strcmp(dir->exclude_per_dir, \".gitignore\"))\n+\t\tdir->flags &= ~DIR_STD_EXCLUDES;\n+\n+\tif (istate->fs_cache && dir->flags & DIR_STD_EXCLUDES) {\n+\t\tdir->flags |= DIR_EXCLUDE_CMDL_ONLY;\n+\t}\n+\n+\tif (istate->fs_cache) {\n+\t\tint len_no_slash = len;\n+\t\tif (len && path[len - 1] == '/')\n+\t\t\tlen_no_slash --;\n+\t\tfe = fs_cache_file_exists(istate->fs_cache, path, len_no_slash);\n+\t}\n+\n+\tif (!len || treat_leading_path(dir, path, len, simplify, fe)) {\n+\t\tread_directory_recursive(dir, path, len, 0, simplify, fe);\n+\t}\n \tfree_simplify(simplify);\n \tqsort(dir->entries, dir->nr, sizeof(struct dir_entry *), cmp_name);\n \tqsort(dir->ignored, dir->ignored_nr, sizeof(struct dir_entry *), cmp_name);\n+\tdir->flags = saved_flags;\n \treturn dir->nr;\n }\n \n@@ -1608,6 +1688,7 @@ void setup_standard_excludes(struct dir_struct *dir)\n {\n \tconst char *path;\n \tchar *xdg_path;\n+\tint previously_empty;\n \n \tdir->exclude_per_dir = \".gitignore\";\n \tpath = git_path(\"info/exclude\");\n@@ -1615,10 +1696,13 @@ void setup_standard_excludes(struct dir_struct *dir)\n \t\thome_config_paths(NULL, &xdg_path, \"ignore\");\n \t\texcludes_file = xdg_path;\n \t}\n+\tpreviously_empty = dir->exclude_list_group[EXC_FILE].nr == 0;\n \tif (!access_or_warn(path, R_OK, 0))\n \t\tadd_excludes_from_file(dir, path);\n \tif (excludes_file && !access_or_warn(excludes_file, R_OK, 0))\n \t\tadd_excludes_from_file(dir, excludes_file);\n+\tif (previously_empty)\n+\t\tdir->flags |= DIR_STD_EXCLUDES;\n }\n \n int remove_path(const char *name)\ndiff --git a/dir.h b/dir.h\nindex 55e5345..6453faf 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -81,7 +81,18 @@ struct dir_struct {\n \t\tDIR_NO_GITLINKS = 1<<3,\n \t\tDIR_COLLECT_IGNORED = 1<<4,\n \t\tDIR_SHOW_IGNORED_TOO = 1<<5,\n-\t\tDIR_COLLECT_KILLED_ONLY = 1<<6\n+\t\tDIR_COLLECT_KILLED_ONLY = 1<<6,\n+\t\t/*\n+\t\t * Whether the standard excludes are the only file\n+\t\t * excludes which have been set up (if so, we can use\n+\t\t * the fs_cache to optimize is_excluded).\n+\t\t */\n+\t\tDIR_STD_EXCLUDES = 1<<7,\n+\t\t/*\n+\t\t * Excludes should only check the command-line (for\n+\t\t * use with fs_cache)\n+\t\t */\n+\t\tDIR_EXCLUDE_CMDL_ONLY = 1<<8\n \t} flags;\n \tstruct dir_entry **entries;\n \tstruct dir_entry **ignored;\n@@ -138,7 +149,7 @@ extern int match_pathspec(const struct pathspec *pathspec,\n extern int within_depth(const char *name, int namelen, int depth, int max_depth);\n \n extern int fill_directory(struct dir_struct *dir, const struct pathspec *pathspec);\n-extern int read_directory(struct dir_struct *, const char *path, int len, const struct pathspec *pathspec);\n+extern int read_directory(struct index_state *index, struct dir_struct *, const char *path, int len, const struct pathspec *pathspec);\n \n extern int is_excluded_from_list(const char *pathname, int pathlen, const char *basename,\n \t\t\t\t int *dtype, struct exclude_list *el);\n@@ -223,4 +234,7 @@ static inline int dir_path_match(const struct dir_entry *ent,\n \t\t\t      has_trailing_dir);\n }\n \n+int get_dtype(struct dirent *de, const char *path, int len);\n+\n+\n #endif\ndiff --git a/environment.c b/environment.c\nindex 5c4815d..060e473 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -73,6 +73,10 @@ char comment_line_char = '#';\n /* Parallel index stat data preload? */\n int core_preload_index = 0;\n \n+/* Use Watchman for faster status queries */\n+int core_use_watchman = 0;\n+int core_watchman_sync_timeout = 300;\n+\n /* This is set by setup_git_dir_gently() and/or git_default_config() */\n char *git_work_tree_cfg;\n static char *work_tree;\n@@ -81,7 +85,7 @@ static const char *namespace;\n static size_t namespace_len;\n \n static const char *git_dir;\n-static char *git_object_dir, *git_index_file, *git_graft_file;\n+static char *git_object_dir, *git_index_file, *git_graft_file, *git_fs_cache_file;\n \n /*\n  * Repository-local GIT_* environment variables; see cache.h for details.\n@@ -143,6 +147,11 @@ static void setup_git_env(void)\n \t\tgit_index_file = xmalloc(strlen(git_dir) + 7);\n \t\tsprintf(git_index_file, \"%s/index\", git_dir);\n \t}\n+\tgit_fs_cache_file = getenv(FS_CACHE_ENVIRONMENT);\n+\tif (!git_fs_cache_file) {\n+\t\tgit_fs_cache_file = xmalloc(strlen(git_dir) + 10);\n+\t\tsprintf(git_fs_cache_file, \"%s/fs_cache\", git_dir);\n+\t}\n \tgit_graft_file = getenv(GRAFT_ENVIRONMENT);\n \tif (!git_graft_file)\n \t\tgit_graft_file = git_pathdup(\"info/grafts\");\n@@ -266,6 +275,13 @@ char *get_graft_file(void)\n \treturn git_graft_file;\n }\n \n+char *get_fs_cache_file(void)\n+{\n+\tif (!git_fs_cache_file)\n+\t\tsetup_git_env();\n+\treturn git_fs_cache_file;\n+}\n+\n int set_git_dir(const char *path)\n {\n \tif (setenv(GIT_DIR_ENVIRONMENT, path, 1))\ndiff --git a/fs_cache.c b/fs_cache.c\nnew file mode 100644\nindex 0000000..3702186\n--- /dev/null\n+++ b/fs_cache.c\n@@ -0,0 +1,645 @@\n+#include \"cache.h\"\n+#include \"fs_cache.h\"\n+#include \"strbuf.h\"\n+#include \"hashmap.h\"\n+#include \"hash-io.h\"\n+\n+#define FS_CACHE_SIGNATURE 0x4D4F4443\t/* \"MODC\" */\n+\n+static int fe_entry_cmp(const struct fsc_entry *f1,\n+\t\t\tconst struct fsc_entry *f2,\n+\t\t\tconst char *name)\n+{\n+\tif (f1->pathlen != f2->pathlen)\n+\t\treturn 1;\n+\tname = name ? name : f2->path;\n+\treturn ignore_case ? strncasecmp(f1->path, name, f1->pathlen) :\n+\t\tstrncmp(f1->path, name, f1->pathlen);\n+\n+}\n+\n+unsigned char fe_dtype(struct fsc_entry *file)\n+{\n+\tif (fe_is_reg(file)) {\n+\t\treturn DT_REG;\n+\t}\n+\tif (fe_is_dir(file)) {\n+\t\treturn DT_DIR;\n+\t}\n+\tif (fe_is_lnk(file)) {\n+\t\treturn DT_LNK;\n+\t}\n+\treturn DT_UNKNOWN;\n+}\n+\n+#define FS_CACHE_FORMAT_LB 1\n+#define FS_CACHE_FORMAT_UB 2\n+\n+static int verify_hdr(struct fs_cache_header *hdr, unsigned long size)\n+{\n+\tvmac_ctx_t c;\n+\tunsigned char sha1[20];\n+\tint hdr_version;\n+\n+\tif (hdr->hdr_signature != htonl(FS_CACHE_SIGNATURE)) {\n+\t\twarning(\"bad fs_cache signature\");\n+\t\treturn -1;\n+\t}\n+\thdr_version = ntohl(hdr->hdr_version);\n+\tif (hdr_version < FS_CACHE_FORMAT_LB || FS_CACHE_FORMAT_UB < hdr_version) {\n+\t\twarning(\"bad fs_cache version %d\", hdr_version);\n+\t\treturn -1;\n+\t}\n+\n+\tunsigned char *key = (unsigned char *)\"abcdefghijklmnop\";\n+\tvmac_set_key(key, &c);\n+\tvmac_update_unaligned(hdr, size - 20, &c);\n+\tvmac_final(sha1, &c);\n+\tif (hashcmp(sha1, (unsigned char *)hdr + size - 20)) {\n+\t\twarning(\"bad fs_cache file vmac signature\");\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static struct fsc_entry *create_from_disk(struct fs_cache *fs_cache, struct ondisk_fsc_entry *disk_fe, unsigned long *consumed)\n+{\n+\tstruct fsc_entry *fe;\n+\tint pathlen = strlen(disk_fe->path);\n+\n+\tfe = obstack_alloc(&fs_cache->obs, sizeof(*fe) + pathlen + 1);\n+\n+\tfe->size = ntohl(disk_fe->size);\n+\tfe->mode = ntohl(disk_fe->mode);\n+\tfe->flags = ntohl(disk_fe->flags);\n+\n+\tfe->ctime.sec = ntohl(disk_fe->ctime.sec);\n+\tfe->mtime.sec = ntohl(disk_fe->mtime.sec);\n+\tfe->ctime.nsec = ntohl(disk_fe->ctime.nsec);\n+\tfe->mtime.nsec = ntohl(disk_fe->mtime.nsec);\n+\n+\tfe->ino = ntohl(disk_fe->ino);\n+\tfe->dev = ntohl(disk_fe->dev);\n+\n+\tfe->uid = ntohl(disk_fe->uid);\n+\tfe->gid = ntohl(disk_fe->gid);\n+\n+\tfe->parent = NULL;\n+\tfe->first_child = NULL;\n+\tfe->next_sibling = NULL;\n+\tmemcpy(fe->path, disk_fe->path, pathlen + 1);\n+\tfe->pathlen = pathlen;\n+\n+\thashmap_entry_init(fe, memihash(fe->path, pathlen));\n+\t*consumed = sizeof(*disk_fe) + pathlen + 1;\n+\treturn fe;\n+}\n+\n+static void copy_fs_cache_entry_to_ondisk(\n+\tstruct ondisk_fsc_entry *ondisk,\n+\tstruct fsc_entry *fe)\n+{\n+\n+\tondisk->size = htonl(fe->size);\n+\tondisk->mode = htonl(fe->mode);\n+\tondisk->flags = htonl(fe->flags & ~FE_NEW);\n+\n+\tondisk->ctime.sec = htonl(fe->ctime.sec);\n+\tondisk->mtime.sec = htonl(fe->mtime.sec);\n+\tondisk->ctime.nsec = htonl(fe->ctime.nsec);\n+\tondisk->mtime.nsec = htonl(fe->mtime.nsec);\n+\n+\tondisk->ino = htonl(fe->ino);\n+\tondisk->dev = htonl(fe->dev);\n+\n+\tondisk->uid = htonl(fe->uid);\n+\tondisk->gid = htonl(fe->gid);\n+\n+\tmemcpy(ondisk->path, fe->path, fe->pathlen + 1);\n+\n+}\n+\n+static int fe_write_entry(struct fsc_entry *fe, int fd, struct hash_context *context)\n+{\n+\tint result;\n+\tstatic struct ondisk_fsc_entry *ondisk = NULL;\n+\tstatic size_t max_size = sizeof(*ondisk) + 1 + PATH_MAX;\n+\tsize_t size;\n+\n+\tsize = sizeof(*ondisk) + fe->pathlen + 1;\n+\tif (size > max_size) {\n+\t\tmax_size = size;\n+\t\tif (ondisk) {\n+\t\t\tondisk = xrealloc(ondisk, max_size);\n+\t\t\tmemset(ondisk, 0, max_size);\n+\t\t}\n+\t}\n+\n+\n+\tif (!ondisk)\n+\t\tondisk = xcalloc(1, max_size);\n+\n+\tcopy_fs_cache_entry_to_ondisk(ondisk, fe);\n+\n+\tresult = write_with_hash(context, fd, ondisk, size);\n+\n+\treturn result ? -1 : 0;\n+}\n+\n+static int fe_write_entry_recursive(struct fsc_entry *fe, int fd, struct hash_context *c)\n+{\n+\tif (fe_write_entry(fe, fd, c))\n+\t\treturn error(\"failed to write some file of fs_cache\");\n+\tfe = fe->first_child;\n+\twhile (fe) {\n+\t\tfe_write_entry_recursive(fe, fd, c);\n+\t\tfe = fe->next_sibling;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static char *xstrcpy(char *dest, const char *src)\n+{\n+\twhile ((*dest++ = *src++)) {\n+\t}\n+\n+\treturn dest;\n+}\n+\n+int write_fs_cache(struct fs_cache *fs_cache)\n+{\n+\tstruct hash_context c;\n+\tstruct fs_cache_header *hdr;\n+\tint hdr_size;\n+\tstruct stat st;\n+\tint fd;\n+\tconst char *path;\n+\tchar *cur;\n+\tint string_size;\n+\n+\tpath = get_fs_cache_file();\n+\n+\tfd = open(path, O_WRONLY|O_TRUNC|O_CREAT, 0666);\n+\tif (fd < 0)\n+\t\tdie_errno(\"failed to open fs_cache file %s\", path);\n+\n+\tstring_size = strlen(fs_cache->last_update) +\n+\t\tstrlen(fs_cache->repo_path) +\n+\t\tstrlen(fs_cache->excludes_file) + 3;\n+\n+\thdr_size = sizeof(*hdr) + string_size;\n+\thdr = xmalloc(hdr_size);\n+\thdr->hdr_signature = htonl(FS_CACHE_SIGNATURE);\n+\thdr->hdr_version = htonl(fs_cache->version);\n+\thdr->hdr_entries = htonl(fs_cache->nr);\n+\thdr->flags = htonl(fs_cache->flags);\n+\thashcpy(hdr->git_excludes_sha1, fs_cache->git_excludes_sha1);\n+\thashcpy(hdr->user_excludes_sha1, fs_cache->user_excludes_sha1);\n+\tcur = xstrcpy(hdr->strings, fs_cache->last_update);\n+\tcur = xstrcpy(cur, fs_cache->repo_path);\n+\tcur = xstrcpy(cur, fs_cache->excludes_file);\n+\n+\thash_context_init(&c, HASH_IO_VMAC);\n+\n+\tif (write_with_hash(&c, fd, hdr, hdr_size) < 0)\n+\t\tdie_errno(\"failed to write header of fs_cache\");\n+\n+\tfe_write_entry_recursive(fs_cache_file_exists(fs_cache, \"\", 0), fd, &c);\n+\tif (write_with_hash_flush(&c, fd) || fstat(fd, &st))\n+\t\treturn error(\"Failed to flush/fstat fs_cache file\");\n+\n+\thash_context_release(&c);\n+\tfree(hdr);\n+\treturn 0;\n+}\n+\n+void *mmap_fs_cache(size_t *mmap_size) {\n+\tstruct stat st;\n+\tvoid *mmap;\n+\tconst char *path = get_fs_cache_file();\n+\tint fd = open(path, O_RDONLY);\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT)\n+\t\t\treturn NULL;\n+\t\tdie_errno(\"fs_cache file open failed\");\n+\t}\n+\n+\tif (fstat(fd, &st))\n+\t\tdie_errno(\"cannot stat the open fs_cache\");\n+\n+\t*mmap_size = xsize_t(st.st_size);\n+\tif (*mmap_size < sizeof(struct fs_cache_header) + 20) {\n+\t\twarning(\"fs_cache file smaller than expected\");\n+\t\treturn NULL;\n+\t}\n+\n+\tmmap = xmmap(NULL, *mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n+\tif (mmap == MAP_FAILED)\n+\t\tdie_errno(\"unable to map fs_cache file\");\n+\tclose(fd);\n+\treturn mmap;\n+}\n+\n+/* Loading the fs_cache can take some time, and we might want to thread\n+ * it with other loads; we need the last-update time and the repo path\n+ * to check whether this is a good idea, so this function will preload\n+ * it.  Note that the caller must free the returned strings.\n+ */\n+void fs_cache_preload_metadata(char **last_update, char **repo_path) {\n+\tsize_t mmap_size;\n+\tvoid *mmap;\n+\tstruct fs_cache_header *hdr;\n+\tint version;\n+\n+\tmmap = mmap_fs_cache(&mmap_size);\n+\tif (!mmap) {\n+\t\t*last_update = *repo_path = NULL;\n+\t\treturn;\n+\t}\n+\thdr = mmap;\n+\tversion = ntohl(hdr->hdr_version);\n+\tif (version < FS_CACHE_FORMAT_LB || FS_CACHE_FORMAT_UB < version) {\n+\t\twarning(\"bad fs_cache version %d\", version);\n+\t\tgoto unmap;\n+\t}\n+\n+\t*last_update = xstrdup(hdr->strings);\n+\t*repo_path = xstrdup(hdr->strings + strlen(*last_update) + 1);\n+\n+unmap:\n+\tmunmap(mmap, mmap_size);\n+}\n+\n+static void write_fs_cache_if_necessary(void)\n+{\n+\tstruct fs_cache *fs_cache = the_index.fs_cache;\n+\tif (fs_cache && fs_cache->needs_write && fs_cache->fully_loaded) {\n+\t\twrite_fs_cache(fs_cache);\n+\t\tthe_index.fs_cache = 0;\n+\t}\n+}\n+\n+static void fe_set_parent(struct fsc_entry *fe, struct fsc_entry *parent)\n+{\n+\tfe->parent = parent;\n+\tfe->next_sibling = fe->parent->first_child;\n+\tfe->parent->first_child = fe;\n+}\n+\n+void set_up_parent(struct fs_cache *fs_cache, struct fsc_entry *fe)\n+{\n+\tchar *last_slash;\n+\tint parent_len;\n+\tstruct fsc_entry *parent;\n+\tif (fe->pathlen == 0)\n+\t\treturn;\n+\n+\tlast_slash = strrchr(fe->path, '/');\n+\n+\tif (last_slash) {\n+\t\tparent_len = last_slash - fe->path;\n+\t} else {\n+\t\tparent_len = 0;\n+\t}\n+\n+\tparent = fs_cache_file_exists(fs_cache, fe->path, parent_len);\n+\tif (!parent) {\n+\t\tdie(\"Missing parent directory for %s\", fe->path);\n+\t}\n+\tfe_set_parent(fe, parent);\n+}\n+\n+static char *read_string(char **out, char *in) {\n+\tint len = strlen(in);\n+\t*out = xstrdup(in);\n+\treturn in + len + 1;\n+}\n+\n+/* Load the modified file cache from disk.  If the cache is corrupt,\n+ * prints a warning and returns NULL; we can safely recreate it.  If\n+ * the cache is missing, also returns NULL.  If there is some other\n+ * problem reading the cache (say it's read-only, or we get an io\n+ * error), die with an error message. */\n+struct fs_cache *read_fs_cache(void)\n+{\n+\tstruct fs_cache *fs_cache;\n+\tstruct fs_cache_header *hdr;\n+\tint i;\n+\tunsigned int nr;\n+\tvoid *mmap;\n+\tvoid *mmap_cur;\n+\tsize_t mmap_size;\n+\n+\tmmap = mmap_fs_cache(&mmap_size);\n+\tif (!mmap) {\n+\t\treturn NULL;\n+\t}\n+\n+\thdr = mmap;\n+\tif (verify_hdr(hdr, mmap_size) < 0)\n+\t\tgoto unmap;\n+\n+\tfs_cache = xcalloc(1, sizeof(*fs_cache));\n+\tobstack_init(&fs_cache->obs);\n+\tnr = ntohl(hdr->hdr_entries);\n+\tfs_cache->flags = ntohl(hdr->flags);\n+\tfs_cache->version = ntohl(hdr->hdr_version);\n+\thashmap_init(&fs_cache->paths, (hashmap_cmp_fn) fe_entry_cmp, nr);\n+\tfs_cache->nr = 0;\n+\thashcpy(fs_cache->git_excludes_sha1, hdr->git_excludes_sha1);\n+\thashcpy(fs_cache->user_excludes_sha1, hdr->user_excludes_sha1);\n+\n+\tmmap_cur = hdr->strings;\n+\tmmap_cur = read_string(&fs_cache->last_update, mmap_cur);\n+\tmmap_cur = read_string(&fs_cache->repo_path, mmap_cur);\n+\tmmap_cur = read_string(&fs_cache->excludes_file, mmap_cur);\n+\n+\tstruct fsc_entry *parent_stack[PATH_MAX];\n+\tint parent_top = -1;\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct ondisk_fsc_entry *disk_fe;\n+\t\tstruct fsc_entry *fe;\n+\t\tunsigned long consumed;\n+\n+\t\tdisk_fe = (struct ondisk_fsc_entry *) mmap_cur;\n+\t\tfe = create_from_disk(fs_cache, disk_fe, &consumed);\n+\t\t/*\n+\t\t * We eliminate deleted cache entries on read because\n+\t\t * otherwise we have to count them in advance to fill\n+\t\t * in nr, and that would be expensive.\n+\t\t */\n+\t\tif (!fe_deleted(fe)) {\n+\t\t\tfs_cache_insert(fs_cache, fe);\n+\t\t\tif (parent_top == -1) {\n+\t\t\t\tparent_top = 0;\n+\t\t\t\tparent_stack[0] = fe;\n+\t\t\t} else {\n+\t\t\t\tchar *p = parent_stack[parent_top]->path;\n+\t\t\t\tchar *c = fe->path;\n+\t\t\t\tparent_top = 1;\n+\t\t\t\tfor (; *p && *c; ++p, ++c) {\n+\t\t\t\t\tif (*p != *c)\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t\tif (*p == '/')\n+\t\t\t\t\t\tparent_top ++;\n+\t\t\t\t}\n+\t\t\t\tif (*p == 0 && *c == '/')\n+\t\t\t\t\tparent_top ++;\n+\t\t\t\tparent_stack[parent_top] = fe;\n+\t\t\t\tfe_set_parent(fe, parent_stack[parent_top - 1]);\n+\t\t\t}\n+\t\t}\n+\t\tmmap_cur += consumed;\n+\t}\n+\n+\tfs_cache->fully_loaded = 1;\n+\n+\tmunmap(mmap, mmap_size);\n+\n+\tatexit(write_fs_cache_if_necessary);\n+\treturn fs_cache;\n+\n+unmap:\n+\tmunmap(mmap, mmap_size);\n+\treturn NULL;\n+}\n+\n+struct fs_cache *empty_fs_cache(void)\n+{\n+\tstruct fs_cache *fs_cache = xcalloc(1, sizeof(*fs_cache));\n+\tfs_cache->version = 1;\n+\tfs_cache->needs_write = 1;\n+\tfs_cache->fully_loaded = 1;\n+\thashmap_init(&fs_cache->paths, (hashmap_cmp_fn) fe_entry_cmp, 1);\n+\tatexit(write_fs_cache_if_necessary);\n+\treturn fs_cache;\n+}\n+\n+struct fsc_entry *fs_cache_file_exists(const struct fs_cache *fs_cache,\n+\t\t\t\t       const char *name, int namelen)\n+{\n+\treturn fs_cache_file_exists_prehash(fs_cache, name, namelen,\n+\t\t\t\t\t  memihash(name, namelen));\n+}\n+\n+struct fsc_entry *fs_cache_file_exists_prehash(const struct fs_cache *fs_cache, const char *path, int pathlen, unsigned int hash)\n+{\n+\tstruct fsc_entry key;\n+\n+\thashmap_entry_init(&key, hash);\n+\tkey.pathlen = pathlen;\n+\treturn hashmap_get(&fs_cache->paths, &key, path);\n+}\n+\n+struct fsc_entry *make_fs_cache_entry(const char *path)\n+{\n+\treturn make_fs_cache_entry_len(path, strlen(path));\n+}\n+\n+struct fsc_entry *make_fs_cache_entry_len(const char *path, int len)\n+{\n+\tstruct fsc_entry *fe = xcalloc(1, sizeof(*fe) + len + 1);\n+\tfe_set_new(fe);\n+\tmemcpy(fe->path, path, len);\n+\tfe->pathlen = len;\n+\thashmap_entry_init(fe, memihash(fe->path, fe->pathlen));\n+\treturn fe;\n+}\n+\n+void fs_cache_insert(struct fs_cache *fs_cache, struct fsc_entry *fe)\n+{\n+\thashmap_add(&fs_cache->paths, fe);\n+\tfs_cache->nr ++;\n+}\n+\n+static void fs_cache_remove_recursive(struct fs_cache *fs_cache,\n+\t\t\t\t    struct fsc_entry *fe)\n+{\n+\tstruct fsc_entry *cur, *next;\n+\tfor (cur = fe->first_child; cur; cur = next) {\n+\t\tfs_cache_remove_recursive(fs_cache, cur);\n+\t\tnext = cur->next_sibling;\n+\t\tcur->next_sibling = NULL;\n+\t\tcur->parent = NULL;\n+\t\tcur->first_child = NULL;\n+\t}\n+\n+\thashmap_remove(&fs_cache->paths, fe, fe);\n+\tfs_cache->nr --;\n+}\n+\n+static void fe_remove_from_parent(struct fsc_entry *fe)\n+{\n+\tstruct fsc_entry *prev, *cur;\n+\tif (fe->parent) {\n+\t\tprev = NULL;\n+\t\tfor (cur = fe->parent->first_child; cur; cur = cur->next_sibling) {\n+\t\t\tif (cur == fe) {\n+\t\t\t\tif (prev)\n+\t\t\t\t\tprev->next_sibling = fe->next_sibling;\n+\t\t\t\telse\n+\t\t\t\t\tfe->parent->first_child = fe->next_sibling;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tprev = cur;\n+\t\t}\n+\t}\n+}\n+\n+void fe_delete_children(struct fsc_entry *fe)\n+{\n+\tfor (fe = fe->first_child; fe; fe = fe->next_sibling) {\n+\t\tfe_set_deleted(fe);\n+\t}\n+}\n+\n+void fe_clear_children(struct fs_cache *fs_cache, struct fsc_entry *fe) {\n+\tfor (fe = fe->first_child; fe; fe = fe->next_sibling) {\n+\t\tfs_cache_remove(fs_cache, fe);\n+\t}\n+\n+}\n+\n+void fe_set_deleted(struct fsc_entry *fe)\n+{\n+\tfe->flags |= FE_DELETED;\n+\tfe_delete_children(fe);\n+}\n+\n+void fs_cache_remove_nonrecursive(struct fs_cache *fs_cache,\n+\t\t\t\t  struct fsc_entry *fe)\n+{\n+\n+\thashmap_remove(&fs_cache->paths, fe, fe);\n+\tfs_cache->nr --;\n+\n+\tfe_remove_from_parent(fe);\n+}\n+\n+void fs_cache_remove(struct fs_cache *fs_cache,\n+\t\t   struct fsc_entry *fe)\n+{\n+\n+\tfs_cache_remove_recursive(fs_cache, fe);\n+\n+\tfe_remove_from_parent(fe);\n+}\n+\n+void free_fs_cache(struct fs_cache *fs_cache)\n+{\n+\tobstack_free(&fs_cache->obs, NULL);\n+\tfree(fs_cache->last_update);\n+\tfree(fs_cache->repo_path);\n+\tfree(fs_cache->excludes_file);\n+}\n+\n+void fe_to_stat(struct fsc_entry *fe, struct stat *st)\n+{\n+\tst->st_mtime = fe->mtime.sec;\n+\tst->st_ctime = fe->ctime.sec;\n+#ifndef NO_NSEC\n+#ifdef USE_ST_TIMESPEC\n+\tst->st_mtimespec.tv_nsec = fe->mtime.nsec;\n+\tst->st_ctimespec.tv_nsec = fe->ctime.nsec;\n+#else\n+\tst->st_mtim.tv_nsec = fe->mtime.nsec;\n+\tst->st_ctim.tv_nsec = fe->ctime.nsec;\n+#endif\n+#endif\n+\tst->st_mode  = fe->mode;\n+\tst->st_ino  = fe->ino;\n+\tst->st_dev  = fe->dev;\n+\tst->st_uid  = fe->uid;\n+\tst->st_gid  = fe->gid;\n+\tst->st_size  = fe->size;\n+}\n+\n+int is_in_dot_git(const char *name)\n+{\n+\tchar *evil = \".git\";\n+\tchar *cur = evil;\n+\twhile (*name) {\n+\t\tif (*name == *cur++) {\n+\t\t\tname++;\n+\t\t\tif (*cur == 0) {\n+\t\t\t\tif (*name == 0 || *name == '/') {\n+\t\t\t\t\treturn 1;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t} else {\n+\t\t\tif (*name == '/') {\n+\t\t\t\tcur = evil;\n+\t\t\t} else {\n+\t\t\t\tcur = \"\";\n+\t\t\t}\n+\t\t\tname++;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int is_path_prefix(const char *putative_parent, const char *fname) {\n+\tconst char* c;\n+\tfor (c = putative_parent; *c && *fname; ++c, ++fname) {\n+\t\tif (*c != *fname) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\treturn *c == 0 && (*fname == 0 || *fname == '/');\n+}\n+\n+int fs_cache_open(struct fs_cache *fs_cache, const char *fname, int flags) {\n+\tif (fs_cache && fname[0] != '/' && !is_path_prefix(get_git_dir(), fname)) {\n+\t\tstruct fsc_entry *fe = fs_cache_file_exists(fs_cache, fname, strlen(fname));\n+\t\tif (!fe || fe_deleted(fe)) {\n+\t\t\terrno = ENOENT;\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\treturn open(fname, flags);\n+}\n+\n+static const int topological_rank[256] = {\n+\t0, /* slash moved here */ 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,\n+\t15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,\n+\t32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,\n+\t1 /* slash is special */,\n+\t48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62,\n+\t63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78,\n+\t79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94,\n+\t95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106,\n+\t107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120,\n+\t121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134,\n+\t135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148,\n+\t149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162,\n+\t163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176,\n+\t177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190,\n+\t191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204,\n+\t205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218,\n+\t219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232,\n+\t233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246,\n+\t247, 248, 249, 250, 251, 252, 253, 254, 255\n+};\n+\n+/*\n+ * Compare fsc_entry structs topologically -- that is, so that parent\n+ * directories come before their children.\n+ */\n+int cmp_fsc_entry(const void *a, const void *b)\n+{\n+\tstruct fsc_entry* const * sa = a;\n+\tstruct fsc_entry* const * sb = b;\n+\tconst unsigned char* pa = (unsigned char *)(*sa)->path;\n+\tconst unsigned char* pb = (unsigned char *)(*sb)->path;\n+\twhile (*pa && *pb) {\n+\t\tint ca = topological_rank[*pa++];\n+\t\tint cb = topological_rank[*pb++];\n+\t\tint diff = ca - cb;\n+\t\tif (diff)\n+\t\t\treturn diff;\n+\t}\n+\treturn topological_rank[*pa] - topological_rank[*pb];\n+}\ndiff --git a/fs_cache.h b/fs_cache.h\nnew file mode 100644\nindex 0000000..fb558e1\n--- /dev/null\n+++ b/fs_cache.h\n@@ -0,0 +1,138 @@\n+#ifndef FS_CACHE_H\n+#define FS_CACHE_H\n+\n+#include <sys/types.h>\n+#include <sys/stat.h>\n+#include <unistd.h>\n+#include \"compat/obstack.h\"\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+#include \"hashmap.h\"\n+\n+#define obstack_chunk_alloc xmalloc\n+#define obstack_chunk_free free\n+\n+/* The filesystem cache (fs_cache) stores the state of every file\n+ * inside the root directory (excluding those in .git).  The state\n+ * includes whether or not the file exists, as well as most of what\n+ * lstat returns.\n+ */\n+\n+#define fe_is_reg(fe) (S_ISREG((fe)->mode))\n+#define fe_is_dir(fe) (S_ISDIR((fe)->mode))\n+#define fe_is_lnk(fe) (S_ISLNK((fe)->mode))\n+\n+/* Directories get very different treatment generally; the normal bits\n+ * don't apply to them, since they have no independent existence in\n+ * git.  Also, they are subject to spooky action at a distance -- if a\n+ * file called x/a/b/c is created (and added to the index), then x\n+ * suddenly must get added to the index.\n+ */\n+\n+struct fsc_entry {\n+\tstruct hashmap_entry ent;\n+\tunsigned int mode;\n+\toff_t size;\n+\tunsigned int flags;\n+\tstruct cache_time ctime;\n+\tstruct cache_time mtime;\n+\tino_t ino;\n+\tdev_t dev;\n+\tuid_t uid;\n+\tgid_t gid;\n+\tstruct fsc_entry *parent;\n+\tstruct fsc_entry *first_child;\n+\tstruct fsc_entry *next_sibling;\n+\tint pathlen;\n+\tchar path[FLEX_ARRAY];\n+};\n+\n+#define FE_DELETED           (1 << 0)\n+\n+/* Excluded by the standard set of gitexcludes */\n+#define FE_EXCLUDED          (1 << 8)\n+\n+/* Not yet saved to disk */\n+#define FE_NEW               (1 << 10)\n+\n+void fe_set_deleted(struct fsc_entry *fe);\n+#define fe_clear_deleted(fe) ((fe)->flags &= ~FE_DELETED)\n+#define fe_deleted(fe) ((fe)->flags & FE_DELETED)\n+\n+#define fe_excluded(fe) ((fe)->flags & FE_EXCLUDED)\n+#define fe_set_excluded(fe) ((fe)->flags |= FE_EXCLUDED)\n+#define fe_clear_excluded(fe) ((fe)->flags &= ~FE_EXCLUDED)\n+\n+#define fe_new(fe) ((fe)->flags & FE_NEW)\n+#define fe_set_new(fe) ((fe)->flags |= FE_NEW)\n+#define fe_clear_new(fe) ((fe)->flags &= ~FE_NEW)\n+\n+struct fs_cache {\n+\tchar *last_update;\n+\tchar *repo_path;\n+\tchar *excludes_file;\n+\tunsigned char git_excludes_sha1[20]; /* for .git/info/exclude */\n+\tunsigned char user_excludes_sha1[20]; /* for core.excludesfile */\n+\tunsigned int version;\n+\tstruct hashmap paths;\n+\tint nr;\n+\tunsigned invalid : 1; /* A commit hook might have made fs\n+\t\t\t       * changes, necessitating a reload. */\n+\tunsigned needs_write : 1;\n+\tunsigned fully_loaded : 1;\n+\tuint32_t flags;\n+\tstruct obstack obs;\n+};\n+\n+struct fs_cache_header {\n+\tuint32_t hdr_signature;\n+\tuint32_t hdr_version;\n+\tuint32_t hdr_entries;\n+\tuint32_t flags;\n+\tunsigned char git_excludes_sha1[20];\n+\tunsigned char user_excludes_sha1[20];\n+\tchar strings[FLEX_ARRAY];\n+};\n+\n+struct ondisk_fsc_entry {\n+\tuint64_t ino;\n+\tuint64_t dev;\n+\tstruct cache_time ctime;\n+\tstruct cache_time mtime;\n+\tuint32_t mode;\n+\tuint32_t size;\n+\tuint32_t flags;\n+\tuint32_t uid;\n+\tuint32_t gid;\n+\tchar path[FLEX_ARRAY];\n+};\n+\n+extern char *get_fs_cache_file(void);\n+\n+unsigned char fe_dtype(struct fsc_entry *file);\n+void fe_to_stat(struct fsc_entry *fe, struct stat *st);\n+void fe_delete_children(struct fsc_entry *fe);\n+void fe_clear_children(struct fs_cache *fs_cache, struct fsc_entry *fe);\n+\n+struct fs_cache *read_fs_cache(void);\n+int write_fs_cache(struct fs_cache *fs_cache);\n+struct fs_cache *empty_fs_cache(void);\n+struct fsc_entry *fs_cache_file_exists(const struct fs_cache *fs_cache, const char *name, int namelen);\n+struct fsc_entry *fs_cache_file_exists_prehash(const struct fs_cache *fs_cache, const char *name, int namelen, unsigned int hash);\n+struct fsc_entry *make_fs_cache_entry(const char *path);\n+struct fsc_entry *make_fs_cache_entry_len(const char *path, int len);\n+void fs_cache_preload_metadata(char **last_update, char **repo_path);\n+\n+void fs_cache_remove(struct fs_cache *fs_cache, struct fsc_entry *fe);\n+void fs_cache_insert(struct fs_cache *fs_cache, struct fsc_entry *fe);\n+void free_fs_cache(struct fs_cache *fs_cache);\n+void set_up_parent(struct fs_cache *fs_cache, struct fsc_entry *fe);\n+\n+int fs_cache_open(struct fs_cache *fs_cache, const char *fname, int flags);\n+\n+int is_in_dot_git(const char *name);\n+\n+int cmp_fsc_entry(const void *a, const void *b);\n+\n+#endif /* FS_CACHE_H */\ndiff --git a/hash-io.c b/hash-io.c\nindex ab4c7e4..907a360 100644\n--- a/hash-io.c\n+++ b/hash-io.c\n@@ -154,11 +154,11 @@ void hash_context_init(struct hash_context *ctx, enum hash_io_type type)\n \tctx->write_buffer_len = 0;\n \tswitch (type) {\n \tcase HASH_IO_VMAC:\n-\t\tctx->c.vc = malloc(sizeof *ctx->c.vc);\n+\t\tctx->c.vc = xmalloc(sizeof *ctx->c.vc);\n \t\tvmac_set_key(VMAC_KEY, ctx->c.vc);\n \t\tbreak;\n \tcase HASH_IO_SHA1:\n-\t\tctx->c.sc = malloc(sizeof *ctx->c.sc);\n+\t\tctx->c.sc = xmalloc(sizeof *ctx->c.sc);\n \t\tgit_SHA1_Init(ctx->c.sc);\n \t\tbreak;\n \tdefault:\ndiff --git a/read-cache.c b/read-cache.c\nindex a124e59..5b1ca78 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -5,6 +5,7 @@\n  */\n #define NO_THE_INDEX_COMPATIBILITY_MACROS\n #include \"cache.h\"\n+#include \"fs_cache.h\"\n #include \"cache-tree.h\"\n #include \"refs.h\"\n #include \"dir.h\"\n@@ -16,6 +17,10 @@\n #include \"varint.h\"\n #include \"hash-io.h\"\n \n+#ifdef USE_WATCHMAN\n+#include \"watchman-support.h\"\n+#endif\n+\n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t       unsigned int options);\n \n@@ -1004,6 +1009,30 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n \treturn 0;\n }\n \n+static int fs_cache_lstat(struct fs_cache *fs_cache,\n+\t\t\tconst char *name, int len, struct stat *st)\n+{\n+\n+\tstruct fsc_entry *fe;\n+\tif (!fs_cache)\n+\t\treturn lstat(name, st);\n+\n+\tfe = fs_cache_file_exists(fs_cache, name, len);\n+\tif (!fe) {\n+\t\t/* This is necessary because children of symlinks are not\n+\t\t * included in the fs_cache. */\n+\t\treturn lstat(name, st);\n+\t}\n+\n+\tif (fe_deleted(fe)) {\n+\t\terrno = ENOENT;\n+\t\treturn -1;\n+\t} else {\n+\t\tfe_to_stat(fe, st);\n+\t}\n+\treturn 0;\n+}\n+\n /*\n  * \"refresh\" does not calculate a new sha1 file or bring the\n  * cache up-to-date for mode/content changes. But what it\n@@ -1045,7 +1074,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\treturn ce;\n \t}\n \n-\tif (lstat(ce->name, &st) < 0) {\n+\tif (fs_cache_lstat(the_index.fs_cache, ce->name, ce_namelen(ce), &st) < 0) {\n \t\tif (ignore_missing && errno == ENOENT)\n \t\t\treturn ce;\n \t\tif (err)\n@@ -1446,6 +1475,25 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static void do_load_fs_cache(struct index_state *istate, int force)\n+{\n+#ifdef USE_WATCHMAN\n+\tif (core_use_watchman && (istate->initialized || force)) {\n+\t\tif (istate->fs_cache) {\n+\t\t\tif (istate->fs_cache->invalid)\n+\t\t\t\twatchman_reload_fs_cache(istate);\n+\t\t} else {\n+\t\t\tif (watchman_load_fs_cache(istate)) {\n+\t\t\t\tif (istate->fs_cache) {\n+\t\t\t\t\tistate->fs_cache->needs_write = 0;\n+\t\t\t\t\tistate->fs_cache = NULL;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t}\n+#endif\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n@@ -1457,6 +1505,8 @@ int read_index_from(struct index_state *istate, const char *path)\n \tsize_t mmap_size;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n \n+\tdo_load_fs_cache(istate, 0);\n+\n \tif (istate->initialized)\n \t\treturn istate->cache_nr;\n \n@@ -1464,8 +1514,10 @@ int read_index_from(struct index_state *istate, const char *path)\n \tistate->timestamp.nsec = 0;\n \tfd = open(path, O_RDONLY);\n \tif (fd < 0) {\n-\t\tif (errno == ENOENT)\n+\t\tif (errno == ENOENT) {\n+\t\t\tdo_load_fs_cache(istate, 1);\n \t\t\treturn 0;\n+\t\t}\n \t\tdie_errno(\"index file open failed\");\n \t}\n \n@@ -1530,6 +1582,9 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n+\n+\tdo_load_fs_cache(istate, 0);\n+\n \tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n@@ -1560,6 +1615,8 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->fs_cache)\n+\t\tistate->fs_cache->invalid = 1;\n \treturn 0;\n }\n \ndiff --git a/t/t1012-read-tree-df.sh b/t/t1012-read-tree-df.sh\nindex a6a04b6..a677053 100755\n--- a/t/t1012-read-tree-df.sh\n+++ b/t/t1012-read-tree-df.sh\n@@ -23,7 +23,7 @@ maketree () {\n }\n \n settree () {\n-\trm -f .git/index .git/index.lock &&\n+\trm -f .git/index .git/index.lock .git/fs_cache &&\n \tgit clean -d -f -f -q -x &&\n \tgit read-tree \"$1\" &&\n \tgit checkout-index -f -q -u -a &&\ndiff --git a/t/t2201-add-update-typechange.sh b/t/t2201-add-update-typechange.sh\nindex 954fc51..ad49535 100755\n--- a/t/t2201-add-update-typechange.sh\n+++ b/t/t2201-add-update-typechange.sh\n@@ -124,7 +124,9 @@ test_expect_success diff-index '\n \n test_expect_success 'add -u' '\n \trm -f \".git/saved-index\" &&\n+\trm -f \".git/saved-fs_cache\" &&\n \tcp -p \".git/index\" \".git/saved-index\" &&\n+\t(test ! -f .git/fs_cache || cp -p \".git/fs_cache\" \".git/saved-fs_cache\") &&\n \tgit add -u &&\n \tgit ls-files -s >actual &&\n \ttest_cmp expect-final actual\n@@ -134,7 +136,8 @@ test_expect_success 'commit -a' '\n \tif test -f \".git/saved-index\"\n \tthen\n \t\trm -f \".git/index\" &&\n-\t\tmv \".git/saved-index\" \".git/index\"\n+\t\tmv \".git/saved-index\" \".git/index\" &&\n+\t\t(test ! -f .git/saved-fs_cache || mv \".git/saved-fs_cache\" \".git/fs_cache\")\n \tfi &&\n \tgit commit -m \"second\" -a &&\n \tgit ls-files -s >actual &&\ndiff --git a/t/t2204-add-ignored.sh b/t/t2204-add-ignored.sh\nindex 8340ac2..6653aa8 100755\n--- a/t/t2204-add-ignored.sh\n+++ b/t/t2204-add-ignored.sh\n@@ -18,7 +18,7 @@ test_expect_success setup '\n for i in file dir/file dir 'd*'\n do\n \ttest_expect_success \"no complaints for unignored $i\" '\n-\t\trm -f .git/index &&\n+\t\trm -f .git/index .git/fs_cache &&\n \t\tgit add \"$i\" &&\n \t\tgit ls-files \"$i\" >out &&\n \t\ttest -s out\n@@ -28,7 +28,7 @@ done\n for i in ign dir/ign dir/sub dir/sub/*ign sub/file sub sub/*\n do\n \ttest_expect_success \"complaints for ignored $i\" '\n-\t\trm -f .git/index &&\n+\t\trm -f .git/index .git/fs_cache &&\n \t\ttest_must_fail git add \"$i\" 2>err &&\n \t\tgit ls-files \"$i\" >out &&\n \t\t! test -s out\n@@ -39,7 +39,7 @@ do\n \t'\n \n \ttest_expect_success \"complaints for ignored $i with unignored file\" '\n-\t\trm -f .git/index &&\n+\t\trm -f .git/index .git/fs_cache &&\n \t\ttest_must_fail git add \"$i\" file 2>err &&\n \t\tgit ls-files \"$i\" >out &&\n \t\t! test -s out\ndiff --git a/t/t6001-rev-list-graft.sh b/t/t6001-rev-list-graft.sh\nindex 8efcd13..56f8ac0 100755\n--- a/t/t6001-rev-list-graft.sh\n+++ b/t/t6001-rev-list-graft.sh\n@@ -20,7 +20,7 @@ test_expect_success setup '\n \tgit commit -a -m \"Third in one history.\" &&\n \tA2=`git rev-parse --verify HEAD` &&\n \n-\trm -f .git/refs/heads/master .git/index &&\n+\trm -f .git/refs/heads/master .git/index .git/fs_cache &&\n \n \techo >fileA fileA again &&\n \techo >subdir/fileB fileB again &&\ndiff --git a/t/t7900-watchman.sh b/t/t7900-watchman.sh\nnew file mode 100755\nindex 0000000..bea6180\n--- /dev/null\n+++ b/t/t7900-watchman.sh\n@@ -0,0 +1,249 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2014 Twitter, Inc\n+#\n+\n+test_description='Watchman'\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq WATCHMAN\n+then\n+\tskip_all='skipping watchman tests - no watchman'\n+\ttest_done\n+fi\n+\n+xpgrep () {\n+\tresult=$(ps xopid,comm | grep \" $1\\$\" | awk '{ print $1 }')\n+\techo $result\n+\ttest \"x$result\" != \"x\"\n+}\n+\n+kill_watchman() {\n+\t#stop watchman\n+\txpgrep watchman | xargs kill\n+}\n+\n+#make sure that watchman is not running, but that it is runnable\n+test_expect_success setup '\n+\tgit config core.usewatchman true &&\n+\t#watchman is maybe running\n+\txpgrep watchman > running-watchman\n+\tgrep . running-watchman > /dev/null && kill $(cat running-watchman)\n+\trm running-watchman &&\n+\tsleep 0.25 &&\n+\t#watchman is stopped\n+\t! xpgrep watchman > /dev/null &&\n+\t#watchman is startable\n+\twatchman &&\n+\tkill_watchman\n+'\n+\n+cat >expect <<\\EOF\n+?? expect\n+?? morx\n+?? output\n+EOF\n+\n+test_expect_success 'watchman sees new files' '\n+\ttouch morx &&\n+\tgit status -s > output &&\n+\ttest_cmp expect output\n+'\n+\n+cat >expect <<\\EOF\n+?? expect\n+?? output\n+EOF\n+\n+test_expect_success 'watchman sees file deletion' '\n+\trm morx &&\n+\tgit status -s > output &&\n+\ttest_cmp expect output\n+'\n+\n+cat >expect <<\\EOF\n+?? .gitignore\n+?? bramp\n+EOF\n+\n+test_expect_success 'watchman understands .gitignore' '\n+\ttouch bramp &&\n+\tcat >.gitignore <<-EOF &&\n+\texpect*\n+\toutput*\n+EOF\n+\tgit status -s > output &&\n+\ttest_cmp expect output\n+'\n+\n+cat >expect <<\\EOF\n+?? .gitignore\n+EOF\n+\n+test_expect_success 'watchman notices changes to .gitignore' '\n+\tcat >.gitignore <<-EOF &&\n+\texpect*\n+\toutput*\n+\tbramp\n+EOF\n+\tgit status -s > output &&\n+\ttest_cmp expect output\n+'\n+\n+cat >expect <<\\EOF\n+?? .gitignore\n+EOF\n+\n+test_expect_success 'watchman understands .git/info/exclude' '\n+\ttouch fleem &&\n+\tcat >.git/info/exclude <<-EOF &&\n+\tfleem\n+EOF\n+\tgit status -s > output &&\n+\ttest_cmp expect output\n+'\n+\n+cat >expect <<\\EOF\n+?? .gitignore\n+?? fleem\n+EOF\n+\n+test_expect_success 'watchman notices changes to .git/info/exclude' '\n+\ttouch crubbins &&\n+\tcat >.git/info/exclude <<-EOF &&\n+\tcrubbins\n+EOF\n+\tgit status -s > output &&\n+\ttest_cmp expect output\n+'\n+\n+cat >expect <<\\EOF\n+?? .gitignore\n+?? crubbins\n+?? fleem\n+EOF\n+\n+test_expect_success 'watchman notices removal of .git/info/exclude' '\n+\trm .git/info/exclude &&\n+\tgit status -s > output &&\n+\ttest_cmp expect output &&\n+\trm crubbins bramp fleem\n+'\n+\n+\n+cat >expect <<\\EOF\n+?? .gitignore\n+?? fleem\n+?? myignore\n+EOF\n+\n+test_expect_success 'watchman notices changes to file configured by core.excludesfile' '\n+\ttouch fleem &&\n+\ttouch crubbins &&\n+\tcat >myignore <<-EOF &&\n+\tcrubbins\n+EOF\n+\tgit config core.excludesfile myignore &&\n+\tgit status -s > output &&\n+\ttest_cmp expect output\n+'\n+\n+cat >expect <<\\EOF\n+?? .gitignore\n+?? crubbins\n+?? myignore\n+?? myignore2\n+EOF\n+\n+test_expect_success 'watchman notices changes to config variable core.excludesfile' '\n+\ttouch fleem &&\n+\ttouch crubbins &&\n+\tcat >myignore2 <<-EOF &&\n+\tfleem\n+EOF\n+\tgit config core.excludesfile myignore2 &&\n+\tgit status -s > output &&\n+\ttest_cmp expect output\n+'\n+\n+cat >expect <<\\EOF\n+?? .gitignore\n+?? crubbins\n+?? fleem\n+?? myignore\n+EOF\n+\n+test_expect_success 'watchman notices removal of file referred to by' '\n+\trm myignore2 &&\n+\tgit status -s > output &&\n+\ttest_cmp expect output &&\n+\trm crubbins fleem myignore\n+'\n+\n+\n+cat >expect.nothing <<\\EOF\n+EOF\n+\n+cat >expect.2 <<\\EOF\n+EOF\n+\n+test_expect_success 'git diff still works' '\n+\techo 1 > diffy &&\n+\tgit add diffy .gitignore &&\n+\tgit commit -m initial &&\n+\tgit status -s > output &&\n+\ttest_cmp expect.nothing output &&\n+\techo 2 >> diffy &&\n+\ttest_cmp expect.2 output\n+'\n+\n+cat >expect <<\\EOF\n+ D diffy\n+EOF\n+\n+test_expect_success 'file to directory changes still work' '\n+\trm diffy &&\n+\tmkdir diffy &&\n+\ttouch diffy/a &&\n+\tgit status -s > output &&\n+\ttest_cmp expect output &&\n+\tgit add diffy/a &&\n+\tgit commit -m two &&\n+\tgit status -s > output.nothing\n+'\n+\n+cat >expect <<\\EOF\n+ D diffy/a\n+?? diffy\n+EOF\n+\n+test_expect_success 'directory to file changes still work' '\n+\trm -r diffy &&\n+\ttouch diffy &&\n+\tgit status -s > output &&\n+\ttest_cmp expect output &&\n+\trm diffy &&\n+\tgit rm diffy/a &&\n+\tgit commit -m \"remove diffy\"\n+'\n+\n+cat >expect <<\\EOF\n+?? dead\n+EOF\n+\n+test_expect_success 'changes while watchman is not running are detected' '\n+\tkill_watchman &&\n+\tsleep 0.25 &&\n+\t! xpgrep watchman > /dev/null &&\n+\ttouch dead &&\n+\tgit status -s > output &&\n+\ttest_cmp expect output\n+'\n+\n+test_expect_success 'Restore default test environment' '\n+\tgit config --unset core.usewatchman &&\n+\tkill_watchman\n+'\n+\n+test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex c081668..19d1ec5 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -781,6 +781,7 @@ test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE\" && test_set_prereq LIBPCRE\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n+test -n \"$USE_WATCHMAN\" && test_set_prereq WATCHMAN\n \n # Can we rely on git's output in the C locale?\n if test -n \"$GETTEXT_POISON\"\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 97fc995..5161de3 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1339,7 +1339,7 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,\n \tmemset(&d, 0, sizeof(d));\n \tif (o->dir)\n \t\td.exclude_per_dir = o->dir->exclude_per_dir;\n-\ti = read_directory(&d, pathbuf, namelen+1, NULL);\n+\ti = read_directory(o->src_index, &d, pathbuf, namelen+1, NULL);\n \tif (i)\n \t\treturn o->gently ? -1 :\n \t\t\tadd_rejected_path(o, ERROR_NOT_UPTODATE_DIR, ce->name);\ndiff --git a/watchman-support.c b/watchman-support.c\nnew file mode 100644\nindex 0000000..9c5a87d\n--- /dev/null\n+++ b/watchman-support.c\n@@ -0,0 +1,640 @@\n+#include <dirent.h>\n+\n+#include \"git-compat-util.h\"\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"fs_cache.h\"\n+#include \"strbuf.h\"\n+#include \"pathspec.h\"\n+#include \"watchman-support.h\"\n+\n+#define NS_PER_SEC 1000000000L\n+\n+#define SET_TIME_FROM_NS(time, ns)\t      \\\n+\tdo {\t\t\t\t      \\\n+\t\t(time).sec = (ns) / NS_PER_SEC;      \\\n+\t\t(time).nsec = (ns) % NS_PER_SEC;     \\\n+\t} while(0)\n+\n+static inline unsigned int create_fe_mode(unsigned int mode)\n+{\n+\tif (S_ISLNK(mode))\n+\t\treturn S_IFLNK;\n+\tif (S_ISDIR(mode))\n+\t\treturn S_IFDIR;\n+\treturn S_IFREG | ce_permissions(mode);\n+}\n+\n+static void copy_wm_stat_to_fe(struct watchman_stat *wm, struct fsc_entry *fe)\n+{\n+\tif (!wm->exists) {\n+\t\tfe_set_deleted(fe);\n+\t\treturn;\n+\t} else\n+\t\tfe_clear_deleted(fe);\n+\tfe->size = wm->size;\n+\tfe->mode = create_fe_mode(wm->mode);\n+\tfe->ino = wm->ino;\n+\tfe->dev = wm->dev;\n+\tfe->uid = wm->uid;\n+\tfe->gid = wm->gid;\n+\tSET_TIME_FROM_NS(fe->mtime, wm->mtime_ns);\n+\tSET_TIME_FROM_NS(fe->ctime, wm->ctime_ns);\n+\treturn;\n+}\n+\n+static struct fsc_entry *wm_stat_to_fe(struct watchman_stat *wm)\n+{\n+\tstruct fsc_entry *fe = make_fs_cache_entry(wm->name);\n+\tfe_set_new(fe);\n+\tcopy_wm_stat_to_fe(wm, fe);\n+\treturn fe;\n+}\n+\n+static void update_exclude(struct dir_struct *dir, struct fsc_entry *fe)\n+{\n+\tint dtype = fe_dtype(fe);\n+\tif (is_excluded(dir, fe->path, &dtype)) {\n+\t\tfe_set_excluded(fe);\n+\t} else {\n+\t\tfe_clear_excluded(fe);\n+\t}\n+\tfor (fe = fe->first_child; fe; fe = fe->next_sibling) {\n+\t\tupdate_exclude(dir, fe);\n+\t}\n+}\n+\n+static struct fsc_entry *fs_cache_file_deleted(struct fs_cache *fs_cache,\n+\t\t\t\t\t       struct watchman_stat *wm)\n+{\n+\tint namelen = strlen(wm->name);\n+\tstruct fsc_entry *fe;\n+\n+\tfe = fs_cache_file_exists(fs_cache, wm->name, namelen);\n+\n+\tif (fe) {\n+\t\tfe_set_deleted(fe);\n+\t\tfe_clear_children(fs_cache, fe);\n+\t}\n+\n+\treturn fe;\n+}\n+\n+static struct fsc_entry *fs_cache_file_modified(struct fs_cache *fs_cache,\n+\t\t\t\t\t\tstruct watchman_stat *wm)\n+{\n+\tint namelen = strlen(wm->name);\n+\tstruct fsc_entry *fe;\n+\tfe = fs_cache_file_exists(fs_cache, wm->name, namelen);\n+\tif (!fe) {\n+\t\tfe = wm_stat_to_fe(wm);\n+\t\tfs_cache_insert(fs_cache, fe);\n+\t\tset_up_parent(fs_cache, fe);\n+\t} else {\n+\t\tint was_dir = fe_is_dir(fe);\n+\t\tif (fe_deleted(fe))\n+\t\t\tfe_set_new(fe);\n+\t\tcopy_wm_stat_to_fe(wm, fe);\n+\t\tif (was_dir && !fe_is_dir(fe)) {\n+\t\t\tfe_clear_children(fs_cache, fe);\n+\t\t}\n+\t}\n+\treturn fe;\n+}\n+\n+static struct watchman_expression *make_expression()\n+{\n+\tstruct watchman_expression *types[3];\n+\ttypes[0] = watchman_type_expression('f');\n+\ttypes[1] = watchman_type_expression('d');\n+\ttypes[2] = watchman_type_expression('l');\n+\tstruct watchman_expression *expr = watchman_anyof_expression(3, types);\n+\n+\treturn expr;\n+}\n+\n+struct watchman_query *make_query(const char *last_update)\n+{\n+\tstruct watchman_query *query = watchman_query();\n+\n+\twatchman_query_set_fields(query,\n+\t\t\t\t  WATCHMAN_FIELD_NAME |\n+\t\t\t\t  WATCHMAN_FIELD_MTIME_NS |\n+\t\t\t\t  WATCHMAN_FIELD_CTIME_NS |\n+\t\t\t\t  WATCHMAN_FIELD_INO |\n+\t\t\t\t  WATCHMAN_FIELD_DEV |\n+\t\t\t\t  WATCHMAN_FIELD_UID |\n+\t\t\t\t  WATCHMAN_FIELD_GID |\n+\t\t\t\t  WATCHMAN_FIELD_EXISTS |\n+\t\t\t\t  WATCHMAN_FIELD_MODE |\n+\t\t\t\t  WATCHMAN_FIELD_SIZE);\n+\twatchman_query_set_empty_on_fresh(query, 1);\n+\n+\tquery->sync_timeout = core_watchman_sync_timeout;\n+\n+\tif (last_update) {\n+\t\twatchman_query_set_since_oclock(query, last_update);\n+\t}\n+\treturn query;\n+}\n+\n+enum path_treatment {\n+\tpath_recurse,\n+\tpath_file\n+};\n+\n+void fe_from_stat(struct fsc_entry *fe, struct stat *st)\n+{\n+\tfe->mode = create_fe_mode(st->st_mode);\n+\tfe->size = st->st_size;\n+\tfe->ino = st->st_ino;\n+\tfe->dev = st->st_dev;\n+\tfe->ctime.sec = st->st_ctime;\n+\tfe->ctime.nsec = ST_CTIME_NSEC(*st);\n+\tfe->mtime.sec = st->st_mtime;\n+\tfe->mtime.nsec = ST_MTIME_NSEC(*st);\n+\tfe->uid = st->st_uid;\n+\tfe->gid = st->st_gid;\n+}\n+\n+static void update_all_excludes(struct fs_cache *fs_cache)\n+{\n+\tstruct fsc_entry *root = fs_cache_file_exists(fs_cache, \"\", 0);\n+\tstruct dir_struct dir;\n+\tchar original_path[PATH_MAX + 1];\n+\tconst char *fs_path = get_git_work_tree();\n+\n+\tif (!getcwd(original_path, PATH_MAX + 1))\n+\t\t\tdie_errno(\"failed to get working directory\\n\");\n+\tif (chdir(fs_path))\n+\t\t\tdie_errno(\"failed to chdir to git work tree\\n\");\n+\n+\tassert (root);\n+\n+\tmemset(&dir, 0, sizeof(dir));\n+\tsetup_standard_excludes(&dir);\n+\tupdate_exclude(&dir, root);\n+\tclear_directory(&dir);\n+\n+\tif (chdir(original_path))\n+\t\t\tdie_errno(\"failed to chdir back to original path\\n\");\n+}\n+\n+static enum path_treatment watchman_handle(struct index_state *istate, struct strbuf *path, struct dirent *de, int rootlen, struct fsc_entry **out)\n+{\n+\tstruct fs_cache *fs_cache = istate->fs_cache;\n+\tstruct fsc_entry *fe;\n+\tstruct stat st;\n+\tint dtype;\n+\n+\tfe = make_fs_cache_entry(path->buf + rootlen);\n+\t*out = fe;\n+\tfs_cache_insert(fs_cache, fe);\n+\tset_up_parent(fs_cache, fe);\n+\tlstat(path->buf, &st);\n+\tfe_from_stat(fe, &st);\n+\n+\tdtype = DTYPE(de);\n+\tif (dtype == DT_UNKNOWN) {\n+\t\t/* this involves an extra stat call, but only on\n+\t\t * Cygwin, which watchman doesn't support anyway. */\n+\t\tdtype = get_dtype(de, path->buf, path->len);\n+\t}\n+\tif (dtype == DT_DIR) {\n+\t\treturn path_recurse;\n+\t}\n+\n+\treturn path_file;\n+}\n+\n+static void path_set_last_component(struct strbuf *path, int baselen, const char *add)\n+{\n+\tstrbuf_setlen(path, baselen);\n+\tif (baselen) {\n+\t\tstrbuf_addch(path, '/');\n+\t}\n+\tstrbuf_addstr(path, add);\n+}\n+\n+static int preload_wt_recursive(struct index_state *istate, struct strbuf *path, int rootlen)\n+{\n+\tDIR *fdir;\n+\tstruct dirent *de;\n+\tint baselen = path->len;\n+\n+\tfdir = opendir(path->buf);\n+\tif (!fdir) {\n+\t\treturn error(\"Failed to open %s\", path->buf);\n+\t}\n+\n+\twhile ((de = readdir(fdir)) != NULL) {\n+\t\tstruct fsc_entry *fe;\n+\t\tif (is_dot_or_dotdot(de->d_name) || is_in_dot_git(de->d_name))\n+\t\t\tcontinue;\n+\n+\t\tpath_set_last_component(path, baselen, de->d_name);\n+\n+\t\t/* recurse into subdir if necessary */\n+\t\tif (watchman_handle(istate, path, de, rootlen, &fe) == path_recurse) {\n+\t\t\tint result = preload_wt_recursive(istate, path, rootlen);\n+\t\t\tif (result) {\n+\t\t\t\tclosedir(fdir);\n+\t\t\t\treturn result;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tclosedir(fdir);\n+\treturn 0;\n+}\n+\n+static void init_excludes_config()\n+{\n+\tchar *xdg_path;\n+\tif (!excludes_file) {\n+\t\thome_config_paths(NULL, &xdg_path, \"ignore\");\n+\t\texcludes_file = xdg_path;\n+\t}\n+}\n+\n+static void compute_sha1(const char *path, unsigned char *sha1)\n+{\n+\tstruct stat st;\n+\tif (stat(path, &st)) {\n+\t\tmemset(sha1, 0, 20);\n+\t} else {\n+\t\tif (index_path(sha1, path, &st, 0)) {\n+\t\t\tmemset(sha1, 0, 20);\n+\t\t}\n+\t}\n+}\n+\n+static void init_excludes_files(struct fs_cache *fs_cache)\n+{\n+\tinit_excludes_config();\n+\tif (fs_cache->excludes_file) {\n+\t\tfree(fs_cache->excludes_file);\n+\t}\n+\tif (excludes_file) {\n+\t\tfs_cache->excludes_file = xstrdup(excludes_file);\n+\t\tcompute_sha1(excludes_file, fs_cache->user_excludes_sha1);\n+\t} else {\n+\t\tfs_cache->excludes_file = xstrdup(\"\");\n+\t\tmemset(fs_cache->user_excludes_sha1, 0, 20);\n+\t}\n+\tcompute_sha1(git_path(\"info/excludes\"), fs_cache->git_excludes_sha1);\n+}\n+\n+static int git_excludes_file_changed(struct fs_cache *fs_cache)\n+{\n+\tunsigned char sha1[20];\n+\n+\tcompute_sha1(git_path(\"info/exclude\"), sha1);\n+\tif (!hashcmp(fs_cache->git_excludes_sha1, sha1))\n+\t\treturn 0;\n+\thashcpy(fs_cache->git_excludes_sha1, sha1);\n+\treturn 1;\n+}\n+\n+static int user_excludes_file_changed(struct fs_cache *fs_cache)\n+{\n+\tunsigned char sha1[20] = {0};\n+\tstruct stat st;\n+\n+\tinit_excludes_config();\n+\n+\tif (!excludes_file) {\n+\t\tif (strlen(fs_cache->excludes_file) == 0) {\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tfs_cache->excludes_file[0] = 0;\n+\t\tif (is_null_sha1(fs_cache->user_excludes_sha1))\n+\t\t\treturn 0;\n+\n+\t\tmemset(fs_cache->user_excludes_sha1, 0, 20);\n+\t\treturn 1;\n+\t}\n+\n+\t/* A change in exclude filename forces an exclude reload */\n+\tif (strcmp(fs_cache->excludes_file, excludes_file)) {\n+\t\tinit_excludes_files(fs_cache);\n+\t\treturn 1;\n+\t}\n+\n+\tif (!strlen(fs_cache->excludes_file)) {\n+\t\treturn 0;\n+\t}\n+\n+\tif (stat(excludes_file, &st)) {\n+\t\t/* There is a problem reading the excludes file; this\n+\t\t * could be a persistent condition, so we need to\n+\t\t * check if the file is presently marked as invalid */\n+\t\tif (is_null_sha1(fs_cache->user_excludes_sha1))\n+\t\t\treturn 0;\n+\t\telse {\n+\t\t\tmemset(fs_cache->user_excludes_sha1, 0, 20);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\tif (index_path(sha1, excludes_file, &st, 0)) {\n+\t\tif (is_null_sha1(fs_cache->user_excludes_sha1)) {\n+\t\t\treturn 0;\n+\t\t} else {\n+\t\t\tmemset(fs_cache->user_excludes_sha1, 0, 20);\n+\t\t\treturn 1;\n+\t\t}\n+\t} else {\n+\t\tif (!hashcmp(fs_cache->user_excludes_sha1, sha1))\n+\t\t\treturn 0;\n+\t\thashcpy(fs_cache->user_excludes_sha1, sha1);\n+\t\treturn 1;\n+\t}\n+}\n+\n+static void create_fs_cache(struct index_state *istate)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *fs_path = get_git_work_tree();\n+\tstruct fsc_entry *root;\n+\n+\tstrbuf_addstr(&buf, fs_path);\n+\tistate->fs_cache = empty_fs_cache();\n+\troot = make_fs_cache_entry(\"\");\n+\troot->mode = 040644;\n+\tfs_cache_insert(istate->fs_cache, root);\n+\tpreload_wt_recursive(istate, &buf, buf.len + 1);\n+\tstrbuf_release(&buf);\n+\n+\tinit_excludes_files(istate->fs_cache);\n+\tupdate_all_excludes(istate->fs_cache);\n+}\n+\n+static void load_fs_cache(struct index_state *istate)\n+{\n+\tif (istate->fs_cache)\n+\t\treturn;\n+\tistate->fs_cache = read_fs_cache();\n+\tif (!istate->fs_cache) {\n+\t\tcreate_fs_cache(istate);\n+\t}\n+}\n+\n+static struct watchman_query_result *watchman_fs_cache_query(struct watchman_connection *connection, const char *fs_path, const char *last_update)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_expression *expr;\n+\tstruct watchman_query *query;\n+\tstruct watchman_query_result *result = NULL;\n+\tstruct stat st;\n+\tint fs_path_len = strlen(fs_path);\n+\tchar *git_path;\n+\n+\texpr = make_expression();\n+\tquery = make_query(last_update);\n+\tif (lstat(fs_path, &st)) {\n+\t\treturn NULL;\n+\t}\n+\n+\tgit_path = xmalloc(fs_path_len + 6);\n+\tstrcpy(git_path, fs_path);\n+\tstrcpy(git_path + fs_path_len, \"/.git\");\n+\n+\tif (lstat(git_path, &st)) {\n+\t\t/* Watchman gets confused if we delete the .git\n+\t\t * directory out from under it, since that's where it\n+\t\t * stores its cookies.  So we'll need to delete the\n+\t\t * watch and then recreate it. It's OK for this to\n+\t\t * fail, as the watch might have already been\n+\t\t * deleted. */\n+\t\twatchman_watch_del(connection, fs_path, &wm_error);\n+\n+\t\tif (watchman_watch(connection, fs_path, &wm_error)) {\n+\t\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\tresult = watchman_do_query(connection, fs_path, query, expr, &wm_error);\n+\tif (!result) {\n+\t\twarning(\"Watchman query error: %s (at %s)\", wm_error.message, last_update);\n+\t\tgoto out;\n+\t}\n+\twatchman_free_expression(expr);\n+\twatchman_free_query(query);\n+\n+out:\n+\tfree(git_path);\n+\treturn result;\n+}\n+\n+static int cmp_stat(const void *a, const void *b)\n+{\n+\tconst struct watchman_stat* sa = a;\n+\tconst struct watchman_stat* sb = b;\n+\treturn strcmp(sa->name, sb->name);\n+}\n+\n+static void append(struct fsc_entry ***list, int* cap, int* len, struct fsc_entry *entry)\n+{\n+\tif (*len >= *cap) {\n+\t\tint sz;\n+\t\t*cap = *cap ? *cap * 2 : 10;\n+\t\tsz = *cap * sizeof(**list);\n+\t\t*list = xrealloc(*list, sz);\n+\t}\n+\t(*list)[(*len)++] = entry;\n+}\n+\n+static int is_child_of(struct fsc_entry *putative_child, struct fsc_entry *parent)\n+{\n+\twhile (putative_child) {\n+\t\tputative_child = putative_child->parent;\n+\t\tif (putative_child == parent) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static void update_fs_cache(struct index_state *istate, struct watchman_query_result *result)\n+{\n+\tstruct fs_cache *fs_cache = istate->fs_cache;\n+\tstruct fsc_entry *fe;\n+\tint i;\n+\tstruct fsc_entry **exclude_dirty = NULL;\n+\tint cap = 0, len = 0, all_dirty = 0;\n+\t/* note that we always want to call both of these functions,\n+\t * since they update the fs_cache's view of files which are\n+\t * not watched by watchman */\n+\tint user_changed = user_excludes_file_changed(fs_cache);\n+\tint git_changed = git_excludes_file_changed(fs_cache);\n+\n+\tall_dirty = user_changed || git_changed;\n+\n+\tqsort(result->stats, result->nr, sizeof(*result->stats), cmp_stat);\n+\n+\tfor (i = 0; i < result->nr; ++i) {\n+\t\t/*for each result in the changed set, we need to check\n+\t\t  it against the index and HEAD */\n+\n+\t\tstruct watchman_stat *wm = result->stats + i;\n+\n+\t\tif (is_in_dot_git(wm->name)) {\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfs_cache->needs_write = 1;\n+\t\tif (wm->exists) {\n+\t\t\tfe = fs_cache_file_modified(fs_cache, wm);\n+\t\t} else {\n+\t\t\tfe = fs_cache_file_deleted(fs_cache, wm);\n+\t\t}\n+\t\tif (fe && !all_dirty) {\n+\t\t\tif (ends_with(wm->name, \"/.gitignore\") ||\n+\t\t\t    !strcmp(wm->name, \".gitignore\")) {\n+\t\t\t\tappend(&exclude_dirty, &cap, &len, fe->parent);\n+\t\t\t} else if (fe_new(fe)) {\n+\t\t\t\tappend(&exclude_dirty, &cap, &len, fe);\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (exclude_dirty) {\n+\t\tstruct dir_struct dir;\n+\t\tstruct fsc_entry *last = NULL;\n+\t\tchar original_path[PATH_MAX + 1];\n+\t\tqsort(exclude_dirty, len, sizeof(*exclude_dirty), cmp_fsc_entry);\n+\n+\t\tif (!getcwd(original_path, PATH_MAX + 1))\n+\t\t\tdie_errno(\"failed to get working directory\\n\");\n+\t\tif (chdir(get_git_work_tree()))\n+\t\t\tdie_errno(\"failed to chdir to git work tree\\n\");\n+\n+\t\tmemset(&dir, 0, sizeof(dir));\n+\t\tsetup_standard_excludes(&dir);\n+\n+\t\tfor (i = 0; i < len; i++) {\n+\t\t\tstruct fsc_entry *fe = exclude_dirty[i];\n+\n+\t\t\tif (i == 0 || !is_child_of(fe, last)) {\n+\t\t\t\tupdate_exclude(&dir, fe);\n+\t\t\t\tlast = fe;\n+\t\t\t}\n+\t\t}\n+\t\tclear_directory(&dir);\n+\t\tfree(exclude_dirty);\n+\t\tif (chdir(original_path))\n+\t\t\tdie_errno(\"failed to chdir back to original path\\n\");\n+\t} else if (all_dirty) {\n+\t\tupdate_all_excludes(fs_cache);\n+\t}\n+\n+}\n+\n+int watchman_reload_fs_cache(struct index_state *istate)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_query_result *result;\n+\tstruct watchman_connection *connection;\n+\tint ret = -1;\n+\tconst char *fs_path;\n+\tconst char *last_update = istate->fs_cache->last_update;\n+\n+\tfs_path = get_git_work_tree();\n+\tif (!fs_path)\n+\t\treturn -1;\n+\n+\tconnection = watchman_connect(&wm_error);\n+\n+\tif (!connection) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\treturn -1;\n+\t}\n+\n+\tresult = watchman_fs_cache_query(connection, fs_path, last_update);\n+\tif (!result) {\n+\t\tgoto done;\n+\t}\n+\tistate->fs_cache->last_update = xstrdup(result->clock);\n+\n+\tupdate_fs_cache(istate, result);\n+\twatchman_free_query_result(result);\n+\tret = 0;\n+done:\n+\twatchman_connection_close(connection);\n+\treturn ret;\n+}\n+\n+int watchman_load_fs_cache(struct index_state *istate)\n+{\n+\tstruct watchman_error wm_error;\n+\tint ret = -1;\n+\tconst char *fs_path;\n+\tchar *last_update = NULL;\n+\tchar *stored_repo_path = NULL;\n+\tstruct watchman_query_result *result;\n+\tstruct watchman_connection *connection;\n+\n+\tfs_path = get_git_work_tree();\n+\tif (!fs_path)\n+\t\treturn -1;\n+\n+\tconnection = watchman_connect(&wm_error);\n+\n+\tif (!connection) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\treturn -1;\n+\t}\n+\n+\tif (watchman_watch(connection, fs_path, &wm_error)) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\tgoto done;\n+\t}\n+\n+\tfs_cache_preload_metadata(&last_update, &stored_repo_path);\n+\tif (!last_update || strcmp(stored_repo_path, fs_path)) {\n+\t\tif (istate->fs_cache) {\n+\t\t\tfree_fs_cache(istate->fs_cache);\n+\t\t\tistate->fs_cache = NULL;\n+\t\t}\n+\t\t/* fs_cache is corrupt, or refers to another repo path;\n+\t\t * let's try recreating it. */\n+\t\tif (last_update)\n+\t\t\tfree(last_update);\n+\t\tlast_update = NULL;\n+\t\t/* now we continue, because we need to get the\n+\t\t * a last-update time from watchman. */\n+\t}\n+\tfree(stored_repo_path);\n+\n+\tresult = watchman_fs_cache_query(connection, fs_path, last_update);\n+\tif (last_update) {\n+\t\tfree(last_update);\n+\t\tlast_update = NULL;\n+\t}\n+\tif (!result) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (result->is_fresh_instance) {\n+\t\tif (istate->fs_cache) {\n+\t\t\tfree_fs_cache(istate->fs_cache);\n+\t\t\tistate->fs_cache = NULL;\n+\t\t}\n+\t\tcreate_fs_cache(istate);\n+\t\tistate->fs_cache->repo_path = xstrdup(fs_path);\n+\t} else {\n+\t\tload_fs_cache(istate);\n+\t\tupdate_fs_cache(istate, result);\n+\t}\n+\n+\tistate->fs_cache->last_update = xstrdup(result->clock);\n+\n+\twatchman_free_query_result(result);\n+\tret = 0;\n+\n+done:\n+\twatchman_connection_close(connection);\n+\treturn ret;\n+\n+}\ndiff --git a/watchman-support.h b/watchman-support.h\nnew file mode 100644\nindex 0000000..1ab865f\n--- /dev/null\n+++ b/watchman-support.h\n@@ -0,0 +1,10 @@\n+#ifndef WATCHMAN_SUPPORT_H\n+#define WATCHMAN_SUPPORT_H\n+\n+#include \"cache.h\"\n+#include <watchman.h>\n+\n+int watchman_load_fs_cache(struct index_state *index);\n+int watchman_reload_fs_cache(struct index_state *index);\n+\n+#endif /* WATCHMAN_SUPPORT_H */\n-- \n2.0.0.rc0.31.g69c1a2d\n"},{"id":"240597","messageId":"536428224adfb_200c12912f010@nysa.notmuch","threadId":"36571","inReplyTo":"1399072451-15561-1-git-send-email-dturner@twopensource.com","subject":"RE: Watchman support for git","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2014-05-02T23:20:02Z","receivedAt":"2014-05-02T23:20:02Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"dturner@ wrote:\n> Test repository 1: Linux\n> \n> Linux is about 45k files in 3k directories.  The average length of a\n> filename is about 32 bytes.\n> \n> Git status timing:\n> no watchman: 125ms\n> watchman: 90ms\n\nThat's very interesting. Do you get similar improvements when doing\nsomething similar in Merurial (watchman vs . no watchman).\n\n-- \nFelipe Contreras\n"},{"id":"240603","messageId":"CACsJy8C72QQZd4v+p4bkFFKHnN2Uj1zD-BELwGtYB7sx1P6q6g@mail.gmail.com","threadId":"36571","inReplyTo":"1399072451-15561-1-git-send-email-dturner@twopensource.com","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-03T00:52:10Z","receivedAt":"2014-05-03T00:52:10Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, May 3, 2014 at 6:14 AM,  <dturner@twopensource.com> wrote:\n> The index format change might be less important with the split index;\n> I haven't investigated that since at the time I wrote these patches,\n> it didn't exist.\n\nThis is the worst case scenario of \"git status\" on webkit.git (182k\nfiles, path name 74 byte long on average), hot cache, no SSD\n\n   366.379ms gitmodules_config:199 if (read_cache() < 0) die(\"index file\n     0.004ms cmd_status:1294 read_cache_preload(&s.pathspec);\n   488.433ms cmd_status:1295 refresh_index(&the_index, REFRESH_QUIE\n   456.495ms cmd_status:1299 update_index_if_able(&the_index, &inde\n    13.088ms wt_status_collect:616 wt_status_collect_changes_worktree(s)\n   706.926ms wt_status_collect:621 wt_status_collect_changes_index(s)\n   100.495ms lazy_init_name_hash:136 { int nr; if (istate->name_hash_initia\n   921.185ms wt_status_collect:622 wt_status_collect_untracked(s)\n\nreal    0m2.969s\nuser    0m1.943s\nsys     0m1.021s\n\nIndex v4 and split index (and the following read-cache daemon,\nhopefully) should help reduce numbers of the 1st and 4th lines, I\nexpect to less than 50ms each line. lazy_init_name_hash could be taken\naway with read-cache daemon also.\n\ncore.preloadindex can cut the total number of 2nd and 3rd lines by\nhalf. Watchman should help in these two lines, but it should do better\nthan core.preloadindex.\n\nwt_status_collect_changes_index() depends on how damaged cache-tree is\n(in this case, totally scraped). watchman does not help this either.\nWe need to try to \"heal\" cache-tree as much as possible to reduce the\nnumber.\n\nThe last line could be a competition between watchman and my coming\n\"untracked cache\" series. I expect to cut the number in that line at\nleast in half without external dependency.\n\nPatch 2/3 did not seem to make it to the list by the way.. initial\nreaction is storing the list of all paths seems too much, but I'll\nneed to play with it a bit to understand it.\n-- \nDuy\n"},{"id":"240605","messageId":"1399083897.5310.0.camel@stross","threadId":"36571","inReplyTo":"536428224adfb_200c12912f010@nysa.notmuch","subject":"RE: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-03T02:24:57Z","receivedAt":"2014-05-03T02:24:57Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2014-05-02 at 18:20 -0500, Felipe Contreras wrote:\n> dturner@ wrote:\n> > Test repository 1: Linux\n> > \n> > Linux is about 45k files in 3k directories.  The average length of a\n> > filename is about 32 bytes.\n> > \n> > Git status timing:\n> > no watchman: 125ms\n> > watchman: 90ms\n> \n> That's very interesting. Do you get similar improvements when doing\n> something similar in Merurial (watchman vs . no watchman).\n\nI have not tried it.  My understanding is that this is why Facebook\nwrote Watchman and added support for it to Mercurial, so I would assume\nthat the improvements are at least this good.\n"},{"id":"240606","messageId":"5364654088dc4_4d2010fb2ec7d@nysa.notmuch","threadId":"36571","inReplyTo":"1399083897.5310.0.camel@stross","subject":"RE: Watchman support for git","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2014-05-03T03:40:48Z","receivedAt":"2014-05-03T03:40:48Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"David Turner wrote:\n> On Fri, 2014-05-02 at 18:20 -0500, Felipe Contreras wrote:\n> > dturner@ wrote:\n> > > Test repository 1: Linux\n> > > \n> > > Linux is about 45k files in 3k directories.  The average length of a\n> > > filename is about 32 bytes.\n> > > \n> > > Git status timing:\n> > > no watchman: 125ms\n> > > watchman: 90ms\n> > \n> > That's very interesting. Do you get similar improvements when doing\n> > something similar in Merurial (watchman vs . no watchman).\n> \n> I have not tried it.  My understanding is that this is why Facebook\n> wrote Watchman and added support for it to Mercurial, so I would assume\n> that the improvements are at least this good.\n\nYeah, my bet is that they are actually much better (because Mercurial\ncan't be so optimized as Git).\n\nI'm interested in this number because if watchman in Git is improving it\nby 30%, but in Mercurial it's improving it by 100% (made up number),\ntherefore it makes sens that you might want it more if you are using hg,\nbut not so much if you are using git.\n\nAlso, if similar repositories with Mercurial+watchman are actually\nfaster than Git+watchman, that means that there's room for improvement\nin your implementation. This is not a big issue at this point of the\nprocess, just something nice to know.\n\n-- \nFelipe Contreras\n"},{"id":"240607","messageId":"1399091986.5310.20.camel@stross","threadId":"36571","inReplyTo":"CACsJy8C72QQZd4v+p4bkFFKHnN2Uj1zD-BELwGtYB7sx1P6q6g@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-03T04:39:46Z","receivedAt":"2014-05-03T04:39:46Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sat, 2014-05-03 at 07:52 +0700, Duy Nguyen wrote:\n> On Sat, May 3, 2014 at 6:14 AM,  <dturner@twopensource.com> wrote:\n> > The index format change might be less important with the split index;\n> > I haven't investigated that since at the time I wrote these patches,\n> > it didn't exist.\n> \n> This is the worst case scenario of \"git status\" on webkit.git (182k\n> files, path name 74 byte long on average), hot cache, no SSD\n> \n>    366.379ms gitmodules_config:199 if (read_cache() < 0) die(\"index file\n>      0.004ms cmd_status:1294 read_cache_preload(&s.pathspec);\n>    488.433ms cmd_status:1295 refresh_index(&the_index, REFRESH_QUIE\n>    456.495ms cmd_status:1299 update_index_if_able(&the_index, &inde\n>     13.088ms wt_status_collect:616 wt_status_collect_changes_worktree(s)\n>    706.926ms wt_status_collect:621 wt_status_collect_changes_index(s)\n>    100.495ms lazy_init_name_hash:136 { int nr; if (istate->name_hash_initia\n>    921.185ms wt_status_collect:622 wt_status_collect_untracked(s)\n> \n> real    0m2.969s\n> user    0m1.943s\n> sys     0m1.021s\n\nFor me, those times are:\n0m0.581s (no watchman, index v4)\n0m0.465s (watchman, index v4)\n0m0.445s (watchman, index v5)\n\nThat's not huge win on its own, but (a) it's better than nothing and (b)\nit lays the groundwork for other improvements.\n\nA fair amount (~12%) of the time seems to be spent in zlib; this varies\nbased on how the data is packed IIRC. \n\n> Index v4 and split index (and the following read-cache daemon,\n> hopefully) \n\nLooking at some of the archives for read-cache daemon, it seems to be\nsomewhat similar to watchman, right?  But I only saw inotify code; what\nabout Mac OS?  Or am I misunderstanding what it is?\n\n> should help reduce numbers of the 1st and 4th lines, I\n> expect to less than 50ms each line. lazy_init_name_hash could be taken\n> away with read-cache daemon also.\n> \n> core.preloadindex can cut the total number of 2nd and 3rd lines by\n> half. Watchman should help in these two lines, but it should do better\n> than core.preloadindex.\n> \n> wt_status_collect_changes_index() depends on how damaged cache-tree is\n> (in this case, totally scraped). watchman does not help this either.\n> We need to try to \"heal\" cache-tree as much as possible to reduce the\n> number.\n> \n> The last line could be a competition between watchman and my coming\n> \"untracked cache\" series. I expect to cut the number in that line at\n> least in half without external dependency.\n\nI hadn't seen the \"untracked cached\" work (I actually finished these\npatches a month or so ago but have been waiting for some internal\nreviews before sending them out).  Looks interesting.  It seems we use a\nsimilar strategy for handling ignores.\n\n> Patch 2/3 did not seem to make it to the list by the way.. \n\nThanks for your comments.  I just tried again to send patch 2/3.  I do\nactually see the CC of it in my @twitter.com mailbox, but I don't see it\nin the archives on the web.  Do you know if there is a reason the\nmailing list would reject it?  At any rate, the contents may be found\nat \nhttps://github.com/dturner-tw/git/commit/cf587d54fc72d82a23267348afa2c4b60f14ce51.diff\n\n> initial\n> reaction is storing the list of all paths seems too much, but I'll\n> need to play with it a bit to understand it.\n\nI wonder if it would make sense to use the untracked cache as the\nstorage strategy, but use watchman as the update strategy.\n"},{"id":"240618","messageId":"CACsJy8B1Q3WEPT+nzDDwS5f7Wx+u5CHfN9JppRHv5VEx5NTxSw@mail.gmail.com","threadId":"36571","inReplyTo":"1399091986.5310.20.camel@stross","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-03T08:49:23Z","receivedAt":"2014-05-03T08:49:23Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, May 3, 2014 at 11:39 AM, David Turner <dturner@twopensource.com> wrote:\n>> Index v4 and split index (and the following read-cache daemon,\n>> hopefully)\n>\n> Looking at some of the archives for read-cache daemon, it seems to be\n> somewhat similar to watchman, right?  But I only saw inotify code; what\n> about Mac OS?  Or am I misunderstanding what it is?\n\nIt's mentioned in [1], the second paragraph, mostly to hide index I/O\nread cost and the SHA-1 hashing cost in the background. In theory it\nshould work on all platforms that support multiple processes and\nefficient IPC. It can help load watchman file cache faster too.\n\n>> The last line could be a competition between watchman and my coming\n>> \"untracked cache\" series. I expect to cut the number in that line at\n>> least in half without external dependency.\n>\n> I hadn't seen the \"untracked cached\" work (I actually finished these\n> patches a month or so ago but have been waiting for some internal\n> reviews before sending them out).  Looks interesting.  It seems we use a\n> similar strategy for handling ignores.\n\nYep, mostly the same at the core, except that I exploit directory\nmtime while you use inotify. Each approach has its own pros and cons,\nI think. Both should face the same traps in caching (e.g. if you \"git\nrm --cached\" a file, that file could be come either untracked, or\nignored).\n\n>> Patch 2/3 did not seem to make it to the list by the way..\n>\n> Thanks for your comments.  I just tried again to send patch 2/3.  I do\n> actually see the CC of it in my @twitter.com mailbox, but I don't see it\n> in the archives on the web.  Do you know if there is a reason the\n> mailing list would reject it?\n\nProbably its size, 131K, which is also an indicator to split it (and\nthe third patch) into smaller patches if you want to merge this\nfeature in master eventually.\n\n>   At any rate, the contents may be found\n> at\n> https://github.com/dturner-tw/git/commit/cf587d54fc72d82a23267348afa2c4b60f14ce51.diff\n\nGood enough for me :)\n\n>\n>> initial\n>> reaction is storing the list of all paths seems too much, but I'll\n>> need to play with it a bit to understand it.\n>\n> I wonder if it would make sense to use the untracked cache as the\n> storage strategy, but use watchman as the update strategy.\n\nI'm afraid not. If a directory mtime is changed, which means\nfiles/dirs have been added or deleted, the untracked code would fall\nback to the opendir/readdir/is_excluded dance again on that directory.\nIf we naively do the same using watchman, we lose its advantage that\nit knows exactly what files/dirs are added/removed. That kind of\nknowledge can help speed up the dance, which is not stored anywhere in\nthe untracked cache.\n\nWe could extend the \"read-cache daemon\" mentioned above though, to\nhide all the hard work in the background and present a good view to\ngit: when a file/dir is added, read-cache daemon classifies the new\nfiles/dirs as tracked/untracked/ignore and update its untracked cache\nin memory. When \"git status\" asks about the index and untracked cache,\nit will receive the _updated_ cache (not the on disk version any more)\nwith latest dir mtime so git can verify the cache is perfect and skip\nopendir/.... All git does is to write the index down in the end to\nmake the updated data permanent. It sounds interesting. But I'm not so\nsure if it's worth the complexity.\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/247268\n-- \nDuy\n"},{"id":"240647","messageId":"1399150149.5310.47.camel@stross","threadId":"36571","inReplyTo":"CACsJy8B1Q3WEPT+nzDDwS5f7Wx+u5CHfN9JppRHv5VEx5NTxSw@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-03T20:49:09Z","receivedAt":"2014-05-03T20:49:09Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sat, 2014-05-03 at 15:49 +0700, Duy Nguyen wrote:\n> On Sat, May 3, 2014 at 11:39 AM, David Turner <dturner@twopensource.com> wrote:\n> >> Index v4 and split index (and the following read-cache daemon,\n> >> hopefully)\n> >\n> > Looking at some of the archives for read-cache daemon, it seems to be\n> > somewhat similar to watchman, right?  But I only saw inotify code; what\n> > about Mac OS?  Or am I misunderstanding what it is?\n> \n> It's mentioned in [1], the second paragraph, mostly to hide index I/O\n> read cost and the SHA-1 hashing cost in the background. In theory it\n> should work on all platforms that support multiple processes and\n> efficient IPC. It can help load watchman file cache faster too.\n\nYes, that seems like a good idea.\n\nI actually wrote some of a more-complicated, weirder version of this\nidea.  In my version, there was a long-running git daemon process that\nheld the index, the watchman file cache, and also objects loaded from\nthe object database.  Other git commands would then send their\ncommand-line and arguments over to the daemon, which would run the\ncommands and send stdin/out/err back.  Of course, this is complicated\nbecause git commands are designed to run then exit, so they often rely\non variables being initialized to zero, or fail to free memory.  I used\nthe Boehm GC to handle the memory freeing problem.  To handle variables\nthat needed to be reinitialized, I used __attribute__(section...) to put\nthem all into one section, which I could save on daemon startup and\nrestore after each command.  I also replaced calls to exit() with a\nfunction that called longjmp() so the daemon could survive commands\nfailing.  Had I continued, I would also have had to track open file\ndescriptors to avoid leaking those.\n\nThis was a giant mess that only sort-of worked: it was difficult to\ntrack down all of the variables that needed to be reinitialized. \n\nThe advantage of my method is that there was somewhat less data to\nmarshall over IPC, and that objects could be easily cached; the\ndisadvantage is complexity, massive code changes, and the fact that it\ndidn't actually totally work at the time I ran out of time. \n\nSo I'm really looking forward to trying your version!\n\n> >> The last line could be a competition between watchman and my coming\n> >> \"untracked cache\" series. I expect to cut the number in that line at\n> >> least in half without external dependency.\n> >\n> > I hadn't seen the \"untracked cached\" work (I actually finished these\n> > patches a month or so ago but have been waiting for some internal\n> > reviews before sending them out).  Looks interesting.  It seems we use a\n> > similar strategy for handling ignores.\n> \n> Yep, mostly the same at the core, except that I exploit directory\n> mtime while you use inotify. Each approach has its own pros and cons,\n> I think. Both should face the same traps in caching (e.g. if you \"git\n> rm --cached\" a file, that file could be come either untracked, or\n> ignored).\n> \n> >> Patch 2/3 did not seem to make it to the list by the way..\n> >\n> > Thanks for your comments.  I just tried again to send patch 2/3.  I do\n> > actually see the CC of it in my @twitter.com mailbox, but I don't see it\n> > in the archives on the web.  Do you know if there is a reason the\n> > mailing list would reject it?\n> \n> Probably its size, 131K, which is also an indicator to split it (and\n> the third patch) into smaller patches if you want to merge this\n> feature in master eventually.\n\nI would like to merge the feature into master.  It works well for me,\nand some of my colleagues who have tried it out.\n\nI can split the vmac patch into two, but one of them will remain quite\nlarge because it contains the code for VMAC and AES, which total a bit\nover 100k.  Since the list will probably reject that, I'll post a link\nto a repository containing the patches.\n\nI'm not 100% sure how to split the watchman patch up.  I could add the\nfs_cache code and then separately add the watchman code that populates\nthe cache.  Do you think there is a need to divide it up beyond this?\n"},{"id":"240651","messageId":"CACsJy8Dr=m5FwmD2gXTM3bN-iYv+fyZ9RBUyvcj3UJJCC1yYtg@mail.gmail.com","threadId":"36571","inReplyTo":"1399150149.5310.47.camel@stross","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-04T00:15:11Z","receivedAt":"2014-05-04T00:15:11Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, May 4, 2014 at 3:49 AM, David Turner <dturner@twopensource.com> wrote:\n> On Sat, 2014-05-03 at 15:49 +0700, Duy Nguyen wrote:\n>> On Sat, May 3, 2014 at 11:39 AM, David Turner <dturner@twopensource.com> wrote:\n>> >> Index v4 and split index (and the following read-cache daemon,\n>> >> hopefully)\n>> >\n>> > Looking at some of the archives for read-cache daemon, it seems to be\n>> > somewhat similar to watchman, right?  But I only saw inotify code; what\n>> > about Mac OS?  Or am I misunderstanding what it is?\n>>\n>> It's mentioned in [1], the second paragraph, mostly to hide index I/O\n>> read cost and the SHA-1 hashing cost in the background. In theory it\n>> should work on all platforms that support multiple processes and\n>> efficient IPC. It can help load watchman file cache faster too.\n>\n> Yes, that seems like a good idea.\n>\n> I actually wrote some of a more-complicated, weirder version of this\n> idea.  In my version, there was a long-running git daemon process that\n> held the index, the watchman file cache, and also objects loaded from\n> the object database.  Other git commands would then send their\n> command-line and arguments over to the daemon, which would run the\n> commands and send stdin/out/err back.  Of course, this is complicated\n> because git commands are designed to run then exit, so they often rely\n> on variables being initialized to zero, or fail to free memory.  I used\n> the Boehm GC to handle the memory freeing problem.  To handle variables\n> that needed to be reinitialized, I used __attribute__(section...) to put\n> them all into one section, which I could save on daemon startup and\n> restore after each command.  I also replaced calls to exit() with a\n> function that called longjmp() so the daemon could survive commands\n> failing.  Had I continued, I would also have had to track open file\n> descriptors to avoid leaking those.\n>\n> This was a giant mess that only sort-of worked: it was difficult to\n> track down all of the variables that needed to be reinitialized.\n>\n> The advantage of my method is that there was somewhat less data to\n> marshall over IPC, and that objects could be easily cached; the\n> disadvantage is complexity, massive code changes, and the fact that it\n> didn't actually totally work at the time I ran out of time.\n>\n> So I'm really looking forward to trying your version!\n\nHm.. I may face the same problem if I'm not careful. So far I think\nthe daemon only holds index data (with on-disk format, not in-memory),\nmainly to cut out SHA-1 hashing cost. This is still at the idea phase\nfor me though, nothing is materialized yet.\n\n> I would like to merge the feature into master.  It works well for me,\n> and some of my colleagues who have tried it out.\n\nHave you tried to turn watchman on by default, then run it with git\ntest suite? That usually helps.\n\n> I can split the vmac patch into two, but one of them will remain quite\n> large because it contains the code for VMAC and AES, which total a bit\n> over 100k.  Since the list will probably reject that, I'll post a link\n> to a repository containing the patches.\n\nWith the read-cache deamon, I think hashing cost is less of an issue,\nso new hashing algorithm becomes less important. If you store the file\ncache in the deamon's memory only, there's no need to hash anything.\nBut I guess you already tried this.\n\n> I'm not 100% sure how to split the watchman patch up.  I could add the\n> fs_cache code and then separately add the watchman code that populates\n> the cache.  Do you think there is a need to divide it up beyond this?\n\nI'll need to have closer look at your patches to give any suggestions.\nAlthough if you don't mind waiting a bit, I can try to put my\nuntracked cache patches in good shape (hopefully in 2 weeks), then you\ncan mostly avoid touching dir.c and reuse my work.\n\nI backed away from watchman support because I was worried about its\noverhead (of watchman itself, and git/watchman IPC because it's not\ndesigned specifically for git), which led me to try optimizing git as\nmuch as possible without watchman first, then see how/if watchman can\nhelp on top of that. I still think it's a good approach (maybe because\nit started to make me doubt if watchman could pull a big performance\nwin on top to justify the changes to support it)\n-- \nDuy\n"},{"id":"240733","messageId":"1399313294.5310.59.camel@stross","threadId":"36571","inReplyTo":"5364654088dc4_4d2010fb2ec7d@nysa.notmuch","subject":"RE: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-05T18:08:14Z","receivedAt":"2014-05-05T18:08:14Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2014-05-02 at 22:40 -0500, Felipe Contreras wrote:\n> David Turner wrote:\n> > On Fri, 2014-05-02 at 18:20 -0500, Felipe Contreras wrote:\n> > > dturner@ wrote:\n> > > > Test repository 1: Linux\n> > > > \n> > > > Linux is about 45k files in 3k directories.  The average length of a\n> > > > filename is about 32 bytes.\n> > > > \n> > > > Git status timing:\n> > > > no watchman: 125ms\n> > > > watchman: 90ms\n> > > \n> > > That's very interesting. Do you get similar improvements when doing\n> > > something similar in Merurial (watchman vs . no watchman).\n> > \n> > I have not tried it.  My understanding is that this is why Facebook\n> > wrote Watchman and added support for it to Mercurial, so I would assume\n> > that the improvements are at least this good.\n> \n> Yeah, my bet is that they are actually much better (because Mercurial\n> can't be so optimized as Git).\n>\n> I'm interested in this number because if watchman in Git is improving it\n> by 30%, but in Mercurial it's improving it by 100% (made up number),\n> therefore it makes sens that you might want it more if you are using hg,\n> but not so much if you are using git.\n> \n> Also, if similar repositories with Mercurial+watchman are actually\n> faster than Git+watchman, that means that there's room for improvement\n> in your implementation. This is not a big issue at this point of the\n> process, just something nice to know.\n\nConverting git repos to hg seems to be incredibly slow.  (I have not yet\ntried doing it with git-remote-hg).  But I did find a hg repository for\nlinux: https://bitbucket.org/orzel/linux-kernel-stable\n\nOn this repo, I get:\nhg without watchman: 620ms\nhg with watchman: 264ms\n(compared to 125ms/90ms for git).\n\nThe number of syscalls is, perhaps also interesting:\nno watchman / with watchman\nhg  77773   /  5421\ngit 59180   /  599 \n\n(About 1/3 of hg's syscalls with watchman seem to be loading Python\nstuff)\n"},{"id":"240726","messageId":"5367d50e333f_25278db2eccf@nysa.notmuch","threadId":"36571","inReplyTo":"1399313294.5310.59.camel@stross","subject":"RE: Watchman support for git","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2014-05-05T18:14:38Z","receivedAt":"2014-05-05T18:14:38Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"David Turner wrote:\n> On Fri, 2014-05-02 at 22:40 -0500, Felipe Contreras wrote:\n> > David Turner wrote:\n> > > On Fri, 2014-05-02 at 18:20 -0500, Felipe Contreras wrote:\n> > > > dturner@ wrote:\n> > > > > Test repository 1: Linux\n> > > > > \n> > > > > Linux is about 45k files in 3k directories.  The average length of a\n> > > > > filename is about 32 bytes.\n> > > > > \n> > > > > Git status timing:\n> > > > > no watchman: 125ms\n> > > > > watchman: 90ms\n> > > > \n> > > > That's very interesting. Do you get similar improvements when doing\n> > > > something similar in Merurial (watchman vs . no watchman).\n> > > \n> > > I have not tried it.  My understanding is that this is why Facebook\n> > > wrote Watchman and added support for it to Mercurial, so I would assume\n> > > that the improvements are at least this good.\n> > \n> > Yeah, my bet is that they are actually much better (because Mercurial\n> > can't be so optimized as Git).\n> >\n> > I'm interested in this number because if watchman in Git is improving it\n> > by 30%, but in Mercurial it's improving it by 100% (made up number),\n> > therefore it makes sens that you might want it more if you are using hg,\n> > but not so much if you are using git.\n> > \n> > Also, if similar repositories with Mercurial+watchman are actually\n> > faster than Git+watchman, that means that there's room for improvement\n> > in your implementation. This is not a big issue at this point of the\n> > process, just something nice to know.\n> \n> Converting git repos to hg seems to be incredibly slow.  (I have not yet\n> tried doing it with git-remote-hg).\n\nI haven't profiled the git->hg conversion, only the other way around,\nbut I don't see why it would be slow in git-remote-hg.\n\nThe only restriction is that octopus merges are not supported, so you\nwould probably need to find another repository.\n\n> But I did find a hg repository for\n> linux: https://bitbucket.org/orzel/linux-kernel-stable\n> \n> On this repo, I get:\n> hg without watchman: 620ms\n> hg with watchman: 264ms\n> (compared to 125ms/90ms for git).\n> \n> The number of syscalls is, perhaps also interesting:\n> no watchman / with watchman\n> hg  77773   /  5421\n> git 59180   /  599 \n> \n> (About 1/3 of hg's syscalls with watchman seem to be loading Python\n> stuff)\n\nExactly what I thought :) 28% improvement for Git, 76% improvement for\nMercurial. Still, the improvement would be welcome by big companies I\nbet.\n\nCheers.\n\n-- \nFelipe Contreras\n"},{"id":"240777","messageId":"CACsJy8DWK9ysCJobh4oQZe2hxdTaE=n7hH=Nm6bfWzx+yJ=KZg@mail.gmail.com","threadId":"36571","inReplyTo":"CACsJy8C72QQZd4v+p4bkFFKHnN2Uj1zD-BELwGtYB7sx1P6q6g@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-06T00:26:26Z","receivedAt":"2014-05-06T00:26:26Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, May 3, 2014 at 7:52 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n> wt_status_collect_changes_index() depends on how damaged cache-tree is\n> (in this case, totally scraped). watchman does not help this either.\n> We need to try to \"heal\" cache-tree as much as possible to reduce the\n> number.\n\nOn the topic of cache-tree, I notice that unpack_trees() will call\ndiscard_index() in the end (or even discard_index() on a merge\nfailure). That destroys cache-tree. unpack_trees() is the core of\nbranch switching, or reset/merge, which I consider frequent\noperations. Cache-tree destruction is bad for \"git diff --cached\",\n\"git commit\" and maybe more. This discard_index() code was added\nrecently in e28f764 (unpack-trees: plug a memory leak - 2013-08-13).\nAs a workaround, perhaps we only do so when the sequencer is running.\n-- \nDuy\n"},{"id":"240727","messageId":"CACsJy8B_NiL8LnYXk_O+tbsZxfNbg5EvkbbPtO-crHMmKdZhEA@mail.gmail.com","threadId":"36571","inReplyTo":"CACsJy8DWK9ysCJobh4oQZe2hxdTaE=n7hH=Nm6bfWzx+yJ=KZg@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-06T00:30:17Z","receivedAt":"2014-05-06T00:30:17Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 6, 2014 at 7:26 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n> This discard_index() code was added\n> recently in e28f764 (unpack-trees: plug a memory leak - 2013-08-13).\n> As a workaround, perhaps we only do so when the sequencer is running.\n\nHmm.. as o->result does not carry cache-tree anyway, the next\nassignment after discard_index() will \"destroy\" cache-tree anyway. I\nwrote and deleted the following sentence, but it looks like I should\nrewrite again. So if cache-tree has always been destroyed before, be\ncareful when you try to keep cache-tree as cache-tree invalidation\ncode in unpack-trees.c may not be well tested.\n-- \nDuy\n"},{"id":"240799","messageId":"1399346024.11843.43.camel@stross","threadId":"36571","inReplyTo":"CACsJy8Dr=m5FwmD2gXTM3bN-iYv+fyZ9RBUyvcj3UJJCC1yYtg@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-06T03:13:44Z","receivedAt":"2014-05-06T03:13:44Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sun, 2014-05-04 at 07:15 +0700, Duy Nguyen wrote:\n> > I would like to merge the feature into master.  It works well for me,\n> > and some of my colleagues who have tried it out.\n> \n> Have you tried to turn watchman on by default, then run it with git\n> test suite? That usually helps.\n\nI have.  The tests work run fine under make, but prove sometimes freezes\ndue to an issue in libwatchman which I just fixed (and which I plan to\nmerge as soon as I can get a colleague to look the changes over).\n\n> > I can split the vmac patch into two, but one of them will remain quite\n> > large because it contains the code for VMAC and AES, which total a bit\n> > over 100k.  Since the list will probably reject that, I'll post a link\n> > to a repository containing the patches.\n> \n> With the read-cache deamon, I think hashing cost is less of an issue,\n> so new hashing algorithm becomes less important. If you store the file\n> cache in the deamon's memory only, there's no need to hash anything.\n> But I guess you already tried this.\n\nI agree that with the daemon, the cost is less of an issue, but I am not\n100% sure it is a non-issue; consecutive commands that need to\nread/write the index can still be slowed down.\n\n> > I'm not 100% sure how to split the watchman patch up.  I could add the\n> > fs_cache code and then separately add the watchman code that populates\n> > the cache.  Do you think there is a need to divide it up beyond this?\n> \n> I'll need to have closer look at your patches to give any suggestions.\n\nI have uploaded a new version (which is about 5-10% faster and which\ncorrects some minor changes) to https://github.com/dturner-tw/git.git on\nthe watchman branch.  \n\n> Although if you don't mind waiting a bit, I can try to put my\n> untracked cache patches in good shape (hopefully in 2 weeks), then you\n> can mostly avoid touching dir.c and reuse my work.\n\nIf the untracked cache patches are going to make it into master, then I\nwould of course be willing to rewrite on top of them.  But I would also\nlike to have a sense of whether there is any interest in watchman\nsupport (outside of Twitter).\n\nFor what it's worth, the numbers today for index version 4 are for my\nsuperscience repo are:\n~380 (no watchman), ~260 (untracked-cache), ~175 (watchman).\n\nThat's because untracked-cache still has to stat every directory.\n\n> I backed away from watchman support because I was worried about its\n> overhead (of watchman itself, and git/watchman IPC because it's not\n> designed specifically for git), which led me to try optimizing git as\n> much as possible without watchman first, then see how/if watchman can\n> help on top of that. I still think it's a good approach (maybe because\n> it started to make me doubt if watchman could pull a big performance\n> win on top to justify the changes to support it)\n\nI think on large repositories (especially deeply-nested ones), with the\ncommon case of a small number of changes, watchman will end up being a\nbig win.  Java tends towards deep nesting\n(src/main/java/com/twitter/common/...), which is probably why my test\nrepo had the largest speedup (>50%).  The IPC overhead might become bad\nif there were a large number of changes, but so far this has not been an\nissue for me in testing.  \n"},{"id":"240846","messageId":"xmqqtx92czqv.fsf@gitster.dls.corp.google.com","threadId":"36571","inReplyTo":"1399072451-15561-2-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 1/3] After chdir to run grep, return to old directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-06T22:24:08Z","receivedAt":"2014-05-06T22:24:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"dturner@twopensource.com writes:\n\n> From: David Turner <dturner@twitter.com>\n>\n> Signed-off-by: David Turner <dturner@twitter.com>\n\nEhh, why?\n\n> ---\n>  builtin/grep.c | 14 ++++++++++++--\n>  1 file changed, 12 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 69ac2d8..e9fe040 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -355,15 +355,25 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n>  {\n>  \tstruct string_list *path_list = opt->output_priv;\n>  \tconst char **argv = xmalloc(sizeof(const char *) * (path_list->nr + 1));\n> +\tstatic char old_directory[PATH_MAX+1];\n>  \tint i, status;\n>  \n>  \tfor (i = 0; i < path_list->nr; i++)\n>  \t\targv[i] = path_list->items[i].string;\n>  \targv[path_list->nr] = NULL;\n>  \n> -\tif (prefix && chdir(prefix))\n> -\t\tdie(_(\"Failed to chdir: %s\"), prefix);\n> +\n> +\tif (prefix) {\n> +\t\tif (!getcwd(old_directory, PATH_MAX+1))\n> +\t\t\tdie(_(\"Failed to get cwd: %s\"), prefix);\n> +\t\tif (chdir(prefix))\n> +\t\t\tdie(_(\"Failed to chdir: %s\"), prefix);\n> +\t}\n>  \tstatus = run_command_v_opt(argv, RUN_USING_SHELL);\n> +\tif (prefix)\n> +\t\tif (chdir(old_directory))\n> +\t\t\tdie(_(\"Failed to chdir: %s\"), old_directory);\n> +\n>  \tif (status)\n>  \t\texit(status);\n>  \tfree(argv);\n"},{"id":"240860","messageId":"1399421211.11843.53.camel@stross","threadId":"36571","inReplyTo":"xmqqtx92czqv.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 1/3] After chdir to run grep, return to old directory","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-07T00:06:51Z","receivedAt":"2014-05-07T00:06:51Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2014-05-06 at 15:24 -0700, Junio C Hamano wrote:\n> dturner@twopensource.com writes:\n> \n> > From: David Turner <dturner@twitter.com>\n> >\n> > Signed-off-by: David Turner <dturner@twitter.com>\n> \n> Ehh, why?\n\nBriefly, because otherwise ./t7811-grep-open.sh fails when run under\nwatchman.\n\nThis is actually something that I think I'm doing wrong, but I can't see\nwhat the sensible way to do it is.  When we go to write the fs_cache (in\nan atexit hook), we use get_fs_cache_file() from environment.c.  This is\na relative path, because all of the other similar paths are.  So if we\nhave chdired, then we fail.\n"},{"id":"240866","messageId":"20140507030037.GB20726@sigill.intra.peff.net","threadId":"36571","inReplyTo":"1399421211.11843.53.camel@stross","subject":"Re: [PATCH 1/3] After chdir to run grep, return to old directory","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-05-07T03:00:37Z","receivedAt":"2014-05-07T03:00:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 06, 2014 at 05:06:51PM -0700, David Turner wrote:\n\n> On Tue, 2014-05-06 at 15:24 -0700, Junio C Hamano wrote:\n> > dturner@twopensource.com writes:\n> > \n> > > From: David Turner <dturner@twitter.com>\n> > >\n> > > Signed-off-by: David Turner <dturner@twitter.com>\n> > \n> > Ehh, why?\n> \n> Briefly, because otherwise ./t7811-grep-open.sh fails when run under\n> watchman.\n> \n> This is actually something that I think I'm doing wrong, but I can't see\n> what the sensible way to do it is.  When we go to write the fs_cache (in\n> an atexit hook), we use get_fs_cache_file() from environment.c.  This is\n> a relative path, because all of the other similar paths are.  So if we\n> have chdired, then we fail.\n\nIt sounds like a reasonable code-hygiene thing, too. It looks like we\nare only doing the chdir to impact the environment of the pager, and\naffecting the global environment of the parent process is a side effect.\nIt happens not to cause problems now because we exit immediately, but\nanybody who adds code after run_pager has to deal with it (and that is\nbasically what you are doing here). Something like cleaning up tempfiles\nwould be similarly affected.\n\nThat being said, this really seems like something that the run-command\ninterface should be doing, since it can handle the chdir in the forked\nchild. And indeed, it seems to support that.\n\nMaybe:\n\n-- >8 --\nSubject: grep: use run-command's \"dir\" option for --open-files-in-pager\n\nGit generally changes directory to the repository root on\nstartup.  When running \"grep --open-files-in-pager\" from a\nsubdirectory, we chdir back to the original directory before\nrunning the pager, so that we can feed the relative\npathnames to the pager.\n\nWe currently do this chdir manually, but we can ask\nrun_command to do it for us. This is fewer lines of code,\nand as a bonus, the chdir is limited to the child process,\nwhich avoids any unexpected surprises for code running after\nthe pager (there isn't any currently, but this is\nfuture-proofing).\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n builtin/grep.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 69ac2d8..43af5b7 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -361,9 +361,7 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \t\targv[i] = path_list->items[i].string;\n \targv[path_list->nr] = NULL;\n \n-\tif (prefix && chdir(prefix))\n-\t\tdie(_(\"Failed to chdir: %s\"), prefix);\n-\tstatus = run_command_v_opt(argv, RUN_USING_SHELL);\n+\tstatus = run_command_v_opt_cd_env(argv, RUN_USING_SHELL, prefix, NULL);\n \tif (status)\n \t\texit(status);\n \tfree(argv);\n-- \n2.0.0.rc1.436.g03cb729\n"},{"id":"240868","messageId":"1399433631.11843.57.camel@stross","threadId":"36571","inReplyTo":"20140507030037.GB20726@sigill.intra.peff.net","subject":"Re: [PATCH 1/3] After chdir to run grep, return to old directory","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-07T03:33:51Z","receivedAt":"2014-05-07T03:33:51Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"This causes my test to pass and generally seems correct to me.\n\nOn Tue, 2014-05-06 at 23:00 -0400, Jeff King wrote:\n...\n> That being said, this really seems like something that the run-command\n> interface should be doing, since it can handle the chdir in the forked\n> child. And indeed, it seems to support that.\n> \n> Maybe:\n> \n> -- >8 --\n> Subject: grep: use run-command's \"dir\" option for --open-files-in-pager\n> \n> Git generally changes directory to the repository root on\n> startup.  When running \"grep --open-files-in-pager\" from a\n> subdirectory, we chdir back to the original directory before\n> running the pager, so that we can feed the relative\n> pathnames to the pager.\n> \n> We currently do this chdir manually, but we can ask\n> run_command to do it for us. This is fewer lines of code,\n> and as a bonus, the chdir is limited to the child process,\n> which avoids any unexpected surprises for code running after\n> the pager (there isn't any currently, but this is\n> future-proofing).\n> \n> Signed-off-by: Jeff King <peff@peff.net>\n> ---\n>  builtin/grep.c | 4 +---\n>  1 file changed, 1 insertion(+), 3 deletions(-)\n> \n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index 69ac2d8..43af5b7 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -361,9 +361,7 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n>  \t\targv[i] = path_list->items[i].string;\n>  \targv[path_list->nr] = NULL;\n>  \n> -\tif (prefix && chdir(prefix))\n> -\t\tdie(_(\"Failed to chdir: %s\"), prefix);\n> -\tstatus = run_command_v_opt(argv, RUN_USING_SHELL);\n> +\tstatus = run_command_v_opt_cd_env(argv, RUN_USING_SHELL, prefix, NULL);\n>  \tif (status)\n>  \t\texit(status);\n>  \tfree(argv);\n"},{"id":"240922","messageId":"xmqqlhudbi4r.fsf@gitster.dls.corp.google.com","threadId":"36571","inReplyTo":"1399433631.11843.57.camel@stross","subject":"Re: [PATCH 1/3] After chdir to run grep, return to old directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-07T17:42:12Z","receivedAt":"2014-05-07T17:42:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner <dturner@twopensource.com> writes:\n\n> This causes my test to pass and generally seems correct to me.\n\nYes, this approach is very sensible, and I'll queue.\n\nBut watchman support _should_ be prepared for a program that does\nnot do this.  Developing your support in on a codebase with this\npatch may be sweeping a bug in your code under the rug.\n\nThanks both for working on your respective changes ;-).\n\n>\n> On Tue, 2014-05-06 at 23:00 -0400, Jeff King wrote:\n> ...\n>> That being said, this really seems like something that the run-command\n>> interface should be doing, since it can handle the chdir in the forked\n>> child. And indeed, it seems to support that.\n>> \n>> Maybe:\n>> \n>> -- >8 --\n>> Subject: grep: use run-command's \"dir\" option for --open-files-in-pager\n>> \n>> Git generally changes directory to the repository root on\n>> startup.  When running \"grep --open-files-in-pager\" from a\n>> subdirectory, we chdir back to the original directory before\n>> running the pager, so that we can feed the relative\n>> pathnames to the pager.\n>> \n>> We currently do this chdir manually, but we can ask\n>> run_command to do it for us. This is fewer lines of code,\n>> and as a bonus, the chdir is limited to the child process,\n>> which avoids any unexpected surprises for code running after\n>> the pager (there isn't any currently, but this is\n>> future-proofing).\n>> \n>> Signed-off-by: Jeff King <peff@peff.net>\n>> ---\n>>  builtin/grep.c | 4 +---\n>>  1 file changed, 1 insertion(+), 3 deletions(-)\n>> \n>> diff --git a/builtin/grep.c b/builtin/grep.c\n>> index 69ac2d8..43af5b7 100644\n>> --- a/builtin/grep.c\n>> +++ b/builtin/grep.c\n>> @@ -361,9 +361,7 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n>>  \t\targv[i] = path_list->items[i].string;\n>>  \targv[path_list->nr] = NULL;\n>>  \n>> -\tif (prefix && chdir(prefix))\n>> -\t\tdie(_(\"Failed to chdir: %s\"), prefix);\n>> -\tstatus = run_command_v_opt(argv, RUN_USING_SHELL);\n>> +\tstatus = run_command_v_opt_cd_env(argv, RUN_USING_SHELL, prefix, NULL);\n>>  \tif (status)\n>>  \t\texit(status);\n>>  \tfree(argv);\n"},{"id":"240962","messageId":"1399496239.11843.78.camel@stross","threadId":"36571","inReplyTo":"xmqqlhudbi4r.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 1/3] After chdir to run grep, return to old directory","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-07T20:57:19Z","receivedAt":"2014-05-07T20:57:19Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2014-05-07 at 10:42 -0700, Junio C Hamano wrote:\n> David Turner <dturner@twopensource.com> writes:\n> \n> > This causes my test to pass and generally seems correct to me.\n> \n> Yes, this approach is very sensible, and I'll queue.\n> \n> But watchman support _should_ be prepared for a program that does\n> not do this.  Developing your support in on a codebase with this\n> patch may be sweeping a bug in your code under the rug.\n\nI agree that good defensive coding practice would be to not depend on\nthe cwd.  That's just what everything in environment.c presently does.\nI don't want to change the rest of the get_*_file functions, because I\ndon't know what their callers expect of them.  Here is my patch to do\nchange just the fs_cache bit:\n\nhttps://github.com/dturner-tw/git/commit/3fe93aeaee9719ee171a253c49af5126a057c513.patch\n\n(I went ahead and made it part of \nthe watchman branch on https://github.com/dturner-tw/git.git )\n\nI wanted to just do the fixup where the path is used, but I couldn't see\na function for that.  If there's no function, that indicates to me that\nit's probably a bad idea.  But maybe I'm just missing it.\n"},{"id":"241020","messageId":"536BD864.5090804@gmail.com","threadId":"36571","inReplyTo":"5364654088dc4_4d2010fb2ec7d@nysa.notmuch","subject":"Re: Watchman support for git","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2014-05-08T19:17:56Z","receivedAt":"2014-05-08T19:17:56Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On 03.05.2014 05:40, Felipe Contreras wrote:\n\n>>> That's very interesting. Do you get similar improvements when doing\n>>> something similar in Merurial (watchman vs . no watchman).\n>>\n>> I have not tried it.  My understanding is that this is why Facebook\n>> wrote Watchman and added support for it to Mercurial, so I would assume\n>> that the improvements are at least this good.\n> \n> Yeah, my bet is that they are actually much better (because Mercurial\n> can't be so optimized as Git).\n> \n> I'm interested in this number because if watchman in Git is improving it\n> by 30%, but in Mercurial it's improving it by 100% (made up number),\n> therefore it makes sens that you might want it more if you are using hg,\n> but not so much if you are using git.\n> \n> Also, if similar repositories with Mercurial+watchman are actually\n> faster than Git+watchman, that means that there's room for improvement\n> in your implementation. This is not a big issue at this point of the\n> process, just something nice to know.\n\nThe article at [1] has some details, they claim \"For our repository, enabling Watchman integration has made Mercurial's status command more than 5x faster than Git's status command\".\n\n[1] https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/\n\n-- \nSebastian Schuberth\n"},{"id":"241106","messageId":"alpine.DEB.2.02.1405090002000.17457@nftneq.ynat.uz","threadId":"36571","inReplyTo":"536BD864.5090804@gmail.com","subject":"Re: Watchman support for git","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2014-05-09T07:08:25Z","receivedAt":"2014-05-09T07:08:25Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Thu, 8 May 2014, Sebastian Schuberth wrote:\n\n> On 03.05.2014 05:40, Felipe Contreras wrote:\n>\n>>>> That's very interesting. Do you get similar improvements when doing\n>>>> something similar in Merurial (watchman vs . no watchman).\n>>>\n>>> I have not tried it.  My understanding is that this is why Facebook\n>>> wrote Watchman and added support for it to Mercurial, so I would assume\n>>> that the improvements are at least this good.\n>>\n>> Yeah, my bet is that they are actually much better (because Mercurial\n>> can't be so optimized as Git).\n>>\n>> I'm interested in this number because if watchman in Git is improving it\n>> by 30%, but in Mercurial it's improving it by 100% (made up number),\n>> therefore it makes sens that you might want it more if you are using hg,\n>> but not so much if you are using git.\n>>\n>> Also, if similar repositories with Mercurial+watchman are actually\n>> faster than Git+watchman, that means that there's room for improvement\n>> in your implementation. This is not a big issue at this point of the\n>> process, just something nice to know.\n>\n> The article at [1] has some details, they claim \"For our repository, enabling Watchman integration has made Mercurial's status command more than 5x faster than Git's status command\".\n>\n> [1] https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/\n\na lot of that speed comparison is going to depend on your storage system and the \nsize of your repository.\n\nif you have a high-end enterprise storage system that tracks metadata very \ndifferently from the file contents (I've seen some that have rackes worth of \nSATA drives for contents and then 'small' arrays of a few dozen flash drives for \nthe metadata), and then you have very large repositories (Facebook has \neverything in a single repo), then you have a perfect storm where something like \nwatchman that talks the proprietary protocol of the storage array can be FAR \nfaster than anything that needs to operate with the standard POSIX calls.\n\nThat can easily account for the difference between the facebook announcement and \nthe results presented for normal disks that show an improvement, but with even \nstock git being faster than improved mercurial.\n\nDavid Lang\n"},{"id":"241179","messageId":"1399655858.11843.119.camel@stross","threadId":"36571","inReplyTo":"alpine.DEB.2.02.1405090002000.17457@nftneq.ynat.uz","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-09T17:17:38Z","receivedAt":"2014-05-09T17:17:38Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2014-05-09 at 00:08 -0700, David Lang wrote:\n> On Thu, 8 May 2014, Sebastian Schuberth wrote:\n> \n> > On 03.05.2014 05:40, Felipe Contreras wrote:\n> >\n> >>>> That's very interesting. Do you get similar improvements when doing\n> >>>> something similar in Merurial (watchman vs . no watchman).\n> >>>\n> >>> I have not tried it.  My understanding is that this is why Facebook\n> >>> wrote Watchman and added support for it to Mercurial, so I would assume\n> >>> that the improvements are at least this good.\n> >>\n> >> Yeah, my bet is that they are actually much better (because Mercurial\n> >> can't be so optimized as Git).\n> >>\n> >> I'm interested in this number because if watchman in Git is improving it\n> >> by 30%, but in Mercurial it's improving it by 100% (made up number),\n> >> therefore it makes sens that you might want it more if you are using hg,\n> >> but not so much if you are using git.\n> >>\n> >> Also, if similar repositories with Mercurial+watchman are actually\n> >> faster than Git+watchman, that means that there's room for improvement\n> >> in your implementation. This is not a big issue at this point of the\n> >> process, just something nice to know.\n> >\n> > The article at [1] has some details, they claim \"For our repository, enabling Watchman integration has made Mercurial's status command more than 5x faster than Git's status command\".\n> >\n> > [1] https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/\n> \n> a lot of that speed comparison is going to depend on your storage system and the \n> size of your repository.\n> \n> if you have a high-end enterprise storage system that tracks metadata very \n> differently from the file contents (I've seen some that have rackes worth of \n> SATA drives for contents and then 'small' arrays of a few dozen flash drives for \n> the metadata), and then you have very large repositories (Facebook has \n> everything in a single repo), then you have a perfect storm where something like \n> watchman that talks the proprietary protocol of the storage array can be FAR \n> faster than anything that needs to operate with the standard POSIX calls.\n> \n> That can easily account for the difference between the facebook announcement and \n> the results presented for normal disks that show an improvement, but with even \n> stock git being faster than improved mercurial.\n\nAs I recall from Facebook's presentation[1] on this (as well as from the\ndiscussion on the git mailing list[2]), Facebook's test respository is\nmuch larger than any known git repository.  In particular, it is larger\nthan WebKit. These performance improvements are not for server-side\ntasks, but for client-side (e.g. git/hg status).  Facebook also made\nother improvements for the client-server communication, and for\nlog/blame, but these are not relevant to watchman.\n\nIt is entirely possible that, as repo size grows, Mercurial with\nwatchman is faster than git without.\n\nWith my patches, git status isn't constant-time; it's merely a roughly\nconstant factor faster. My initial design was to make git status\nconstant-time by caching the results of the wt_status_collect calls.\nBut there were so many cases with the various options that I got a bit\nlost in the wilderness and made a big mess. Maybe I would do better if I\ntried it again today.  And maybe if I just build on top of the\nuntracked-cache code, I would be able to get to constant-time; I'll have\nto try that at some point.  \n\n[1] http://www.youtube.com/watch?v=Dlguc63cRXg\n[2] http://comments.gmane.org/gmane.comp.version-control.git/189776\n"},{"id":"241190","messageId":"alpine.DEB.2.02.1405091103200.5876@nftneq.ynat.uz","threadId":"36571","inReplyTo":"1399655858.11843.119.camel@stross","subject":"Re: Watchman support for git","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2014-05-09T18:08:01Z","receivedAt":"2014-05-09T18:08:01Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Fri, 9 May 2014, David Turner wrote:\n\n> On Fri, 2014-05-09 at 00:08 -0700, David Lang wrote:\n>> On Thu, 8 May 2014, Sebastian Schuberth wrote:\n>>\n>>> On 03.05.2014 05:40, Felipe Contreras wrote:\n>>>\n>>>>>> That's very interesting. Do you get similar improvements when doing\n>>>>>> something similar in Merurial (watchman vs . no watchman).\n>>>>>\n>>>>> I have not tried it.  My understanding is that this is why Facebook\n>>>>> wrote Watchman and added support for it to Mercurial, so I would assume\n>>>>> that the improvements are at least this good.\n>>>>\n>>>> Yeah, my bet is that they are actually much better (because Mercurial\n>>>> can't be so optimized as Git).\n>>>>\n>>>> I'm interested in this number because if watchman in Git is improving it\n>>>> by 30%, but in Mercurial it's improving it by 100% (made up number),\n>>>> therefore it makes sens that you might want it more if you are using hg,\n>>>> but not so much if you are using git.\n>>>>\n>>>> Also, if similar repositories with Mercurial+watchman are actually\n>>>> faster than Git+watchman, that means that there's room for improvement\n>>>> in your implementation. This is not a big issue at this point of the\n>>>> process, just something nice to know.\n>>>\n>>> The article at [1] has some details, they claim \"For our repository, enabling Watchman integration has made Mercurial's status command more than 5x faster than Git's status command\".\n>>>\n>>> [1] https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/\n>>\n>> a lot of that speed comparison is going to depend on your storage system and the\n>> size of your repository.\n>>\n>> if you have a high-end enterprise storage system that tracks metadata very\n>> differently from the file contents (I've seen some that have rackes worth of\n>> SATA drives for contents and then 'small' arrays of a few dozen flash drives for\n>> the metadata), and then you have very large repositories (Facebook has\n>> everything in a single repo), then you have a perfect storm where something like\n>> watchman that talks the proprietary protocol of the storage array can be FAR\n>> faster than anything that needs to operate with the standard POSIX calls.\n>>\n>> That can easily account for the difference between the facebook announcement and\n>> the results presented for normal disks that show an improvement, but with even\n>> stock git being faster than improved mercurial.\n>\n> As I recall from Facebook's presentation[1] on this (as well as from the\n> discussion on the git mailing list[2]), Facebook's test respository is\n> much larger than any known git repository.  In particular, it is larger\n> than WebKit.\n\nagreed, it's huge, it's the entire codebase history of every tool that they use \ncrammed together in one rep\n\n> These performance improvements are not for server-side\n> tasks, but for client-side (e.g. git/hg status).  Facebook also made\n> other improvements for the client-server communication, and for\n> log/blame, but these are not relevant to watchman.\n\nwell, in their situation they have shared storage that clients use for this huge \nrepo, so I don't think they have a clear client/server boundry the way you are \nthinking. Even clients have this huge repo to deal with, and they can do so \nefficiently by querying the storage device rather than trying to walk the tree \nor monitor access directly.\n\n> It is entirely possible that, as repo size grows, Mercurial with\n> watchman is faster than git without.\n>\n> With my patches, git status isn't constant-time; it's merely a roughly\n> constant factor faster. My initial design was to make git status\n> constant-time by caching the results of the wt_status_collect calls.\n\nThis is what you would have to do with traditional storage. My understanding is \nthat the real benefits of watchman show up when you have non-traditional storage \nand can take advantage of the knowledge that the storage system gathers for it's \nown use.\n\nDavid Lang\n\n> But there were so many cases with the various options that I got a bit\n> lost in the wilderness and made a big mess. Maybe I would do better if I\n> tried it again today.  And maybe if I just build on top of the\n> untracked-cache code, I would be able to get to constant-time; I'll have\n> to try that at some point.\n>\n> [1] http://www.youtube.com/watch?v=Dlguc63cRXg\n> [2] http://comments.gmane.org/gmane.comp.version-control.git/189776\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"241191","messageId":"1399659442.11843.125.camel@stross","threadId":"36571","inReplyTo":"alpine.DEB.2.02.1405091103200.5876@nftneq.ynat.uz","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-09T18:17:22Z","receivedAt":"2014-05-09T18:17:22Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2014-05-09 at 11:08 -0700, David Lang wrote:\n> On Fri, 9 May 2014, David Turner wrote:\n> \n> > On Fri, 2014-05-09 at 00:08 -0700, David Lang wrote:\n> >> On Thu, 8 May 2014, Sebastian Schuberth wrote:\n> >>\n> >>> On 03.05.2014 05:40, Felipe Contreras wrote:\n> >>>\n> >>>>>> That's very interesting. Do you get similar improvements when doing\n> >>>>>> something similar in Merurial (watchman vs . no watchman).\n> >>>>>\n> >>>>> I have not tried it.  My understanding is that this is why Facebook\n> >>>>> wrote Watchman and added support for it to Mercurial, so I would assume\n> >>>>> that the improvements are at least this good.\n> >>>>\n> >>>> Yeah, my bet is that they are actually much better (because Mercurial\n> >>>> can't be so optimized as Git).\n> >>>>\n> >>>> I'm interested in this number because if watchman in Git is improving it\n> >>>> by 30%, but in Mercurial it's improving it by 100% (made up number),\n> >>>> therefore it makes sens that you might want it more if you are using hg,\n> >>>> but not so much if you are using git.\n> >>>>\n> >>>> Also, if similar repositories with Mercurial+watchman are actually\n> >>>> faster than Git+watchman, that means that there's room for improvement\n> >>>> in your implementation. This is not a big issue at this point of the\n> >>>> process, just something nice to know.\n> >>>\n> >>> The article at [1] has some details, they claim \"For our repository, enabling Watchman integration has made Mercurial's status command more than 5x faster than Git's status command\".\n> >>>\n> >>> [1] https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/\n> >>\n> >> a lot of that speed comparison is going to depend on your storage system and the\n> >> size of your repository.\n> >>\n> >> if you have a high-end enterprise storage system that tracks metadata very\n> >> differently from the file contents (I've seen some that have rackes worth of\n> >> SATA drives for contents and then 'small' arrays of a few dozen flash drives for\n> >> the metadata), and then you have very large repositories (Facebook has\n> >> everything in a single repo), then you have a perfect storm where something like\n> >> watchman that talks the proprietary protocol of the storage array can be FAR\n> >> faster than anything that needs to operate with the standard POSIX calls.\n> >>\n> >> That can easily account for the difference between the facebook announcement and\n> >> the results presented for normal disks that show an improvement, but with even\n> >> stock git being faster than improved mercurial.\n> >\n> > As I recall from Facebook's presentation[1] on this (as well as from the\n> > discussion on the git mailing list[2]), Facebook's test respository is\n> > much larger than any known git repository.  In particular, it is larger\n> > than WebKit.\n> \n> agreed, it's huge, it's the entire codebase history of every tool that they use \n> crammed together in one rep\n> \n> > These performance improvements are not for server-side\n> > tasks, but for client-side (e.g. git/hg status).  Facebook also made\n> > other improvements for the client-server communication, and for\n> > log/blame, but these are not relevant to watchman.\n> \n> well, in their situation they have shared storage that clients use for this huge \n> repo, so I don't think they have a clear client/server boundry the way you are \n> thinking. Even clients have this huge repo to deal with, and they can do so \n> efficiently by querying the storage device rather than trying to walk the tree \n> or monitor access directly.\n\nThat's not my understanding from Durham Goode's talk in January.  Yes,\noperations involving history go to the server.  But the client also\nmaintains a copy of the working tree, and it is for this that watchman\nis used.  Otherwise, why bother with watchman at all?  The server knows\nwhen it changes files and could simply maintain its own index of what's\nchanged.  Watchman is built on inotify/fsevents -- it doesn't have\nanything to do with any sort of storage device beyond a vanilla hard\ndrive.\n"},{"id":"241193","messageId":"alpine.DEB.2.02.1405091124150.5876@nftneq.ynat.uz","threadId":"36571","inReplyTo":"1399659442.11843.125.camel@stross","subject":"Re: Watchman support for git","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2014-05-09T18:27:12Z","receivedAt":"2014-05-09T18:27:12Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Fri, 9 May 2014, David Turner wrote:\n\n> On Fri, 2014-05-09 at 11:08 -0700, David Lang wrote:\n>> On Fri, 9 May 2014, David Turner wrote:\n>>\n>>> On Fri, 2014-05-09 at 00:08 -0700, David Lang wrote:\n>>>> On Thu, 8 May 2014, Sebastian Schuberth wrote:\n>>>>\n>>>>> On 03.05.2014 05:40, Felipe Contreras wrote:\n>>>>>\n>>>>>>>> That's very interesting. Do you get similar improvements when doing\n>>>>>>>> something similar in Merurial (watchman vs . no watchman).\n>>>>>>>\n>>>>>>> I have not tried it.  My understanding is that this is why Facebook\n>>>>>>> wrote Watchman and added support for it to Mercurial, so I would assume\n>>>>>>> that the improvements are at least this good.\n>>>>>>\n>>>>>> Yeah, my bet is that they are actually much better (because Mercurial\n>>>>>> can't be so optimized as Git).\n>>>>>>\n>>>>>> I'm interested in this number because if watchman in Git is improving it\n>>>>>> by 30%, but in Mercurial it's improving it by 100% (made up number),\n>>>>>> therefore it makes sens that you might want it more if you are using hg,\n>>>>>> but not so much if you are using git.\n>>>>>>\n>>>>>> Also, if similar repositories with Mercurial+watchman are actually\n>>>>>> faster than Git+watchman, that means that there's room for improvement\n>>>>>> in your implementation. This is not a big issue at this point of the\n>>>>>> process, just something nice to know.\n>>>>>\n>>>>> The article at [1] has some details, they claim \"For our repository, enabling Watchman integration has made Mercurial's status command more than 5x faster than Git's status command\".\n>>>>>\n>>>>> [1] https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/\n>>>>\n>>>> a lot of that speed comparison is going to depend on your storage system and the\n>>>> size of your repository.\n>>>>\n>>>> if you have a high-end enterprise storage system that tracks metadata very\n>>>> differently from the file contents (I've seen some that have rackes worth of\n>>>> SATA drives for contents and then 'small' arrays of a few dozen flash drives for\n>>>> the metadata), and then you have very large repositories (Facebook has\n>>>> everything in a single repo), then you have a perfect storm where something like\n>>>> watchman that talks the proprietary protocol of the storage array can be FAR\n>>>> faster than anything that needs to operate with the standard POSIX calls.\n>>>>\n>>>> That can easily account for the difference between the facebook announcement and\n>>>> the results presented for normal disks that show an improvement, but with even\n>>>> stock git being faster than improved mercurial.\n>>>\n>>> As I recall from Facebook's presentation[1] on this (as well as from the\n>>> discussion on the git mailing list[2]), Facebook's test respository is\n>>> much larger than any known git repository.  In particular, it is larger\n>>> than WebKit.\n>>\n>> agreed, it's huge, it's the entire codebase history of every tool that they use\n>> crammed together in one rep\n>>\n>>> These performance improvements are not for server-side\n>>> tasks, but for client-side (e.g. git/hg status).  Facebook also made\n>>> other improvements for the client-server communication, and for\n>>> log/blame, but these are not relevant to watchman.\n>>\n>> well, in their situation they have shared storage that clients use for this huge\n>> repo, so I don't think they have a clear client/server boundry the way you are\n>> thinking. Even clients have this huge repo to deal with, and they can do so\n>> efficiently by querying the storage device rather than trying to walk the tree\n>> or monitor access directly.\n>\n> That's not my understanding from Durham Goode's talk in January.  Yes,\n> operations involving history go to the server.  But the client also\n> maintains a copy of the working tree, and it is for this that watchman\n> is used.  Otherwise, why bother with watchman at all?  The server knows\n> when it changes files and could simply maintain its own index of what's\n> changed.  Watchman is built on inotify/fsevents -- it doesn't have\n> anything to do with any sort of storage device beyond a vanilla hard\n> drive.\n\nWhen you have such a massive repo, your clients aren't storing the data on their \nlocal drives, they are accessing the data on a network attached storage (via NFS \nor through a fuse mount). So they can have their watchman send queries to the \nstorage server to find out what has changed in this massive repo rather than \nhaving to walk the directory tree (or try to monitor it for changes on the \nclient machine)\n\nDavid Lang\n"},{"id":"241198","messageId":"1399661273.11843.133.camel@stross","threadId":"36571","inReplyTo":"alpine.DEB.2.02.1405091124150.5876@nftneq.ynat.uz","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-09T18:47:53Z","receivedAt":"2014-05-09T18:47:53Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2014-05-09 at 11:27 -0700, David Lang wrote:\n> >\n> > That's not my understanding from Durham Goode's talk in January.  Yes,\n> > operations involving history go to the server.  But the client also\n> > maintains a copy of the working tree, and it is for this that watchman\n> > is used.  Otherwise, why bother with watchman at all?  The server knows\n> > when it changes files and could simply maintain its own index of what's\n> > changed.  Watchman is built on inotify/fsevents -- it doesn't have\n> > anything to do with any sort of storage device beyond a vanilla hard\n> > drive.\n> \n> When you have such a massive repo, your clients aren't storing the data on their \n> local drives, they are accessing the data on a network attached storage (via NFS \n> or through a fuse mount). So they can have their watchman send queries to the \n> storage server to find out what has changed in this massive repo rather than \n> having to walk the directory tree (or try to monitor it for changes on the \n> client machine)\n\nYes, you could do that.  But I repeat: that is not what Facebook is\nactually doing.  If it were, they would have no need for inotify or\nFSEvents, since neither even works with NFS (and a FUSE-based solution\nwould need special support at which point it might as well just\nimplement watchman itself). \n\nPlease see this section of Durham Goode's talk for evidence that\nFacebook is not doing that:\nhttp://www.youtube.com/watch?v=Dlguc63cRXg&t=10m40s\n\n(it's at 10:40 in case that link doesn't work)\n\nMaybe things have changed since at Facebook since Durham gave that talk\nin January and you have knowledge of that. If so, say so.  But from what\nFacebook has said publicly, this is simply not what's going on.\n"},{"id":"241238","messageId":"CACsJy8B6AVOHH7HhreqvusQN=UFZzj1mkjqekrOb62Lmq_8VQw@mail.gmail.com","threadId":"36571","inReplyTo":"1399072451-15561-1-git-send-email-dturner@twopensource.com","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-10T05:26:36Z","receivedAt":"2014-05-10T05:26:36Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, May 3, 2014 at 6:14 AM,  <dturner@twopensource.com> wrote:\n> The most sigificant patch uses Facebook's watchman daemon[1] to monitor\n> the repository work tree for changes.  This makes allows git status\n> to avoid traversing the entire work tree to find changes.\n\nI got \"warning: Watchman watch error: Got bad JSON from watchman\nget-sockname: '[' or '{' expected near end of file\". Any ideas what I\ndid wrong? I'm using watchman.git and libwatchman.git. check-0.9.11\nand jansson-2.4 were installed by system (gentoo).\n-- \nDuy\n"},{"id":"241239","messageId":"CACsJy8DVVpfYEmE8pSZNyXy1m5WRkdm08deW3EXgAy_0Gn72zw@mail.gmail.com","threadId":"36571","inReplyTo":"1399072451-15561-1-git-send-email-dturner@twopensource.com","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-10T08:16:49Z","receivedAt":"2014-05-10T08:16:49Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, May 3, 2014 at 6:14 AM,  <dturner@twopensource.com> wrote:\n> The most sigificant patch uses Facebook's watchman daemon[1] to monitor\n> the repository work tree for changes.  This makes allows git status\n> to avoid traversing the entire work tree to find changes.\n\nSome comments on this series. I still haven't been able to run it with\nwatchman so there are many guesses from my side.\n\nFirst of all, when I set USE_WATCHMAN=Yes in config.mak I expect it to\nwork out of the box, provided that all dependencies are installed. I\nstill need to set WATCHMAN_LIBS for it to build because you only set\nit with configure script. Would be nice to have a default value for\nnon-configure people too.\n\nI'm not so happy that git now needs to link to libjansson.so and\nlibwatchman.so. I would load libwatchman.so at runtime only when\ncore.usewatchman is on, but this is more of personal taste.\n\nI still prefer my old tracking model, where the majority of lstat() is\ndone by refresh operation and we only need to optimize those lstat\ncalls, not every single lstat statement in the code base. With that in\nmind, I think you don't need to keep a fs cache on disk at all. All\nyou need to store (in the index) is the clock value from watchman.\nAfter we parse the index, we perform a \"since\" query to get path names\n(and perhaps \"exists\" and \"mode\" for later). Then we set CE_VALID bit\non entries that are _not_ in the query result. The remaining items\nwill be lstat'd by git (see [1] and read-cache.c changes onthe next\nfew patches). Assuming the number of updated files is reasonably\nsmall, we won't  be punished by lookup time. To avoid write time cost\ndue to possible mass CE_VALID change, assuming split-index is already\nused, we could store this bit separately as an extension and apply it\nback at read time.\n\nYour changes in dir.c. If I didn't misread it, you still do\nlast_exclude_matching when using fs-cache. That call is where all your\nCPU is spent and probably explains why you didn't see big perf gain\nwith watchman. I think my untracked cache has dealt correctly with\ncaching in this area. So when you do the watchman query earlier, you\nalso check if any of the directories are updated and force using\nuntracked cache for the rest of the directories.\n\nHope it helps,\nDuy\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/240339\n"},{"id":"241250","messageId":"1399747109.11843.137.camel@stross","threadId":"36571","inReplyTo":"CACsJy8B6AVOHH7HhreqvusQN=UFZzj1mkjqekrOb62Lmq_8VQw@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-10T18:38:29Z","receivedAt":"2014-05-10T18:38:29Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sat, 2014-05-10 at 12:26 +0700, Duy Nguyen wrote:\n> On Sat, May 3, 2014 at 6:14 AM,  <dturner@twopensource.com> wrote:\n> > The most sigificant patch uses Facebook's watchman daemon[1] to monitor\n> > the repository work tree for changes.  This makes allows git status\n> > to avoid traversing the entire work tree to find changes.\n> \n> I got \"warning: Watchman watch error: Got bad JSON from watchman\n> get-sockname: '[' or '{' expected near end of file\". Any ideas what I\n> did wrong? I'm using watchman.git and libwatchman.git. check-0.9.11\n> and jansson-2.4 were installed by system (gentoo).\n\nWhat do you get from watchman get-sockname on the command-line?  Do the\nwatchman tests pass?\n\nI don't know that I have tested with Jansson 2.4, but I've tried 2.2.1,\n2.5, and 2.6.  \n\nI also haven't tried on gentoo (I use Ubuntu).  But if you still have\nproblems, I can try installing gentoo in a vm and seeing if it's\nsomething gentoo-specific.\n"},{"id":"241254","messageId":"CACsJy8Cazm+6ixw3r8WYfdFYeD01Lmf0PSF0sdsh7PGy_6WDTQ@mail.gmail.com","threadId":"36571","inReplyTo":"1399747109.11843.137.camel@stross","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-11T00:21:11Z","receivedAt":"2014-05-11T00:21:11Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, May 11, 2014 at 1:38 AM, David Turner <dturner@twopensource.com> wrote:\n>> I got \"warning: Watchman watch error: Got bad JSON from watchman\n>> get-sockname: '[' or '{' expected near end of file\". Any ideas what I\n>> did wrong? I'm using watchman.git and libwatchman.git. check-0.9.11\n>> and jansson-2.4 were installed by system (gentoo).\n>\n> What do you get from watchman get-sockname on the command-line?  Do the\n> watchman tests pass?\n\nFound the problem. \"watchman\" binary is not in $PATH but popen() did\nnot complain (or it did but your \"2>/dev/null\" in watchman_connect\nsuppressed it). BTW you need to update the array size of \"expressions\"\nin test_watchman_misc().\n\nSo without watchman I got\n\n   299.871ms read_index_from:1538 if (verify_hdr(hdr, mmap_size) < 0) go\n   498.205ms cmd_status:1300 refresh_index(&the_index, REFRESH_QUIE\n   796.050ms wt_status_collect:622 wt_status_collect_untracked(s)\n\nand with watchman (\"git status\" ran several times to make sure it's cached)\n\n   301.950ms read_index_from:1538 if (verify_hdr(hdr, mmap_size) < 0) go\n    34.918ms  read_fs_cache:347 if (verify_hdr(hdr, mmap_size) < 0) go\n  1564.096ms  watchman_load_fs_cache:628 update_fs_cache(istate, result);\n   161.930ms cmd_status:1300 refresh_index(&the_index, REFRESH_QUIE\n   251.614ms wt_status_collect:622 wt_status_collect_untracked(s)\n\nGiven the total time of \"git status\" without watchman is 1.9s,,\nupdate_fs_cache() nearly matches that number alone. All that is spent\nin the exclude update code in the function, but if you do\nlast_excluding_matching() anyway, why cache it?\n\nI think we're paying lookup cost in refresh_index(). I forced CE_VALID\nbit on as an experiment and refresh_index() went down to 33ms.\n\nA bit surprised about wt_status_collect_untracked number. I verified\nthat last_excluding_matching() still runs (on the same number of\nentries like in no-watchman case). Replacing fs_cache_open() in\nadd_excludes_from_file_to_list() to plain open() does not change the\nnumber, so we probably won't need that (unless your worktree is filled\nwith .gitignore, which I doubt it's a norm). So about 500ms saved is\nin opendir/readdir.. A simple/separate opendir/readdir/closedir\nprogram confirms that (ugh!).\n-- \nDuy\n"},{"id":"241269","messageId":"1399848982.11843.161.camel@stross","threadId":"36571","inReplyTo":"CACsJy8Cazm+6ixw3r8WYfdFYeD01Lmf0PSF0sdsh7PGy_6WDTQ@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-11T22:56:22Z","receivedAt":"2014-05-11T22:56:22Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sun, 2014-05-11 at 07:21 +0700, Duy Nguyen wrote:\n> On Sun, May 11, 2014 at 1:38 AM, David Turner <dturner@twopensource.com> wrote:\n> >> I got \"warning: Watchman watch error: Got bad JSON from watchman\n> >> get-sockname: '[' or '{' expected near end of file\". Any ideas what I\n> >> did wrong? I'm using watchman.git and libwatchman.git. check-0.9.11\n> >> and jansson-2.4 were installed by system (gentoo).\n> >\n> > What do you get from watchman get-sockname on the command-line?  Do the\n> > watchman tests pass?\n> \n> Found the problem. \"watchman\" binary is not in $PATH but popen() did\n> not complain (or it did but your \"2>/dev/null\" in watchman_connect\n> suppressed it). \n\nI should probably not be using popen, since it doesn't offer good error\nreporting.  I'll try to fix that in the next few days.\n\n> BTW you need to update the array size of \"expressions\"\n> in test_watchman_misc().\n\nThanks, fixed.\n\n> So without watchman I got\n> \n>    299.871ms read_index_from:1538 if (verify_hdr(hdr, mmap_size) < 0) go\n>    498.205ms cmd_status:1300 refresh_index(&the_index, REFRESH_QUIE\n>    796.050ms wt_status_collect:622 wt_status_collect_untracked(s)\n> \n> and with watchman (\"git status\" ran several times to make sure it's cached)\n> \n>    301.950ms read_index_from:1538 if (verify_hdr(hdr, mmap_size) < 0) go\n>     34.918ms  read_fs_cache:347 if (verify_hdr(hdr, mmap_size) < 0) go\n>   1564.096ms  watchman_load_fs_cache:628 update_fs_cache(istate, result);\n>    161.930ms cmd_status:1300 refresh_index(&the_index, REFRESH_QUIE\n>    251.614ms wt_status_collect:622 wt_status_collect_untracked(s)\n> \n> Given the total time of \"git status\" without watchman is 1.9s,,\n> update_fs_cache() nearly matches that number alone. All that is spent\n> in the exclude update code in the function, but if you do\n> last_excluding_matching() anyway, why cache it?\n\nMy numbers are different (for my test repository):\n\n---\n    30.031ms read_index:1386 r = read_index_from(istate, get_index_\n    71.625ms cmd_status:1302 refresh_index(&the_index, REFRESH_QUIE\n   259.712ms wt_status_collect:622 wt_status_collect_untracked(s)\n----\n    41.110ms read_index:1386 r = read_index_from(istate, get_index_\n     9.294ms read_fs_cache:347 if (verify_hdr(hdr, mmap_size) < 0) go\n     0.173ms watchman_load_fs_cache:628 update_fs_cache(istate, result)\n    41.901ms read_index:1386 r = read_index_from(istate, get_index_\n    18.355ms cmd_status:1302 refresh_index(&the_index, REFRESH_QUIE\n    50.911ms wt_status_collect:622 wt_status_collect_untracked(s)\n---\n\nI think something must be going wrong with update_fs_cache on your\nmachine.  I have a few hypotheses:\n\n1. Maybe watchman isn't fully started up when you run your tests.\n2. Maybe there is a bug.\n\nupdate_fs_cache should only have to update based on what it has learned\nfrom watchman.  So if no .gitignore has been changed, it should not have\nto do very much work.\n\nI could take the fe_excluded check and move it above the\nlast_exclude_matching check in fs_cache_is_excluded; it causes t7300 to\nfail when run under watchman but presumably that's fixable\n\n> I think we're paying lookup cost in refresh_index(). I forced CE_VALID\n> bit on as an experiment and refresh_index() went down to 33ms.\n\nYes, it is possible to use Watchman to set CE_VALID as well, and I\nshould probably do that.  It's a bit more complicated (at least, I\nrecall running into problems), but probably doable.\n\n> A bit surprised about wt_status_collect_untracked number. I verified\n> that last_excluding_matching() still runs (on the same number of\n> entries like in no-watchman case). Replacing fs_cache_open() in\n> add_excludes_from_file_to_list() to plain open() does not change the\n> number, so we probably won't need that (unless your worktree is filled\n> with .gitignore, which I doubt it's a norm). \n\nMy test repo has a couple hundred of them.  Maybe that's unusual?  A\nrepo with a lot of projects will tend to have lots of gitignore files,\nbecause each project will want to maintain them independently.\n"},{"id":"241282","messageId":"CACsJy8C_j2bKVwqOQtOqGFkc_-_AmY=bQXquRfL-aqk=z9YKdw@mail.gmail.com","threadId":"36571","inReplyTo":"1399848982.11843.161.camel@stross","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-12T10:45:42Z","receivedAt":"2014-05-12T10:45:42Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, May 12, 2014 at 5:56 AM, David Turner <dturner@twopensource.com> wrote:\n>> So without watchman I got\n>>\n>>    299.871ms read_index_from:1538 if (verify_hdr(hdr, mmap_size) < 0) go\n>>    498.205ms cmd_status:1300 refresh_index(&the_index, REFRESH_QUIE\n>>    796.050ms wt_status_collect:622 wt_status_collect_untracked(s)\n>>\n>> and with watchman (\"git status\" ran several times to make sure it's cached)\n>>\n>>    301.950ms read_index_from:1538 if (verify_hdr(hdr, mmap_size) < 0) go\n>>     34.918ms  read_fs_cache:347 if (verify_hdr(hdr, mmap_size) < 0) go\n>>   1564.096ms  watchman_load_fs_cache:628 update_fs_cache(istate, result);\n>>    161.930ms cmd_status:1300 refresh_index(&the_index, REFRESH_QUIE\n>>    251.614ms wt_status_collect:622 wt_status_collect_untracked(s)\n>>\n>> Given the total time of \"git status\" without watchman is 1.9s,,\n>> update_fs_cache() nearly matches that number alone. All that is spent\n>> in the exclude update code in the function, but if you do\n>> last_excluding_matching() anyway, why cache it?\n>\n> My numbers are different (for my test repository):\n>\n> ---\n>     30.031ms read_index:1386 r = read_index_from(istate, get_index_\n>     71.625ms cmd_status:1302 refresh_index(&the_index, REFRESH_QUIE\n>    259.712ms wt_status_collect:622 wt_status_collect_untracked(s)\n> ----\n>     41.110ms read_index:1386 r = read_index_from(istate, get_index_\n>      9.294ms read_fs_cache:347 if (verify_hdr(hdr, mmap_size) < 0) go\n>      0.173ms watchman_load_fs_cache:628 update_fs_cache(istate, result)\n>     41.901ms read_index:1386 r = read_index_from(istate, get_index_\n>     18.355ms cmd_status:1302 refresh_index(&the_index, REFRESH_QUIE\n>     50.911ms wt_status_collect:622 wt_status_collect_untracked(s)\n> ---\n>\n> I think something must be going wrong with update_fs_cache on your\n> machine.  I have a few hypotheses:\n>\n> 1. Maybe watchman isn't fully started up when you run your tests.\n> 2. Maybe there is a bug.\n\nIt's probably me doing something wrong (I ran it more than a couple\ntimes so watchman must have loaded the whole thing). I got small\nnumbers in update_fs_cache() now.\n\n>> A bit surprised about wt_status_collect_untracked number. I verified\n>> that last_excluding_matching() still runs (on the same number of\n>> entries like in no-watchman case). Replacing fs_cache_open() in\n>> add_excludes_from_file_to_list() to plain open() does not change the\n>> number, so we probably won't need that (unless your worktree is filled\n>> with .gitignore, which I doubt it's a norm).\n>\n> My test repo has a couple hundred of them.  Maybe that's unusual?  A\n> repo with a lot of projects will tend to have lots of gitignore files,\n> because each project will want to maintain them independently.\n\nI tried the worst case, every directory had an empty .gitignore. The\nnumbers did not change much. And I found out that because\nadd_excludes.. were called twice, not on every .gitignore because of\nthe condition \"!(dir->flags & DIR_EXCLUDE_CMDL_ONLY)\". So the number\nof .gitignore does not matter (yet).\n\nThis is your quote from above, moved down a bit:\n\n> update_fs_cache should only have to update based on what it has learned\n> from watchman.  So if no .gitignore has been changed, it should not have\n> to do very much work.\n>\n> I could take the fe_excluded check and move it above the\n> last_exclude_matching check in fs_cache_is_excluded; it causes t7300 to\n> fail when run under watchman but presumably that's fixable\n\nSo you exclude files early and make the real read_directory() pass do\npretty much nothing. This is probably not a good idea. Assume that I\ntouch $TOP/.gitignore then do something other than \"git status\" (or\n\"git add\") then I have to pay read_directory() cost.\n\nBack to the open vs fs_cache_open and the number of .gitignore files\nabove. I touch $TOP/.gitignore then do \"git status\" to make it read\nall .gitignore files (6k of them) and change between open and\nfs_cache_open. I think the numbers still  do not make any visible\ndifference (~1620-1630ms).\n-- \nDuy\n"},{"id":"241461","messageId":"1400020707.14179.13.camel@stross","threadId":"36571","inReplyTo":"CACsJy8C_j2bKVwqOQtOqGFkc_-_AmY=bQXquRfL-aqk=z9YKdw@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-13T22:38:27Z","receivedAt":"2014-05-13T22:38:27Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2014-05-12 at 17:45 +0700, Duy Nguyen wrote:\n> This is your quote from above, moved down a bit:\n> \n> > update_fs_cache should only have to update based on what it has learned\n> > from watchman.  So if no .gitignore has been changed, it should not have\n> > to do very much work.\n> >\n> > I could take the fe_excluded check and move it above the\n> > last_exclude_matching check in fs_cache_is_excluded; it causes t7300 to\n> > fail when run under watchman but presumably that's fixable\n> \n> So you exclude files early and make the real read_directory() pass do\n> pretty much nothing. This is probably not a good idea. Assume that I\n> touch $TOP/.gitignore then do something other than \"git status\" (or\n> \"git add\") then I have to pay read_directory() cost.\n\nI'm not sure I understand this. read_directory does something: it checks\nthe fs_cache (instead of the filesystem) for untracked files.\n\n> Back to the open vs fs_cache_open and the number of .gitignore files\n> above. I touch $TOP/.gitignore then do \"git status\" to make it read\n> all .gitignore files (6k of them) and change between open and\n> fs_cache_open. I think the numbers still  do not make any visible\n> difference (~1620-1630ms).\n\nYes, I would expect no win in that case.  fs_cache_open will only save\ntime in the common case where there is no .gitignore file, because it\nsaves an open() call.  If every possible .gitignore file exists, of\ncourse it makes no difference.  But also, your processor may be\nsufficiently slow that the context-switch penalty for open() is less\nthan the hash table lookup.  \n\nFor me, the win from fs_cache_open is about 7%.  \n"},{"id":"241466","messageId":"CACsJy8CVK1codmuSvUyeVYo1d-nQ6thoV8pn1WN_8D8doYXu5g@mail.gmail.com","threadId":"36571","inReplyTo":"1400020707.14179.13.camel@stross","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T22:54:40Z","receivedAt":"2014-05-13T22:54:40Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, May 14, 2014 at 5:38 AM, David Turner <dturner@twopensource.com> wrote:\n> On Mon, 2014-05-12 at 17:45 +0700, Duy Nguyen wrote:\n>> This is your quote from above, moved down a bit:\n>>\n>> > update_fs_cache should only have to update based on what it has learned\n>> > from watchman.  So if no .gitignore has been changed, it should not have\n>> > to do very much work.\n>> >\n>> > I could take the fe_excluded check and move it above the\n>> > last_exclude_matching check in fs_cache_is_excluded; it causes t7300 to\n>> > fail when run under watchman but presumably that's fixable\n>>\n>> So you exclude files early and make the real read_directory() pass do\n>> pretty much nothing. This is probably not a good idea. Assume that I\n>> touch $TOP/.gitignore then do something other than \"git status\" (or\n>> \"git add\") then I have to pay read_directory() cost.\n>\n> I'm not sure I understand this. read_directory does something: it checks\n> the fs_cache (instead of the filesystem) for untracked files.\n\nA lot of commands do read_cache() that that eventually calls\nupdate_fs_cache, which does part of read_directory's work (the\nfe_excluded thing). But not many of those commands actually call\nread_directory(). It'd better if there's a way to mark that \"this\n.gitignore is changed\", but delay the actual exclude processsing until\nwe are sure read_directory() will be used.\n-- \nDuy\n"},{"id":"241472","messageId":"1400023182.14179.23.camel@stross","threadId":"36571","inReplyTo":"CACsJy8CVK1codmuSvUyeVYo1d-nQ6thoV8pn1WN_8D8doYXu5g@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-13T23:19:42Z","receivedAt":"2014-05-13T23:19:42Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2014-05-14 at 05:54 +0700, Duy Nguyen wrote:\n> On Wed, May 14, 2014 at 5:38 AM, David Turner <dturner@twopensource.com> wrote:\n> > On Mon, 2014-05-12 at 17:45 +0700, Duy Nguyen wrote:\n> >> This is your quote from above, moved down a bit:\n> >>\n> >> > update_fs_cache should only have to update based on what it has learned\n> >> > from watchman.  So if no .gitignore has been changed, it should not have\n> >> > to do very much work.\n> >> >\n> >> > I could take the fe_excluded check and move it above the\n> >> > last_exclude_matching check in fs_cache_is_excluded; it causes t7300 to\n> >> > fail when run under watchman but presumably that's fixable\n> >>\n> >> So you exclude files early and make the real read_directory() pass do\n> >> pretty much nothing. This is probably not a good idea. Assume that I\n> >> touch $TOP/.gitignore then do something other than \"git status\" (or\n> >> \"git add\") then I have to pay read_directory() cost.\n> >\n> > I'm not sure I understand this. read_directory does something: it checks\n> > the fs_cache (instead of the filesystem) for untracked files.\n> \n> A lot of commands do read_cache() that that eventually calls\n> update_fs_cache, which does part of read_directory's work (the\n> fe_excluded thing). But not many of those commands actually call\n> read_directory(). It'd better if there's a way to mark that \"this\n> .gitignore is changed\", but delay the actual exclude processsing until\n> we are sure read_directory() will be used.\n\nOK, that would be straighforward, I think.\n"},{"id":"241475","messageId":"1400024691.14179.40.camel@stross","threadId":"36571","inReplyTo":"CACsJy8DVVpfYEmE8pSZNyXy1m5WRkdm08deW3EXgAy_0Gn72zw@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-13T23:44:51Z","receivedAt":"2014-05-13T23:44:51Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sat, 2014-05-10 at 15:16 +0700, Duy Nguyen wrote:\n> On Sat, May 3, 2014 at 6:14 AM,  <dturner@twopensource.com> wrote:\n> > The most sigificant patch uses Facebook's watchman daemon[1] to monitor\n> > the repository work tree for changes.  This makes allows git status\n> > to avoid traversing the entire work tree to find changes.\n> \n> Some comments on this series. I still haven't been able to run it with\n> watchman so there are many guesses from my side.\n> \n> First of all, when I set USE_WATCHMAN=Yes in config.mak I expect it to\n> work out of the box, provided that all dependencies are installed. I\n> still need to set WATCHMAN_LIBS for it to build because you only set\n> it with configure script. Would be nice to have a default value for\n> non-configure people too.\n\nI'll fix that, thanks.\n\n> I'm not so happy that git now needs to link to libjansson.so and\n> libwatchman.so. I would load libwatchman.so at runtime only when\n> core.usewatchman is on, but this is more of personal taste.\n\nI assume you mean something with dlopen? I don't really like that because\n(a) nothing else in git works that way and\n(b) you would still need the libwatchman headers to build git (or the\nstructs could be copied, but that is ugly).\n\n> I still prefer my old tracking model, where the majority of lstat() is\n> done by refresh operation and we only need to optimize those lstat\n> calls, not every single lstat statement in the code base.\n\nThe lstat calls are only one of the problems.  The others are\nopendir/readdir and open(.gitignore). \n\n>  With that in\n> mind, I think you don't need to keep a fs cache on disk at all. All\n> you need to store (in the index) is the clock value from watchman.\n> After we parse the index, we perform a \"since\" query to get path names\n> (and perhaps \"exists\" and \"mode\" for later). Then we set CE_VALID bit\n> on entries that are _not_ in the query result. The remaining items\n> will be lstat'd by git (see [1] and read-cache.c changes on the next\n> few patches). Assuming the number of updated files is reasonably\n> small, we won't  be punished by lookup time. \n\nI considered this, but rejected it for a few reasons:\n1. We still need to know about the untracked files.  I guess we could \ndo an index extension for just that, like your untracked cache.\n\n2. That doesn't eliminate opendir/readdir, I think.  Those are a major \ncost. I did not thoroughly review your patches from January/February, \nbut I didn't notice anything in there about this part of dir.c.  \nAlso the cost of open(nonexistent .gitignore) to do ignore processing.\n\n3. Changing a file in the index (e.g. git add) requires clearing\nthe CE_VALID bit; this means that third-party libraries (e.g. jgit) \nthat change the git repo need to understand this extension for correct\noperation.  Maybe that's just the nature of extensions, but it's not\nsomething that my present patch set requires.\n"},{"id":"241493","messageId":"CACsJy8C49EDwjtv_L2pTRy3XbPptp7+nNzT=Jp4BaH_TOZtvnQ@mail.gmail.com","threadId":"36571","inReplyTo":"1400024691.14179.40.camel@stross","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-14T10:36:40Z","receivedAt":"2014-05-14T10:36:40Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, May 14, 2014 at 6:44 AM, David Turner <dturner@twopensource.com> wrote:\n>> I'm not so happy that git now needs to link to libjansson.so and\n>> libwatchman.so. I would load libwatchman.so at runtime only when\n>> core.usewatchman is on, but this is more of personal taste.\n>\n> I assume you mean something with dlopen? I don't really like that because\n> (a) nothing else in git works that way and\n> (b) you would still need the libwatchman headers to build git (or the\n> structs could be copied, but that is ugly).\n\nAnd because we don't link to libdl.so anyway, using dlopen means\ntrading libjansson.so and libwatchman.so for libdl.so. And it's\nuglier. So forget what I wrote.\n\n>>  With that in\n>> mind, I think you don't need to keep a fs cache on disk at all. All\n>> you need to store (in the index) is the clock value from watchman.\n>> After we parse the index, we perform a \"since\" query to get path names\n>> (and perhaps \"exists\" and \"mode\" for later). Then we set CE_VALID bit\n>> on entries that are _not_ in the query result. The remaining items\n>> will be lstat'd by git (see [1] and read-cache.c changes on the next\n>> few patches). Assuming the number of updated files is reasonably\n>> small, we won't  be punished by lookup time.\n>\n> I considered this, but rejected it for a few reasons:\n> 1. We still need to know about the untracked files.  I guess we could\n> do an index extension for just that, like your untracked cache.\n\nYes. But consider that the number of untracked files is usually small\n(otherwise 'git status' would look very messy). And your fscache would\nneed to store excluded file list too, which could be a lot bigger (one\npair of  .[ch] -> one .o). _But_ yours would make 'git status\n--ignored' work. I don't consider that a major use case for\noptimization, but people may have different opinions.\n\n> 2. That doesn't eliminate opendir/readdir, I think.  Those are a major\n> cost. I did not thoroughly review your patches from January/February,\n> but I didn't notice anything in there about this part of dir.c.\n> Also the cost of open(nonexistent .gitignore) to do ignore processing.\n\nAssuming untracked cache is in use, opendir/readdir is replaced with\nlstat. And cheap lstat can be solved with watchman without storing\nanything extra. I solve open(.gitignore) too by checking its stat data\nwith the one in index. If matches, I reuse the version in tree. This\ndoes not necessarily make it cheaper, but it increases locality so it\nmight be. _Modified_ .gitignore files have to go through\nopen(.gitignore), but people don't update .gitignore often.\n\n> 3. Changing a file in the index (e.g. git add) requires clearing\n> the CE_VALID bit; this means that third-party libraries (e.g. jgit)\n> that change the git repo need to understand this extension for correct\n> operation.  Maybe that's just the nature of extensions, but it's not\n> something that my present patch set requires.\n\nI don't store CE_VALID on disk. Instead I store a new bit CE_WATCHED.\nOnly after loading and validating against watchman that I turn\nCE_WATCHED to CE_VALID in memory. So JGit, libgit2 are not confused.\n\nI assume you won't change your mind about this. Which is fine to me. I\nwill still try out my approach with your libwatchman though. Just\ncurious about its performance and complexity, compared to your\napproach.\n\nA bit off topic, but msys fork has another \"fscache\" in compat/win32.\nIf you could come up with a different name, maybe it'll be less\nconfusing for them after merging. But this is not a big deal, as this\nfscache won't work on windows anyway.\n-- \nDuy\n"},{"id":"241494","messageId":"CACsJy8CDGP1QwX4C3yKVUOw3h90EcLSXXXy=t6XFSVOrVjxA4A@mail.gmail.com","threadId":"36571","inReplyTo":"CACsJy8C49EDwjtv_L2pTRy3XbPptp7+nNzT=Jp4BaH_TOZtvnQ@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-14T10:52:12Z","receivedAt":"2014-05-14T10:52:12Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"One more thing I think we can solve with watchman is the racy\ntimestamp issue (see 29e4d36 and 407c8eb). Basically if a file is\nupdated within a second (and the system only supports timestamp\ngranuarity down to a second) then git can't know if a file is changed\nby comparing stat data, so it compares content anyway. With watchman,\nI think we know that a file is updated or not and can disable racy\ntest. How often this happens and whether it is worth supporting is\nsomething to think about.\n-- \nDuy\n"},{"id":"241775","messageId":"1400182971.14179.49.camel@stross","threadId":"36571","inReplyTo":"CACsJy8C49EDwjtv_L2pTRy3XbPptp7+nNzT=Jp4BaH_TOZtvnQ@mail.gmail.com","subject":"Re: Watchman support for git","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-15T19:42:51Z","receivedAt":"2014-05-15T19:42:51Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2014-05-14 at 17:36 +0700, Duy Nguyen wrote:\n> >>  With that in\n> >> mind, I think you don't need to keep a fs cache on disk at all. All\n> >> you need to store (in the index) is the clock value from watchman.\n> >> After we parse the index, we perform a \"since\" query to get path names\n> >> (and perhaps \"exists\" and \"mode\" for later). Then we set CE_VALID bit\n> >> on entries that are _not_ in the query result. The remaining items\n> >> will be lstat'd by git (see [1] and read-cache.c changes on the next\n> >> few patches). Assuming the number of updated files is reasonably\n> >> small, we won't  be punished by lookup time.\n> >\n> > I considered this, but rejected it for a few reasons:\n> > 1. We still need to know about the untracked files.  I guess we could\n> > do an index extension for just that, like your untracked cache.\n> \n> Yes. But consider that the number of untracked files is usually small\n> (otherwise 'git status' would look very messy). And your fscache would\n> need to store excluded file list too, which could be a lot bigger (one\n> pair of  .[ch] -> one .o). _But_ yours would make 'git status\n> --ignored' work. I don't consider that a major use case for\n> optimization, but people may have different opinions.\n\nI don't think --ignored is a major use case.  But I do think it's nice\nto get as much mileage as possible out of a change like this.  \n\n> > 2. That doesn't eliminate opendir/readdir, I think.  Those are a major\n> > cost. I did not thoroughly review your patches from January/February,\n> > but I didn't notice anything in there about this part of dir.c.\n> > Also the cost of open(nonexistent .gitignore) to do ignore processing.\n> \n> Assuming untracked cache is in use, opendir/readdir is replaced with\n> lstat. And cheap lstat can be solved with watchman without storing\n> anything extra. I solve open(.gitignore) too by checking its stat data\n> with the one in index. If matches, I reuse the version in tree. This\n> does not necessarily make it cheaper, but it increases locality so it\n> might be. _Modified_ .gitignore files have to go through\n> open(.gitignore), but people don't update .gitignore often.\n\nInteresting -- if all that actually works, then it's probably the right\napproach.  \n\n> > 3. Changing a file in the index (e.g. git add) requires clearing\n> > the CE_VALID bit; this means that third-party libraries (e.g. jgit)\n> > that change the git repo need to understand this extension for correct\n> > operation.  Maybe that's just the nature of extensions, but it's not\n> > something that my present patch set requires.\n> \n> I don't store CE_VALID on disk. Instead I store a new bit CE_WATCHED.\n> Only after loading and validating against watchman that I turn\n> CE_WATCHED to CE_VALID in memory. So JGit, libgit2 are not confused.\n> \n> I assume you won't change your mind about this. Which is fine to me. I\n> will still try out my approach with your libwatchman though. Just\n> curious about its performance and complexity, compared to your\n> approach.\n\nI am open-minded here. This code is really the first time I have looked\nat git's internals, and I accept that your way might be better.  If\nyou're going to try the watchman version of your approach, then we do a\ndirect comparison.  Let me know if there is something I can do to help\nout on that.\n\n> A bit off topic, but msys fork has another \"fscache\" in compat/win32.\n> If you could come up with a different name, maybe it'll be less\n> confusing for them after merging. But this is not a big deal, as this\n> fscache won't work on windows anyway.\n\nDoes wtcache sounds like a better name?\n"},{"id":"242153","messageId":"CACsJy8C5xvs1ouO=1P8i61i1QCWNu67NAebFpC8F3M91qq2AVw@mail.gmail.com","threadId":"36571","inReplyTo":"1400182971.14179.49.camel@stross","subject":"Re: Watchman support for git","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-19T10:10:38Z","receivedAt":"2014-05-19T10:10:38Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, May 16, 2014 at 2:42 AM, David Turner <dturner@twopensource.com> wrote:\n>> I assume you won't change your mind about this. Which is fine to me. I\n>> will still try out my approach with your libwatchman though. Just\n>> curious about its performance and complexity, compared to your\n>> approach.\n>\n> I am open-minded here. This code is really the first time I have looked\n> at git's internals, and I accept that your way might be better.  If\n> you're going to try the watchman version of your approach, then we do a\n> direct comparison.  Let me know if there is something I can do to help\n> out on that.\n\nYou already helped by publishing your patches (and letting me know\nabout libwatchman) so I can steal bits here and there ;-)\n\n>> A bit off topic, but msys fork has another \"fscache\" in compat/win32.\n>> If you could come up with a different name, maybe it'll be less\n>> confusing for them after merging. But this is not a big deal, as this\n>> fscache won't work on windows anyway.\n>\n> Does wtcache sounds like a better name?\n>\n\nHeh i'm bad at naming. But that sounds a bit cryptic. Maybe\nwatchman-cache.c (with funtions starting with prefix wmc_)? Should not\nworry too much about this though, unless some msysgit guys yell up.\n-- \nDuy\n"}]}