{"thread":{"id":"40692","subject":"[PATCH 0/5] Use watchman to reduce index refresh time","startedAt":"2015-11-01T13:55:41Z","lastAt":"2015-11-20T09:45:21Z","messageCount":17,"participants":["Nguyễn Thái Ngọc Duy","Paolo Ciarrocchi","Duy Nguyen","David Turner","Christian Couder"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"272666","messageId":"1446386146-10438-1-git-send-email-pclouds@gmail.com","threadId":"40692","inReplyTo":null,"subject":"[PATCH 0/5] Use watchman to reduce index refresh time","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-01T13:55:41Z","receivedAt":"2015-11-01T13:55:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This series builds on top of the index-helper series I just sent and\nuses watchman to keep track of file changes in order to avoid lstat()\nat refresh time. The series can also be found at [1]\n\nWhen I started this work, watchman did not support Windows yet. It\ndoes now, even if still experimental [2]. So Windows people, please\ntry it out if you have time.\n\nTo put all pieces so far together, we have split-index to reduce index\nwrite time, untracked cache to reduce I/O as well as computation for\n.gitignore, index-helper for index read time and this series for\nlstat() at refresh time. The remaining piece is killing lstat() from\nuntracked cache, but right now it's just some idea and incomplete\ncode.\n\n[1] https://github.com/pclouds/git/commits/refresh-with-watchman\n[2] https://github.com/facebook/watchman/issues/19\n\nNguyễn Thái Ngọc Duy (5):\n  read-cache: add watchman 'WAMA' extension\n  Add watchman support to reduce index refresh cost\n  read-cache: allow index-helper to prepare shm before git reads it\n  index-helper: use watchman to avoid refreshing index with lstat()\n  update-index: enable/disable watchman support\n\n Makefile                 |  12 +++++\n builtin/update-index.c   |  11 +++++\n cache.h                  |   7 +++\n config.c                 |   5 ++\n configure.ac             |   8 +++\n environment.c            |   3 ++\n index-helper.c           |  84 +++++++++++++++++++++++++++++--\n read-cache.c             | 126 ++++++++++++++++++++++++++++++++++++++++++++---\n watchman-support.c (new) | 108 ++++++++++++++++++++++++++++++++++++++++\n watchman-support.h (new) |   7 +++\n 10 files changed, 361 insertions(+), 10 deletions(-)\n create mode 100644 watchman-support.c\n create mode 100644 watchman-support.h\n\n-- \n2.2.0.513.g477eb31\n"},{"id":"272667","messageId":"1446386146-10438-2-git-send-email-pclouds@gmail.com","threadId":"40692","inReplyTo":"1446386146-10438-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 1/5] read-cache: add watchman 'WAMA' extension","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-01T13:55:42Z","receivedAt":"2015-11-01T13:55:42Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The extension contains a bitmap, one bit for each entry in the\nindex. If the n-th bit is zero, the n-th entry is considered\nunchanged, we can ce_mark_uptodate() it without refreshing. If the bit\nis non-zero and we found out the corresponding file is clean after\nrefresh, we can clear the bit.\n\nThe 'skipping refresh' bit is not in this patch yet as we would need\nwatchman. More details in later patches.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n cache.h      |  4 ++++\n read-cache.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 73 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 9633acc..a05fd31 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -163,6 +163,8 @@ struct cache_entry {\n #define CE_VALID     (0x8000)\n #define CE_STAGESHIFT 12\n \n+#define CE_NO_WATCH  (0x0001)\n+\n /*\n  * Range 0xFFFF0FFF in ce_flags is divided into\n  * two parts: in-memory flags and on-disk ones.\n@@ -298,6 +300,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n #define UNTRACKED_CHANGED\t(1 << 7)\n+#define WATCHMAN_CHANGED\t(1 << 8)\n \n struct split_index;\n struct untracked_cache;\n@@ -322,6 +325,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tvoid *mmap;\n \tsize_t mmap_size;\n+\tchar *last_update;\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex f609776..893223e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -19,6 +19,7 @@\n #include \"split-index.h\"\n #include \"utf8.h\"\n #include \"shm.h\"\n+#include \"ewah/ewok.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t       unsigned int options);\n@@ -41,11 +42,13 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t  /* \"link\" */\n #define CACHE_EXT_UNTRACKED 0x554E5452\t  /* \"UNTR\" */\n+#define CACHE_EXT_WATCHMAN 0x57414D41\t  /* \"WAMA\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | \\\n+\t\t WATCHMAN_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -1224,8 +1227,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\tcontinue;\n \n \t\tnew = refresh_cache_ent(istate, ce, options, &cache_errno, &changed);\n-\t\tif (new == ce)\n+\t\tif (new == ce) {\n+\t\t\tif (ce->ce_flags & CE_NO_WATCH) {\n+\t\t\t\tce->ce_flags          &= ~CE_NO_WATCH;\n+\t\t\t\tistate->cache_changed |= WATCHMAN_CHANGED;\n+\t\t\t}\n \t\t\tcontinue;\n+\t\t}\n \t\tif (!new) {\n \t\t\tconst char *fmt;\n \n@@ -1369,6 +1377,48 @@ static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n \treturn 0;\n }\n \n+static void mark_no_watchman(size_t pos, void *data)\n+{\n+\tstruct index_state *istate = data;\n+\tassert(pos < istate->cache_nr);\n+\tistate->cache[pos]->ce_flags |= CE_NO_WATCH;\n+}\n+\n+static int read_watchman_ext(struct index_state *istate, const void *data,\n+\t\t\t     unsigned long sz)\n+{\n+\tstruct ewah_bitmap *bitmap;\n+\tint ret, len;\n+\n+\tif (memchr(data, 0, sz) == NULL)\n+\t\treturn error(\"invalid extension\");\n+\tlen = strlen(data) + 1;\n+\tbitmap = ewah_new();\n+\tret = ewah_read_mmap(bitmap, (const char *)data + len, sz - len);\n+\tif (ret != sz - len) {\n+\t\tewah_free(bitmap);\n+\t\treturn error(\"fail to parse ewah bitmap\");\n+\t}\n+\tistate->last_update = xstrdup(data);\n+\tewah_each_bit(bitmap, mark_no_watchman, istate);\n+\tewah_free(bitmap);\n+\treturn 0;\n+}\n+\n+static void write_watchman_ext(struct strbuf *sb, struct index_state* istate)\n+{\n+\tstruct ewah_bitmap *bitmap;\n+\tint i;\n+\n+\tstrbuf_add(sb, istate->last_update, strlen(istate->last_update) + 1);\n+\tbitmap = ewah_new();\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_NO_WATCH)\n+\t\t\tewah_set(bitmap, i);\n+\tewah_serialize_strbuf(bitmap, sb);\n+\tewah_free(bitmap);\n+}\n+\n static int read_index_extension(struct index_state *istate,\n \t\t\t\tconst char *ext, void *data, unsigned long sz)\n {\n@@ -1386,6 +1436,11 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_UNTRACKED:\n \t\tistate->untracked = read_untracked_extension(data, sz);\n \t\tbreak;\n+\n+\tcase CACHE_EXT_WATCHMAN:\n+\t\tread_watchman_ext(istate, data, sz);\n+\t\tbreak;\n+\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1794,6 +1849,8 @@ int discard_index(struct index_state *istate)\n \tistate->untracked = NULL;\n \tistate->from_shm = 0;\n \tistate->to_shm   = 0;\n+\tfree(istate->last_update);\n+\tistate->last_update = NULL;\n \treturn 0;\n }\n \n@@ -2191,6 +2248,16 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->last_update) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_watchman_ext(&sb, istate);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCHMAN, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st))\n \t\treturn -1;\n-- \n2.2.0.513.g477eb31\n"},{"id":"272668","messageId":"1446386146-10438-3-git-send-email-pclouds@gmail.com","threadId":"40692","inReplyTo":"1446386146-10438-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/5] Add watchman support to reduce index refresh cost","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-01T13:55:43Z","receivedAt":"2015-11-01T13:55:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The previous patch has the logic to clear bits in 'WAMA' bitmap. This\npatch has logic to set bits as told by watchman. The missing bit,\n_using_ these bits, are not here yet.\n\nA lot of this code is written by David Turner originally, mostly from\n[1]. I'm just copying and polishing it a bit.\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/248006\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Makefile                 |   7 +++\n cache.h                  |   1 +\n config.c                 |   5 +++\n configure.ac             |   8 ++++\n environment.c            |   3 ++\n watchman-support.c (new) | 108 +++++++++++++++++++++++++++++++++++++++++++++++\n watchman-support.h (new) |   8 ++++\n 7 files changed, 140 insertions(+)\n create mode 100644 watchman-support.c\n create mode 100644 watchman-support.h\n\ndiff --git a/Makefile b/Makefile\nindex c01cd2e..761acb6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1389,6 +1389,12 @@ else\n \tLIB_OBJS += thread-utils.o\n endif\n \n+ifdef USE_WATCHMAN\n+\tLIB_H += watchman-support.h\n+\tLIB_OBJS += watchman-support.o\n+\tBASIC_CFLAGS += -DUSE_WATCHMAN\n+endif\n+\n ifdef HAVE_PATHS_H\n \tBASIC_CFLAGS += -DHAVE_PATHS_H\n endif\n@@ -2135,6 +2141,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n+\t@echo USE_WATCHMAN=\\''$(subst ','\\'',$(subst ','\\'',$(USE_WATCHMAN)))'\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/cache.h b/cache.h\nindex a05fd31..572299c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -648,6 +648,7 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n+extern int core_watchman_sync_timeout;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex 248a21a..6b63f66 100644\n--- a/config.c\n+++ b/config.c\n@@ -881,6 +881,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.watchmansynctimeout\")) {\n+\t\tcore_watchman_sync_timeout = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.createobject\")) {\n \t\tif (!strcmp(value, \"rename\"))\n \t\t\tobject_creation_mode = OBJECT_CREATION_USES_RENAMES;\ndiff --git a/configure.ac b/configure.ac\nindex 76170ad..9772f79 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -1092,6 +1092,14 @@ AC_COMPILE_IFELSE([BSD_SYSCTL_SRC],\n \tHAVE_BSD_SYSCTL=])\n GIT_CONF_SUBST([HAVE_BSD_SYSCTL])\n \n+#\n+# Check for watchman client library\n+\n+AC_CHECK_LIB([watchman], [watchman_connect],\n+\t[USE_WATCHMAN=YesPlease],\n+\t[USE_WATCHMAN=])\n+GIT_CONF_SUBST([USE_WATCHMAN])\n+\n ## Other checks.\n # Define USE_PIC if you need the main git objects to be built with -fPIC\n # in order to build and link perl/Git.so.  x86-64 seems to need this.\ndiff --git a/environment.c b/environment.c\nindex 2da7fe2..84df431 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -87,6 +87,9 @@ int auto_comment_line_char;\n /* Parallel index stat data preload? */\n int core_preload_index = 1;\n \n+int core_watchman_sync_timeout = 300;\n+\n+\n /* This is set by setup_git_dir_gently() and/or git_default_config() */\n char *git_work_tree_cfg;\n static char *work_tree;\ndiff --git a/watchman-support.c b/watchman-support.c\nnew file mode 100644\nindex 0000000..7f6c0a9\n--- /dev/null\n+++ b/watchman-support.c\n@@ -0,0 +1,108 @@\n+#include \"cache.h\"\n+#include \"watchman-support.h\"\n+#include \"strbuf.h\"\n+#include <watchman.h>\n+\n+static struct watchman_query *make_query(const char *last_update)\n+{\n+\tstruct watchman_query *query = watchman_query();\n+\twatchman_query_set_fields(query, WATCHMAN_FIELD_NAME |\n+\t\t\t\t\t WATCHMAN_FIELD_EXISTS |\n+\t\t\t\t\t WATCHMAN_FIELD_NEWER);\n+\twatchman_query_set_empty_on_fresh(query, 1);\n+\tquery->sync_timeout = core_watchman_sync_timeout;\n+\tif (*last_update)\n+\t\twatchman_query_set_since_oclock(query, last_update);\n+\treturn query;\n+}\n+\n+static struct watchman_query_result* query_watchman(\n+\tstruct index_state *istate, struct watchman_connection *connection,\n+\tconst char *fs_path, const char *last_update)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_query *query;\n+\tstruct watchman_expression *expr;\n+\tstruct watchman_query_result *result;\n+\n+\tquery = make_query(last_update);\n+\texpr = watchman_true_expression();\n+\tresult = watchman_do_query(connection, fs_path, query, expr, &wm_error);\n+\twatchman_free_query(query);\n+\twatchman_free_expression(expr);\n+\n+\tif (!result)\n+\t\twarning(\"Watchman query error: %s (at %s)\",\n+\t\t\twm_error.message,\n+\t\t\t*last_update ? last_update : \"the beginning\");\n+\n+\treturn result;\n+}\n+\n+static void update_index(struct index_state *istate,\n+\t\t\t struct watchman_query_result *result)\n+{\n+\tint i;\n+\n+\tif (result->is_fresh_instance) {\n+\t\t/* let refresh clear them later */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags |= CE_NO_WATCH;\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < result->nr; i++) {\n+\t\tstruct watchman_stat *wm = result->stats + i;\n+\t\tint pos;\n+\n+\t\tif (!strncmp(wm->name, \".git/\", 5) ||\n+\t\t    strstr(wm->name, \"/.git/\"))\n+\t\t\tcontinue;\n+\n+\t\tpos = index_name_pos(istate, wm->name, strlen(wm->name));\n+\t\tif (pos < 0)\n+\t\t\tcontinue;\n+\t\t/* FIXME: ignore staged entries and gitlinks too? */\n+\n+\t\tistate->cache[pos]->ce_flags |= CE_NO_WATCH;\n+\t}\n+\n+done:\n+\tfree(istate->last_update);\n+\tistate->last_update    = xstrdup(result->clock);\n+\tistate->cache_changed |= WATCHMAN_CHANGED;\n+}\n+\n+int check_watchman(struct index_state *istate)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_connection *connection;\n+\tstruct watchman_query_result *result;\n+\tconst char *fs_path;\n+\n+\tfs_path = get_git_work_tree();\n+\tif (!fs_path)\n+\t\treturn -1;\n+\n+\tconnection = watchman_connect(&wm_error);\n+\n+\tif (!connection) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\treturn -1;\n+\t}\n+\n+\tif (watchman_watch(connection, fs_path, &wm_error)) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\twatchman_connection_close(connection);\n+\t\treturn -1;\n+\t}\n+\n+\n+\tresult = query_watchman(istate, connection, fs_path, istate->last_update);\n+\twatchman_connection_close(connection);\n+\tif (!result)\n+\t\treturn -1;\n+\tupdate_index(istate, result);\n+\twatchman_free_query_result(result);\n+\treturn 0;\n+}\ndiff --git a/watchman-support.h b/watchman-support.h\nnew file mode 100644\nindex 0000000..5610409\n--- /dev/null\n+++ b/watchman-support.h\n@@ -0,0 +1,8 @@\n+#ifndef WATCHMAN_SUPPORT_H\n+#define WATCHMAN_SUPPORT_H\n+\n+struct index_state;\n+int check_watchman(struct index_state *index);\n+\n+\n+#endif /* WATCHMAN_SUPPORT_H */\n-- \n2.2.0.513.g477eb31\n"},{"id":"272669","messageId":"1446386146-10438-4-git-send-email-pclouds@gmail.com","threadId":"40692","inReplyTo":"1446386146-10438-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/5] read-cache: allow index-helper to prepare shm before git reads it","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-01T13:55:44Z","receivedAt":"2015-11-01T13:55:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If index-helper puts 'W' before pid in $GIT_DIR/index-helper.pid, then\ngit will sleep for a while, expecting to be waken up by SIGUSR1 when\nindex-helper has done shm preparation, or after the timeout.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n read-cache.c | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 893223e..ae33951 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1591,14 +1591,24 @@ static void do_poke(struct strbuf *sb, int refresh_cache)\n \tPostMessage(hwnd, refresh_cache ? WM_USER : WM_USER + 1, 0, 0);\n }\n #else\n+static void do_nothing(int sig)\n+{\n+}\n+\n static void do_poke(struct strbuf *sb, int refresh_cache)\n {\n-\tchar\t*start = sb->buf;\n+\tint\t wait  = sb->buf[0] == 'W';\n+\tchar\t*start = wait ? sb->buf + 1 : sb->buf;\n \tchar\t*end   = NULL;\n \tpid_t\t pid   = strtoul(start, &end, 10);\n+\tint\t ret;\n \tif (!end || end != sb->buf + sb->len)\n \t\treturn;\n-\tkill(pid, refresh_cache ? SIGHUP : SIGUSR1);\n+\tret = kill(pid, refresh_cache ? SIGHUP : SIGUSR1);\n+\tif (!refresh_cache && !ret && wait) {\n+\t\tsignal(SIGHUP, do_nothing);\n+\t\tsleep(1);\n+\t}\n }\n #endif\n \n-- \n2.2.0.513.g477eb31\n"},{"id":"272671","messageId":"1446386146-10438-5-git-send-email-pclouds@gmail.com","threadId":"40692","inReplyTo":"1446386146-10438-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 4/5] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-01T13:55:45Z","receivedAt":"2015-11-01T13:55:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Watchman is hidden behind index-helper. Before git tries to read the\nindex from shm, it notifies index-helper with SIGHUP and sleep,\nwaiting for index-helper to prepare shm. index-helper then contacts\nwatchman, updates 'WAMA' extension and put it in a separate shm and\nwakes git up with SIGHUP.\n\nGit uses this extension to not lstat unchanged entries. Git only trust\n'WAMA' extension when it's received from the separate shm, not from\ndisk. Unmarked entries are \"clean\". Marked entries are dirty from\nwatchman point of view. If it finds out some entries are\n'watchman-dirty', but are really unchanged (e.g. the file was changed,\nthen reverted back), then Git will clear the marking in 'WAMA' before\nwriting it down.\n\nHiding watchman behind index-helper means you need both daemons. You\ncan't run watchman alone. Not so good. But on the other hand, 'git'\nbinary is not linked to watchman/json libraries, which is good for\npackaging. Core git package will run fine without watchman-related\npackages. If they need watchman, they can install git-index-helper and\ndependencies.\n\nAnother reason for tying watchman to index-helper is, when used with\nuntracked cache, we need to keep track of $GIT_WORK_TREE file\nlisting. That kind of list can be kept in index-helper.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Makefile           |  5 ++++\n cache.h            |  2 ++\n index-helper.c     | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++---\n read-cache.c       | 43 ++++++++++++++++++++++++++--\n watchman-support.h |  1 -\n 5 files changed, 127 insertions(+), 8 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 761acb6..3f5eac8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -434,6 +434,7 @@ MSGFMT = msgfmt\n CURL_CONFIG = curl-config\n PTHREAD_LIBS = -lpthread\n PTHREAD_CFLAGS =\n+WATCHMAN_LIBS =\n GCOV = gcov\n \n export TCL_PATH TCLTK_PATH\n@@ -1392,6 +1393,7 @@ endif\n ifdef USE_WATCHMAN\n \tLIB_H += watchman-support.h\n \tLIB_OBJS += watchman-support.o\n+\tWATCHMAN_LIBS = -lwatchman\n \tBASIC_CFLAGS += -DUSE_WATCHMAN\n endif\n \n@@ -2005,6 +2007,9 @@ git-remote-testsvn$X: remote-testsvn.o GIT-LDFLAGS $(GITLIBS) $(VCSSVN_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) \\\n \t$(VCSSVN_LIB)\n \n+git-index-helper$X: index-helper.o GIT-LDFLAGS $(GITLIBS)\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) $(WATCHMAN_LIBS)\n+\n $(REMOTE_CURL_ALIASES): $(REMOTE_CURL_PRIMARY)\n \t$(QUIET_LNCP)$(RM) $@ && \\\n \tln $< $@ 2>/dev/null || \\\ndiff --git a/cache.h b/cache.h\nindex 572299c..c04141b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -518,6 +518,7 @@ extern int daemonize(int *);\n \n /* Initialize and use the cache information */\n struct lock_file;\n+extern int verify_index(const struct index_state *);\n extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);\n extern int do_read_index(struct index_state *istate, const char *path,\n@@ -525,6 +526,7 @@ extern int do_read_index(struct index_state *istate, const char *path,\n extern int read_index_from(struct index_state *, const char *path);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n+extern void write_watchman_ext(struct strbuf *sb, struct index_state* istate);\n #define COMMIT_LOCK\t\t(1 << 0)\n #define CLOSE_LOCK\t\t(1 << 1)\n #define REFRESH_DAEMON\t\t(1 << 2)\ndiff --git a/index-helper.c b/index-helper.c\nindex cf26da7..421887e 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -5,15 +5,18 @@\n #include \"split-index.h\"\n #include \"shm.h\"\n #include \"lockfile.h\"\n+#include \"watchman-support.h\"\n \n struct shm {\n \tunsigned char sha1[20];\n \tvoid *shm;\n \tsize_t size;\n+\tpid_t pid;\n };\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n+static struct shm shm_watchman;\n static int daemonized, to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n@@ -25,10 +28,21 @@ static void release_index_shm(struct shm *is)\n \tis->shm = NULL;\n }\n \n+static void release_watchman_shm(struct shm *is)\n+{\n+\tif (!is->shm)\n+\t\treturn;\n+\tmunmap(is->shm, is->size);\n+\tgit_shm_unlink(\"git-watchman-%s-%\" PRIuMAX,\n+\t\t       sha1_to_hex(is->sha1), (uintmax_t)is->pid);\n+\tis->shm = NULL;\n+}\n+\n static void cleanup_shm(void)\n {\n \trelease_index_shm(&shm_index);\n \trelease_index_shm(&shm_base_index);\n+\trelease_watchman_shm(&shm_watchman);\n }\n \n static void cleanup(void)\n@@ -120,13 +134,15 @@ static void share_the_index(void)\n \tif (the_index.split_index && the_index.split_index->base)\n \t\tshare_index(the_index.split_index->base, &shm_base_index);\n \tshare_index(&the_index, &shm_index);\n-\tif (to_verify && !verify_shm())\n+\tif (to_verify && !verify_shm()) {\n \t\tcleanup_shm();\n-\tdiscard_index(&the_index);\n+\t\tdiscard_index(&the_index);\n+\t}\n }\n \n static void refresh(int sig)\n {\n+\tdiscard_index(&the_index);\n \tthe_index.keep_mmap = 1;\n \tthe_index.to_shm    = 1;\n \tif (read_cache() < 0)\n@@ -136,7 +152,55 @@ static void refresh(int sig)\n \n #ifdef HAVE_SHM\n \n-static void do_nothing(int sig)\n+#ifdef USE_WATCHMAN\n+static void share_watchman(struct index_state *istate,\n+\t\t\t   struct shm *is, pid_t pid)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tvoid *shm;\n+\n+\twrite_watchman_ext(&sb, istate);\n+\tif (git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, sb.len + 20,\n+\t\t\t&shm, PROT_READ | PROT_WRITE, MAP_SHARED,\n+\t\t\t\"git-watchman-%s-%\" PRIuMAX,\n+\t\t\tsha1_to_hex(istate->sha1), (uintmax_t)pid) == sb.len + 20) {\n+\t\tis->size = sb.len + 20;\n+\t\tis->shm = shm;\n+\t\tis->pid = pid;\n+\t\thashcpy(is->sha1, istate->sha1);\n+\n+\t\tmemcpy(shm, sb.buf, sb.len);\n+\t\thashcpy((unsigned char *)shm + is->size - 20, is->sha1);\n+\t}\n+\tstrbuf_release(&sb);\n+}\n+\n+static void prepare_with_watchman(pid_t pid)\n+{\n+\t/*\n+\t * with the help of watchman, maybe we could detect if\n+\t * $GIT_DIR/index is updated..\n+\t */\n+\tif (!verify_index(&the_index))\n+\t\trefresh(0);\n+\n+\tif (check_watchman(&the_index))\n+\t\treturn;\n+\n+\tshare_watchman(&the_index, &shm_watchman, pid);\n+}\n+\n+static void prepare_index(int sig, siginfo_t *si, void *context)\n+{\n+\tfree_watchman_shm(&shm_watchman);\n+\tif (the_index.last_update)\n+\t\tprepare_with_watchman(si->si_pid);\n+\tkill(si->si_pid, SIGHUP); /* stop the waiting in poke_daemon() */\n+}\n+\n+#else\n+\n+static void prepare_index(int sig, siginfo_t *si, void *context)\n {\n \t/*\n \t * what we need is the signal received and interrupts\n@@ -145,11 +209,21 @@ static void do_nothing(int sig)\n \t */\n }\n \n+#endif\n+\n static void loop(const char *pid_file, int idle_in_seconds)\n {\n+\tstruct sigaction sa;\n+\n \tsigchain_pop(SIGHUP);\t/* pushed by sigchain_push_common */\n \tsigchain_push(SIGHUP, refresh);\n-\tsigchain_push(SIGUSR1, do_nothing);\n+\n+\tmemset(&sa, 0, sizeof(sa));\n+\tsa.sa_sigaction = prepare_index;\n+\tsigemptyset(&sa.sa_mask);\n+\tsa.sa_flags = SA_SIGINFO;\n+\tsigaction(SIGUSR1, &sa, NULL);\n+\n \trefresh(0);\n \twhile (sleep(idle_in_seconds))\n \t\t; /* do nothing, all is handled by signal handlers already */\n@@ -245,6 +319,8 @@ int main(int argc, char **argv)\n \t\t\t\t       LOCK_DIE_ON_ERROR);\n #ifdef GIT_WINDOWS_NATIVE\n \tstrbuf_addstr(&sb, \"HWND\");\n+#elif defined(USE_WATCHMAN)\n+\tstrbuf_addch(&sb, 'W');\t/* see poke_daemon() */\n #endif\n \tstrbuf_addf(&sb, \"%\" PRIuMAX, (uintmax_t) getpid());\n \twrite_in_full(fd, sb.buf, sb.len);\ndiff --git a/read-cache.c b/read-cache.c\nindex ae33951..1b84538 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1405,7 +1405,7 @@ static int read_watchman_ext(struct index_state *istate, const void *data,\n \treturn 0;\n }\n \n-static void write_watchman_ext(struct strbuf *sb, struct index_state* istate)\n+void write_watchman_ext(struct strbuf *sb, struct index_state* istate)\n {\n \tstruct ewah_bitmap *bitmap;\n \tint i;\n@@ -1678,6 +1678,39 @@ static int try_shm(struct index_state *istate)\n \treturn 0;\n }\n \n+static void refresh_by_watchman(struct index_state *istate)\n+{\n+\tvoid *shm = NULL;\n+\tint length;\n+\tint i;\n+\n+\tlength = git_shm_map(O_RDONLY, 0700, -1, &shm,\n+\t\t\t     PROT_READ, MAP_SHARED,\n+\t\t\t     \"git-watchman-%s-%\" PRIuMAX,\n+\t\t\t     sha1_to_hex(istate->sha1),\n+\t\t\t     (uintmax_t)getpid());\n+\n+\tif (length <= 20 ||\n+\t    hashcmp(istate->sha1, (unsigned char *)shm + length - 20) ||\n+\t    /*\n+\t     * No need to clear CE_NO_WATCH set by 'WAMA' on\n+\t     * disk. Watchman can only set more, not clear any, so\n+\t     * this is OR mask.\n+\t     */\n+\t    read_watchman_ext(istate, shm, length - 20))\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\tif (ce_stage(ce) || (ce->ce_flags & CE_NO_WATCH))\n+\t\t\tcontinue;\n+\t\tce_mark_uptodate(ce);\n+\t}\n+done:\n+\tif (shm)\n+\t\tmunmap(shm, length);\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1797,7 +1830,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tsplit_index = istate->split_index;\n \tif (!split_index || is_null_sha1(split_index->base_sha1)) {\n \t\tcheck_ce_order(istate);\n-\t\treturn ret;\n+\t\tgoto done;\n \t}\n \n \tif (split_index->base)\n@@ -1818,6 +1851,10 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\t    sha1_to_hex(split_index->base->sha1));\n \tmerge_base_index(istate);\n \tcheck_ce_order(istate);\n+\n+done:\n+\tif (ret > 0 && istate->from_shm && istate->last_update)\n+\t\trefresh_by_watchman(istate);\n \treturn ret;\n }\n \n@@ -2119,7 +2156,7 @@ out:\n \treturn 0;\n }\n \n-static int verify_index(const struct index_state *istate)\n+int verify_index(const struct index_state *istate)\n {\n \treturn verify_index_from(istate, get_index_file());\n }\ndiff --git a/watchman-support.h b/watchman-support.h\nindex 5610409..ee1ef2c 100644\n--- a/watchman-support.h\n+++ b/watchman-support.h\n@@ -4,5 +4,4 @@\n struct index_state;\n int check_watchman(struct index_state *index);\n \n-\n #endif /* WATCHMAN_SUPPORT_H */\n-- \n2.2.0.513.g477eb31\n"},{"id":"272670","messageId":"1446386146-10438-6-git-send-email-pclouds@gmail.com","threadId":"40692","inReplyTo":"1446386146-10438-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 5/5] update-index: enable/disable watchman support","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-01T13:55:46Z","receivedAt":"2015-11-01T13:55:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n builtin/update-index.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 7431938..86aec21 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -903,6 +903,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n {\n \tint newfd, entries, has_errors = 0, line_termination = '\\n';\n \tint untracked_cache = -1;\n+\tint use_watchman = -1;\n \tint read_from_stdin = 0;\n \tint prefix_length = prefix ? strlen(prefix) : 0;\n \tint preferred_index_format = 0;\n@@ -998,6 +999,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"enable/disable untracked cache\")),\n \t\tOPT_SET_INT(0, \"force-untracked-cache\", &untracked_cache,\n \t\t\t    N_(\"enable untracked cache without testing the filesystem\"), 2),\n+\t\tOPT_BOOL(0, \"watchman\", &use_watchman,\n+\t\t\tN_(\"use or not use watchman to reduce refresh cost\")),\n \t\tOPT_END()\n \t};\n \n@@ -1127,6 +1130,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tthe_index.cache_changed |= UNTRACKED_CHANGED;\n \t}\n \n+\tif (use_watchman > 0) {\n+\t\tthe_index.last_update    = xstrdup(\"\");\n+\t\tthe_index.cache_changed |= WATCHMAN_CHANGED;\n+\t} else if (!use_watchman) {\n+\t\tthe_index.last_update    = NULL;\n+\t\tthe_index.cache_changed |= WATCHMAN_CHANGED;\n+\t}\n+\n \tif (active_cache_changed) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n-- \n2.2.0.513.g477eb31\n"},{"id":"272730","messageId":"CAHVLzcn3j8eLi9VrNoZjyOZ2UzE7=NYF1bqB9UyKmghUoUw1Zg@mail.gmail.com","threadId":"40692","inReplyTo":"1446386146-10438-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 0/5] Use watchman to reduce index refresh time","fromName":"Paolo Ciarrocchi","fromEmail":"paolo.ciarrocchi@gmail.com","sentAt":"2015-11-02T14:54:57Z","receivedAt":"2015-11-02T14:54:57Z","isPatch":true,"sender":{"key":"paolo.ciarrocchi@gmail.com","avatar":null},"body":"On Sun, Nov 1, 2015 at 2:55 PM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n\nHi Duy,\n\n> This series builds on top of the index-helper series I just sent and\n> uses watchman to keep track of file changes in order to avoid lstat()\n> at refresh time. The series can also be found at [1]\n>\n> When I started this work, watchman did not support Windows yet. It\n> does now, even if still experimental [2]. So Windows people, please\n> try it out if you have time.\n>\n> To put all pieces so far together, we have split-index to reduce index\n> write time, untracked cache to reduce I/O as well as computation for\n> .gitignore, index-helper for index read time and this series for\n> lstat() at refresh time. The remaining piece is killing lstat() from\n> untracked cache, but right now it's just some idea and incomplete\n> code.\n\nDid you manage to measure the speedup introduced by this series?\n\nCiao,\n-- \nPaolo\n"},{"id":"272748","messageId":"CACsJy8DzSEVJYc85-3vSAZ8wB1pR9TLz0RrKyKFfHth9Tq+xyg@mail.gmail.com","threadId":"40692","inReplyTo":"CAHVLzcn3j8eLi9VrNoZjyOZ2UzE7=NYF1bqB9UyKmghUoUw1Zg@mail.gmail.com","subject":"Re: [PATCH 0/5] Use watchman to reduce index refresh time","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-02T19:23:52Z","receivedAt":"2015-11-02T19:23:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 2, 2015 at 3:54 PM, Paolo Ciarrocchi\n<paolo.ciarrocchi@gmail.com> wrote:\n> On Sun, Nov 1, 2015 at 2:55 PM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n>\n> Hi Duy,\n>\n>> This series builds on top of the index-helper series I just sent and\n>> uses watchman to keep track of file changes in order to avoid lstat()\n>> at refresh time. The series can also be found at [1]\n>>\n>> When I started this work, watchman did not support Windows yet. It\n>> does now, even if still experimental [2]. So Windows people, please\n>> try it out if you have time.\n>>\n>> To put all pieces so far together, we have split-index to reduce index\n>> write time, untracked cache to reduce I/O as well as computation for\n>> .gitignore, index-helper for index read time and this series for\n>> lstat() at refresh time. The remaining piece is killing lstat() from\n>> untracked cache, but right now it's just some idea and incomplete\n>> code.\n>\n> Did you manage to measure the speedup introduced by this series?\n\nIt was from last year. I may have measured it but because I didn't\nsave it in the commit message, it was lost anyway. Installing watchman\nand measuring with webkit.git soon..\n-- \nDuy\n"},{"id":"272758","messageId":"1446499174.4131.20.camel@twopensource.com","threadId":"40692","inReplyTo":"1446386146-10438-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 2/5] Add watchman support to reduce index refresh cost","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-11-02T21:19:34Z","receivedAt":"2015-11-02T21:19:34Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sun, 2015-11-01 at 14:55 +0100, Nguyễn Thái Ngọc Duy wrote:\n> The previous patch has the logic to clear bits in 'WAMA' bitmap. This\n> patch has logic to set bits as told by watchman. The missing bit,\n> _using_ these bits, are not here yet.\n> \n> A lot of this code is written by David Turner originally, mostly from\n> [1]. I'm just copying and polishing it a bit.\n>\n> [1] http://article.gmane.org/gmane.comp.version-control.git/248006\n\nOur code has evolved somewhat from there[1].  It looks like you've\nincorporated some of these updates.  But I wanted to call out one thing\nin particular that it doesn't look like this patch handles: there's some\nreal ugliness on at least OSX around case-changing renames. \n\nYou probably won't be able to use our code directly as-is, but we'll\nwant to do something about this situation.  This is thicket of TOCTOU\nissues.  For instance, watchman learns that a file called FOO has\nchanged, but by the time it goes to the filesystem to look up the\ncanonical case, FOO has been renamed to foo already (or renamed and then\ndeleted).  \n\nI won't say for sure that your code is insufficient as I haven't yet\ntried it out, but we should ensure this case is handled before this is\nmerged.\n\n[1] https://github.com/dturner-tw/git/tree/dturner/watchman\n\n> +\n> +\t\tpos = index_name_pos(istate, wm->name, strlen(wm->name));\n\nThis is the bit where case matters.\n"},{"id":"272763","messageId":"1446501799.4131.24.camel@twopensource.com","threadId":"40692","inReplyTo":"1446386146-10438-2-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 1/5] read-cache: add watchman 'WAMA' extension","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-11-02T22:03:19Z","receivedAt":"2015-11-02T22:03:19Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Sun, 2015-11-01 at 14:55 +0100, Nguyễn Thái Ngọc Duy wrote:\n>\n>+#define CE_NO_WATCH  (0x0001)\n\nThis name seems very confusing to me.  CE_NO_WATCHMAN_STAT?\nCE_UNKNOWN_TO_WATCHMAN?  \n\n(one reason it may seem more confusing to me than to others is that\nTwitter's code has a concept of files that we don't watch at all e.g.\nIntellij's .idea dir).  \n\n> @@ -322,6 +325,7 @@ struct index_state {\n>  \tstruct untracked_cache *untracked;\n>  \tvoid *mmap;\n>  \tsize_t mmap_size;\n> +\tchar *last_update;\n\nMight be worth a comment explaining what this is.\n"},{"id":"272823","messageId":"CACsJy8Dz17gRSHch9e=iB2Kq2t4FbssatW84DF8pHTuFMgEgjQ@mail.gmail.com","threadId":"40692","inReplyTo":"CACsJy8DzSEVJYc85-3vSAZ8wB1pR9TLz0RrKyKFfHth9Tq+xyg@mail.gmail.com","subject":"Re: [PATCH 0/5] Use watchman to reduce index refresh time","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-03T09:21:36Z","receivedAt":"2015-11-03T09:21:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 2, 2015 at 8:23 PM, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Mon, Nov 2, 2015 at 3:54 PM, Paolo Ciarrocchi\n> <paolo.ciarrocchi@gmail.com> wrote:\n>> On Sun, Nov 1, 2015 at 2:55 PM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n>>\n>> Hi Duy,\n>>\n>>> This series builds on top of the index-helper series I just sent and\n>>> uses watchman to keep track of file changes in order to avoid lstat()\n>>> at refresh time. The series can also be found at [1]\n>>>\n>>> When I started this work, watchman did not support Windows yet. It\n>>> does now, even if still experimental [2]. So Windows people, please\n>>> try it out if you have time.\n>>>\n>>> To put all pieces so far together, we have split-index to reduce index\n>>> write time, untracked cache to reduce I/O as well as computation for\n>>> .gitignore, index-helper for index read time and this series for\n>>> lstat() at refresh time. The remaining piece is killing lstat() from\n>>> untracked cache, but right now it's just some idea and incomplete\n>>> code.\n>>\n>> Did you manage to measure the speedup introduced by this series?\n>\n> It was from last year. I may have measured it but because I didn't\n> save it in the commit message, it was lost anyway. Installing watchman\n> and measuring with webkit.git soon..\n\nTest repo: webkit.git with 104665 tracked files, 5615 untracked,\n3517 dirs. Best numbers out of a few tries. This is best case\nscenario. Normal usage could have worse numbers.\n\nThere is something strange about the \"-uno\" measurements. I don't\nthink watchman+untracked cache can beat -uno..  Maybe I did something\nwrong.\n\n0m0.383s   index v2\n0m0.351s   index v4\n0m0.352s   v2 split-index\n0m0.309s   v2 split index-helper\n0m0.159s   v2 split helper untracked-cache\n0m0.123s   v2 split helper \"status -uno\"\n0m0.098s   v2 split helper untracked watchman\n0m0.071s   v2 split helper watchman \"status -uno\"\n\nNote, the watchman series needs\ns/free_watchman_shm/release_watchman_shm/ (I didn't do a good job\nof testing after rebase). And there's a small bug in index-helper\n--detach code writing incorrect PID..\n-- \nDuy\n"},{"id":"272825","messageId":"CAHVLzcnm6qUhcuuP36Q7Yvf9rUAu-KK4j5-Rzmzd_CFCaRwYug@mail.gmail.com","threadId":"40692","inReplyTo":"CACsJy8Dz17gRSHch9e=iB2Kq2t4FbssatW84DF8pHTuFMgEgjQ@mail.gmail.com","subject":"Re: [PATCH 0/5] Use watchman to reduce index refresh time","fromName":"Paolo Ciarrocchi","fromEmail":"paolo.ciarrocchi@gmail.com","sentAt":"2015-11-03T10:26:47Z","receivedAt":"2015-11-03T10:26:47Z","isPatch":true,"sender":{"key":"paolo.ciarrocchi@gmail.com","avatar":null},"body":"On Tue, Nov 3, 2015 at 10:21 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n>> It was from last year. I may have measured it but because I didn't\n>> save it in the commit message, it was lost anyway. Installing watchman\n>> and measuring with webkit.git soon..\n>\n> Test repo: webkit.git with 104665 tracked files, 5615 untracked,\n> 3517 dirs. Best numbers out of a few tries. This is best case\n> scenario. Normal usage could have worse numbers.\n>\n> There is something strange about the \"-uno\" measurements. I don't\n> think watchman+untracked cache can beat -uno..  Maybe I did something\n> wrong.\n>\n> 0m0.383s   index v2\n> 0m0.351s   index v4\n> 0m0.352s   v2 split-index\n> 0m0.309s   v2 split index-helper\n> 0m0.159s   v2 split helper untracked-cache\n> 0m0.123s   v2 split helper \"status -uno\"\n> 0m0.098s   v2 split helper untracked watchman\n> 0m0.071s   v2 split helper watchman \"status -uno\"\n>\n> Note, the watchman series needs\n> s/free_watchman_shm/release_watchman_shm/ (I didn't do a good job\n> of testing after rebase). And there's a small bug in index-helper\n> --detach code writing incorrect PID..\n\n\nImpressive improvements!\n\nCiao,\nPaolo\n\n-- \nPaolo\n"},{"id":"272843","messageId":"CACsJy8Bb0BVsCdgDQe9JWPUOR9Hw+963Zph3HMZA6Zmzr2THMA@mail.gmail.com","threadId":"40692","inReplyTo":"1446501799.4131.24.camel@twopensource.com","subject":"Re: [PATCH 1/5] read-cache: add watchman 'WAMA' extension","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-03T19:17:38Z","receivedAt":"2015-11-03T19:17:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 2, 2015 at 11:03 PM, David Turner <dturner@twopensource.com> wrote:\n> On Sun, 2015-11-01 at 14:55 +0100, Nguyễn Thái Ngọc Duy wrote:\n>>\n>>+#define CE_NO_WATCH  (0x0001)\n>\n> This name seems very confusing to me.  CE_NO_WATCHMAN_STAT?\n> CE_UNKNOWN_TO_WATCHMAN?\n\nFiles that are known updated. Maybe CE_WATCHMAN_DIRTY?\n\n> (one reason it may seem more confusing to me than to others is that\n> Twitter's code has a concept of files that we don't watch at all e.g.\n> Intellij's .idea dir).\n>\n>> @@ -322,6 +325,7 @@ struct index_state {\n>>       struct untracked_cache *untracked;\n>>       void *mmap;\n>>       size_t mmap_size;\n>> +     char *last_update;\n>\n> Might be worth a comment explaining what this is.\n\nIt's the clock value from watchman when we query file status. Will\nmake a note. Or maybe I should just rename it to watchman_clock.\n-- \nDuy\n"},{"id":"272846","messageId":"1446580181.4131.36.camel@twopensource.com","threadId":"40692","inReplyTo":"CACsJy8Bb0BVsCdgDQe9JWPUOR9Hw+963Zph3HMZA6Zmzr2THMA@mail.gmail.com","subject":"Re: [PATCH 1/5] read-cache: add watchman 'WAMA' extension","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-11-03T19:49:41Z","receivedAt":"2015-11-03T19:49:41Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2015-11-03 at 20:17 +0100, Duy Nguyen wrote:\n> On Mon, Nov 2, 2015 at 11:03 PM, David Turner <dturner@twopensource.com> wrote:\n> > On Sun, 2015-11-01 at 14:55 +0100, Nguyễn Thái Ngọc Duy wrote:\n> >>\n> >>+#define CE_NO_WATCH  (0x0001)\n> >\n> > This name seems very confusing to me.  CE_NO_WATCHMAN_STAT?\n> > CE_UNKNOWN_TO_WATCHMAN?\n> \n> Files that are known updated. Maybe CE_WATCHMAN_DIRTY?\n\n+1\n\n> > (one reason it may seem more confusing to me than to others is that\n> > Twitter's code has a concept of files that we don't watch at all e.g.\n> > Intellij's .idea dir).\n> >\n> >> @@ -322,6 +325,7 @@ struct index_state {\n> >>       struct untracked_cache *untracked;\n> >>       void *mmap;\n> >>       size_t mmap_size;\n> >> +     char *last_update;\n> >\n> > Might be worth a comment explaining what this is.\n> \n> It's the clock value from watchman when we query file status. Will\n> make a note. Or maybe I should just rename it to watchman_clock.\n\nEither way would work for me.\n"},{"id":"273109","messageId":"CAP8UFD0rAL62eFdtLo__ta=_Yhq4cdRcqmfYTqw3imNa5Lde-A@mail.gmail.com","threadId":"40692","inReplyTo":"CACsJy8Dz17gRSHch9e=iB2Kq2t4FbssatW84DF8pHTuFMgEgjQ@mail.gmail.com","subject":"Re: [PATCH 0/5] Use watchman to reduce index refresh time","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2015-11-09T20:06:53Z","receivedAt":"2015-11-09T20:06:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Nov 3, 2015 at 10:21 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Mon, Nov 2, 2015 at 8:23 PM, Duy Nguyen <pclouds@gmail.com> wrote:\n>> On Mon, Nov 2, 2015 at 3:54 PM, Paolo Ciarrocchi\n>> <paolo.ciarrocchi@gmail.com> wrote:\n>>> On Sun, Nov 1, 2015 at 2:55 PM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n>>>\n>>> Hi Duy,\n>>>\n>>>> This series builds on top of the index-helper series I just sent and\n>>>> uses watchman to keep track of file changes in order to avoid lstat()\n>>>> at refresh time. The series can also be found at [1]\n>>>>\n>>>> When I started this work, watchman did not support Windows yet. It\n>>>> does now, even if still experimental [2]. So Windows people, please\n>>>> try it out if you have time.\n>>>>\n>>>> To put all pieces so far together, we have split-index to reduce index\n>>>> write time, untracked cache to reduce I/O as well as computation for\n>>>> .gitignore, index-helper for index read time and this series for\n>>>> lstat() at refresh time. The remaining piece is killing lstat() from\n>>>> untracked cache, but right now it's just some idea and incomplete\n>>>> code.\n>>>\n>>> Did you manage to measure the speedup introduced by this series?\n>>\n>> It was from last year. I may have measured it but because I didn't\n>> save it in the commit message, it was lost anyway. Installing watchman\n>> and measuring with webkit.git soon..\n>\n> Test repo: webkit.git with 104665 tracked files, 5615 untracked,\n> 3517 dirs. Best numbers out of a few tries. This is best case\n> scenario. Normal usage could have worse numbers.\n\nThank you for the tests!\n\nI tried to replicate your results on webkit.git with 184672 tracked\nfiles, 5631 untracked, 9330 dirs. Also best numbers out of a few\ntries.\n\n> There is something strange about the \"-uno\" measurements. I don't\n> think watchman+untracked cache can beat -uno..  Maybe I did something\n> wrong.\n>\n> 0m0.383s   index v2\n> 0m0.351s   index v4\n> 0m0.352s   v2 split-index\n> 0m0.309s   v2 split index-helper\n> 0m0.159s   v2 split helper untracked-cache\n> 0m0.123s   v2 split helper \"status -uno\"\n> 0m0.098s   v2 split helper untracked watchman\n> 0m0.071s   v2 split helper watchman \"status -uno\"\n\nI got the following results from \"time git status ...\":\n\n0m0.774s\n0m0.799s split\n0m0.766s split helper\n0m0.335s split helper untracked\n0m0.232s split helper untracked -uno\n0m0.284s split helper untracked watchman\n0m0.188s split helper untracked watchman -uno\n\nUsing David's series I get worse results than all of the above but I\nguess it's because his series is based on an ancient git version\n(v2.0.0-rc0).\n\n> Note, the watchman series needs\n> s/free_watchman_shm/release_watchman_shm/ (I didn't do a good job\n> of testing after rebase). And there's a small bug in index-helper\n> --detach code writing incorrect PID..\n\nI did the s/free_watchman_shm/release_watchman_shm/ change, but I\ndidn't notice the index-helper bug.\n\nThanks,\nChristian.\n"},{"id":"273161","messageId":"1447189468.20147.32.camel@twopensource.com","threadId":"40692","inReplyTo":"CAP8UFD0rAL62eFdtLo__ta=_Yhq4cdRcqmfYTqw3imNa5Lde-A@mail.gmail.com","subject":"Re: [PATCH 0/5] Use watchman to reduce index refresh time","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-11-10T21:04:28Z","receivedAt":"2015-11-10T21:04:28Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Mon, 2015-11-09 at 21:06 +0100, Christian Couder wrote:\n> Using David's series I get worse results than all of the above but I\n> guess it's because his series is based on an ancient git version\n> (v2.0.0-rc0).\n\nMy more-recent series is on top of 2.4, but (for webkit):\nmine: 0m0.206s \nduy's: 0m0.107s\n\nHowever, I'm getting occasional index-helper segfaults due to\nistate->last_update being NULL.  (I'm using Duy's  index-helper branch\nfrom his github + this patchset + a function signature fix due to a\nnewer version of libwatchman).  I haven't looked into why this is --\nmaybe accidentally mixing git versions while testing?  \n\nAlso, after messing around for a while (on Duy's branch), I ended up in\na state where git status would take ~2.5s every time.  The index helper\nwas alive but evidently not working right.  Killing and restarting it\nworked.  Actually, I think I can repro this more easily: \"git rm\nChangelog\" seems to put the index-helper into this state.\n"},{"id":"273527","messageId":"CAP8UFD00SViJ3fdj2s+z1-GWtrcgrF_pXuUUsEGjzmnxe5_gmQ@mail.gmail.com","threadId":"40692","inReplyTo":"1447189468.20147.32.camel@twopensource.com","subject":"Re: [PATCH 0/5] Use watchman to reduce index refresh time","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2015-11-20T09:45:21Z","receivedAt":"2015-11-20T09:45:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Nov 10, 2015 at 10:04 PM, David Turner <dturner@twopensource.com> wrote:\n> On Mon, 2015-11-09 at 21:06 +0100, Christian Couder wrote:\n>> Using David's series I get worse results than all of the above but I\n>> guess it's because his series is based on an ancient git version\n>> (v2.0.0-rc0).\n>\n> My more-recent series is on top of 2.4, but (for webkit):\n> mine: 0m0.206s\n> duy's: 0m0.107s\n\nI tried using your more recent series and I get basically no change\ncompared to the current git (without using untracked cache) on Webkit\nwith around 5600 untracked files. Maybe I am doing something wrong.\n\nI compiled watchman and installed it. It is in /usr/local/bin/watchman\nbut I am not sure it is actually being used though I have configured\n\"core.usewatchman\" to \"true\".\n\n> However, I'm getting occasional index-helper segfaults due to\n> istate->last_update being NULL.  (I'm using Duy's  index-helper branch\n> from his github + this patchset + a function signature fix due to a\n> newer version of libwatchman).  I haven't looked into why this is --\n> maybe accidentally mixing git versions while testing?\n>\n> Also, after messing around for a while (on Duy's branch), I ended up in\n> a state where git status would take ~2.5s every time.  The index helper\n> was alive but evidently not working right.  Killing and restarting it\n> worked.  Actually, I think I can repro this more easily: \"git rm\n> Changelog\" seems to put the index-helper into this state.\n\nThanks for this. I have also seen strange things happening with the\nindex-helper.\nWhen it is working I found that it provides around 10% improvement on\ngit rebase time.\n"}]}