{"thread":{"id":"42713","subject":"[PATCH v13 04/20] index-helper: new daemon for caching index and related stuff","startedAt":"2016-06-26T04:14:42Z","lastAt":"2016-07-08T16:40:19Z","messageCount":35,"participants":["David Turner","Nguyễn Thái Ngọc Duy","Johannes Schindelin","Duy Nguyen","Ben Peart"],"isPatch":true,"patchVersion":13,"patchTotal":20},"messages":[{"id":"290154","messageId":"1466914464-10358-4-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 04/20] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:08Z","receivedAt":"2016-06-26T04:14:42Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nInstead of reading the index from disk and worrying about disk\ncorruption, the index is cached in memory (memory bit-flips happen\ntoo, but hopefully less often). The result is faster read. Read time\nis reduced by 70%.\n\nThe biggest gain is not having to verify the trailing SHA-1, which\ntakes lots of time especially on large index files. But this also\nopens doors for further optimizations:\n\n - we could create an in-memory format that's essentially the memory\n dump of the index to eliminate most of parsing/allocation\n overhead. The mmap'd memory can be used straight away. Experiment\n [1] shows we could reduce read time by 88%.\n\n - we could cache non-index info such as name hash\n\nShared memory is done by storing files in a per-repository temporary\ndirectory. This is more portable than shm (which requires\nposix-realtime and has various quirks on OS X). It might even work on\nWindows, although this has not been tested. The shared memory file's\nname follows the template \"shm--\" where is the\ntrailing SHA-1 of the index file. is \"index\" for cached index\nfiles (and might later be \"name-hash\" for name-hash cache). If such\nshared memory exists, it contains the same index content as on\ndisk. The content is already validated by the daemon and git won't\nvalidate it again (except comparing the trailing SHA-1s).\n\nWe also add some bits to the index (to_shm and from_shm) to track\nwhen an index came from shared memory or is going to shared memory.\n\nWe keep this daemon's logic as thin as possible. The \"brain\" stays in\ngit. So the daemon can read and validate stuff, but that's all it's\nallowed to do. It does not add/create new information. It doesn't even\naccept direct updates from git.\n\nGit can poke the daemon via unix domain sockets to tell it to refresh\nthe index cache, or to keep it alive some more minutes. It can't give\nany real index data directly to the daemon. Real data goes to disk\nfirst, then the daemon reads and verifies it from there. The daemon\nonly handles $GIT_DIR/index, not temporary index files; it only gets\npoked for the former.\n\n$GIT_DIR/index-helper.sock is the socket for the daemon process. The\ndaemon reads from the socket and executes commands.\n\nNamed pipes were considered for portability reasons, but then commands\nthat need replies from the daemon would have to open their own pipes,\nsince a named pipe should only have one reader. Unix domain sockets\ndon't have this problem.\n\nOn webkit.git with index format v2, duplicating 8 times to 1.5m\nentries and 236MB in size:\n\n(vanilla) 0.50 s: read_index_from .git/index\n(index-helper) 0.18 s: read_index_from .git/index\n\nInterestingly with index v4, we get less out of index-helper. It makes\nsense as v4 requires more processing after loading the index:\n\n(vanilla) 0.37 s: read_index_from .git/index\n(index-helper) 0.22 s: read_index_from .git/index\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/247268/focus=248771\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Ramsay Jones \nSigned-off-by: Junio C Hamano \n---\n .gitignore | 1 +\n Documentation/git-index-helper.txt | 50 ++++++\n Makefile | 7 +\n cache.h | 11 ++\n command-list.txt | 1 +\n contrib/completion/git-completion.bash | 1 +\n git-compat-util.h | 1 +\n index-helper.c | 292 +++++++++++++++++++++++++++++++++\n read-cache.c | 123 +++++++++++++-\n t/t7900-index-helper.sh | 23 +++\n 10 files changed, 501 insertions(+), 9 deletions(-)\n create mode 100644 Documentation/git-index-helper.txt\n create mode 100644 index-helper.c\n create mode 100755 t/t7900-index-helper.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 5087ce1..b92f122 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -71,6 +71,7 @@\n /git-http-fetch\n /git-http-push\n /git-imap-send\n+/git-index-helper\n /git-index-pack\n /git-init\n /git-init-db\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nnew file mode 100644\nindex 0000000..fa6e347\n--- /dev/null\n+++ b/Documentation/git-index-helper.txt\n@@ -0,0 +1,50 @@\n+git-index-helper(1)\n+===================\n+\n+NAME\n+----\n+git-index-helper - A simple cache daemon for speeding up index file access\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git index-helper' [options]\n+\n+DESCRIPTION\n+-----------\n+Keep the index file in memory for faster access. This daemon is per\n+repository and per working tree. So if you have two working trees\n+each with a submodule, you might need four index-helpers. (In practice,\n+this is only worthwhile for large indexes, so only use it if you notice\n+that git status is slow).\n+\n+OPTIONS\n+-------\n+\n+--exit-after=::\n+\tExit if the cached index is not accessed for ``\n+\tseconds. Specify 0 to wait forever. Default is 600.\n+\n+NOTES\n+-----\n+\n+$GIT_DIR/index-helper.sock a Unix domain socket that the daemon reads\n+commands from. The directory will also contain files named\n+\"shm-index-\". These are used as backing stores for shared\n+memory. Normally the daemon will clean up these files when it exits\n+or when they are no longer relevant. But if it crashes, some objects\n+could remain there and they can be safely deleted with \"rm\"\n+command. The following commands are used to control the daemon:\n+\n+\"refresh\"::\n+\tReread the index.\n+\n+\"poke\":\n+\tLet the daemon know the index is to be read. It keeps the\n+\tdaemon alive longer, unless `--exit-after=0` is used.\n+\n+All commands and replies are in pkt-line format.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 2742a69..2b2ac05 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1433,6 +1433,12 @@ ifdef HAVE_DEV_TTY\n \tBASIC_CFLAGS += -DHAVE_DEV_TTY\n endif\n \n+ifndef NO_MMAP\n+ifndef NO_UNIX_SOCKETS\n+\tPROGRAM_OBJS += index-helper.o\n+endif\n+endif\n+\n ifdef DIR_HAS_BSD_GROUP_SEMANTICS\n \tCOMPAT_CFLAGS += -DDIR_HAS_BSD_GROUP_SEMANTICS\n endif\n@@ -2159,6 +2165,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n+\t@echo NO_MMAP=\\''$(subst ','\\'',$(subst ','\\'',$(NO_MMAP)))'\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/cache.h b/cache.h\nindex 4180e2b..2d7af6f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -334,6 +334,17 @@ struct index_state {\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n \t\t keep_mmap : 1,\n+\n+\t\t /*\n+\t\t * This index came from index-helper originally.\n+\t\t */\n+\t\t from_shm : 1,\n+\t\t /*\n+\t\t * We're moving this index to shared memory, so we\n+\t\t * don't need to poke the daemon to request updates\n+\t\t * on it.\n+\t\t */\n+\t\t to_shm : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\ndiff --git a/command-list.txt b/command-list.txt\nindex 2a94137..d9ad7d1 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -69,6 +69,7 @@ git-http-backend synchingrepositories\n git-http-fetch synchelpers\n git-http-push synchelpers\n git-imap-send foreignscminterface\n+git-index-helper purehelpers\n git-index-pack plumbingmanipulators\n git-init mainporcelain init\n git-instaweb ancillaryinterrogators\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex e3918c8..98794d2 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -684,6 +684,7 @@ __git_list_porcelain_commands ()\n \t\tfor-each-ref) : plumbing;;\n \t\thash-object) : plumbing;;\n \t\thttp-*) : transport;;\n+\t\tindex-helper) : plumbing;;\n \t\tindex-pack) : plumbing;;\n \t\tinit-db) : deprecated;;\n \t\tlocal-fetch) : plumbing;;\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 4743954..41a0efa 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -513,6 +513,7 @@ static inline int ends_with(const char *str, const char *suffix)\n #define PROT_READ 1\n #define PROT_WRITE 2\n #define MAP_PRIVATE 1\n+#define MAP_SHARED 2\n #endif\n \n #define mmap git_mmap\ndiff --git a/index-helper.c b/index-helper.c\nnew file mode 100644\nindex 0000000..baf5bc6\n--- /dev/null\n+++ b/index-helper.c\n@@ -0,0 +1,292 @@\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"sigchain.h\"\n+#include \"strbuf.h\"\n+#include \"exec_cmd.h\"\n+#include \"split-index.h\"\n+#include \"lockfile.h\"\n+#include \"cache.h\"\n+#include \"unix-socket.h\"\n+#include \"pkt-line.h\"\n+\n+struct shm {\n+\tunsigned char sha1[20];\n+\tvoid *shm;\n+\tsize_t size;\n+};\n+\n+static struct shm shm_index;\n+static struct shm shm_base_index;\n+\n+static void release_index_shm(struct shm *is)\n+{\n+\tif (!is->shm)\n+\t\treturn;\n+\tmunmap(is->shm, is->size);\n+\tunlink(git_path(\"shm-index-%s\", sha1_to_hex(is->sha1)));\n+\tis->shm = NULL;\n+}\n+\n+static void cleanup_shm(void)\n+{\n+\trelease_index_shm(&shm_index);\n+\trelease_index_shm(&shm_base_index);\n+}\n+\n+static void cleanup(void)\n+{\n+\tunlink(git_path(\"index-helper.sock\"));\n+\tcleanup_shm();\n+}\n+\n+static void cleanup_on_signal(int sig)\n+{\n+\t/*\n+\t * We ignore sigpipes -- that's just a client suddenly dying.\n+\t */\n+\tif (sig == SIGPIPE)\n+\t\treturn;\n+\tcleanup();\n+\tsigchain_pop(sig);\n+\traise(sig);\n+}\n+\n+static int shared_mmap_create(size_t size, void **new_mmap, const char *path)\n+{\n+\tint fd = -1;\n+\tint ret = -1;\n+\n+\tfd = open(path, O_CREAT | O_EXCL | O_RDWR, 0600);\n+\n+\tif (fd < 0) {\n+\t\tif (errno != EEXIST)\n+\t\t\tgoto done;\n+\n+\t\tunlink(path);\n+\t\tfd = open(path, O_CREAT | O_EXCL | O_RDWR, 0600);\n+\t}\n+\n+\tif (fd < 0)\n+\t\tgoto done;\n+\n+\tif (ftruncate(fd, size))\n+\t\tgoto done;\n+\n+\t*new_mmap = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED,\n+\t\t\t fd, 0);\n+\n+\tif (*new_mmap == MAP_FAILED) {\n+\t\t*new_mmap = NULL;\n+\t\tgoto done;\n+\t}\n+\tmadvise(new_mmap, size, MADV_WILLNEED);\n+\n+\tret = 0;\n+done:\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+\treturn ret;\n+}\n+\n+static void share_index(struct index_state *istate, struct shm *is)\n+{\n+\tvoid *new_mmap;\n+\tif (istate->mmap_size <= 20 ||\n+\t hashcmp(istate->sha1,\n+\t\t (unsigned char *)istate->mmap + istate->mmap_size - 20) ||\n+\t !hashcmp(istate->sha1, is->sha1))\n+\t\t/* Nothing to do */\n+\t\treturn;\n+\tif (shared_mmap_create(istate->mmap_size, &new_mmap,\n+\t\t\t git_path(\"shm-index-%s\",\n+\t\t\t\t\tsha1_to_hex(istate->sha1))) < 0) {\n+\t\tdie(\"Failed to create shm-index file\");\n+\t}\n+\n+\n+\trelease_index_shm(is);\n+\tis->size = istate->mmap_size;\n+\tis->shm = new_mmap;\n+\thashcpy(is->sha1, istate->sha1);\n+\n+\tmemcpy(new_mmap, istate->mmap, istate->mmap_size - 20);\n+\n+\t/*\n+\t * The trailing hash must be written last after everything is\n+\t * written. It's the indication that the shared memory is now\n+\t * ready.\n+\t * The memory barrier here matches read-cache.c:try_shm.\n+\t */\n+\t__sync_synchronize();\n+\n+\thashcpy((unsigned char *)new_mmap + istate->mmap_size - 20, is->sha1);\n+}\n+\n+static void share_the_index(void)\n+{\n+\tif (the_index.split_index && the_index.split_index->base)\n+\t\tshare_index(the_index.split_index->base, &shm_base_index);\n+\tshare_index(&the_index, &shm_index);\n+\tdiscard_index(&the_index);\n+}\n+\n+static void set_socket_blocking_flag(int fd, int make_nonblocking)\n+{\n+\tint flags;\n+\n+\tflags = fcntl(fd, F_GETFL, NULL);\n+\n+\tif (flags < 0)\n+\t\tdie(_(\"fcntl failed\"));\n+\n+\tif (make_nonblocking)\n+\t\tflags |= O_NONBLOCK;\n+\telse\n+\t\tflags &= ~O_NONBLOCK;\n+\n+\tif (fcntl(fd, F_SETFL, flags) < 0)\n+\t\tdie(_(\"fcntl failed\"));\n+}\n+\n+static void refresh(void)\n+{\n+\tdiscard_index(&the_index);\n+\tthe_index.keep_mmap = 1;\n+\tthe_index.to_shm = 1;\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"could not read index\"));\n+\tshare_the_index();\n+}\n+\n+#ifndef NO_MMAP\n+\n+static void loop(int fd, int idle_in_seconds)\n+{\n+\tassert(idle_in_seconds < INT_MAX / 1000);\n+\n+\tif (idle_in_seconds == 0)\n+\t\tidle_in_seconds = -1;\n+\n+\twhile (1) {\n+\t\tstruct pollfd pollfd;\n+\t\tint result, client_fd;\n+\t\tint flags;\n+\t\tchar buf[4096];\n+\t\tint bytes_read;\n+\n+\t\t/* Wait for a request */\n+\t\tpollfd.fd = fd;\n+\t\tpollfd.events = POLLIN;\n+\t\tresult = poll(&pollfd, 1, idle_in_seconds * 1000);\n+\t\tif (result < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\t/*\n+\t\t\t\t * This can lead to an overlong keepalive,\n+\t\t\t\t * but that is better than a premature exit.\n+\t\t\t\t */\n+\t\t\t\tcontinue;\n+\t\t\tdie_errno(_(\"poll() failed\"));\n+\t\t} else if (result == 0)\n+\t\t\t/* timeout */\n+\t\t\tbreak;\n+\n+\t\tclient_fd = accept(fd, NULL, NULL);\n+\t\tif (client_fd < 0)\n+\t\t\t/*\n+\t\t\t * An error here is unlikely -- it probably\n+\t\t\t * indicates that the connecting process has\n+\t\t\t * already dropped the connection.\n+\t\t\t */\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * Our connection to the client is blocking since a client\n+\t\t * can always be killed by SIGINT or similar.\n+\t\t */\n+\t\tset_socket_blocking_flag(client_fd, 0);\n+\n+\t\tflags = PACKET_READ_GENTLE_ON_EOF | PACKET_READ_CHOMP_NEWLINE;\n+\t\tbytes_read = packet_read(client_fd, NULL, NULL, buf,\n+\t\t\t\t\t sizeof(buf), flags);\n+\n+\t\tif (bytes_read > 0) {\n+\t\t\t/* ensure string termination */\n+\t\t\tbuf[bytes_read] = 0;\n+\t\t\tif (!strcmp(buf, \"refresh\")) {\n+\t\t\t\trefresh();\n+\t\t\t} else if (!strcmp(buf, \"poke\")) {\n+\t\t\t\t/*\n+\t\t\t\t * Just a poke to keep us\n+\t\t\t\t * alive, nothing to do.\n+\t\t\t\t */\n+\t\t\t} else {\n+\t\t\t\twarning(\"BUG: Bogus command %s\", buf);\n+\t\t\t}\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * No command from client. Probably it's just\n+\t\t\t * a liveness check or client error. Just\n+\t\t\t * close up.\n+\t\t\t */\n+\t\t}\n+\t\tclose(client_fd);\n+\t}\n+\n+\tclose(fd);\n+}\n+\n+#else\n+\n+static void loop(int fd, int idle_in_seconds)\n+{\n+\tdie(_(\"index-helper is not supported on this platform\"));\n+}\n+\n+#endif\n+\n+static const char * const usage_text[] = {\n+\tN_(\"git index-helper [options]\"),\n+\tNULL\n+};\n+\n+int main(int argc, char **argv)\n+{\n+\tconst char *prefix;\n+\tint idle_in_seconds = 600;\n+\tint fd;\n+\tstruct strbuf socket_path = STRBUF_INIT;\n+\tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"exit-after\", &idle_in_seconds,\n+\t\t\t N_(\"exit if not used after some seconds\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_extract_argv0_path(argv[0]);\n+\tgit_setup_gettext();\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(usage_text, options);\n+\n+\tprefix = setup_git_directory();\n+\tif (parse_options(argc, (const char **)argv, prefix,\n+\t\t\t options, usage_text, 0))\n+\t\tdie(_(\"too many arguments\"));\n+\n+\tatexit(cleanup);\n+\tsigchain_push_common(cleanup_on_signal);\n+\n+\tstrbuf_git_path(&socket_path, \"index-helper.sock\");\n+\n+\tfd = unix_stream_listen(socket_path.buf);\n+\tif (fd < 0)\n+\t\tdie_errno(_(\"could not set up index-helper socket\"));\n+\n+\tif (idle_in_seconds >= INT_MAX / 1000)\n+\t\tdie(\"--exit-after value must be less than %d seconds\",\n+\t\t INT_MAX / 1000);\n+\n+\tloop(fd, idle_in_seconds);\n+\n+\tclose(fd);\n+\treturn 0;\n+}\ndiff --git a/read-cache.c b/read-cache.c\nindex 3cb0ec3..10d5465 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -18,6 +18,9 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"utf8.h\"\n+#include \"unix-socket.h\"\n+#include \"pkt-line.h\"\n+#include \"sigchain.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -1541,6 +1544,96 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_untracked_cache(istate);\n }\n \n+static int poke_daemon(struct index_state *istate,\n+\t\t const struct stat *st, int refresh_cache)\n+{\n+\tint fd;\n+\tint ret = 0;\n+\tconst char *socket_path;\n+\n+\t/* if this is from index-helper, do not poke itself (recursively) */\n+\tif (istate->to_shm)\n+\t\treturn 0;\n+\n+\tsocket_path = git_path(\"index-helper.sock\");\n+\n+\tfd = unix_stream_connect(socket_path);\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tif (refresh_cache) {\n+\t\tpacket_write_gently(fd, \"refresh\");\n+\t} else {\n+\t\tpacket_write_gently(fd, \"poke\");\n+\t}\n+\tpacket_flush_gently(fd);\n+\n+\tclose(fd);\n+\tsigchain_pop(SIGPIPE);\n+\treturn ret;\n+}\n+\n+static int is_main_index(struct index_state *istate)\n+{\n+\treturn istate == &the_index ||\n+\t\t(the_index.split_index &&\n+\t\t istate == the_index.split_index->base);\n+}\n+\n+/*\n+ * Try to open and verify a cached shm index if available. Return 0 if\n+ * succeeds (istate->mmap and istate->mmap_size are updated). Return\n+ * negative otherwise.\n+ */\n+static int try_shm(struct index_state *istate)\n+{\n+\tvoid *new_mmap = NULL;\n+\tsize_t old_size = istate->mmap_size;\n+\tssize_t new_size;\n+\tconst unsigned char *sha1;\n+\tstruct stat st;\n+\tint fd = -1;\n+\n+\tif (!is_main_index(istate) ||\n+\t old_size <= 20 ||\n+\t stat(git_path(\"index-helper.sock\"), &st))\n+\t\treturn -1;\n+\tif (poke_daemon(istate, &st, 0))\n+\t\treturn -1;\n+\tsha1 = (unsigned char *)istate->mmap + old_size - 20;\n+\n+\tfd = open(git_path(\"shm-index-%s\", sha1_to_hex(sha1)), O_RDONLY);\n+\tif (fd < 0)\n+\t\tgoto fail;\n+\n+\tif (fstat(fd, &st))\n+\t\tgoto fail;\n+\n+\tnew_size = st.st_size;\n+\tnew_mmap = mmap(NULL, new_size, PROT_READ, MAP_SHARED, fd, 0);\n+\tif (new_size <= 20 ||\n+\t hashcmp((unsigned char *)istate->mmap + old_size - 20,\n+\t\t (unsigned char *)new_mmap + new_size - 20)) {\n+\t\tif (new_mmap)\n+\t\t\tmunmap(new_mmap, new_size);\n+\t\tgoto fail;\n+\t}\n+\n+\t/* The memory barrier here matches index-helper.c:share_index. */\n+\t__sync_synchronize();\n+\n+\tmunmap(istate->mmap, istate->mmap_size);\n+\tistate->mmap = new_mmap;\n+\tistate->mmap_size = new_size;\n+\tistate->from_shm = 1;\n+\tclose(fd);\n+\treturn 0;\n+\n+fail:\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+\tpoke_daemon(istate, &st, 1);\n+\treturn -1;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1555,6 +1648,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tif (istate->initialized)\n \t\treturn istate->cache_nr;\n \n+\tistate->from_shm = 0;\n \tistate->timestamp.sec = 0;\n \tistate->timestamp.nsec = 0;\n \tfd = open(path, O_RDONLY);\n@@ -1574,15 +1668,17 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tmmap = xmmap(NULL, mmap_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n-\tif (istate->keep_mmap) {\n-\t\tistate->mmap = mmap;\n-\t\tistate->mmap_size = mmap_size;\n-\t}\n \tclose(fd);\n \n-\thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n+\tistate->mmap = mmap;\n+\tistate->mmap_size = mmap_size;\n+\tif (try_shm(istate) &&\n+\t verify_hdr(istate->mmap, istate->mmap_size) < 0)\n \t\tgoto unmap;\n+\thdr = mmap = istate->mmap;\n+\tmmap_size = istate->mmap_size;\n+\tif (!istate->keep_mmap)\n+\t\tistate->mmap = NULL;\n \n \thashcpy(istate->sha1, (const unsigned char *)hdr + mmap_size - 20);\n \tistate->version = ntohl(hdr->hdr_version);\n@@ -1662,6 +1758,8 @@ int read_index_from(struct index_state *istate, const char *path)\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n \tsplit_index->base->keep_mmap = istate->keep_mmap;\n+\tsplit_index->base->to_shm = istate->to_shm;\n+\tsplit_index->base->from_shm = istate->from_shm;\n \tret = do_read_index(split_index->base,\n \t\t\t git_path(\"sharedindex.%s\",\n \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n@@ -1712,6 +1810,8 @@ int discard_index(struct index_state *istate)\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n+\tistate->from_shm = 0;\n+\tistate->to_shm = 0;\n \treturn 0;\n }\n \n@@ -2138,9 +2238,14 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \t\treturn ret;\n \tassert((flags & (COMMIT_LOCK | CLOSE_LOCK)) !=\n \t (COMMIT_LOCK | CLOSE_LOCK));\n-\tif (flags & COMMIT_LOCK)\n-\t\treturn commit_locked_index(lock);\n-\telse if (flags & CLOSE_LOCK)\n+\tif (flags & COMMIT_LOCK) {\n+\t\tstruct stat st;\n+\t\tret = commit_locked_index(lock);\n+\t\tif (!ret && is_main_index(istate) &&\n+\t\t !stat(git_path(\"index-helper.sock\"), &st))\n+\t\t\tpoke_daemon(istate, &st, 1);\n+\t\treturn ret;\n+\t} else if (flags & CLOSE_LOCK)\n \t\treturn close_lock_file(lock);\n \telse\n \t\treturn ret;\ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nnew file mode 100755\nindex 0000000..114c112\n--- /dev/null\n+++ b/t/t7900-index-helper.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2016, Twitter, Inc\n+#\n+\n+test_description='git-index-helper\n+\n+Testing git index-helper\n+'\n+\n+. ./test-lib.sh\n+\n+test -n \"$NO_MMAP\" && {\n+\tskip_all='skipping index-helper tests: no mmap'\n+\ttest_done\n+}\n+\n+test_expect_success 'index-helper smoke test' '\n+\tgit index-helper --exit-after 1 &&\n+\ttest_path_is_missing .git/index-helper.sock\n+'\n+\n+test_done\n-- \n1.9.1\n\n"},{"id":"290155","messageId":"1466914464-10358-1-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":null,"subject":"[PATCH v13 01/20] read-cache.c: fix constness of verify_hdr()","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:05Z","receivedAt":"2016-06-26T04:14:50Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n read-cache.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex d9fb78b..16cc487 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1345,7 +1345,7 @@ struct ondisk_cache_entry_extended {\n \t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr(struct cache_header *hdr, unsigned long size)\n+static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n {\n \tgit_SHA_CTX c;\n \tunsigned char sha1[20];\n-- \n1.9.1\n\n"},{"id":"290156","messageId":"1466914464-10358-3-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 03/20] pkt-line: add gentle version of packet_write","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:07Z","receivedAt":"2016-06-26T04:14:52Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner \n\npacket_write calls write_or_die, which dies with a sigpipe even if\ncalling code has explicitly blocked that signal.\n\nAdd packet_write_gently and packet_flush_gently, which don't. Soon,\nwe will use this for communication with git index-helper, which, being\nmerely an optimization, should be permitted to die without disrupting\nclients.\n\nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n pkt-line.c | 18 ++++++++++++++++++\n pkt-line.h | 2 ++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 62fdb37..f964446 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -91,6 +91,12 @@ void packet_flush(int fd)\n \twrite_or_die(fd, \"0000\", 4);\n }\n \n+int packet_flush_gently(int fd)\n+{\n+\tpacket_trace(\"0000\", 4, 1);\n+\treturn write_in_full(fd, \"0000\", 4) != 4;\n+}\n+\n void packet_buf_flush(struct strbuf *buf)\n {\n \tpacket_trace(\"0000\", 4, 1);\n@@ -130,6 +136,18 @@ void packet_write(int fd, const char *fmt, ...)\n \twrite_or_die(fd, buf.buf, buf.len);\n }\n \n+int packet_write_gently(int fd, const char *fmt, ...)\n+{\n+\tstatic struct strbuf buf = STRBUF_INIT;\n+\tva_list args;\n+\n+\tstrbuf_reset(&buf);\n+\tva_start(args, fmt);\n+\tformat_packet(&buf, fmt, args);\n+\tva_end(args);\n+\treturn write_in_full(fd, buf.buf, buf.len) != buf.len;\n+}\n+\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n {\n \tva_list args;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3cb9d91..deffcb5 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -20,7 +20,9 @@\n * side can't, we stay with pure read/write interfaces.\n */\n void packet_flush(int fd);\n+int packet_flush_gently(int fd);\n void packet_write(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+int packet_write_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n \n-- \n1.9.1\n\n"},{"id":"290157","messageId":"1466914464-10358-2-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 02/20] read-cache: allow to keep mmap'd memory after reading","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:06Z","receivedAt":"2016-06-26T04:14:53Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nLater, we will introduce git index-helper to share this memory with\nother git processes.\n\nWe only unmap it when we discard the index (although the kernel may of\ncourse choose to page it out).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n cache.h | 3 +++\n read-cache.c | 13 ++++++++++++-\n 2 files changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex b829410..4180e2b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -333,11 +333,14 @@ struct index_state {\n \tstruct split_index *split_index;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n+\t\t keep_mmap : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n \tstruct untracked_cache *untracked;\n+\tvoid *mmap;\n+\tsize_t mmap_size;\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 16cc487..3cb0ec3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1574,6 +1574,10 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tmmap = xmmap(NULL, mmap_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n+\tif (istate->keep_mmap) {\n+\t\tistate->mmap = mmap;\n+\t\tistate->mmap_size = mmap_size;\n+\t}\n \tclose(fd);\n \n \thdr = mmap;\n@@ -1626,11 +1630,13 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n-\tmunmap(mmap, mmap_size);\n+\tif (!istate->keep_mmap)\n+\t\tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n unmap:\n \tmunmap(mmap, mmap_size);\n+\tistate->mmap = NULL;\n \tdie(\"index file corrupt\");\n }\n \n@@ -1655,6 +1661,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdiscard_index(split_index->base);\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n+\tsplit_index->base->keep_mmap = istate->keep_mmap;\n \tret = do_read_index(split_index->base,\n \t\t\t git_path(\"sharedindex.%s\",\n \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n@@ -1698,6 +1705,10 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->keep_mmap && istate->mmap) {\n+\t\tmunmap(istate->mmap, istate->mmap_size);\n+\t\tistate->mmap = NULL;\n+\t}\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n-- \n1.9.1\n\n"},{"id":"290158","messageId":"1466914464-10358-6-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 06/20] daemonize(): set a flag before exiting the main process","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:10Z","receivedAt":"2016-06-26T04:14:57Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThis allows signal handlers and atexit functions to realize this\nsituation and not clean up.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n builtin/gc.c | 2 +-\n cache.h | 2 +-\n daemon.c | 2 +-\n setup.c | 4 +++-\n 4 files changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c583aad..37180de 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -385,7 +385,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t * failure to daemonize is ok, we'll continue\n \t\t\t * in foreground\n \t\t\t */\n-\t\t\tdaemonized = !daemonize();\n+\t\t\tdaemonized = !daemonize(NULL);\n \t\t}\n \t} else\n \t\tadd_repack_all_option();\ndiff --git a/cache.h b/cache.h\nindex 6cb0d02..4c1529a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -539,7 +539,7 @@ extern int set_git_dir_init(const char *git_dir, const char *real_git_dir, int);\n extern int init_db(const char *template_dir, unsigned int flags);\n \n extern void sanitize_stdfds(void);\n-extern int daemonize(void);\n+extern int daemonize(int *);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \ndiff --git a/daemon.c b/daemon.c\nindex 8d45c33..a5cf954 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -1365,7 +1365,7 @@ int main(int argc, char **argv)\n \t\treturn execute();\n \n \tif (detach) {\n-\t\tif (daemonize())\n+\t\tif (daemonize(NULL))\n \t\t\tdie(\"--detach not supported on this platform\");\n \t} else\n \t\tsanitize_stdfds();\ndiff --git a/setup.c b/setup.c\nindex de1a2a7..9adf13f 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1017,7 +1017,7 @@ void sanitize_stdfds(void)\n \t\tclose(fd);\n }\n \n-int daemonize(void)\n+int daemonize(int *daemonized)\n {\n #ifdef NO_POSIX_GOODIES\n \terrno = ENOSYS;\n@@ -1029,6 +1029,8 @@ int daemonize(void)\n \t\tcase -1:\n \t\t\tdie_errno(\"fork failed\");\n \t\tdefault:\n+\t\t\tif (daemonized)\n+\t\t\t\t*daemonized = 1;\n \t\t\texit(0);\n \t}\n \tif (setsid() == -1)\n-- \n1.9.1\n\n"},{"id":"290159","messageId":"1466914464-10358-5-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 05/20] index-helper: add --strict","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:09Z","receivedAt":"2016-06-26T04:15:00Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThere are \"holes\" in the index-helper approach because the shared\nmemory is not verified again by git. If $USER is compromised, shared\nmemory could be modified. But anyone who could do this could already\nmodify $GIT_DIR/index. A more realistic risk is some bugs in\nindex-helper that produce corrupt shared memory. --strict is added to\navoid that.\n\nStrictly speaking there's still a very small gap where corrupt shared\nmemory could still be read by git: after we write the trailing SHA-1 in\nthe shared memory (thus signaling \"this shm is ready\") and before\nverify_shm() detects an error.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-index-helper.txt | 9 +++++++\n cache.h | 1 +\n index-helper.c | 48 ++++++++++++++++++++++++++++++++++++++\n read-cache.c | 9 ++++---\n 4 files changed, 64 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex fa6e347..ca5a9de 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -25,6 +25,15 @@ OPTIONS\n \tExit if the cached index is not accessed for ``\n \tseconds. Specify 0 to wait forever. Default is 600.\n \n+--strict::\n+--no-strict::\n+\tStrict mode makes index-helper verify the shared memory after\n+\tit's created. If the result does not match what's read from\n+\t$GIT_DIR/index, the shared memory is destroyed. This makes\n+\tindex-helper take more than double the amount of time required\n+\tfor reading an index, but because it will happen in the\n+\tbackground, it's not noticable. `--strict` is enabled by default.\n+\n NOTES\n -----\n \ndiff --git a/cache.h b/cache.h\nindex 2d7af6f..6cb0d02 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -345,6 +345,7 @@ struct index_state {\n \t\t * on it.\n \t\t */\n \t\t to_shm : 1,\n+\t\t always_verify_trailing_sha1 : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\ndiff --git a/index-helper.c b/index-helper.c\nindex baf5bc6..d1e9287 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -17,6 +17,7 @@ struct shm {\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n+static int to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n {\n@@ -122,11 +123,56 @@ static void share_index(struct index_state *istate, struct shm *is)\n \thashcpy((unsigned char *)new_mmap + istate->mmap_size - 20, is->sha1);\n }\n \n+static int verify_shm(void)\n+{\n+\tint i;\n+\tstruct index_state istate;\n+\tmemset(&istate, 0, sizeof(istate));\n+\tistate.always_verify_trailing_sha1 = 1;\n+\tistate.to_shm = 1;\n+\ti = read_index(&istate);\n+\tif (i != the_index.cache_nr)\n+\t\tgoto done;\n+\tfor (; i < the_index.cache_nr; i++) {\n+\t\tstruct cache_entry *base, *ce;\n+\t\t/* namelen is checked separately */\n+\t\tconst unsigned int ondisk_flags =\n+\t\t\tCE_STAGEMASK | CE_VALID | CE_EXTENDED_FLAGS;\n+\t\tunsigned int ce_flags, base_flags, ret;\n+\t\tbase = the_index.cache[i];\n+\t\tce = istate.cache[i];\n+\t\tif (ce->ce_namelen != base->ce_namelen ||\n+\t\t strcmp(ce->name, base->name)) {\n+\t\t\twarning(\"mismatch at entry %d\", i);\n+\t\t\tbreak;\n+\t\t}\n+\t\tce_flags = ce->ce_flags;\n+\t\tbase_flags = base->ce_flags;\n+\t\t/* only on-disk flags matter */\n+\t\tce->ce_flags &= ondisk_flags;\n+\t\tbase->ce_flags &= ondisk_flags;\n+\t\tret = memcmp(&ce->ce_stat_data, &base->ce_stat_data,\n+\t\t\t offsetof(struct cache_entry, name) -\n+\t\t\t offsetof(struct cache_entry, ce_stat_data));\n+\t\tce->ce_flags = ce_flags;\n+\t\tbase->ce_flags = base_flags;\n+\t\tif (ret) {\n+\t\t\twarning(\"mismatch at entry %d\", i);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+done:\n+\tdiscard_index(&istate);\n+\treturn i == the_index.cache_nr;\n+}\n+\n static void share_the_index(void)\n {\n \tif (the_index.split_index && the_index.split_index->base)\n \t\tshare_index(the_index.split_index->base, &shm_base_index);\n \tshare_index(&the_index, &shm_index);\n+\tif (to_verify && !verify_shm())\n+\t\tcleanup_shm();\n \tdiscard_index(&the_index);\n }\n \n@@ -258,6 +304,8 @@ int main(int argc, char **argv)\n \tstruct option options[] = {\n \t\tOPT_INTEGER(0, \"exit-after\", &idle_in_seconds,\n \t\t\t N_(\"exit if not used after some seconds\")),\n+\t\tOPT_BOOL(0, \"strict\", &to_verify,\n+\t\t\t N_(\"verify shared memory after creating\")),\n \t\tOPT_END()\n \t};\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 10d5465..befc499 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1672,9 +1672,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \n \tistate->mmap = mmap;\n \tistate->mmap_size = mmap_size;\n-\tif (try_shm(istate) &&\n-\t verify_hdr(istate->mmap, istate->mmap_size) < 0)\n-\t\tgoto unmap;\n+\tif (try_shm(istate)) {\n+\t\tif (verify_hdr(istate->mmap, istate->mmap_size) < 0)\n+\t\t\tgoto unmap;\n+\t} else if (istate->always_verify_trailing_sha1 &&\n+\t\t verify_hdr(istate->mmap, istate->mmap_size) < 0)\n+\t\t\tgoto unmap;\n \thdr = mmap = istate->mmap;\n \tmmap_size = istate->mmap_size;\n \tif (!istate->keep_mmap)\n-- \n1.9.1\n\n"},{"id":"290160","messageId":"1466914464-10358-18-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 18/20] index-helper: optionally automatically run","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:22Z","receivedAt":"2016-06-26T04:15:06Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner \n\nIntroduce a new config option, indexhelper.autorun, to automatically\nrun git index-helper before starting up a builtin git command. This\nenables users to keep index-helper running without manual\nintervention.\n\nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Documentation/config.txt | 4 ++++\n read-cache.c | 35 +++++++++++++++++++++++++++++++++--\n t/t7900-index-helper.sh | 20 ++++++++++++++++++++\n 3 files changed, 57 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 15001ce..385ea66 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1856,6 +1856,10 @@ index.version::\n \tSpecify the version with which new index files should be\n \tinitialized. This does not affect existing repositories.\n \n+indexhelper.autorun::\n+\tAutomatically run git index-helper when any builtin git\n+\tcommand is run inside a repository.\n+\n init.templateDir::\n \tSpecify the directory from which templates will be copied.\n \t(See the \"TEMPLATE DIRECTORY\" section of linkgit:git-init[1].)\ndiff --git a/read-cache.c b/read-cache.c\nindex 8141559..fde778b 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -22,6 +22,7 @@\n #include \"pkt-line.h\"\n #include \"sigchain.h\"\n #include \"ewah/ewok.h\"\n+#include \"run-command.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -1733,6 +1734,31 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_untracked_cache(istate);\n }\n \n+static int want_auto_index_helper(void)\n+{\n+\tint want = -1;\n+\tconst char *value = NULL;\n+\tconst char *key = \"indexhelper.autorun\";\n+\n+\tif (git_config_key_is_valid(key) &&\n+\t !git_config_get_value(key, &value)) {\n+\t\tint b = git_config_maybe_bool(key, value);\n+\t\twant = b >= 0 ? b : 0;\n+\t}\n+\treturn want;\n+}\n+\n+static void autorun_index_helper(void)\n+{\n+\tconst char *argv[] = {\"git-index-helper\", \"--detach\", \"--autorun\", NULL};\n+\tif (want_auto_index_helper() <= 0)\n+\t\treturn;\n+\n+\tif (run_command_v_opt(argv,\n+\t\t\t RUN_SILENT_EXEC_FAILURE | RUN_CLEAN_ON_EXIT))\n+\t\twarning(_(\"You specified indexhelper.autorun, but running git-index-helper failed.\"));\n+}\n+\n /* in ms */\n #define WATCHMAN_TIMEOUT 1000\n \n@@ -1804,6 +1830,7 @@ static int poke_daemon(struct index_state *istate,\n \tif (fd < 0) {\n \t\twarning(\"Failed to connect to index-helper socket\");\n \t\tunlink(git_path(\"index-helper.sock\"));\n+\t\tautorun_index_helper();\n \t\treturn -1;\n \t}\n \tsigchain_push(SIGPIPE, SIG_IGN);\n@@ -1843,9 +1870,13 @@ static int try_shm(struct index_state *istate)\n \tint fd = -1;\n \n \tif (!is_main_index(istate) ||\n-\t old_size <= 20 ||\n-\t stat(git_path(\"index-helper.sock\"), &st))\n+\t old_size <= 20)\n \t\treturn -1;\n+\n+\tif (stat(git_path(\"index-helper.sock\"), &st)) {\n+\t\tautorun_index_helper();\n+\t\treturn -1;\n+\t}\n \tif (poke_daemon(istate, &st, 0))\n \t\treturn -1;\n \tsha1 = (unsigned char *)istate->mmap + old_size - 20;\ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex aa6e5fc..3cfdf63 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -16,6 +16,9 @@ test -n \"$NO_MMAP\" && {\n }\n \n test_expect_success 'index-helper smoke test' '\n+\t# We need an existing commit so that the index exists (otherwise,\n+\t# the index-helper will not be autostarted)\n+\ttest_commit x &&\n \tgit index-helper --exit-after 1 &&\n \ttest_path_is_missing .git/index-helper.sock\n '\n@@ -46,4 +49,21 @@ test_expect_success 'index-helper is quiet with --autorun' '\n \tgit index-helper --autorun\n '\n \n+test_expect_success 'index-helper autorun works' '\n+\ttest_when_finished \"git index-helper --kill\" &&\n+\trm -f .git/index-helper.sock &&\n+\tgit status &&\n+\ttest_path_is_missing .git/index-helper.sock &&\n+\ttest_config indexhelper.autorun true &&\n+\tgit status &&\n+\ttest -S .git/index-helper.sock &&\n+\tgit status 2>err &&\n+\ttest -S .git/index-helper.sock &&\n+\ttest_must_be_empty err &&\n+\tgit index-helper --kill &&\n+\ttest_config indexhelper.autorun false &&\n+\tgit status &&\n+\ttest_path_is_missing .git/index-helper.sock\n+'\n+\n test_done\n-- \n1.9.1\n\n"},{"id":"290161","messageId":"1466914464-10358-17-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 17/20] index-helper: autorun mode","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:21Z","receivedAt":"2016-06-26T04:15:09Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner \n\nSoon, we'll want to automatically start index-helper, so we need\na mode that silently exits if it can't start up (either because\nit's not in a git dir, or because another one is already running).\n\nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-index-helper.txt | 4 ++++\n index-helper.c | 29 +++++++++++++++++++++++------\n t/t7900-index-helper.sh | 8 ++++++++\n 3 files changed, 35 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex 6f63a9e..eea5a38 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -43,6 +43,10 @@ OPTIONS\n --kill::\n \tKill any running index-helper and clean up the socket\n \n+--autorun::\n+\tIf index-helper is not already running, start it. Else, do\n+\tnothing.\n+\n NOTES\n -----\n \ndiff --git a/index-helper.c b/index-helper.c\nindex 9cfcb9e..f57fc14 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -415,8 +415,9 @@ static void request_kill(void)\n int main(int argc, char **argv)\n {\n \tconst char *prefix;\n-\tint idle_in_seconds = 600, detach = 0, kill = 0;\n+\tint idle_in_seconds = 600, detach = 0, kill = 0, autorun = 0;\n \tint fd;\n+\tint nongit;\n \tstruct strbuf socket_path = STRBUF_INIT;\n \tstruct option options[] = {\n \t\tOPT_INTEGER(0, \"exit-after\", &idle_in_seconds,\n@@ -425,6 +426,7 @@ int main(int argc, char **argv)\n \t\t\t N_(\"verify shared memory after creating\")),\n \t\tOPT_BOOL(0, \"detach\", &detach, N_(\"detach the process\")),\n \t\tOPT_BOOL(0, \"kill\", &kill, N_(\"request that existing index helper processes exit\")),\n+\t\tOPT_BOOL(0, \"autorun\", &autorun, N_(\"this is an automatic run of git index-helper, so certain errors can be solved by silently exiting\")),\n \t\tOPT_END()\n \t};\n \n@@ -434,7 +436,14 @@ int main(int argc, char **argv)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(usage_text, options);\n \n-\tprefix = setup_git_directory();\n+\tprefix = setup_git_directory_gently(&nongit);\n+\tif (nongit) {\n+\t\tif (autorun)\n+\t\t\texit(0);\n+\t\telse\n+\t\t\tdie(_(\"not a git repository\"));\n+\t}\n+\n \tif (parse_options(argc, (const char **)argv, prefix,\n \t\t\t options, usage_text, 0))\n \t\tdie(_(\"too many arguments\"));\n@@ -448,10 +457,18 @@ int main(int argc, char **argv)\n \n \t/* check that no other copy is running */\n \tfd = unix_stream_connect(git_path(\"index-helper.sock\"));\n-\tif (fd >= 0)\n-\t\tdie(_(\"Already running\"));\n-\tif (errno != ECONNREFUSED && errno != ENOENT)\n-\t\tdie_errno(_(\"Unexpected error checking socket\"));\n+\tif (fd >= 0) {\n+\t\tif (autorun)\n+\t\t\texit(0);\n+\t\telse\n+\t\t\tdie(_(\"Already running\"));\n+\t}\n+\tif (errno != ECONNREFUSED && errno != ENOENT) {\n+\t\tif (autorun)\n+\t\t\treturn 0;\n+\t\telse\n+\t\t\tdie_errno(_(\"Unexpected error checking socket\"));\n+\t}\n \n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex 7159971..aa6e5fc 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -38,4 +38,12 @@ test_expect_success 'index-helper will not start if already running' '\n \tgrep \"Already running\" err\n '\n \n+test_expect_success 'index-helper is quiet with --autorun' '\n+\ttest_when_finished \"git index-helper --kill\" &&\n+\tgit index-helper --kill &&\n+\tgit index-helper --detach &&\n+\ttest -S .git/index-helper.sock &&\n+\tgit index-helper --autorun\n+'\n+\n test_done\n-- \n1.9.1\n\n"},{"id":"290162","messageId":"1466914464-10358-15-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 15/20] index-helper: kill mode","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:19Z","receivedAt":"2016-06-26T04:15:12Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner \n\nAdd a new command (and command-line arg) to allow index-helpers to\nexit cleanly.\n\nThis is mainly useful for tests.\n\nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-index-helper.txt | 3 +++\n index-helper.c | 31 ++++++++++++++++++++++++++++++-\n t/t7900-index-helper.sh | 9 +++++++++\n 3 files changed, 42 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex b2ca511..6f63a9e 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -40,6 +40,9 @@ OPTIONS\n --detach::\n \tDetach from the shell.\n \n+--kill::\n+\tKill any running index-helper and clean up the socket\n+\n NOTES\n -----\n \ndiff --git a/index-helper.c b/index-helper.c\nindex bb39326..f7a002b 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -361,6 +361,8 @@ static void loop(int fd, int idle_in_seconds)\n \t\t\t\t\t * alive, nothing to do.\n \t\t\t\t\t */\n \t\t\t\t}\n+\t\t\t} else if (!strcmp(buf, \"die\")) {\n+\t\t\t\tbreak;\n \t\t\t} else {\n \t\t\t\twarning(\"BUG: Bogus command %s\", buf);\n \t\t\t}\n@@ -391,10 +393,29 @@ static const char * const usage_text[] = {\n \tNULL\n };\n \n+static void request_kill(void)\n+{\n+\tint fd = unix_stream_connect(git_path(\"index-helper.sock\"));\n+\n+\tif (fd >= 0) {\n+\t\twrite_in_full(fd, \"die\", 4);\n+\t\tclose(fd);\n+\t}\n+\n+\t/*\n+\t * The child will try to do this anyway, but we want to be\n+\t * ready to launch a new daemon immediately after this command\n+\t * returns.\n+\t */\n+\n+\tunlink(git_path(\"index-helper.sock\"));\n+\treturn;\n+}\n+\n int main(int argc, char **argv)\n {\n \tconst char *prefix;\n-\tint idle_in_seconds = 600, detach = 0;\n+\tint idle_in_seconds = 600, detach = 0, kill = 0;\n \tint fd;\n \tstruct strbuf socket_path = STRBUF_INIT;\n \tstruct option options[] = {\n@@ -403,6 +424,7 @@ int main(int argc, char **argv)\n \t\tOPT_BOOL(0, \"strict\", &to_verify,\n \t\t\t N_(\"verify shared memory after creating\")),\n \t\tOPT_BOOL(0, \"detach\", &detach, N_(\"detach the process\")),\n+\t\tOPT_BOOL(0, \"kill\", &kill, N_(\"request that existing index helper processes exit\")),\n \t\tOPT_END()\n \t};\n \n@@ -417,6 +439,13 @@ int main(int argc, char **argv)\n \t\t\t options, usage_text, 0))\n \t\tdie(_(\"too many arguments\"));\n \n+\tif (kill) {\n+\t\tif (detach)\n+\t\t\tdie(_(\"--kill doesn't want any other options\"));\n+\t\trequest_kill();\n+\t\treturn 0;\n+\t}\n+\n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n \ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex 114c112..e71b5af 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -20,4 +20,13 @@ test_expect_success 'index-helper smoke test' '\n \ttest_path_is_missing .git/index-helper.sock\n '\n \n+test_expect_success 'index-helper creates usable path file and can be killed' '\n+\ttest_when_finished \"git index-helper --kill\" &&\n+\ttest_path_is_missing .git/index-helper.sock &&\n+\tgit index-helper --detach &&\n+\ttest -S .git/index-helper.sock &&\n+\tgit index-helper --kill &&\n+\ttest_path_is_missing .git/index-helper.sock\n+'\n+\n test_done\n-- \n1.9.1\n\n"},{"id":"290163","messageId":"1466914464-10358-14-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 14/20] watchman: add a config option to enable the extension","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:18Z","receivedAt":"2016-06-26T04:15:15Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner \n\nFor installations that have centrally-managed configuration, it's\neasier to set a config once than to run update-index on every\nrepository.\n\nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n .gitignore | 1 +\n Documentation/config.txt | 4 ++++\n Makefile | 1 +\n read-cache.c | 6 ++++++\n t/t1701-watchman-extension.sh | 37 +++++++++++++++++++++++++++++++++++++\n test-dump-watchman.c | 16 ++++++++++++++++\n 6 files changed, 65 insertions(+)\n create mode 100755 t/t1701-watchman-extension.sh\n create mode 100644 test-dump-watchman.c\n\ndiff --git a/.gitignore b/.gitignore\nindex b92f122..e6a5b2c 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -188,6 +188,7 @@\n /test-dump-cache-tree\n /test-dump-split-index\n /test-dump-untracked-cache\n+/test-dump-watchman\n /test-fake-ssh\n /test-scrap-cache-tree\n /test-genrandom\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 2cd6bdd..15001ce 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1848,6 +1848,10 @@ imap::\n \tThe configuration variables in the 'imap' section are described\n \tin linkgit:git-imap-send[1].\n \n+index.addwatchmanextension::\n+\tAutomatically add the watchman extension to the index whenever\n+\tit is written.\n+\n index.version::\n \tSpecify the version with which new index files should be\n \tinitialized. This does not affect existing repositories.\ndiff --git a/Makefile b/Makefile\nindex 7f4ab41..25ab7b4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -599,6 +599,7 @@ TEST_PROGRAMS_NEED_X += test-delta\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n TEST_PROGRAMS_NEED_X += test-dump-split-index\n TEST_PROGRAMS_NEED_X += test-dump-untracked-cache\n+TEST_PROGRAMS_NEED_X += test-dump-watchman\n TEST_PROGRAMS_NEED_X += test-fake-ssh\n TEST_PROGRAMS_NEED_X += test-genrandom\n TEST_PROGRAMS_NEED_X += test-hashmap\ndiff --git a/read-cache.c b/read-cache.c\nindex bc3c989..8141559 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2436,6 +2436,7 @@ static int do_write_index(struct index_state *istate, int newfd,\n \tint entries = istate->cache_nr;\n \tstruct stat st;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n+\tint watchman = 0;\n \n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n@@ -2459,6 +2460,11 @@ static int do_write_index(struct index_state *istate, int newfd,\n \tif (istate->version == 3 || istate->version == 2)\n \t\tistate->version = extended ? 3 : 2;\n \n+\tif (!git_config_get_bool(\"index.addwatchmanextension\", &watchman) &&\n+\t watchman &&\n+\t !the_index.last_update)\n+\t\tthe_index.last_update = xstrdup(\"\");\n+\n \thdr_version = istate->version;\n \n \thdr.hdr_signature = htonl(CACHE_SIGNATURE);\ndiff --git a/t/t1701-watchman-extension.sh b/t/t1701-watchman-extension.sh\nnew file mode 100755\nindex 0000000..71f1d46\n--- /dev/null\n+++ b/t/t1701-watchman-extension.sh\n@@ -0,0 +1,37 @@\n+#!/bin/sh\n+\n+test_description='watchman extension smoke tests'\n+\n+# These don't actually test watchman interaction -- just the\n+# index extension\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'enable watchman' '\n+\ttest_commit a &&\n+\ttest-dump-watchman .git/index >actual &&\n+\techo \"last_update: (null)\" >expect &&\n+\ttest_cmp expect actual &&\n+\tgit update-index --watchman &&\n+\ttest-dump-watchman .git/index >actual &&\n+\techo \"last_update: \" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'disable watchman' '\n+\tgit update-index --no-watchman &&\n+\ttest-dump-watchman .git/index >actual &&\n+\techo \"last_update: (null)\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'auto-enable watchman' '\n+\ttest_config index.addwatchmanextension true &&\n+\ttest_commit c &&\n+\ttest-dump-watchman .git/index >actual &&\n+\techo \"last_update: \" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_done\ndiff --git a/test-dump-watchman.c b/test-dump-watchman.c\nnew file mode 100644\nindex 0000000..0314fa5\n--- /dev/null\n+++ b/test-dump-watchman.c\n@@ -0,0 +1,16 @@\n+#include \"cache.h\"\n+#include \"ewah/ewok.h\"\n+\n+int main(int argc, char **argv)\n+{\n+\tdo_read_index(&the_index, argv[1], 1);\n+\tprintf(\"last_update: %s\\n\", the_index.last_update ?\n+\t the_index.last_update : \"(null)\");\n+\n+\t/*\n+\t * For now, we just dump last_update, since it is not reasonable\n+\t * to populate the extension itself in tests.\n+\t */\n+\n+\treturn 0;\n+}\n-- \n1.9.1\n\n"},{"id":"290164","messageId":"1466914464-10358-13-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 13/20] unpack-trees: preserve index extensions","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:17Z","receivedAt":"2016-06-26T04:15:19Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner \n\nMake git checkout (and other unpack_tree operations) preserve the\nuntracked cache and watchman status. This is valuable for two reasons:\n\n1. Often, an unpack_tree operation will not touch large parts of the\nworking tree, and thus most of the untracked cache will continue to be\nvalid.\n\n2. Even if the untracked cache were entirely invalidated by such an\noperation, the user has signaled their intention to have such a cache,\nand we don't want to throw it away.\n\nThe same logic applies to the watchman state.\n\nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n cache.h | 1 +\n read-cache.c | 8 ++++++++\n t/t7063-status-untracked-cache.sh | 22 ++++++++++++++++++++++\n t/test-lib-functions.sh | 4 ++++\n unpack-trees.c | 1 +\n 5 files changed, 36 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 633e1dd..1b372ed 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -580,6 +580,7 @@ extern void write_watchman_ext(struct strbuf *sb, struct index_state *istate);\n #define CLOSE_LOCK\t\t(1 << 1)\n extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\n extern int discard_index(struct index_state *);\n+extern void move_index_extensions(struct index_state *dst, struct index_state *src);\n extern int unmerged_index(const struct index_state *);\n extern int verify_path(const char *path);\n extern int index_dir_exists(struct index_state *istate, const char *name, int namelen);\ndiff --git a/read-cache.c b/read-cache.c\nindex 8521e85..bc3c989 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2777,3 +2777,11 @@ void stat_validity_update(struct stat_validity *sv, int fd)\n \t\tfill_stat_data(sv->sd, &st);\n \t}\n }\n+\n+void move_index_extensions(struct index_state *dst, struct index_state *src)\n+{\n+\tdst->untracked = src->untracked;\n+\tsrc->untracked = NULL;\n+\tdst->last_update = src->last_update;\n+\tsrc->last_update = NULL;\n+}\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex a971884..083516d 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -646,4 +646,26 @@ test_expect_success 'test ident field is working' '\n \ttest_cmp ../expect ../err\n '\n \n+test_expect_success 'untracked cache survives a checkout' '\n+\tgit commit --allow-empty -m empty &&\n+\ttest-dump-untracked-cache >../before &&\n+\ttest_when_finished \"git checkout master\" &&\n+\tgit checkout -b other_branch &&\n+\ttest-dump-untracked-cache >../after &&\n+\ttest_cmp ../before ../after &&\n+\ttest_commit test &&\n+\ttest-dump-untracked-cache >../before &&\n+\tgit checkout master &&\n+\ttest-dump-untracked-cache >../after &&\n+\ttest_cmp ../before ../after\n+'\n+\n+test_expect_success 'untracked cache survives a commit' '\n+\ttest-dump-untracked-cache >../before &&\n+\tgit add done/two &&\n+\tgit commit -m commit &&\n+\ttest-dump-untracked-cache >../after &&\n+\ttest_cmp ../before ../after\n+'\n+\n test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 8d99eb3..e974b5b 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -186,6 +186,10 @@ test_commit () {\n \t\ttest_tick\n \tfi &&\n \tgit commit $signoff -m \"$1\" &&\n+\tif [ \"$(git config core.bare)\" = false ]\n+\tthen\n+\t git update-index --force-untracked-cache\n+\tfi\n \tgit tag \"${4:-$1}\"\n }\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 9f55cc2..fc90eb3 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1215,6 +1215,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t\t\t\t\t WRITE_TREE_SILENT |\n \t\t\t\t\t\t WRITE_TREE_REPAIR);\n \t\t}\n+\t\tmove_index_extensions(&o->result, o->dst_index);\n \t\tdiscard_index(o->dst_index);\n \t\t*o->dst_index = o->result;\n \t} else {\n-- \n1.9.1\n\n"},{"id":"290165","messageId":"1466914464-10358-16-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 16/20] index-helper: don't run if already running","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:20Z","receivedAt":"2016-06-26T04:15:25Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner \n\nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n index-helper.c | 7 +++++++\n t/t7900-index-helper.sh | 9 +++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/index-helper.c b/index-helper.c\nindex f7a002b..9cfcb9e 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -446,6 +446,13 @@ int main(int argc, char **argv)\n \t\treturn 0;\n \t}\n \n+\t/* check that no other copy is running */\n+\tfd = unix_stream_connect(git_path(\"index-helper.sock\"));\n+\tif (fd >= 0)\n+\t\tdie(_(\"Already running\"));\n+\tif (errno != ECONNREFUSED && errno != ENOENT)\n+\t\tdie_errno(_(\"Unexpected error checking socket\"));\n+\n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n \ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex e71b5af..7159971 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -29,4 +29,13 @@ test_expect_success 'index-helper creates usable path file and can be killed' '\n \ttest_path_is_missing .git/index-helper.sock\n '\n \n+test_expect_success 'index-helper will not start if already running' '\n+\ttest_when_finished \"git index-helper --kill\" &&\n+\tgit index-helper --detach &&\n+\ttest -S .git/index-helper.sock &&\n+\ttest_must_fail git index-helper 2>err &&\n+\ttest -S .git/index-helper.sock &&\n+\tgrep \"Already running\" err\n+'\n+\n test_done\n-- \n1.9.1\n\n"},{"id":"290166","messageId":"1466914464-10358-12-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 12/20] update-index: enable/disable watchman support","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:16Z","receivedAt":"2016-06-26T04:15:28Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-index-helper.txt | 3 +++\n Documentation/git-update-index.txt | 6 ++++++\n builtin/update-index.c | 15 +++++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex 2982e03..b2ca511 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -18,6 +18,9 @@ each with a submodule, you might need four index-helpers. (In practice,\n this is only worthwhile for large indexes, so only use it if you notice\n that git status is slow).\n \n+If you want the index-helper to accelerate untracked file checking,\n+run git update-index --watchman before using it.\n+\n OPTIONS\n -------\n \ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex c6cbed1..6736487 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -19,6 +19,7 @@ SYNOPSIS\n \t [--ignore-submodules]\n \t [--[no-]split-index]\n \t [--[no-|test-|force-]untracked-cache]\n+\t [--[no-]watchman]\n \t [--really-refresh] [--unresolve] [--again | -g]\n \t [--info-only] [--index-info]\n \t [-z] [--stdin] [--index-version ]\n@@ -176,6 +177,11 @@ may not support it yet.\n --no-untracked-cache::\n \tEnable or disable untracked cache feature. Please use\n \t`--test-untracked-cache` before enabling it.\n+\n+--watchman::\n+--no-watchman::\n+\tEnable or disable watchman support. This is, at present,\n+\tonly useful with git index-helper.\n +\n These options take effect whatever the value of the `core.untrackedCache`\n configuration variable (see linkgit:git-config[1]). But a warning is\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 1c94ca5..55722b9 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -914,6 +914,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n {\n \tint newfd, entries, has_errors = 0, nul_term_line = 0;\n \tenum uc_mode untracked_cache = UC_UNSPECIFIED;\n+\tint use_watchman = -1;\n \tint read_from_stdin = 0;\n \tint prefix_length = prefix ? strlen(prefix) : 0;\n \tint preferred_index_format = 0;\n@@ -1012,6 +1013,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"test if the filesystem supports untracked cache\"), UC_TEST),\n \t\tOPT_SET_INT(0, \"force-untracked-cache\", &untracked_cache,\n \t\t\t N_(\"enable untracked cache without testing the filesystem\"), UC_FORCE),\n+\t\tOPT_BOOL(0, \"watchman\", &use_watchman,\n+\t\t\tN_(\"use or not use watchman to reduce refresh cost\")),\n \t\tOPT_END()\n \t};\n \n@@ -1149,6 +1152,18 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tdie(\"Bug: bad untracked_cache value: %d\", untracked_cache);\n \t}\n \n+\tif (use_watchman > 0) {\n+\t\tthe_index.last_update = xstrdup(\"\");\n+\t\tthe_index.cache_changed |= WATCHMAN_CHANGED;\n+#ifndef USE_WATCHMAN\n+\t\twarning(_(\"git was built without watchman support, so this \"\n+\t\t\t \"command will probably not result in any speedup.\"));\n+#endif\n+\t} else if (!use_watchman) {\n+\t\tthe_index.last_update = NULL;\n+\t\tthe_index.cache_changed |= WATCHMAN_CHANGED;\n+\t}\n+\n \tif (active_cache_changed) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n-- \n1.9.1\n\n"},{"id":"290167","messageId":"1466914464-10358-9-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 09/20] read-cache: add watchman 'WAMA' extension","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:13Z","receivedAt":"2016-06-26T04:15:52Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThe extension contains a bitmap, one bit for each entry in the\nindex. If the n-th bit is zero, the n-th entry is considered\nunchanged, we can ce_mark_uptodate() it without refreshing. If the bit\nis non-zero and we found out the corresponding file is clean after\nrefresh, we can clear the bit.\n\nIn addition, there's a list of directories in the untracked-cache\nto invalidate (because they have new or modified entries).\n\nThe 'skipping refresh' bit is not in this patch yet as we would need\nwatchman. More details in later patches.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Documentation/technical/index-format.txt | 22 ++++++\n cache.h | 4 +\n dir.h | 3 +\n read-cache.c | 127 ++++++++++++++++++++++++++++++-\n 4 files changed, 154 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex ade0b0c..86ed3a6 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -295,3 +295,25 @@ The remaining data of each directory block is grouped by type:\n in the previous ewah bitmap.\n \n - One NUL.\n+\n+== Watchman cache\n+\n+ The watchman cache tracks files for which watchman has told us about\n+ changes. The signature for this extension is { 'W', 'A', 'M', 'A' }.\n+\n+ The extension starts with\n+\n+ - A NUL-terminated string: the watchman vector clock at the last\n+ time we heard from watchman.\n+\n+ - 32-bit bitmap size: the size of the CE_WATCHMAN_DIRTY bitmap\n+\n+ - 32-bit untracked cache entry count: the number of dirty untracked\n+ cache entries\n+\n+ - An ewah bitmap, the n-th bit indicates whether the n-th index entry\n+ is CE_WATCHMAN_DIRTY.\n+\n+ - a list of N NUL-terminated strings. Each is a directory that should\n+ be marked dirty in the untracked cache because watchman has told us\n+ about an update to a file in it.\ndiff --git a/cache.h b/cache.h\nindex 4c1529a..f10992d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -182,6 +182,8 @@ struct cache_entry {\n #define CE_VALID (0x8000)\n #define CE_STAGESHIFT 12\n \n+#define CE_WATCHMAN_DIRTY (0x0001)\n+\n /*\n * Range 0xFFFF0FFF in ce_flags is divided into\n * two parts: in-memory flags and on-disk ones.\n@@ -320,6 +322,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n #define UNTRACKED_CHANGED\t(1 << 7)\n+#define WATCHMAN_CHANGED\t(1 << 8)\n \n struct split_index;\n struct untracked_cache;\n@@ -353,6 +356,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tvoid *mmap;\n \tsize_t mmap_size;\n+\tchar *last_update;\n };\n \n extern struct index_state the_index;\ndiff --git a/dir.h b/dir.h\nindex cd46f30..896b64a 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -139,6 +139,9 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n+\t/* watchman invalidation data */\n+\tunsigned int use_watchman : 1;\n+\tstruct string_list invalid_untracked;\n };\n \n struct dir_struct {\ndiff --git a/read-cache.c b/read-cache.c\nindex befc499..e0fc634 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -21,6 +21,7 @@\n #include \"unix-socket.h\"\n #include \"pkt-line.h\"\n #include \"sigchain.h\"\n+#include \"ewah/ewok.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -43,11 +44,13 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n #define CACHE_EXT_UNTRACKED 0x554E5452\t /* \"UNTR\" */\n+#define CACHE_EXT_WATCHMAN 0x57414D41\t /* \"WAMA\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | \\\n+\t\t WATCHMAN_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -1222,8 +1225,23 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\tcontinue;\n \n \t\tnew = refresh_cache_ent(istate, ce, options, &cache_errno, &changed);\n-\t\tif (new == ce)\n+\t\tif (new == ce) {\n+\t\t\tif (ce->ce_flags & CE_WATCHMAN_DIRTY) {\n+\t\t\t\t/* The rule is index-helper sets\n+\t\t\t\t * CE_WATCHMAN_DIRTY when a file is changed,\n+\t\t\t\t * then git clears it when it has verified\n+\t\t\t\t * that the in-index entry now matches the\n+\t\t\t\t * worktree version. index-helper does not\n+\t\t\t\t * clear the bit and git does not set it. We\n+\t\t\t\t * have verified here that stat info (and even\n+\t\t\t\t * content) matches, so it's safe to clear\n+\t\t\t\t * CE_WATCHMAN_DIRTY now.\n+\t\t\t\t */\n+\t\t\t\tce->ce_flags &= ~CE_WATCHMAN_DIRTY;\n+\t\t\t\tistate->cache_changed |= WATCHMAN_CHANGED;\n+\t\t\t}\n \t\t\tcontinue;\n+\t\t}\n \t\tif (!new) {\n \t\t\tconst char *fmt;\n \n@@ -1367,6 +1385,94 @@ static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n \treturn 0;\n }\n \n+static void mark_no_watchman(size_t pos, void *data)\n+{\n+\tstruct index_state *istate = data;\n+\tassert(pos < istate->cache_nr);\n+\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n+}\n+\n+static int read_watchman_ext(struct index_state *istate, const void *data,\n+\t\t\t unsigned long sz)\n+{\n+\tstruct ewah_bitmap *bitmap;\n+\tint ret, len;\n+\tuint32_t bitmap_size;\n+\tuint32_t untracked_nr;\n+\n+\tif (memchr(data, 0, sz) == NULL)\n+\t\treturn error(\"invalid extension\");\n+\n+\tlen = strlen(data) + 1;\n+\tmemcpy(&bitmap_size, (const char *)data + len, 4);\n+\tmemcpy(&untracked_nr, (const char *)data + len + 4, 4);\n+\tuntracked_nr = ntohl(untracked_nr);\n+\tbitmap_size = ntohl(bitmap_size);\n+\n+\tbitmap = ewah_new();\n+\tret = ewah_read_mmap(bitmap, (const char *)data + len + 8, bitmap_size);\n+\tif (ret != bitmap_size) {\n+\t\tewah_free(bitmap);\n+\t\treturn error(\"failed to parse ewah bitmap reading watchman index extension\");\n+\t}\n+\tistate->last_update = xstrdup(data);\n+\tewah_each_bit(bitmap, mark_no_watchman, istate);\n+\tewah_free(bitmap);\n+\n+\t/*\n+\t * TODO: update the untracked cache from the untracked data in this\n+\t * extension.\n+\t */\n+\treturn 0;\n+}\n+\n+static int untracked_entry_append(struct string_list_item *item, void *sbvoid)\n+{\n+\tstruct strbuf *sb = sbvoid;\n+\n+\tstrbuf_addstr(sb, item->string);\n+\tstrbuf_addch(sb, 0);\n+\treturn 0;\n+}\n+\n+void write_watchman_ext(struct strbuf *sb, struct index_state *istate)\n+{\n+\tstruct ewah_bitmap *bitmap;\n+\tint i;\n+\tint ewah_start;\n+\tint ewah_size = 0;\n+\tint fixup = 0;\n+\n+\tstrbuf_add(sb, istate->last_update, strlen(istate->last_update) + 1);\n+\tfixup = sb->len;\n+\tstrbuf_add(sb, &ewah_size, 4); /* we'll fix this up later */\n+\tif (istate->untracked) {\n+\t\tuint32_t nr = istate->untracked->invalid_untracked.nr;\n+\t\tnr = htonl(nr);\n+\t\tstrbuf_add(sb, &nr, 4);\n+\t} else {\n+\t\t/* zero */\n+\t\tstrbuf_add(sb, &ewah_size, 4);\n+\t}\n+\n+\tewah_start = sb->len;\n+\tbitmap = ewah_new();\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHMAN_DIRTY)\n+\t\t\tewah_set(bitmap, i);\n+\tewah_serialize_strbuf(bitmap, sb);\n+\tewah_free(bitmap);\n+\n+\t/* fix up size field */\n+\tewah_size = sb->len - ewah_start;\n+\tewah_size = htonl(ewah_size);\n+\tmemcpy(sb->buf + fixup, &ewah_size, 4);\n+\n+\tif (istate->untracked)\n+\t\tfor_each_string_list(&istate->untracked->invalid_untracked,\n+\t\t\t\t untracked_entry_append, sb);\n+}\n+\n static int read_index_extension(struct index_state *istate,\n \t\t\t\tconst char *ext, void *data, unsigned long sz)\n {\n@@ -1384,6 +1490,11 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_UNTRACKED:\n \t\tistate->untracked = read_untracked_extension(data, sz);\n \t\tbreak;\n+\n+\tcase CACHE_EXT_WATCHMAN:\n+\t\tread_watchman_ext(istate, data, sz);\n+\t\tbreak;\n+\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1815,6 +1926,8 @@ int discard_index(struct index_state *istate)\n \tistate->untracked = NULL;\n \tistate->from_shm = 0;\n \tistate->to_shm = 0;\n+\tfree(istate->last_update);\n+\tistate->last_update = NULL;\n \treturn 0;\n }\n \n@@ -2212,6 +2325,16 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->last_update) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_watchman_ext(&sb, istate);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCHMAN, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st))\n \t\treturn -1;\n-- \n1.9.1\n\n"},{"id":"290168","messageId":"1466914464-10358-10-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 10/20] watchman: support watchman to reduce index refresh cost","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:14Z","receivedAt":"2016-06-26T04:15:55Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThe previous patch has the logic to clear bits in 'WAMA' bitmap. This\npatch has logic to set bits as told by watchman. The missing bit,\n_using_ these bits, are not here yet.\n\nA lot of this code is written by David Turner originally, mostly from\n[1]. I'm just copying and polishing it a bit.\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/248006\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Makefile | 12 +++++\n cache.h | 1 +\n config.c | 5 ++\n configure.ac | 8 ++++\n environment.c | 2 +\n watchman-support.c | 135 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n watchman-support.h | 7 +++\n 7 files changed, 170 insertions(+)\n create mode 100644 watchman-support.c\n create mode 100644 watchman-support.h\n\ndiff --git a/Makefile b/Makefile\nindex 2b2ac05..7f4ab41 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -451,6 +451,7 @@ MSGFMT = msgfmt\n CURL_CONFIG = curl-config\n PTHREAD_LIBS = -lpthread\n PTHREAD_CFLAGS =\n+WATCHMAN_LIBS =\n GCOV = gcov\n \n export TCL_PATH TCLTK_PATH\n@@ -1416,6 +1417,13 @@ else\n \tLIB_OBJS += thread-utils.o\n endif\n \n+ifdef USE_WATCHMAN\n+\tLIB_H += watchman-support.h\n+\tLIB_OBJS += watchman-support.o\n+\tWATCHMAN_LIBS = -lwatchman\n+\tBASIC_CFLAGS += -DUSE_WATCHMAN\n+endif\n+\n ifdef HAVE_PATHS_H\n \tBASIC_CFLAGS += -DHAVE_PATHS_H\n endif\n@@ -2027,6 +2035,9 @@ git-remote-testsvn$X: remote-testsvn.o GIT-LDFLAGS $(GITLIBS) $(VCSSVN_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) \\\n \t$(VCSSVN_LIB)\n \n+git-index-helper$X: index-helper.o GIT-LDFLAGS $(GITLIBS)\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) $(WATCHMAN_LIBS)\n+\n $(REMOTE_CURL_ALIASES): $(REMOTE_CURL_PRIMARY)\n \t$(QUIET_LNCP)$(RM) $@ && \\\n \tln $< $@ 2>/dev/null || \\\n@@ -2166,6 +2177,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo NO_MMAP=\\''$(subst ','\\'',$(subst ','\\'',$(NO_MMAP)))'\\' >>$@+\n+\t@echo USE_WATCHMAN=\\''$(subst ','\\'',$(subst ','\\'',$(USE_WATCHMAN)))'\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/cache.h b/cache.h\nindex f10992d..452aea2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -696,6 +696,7 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n+extern int core_watchman_sync_timeout;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex 9ba40bc..e6dc141 100644\n--- a/config.c\n+++ b/config.c\n@@ -882,6 +882,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.watchmansynctimeout\")) {\n+\t\tcore_watchman_sync_timeout = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.createobject\")) {\n \t\tif (!strcmp(value, \"rename\"))\n \t\t\tobject_creation_mode = OBJECT_CREATION_USES_RENAMES;\ndiff --git a/configure.ac b/configure.ac\nindex 0cd9f46..334d63b 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -1099,6 +1099,14 @@ AC_COMPILE_IFELSE([BSD_SYSCTL_SRC],\n \tHAVE_BSD_SYSCTL=])\n GIT_CONF_SUBST([HAVE_BSD_SYSCTL])\n \n+#\n+# Check for watchman client library\n+\n+AC_CHECK_LIB([watchman], [watchman_connect],\n+\t[USE_WATCHMAN=YesPlease],\n+\t[USE_WATCHMAN=])\n+GIT_CONF_SUBST([USE_WATCHMAN])\n+\n ## Other checks.\n # Define USE_PIC if you need the main git objects to be built with -fPIC\n # in order to build and link perl/Git.so. x86-64 seems to need this.\ndiff --git a/environment.c b/environment.c\nindex 6dec9d0..ad41015 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -94,6 +94,8 @@ int core_preload_index = 1;\n */\n int ignore_untracked_cache_config;\n \n+int core_watchman_sync_timeout = 300;\n+\n /* This is set by setup_git_dir_gently() and/or git_default_config() */\n char *git_work_tree_cfg;\n static char *work_tree;\ndiff --git a/watchman-support.c b/watchman-support.c\nnew file mode 100644\nindex 0000000..8421fe0\n--- /dev/null\n+++ b/watchman-support.c\n@@ -0,0 +1,135 @@\n+#include \"cache.h\"\n+#include \"watchman-support.h\"\n+#include \"strbuf.h\"\n+#include \"dir.h\"\n+#include \n+\n+static struct watchman_query *make_query(const char *last_update)\n+{\n+\tstruct watchman_query *query = watchman_query();\n+\twatchman_query_set_fields(query, WATCHMAN_FIELD_NAME |\n+\t\t\t\t\t WATCHMAN_FIELD_EXISTS |\n+\t\t\t\t\t WATCHMAN_FIELD_NEWER);\n+\twatchman_query_set_empty_on_fresh(query, 1);\n+\tquery->sync_timeout = core_watchman_sync_timeout;\n+\tif (*last_update)\n+\t\twatchman_query_set_since_oclock(query, last_update);\n+\treturn query;\n+}\n+\n+static struct watchman_query_result *query_watchman(\n+\tstruct index_state *istate, struct watchman_connection *connection,\n+\tconst char *fs_path, const char *last_update)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_query *query;\n+\tstruct watchman_expression *expr;\n+\tstruct watchman_query_result *result;\n+\n+\tquery = make_query(last_update);\n+\texpr = watchman_true_expression();\n+\tresult = watchman_do_query(connection, fs_path, query, expr, &wm_error);\n+\twatchman_free_query(query);\n+\twatchman_free_expression(expr);\n+\n+\tif (!result)\n+\t\twarning(\"Watchman query error: %s (at %s)\",\n+\t\t\twm_error.message,\n+\t\t\t*last_update ? last_update : \"the beginning\");\n+\n+\treturn result;\n+}\n+\n+static void update_index(struct index_state *istate,\n+\t\t\t struct watchman_query_result *result)\n+{\n+\tint i;\n+\n+\tif (result->is_fresh_instance) {\n+\t\t/* let refresh clear them later */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags |= CE_WATCHMAN_DIRTY;\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < result->nr; i++) {\n+\t\tstruct watchman_stat *wm = result->stats + i;\n+\t\tint pos;\n+\n+\t\tif (S_ISDIR(wm->mode) ||\n+\t\t !strncmp(wm->name, \".git/\", 5) ||\n+\t\t strstr(wm->name, \"/.git/\"))\n+\t\t\tcontinue;\n+\n+\t\tpos = index_name_pos(istate, wm->name, strlen(wm->name));\n+\t\tif (pos < 0) {\n+\t\t\tif (istate->untracked) {\n+\t\t\t\tchar *name = xstrdup(wm->name);\n+\t\t\t\tchar *dname = dirname(name);\n+\n+\t\t\t\t/*\n+\t\t\t\t * dirname() returns '.' for the root,\n+\t\t\t\t * but we call it ''.\n+\t\t\t\t */\n+\t\t\t\tif (dname[0] == '.' && dname[1] == 0)\n+\t\t\t\t\tstring_list_append(&istate->untracked->invalid_untracked, \"\");\n+\t\t\t\telse\n+\t\t\t\t\tstring_list_append(&istate->untracked->invalid_untracked,\n+\t\t\t\t\t\t\t dname);\n+\t\t\t\tfree(name);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* FIXME: ignore staged entries and gitlinks too? */\n+\n+\t\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n+\t}\n+\n+done:\n+\tfree(istate->last_update);\n+\tistate->last_update = xstrdup(result->clock);\n+\tistate->cache_changed |= WATCHMAN_CHANGED;\n+\tif (istate->untracked)\n+\t\tstring_list_remove_duplicates(&istate->untracked->invalid_untracked, 0);\n+}\n+\n+int check_watchman(struct index_state *istate)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_connection *connection;\n+\tstruct watchman_query_result *result;\n+\tconst char *fs_path;\n+\tstruct timeval timeout;\n+\t/*\n+\t * Convert core_watchman_sync_timeout, in milliseconds, to\n+\t * struct timeval, in seconds and microseconds.\n+\t */\n+\n+\tfs_path = get_git_work_tree();\n+\tif (!fs_path)\n+\t\treturn -1;\n+\n+\ttimeout.tv_sec = core_watchman_sync_timeout / 1000;\n+\ttimeout.tv_usec = (core_watchman_sync_timeout % 1000) * 1000;\n+\tconnection = watchman_connect(timeout, &wm_error);\n+\n+\tif (!connection) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\treturn -1;\n+\t}\n+\n+\tif (watchman_watch(connection, fs_path, &wm_error)) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\twatchman_connection_close(connection);\n+\t\treturn -1;\n+\t}\n+\n+\tresult = query_watchman(istate, connection, fs_path,\n+\t\t\t\tistate->last_update ? istate->last_update : \"\");\n+\twatchman_connection_close(connection);\n+\tif (!result)\n+\t\treturn -1;\n+\tupdate_index(istate, result);\n+\twatchman_free_query_result(result);\n+\treturn 0;\n+}\ndiff --git a/watchman-support.h b/watchman-support.h\nnew file mode 100644\nindex 0000000..ee1ef2c\n--- /dev/null\n+++ b/watchman-support.h\n@@ -0,0 +1,7 @@\n+#ifndef WATCHMAN_SUPPORT_H\n+#define WATCHMAN_SUPPORT_H\n+\n+struct index_state;\n+int check_watchman(struct index_state *index);\n+\n+#endif /* WATCHMAN_SUPPORT_H */\n-- \n1.9.1\n\n"},{"id":"290169","messageId":"1466914464-10358-11-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 11/20] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:15Z","receivedAt":"2016-06-26T04:15:57Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nWatchman is hidden behind index-helper. Before git tries to read the\nindex from shm, it notifies index-helper through the socket and waits\nfor index-helper to prepare a file for sharing memory (with\nMAP_SHARED). index-helper then contacts watchman, updates 'WAMA'\nextension and put it in a separate file and wakes git up with a reply\nto git's socket.\n\nGit uses this extension to not lstat unchanged entries. Git only\ntrusts the 'WAMA' extension when it's received from the separate file,\nnot from disk. Unmarked entries are \"clean\". Marked entries are dirty\nfrom watchman point of view. If it finds out some entries are\n'watchman-dirty', but are really unchanged (e.g. the file was changed,\nthen reverted back), then Git will clear the marking in 'WAMA' before\nwriting it down.\n\nHiding watchman behind index-helper means you need both daemons. You\ncan't run watchman alone. Not so good. But on the other hand, 'git'\nbinary is not linked to watchman/json libraries, which is good for\npackaging. Core git package will run fine without watchman-related\npackages. If they need watchman, they can install git-index-helper and\ndependencies.\n\nThis also lets us trust anything in the untracked cache that we haven't\nmarked invalid, saving those stat() calls.\n\nAnother reason for tying watchman to index-helper is, when used with\nuntracked cache, we need to keep track of $GIT_WORK_TREE file\nlisting. That kind of list can be kept in index-helper.\n\nHelped-by: Ramsay Jones \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-index-helper.txt | 11 ++\n cache.h | 2 +\n dir.c | 23 +++-\n dir.h | 3 +\n index-helper.c | 107 +++++++++++++++--\n pkt-line.c | 2 +-\n read-cache.c | 239 ++++++++++++++++++++++++++++++++++---\n 7 files changed, 361 insertions(+), 26 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex 3d2c829..2982e03 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -55,6 +55,17 @@ command. The following commands are used to control the daemon:\n \tLet the daemon know the index is to be read. It keeps the\n \tdaemon alive longer, unless `--exit-after=0` is used.\n \n+\"poke \":\n+\tLike \"poke\", but replies with \"OK\". If the \"watchman\"\n+\tcapability is requested, and git is built with watchman, and\n+\tthe index has the watchman extension, then index-helper\n+\tqueries watchman. When watchman replies, index-helper\n+\tprepares a shared memory object with the watchman index\n+\textension, then replies \"OK watchman\". If git was built\n+\twithout watchman, \"NAK watchman\" is sent. This shouldn't\n+\thappen very often, since a git built without watchman won't\n+\trequest the watchman capability.\n+\n All commands and replies are in pkt-line format.\n \n In the event of an error, messages may be written to\ndiff --git a/cache.h b/cache.h\nindex 452aea2..633e1dd 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -567,6 +567,7 @@ extern int daemonize(int *);\n \n /* Initialize and use the cache information */\n struct lock_file;\n+extern int verify_index(const struct index_state *);\n extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);\n extern int do_read_index(struct index_state *istate, const char *path,\n@@ -574,6 +575,7 @@ extern int do_read_index(struct index_state *istate, const char *path,\n extern int read_index_from(struct index_state *, const char *path);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n+extern void write_watchman_ext(struct strbuf *sb, struct index_state *istate);\n #define COMMIT_LOCK\t\t(1 << 0)\n #define CLOSE_LOCK\t\t(1 << 1)\n extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\ndiff --git a/dir.c b/dir.c\nindex a4a9d9f..850d4fb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -592,9 +592,9 @@ static void trim_trailing_spaces(char *buf)\n *\n * If \"name\" has the trailing slash, it'll be excluded in the search.\n */\n-static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n-\t\t\t\t\t\t struct untracked_cache_dir *dir,\n-\t\t\t\t\t\t const char *name, int len)\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+\t\t\t\t\t struct untracked_cache_dir *dir,\n+\t\t\t\t\t const char *name, int len)\n {\n \tint first, last;\n \tstruct untracked_cache_dir *d;\n@@ -1559,6 +1559,17 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n+\tif (dir->untracked->use_watchman) {\n+\t\t/*\n+\t\t * With watchman, we can trust the untracked cache's\n+\t\t * valid field.\n+\t\t */\n+\t\tif (untracked->valid)\n+\t\t\tgoto skip_stat;\n+\t\telse\n+\t\t\tinvalidate_directory(dir->untracked, untracked);\n+\t}\n+\n \tif (stat(path->len ? path->buf : \".\", &st)) {\n \t\tinvalidate_directory(dir->untracked, untracked);\n \t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n@@ -1572,6 +1583,7 @@ static int valid_cached_dir(struct dir_struct *dir,\n \t\treturn 0;\n \t}\n \n+skip_stat:\n \tif (untracked->check_only != !!check_only) {\n \t\tinvalidate_directory(dir->untracked, untracked);\n \t\treturn 0;\n@@ -2427,8 +2439,10 @@ static void free_untracked(struct untracked_cache_dir *ucd)\n \n void free_untracked_cache(struct untracked_cache *uc)\n {\n-\tif (uc)\n+\tif (uc) {\n \t\tfree_untracked(uc->root);\n+\t\tstring_list_clear(&uc->invalid_untracked, 0);\n+\t}\n \tfree(uc);\n }\n \n@@ -2577,6 +2591,7 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n \t\treturn NULL;\n \n \tuc = xcalloc(1, sizeof(*uc));\n+\tstring_list_init(&uc->invalid_untracked, 1);\n \tstrbuf_init(&uc->ident, ident_len);\n \tstrbuf_add(&uc->ident, ident, ident_len);\n \tload_sha1_stat(&uc->ss_info_exclude, &ouc->info_exclude_stat,\ndiff --git a/dir.h b/dir.h\nindex 896b64a..ed16746 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -312,4 +312,7 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked);\n void add_untracked_cache(struct index_state *istate);\n void remove_untracked_cache(struct index_state *istate);\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+\t\t\t\t\t struct untracked_cache_dir *dir,\n+\t\t\t\t\t const char *name, int len);\n #endif\ndiff --git a/index-helper.c b/index-helper.c\nindex 142af7a..bb39326 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -8,15 +8,18 @@\n #include \"cache.h\"\n #include \"unix-socket.h\"\n #include \"pkt-line.h\"\n+#include \"watchman-support.h\"\n \n struct shm {\n \tunsigned char sha1[20];\n \tvoid *shm;\n \tsize_t size;\n+\tpid_t pid;\n };\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n+static struct shm shm_watchman;\n static int daemonized, to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n@@ -28,10 +31,21 @@ static void release_index_shm(struct shm *is)\n \tis->shm = NULL;\n }\n \n+static void release_watchman_shm(struct shm *is)\n+{\n+\tif (!is->shm)\n+\t\treturn;\n+\tmunmap(is->shm, is->size);\n+\tunlink(git_path(\"shm-watchman-%s-%\" PRIuMAX,\n+\t\t\tsha1_to_hex(is->sha1), (uintmax_t)is->pid));\n+\tis->shm = NULL;\n+}\n+\n static void cleanup_shm(void)\n {\n \trelease_index_shm(&shm_index);\n \trelease_index_shm(&shm_base_index);\n+\trelease_watchman_shm(&shm_watchman);\n }\n \n static void cleanup(void)\n@@ -174,9 +188,10 @@ static void share_the_index(void)\n \tif (the_index.split_index && the_index.split_index->base)\n \t\tshare_index(the_index.split_index->base, &shm_base_index);\n \tshare_index(&the_index, &shm_index);\n-\tif (to_verify && !verify_shm())\n+\tif (to_verify && !verify_shm()) {\n \t\tcleanup_shm();\n-\tdiscard_index(&the_index);\n+\t\tdiscard_index(&the_index);\n+\t}\n }\n \n static void set_socket_blocking_flag(int fd, int make_nonblocking)\n@@ -209,6 +224,80 @@ static void refresh(void)\n \n #ifndef NO_MMAP\n \n+#ifdef USE_WATCHMAN\n+static void share_watchman(struct index_state *istate,\n+\t\t\t struct shm *is, pid_t pid)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tvoid *shm;\n+\n+\twrite_watchman_ext(&sb, istate);\n+\tif (!shared_mmap_create(sb.len + 20, &shm,\n+\t\t\t\tgit_path(\"shm-watchman-%s-%\" PRIuMAX,\n+\t\t\t\t\t sha1_to_hex(istate->sha1),\n+\t\t\t\t\t (uintmax_t)pid))) {\n+\t\tis->size = sb.len + 20;\n+\t\tis->shm = shm;\n+\t\tis->pid = pid;\n+\t\thashcpy(is->sha1, istate->sha1);\n+\n+\t\tmemcpy(shm, sb.buf, sb.len);\n+\t\thashcpy((unsigned char *)shm + is->size - 20, is->sha1);\n+\t}\n+\tstrbuf_release(&sb);\n+}\n+\n+\n+static void prepare_with_watchman(pid_t pid)\n+{\n+\t/*\n+\t * TODO: with the help of watchman, maybe we could detect if\n+\t * $GIT_DIR/index is updated.\n+\t */\n+\tif (!verify_index(&the_index))\n+\t\trefresh();\n+\n+\tif (check_watchman(&the_index))\n+\t\treturn;\n+\n+\tshare_watchman(&the_index, &shm_watchman, pid);\n+}\n+\n+static void prepare_index(pid_t pid)\n+{\n+\tif (shm_index.shm == NULL)\n+\t\trefresh();\n+\trelease_watchman_shm(&shm_watchman);\n+\tif (the_index.last_update)\n+\t\tprepare_with_watchman(pid);\n+}\n+\n+#endif\n+\n+static void reply_to_poke(int client_fd, const char *pid_buf)\n+{\n+\tchar *capabilities;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+#ifdef USE_WATCHMAN\n+\tpid_t client_pid = strtoull(pid_buf, NULL, 10);\n+\n+\tprepare_index(client_pid);\n+#endif\n+\tcapabilities = strchr(pid_buf, ' ');\n+\n+\tif (capabilities && !strcmp(capabilities, \" watchman\"))\n+#ifdef USE_WATCHMAN\n+\t\tpacket_buf_write(&sb, \"OK watchman\");\n+#else\n+\t\tpacket_buf_write(&sb, \"NAK watchman\");\n+#endif\n+\telse\n+\t\tpacket_buf_write(&sb, \"OK\");\n+\tif (write_in_full(client_fd, sb.buf, sb.len) != sb.len)\n+\t\twarning(_(\"client write failed\"));\n+}\n+\n static void loop(int fd, int idle_in_seconds)\n {\n \tassert(idle_in_seconds < INT_MAX / 1000);\n@@ -263,11 +352,15 @@ static void loop(int fd, int idle_in_seconds)\n \t\t\tbuf[bytes_read] = 0;\n \t\t\tif (!strcmp(buf, \"refresh\")) {\n \t\t\t\trefresh();\n-\t\t\t} else if (!strcmp(buf, \"poke\")) {\n-\t\t\t\t/*\n-\t\t\t\t * Just a poke to keep us\n-\t\t\t\t * alive, nothing to do.\n-\t\t\t\t */\n+\t\t\t} else if (starts_with(buf, \"poke\")) {\n+\t\t\t\tif (buf[4] == ' ') {\n+\t\t\t\t\treply_to_poke(client_fd, buf + 5);\n+\t\t\t\t} else {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * Just a poke to keep us\n+\t\t\t\t\t * alive, nothing to do.\n+\t\t\t\t\t */\n+\t\t\t\t}\n \t\t\t} else {\n \t\t\t\twarning(\"BUG: Bogus command %s\", buf);\n \t\t\t}\ndiff --git a/pkt-line.c b/pkt-line.c\nindex f964446..8aa6c09 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -94,7 +94,7 @@ void packet_flush(int fd)\n int packet_flush_gently(int fd)\n {\n \tpacket_trace(\"0000\", 4, 1);\n-\treturn write_in_full(fd, \"0000\", 4) != 4;\n+\treturn write_in_full(fd, \"0000\", 4) == 4 ? 0 : -1;\n }\n \n void packet_buf_flush(struct strbuf *buf)\ndiff --git a/read-cache.c b/read-cache.c\nindex e0fc634..8521e85 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1385,11 +1385,75 @@ static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n \treturn 0;\n }\n \n+static struct untracked_cache_dir *find_untracked_cache_dir(\n+\tstruct untracked_cache *uc, struct untracked_cache_dir *ucd,\n+\tconst char *name)\n+{\n+\tint component_len;\n+\tconst char *end;\n+\tstruct untracked_cache_dir *dir;\n+\n+\tif (!*name)\n+\t\treturn ucd;\n+\n+\tend = strchr(name, '/');\n+\tif (end)\n+\t\tcomponent_len = end - name;\n+\telse\n+\t\tcomponent_len = strlen(name);\n+\n+\tdir = lookup_untracked(uc, ucd, name, component_len);\n+\tif (dir)\n+\t\treturn find_untracked_cache_dir(uc, dir, name + component_len + 1);\n+\n+\treturn NULL;\n+}\n+\n static void mark_no_watchman(size_t pos, void *data)\n {\n \tstruct index_state *istate = data;\n+\tstruct cache_entry *ce = istate->cache[pos];\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tchar *c;\n+\tstruct untracked_cache_dir *dir;\n+\n \tassert(pos < istate->cache_nr);\n-\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n+\tce->ce_flags |= CE_WATCHMAN_DIRTY;\n+\n+\tif (!istate->untracked || !istate->untracked->root)\n+\t\treturn;\n+\n+\tstrbuf_add(&sb, ce->name, ce_namelen(ce));\n+\n+\tfor (c = sb.buf + sb.len - 1; c > sb.buf; c--) {\n+\t\tif (*c == '/') {\n+\t\t\tstrbuf_setlen(&sb, c - sb.buf);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (c == sb.buf)\n+\t\tstrbuf_setlen(&sb, 0);\n+\n+\tdir = find_untracked_cache_dir(istate->untracked,\n+\t\t\t\t istate->untracked->root, sb.buf);\n+\tif (dir)\n+\t\tdir->valid = 0;\n+\n+\tstrbuf_release(&sb);\n+}\n+\n+static int mark_untracked_invalid(struct string_list_item *item, void *uc)\n+{\n+\tstruct untracked_cache *untracked = uc;\n+\tstruct untracked_cache_dir *dir;\n+\n+\tdir = find_untracked_cache_dir(untracked, untracked->root,\n+\t\t\t\t item->string);\n+\tif (dir)\n+\t\tdir->valid = 0;\n+\n+\treturn 0;\n }\n \n static int read_watchman_ext(struct index_state *istate, const void *data,\n@@ -1419,10 +1483,24 @@ static int read_watchman_ext(struct index_state *istate, const void *data,\n \tewah_each_bit(bitmap, mark_no_watchman, istate);\n \tewah_free(bitmap);\n \n-\t/*\n-\t * TODO: update the untracked cache from the untracked data in this\n-\t * extension.\n-\t */\n+\tif (istate->untracked && istate->untracked->root) {\n+\t\tint i;\n+\t\tconst char *untracked;\n+\n+\t\tuntracked = (const char *)data + len + 8 + bitmap_size;\n+\t\tfor (i = 0; i < untracked_nr; ++i) {\n+\t\t\tint len = strlen(untracked);\n+\t\t\tstring_list_append(&istate->untracked->invalid_untracked,\n+\t\t\t\t\t untracked);\n+\t\t\tuntracked += len + 1;\n+\t\t}\n+\n+\t\tfor_each_string_list(&istate->untracked->invalid_untracked,\n+\t\t\t mark_untracked_invalid, istate->untracked);\n+\n+\t\tif (untracked_nr)\n+\t\t\tistate->cache_changed |= WATCHMAN_CHANGED;\n+\t}\n \treturn 0;\n }\n \n@@ -1655,27 +1733,88 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_untracked_cache(istate);\n }\n \n+/* in ms */\n+#define WATCHMAN_TIMEOUT 1000\n+\n+static int poke_and_wait_for_reply(int fd)\n+{\n+\tint ret = -1;\n+\tstruct pollfd pollfd;\n+\tint bytes_read;\n+\tchar reply_buf[4096];\n+\tconst char *requested_capabilities = \"\";\n+\n+#ifdef USE_WATCHMAN\n+\trequested_capabilities = \"watchman\";\n+#endif\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\n+\tif (packet_write_gently(fd, \"poke %d %s\", getpid(), requested_capabilities))\n+\t\treturn -1;\n+\tif (packet_flush_gently(fd))\n+\t\treturn -1;\n+\n+\t/* Now wait for a reply */\n+\tpollfd.fd = fd;\n+\tpollfd.events = POLLIN;\n+\tif (poll(&pollfd, 1, WATCHMAN_TIMEOUT) <= 0)\n+\t\t/* No reply or error, giving up */\n+\t\tgoto done_poke;\n+\n+\tbytes_read = packet_read(fd, NULL, NULL, reply_buf, sizeof(reply_buf),\n+\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\tif (bytes_read < 0)\n+\t\tgoto done_poke;\n+\n+\tif (!strcmp(reply_buf, \"NAK watchman\")\n+#ifdef USE_WATCHMAN\n+\t || !ends_with(reply_buf, \"watchman\")\n+#endif\n+\t\t) {\n+\t\twarning(\"We requested watchman support from index-helper, but \"\n+\t\t\t\"it doesn't support it. Please use a version of git \"\n+\t\t\t\"index-helper with watchman support.\");\n+\t\tgoto done_poke;\n+\t}\n+\n+\tif (!starts_with(reply_buf, \"OK\"))\n+\t\tgoto done_poke;\n+\n+\tret = 0;\n+done_poke:\n+\tclose(fd);\n+\treturn ret;\n+}\n+\n static int poke_daemon(struct index_state *istate,\n \t\t const struct stat *st, int refresh_cache)\n {\n \tint fd;\n-\tint ret = 0;\n-\tconst char *socket_path;\n+\tint ret = -1;\n \n \t/* if this is from index-helper, do not poke itself (recursively) */\n \tif (istate->to_shm)\n \t\treturn 0;\n \n-\tsocket_path = git_path(\"index-helper.sock\");\n-\n-\tfd = unix_stream_connect(socket_path);\n+\tfd = unix_stream_connect(git_path(\"index-helper.sock\"));\n+\tif (fd < 0) {\n+\t\twarning(\"Failed to connect to index-helper socket\");\n+\t\tunlink(git_path(\"index-helper.sock\"));\n+\t\treturn -1;\n+\t}\n \tsigchain_push(SIGPIPE, SIG_IGN);\n+\n \tif (refresh_cache) {\n \t\tpacket_write_gently(fd, \"refresh\");\n+\t\tpacket_flush_gently(fd);\n+\t\tret = 0;\n \t} else {\n-\t\tpacket_write_gently(fd, \"poke\");\n+\t\tret = poke_and_wait_for_reply(fd);\n \t}\n-\tpacket_flush_gently(fd);\n \n \tclose(fd);\n \tsigchain_pop(SIGPIPE);\n@@ -1745,6 +1884,74 @@ fail:\n \treturn -1;\n }\n \n+static void refresh_by_watchman(struct index_state *istate)\n+{\n+\tvoid *shm = NULL;\n+\tint length;\n+\tint i;\n+\tstruct stat st;\n+\tint fd = -1;\n+\tconst char *path = git_path(\"shm-watchman-%s-%\"PRIuMAX,\n+\t\t\t\t sha1_to_hex(istate->sha1),\n+\t\t\t\t (uintmax_t)getpid());\n+\n+\tfd = open(path, O_RDONLY);\n+\tif (fd < 0)\n+\t\treturn;\n+\n+\t/*\n+\t * This watchman data is just for us -- no need to keep it\n+\t * around once we've got it open.\n+\t */\n+\tunlink(path);\n+\n+\tif (fstat(fd, &st) < 0)\n+\t\tgoto done;\n+\n+\tlength = st.st_size;\n+\tshm = mmap(NULL, length, PROT_READ, MAP_SHARED, fd, 0);\n+\n+\tif (shm == MAP_FAILED)\n+\t\tgoto done;\n+\n+\tclose(fd);\n+\tfd = -1;\n+\n+\tif (length <= 20 ||\n+\t hashcmp(istate->sha1, (unsigned char *)shm + length - 20) ||\n+\t /*\n+\t * No need to clear CE_WATCHMAN_DIRTY set by 'WAMA' on\n+\t * disk. Watchman can only set more, not clear any, so\n+\t * this is OR mask.\n+\t */\n+\t read_watchman_ext(istate, shm, length - 20))\n+\t\tgoto done;\n+\n+\t/*\n+\t * Now that we've marked the invalid entries in the\n+\t * untracked-cache itself, we can erase them from the list of\n+\t * entries to be processed and mark the untracked cache for\n+\t * watchman usage.\n+\t */\n+\tif (istate->untracked) {\n+\t\tstring_list_clear(&istate->untracked->invalid_untracked, 0);\n+\t\tistate->untracked->use_watchman = 1;\n+\t}\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\tif (ce_stage(ce) || (ce->ce_flags & CE_WATCHMAN_DIRTY))\n+\t\t\tcontinue;\n+\t\tce_mark_uptodate(ce);\n+\t}\n+done:\n+\tif (shm)\n+\t\tmunmap(shm, length);\n+\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1864,7 +2071,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tsplit_index = istate->split_index;\n \tif (!split_index || is_null_sha1(split_index->base_sha1)) {\n \t\tpost_read_index_from(istate);\n-\t\treturn ret;\n+\t\tgoto done;\n \t}\n \n \tif (split_index->base)\n@@ -1885,6 +2092,10 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\t sha1_to_hex(split_index->base->sha1));\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n+\n+done:\n+\tif (ret > 0 && istate->from_shm && istate->last_update)\n+\t\trefresh_by_watchman(istate);\n \treturn ret;\n }\n \n@@ -2186,7 +2397,7 @@ out:\n \treturn 0;\n }\n \n-static int verify_index(const struct index_state *istate)\n+int verify_index(const struct index_state *istate)\n {\n \treturn verify_index_from(istate, get_index_file());\n }\n-- \n1.9.1\n\n"},{"id":"290170","messageId":"1466914464-10358-7-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 07/20] index-helper: add --detach","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:11Z","receivedAt":"2016-06-26T04:16:13Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nWe detach after creating and opening the socket, because otherwise\nwe might return control to the shell before index-helper is ready to\naccept commands. This might lead to flaky tests.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-index-helper.txt | 3 +++\n index-helper.c | 9 +++++++--\n 2 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex ca5a9de..228b1df 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -34,6 +34,9 @@ OPTIONS\n \tfor reading an index, but because it will happen in the\n \tbackground, it's not noticable. `--strict` is enabled by default.\n \n+--detach::\n+\tDetach from the shell.\n+\n NOTES\n -----\n \ndiff --git a/index-helper.c b/index-helper.c\nindex d1e9287..7cc8166 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -17,7 +17,7 @@ struct shm {\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n-static int to_verify = 1;\n+static int daemonized, to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n {\n@@ -36,6 +36,8 @@ static void cleanup_shm(void)\n \n static void cleanup(void)\n {\n+\tif (daemonized)\n+\t\treturn;\n \tunlink(git_path(\"index-helper.sock\"));\n \tcleanup_shm();\n }\n@@ -298,7 +300,7 @@ static const char * const usage_text[] = {\n int main(int argc, char **argv)\n {\n \tconst char *prefix;\n-\tint idle_in_seconds = 600;\n+\tint idle_in_seconds = 600, detach = 0;\n \tint fd;\n \tstruct strbuf socket_path = STRBUF_INIT;\n \tstruct option options[] = {\n@@ -306,6 +308,7 @@ int main(int argc, char **argv)\n \t\t\t N_(\"exit if not used after some seconds\")),\n \t\tOPT_BOOL(0, \"strict\", &to_verify,\n \t\t\t N_(\"verify shared memory after creating\")),\n+\t\tOPT_BOOL(0, \"detach\", &detach, N_(\"detach the process\")),\n \t\tOPT_END()\n \t};\n \n@@ -333,6 +336,8 @@ int main(int argc, char **argv)\n \t\tdie(\"--exit-after value must be less than %d seconds\",\n \t\t INT_MAX / 1000);\n \n+\tif (detach && daemonize(&daemonized))\n+\t\tdie_errno(_(\"unable to detach\"));\n \tloop(fd, idle_in_seconds);\n \n \tclose(fd);\n-- \n1.9.1\n\n"},{"id":"290171","messageId":"1466914464-10358-8-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 08/20] index-helper: log warnings","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:12Z","receivedAt":"2016-06-26T04:16:16Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: David Turner \n\nInstead of writing warnings to stderr, write them to a log. Later, we'll\nprobably be daemonized, so writing to stderr will be pointless.\n\nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-index-helper.txt | 3 +++\n index-helper.c | 12 +++++++++++-\n 2 files changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex 228b1df..3d2c829 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -57,6 +57,9 @@ command. The following commands are used to control the daemon:\n \n All commands and replies are in pkt-line format.\n \n+In the event of an error, messages may be written to\n+$GIT_DIR/index-helper.log.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/index-helper.c b/index-helper.c\nindex 7cc8166..142af7a 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -103,7 +103,8 @@ static void share_index(struct index_state *istate, struct shm *is)\n \tif (shared_mmap_create(istate->mmap_size, &new_mmap,\n \t\t\t git_path(\"shm-index-%s\",\n \t\t\t\t\tsha1_to_hex(istate->sha1))) < 0) {\n-\t\tdie(\"Failed to create shm-index file\");\n+\t\twarning(\"Failed to create shm-index file\");\n+\t\texit(1);\n \t}\n \n \n@@ -336,8 +337,17 @@ int main(int argc, char **argv)\n \t\tdie(\"--exit-after value must be less than %d seconds\",\n \t\t INT_MAX / 1000);\n \n+\tif (detach) {\n+\t\tFILE *fp = fopen(git_path(\"index-helper.log\"), \"a\");\n+\t\tif (!fp)\n+\t\t\tdie(\"failed to open %s for writing\",\n+\t\t\t git_path(\"index-helper.log\"));\n+\t\tset_error_handle(fp);\n+\t}\n+\n \tif (detach && daemonize(&daemonized))\n \t\tdie_errno(_(\"unable to detach\"));\n+\n \tloop(fd, idle_in_seconds);\n \n \tclose(fd);\n-- \n1.9.1\n\n"},{"id":"290172","messageId":"1466914464-10358-19-git-send-email-novalis@novalis.org","threadId":"42713","inReplyTo":"1466914464-10358-1-git-send-email-novalis@novalis.org","subject":"[PATCH v13 19/20] trace: measure where the time is spent in the index-heavy operations","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-26T04:14:23Z","receivedAt":"2016-06-26T04:16:22Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nAll the known heavy code blocks are measured (except object database\naccess). This should help identify if an optimization is effective or\nnot. An unoptimized git-status would give something like below (92% of\ntime is accounted). To sum up the effort of making git scale better:\n\n - read cache line is being addressed by index-helper\n - preload/refresh index by watchman\n - read packed refs by lmdb backend\n - diff-index by rebuilding cache-tree\n - read directory by untracked cache and watchman\n - write index by split index\n - name hash potientially by index-helper\n\nread-cache.c:2075 performance: 0.004058570 s: read cache .../index\npreload-index.c:104 performance: 0.004419864 s: preload index\nread-cache.c:1265 performance: 0.000185224 s: refresh index\nrefs/files-backend.c:1100 performance: 0.001935788 s: read packed refs\ndiff-lib.c:240 performance: 0.000144132 s: diff-files\ndiff-lib.c:506 performance: 0.013592000 s: diff-index\nname-hash.c:128 performance: 0.000614177 s: initialize name hash\ndir.c:2030 performance: 0.015814103 s: read directory\nread-cache.c:2565 performance: 0.004052343 s: write index, changed mask = 2\ntrace.c:420 performance: 0.048365509 s: git command: './git' 'status'\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \nSigned-off-by: Junio C Hamano \n---\n diff-lib.c | 4 ++++\n dir.c | 2 ++\n name-hash.c | 2 ++\n preload-index.c | 2 ++\n read-cache.c | 11 +++++++++++\n refs/files-backend.c | 2 ++\n 6 files changed, 23 insertions(+)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex bc49c70..7af7f9a 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -90,6 +90,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \tint diff_unmerged_stage = revs->max_count;\n \tunsigned ce_option = ((option & DIFF_RACY_IS_MODIFIED)\n \t\t\t ? CE_MATCH_RACY_IS_DIRTY : 0);\n+\tuint64_t start = getnanotime();\n \n \tdiff_set_mnemonic_prefix(&revs->diffopt, \"i/\", \"w/\");\n \n@@ -236,6 +237,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t}\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n+\ttrace_performance_since(start, \"diff-files\");\n \treturn 0;\n }\n \n@@ -491,6 +493,7 @@ static int diff_cache(struct rev_info *revs,\n int run_diff_index(struct rev_info *revs, int cached)\n {\n \tstruct object_array_entry *ent;\n+\tuint64_t start = getnanotime();\n \n \tent = revs->pending.objects;\n \tif (diff_cache(revs, ent->item->oid.hash, ent->name, cached))\n@@ -500,6 +503,7 @@ int run_diff_index(struct rev_info *revs, int cached)\n \tdiffcore_fix_diff_index(&revs->diffopt);\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n+\ttrace_performance_since(start, \"diff-index\");\n \treturn 0;\n }\n \ndiff --git a/dir.c b/dir.c\nindex 850d4fb..024b13c 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1991,6 +1991,7 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const stru\n {\n \tstruct path_simplify *simplify;\n \tstruct untracked_cache_dir *untracked;\n+\tuint64_t start = getnanotime();\n \n \t/*\n \t * Check out create_simplify()\n@@ -2026,6 +2027,7 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const stru\n \tfree_simplify(simplify);\n \tqsort(dir->entries, dir->nr, sizeof(struct dir_entry *), cmp_name);\n \tqsort(dir->ignored, dir->ignored_nr, sizeof(struct dir_entry *), cmp_name);\n+\ttrace_performance_since(start, \"read directory %.*s\", len, path);\n \tif (dir->untracked) {\n \t\tstatic struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n \t\ttrace_printf_key(&trace_untracked_stats,\ndiff --git a/name-hash.c b/name-hash.c\nindex 6d9f23e..b3966d8 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -115,6 +115,7 @@ static int cache_entry_cmp(const struct cache_entry *ce1,\n static void lazy_init_name_hash(struct index_state *istate)\n {\n \tint nr;\n+\tuint64_t start = getnanotime();\n \n \tif (istate->name_hash_initialized)\n \t\treturn;\n@@ -124,6 +125,7 @@ static void lazy_init_name_hash(struct index_state *istate)\n \tfor (nr = 0; nr < istate->cache_nr; nr++)\n \t\thash_index_entry(istate, istate->cache[nr]);\n \tistate->name_hash_initialized = 1;\n+\ttrace_performance_since(start, \"initialize name hash\");\n }\n \n void add_name_hash(struct index_state *istate, struct cache_entry *ce)\ndiff --git a/preload-index.c b/preload-index.c\nindex c1fe3a3..7bb4809 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -72,6 +72,7 @@ static void preload_index(struct index_state *index,\n {\n \tint threads, i, work, offset;\n \tstruct thread_data data[MAX_PARALLEL];\n+\tuint64_t start = getnanotime();\n \n \tif (!core_preload_index)\n \t\treturn;\n@@ -100,6 +101,7 @@ static void preload_index(struct index_state *index,\n \t\tif (pthread_join(p->pthread, NULL))\n \t\t\tdie(\"unable to join threaded lstat\");\n \t}\n+\ttrace_performance_since(start, \"preload index\");\n }\n #endif\n \ndiff --git a/read-cache.c b/read-cache.c\nindex fde778b..ee777c1 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1189,6 +1189,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tconst char *typechange_fmt;\n \tconst char *added_fmt;\n \tconst char *unmerged_fmt;\n+\tuint64_t start = getnanotime();\n \n \tmodified_fmt = (in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\");\n \tdeleted_fmt = (in_porcelain ? \"D\\t%s\\n\" : \"%s: needs update\\n\");\n@@ -1273,6 +1274,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \n \t\treplace_index_entry(istate, i, new);\n \t}\n+\ttrace_performance_since(start, \"refresh index\");\n \treturn has_errors;\n }\n \n@@ -2092,12 +2094,15 @@ int read_index_from(struct index_state *istate, const char *path)\n {\n \tstruct split_index *split_index;\n \tint ret;\n+\tuint64_t start;\n \n \t/* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n \tif (istate->initialized)\n \t\treturn istate->cache_nr;\n \n+\tstart = getnanotime();\n \tret = do_read_index(istate, path, 0);\n+\ttrace_performance_since(start, \"read cache %s\", path);\n \n \tsplit_index = istate->split_index;\n \tif (!split_index || is_null_sha1(split_index->base_sha1)) {\n@@ -2112,6 +2117,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tsplit_index->base->keep_mmap = istate->keep_mmap;\n \tsplit_index->base->to_shm = istate->to_shm;\n \tsplit_index->base->from_shm = istate->from_shm;\n+\tstart = getnanotime();\n \tret = do_read_index(split_index->base,\n \t\t\t git_path(\"sharedindex.%s\",\n \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n@@ -2123,6 +2129,9 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\t sha1_to_hex(split_index->base->sha1));\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n+\ttrace_performance_since(start, \"read cache %s\",\n+\t\t\t\tgit_path(\"sharedindex.%s\",\n+\t\t\t\t\t sha1_to_hex(split_index->base_sha1)));\n \n done:\n \tif (ret > 0 && istate->from_shm && istate->last_update)\n@@ -2468,6 +2477,7 @@ static int do_write_index(struct index_state *istate, int newfd,\n \tstruct stat st;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n \tint watchman = 0;\n+\tuint64_t start = getnanotime();\n \n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n@@ -2588,6 +2598,7 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\treturn -1;\n \tistate->timestamp.sec = (unsigned int)st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n+\ttrace_performance_since(start, \"write index, changed mask = %x\", istate->cache_changed);\n \treturn 0;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 81f68f8..57571ce 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1048,6 +1048,7 @@ static void read_packed_refs(FILE *f, struct ref_dir *dir)\n \tstruct ref_entry *last = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n \tenum { PEELED_NONE, PEELED_TAGS, PEELED_FULLY } peeled = PEELED_NONE;\n+\tuint64_t start = getnanotime();\n \n \twhile (strbuf_getwholeline(&line, f, '\\n') != EOF) {\n \t\tunsigned char sha1[20];\n@@ -1096,6 +1097,7 @@ static void read_packed_refs(FILE *f, struct ref_dir *dir)\n \t}\n \n \tstrbuf_release(&line);\n+\ttrace_performance_since(start, \"read packed refs\");\n }\n \n /*\n-- \n1.9.1\n\n"},{"id":"290186","messageId":"20160626120928.14950-1-pclouds@gmail.com","threadId":"42713","inReplyTo":"1466914464-10358-19-git-send-email-novalis@novalis.org","subject":"[PATCH v13 21/20] unix-socket.c: add stub implementation when unix sockets are not supported","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-26T12:09:28Z","receivedAt":"2016-06-26T12:09:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This keeps #ifdef at the callee instead of caller, it's less messier.\n\nThe caller in question is in read-cache.c which, unlike other\nunix-socket callers so far, is always built regardless of unix socket\nsupport. No extra handling (for ENOSYS) is needed because in this\nbuild, index-helper does not exist, $GIT_DIR/index-helper.sock does\nnot exist, so no unix socket call is made by read-cache.c in the first\nplace.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n BTW 20/20 didn't seem to make it to the list (or my mailbox). And you\n probably want to update .mailmap with @novalis.org as main address\n too.\n\n Makefile | 2 ++\n unix-socket.h | 18 ++++++++++++++++++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex d1309b8..d8cfc51 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1337,6 +1337,8 @@ ifndef NO_UNIX_SOCKETS\n \tLIB_OBJS += unix-socket.o\n \tPROGRAM_OBJS += credential-cache.o\n \tPROGRAM_OBJS += credential-cache--daemon.o\n+else\n+\tBASIC_CFLAGS += -DNO_UNIX_SOCKETS\n endif\n \n ifdef NO_ICONV\ndiff --git a/unix-socket.h b/unix-socket.h\nindex e271aee..f1cba70 100644\n--- a/unix-socket.h\n+++ b/unix-socket.h\n@@ -1,7 +1,25 @@\n #ifndef UNIX_SOCKET_H\n #define UNIX_SOCKET_H\n \n+#ifndef NO_UNIX_SOCKETS\n+\n int unix_stream_connect(const char *path);\n int unix_stream_listen(const char *path);\n \n+#else\n+\n+static inline int unix_stream_connect(const char *path)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+\n+static inline int unix_stream_listen(const char *path)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+\n+#endif\n+\n #endif /* UNIX_SOCKET_H */\n-- \n2.8.2.531.gd073806\n\n"},{"id":"290228","messageId":"alpine.DEB.2.20.1606271413350.12947@virtualbox","threadId":"42713","inReplyTo":"20160626120928.14950-1-pclouds@gmail.com","subject":"Re: [PATCH v13 21/20] unix-socket.c: add stub implementation when unix sockets are not supported","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-06-27T12:14:40Z","receivedAt":"2016-06-27T12:15:06Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Sun, 26 Jun 2016, Nguyễn Thái Ngọc Duy wrote:\n\n> This keeps #ifdef at the callee instead of caller, it's less messier.\n> \n> The caller in question is in read-cache.c which, unlike other\n> unix-socket callers so far, is always built regardless of unix socket\n> support. No extra handling (for ENOSYS) is needed because in this\n> build, index-helper does not exist, $GIT_DIR/index-helper.sock does\n> not exist, so no unix socket call is made by read-cache.c in the first\n> place.\n> \n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n\nHeh, I made something very similar (although I did not update the errno as\nyour patch does): https://github.com/git-for-windows/git/commit/919cb1d79\n\nCiao,\nDscho"},{"id":"290248","messageId":"CACsJy8A2i9enBUtK+jcJmnChTddJBjd6O9LZfU9Tj2ikn8JqRg@mail.gmail.com","threadId":"42713","inReplyTo":"alpine.DEB.2.20.1606271413350.12947@virtualbox","subject":"Re: [PATCH v13 21/20] unix-socket.c: add stub implementation when unix sockets are not supported","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-27T16:16:29Z","receivedAt":"2016-06-27T16:17:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jun 27, 2016 at 2:14 PM, Johannes Schindelin\n wrote:\n> Hi Duy,\n>\n> On Sun, 26 Jun 2016, Nguyễn Thái Ngọc Duy wrote:\n>\n>> This keeps #ifdef at the callee instead of caller, it's less messier.\n>>\n>> The caller in question is in read-cache.c which, unlike other\n>> unix-socket callers so far, is always built regardless of unix socket\n>> support. No extra handling (for ENOSYS) is needed because in this\n>> build, index-helper does not exist, $GIT_DIR/index-helper.sock does\n>> not exist, so no unix socket call is made by read-cache.c in the first\n>> place.\n>>\n>> Signed-off-by: Nguyễn Thái Ngọc Duy \n>> ---\n>\n> Heh, I made something very similar (although I did not update the errno as\n> your patch does): https://github.com/git-for-windows/git/commit/919cb1d79\n\nYours lacks the important \"else\" line in Makefile, so pick mine! :-D\n-- \nDuy\n"},{"id":"290255","messageId":"57715A1D.3030705@novalis.org","threadId":"42713","inReplyTo":"CAFUO74nENapwVsM3CUst9AHqy5LcKTFBCnJxGXPk8E952t+X5Q@mail.gmail.com","subject":"Re: [PATCH v13 04/20] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"novalis@novalis.org","sentAt":"2016-06-27T16:53:49Z","receivedAt":"2016-06-27T16:54:18Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On 06/27/2016 08:58 AM, Keith McGuigan wrote:\n>\n> If 'indexhelper.exitafter' is set to 0, then loop is called with\n> idle_in_seconds == 0. It gets converted to -1, but then\n> 'idle_in_seconds * 1000' is passed to poll(), so poll gets an argument\n> of -1000, which it EINVALs on.\n\nSilly OS X!\n\nWill squash when I re-roll thanks.\n\n\n\n"},{"id":"290360","messageId":"alpine.DEB.2.20.1606281206100.12947@virtualbox","threadId":"42713","inReplyTo":"CACsJy8A2i9enBUtK+jcJmnChTddJBjd6O9LZfU9Tj2ikn8JqRg@mail.gmail.com","subject":"Re: [PATCH v13 21/20] unix-socket.c: add stub implementation when unix sockets are not supported","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-06-28T10:06:44Z","receivedAt":"2016-06-28T10:07:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duym\n\nOn Mon, 27 Jun 2016, Duy Nguyen wrote:\n\n> On Mon, Jun 27, 2016 at 2:14 PM, Johannes Schindelin\n> wrote:\n> >\n> > On Sun, 26 Jun 2016, Nguyễn Thái Ngọc Duy wrote:\n> >\n> >> This keeps #ifdef at the callee instead of caller, it's less messier.\n> >>\n> >> The caller in question is in read-cache.c which, unlike other\n> >> unix-socket callers so far, is always built regardless of unix socket\n> >> support. No extra handling (for ENOSYS) is needed because in this\n> >> build, index-helper does not exist, $GIT_DIR/index-helper.sock does\n> >> not exist, so no unix socket call is made by read-cache.c in the first\n> >> place.\n> >>\n> >> Signed-off-by: Nguyễn Thái Ngọc Duy \n> >> ---\n> >\n> > Heh, I made something very similar (although I did not update the errno as\n> > your patch does): https://github.com/git-for-windows/git/commit/919cb1d79\n> \n> Yours lacks the important \"else\" line in Makefile, so pick mine! :-D\n\nOh, I definitely prefer your patch over mine. I just meant to concur that\nthis needs fixing.\n\nCiao,\nDscho"},{"id":"290592","messageId":"alpine.DEB.2.20.1606301457340.12947@virtualbox","threadId":"42713","inReplyTo":"1466914464-10358-4-git-send-email-novalis@novalis.org","subject":"Re: [PATCH v13 04/20] index-helper: new daemon for caching index and related stuff","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-06-30T13:06:45Z","receivedAt":"2016-06-30T13:07:08Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Dave,\n\nOn Sun, 26 Jun 2016, David Turner wrote:\n\n> diff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\n> new file mode 100755\n> index 0000000..114c112\n> --- /dev/null\n> +++ b/t/t7900-index-helper.sh\n> @@ -0,0 +1,23 @@\n> +#!/bin/sh\n> +#\n> +# Copyright (c) 2016, Twitter, Inc\n> +#\n> +\n> +test_description='git-index-helper\n> +\n> +Testing git index-helper\n> +'\n> +\n> +. ./test-lib.sh\n> +\n> +test -n \"$NO_MMAP\" && {\n> +\tskip_all='skipping index-helper tests: no mmap'\n> +\ttest_done\n> +}\n\nEven when NO_MMAP is empty, there might be no Unix sockets available (such\nas is the case on Windows). In any case, you really only want to skip\nthese tests when index-helper is not available, so would you mind\nsquashing this patch in?\n\n-- snipsnap --\nFrom: Johannes Schindelin \nSubject: [PATCH] fixup! index-helper: new daemon for caching index and related\n stuff\n\n---\n t/t7900-index-helper.sh | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex 6c9b4dd..12b5bf7 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -10,8 +10,10 @@ Testing git index-helper\n \n . ./test-lib.sh\n \n-test -n \"$NO_MMAP\" && {\n-\tskip_all='skipping index-helper tests: no mmap'\n+git index-helper -h 2>/dev/null\n+test $? = 129 ||\n+{\n+\tskip_all='skipping index-helper tests: no index-helper executable'\n \ttest_done\n }\n \n-- \n2.9.0.270.g810e421\n\n"},{"id":"290596","messageId":"CACsJy8DuTsDAYzRVk=mW7WX5CZb0Z5bAPnzV_2KXK-BcX=tcVg@mail.gmail.com","threadId":"42713","inReplyTo":"alpine.DEB.2.20.1606301457340.12947@virtualbox","subject":"Re: [PATCH v13 04/20] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-30T15:04:35Z","receivedAt":"2016-06-30T15:05:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 30, 2016 at 3:06 PM, Johannes Schindelin\n wrote:\n> Even when NO_MMAP is empty, there might be no Unix sockets available (such\n> as is the case on Windows). In any case, you really only want to skip\n> these tests when index-helper is not available, so would you mind\n> squashing this patch in?\n>\n> -- snipsnap --\n> From: Johannes Schindelin \n> Subject: [PATCH] fixup! index-helper: new daemon for caching index and related\n> stuff\n>\n> ---\n> t/t7900-index-helper.sh | 6 ++++--\n> 1 file changed, 4 insertions(+), 2 deletions(-)\n>\n> diff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\n> index 6c9b4dd..12b5bf7 100755\n> --- a/t/t7900-index-helper.sh\n> +++ b/t/t7900-index-helper.sh\n> @@ -10,8 +10,10 @@ Testing git index-helper\n>\n> . ./test-lib.sh\n>\n> -test -n \"$NO_MMAP\" && {\n> - skip_all='skipping index-helper tests: no mmap'\n> +git index-helper -h 2>/dev/null\n> +test $? = 129 ||\n\nSo when NO_MMAP is set, \"git index-helper -h\" will set $? to 1. And we\ncorrectly skip the tests as well. It's a bit subtle though. How about\n\"git help -a|grep index-helper\"?\n\n> +{\n> + skip_all='skipping index-helper tests: no index-helper executable'\n> test_done\n> }\n>\n> --\n> 2.9.0.270.g810e421\n>\n\n\n\n-- \nDuy\n"},{"id":"290606","messageId":"loom.20160630T192750-125@post.gmane.org","threadId":"42713","inReplyTo":"1466914464-10358-11-git-send-email-novalis@novalis.org","subject":"Re: [PATCH v13 11/20] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2016-06-30T17:55:20Z","receivedAt":"2016-06-30T18:01:09Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"David Turner novalis.org> writes:\n\n> \n> Hiding watchman behind index-helper means you need both daemons. You\n> can't run watchman alone. Not so good. But on the other hand, 'git'\n> binary is not linked to watchman/json libraries, which is good for\n> packaging. Core git package will run fine without watchman-related\n> packages. If they need watchman, they can install git-index-helper and\n> dependencies.\n> \n\nHave you considered splitting index-helper and watchman apart? Using \nWatchman to not lstat unchanged entries is a huge perf win with very \nlarge repos.\n\nIt would also be interesting to make the Watchman backend replaceable by\nusing an extensible API. This has the benefit of not having to link the\n'git' binary to the watchman/json libraries. Is there any pattern \nalready in git for accomplishing this?\n\n"},{"id":"290608","messageId":"CACsJy8DjYLQCBRc9CzFSWNqkVnhbAfnxd1mnQh4oEfJwKjPd1A@mail.gmail.com","threadId":"42713","inReplyTo":"loom.20160630T192750-125@post.gmane.org","subject":"Re: [PATCH v13 11/20] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-30T19:14:44Z","receivedAt":"2016-06-30T19:15:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jun 30, 2016 at 7:55 PM, Ben Peart wrote:\n> David Turner novalis.org> writes:\n>\n>>\n>> Hiding watchman behind index-helper means you need both daemons. You\n>> can't run watchman alone. Not so good. But on the other hand, 'git'\n>> binary is not linked to watchman/json libraries, which is good for\n>> packaging. Core git package will run fine without watchman-related\n>> packages. If they need watchman, they can install git-index-helper and\n>> dependencies.\n>>\n>\n> Have you considered splitting index-helper and watchman apart? Using\n> Watchman to not lstat unchanged entries is a huge perf win with very\n> large repos.\n\nOn large repos (i.e. lots of files/dirs on worktree), the cost of\nreading index will increase proportionally. Yes lstat costs, but I\nsuspect index reading (integrity verification actually) may cost more,\nespecially on platforms with cheap lstat like linux. On these repos\nyou really want to enable all four: index-helper (with watchman),\nsplit-index (I still need to work out pruning on split-index) and\nuntracked cache. There's still a lot more to make git run fast on\nlarge repos though.\n\n> It would also be interesting to make the Watchman backend replaceable by\n> using an extensible API. This has the benefit of not having to link the\n> 'git' binary to the watchman/json libraries.\n\n'git' binary is not linked to watchman libraries. git-index-helper is\na separate binary, by design. In theory you can create a\n'git-index-helper' replacement binary with something other than\nwatchman. I think David documented the protocol well (it may change in\nthe future though and we are not prepared for capability progression)\n\n> Is there any pattern already in git for accomplishing this?\n-- \nDuy\n"},{"id":"290620","messageId":"loom.20160701T013515-311@post.gmane.org","threadId":"42713","inReplyTo":"CACsJy8DjYLQCBRc9CzFSWNqkVnhbAfnxd1mnQh4oEfJwKjPd1A@mail.gmail.com","subject":"Re: [PATCH v13 11/20] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2016-06-30T23:54:50Z","receivedAt":"2016-06-30T23:55:07Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Duy Nguyen gmail.com> writes:\n\n> \n> On Thu, Jun 30, 2016 at 7:55 PM, Ben Peart gmail.com> wrote:\n> > David Turner novalis.org> writes:\n> >\n> >>\n> >> Hiding watchman behind index-helper means you need both daemons. You\n> >> can't run watchman alone. Not so good. But on the other hand, 'git'\n> >> binary is not linked to watchman/json libraries, which is good for\n> >> packaging. Core git package will run fine without watchman-related\n> >> packages. If they need watchman, they can install git-index-helper and\n> >> dependencies.\n> >>\n> >\n> > Have you considered splitting index-helper and watchman apart? Using\n> > Watchman to not lstat unchanged entries is a huge perf win with very\n> > large repos.\n> \n> On large repos (i.e. lots of files/dirs on worktree), the cost of\n> reading index will increase proportionally. Yes lstat costs, but I\n> suspect index reading (integrity verification actually) may cost more,\n> especially on platforms with cheap lstat like linux. On these repos\n> you really want to enable all four: index-helper (with watchman),\n> split-index (I still need to work out pruning on split-index) and\n> untracked cache. There's still a lot more to make git run fast on\n> large repos though.\n> \n\nI've found (at least on Windows) that as the repo size gets larger, the\ntime to read the index becomes a much smaller percentage of the overall\ntime. I just captured some perf traces of git status on a large repo we\nhave. Of that, 92.5% was spent in git!read_directory and only 4.0% was \nspent in git!read_index. Of that 4%, 2.6% was git!glk_SHA1_Update.\n\nGiven there were no dirty files, Watchman would have made a huge \nimprovement in the overall time but index helper would have had\nrelatively little impact. We've noticed this same pattern in all our\nrepos which is what is driving our interest in the Watchman model and\nalso shows why index-helper is of less interest.\n\n> > It would also be interesting to make the Watchman backend replaceable by\n> > using an extensible API. This has the benefit of not having to link the\n> > 'git' binary to the watchman/json libraries.\n> \n> 'git' binary is not linked to watchman libraries. git-index-helper is\n> a separate binary, by design. In theory you can create a\n> 'git-index-helper' replacement binary with something other than\n> watchman. I think David documented the protocol well (it may change in\n> the future though and we are not prepared for capability progression)\n> \n> > Is there any pattern already in git for accomplishing this?\n\nWhile the current design hides watchman behind index-helper, if we were\nto change that model so they were independent, we would be interested\nin doing it in such a way that provided some abstraction so that it \ncould be replaced with another file watching daemon.\n\n"},{"id":"290746","messageId":"alpine.DEB.2.20.1607021312210.12947@virtualbox","threadId":"42713","inReplyTo":"CACsJy8DuTsDAYzRVk=mW7WX5CZb0Z5bAPnzV_2KXK-BcX=tcVg@mail.gmail.com","subject":"Re: [PATCH v13 04/20] index-helper: new daemon for caching index and related stuff","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-02T11:20:44Z","receivedAt":"2016-07-02T11:21:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Thu, 30 Jun 2016, Duy Nguyen wrote:\n\n> On Thu, Jun 30, 2016 at 3:06 PM, Johannes Schindelin\n> wrote:\n> > Even when NO_MMAP is empty, there might be no Unix sockets available (such\n> > as is the case on Windows). In any case, you really only want to skip\n> > these tests when index-helper is not available, so would you mind\n> > squashing this patch in?\n> >\n> > -- snipsnap --\n> > From: Johannes Schindelin \n> > Subject: [PATCH] fixup! index-helper: new daemon for caching index and related\n> > stuff\n> >\n> > ---\n> > t/t7900-index-helper.sh | 6 ++++--\n> > 1 file changed, 4 insertions(+), 2 deletions(-)\n> >\n> > diff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\n> > index 6c9b4dd..12b5bf7 100755\n> > --- a/t/t7900-index-helper.sh\n> > +++ b/t/t7900-index-helper.sh\n> > @@ -10,8 +10,10 @@ Testing git index-helper\n> >\n> > . ./test-lib.sh\n> >\n> > -test -n \"$NO_MMAP\" && {\n> > - skip_all='skipping index-helper tests: no mmap'\n> > +git index-helper -h 2>/dev/null\n> > +test $? = 129 ||\n> \n> So when NO_MMAP is set, \"git index-helper -h\" will set $? to 1.\n\nNot quite.\n\nWhen NO_MMAP is set, index-helper will not be compiled. Or at least it\nshould not be:\n\n> +ifndef NO_MMAP\n> +ifndef NO_UNIX_SOCKETS\n> + PROGRAM_OBJS += index-helper.o\n> +endif\n> +endif\n\nIf it is *unset*, *and* if NO_UNIX_SOCKETS is *also* unset, index-helper\ngets compiled, and -h triggers code in parse-options.c or usage.c that\nexits with status 129.\n\nSo I do not think that this is subtle.\n\nBut it just occurred to me that the #ifndef NO_MMAP in index-helper.c is\nunnecessary and that its #else clause (containing a loop() that fails)\ncontains dead code: we never compile this code with NO_MMAP, and neither\nshould we.\n\nDave, would you mind taking that #ifndef NO_MMAP out of \"index-helper: new\ndaemon for caching index and related stuff\" when you re-roll?\n\nThanks,\nDscho\n"},{"id":"290748","messageId":"CACsJy8Bke0EQ9rOEBmz6BcvcfPe-X_rktaOku9F-wUOrnVMJDw@mail.gmail.com","threadId":"42713","inReplyTo":"alpine.DEB.2.20.1607021312210.12947@virtualbox","subject":"Re: [PATCH v13 04/20] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-02T12:43:23Z","receivedAt":"2016-07-02T12:43:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 2, 2016 at 1:20 PM, Johannes Schindelin\n wrote:\n>> > -test -n \"$NO_MMAP\" && {\n>> > - skip_all='skipping index-helper tests: no mmap'\n>> > +git index-helper -h 2>/dev/null\n>> > +test $? = 129 ||\n>>\n>> So when NO_MMAP is set, \"git index-helper -h\" will set $? to 1.\n>\n> Not quite.\n>\n> When NO_MMAP is set, index-helper will not be compiled. Or at least it\n> should not be:\n>\n>> +ifndef NO_MMAP\n>> +ifndef NO_UNIX_SOCKETS\n>> + PROGRAM_OBJS += index-helper.o\n>> +endif\n>> +endif\n>\n> If it is *unset*, *and* if NO_UNIX_SOCKETS is *also* unset, index-helper\n> gets compiled, and -h triggers code in parse-options.c or usage.c that\n> exits with status 129.\n>\n> So I do not think that this is subtle.\n\nThe question is \"whether index-helper is supported?\" but you need to\ngo through parse-options.c and that strange (to me) exit status 129 to\nconclude that index-helper is in fact built. It's not just as straight\nforwarding from reading the test.\n\n> But it just occurred to me that the #ifndef NO_MMAP in index-helper.c is\n> unnecessary and that its #else clause (containing a loop() that fails)\n> contains dead code: we never compile this code with NO_MMAP, and neither\n> should we.\n\nYeah that's probably old code from POSIX shm time. Back then there\nwere calls to shm_open and stuff, not just mmap.\n\n> Dave, would you mind taking that #ifndef NO_MMAP out of \"index-helper: new\n> daemon for caching index and related stuff\" when you re-roll?\n-- \nDuy\n"},{"id":"290778","messageId":"CACsJy8DvZgj0w8z8LdeuY8HdTTcEu7ABPyFi_wnmEZOGsoMkDQ@mail.gmail.com","threadId":"42713","inReplyTo":"loom.20160701T013515-311@post.gmane.org","subject":"Re: [PATCH v13 11/20] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-03T12:28:20Z","receivedAt":"2016-07-03T12:28:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jul 1, 2016 at 1:54 AM, Ben Peart wrote:\n> I've found (at least on Windows) that as the repo size gets larger, the\n> time to read the index becomes a much smaller percentage of the overall\n> time. I just captured some perf traces of git status on a large repo we\n> have. Of that, 92.5% was spent in git!read_directory and only 4.0% was\n> spent in git!read_index. Of that 4%, 2.6% was git!glk_SHA1_Update.\n\nOK.. Windows. I don't have Windows numbers, but \"git!read_directory\"\n(is that read_directory() in \"git\" process?) does not do a lot of\nstat'ing. Although I agree that it dominates git-status. I recorded\nsome numbers in nd/untracked-cache topic (see 9e59724 (update-index:\nmanually enable or disable untracked cache - 2015-03-08)).\n\nFirst step would be enabling that because besides directory\ntraversing, this code does a lot of string processing that's hopefully\neliminated by untracked cache extension. I cut git-status' time in\nhalf with it. The side effect though, is that it creates a new flow of\nstat(), one per directory. It would be great if you could do some\nmeasurements with untracked cache on Windows and see if we get similar\ngain.\n\n> Given there were no dirty files, Watchman would have made a huge\n> improvement in the overall time but index helper would have had\n> relatively little impact. We've noticed this same pattern in all our\n> repos which is what is driving our interest in the Watchman model and\n> also shows why index-helper is of less interest.\n\nAssuming that untracked cache cuts git-status time by half on Windows\nas well, then index read time now takes a bigger percentage, 8%, where\nit starts to make sense to optimize it.\n\nOn a quiet repository, having watchman does not help so much because\nwe already reduce the significant number of filesystem-related system\ncalls. Yes there is lstat() and eliminating it may gain some more\n(with watchman) and it matters on a repo with lots of directories. You\nprobably can just take these lstat out (I can help point out where)\nand see how much the gain is.\n\nAssuming (blindly again) that removing lstat helps like 10% of\nread_index(), we would need to profile untracked cache code and see\nwhere what we can do next. There are still a lot of directory\ntraversing there (except that it traverses the cache instead of\nfilesystem) and maybe we can do something. But I haven't gotten that\nfar.\n\n> While the current design hides watchman behind index-helper, if we were\n> to change that model so they were independent, we would be interested\n> in doing it in such a way that provided some abstraction so that it\n> could be replaced with another file watching daemon.\n\nFrankly I'm not that interested in replacing another file watching\ndaemon. My first attempt at this problem was \"file-watcher\" which was\ntied to Linux inotify system only and it would make sense to make it\nreplaceable. But watchman supports OS X, Linux, FreeBSD and Windows\nnow, not just Linux only as before, why another abstraction layer? You\ncould even replace \"watchman.exe\" binary. As long as you produce the\nsame json data, your new daemon should still work.\n\nTying index caching daemon and file watching daemon (let's avoid\n\"watchman\" for now) gives us a bonus. Because both git and the caching\ndaemon know that they read the same index, we could answer the\nquestion \"what files are dirty?\" with \"file number 1, 5, 9 in the\nindex\" instead of sending full paths and git has to make some more\nlookups to identify them. In this series we send it as a compressed\nbit map. Probably not a big deal in terms of real performance, but it\nfeels nice to do lookups less.\n\nThe second reason is because watchman daemon alone does not provide\nenough information to reduce untracked cache's lstat() as much as\npossible. The current approach in this series is a naive one, which\nworks for some cases, but not optimal (I'll get to that). We need a\nseparate long-running daemon to maintain extra info to reduce lstat().\nBecause our target is watchman, it does not make sense to add yet\nanother daemon besides index-helper to do this.\n\nOK the optimal lstat() reduction thing. Right now, if any file in a\ndirectory is updated, the directory is invalidated in untracked cache\nand we need to traverse it to collect excluded files again. But it\ndoes not have to be that way. We don't care if any file is _updated_\nbecause it will not change untracked cache output. We care about what\nfiles are _added_ or _deleted_. New files will need to be classified\nas either tracked, untracked or ignored. Deleted files may invalid\neither three file lists. Watchman cannot answer \"what files are added\nor deleted since the point X in time\" and I agree that it's not\nwatchman's job (watchman issue 65). So we have to maintain some more\ninfo by ourselves, e.g. the list of files at any requested \"clock\".\nWith that we can compare the file lists of two \"clock\"s and tell git\nwhat files are added or deleted.\n-- \nDuy\n"},{"id":"290890","messageId":"loom.20160706T170547-355@post.gmane.org","threadId":"42713","inReplyTo":"CACsJy8DvZgj0w8z8LdeuY8HdTTcEu7ABPyFi_wnmEZOGsoMkDQ@mail.gmail.com","subject":"Re: [PATCH v13 11/20] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2016-07-06T16:54:00Z","receivedAt":"2016-07-06T16:54:20Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Duy Nguyen gmail.com> writes:\n\n> \n> First step would be enabling that because besides directory\n> traversing, this code does a lot of string processing that's hopefully\n> eliminated by untracked cache extension. I cut git-status' time in\n> half with it. The side effect though, is that it creates a new flow of\n> stat(), one per directory. It would be great if you could do some\n> measurements with untracked cache on Windows and see if we get similar\n> gain.\n \nThese numbers were captured with core.fscache and core.untrackedcache \nboth set to true in the belief that it would produce the best \nperformance. If that is incorrect, I'm happy to capture numbers with \nother options set. \n\nIf you drill the next step down into the call tree, the bulk of the cost\nof read_directory is coming from mingw_stat (85.6%). These numbers are \ninclusive in that they reflect the of the function plus any of its \ncallees. \n\nIf you look at the overall cost of functions exclusively (ie that only \ninclude the cost of the function and not it's children), \nkernelbase!CreateFileW, kernelbase!CloseHandle, and \nkernebase!GetFileInformationByHandle dominate (84.9% of the time). \nThese functions form the basis of the stat emulation on Windows although\nthe fscache certainly has an impact on what is happening here as well.\n\nWhile we can certainly work to speed these up, the biggest performance \nwin will be eliminating as many of these calls as possible which is what\nwe are hoping to accomplish by using Watchman to limit the files that \n_need_ a stat call.\n\n> > Given there were no dirty files, Watchman would have made a huge\n> > improvement in the overall time but index helper would have had\n> > relatively little impact. We've noticed this same pattern in all our\n> > repos which is what is driving our interest in the Watchman model and\n> > also shows why index-helper is of less interest.\n> \n> Assuming that untracked cache cuts git-status time by half on Windows\n> as well, then index read time now takes a bigger percentage, 8%, where\n> it starts to make sense to optimize it.\n> \n> On a quiet repository, having watchman does not help so much because\n> we already reduce the significant number of filesystem-related system\n> calls. Yes there is lstat() and eliminating it may gain some more\n> (with watchman) and it matters on a repo with lots of directories. You\n> probably can just take these lstat out (I can help point out where)\n> and see how much the gain is.\n\nI don't understand why Watchman won't provide a _significant_ improvement\nhere. My understanding is that it will minimize the stat calls to those\nfiles that may have changed as well as limiting the untracked cache to\nonly scanning those directories that may have changes in them. In this\nparticular scenario, _no_ files have changed so Watchman would return\nan empty set thus eliminating virtually the stat calls and directory\nenumerations. I'd expect this to result in a >90% savings. Am I \nmissing something?\n\n> Assuming (blindly again) that removing lstat helps like 10% of\n> read_index(), we would need to profile untracked cache code and see\n> where what we can do next. There are still a lot of directory\n> traversing there (except that it traverses the cache instead of\n> filesystem) and maybe we can do something. But I haven't gotten that\n> far.\n> \n> > While the current design hides watchman behind index-helper, if we were\n> > to change that model so they were independent, we would be interested\n> > in doing it in such a way that provided some abstraction so that it\n> > could be replaced with another file watching daemon.\n> \n> Frankly I'm not that interested in replacing another file watching\n> daemon. My first attempt at this problem was \"file-watcher\" which was\n> tied to Linux inotify system only and it would make sense to make it\n> replaceable. But watchman supports OS X, Linux, FreeBSD and Windows\n> now, not just Linux only as before, why another abstraction layer? You\n> could even replace \"watchman.exe\" binary. As long as you produce the\n> same json data, your new daemon should still work.\n\nThis is a simplification it would be nice to make as we have other code\nalready running that can report on all the changes happening. It would\nenable us to remove the additional complexity of the Watchman daemon. \nI'm sure we can make it work either way by emulating the Watchman \ninterface and output.\n\n> Tying index caching daemon and file watching daemon (let's avoid\n> \"watchman\" for now) gives us a bonus. Because both git and the caching\n> daemon know that they read the same index, we could answer the\n> question \"what files are dirty?\" with \"file number 1, 5, 9 in the\n> index\" instead of sending full paths and git has to make some more\n> lookups to identify them. In this series we send it as a compressed\n> bit map. Probably not a big deal in terms of real performance, but it\n> feels nice to do lookups less.\n\nToday, Watchman returns a list of files and directories and then creates\nthe compressed bitmap that index-helper uses. The work of looking those\nentries up in the index and then creating the bitmap still has to happen\nso I suspect you are correct that it doesn't make much of a real \nperformance difference. It's just moving where that lookup and bitmap\ncreation happens.\n\nI'm in the process of prototyping this, and currently skip much of the \nprocess of iterating through the list of changed files, looking up the \nentry in the index, creating the bitmap, passing that bitmap through the \nWAMA section to index-helper reading the WAMA section in git and then \nlooping through the bitmap to set the CE_WATCHMAN_DIRTY bit on the \ncorresponding index entries and updating the untracked cache.\n\nInstead, I iterate through the list of changed files, look up the entry \nin the index and directly set the dirty bit all within read-cache.c. \nAt this point, it's <100 lines of code. I'll keep fleshing this out\nand get some perf numbers once it's working.\n\nWhile this saves several steps and reduces complexity (index-helper and \nthe WAMA is section are no longer involved), it does mean that \nread-cache.c needs a platform independent way to query and receive the \nlist of modified files and directories. It also means we need a way to \nstore the date/time string passed to the file watching daemon in the \nindex. Using the WAMA section just to store the date/time seems a bit \nheavy but it can work.\n\n> The second reason is because watchman daemon alone does not provide\n> enough information to reduce untracked cache's lstat() as much as\n> possible. The current approach in this series is a naive one, which\n> works for some cases, but not optimal (I'll get to that). We need a\n> separate long-running daemon to maintain extra info to reduce lstat().\n> Because our target is watchman, it does not make sense to add yet\n> another daemon besides index-helper to do this.\n> \n> OK the optimal lstat() reduction thing. Right now, if any file in a\n> directory is updated, the directory is invalidated in untracked cache\n> and we need to traverse it to collect excluded files again. But it\n> does not have to be that way. We don't care if any file is _updated_\n> because it will not change untracked cache output. We care about what\n> files are _added_ or _deleted_. New files will need to be classified\n> as either tracked, untracked or ignored. Deleted files may invalid\n> either three file lists. Watchman cannot answer \"what files are added\n> or deleted since the point X in time\" and I agree that it's not\n> watchman's job (watchman issue 65). So we have to maintain some more\n> info by ourselves, e.g. the list of files at any requested \"clock\".\n> With that we can compare the file lists of two \"clock\"s and tell git\n> what files are added or deleted.\n\nThis sounds like a nice optimization but right now I'm focused on how we\ncan scope the cost of status to be limited to the files and directories\nthat may have changes. With large repos, this is a small subset of the\noverall repo and will hopefully be enough to make the performance \nreasonable.\n\n\n"},{"id":"291034","messageId":"20160708163240.GA20978@duynguyen","threadId":"42713","inReplyTo":"loom.20160706T170547-355@post.gmane.org","subject":"Re: [PATCH v13 11/20] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-08T16:32:40Z","receivedAt":"2016-07-08T16:32:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 06, 2016 at 04:54:00PM +0000, Ben Peart wrote:\n> Duy Nguyen gmail.com> writes:\n>\n> >\n> > First step would be enabling that because besides directory\n> > traversing, this code does a lot of string processing that's hopefully\n> > eliminated by untracked cache extension. I cut git-status' time in\n> > half with it. The side effect though, is that it creates a new flow of\n> > stat(), one per directory. It would be great if you could do some\n> > measurements with untracked cache on Windows and see if we get similar\n> > gain.\n>\n> These numbers were captured with core.fscache and core.untrackedcache\n> both set to true in the belief that it would produce the best\n> performance.\n\nSo I guessed it wrong :-) I thought untracked cache was still not popular.\n\n> If that is incorrect, I'm happy to capture numbers with other\n> options set.\n\nIt's not incorrect. But if you are happy to provide numbers, I'm happy\nto read them and provide my opinions.\n\n> If you drill the next step down into the call tree, the bulk of the cost\n> of read_directory is coming from mingw_stat (85.6%). These numbers are\n> inclusive in that they reflect the of the function plus any of its\n> callees.\n\nIf I remember it correctly, fscache changes mingw_stat() to\nFindFirst.. and FindNext... basically read the whole directory up. The\nassumption is, if you stat(a/b) you probably want to stat(a/c) and\nstat(a/d) soon too. And stat() on Windows is slower in that case\n(which is a real case, index refresh).\n\nUntracked cache does not follow this pattern, it stat() all\ndirectories an no files (ok, _some_ files, like .gitignore), which is\nnot what fscache is optimized for. It would be good if you can get\nsome numbers with untracked cache but _without_ fscache.\n\nYou should kill the lstat() stream from index refresh though because\nit may interfere. Doing that is as simple as this. This could provide\nsome good base line for measuring read_directory().\n\ndiff --git a/read-cache.c b/read-cache.c\nindex db27766..1cef379 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1079,7 +1079,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t * that the change to the work tree does not matter and told\n \t * us not to worry.\n \t */\n-\tif (!ignore_skip_worktree && ce_skip_worktree(ce)) {\n+\tif (1 || !ignore_skip_worktree && ce_skip_worktree(ce)) {\n \t\tce_mark_uptodate(ce);\n \t\treturn ce;\n \t}\n\n\n> If you look at the overall cost of functions exclusively (ie that only\n> include the cost of the function and not it's children),\n> kernelbase!CreateFileW, kernelbase!CloseHandle, and\n> kernebase!GetFileInformationByHandle dominate (84.9% of the time).\n> These functions form the basis of the stat emulation on Windows although\n> the fscache certainly has an impact on what is happening here as well.\n\nYeah.. I'm hoping to see something different without fscache. But\nprobably not much.\n\n> > > Given there were no dirty files, Watchman would have made a huge\n> > > improvement in the overall time but index helper would have had\n> > > relatively little impact. We've noticed this same pattern in all our\n> > > repos which is what is driving our interest in the Watchman model and\n> > > also shows why index-helper is of less interest.\n> >\n> > Assuming that untracked cache cuts git-status time by half on Windows\n> > as well, then index read time now takes a bigger percentage, 8%, where\n> > it starts to make sense to optimize it.\n> >\n> > On a quiet repository, having watchman does not help so much because\n> > we already reduce the significant number of filesystem-related system\n> > calls. Yes there is lstat() and eliminating it may gain some more\n> > (with watchman) and it matters on a repo with lots of directories. You\n> > probably can just take these lstat out (I can help point out where)\n> > and see how much the gain is.\n>\n> I don't understand why Watchman won't provide a _significant_ improvement\n> here. My understanding is that it will minimize the stat calls to those\n> files that may have changed as well as limiting the untracked cache to\n> only scanning those directories that may have changes in them. In this\n> particular scenario, _no_ files have changed so Watchman would return\n> an empty set thus eliminating virtually the stat calls and directory\n> enumerations. I'd expect this to result in a >90% savings. Am I\n> missing something?\n\nIf stat() calls consume most of the time by read_directory() then yes\nwatchman provides significant improvement. I was writing that thinking\nyou did not enable untracked cache. It's a big leap with and without\nuntracked cache. Watchman's performance gain is smaller in comparison\nto that (again I can be really wrong when it comes to Windows stuff).\n\n\n> > > While the current design hides watchman behind index-helper, if we were\n> > > to change that model so they were independent, we would be interested\n> > > in doing it in such a way that provided some abstraction so that it\n> > > could be replaced with another file watching daemon.\n> >\n> > Frankly I'm not that interested in replacing another file watching\n> > daemon. My first attempt at this problem was \"file-watcher\" which was\n> > tied to Linux inotify system only and it would make sense to make it\n> > replaceable. But watchman supports OS X, Linux, FreeBSD and Windows\n> > now, not just Linux only as before, why another abstraction layer? You\n> > could even replace \"watchman.exe\" binary. As long as you produce the\n> > same json data, your new daemon should still work.\n>\n> This is a simplification it would be nice to make as we have other code\n> already running that can report on all the changes happening. It would\n> enable us to remove the additional complexity of the Watchman daemon.\n> I'm sure we can make it work either way by emulating the Watchman\n> interface and output.\n\nIn the spririt of scratching own itches, please feel free to suggest\nsome changes to split the watchman interface out of index-helper. It\nfeels like a burden to me, but I am just a contributor like you.\n\n>\n> > Tying index caching daemon and file watching daemon (let's avoid\n> > \"watchman\" for now) gives us a bonus. Because both git and the caching\n> > daemon know that they read the same index, we could answer the\n> > question \"what files are dirty?\" with \"file number 1, 5, 9 in the\n> > index\" instead of sending full paths and git has to make some more\n> > lookups to identify them. In this series we send it as a compressed\n> > bit map. Probably not a big deal in terms of real performance, but it\n> > feels nice to do lookups less.\n>\n> Today, Watchman returns a list of files and directories and then creates\n> the compressed bitmap that index-helper uses. The work of looking those\n> entries up in the index and then creating the bitmap still has to happen\n> so I suspect you are correct that it doesn't make much of a real\n> performance difference. It's just moving where that lookup and bitmap\n> creation happens.\n\nWe could do the lookups in background, before git requests it, so the\ncost would be hidden. But yeah, numbers decide whether this is an evil\nmicro-optimization or not.\n\n\n> I'm in the process of prototyping this, and currently skip much of the\n> process of iterating through the list of changed files, looking up the\n> entry in the index, creating the bitmap, passing that bitmap through the\n> WAMA section to index-helper reading the WAMA section in git and then\n> looping through the bitmap to set the CE_WATCHMAN_DIRTY bit on the\n> corresponding index entries and updating the untracked cache.\n>\n> Instead, I iterate through the list of changed files, look up the entry\n> in the index and directly set the dirty bit all within read-cache.c.\n> At this point, it's <100 lines of code. I'll keep fleshing this out\n> and get some perf numbers once it's working.\n\nGreat!\n\n> > The second reason is because watchman daemon alone does not provide\n> > enough information to reduce untracked cache's lstat() as much as\n> > possible. The current approach in this series is a naive one, which\n> > works for some cases, but not optimal (I'll get to that). We need a\n> > separate long-running daemon to maintain extra info to reduce lstat().\n> > Because our target is watchman, it does not make sense to add yet\n> > another daemon besides index-helper to do this.\n> >\n> > OK the optimal lstat() reduction thing. Right now, if any file in a\n> > directory is updated, the directory is invalidated in untracked cache\n> > and we need to traverse it to collect excluded files again. But it\n> > does not have to be that way. We don't care if any file is _updated_\n> > because it will not change untracked cache output. We care about what\n> > files are _added_ or _deleted_. New files will need to be classified\n> > as either tracked, untracked or ignored. Deleted files may invalid\n> > either three file lists. Watchman cannot answer \"what files are added\n> > or deleted since the point X in time\" and I agree that it's not\n> > watchman's job (watchman issue 65). So we have to maintain some more\n> > info by ourselves, e.g. the list of files at any requested \"clock\".\n> > With that we can compare the file lists of two \"clock\"s and tell git\n> > what files are added or deleted.\n>\n> This sounds like a nice optimization but right now I'm focused on how we\n> can scope the cost of status to be limited to the files and directories\n> that may have changes. With large repos, this is a small subset of the\n> overall repo and will hopefully be enough to make the performance\n> reasonable.\n\nThe use case in my mind is building stuff where .o files are put in\nthe same place as source files, and such a build would touch everything.\nI certainly am not familar with what most real large repos look like\nand how they are used. So I may be wrong.\n--\nDuy\n"},{"id":"291037","messageId":"CACsJy8Dz7LHPQp2drE9EeqFzvjm1RHo_mA0ytfhXMgrcZBMgPw@mail.gmail.com","threadId":"42713","inReplyTo":"loom.20160706T170547-355@post.gmane.org","subject":"Re: [PATCH v13 11/20] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-08T16:39:44Z","receivedAt":"2016-07-08T16:40:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"BTW just for me to have some perspective, roughly how many directories\nand files are there in the worktree of this repo?\n-- \nDuy\n"}]}