{"thread":{"id":"41948","subject":"[PATCH v3 01/16] read-cache.c: fix constness of verify_hdr()","startedAt":"2016-04-06T22:11:46Z","lastAt":"2016-04-15T01:31:53Z","messageCount":37,"participants":["David Turner","Johannes Sixt","Johannes Schindelin","Junio C Hamano","Duy Nguyen"],"isPatch":true,"patchVersion":3,"patchTotal":16},"messages":[{"id":"282796","messageId":"1459980722-4836-1-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":null,"subject":"[PATCH v3 00/16] index-helper, watchman","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:46Z","receivedAt":"2016-04-06T22:11:46Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"The major change here is to switch from named pipes to sockets. But somehow\nwhile doing this, I noticed a few other things (and others noticed some too):\n\nFixed bugs in successful write testing\nAdded a bit more doco\nImproved tests slightly\nMemory barriers\n.git vs .git/ from watchman\none patch moved/squashed at Duy's suggestion\n(some of?) my spelling errors corrected\n\nDavid Turner (6):\n unpack-trees: preserve index extensions\n index-helper: kill mode\n index-helper: don't run if already running\n index-helper: autorun mode\n index-helper: optionally automatically run\n read-cache: config for waiting for index-helper\n\nNguyễn Thái Ngọc Duy (10):\n read-cache.c: fix constness of verify_hdr()\n read-cache: allow to keep mmap'd memory after reading\n index-helper: new daemon for caching index and related stuff\n index-helper: add --strict\n daemonize(): set a flag before exiting the main process\n index-helper: add --detach\n read-cache: add watchman 'WAMA' extension\n Add watchman support to reduce index refresh cost\n index-helper: use watchman to avoid refreshing index with lstat()\n update-index: enable/disable watchman support\n\n .gitignore | 1 +\n Documentation/config.txt | 10 +\n Documentation/git-index-helper.txt | 64 +++++\n Makefile | 22 ++\n builtin/gc.c | 2 +-\n builtin/update-index.c | 11 +\n cache.h | 23 +-\n config.c | 10 +\n config.mak.uname | 1 +\n configure.ac | 8 +\n daemon.c | 2 +-\n dir.c | 23 +-\n dir.h | 6 +\n environment.c | 8 +\n git-compat-util.h | 18 ++\n git.c | 35 ++-\n index-helper.c | 462 +++++++++++++++++++++++++++++++++++++\n read-cache.c | 438 +++++++++++++++++++++++++++++++++--\n setup.c | 4 +-\n shm.c | 67 ++++++\n shm.h | 23 ++\n t/t7063-status-untracked-cache.sh | 22 ++\n t/t7900-index-helper.sh | 69 ++++++\n t/test-lib-functions.sh | 4 +\n unpack-trees.c | 1 +\n watchman-support.c | 135 +++++++++++\n watchman-support.h | 7 +\n 27 files changed, 1454 insertions(+), 22 deletions(-)\n create mode 100644 Documentation/git-index-helper.txt\n create mode 100644 index-helper.c\n create mode 100644 shm.c\n create mode 100644 shm.h\n create mode 100755 t/t7900-index-helper.sh\n create mode 100644 watchman-support.c\n create mode 100644 watchman-support.h\n\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282794","messageId":"1459980722-4836-2-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 01/16] read-cache.c: fix constness of verify_hdr()","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:47Z","receivedAt":"2016-04-06T22:11:47Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex d9fb78b..16cc487 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1345,7 +1345,7 @@ struct ondisk_cache_entry_extended {\n \t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr(struct cache_header *hdr, unsigned long size)\n+static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n {\n \tgit_SHA_CTX c;\n \tunsigned char sha1[20];\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282795","messageId":"1459980722-4836-3-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 02/16] read-cache: allow to keep mmap'd memory after reading","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:48Z","receivedAt":"2016-04-06T22:11:48Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nLater, we will introduce git index-helper to share this memory with\nother git processes.\n\nSince the memory will be shared, it will never be unmapped (although\nthe kernel may of course choose to page it out).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 3 +++\n read-cache.c | 13 ++++++++++++-\n 2 files changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex b829410..4180e2b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -333,11 +333,14 @@ struct index_state {\n \tstruct split_index *split_index;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n+\t\t keep_mmap : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n \tstruct untracked_cache *untracked;\n+\tvoid *mmap;\n+\tsize_t mmap_size;\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 16cc487..7e387e9 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1574,6 +1574,10 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tmmap = xmmap(NULL, mmap_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n+\tif (istate->keep_mmap) {\n+\t\tistate->mmap = mmap;\n+\t\tistate->mmap_size = mmap_size;\n+\t}\n \tclose(fd);\n \n \thdr = mmap;\n@@ -1626,10 +1630,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n-\tmunmap(mmap, mmap_size);\n+\tif (!istate->keep_mmap)\n+\t\tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n unmap:\n+\tistate->mmap = NULL;\n \tmunmap(mmap, mmap_size);\n \tdie(\"index file corrupt\");\n }\n@@ -1655,6 +1661,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdiscard_index(split_index->base);\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n+\tsplit_index->base->keep_mmap = istate->keep_mmap;\n \tret = do_read_index(split_index->base,\n \t\t\t git_path(\"sharedindex.%s\",\n \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n@@ -1698,6 +1705,10 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->keep_mmap && istate->mmap) {\n+\t\tmunmap(istate->mmap, istate->mmap_size);\n+\t\tistate->mmap = NULL;\n+\t}\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282798","messageId":"1459980722-4836-4-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:49Z","receivedAt":"2016-04-06T22:11:49Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nInstead of reading the index from disk and worrying about disk\ncorruption, the index is cached in memory (memory bit-flips happen\ntoo, but hopefully less often). The result is faster read. Read time\nis reduced by 70%.\n\nThe biggest gain is not having to verify the trailing SHA-1, which\ntakes lots of time especially on large index files. But this also\nopens doors for further optimiztions:\n\n - we could create an in-memory format that's essentially the memory\n dump of the index to eliminate most of parsing/allocation\n overhead. The mmap'd memory can be used straight away. Experiment\n [1] shows we could reduce read time by 88%.\n\n - we could cache non-index info such as name hash\n\nThe shared memory's name folows the template \"git--\"\nwhere is the trailing SHA-1 of the index file. is\n\"index\" for cached index files (and may be \"name-hash\" for name-hash\ncache). If such shared memory exists, it contains the same index\ncontent as on disk. The content is already validated by the daemon and\ngit won't validate it again (except comparing the trailing SHA-1s).\n\nWe keep this daemon's logic as thin as possible. The \"brain\" stays in\ngit. So the daemon can read and validate stuff, but that's all it's\nallowed to do. It does not add/create new information. It doesn't even\naccept direct updates from git.\n\nGit can poke the daemon via unix domain sockets to tell it to refresh\nthe index cache, or to keep it alive some more minutes. It can't give\nany real index data directly to the daemon. Real data goes to disk\nfirst, then the daemon reads and verifies it from there. Poking only\nhappens for $GIT_DIR/index, not temporary index files.\n\n$GIT_DIR/index-helper.path is a symlink to the socket for the daemon\nprocess. The daemon reads from the socket and executes commands.\n\nNamed pipes were considered for portability reasons, but then commands\nthat need replies from the daemon would have open their own pipes,\nsince a named pipe should only have one reader. Unix domain sockets\ndon't have this problem.\n\nindex-helper requires POSIX realtime extension. POSIX shm interface\nhowever is abstracted away so that Windows support can be added later.\n\nOn webkit.git with index format v2, duplicating 8 times to 1.4m\nentries and 200MB in size:\n\n(vanilla) 0.986986364 s: read_index_from .git/index\n(index-helper) 0.267850279 s: read_index_from .git/index\n\nInterestingly with index v4, we get less out of index-helper. It makes\nsense as v4 requires more processing after loading the index:\n\n(vanilla) 0.722496666 s: read_index_from .git/index\n(index-helper) 0.302741500 s: read_index_from .git/index\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/247268/focus=248771\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \n---\n .gitignore | 1 +\n Documentation/git-index-helper.txt | 46 ++++++\n Makefile | 10 ++\n cache.h | 2 +\n config.mak.uname | 1 +\n git-compat-util.h | 18 +++\n index-helper.c | 284 +++++++++++++++++++++++++++++++++++++\n read-cache.c | 116 +++++++++++++--\n shm.c | 67 +++++++++\n shm.h | 23 +++\n t/t7900-index-helper.sh | 23 +++\n 11 files changed, 582 insertions(+), 9 deletions(-)\n create mode 100644 Documentation/git-index-helper.txt\n create mode 100644 index-helper.c\n create mode 100644 shm.c\n create mode 100644 shm.h\n create mode 100755 t/t7900-index-helper.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 5087ce1..b92f122 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -71,6 +71,7 @@\n /git-http-fetch\n /git-http-push\n /git-imap-send\n+/git-index-helper\n /git-index-pack\n /git-init\n /git-init-db\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nnew file mode 100644\nindex 0000000..61605e9\n--- /dev/null\n+++ b/Documentation/git-index-helper.txt\n@@ -0,0 +1,46 @@\n+git-index-helper(1)\n+===================\n+\n+NAME\n+----\n+git-index-helper - A simple cache daemon for speeding up index file access\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git index-helper' [options]\n+\n+DESCRIPTION\n+-----------\n+Keep the index file in memory for faster access. This daemon is per\n+repository.\n+\n+OPTIONS\n+-------\n+\n+--exit-after=::\n+\tExit if the cached index is not accessed for ``\n+\tseconds. Specify 0 to wait forever. Default is 600.\n+\n+NOTES\n+-----\n+$GIT_DIR/index-helper.path is a symlink to a Unix domain socket in\n+$TMPDIR that the daemon reads commands from. At least on Linux, shared\n+memory objects are available via /dev/shm with the name pattern\n+\"git--\". Normally the daemon will clean up shared\n+memory objects when it exits. But if it crashes, some objects could\n+remain there and they can be safely deleted with \"rm\" command. The\n+following commands are used to control the daemon:\n+\n+\"refresh\"::\n+\tReread the index.\n+\n+\"poke\":\n+\tLet the daemon know the index is to be read. It keeps the\n+\tdaemon alive longer, unless `--exit-after=0` is used.\n+\n+All commands and replies are terminated by a 0 byte.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 2742a69..24d6c0e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -370,6 +370,8 @@ all::\n # Define HAVE_BSD_SYSCTL if your platform has a BSD-compatible sysctl function.\n #\n # Define HAVE_GETDELIM if your system has the getdelim() function.\n+#\n+# Define HAVE_SHM if you platform support shm_* functions in librt.\n \n GIT-VERSION-FILE: FORCE\n \t@$(SHELL_PATH) ./GIT-VERSION-GEN\n@@ -805,6 +807,7 @@ LIB_OBJS += sha1-lookup.o\n LIB_OBJS += sha1_file.o\n LIB_OBJS += sha1_name.o\n LIB_OBJS += shallow.o\n+LIB_OBJS += shm.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n LIB_OBJS += split-index.o\n@@ -1433,6 +1436,12 @@ ifdef HAVE_DEV_TTY\n \tBASIC_CFLAGS += -DHAVE_DEV_TTY\n endif\n \n+ifdef HAVE_SHM\n+\tBASIC_CFLAGS\t+= -DHAVE_SHM\n+\tEXTLIBS\t += -lrt\n+\tPROGRAM_OBJS\t+= index-helper.o\n+endif\n+\n ifdef DIR_HAS_BSD_GROUP_SEMANTICS\n \tCOMPAT_CFLAGS += -DDIR_HAS_BSD_GROUP_SEMANTICS\n endif\n@@ -2159,6 +2168,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n+\t@echo HAVE_SHM=\\''$(subst ','\\'',$(subst ','\\'',$(HAVE_SHM)))'\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/cache.h b/cache.h\nindex 4180e2b..43fb314 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -334,6 +334,8 @@ struct index_state {\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n \t\t keep_mmap : 1,\n+\t\t from_shm : 1,\n+\t\t to_shm : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 1139b44..d28d05d 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -38,6 +38,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_CLOCK_MONOTONIC = YesPlease\n \tHAVE_GETDELIM = YesPlease\n \tSANE_TEXT_GREP=-a\n+\tHAVE_SHM = YesPlease\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tHAVE_ALLOCA_H = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex c07e0c1..8b878fe 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -513,6 +513,7 @@ static inline int ends_with(const char *str, const char *suffix)\n #define PROT_READ 1\n #define PROT_WRITE 2\n #define MAP_PRIVATE 1\n+#define MAP_SHARED 2\n #endif\n \n #define mmap git_mmap\n@@ -1045,4 +1046,21 @@ struct tm *git_gmtime_r(const time_t *, struct tm *);\n #define getc_unlocked(fh) getc(fh)\n #endif\n \n+#ifdef __linux__\n+#define UNIX_PATH_MAX 108\n+#elif defined(__APPLE__) || defined(BSD)\n+#define UNIX_PATH_MAX 104\n+#else\n+/*\n+ * Quoth POSIX: The size of sun_path has intentionally been left\n+ * undefined. This is because different implementations use different\n+ * sizes. For example, 4.3 BSD uses a size of 108, and 4.4 BSD uses a\n+ * size of 104. Since most implementations originate from BSD\n+ * versions, the size is typically in the range 92 to 108.\n+ *\n+ * Thanks, POSIX! Super-helpful! Hope we don't overflow any buffers!\n+ */\n+#define UNIX_PATH_MAX 92\n+#endif\n+\n #endif\ndiff --git a/index-helper.c b/index-helper.c\nnew file mode 100644\nindex 0000000..263b066\n--- /dev/null\n+++ b/index-helper.c\n@@ -0,0 +1,284 @@\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"sigchain.h\"\n+#include \"strbuf.h\"\n+#include \"exec_cmd.h\"\n+#include \"split-index.h\"\n+#include \"shm.h\"\n+#include \"lockfile.h\"\n+\n+struct shm {\n+\tunsigned char sha1[20];\n+\tvoid *shm;\n+\tsize_t size;\n+};\n+\n+static struct shm shm_index;\n+static struct shm shm_base_index;\n+\n+static void release_index_shm(struct shm *is)\n+{\n+\tif (!is->shm)\n+\t\treturn;\n+\tmunmap(is->shm, is->size);\n+\tgit_shm_unlink(\"git-index-%s\", sha1_to_hex(is->sha1));\n+\tis->shm = NULL;\n+}\n+\n+static void cleanup_shm(void)\n+{\n+\trelease_index_shm(&shm_index);\n+\trelease_index_shm(&shm_base_index);\n+}\n+\n+static void cleanup(void)\n+{\n+\tunlink(git_path(\"index-helper.path\"));\n+\tcleanup_shm();\n+}\n+\n+static void cleanup_on_signal(int sig)\n+{\n+\t/* We ignore sigpipes -- that's just a client being broken. */\n+\tif (sig == SIGPIPE)\n+\t\treturn;\n+\tcleanup();\n+\tsigchain_pop(sig);\n+\traise(sig);\n+}\n+\n+static void share_index(struct index_state *istate, struct shm *is)\n+{\n+\tvoid *new_mmap;\n+\tif (istate->mmap_size <= 20 ||\n+\t hashcmp(istate->sha1,\n+\t\t (unsigned char *)istate->mmap + istate->mmap_size - 20) ||\n+\t !hashcmp(istate->sha1, is->sha1) ||\n+\t git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, istate->mmap_size,\n+\t\t\t&new_mmap, PROT_READ | PROT_WRITE, MAP_SHARED,\n+\t\t\t\"git-index-%s\", sha1_to_hex(istate->sha1)) < 0)\n+\t\treturn;\n+\n+\trelease_index_shm(is);\n+\tis->size = istate->mmap_size;\n+\tis->shm = new_mmap;\n+\thashcpy(is->sha1, istate->sha1);\n+\tmemcpy(new_mmap, istate->mmap, istate->mmap_size - 20);\n+\n+\t/*\n+\t * The trailing hash must be written last after everything is\n+\t * written. It's the indication that the shared memory is now\n+\t * ready.\n+\t * The memory barrier here matches read-cache.c:try_shm.\n+\t */\n+\t__sync_synchronize();\n+\n+\thashcpy((unsigned char *)new_mmap + istate->mmap_size - 20, is->sha1);\n+}\n+\n+static void share_the_index(void)\n+{\n+\tif (the_index.split_index && the_index.split_index->base)\n+\t\tshare_index(the_index.split_index->base, &shm_base_index);\n+\tshare_index(&the_index, &shm_index);\n+\tdiscard_index(&the_index);\n+}\n+\n+static void refresh(void)\n+{\n+\tthe_index.keep_mmap = 1;\n+\tthe_index.to_shm = 1;\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"could not read index\"));\n+\tshare_the_index();\n+}\n+\n+static void set_socket_nonblocking(int fd)\n+{\n+\tint flags;\n+\n+\tflags = fcntl(fd, F_GETFL, NULL);\n+\n+\tif (flags < 0)\n+\t\tdie(_(\"fcntl failed\"));\n+\n+\tif (fcntl(fd, F_SETFL, flags | O_NONBLOCK) < 0)\n+\t\tdie(_(\"fcntl failed\"));\n+}\n+\n+#ifdef HAVE_SHM\n+\n+static void loop(int fd, int idle_in_seconds)\n+{\n+\tstruct timeval timeout;\n+\tstruct timeval *timeout_p;\n+\n+\twhile (1) {\n+\t\tfd_set readfds;\n+\t\tint result, client_fd;\n+\t\tstruct strbuf command = STRBUF_INIT;\n+\n+\t\t/* need to reset timer in case select() decremented it */\n+\t\tif (idle_in_seconds) {\n+\t\t\ttimeout.tv_usec = 0;\n+\t\t\ttimeout.tv_sec = idle_in_seconds;\n+\t\t\ttimeout_p = &timeout;\n+\t\t} else {\n+\t\t\ttimeout_p = NULL;\n+\t\t}\n+\n+\t\t/* Wait for a request */\n+\t\tFD_ZERO(&readfds);\n+\t\tFD_SET(fd, &readfds);\n+\t\tresult = select(fd + 1, &readfds, NULL, NULL, timeout_p);\n+\t\tif (result < 0)\n+\t\t\tdie_errno(_(\"select() failed\"));\n+\t\tif (result == 0)\n+\t\t\t/* timeout */\n+\t\t\tbreak;\n+\n+\t\tclient_fd = accept(fd, NULL, NULL);\n+\t\tif (client_fd < 0)\n+\t\t\t/*\n+\t\t\t * An error here is unlikely -- it probably\n+\t\t\t * indicates that the connecting process has\n+\t\t\t * already dropped the connection.\n+\t\t\t */\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * Our connection to the client is blocking since a client\n+\t\t * can always be killed by SIGINT or similar.\n+\t\t */\n+\t\tif (strbuf_getwholeline_fd(&command, client_fd, '\\0') == 0) {\n+\t\t\tif (!strcmp(command.buf, \"refresh\")) {\n+\t\t\t\trefresh();\n+\t\t\t} else if (!strcmp(command.buf, \"poke\")) {\n+\t\t\t\t/*\n+\t\t\t\t * Just a poke to keep us\n+\t\t\t\t * alive, nothing to do.\n+\t\t\t\t */\n+\t\t\t} else {\n+\t\t\t\twarning(\"BUG: Bogus command %s\", command.buf);\n+\t\t\t}\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * No command from client. Probably it's just\n+\t\t\t * a liveness check. Just close up.\n+\t\t\t */\n+\t\t}\n+\t\tclose(client_fd);\n+\t}\n+\n+\tclose(fd);\n+}\n+\n+#else\n+\n+static void loop(int fd, int idle_in_seconds)\n+{\n+\tdie(_(\"index-helper is not supported on this platform\"));\n+}\n+\n+#endif\n+\n+static int setup_socket(const char *socket_path)\n+{\n+\tstruct sockaddr_un address = {0};\n+\tint fd;\n+\tint len;\n+\n+\tlen = strlen(socket_path);\n+\tif (len > UNIX_PATH_MAX - 1)\n+\t\tdie(\"path %s is too long for a socket\", socket_path);\n+\n+\tfd = socket(PF_UNIX, SOCK_STREAM, 0);\n+\tif (fd < 0)\n+\t\treturn -1;\n+\n+\taddress.sun_family = AF_UNIX;\n+\tstrncpy(address.sun_path, socket_path, UNIX_PATH_MAX);\n+\n+\tif (bind(fd, (struct sockaddr *) &address, sizeof(address)))\n+\t\tdie_errno(_(\"failed to bind to socket %s\"), socket_path);\n+\n+\tset_socket_nonblocking(fd);\n+\n+\tif (listen(fd, 3))\n+\t\tdie_errno(_(\"failed to listen on socket %s\"), socket_path);\n+\n+\treturn fd;\n+}\n+\n+static const char * const usage_text[] = {\n+\tN_(\"git index-helper [options]\"),\n+\tNULL\n+};\n+\n+static void make_socket_path(struct strbuf *path)\n+{\n+\tconst char *tmpdir;\n+\n+\ttmpdir = getenv(\"TMPDIR\");\n+\tif (!tmpdir)\n+\t\ttmpdir = \"/tmp\";\n+\n+\t/*\n+\t * We need to make a dir that we can own, so that users other\n+\t * than us cannot poke the daemon, and so that, if the daemon\n+\t * dies, no other process can recreate the socket and trick us\n+\t * into talking to an imposter.\n+\t */\n+\tstrbuf_addf(path, \"%s/XXXXXX\", tmpdir);\n+\tif (!mkdtemp(path->buf))\n+\t\tdie((\"failed to make temp dir for socket\"));\n+\n+\t/*\n+\t * Use a stupid filename because we want to minimize the path\n+\t * length since socket filenames must be short.\n+\t */\n+\tstrbuf_addstr(path, \"/s\");\n+}\n+\n+int main(int argc, char **argv)\n+{\n+\tconst char *prefix;\n+\tint idle_in_seconds = 600;\n+\tint fd;\n+\tstruct strbuf socket_path = STRBUF_INIT;\n+\tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"exit-after\", &idle_in_seconds,\n+\t\t\t N_(\"exit if not used after some seconds\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_extract_argv0_path(argv[0]);\n+\tgit_setup_gettext();\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(usage_text, options);\n+\n+\tprefix = setup_git_directory();\n+\tif (parse_options(argc, (const char **)argv, prefix,\n+\t\t\t options, usage_text, 0))\n+\t\tdie(_(\"too many arguments\"));\n+\n+\tatexit(cleanup);\n+\tsigchain_push_common(cleanup_on_signal);\n+\n+\tmake_socket_path(&socket_path);\n+\tfd = setup_socket(socket_path.buf);\n+\tif (fd < 0)\n+\t\tdie_errno(_(\"could not set up index-helper socket\"));\n+\t/*\n+\t * Now that the socket is set up, we symlink it into\n+\t * GIT_DIR so clients can find it.\n+\t */\n+\tif (unlink(git_path(\"index-helper.path\")) && errno != ENOENT)\n+\t\tdie(_(\"failed to delete old index-helper.path\"));\n+\tif (symlink(socket_path.buf, git_path(\"index-helper.path\")))\n+\t\tdie(_(\"failed to symlink socket path into index-helper.path\"));\n+\tloop(fd, idle_in_seconds);\n+\n+\treturn 0;\n+}\ndiff --git a/read-cache.c b/read-cache.c\nindex 7e387e9..66f2874 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -18,6 +18,7 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"utf8.h\"\n+#include \"shm.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -1541,6 +1542,91 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_untracked_cache(istate);\n }\n \n+static int poke_daemon(struct index_state *istate,\n+\t\t const struct stat *st, int refresh_cache)\n+{\n+\tint fd;\n+\tint ret = 0;\n+\tstruct sockaddr_un address = {0};\n+\n+\t/* if this is from index-helper, do not poke itself (recursively) */\n+\tif (istate->to_shm)\n+\t\treturn 0;\n+\n+\tif (readlink(git_path(\"index-helper.path\"), address.sun_path,\n+\t\t UNIX_PATH_MAX) < 0)\n+\t\treturn -1;\n+\n+\tfd = socket(PF_UNIX, SOCK_STREAM, 0);\n+\tif (fd < 0)\n+\t\treturn -1;\n+\n+\taddress.sun_family = AF_UNIX;\n+\tif (connect(fd, (struct sockaddr *) &address, sizeof(address))) {\n+\t\twarning(\"Failed to connect to socket %s\", address.sun_path);\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\n+\tif (refresh_cache) {\n+\t\tret = write_in_full(fd, \"refresh\", 8) != 8;\n+\t} else {\n+\t\tret = write_in_full(fd, \"poke\", 5) != 5;\n+\t}\n+\n+\tclose(fd);\n+\treturn ret;\n+}\n+\n+static int is_main_index(struct index_state *istate)\n+{\n+\treturn istate == &the_index ||\n+\t\t(the_index.split_index &&\n+\t\t istate == the_index.split_index->base);\n+}\n+\n+/*\n+ * Try to open and verify a cached shm index if available. Return 0 if\n+ * succeeds (istate->mmap and istate->mmap_size are updated). Return\n+ * negative otherwise.\n+ */\n+static int try_shm(struct index_state *istate)\n+{\n+\tvoid *new_mmap = NULL;\n+\tsize_t old_size = istate->mmap_size;\n+\tssize_t new_size;\n+\tconst unsigned char *sha1;\n+\tstruct stat st;\n+\n+\tif (!is_main_index(istate) ||\n+\t old_size <= 20 ||\n+\t stat(git_path(\"index-helper.path\"), &st))\n+\t\treturn -1;\n+\tif (poke_daemon(istate, &st, 0))\n+\t\treturn -1;\n+\tsha1 = (unsigned char *)istate->mmap + old_size - 20;\n+\tnew_size = git_shm_map(O_RDONLY, 0700, -1, &new_mmap,\n+\t\t\t\t PROT_READ, MAP_SHARED,\n+\t\t\t\t \"git-index-%s\", sha1_to_hex(sha1));\n+\tif (new_size <= 20 ||\n+\t hashcmp((unsigned char *)istate->mmap + old_size - 20,\n+\t\t (unsigned char *)new_mmap + new_size - 20)) {\n+\t\tif (new_mmap)\n+\t\t\tmunmap(new_mmap, new_size);\n+\t\tpoke_daemon(istate, &st, 1);\n+\t\treturn -1;\n+\t}\n+\n+\t/* The memory barrier here matches index-helper.c:share_index. */\n+\t__sync_synchronize();\n+\n+\tmunmap(istate->mmap, istate->mmap_size);\n+\tistate->mmap = new_mmap;\n+\tistate->mmap_size = new_size;\n+\tistate->from_shm = 1;\n+\treturn 0;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1555,6 +1641,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tif (istate->initialized)\n \t\treturn istate->cache_nr;\n \n+\tistate->from_shm = 0;\n \tistate->timestamp.sec = 0;\n \tistate->timestamp.nsec = 0;\n \tfd = open(path, O_RDONLY);\n@@ -1574,15 +1661,17 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tmmap = xmmap(NULL, mmap_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n-\tif (istate->keep_mmap) {\n-\t\tistate->mmap = mmap;\n-\t\tistate->mmap_size = mmap_size;\n-\t}\n \tclose(fd);\n \n-\thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n+\tistate->mmap = mmap;\n+\tistate->mmap_size = mmap_size;\n+\tif (try_shm(istate) &&\n+\t verify_hdr(istate->mmap, istate->mmap_size) < 0)\n \t\tgoto unmap;\n+\thdr = mmap = istate->mmap;\n+\tmmap_size = istate->mmap_size;\n+\tif (!istate->keep_mmap)\n+\t\tistate->mmap = NULL;\n \n \thashcpy(istate->sha1, (const unsigned char *)hdr + mmap_size - 20);\n \tistate->version = ntohl(hdr->hdr_version);\n@@ -1662,6 +1751,8 @@ int read_index_from(struct index_state *istate, const char *path)\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n \tsplit_index->base->keep_mmap = istate->keep_mmap;\n+\tsplit_index->base->to_shm = istate->to_shm;\n+\tsplit_index->base->from_shm = istate->from_shm;\n \tret = do_read_index(split_index->base,\n \t\t\t git_path(\"sharedindex.%s\",\n \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n@@ -1712,6 +1803,8 @@ int discard_index(struct index_state *istate)\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n+\tistate->from_shm = 0;\n+\tistate->to_shm = 0;\n \treturn 0;\n }\n \n@@ -2138,9 +2231,14 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \t\treturn ret;\n \tassert((flags & (COMMIT_LOCK | CLOSE_LOCK)) !=\n \t (COMMIT_LOCK | CLOSE_LOCK));\n-\tif (flags & COMMIT_LOCK)\n-\t\treturn commit_locked_index(lock);\n-\telse if (flags & CLOSE_LOCK)\n+\tif (flags & COMMIT_LOCK) {\n+\t\tstruct stat st;\n+\t\tret = commit_locked_index(lock);\n+\t\tif (!ret && is_main_index(istate) &&\n+\t\t !stat(git_path(\"index-helper.path\"), &st))\n+\t\t\tpoke_daemon(istate, &st, 1);\n+\t\treturn ret;\n+\t} else if (flags & CLOSE_LOCK)\n \t\treturn close_lock_file(lock);\n \telse\n \t\treturn ret;\ndiff --git a/shm.c b/shm.c\nnew file mode 100644\nindex 0000000..4ec1a00\n--- /dev/null\n+++ b/shm.c\n@@ -0,0 +1,67 @@\n+#include \"git-compat-util.h\"\n+#include \"shm.h\"\n+\n+#ifdef HAVE_SHM\n+\n+#define SHM_PATH_LEN 72\t\t/* we don't create very long paths.. */\n+\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t int prot, int flags, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tchar path[SHM_PATH_LEN];\n+\tint fd;\n+\n+\tpath[0] = '/';\n+\tva_start(ap, fmt);\n+\tvsprintf(path + 1, fmt, ap);\n+\tva_end(ap);\n+\tfd = shm_open(path, oflag, perm);\n+\tif (fd < 0)\n+\t\treturn -1;\n+\tif (length > 0 && ftruncate(fd, length)) {\n+\t\tshm_unlink(path);\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\tif (length < 0 && !(oflag & O_CREAT)) {\n+\t\tstruct stat st;\n+\t\tif (fstat(fd, &st))\n+\t\t\tdie_errno(\"unable to stat %s\", path);\n+\t\tlength = st.st_size;\n+\t}\n+\t*mmap = xmmap(NULL, length, prot, flags, fd, 0);\n+\tclose(fd);\n+\tif (*mmap == MAP_FAILED) {\n+\t\t*mmap = NULL;\n+\t\tshm_unlink(path);\n+\t\treturn -1;\n+\t}\n+\treturn length;\n+}\n+\n+void git_shm_unlink(const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tchar path[SHM_PATH_LEN];\n+\n+\tpath[0] = '/';\n+\tva_start(ap, fmt);\n+\tvsprintf(path + 1, fmt, ap);\n+\tva_end(ap);\n+\tshm_unlink(path);\n+}\n+\n+#else\n+\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t int prot, int flags, const char *fmt, ...)\n+{\n+\treturn -1;\n+}\n+\n+void git_shm_unlink(const char *fmt, ...)\n+{\n+}\n+\n+#endif\ndiff --git a/shm.h b/shm.h\nnew file mode 100644\nindex 0000000..798d3fd\n--- /dev/null\n+++ b/shm.h\n@@ -0,0 +1,23 @@\n+#ifndef SHM_H\n+#define SHM_H\n+\n+/*\n+ * Create or open a shared memory and mmap it. Return mmap size if\n+ * successful, -1 otherwise. If successful mmap contains the mmap'd\n+ * pointer. If oflag does not contain O_CREAT and length is negative,\n+ * the mmap size is retrieved from existing shared memory object.\n+ *\n+ * The mmap could be freed by munmap, even on Windows. Note that on\n+ * Windows, git_shm_unlink() is no-op, so the last unmap will destroy\n+ * the shared memory.\n+ */\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t int prot, int flags, const char *fmt, ...);\n+\n+/*\n+ * Unlink a shared memory object. Only needed on POSIX platforms. On\n+ * Windows this is no-op.\n+ */\n+void git_shm_unlink(const char *fmt, ...);\n+\n+#endif\ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nnew file mode 100755\nindex 0000000..6a06094\n--- /dev/null\n+++ b/t/t7900-index-helper.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2016, Twitter, Inc\n+#\n+\n+test_description='git-index-helper\n+\n+Testing git index-helper\n+'\n+\n+. ./test-lib.sh\n+\n+test -z \"$HAVE_SHM\" && {\n+\tskip_all='skipping index-helper tests: no SHM'\n+\ttest_done\n+}\n+\n+test_expect_success 'index-helper smoke test' '\n+\tgit index-helper --exit-after 1 &&\n+\ttest_path_is_missing .git/index-helper.path\n+'\n+\n+test_done\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282800","messageId":"1459980722-4836-5-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 04/16] index-helper: add --strict","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:50Z","receivedAt":"2016-04-06T22:11:50Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThere are \"holes\" in the index-helper approach because the shared\nmemory is not verified again by git. If $USER is compromised, shared\nmemory could be modified. But anyone who could do this could already\nmodify $GIT_DIR/index. A more realistic risk is some bugs in\nindex-helper that produce corrupt shared memory. --strict is added to\navoid that.\n\nStrictly speaking there's still a very small gap where corrupt shared\nmemory could still be read by git: after we write the trailing SHA-1 in\nthe shared memory (thus signaling \"this shm is ready\") and before\nverify_shm() detects an error.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-index-helper.txt | 9 +++++++\n cache.h | 1 +\n index-helper.c | 48 ++++++++++++++++++++++++++++++++++++++\n read-cache.c | 9 ++++---\n 4 files changed, 64 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex 61605e9..b177fb8 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -22,6 +22,15 @@ OPTIONS\n \tExit if the cached index is not accessed for ``\n \tseconds. Specify 0 to wait forever. Default is 600.\n \n+--strict::\n+--no-strict::\n+\tStrict mode makes index-helper verify the shared memory after\n+\tit's created. If the result does not match what's read from\n+\t$GIT_DIR/index, the shared memory is destroyed. This makes\n+\tindex-helper take more than double the amount of time required\n+\tfor reading an index, but because it will happen in the\n+\tbackground, it's not noticable. `--strict` is enabled by default.\n+\n NOTES\n -----\n $GIT_DIR/index-helper.path is a symlink to a Unix domain socket in\ndiff --git a/cache.h b/cache.h\nindex 43fb314..4b678e9 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -336,6 +336,7 @@ struct index_state {\n \t\t keep_mmap : 1,\n \t\t from_shm : 1,\n \t\t to_shm : 1,\n+\t\t always_verify_trailing_sha1 : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\ndiff --git a/index-helper.c b/index-helper.c\nindex 263b066..8288e30 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -15,6 +15,7 @@ struct shm {\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n+static int to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n {\n@@ -76,11 +77,56 @@ static void share_index(struct index_state *istate, struct shm *is)\n \thashcpy((unsigned char *)new_mmap + istate->mmap_size - 20, is->sha1);\n }\n \n+static int verify_shm(void)\n+{\n+\tint i;\n+\tstruct index_state istate;\n+\tmemset(&istate, 0, sizeof(istate));\n+\tistate.always_verify_trailing_sha1 = 1;\n+\tistate.to_shm = 1;\n+\ti = read_index(&istate);\n+\tif (i != the_index.cache_nr)\n+\t\tgoto done;\n+\tfor (; i < the_index.cache_nr; i++) {\n+\t\tstruct cache_entry *base, *ce;\n+\t\t/* namelen is checked separately */\n+\t\tconst unsigned int ondisk_flags =\n+\t\t\tCE_STAGEMASK | CE_VALID | CE_EXTENDED_FLAGS;\n+\t\tunsigned int ce_flags, base_flags, ret;\n+\t\tbase = the_index.cache[i];\n+\t\tce = istate.cache[i];\n+\t\tif (ce->ce_namelen != base->ce_namelen ||\n+\t\t strcmp(ce->name, base->name)) {\n+\t\t\twarning(\"mismatch at entry %d\", i);\n+\t\t\tbreak;\n+\t\t}\n+\t\tce_flags = ce->ce_flags;\n+\t\tbase_flags = base->ce_flags;\n+\t\t/* only on-disk flags matter */\n+\t\tce->ce_flags &= ondisk_flags;\n+\t\tbase->ce_flags &= ondisk_flags;\n+\t\tret = memcmp(&ce->ce_stat_data, &base->ce_stat_data,\n+\t\t\t offsetof(struct cache_entry, name) -\n+\t\t\t offsetof(struct cache_entry, ce_stat_data));\n+\t\tce->ce_flags = ce_flags;\n+\t\tbase->ce_flags = base_flags;\n+\t\tif (ret) {\n+\t\t\twarning(\"mismatch at entry %d\", i);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+done:\n+\tdiscard_index(&istate);\n+\treturn i == the_index.cache_nr;\n+}\n+\n static void share_the_index(void)\n {\n \tif (the_index.split_index && the_index.split_index->base)\n \t\tshare_index(the_index.split_index->base, &shm_base_index);\n \tshare_index(&the_index, &shm_index);\n+\tif (to_verify && !verify_shm())\n+\t\tcleanup_shm();\n \tdiscard_index(&the_index);\n }\n \n@@ -249,6 +295,8 @@ int main(int argc, char **argv)\n \tstruct option options[] = {\n \t\tOPT_INTEGER(0, \"exit-after\", &idle_in_seconds,\n \t\t\t N_(\"exit if not used after some seconds\")),\n+\t\tOPT_BOOL(0, \"strict\", &to_verify,\n+\t\t\t \"verify shared memory after creating\"),\n \t\tOPT_END()\n \t};\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 66f2874..7215a17 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1665,9 +1665,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \n \tistate->mmap = mmap;\n \tistate->mmap_size = mmap_size;\n-\tif (try_shm(istate) &&\n-\t verify_hdr(istate->mmap, istate->mmap_size) < 0)\n-\t\tgoto unmap;\n+\tif (try_shm(istate)) {\n+\t\tif (verify_hdr(istate->mmap, istate->mmap_size) < 0)\n+\t\t\tgoto unmap;\n+\t} else if (istate->always_verify_trailing_sha1 &&\n+\t\t verify_hdr(istate->mmap, istate->mmap_size) < 0)\n+\t\t\tgoto unmap;\n \thdr = mmap = istate->mmap;\n \tmmap_size = istate->mmap_size;\n \tif (!istate->keep_mmap)\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282810","messageId":"1459980722-4836-6-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 05/16] daemonize(): set a flag before exiting the main process","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:51Z","receivedAt":"2016-04-06T22:11:51Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThis allows signal handlers and atexit functions to realize this\nsituation and not clean up.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/gc.c | 2 +-\n cache.h | 2 +-\n daemon.c | 2 +-\n setup.c | 4 +++-\n 4 files changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c583aad..37180de 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -385,7 +385,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t * failure to daemonize is ok, we'll continue\n \t\t\t * in foreground\n \t\t\t */\n-\t\t\tdaemonized = !daemonize();\n+\t\t\tdaemonized = !daemonize(NULL);\n \t\t}\n \t} else\n \t\tadd_repack_all_option();\ndiff --git a/cache.h b/cache.h\nindex 4b678e9..0aeb994 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -530,7 +530,7 @@ extern int set_git_dir_init(const char *git_dir, const char *real_git_dir, int);\n extern int init_db(const char *template_dir, unsigned int flags);\n \n extern void sanitize_stdfds(void);\n-extern int daemonize(void);\n+extern int daemonize(int *);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \ndiff --git a/daemon.c b/daemon.c\nindex 8d45c33..a5cf954 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -1365,7 +1365,7 @@ int main(int argc, char **argv)\n \t\treturn execute();\n \n \tif (detach) {\n-\t\tif (daemonize())\n+\t\tif (daemonize(NULL))\n \t\t\tdie(\"--detach not supported on this platform\");\n \t} else\n \t\tsanitize_stdfds();\ndiff --git a/setup.c b/setup.c\nindex de1a2a7..9adf13f 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1017,7 +1017,7 @@ void sanitize_stdfds(void)\n \t\tclose(fd);\n }\n \n-int daemonize(void)\n+int daemonize(int *daemonized)\n {\n #ifdef NO_POSIX_GOODIES\n \terrno = ENOSYS;\n@@ -1029,6 +1029,8 @@ int daemonize(void)\n \t\tcase -1:\n \t\t\tdie_errno(\"fork failed\");\n \t\tdefault:\n+\t\t\tif (daemonized)\n+\t\t\t\t*daemonized = 1;\n \t\t\texit(0);\n \t}\n \tif (setsid() == -1)\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282809","messageId":"1459980722-4836-7-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 06/16] index-helper: add --detach","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:52Z","receivedAt":"2016-04-06T22:11:52Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nWe detach after creating and opening the socket, because otherwise\nwe might return control to the shell before index-helper is ready to\naccept commands. This might lead to flaky tests.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-index-helper.txt | 3 +++\n index-helper.c | 11 +++++++++--\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex b177fb8..bb344cf 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -31,6 +31,9 @@ OPTIONS\n \tfor reading an index, but because it will happen in the\n \tbackground, it's not noticable. `--strict` is enabled by default.\n \n+--detach::\n+\tDetach from the shell.\n+\n NOTES\n -----\n $GIT_DIR/index-helper.path is a symlink to a Unix domain socket in\ndiff --git a/index-helper.c b/index-helper.c\nindex 8288e30..10f29f5 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -15,7 +15,7 @@ struct shm {\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n-static int to_verify = 1;\n+static int daemonized, to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n {\n@@ -34,6 +34,8 @@ static void cleanup_shm(void)\n \n static void cleanup(void)\n {\n+\tif (daemonized)\n+\t\treturn;\n \tunlink(git_path(\"index-helper.path\"));\n \tcleanup_shm();\n }\n@@ -289,7 +291,7 @@ static void make_socket_path(struct strbuf *path)\n int main(int argc, char **argv)\n {\n \tconst char *prefix;\n-\tint idle_in_seconds = 600;\n+\tint idle_in_seconds = 600, detach = 0;\n \tint fd;\n \tstruct strbuf socket_path = STRBUF_INIT;\n \tstruct option options[] = {\n@@ -297,6 +299,7 @@ int main(int argc, char **argv)\n \t\t\t N_(\"exit if not used after some seconds\")),\n \t\tOPT_BOOL(0, \"strict\", &to_verify,\n \t\t\t \"verify shared memory after creating\"),\n+\t\tOPT_BOOL(0, \"detach\", &detach, \"detach the process\"),\n \t\tOPT_END()\n \t};\n \n@@ -326,6 +329,10 @@ int main(int argc, char **argv)\n \t\tdie(_(\"failed to delete old index-helper.path\"));\n \tif (symlink(socket_path.buf, git_path(\"index-helper.path\")))\n \t\tdie(_(\"failed to symlink socket path into index-helper.path\"));\n+\n+\tif (detach && daemonize(&daemonized))\n+\t\tdie_errno(_(\"unable to detach\"));\n+\n \tloop(fd, idle_in_seconds);\n \n \treturn 0;\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282802","messageId":"1459980722-4836-8-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 07/16] read-cache: add watchman 'WAMA' extension","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:53Z","receivedAt":"2016-04-06T22:11:53Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThe extension contains a bitmap, one bit for each entry in the\nindex. If the n-th bit is zero, the n-th entry is considered\nunchanged, we can ce_mark_uptodate() it without refreshing. If the bit\nis non-zero and we found out the corresponding file is clean after\nrefresh, we can clear the bit.\n\nIn addition, there's a list of directories in the untracked-cache\nto invalidate (because they have new or modified entries).\n\nThe 'skipping refresh' bit is not in this patch yet as we would need\nwatchman. More details in later patches.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 4 ++\n dir.h | 3 ++\n read-cache.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 3 files changed, 122 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 0aeb994..f4f7eef 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -182,6 +182,8 @@ struct cache_entry {\n #define CE_VALID (0x8000)\n #define CE_STAGESHIFT 12\n \n+#define CE_WATCHMAN_DIRTY (0x0001)\n+\n /*\n * Range 0xFFFF0FFF in ce_flags is divided into\n * two parts: in-memory flags and on-disk ones.\n@@ -320,6 +322,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n #define UNTRACKED_CHANGED\t(1 << 7)\n+#define WATCHMAN_CHANGED\t(1 << 8)\n \n struct split_index;\n struct untracked_cache;\n@@ -344,6 +347,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tvoid *mmap;\n \tsize_t mmap_size;\n+\tchar *last_update;\n };\n \n extern struct index_state the_index;\ndiff --git a/dir.h b/dir.h\nindex 3ec3fb0..3d540de 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -142,6 +142,9 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n+\t/* watchman invalidation data */\n+\tunsigned int use_watchman : 1;\n+\tstruct string_list invalid_untracked;\n };\n \n struct dir_struct {\ndiff --git a/read-cache.c b/read-cache.c\nindex 7215a17..59d892e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -19,6 +19,7 @@\n #include \"split-index.h\"\n #include \"utf8.h\"\n #include \"shm.h\"\n+#include \"ewah/ewok.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -41,11 +42,13 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n #define CACHE_EXT_UNTRACKED 0x554E5452\t /* \"UNTR\" */\n+#define CACHE_EXT_WATCHMAN 0x57414D41\t /* \"WAMA\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | \\\n+\t\t WATCHMAN_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -1220,8 +1223,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\tcontinue;\n \n \t\tnew = refresh_cache_ent(istate, ce, options, &cache_errno, &changed);\n-\t\tif (new == ce)\n+\t\tif (new == ce) {\n+\t\t\tif (ce->ce_flags & CE_WATCHMAN_DIRTY) {\n+\t\t\t\tce->ce_flags &= ~CE_WATCHMAN_DIRTY;\n+\t\t\t\tistate->cache_changed |= WATCHMAN_CHANGED;\n+\t\t\t}\n \t\t\tcontinue;\n+\t\t}\n \t\tif (!new) {\n \t\t\tconst char *fmt;\n \n@@ -1365,6 +1373,94 @@ static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n \treturn 0;\n }\n \n+static void mark_no_watchman(size_t pos, void *data)\n+{\n+\tstruct index_state *istate = data;\n+\tassert(pos < istate->cache_nr);\n+\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n+}\n+\n+static int read_watchman_ext(struct index_state *istate, const void *data,\n+\t\t\t unsigned long sz)\n+{\n+\tstruct ewah_bitmap *bitmap;\n+\tint ret, len;\n+\tuint32_t bitmap_size;\n+\tuint32_t untracked_nr;\n+\n+\tif (memchr(data, 0, sz) == NULL)\n+\t\treturn error(\"invalid extension\");\n+\n+\tlen = strlen(data) + 1;\n+\tmemcpy(&bitmap_size, (const char *)data + len, 4);\n+\tmemcpy(&untracked_nr, (const char *)data + len + 4, 4);\n+\tuntracked_nr = ntohl(untracked_nr);\n+\tbitmap_size = ntohl(bitmap_size);\n+\n+\tbitmap = ewah_new();\n+\tret = ewah_read_mmap(bitmap, (const char *)data + len + 8, bitmap_size);\n+\tif (ret != bitmap_size) {\n+\t\tewah_free(bitmap);\n+\t\treturn error(\"failed to parse ewah bitmap reading watchman index extension\");\n+\t}\n+\tistate->last_update = xstrdup(data);\n+\tewah_each_bit(bitmap, mark_no_watchman, istate);\n+\tewah_free(bitmap);\n+\n+\t/*\n+\t * TODO: update the untracked cache from the untracked data in this\n+\t * extension.\n+\t */\n+\treturn 0;\n+}\n+\n+static int untracked_entry_append(struct string_list_item *item, void *sbvoid)\n+{\n+\tstruct strbuf *sb = sbvoid;\n+\n+\tstrbuf_addstr(sb, item->string);\n+\tstrbuf_addch(sb, 0);\n+\treturn 0;\n+}\n+\n+void write_watchman_ext(struct strbuf *sb, struct index_state* istate)\n+{\n+\tstruct ewah_bitmap *bitmap;\n+\tint i;\n+\tint ewah_start;\n+\tint ewah_size = 0;\n+\tint fixup = 0;\n+\n+\tstrbuf_add(sb, istate->last_update, strlen(istate->last_update) + 1);\n+\tfixup = sb->len;\n+\tstrbuf_add(sb, &ewah_size, 4); /* we'll fix this up later */\n+\tif (istate->untracked) {\n+\t\tuint32_t nr = istate->untracked->invalid_untracked.nr;\n+\t\tnr = htonl(nr);\n+\t\tstrbuf_add(sb, &nr, 4);\n+\t} else {\n+\t\t/* zero */\n+\t\tstrbuf_add(sb, &ewah_size, 4);\n+\t}\n+\n+\tewah_start = sb->len;\n+\tbitmap = ewah_new();\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHMAN_DIRTY)\n+\t\t\tewah_set(bitmap, i);\n+\tewah_serialize_strbuf(bitmap, sb);\n+\tewah_free(bitmap);\n+\n+\t/* fix up size field */\n+\tewah_size = sb->len - ewah_start;\n+\tewah_size = htonl(ewah_size);\n+\tmemcpy(sb->buf + fixup, &ewah_size, 4);\n+\n+\tif (istate->untracked)\n+\t\tfor_each_string_list(&istate->untracked->invalid_untracked,\n+\t\t\t\t untracked_entry_append, sb);\n+}\n+\n static int read_index_extension(struct index_state *istate,\n \t\t\t\tconst char *ext, void *data, unsigned long sz)\n {\n@@ -1382,6 +1478,11 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_UNTRACKED:\n \t\tistate->untracked = read_untracked_extension(data, sz);\n \t\tbreak;\n+\n+\tcase CACHE_EXT_WATCHMAN:\n+\t\tread_watchman_ext(istate, data, sz);\n+\t\tbreak;\n+\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1808,6 +1909,8 @@ int discard_index(struct index_state *istate)\n \tistate->untracked = NULL;\n \tistate->from_shm = 0;\n \tistate->to_shm = 0;\n+\tfree(istate->last_update);\n+\tistate->last_update = NULL;\n \treturn 0;\n }\n \n@@ -2205,6 +2308,16 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->last_update) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_watchman_ext(&sb, istate);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCHMAN, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st))\n \t\treturn -1;\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282797","messageId":"1459980722-4836-9-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 08/16] Add watchman support to reduce index refresh cost","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:54Z","receivedAt":"2016-04-06T22:11:54Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThe previous patch has the logic to clear bits in 'WAMA' bitmap. This\npatch has logic to set bits as told by watchman. The missing bit,\n_using_ these bits, are not here yet.\n\nA lot of this code is written by David Turner originally, mostly from\n[1]. I'm just copying and polishing it a bit.\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/248006\n\nSigned-off-by: David Turner \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Makefile | 12 +++++\n cache.h | 1 +\n config.c | 5 ++\n configure.ac | 8 ++++\n environment.c | 3 ++\n watchman-support.c | 135 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n watchman-support.h | 7 +++\n 7 files changed, 171 insertions(+)\n create mode 100644 watchman-support.c\n create mode 100644 watchman-support.h\n\ndiff --git a/Makefile b/Makefile\nindex 24d6c0e..8ad8e1f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -453,6 +453,7 @@ MSGFMT = msgfmt\n CURL_CONFIG = curl-config\n PTHREAD_LIBS = -lpthread\n PTHREAD_CFLAGS =\n+WATCHMAN_LIBS =\n GCOV = gcov\n \n export TCL_PATH TCLTK_PATH\n@@ -1419,6 +1420,13 @@ else\n \tLIB_OBJS += thread-utils.o\n endif\n \n+ifdef USE_WATCHMAN\n+\tLIB_H += watchman-support.h\n+\tLIB_OBJS += watchman-support.o\n+\tWATCHMAN_LIBS = -lwatchman\n+\tBASIC_CFLAGS += -DUSE_WATCHMAN\n+endif\n+\n ifdef HAVE_PATHS_H\n \tBASIC_CFLAGS += -DHAVE_PATHS_H\n endif\n@@ -2030,6 +2038,9 @@ git-remote-testsvn$X: remote-testsvn.o GIT-LDFLAGS $(GITLIBS) $(VCSSVN_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) \\\n \t$(VCSSVN_LIB)\n \n+git-index-helper$X: index-helper.o GIT-LDFLAGS $(GITLIBS)\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) $(WATCHMAN_LIBS)\n+\n $(REMOTE_CURL_ALIASES): $(REMOTE_CURL_PRIMARY)\n \t$(QUIET_LNCP)$(RM) $@ && \\\n \tln $< $@ 2>/dev/null || \\\n@@ -2169,6 +2180,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo HAVE_SHM=\\''$(subst ','\\'',$(subst ','\\'',$(HAVE_SHM)))'\\' >>$@+\n+\t@echo USE_WATCHMAN=\\''$(subst ','\\'',$(subst ','\\'',$(USE_WATCHMAN)))'\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/cache.h b/cache.h\nindex f4f7eef..37f211b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -687,6 +687,7 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n+extern int core_watchman_sync_timeout;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex 9ba40bc..e6dc141 100644\n--- a/config.c\n+++ b/config.c\n@@ -882,6 +882,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.watchmansynctimeout\")) {\n+\t\tcore_watchman_sync_timeout = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.createobject\")) {\n \t\tif (!strcmp(value, \"rename\"))\n \t\t\tobject_creation_mode = OBJECT_CREATION_USES_RENAMES;\ndiff --git a/configure.ac b/configure.ac\nindex 0cd9f46..334d63b 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -1099,6 +1099,14 @@ AC_COMPILE_IFELSE([BSD_SYSCTL_SRC],\n \tHAVE_BSD_SYSCTL=])\n GIT_CONF_SUBST([HAVE_BSD_SYSCTL])\n \n+#\n+# Check for watchman client library\n+\n+AC_CHECK_LIB([watchman], [watchman_connect],\n+\t[USE_WATCHMAN=YesPlease],\n+\t[USE_WATCHMAN=])\n+GIT_CONF_SUBST([USE_WATCHMAN])\n+\n ## Other checks.\n # Define USE_PIC if you need the main git objects to be built with -fPIC\n # in order to build and link perl/Git.so. x86-64 seems to need this.\ndiff --git a/environment.c b/environment.c\nindex 6dec9d0..35e03c7 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -94,6 +94,9 @@ int core_preload_index = 1;\n */\n int ignore_untracked_cache_config;\n \n+int core_watchman_sync_timeout = 300;\n+\n+\n /* This is set by setup_git_dir_gently() and/or git_default_config() */\n char *git_work_tree_cfg;\n static char *work_tree;\ndiff --git a/watchman-support.c b/watchman-support.c\nnew file mode 100644\nindex 0000000..b168e88\n--- /dev/null\n+++ b/watchman-support.c\n@@ -0,0 +1,135 @@\n+#include \"cache.h\"\n+#include \"watchman-support.h\"\n+#include \"strbuf.h\"\n+#include \"dir.h\"\n+#include \n+\n+static struct watchman_query *make_query(const char *last_update)\n+{\n+\tstruct watchman_query *query = watchman_query();\n+\twatchman_query_set_fields(query, WATCHMAN_FIELD_NAME |\n+\t\t\t\t\t WATCHMAN_FIELD_EXISTS |\n+\t\t\t\t\t WATCHMAN_FIELD_NEWER);\n+\twatchman_query_set_empty_on_fresh(query, 1);\n+\tquery->sync_timeout = core_watchman_sync_timeout;\n+\tif (*last_update)\n+\t\twatchman_query_set_since_oclock(query, last_update);\n+\treturn query;\n+}\n+\n+static struct watchman_query_result* query_watchman(\n+\tstruct index_state *istate, struct watchman_connection *connection,\n+\tconst char *fs_path, const char *last_update)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_query *query;\n+\tstruct watchman_expression *expr;\n+\tstruct watchman_query_result *result;\n+\n+\tquery = make_query(last_update);\n+\texpr = watchman_true_expression();\n+\tresult = watchman_do_query(connection, fs_path, query, expr, &wm_error);\n+\twatchman_free_query(query);\n+\twatchman_free_expression(expr);\n+\n+\tif (!result)\n+\t\twarning(\"Watchman query error: %s (at %s)\",\n+\t\t\twm_error.message,\n+\t\t\t*last_update ? last_update : \"the beginning\");\n+\n+\treturn result;\n+}\n+\n+static void update_index(struct index_state *istate,\n+\t\t\t struct watchman_query_result *result)\n+{\n+\tint i;\n+\n+\tif (result->is_fresh_instance) {\n+\t\t/* let refresh clear them later */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags |= CE_WATCHMAN_DIRTY;\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < result->nr; i++) {\n+\t\tstruct watchman_stat *wm = result->stats + i;\n+\t\tint pos;\n+\n+\t\tif (S_ISDIR(wm->mode) ||\n+\t\t !strncmp(wm->name, \".git/\", 5) ||\n+\t\t strstr(wm->name, \"/.git/\"))\n+\t\t\tcontinue;\n+\n+\t\tpos = index_name_pos(istate, wm->name, strlen(wm->name));\n+\t\tif (pos < 0) {\n+\t\t\tif (istate->untracked) {\n+\t\t\t\tchar *name = xstrdup(wm->name);\n+\t\t\t\tchar *dname = dirname(name);\n+\n+\t\t\t\t/*\n+\t\t\t\t * dirname() returns '.' for the root,\n+\t\t\t\t * but we call it ''.\n+\t\t\t\t */\n+\t\t\t\tif (dname[0] == '.' && dname[1] == 0)\n+\t\t\t\t\tstring_list_append(&istate->untracked->invalid_untracked, \"\");\n+\t\t\t\telse\n+\t\t\t\t\tstring_list_append(&istate->untracked->invalid_untracked,\n+\t\t\t\t\t\t\t dname);\n+\t\t\t\tfree(name);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* FIXME: ignore staged entries and gitlinks too? */\n+\n+\t\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n+\t}\n+\n+done:\n+\tfree(istate->last_update);\n+\tistate->last_update = xstrdup(result->clock);\n+\tistate->cache_changed |= WATCHMAN_CHANGED;\n+\tif (istate->untracked)\n+\t\tstring_list_remove_duplicates(&istate->untracked->invalid_untracked, 0);\n+}\n+\n+int check_watchman(struct index_state *istate)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_connection *connection;\n+\tstruct watchman_query_result *result;\n+\tconst char *fs_path;\n+\tstruct timeval timeout;\n+\t/*\n+\t * Convert core_watchman_sync_timeout, in milliseconds, to\n+\t * struct timeval, in seconds and microseconds.\n+\t */\n+\n+\tfs_path = get_git_work_tree();\n+\tif (!fs_path)\n+\t\treturn -1;\n+\n+\ttimeout.tv_sec = core_watchman_sync_timeout / 1000;\n+\ttimeout.tv_usec = (core_watchman_sync_timeout % 1000) * 1000;\n+\tconnection = watchman_connect(timeout, &wm_error);\n+\n+\tif (!connection) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\treturn -1;\n+\t}\n+\n+\tif (watchman_watch(connection, fs_path, &wm_error)) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\twatchman_connection_close(connection);\n+\t\treturn -1;\n+\t}\n+\n+\n+\tresult = query_watchman(istate, connection, fs_path, istate->last_update);\n+\twatchman_connection_close(connection);\n+\tif (!result)\n+\t\treturn -1;\n+\tupdate_index(istate, result);\n+\twatchman_free_query_result(result);\n+\treturn 0;\n+}\ndiff --git a/watchman-support.h b/watchman-support.h\nnew file mode 100644\nindex 0000000..ee1ef2c\n--- /dev/null\n+++ b/watchman-support.h\n@@ -0,0 +1,7 @@\n+#ifndef WATCHMAN_SUPPORT_H\n+#define WATCHMAN_SUPPORT_H\n+\n+struct index_state;\n+int check_watchman(struct index_state *index);\n+\n+#endif /* WATCHMAN_SUPPORT_H */\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282807","messageId":"1459980722-4836-10-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 09/16] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:55Z","receivedAt":"2016-04-06T22:11:55Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nWatchman is hidden behind index-helper. Before git tries to read the\nindex from shm, it notifies index-helper through the socket and\nwaits for index-helper to prepare shm. index-helper then contacts\nwatchman, updates 'WAMA' extension and put it in a separate shm and\nwakes git up with a reply to git's socket.\n\nGit uses this extension to not lstat unchanged entries. Git only\ntrusts the 'WAMA' extension when it's received from the separate shm,\nnot from disk. Unmarked entries are \"clean\". Marked entries are dirty\nfrom watchman point of view. If it finds out some entries are\n'watchman-dirty', but are really unchanged (e.g. the file was changed,\nthen reverted back), then Git will clear the marking in 'WAMA' before\nwriting it down.\n\nHiding watchman behind index-helper means you need both daemons. You\ncan't run watchman alone. Not so good. But on the other hand, 'git'\nbinary is not linked to watchman/json libraries, which is good for\npackaging. Core git package will run fine without watchman-related\npackages. If they need watchman, they can install git-index-helper and\ndependencies.\n\nThis also lets us trust anything in the untracked cache that we haven't\nmarked invalid, saving those stat() calls.\n\nAnother reason for tying watchman to index-helper is, when used with\nuntracked cache, we need to keep track of $GIT_WORK_TREE file\nlisting. That kind of list can be kept in index-helper.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \n---\n Documentation/git-index-helper.txt | 6 +\n cache.h | 8 ++\n dir.c | 23 +++-\n dir.h | 3 +\n index-helper.c | 102 ++++++++++++++---\n read-cache.c | 218 +++++++++++++++++++++++++++++++++----\n 6 files changed, 319 insertions(+), 41 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex bb344cf..9d1ad0e 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -51,6 +51,12 @@ following commands are used to control the daemon:\n \tLet the daemon know the index is to be read. It keeps the\n \tdaemon alive longer, unless `--exit-after=0` is used.\n \n+\"poke \":\n+\tLike \"poke\", but replies with \"OK\". If watchman is\n+\tconfigured, index-helper queries watchman, then prepares a\n+\tshared memory object with the watchman index extension before\n+\treplying.\n+\n All commands and replies are terminated by a 0 byte.\n \n GIT\ndiff --git a/cache.h b/cache.h\nindex 37f211b..e43a6e1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -558,6 +558,7 @@ extern int daemonize(int *);\n \n /* Initialize and use the cache information */\n struct lock_file;\n+extern int verify_index(const struct index_state *);\n extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);\n extern int do_read_index(struct index_state *istate, const char *path,\n@@ -565,6 +566,7 @@ extern int do_read_index(struct index_state *istate, const char *path,\n extern int read_index_from(struct index_state *, const char *path);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n+extern void write_watchman_ext(struct strbuf *sb, struct index_state* istate);\n #define COMMIT_LOCK\t\t(1 << 0)\n #define CLOSE_LOCK\t\t(1 << 1)\n extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\n@@ -1839,4 +1841,10 @@ void sleep_millisec(int millisec);\n */\n void safe_create_dir(const char *dir, int share);\n \n+/*\n+ * Open a socket to the git index-helper. Return the fd of that\n+ * socket, or -1 on error.\n+ */\n+int connect_to_index_helper(void);\n+\n #endif /* CACHE_H */\ndiff --git a/dir.c b/dir.c\nindex 69e0be6..5058b29 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -597,9 +597,9 @@ static void trim_trailing_spaces(char *buf)\n *\n * If \"name\" has the trailing slash, it'll be excluded in the search.\n */\n-static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n-\t\t\t\t\t\t struct untracked_cache_dir *dir,\n-\t\t\t\t\t\t const char *name, int len)\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+\t\t\t\t\t struct untracked_cache_dir *dir,\n+\t\t\t\t\t const char *name, int len)\n {\n \tint first, last;\n \tstruct untracked_cache_dir *d;\n@@ -1726,6 +1726,17 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n+\tif (dir->untracked->use_watchman) {\n+\t\t/*\n+\t\t * With watchman, we can trust the untracked cache's\n+\t\t * valid field.\n+\t\t */\n+\t\tif (untracked->valid)\n+\t\t\tgoto skip_stat;\n+\t\telse\n+\t\t\tinvalidate_directory(dir->untracked, untracked);\n+\t}\n+\n \tif (stat(path->len ? path->buf : \".\", &st)) {\n \t\tinvalidate_directory(dir->untracked, untracked);\n \t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n@@ -1739,6 +1750,7 @@ static int valid_cached_dir(struct dir_struct *dir,\n \t\treturn 0;\n \t}\n \n+skip_stat:\n \tif (untracked->check_only != !!check_only) {\n \t\tinvalidate_directory(dir->untracked, untracked);\n \t\treturn 0;\n@@ -2625,8 +2637,10 @@ static void free_untracked(struct untracked_cache_dir *ucd)\n \n void free_untracked_cache(struct untracked_cache *uc)\n {\n-\tif (uc)\n+\tif (uc) {\n \t\tfree_untracked(uc->root);\n+\t\tstring_list_clear(&uc->invalid_untracked, 0);\n+\t}\n \tfree(uc);\n }\n \n@@ -2775,6 +2789,7 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n \t\treturn NULL;\n \n \tuc = xcalloc(1, sizeof(*uc));\n+\tstring_list_init(&uc->invalid_untracked, 1);\n \tstrbuf_init(&uc->ident, ident_len);\n \tstrbuf_add(&uc->ident, ident, ident_len);\n \tload_sha1_stat(&uc->ss_info_exclude, &ouc->info_exclude_stat,\ndiff --git a/dir.h b/dir.h\nindex 3d540de..8fd3f9e 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -315,4 +315,7 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked);\n void add_untracked_cache(struct index_state *istate);\n void remove_untracked_cache(struct index_state *istate);\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+\t\t\t\t\t struct untracked_cache_dir *dir,\n+\t\t\t\t\t const char *name, int len);\n #endif\ndiff --git a/index-helper.c b/index-helper.c\nindex 10f29f5..f993ae6 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -6,15 +6,18 @@\n #include \"split-index.h\"\n #include \"shm.h\"\n #include \"lockfile.h\"\n+#include \"watchman-support.h\"\n \n struct shm {\n \tunsigned char sha1[20];\n \tvoid *shm;\n \tsize_t size;\n+\tpid_t pid;\n };\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n+static struct shm shm_watchman;\n static int daemonized, to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n@@ -26,10 +29,21 @@ static void release_index_shm(struct shm *is)\n \tis->shm = NULL;\n }\n \n+static void release_watchman_shm(struct shm *is)\n+{\n+\tif (!is->shm)\n+\t\treturn;\n+\tmunmap(is->shm, is->size);\n+\tgit_shm_unlink(\"git-watchman-%s-%\" PRIuMAX,\n+\t\t sha1_to_hex(is->sha1), (uintmax_t)is->pid);\n+\tis->shm = NULL;\n+}\n+\n static void cleanup_shm(void)\n {\n \trelease_index_shm(&shm_index);\n \trelease_index_shm(&shm_base_index);\n+\trelease_watchman_shm(&shm_watchman);\n }\n \n static void cleanup(void)\n@@ -127,18 +141,10 @@ static void share_the_index(void)\n \tif (the_index.split_index && the_index.split_index->base)\n \t\tshare_index(the_index.split_index->base, &shm_base_index);\n \tshare_index(&the_index, &shm_index);\n-\tif (to_verify && !verify_shm())\n+\tif (to_verify && !verify_shm()) {\n \t\tcleanup_shm();\n-\tdiscard_index(&the_index);\n-}\n-\n-static void refresh(void)\n-{\n-\tthe_index.keep_mmap = 1;\n-\tthe_index.to_shm = 1;\n-\tif (read_cache() < 0)\n-\t\tdie(_(\"could not read index\"));\n-\tshare_the_index();\n+\t\tdiscard_index(&the_index);\n+\t}\n }\n \n static void set_socket_nonblocking(int fd)\n@@ -156,6 +162,61 @@ static void set_socket_nonblocking(int fd)\n \n #ifdef HAVE_SHM\n \n+#ifdef USE_WATCHMAN\n+static void share_watchman(struct index_state *istate,\n+\t\t\t struct shm *is, pid_t pid)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tvoid *shm;\n+\n+\twrite_watchman_ext(&sb, istate);\n+\tif (git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, sb.len + 20,\n+\t\t\t&shm, PROT_READ | PROT_WRITE, MAP_SHARED,\n+\t\t\t\"git-watchman-%s-%\" PRIuMAX,\n+\t\t\tsha1_to_hex(istate->sha1), (uintmax_t)pid) == sb.len + 20) {\n+\t\tis->size = sb.len + 20;\n+\t\tis->shm = shm;\n+\t\tis->pid = pid;\n+\t\thashcpy(is->sha1, istate->sha1);\n+\n+\t\tmemcpy(shm, sb.buf, sb.len);\n+\t\thashcpy((unsigned char *)shm + is->size - 20, is->sha1);\n+\t}\n+\tstrbuf_release(&sb);\n+}\n+\n+\n+static void prepare_with_watchman(pid_t pid)\n+{\n+\t/*\n+\t * TODO: with the help of watchman, maybe we could detect if\n+\t * $GIT_DIR/index is updated.\n+\t */\n+\tif (check_watchman(&the_index))\n+\t\treturn;\n+\n+\tshare_watchman(&the_index, &shm_watchman, pid);\n+}\n+\n+static void prepare_index(pid_t pid)\n+{\n+\trelease_watchman_shm(&shm_watchman);\n+\tif (the_index.last_update)\n+\t\tprepare_with_watchman(pid);\n+}\n+\n+#endif\n+\n+static void refresh(void)\n+{\n+\tdiscard_index(&the_index);\n+\tthe_index.keep_mmap = 1;\n+\tthe_index.to_shm = 1;\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"could not read index\"));\n+\tshare_the_index();\n+}\n+\n static void loop(int fd, int idle_in_seconds)\n {\n \tstruct timeval timeout;\n@@ -201,11 +262,20 @@ static void loop(int fd, int idle_in_seconds)\n \t\tif (strbuf_getwholeline_fd(&command, client_fd, '\\0') == 0) {\n \t\t\tif (!strcmp(command.buf, \"refresh\")) {\n \t\t\t\trefresh();\n-\t\t\t} else if (!strcmp(command.buf, \"poke\")) {\n-\t\t\t\t/*\n-\t\t\t\t * Just a poke to keep us\n-\t\t\t\t * alive, nothing to do.\n-\t\t\t\t */\n+\t\t\t} else if (starts_with(command.buf, \"poke\")) {\n+\t\t\t\tif (command.buf[4] == ' ') {\n+#ifdef USE_WATCHMAN\n+\t\t\t\t\tint client_pid = strtoull(command.buf + 5, NULL, 10);\n+\t\t\t\t\tprepare_index(client_pid);\n+#endif\n+\t\t\t\t\tif (write_in_full(client_fd, \"OK\", 3) != 3)\n+\t\t\t\t\t\twarning(_(\"client write failed\"));\n+\t\t\t\t} else {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * Just a poke to keep us\n+\t\t\t\t\t * alive, nothing to do.\n+\t\t\t\t\t */\n+\t\t\t\t}\n \t\t\t} else {\n \t\t\t\twarning(\"BUG: Bogus command %s\", command.buf);\n \t\t\t}\ndiff --git a/read-cache.c b/read-cache.c\nindex 59d892e..b6e9244 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1233,7 +1233,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (!new) {\n \t\t\tconst char *fmt;\n \n-\t\t\tif (really && cache_errno == EINVAL) {\n+\t\t\tif (really || cache_errno == EINVAL) {\n \t\t\t\t/* If we are doing --really-refresh that\n \t\t\t\t * means the index is not valid anymore.\n \t\t\t\t */\n@@ -1373,11 +1373,76 @@ static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n \treturn 0;\n }\n \n+static struct untracked_cache_dir *find_untracked_cache_dir(\n+\tstruct untracked_cache *uc, struct untracked_cache_dir *ucd,\n+\tconst char *name)\n+{\n+\tint component_len;\n+\tconst char *end;\n+\tstruct untracked_cache_dir *dir;\n+\n+\tif (!*name)\n+\t\treturn ucd;\n+\n+\tend = strchr(name, '/');\n+\tif (end)\n+\t\tcomponent_len = end - name;\n+\telse\n+\t\tcomponent_len = strlen(name);\n+\n+\tdir = lookup_untracked(uc, ucd, name, component_len);\n+\tif (dir)\n+\t\treturn find_untracked_cache_dir(uc, dir, name + component_len + 1);\n+\n+\treturn NULL;\n+}\n+\n static void mark_no_watchman(size_t pos, void *data)\n {\n \tstruct index_state *istate = data;\n+\tstruct cache_entry *ce = istate->cache[pos];\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tchar *c;\n+\tstruct untracked_cache_dir *dir;\n+\n \tassert(pos < istate->cache_nr);\n-\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n+\tce->ce_flags |= CE_WATCHMAN_DIRTY;\n+\n+\tif (!istate->untracked || !istate->untracked->root)\n+\t\treturn;\n+\n+\tstrbuf_add(&sb, ce->name, ce_namelen(ce));\n+\n+\tfor (c = sb.buf + sb.len - 1; c > sb.buf; c--) {\n+\t\tif (*c == '/') {\n+\t\t\tstrbuf_setlen(&sb, c - sb.buf);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (c == sb.buf) {\n+\t\tstrbuf_setlen(&sb, 0);\n+\t}\n+\n+\tdir = find_untracked_cache_dir(istate->untracked,\n+\t\t\t\t istate->untracked->root, sb.buf);\n+\tif (dir)\n+\t\tdir->valid = 0;\n+\n+\tstrbuf_release(&sb);\n+}\n+\n+static int mark_untracked_invalid(struct string_list_item *item, void *uc)\n+{\n+\tstruct untracked_cache *untracked = uc;\n+\tstruct untracked_cache_dir *dir;\n+\n+\tdir = find_untracked_cache_dir(untracked, untracked->root,\n+\t\t\t\t item->string);\n+\tif (dir)\n+\t\tdir->valid = 0;\n+\n+\treturn 0;\n }\n \n static int read_watchman_ext(struct index_state *istate, const void *data,\n@@ -1407,10 +1472,24 @@ static int read_watchman_ext(struct index_state *istate, const void *data,\n \tewah_each_bit(bitmap, mark_no_watchman, istate);\n \tewah_free(bitmap);\n \n-\t/*\n-\t * TODO: update the untracked cache from the untracked data in this\n-\t * extension.\n-\t */\n+\tif (istate->untracked && istate->untracked->root) {\n+\t\tint i;\n+\t\tconst char *untracked;\n+\n+\t\tuntracked = data + len + 8 + bitmap_size;\n+\t\tfor (i = 0; i < untracked_nr; ++i) {\n+\t\t\tint len = strlen(untracked);\n+\t\t\tstring_list_append(&istate->untracked->invalid_untracked,\n+\t\t\t\t\t untracked);\n+\t\t\tuntracked += len + 1;\n+\t\t}\n+\n+\t\tfor_each_string_list(&istate->untracked->invalid_untracked,\n+\t\t\t mark_untracked_invalid, istate->untracked);\n+\n+\t\tif (untracked_nr)\n+\t\t\tistate->cache_changed |= WATCHMAN_CHANGED;\n+\t}\n \treturn 0;\n }\n \n@@ -1643,16 +1722,10 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_untracked_cache(istate);\n }\n \n-static int poke_daemon(struct index_state *istate,\n-\t\t const struct stat *st, int refresh_cache)\n+int connect_to_index_helper(void)\n {\n-\tint fd;\n-\tint ret = 0;\n \tstruct sockaddr_un address = {0};\n-\n-\t/* if this is from index-helper, do not poke itself (recursively) */\n-\tif (istate->to_shm)\n-\t\treturn 0;\n+\tint fd;\n \n \tif (readlink(git_path(\"index-helper.path\"), address.sun_path,\n \t\t UNIX_PATH_MAX) < 0)\n@@ -1663,16 +1736,71 @@ static int poke_daemon(struct index_state *istate,\n \t\treturn -1;\n \n \taddress.sun_family = AF_UNIX;\n-\tif (connect(fd, (struct sockaddr *) &address, sizeof(address))) {\n-\t\twarning(\"Failed to connect to socket %s\", address.sun_path);\n-\t\tclose(fd);\n+\tif (!connect(fd, (struct sockaddr *) &address, sizeof(address)))\n+\t\treturn fd;\n+\n+\n+\twarning(\"Failed to connect to socket %s\", address.sun_path);\n+\tclose(fd);\n+\treturn -1;\n+}\n+\n+static int poke_and_wait_for_reply(int fd)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct strbuf reply = STRBUF_INIT;\n+\tint ret = -1;\n+\tfd_set fds;\n+\tstruct timeval timeout;\n+\n+\ttimeout.tv_usec = 0;\n+\ttimeout.tv_sec = 1;\n+\n+\tif (fd < 0)\n \t\treturn -1;\n-\t}\n+\n+\tstrbuf_addf(&buf, \"poke %d\", getpid());\n+\tif (write_in_full(fd, buf.buf, buf.len + 1) != buf.len + 1)\n+\t\tgoto done_poke;\n+\n+\t/* Now wait for a reply */\n+\tFD_ZERO(&fds);\n+\tFD_SET(fd, &fds);\n+\tif (select(fd + 1, &fds, NULL, NULL, &timeout) == 0)\n+\t\t/* No reply, giving up */\n+\t\tgoto done_poke;\n+\n+\tif (strbuf_getwholeline_fd(&reply, fd, 0))\n+\t\tgoto done_poke;\n+\n+\tif (!starts_with(reply.buf, \"OK\"))\n+\t\tgoto done_poke;\n+\n+\tret = 0;\n+done_poke:\n+\tclose(fd);\n+\tstrbuf_release(&buf);\n+\tstrbuf_release(&reply);\n+\n+\treturn ret;\n+}\n+\n+static int poke_daemon(struct index_state *istate,\n+\t\t const struct stat *st, int refresh_cache)\n+{\n+\tint fd;\n+\tint ret = -1;\n+\n+\t/* if this is from index-helper, do not poke itself (recursively) */\n+\tif (istate->to_shm)\n+\t\treturn 0;\n+\n+\tfd = connect_to_index_helper();\n \n \tif (refresh_cache) {\n \t\tret = write_in_full(fd, \"refresh\", 8) != 8;\n \t} else {\n-\t\tret = write_in_full(fd, \"poke\", 5) != 5;\n+\t\tret = poke_and_wait_for_reply(fd);\n \t}\n \n \tclose(fd);\n@@ -1728,6 +1856,50 @@ static int try_shm(struct index_state *istate)\n \treturn 0;\n }\n \n+static void refresh_by_watchman(struct index_state *istate)\n+{\n+\tvoid *shm = NULL;\n+\tint length;\n+\tint i;\n+\n+\tlength = git_shm_map(O_RDONLY, 0700, -1, &shm,\n+\t\t\t PROT_READ, MAP_SHARED,\n+\t\t\t \"git-watchman-%s-%\" PRIuMAX,\n+\t\t\t sha1_to_hex(istate->sha1),\n+\t\t\t (uintmax_t)getpid());\n+\n+\tif (length <= 20 ||\n+\t hashcmp(istate->sha1, (unsigned char *)shm + length - 20) ||\n+\t /*\n+\t * No need to clear CE_WATCHMAN_DIRTY set by 'WAMA' on\n+\t * disk. Watchman can only set more, not clear any, so\n+\t * this is OR mask.\n+\t */\n+\t read_watchman_ext(istate, shm, length - 20))\n+\t\tgoto done;\n+\n+\t/*\n+\t * Now that we've marked the invalid entries in the\n+\t * untracked-cache itself, we can erase them from the list of\n+\t * entries to be processed and mark the untracked cache for\n+\t * watchman usage.\n+\t */\n+\tif (istate->untracked) {\n+\t\tstring_list_clear(&istate->untracked->invalid_untracked, 0);\n+\t\tistate->untracked->use_watchman = 1;\n+\t}\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\tif (ce_stage(ce) || (ce->ce_flags & CE_WATCHMAN_DIRTY))\n+\t\t\tcontinue;\n+\t\tce_mark_uptodate(ce);\n+\t}\n+done:\n+\tif (shm)\n+\t\tmunmap(shm, length);\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1847,7 +2019,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tsplit_index = istate->split_index;\n \tif (!split_index || is_null_sha1(split_index->base_sha1)) {\n \t\tpost_read_index_from(istate);\n-\t\treturn ret;\n+\t\tgoto done;\n \t}\n \n \tif (split_index->base)\n@@ -1868,6 +2040,10 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\t sha1_to_hex(split_index->base->sha1));\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n+\n+done:\n+\tif (ret > 0 && istate->from_shm && istate->last_update)\n+\t\trefresh_by_watchman(istate);\n \treturn ret;\n }\n \n@@ -2169,7 +2345,7 @@ out:\n \treturn 0;\n }\n \n-static int verify_index(const struct index_state *istate)\n+int verify_index(const struct index_state *istate)\n {\n \treturn verify_index_from(istate, get_index_file());\n }\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282801","messageId":"1459980722-4836-11-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 10/16] update-index: enable/disable watchman support","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:56Z","receivedAt":"2016-04-06T22:11:56Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/update-index.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 1c94ca5..7c08e1c 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -914,6 +914,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n {\n \tint newfd, entries, has_errors = 0, nul_term_line = 0;\n \tenum uc_mode untracked_cache = UC_UNSPECIFIED;\n+\tint use_watchman = -1;\n \tint read_from_stdin = 0;\n \tint prefix_length = prefix ? strlen(prefix) : 0;\n \tint preferred_index_format = 0;\n@@ -1012,6 +1013,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"test if the filesystem supports untracked cache\"), UC_TEST),\n \t\tOPT_SET_INT(0, \"force-untracked-cache\", &untracked_cache,\n \t\t\t N_(\"enable untracked cache without testing the filesystem\"), UC_FORCE),\n+\t\tOPT_BOOL(0, \"watchman\", &use_watchman,\n+\t\t\tN_(\"use or not use watchman to reduce refresh cost\")),\n \t\tOPT_END()\n \t};\n \n@@ -1149,6 +1152,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tdie(\"Bug: bad untracked_cache value: %d\", untracked_cache);\n \t}\n \n+\tif (use_watchman > 0) {\n+\t\tthe_index.last_update = xstrdup(\"\");\n+\t\tthe_index.cache_changed |= WATCHMAN_CHANGED;\n+\t} else if (!use_watchman) {\n+\t\tthe_index.last_update = NULL;\n+\t\tthe_index.cache_changed |= WATCHMAN_CHANGED;\n+\t}\n+\n \tif (active_cache_changed) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282799","messageId":"1459980722-4836-12-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 11/16] unpack-trees: preserve index extensions","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:57Z","receivedAt":"2016-04-06T22:11:57Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Make git checkout (and other unpack_tree operations) preserve the\nuntracked cache and watchman status. This is valuable for two reasons:\n\n1. Often, an unpack_tree operation will not touch large parts of the\nworking tree, and thus most of the untracked cache will continue to be\nvalid.\n\n2. Even if the untracked cache were entirely invalidated by such an\noperation, the user has signaled their intention to have such a cache,\nand we don't want to throw it away.\n\nThe same logic applies to the watchman state.\n\nSigned-off-by: David Turner \n---\n cache.h | 1 +\n read-cache.c | 8 ++++++++\n t/t7063-status-untracked-cache.sh | 22 ++++++++++++++++++++++\n t/test-lib-functions.sh | 4 ++++\n unpack-trees.c | 1 +\n 5 files changed, 36 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex e43a6e1..5713835 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -571,6 +571,7 @@ extern void write_watchman_ext(struct strbuf *sb, struct index_state* istate);\n #define CLOSE_LOCK\t\t(1 << 1)\n extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\n extern int discard_index(struct index_state *);\n+extern void move_index_extensions(struct index_state *dst, struct index_state *src);\n extern int unmerged_index(const struct index_state *);\n extern int verify_path(const char *path);\n extern int index_dir_exists(struct index_state *istate, const char *name, int namelen);\ndiff --git a/read-cache.c b/read-cache.c\nindex b6e9244..470cd7b 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2725,3 +2725,11 @@ void stat_validity_update(struct stat_validity *sv, int fd)\n \t\tfill_stat_data(sv->sd, &st);\n \t}\n }\n+\n+void move_index_extensions(struct index_state *dst, struct index_state *src)\n+{\n+\tdst->untracked = src->untracked;\n+\tsrc->untracked = NULL;\n+\tdst->last_update = src->last_update;\n+\tsrc->last_update = NULL;\n+}\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex a971884..083516d 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -646,4 +646,26 @@ test_expect_success 'test ident field is working' '\n \ttest_cmp ../expect ../err\n '\n \n+test_expect_success 'untracked cache survives a checkout' '\n+\tgit commit --allow-empty -m empty &&\n+\ttest-dump-untracked-cache >../before &&\n+\ttest_when_finished \"git checkout master\" &&\n+\tgit checkout -b other_branch &&\n+\ttest-dump-untracked-cache >../after &&\n+\ttest_cmp ../before ../after &&\n+\ttest_commit test &&\n+\ttest-dump-untracked-cache >../before &&\n+\tgit checkout master &&\n+\ttest-dump-untracked-cache >../after &&\n+\ttest_cmp ../before ../after\n+'\n+\n+test_expect_success 'untracked cache survives a commit' '\n+\ttest-dump-untracked-cache >../before &&\n+\tgit add done/two &&\n+\tgit commit -m commit &&\n+\ttest-dump-untracked-cache >../after &&\n+\ttest_cmp ../before ../after\n+'\n+\n test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 8d99eb3..e974b5b 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -186,6 +186,10 @@ test_commit () {\n \t\ttest_tick\n \tfi &&\n \tgit commit $signoff -m \"$1\" &&\n+\tif [ \"$(git config core.bare)\" = false ]\n+\tthen\n+\t git update-index --force-untracked-cache\n+\tfi\n \tgit tag \"${4:-$1}\"\n }\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 9f55cc2..fc90eb3 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1215,6 +1215,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t\t\t\t\t WRITE_TREE_SILENT |\n \t\t\t\t\t\t WRITE_TREE_REPAIR);\n \t\t}\n+\t\tmove_index_extensions(&o->result, o->dst_index);\n \t\tdiscard_index(o->dst_index);\n \t\t*o->dst_index = o->result;\n \t} else {\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282804","messageId":"1459980722-4836-13-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 12/16] index-helper: kill mode","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:58Z","receivedAt":"2016-04-06T22:11:58Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Add a new command (and command-line arg) to allow index-helpers to\nexit cleanly.\n\nThis is mainly useful for tests.\n\nSigned-off-by: David Turner \n---\n index-helper.c | 31 ++++++++++++++++++++++++++++++-\n t/t7900-index-helper.sh | 13 +++++++++++++\n 2 files changed, 43 insertions(+), 1 deletion(-)\n\ndiff --git a/index-helper.c b/index-helper.c\nindex f993ae6..a3f6a74 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -276,6 +276,8 @@ static void loop(int fd, int idle_in_seconds)\n \t\t\t\t\t * alive, nothing to do.\n \t\t\t\t\t */\n \t\t\t\t}\n+\t\t\t} else if (!strcmp(command.buf, \"die\")) {\n+\t\t\t\tbreak;\n \t\t\t} else {\n \t\t\t\twarning(\"BUG: Bogus command %s\", command.buf);\n \t\t\t}\n@@ -358,10 +360,29 @@ static void make_socket_path(struct strbuf *path)\n \tstrbuf_addstr(path, \"/s\");\n }\n \n+static void request_kill(void)\n+{\n+\tint fd = connect_to_index_helper();\n+\n+\tif (fd >= 0) {\n+\t\twrite_in_full(fd, \"die\", 4);\n+\t\tclose(fd);\n+\t}\n+\n+\t/*\n+\t * The child will try to do this anyway, but we want to be\n+\t * ready to launch a new daemon immediately after this command\n+\t * returns.\n+\t */\n+\n+\tunlink(git_path(\"index-helper.path\"));\n+\treturn;\n+}\n+\n int main(int argc, char **argv)\n {\n \tconst char *prefix;\n-\tint idle_in_seconds = 600, detach = 0;\n+\tint idle_in_seconds = 600, detach = 0, kill = 0;\n \tint fd;\n \tstruct strbuf socket_path = STRBUF_INIT;\n \tstruct option options[] = {\n@@ -370,6 +391,7 @@ int main(int argc, char **argv)\n \t\tOPT_BOOL(0, \"strict\", &to_verify,\n \t\t\t \"verify shared memory after creating\"),\n \t\tOPT_BOOL(0, \"detach\", &detach, \"detach the process\"),\n+\t\tOPT_BOOL(0, \"kill\", &kill, \"request that existing index helper processes exit\"),\n \t\tOPT_END()\n \t};\n \n@@ -384,6 +406,13 @@ int main(int argc, char **argv)\n \t\t\t options, usage_text, 0))\n \t\tdie(_(\"too many arguments\"));\n \n+\tif (kill) {\n+\t\tif (detach)\n+\t\t\tdie(_(\"--kill doesn't want any other options\"));\n+\t\trequest_kill();\n+\t\treturn 0;\n+\t}\n+\n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n \ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex 6a06094..b1fa96e 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -20,4 +20,17 @@ test_expect_success 'index-helper smoke test' '\n \ttest_path_is_missing .git/index-helper.path\n '\n \n+test_expect_success 'index-helper creates usable path file and can be killed' '\n+\ttest_when_finished \"git index-helper --kill\" &&\n+\ttest_path_is_missing .git/index-helper.path &&\n+\tgit index-helper --detach &&\n+\ttest -L .git/index-helper.path &&\n+\tsock=\"$(readlink .git/index-helper.path)\" &&\n+\ttest -S \"$sock\" &&\n+\tdir=$(dirname \"$sock\") &&\n+\tls -ld \"$dir\" | grep ^drwx...--- &&\n+\tgit index-helper --kill &&\n+\ttest_path_is_missing .git/index-helper.path\n+'\n+\n test_done\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282803","messageId":"1459980722-4836-14-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 13/16] index-helper: don't run if already running","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:11:59Z","receivedAt":"2016-04-06T22:11:59Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Signed-off-by: David Turner \n---\n index-helper.c | 7 +++++++\n t/t7900-index-helper.sh | 9 +++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/index-helper.c b/index-helper.c\nindex a3f6a74..b62d805 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -413,6 +413,13 @@ int main(int argc, char **argv)\n \t\treturn 0;\n \t}\n \n+\t/* check that no other copy is running */\n+\tfd = connect_to_index_helper();\n+\tif (fd > 0)\n+\t\tdie(_(\"Already running\"));\n+\tif (errno != ECONNREFUSED && errno != ENOENT)\n+\t\tdie_errno(_(\"Unexpected error checking socket\"));\n+\n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n \ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex b1fa96e..3f94d8a 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -33,4 +33,13 @@ test_expect_success 'index-helper creates usable path file and can be killed' '\n \ttest_path_is_missing .git/index-helper.path\n '\n \n+test_expect_success 'index-helper will not start if already running' '\n+\ttest_when_finished \"git index-helper --kill\" &&\n+\tgit index-helper --detach &&\n+\ttest -L .git/index-helper.path &&\n+\ttest_must_fail git index-helper 2>err &&\n+\ttest -L .git/index-helper.path &&\n+\tgrep \"Already running\" err\n+'\n+\n test_done\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282808","messageId":"1459980722-4836-15-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 14/16] index-helper: autorun mode","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:12:00Z","receivedAt":"2016-04-06T22:12:00Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Soon, we'll want to automatically start index-helper, so we need\na mode that silently exits if it can't start up (either because\nit's not in a git dir, or because another one is already running).\n\nSigned-off-by: David Turner \n---\n index-helper.c | 29 +++++++++++++++++++++++------\n t/t7900-index-helper.sh | 8 ++++++++\n 2 files changed, 31 insertions(+), 6 deletions(-)\n\ndiff --git a/index-helper.c b/index-helper.c\nindex b62d805..4d08ff4 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -382,8 +382,9 @@ static void request_kill(void)\n int main(int argc, char **argv)\n {\n \tconst char *prefix;\n-\tint idle_in_seconds = 600, detach = 0, kill = 0;\n+\tint idle_in_seconds = 600, detach = 0, kill = 0, autorun = 0;\n \tint fd;\n+\tint nongit;\n \tstruct strbuf socket_path = STRBUF_INIT;\n \tstruct option options[] = {\n \t\tOPT_INTEGER(0, \"exit-after\", &idle_in_seconds,\n@@ -392,6 +393,7 @@ int main(int argc, char **argv)\n \t\t\t \"verify shared memory after creating\"),\n \t\tOPT_BOOL(0, \"detach\", &detach, \"detach the process\"),\n \t\tOPT_BOOL(0, \"kill\", &kill, \"request that existing index helper processes exit\"),\n+\t\tOPT_BOOL(0, \"autorun\", &autorun, \"this is an automatic run of git index-helper, so certain errors can be solved by silently exiting\"),\n \t\tOPT_END()\n \t};\n \n@@ -401,7 +403,14 @@ int main(int argc, char **argv)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(usage_text, options);\n \n-\tprefix = setup_git_directory();\n+\tprefix = setup_git_directory_gently(&nongit);\n+\tif (nongit) {\n+\t\tif (autorun)\n+\t\t\texit(0);\n+\t\telse\n+\t\t\tdie(_(\"not a git repository\"));\n+\t}\n+\n \tif (parse_options(argc, (const char **)argv, prefix,\n \t\t\t options, usage_text, 0))\n \t\tdie(_(\"too many arguments\"));\n@@ -415,10 +424,18 @@ int main(int argc, char **argv)\n \n \t/* check that no other copy is running */\n \tfd = connect_to_index_helper();\n-\tif (fd > 0)\n-\t\tdie(_(\"Already running\"));\n-\tif (errno != ECONNREFUSED && errno != ENOENT)\n-\t\tdie_errno(_(\"Unexpected error checking socket\"));\n+\tif (fd > 0) {\n+\t\tif (autorun)\n+\t\t\texit(0);\n+\t\telse\n+\t\t\tdie(_(\"Already running\"));\n+\t}\n+\tif (errno != ECONNREFUSED && errno != ENOENT) {\n+\t\tif (autorun)\n+\t\t\treturn 0;\n+\t\telse\n+\t\t\tdie_errno(_(\"Unexpected error checking socket\"));\n+\t}\n \n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex 3f94d8a..3925ee3 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -42,4 +42,12 @@ test_expect_success 'index-helper will not start if already running' '\n \tgrep \"Already running\" err\n '\n \n+test_expect_success 'index-helper is quiet with --autorun' '\n+\ttest_when_finished \"git index-helper --kill\" &&\n+\tgit index-helper --kill &&\n+\tgit index-helper --detach &&\n+\ttest -L .git/index-helper.path &&\n+\tgit index-helper --autorun\n+'\n+\n test_done\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282805","messageId":"1459980722-4836-16-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 15/16] index-helper: optionally automatically run","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:12:01Z","receivedAt":"2016-04-06T22:12:01Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Introduce a new config option, indexhelper.autorun, to automatically\nrun git index-helper before starting up a builtin git command. This\nenables users to keep index-helper running without manual\nintervention.\n\nSigned-off-by: David Turner \n---\n Documentation/config.txt | 4 ++++\n git.c | 35 ++++++++++++++++++++++++++++++++++-\n t/t7900-index-helper.sh | 16 ++++++++++++++++\n 3 files changed, 54 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 2cd6bdd..8ec8824 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1852,6 +1852,10 @@ index.version::\n \tSpecify the version with which new index files should be\n \tinitialized. This does not affect existing repositories.\n \n+indexhelper.autorun::\n+\tAutomatically run git index-helper when any builtin git\n+\tcommand is run inside a repository.\n+\n init.templateDir::\n \tSpecify the directory from which templates will be copied.\n \t(See the \"TEMPLATE DIRECTORY\" section of linkgit:git-init[1].)\ndiff --git a/git.c b/git.c\nindex 6cc0c07..7d27782 100644\n--- a/git.c\n+++ b/git.c\n@@ -521,6 +521,37 @@ static void strip_extension(const char **argv)\n #define strip_extension(cmd)\n #endif\n \n+static int want_auto_index_helper(void)\n+{\n+\tint want = -1;\n+\tconst char *value = NULL;\n+\tconst char *key = \"indexhelper.autorun\";\n+\n+\tif (git_config_key_is_valid(key) &&\n+\t !git_config_get_value(key, &value)) {\n+\t\tint b = git_config_maybe_bool(key, value);\n+\t\twant = b >= 0 ? b : 0;\n+\t}\n+\treturn want;\n+}\n+\n+static void maybe_run_index_helper(struct cmd_struct *cmd)\n+{\n+\tconst char *argv[] = {\"git-index-helper\", \"--detach\", \"--autorun\", NULL};\n+\n+\tif (!(cmd->option & NEED_WORK_TREE))\n+\t\treturn;\n+\n+\tif (want_auto_index_helper() <= 0)\n+\t\treturn;\n+\n+\ttrace_argv_printf(argv, \"trace: auto index-helper:\");\n+\n+\tif (run_command_v_opt(argv,\n+\t\t\t RUN_SILENT_EXEC_FAILURE | RUN_CLEAN_ON_EXIT))\n+\t\twarning(_(\"You specified indexhelper.autorun, but running git-index-helper failed.\"));\n+}\n+\n static void handle_builtin(int argc, const char **argv)\n {\n \tconst char *cmd;\n@@ -536,8 +567,10 @@ static void handle_builtin(int argc, const char **argv)\n \t}\n \n \tbuiltin = get_builtin(cmd);\n-\tif (builtin)\n+\tif (builtin) {\n+\t\tmaybe_run_index_helper(builtin);\n \t\texit(run_builtin(builtin, argc, argv));\n+\t}\n }\n \n static void execv_dashed_external(const char **argv)\ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex 3925ee3..6e11f8c 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -50,4 +50,20 @@ test_expect_success 'index-helper is quiet with --autorun' '\n \tgit index-helper --autorun\n '\n \n+test_expect_success 'index-helper autorun works' '\n+\trm -f .git/index-helper.path &&\n+\tgit status &&\n+\ttest_path_is_missing .git/index-helper.path &&\n+\ttest_config indexhelper.autorun true &&\n+\tgit status &&\n+\ttest -L .git/index-helper.path &&\n+\tgit status 2>err &&\n+\ttest -L .git/index-helper.path &&\n+\t! grep -q . err &&\n+\tgit index-helper --kill &&\n+\ttest_config indexhelper.autorun false &&\n+\tgit status &&\n+\ttest_path_is_missing .git/index-helper.path\n+'\n+\n test_done\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282806","messageId":"1459980722-4836-17-git-send-email-dturner@twopensource.com","threadId":"41948","inReplyTo":"1459980722-4836-1-git-send-email-dturner@twopensource.com","subject":"[PATCH v3 16/16] read-cache: config for waiting for index-helper","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-06T22:12:02Z","receivedAt":"2016-04-06T22:12:02Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"When we poke index-helper, and index-helper is using watchman, we want\nto wait for a response (showing that the watchman extension shm has\nbeen prepared). If it's not using watchman, we don't.\n\nSo add a new config, core.waitforindexhelper, with sensible defaults, to\nconfigure this behavior.\n\nSigned-off-by: David Turner \n---\n Documentation/config.txt | 6 ++++++\n cache.h | 1 +\n config.c | 5 +++++\n environment.c | 5 +++++\n read-cache.c | 5 ++++-\n 5 files changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 8ec8824..f61e6fe 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -670,6 +670,12 @@ Likewise, when the `LV` environment variable is unset, Git sets it\n to `-c`. You can override this setting by exporting `LV` with\n another value or setting `core.pager` to `lv +c`.\n \n+core.waitforindexhelper::\n+\tIf true, git status and other commands which use the index\n+\twill wait for a response from the index-helper before continuing;\n+\tthis gives time for the index-helper to communicate with watchman\n+\tand give information about modified files.\n+\n core.whitespace::\n \tA comma separated list of common whitespace problems to\n \tnotice. 'git diff' will use `color.diff.whitespace` to\ndiff --git a/cache.h b/cache.h\nindex 5713835..db66451 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -691,6 +691,7 @@ extern char *git_replace_ref_base;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_watchman_sync_timeout;\n+extern int wait_for_index_helper;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex e6dc141..5f1b8bd 100644\n--- a/config.c\n+++ b/config.c\n@@ -887,6 +887,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.waitforindexhelper\")) {\n+\t\twait_for_index_helper = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.createobject\")) {\n \t\tif (!strcmp(value, \"rename\"))\n \t\t\tobject_creation_mode = OBJECT_CREATION_USES_RENAMES;\ndiff --git a/environment.c b/environment.c\nindex 35e03c7..c7fb0a9 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -95,6 +95,11 @@ int core_preload_index = 1;\n int ignore_untracked_cache_config;\n \n int core_watchman_sync_timeout = 300;\n+#ifdef USE_WATCHMAN\n+int wait_for_index_helper = 1;\n+#else\n+int wait_for_index_helper = 0;\n+#endif\n \n \n /* This is set by setup_git_dir_gently() and/or git_default_config() */\ndiff --git a/read-cache.c b/read-cache.c\nindex 470cd7b..23dbb73 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1800,7 +1800,10 @@ static int poke_daemon(struct index_state *istate,\n \tif (refresh_cache) {\n \t\tret = write_in_full(fd, \"refresh\", 8) != 8;\n \t} else {\n-\t\tret = poke_and_wait_for_reply(fd);\n+\t\tif (wait_for_index_helper)\n+\t\t\tret = poke_and_wait_for_reply(fd);\n+\t\telse\n+\t\t\tret = write_in_full(fd, \"poke\", 5) != 5;\n \t}\n \n \tclose(fd);\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"282831","messageId":"5705FC59.2050801@kdbg.org","threadId":"41948","inReplyTo":"1459980722-4836-4-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2016-04-07T06:21:13Z","receivedAt":"2016-04-07T06:21:13Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 07.04.2016 um 00:11 schrieb David Turner:\n> +static void share_index(struct index_state *istate, struct shm *is)\n> +{\n> +\tvoid *new_mmap;\n> +\tif (istate->mmap_size <= 20 ||\n> +\t hashcmp(istate->sha1,\n> +\t\t (unsigned char *)istate->mmap + istate->mmap_size - 20) ||\n> +\t !hashcmp(istate->sha1, is->sha1) ||\n> +\t git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, istate->mmap_size,\n> +\t\t\t&new_mmap, PROT_READ | PROT_WRITE, MAP_SHARED,\n> +\t\t\t\"git-index-%s\", sha1_to_hex(istate->sha1)) < 0)\n\nBuilds which have NO_MMAP set require that MAP_PRIVATE is set. So I \nwould guess that at this point you leave those builds behind. Unless we \ndeclare such systems as hopelessly outdated and remove NO_MMAP and \ncompat/mmap.c or you support index-helper only when NO_MMAP is not set.\n\n-- Hannes\n"},{"id":"282836","messageId":"alpine.DEB.2.20.1604071614250.2967@virtualbox","threadId":"41948","inReplyTo":"5705FC59.2050801@kdbg.org","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-04-07T14:14:50Z","receivedAt":"2016-04-07T14:14:50Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 7 Apr 2016, Johannes Sixt wrote:\n\n> Am 07.04.2016 um 00:11 schrieb David Turner:\n> > +static void share_index(struct index_state *istate, struct shm *is)\n> > +{\n> > +\tvoid *new_mmap;\n> > +\tif (istate->mmap_size <= 20 ||\n> > +\t hashcmp(istate->sha1,\n> > +\t\t (unsigned char *)istate->mmap + istate->mmap_size - 20)\n> > ||\n> > +\t !hashcmp(istate->sha1, is->sha1) ||\n> > +\t git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, istate->mmap_size,\n> > +\t\t\t&new_mmap, PROT_READ | PROT_WRITE, MAP_SHARED,\n> > +\t\t\t\"git-index-%s\", sha1_to_hex(istate->sha1)) < 0)\n> \n> Builds which have NO_MMAP set require that MAP_PRIVATE is set. So I would\n> guess that at this point you leave those builds behind. Unless we declare\n> such systems as hopelessly outdated and remove NO_MMAP and compat/mmap.c or\n> you support index-helper only when NO_MMAP is not set.\n\nI vote for the latter: support index-helper only when NO_MMAP is unset.\n\nCiao,\nDscho\n"},{"id":"282868","messageId":"1460054824.5540.2.camel@twopensource.com","threadId":"41948","inReplyTo":"alpine.DEB.2.20.1604071614250.2967@virtualbox","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-07T18:47:04Z","receivedAt":"2016-04-07T18:47:04Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-04-07 at 16:14 +0200, Johannes Schindelin wrote:\n> Hi,\n> \n> On Thu, 7 Apr 2016, Johannes Sixt wrote:\n> \n> > Am 07.04.2016 um 00:11 schrieb David Turner:\n> > > +static void share_index(struct index_state *istate, struct shm\n> > > *is)\n> > > +{\n> > > +\tvoid *new_mmap;\n> > > +\tif (istate->mmap_size <= 20 ||\n> > > +\t hashcmp(istate->sha1,\n> > > +\t\t (unsigned char *)istate->mmap + istate\n> > > ->mmap_size - 20)\n> > > > > \n> > > +\t !hashcmp(istate->sha1, is->sha1) ||\n> > > +\t git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, istate\n> > > ->mmap_size,\n> > > +\t\t\t&new_mmap, PROT_READ | PROT_WRITE,\n> > > MAP_SHARED,\n> > > +\t\t\t\"git-index-%s\", sha1_to_hex(istate\n> > > ->sha1)) < 0)\n> > \n> > Builds which have NO_MMAP set require that MAP_PRIVATE is set. So I\n> > would\n> > guess that at this point you leave those builds behind. Unless we\n> > declare\n> > such systems as hopelessly outdated and remove NO_MMAP and\n> > compat/mmap.c or\n> > you support index-helper only when NO_MMAP is not set.\n> \n> I vote for the latter: support index-helper only when NO_MMAP is\n> unset.\n\nWill fix, thanks.\n"},{"id":"282916","messageId":"xmqqh9fdxcht.fsf@gitster.mtv.corp.google.com","threadId":"41948","inReplyTo":"1459980722-4836-10-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v3 09/16] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-04-07T22:47:26Z","receivedAt":"2016-04-07T22:47:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> diff --git a/read-cache.c b/read-cache.c\n> index 59d892e..b6e9244 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1407,10 +1472,24 @@ static int read_watchman_ext(struct index_state *istate, const void *data,\n> \tewah_each_bit(bitmap, mark_no_watchman, istate);\n> \tewah_free(bitmap);\n> \n> -\t/*\n> -\t * TODO: update the untracked cache from the untracked data in this\n> -\t * extension.\n> -\t */\n> +\tif (istate->untracked && istate->untracked->root) {\n> +\t\tint i;\n> +\t\tconst char *untracked;\n> +\n> +\t\tuntracked = data + len + 8 + bitmap_size;\n\nThis breaks compilation as data here is of type (void *).\n\nThere may be similar breakages in this patch.\n"},{"id":"282917","messageId":"xmqqd1q1xc8r.fsf@gitster.mtv.corp.google.com","threadId":"41948","inReplyTo":"xmqqh9fdxcht.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 09/16] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-04-07T22:52:52Z","receivedAt":"2016-04-07T22:52:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> +\t\tuntracked = data + len + 8 + bitmap_size;\n>\n> This breaks compilation as data here is of type (void *).\n>\n> There may be similar breakages in this patch.\n\nIt turns out that this is the only one, and\n\n\tuntracked = (char *)data + len + 8 + bitmap_size;\n\nis a sufficient fix (I've squashed it in to my copy).\n\nThanks.\n"},{"id":"282918","messageId":"1460074483.5540.9.camel@twopensource.com","threadId":"41948","inReplyTo":"xmqqd1q1xc8r.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 09/16] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-08T00:14:43Z","receivedAt":"2016-04-08T00:14:43Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-04-07 at 15:52 -0700, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > > +\t\tuntracked = data + len + 8 + bitmap_size;\n> > \n> > This breaks compilation as data here is of type (void *).\n> > \n> > There may be similar breakages in this patch.\n> \n> It turns out that this is the only one, and\n> \n> \tuntracked = (char *)data + len + 8 + bitmap_size;\n> \n> is a sufficient fix (I've squashed it in to my copy).\n> \n> Thanks.\n\nThanks. I've squashed it into mine as well but with const char *\ninstead (seems maybe cleaner to me)\n"},{"id":"282920","messageId":"xmqq8u0oy6vl.fsf@gitster.mtv.corp.google.com","threadId":"41948","inReplyTo":"1460074483.5540.9.camel@twopensource.com","subject":"Re: [PATCH v3 09/16] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-04-08T06:03:26Z","receivedAt":"2016-04-08T06:03:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> On Thu, 2016-04-07 at 15:52 -0700, Junio C Hamano wrote:\n>> Junio C Hamano writes:\n>> \n>> > > +\t\tuntracked = data + len + 8 + bitmap_size;\n>> > \n>> > This breaks compilation as data here is of type (void *).\n>> > \n>> > There may be similar breakages in this patch.\n>> \n>> It turns out that this is the only one, and\n>> \n>> \tuntracked = (char *)data + len + 8 + bitmap_size;\n>> \n>> is a sufficient fix (I've squashed it in to my copy).\n>> \n>> Thanks.\n>\n> Thanks. I've squashed it into mine as well but with const char *\n> instead (seems maybe cleaner to me)\n\nYeah, I agree, as \"data\" is \"const void *\", so even though the left\nhand side of the assignment, \"untracked\", is \"const char *\", my cast\nis (temporarily) casting the const away--it would be better to use\n\"const char *\" there.\n\nThanks.\n"},{"id":"282945","messageId":"CACsJy8C5NhaAAW=wzpwkBdLvVZz8wVM7QX==n_CL5g+LLAKY=A@mail.gmail.com","threadId":"41948","inReplyTo":"1459980722-4836-4-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-04-08T11:26:38Z","receivedAt":"2016-04-08T11:26:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Apr 7, 2016 at 5:11 AM, David Turner wrote:\n> @@ -1045,4 +1046,21 @@ struct tm *git_gmtime_r(const time_t *, struct tm *);\n> #define getc_unlocked(fh) getc(fh)\n> #endif\n>\n> +#ifdef __linux__\n> +#define UNIX_PATH_MAX 108\n> +#elif defined(__APPLE__) || defined(BSD)\n> +#define UNIX_PATH_MAX 104\n> +#else\n> +/*\n> + * Quoth POSIX: The size of sun_path has intentionally been left\n> + * undefined. This is because different implementations use different\n> + * sizes. For example, 4.3 BSD uses a size of 108, and 4.4 BSD uses a\n> + * size of 104. Since most implementations originate from BSD\n> + * versions, the size is typically in the range 92 to 108.\n> + *\n> + * Thanks, POSIX! Super-helpful! Hope we don't overflow any buffers!\n> + */\n> +#define UNIX_PATH_MAX 92\n> +#endif\n\nOkay. You probably want to fix unix-socket.c too, and maybe reuse that\ncode instead of opening unix sockets manually. There's a check in\nunix_sockaddr_init() about sizeof(sun_path) but I'm not sure if it\nsuffices.\n\nBTW, it looks like you tested this on Mac. But config.mak.uname only\nenables HAVE_SHM on Linux. Should it be enabled on Mac as well at\nleast?\n--\nDuy\n"},{"id":"282960","messageId":"alpine.DEB.2.20.1604081616550.2967@virtualbox","threadId":"41948","inReplyTo":"1460054824.5540.2.camel@twopensource.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-04-08T14:17:47Z","receivedAt":"2016-04-08T14:17:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi David,\n\nOn Thu, 7 Apr 2016, David Turner wrote:\n\n> On Thu, 2016-04-07 at 16:14 +0200, Johannes Schindelin wrote:\n> > \n> > On Thu, 7 Apr 2016, Johannes Sixt wrote:\n> > \n> > > Am 07.04.2016 um 00:11 schrieb David Turner:\n> > > > +static void share_index(struct index_state *istate, struct shm\n> > > > *is)\n> > > > +{\n> > > > +\tvoid *new_mmap;\n> > > > +\tif (istate->mmap_size <= 20 ||\n> > > > +\t hashcmp(istate->sha1,\n> > > > +\t\t (unsigned char *)istate->mmap + istate\n> > > > ->mmap_size - 20)\n> > > > > > \n> > > > +\t !hashcmp(istate->sha1, is->sha1) ||\n> > > > +\t git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, istate\n> > > > ->mmap_size,\n> > > > +\t\t\t&new_mmap, PROT_READ | PROT_WRITE,\n> > > > MAP_SHARED,\n> > > > +\t\t\t\"git-index-%s\", sha1_to_hex(istate\n> > > > ->sha1)) < 0)\n> > > \n> > > Builds which have NO_MMAP set require that MAP_PRIVATE is set. So I\n> > > would\n> > > guess that at this point you leave those builds behind. Unless we\n> > > declare\n> > > such systems as hopelessly outdated and remove NO_MMAP and\n> > > compat/mmap.c or\n> > > you support index-helper only when NO_MMAP is not set.\n> > \n> > I vote for the latter: support index-helper only when NO_MMAP is\n> > unset.\n> \n> Will fix, thanks.\n\nThank you!\n\nJust for the record, I was really pushing forward with my rebase--helper\nwork, is the only reason I did not review your index-helper series yet!\n\nThanks,\nDscho\n"},{"id":"282997","messageId":"1460153784.5540.19.camel@twopensource.com","threadId":"41948","inReplyTo":"CACsJy8C5NhaAAW=wzpwkBdLvVZz8wVM7QX==n_CL5g+LLAKY=A@mail.gmail.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-08T22:16:24Z","receivedAt":"2016-04-08T22:16:24Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2016-04-08 at 18:26 +0700, Duy Nguyen wrote:\n> On Thu, Apr 7, 2016 at 5:11 AM, David Turner <\n> dturner@twopensource.com> wrote:\n> > @@ -1045,4 +1046,21 @@ struct tm *git_gmtime_r(const time_t *,\n> > struct tm *);\n> > #define getc_unlocked(fh) getc(fh)\n> > #endif\n> > \n> > +#ifdef __linux__\n> > +#define UNIX_PATH_MAX 108\n> > +#elif defined(__APPLE__) || defined(BSD)\n> > +#define UNIX_PATH_MAX 104\n> > +#else\n> > +/*\n> > + * Quoth POSIX: The size of sun_path has intentionally been left\n> > + * undefined. This is because different implementations use\n> > different\n> > + * sizes. For example, 4.3 BSD uses a size of 108, and 4.4 BSD\n> > uses a\n> > + * size of 104. Since most implementations originate from BSD\n> > + * versions, the size is typically in the range 92 to 108.\n> > + *\n> > + * Thanks, POSIX! Super-helpful! Hope we don't overflow any\n> > buffers!\n> > + */\n> > +#define UNIX_PATH_MAX 92\n> > +#endif\n> \n> Okay. You probably want to fix unix-socket.c too, and maybe reuse\n> that\n> code instead of opening unix sockets manually. There's a check in\n> unix_sockaddr_init() about sizeof(sun_path) but I'm not sure if it\n> suffices.\n> \n> BTW, it looks like you tested this on Mac. But config.mak.uname only\n> enables HAVE_SHM on Linux. Should it be enabled on Mac as well at\n> least?\n\nI did not in fact test on Mac. I should have. And indeed I need to\nchange config.mak.uname. \n\nAlso I was leaking some memory. And had some whiny warnings. And had\na bug around how non-blocking sockets work on Mac. \n\nAnd SHM on Macs works a bit differently than on Linux in at least two\nirritating ways. \n\nSo, uh, new version to come once I actually make it work on Mac.\nProbably Monday.\n"},{"id":"283161","messageId":"1460417232.5540.53.camel@twopensource.com","threadId":"41948","inReplyTo":"1460153784.5540.19.camel@twopensource.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-11T23:27:12Z","receivedAt":"2016-04-11T23:27:12Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2016-04-08 at 18:16 -0400, David Turner wrote:\n> And SHM on Macs works a bit differently than on Linux in at least two\n> irritating ways. \n> \n> So, uh, new version to come once I actually make it work on Mac.\n> Probably Monday.\n\nI was chatting with a friend about this and he mentioned that SHM does\nnot really fit well into the Unix \"everything is a file\" model. It\nlives in a separate namespace, and still requires most of the file-like\noperations just with funny names and a separate namespace: shm_open,\nshm_unlink. This weirdness is something I noticed in my porting work:\non OS X, a shm name can only be 32 bytes long, requiring weird hacks.\nAnd on OSX, fstat on a shm fd is rounded up to the page size (!). \n There may also be other portability issues that I have not yet\ndiscovered.\n\nInstead, my friend suggests that we should just use files. For\ninstance, we could do $TMPDIR/$index_helper_pid/shm-index.$sha. \n\n(I'm proposing $TMPDIR because it's cleaned up on reboot so we don't\nneed any manual intervention or complicated gc schemes)\n\nWhat do folks think of this?\n"},{"id":"283174","messageId":"CACsJy8C2FtdetQ_NJSKR_JCZ5Ju0E3rV7Du=J4f2_kn5qrcHxg@mail.gmail.com","threadId":"41948","inReplyTo":"1460417232.5540.53.camel@twopensource.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-04-12T09:40:30Z","receivedAt":"2016-04-12T09:40:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Apr 12, 2016 at 6:27 AM, David Turner wrote:\n> On Fri, 2016-04-08 at 18:16 -0400, David Turner wrote:\n>> And SHM on Macs works a bit differently than on Linux in at least two\n>> irritating ways.\n>>\n>> So, uh, new version to come once I actually make it work on Mac.\n>> Probably Monday.\n>\n> I was chatting with a friend about this and he mentioned that SHM does\n> not really fit well into the Unix \"everything is a file\" model. It\n> lives in a separate namespace, and still requires most of the file-like\n> operations just with funny names and a separate namespace: shm_open,\n> shm_unlink. This weirdness is something I noticed in my porting work:\n> on OS X, a shm name can only be 32 bytes long, requiring weird hacks.\n> And on OSX, fstat on a shm fd is rounded up to the page size (!).\n> There may also be other portability issues that I have not yet\n> discovered.\n>\n> Instead, my friend suggests that we should just use files. For\n> instance, we could do $TMPDIR/$index_helper_pid/shm-index.$sha.\n>\n> (I'm proposing $TMPDIR because it's cleaned up on reboot so we don't\n> need any manual intervention or complicated gc schemes)\n>\n> What do folks think of this?\n\nI avoided actual files for two reasons\n\n - disk error rate is higher than memory one, and we might need\ntrailing SHA-1 back\n - access is slow (unless cached, but we can't be sure)\n\nIf we can keep index-helper stuff on tmpfs or similar, then it would\naddress both, but that's even more OS-specific than shm. If we have a\ngood abstraction layer then people can put stuff on $TMPDIR on Mac,\nfor example. But then it's not full POSIX file interface anymore...\nAnd you forgot Windows which does not strictly follow UNIX design.\n-- \nDuy\n"},{"id":"283207","messageId":"1460482108.5540.59.camel@twopensource.com","threadId":"41948","inReplyTo":"CACsJy8C2FtdetQ_NJSKR_JCZ5Ju0E3rV7Du=J4f2_kn5qrcHxg@mail.gmail.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-12T17:28:28Z","receivedAt":"2016-04-12T17:28:28Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2016-04-12 at 16:40 +0700, Duy Nguyen wrote:\n> On Tue, Apr 12, 2016 at 6:27 AM, David Turner <\n> dturner@twopensource.com> wrote:\n> > On Fri, 2016-04-08 at 18:16 -0400, David Turner wrote:\n> > > And SHM on Macs works a bit differently than on Linux in at least\n> > > two\n> > > irritating ways.\n> > > \n> > > So, uh, new version to come once I actually make it work on Mac.\n> > > Probably Monday.\n> > \n> > I was chatting with a friend about this and he mentioned that SHM\n> > does\n> > not really fit well into the Unix \"everything is a file\" model. It\n> > lives in a separate namespace, and still requires most of the file\n> > -like\n> > operations just with funny names and a separate namespace:\n> > shm_open,\n> > shm_unlink. This weirdness is something I noticed in my porting\n> > work:\n> > on OS X, a shm name can only be 32 bytes long, requiring weird\n> > hacks.\n> > And on OSX, fstat on a shm fd is rounded up to the page size (!).\n> > There may also be other portability issues that I have not yet\n> > discovered.\n> > \n> > Instead, my friend suggests that we should just use files. For\n> > instance, we could do $TMPDIR/$index_helper_pid/shm-index.$sha.\n> > \n> > (I'm proposing $TMPDIR because it's cleaned up on reboot so we\n> > don't\n> > need any manual intervention or complicated gc schemes)\n> > \n> > What do folks think of this?\n> \n> I avoided actual files for two reasons\n> \n> - disk error rate is higher than memory one, and we might need\n> trailing SHA-1 back\n\nThis only matters if we ever *read* the mmap off disk. But that will\nrarely happen -- usually, everything will stay in memory. Also, we\nnever worry about disk errors for other git objects (e.g. blobs,\ncommits, or trees), so it seems silly to worry for the index.\n\n> - access is slow (unless cached, but we can't be sure)\n\nWe could solve this (and the other problem) with mlock.\n\n> If we can keep index-helper stuff on tmpfs or similar, then it would\n> address both, but that's even more OS-specific than shm. If we have a\n> good abstraction layer then people can put stuff on $TMPDIR on Mac,\n> for example. But then it's not full POSIX file interface anymore...\n> And you forgot Windows which does not strictly follow UNIX design.\n\nAs I understand it, Windows does support MAP_SHARED but not shm_open. \n So Windows-specific stuff has to be done for shm_open but not for\nfiles.\n"},{"id":"283209","messageId":"xmqqfuuqr9ca.fsf@gitster.mtv.corp.google.com","threadId":"41948","inReplyTo":"1460482108.5540.59.camel@twopensource.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-04-12T18:05:41Z","receivedAt":"2016-04-12T18:05:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n>> I avoided actual files for two reasons\n>> \n>> - disk error rate is higher than memory one, and we might need\n>> trailing SHA-1 back\n>\n> This only matters if we ever *read* the mmap off disk. But that will\n> rarely happen -- usually, everything will stay in memory.\n\nTrue, and I do not think I'd terribly mind if we decided to use a\non-disk file mmaped with MAP_SHARED as a more portable substitute\nfor shm_* for this reason.\n\n> Also, we\n> never worry about disk errors for other git objects (e.g. blobs,\n> commits, or trees), so it seems silly to worry for the index.\n\nThe objects are protected by checksums and fsck will notice errors,\nso the argument itself is bogus, I'd think, but you do not have to\ndepend on that argument to support \"let's use a mmaped file as a\nmore portable substitute for shm*\" in the first place, I would say.\n\n>> - access is slow (unless cached, but we can't be sure)\n>\n> We could solve this (and the other problem) with mlock.\n\nProbably you meant madvise(2)?\n\nFor something of a size comparable to the index file held by\nindex-helper-daemon in-core, I'd expect we wouldn't page too\nbadly.\n"},{"id":"283259","messageId":"1460507524.5540.73.camel@twopensource.com","threadId":"41948","inReplyTo":"xmqqfuuqr9ca.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-13T00:32:04Z","receivedAt":"2016-04-13T00:32:04Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2016-04-12 at 11:05 -0700, Junio C Hamano wrote:\n> > > - access is slow (unless cached, but we can't be sure)\n> > \n> > We could solve this (and the other problem) with mlock.\n> \n> Probably you meant madvise(2)?\n> \n> For something of a size comparable to the index file held by\n> index-helper-daemon in-core, I'd expect we wouldn't page too\n> badly.\n\nNo, I meant mlock -- but I guess that's not portable. madvise with\nMADV_WILLNEED will probably do the job.\n"},{"id":"283444","messageId":"CACsJy8B6o-8YJYemcYX3NwyMaY9jV666ZwKJYwfbwMSmzM-YGw@mail.gmail.com","threadId":"41948","inReplyTo":"xmqqfuuqr9ca.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-04-14T10:43:23Z","receivedAt":"2016-04-14T10:43:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Apr 13, 2016 at 1:05 AM, Junio C Hamano wrote:\n>>> - access is slow (unless cached, but we can't be sure)\n>>\n>> We could solve this (and the other problem) with mlock.\n>\n> Probably you meant madvise(2)?\n>\n> For something of a size comparable to the index file held by\n> index-helper-daemon in-core, I'd expect we wouldn't page too\n> badly.\n\nI had a look at linux implementation of madvise(MADV_WILLNEED). All it\ndoes is force populating the entire memory region, which is good. But\nI suspect when memory is under pressure, some pages may be reclaimed.\nindex files in monster repo case can go up to a few hundred megabytes,\nchances of being reclaimed rise accordingly. But we can reconsider\nmlock() later when/if real problems happen.\n-- \nDuy\n"},{"id":"283466","messageId":"xmqqd1psgiuc.fsf@gitster.mtv.corp.google.com","threadId":"41948","inReplyTo":"CACsJy8B6o-8YJYemcYX3NwyMaY9jV666ZwKJYwfbwMSmzM-YGw@mail.gmail.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-04-14T18:12:43Z","receivedAt":"2016-04-14T18:12:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On Wed, Apr 13, 2016 at 1:05 AM, Junio C Hamano wrote:\n>>>> - access is slow (unless cached, but we can't be sure)\n>>>\n>>> We could solve this (and the other problem) with mlock.\n>>\n>> Probably you meant madvise(2)?\n>>\n>> For something of a size comparable to the index file held by\n>> index-helper-daemon in-core, I'd expect we wouldn't page too\n>> badly.\n>\n> I had a look at linux implementation of madvise(MADV_WILLNEED). All it\n> does is force populating the entire memory region, which is good. But\n> I suspect when memory is under pressure, some pages may be reclaimed.\n\nI share that suspicion. Why is such a reclamation bad thing, though?\n\n> index files in monster repo case can go up to a few hundred megabytes,\n> chances of being reclaimed rise accordingly. But we can reconsider\n> mlock() later when/if real problems happen.\n\nHolding onto \"a few hundred megabytes\" just so that occasional Git\noperations will not stall with the daemon and slowing down the\noverall work of the user by panalizing other elements in the user's\nworkflow does not sound like a good trade-off to me. Wouldn't the\nuser better off by not using the daemon at that point, which would\ngive the few hundred megabytes back to the system for better uses?\n"},{"id":"283472","messageId":"1460658054.5540.89.camel@twopensource.com","threadId":"41948","inReplyTo":"xmqqd1psgiuc.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-04-14T18:20:54Z","receivedAt":"2016-04-14T18:20:54Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-04-14 at 11:12 -0700, Junio C Hamano wrote:\n> Duy Nguyen writes:\n> \n> > On Wed, Apr 13, 2016 at 1:05 AM, Junio C Hamano \n> > wrote:\n> > > > > - access is slow (unless cached, but we can't be sure)\n> > > > \n> > > > We could solve this (and the other problem) with mlock.\n> > > \n> > > Probably you meant madvise(2)?\n> > > \n> > > For something of a size comparable to the index file held by\n> > > index-helper-daemon in-core, I'd expect we wouldn't page too\n> > > badly.\n> > \n> > I had a look at linux implementation of madvise(MADV_WILLNEED). All\n> > it\n> > does is force populating the entire memory region, which is good.\n> > But\n> > I suspect when memory is under pressure, some pages may be\n> > reclaimed.\n> \n> I share that suspicion. Why is such a reclamation bad thing, though?\n> \n> > index files in monster repo case can go up to a few hundred\n> > megabytes,\n> > chances of being reclaimed rise accordingly. But we can reconsider\n> > mlock() later when/if real problems happen.\n> \n> Holding onto \"a few hundred megabytes\" just so that occasional Git\n> operations will not stall with the daemon and slowing down the\n> overall work of the user by panalizing other elements in the user's\n> workflow does not sound like a good trade-off to me. Wouldn't the\n> user better off by not using the daemon at that point, which would\n> give the few hundred megabytes back to the system for better uses?\n\nBy running git index-helper, the user has signaled their intent about\nmemory usage. It's probably reasonable to honor this intent (even if\nthe user might be mistaken about what the right trade-off is).\n"},{"id":"283505","messageId":"CACsJy8C9wA=Yig6n4evqYnpOTqOWKbH4kXH5OaYTRvS_5a+a8A@mail.gmail.com","threadId":"41948","inReplyTo":"xmqqd1psgiuc.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-04-15T00:16:03Z","receivedAt":"2016-04-15T00:16:03Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Apr 15, 2016 at 1:12 AM, Junio C Hamano wrote:\n> Duy Nguyen writes:\n>\n>> On Wed, Apr 13, 2016 at 1:05 AM, Junio C Hamano wrote:\n>>>>> - access is slow (unless cached, but we can't be sure)\n>>>>\n>>>> We could solve this (and the other problem) with mlock.\n>>>\n>>> Probably you meant madvise(2)?\n>>>\n>>> For something of a size comparable to the index file held by\n>>> index-helper-daemon in-core, I'd expect we wouldn't page too\n>>> badly.\n>>\n>> I had a look at linux implementation of madvise(MADV_WILLNEED). All it\n>> does is force populating the entire memory region, which is good. But\n>> I suspect when memory is under pressure, some pages may be reclaimed.\n>\n> I share that suspicion. Why is such a reclamation bad thing, though?\n\nBecause the index is read entirely in memory in the next git process,\nit will be slowed down a bit (and the disk error factor creeps in),\nbut the next one would be fast again. So no, not so bad.\n\n>> index files in monster repo case can go up to a few hundred megabytes,\n>> chances of being reclaimed rise accordingly. But we can reconsider\n>> mlock() later when/if real problems happen.\n>\n> Holding onto \"a few hundred megabytes\" just so that occasional Git\n> operations will not stall with the daemon and slowing down the\n> overall work of the user by panalizing other elements in the user's\n> workflow does not sound like a good trade-off to me. Wouldn't the\n> user better off by not using the daemon at that point, which would\n> give the few hundred megabytes back to the system for better uses?\n\nIn an ideal world, I would go with multiple index backends and use\nsomething else, maybe lmdb again, more suitable for indexes this size.\nBut that touches many many places.\n\nWe don't hold this memory forever though. If the daemon is idle for a\nwhile, it exits, releasing memory back to system. And not using\nmlock() already gives the OS more freedom in memory usage.\n-- \nDuy\n"},{"id":"283511","messageId":"xmqqfuunejxy.fsf@gitster.mtv.corp.google.com","threadId":"41948","inReplyTo":"CACsJy8C9wA=Yig6n4evqYnpOTqOWKbH4kXH5OaYTRvS_5a+a8A@mail.gmail.com","subject":"Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-04-15T01:31:53Z","receivedAt":"2016-04-15T01:31:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> We don't hold this memory forever though. If the daemon is idle for a\n> while, it exits, releasing memory back to system. And not using\n> mlock() already gives the OS more freedom in memory usage.\n\nYup, that is why I suspected you wanted to use something other than\nmlock().\n"}]}