{"thread":{"id":"41653","subject":"[PATCH 00/19] index-helper, watchman","startedAt":"2016-03-09T18:36:03Z","lastAt":"2016-03-29T21:51:20Z","messageCount":60,"participants":["David Turner","Junio C Hamano","Duy Nguyen","Johannes Schindelin","Torsten Bögershausen"],"isPatch":true,"patchVersion":1,"patchTotal":19},"messages":[{"id":"280482","messageId":"1457548582-28302-1-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":null,"subject":"[PATCH 00/19] index-helper, watchman","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:03Z","receivedAt":"2016-03-09T18:36:03Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"This is a rebase and extension of Duy's work on git index-helper and\nwatchman support.\n\nPatches 1-14 and are more-or-less just Duy's patches, rebased. I fixed\nup a race condition with signal handling, but otherwise didn't change\nmuch.\n\nPatch 15 preserves the untracked cache and watchman extensions across\ncheckouts.\n\nPatches 16-18 are new index-helper patches that I wrote to improve the\ngeneral usability of index-helper.\n\nAnd patch 19 is for discussion only -- if the overall concept is\ncorrect, I'll rewrite it into the places where it should go. I didn't\ndo it yet in part because I am not sure folks will agree with it and\nin part because I didn't want to mess too much with Duy's code before\nre-sending it.\n\nI haven't used this code much. I just wrote patches 16-18 today, in\nfact. It's got fewer tests than I would like, in part because it is\nsomewhat difficult to test since it involves two separate persistent\ndaemons (index-helper, watchman), and the interaction between them.\n\nAt Twitter, we're still using something based on my earlier watchman\npatches[1]. We would like to switch to this (especially if this is\nthe version that mainstream git is going towards). But we have other\nlocal patches[2], and I haven't fully finished migrating them to 2.8.\nMy informal testing shows that with the untracked cache and watchman\nindex extensions, and git index-helper[3], performance is slightly\nbetter than my earlier code. That's not a surprise, since the\nindex-helper eliminates index verification time.\n\n[1] https://github.com/dturner-tw/git/tree/dturner/watchman\n\n[2] e.g. something like this:\nhttps://github.com/dturner-tw/git/tree/dturner/journal\n\n[3] In both cases, I'm using a version which has the SSE ref-parsing\nand hashmap code; these were rejected due to complexity but they provide\na benefit for us, so we still use them.\n\nDavid Turner (5):\n unpack-trees: preserve index extensions\n index-helper: rewrite pidfile after daemonizing\n index-helper: process management\n index-helper: autorun\n hack: watchman/untracked cache mashup\n\nNguyễn Thái Ngọc Duy (14):\n trace.c: add GIT_TRACE_PACK_STATS for pack usage statistics\n read-cache.c: fix constness of verify_hdr()\n read-cache: allow to keep mmap'd memory after reading\n index-helper: new daemon for caching index and related stuff\n trace.c: add GIT_TRACE_INDEX_STATS for index statistics\n index-helper: add --strict\n daemonize(): set a flag before exiting the main process\n index-helper: add --detach\n index-helper: add Windows support\n read-cache: add watchman 'WAMA' extension\n Add watchman support to reduce index refresh cost\n read-cache: allow index-helper to prepare shm before git reads it\n index-helper: use watchman to avoid refreshing index with lstat()\n update-index: enable/disable watchman support\n\n .gitignore | 1 +\n Documentation/git-index-helper.txt | 56 +++++\n Documentation/git.txt | 4 +\n Makefile | 21 ++\n builtin/gc.c | 2 +-\n builtin/update-index.c | 11 +\n cache.h | 20 +-\n config.c | 5 +\n config.mak.uname | 3 +\n configure.ac | 8 +\n daemon.c | 2 +-\n dir.c | 23 +-\n dir.h | 6 +\n environment.c | 3 +\n git-compat-util.h | 1 +\n git.c | 37 +++-\n index-helper.c | 437 ++++++++++++++++++++++++++++++++++++\n read-cache.c | 441 +++++++++++++++++++++++++++++++++++--\n setup.c | 4 +-\n sha1_file.c | 24 ++\n shm.c | 163 ++++++++++++++\n shm.h | 23 ++\n t/t7063-status-untracked-cache.sh | 23 ++\n t/t7900-index-helper.sh | 23 ++\n t/test-lib-functions.sh | 4 +\n trace.c | 16 ++\n trace.h | 1 +\n unpack-trees.c | 1 +\n watchman-support.c | 134 +++++++++++\n watchman-support.h | 7 +\n 30 files changed, 1481 insertions(+), 23 deletions(-)\n create mode 100644 Documentation/git-index-helper.txt\n create mode 100644 index-helper.c\n create mode 100644 shm.c\n create mode 100644 shm.h\n create mode 100755 t/t7900-index-helper.sh\n create mode 100644 watchman-support.c\n create mode 100644 watchman-support.h\n\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280486","messageId":"1457548582-28302-2-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 01/19] trace.c: add GIT_TRACE_PACK_STATS for pack usage statistics","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:04Z","receivedAt":"2016-03-09T18:36:04Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\ntrace_stats() is intended for GIT_TRACE_*_STATS variable group and\nGIT_TRACE_PACK_STATS is more like an example how new vars can be added.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git.txt | 3 +++\n cache.h | 2 ++\n git.c | 1 +\n sha1_file.c | 24 ++++++++++++++++++++++++\n trace.c | 13 +++++++++++++\n trace.h | 1 +\n 6 files changed, 44 insertions(+)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 2754af8..794271e 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -1055,6 +1055,9 @@ of clones and fetches.\n \ttime of each Git command.\n \tSee 'GIT_TRACE' for available trace output options.\n \n+'GIT_TRACE_PACK_STATS'::\n+\tPrint various statistics.\n+\n 'GIT_TRACE_SETUP'::\n \tEnables trace messages printing the .git, working tree and current\n \tworking directory after Git has completed its setup phase.\ndiff --git a/cache.h b/cache.h\nindex b829410..7e01403 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1828,4 +1828,6 @@ void sleep_millisec(int millisec);\n */\n void safe_create_dir(const char *dir, int share);\n \n+void report_pack_stats(struct trace_key *key);\n+\n #endif /* CACHE_H */\ndiff --git a/git.c b/git.c\nindex 6cc0c07..a4f6f71 100644\n--- a/git.c\n+++ b/git.c\n@@ -655,6 +655,7 @@ int main(int argc, char **av)\n \tgit_setup_gettext();\n \n \ttrace_command_performance(argv);\n+\ttrace_stats();\n \n \t/*\n \t * \"git-xxxx\" is the same as \"git xxxx\", but we obviously:\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d0f2aa0..14cebdf 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -518,6 +518,7 @@ static unsigned int peak_pack_open_windows;\n static unsigned int pack_open_windows;\n static unsigned int pack_open_fds;\n static unsigned int pack_max_fds;\n+static unsigned int pack_access_nr;\n static size_t peak_pack_mapped;\n static size_t pack_mapped;\n struct packed_git *packed_git;\n@@ -543,6 +544,28 @@ void pack_report(void)\n \t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n }\n \n+void report_pack_stats(struct trace_key *key)\n+{\n+\ttrace_printf_key(key, \"\\n\"\n+\t\t\t \"pack_report: getpagesize() = %10\" SZ_FMT \"\\n\"\n+\t\t\t \"pack_report: core.packedGitWindowSize = %10\" SZ_FMT \"\\n\"\n+\t\t\t \"pack_report: core.packedGitLimit = %10\" SZ_FMT \"\\n\"\n+\t\t\t \"pack_report: pack_used_ctr = %10u\\n\"\n+\t\t\t \"pack_report: pack_mmap_calls = %10u\\n\"\n+\t\t\t \"pack_report: pack_open_windows = %10u / %10u\\n\"\n+\t\t\t \"pack_report: pack_mapped = \"\n+\t\t\t \"%10\" SZ_FMT \" / %10\" SZ_FMT \"\\n\"\n+\t\t\t \"pack_report: pack accesss = %10u\\n\",\n+\t\t\t sz_fmt(getpagesize()),\n+\t\t\t sz_fmt(packed_git_window_size),\n+\t\t\t sz_fmt(packed_git_limit),\n+\t\t\t pack_used_ctr,\n+\t\t\t pack_mmap_calls,\n+\t\t\t pack_open_windows, peak_pack_open_windows,\n+\t\t\t sz_fmt(pack_mapped), sz_fmt(peak_pack_mapped),\n+\t\t\t pack_access_nr);\n+}\n+\n /*\n * Open and mmap the index file at path, perform a couple of\n * consistency checks, then record its information to p. Return 0 on\n@@ -2238,6 +2261,7 @@ static void write_pack_access_log(struct packed_git *p, off_t obj_offset)\n \tstatic struct trace_key pack_access = TRACE_KEY_INIT(PACK_ACCESS);\n \ttrace_printf_key(&pack_access, \"%s %\"PRIuMAX\"\\n\",\n \t\t\t p->pack_name, (uintmax_t)obj_offset);\n+\tpack_access_nr++;\n }\n \n int do_check_packed_object_crc;\ndiff --git a/trace.c b/trace.c\nindex 4aeea60..b1d0885 100644\n--- a/trace.c\n+++ b/trace.c\n@@ -432,3 +432,16 @@ void trace_command_performance(const char **argv)\n \tsq_quote_argv(&command_line, argv, 0);\n \tcommand_start_time = getnanotime();\n }\n+\n+static struct trace_key trace_pack_stats = TRACE_KEY_INIT(PACK_STATS);\n+\n+static void print_stats_atexit(void)\n+{\n+\treport_pack_stats(&trace_pack_stats);\n+}\n+\n+void trace_stats(void)\n+{\n+\tif (trace_want(&trace_pack_stats))\n+\t\tatexit(print_stats_atexit);\n+}\ndiff --git a/trace.h b/trace.h\nindex 179b249..52bda4e 100644\n--- a/trace.h\n+++ b/trace.h\n@@ -19,6 +19,7 @@ extern void trace_disable(struct trace_key *key);\n extern uint64_t getnanotime(void);\n extern void trace_command_performance(const char **argv);\n extern void trace_verbatim(struct trace_key *key, const void *buf, unsigned len);\n+extern void trace_stats(void);\n \n #ifndef HAVE_VARIADIC_MACROS\n \n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280483","messageId":"1457548582-28302-3-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 02/19] read-cache.c: fix constness of verify_hdr()","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:05Z","receivedAt":"2016-03-09T18:36:05Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex d9fb78b..16cc487 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1345,7 +1345,7 @@ struct ondisk_cache_entry_extended {\n \t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr(struct cache_header *hdr, unsigned long size)\n+static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n {\n \tgit_SHA_CTX c;\n \tunsigned char sha1[20];\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280484","messageId":"1457548582-28302-4-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 03/19] read-cache: allow to keep mmap'd memory after reading","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:06Z","receivedAt":"2016-03-09T18:36:06Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 3 +++\n read-cache.c | 13 ++++++++++++-\n 2 files changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7e01403..c43ef3d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -333,11 +333,14 @@ struct index_state {\n \tstruct split_index *split_index;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n+\t\t keep_mmap : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n \tstruct untracked_cache *untracked;\n+\tvoid *mmap;\n+\tsize_t mmap_size;\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 16cc487..7e387e9 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1574,6 +1574,10 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tmmap = xmmap(NULL, mmap_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n+\tif (istate->keep_mmap) {\n+\t\tistate->mmap = mmap;\n+\t\tistate->mmap_size = mmap_size;\n+\t}\n \tclose(fd);\n \n \thdr = mmap;\n@@ -1626,10 +1630,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n-\tmunmap(mmap, mmap_size);\n+\tif (!istate->keep_mmap)\n+\t\tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n unmap:\n+\tistate->mmap = NULL;\n \tmunmap(mmap, mmap_size);\n \tdie(\"index file corrupt\");\n }\n@@ -1655,6 +1661,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdiscard_index(split_index->base);\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n+\tsplit_index->base->keep_mmap = istate->keep_mmap;\n \tret = do_read_index(split_index->base,\n \t\t\t git_path(\"sharedindex.%s\",\n \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n@@ -1698,6 +1705,10 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->keep_mmap && istate->mmap) {\n+\t\tmunmap(istate->mmap, istate->mmap_size);\n+\t\tistate->mmap = NULL;\n+\t}\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280487","messageId":"1457548582-28302-5-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:07Z","receivedAt":"2016-03-09T18:36:07Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nInstead of reading the index from disk and worrying about disk\ncorruption, the index is cached in memory (memory bit-flips happen\ntoo, but hopefully less often). The result is faster read. Read time\nis reduced by 70%.\n\nThe biggest gain is not having to verify the trailing SHA-1, which\ntakes lots of time especially on large index files. But this also\nopens doors for further optimiztions:\n\n - we could create an in-memory format that's essentially the memory\n dump of the index to eliminate most of parsing/allocation\n overhead. The mmap'd memory can be used straight away. Experiment\n [1] shows we could reduce read time by 88%.\n\n - we could cache non-index info such as name hash\n\nThe shared memory's name folows the template \"git--\"\nwhere is the trailing SHA-1 of the index file. is\n\"index\" for cached index files (and may be \"name-hash\" for name-hash\ncache). If such shared memory exists, it contains the same index\ncontent as on disk. The content is already validated by the daemon and\ngit won't validate it again (except comparing the trailing SHA-1s).\n\nGit can poke the daemon to tell it to refresh the index cache, or to\nkeep it alive some more minutes via UNIX signals. It can't give any\nreal index data directly to the daemon. Real data goes to disk first,\nthen the daemon reads and verifies it from there. Poking only happens\nfor $GIT_DIR/index, not temporary index files.\n\n$GIT_DIR/index-helper.pid contains a reference to daemon process (and\nit's pid on *nix). The file's mtime is updated every time it's accessed\n(or should be updated often enough). Old index-helper.pid is considered\nstale and ignored.\n\nindex-helper requires POSIX realtime extension. POSIX shm interface\nhowever is abstracted away so that Windows support can be added later.\n\nOn webkit.git with index format v2, duplicating 8 times to 1.4m\nentries and 200MB in size:\n\n(vanilla) 0.986986364 s: read_index_from .git/index\n(index-helper) 0.267850279 s: read_index_from .git/index\n\nInterestingly with index v4, we get less out of index-helper. It makes\nsense as v4 requires more processing after loading the index:\n\n(vanilla) 0.722496666 s: read_index_from .git/index\n(index-helper) 0.302741500 s: read_index_from .git/index\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n .gitignore | 1 +\n Documentation/git-index-helper.txt | 44 ++++++++++\n Makefile | 9 +++\n cache.h | 3 +\n config.mak.uname | 1 +\n git-compat-util.h | 1 +\n index-helper.c | 162 +++++++++++++++++++++++++++++++++++++\n read-cache.c | 106 +++++++++++++++++++++---\n shm.c | 67 +++++++++++++++\n shm.h | 23 ++++++\n 10 files changed, 408 insertions(+), 9 deletions(-)\n create mode 100644 Documentation/git-index-helper.txt\n create mode 100644 index-helper.c\n create mode 100644 shm.c\n create mode 100644 shm.h\n\ndiff --git a/.gitignore b/.gitignore\nindex 5087ce1..b92f122 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -71,6 +71,7 @@\n /git-http-fetch\n /git-http-push\n /git-imap-send\n+/git-index-helper\n /git-index-pack\n /git-init\n /git-init-db\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nnew file mode 100644\nindex 0000000..9db28cf\n--- /dev/null\n+++ b/Documentation/git-index-helper.txt\n@@ -0,0 +1,44 @@\n+git-index-helper(1)\n+===================\n+\n+NAME\n+----\n+git-index-helper - A simple cache daemon for speeding up index file access\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git index-helper' [options]\n+\n+DESCRIPTION\n+-----------\n+Keep the index file in memory for faster access. This daemon is per\n+repository.\n+\n+OPTIONS\n+-------\n+\n+--exit-after=::\n+\tExit if the cached index is not accessed for ``\n+\tminutes. Specify 0 to wait forever. Default is 10.\n+\n+NOTES\n+-----\n+On UNIX-like systems, $GIT_DIR/index-helper.pid contains the process\n+id of the daemon. At least on Linux, shared memory objects are\n+availble via /dev/shm with the name pattern \"git--\".\n+Normally the daemon will clean up shared memory objects when it exits.\n+But if it crashes, some objects could remain there and they can be\n+safely deleted with \"rm\" command. The following signals are used to\n+control the daemon:\n+\n+SIGHUP::\n+\tReread the index.\n+\n+SIGUSR1::\n+\tLet the daemon know the index is to be read. It keeps the\n+\tdaemon alive longer, unless `--exit-after=0` is used.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 24bef8d..a6c668b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -367,6 +367,8 @@ all::\n # Define HAVE_BSD_SYSCTL if your platform has a BSD-compatible sysctl function.\n #\n # Define HAVE_GETDELIM if your system has the getdelim() function.\n+#\n+# Define HAVE_SHM if you platform support shm_* functions in librt.\n \n GIT-VERSION-FILE: FORCE\n \t@$(SHELL_PATH) ./GIT-VERSION-GEN\n@@ -802,6 +804,7 @@ LIB_OBJS += sha1-lookup.o\n LIB_OBJS += sha1_file.o\n LIB_OBJS += sha1_name.o\n LIB_OBJS += shallow.o\n+LIB_OBJS += shm.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n LIB_OBJS += split-index.o\n@@ -1430,6 +1433,12 @@ ifdef HAVE_DEV_TTY\n \tBASIC_CFLAGS += -DHAVE_DEV_TTY\n endif\n \n+ifdef HAVE_SHM\n+\tBASIC_CFLAGS\t+= -DHAVE_SHM\n+\tEXTLIBS\t += -lrt\n+\tPROGRAM_OBJS\t+= index-helper.o\n+endif\n+\n ifdef DIR_HAS_BSD_GROUP_SEMANTICS\n \tCOMPAT_CFLAGS += -DDIR_HAS_BSD_GROUP_SEMANTICS\n endif\ndiff --git a/cache.h b/cache.h\nindex c43ef3d..bc2f529 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -334,6 +334,8 @@ struct index_state {\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n \t\t keep_mmap : 1,\n+\t\t from_shm : 1,\n+\t\t to_shm : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n@@ -560,6 +562,7 @@ extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n #define COMMIT_LOCK\t\t(1 << 0)\n #define CLOSE_LOCK\t\t(1 << 1)\n+#define REFRESH_DAEMON\t\t(1 << 2)\n extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\n extern int discard_index(struct index_state *);\n extern int unmerged_index(const struct index_state *);\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 4c68e07..b5108e1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -37,6 +37,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_CLOCK_GETTIME = YesPlease\n \tHAVE_CLOCK_MONOTONIC = YesPlease\n \tHAVE_GETDELIM = YesPlease\n+\tHAVE_SHM = YesPlease\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tHAVE_ALLOCA_H = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex c07e0c1..56945a7 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -513,6 +513,7 @@ static inline int ends_with(const char *str, const char *suffix)\n #define PROT_READ 1\n #define PROT_WRITE 2\n #define MAP_PRIVATE 1\n+#define MAP_SHARED 2\n #endif\n \n #define mmap git_mmap\ndiff --git a/index-helper.c b/index-helper.c\nnew file mode 100644\nindex 0000000..cf2971d\n--- /dev/null\n+++ b/index-helper.c\n@@ -0,0 +1,162 @@\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"sigchain.h\"\n+#include \"exec_cmd.h\"\n+#include \"split-index.h\"\n+#include \"shm.h\"\n+#include \"lockfile.h\"\n+\n+struct shm {\n+\tunsigned char sha1[20];\n+\tvoid *shm;\n+\tsize_t size;\n+};\n+\n+static struct shm shm_index;\n+static struct shm shm_base_index;\n+\n+static void release_index_shm(struct shm *is)\n+{\n+\tif (!is->shm)\n+\t\treturn;\n+\tmunmap(is->shm, is->size);\n+\tgit_shm_unlink(\"git-index-%s\", sha1_to_hex(is->sha1));\n+\tis->shm = NULL;\n+}\n+\n+static void cleanup_shm(void)\n+{\n+\trelease_index_shm(&shm_index);\n+\trelease_index_shm(&shm_base_index);\n+}\n+\n+static void cleanup(void)\n+{\n+\tunlink(git_path(\"index-helper.pid\"));\n+\tcleanup_shm();\n+}\n+\n+static void cleanup_on_signal(int sig)\n+{\n+\tcleanup();\n+\tsigchain_pop(sig);\n+\traise(sig);\n+}\n+\n+static void share_index(struct index_state *istate, struct shm *is)\n+{\n+\tvoid *new_mmap;\n+\tif (istate->mmap_size <= 20 ||\n+\t hashcmp(istate->sha1,\n+\t\t (unsigned char *)istate->mmap + istate->mmap_size - 20) ||\n+\t !hashcmp(istate->sha1, is->sha1) ||\n+\t git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, istate->mmap_size,\n+\t\t\t&new_mmap, PROT_READ | PROT_WRITE, MAP_SHARED,\n+\t\t\t\"git-index-%s\", sha1_to_hex(istate->sha1)) < 0)\n+\t\treturn;\n+\n+\trelease_index_shm(is);\n+\tis->size = istate->mmap_size;\n+\tis->shm = new_mmap;\n+\thashcpy(is->sha1, istate->sha1);\n+\tmemcpy(new_mmap, istate->mmap, istate->mmap_size - 20);\n+\n+\t/*\n+\t * The trailing hash must be written last after everything is\n+\t * written. It's the indication that the shared memory is now\n+\t * ready.\n+\t */\n+\thashcpy((unsigned char *)new_mmap + istate->mmap_size - 20, is->sha1);\n+}\n+\n+static void share_the_index(void)\n+{\n+\tif (the_index.split_index && the_index.split_index->base)\n+\t\tshare_index(the_index.split_index->base, &shm_base_index);\n+\tshare_index(&the_index, &shm_index);\n+\tdiscard_index(&the_index);\n+}\n+\n+static void refresh(int sig)\n+{\n+\tthe_index.keep_mmap = 1;\n+\tthe_index.to_shm = 1;\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"could not read index\"));\n+\tshare_the_index();\n+}\n+\n+#ifdef HAVE_SHM\n+\n+static void do_nothing(int sig)\n+{\n+\t/*\n+\t * what we need is the signal received and interrupts\n+\t * sleep(). We don't need to do anything else when receving\n+\t * the signal\n+\t */\n+}\n+\n+static void loop(const char *pid_file, int idle_in_seconds)\n+{\n+\tsigchain_pop(SIGHUP);\t/* pushed by sigchain_push_common */\n+\tsigchain_push(SIGHUP, refresh);\n+\tsigchain_push(SIGUSR1, do_nothing);\n+\trefresh(0);\n+\twhile (sleep(idle_in_seconds))\n+\t\t; /* do nothing, all is handled by signal handlers already */\n+}\n+\n+#else\n+\n+static void loop(const char *pid_file, int idle_in_seconds)\n+{\n+\tdie(_(\"index-helper is not supported on this platform\"));\n+}\n+\n+#endif\n+\n+static const char * const usage_text[] = {\n+\tN_(\"git index-helper [options]\"),\n+\tNULL\n+};\n+\n+int main(int argc, char **argv)\n+{\n+\tstatic struct lock_file lock;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *prefix;\n+\tint fd, idle_in_minutes = 10;\n+\tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"exit-after\", &idle_in_minutes,\n+\t\t\t N_(\"exit if not used after some minutes\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_extract_argv0_path(argv[0]);\n+\tgit_setup_gettext();\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(usage_text, options);\n+\n+\tprefix = setup_git_directory();\n+\tif (parse_options(argc, (const char **)argv, prefix,\n+\t\t\t options, usage_text, 0))\n+\t\tdie(_(\"too many arguments\"));\n+\n+\tfd = hold_lock_file_for_update(&lock,\n+\t\t\t\t git_path(\"index-helper.pid\"),\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n+\tstrbuf_addf(&sb, \"%\" PRIuMAX, (uintmax_t) getpid());\n+\twrite_in_full(fd, sb.buf, sb.len);\n+\tcommit_lock_file(&lock);\n+\n+\tatexit(cleanup);\n+\tsigchain_push_common(cleanup_on_signal);\n+\n+\tif (!idle_in_minutes)\n+\t\tidle_in_minutes = 0xffffffff / 60;\n+\tloop(sb.buf, idle_in_minutes * 60);\n+\tstrbuf_release(&sb);\n+\treturn 0;\n+}\ndiff --git a/read-cache.c b/read-cache.c\nindex 7e387e9..eb4b9b4 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -18,6 +18,7 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"utf8.h\"\n+#include \"shm.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -1541,6 +1542,81 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_untracked_cache(istate);\n }\n \n+static void do_poke(struct strbuf *sb, int refresh_cache)\n+{\n+\tchar\t*start = sb->buf;\n+\tchar\t*end = NULL;\n+\tpid_t\t pid = strtoul(start, &end, 10);\n+\tif (!end || end != sb->buf + sb->len)\n+\t\treturn;\n+\tkill(pid, refresh_cache ? SIGHUP : SIGUSR1);\n+}\n+\n+static void poke_daemon(struct index_state *istate,\n+\t\t\tconst struct stat *st, int refresh_cache)\n+{\n+\tint fd;\n+\tstruct strbuf sb;\n+\n+\t/* if this is from index-helper, do not poke itself (recursively) */\n+\tif (istate->to_shm)\n+\t\treturn;\n+\n+\tfd = open(git_path(\"index-helper.pid\"), O_RDONLY);\n+\tif (fd < 0)\n+\t\treturn;\n+\tstrbuf_init(&sb, st->st_size + 1);\n+\tstrbuf_setlen(&sb, st->st_size);\n+\tif (read_in_full(fd, sb.buf, st->st_size) == st->st_size)\n+\t\tdo_poke(&sb, refresh_cache);\n+\tclose(fd);\n+\tstrbuf_release(&sb);\n+}\n+\n+static int is_main_index(struct index_state *istate)\n+{\n+\treturn istate == &the_index ||\n+\t\t(the_index.split_index &&\n+\t\t istate == the_index.split_index->base);\n+}\n+\n+/*\n+ * Try to open and verify a cached shm index if available. Return 0 if\n+ * succeeds (istate->mmap and istate->mmap_size are updated). Return\n+ * negative otherwise.\n+ */\n+static int try_shm(struct index_state *istate)\n+{\n+\tvoid *new_mmap = NULL;\n+\tsize_t old_size = istate->mmap_size;\n+\tssize_t new_size;\n+\tconst unsigned char *sha1;\n+\tstruct stat st;\n+\n+\tif (!is_main_index(istate) ||\n+\t old_size <= 20 ||\n+\t stat(git_path(\"index-helper.pid\"), &st))\n+\t\treturn -1;\n+\tpoke_daemon(istate, &st, 0);\n+\tsha1 = (unsigned char *)istate->mmap + old_size - 20;\n+\tnew_size = git_shm_map(O_RDONLY, 0700, -1, &new_mmap,\n+\t\t\t\t PROT_READ, MAP_SHARED,\n+\t\t\t\t \"git-index-%s\", sha1_to_hex(sha1));\n+\tif (new_size <= 20 ||\n+\t hashcmp((unsigned char *)istate->mmap + old_size - 20,\n+\t\t (unsigned char *)new_mmap + new_size - 20)) {\n+\t\tif (new_mmap)\n+\t\t\tmunmap(new_mmap, new_size);\n+\t\tpoke_daemon(istate, &st, 1);\n+\t\treturn -1;\n+\t}\n+\tmunmap(istate->mmap, istate->mmap_size);\n+\tistate->mmap = new_mmap;\n+\tistate->mmap_size = new_size;\n+\tistate->from_shm = 1;\n+\treturn 0;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1555,6 +1631,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tif (istate->initialized)\n \t\treturn istate->cache_nr;\n \n+\tistate->from_shm = 0;\n \tistate->timestamp.sec = 0;\n \tistate->timestamp.nsec = 0;\n \tfd = open(path, O_RDONLY);\n@@ -1574,15 +1651,17 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tmmap = xmmap(NULL, mmap_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n-\tif (istate->keep_mmap) {\n-\t\tistate->mmap = mmap;\n-\t\tistate->mmap_size = mmap_size;\n-\t}\n \tclose(fd);\n \n-\thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n+\tistate->mmap = mmap;\n+\tistate->mmap_size = mmap_size;\n+\tif (try_shm(istate) &&\n+\t verify_hdr(istate->mmap, istate->mmap_size) < 0)\n \t\tgoto unmap;\n+\thdr = mmap = istate->mmap;\n+\tmmap_size = istate->mmap_size;\n+\tif (!istate->keep_mmap)\n+\t\tistate->mmap = NULL;\n \n \thashcpy(istate->sha1, (const unsigned char *)hdr + mmap_size - 20);\n \tistate->version = ntohl(hdr->hdr_version);\n@@ -1662,6 +1741,8 @@ int read_index_from(struct index_state *istate, const char *path)\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n \tsplit_index->base->keep_mmap = istate->keep_mmap;\n+\tsplit_index->base->to_shm = istate->to_shm;\n+\tsplit_index->base->from_shm = istate->from_shm;\n \tret = do_read_index(split_index->base,\n \t\t\t git_path(\"sharedindex.%s\",\n \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n@@ -1712,6 +1793,8 @@ int discard_index(struct index_state *istate)\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n+\tistate->from_shm = 0;\n+\tistate->to_shm = 0;\n \treturn 0;\n }\n \n@@ -2138,9 +2221,14 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \t\treturn ret;\n \tassert((flags & (COMMIT_LOCK | CLOSE_LOCK)) !=\n \t (COMMIT_LOCK | CLOSE_LOCK));\n-\tif (flags & COMMIT_LOCK)\n-\t\treturn commit_locked_index(lock);\n-\telse if (flags & CLOSE_LOCK)\n+\tif (flags & COMMIT_LOCK) {\n+\t\tstruct stat st;\n+\t\tret = commit_locked_index(lock);\n+\t\tif (!ret && is_main_index(istate) &&\n+\t\t !stat(git_path(\"index-helper.pid\"), &st))\n+\t\t\tpoke_daemon(istate, &st, 1);\n+\t\treturn ret;\n+\t} else if (flags & CLOSE_LOCK)\n \t\treturn close_lock_file(lock);\n \telse\n \t\treturn ret;\ndiff --git a/shm.c b/shm.c\nnew file mode 100644\nindex 0000000..4ec1a00\n--- /dev/null\n+++ b/shm.c\n@@ -0,0 +1,67 @@\n+#include \"git-compat-util.h\"\n+#include \"shm.h\"\n+\n+#ifdef HAVE_SHM\n+\n+#define SHM_PATH_LEN 72\t\t/* we don't create very long paths.. */\n+\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t int prot, int flags, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tchar path[SHM_PATH_LEN];\n+\tint fd;\n+\n+\tpath[0] = '/';\n+\tva_start(ap, fmt);\n+\tvsprintf(path + 1, fmt, ap);\n+\tva_end(ap);\n+\tfd = shm_open(path, oflag, perm);\n+\tif (fd < 0)\n+\t\treturn -1;\n+\tif (length > 0 && ftruncate(fd, length)) {\n+\t\tshm_unlink(path);\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\tif (length < 0 && !(oflag & O_CREAT)) {\n+\t\tstruct stat st;\n+\t\tif (fstat(fd, &st))\n+\t\t\tdie_errno(\"unable to stat %s\", path);\n+\t\tlength = st.st_size;\n+\t}\n+\t*mmap = xmmap(NULL, length, prot, flags, fd, 0);\n+\tclose(fd);\n+\tif (*mmap == MAP_FAILED) {\n+\t\t*mmap = NULL;\n+\t\tshm_unlink(path);\n+\t\treturn -1;\n+\t}\n+\treturn length;\n+}\n+\n+void git_shm_unlink(const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tchar path[SHM_PATH_LEN];\n+\n+\tpath[0] = '/';\n+\tva_start(ap, fmt);\n+\tvsprintf(path + 1, fmt, ap);\n+\tva_end(ap);\n+\tshm_unlink(path);\n+}\n+\n+#else\n+\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t int prot, int flags, const char *fmt, ...)\n+{\n+\treturn -1;\n+}\n+\n+void git_shm_unlink(const char *fmt, ...)\n+{\n+}\n+\n+#endif\ndiff --git a/shm.h b/shm.h\nnew file mode 100644\nindex 0000000..798d3fd\n--- /dev/null\n+++ b/shm.h\n@@ -0,0 +1,23 @@\n+#ifndef SHM_H\n+#define SHM_H\n+\n+/*\n+ * Create or open a shared memory and mmap it. Return mmap size if\n+ * successful, -1 otherwise. If successful mmap contains the mmap'd\n+ * pointer. If oflag does not contain O_CREAT and length is negative,\n+ * the mmap size is retrieved from existing shared memory object.\n+ *\n+ * The mmap could be freed by munmap, even on Windows. Note that on\n+ * Windows, git_shm_unlink() is no-op, so the last unmap will destroy\n+ * the shared memory.\n+ */\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t int prot, int flags, const char *fmt, ...);\n+\n+/*\n+ * Unlink a shared memory object. Only needed on POSIX platforms. On\n+ * Windows this is no-op.\n+ */\n+void git_shm_unlink(const char *fmt, ...);\n+\n+#endif\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280485","messageId":"1457548582-28302-6-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 05/19] trace.c: add GIT_TRACE_INDEX_STATS for index statistics","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:08Z","receivedAt":"2016-03-09T18:36:08Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git.txt | 1 +\n cache.h | 1 +\n read-cache.c | 16 ++++++++++++++++\n trace.c | 5 ++++-\n 4 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 794271e..71a88a8 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -1056,6 +1056,7 @@ of clones and fetches.\n \tSee 'GIT_TRACE' for available trace output options.\n \n 'GIT_TRACE_PACK_STATS'::\n+'GIT_TRACE_INDEX_STATS'::\n \tPrint various statistics.\n \n 'GIT_TRACE_SETUP'::\ndiff --git a/cache.h b/cache.h\nindex bc2f529..e22296c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1835,5 +1835,6 @@ void sleep_millisec(int millisec);\n void safe_create_dir(const char *dir, int share);\n \n void report_pack_stats(struct trace_key *key);\n+void report_index_stats(struct trace_key *key);\n \n #endif /* CACHE_H */\ndiff --git a/read-cache.c b/read-cache.c\nindex eb4b9b4..7bd3ce4 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -50,6 +50,10 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n struct index_state the_index;\n static const char *alternate_index_output;\n \n+static unsigned int nr_read_index;\n+static unsigned int nr_read_shm_index;\n+static unsigned int nr_write_index;\n+\n static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n {\n \tistate->cache[nr] = ce;\n@@ -1614,6 +1618,7 @@ static int try_shm(struct index_state *istate)\n \tistate->mmap = new_mmap;\n \tistate->mmap_size = new_size;\n \tistate->from_shm = 1;\n+\tnr_read_shm_index++;\n \treturn 0;\n }\n \n@@ -1711,6 +1716,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t}\n \tif (!istate->keep_mmap)\n \t\tmunmap(mmap, mmap_size);\n+\tnr_read_index++;\n \treturn istate->cache_nr;\n \n unmap:\n@@ -2197,6 +2203,7 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\treturn -1;\n \tistate->timestamp.sec = (unsigned int)st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n+\tnr_write_index++;\n \treturn 0;\n }\n \n@@ -2423,3 +2430,12 @@ void stat_validity_update(struct stat_validity *sv, int fd)\n \t\tfill_stat_data(sv->sd, &st);\n \t}\n }\n+\n+void report_index_stats(struct trace_key *key)\n+{\n+\ttrace_printf_key(key, \"\\n\"\n+\t\t\t \"index stats: file reads = %10u\\n\"\n+\t\t\t \"index stats: cache reads = %10u\\n\"\n+\t\t\t \"index stats: file writes = %10u\\n\",\n+\t\t\t nr_read_index, nr_read_shm_index, nr_write_index);\n+}\ndiff --git a/trace.c b/trace.c\nindex b1d0885..eea1fa8 100644\n--- a/trace.c\n+++ b/trace.c\n@@ -434,14 +434,17 @@ void trace_command_performance(const char **argv)\n }\n \n static struct trace_key trace_pack_stats = TRACE_KEY_INIT(PACK_STATS);\n+static struct trace_key trace_index_stats = TRACE_KEY_INIT(INDEX_STATS);\n \n static void print_stats_atexit(void)\n {\n \treport_pack_stats(&trace_pack_stats);\n+\treport_index_stats(&trace_index_stats);\n }\n \n void trace_stats(void)\n {\n-\tif (trace_want(&trace_pack_stats))\n+\tif (trace_want(&trace_pack_stats) ||\n+\t trace_want(&trace_index_stats))\n \t\tatexit(print_stats_atexit);\n }\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280489","messageId":"1457548582-28302-7-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 06/19] index-helper: add --strict","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:09Z","receivedAt":"2016-03-09T18:36:09Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThere's are \"holes\" in the index-helper approach because the shared\nmemory is not verified again by git. If $USER is compromised, shared\nmemory could be modified. But then they can already modify\n$GIT_DIR/index. A more realistic risk is some bugs in index-helper\nproduce corrupt shared memory. --strict is added to avoid that\n\nStrictly speaking there's still a very small gap where corrupt shared\nmemory could still be read by git: after we write the trailing SHA-1 in\nthe shared memory (thus signaling \"this shm is ready\") and before\nverify_shm() detects an error.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-index-helper.txt | 9 +++++++\n cache.h | 1 +\n index-helper.c | 48 ++++++++++++++++++++++++++++++++++++++\n read-cache.c | 9 ++++---\n 4 files changed, 64 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex 9db28cf..ad40366 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -22,6 +22,15 @@ OPTIONS\n \tExit if the cached index is not accessed for ``\n \tminutes. Specify 0 to wait forever. Default is 10.\n \n+--strict::\n+--no-strict::\n+\tStrict mode makes index-helper verify the shared memory after\n+\tit's created. If the result does not match what's read from\n+\t$GIT_DIR/index, the shared memory is destroyed. This makes\n+\tindex-helper take more than double the amount of time required\n+\tfor reading an index, but because it will happen in the\n+\tbackground, it's not noticable. `--strict` is enabled by default.\n+\n NOTES\n -----\n On UNIX-like systems, $GIT_DIR/index-helper.pid contains the process\ndiff --git a/cache.h b/cache.h\nindex e22296c..1d7e561 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -336,6 +336,7 @@ struct index_state {\n \t\t keep_mmap : 1,\n \t\t from_shm : 1,\n \t\t to_shm : 1,\n+\t\t always_verify_trailing_sha1 : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\ndiff --git a/index-helper.c b/index-helper.c\nindex cf2971d..1140bc0 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -14,6 +14,7 @@ struct shm {\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n+static int to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n {\n@@ -69,11 +70,56 @@ static void share_index(struct index_state *istate, struct shm *is)\n \thashcpy((unsigned char *)new_mmap + istate->mmap_size - 20, is->sha1);\n }\n \n+static int verify_shm(void)\n+{\n+\tint i;\n+\tstruct index_state istate;\n+\tmemset(&istate, 0, sizeof(istate));\n+\tistate.always_verify_trailing_sha1 = 1;\n+\tistate.to_shm = 1;\n+\ti = read_index(&istate);\n+\tif (i != the_index.cache_nr)\n+\t\tgoto done;\n+\tfor (; i < the_index.cache_nr; i++) {\n+\t\tstruct cache_entry *base, *ce;\n+\t\t/* namelen is checked separately */\n+\t\tconst unsigned int ondisk_flags =\n+\t\t\tCE_STAGEMASK | CE_VALID | CE_EXTENDED_FLAGS;\n+\t\tunsigned int ce_flags, base_flags, ret;\n+\t\tbase = the_index.cache[i];\n+\t\tce = istate.cache[i];\n+\t\tif (ce->ce_namelen != base->ce_namelen ||\n+\t\t strcmp(ce->name, base->name)) {\n+\t\t\twarning(\"mismatch at entry %d\", i);\n+\t\t\tbreak;\n+\t\t}\n+\t\tce_flags = ce->ce_flags;\n+\t\tbase_flags = base->ce_flags;\n+\t\t/* only on-disk flags matter */\n+\t\tce->ce_flags &= ondisk_flags;\n+\t\tbase->ce_flags &= ondisk_flags;\n+\t\tret = memcmp(&ce->ce_stat_data, &base->ce_stat_data,\n+\t\t\t offsetof(struct cache_entry, name) -\n+\t\t\t offsetof(struct cache_entry, ce_stat_data));\n+\t\tce->ce_flags = ce_flags;\n+\t\tbase->ce_flags = base_flags;\n+\t\tif (ret) {\n+\t\t\twarning(\"mismatch at entry %d\", i);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+done:\n+\tdiscard_index(&istate);\n+\treturn i == the_index.cache_nr;\n+}\n+\n static void share_the_index(void)\n {\n \tif (the_index.split_index && the_index.split_index->base)\n \t\tshare_index(the_index.split_index->base, &shm_base_index);\n \tshare_index(&the_index, &shm_index);\n+\tif (to_verify && !verify_shm())\n+\t\tcleanup_shm();\n \tdiscard_index(&the_index);\n }\n \n@@ -130,6 +176,8 @@ int main(int argc, char **argv)\n \tstruct option options[] = {\n \t\tOPT_INTEGER(0, \"exit-after\", &idle_in_minutes,\n \t\t\t N_(\"exit if not used after some minutes\")),\n+\t\tOPT_BOOL(0, \"strict\", &to_verify,\n+\t\t\t \"verify shared memory after creating\"),\n \t\tOPT_END()\n \t};\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 7bd3ce4..1a0ab0c 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1660,9 +1660,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \n \tistate->mmap = mmap;\n \tistate->mmap_size = mmap_size;\n-\tif (try_shm(istate) &&\n-\t verify_hdr(istate->mmap, istate->mmap_size) < 0)\n-\t\tgoto unmap;\n+\tif (try_shm(istate)) {\n+\t\tif (verify_hdr(istate->mmap, istate->mmap_size) < 0)\n+\t\t\tgoto unmap;\n+\t} else if (istate->always_verify_trailing_sha1 &&\n+\t\t verify_hdr(istate->mmap, istate->mmap_size) < 0)\n+\t\t\tgoto unmap;\n \thdr = mmap = istate->mmap;\n \tmmap_size = istate->mmap_size;\n \tif (!istate->keep_mmap)\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280488","messageId":"1457548582-28302-8-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 07/19] daemonize(): set a flag before exiting the main process","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:10Z","receivedAt":"2016-03-09T18:36:10Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThis allows signal handlers and atexit functions to realize this\nsituation and not clean up.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/gc.c | 2 +-\n cache.h | 2 +-\n daemon.c | 2 +-\n setup.c | 4 +++-\n 4 files changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c583aad..37180de 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -385,7 +385,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t * failure to daemonize is ok, we'll continue\n \t\t\t * in foreground\n \t\t\t */\n-\t\t\tdaemonized = !daemonize();\n+\t\t\tdaemonized = !daemonize(NULL);\n \t\t}\n \t} else\n \t\tadd_repack_all_option();\ndiff --git a/cache.h b/cache.h\nindex 1d7e561..cc1f6f5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -530,7 +530,7 @@ extern int set_git_dir_init(const char *git_dir, const char *real_git_dir, int);\n extern int init_db(const char *template_dir, unsigned int flags);\n \n extern void sanitize_stdfds(void);\n-extern int daemonize(void);\n+extern int daemonize(int *);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \ndiff --git a/daemon.c b/daemon.c\nindex 8d45c33..a5cf954 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -1365,7 +1365,7 @@ int main(int argc, char **argv)\n \t\treturn execute();\n \n \tif (detach) {\n-\t\tif (daemonize())\n+\t\tif (daemonize(NULL))\n \t\t\tdie(\"--detach not supported on this platform\");\n \t} else\n \t\tsanitize_stdfds();\ndiff --git a/setup.c b/setup.c\nindex de1a2a7..9adf13f 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -1017,7 +1017,7 @@ void sanitize_stdfds(void)\n \t\tclose(fd);\n }\n \n-int daemonize(void)\n+int daemonize(int *daemonized)\n {\n #ifdef NO_POSIX_GOODIES\n \terrno = ENOSYS;\n@@ -1029,6 +1029,8 @@ int daemonize(void)\n \t\tcase -1:\n \t\t\tdie_errno(\"fork failed\");\n \t\tdefault:\n+\t\t\tif (daemonized)\n+\t\t\t\t*daemonized = 1;\n \t\t\texit(0);\n \t}\n \tif (setsid() == -1)\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280494","messageId":"1457548582-28302-9-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 08/19] index-helper: add --detach","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:11Z","receivedAt":"2016-03-09T18:36:11Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-index-helper.txt | 3 +++\n index-helper.c | 10 ++++++++--\n 2 files changed, 11 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex ad40366..9ced091 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -31,6 +31,9 @@ OPTIONS\n \tfor reading an index, but because it will happen in the\n \tbackground, it's not noticable. `--strict` is enabled by default.\n \n+--detach::\n+\tDetach from the shell.\n+\n NOTES\n -----\n On UNIX-like systems, $GIT_DIR/index-helper.pid contains the process\ndiff --git a/index-helper.c b/index-helper.c\nindex 1140bc0..4dd9656 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -14,7 +14,7 @@ struct shm {\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n-static int to_verify = 1;\n+static int daemonized, to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n {\n@@ -33,6 +33,8 @@ static void cleanup_shm(void)\n \n static void cleanup(void)\n {\n+\tif (daemonized)\n+\t\treturn;\n \tunlink(git_path(\"index-helper.pid\"));\n \tcleanup_shm();\n }\n@@ -172,12 +174,13 @@ int main(int argc, char **argv)\n \tstatic struct lock_file lock;\n \tstruct strbuf sb = STRBUF_INIT;\n \tconst char *prefix;\n-\tint fd, idle_in_minutes = 10;\n+\tint fd, idle_in_minutes = 10, detach = 0;\n \tstruct option options[] = {\n \t\tOPT_INTEGER(0, \"exit-after\", &idle_in_minutes,\n \t\t\t N_(\"exit if not used after some minutes\")),\n \t\tOPT_BOOL(0, \"strict\", &to_verify,\n \t\t\t \"verify shared memory after creating\"),\n+\t\tOPT_BOOL(0, \"detach\", &detach, \"detach the process\"),\n \t\tOPT_END()\n \t};\n \n@@ -202,6 +205,9 @@ int main(int argc, char **argv)\n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n \n+\tif (detach && daemonize(&daemonized))\n+\t\tdie_errno(\"unable to detach\");\n+\n \tif (!idle_in_minutes)\n \t\tidle_in_minutes = 0xffffffff / 60;\n \tloop(sb.buf, idle_in_minutes * 60);\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280491","messageId":"1457548582-28302-10-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 09/19] index-helper: add Windows support","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:12Z","receivedAt":"2016-03-09T18:36:12Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nWindows supports shared memory, but the semantics is a bit different\nthan POSIX shm. The most noticeable thing is there's no way to get the\nshared memory's size by the reader, and wrapping fstat to do that\nwould be hell. So the shm size is added near the end, hidden away from\nshm users (storing it in headers would cause more problems with munmap,\nstoring it as a separate shm is even worse).\n\nPostMessage is used instead of UNIX signals for\nnotification. Lightweight (at least code-wise) on the client side.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n config.mak.uname | 2 ++\n index-helper.c | 48 ++++++++++++++++++++++++++++\n read-cache.c | 13 ++++++++\n shm.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 159 insertions(+)\n\ndiff --git a/config.mak.uname b/config.mak.uname\nindex b5108e1..49320c7 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -394,6 +394,7 @@ ifndef DEBUG\n else\n \tBASIC_CFLAGS += -Zi -MDd\n endif\n+\tPROGRAM_OBJS += index-helper.o\n \tX = .exe\n endif\n ifeq ($(uname_S),Interix)\n@@ -574,6 +575,7 @@ else\n \t\tNO_CURL = YesPlease\n \tendif\n endif\n+\tPROGRAM_OBJS += index-helper.o\n endif\n ifeq ($(uname_S),QNX)\n \tCOMPAT_CFLAGS += -DSA_RESTART=0\ndiff --git a/index-helper.c b/index-helper.c\nindex 4dd9656..cf26da7 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -155,6 +155,51 @@ static void loop(const char *pid_file, int idle_in_seconds)\n \t\t; /* do nothing, all is handled by signal handlers already */\n }\n \n+#elif defined(GIT_WINDOWS_NATIVE)\n+\n+static void loop(const char *pid_file, int idle_in_seconds)\n+{\n+\tHWND hwnd;\n+\tUINT_PTR timer = 0;\n+\tMSG msg;\n+\tHINSTANCE hinst = GetModuleHandle(NULL);\n+\tWNDCLASS wc;\n+\n+\t/*\n+\t * Emulate UNIX signals by sending WM_USER+x to a\n+\t * window. Register window class and create a new window to\n+\t * catch these messages.\n+\t */\n+\tmemset(&wc, 0, sizeof(wc));\n+\twc.lpfnWndProc\t = DefWindowProc;\n+\twc.hInstance\t = hinst;\n+\twc.lpszClassName = \"git-index-helper\";\n+\tif (!RegisterClass(&wc))\n+\t\tdie_errno(_(\"could not register new window class\"));\n+\n+\thwnd = CreateWindow(\"git-index-helper\", pid_file,\n+\t\t\t 0, 0, 0, 1, 1, NULL, NULL, hinst, NULL);\n+\tif (!hwnd)\n+\t\tdie_errno(_(\"could not register new window\"));\n+\n+\trefresh(0);\n+\twhile (1) {\n+\t\ttimer = SetTimer(hwnd, timer, idle_in_seconds * 1000, NULL);\n+\t\tif (!timer)\n+\t\t\tdie(_(\"no timer!\"));\n+\t\tif (!GetMessage(&msg, hwnd, 0, 0) || msg.message == WM_TIMER)\n+\t\t\tbreak;\n+\t\tswitch (msg.message) {\n+\t\tcase WM_USER:\n+\t\t\trefresh(0);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* just reset the timer */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\n+\n #else\n \n static void loop(const char *pid_file, int idle_in_seconds)\n@@ -198,6 +243,9 @@ int main(int argc, char **argv)\n \tfd = hold_lock_file_for_update(&lock,\n \t\t\t\t git_path(\"index-helper.pid\"),\n \t\t\t\t LOCK_DIE_ON_ERROR);\n+#ifdef GIT_WINDOWS_NATIVE\n+\tstrbuf_addstr(&sb, \"HWND\");\n+#endif\n \tstrbuf_addf(&sb, \"%\" PRIuMAX, (uintmax_t) getpid());\n \twrite_in_full(fd, sb.buf, sb.len);\n \tcommit_lock_file(&lock);\ndiff --git a/read-cache.c b/read-cache.c\nindex 1a0ab0c..16fbdf6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1546,6 +1546,18 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_untracked_cache(istate);\n }\n \n+#if defined(GIT_WINDOWS_NATIVE)\n+static void do_poke(struct strbuf *sb, int refresh_cache)\n+{\n+\tHWND hwnd;\n+\tif (!starts_with(sb->buf, \"HWND\"))\n+\t\treturn;\n+\thwnd = FindWindow(\"git-index-helper\", sb->buf);\n+\tif (!hwnd)\n+\t\treturn;\n+\tPostMessage(hwnd, refresh_cache ? WM_USER : WM_USER + 1, 0, 0);\n+}\n+#else\n static void do_poke(struct strbuf *sb, int refresh_cache)\n {\n \tchar\t*start = sb->buf;\n@@ -1555,6 +1567,7 @@ static void do_poke(struct strbuf *sb, int refresh_cache)\n \t\treturn;\n \tkill(pid, refresh_cache ? SIGHUP : SIGUSR1);\n }\n+#endif\n \n static void poke_daemon(struct index_state *istate,\n \t\t\tconst struct stat *st, int refresh_cache)\ndiff --git a/shm.c b/shm.c\nindex 4ec1a00..04d8a35 100644\n--- a/shm.c\n+++ b/shm.c\n@@ -52,6 +52,102 @@ void git_shm_unlink(const char *fmt, ...)\n \tshm_unlink(path);\n }\n \n+#elif defined(GIT_WINDOWS_NATIVE)\n+\n+#define SHM_PATH_LEN 82\t/* a little bit longer than POSIX because of \"Local\\\\\" */\n+\n+static ssize_t create_shm_map(int oflag, int perm, ssize_t length,\n+\t\t\t void **mmap, int prot, int flags,\n+\t\t\t const char *path, unsigned long page_size)\n+{\n+\tsize_t real_length;\n+\tvoid *last_page;\n+\tHANDLE h;\n+\n+\tassert(perm == 0700);\n+\tassert(oflag == (O_CREAT | O_EXCL | O_RDWR));\n+\tassert(prot == (PROT_READ | PROT_WRITE));\n+\tassert(flags == MAP_SHARED);\n+\tassert(length >= 0);\n+\n+\treal_length = length;\n+\tif (real_length % page_size)\n+\t\treal_length += page_size - (real_length % page_size);\n+\treal_length += page_size;\n+\th = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0,\n+\t\t\t real_length, path);\n+\tif (!h)\n+\t\treturn -1;\n+\t*mmap = MapViewOfFile(h, FILE_MAP_ALL_ACCESS, 0, 0, real_length);\n+\tCloseHandle(h);\n+\tif (!*mmap)\n+\t\treturn -1;\n+\tlast_page = (unsigned char *)*mmap + real_length - page_size;\n+\t*(unsigned long *)last_page = length;\n+\treturn length;\n+}\n+\n+static ssize_t open_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t\t int prot, int flags, const char *path,\n+\t\t\t unsigned long page_size)\n+{\n+\tvoid *last_page;\n+\tHANDLE h;\n+\n+\tassert(perm == 0700);\n+\tassert(oflag == O_RDONLY);\n+\tassert(prot == PROT_READ);\n+\tassert(flags == MAP_SHARED);\n+\tassert(length <= 0);\n+\n+\th = OpenFileMapping(FILE_MAP_READ, FALSE, path);\n+\tif (!h)\n+\t\treturn -1;\n+\t*mmap = MapViewOfFile(h, FILE_MAP_READ, 0, 0, 0);\n+\tCloseHandle(h);\n+\tif (!*mmap)\n+\t\treturn -1;\n+\tif (length < 0) {\n+\t\tMEMORY_BASIC_INFORMATION mbi;\n+\t\tif (!VirtualQuery(*mmap, &mbi, sizeof(mbi))) {\n+\t\t\tUnmapViewOfFile(*mmap);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (mbi.RegionSize % page_size)\n+\t\t\tdie(\"expected size %lu to be %lu aligned\",\n+\t\t\t\t mbi.RegionSize, page_size);\n+\t\tlast_page = (unsigned char *)*mmap + mbi.RegionSize - page_size;\n+\t\tlength = *(unsigned long *)last_page;\n+\t}\n+\treturn length;\n+}\n+\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t int prot, int flags, const char *fmt, ...)\n+{\n+\tSYSTEM_INFO si;\n+\tva_list ap;\n+\tchar path[SHM_PATH_LEN];\n+\n+\tGetSystemInfo(&si);\n+\n+\tstrcpy(path, \"Local\\\\\");\n+\tva_start(ap, fmt);\n+\tvsprintf(path + strlen(path), fmt, ap);\n+\tva_end(ap);\n+\n+\tif (oflag & O_CREAT)\n+\t\treturn create_shm_map(oflag, perm, length, mmap, prot,\n+\t\t\t\t flags, path, si.dwPageSize);\n+\telse\n+\t\treturn open_shm_map(oflag, perm, length, mmap, prot,\n+\t\t\t\t flags, path, si.dwPageSize);\n+}\n+\n+void git_shm_unlink(const char *fmt, ...)\n+{\n+}\n+\n #else\n \n ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280490","messageId":"1457548582-28302-11-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 10/19] read-cache: add watchman 'WAMA' extension","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:13Z","receivedAt":"2016-03-09T18:36:13Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThe extension contains a bitmap, one bit for each entry in the\nindex. If the n-th bit is zero, the n-th entry is considered\nunchanged, we can ce_mark_uptodate() it without refreshing. If the bit\nis non-zero and we found out the corresponding file is clean after\nrefresh, we can clear the bit.\n\nThe 'skipping refresh' bit is not in this patch yet as we would need\nwatchman. More details in later patches.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 4 ++++\n read-cache.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 73 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex cc1f6f5..8f7b4b1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -182,6 +182,8 @@ struct cache_entry {\n #define CE_VALID (0x8000)\n #define CE_STAGESHIFT 12\n \n+#define CE_WATCHMAN_DIRTY (0x0001)\n+\n /*\n * Range 0xFFFF0FFF in ce_flags is divided into\n * two parts: in-memory flags and on-disk ones.\n@@ -320,6 +322,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n #define UNTRACKED_CHANGED\t(1 << 7)\n+#define WATCHMAN_CHANGED\t(1 << 8)\n \n struct split_index;\n struct untracked_cache;\n@@ -344,6 +347,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tvoid *mmap;\n \tsize_t mmap_size;\n+\tchar *last_update;\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 16fbdf6..85ef15b 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -19,6 +19,7 @@\n #include \"split-index.h\"\n #include \"utf8.h\"\n #include \"shm.h\"\n+#include \"ewah/ewok.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -41,11 +42,13 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n #define CACHE_EXT_UNTRACKED 0x554E5452\t /* \"UNTR\" */\n+#define CACHE_EXT_WATCHMAN 0x57414D41\t /* \"WAMA\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | \\\n+\t\t WATCHMAN_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -1224,8 +1227,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\tcontinue;\n \n \t\tnew = refresh_cache_ent(istate, ce, options, &cache_errno, &changed);\n-\t\tif (new == ce)\n+\t\tif (new == ce) {\n+\t\t\tif (ce->ce_flags & CE_WATCHMAN_DIRTY) {\n+\t\t\t\tce->ce_flags &= ~CE_WATCHMAN_DIRTY;\n+\t\t\t\tistate->cache_changed |= WATCHMAN_CHANGED;\n+\t\t\t}\n \t\t\tcontinue;\n+\t\t}\n \t\tif (!new) {\n \t\t\tconst char *fmt;\n \n@@ -1369,6 +1377,48 @@ static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n \treturn 0;\n }\n \n+static void mark_no_watchman(size_t pos, void *data)\n+{\n+\tstruct index_state *istate = data;\n+\tassert(pos < istate->cache_nr);\n+\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n+}\n+\n+static int read_watchman_ext(struct index_state *istate, const void *data,\n+\t\t\t unsigned long sz)\n+{\n+\tstruct ewah_bitmap *bitmap;\n+\tint ret, len;\n+\n+\tif (memchr(data, 0, sz) == NULL)\n+\t\treturn error(\"invalid extension\");\n+\tlen = strlen(data) + 1;\n+\tbitmap = ewah_new();\n+\tret = ewah_read_mmap(bitmap, (const char *)data + len, sz - len);\n+\tif (ret != sz - len) {\n+\t\tewah_free(bitmap);\n+\t\treturn error(\"failed to parse ewah bitmap reading watchman index extension\");\n+\t}\n+\tistate->last_update = xstrdup(data);\n+\tewah_each_bit(bitmap, mark_no_watchman, istate);\n+\tewah_free(bitmap);\n+\treturn 0;\n+}\n+\n+static void write_watchman_ext(struct strbuf *sb, struct index_state* istate)\n+{\n+\tstruct ewah_bitmap *bitmap;\n+\tint i;\n+\n+\tstrbuf_add(sb, istate->last_update, strlen(istate->last_update) + 1);\n+\tbitmap = ewah_new();\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHMAN_DIRTY)\n+\t\t\tewah_set(bitmap, i);\n+\tewah_serialize_strbuf(bitmap, sb);\n+\tewah_free(bitmap);\n+}\n+\n static int read_index_extension(struct index_state *istate,\n \t\t\t\tconst char *ext, void *data, unsigned long sz)\n {\n@@ -1386,6 +1436,11 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_UNTRACKED:\n \t\tistate->untracked = read_untracked_extension(data, sz);\n \t\tbreak;\n+\n+\tcase CACHE_EXT_WATCHMAN:\n+\t\tread_watchman_ext(istate, data, sz);\n+\t\tbreak;\n+\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1817,6 +1872,8 @@ int discard_index(struct index_state *istate)\n \tistate->untracked = NULL;\n \tistate->from_shm = 0;\n \tistate->to_shm = 0;\n+\tfree(istate->last_update);\n+\tistate->last_update = NULL;\n \treturn 0;\n }\n \n@@ -2214,6 +2271,16 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->last_update) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_watchman_ext(&sb, istate);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCHMAN, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st))\n \t\treturn -1;\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280495","messageId":"1457548582-28302-12-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 11/19] Add watchman support to reduce index refresh cost","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:14Z","receivedAt":"2016-03-09T18:36:14Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nThe previous patch has the logic to clear bits in 'WAMA' bitmap. This\npatch has logic to set bits as told by watchman. The missing bit,\n_using_ these bits, are not here yet.\n\nA lot of this code is written by David Turner originally, mostly from\n[1]. I'm just copying and polishing it a bit.\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/248006\n\nSigned-off-by: David Turner \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Makefile | 7 ++++\n cache.h | 1 +\n config.c | 5 +++\n configure.ac | 8 ++++\n environment.c | 3 ++\n watchman-support.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n watchman-support.h | 7 ++++\n 7 files changed, 146 insertions(+)\n create mode 100644 watchman-support.c\n create mode 100644 watchman-support.h\n\ndiff --git a/Makefile b/Makefile\nindex a6c668b..e51331c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1416,6 +1416,12 @@ else\n \tLIB_OBJS += thread-utils.o\n endif\n \n+ifdef USE_WATCHMAN\n+\tLIB_H += watchman-support.h\n+\tLIB_OBJS += watchman-support.o\n+\tBASIC_CFLAGS += -DUSE_WATCHMAN\n+endif\n+\n ifdef HAVE_PATHS_H\n \tBASIC_CFLAGS += -DHAVE_PATHS_H\n endif\n@@ -2164,6 +2170,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n+\t@echo USE_WATCHMAN=\\''$(subst ','\\'',$(subst ','\\'',$(USE_WATCHMAN)))'\\' >>$@+\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/cache.h b/cache.h\nindex 8f7b4b1..bf20652 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -688,6 +688,7 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n+extern int core_watchman_sync_timeout;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex 9ba40bc..e6dc141 100644\n--- a/config.c\n+++ b/config.c\n@@ -882,6 +882,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.watchmansynctimeout\")) {\n+\t\tcore_watchman_sync_timeout = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.createobject\")) {\n \t\tif (!strcmp(value, \"rename\"))\n \t\t\tobject_creation_mode = OBJECT_CREATION_USES_RENAMES;\ndiff --git a/configure.ac b/configure.ac\nindex 89e2590..6f10a15 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -1092,6 +1092,14 @@ AC_COMPILE_IFELSE([BSD_SYSCTL_SRC],\n \tHAVE_BSD_SYSCTL=])\n GIT_CONF_SUBST([HAVE_BSD_SYSCTL])\n \n+#\n+# Check for watchman client library\n+\n+AC_CHECK_LIB([watchman], [watchman_connect],\n+\t[USE_WATCHMAN=YesPlease],\n+\t[USE_WATCHMAN=])\n+GIT_CONF_SUBST([USE_WATCHMAN])\n+\n ## Other checks.\n # Define USE_PIC if you need the main git objects to be built with -fPIC\n # in order to build and link perl/Git.so. x86-64 seems to need this.\ndiff --git a/environment.c b/environment.c\nindex 6dec9d0..35e03c7 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -94,6 +94,9 @@ int core_preload_index = 1;\n */\n int ignore_untracked_cache_config;\n \n+int core_watchman_sync_timeout = 300;\n+\n+\n /* This is set by setup_git_dir_gently() and/or git_default_config() */\n char *git_work_tree_cfg;\n static char *work_tree;\ndiff --git a/watchman-support.c b/watchman-support.c\nnew file mode 100644\nindex 0000000..08e37ae\n--- /dev/null\n+++ b/watchman-support.c\n@@ -0,0 +1,115 @@\n+#include \"cache.h\"\n+#include \"watchman-support.h\"\n+#include \"strbuf.h\"\n+#include \n+\n+static struct watchman_query *make_query(const char *last_update)\n+{\n+\tstruct watchman_query *query = watchman_query();\n+\twatchman_query_set_fields(query, WATCHMAN_FIELD_NAME |\n+\t\t\t\t\t WATCHMAN_FIELD_EXISTS |\n+\t\t\t\t\t WATCHMAN_FIELD_NEWER);\n+\twatchman_query_set_empty_on_fresh(query, 1);\n+\tquery->sync_timeout = core_watchman_sync_timeout;\n+\tif (*last_update)\n+\t\twatchman_query_set_since_oclock(query, last_update);\n+\treturn query;\n+}\n+\n+static struct watchman_query_result* query_watchman(\n+\tstruct index_state *istate, struct watchman_connection *connection,\n+\tconst char *fs_path, const char *last_update)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_query *query;\n+\tstruct watchman_expression *expr;\n+\tstruct watchman_query_result *result;\n+\n+\tquery = make_query(last_update);\n+\texpr = watchman_true_expression();\n+\tresult = watchman_do_query(connection, fs_path, query, expr, &wm_error);\n+\twatchman_free_query(query);\n+\twatchman_free_expression(expr);\n+\n+\tif (!result)\n+\t\twarning(\"Watchman query error: %s (at %s)\",\n+\t\t\twm_error.message,\n+\t\t\t*last_update ? last_update : \"the beginning\");\n+\n+\treturn result;\n+}\n+\n+static void update_index(struct index_state *istate,\n+\t\t\t struct watchman_query_result *result)\n+{\n+\tint i;\n+\n+\tif (result->is_fresh_instance) {\n+\t\t/* let refresh clear them later */\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tistate->cache[i]->ce_flags |= CE_WATCHMAN_DIRTY;\n+\t\tgoto done;\n+\t}\n+\n+\tfor (i = 0; i < result->nr; i++) {\n+\t\tstruct watchman_stat *wm = result->stats + i;\n+\t\tint pos;\n+\n+\t\tif (!strncmp(wm->name, \".git/\", 5) ||\n+\t\t strstr(wm->name, \"/.git/\"))\n+\t\t\tcontinue;\n+\n+\t\tpos = index_name_pos(istate, wm->name, strlen(wm->name));\n+\t\tif (pos < 0)\n+\t\t\tcontinue;\n+\t\t/* FIXME: ignore staged entries and gitlinks too? */\n+\n+\t\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n+\t}\n+\n+done:\n+\tfree(istate->last_update);\n+\tistate->last_update = xstrdup(result->clock);\n+\tistate->cache_changed |= WATCHMAN_CHANGED;\n+}\n+\n+int check_watchman(struct index_state *istate)\n+{\n+\tstruct watchman_error wm_error;\n+\tstruct watchman_connection *connection;\n+\tstruct watchman_query_result *result;\n+\tconst char *fs_path;\n+\tstruct timeval timeout;\n+\t/*\n+\t * Convert core_watchman_sync_timeout, in milliseconds, to\n+\t * struct timeval, in seconds and microseconds.\n+\t */\n+\n+\tfs_path = get_git_work_tree();\n+\tif (!fs_path)\n+\t\treturn -1;\n+\n+\ttimeout.tv_sec = core_watchman_sync_timeout / 1000;\n+\ttimeout.tv_usec = (core_watchman_sync_timeout % 1000) * 1000;\n+\tconnection = watchman_connect(timeout, &wm_error);\n+\n+\tif (!connection) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\treturn -1;\n+\t}\n+\n+\tif (watchman_watch(connection, fs_path, &wm_error)) {\n+\t\twarning(\"Watchman watch error: %s\", wm_error.message);\n+\t\twatchman_connection_close(connection);\n+\t\treturn -1;\n+\t}\n+\n+\n+\tresult = query_watchman(istate, connection, fs_path, istate->last_update);\n+\twatchman_connection_close(connection);\n+\tif (!result)\n+\t\treturn -1;\n+\tupdate_index(istate, result);\n+\twatchman_free_query_result(result);\n+\treturn 0;\n+}\ndiff --git a/watchman-support.h b/watchman-support.h\nnew file mode 100644\nindex 0000000..ee1ef2c\n--- /dev/null\n+++ b/watchman-support.h\n@@ -0,0 +1,7 @@\n+#ifndef WATCHMAN_SUPPORT_H\n+#define WATCHMAN_SUPPORT_H\n+\n+struct index_state;\n+int check_watchman(struct index_state *index);\n+\n+#endif /* WATCHMAN_SUPPORT_H */\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280492","messageId":"1457548582-28302-13-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 12/19] read-cache: allow index-helper to prepare shm before git reads it","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:15Z","receivedAt":"2016-03-09T18:36:15Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nIf index-helper puts 'W' before pid in $GIT_DIR/index-helper.pid, then\ngit will sleep for a while, expecting to be waken up by SIGUSR1 when\nindex-helper has done shm preparation, or after the timeout.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: David Turner \n---\n read-cache.c | 36 ++++++++++++++++++++++++++++++++++--\n 1 file changed, 34 insertions(+), 2 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 85ef15b..57c5df9 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1613,14 +1613,46 @@ static void do_poke(struct strbuf *sb, int refresh_cache)\n \tPostMessage(hwnd, refresh_cache ? WM_USER : WM_USER + 1, 0, 0);\n }\n #else\n+\n+static volatile int done_sleeping;\n+\n+static void mark_done_sleeping(int sig)\n+{\n+\tdone_sleeping = 1;\n+}\n+\n+/*\n+ * Send a message to the index-helper to let it know that we're going\n+ * to read the index. If refresh_cache is true, then the index-helper\n+ * should re-read the index; otherwise, it should just stay alive.\n+ *\n+ * If the index-helper supports watchman, it will refresh the index\n+ * before it hands it over. Wait up to one second for a response\n+ * indicating that the index has been successfully refreshed.\n+ *\n+ */\n static void do_poke(struct strbuf *sb, int refresh_cache)\n {\n-\tchar\t*start = sb->buf;\n+\tint\t wait = sb->buf[0] == 'W';\n+\tchar\t*start = wait ? sb->buf + 1 : sb->buf;\n \tchar\t*end = NULL;\n \tpid_t\t pid = strtoul(start, &end, 10);\n+\tint\t ret;\n+\tint\t count = 0;\n+\n+\tdone_sleeping = 0;\n \tif (!end || end != sb->buf + sb->len)\n \t\treturn;\n-\tkill(pid, refresh_cache ? SIGHUP : SIGUSR1);\n+\tif (!refresh_cache && wait)\n+\t\tsignal(SIGHUP, mark_done_sleeping);\n+\tret = kill(pid, refresh_cache ? SIGHUP : SIGUSR1);\n+\tif (!refresh_cache && wait) {\n+\t\tif (!ret)\n+\t\t\twhile (!done_sleeping && count++ < 1000)\n+\t\t\t\tsleep_millisec(1);\n+\n+\t\tsigaction(SIGHUP, NULL, NULL);\n+\t}\n }\n #endif\n \n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280496","messageId":"1457548582-28302-14-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 13/19] index-helper: use watchman to avoid refreshing index with lstat()","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:16Z","receivedAt":"2016-03-09T18:36:16Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nWatchman is hidden behind index-helper. Before git tries to read the\nindex from shm, it notifies index-helper with SIGHUP and sleep,\nwaiting for index-helper to prepare shm. index-helper then contacts\nwatchman, updates 'WAMA' extension and put it in a separate shm and\nwakes git up with SIGHUP.\n\nGit uses this extension to not lstat unchanged entries. Git only trust\n'WAMA' extension when it's received from the separate shm, not from\ndisk. Unmarked entries are \"clean\". Marked entries are dirty from\nwatchman point of view. If it finds out some entries are\n'watchman-dirty', but are really unchanged (e.g. the file was changed,\nthen reverted back), then Git will clear the marking in 'WAMA' before\nwriting it down.\n\nHiding watchman behind index-helper means you need both daemons. You\ncan't run watchman alone. Not so good. But on the other hand, 'git'\nbinary is not linked to watchman/json libraries, which is good for\npackaging. Core git package will run fine without watchman-related\npackages. If they need watchman, they can install git-index-helper and\ndependencies.\n\nAnother reason for tying watchman to index-helper is, when used with\nuntracked cache, we need to keep track of $GIT_WORK_TREE file\nlisting. That kind of list can be kept in index-helper.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Makefile | 5 ++++\n cache.h | 2 ++\n index-helper.c | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n read-cache.c | 43 +++++++++++++++++++++++++++---\n 4 files changed, 127 insertions(+), 7 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex e51331c..d79fc0c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -450,6 +450,7 @@ MSGFMT = msgfmt\n CURL_CONFIG = curl-config\n PTHREAD_LIBS = -lpthread\n PTHREAD_CFLAGS =\n+WATCHMAN_LIBS =\n GCOV = gcov\n \n export TCL_PATH TCLTK_PATH\n@@ -1419,6 +1420,7 @@ endif\n ifdef USE_WATCHMAN\n \tLIB_H += watchman-support.h\n \tLIB_OBJS += watchman-support.o\n+\tWATCHMAN_LIBS = -lwatchman\n \tBASIC_CFLAGS += -DUSE_WATCHMAN\n endif\n \n@@ -2032,6 +2034,9 @@ git-remote-testsvn$X: remote-testsvn.o GIT-LDFLAGS $(GITLIBS) $(VCSSVN_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) \\\n \t$(VCSSVN_LIB)\n \n+git-index-helper$X: index-helper.o GIT-LDFLAGS $(GITLIBS)\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) $(WATCHMAN_LIBS)\n+\n $(REMOTE_CURL_ALIASES): $(REMOTE_CURL_PRIMARY)\n \t$(QUIET_LNCP)$(RM) $@ && \\\n \tln $< $@ 2>/dev/null || \\\ndiff --git a/cache.h b/cache.h\nindex bf20652..272c928 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -558,6 +558,7 @@ extern int daemonize(int *);\n \n /* Initialize and use the cache information */\n struct lock_file;\n+extern int verify_index(const struct index_state *);\n extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);\n extern int do_read_index(struct index_state *istate, const char *path,\n@@ -565,6 +566,7 @@ extern int do_read_index(struct index_state *istate, const char *path,\n extern int read_index_from(struct index_state *, const char *path);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n+extern void write_watchman_ext(struct strbuf *sb, struct index_state* istate);\n #define COMMIT_LOCK\t\t(1 << 0)\n #define CLOSE_LOCK\t\t(1 << 1)\n #define REFRESH_DAEMON\t\t(1 << 2)\ndiff --git a/index-helper.c b/index-helper.c\nindex cf26da7..7e7ce9b 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -5,15 +5,18 @@\n #include \"split-index.h\"\n #include \"shm.h\"\n #include \"lockfile.h\"\n+#include \"watchman-support.h\"\n \n struct shm {\n \tunsigned char sha1[20];\n \tvoid *shm;\n \tsize_t size;\n+\tpid_t pid;\n };\n \n static struct shm shm_index;\n static struct shm shm_base_index;\n+static struct shm shm_watchman;\n static int daemonized, to_verify = 1;\n \n static void release_index_shm(struct shm *is)\n@@ -25,10 +28,21 @@ static void release_index_shm(struct shm *is)\n \tis->shm = NULL;\n }\n \n+static void release_watchman_shm(struct shm *is)\n+{\n+\tif (!is->shm)\n+\t\treturn;\n+\tmunmap(is->shm, is->size);\n+\tgit_shm_unlink(\"git-watchman-%s-%\" PRIuMAX,\n+\t\t sha1_to_hex(is->sha1), (uintmax_t)is->pid);\n+\tis->shm = NULL;\n+}\n+\n static void cleanup_shm(void)\n {\n \trelease_index_shm(&shm_index);\n \trelease_index_shm(&shm_base_index);\n+\trelease_watchman_shm(&shm_watchman);\n }\n \n static void cleanup(void)\n@@ -120,13 +134,15 @@ static void share_the_index(void)\n \tif (the_index.split_index && the_index.split_index->base)\n \t\tshare_index(the_index.split_index->base, &shm_base_index);\n \tshare_index(&the_index, &shm_index);\n-\tif (to_verify && !verify_shm())\n+\tif (to_verify && !verify_shm()) {\n \t\tcleanup_shm();\n-\tdiscard_index(&the_index);\n+\t\tdiscard_index(&the_index);\n+\t}\n }\n \n static void refresh(int sig)\n {\n+\tdiscard_index(&the_index);\n \tthe_index.keep_mmap = 1;\n \tthe_index.to_shm = 1;\n \tif (read_cache() < 0)\n@@ -136,7 +152,55 @@ static void refresh(int sig)\n \n #ifdef HAVE_SHM\n \n-static void do_nothing(int sig)\n+#ifdef USE_WATCHMAN\n+static void share_watchman(struct index_state *istate,\n+\t\t\t struct shm *is, pid_t pid)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tvoid *shm;\n+\n+\twrite_watchman_ext(&sb, istate);\n+\tif (git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, sb.len + 20,\n+\t\t\t&shm, PROT_READ | PROT_WRITE, MAP_SHARED,\n+\t\t\t\"git-watchman-%s-%\" PRIuMAX,\n+\t\t\tsha1_to_hex(istate->sha1), (uintmax_t)pid) == sb.len + 20) {\n+\t\tis->size = sb.len + 20;\n+\t\tis->shm = shm;\n+\t\tis->pid = pid;\n+\t\thashcpy(is->sha1, istate->sha1);\n+\n+\t\tmemcpy(shm, sb.buf, sb.len);\n+\t\thashcpy((unsigned char *)shm + is->size - 20, is->sha1);\n+\t}\n+\tstrbuf_release(&sb);\n+}\n+\n+static void prepare_with_watchman(pid_t pid)\n+{\n+\t/*\n+\t * with the help of watchman, maybe we could detect if\n+\t * $GIT_DIR/index is updated..\n+\t */\n+\tif (!verify_index(&the_index))\n+\t\trefresh(0);\n+\n+\tif (check_watchman(&the_index))\n+\t\treturn;\n+\n+\tshare_watchman(&the_index, &shm_watchman, pid);\n+}\n+\n+static void prepare_index(int sig, siginfo_t *si, void *context)\n+{\n+\trelease_watchman_shm(&shm_watchman);\n+\tif (the_index.last_update)\n+\t\tprepare_with_watchman(si->si_pid);\n+\tkill(si->si_pid, SIGHUP); /* stop the waiting in poke_daemon() */\n+}\n+\n+#else\n+\n+static void prepare_index(int sig, siginfo_t *si, void *context)\n {\n \t/*\n \t * what we need is the signal received and interrupts\n@@ -145,11 +209,21 @@ static void do_nothing(int sig)\n \t */\n }\n \n+#endif\n+\n static void loop(const char *pid_file, int idle_in_seconds)\n {\n+\tstruct sigaction sa;\n+\n \tsigchain_pop(SIGHUP);\t/* pushed by sigchain_push_common */\n \tsigchain_push(SIGHUP, refresh);\n-\tsigchain_push(SIGUSR1, do_nothing);\n+\n+\tmemset(&sa, 0, sizeof(sa));\n+\tsa.sa_sigaction = prepare_index;\n+\tsigemptyset(&sa.sa_mask);\n+\tsa.sa_flags = SA_SIGINFO;\n+\tsigaction(SIGUSR1, &sa, NULL);\n+\n \trefresh(0);\n \twhile (sleep(idle_in_seconds))\n \t\t; /* do nothing, all is handled by signal handlers already */\n@@ -245,6 +319,8 @@ int main(int argc, char **argv)\n \t\t\t\t LOCK_DIE_ON_ERROR);\n #ifdef GIT_WINDOWS_NATIVE\n \tstrbuf_addstr(&sb, \"HWND\");\n+#elif defined(USE_WATCHMAN)\n+\tstrbuf_addch(&sb, 'W');\t/* see poke_daemon() */\n #endif\n \tstrbuf_addf(&sb, \"%\" PRIuMAX, (uintmax_t) getpid());\n \twrite_in_full(fd, sb.buf, sb.len);\ndiff --git a/read-cache.c b/read-cache.c\nindex 57c5df9..78f5f0e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1405,7 +1405,7 @@ static int read_watchman_ext(struct index_state *istate, const void *data,\n \treturn 0;\n }\n \n-static void write_watchman_ext(struct strbuf *sb, struct index_state* istate)\n+void write_watchman_ext(struct strbuf *sb, struct index_state* istate)\n {\n \tstruct ewah_bitmap *bitmap;\n \tint i;\n@@ -1722,6 +1722,39 @@ static int try_shm(struct index_state *istate)\n \treturn 0;\n }\n \n+static void refresh_by_watchman(struct index_state *istate)\n+{\n+\tvoid *shm = NULL;\n+\tint length;\n+\tint i;\n+\n+\tlength = git_shm_map(O_RDONLY, 0700, -1, &shm,\n+\t\t\t PROT_READ, MAP_SHARED,\n+\t\t\t \"git-watchman-%s-%\" PRIuMAX,\n+\t\t\t sha1_to_hex(istate->sha1),\n+\t\t\t (uintmax_t)getpid());\n+\n+\tif (length <= 20 ||\n+\t hashcmp(istate->sha1, (unsigned char *)shm + length - 20) ||\n+\t /*\n+\t * No need to clear CE_WATCHMAN_DIRTY set by 'WAMA' on\n+\t * disk. Watchman can only set more, not clear any, so\n+\t * this is OR mask.\n+\t */\n+\t read_watchman_ext(istate, shm, length - 20))\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\tif (ce_stage(ce) || (ce->ce_flags & CE_WATCHMAN_DIRTY))\n+\t\t\tcontinue;\n+\t\tce_mark_uptodate(ce);\n+\t}\n+done:\n+\tif (shm)\n+\t\tmunmap(shm, length);\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1842,7 +1875,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tsplit_index = istate->split_index;\n \tif (!split_index || is_null_sha1(split_index->base_sha1)) {\n \t\tpost_read_index_from(istate);\n-\t\treturn ret;\n+\t\tgoto done;\n \t}\n \n \tif (split_index->base)\n@@ -1863,6 +1896,10 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\t sha1_to_hex(split_index->base->sha1));\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n+\n+done:\n+\tif (ret > 0 && istate->from_shm && istate->last_update)\n+\t\trefresh_by_watchman(istate);\n \treturn ret;\n }\n \n@@ -2164,7 +2201,7 @@ out:\n \treturn 0;\n }\n \n-static int verify_index(const struct index_state *istate)\n+int verify_index(const struct index_state *istate)\n {\n \treturn verify_index_from(istate, get_index_file());\n }\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280493","messageId":"1457548582-28302-15-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 14/19] update-index: enable/disable watchman support","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:17Z","receivedAt":"2016-03-09T18:36:17Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"From: Nguyễn Thái Ngọc Duy \n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/update-index.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 1c94ca5..7c08e1c 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -914,6 +914,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n {\n \tint newfd, entries, has_errors = 0, nul_term_line = 0;\n \tenum uc_mode untracked_cache = UC_UNSPECIFIED;\n+\tint use_watchman = -1;\n \tint read_from_stdin = 0;\n \tint prefix_length = prefix ? strlen(prefix) : 0;\n \tint preferred_index_format = 0;\n@@ -1012,6 +1013,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"test if the filesystem supports untracked cache\"), UC_TEST),\n \t\tOPT_SET_INT(0, \"force-untracked-cache\", &untracked_cache,\n \t\t\t N_(\"enable untracked cache without testing the filesystem\"), UC_FORCE),\n+\t\tOPT_BOOL(0, \"watchman\", &use_watchman,\n+\t\t\tN_(\"use or not use watchman to reduce refresh cost\")),\n \t\tOPT_END()\n \t};\n \n@@ -1149,6 +1152,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tdie(\"Bug: bad untracked_cache value: %d\", untracked_cache);\n \t}\n \n+\tif (use_watchman > 0) {\n+\t\tthe_index.last_update = xstrdup(\"\");\n+\t\tthe_index.cache_changed |= WATCHMAN_CHANGED;\n+\t} else if (!use_watchman) {\n+\t\tthe_index.last_update = NULL;\n+\t\tthe_index.cache_changed |= WATCHMAN_CHANGED;\n+\t}\n+\n \tif (active_cache_changed) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280499","messageId":"1457548582-28302-16-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 15/19] unpack-trees: preserve index extensions","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:18Z","receivedAt":"2016-03-09T18:36:18Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Make git checkout (and other unpack_tree operations) preserve the\nuntracked cache and watchman status. This is valuable for two reasons:\n\n1. Often, an unpack_tree operation will not touch large parts of the\nworking tree, and thus most of the untracked cache will continue to be\nvalid.\n\n2. Even if the untracked cache were entirely invalidated by such an\noperation, the user has signaled their intention to have such a cache,\nand we don't want to throw it away.\n\nThe same logic applies to the watchman state.\n\nSigned-off-by: David Turner \n---\n cache.h | 1 +\n read-cache.c | 8 ++++++++\n t/t7063-status-untracked-cache.sh | 23 +++++++++++++++++++++++\n t/test-lib-functions.sh | 4 ++++\n unpack-trees.c | 1 +\n 5 files changed, 37 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 272c928..03c34f1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -572,6 +572,7 @@ extern void write_watchman_ext(struct strbuf *sb, struct index_state* istate);\n #define REFRESH_DAEMON\t\t(1 << 2)\n extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\n extern int discard_index(struct index_state *);\n+extern void move_index_extensions(struct index_state *dst, struct index_state *src);\n extern int unmerged_index(const struct index_state *);\n extern int verify_path(const char *path);\n extern int index_dir_exists(struct index_state *istate, const char *name, int namelen);\ndiff --git a/read-cache.c b/read-cache.c\nindex 78f5f0e..40d789a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2591,3 +2591,11 @@ void report_index_stats(struct trace_key *key)\n \t\t\t \"index stats: file writes = %10u\\n\",\n \t\t\t nr_read_index, nr_read_shm_index, nr_write_index);\n }\n+\n+void move_index_extensions(struct index_state *dst, struct index_state *src)\n+{\n+\tdst->untracked = src->untracked;\n+\tsrc->untracked = NULL;\n+\tdst->last_update = src->last_update;\n+\tsrc->last_update = NULL;\n+}\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex a971884..a2c8535 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -646,4 +646,27 @@ test_expect_success 'test ident field is working' '\n \ttest_cmp ../expect ../err\n '\n \n+test_expect_success 'untracked cache survives a checkout' '\n+\tgit commit --allow-empty -m empty &&\n+\ttest-dump-untracked-cache >../before &&\n+\ttest_when_finished \"git checkout master\" &&\n+\tgit checkout -b other_branch &&\n+\ttest-dump-untracked-cache >../after &&\n+\ttest_cmp ../before ../after &&\n+\ttest_commit test &&\n+\ttest-dump-untracked-cache >../before &&\n+\tgit checkout master &&\n+\ttest-dump-untracked-cache >../after &&\n+\ttest_cmp ../before ../after\n+'\n+\n+\n+test_expect_success 'untracked cache survives a commit' '\n+\ttest-dump-untracked-cache >../before &&\n+\tgit add done/two &&\n+\tgit commit -m commit &&\n+\ttest-dump-untracked-cache >../after &&\n+\ttest_cmp ../before ../after\n+'\n+\n test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 8d99eb3..e974b5b 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -186,6 +186,10 @@ test_commit () {\n \t\ttest_tick\n \tfi &&\n \tgit commit $signoff -m \"$1\" &&\n+\tif [ \"$(git config core.bare)\" = false ]\n+\tthen\n+\t git update-index --force-untracked-cache\n+\tfi\n \tgit tag \"${4:-$1}\"\n }\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 9f55cc2..fc90eb3 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1215,6 +1215,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t\t\t\t\t WRITE_TREE_SILENT |\n \t\t\t\t\t\t WRITE_TREE_REPAIR);\n \t\t}\n+\t\tmove_index_extensions(&o->result, o->dst_index);\n \t\tdiscard_index(o->dst_index);\n \t\t*o->dst_index = o->result;\n \t} else {\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280498","messageId":"1457548582-28302-17-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 16/19] index-helper: rewrite pidfile after daemonizing","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:19Z","receivedAt":"2016-03-09T18:36:19Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"When we daemonize, our pid changes, so we need to rewrite the pid file.\n\nSigned-off-by: David Turner \n---\n index-helper.c | 48 ++++++++++++++++++++++++++++++++----------------\n 1 file changed, 32 insertions(+), 16 deletions(-)\n\ndiff --git a/index-helper.c b/index-helper.c\nindex 7e7ce9b..dc03e1e 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -288,12 +288,33 @@ static const char * const usage_text[] = {\n \tNULL\n };\n \n-int main(int argc, char **argv)\n+static const char *write_pid(void)\n {\n+\tstatic struct strbuf sb = STRBUF_INIT;\n \tstatic struct lock_file lock;\n-\tstruct strbuf sb = STRBUF_INIT;\n+\tint fd;\n+\n+\tstrbuf_reset(&sb);\n+\tfd = hold_lock_file_for_update(&lock,\n+\t\t\t\t git_path(\"index-helper.pid\"),\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n+#ifdef GIT_WINDOWS_NATIVE\n+\tstrbuf_addstr(&sb, \"HWND\");\n+#elif defined(USE_WATCHMAN)\n+\tstrbuf_addch(&sb, 'W');\t/* see poke_daemon() */\n+#endif\n+\tstrbuf_addf(&sb, \"%\" PRIuMAX, (uintmax_t) getpid());\n+\twrite_in_full(fd, sb.buf, sb.len);\n+\tcommit_lock_file(&lock);\n+\n+\treturn sb.buf;\n+}\n+\n+int main(int argc, char **argv)\n+{\n \tconst char *prefix;\n-\tint fd, idle_in_minutes = 10, detach = 0;\n+\tint idle_in_minutes = 10, detach = 0;\n+\tconst char *pid_file;\n \tstruct option options[] = {\n \t\tOPT_INTEGER(0, \"exit-after\", &idle_in_minutes,\n \t\t\t N_(\"exit if not used after some minutes\")),\n@@ -314,17 +335,7 @@ int main(int argc, char **argv)\n \t\t\t options, usage_text, 0))\n \t\tdie(_(\"too many arguments\"));\n \n-\tfd = hold_lock_file_for_update(&lock,\n-\t\t\t\t git_path(\"index-helper.pid\"),\n-\t\t\t\t LOCK_DIE_ON_ERROR);\n-#ifdef GIT_WINDOWS_NATIVE\n-\tstrbuf_addstr(&sb, \"HWND\");\n-#elif defined(USE_WATCHMAN)\n-\tstrbuf_addch(&sb, 'W');\t/* see poke_daemon() */\n-#endif\n-\tstrbuf_addf(&sb, \"%\" PRIuMAX, (uintmax_t) getpid());\n-\twrite_in_full(fd, sb.buf, sb.len);\n-\tcommit_lock_file(&lock);\n+\twrite_pid();\n \n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n@@ -332,9 +343,14 @@ int main(int argc, char **argv)\n \tif (detach && daemonize(&daemonized))\n \t\tdie_errno(\"unable to detach\");\n \n+\t/*\n+\t * Now that we're daemonized, we need to rewrite the PID file\n+\t * because we have a new PID.\n+\t */\n+\tpid_file = write_pid();\n+\n \tif (!idle_in_minutes)\n \t\tidle_in_minutes = 0xffffffff / 60;\n-\tloop(sb.buf, idle_in_minutes * 60);\n-\tstrbuf_release(&sb);\n+\tloop(pid_file, idle_in_minutes * 60);\n \treturn 0;\n }\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280497","messageId":"1457548582-28302-18-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 17/19] index-helper: process management","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:20Z","receivedAt":"2016-03-09T18:36:20Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Don't start index-helper when it is already running for a repository.\n\nKeep the index-helper pid file's mtime up-to-date. Before starting\nindex-helper, check that (a) the pid file's mtime is recent and (b)\nthere is some process with that pid. Of course, it's possible that\nthe mtime is recent but the index-helper has died anyway (and some other\nprocess has taken its pid), but that's pretty unlikely.\n\nProvide index-helper --kill (to kill the running index-helper) and\nindex-helper --ignore-existing (to start up a new index-helper even\nif we believe that there's one already running).\n\nAdd a test for index-helper's pid-file writing and for --kill.\n\nSigned-off-by: David Turner \n---\n index-helper.c | 95 +++++++++++++++++++++++++++++++++++++++++++++----\n t/t7900-index-helper.sh | 23 ++++++++++++\n 2 files changed, 111 insertions(+), 7 deletions(-)\n create mode 100755 t/t7900-index-helper.sh\n\ndiff --git a/index-helper.c b/index-helper.c\nindex dc03e1e..a75da60 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -152,6 +152,17 @@ static void refresh(int sig)\n \n #ifdef HAVE_SHM\n \n+static void touch_pid_file(void)\n+{\n+\t/*\n+\t * If we fail to update the times on index-helper.pid, we've\n+\t * probably had the repo deleted out from under us or otherwise\n+\t * lost access; might as well give up.\n+\t */\n+\tif (utimes(git_path(\"index-helper.pid\"), NULL))\n+\t\texit(0);\n+}\n+\n #ifdef USE_WATCHMAN\n static void share_watchman(struct index_state *istate,\n \t\t\t struct shm *is, pid_t pid)\n@@ -211,9 +222,10 @@ static void prepare_index(int sig, siginfo_t *si, void *context)\n \n #endif\n \n-static void loop(const char *pid_file, int idle_in_seconds)\n+static void loop(const char *pid_file, int idle_in_minutes)\n {\n \tstruct sigaction sa;\n+\tint timeout_count = 0;\n \n \tsigchain_pop(SIGHUP);\t/* pushed by sigchain_push_common */\n \tsigchain_push(SIGHUP, refresh);\n@@ -225,19 +237,25 @@ static void loop(const char *pid_file, int idle_in_seconds)\n \tsigaction(SIGUSR1, &sa, NULL);\n \n \trefresh(0);\n-\twhile (sleep(idle_in_seconds))\n-\t\t; /* do nothing, all is handled by signal handlers already */\n+\twhile (timeout_count < idle_in_minutes) {\n+\t\tif (sleep(60) == EINTR)\n+\t\t\ttimeout_count = 0;\n+\t\telse\n+\t\t\ttimeout_count ++;\n+\t\ttouch_pid_file();\n+\t}\n }\n \n #elif defined(GIT_WINDOWS_NATIVE)\n \n-static void loop(const char *pid_file, int idle_in_seconds)\n+static void loop(const char *pid_file, int idle_in_minutes)\n {\n \tHWND hwnd;\n \tUINT_PTR timer = 0;\n \tMSG msg;\n \tHINSTANCE hinst = GetModuleHandle(NULL);\n \tWNDCLASS wc;\n+\tint timeout_count = 0;\n \n \t/*\n \t * Emulate UNIX signals by sending WM_USER+x to a\n@@ -258,11 +276,18 @@ static void loop(const char *pid_file, int idle_in_seconds)\n \n \trefresh(0);\n \twhile (1) {\n-\t\ttimer = SetTimer(hwnd, timer, idle_in_seconds * 1000, NULL);\n+\t\ttimer = SetTimer(hwnd, timer, 60 * 1000, NULL);\n \t\tif (!timer)\n \t\t\tdie(_(\"no timer!\"));\n-\t\tif (!GetMessage(&msg, hwnd, 0, 0) || msg.message == WM_TIMER)\n+\t\tif (!GetMessage(&msg, hwnd, 0, 0))\n \t\t\tbreak;\n+\t\tif (msg.message == WM_TIMER) {\n+\t\t\ttimeout_count ++;\n+\t\t\tif (timeout_count == idle_in_minutes)\n+\t\t\t\tbreak;\n+\t\t}\n+\t\ttimeout_count = 0;\n+\t\ttouch_pid_file();\n \t\tswitch (msg.message) {\n \t\tcase WM_USER:\n \t\t\trefresh(0);\n@@ -288,6 +313,44 @@ static const char * const usage_text[] = {\n \tNULL\n };\n \n+static int already_running(void)\n+{\n+\tchar contents[20] = {0};\n+\tchar *end;\n+\tint fd, dead, i = 0;\n+\ttime_t now;\n+\tpid_t pid;\n+\tstruct stat st;\n+\n+\tfd = open(git_path(\"index-helper.pid\"), O_RDONLY);\n+\tif (fd < 0)\n+\t\treturn 0;\n+\n+\tif (fstat(fd, &st) < 0)\n+\t\treturn 0;\n+\n+\tnow = time(&now);\n+\n+\t/*\n+\t * We touch the pid file every minute, so if a pid file hasn't\n+\t * been updated in two minutes, it's out-of-date.\n+\t */\n+\tif (st.st_mtime + 120 < now)\n+\t\treturn 0;\n+\n+\tread_in_full(fd, &contents, sizeof(contents));\n+\n+\twhile (!isdigit(contents[i]) && contents[i])\n+\t\ti++;\n+\tpid = strtoll(contents + i, &end, 10);\n+\tif (contents == end)\n+\t\treturn 0;\n+\tdead = kill(pid, 0);\n+\tif (!dead)\n+\t\treturn pid;\n+\treturn 0;\n+}\n+\n static const char *write_pid(void)\n {\n \tstatic struct strbuf sb = STRBUF_INIT;\n@@ -314,6 +377,8 @@ int main(int argc, char **argv)\n {\n \tconst char *prefix;\n \tint idle_in_minutes = 10, detach = 0;\n+\tint ignore_existing = 0;\n+\tint kill_existing = 0;\n \tconst char *pid_file;\n \tstruct option options[] = {\n \t\tOPT_INTEGER(0, \"exit-after\", &idle_in_minutes,\n@@ -321,6 +386,8 @@ int main(int argc, char **argv)\n \t\tOPT_BOOL(0, \"strict\", &to_verify,\n \t\t\t \"verify shared memory after creating\"),\n \t\tOPT_BOOL(0, \"detach\", &detach, \"detach the process\"),\n+\t\tOPT_BOOL(0, \"ignore-existing\", &ignore_existing, \"run even if another index-helper seems to be running for this repo\"),\n+\t\tOPT_BOOL(0, \"kill\", &kill_existing, \"kill any running index-helper for this repo\"),\n \t\tOPT_END()\n \t};\n \n@@ -335,6 +402,20 @@ int main(int argc, char **argv)\n \t\t\t options, usage_text, 0))\n \t\tdie(_(\"too many arguments\"));\n \n+\tif (ignore_existing && kill_existing)\n+\t\tdie(_(\"--ignore-existing and --kill don't make sense together\"));\n+\n+\tif (!ignore_existing) {\n+\t\tpid_t pid = already_running();\n+\t\tif (pid) {\n+\t\t\tif (kill_existing)\n+\t\t\t\texit(kill(pid, SIGKILL));\n+\t\t\telse if (!detach)\n+\t\t\t\twarning(\"index-helper is apparently already running with pid %d\", (int)pid);\n+\t\t\texit(0);\n+\t\t}\n+\t}\n+\n \twrite_pid();\n \n \tatexit(cleanup);\n@@ -351,6 +432,6 @@ int main(int argc, char **argv)\n \n \tif (!idle_in_minutes)\n \t\tidle_in_minutes = 0xffffffff / 60;\n-\tloop(pid_file, idle_in_minutes * 60);\n+\tloop(pid_file, idle_in_minutes);\n \treturn 0;\n }\ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nnew file mode 100755\nindex 0000000..6708180\n--- /dev/null\n+++ b/t/t7900-index-helper.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2016, Twitter, Inc\n+#\n+\n+test_description='git-index-helper\n+\n+Testing git index-helper\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'index-helper creates usable pid file and can be killed' '\n+\ttest_path_is_missing .git/index-helper.pid &&\n+\tgit index-helper --detach &&\n+\ttest_path_is_file .git/index-helper.pid &&\n+\tpid=$(sed s/[^0-9]//g .git/index-helper.pid) &&\n+\tkill -0 $pid &&\n+\tgit index-helper --kill &&\n+\t! kill -0 $pid\n+'\n+\n+test_done\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280500","messageId":"1457548582-28302-19-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 18/19] index-helper: autorun","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:21Z","receivedAt":"2016-03-09T18:36:21Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"Introduce a new config option, indexhelper.autorun, to automatically\nrun git index-helper before starting up a builtin git command. This\nenables users to keep index-helper running without manual\nintervention.\n\nSigned-off-by: David Turner \n---\n git.c | 38 +++++++++++++++++++++++++++++++++++++-\n index-helper.c | 11 ++++++++++-\n t/t7900-index-helper.sh | 10 ++++++++++\n 3 files changed, 57 insertions(+), 2 deletions(-)\n\ndiff --git a/git.c b/git.c\nindex a4f6f71..ccf04ab 100644\n--- a/git.c\n+++ b/git.c\n@@ -521,6 +521,40 @@ static void strip_extension(const char **argv)\n #define strip_extension(cmd)\n #endif\n \n+static int want_auto_index_helper(void)\n+{\n+\tint want = -1;\n+\tconst char *value = NULL;\n+\tconst char *key = \"indexhelper.autorun\";\n+\n+\tif (git_config_key_is_valid(key) &&\n+\t !git_config_get_value(key, &value)) {\n+\t\tint b = git_config_maybe_bool(key, value);\n+\t\twant = b >= 0 ? b : 0;\n+\t}\n+\treturn want;\n+}\n+\n+static void maybe_run_index_helper(struct cmd_struct *cmd)\n+{\n+\tconst char *argv[] = {\"git-index-helper\", \"--detach\", \"--auto\", NULL};\n+\tint status;\n+\n+\tif (!(cmd->option & NEED_WORK_TREE))\n+\t\treturn;\n+\n+\tif (want_auto_index_helper() <= 0)\n+\t\treturn;\n+\n+\ttrace_argv_printf(argv, \"trace: auto index-helper:\");\n+\n+\tstatus = run_command_v_opt(argv, RUN_SILENT_EXEC_FAILURE | RUN_CLEAN_ON_EXIT);\n+\n+\tif (status) {\n+\t\twarning(\"You specified indexhelper.autorun, but running git-index-helper failed\");\n+\t}\n+}\n+\n static void handle_builtin(int argc, const char **argv)\n {\n \tconst char *cmd;\n@@ -536,8 +570,10 @@ static void handle_builtin(int argc, const char **argv)\n \t}\n \n \tbuiltin = get_builtin(cmd);\n-\tif (builtin)\n+\tif (builtin) {\n+\t\tmaybe_run_index_helper(builtin);\n \t\texit(run_builtin(builtin, argc, argv));\n+\t}\n }\n \n static void execv_dashed_external(const char **argv)\ndiff --git a/index-helper.c b/index-helper.c\nindex a75da60..bc5c328 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -379,6 +379,7 @@ int main(int argc, char **argv)\n \tint idle_in_minutes = 10, detach = 0;\n \tint ignore_existing = 0;\n \tint kill_existing = 0;\n+\tint nongit = 0, autorun = 0;\n \tconst char *pid_file;\n \tstruct option options[] = {\n \t\tOPT_INTEGER(0, \"exit-after\", &idle_in_minutes,\n@@ -388,6 +389,7 @@ int main(int argc, char **argv)\n \t\tOPT_BOOL(0, \"detach\", &detach, \"detach the process\"),\n \t\tOPT_BOOL(0, \"ignore-existing\", &ignore_existing, \"run even if another index-helper seems to be running for this repo\"),\n \t\tOPT_BOOL(0, \"kill\", &kill_existing, \"kill any running index-helper for this repo\"),\n+\t\tOPT_BOOL(0, \"auto\", &autorun, \"this is an automatic run of git index-helper, so certain errors can be solved by silently exiting\"),\n \t\tOPT_END()\n \t};\n \n@@ -397,11 +399,18 @@ int main(int argc, char **argv)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(usage_text, options);\n \n-\tprefix = setup_git_directory();\n+\tprefix = setup_git_directory_gently(&nongit);\n \tif (parse_options(argc, (const char **)argv, prefix,\n \t\t\t options, usage_text, 0))\n \t\tdie(_(\"too many arguments\"));\n \n+\tif (nongit) {\n+\t\tif (autorun)\n+\t\t\texit(0);\n+\t\telse\n+\t\t\tdie(\"Not a git repository\");\n+\t}\n+\n \tif (ignore_existing && kill_existing)\n \t\tdie(_(\"--ignore-existing and --kill don't make sense together\"));\n \ndiff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\nindex 6708180..e4f9564 100755\n--- a/t/t7900-index-helper.sh\n+++ b/t/t7900-index-helper.sh\n@@ -20,4 +20,14 @@ test_expect_success 'index-helper creates usable pid file and can be killed' '\n \t! kill -0 $pid\n '\n \n+test_expect_success 'index-helper autorun works' '\n+\trm -f .git/index-helper.pid &&\n+\tgit status &&\n+\ttest_path_is_missing .git/index-helper.pid &&\n+\ttest_config indexhelper.autorun true &&\n+\tgit status &&\n+\ttest_path_is_file .git/index-helper.pid &&\n+\tgit index-helper --kill\n+'\n+\n test_done\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280501","messageId":"1457548582-28302-20-git-send-email-dturner@twopensource.com","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"[PATCH 19/19] hack: watchman/untracked cache mashup","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-09T18:36:22Z","receivedAt":"2016-03-09T18:36:22Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"DO NOT APPLY THIS\n\nThis is a hack to add untracked cache info to the watchman WAMA\nextension. The idea is that we don't have to stat directories if\nwatchman hasn't told us of any changes in those dirs.\n\nIf this is the right idea, I can go back and add it to the relevant\npatch(es). But I want to see if other folks think it's the right idea\nfirst.\n\nSigned-off-by: David Turner \n---\n dir.c | 23 +++++++--\n dir.h | 6 +++\n read-cache.c | 142 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n watchman-support.c | 21 +++++++-\n 4 files changed, 182 insertions(+), 10 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 69e0be6..5058b29 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -597,9 +597,9 @@ static void trim_trailing_spaces(char *buf)\n *\n * If \"name\" has the trailing slash, it'll be excluded in the search.\n */\n-static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n-\t\t\t\t\t\t struct untracked_cache_dir *dir,\n-\t\t\t\t\t\t const char *name, int len)\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+\t\t\t\t\t struct untracked_cache_dir *dir,\n+\t\t\t\t\t const char *name, int len)\n {\n \tint first, last;\n \tstruct untracked_cache_dir *d;\n@@ -1726,6 +1726,17 @@ static int valid_cached_dir(struct dir_struct *dir,\n \tif (!untracked)\n \t\treturn 0;\n \n+\tif (dir->untracked->use_watchman) {\n+\t\t/*\n+\t\t * With watchman, we can trust the untracked cache's\n+\t\t * valid field.\n+\t\t */\n+\t\tif (untracked->valid)\n+\t\t\tgoto skip_stat;\n+\t\telse\n+\t\t\tinvalidate_directory(dir->untracked, untracked);\n+\t}\n+\n \tif (stat(path->len ? path->buf : \".\", &st)) {\n \t\tinvalidate_directory(dir->untracked, untracked);\n \t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n@@ -1739,6 +1750,7 @@ static int valid_cached_dir(struct dir_struct *dir,\n \t\treturn 0;\n \t}\n \n+skip_stat:\n \tif (untracked->check_only != !!check_only) {\n \t\tinvalidate_directory(dir->untracked, untracked);\n \t\treturn 0;\n@@ -2625,8 +2637,10 @@ static void free_untracked(struct untracked_cache_dir *ucd)\n \n void free_untracked_cache(struct untracked_cache *uc)\n {\n-\tif (uc)\n+\tif (uc) {\n \t\tfree_untracked(uc->root);\n+\t\tstring_list_clear(&uc->invalid_untracked, 0);\n+\t}\n \tfree(uc);\n }\n \n@@ -2775,6 +2789,7 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n \t\treturn NULL;\n \n \tuc = xcalloc(1, sizeof(*uc));\n+\tstring_list_init(&uc->invalid_untracked, 1);\n \tstrbuf_init(&uc->ident, ident_len);\n \tstrbuf_add(&uc->ident, ident, ident_len);\n \tload_sha1_stat(&uc->ss_info_exclude, &ouc->info_exclude_stat,\ndiff --git a/dir.h b/dir.h\nindex 3ec3fb0..8fd3f9e 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -142,6 +142,9 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n+\t/* watchman invalidation data */\n+\tunsigned int use_watchman : 1;\n+\tstruct string_list invalid_untracked;\n };\n \n struct dir_struct {\n@@ -312,4 +315,7 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked);\n void add_untracked_cache(struct index_state *istate);\n void remove_untracked_cache(struct index_state *istate);\n+struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+\t\t\t\t\t struct untracked_cache_dir *dir,\n+\t\t\t\t\t const char *name, int len);\n #endif\ndiff --git a/read-cache.c b/read-cache.c\nindex 40d789a..66438d8 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1237,7 +1237,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (!new) {\n \t\t\tconst char *fmt;\n \n-\t\t\tif (really && cache_errno == EINVAL) {\n+\t\t\tif (really || cache_errno == EINVAL) {\n \t\t\t\t/* If we are doing --really-refresh that\n \t\t\t\t * means the index is not valid anymore.\n \t\t\t\t */\n@@ -1377,11 +1377,82 @@ static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n \treturn 0;\n }\n \n+static struct untracked_cache_dir *find_untracked_cache_dir(\n+\tstruct untracked_cache *uc, struct untracked_cache_dir *ucd,\n+\tconst char *name)\n+{\n+\tint component_len;\n+\tconst char *end;\n+\tstruct untracked_cache_dir *dir;\n+\n+\tif (!*name)\n+\t\treturn ucd;\n+\n+\tend = strchr(name, '/');\n+\tif (end)\n+\t\tcomponent_len = end - name;\n+\telse\n+\t\tcomponent_len = strlen(name);\n+\n+\tdir = lookup_untracked(uc, ucd, name, component_len);\n+\tif (dir)\n+\t\treturn find_untracked_cache_dir(uc, dir, name + component_len + 1);\n+\n+\treturn NULL;\n+}\n+\n static void mark_no_watchman(size_t pos, void *data)\n {\n \tstruct index_state *istate = data;\n+\tstruct cache_entry *ce = istate->cache[pos];\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tchar *c;\n+\tstruct untracked_cache_dir *dir;\n+\n \tassert(pos < istate->cache_nr);\n-\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n+\tce->ce_flags |= CE_WATCHMAN_DIRTY;\n+\n+\tif (!istate->untracked || !istate->untracked->root)\n+\t\treturn;\n+\n+\tstrbuf_add(&sb, ce->name, ce_namelen(ce));\n+\n+\tfor (c = sb.buf + sb.len - 1; c > sb.buf; c--) {\n+\t\tif (*c == '/') {\n+\t\t\tstrbuf_setlen(&sb, c - sb.buf);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (c == sb.buf) {\n+\t\tstrbuf_setlen(&sb, 0);\n+\t}\n+\n+\tdir = find_untracked_cache_dir(istate->untracked,\n+\t\t\t\t istate->untracked->root, sb.buf);\n+\tif (dir) {\n+\t\tfprintf(stderr, \"from '%s', dir '%s', invalidated '%s'\\n\",\n+\t\t\tce->name, sb.buf, dir->name);\n+\t\tdir->valid = 0;\n+\t} else {\n+\t\tfprintf(stderr, \"from '%s', dir '%s', not found\\n\",\n+\t\t\tce->name, sb.buf);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+}\n+\n+static int mark_untracked_invalid(struct string_list_item *item, void *uc)\n+{\n+\tstruct untracked_cache *untracked = uc;\n+\tstruct untracked_cache_dir *dir;\n+\n+\tdir = find_untracked_cache_dir(untracked, untracked->root,\n+\t\t\t\t item->string);\n+\tif (dir)\n+\t\tdir->valid = 0;\n+\n+\treturn 0;\n }\n \n static int read_watchman_ext(struct index_state *istate, const void *data,\n@@ -1389,19 +1460,56 @@ static int read_watchman_ext(struct index_state *istate, const void *data,\n {\n \tstruct ewah_bitmap *bitmap;\n \tint ret, len;\n+\tuint32_t bitmap_size;\n+\tuint32_t untracked_nr;\n+\tconst char *untracked;\n \n \tif (memchr(data, 0, sz) == NULL)\n \t\treturn error(\"invalid extension\");\n+\n \tlen = strlen(data) + 1;\n+\tmemcpy(&bitmap_size, (const char *)data + len, 4);\n+\tmemcpy(&untracked_nr, (const char *)data + len + 4, 4);\n+\tuntracked_nr = ntohl(untracked_nr);\n+\tbitmap_size = ntohl(bitmap_size);\n+\n \tbitmap = ewah_new();\n-\tret = ewah_read_mmap(bitmap, (const char *)data + len, sz - len);\n-\tif (ret != sz - len) {\n+\tret = ewah_read_mmap(bitmap, (const char *)data + len + 8, bitmap_size);\n+\tif (ret != bitmap_size) {\n \t\tewah_free(bitmap);\n \t\treturn error(\"failed to parse ewah bitmap reading watchman index extension\");\n \t}\n \tistate->last_update = xstrdup(data);\n \tewah_each_bit(bitmap, mark_no_watchman, istate);\n \tewah_free(bitmap);\n+\n+\tif (istate->untracked && istate->untracked->root) {\n+\t\tint i;\n+\n+\t\tistate->untracked->use_watchman = 1;\n+\t\tuntracked = data + len + 8 + bitmap_size;\n+\t\tfor (i = 0; i < untracked_nr; ++i) {\n+\t\t\tint len = strlen(untracked);\n+\t\t\tstring_list_append(&istate->untracked->invalid_untracked,\n+\t\t\t\t\t untracked);\n+\t\t\tuntracked += len + 1;\n+\t\t}\n+\n+\t\tfor_each_string_list(&istate->untracked->invalid_untracked,\n+\t\t\t mark_untracked_invalid, istate->untracked);\n+\n+\t\tstring_list_clear(&istate->untracked->invalid_untracked, 0);\n+\t\tistate->cache_changed |= WATCHMAN_CHANGED;\n+\t}\n+\treturn 0;\n+}\n+\n+int untracked_entry_append(struct string_list_item *item, void *sbvoid)\n+{\n+\tstruct strbuf *sb = sbvoid;\n+\n+\tstrbuf_addstr(sb, item->string);\n+\tstrbuf_addch(sb, 0);\n \treturn 0;\n }\n \n@@ -1409,14 +1517,38 @@ void write_watchman_ext(struct strbuf *sb, struct index_state* istate)\n {\n \tstruct ewah_bitmap *bitmap;\n \tint i;\n+\tint ewah_start;\n+\tint ewah_size = 0;\n+\tint fixup = 0;\n \n \tstrbuf_add(sb, istate->last_update, strlen(istate->last_update) + 1);\n+\tfixup = sb->len;\n+\tstrbuf_add(sb, &ewah_size, 4); /* we'll fix this up later */\n+\tif (istate->untracked) {\n+\t\tuint32_t nr = istate->untracked->invalid_untracked.nr;\n+\t\tnr = htonl(nr);\n+\t\tstrbuf_add(sb, &nr, 4);\n+\t} else {\n+\t\t/* zero */\n+\t\tstrbuf_add(sb, &ewah_size, 4);\n+\t}\n+\n+\tewah_start = sb->len;\n \tbitmap = ewah_new();\n \tfor (i = 0; i < istate->cache_nr; i++)\n \t\tif (istate->cache[i]->ce_flags & CE_WATCHMAN_DIRTY)\n \t\t\tewah_set(bitmap, i);\n \tewah_serialize_strbuf(bitmap, sb);\n \tewah_free(bitmap);\n+\n+\t/* fix up size field */\n+\tewah_size = sb->len - ewah_start;\n+\tewah_size = htonl(ewah_size);\n+\tmemcpy(sb->buf + fixup, &ewah_size, 4);\n+\n+\tif (istate->untracked)\n+\t\tfor_each_string_list(&istate->untracked->invalid_untracked,\n+\t\t\t\t untracked_entry_append, sb);\n }\n \n static int read_index_extension(struct index_state *istate,\n@@ -1956,7 +2088,7 @@ int unmerged_index(const struct index_state *istate)\n \treturn 0;\n }\n \n-#define WRITE_BUFFER_SIZE 8192\n+#define WRITE_BUFFER_SIZE 8192*16\n static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n static unsigned long write_buffer_len;\n \ndiff --git a/watchman-support.c b/watchman-support.c\nindex 08e37ae..b7302b9 100644\n--- a/watchman-support.c\n+++ b/watchman-support.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"watchman-support.h\"\n #include \"strbuf.h\"\n+#include \"dir.h\"\n #include \n \n static struct watchman_query *make_query(const char *last_update)\n@@ -60,8 +61,24 @@ static void update_index(struct index_state *istate,\n \t\t\tcontinue;\n \n \t\tpos = index_name_pos(istate, wm->name, strlen(wm->name));\n-\t\tif (pos < 0)\n+\t\tif (pos < 0) {\n+\t\t\tif (istate->untracked) {\n+\t\t\t\tchar *name = xstrdup(wm->name);\n+\t\t\t\tchar *dname = dirname(name);\n+\n+\t\t\t\t/*\n+\t\t\t\t * dirname() returns '.' for the root,\n+\t\t\t\t * but we call it ''.\n+\t\t\t\t */\n+\t\t\t\tif (dname[0] == '.' && dname[1] == 0)\n+\t\t\t\t\tstring_list_append(&istate->untracked->invalid_untracked, \"\");\n+\t\t\t\telse\n+\t\t\t\t\tstring_list_append(&istate->untracked->invalid_untracked,\n+\t\t\t\t\t\t\t dname);\n+\t\t\t\tfree(name);\n+\t\t\t}\n \t\t\tcontinue;\n+\t\t}\n \t\t/* FIXME: ignore staged entries and gitlinks too? */\n \n \t\tistate->cache[pos]->ce_flags |= CE_WATCHMAN_DIRTY;\n@@ -71,6 +88,8 @@ done:\n \tfree(istate->last_update);\n \tistate->last_update = xstrdup(result->clock);\n \tistate->cache_changed |= WATCHMAN_CHANGED;\n+\tif (istate->untracked)\n+\t\tstring_list_remove_duplicates(&istate->untracked->invalid_untracked, 0);\n }\n \n int check_watchman(struct index_state *istate)\n-- \n2.4.2.767.g62658d5-twtrsrc\n"},{"id":"280513","messageId":"xmqq4mcfgsxn.fsf@gitster.mtv.corp.google.com","threadId":"41653","inReplyTo":"1457548582-28302-2-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 01/19] trace.c: add GIT_TRACE_PACK_STATS for pack usage statistics","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-09T22:58:12Z","receivedAt":"2016-03-09T22:58:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> From: Nguyễn Thái Ngọc Duy \n>\n> trace_stats() is intended for GIT_TRACE_*_STATS variable group and\n> GIT_TRACE_PACK_STATS is more like an example how new vars can be added.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> Documentation/git.txt | 3 +++\n> cache.h | 2 ++\n> git.c | 1 +\n> sha1_file.c | 24 ++++++++++++++++++++++++\n> trace.c | 13 +++++++++++++\n> trace.h | 1 +\n> 6 files changed, 44 insertions(+)\n>\n> diff --git a/Documentation/git.txt b/Documentation/git.txt\n> index 2754af8..794271e 100644\n> --- a/Documentation/git.txt\n> +++ b/Documentation/git.txt\n> @@ -1055,6 +1055,9 @@ of clones and fetches.\n> \ttime of each Git command.\n> \tSee 'GIT_TRACE' for available trace output options.\n> \n> +'GIT_TRACE_PACK_STATS'::\n> +\tPrint various statistics.\n\n\"Various\" is quite vague.\n\nPerhaps a new description in Documentation/technical/ might be a\ngood thing to have?\n\n> +void report_pack_stats(struct trace_key *key)\n> +{\n> +\ttrace_printf_key(key, \"\\n\"\n> +\t\t\t \"pack_report: getpagesize() = %10\" SZ_FMT \"\\n\"\n> +\t\t\t \"pack_report: core.packedGitWindowSize = %10\" SZ_FMT \"\\n\"\n> +\t\t\t \"pack_report: core.packedGitLimit = %10\" SZ_FMT \"\\n\"\n> +\t\t\t \"pack_report: pack_used_ctr = %10u\\n\"\n> +\t\t\t \"pack_report: pack_mmap_calls = %10u\\n\"\n> +\t\t\t \"pack_report: pack_open_windows = %10u / %10u\\n\"\n> +\t\t\t \"pack_report: pack_mapped = \"\n> +\t\t\t \"%10\" SZ_FMT \" / %10\" SZ_FMT \"\\n\"\n> +\t\t\t \"pack_report: pack accesss = %10u\\n\",\n> +\t\t\t sz_fmt(getpagesize()),\n> +\t\t\t sz_fmt(packed_git_window_size),\n> +\t\t\t sz_fmt(packed_git_limit),\n> +\t\t\t pack_used_ctr,\n> +\t\t\t pack_mmap_calls,\n> +\t\t\t pack_open_windows, peak_pack_open_windows,\n> +\t\t\t sz_fmt(pack_mapped), sz_fmt(peak_pack_mapped),\n> +\t\t\t pack_access_nr);\n> +}\n> +\n> /*\n> * Open and mmap the index file at path, perform a couple of\n> * consistency checks, then record its information to p. Return 0 on\n> @@ -2238,6 +2261,7 @@ static void write_pack_access_log(struct packed_git *p, off_t obj_offset)\n> \tstatic struct trace_key pack_access = TRACE_KEY_INIT(PACK_ACCESS);\n> \ttrace_printf_key(&pack_access, \"%s %\"PRIuMAX\"\\n\",\n> \t\t\t p->pack_name, (uintmax_t)obj_offset);\n> +\tpack_access_nr++;\n> }\n\nThis looks rather half-hearted, in that those who are interested in\nthis new number can run \"wc -l\" on the pack-access trace log without\nadding a new \"stats\" anyway. It may make the \"stats\" far more\nuseful to keep track of the summary information of what would be\nwritten to the pack access log and add to the report_pack_stats()\noutput things like the average and median distance of seeks\n(i.e. differences in the \"obj_offset\" into the same pack by two\nadjacent pack accesse) and the variance, for example?\n"},{"id":"280514","messageId":"xmqqziu7fe6u.fsf@gitster.mtv.corp.google.com","threadId":"41653","inReplyTo":"1457548582-28302-4-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 03/19] read-cache: allow to keep mmap'd memory after reading","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-09T23:02:01Z","receivedAt":"2016-03-09T23:02:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> From: Nguyễn Thái Ngọc Duy \n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n\nAs usual in Duy's patches, this one seriously lacks \"why\". And also\nmakes the reader wonder if the memory region is ever unmapped() and\nif so under what condition.\n\n> diff --git a/cache.h b/cache.h\n> index 7e01403..c43ef3d 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -333,11 +333,14 @@ struct index_state {\n> \tstruct split_index *split_index;\n> \tstruct cache_time timestamp;\n> \tunsigned name_hash_initialized : 1,\n> +\t\t keep_mmap : 1,\n> \t\t initialized : 1;\n> \tstruct hashmap name_hash;\n> \tstruct hashmap dir_hash;\n> \tunsigned char sha1[20];\n> \tstruct untracked_cache *untracked;\n> +\tvoid *mmap;\n> +\tsize_t mmap_size;\n> };\n> \n> extern struct index_state the_index;\n> diff --git a/read-cache.c b/read-cache.c\n> index 16cc487..7e387e9 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1574,6 +1574,10 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n> \tmmap = xmmap(NULL, mmap_size, PROT_READ, MAP_PRIVATE, fd, 0);\n> \tif (mmap == MAP_FAILED)\n> \t\tdie_errno(\"unable to map index file\");\n> +\tif (istate->keep_mmap) {\n> +\t\tistate->mmap = mmap;\n> +\t\tistate->mmap_size = mmap_size;\n> +\t}\n> \tclose(fd);\n> \n> \thdr = mmap;\n> @@ -1626,10 +1630,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n> \t\tsrc_offset += 8;\n> \t\tsrc_offset += extsize;\n> \t}\n> -\tmunmap(mmap, mmap_size);\n> +\tif (!istate->keep_mmap)\n> +\t\tmunmap(mmap, mmap_size);\n> \treturn istate->cache_nr;\n> \n> unmap:\n> +\tistate->mmap = NULL;\n> \tmunmap(mmap, mmap_size);\n> \tdie(\"index file corrupt\");\n> }\n> @@ -1655,6 +1661,7 @@ int read_index_from(struct index_state *istate, const char *path)\n> \t\tdiscard_index(split_index->base);\n> \telse\n> \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n> +\tsplit_index->base->keep_mmap = istate->keep_mmap;\n> \tret = do_read_index(split_index->base,\n> \t\t\t git_path(\"sharedindex.%s\",\n> \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n> @@ -1698,6 +1705,10 @@ int discard_index(struct index_state *istate)\n> \tfree(istate->cache);\n> \tistate->cache = NULL;\n> \tistate->cache_alloc = 0;\n> +\tif (istate->keep_mmap && istate->mmap) {\n> +\t\tmunmap(istate->mmap, istate->mmap_size);\n> +\t\tistate->mmap = NULL;\n> +\t}\n> \tdiscard_split_index(istate);\n> \tfree_untracked_cache(istate->untracked);\n> \tistate->untracked = NULL;\n"},{"id":"280515","messageId":"xmqqtwkffdus.fsf@gitster.mtv.corp.google.com","threadId":"41653","inReplyTo":"1457548582-28302-5-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-09T23:09:15Z","receivedAt":"2016-03-09T23:09:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Turner writes:\n\n> From: Nguyễn Thái Ngọc Duy \n>\n> Instead of reading the index from disk and worrying about disk\n> corruption, the index is cached in memory (memory bit-flips happen\n> too, but hopefully less often). The result is faster read. Read time\n> is reduced by 70%.\n>\n> The biggest gain is not having to verify the trailing SHA-1, which\n> takes lots of time especially on large index files. But this also\n> opens doors for further optimiztions:\n>\n> - we could create an in-memory format that's essentially the memory\n> dump of the index to eliminate most of parsing/allocation\n> overhead. The mmap'd memory can be used straight away. Experiment\n> [1] shows we could reduce read time by 88%.\n>\n> - we could cache non-index info such as name hash\n>\n> The shared memory's name folows the template \"git--\"\n> where is the trailing SHA-1 of the index file. is\n> \"index\" for cached index files (and may be \"name-hash\" for name-hash\n> cache). If such shared memory exists, it contains the same index\n> content as on disk. The content is already validated by the daemon and\n> git won't validate it again (except comparing the trailing SHA-1s).\n\nThis indeed is an interesting approach; what is not explained but\nmust be is when the on-disk index is updated to reflect the reality\n(if I am reading the explanation and the code right, while the\ndaemon is running, its in-core cache becomes the source of truth by\nforcing everybody's read-index-from() to go to the daemon). The\nexplanation could be \"this is only for read side, and updating the\nindex happens via the traditional 'write a new file and rename it to\nthe final place' codepath, at which time the daemon must be told to\nre-read it.\"\n"},{"id":"280516","messageId":"xmqqpov3fdb6.fsf@gitster.mtv.corp.google.com","threadId":"41653","inReplyTo":"xmqqtwkffdus.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-09T23:21:01Z","receivedAt":"2016-03-09T23:21:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> David Turner writes:\n>\n>> From: Nguyễn Thái Ngọc Duy \n>>\n>> Instead of reading the index from disk and worrying about disk\n>> corruption, the index is cached in memory (memory bit-flips happen\n>> too, but hopefully less often). The result is faster read. Read time\n>> is reduced by 70%.\n>>\n>> The biggest gain is not having to verify the trailing SHA-1, which\n>> takes lots of time especially on large index files.\n\nCome to think of it, wouldn't it be far less intrusive change to\njust put the index on a ramdisk and skip the trailing SHA-1\nverification, to obtain a similar result with the same trade off\n(i.e. blindly trusting memory instead of being paranoid)?\n\n \n"},{"id":"280519","messageId":"1457568079.13557.7.camel@twopensource.com","threadId":"41653","inReplyTo":"xmqqpov3fdb6.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-10T00:01:19Z","receivedAt":"2016-03-10T00:01:19Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2016-03-09 at 15:21 -0800, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > David Turner writes:\n> > \n> > > From: Nguyễn Thái Ngọc Duy \n> > > \n> > > Instead of reading the index from disk and worrying about disk\n> > > corruption, the index is cached in memory (memory bit-flips\n> > > happen\n> > > too, but hopefully less often). The result is faster read. Read\n> > > time\n> > > is reduced by 70%.\n> > > \n> > > The biggest gain is not having to verify the trailing SHA-1,\n> > > which\n> > > takes lots of time especially on large index files.\n> \n> Come to think of it, wouldn't it be far less intrusive change to\n> just put the index on a ramdisk and skip the trailing SHA-1\n> verification, to obtain a similar result with the same trade off\n> (i.e. blindly trusting memory instead of being paranoid)?\n> \n\n1. If the index were stored on a ramdisk, we would *also* have to write\nit to durable storage to avoid losing the user's work when they power\noff. That's more complicated. \n\n2. Duy notes that it is easier to add further optimizations to this --\nfor instance, we could share a pre-parsed version of the index. It\nwould be harder to add these optimizations to the disk format, because\n(a) they would take up more space, and (b) they would probably be\nharder to write in a single pass, which is presently how index writing\nworks.\n\n3. If we wanted to just skip SHA-1 verification, we would not need a\nramdisk; it's almost certain that the index would be in the disk cache\nmost of the time, so regular storage should be very nearly as fast as a\nramdisk. I think mlock might help ensure that the data remains in the\ncache, although I'm not sure what the permissions story is on that.\n"},{"id":"280520","messageId":"1457568342.13557.11.camel@twopensource.com","threadId":"41653","inReplyTo":"xmqq4mcfgsxn.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 01/19] trace.c: add GIT_TRACE_PACK_STATS for pack usage statistics","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-10T00:05:42Z","receivedAt":"2016-03-10T00:05:42Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2016-03-09 at 14:58 -0800, Junio C Hamano wrote:\n> David Turner writes:\n...\n> > trace_stats() is intended for GIT_TRACE_*_STATS variable group and\n> > GIT_TRACE_PACK_STATS is more like an example how new vars can be\n> > added.\n...\n\n> > +\tpack_access_nr++;\n> > }\n> \n> This looks rather half-hearted, in that those who are interested in\n> this new number can run \"wc -l\" on the pack-access trace log without\n> adding a new \"stats\" anyway. It may make the \"stats\" far more\n> useful to keep track of the summary information of what would be\n> written to the pack access log and add to the report_pack_stats()\n> output things like the average and median distance of seeks\n> (i.e. differences in the \"obj_offset\" into the same pack by two\n> adjacent pack accesse) and the variance, for example?\n\nOn reflection, I do not think I need this patch; it was in Duy's series\nand the index stats patch would need to be rewritten slightly to get\nrid of it, but I'm OK with that. I wanted to make minimal changes to\nDuy's patches, but I'd rather make larger changes than do work on\npatches I don't need.\n\nSo unless Duy objects, I'm going to drop this from the series.\n"},{"id":"280521","messageId":"1457568548.13557.13.camel@twopensource.com","threadId":"41653","inReplyTo":"xmqqziu7fe6u.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 03/19] read-cache: allow to keep mmap'd memory after reading","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-10T00:09:08Z","receivedAt":"2016-03-10T00:09:08Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2016-03-09 at 15:02 -0800, Junio C Hamano wrote:\n> David Turner writes:\n> \n> > From: Nguyễn Thái Ngọc Duy \n> > \n> > Signed-off-by: Nguyễn Thái Ngọc Duy \n> > ---\n> \n> As usual in Duy's patches, this one seriously lacks \"why\". And also\n> makes the reader wonder if the memory region is ever unmapped() and\n> if so under what condition.\n\nHow about this:\n\nLater, we will introduce git index-helper to share this memory with\nother git processes.\n\nSince the memory will be shared, it will never be unmapped (although\nthe kernel may of course choose to page it out).\n"},{"id":"280522","messageId":"1457569105.13557.19.camel@twopensource.com","threadId":"41653","inReplyTo":"xmqqtwkffdus.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-10T00:18:25Z","receivedAt":"2016-03-10T00:18:25Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Wed, 2016-03-09 at 15:09 -0800, Junio C Hamano wrote:\n> David Turner writes:\n> \n> > From: Nguyễn Thái Ngọc Duy \n> > \n> > Instead of reading the index from disk and worrying about disk\n> > corruption, the index is cached in memory (memory bit-flips happen\n> > too, but hopefully less often). The result is faster read. Read\n> > time\n> > is reduced by 70%.\n> > \n> > The biggest gain is not having to verify the trailing SHA-1, which\n> > takes lots of time especially on large index files. But this also\n> > opens doors for further optimiztions:\n> > \n> > - we could create an in-memory format that's essentially the\n> > memory\n> > dump of the index to eliminate most of parsing/allocation\n> > overhead. The mmap'd memory can be used straight away.\n> > Experiment\n> > [1] shows we could reduce read time by 88%.\n> > \n> > - we could cache non-index info such as name hash\n> > \n> > The shared memory's name folows the template \"git--\"\n> > where is the trailing SHA-1 of the index file. is\n> > \"index\" for cached index files (and may be \"name-hash\" for name\n> > -hash\n> > cache). If such shared memory exists, it contains the same index\n> > content as on disk. The content is already validated by the daemon\n> > and\n> > git won't validate it again (except comparing the trailing SHA-1s).\n> \n> This indeed is an interesting approach; what is not explained but\n> must be is when the on-disk index is updated to reflect the reality\n> (if I am reading the explanation and the code right, while the\n> daemon is running, its in-core cache becomes the source of truth by\n> forcing everybody's read-index-from() to go to the daemon). The\n> explanation could be \"this is only for read side, and updating the\n> index happens via the traditional 'write a new file and rename it to\n> the final place' codepath, at which time the daemon must be told to\n> re-read it.\"\n\nThis seems like the explanation (from the current commit message):\n\n\"Git can poke the daemon to tell it to refresh the index cache, or to\nkeep it alive some more minutes via UNIX signals. It can't give any\nreal index data directly to the daemon. Real data goes to disk first,\nthen the daemon reads and verifies it from there. Poking only happens\nfor $GIT_DIR/index, not temporary index files.\"\n\nI guess this could be rewritten as:\n\n\"Index validity is ensured by the following method: When a read is\nrequested from the index-helper, it checks the SHA1 of its cached index\ncopy against the on-disk version. If they differ, index-helper rereads\nthe index. In addition, any git process may explicitly suggest a\nreread via a UNIX signal, but this is only an optimization and it is\nnot necessary for correctness.\n\nIn addition, Git can signal the daemon with a heartbeat signal, to keep\nthe daemon alive longer.\"\n\nHow does that sound?\n"},{"id":"280545","messageId":"CACsJy8BdbhPXAoP9ZSxtK3+gnmDmfJ_wB1w2od2QPL5HACRfWA@mail.gmail.com","threadId":"41653","inReplyTo":"1457568342.13557.11.camel@twopensource.com","subject":"Re: [PATCH 01/19] trace.c: add GIT_TRACE_PACK_STATS for pack usage statistics","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-10T10:59:43Z","receivedAt":"2016-03-10T10:59:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 10, 2016 at 7:05 AM, David Turner wrote:\n> On Wed, 2016-03-09 at 14:58 -0800, Junio C Hamano wrote:\n>> David Turner writes:\n> ...\n>> > trace_stats() is intended for GIT_TRACE_*_STATS variable group and\n>> > GIT_TRACE_PACK_STATS is more like an example how new vars can be\n>> > added.\n> ...\n>\n>> > + pack_access_nr++;\n>> > }\n>>\n>> This looks rather half-hearted, in that those who are interested in\n>> this new number can run \"wc -l\" on the pack-access trace log without\n>> adding a new \"stats\" anyway. It may make the \"stats\" far more\n>> useful to keep track of the summary information of what would be\n>> written to the pack access log and add to the report_pack_stats()\n>> output things like the average and median distance of seeks\n>> (i.e. differences in the \"obj_offset\" into the same pack by two\n>> adjacent pack accesse) and the variance, for example?\n>\n> On reflection, I do not think I need this patch; it was in Duy's series\n> and the index stats patch would need to be rewritten slightly to get\n> rid of it, but I'm OK with that. I wanted to make minimal changes to\n> Duy's patches, but I'd rather make larger changes than do work on\n> patches I don't need.\n>\n> So unless Duy objects, I'm going to drop this from the series.\n\nI wanted something to verify from the tests and this was the best I\ncould come up. But since I added trace_printf_key(&trace_exclude,... I\nthink that could serve better to show what's going on, and we can\neasily summarize from the trace if we want to. That's one option. If\nyou can adjust the tests another way, I'm fine too.\n-- \nDuy\n"},{"id":"280546","messageId":"CACsJy8CqOOB6C6bWkVSEvtPuHxZR_avSb7z7hgo-vtv3gu44Cg@mail.gmail.com","threadId":"41653","inReplyTo":"xmqqpov3fdb6.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-10T11:17:30Z","receivedAt":"2016-03-10T11:17:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 10, 2016 at 6:21 AM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n>> David Turner writes:\n>>\n>>> From: Nguyễn Thái Ngọc Duy \n>>>\n>>> Instead of reading the index from disk and worrying about disk\n>>> corruption, the index is cached in memory (memory bit-flips happen\n>>> too, but hopefully less often). The result is faster read. Read time\n>>> is reduced by 70%.\n>>>\n>>> The biggest gain is not having to verify the trailing SHA-1, which\n>>> takes lots of time especially on large index files.\n>\n> Come to think of it, wouldn't it be far less intrusive change to\n> just put the index on a ramdisk and skip the trailing SHA-1\n> verification, to obtain a similar result with the same trade off\n> (i.e. blindly trusting memory instead of being paranoid)?\n\nI'm straying off-topic again because this reminded me about lmdb :)\nFor the record when I looked at lmdb I thought of using it as an index\nformat too. It has everything we wanted in index-v5. Good enough that\nwe would not need index-helper and split-index to work around current\nindex format. Though I finally decided it didn't fit well.\n\nOn the good side, lmdb is b+ trees, we can update directly on disk\n(without rewriting whole file), read directly from mmap'd file and not\nworry about corrupting it. There's no SHA-1 verification either (and\nwe can do crc32 per entry if we want too). It's good.\n\nBut, it does not fit well the our locking model (but we can change\nthis). And we sometimes want to create temporary throw-away indexes\n(e.g. partial commits) which I don't think is easy to do. And the\nreading directly from mmap does not give us much because we have to do\nbyte endian conversion anyway.\n\nHmm.. on second thought, even though lmdb can't be the default format\n(for a bunch of other limitations), it can still be an option for\nsuper big worktrees, just like index-helper being an optional\noptimization. Hm.. hm.. if we can support lmdb as index format in\naddition to the current format, bringing back some work from Thomas..\nWe may still need a daemon or something to deal with watchman, but the\nunderlying mechanism is exactly the same... David, Junio, what do you\nthink?\n-- \nDuy\n"},{"id":"280569","messageId":"1457641341.13557.32.camel@twopensource.com","threadId":"41653","inReplyTo":"CACsJy8CqOOB6C6bWkVSEvtPuHxZR_avSb7z7hgo-vtv3gu44Cg@mail.gmail.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-10T20:22:21Z","receivedAt":"2016-03-10T20:22:21Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-03-10 at 18:17 +0700, Duy Nguyen wrote:\n> On Thu, Mar 10, 2016 at 6:21 AM, Junio C Hamano \n> wrote:\n> > Junio C Hamano writes:\n> > \n> > > David Turner writes:\n> > > \n> > > > From: Nguyễn Thái Ngọc Duy \n> > > > \n> > > > Instead of reading the index from disk and worrying about disk\n> > > > corruption, the index is cached in memory (memory bit-flips\n> > > > happen\n> > > > too, but hopefully less often). The result is faster read. Read\n> > > > time\n> > > > is reduced by 70%.\n> > > > \n> > > > The biggest gain is not having to verify the trailing SHA-1,\n> > > > which\n> > > > takes lots of time especially on large index files.\n> > \n> > Come to think of it, wouldn't it be far less intrusive change to\n> > just put the index on a ramdisk and skip the trailing SHA-1\n> > verification, to obtain a similar result with the same trade off\n> > (i.e. blindly trusting memory instead of being paranoid)?\n> \n> I'm straying off-topic again because this reminded me about lmdb :)\n> For the record when I looked at lmdb I thought of using it as an\n> index\n> format too. It has everything we wanted in index-v5. Good enough that\n> we would not need index-helper and split-index to work around current\n> index format. Though I finally decided it didn't fit well.\n> \n> On the good side, lmdb is b+ trees, we can update directly on disk\n> (without rewriting whole file), read directly from mmap'd file and\n> not\n> worry about corrupting it. There's no SHA-1 verification either (and\n> we can do crc32 per entry if we want too). It's good.\n> \n> But, it does not fit well the our locking model (but we can change\n> this). And we sometimes want to create temporary throw-away indexes\n> (e.g. partial commits) which I don't think is easy to do. And the\n> reading directly from mmap does not give us much because we have to\n> do\n> byte endian conversion anyway.\n> \n> Hmm.. on second thought, even though lmdb can't be the default format\n> (for a bunch of other limitations), it can still be an option for\n> super big worktrees, just like index-helper being an optional\n> optimization. Hm.. hm.. if we can support lmdb as index format in\n> addition to the current format, bringing back some work from Thomas..\n> We may still need a daemon or something to deal with watchman, but\n> the\n> underlying mechanism is exactly the same... David, Junio, what do you\n> think?\n\nLMDB makes writes faster, since we only have to write the stuff that's\nchanged. That's good.\n\nReads (ignoring SHA verification) will be slightly slower (due to the\nbtree overhead). If, in general, we only had to read part of the\nindex, that would be faster. But a fair amount of git code is written\nto assume that it is reasonable to iterate over the whole index. So I\ndon't see a win from just switching to LMDB without other changes.\n\n(I guess we could parallelize index deserialization more easily with\nlmdb, but I don't know \n\nThe original intent of the index was to be a \"staging area\" to build up\na tree before committing it. This implies an alternate view of the\nindex as a diff against some (possibly-empty) tree. An index diff or\nstatus operation, then, just requires iterating over the changes. \n\nI haven't really dug into merges to understand whether it would be\nreasonable for them to use a representation like this, and what that\nwould do to speed there.\n\nIn general, git takes the commit side of the commit-diff duality. This\nis generally a good thing, because commits are simpler to reason about.\nBut I wonder if we could do better for this specific case.\n\nBut I don't think switching to LMDB would replace index-helper.\n"},{"id":"280602","messageId":"CACsJy8CT7x3p2TLvMbJ2OkNbd=PZ7bxZQJv1wSU1j_8s826J2A@mail.gmail.com","threadId":"41653","inReplyTo":"1457641341.13557.32.camel@twopensource.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-11T01:11:57Z","receivedAt":"2016-03-11T01:11:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 11, 2016 at 3:22 AM, David Turner wrote:\n> Reads (ignoring SHA verification) will be slightly slower (due to the\n> btree overhead). If, in general, we only had to read part of the\n> index, that would be faster. But a fair amount of git code is written\n> to assume that it is reasonable to iterate over the whole index. So I\n> don't see a win from just switching to LMDB without other changes.\n\nI didn't think of the btree overhead. Will need to run some test.. I\nthink lmdb just gives us pointers to mmap'd file, so an entire index\nreading can be fast. We just record where the on-disk data is. We need\nto make index-on-lmdb use native byte endian though to avoid\nconversion and actually reading all index entries because of that.\n\n> (I guess we could parallelize index deserialization more easily with\n> lmdb, but I don't know\n>\n> The original intent of the index was to be a \"staging area\" to build up\n> a tree before committing it. This implies an alternate view of the\n> index as a diff against some (possibly-empty) tree. An index diff or\n> status operation, then, just requires iterating over the changes.\n>\n> I haven't really dug into merges to understand whether it would be\n> reasonable for them to use a representation like this, and what that\n> would do to speed there.\n\nI think we keep the same internal representation of the index, a list\nof paths with extra info like stat and some flags. Minimal impacts to\nthe code base that way. There are some cases (I think) where we just\nneed to read a portion of the index (filtered by pathspec). If we\ndon't update the index afterwards, lmdb partial reads definitely help.\nI need to check if those cases are common (and worth optimizing for)\nthough.\n\n> In general, git takes the commit side of the commit-diff duality. This\n> is generally a good thing, because commits are simpler to reason about.\n> But I wonder if we could do better for this specific case.\n>\n> But I don't think switching to LMDB would replace index-helper.\n\nThe point of index-helper is reduce reading cost (let's leave watchman\nout for now). But it looks like we're not certain if using lmdb can\nreduce reading cost at lower complexity. I'll give it some thought\nover the weekend. Maybe in the end we better just go with\nindex-helper...\n-- \nDuy\n"},{"id":"280776","messageId":"CACsJy8BxDZg93ssBZGcC7Jsm_LpDgRtHUBfCm0agbvkPX9hUHw@mail.gmail.com","threadId":"41653","inReplyTo":"1457548582-28302-5-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-15T11:52:30Z","receivedAt":"2016-03-15T11:52:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 10, 2016 at 1:36 AM, David Turner wrote:\n> Git can poke the daemon to tell it to refresh the index cache, or to\n> keep it alive some more minutes via UNIX signals.\n\nThe reason I went with UNIX signals was because it made it possible to\nmake a simple GetMessage loop, the only thing I can remember from my\nWindows time, on Windows later. It sounded clever, but because this is\nmore like UDP (vs TCP) it's harder for communication. For example, we\ncan't get a confirmation after a request... UNIX sockets would be more\nnatural.\n\nSince this patch was written, watchman has gained Windows support. I\njust looked at the code, it uses named pipe on Windows. So maybe we\ncan just go with that too (if only because it has been proven working\nin practice) and we can go back to UNIX sockets on the *nix side. Too\nbad we can't just copy some functions from watchman because of license\nincompatibility. But we can leave Windows support to gfw team now, I\nthink.\n-- \nDuy\n"},{"id":"280777","messageId":"CACsJy8Cgz0zTryCCXoPMh40Rq4yOCy1L8Nih-YbKaZLyV=Gi-w@mail.gmail.com","threadId":"41653","inReplyTo":"xmqqtwkffdus.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-15T11:56:55Z","receivedAt":"2016-03-15T11:56:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 10, 2016 at 6:09 AM, Junio C Hamano wrote:\n> David Turner writes:\n>\n>> From: Nguyễn Thái Ngọc Duy \n>>\n>> Instead of reading the index from disk and worrying about disk\n>> corruption, the index is cached in memory (memory bit-flips happen\n>> too, but hopefully less often). The result is faster read. Read time\n>> is reduced by 70%.\n>>\n>> The biggest gain is not having to verify the trailing SHA-1, which\n>> takes lots of time especially on large index files. But this also\n>> opens doors for further optimiztions:\n>>\n>> - we could create an in-memory format that's essentially the memory\n>> dump of the index to eliminate most of parsing/allocation\n>> overhead. The mmap'd memory can be used straight away. Experiment\n>> [1] shows we could reduce read time by 88%.\n>>\n>> - we could cache non-index info such as name hash\n>>\n>> The shared memory's name folows the template \"git--\"\n>> where is the trailing SHA-1 of the index file. is\n>> \"index\" for cached index files (and may be \"name-hash\" for name-hash\n>> cache). If such shared memory exists, it contains the same index\n>> content as on disk. The content is already validated by the daemon and\n>> git won't validate it again (except comparing the trailing SHA-1s).\n>\n> This indeed is an interesting approach; what is not explained but\n> must be is when the on-disk index is updated to reflect the reality\n> (if I am reading the explanation and the code right, while the\n> daemon is running, its in-core cache becomes the source of truth by\n> forcing everybody's read-index-from() to go to the daemon). The\n> explanation could be \"this is only for read side, and updating the\n> index happens via the traditional 'write a new file and rename it to\n> the final place' codepath, at which time the daemon must be told to\n> re-read it.\"\n\nAnother aspect that's not mentioned is, we keep this daemon's logic as\nthin as possible. The \"brain\" stays in git. So the daemon can read and\nvalidate stuff, but that's about all it's allowed to do. It's not\nsupposed to add/create new contents. It's not even allowed to accept\ndirect updates from git.\n-- \nDuy\n"},{"id":"280778","messageId":"CACsJy8CaWFhCzrH3imz+BRMTESSmyUB4jeAaYUDNk+Tmpj-VrQ@mail.gmail.com","threadId":"41653","inReplyTo":"1457548582-28302-19-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-15T12:12:36Z","receivedAt":"2016-03-15T12:12:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 10, 2016 at 1:36 AM, David Turner wrote:\n> Introduce a new config option, indexhelper.autorun, to automatically\n> run git index-helper before starting up a builtin git command. This\n> enables users to keep index-helper running without manual\n> intervention.\n\nThis could be a problem on Windows because \"index-helper --detach\"\ndoes not work there. I have no idea how \"daemons\" are managed on\nWindows and not sure if our design is still good when such a \"daemon\"\nis added on Windows. So I'm pulling Johannes in for his opinions.\n\nBackground for Johannes. We're adding \"git index-helper\" daemon (one\nper repo) to cache the index in memory to speed up index load time\n(and in future probably name-hash too, I think it's also more often\nused on Windows because of case-insensitive fs). It also enables\nwatchman (on Windows) for faster refresh. This patch allows to start\nthe daemon automatically if it's not running. But I don't know it will\nwork ok on Windows.\n\nAssuming that \"index-helper\" service has to be installed and started\nfrom system, there can only be one service running right? This clashes\nwith the per-repo daemon design... I think it can stilf work, if the\nmain service just spawns new process, one for each repo. But again I'm\nnot sure.\n\n> Signed-off-by: David Turner \n> ---\n> git.c | 38 +++++++++++++++++++++++++++++++++++++-\n> index-helper.c | 11 ++++++++++-\n> t/t7900-index-helper.sh | 10 ++++++++++\n> 3 files changed, 57 insertions(+), 2 deletions(-)\n>\n> diff --git a/git.c b/git.c\n> index a4f6f71..ccf04ab 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -521,6 +521,40 @@ static void strip_extension(const char **argv)\n> #define strip_extension(cmd)\n> #endif\n>\n> +static int want_auto_index_helper(void)\n> +{\n> + int want = -1;\n> + const char *value = NULL;\n> + const char *key = \"indexhelper.autorun\";\n> +\n> + if (git_config_key_is_valid(key) &&\n> + !git_config_get_value(key, &value)) {\n> + int b = git_config_maybe_bool(key, value);\n> + want = b >= 0 ? b : 0;\n> + }\n> + return want;\n> +}\n> +\n> +static void maybe_run_index_helper(struct cmd_struct *cmd)\n> +{\n> + const char *argv[] = {\"git-index-helper\", \"--detach\", \"--auto\", NULL};\n> + int status;\n> +\n> + if (!(cmd->option & NEED_WORK_TREE))\n> + return;\n> +\n> + if (want_auto_index_helper() <= 0)\n> + return;\n> +\n> + trace_argv_printf(argv, \"trace: auto index-helper:\");\n> +\n> + status = run_command_v_opt(argv, RUN_SILENT_EXEC_FAILURE | RUN_CLEAN_ON_EXIT);\n> +\n> + if (status) {\n> + warning(\"You specified indexhelper.autorun, but running git-index-helper failed\");\n> + }\n> +}\n> +\n> static void handle_builtin(int argc, const char **argv)\n> {\n> const char *cmd;\n> @@ -536,8 +570,10 @@ static void handle_builtin(int argc, const char **argv)\n> }\n>\n> builtin = get_builtin(cmd);\n> - if (builtin)\n> + if (builtin) {\n> + maybe_run_index_helper(builtin);\n> exit(run_builtin(builtin, argc, argv));\n> + }\n> }\n>\n> static void execv_dashed_external(const char **argv)\n> diff --git a/index-helper.c b/index-helper.c\n> index a75da60..bc5c328 100644\n> --- a/index-helper.c\n> +++ b/index-helper.c\n> @@ -379,6 +379,7 @@ int main(int argc, char **argv)\n> int idle_in_minutes = 10, detach = 0;\n> int ignore_existing = 0;\n> int kill_existing = 0;\n> + int nongit = 0, autorun = 0;\n> const char *pid_file;\n> struct option options[] = {\n> OPT_INTEGER(0, \"exit-after\", &idle_in_minutes,\n> @@ -388,6 +389,7 @@ int main(int argc, char **argv)\n> OPT_BOOL(0, \"detach\", &detach, \"detach the process\"),\n> OPT_BOOL(0, \"ignore-existing\", &ignore_existing, \"run even if another index-helper seems to be running for this repo\"),\n> OPT_BOOL(0, \"kill\", &kill_existing, \"kill any running index-helper for this repo\"),\n> + OPT_BOOL(0, \"auto\", &autorun, \"this is an automatic run of git index-helper, so certain errors can be solved by silently exiting\"),\n> OPT_END()\n> };\n>\n> @@ -397,11 +399,18 @@ int main(int argc, char **argv)\n> if (argc == 2 && !strcmp(argv[1], \"-h\"))\n> usage_with_options(usage_text, options);\n>\n> - prefix = setup_git_directory();\n> + prefix = setup_git_directory_gently(&nongit);\n> if (parse_options(argc, (const char **)argv, prefix,\n> options, usage_text, 0))\n> die(_(\"too many arguments\"));\n>\n> + if (nongit) {\n> + if (autorun)\n> + exit(0);\n> + else\n> + die(\"Not a git repository\");\n> + }\n> +\n> if (ignore_existing && kill_existing)\n> die(_(\"--ignore-existing and --kill don't make sense together\"));\n>\n> diff --git a/t/t7900-index-helper.sh b/t/t7900-index-helper.sh\n> index 6708180..e4f9564 100755\n> --- a/t/t7900-index-helper.sh\n> +++ b/t/t7900-index-helper.sh\n> @@ -20,4 +20,14 @@ test_expect_success 'index-helper creates usable pid file and can be killed' '\n> ! kill -0 $pid\n> '\n>\n> +test_expect_success 'index-helper autorun works' '\n> + rm -f .git/index-helper.pid &&\n> + git status &&\n> + test_path_is_missing .git/index-helper.pid &&\n> + test_config indexhelper.autorun true &&\n> + git status &&\n> + test_path_is_file .git/index-helper.pid &&\n> + git index-helper --kill\n> +'\n> +\n> test_done\n> --\n> 2.4.2.767.g62658d5-twtrsrc\n>\n\n\n\n-- \nDuy\n"},{"id":"280779","messageId":"CACsJy8DGEQJmhxZpX3Zd=tGk_9T0n+ZhcaaEahaqFuh6NRvgSQ@mail.gmail.com","threadId":"41653","inReplyTo":"1457548582-28302-20-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 19/19] hack: watchman/untracked cache mashup","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-15T12:31:00Z","receivedAt":"2016-03-15T12:31:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 10, 2016 at 1:36 AM, David Turner wrote:\n> static struct watchman_query *make_query(const char *last_update)\n> @@ -60,8 +61,24 @@ static void update_index(struct index_state *istate,\n> continue;\n>\n> pos = index_name_pos(istate, wm->name, strlen(wm->name));\n> - if (pos < 0)\n> + if (pos < 0) {\n> + if (istate->untracked) {\n> + char *name = xstrdup(wm->name);\n> + char *dname = dirname(name);\n> +\n> + /*\n> + * dirname() returns '.' for the root,\n> + * but we call it ''.\n> + */\n> + if (dname[0] == '.' && dname[1] == 0)\n> + string_list_append(&istate->untracked->invalid_untracked, \"\");\n> + else\n> + string_list_append(&istate->untracked->invalid_untracked,\n> + dname);\n> + free(name);\n> + }\n> continue;\n> + }\n\nSo if we detect an updated file that's not in the index, we are\nprepared to invalidate that path, correct? We may invalidate more than\nnecessary if that's true. Imagine a.o is already ignored. If it's\nrebuilt, we should not need to update untracked cache.\n\nWhat I had in mind (and argued with watchman devs a bit [1]) was\nmaintain the file list of each clock and compare the file list of\ndifferent clocks to figure out what files are added or deleted. Then\nwe can generate invalidation list more accurately. We need to keep at\nleast one file list corresponds to the clock saved in the index. When\nwe get the refresh request, we get a new file list (with new clock),\ndo a diff then save the invalidation list as an extension. Once we\nnotice that new clock is written back in index, we can discard older\nfile lists. In theory we should not need to keep too many file lists,\nso even if one list is big, it should not be a big problem.\n\nI have a note with me about race conditions with this approach, but I\nhaven't remembered exactly why or how yet.. My recent thoughts about\nit though, are race conditions when you do \"git status\" is probably\ntolerable. After all if you're doing some I/O when you do git-status,\nyou're guaranteed to miss some updates.\n\n[1] https://github.com/facebook/watchman/issues/65\n-- \nDuy\n"},{"id":"280783","messageId":"alpine.DEB.2.20.1603151517590.4690@virtualbox","threadId":"41653","inReplyTo":"CACsJy8CaWFhCzrH3imz+BRMTESSmyUB4jeAaYUDNk+Tmpj-VrQ@mail.gmail.com","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-03-15T14:26:44Z","receivedAt":"2016-03-15T14:26:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Tue, 15 Mar 2016, Duy Nguyen wrote:\n\n> On Thu, Mar 10, 2016 at 1:36 AM, David Turner wrote:\n> > Introduce a new config option, indexhelper.autorun, to automatically\n> > run git index-helper before starting up a builtin git command. This\n> > enables users to keep index-helper running without manual\n> > intervention.\n> \n> This could be a problem on Windows because \"index-helper --detach\"\n> does not work there. I have no idea how \"daemons\" are managed on\n> Windows and not sure if our design is still good when such a \"daemon\"\n> is added on Windows. So I'm pulling Johannes in for his opinions.\n> \n> Background for Johannes. We're adding \"git index-helper\" daemon (one\n> per repo) to cache the index in memory to speed up index load time\n> (and in future probably name-hash too, I think it's also more often\n> used on Windows because of case-insensitive fs). It also enables\n> watchman (on Windows) for faster refresh. This patch allows to start\n> the daemon automatically if it's not running. But I don't know it will\n> work ok on Windows.\n> \n> Assuming that \"index-helper\" service has to be installed and started\n> from system, there can only be one service running right? This clashes\n> with the per-repo daemon design... I think it can stilf work, if the\n> main service just spawns new process, one for each repo. But again I'm\n> not sure.\n\nIf we want to run the process as a Windows service, you are correct, there\nreally can only be one. Worse: it runs with admin privileges.\n\nBut why not just keep it running as a detached process? We can run those\non Windows, and if we're opening a named pipe whose name reveals the\none-to-one mapping with the index in question, I think we are fine (read:\nwe can detect whether the process is running already).\n\nWe can even tell those processes to have a timeout, or to react to other\nsystem events.\n\nPlease note that I am *very* interested in this feature (speeding up index\noperations).\n\nCiao,\nDscho\n"},{"id":"280784","messageId":"xmqqk2l391m2.fsf@gitster.mtv.corp.google.com","threadId":"41653","inReplyTo":"CACsJy8Cgz0zTryCCXoPMh40Rq4yOCy1L8Nih-YbKaZLyV=Gi-w@mail.gmail.com","subject":"Re: [PATCH 04/19] index-helper: new daemon for caching index and related stuff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-15T15:56:05Z","receivedAt":"2016-03-15T15:56:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> Another aspect that's not mentioned is, we keep this daemon's logic as\n> thin as possible. The \"brain\" stays in git. So the daemon can read and\n> validate stuff, but that's about all it's allowed to do. It's not\n> supposed to add/create new contents. It's not even allowed to accept\n> direct updates from git.\n\nThat explanation does make the intent clear. It is a kind of design\ndecision that needs to be made early and that is hard to change\nlater (I am not saying I see the need of changing, though), so it is\nworth stating explicitly to guide future readers and updaters of the\ncode.\n\nThanks.\n"},{"id":"280886","messageId":"CACsJy8D7kg0Y2Yj=9N2uS-H6YNinD_GSkUbQpAmV0BcC77TuWw@mail.gmail.com","threadId":"41653","inReplyTo":"alpine.DEB.2.20.1603151517590.4690@virtualbox","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-16T11:37:01Z","receivedAt":"2016-03-16T11:37:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Mar 15, 2016 at 9:26 PM, Johannes Schindelin\n wrote:\n> Hi Duy,\n>\n> On Tue, 15 Mar 2016, Duy Nguyen wrote:\n>\n>> On Thu, Mar 10, 2016 at 1:36 AM, David Turner wrote:\n>> > Introduce a new config option, indexhelper.autorun, to automatically\n>> > run git index-helper before starting up a builtin git command. This\n>> > enables users to keep index-helper running without manual\n>> > intervention.\n>>\n>> This could be a problem on Windows because \"index-helper --detach\"\n>> does not work there. I have no idea how \"daemons\" are managed on\n>> Windows and not sure if our design is still good when such a \"daemon\"\n>> is added on Windows. So I'm pulling Johannes in for his opinions.\n>>\n>> Background for Johannes. We're adding \"git index-helper\" daemon (one\n>> per repo) to cache the index in memory to speed up index load time\n>> (and in future probably name-hash too, I think it's also more often\n>> used on Windows because of case-insensitive fs). It also enables\n>> watchman (on Windows) for faster refresh. This patch allows to start\n>> the daemon automatically if it's not running. But I don't know it will\n>> work ok on Windows.\n>>\n>> Assuming that \"index-helper\" service has to be installed and started\n>> from system, there can only be one service running right? This clashes\n>> with the per-repo daemon design... I think it can stilf work, if the\n>> main service just spawns new process, one for each repo. But again I'm\n>> not sure.\n>\n> If we want to run the process as a Windows service, you are correct, there\n> really can only be one. Worse: it runs with admin privileges.\n>\n> But why not just keep it running as a detached process?\n\nBecause I'm a Windows ignorant who assumed that it's the only way.\n\n> We can run those\n> on Windows, and if we're opening a named pipe whose name reveals the\n> one-to-one mapping with the index in question, I think we are fine (read:\n> we can detect whether the process is running already).\n>\n> We can even tell those processes to have a timeout, or to react to other\n> system events.\n\nSo what's you're saying is, we can make this --detach option works,\nlike how it's daemonized in *nix, right? Then we should have no\nproblem with this autorun patch. Thanks for checking.\n\n> Please note that I am *very* interested in this feature (speeding up index\n> operations).\n\nYeah I thought so ;-) I'm counting on gfw devs to complete the Windows\npart, if this series gets in.\n-- \nDuy\n"},{"id":"280887","messageId":"CACsJy8DSORe6chXkufJB8ZYxi0kG6dUUbW_Spb=60bcxETasLQ@mail.gmail.com","threadId":"41653","inReplyTo":"1457548582-28302-10-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 09/19] index-helper: add Windows support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-16T11:42:54Z","receivedAt":"2016-03-16T11:42:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"I think it's safe to drop this patch now. It's mostly to prove that it\ncould work on Windows. But I don't think it's tested a lot (even I\nonly occasionally test it under wine).\n\nOn Thu, Mar 10, 2016 at 1:36 AM, David Turner wrote:\n> From: Nguyễn Thái Ngọc Duy \n>\n> Windows supports shared memory, but the semantics is a bit different\n> than POSIX shm. The most noticeable thing is there's no way to get the\n> shared memory's size by the reader, and wrapping fstat to do that\n> would be hell. So the shm size is added near the end, hidden away from\n> shm users (storing it in headers would cause more problems with munmap,\n> storing it as a separate shm is even worse).\n>\n> PostMessage is used instead of UNIX signals for\n> notification. Lightweight (at least code-wise) on the client side.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> config.mak.uname | 2 ++\n> index-helper.c | 48 ++++++++++++++++++++++++++++\n> read-cache.c | 13 ++++++++\n> shm.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> 4 files changed, 159 insertions(+)\n>\n> diff --git a/config.mak.uname b/config.mak.uname\n> index b5108e1..49320c7 100644\n> --- a/config.mak.uname\n> +++ b/config.mak.uname\n> @@ -394,6 +394,7 @@ ifndef DEBUG\n> else\n> BASIC_CFLAGS += -Zi -MDd\n> endif\n> + PROGRAM_OBJS += index-helper.o\n> X = .exe\n> endif\n> ifeq ($(uname_S),Interix)\n> @@ -574,6 +575,7 @@ else\n> NO_CURL = YesPlease\n> endif\n> endif\n> + PROGRAM_OBJS += index-helper.o\n> endif\n> ifeq ($(uname_S),QNX)\n> COMPAT_CFLAGS += -DSA_RESTART=0\n> diff --git a/index-helper.c b/index-helper.c\n> index 4dd9656..cf26da7 100644\n> --- a/index-helper.c\n> +++ b/index-helper.c\n> @@ -155,6 +155,51 @@ static void loop(const char *pid_file, int idle_in_seconds)\n> ; /* do nothing, all is handled by signal handlers already */\n> }\n>\n> +#elif defined(GIT_WINDOWS_NATIVE)\n> +\n> +static void loop(const char *pid_file, int idle_in_seconds)\n> +{\n> + HWND hwnd;\n> + UINT_PTR timer = 0;\n> + MSG msg;\n> + HINSTANCE hinst = GetModuleHandle(NULL);\n> + WNDCLASS wc;\n> +\n> + /*\n> + * Emulate UNIX signals by sending WM_USER+x to a\n> + * window. Register window class and create a new window to\n> + * catch these messages.\n> + */\n> + memset(&wc, 0, sizeof(wc));\n> + wc.lpfnWndProc = DefWindowProc;\n> + wc.hInstance = hinst;\n> + wc.lpszClassName = \"git-index-helper\";\n> + if (!RegisterClass(&wc))\n> + die_errno(_(\"could not register new window class\"));\n> +\n> + hwnd = CreateWindow(\"git-index-helper\", pid_file,\n> + 0, 0, 0, 1, 1, NULL, NULL, hinst, NULL);\n> + if (!hwnd)\n> + die_errno(_(\"could not register new window\"));\n> +\n> + refresh(0);\n> + while (1) {\n> + timer = SetTimer(hwnd, timer, idle_in_seconds * 1000, NULL);\n> + if (!timer)\n> + die(_(\"no timer!\"));\n> + if (!GetMessage(&msg, hwnd, 0, 0) || msg.message == WM_TIMER)\n> + break;\n> + switch (msg.message) {\n> + case WM_USER:\n> + refresh(0);\n> + break;\n> + default:\n> + /* just reset the timer */\n> + break;\n> + }\n> + }\n> +}\n> +\n> #else\n>\n> static void loop(const char *pid_file, int idle_in_seconds)\n> @@ -198,6 +243,9 @@ int main(int argc, char **argv)\n> fd = hold_lock_file_for_update(&lock,\n> git_path(\"index-helper.pid\"),\n> LOCK_DIE_ON_ERROR);\n> +#ifdef GIT_WINDOWS_NATIVE\n> + strbuf_addstr(&sb, \"HWND\");\n> +#endif\n> strbuf_addf(&sb, \"%\" PRIuMAX, (uintmax_t) getpid());\n> write_in_full(fd, sb.buf, sb.len);\n> commit_lock_file(&lock);\n> diff --git a/read-cache.c b/read-cache.c\n> index 1a0ab0c..16fbdf6 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1546,6 +1546,18 @@ static void post_read_index_from(struct index_state *istate)\n> tweak_untracked_cache(istate);\n> }\n>\n> +#if defined(GIT_WINDOWS_NATIVE)\n> +static void do_poke(struct strbuf *sb, int refresh_cache)\n> +{\n> + HWND hwnd;\n> + if (!starts_with(sb->buf, \"HWND\"))\n> + return;\n> + hwnd = FindWindow(\"git-index-helper\", sb->buf);\n> + if (!hwnd)\n> + return;\n> + PostMessage(hwnd, refresh_cache ? WM_USER : WM_USER + 1, 0, 0);\n> +}\n> +#else\n> static void do_poke(struct strbuf *sb, int refresh_cache)\n> {\n> char *start = sb->buf;\n> @@ -1555,6 +1567,7 @@ static void do_poke(struct strbuf *sb, int refresh_cache)\n> return;\n> kill(pid, refresh_cache ? SIGHUP : SIGUSR1);\n> }\n> +#endif\n>\n> static void poke_daemon(struct index_state *istate,\n> const struct stat *st, int refresh_cache)\n> diff --git a/shm.c b/shm.c\n> index 4ec1a00..04d8a35 100644\n> --- a/shm.c\n> +++ b/shm.c\n> @@ -52,6 +52,102 @@ void git_shm_unlink(const char *fmt, ...)\n> shm_unlink(path);\n> }\n>\n> +#elif defined(GIT_WINDOWS_NATIVE)\n> +\n> +#define SHM_PATH_LEN 82 /* a little bit longer than POSIX because of \"Local\\\\\" */\n> +\n> +static ssize_t create_shm_map(int oflag, int perm, ssize_t length,\n> + void **mmap, int prot, int flags,\n> + const char *path, unsigned long page_size)\n> +{\n> + size_t real_length;\n> + void *last_page;\n> + HANDLE h;\n> +\n> + assert(perm == 0700);\n> + assert(oflag == (O_CREAT | O_EXCL | O_RDWR));\n> + assert(prot == (PROT_READ | PROT_WRITE));\n> + assert(flags == MAP_SHARED);\n> + assert(length >= 0);\n> +\n> + real_length = length;\n> + if (real_length % page_size)\n> + real_length += page_size - (real_length % page_size);\n> + real_length += page_size;\n> + h = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0,\n> + real_length, path);\n> + if (!h)\n> + return -1;\n> + *mmap = MapViewOfFile(h, FILE_MAP_ALL_ACCESS, 0, 0, real_length);\n> + CloseHandle(h);\n> + if (!*mmap)\n> + return -1;\n> + last_page = (unsigned char *)*mmap + real_length - page_size;\n> + *(unsigned long *)last_page = length;\n> + return length;\n> +}\n> +\n> +static ssize_t open_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n> + int prot, int flags, const char *path,\n> + unsigned long page_size)\n> +{\n> + void *last_page;\n> + HANDLE h;\n> +\n> + assert(perm == 0700);\n> + assert(oflag == O_RDONLY);\n> + assert(prot == PROT_READ);\n> + assert(flags == MAP_SHARED);\n> + assert(length <= 0);\n> +\n> + h = OpenFileMapping(FILE_MAP_READ, FALSE, path);\n> + if (!h)\n> + return -1;\n> + *mmap = MapViewOfFile(h, FILE_MAP_READ, 0, 0, 0);\n> + CloseHandle(h);\n> + if (!*mmap)\n> + return -1;\n> + if (length < 0) {\n> + MEMORY_BASIC_INFORMATION mbi;\n> + if (!VirtualQuery(*mmap, &mbi, sizeof(mbi))) {\n> + UnmapViewOfFile(*mmap);\n> + return -1;\n> + }\n> + if (mbi.RegionSize % page_size)\n> + die(\"expected size %lu to be %lu aligned\",\n> + mbi.RegionSize, page_size);\n> + last_page = (unsigned char *)*mmap + mbi.RegionSize - page_size;\n> + length = *(unsigned long *)last_page;\n> + }\n> + return length;\n> +}\n> +\n> +ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n> + int prot, int flags, const char *fmt, ...)\n> +{\n> + SYSTEM_INFO si;\n> + va_list ap;\n> + char path[SHM_PATH_LEN];\n> +\n> + GetSystemInfo(&si);\n> +\n> + strcpy(path, \"Local\\\\\");\n> + va_start(ap, fmt);\n> + vsprintf(path + strlen(path), fmt, ap);\n> + va_end(ap);\n> +\n> + if (oflag & O_CREAT)\n> + return create_shm_map(oflag, perm, length, mmap, prot,\n> + flags, path, si.dwPageSize);\n> + else\n> + return open_shm_map(oflag, perm, length, mmap, prot,\n> + flags, path, si.dwPageSize);\n> +}\n> +\n> +void git_shm_unlink(const char *fmt, ...)\n> +{\n> +}\n> +\n> #else\n>\n> ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n> --\n> 2.4.2.767.g62658d5-twtrsrc\n>\n\n\n\n-- \nDuy\n"},{"id":"280938","messageId":"1458151880.9385.1.camel@twopensource.com","threadId":"41653","inReplyTo":"alpine.DEB.2.20.1603151517590.4690@virtualbox","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-16T18:11:20Z","receivedAt":"2016-03-16T18:11:20Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2016-03-15 at 15:26 +0100, Johannes Schindelin wrote:\n> Hi Duy,\n> \n> On Tue, 15 Mar 2016, Duy Nguyen wrote:\n> \n> > On Thu, Mar 10, 2016 at 1:36 AM, David Turner <\n> > dturner@twopensource.com> wrote:\n> > > Introduce a new config option, indexhelper.autorun, to\n> > > automatically\n> > > run git index-helper before starting up a builtin git command. \n> > > This\n> > > enables users to keep index-helper running without manual\n> > > intervention.\n> > \n> > This could be a problem on Windows because \"index-helper --detach\"\n> > does not work there. I have no idea how \"daemons\" are managed on\n> > Windows and not sure if our design is still good when such a\n> > \"daemon\"\n> > is added on Windows. So I'm pulling Johannes in for his opinions.\n> > \n> > Background for Johannes. We're adding \"git index-helper\" daemon\n> > (one\n> > per repo) to cache the index in memory to speed up index load time\n> > (and in future probably name-hash too, I think it's also more often\n> > used on Windows because of case-insensitive fs). It also enables\n> > watchman (on Windows) for faster refresh. This patch allows to\n> > start\n> > the daemon automatically if it's not running. But I don't know it\n> > will\n> > work ok on Windows.\n> > \n> > Assuming that \"index-helper\" service has to be installed and\n> > started\n> > from system, there can only be one service running right? This\n> > clashes\n> > with the per-repo daemon design... I think it can stilf work, if\n> > the\n> > main service just spawns new process, one for each repo. But again\n> > I'm\n> > not sure.\n> \n> If we want to run the process as a Windows service, you are correct,\n> there\n> really can only be one. Worse: it runs with admin privileges.\n> \n> But why not just keep it running as a detached process? We can run\n> those\n> on Windows, and if we're opening a named pipe whose name reveals the\n> one-to-one mapping with the index in question, I think we are fine\n> (read:\n> we can detect whether the process is running already).\n> \n> We can even tell those processes to have a timeout, or to react to\n> other\n> system events.\n> \n> Please note that I am *very* interested in this feature (speeding up\n> index\n> operations).\n\nI don't understand what a \"detached process\" is on Windows (I have\nnever done any real Windows programming). Does that mean \"call\ndaemonize() and it'll take care of it?\" Or something else? Or should\nI just not worry about it and let you take care of it?\n\nAlso, I'll figure out how to switch to named pipes. \n"},{"id":"280941","messageId":"alpine.DEB.2.20.1603161923380.4690@virtualbox","threadId":"41653","inReplyTo":"1458151880.9385.1.camel@twopensource.com","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-03-16T18:27:56Z","receivedAt":"2016-03-16T18:27:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi David,\n\nOn Wed, 16 Mar 2016, David Turner wrote:\n\n> I don't understand what a \"detached process\" is on Windows (I have never\n> done any real Windows programming). Does that mean \"call daemonize() and\n> it'll take care of it?\" Or something else? Or should I just not worry\n> about it and let you take care of it?\n\nEvery process is detached by default, when created via CreateProcess().\nThere is no fork() followed by a daemonize(). If you want to wait for a\nprocess to finish, you have to wait for it explicitly.\n\nBut yeah, why don't you let me worry about this?\n\nI am much more concerned about concurrent accesses and the communication\nbetween the Git processes and the index-helper. Writing to the .pid file\nsounds very fragile to me, in particular when multiple processes can poke\nthe index-helper in succession and some readers are unaware that the index\nis being refreshed.\n\nThis is just a quick note, and I should go into more detail, also about\nthe topic branch my colleague Jeff Hostetler and I are working on, that is\nsimilar in spirit but wants to use a home-rolled alternative to watchman.\nI will go into more detail, but that will have to wait for tomorrow, as I\nneed to log off for the day.\n\nCiao,\nDscho\n"},{"id":"280991","messageId":"1458176165.9385.9.camel@twopensource.com","threadId":"41653","inReplyTo":"CACsJy8DGEQJmhxZpX3Zd=tGk_9T0n+ZhcaaEahaqFuh6NRvgSQ@mail.gmail.com","subject":"Re: [PATCH 19/19] hack: watchman/untracked cache mashup","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-17T00:56:05Z","receivedAt":"2016-03-17T00:56:05Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2016-03-15 at 19:31 +0700, Duy Nguyen wrote:\n> On Thu, Mar 10, 2016 at 1:36 AM, David Turner <\n> dturner@twopensource.com> wrote:\n> > static struct watchman_query *make_query(const char *last_update)\n> > @@ -60,8 +61,24 @@ static void update_index(struct index_state\n> > *istate,\n> > continue;\n> > \n> > pos = index_name_pos(istate, wm->name, strlen(wm\n> > ->name));\n> > - if (pos < 0)\n> > + if (pos < 0) {\n> > + if (istate->untracked) {\n> > + char *name = xstrdup(wm->name);\n> > + char *dname = dirname(name);\n> > +\n> > + /*\n> > + * dirname() returns '.' for the\n> > root,\n> > + * but we call it ''.\n> > + */\n> > + if (dname[0] == '.' && dname[1] ==\n> > 0)\n> > + string_list_append(&istate\n> > ->untracked->invalid_untracked, \"\");\n> > + else\n> > + string_list_append(&istate\n> > ->untracked->invalid_untracked,\n> > + dname);\n> > + free(name);\n> > + }\n> > continue;\n> > + }\n> \n> So if we detect an updated file that's not in the index, we are\n> prepared to invalidate that path, correct? We may invalidate more\n> than\n> necessary if that's true. Imagine a.o is already ignored. If it's\n> rebuilt, we should not need to update untracked cache.\n\nYes, that's true. But it would be true with the mtime system too. This\nis no worse, even if it's no better. In-tree builds are a hard case to\nsupport, and I'm totally OK with a system that encourages out-of-tree\nbuilds.\n\nWe could check if it's ignored, but then if someone changes their\ngitignore, we could be wrong.\n\nOr we could suggest that people make their watchmanignore match their\ngitignore.\n\n\n> What I had in mind (and argued with watchman devs a bit [1]) was\n> maintain the file list of each clock and compare the file list of\n> different clocks to figure out what files are added or deleted. Then\n> we can generate invalidation list more accurately. We need to keep at\n> least one file list corresponds to the clock saved in the index.\n> When\n> we get the refresh request, we get a new file list (with new clock),\n> do a diff then save the invalidation list as an extension. Once we\n> notice that new clock is written back in index, we can discard older\n> file lists. In theory we should not need to keep too many file lists,\n> so even if one list is big, it should not be a big problem.\n> \n> I have a note with me about race conditions with this approach, but I\n> haven't remembered exactly why or how yet.. My recent thoughts about\n> it though, are race conditions when you do \"git status\" is probably\n> tolerable. After all if you're doing some I/O when you do git-status,\n> you're guaranteed to miss some updates.\n> \n> [1] https://github.com/facebook/watchman/issues/65\n\nI think it would be possible to just check the UNTR extension and only\nadd a dir to it if that dir doesn't already contain (untracked) the\nfile that's being modified. Or is that also racy?\n"},{"id":"281025","messageId":"alpine.DEB.2.20.1603171309301.4690@virtualbox","threadId":"41653","inReplyTo":"CACsJy8DSORe6chXkufJB8ZYxi0kG6dUUbW_Spb=60bcxETasLQ@mail.gmail.com","subject":"Re: [PATCH 09/19] index-helper: add Windows support","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-03-17T12:18:36Z","receivedAt":"2016-03-17T12:18:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Wed, 16 Mar 2016, Duy Nguyen wrote:\n\n> I think it's safe to drop this patch now. It's mostly to prove that it\n> could work on Windows. But I don't think it's tested a lot (even I\n> only occasionally test it under wine).\n\nI agree that it is more important to get the basics right (such as:\nconcurrent access, elegant interface to watchman, pluggable backends\ndifferent than watchman, support for *other* Git clients modifying the\nindex under index-helper's feet, etc).\n\nWe will try to keep the Windows end going at the same time.\n\nPlease note that there is one very important difference between Windows vs\nLinux when it comes to shared memory: on Windows, the memory is released\nas soon as the last user goes away.\n\nCiao,\nDscho\n"},{"id":"281032","messageId":"CACsJy8BimGG6nVV2kMPG5vJqVTt2mXEtoKv_nKB4LRsVvdF1NA@mail.gmail.com","threadId":"41653","inReplyTo":"alpine.DEB.2.20.1603171309301.4690@virtualbox","subject":"Re: [PATCH 09/19] index-helper: add Windows support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-17T12:59:10Z","receivedAt":"2016-03-17T12:59:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 17, 2016 at 7:18 PM, Johannes Schindelin\n wrote:\n> Please note that there is one very important difference between Windows vs\n> Linux when it comes to shared memory: on Windows, the memory is released\n> as soon as the last user goes away.\n\nI assume that can be taken care of without changing anything else\nfundamentally, or you need to tell us now. I guess that by making\nindex-helper hold the shared memory all the time, we can avoid that.\nYou may also want to check if my solution for the lack of ftruncate on\nWindows shm is correct.\n-- \nDuy\n"},{"id":"281034","messageId":"CACsJy8AsJKmsPm8Y1LRZdmyH60n3OT5X=42RGK5GXNBDfn8j8g@mail.gmail.com","threadId":"41653","inReplyTo":"alpine.DEB.2.20.1603161923380.4690@virtualbox","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-17T13:02:55Z","receivedAt":"2016-03-17T13:02:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 17, 2016 at 1:27 AM, Johannes Schindelin\n wrote:\n> I am much more concerned about concurrent accesses and the communication\n> between the Git processes and the index-helper. Writing to the .pid file\n> sounds very fragile to me, in particular when multiple processes can poke\n> the index-helper in succession and some readers are unaware that the index\n> is being refreshed.\n\nIt's not that bad. We should have protection in place to deal with\nthis and fall back to reading directly from file when things get\nsuspicious. But I agree that sending UNIX signals (or PostMessage) is\nnot really good communication.\n-- \nDuy\n"},{"id":"281036","messageId":"CACsJy8Cx7LnPC1+17qov_pJK9OMpYtdOzS_uTBh5Fnr=A8z_OQ@mail.gmail.com","threadId":"41653","inReplyTo":"1458176165.9385.9.camel@twopensource.com","subject":"Re: [PATCH 19/19] hack: watchman/untracked cache mashup","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-17T13:06:14Z","receivedAt":"2016-03-17T13:06:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 17, 2016 at 7:56 AM, David Turner wrote:\n>> So if we detect an updated file that's not in the index, we are\n>> prepared to invalidate that path, correct? We may invalidate more\n>> than\n>> necessary if that's true. Imagine a.o is already ignored. If it's\n>> rebuilt, we should not need to update untracked cache.\n>\n> Yes, that's true. But it would be true with the mtime system too. This\n> is no worse, even if it's no better. In-tree builds are a hard case to\n> support, and I'm totally OK with a system that encourages out-of-tree\n> builds.\n>\n> We could check if it's ignored, but then if someone changes their\n> gitignore, we could be wrong.\n>\n> Or we could suggest that people make their watchmanignore match their\n> gitignore.\n\nSo your purpose is to reduce stat() on those \"quiet\" directories?\nPutting it that way, yes it's reasonable to do this way. I want to \"do\nbetter\" but we can do in steps and the first step \"no worse, no\nbetter\" is already good.\n\n>> What I had in mind (and argued with watchman devs a bit [1]) was\n>> maintain the file list of each clock and compare the file list of\n>> different clocks to figure out what files are added or deleted. Then\n>> we can generate invalidation list more accurately. We need to keep at\n>> least one file list corresponds to the clock saved in the index.\n>> When\n>> we get the refresh request, we get a new file list (with new clock),\n>> do a diff then save the invalidation list as an extension. Once we\n>> notice that new clock is written back in index, we can discard older\n>> file lists. In theory we should not need to keep too many file lists,\n>> so even if one list is big, it should not be a big problem.\n>>\n>> I have a note with me about race conditions with this approach, but I\n>> haven't remembered exactly why or how yet.. My recent thoughts about\n>> it though, are race conditions when you do \"git status\" is probably\n>> tolerable. After all if you're doing some I/O when you do git-status,\n>> you're guaranteed to miss some updates.\n>>\n>> [1] https://github.com/facebook/watchman/issues/65\n>\n> I think it would be possible to just check the UNTR extension and only\n> add a dir to it if that dir doesn't already contain (untracked) the\n> file that's being modified. Or is that also racy?\n\nSorry can't comment on this racy thing yet. Still haven't sat down,\nredrawn the timeline and retraced what I thought months ago :)\n-- \nDuy\n"},{"id":"281043","messageId":"alpine.DEB.2.20.1603171536420.4690@virtualbox","threadId":"41653","inReplyTo":"CACsJy8AsJKmsPm8Y1LRZdmyH60n3OT5X=42RGK5GXNBDfn8j8g@mail.gmail.com","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-03-17T14:43:34Z","receivedAt":"2016-03-17T14:43:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Thu, 17 Mar 2016, Duy Nguyen wrote:\n\n> On Thu, Mar 17, 2016 at 1:27 AM, Johannes Schindelin\n> wrote:\n> > I am much more concerned about concurrent accesses and the communication\n> > between the Git processes and the index-helper. Writing to the .pid file\n> > sounds very fragile to me, in particular when multiple processes can poke\n> > the index-helper in succession and some readers are unaware that the index\n> > is being refreshed.\n> \n> It's not that bad.\n\nWell, the way I read the code it is possible that:\n\n1. Git process 1 starts, reading the index\n2. Git process 2 starts, poking the index-helper\n3. The index-helper updates the .pid file (why not set a bit in the shared\n memory?) with a prefix \"W\"\n4. Git process 2 reads the .pid file and waits for the \"W\" to go away\n (what if index-helper is not fast enough to write the \"W\"?)\n5. Git process 1 access the index, happily oblivious that it is being\n updated and the data is in an inconsistent state\n\n> We should have protection in place to deal with this and fall back to\n> reading directly from file when things get suspicious.\n\nI really want to prevent that. I know of use cases where the index weighs\n300MB, and falling back to reading it directly *really* hurts.\n\n> But I agree that sending UNIX signals (or PostMessage) is not really\n> good communication.\n\nYeah, I really would like two-way communication instead. Named pipes?\nThey'd have the advantage that you could use the full path to the index as\nidentifier.\n\nThe way I read the current code, we would actually create a different\nshared memory every time the index changes because its checksum is part of\nthe shared memory's \"path\"...\n\nCiao,\nDscho\n"},{"id":"281061","messageId":"1458238119.9385.15.camel@twopensource.com","threadId":"41653","inReplyTo":"CACsJy8Cx7LnPC1+17qov_pJK9OMpYtdOzS_uTBh5Fnr=A8z_OQ@mail.gmail.com","subject":"Re: [PATCH 19/19] hack: watchman/untracked cache mashup","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-17T18:08:39Z","receivedAt":"2016-03-17T18:08:39Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-03-17 at 20:06 +0700, Duy Nguyen wrote:\n> On Thu, Mar 17, 2016 at 7:56 AM, David Turner <\n> dturner@twopensource.com> wrote:\n> > > So if we detect an updated file that's not in the index, we are\n> > > prepared to invalidate that path, correct? We may invalidate more\n> > > than\n> > > necessary if that's true. Imagine a.o is already ignored. If it's\n> > > rebuilt, we should not need to update untracked cache.\n> > \n> > Yes, that's true. But it would be true with the mtime system too.\n> > This\n> > is no worse, even if it's no better. In-tree builds are a hard\n> > case to\n> > support, and I'm totally OK with a system that encourages out-of\n> > -tree\n> > builds.\n> > \n> > We could check if it's ignored, but then if someone changes their\n> > gitignore, we could be wrong.\n> > \n> > Or we could suggest that people make their watchmanignore match\n> > their\n> > gitignore.\n> \n> So your purpose is to reduce stat() on those \"quiet\" directories?\n\nYes. Twitter's repo is perhaps somewhat unusual in that it has a very\n\"bushy\" directory structure -- tens of thousands of directories.\n"},{"id":"281064","messageId":"1458239467.9385.21.camel@twopensource.com","threadId":"41653","inReplyTo":"alpine.DEB.2.20.1603171536420.4690@virtualbox","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-17T18:31:07Z","receivedAt":"2016-03-17T18:31:07Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2016-03-17 at 15:43 +0100, Johannes Schindelin wrote:\n> Hi Duy,\n> \n> On Thu, 17 Mar 2016, Duy Nguyen wrote:\n> \n> > On Thu, Mar 17, 2016 at 1:27 AM, Johannes Schindelin\n> > wrote:\n> > > I am much more concerned about concurrent accesses and the\n> > > communication\n> > > between the Git processes and the index-helper. Writing to the\n> > > .pid file\n> > > sounds very fragile to me, in particular when multiple processes\n> > > can poke\n> > > the index-helper in succession and some readers are unaware that\n> > > the index\n> > > is being refreshed.\n> > \n> > It's not that bad.\n> \n> Well, the way I read the code it is possible that:\n> \n> 1. Git process 1 starts, reading the index\n> 2. Git process 2 starts, poking the index-helper\n> 3. The index-helper updates the .pid file (why not set a bit in the\n> shared\n> memory?) with a prefix \"W\"\n> 4. Git process 2 reads the .pid file and waits for the \"W\" to go away\n> (what if index-helper is not fast enough to write the \"W\"?)\n> 5. Git process 1 access the index, happily oblivious that it is being\n> updated and the data is in an inconsistent state\n\n\nThat's not quite how I understand it. It's more like MVCC. Writes to\nthe index go to a new index file. Index files are identified by their\nSHA. Reads from the index go into a new shm, identified by SHA. \n\nThe \"W\" is set only once -- it just means \"this index helper knows how\nto talk to watchman\". It's a compile-time option.\n\n(I'm going to change this anyway when I switch to named pipes).\n\nThe watchman data is shared independently; if it's not ready in time\n(whatever that means -- it's 1s in the current code), then read-cache\nshould fall back to brute-force checking every file.\n\n> > We should have protection in place to deal with this and fall back\n> > to\n> > reading directly from file when things get suspicious.\n> \n> I really want to prevent that. I know of use cases where the index\n> weighs\n> 300MB, and falling back to reading it directly *really* hurts.\n\n\n\n\n\n> > But I agree that sending UNIX signals (or PostMessage) is not\n> > really\n> > good communication.\n> \n> Yeah, I really would like two-way communication instead. Named pipes?\n> They'd have the advantage that you could use the full path to the\n> index as\n> identifier.\n> \n> The way I read the current code, we would actually create a different\n> shared memory every time the index changes because its checksum is\n> part of\n> the shared memory's \"path\"...\n> \n> Ciao,\n> Dscho\n"},{"id":"281102","messageId":"CACsJy8Dx4=igm3YVYkTDdRSxevDo2xRij9P5m7VPxkVrq3oq8Q@mail.gmail.com","threadId":"41653","inReplyTo":"alpine.DEB.2.20.1603171536420.4690@virtualbox","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-18T00:50:27Z","receivedAt":"2016-03-18T00:50:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 17, 2016 at 9:43 PM, Johannes Schindelin\n wrote:\n> Hi Duy,\n>\n> On Thu, 17 Mar 2016, Duy Nguyen wrote:\n>\n>> On Thu, Mar 17, 2016 at 1:27 AM, Johannes Schindelin\n>> wrote:\n>> > I am much more concerned about concurrent accesses and the communication\n>> > between the Git processes and the index-helper. Writing to the .pid file\n>> > sounds very fragile to me, in particular when multiple processes can poke\n>> > the index-helper in succession and some readers are unaware that the index\n>> > is being refreshed.\n>>\n>> It's not that bad.\n>\n> Well, the way I read the code it is possible that:\n>\n> 1. Git process 1 starts, reading the index\n> 2. Git process 2 starts, poking the index-helper\n> 3. The index-helper updates the .pid file (why not set a bit in the shared\n> memory?) with a prefix \"W\"\n> 4. Git process 2 reads the .pid file and waits for the \"W\" to go away\n> (what if index-helper is not fast enough to write the \"W\"?)\n> 5. Git process 1 access the index, happily oblivious that it is being\n> updated and the data is in an inconsistent state\n\nNo, if process 1 reads the index file, then that file will remain\nconsistent/unchanged all the time. index-helper is not allowed to\ntouch that file at all.\n\nThe process 2 gets the index content from shm (cached by the index\nhelper), verifies that it's good (with the signature at the end of the\nshm). If watchman is used, process 2 can also read the list of\nmodified files from another shm, combine it with the in-core index,\nthen write it down the normal way. Only then process 1 (or process 3)\ncan see the new index content from the file.\n\n>> We should have protection in place to deal with this and fall back to\n>> reading directly from file when things get suspicious.\n>\n> I really want to prevent that. I know of use cases where the index weighs\n> 300MB, and falling back to reading it directly *really* hurts.\n\nFor crying out loud, what do you store in that repo? What I have in\nmind for all these works are indexes in 10MB range, or maybe 50MB max.\n\nVery unscientifically, git.git index is about 274kb and contains ~3000\nentries, so 94 bytes per entry on average. With a 300MB index , the\nextrapolated number of entries is about 3 millions! At around 1\nmillion index entries, I think it's time to just use a database as\nindex.\n\n>> But I agree that sending UNIX signals (or PostMessage) is not really\n>> good communication.\n>\n> Yeah, I really would like two-way communication instead. Named pipes?\n> They'd have the advantage that you could use the full path to the index as\n> identifier.\n\nYep.\n\n> The way I read the current code, we would actually create a different\n> shared memory every time the index changes because its checksum is part of\n> the shared memory's \"path\"...\n\nYep. shm objects are \"immutable\", pretty much like git objects. But\nnow that I think of it, I don't know how cheap/expensive shm creation\noperation is on Windows.\n-- \nDuy\n"},{"id":"281132","messageId":"alpine.DEB.2.20.1603180752540.4690@virtualbox","threadId":"41653","inReplyTo":"CACsJy8Dx4=igm3YVYkTDdRSxevDo2xRij9P5m7VPxkVrq3oq8Q@mail.gmail.com","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-03-18T07:14:51Z","receivedAt":"2016-03-18T07:14:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Fri, 18 Mar 2016, Duy Nguyen wrote:\n\n> On Thu, Mar 17, 2016 at 9:43 PM, Johannes Schindelin\n> wrote:\n>\n> > I know of use cases where the index weighs 300MB, and falling back to\n> > reading it directly *really* hurts.\n> \n> For crying out loud, what do you store in that repo? What I have in\n> mind for all these works are indexes in 10MB range, or maybe 50MB max.\n\nWelcome to the real world.\n\n> Very unscientifically, git.git index is about 274kb and contains ~3000\n> entries, so 94 bytes per entry on average.\n\nIn terms of software projects' size, git.git is but a toy. Most developers\ndeal with vastly larger (and often messier) repositories. This is\nespecially true outside Open Source. Even the Linux kernel's repository is\n*tiny* compared to real-world repositories.\n\nI am sure that David could tell many a tale about repository/working\ndirectory size, too.\n\nSo yeah, this is the challenge: to make Git work at real-world scale\n(didn't we hear a lot about this at the latest Git Merge?)\n\nCiao,\nDscho\n"},{"id":"281134","messageId":"alpine.DEB.2.20.1603180815160.4690@virtualbox","threadId":"41653","inReplyTo":"CACsJy8Dx4=igm3YVYkTDdRSxevDo2xRij9P5m7VPxkVrq3oq8Q@mail.gmail.com","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-03-18T07:17:37Z","receivedAt":"2016-03-18T07:17:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Fri, 18 Mar 2016, Duy Nguyen wrote:\n\n> > Well, the way I read the code it is possible that:\n> >\n> > 1. Git process 1 starts, reading the index\n> > 2. Git process 2 starts, poking the index-helper\n> > 3. The index-helper updates the .pid file (why not set a bit in the shared\n> > memory?) with a prefix \"W\"\n> > 4. Git process 2 reads the .pid file and waits for the \"W\" to go away\n> > (what if index-helper is not fast enough to write the \"W\"?)\n> > 5. Git process 1 access the index, happily oblivious that it is being\n> > updated and the data is in an inconsistent state\n> \n> No, if process 1 reads the index file, then that file will remain\n> consistent/unchanged all the time. index-helper is not allowed to\n> touch that file at all.\n> \n> The process 2 gets the index content from shm (cached by the index\n> helper), verifies that it's good (with the signature at the end of the\n> shm). If watchman is used, process 2 can also read the list of\n> modified files from another shm, combine it with the in-core index,\n> then write it down the normal way. Only then process 1 (or process 3)\n> can see the new index content from the file.\n\nSo how do you deal with releasing the shared memory instances that are\nessentially created for every index update?\n\nCiao,\nDscho\n"},{"id":"281135","messageId":"CACsJy8Bu6QyYmJpPHpEgDx5djt=9XjXzckzJKL_mx8yHeyHdCA@mail.gmail.com","threadId":"41653","inReplyTo":"alpine.DEB.2.20.1603180815160.4690@virtualbox","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-18T07:34:49Z","receivedAt":"2016-03-18T07:34:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 18, 2016 at 2:17 PM, Johannes Schindelin\n wrote:\n> Hi Duy,\n>\n> On Fri, 18 Mar 2016, Duy Nguyen wrote:\n>\n>> > Well, the way I read the code it is possible that:\n>> >\n>> > 1. Git process 1 starts, reading the index\n>> > 2. Git process 2 starts, poking the index-helper\n>> > 3. The index-helper updates the .pid file (why not set a bit in the shared\n>> > memory?) with a prefix \"W\"\n>> > 4. Git process 2 reads the .pid file and waits for the \"W\" to go away\n>> > (what if index-helper is not fast enough to write the \"W\"?)\n>> > 5. Git process 1 access the index, happily oblivious that it is being\n>> > updated and the data is in an inconsistent state\n>>\n>> No, if process 1 reads the index file, then that file will remain\n>> consistent/unchanged all the time. index-helper is not allowed to\n>> touch that file at all.\n>>\n>> The process 2 gets the index content from shm (cached by the index\n>> helper), verifies that it's good (with the signature at the end of the\n>> shm). If watchman is used, process 2 can also read the list of\n>> modified files from another shm, combine it with the in-core index,\n>> then write it down the normal way. Only then process 1 (or process 3)\n>> can see the new index content from the file.\n>\n> So how do you deal with releasing the shared memory instances that are\n> essentially created for every index update?\n\nWhen index-helper reads the index file and realizes the file has been\nupdated (based on trailing SHA-1), it unlink()s the old shm and\nprepares new shm. If no process is accessing old shm, it's gone. If\nnot, it stays until nobody elses uses it. shm on Windows behaves the\nsame way, I believe.\n-- \nDuy\n"},{"id":"281136","messageId":"CACsJy8Amdr-2WqwYjYjyaag0jR_pq=h36QFKMk3BYQmE_A-DOw@mail.gmail.com","threadId":"41653","inReplyTo":"alpine.DEB.2.20.1603180752540.4690@virtualbox","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-18T07:44:41Z","receivedAt":"2016-03-18T07:44:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 18, 2016 at 2:14 PM, Johannes Schindelin\n wrote:\n> Hi Duy,\n>\n> On Fri, 18 Mar 2016, Duy Nguyen wrote:\n>\n>> On Thu, Mar 17, 2016 at 9:43 PM, Johannes Schindelin\n>> wrote:\n>>\n>> > I know of use cases where the index weighs 300MB, and falling back to\n>> > reading it directly *really* hurts.\n>>\n>> For crying out loud, what do you store in that repo? What I have in\n>> mind for all these works are indexes in 10MB range, or maybe 50MB max.\n>\n> Welcome to the real world.\n>\n>> Very unscientifically, git.git index is about 274kb and contains ~3000\n>> entries, so 94 bytes per entry on average.\n>\n> In terms of software projects' size, git.git is but a toy. Most developers\n> deal with vastly larger (and often messier) repositories. This is\n> especially true outside Open Source. Even the Linux kernel's repository is\n> *tiny* compared to real-world repositories.\n>\n> I am sure that David could tell many a tale about repository/working\n> directory size, too.\n\nI know a few real-world repos, decades old, but I don't remember if\nany of them reached this size. Did I get that 3 million entry number\nright? Because even my whole /home uses over 1m inodes. And whole /usr\nonly has 300k files. If the number of entries is lower, maybe there's\nsome improvement we can do to reduce index size a bit. There's some\nfast compression we can do, for starter.\n\n> So yeah, this is the challenge: to make Git work at real-world scale\n> (didn't we hear a lot about this at the latest Git Merge?)\n\nI'm all for making Junio cry by using Git for what it is (or was) not\nintended for, but this seems too much. A repo about 500k files or\nless, I think I can deal with, not those in million range.\n-- \nDuy\n"},{"id":"281153","messageId":"alpine.DEB.2.20.1603181656160.4690@virtualbox","threadId":"41653","inReplyTo":"CACsJy8Bu6QyYmJpPHpEgDx5djt=9XjXzckzJKL_mx8yHeyHdCA@mail.gmail.com","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-03-18T15:57:07Z","receivedAt":"2016-03-18T15:57:07Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Fri, 18 Mar 2016, Duy Nguyen wrote:\n\n> On Fri, Mar 18, 2016 at 2:17 PM, Johannes Schindelin\n> wrote:\n> >\n> > On Fri, 18 Mar 2016, Duy Nguyen wrote:\n> >\n> >> > Well, the way I read the code it is possible that:\n> >> >\n> >> > 1. Git process 1 starts, reading the index\n> >> > 2. Git process 2 starts, poking the index-helper\n> >> > 3. The index-helper updates the .pid file (why not set a bit in the shared\n> >> > memory?) with a prefix \"W\"\n> >> > 4. Git process 2 reads the .pid file and waits for the \"W\" to go away\n> >> > (what if index-helper is not fast enough to write the \"W\"?)\n> >> > 5. Git process 1 access the index, happily oblivious that it is being\n> >> > updated and the data is in an inconsistent state\n> >>\n> >> No, if process 1 reads the index file, then that file will remain\n> >> consistent/unchanged all the time. index-helper is not allowed to\n> >> touch that file at all.\n> >>\n> >> The process 2 gets the index content from shm (cached by the index\n> >> helper), verifies that it's good (with the signature at the end of the\n> >> shm). If watchman is used, process 2 can also read the list of\n> >> modified files from another shm, combine it with the in-core index,\n> >> then write it down the normal way. Only then process 1 (or process 3)\n> >> can see the new index content from the file.\n> >\n> > So how do you deal with releasing the shared memory instances that are\n> > essentially created for every index update?\n> \n> When index-helper reads the index file and realizes the file has been\n> updated (based on trailing SHA-1), it unlink()s the old shm and\n> prepares new shm. If no process is accessing old shm, it's gone. If\n> not, it stays until nobody elses uses it. shm on Windows behaves the\n> same way, I believe.\n\nAh. On Windows, you do not even have to unlink: once the last user's gone,\nthe shared memory's gone, too.\n\nCiao,\nDscho\n"},{"id":"281159","messageId":"1458321734.9385.26.camel@twopensource.com","threadId":"41653","inReplyTo":"CACsJy8Amdr-2WqwYjYjyaag0jR_pq=h36QFKMk3BYQmE_A-DOw@mail.gmail.com","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-18T17:22:14Z","receivedAt":"2016-03-18T17:22:14Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Fri, 2016-03-18 at 14:44 +0700, Duy Nguyen wrote:\n> > So yeah, this is the challenge: to make Git work at real-world\n> > scale\n> > (didn't we hear a lot about this at the latest Git Merge?)\n> \n> I'm all for making Junio cry by using Git for what it is (or was) not\n> intended for, but this seems too much. A repo about 500k files or\n> less, I think I can deal with, not those in million range.\n\nSad news: Facebook's repo was already getting towards that size three\nyears ago:\nhttp://comments.gmane.org/gmane.comp.version-control.git/189776\n"},{"id":"281189","messageId":"CACsJy8BiXLXuU1CNJ3sdo1JVDGYN7fQXu4G1effN+NwGAKCq_g@mail.gmail.com","threadId":"41653","inReplyTo":"1458321734.9385.26.camel@twopensource.com","subject":"Re: [PATCH 18/19] index-helper: autorun","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-03-18T23:09:43Z","receivedAt":"2016-03-18T23:09:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Mar 19, 2016 at 12:22 AM, David Turner wrote:\n> On Fri, 2016-03-18 at 14:44 +0700, Duy Nguyen wrote:\n>> > So yeah, this is the challenge: to make Git work at real-world\n>> > scale\n>> > (didn't we hear a lot about this at the latest Git Merge?)\n>>\n>> I'm all for making Junio cry by using Git for what it is (or was) not\n>> intended for, but this seems too much. A repo about 500k files or\n>> less, I think I can deal with, not those in million range.\n>\n> Sad news: Facebook's repo was already getting towards that size three\n> years ago:\n> http://comments.gmane.org/gmane.comp.version-control.git/189776\n\nI've been doing this for too long I forgot about numbers in that mail.\nThe good news is I also forgot that I tried an artificial index of 200\nMB nearly two years ago [1]. 100ms read time from that test is not\nbad. But Johannes' point stands, it will hurt if index-helper is\nbypassed at this size.\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/248771\n-- \nDuy\n"},{"id":"282070","messageId":"56FAB6B2.5000200@web.de","threadId":"41653","inReplyTo":"1457548582-28302-1-git-send-email-dturner@twopensource.com","subject":"Re: [PATCH 00/19] index-helper, watchman","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-03-29T17:09:06Z","receivedAt":"2016-03-29T17:09:06Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2016-03-09 19.36, David Turner wrote:\n> This is a rebase and extension of Duy's work on git index-helper and\n> watchman support.\n>\nSomewhere we need to tweak something:\nt7900 do all fail under Mac OS, because the index-helper is not build.\n\nThe best would be to have a precondition when running the tests ?\n\nt7900-index-helper.sh not ok 1 - index-helper smoke test\nt7900-index-helper.sh not ok 2 - index-helper creates usable pipe file and can\nbe killed\nt7900-index-helper.sh not ok 3 - index-helper will not start if already running\nt7900-index-helper.sh not ok 4 - index-helper is quiet with --autorun\nt7900-index-helper.sh not ok 5 - index-helper autorun works\n\n\nThe other thing is to enable SHM on other platforms, but first things first.\n"},{"id":"282122","messageId":"1459288280.2976.15.camel@twopensource.com","threadId":"41653","inReplyTo":"56FAB6B2.5000200@web.de","subject":"Re: [PATCH 00/19] index-helper, watchman","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2016-03-29T21:51:20Z","receivedAt":"2016-03-29T21:51:20Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2016-03-29 at 19:09 +0200, Torsten Bögershausen wrote:\n> On 2016-03-09 19.36, David Turner wrote:\n> > This is a rebase and extension of Duy's work on git index-helper\n> > and\n> > watchman support.\n> > \n> Somewhere we need to tweak something:\n> t7900 do all fail under Mac OS, because the index-helper is not\n> build.\n> \n> The best would be to have a precondition when running the tests ?\n> \n> t7900-index-helper.sh not ok 1 - index-helper smoke test\n> t7900-index-helper.sh not ok 2 - index-helper creates usable pipe\n> file and can\n> be killed\n> t7900-index-helper.sh not ok 3 - index-helper will not start if\n> already running\n> t7900-index-helper.sh not ok 4 - index-helper is quiet with -\n> -autorun\n> t7900-index-helper.sh not ok 5 - index-helper autorun works\n> \n> \n> The other thing is to enable SHM on other platforms, but first things\n> first.\n\nWill fix, thanks.\n"}]}