{"thread":{"id":"35642","subject":"[PATCH 0/6] inotify support","startedAt":"2014-01-12T11:03:36Z","lastAt":"2014-02-19T23:45:17Z","messageCount":72,"participants":["Nguyễn Thái Ngọc Duy","Jonathan Nieder","Duy Nguyen","Jeff King","Thomas Gummerer","Torsten Bögershausen","Thomas Rast","Shawn Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"233031","messageId":"1389524622-6702-1-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":null,"subject":"[PATCH 0/6] inotify support","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-12T11:03:36Z","receivedAt":"2014-01-12T11:03:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"It's been 37 weeks since Robert Zeh's attempt to bring inotify support\nto Git [1] and unless I missed some mails, no updates since. So here's\nanother attempt with my preferred approach (can't help it, playing\nwith your own ideas is more fun than improving other people's code)\n\nTo compare to Robert's approach:\n\n- This one uses UNIX datagram socket. If I read its man page right,\n unix socket respects the containing directory's permission. Which\n means on normal repos, only the user process can access. On shared\n repos, multiple users can access it. This should work on Mac.\n Windows will need a different transport.\n\n- The daemon is dumb. It passes the paths around and that's it.\n lstat() is done by git. If I design it right, there's should not be\n any race conditions that make git miss file updates.\n\n- CE_VALID is reused to avoid mass changes (granted there's other\n neat ways as well). I quite like the idea of machine-controlled\n CE_VALID.\n\ninotify support has the potential of reducing syscalls in\nread_directory() as well. I wrote about using lstat() to reduce\nreaddir() a while back, if that's implemented then inotify will fit in\nnicely.\n\nThis is just a proof of concept. I'm sure I haven't handled all error\ncases very well. The first five patches show the protocol and git\nside's changes. The last one fills inotify in.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/215820/focus=222278\n\nNguyễn Thái Ngọc Duy (6):\n read-cache: save trailing sha-1\n read-cache: new extension to mark what file is watched\n read-cache: connect to file watcher\n read-cache: get \"updated\" path list from file watcher\n read-cache: ask file watcher to watch files\n file-watcher: support inotify\n\n .gitignore | 1 +\n Makefile | 1 +\n cache.h | 4 +\n config.mak.uname | 1 +\n file-watcher.c (new) | 329 +++++++++++++++++++++++++++++++++++++++++++++++++++\n git-compat-util.h | 5 +\n pkt-line.c | 2 +-\n pkt-line.h | 2 +\n read-cache.c | 280 ++++++++++++++++++++++++++++++++++++++++++-\n wrapper.c | 27 +++++\n 10 files changed, 645 insertions(+), 7 deletions(-)\n create mode 100644 file-watcher.c\n\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"233032","messageId":"1389524622-6702-2-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389524622-6702-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 1/6] read-cache: save trailing sha-1","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-12T11:03:37Z","receivedAt":"2014-01-12T11:03:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This will be used as signature to know if the index has changed.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n read-cache.c | 7 ++++---\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ce377e1..7f7f306 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -279,6 +279,7 @@ struct index_state {\n \t\t initialized : 1;\n \tstruct hash_table name_hash;\n \tstruct hash_table dir_hash;\n+\tunsigned char sha1[20];\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 33dd676..3b6daf1 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1269,10 +1269,11 @@ struct ondisk_cache_entry_extended {\n \t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr(struct cache_header *hdr, unsigned long size)\n+static int verify_hdr(struct cache_header *hdr,\n+\t\t unsigned long size,\n+\t\t unsigned char *sha1)\n {\n \tgit_SHA_CTX c;\n-\tunsigned char sha1[20];\n \tint hdr_version;\n \n \tif (hdr->hdr_signature != htonl(CACHE_SIGNATURE))\n@@ -1461,7 +1462,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tclose(fd);\n \n \thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n+\tif (verify_hdr(hdr, mmap_size, istate->sha1) < 0)\n \t\tgoto unmap;\n \n \tistate->version = ntohl(hdr->hdr_version);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"233033","messageId":"1389524622-6702-3-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389524622-6702-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/6] read-cache: new extension to mark what file is watched","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-12T11:03:38Z","receivedAt":"2014-01-12T11:03:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If an entry is \"watched\", git lets an external program decide if the\nentry is modified or not. It's more like --assume-unchanged, but\ndesigned to be controlled by machine.\n\nWe are running out of on-disk ce_flags, so instead of extending\non-disk entry format again, \"watched\" flags are in-core only and\nstored as extension instead.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n read-cache.c | 41 ++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 41 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7f7f306..dfa8622 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -168,6 +168,7 @@ struct cache_entry {\n \n /* used to temporarily mark paths matched by pathspecs */\n #define CE_MATCHED (1 << 26)\n+#define CE_WATCHED (1 << 27)\n \n /*\n * Extended on-disk flags\ndiff --git a/read-cache.c b/read-cache.c\nindex 3b6daf1..098d3b6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -33,6 +33,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n #define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n #define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n+#define CACHE_EXT_WATCH 0x57415443\t /* \"WATC\" */\n \n struct index_state the_index;\n \n@@ -1289,6 +1290,19 @@ static int verify_hdr(struct cache_header *hdr,\n \treturn 0;\n }\n \n+static void read_watch_extension(struct index_state *istate, uint8_t *data,\n+\t\t\t\t unsigned long sz)\n+{\n+\tint i;\n+\tif ((istate->cache_nr + 7) / 8 != sz) {\n+\t\terror(\"invalid 'WATC' extension\");\n+\t\treturn;\n+\t}\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (data[i / 8] & (1 << (i % 8)))\n+\t\t\tistate->cache[i]->ce_flags |= CE_WATCHED;\n+}\n+\n static int read_index_extension(struct index_state *istate,\n \t\t\t\tconst char *ext, void *data, unsigned long sz)\n {\n@@ -1299,6 +1313,9 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_RESOLVE_UNDO:\n \t\tistate->resolve_undo = resolve_undo_read(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_WATCH:\n+\t\tread_watch_extension(istate, data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1777,7 +1794,7 @@ int write_index(struct index_state *istate, int newfd)\n {\n \tgit_SHA_CTX c;\n \tstruct cache_header hdr;\n-\tint i, err, removed, extended, hdr_version;\n+\tint i, err, removed, extended, hdr_version, has_watches = 0;\n \tstruct cache_entry **cache = istate->cache;\n \tint entries = istate->cache_nr;\n \tstruct stat st;\n@@ -1786,6 +1803,8 @@ int write_index(struct index_state *istate, int newfd)\n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n \t\t\tremoved++;\n+\t\telse if (cache[i]->ce_flags & CE_WATCHED)\n+\t\t\thas_watches++;\n \n \t\t/* reduce extended entries if possible */\n \t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n@@ -1857,6 +1876,26 @@ int write_index(struct index_state *istate, int newfd)\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (has_watches) {\n+\t\tint id, sz = (entries - removed + 7) / 8;\n+\t\tuint8_t *data = xmalloc(sz);\n+\t\tmemset(data, 0, sz);\n+\t\tfor (i = 0, id = 0; i < entries && has_watches; i++) {\n+\t\t\tstruct cache_entry *ce = cache[i];\n+\t\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\t\tcontinue;\n+\t\t\tif (ce->ce_flags & CE_WATCHED) {\n+\t\t\t\tdata[id / 8] |= 1 << (id % 8);\n+\t\t\t\thas_watches--;\n+\t\t\t}\n+\t\t\tid++;\n+\t\t}\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCH, sz) < 0\n+\t\t\t|| ce_write(&c, newfd, data, sz) < 0;\n+\t\tfree(data);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n \t\treturn -1;\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"233034","messageId":"1389524622-6702-4-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389524622-6702-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/6] read-cache: connect to file watcher","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-12T11:03:39Z","receivedAt":"2014-01-12T11:03:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This patch establishes a connection between a new file watcher daemon\nand git. Each index file may have at most one file watcher attached to\nit. The file watcher maintains a UNIX socket at\n$GIT_DIR/index.watcher. Any process that has write access to $GIT_DIR\ncan talk to the file watcher.\n\nA validation is performed after git connects to the file watcher to\nmake sure both sides have the same view. This is done by exchanging\nthe index signature (*) The file watcher keeps a copy of the signature\nlocally while git computes the signature from the index. If the\nsignatures do not match, something has gone wrong so both sides\nreinitialize wrt. to file watching: the file watcher clears all\nwatches while git clears CE_WATCHED flags.\n\nIf the signatures match, we can trust the file watcher and git can\nstart asking questions that are not important to this patch.\n\nTODO: do not let git hang if the file watcher refuses to\nanswer. Timeout and move on without file watcher support after 20ms or\nso.\n\n(*) for current index versions, the signature is the index SHA-1\ntrailer. But it could be something else (e.g. v5 does not have SHA-1\ntrailer)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n .gitignore | 1 +\n Makefile | 1 +\n cache.h | 1 +\n file-watcher.c (new) | 136 +++++++++++++++++++++++++++++++++++++++++++++++++++\n git-compat-util.h | 5 ++\n read-cache.c | 48 ++++++++++++++++++\n wrapper.c | 27 ++++++++++\n 7 files changed, 219 insertions(+)\n create mode 100644 file-watcher.c\n\ndiff --git a/.gitignore b/.gitignore\nindex b5f9def..12c78f0 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -56,6 +56,7 @@\n /git-fast-import\n /git-fetch\n /git-fetch-pack\n+/git-file-watcher\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\ndiff --git a/Makefile b/Makefile\nindex b4af1e2..ca5dc96 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -536,6 +536,7 @@ PROGRAMS += $(EXTRA_PROGRAMS)\n PROGRAM_OBJS += credential-store.o\n PROGRAM_OBJS += daemon.o\n PROGRAM_OBJS += fast-import.o\n+PROGRAM_OBJS += file-watcher.o\n PROGRAM_OBJS += http-backend.o\n PROGRAM_OBJS += imap-send.o\n PROGRAM_OBJS += sh-i18n--envsubst.o\ndiff --git a/cache.h b/cache.h\nindex dfa8622..6a182b5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -281,6 +281,7 @@ struct index_state {\n \tstruct hash_table name_hash;\n \tstruct hash_table dir_hash;\n \tunsigned char sha1[20];\n+\tint watcher;\n };\n \n extern struct index_state the_index;\ndiff --git a/file-watcher.c b/file-watcher.c\nnew file mode 100644\nindex 0000000..66b44e5\n--- /dev/null\n+++ b/file-watcher.c\n@@ -0,0 +1,136 @@\n+#include \"cache.h\"\n+#include \"sigchain.h\"\n+\n+static char index_signature[41];\n+\n+static int handle_command(int fd, char *msg, int msgsize)\n+{\n+\tstruct sockaddr_un sun;\n+\tint len;\n+\tsocklen_t socklen;\n+\tconst char *arg;\n+\n+\tsocklen = sizeof(sun);\n+\tlen = recvfrom(fd, msg, msgsize, 0, &sun, &socklen);\n+\tif (!len)\n+\t\treturn -1;\n+\tif (len == -1)\n+\t\tdie_errno(\"read\");\n+\tmsg[len] = '\\0';\n+\n+\tif ((arg = skip_prefix(msg, \"hello \"))) {\n+\t\tsendtof(fd, 0, &sun, socklen, \"hello %s\", index_signature);\n+\t\tif (!strcmp(index_signature, arg))\n+\t\t\treturn 0;\n+\t\t/*\n+\t\t * Index SHA-1 mismatch, something has gone\n+\t\t * wrong. Clean up and start over.\n+\t\t */\n+\t\tstrlcpy(index_signature, arg, sizeof(index_signature));\n+\t} else {\n+\t\tdie(\"unrecognized command %s\", msg);\n+\t}\n+\treturn 0;\n+}\n+\n+static const char *socket_path;\n+static int do_not_clean_up;\n+\n+static void cleanup(void)\n+{\n+\tif (do_not_clean_up)\n+\t\treturn;\n+\tunlink(socket_path);\n+}\n+\n+static void cleanup_on_signal(int signo)\n+{\n+\tcleanup();\n+\tsigchain_pop(signo);\n+\traise(signo);\n+}\n+\n+static void daemonize(void)\n+{\n+#ifdef NO_POSIX_GOODIES\n+\tdie(\"fork not supported on this platform\");\n+#else\n+\tswitch (fork()) {\n+\t\tcase 0:\n+\t\t\tbreak;\n+\t\tcase -1:\n+\t\t\tdie_errno(\"fork failed\");\n+\t\tdefault:\n+\t\t\tdo_not_clean_up = 1;\n+\t\t\texit(0);\n+\t}\n+\tif (setsid() == -1)\n+\t\tdie_errno(\"setsid failed\");\n+\tclose(0);\n+\tclose(1);\n+\tclose(2);\n+\tsanitize_stdfds();\n+#endif\n+}\n+\n+int main(int argc, char **argv)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct sockaddr_un sun;\n+\tstruct pollfd pfd[2];\n+\tint fd, err, nr;\n+\tint msgsize;\n+\tchar *msg;\n+\tsocklen_t vallen = sizeof(msgsize);\n+\tint no_daemon = 0;\n+\n+\tif (!strcmp(argv[1], \"--no-daemon\")) {\n+\t\tno_daemon =1;\n+\t\targv++;\n+\t\targc--;\n+\t}\n+\tif (argc < 2)\n+\t\tdie(\"insufficient arguments\");\n+\tsocket_path = argv[1];\n+\tmemset(index_signature, 0, sizeof(index_signature));\n+\tfd = socket(AF_UNIX, SOCK_DGRAM, 0);\n+\tsun.sun_family = AF_UNIX;\n+\tstrlcpy(sun.sun_path, socket_path, sizeof(sun.sun_path));\n+\tif (bind(fd, (struct sockaddr *)&sun, sizeof(sun)))\n+\t\tdie_errno(\"unable to bind to %s\", socket_path);\n+\tatexit(cleanup);\n+\tsigchain_push_common(cleanup_on_signal);\n+\n+\tif (getsockopt(fd, SOL_SOCKET, SO_SNDBUF, &msgsize, &vallen))\n+\t\tdie_errno(\"could not get SO_SNDBUF\");\n+\tmsg = xmalloc(msgsize + 1);\n+\n+\tif (!no_daemon) {\n+\t\tstrbuf_addf(&sb, \"%s.log\", socket_path);\n+\t\terr = open(sb.buf, O_CREAT | O_TRUNC | O_WRONLY, 0600);\n+\t\tif (err == -1)\n+\t\t\tdie_errno(\"unable to create %s\", sb.buf);\n+\t\tdaemonize();\n+\t\tdup2(err, 1);\n+\t\tdup2(err, 2);\n+\t\tclose(err);\n+\t}\n+\n+\tnr = 0;\n+\tpfd[nr].fd = fd;\n+\tpfd[nr++].events = POLLIN;\n+\n+\tfor (;;) {\n+\t\tif (poll(pfd, nr, -1) < 0) {\n+\t\t\tif (errno != EINTR) {\n+\t\t\t\terror(\"Poll failed, resuming: %s\", strerror(errno));\n+\t\t\t\tsleep(1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif ((pfd[0].revents & POLLIN) && handle_command(fd, msg, msgsize))\n+\t\t\tbreak;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b73916b..c119a94 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -536,6 +536,11 @@ extern void *xcalloc(size_t nmemb, size_t size);\n extern void *xmmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n extern ssize_t xread(int fd, void *buf, size_t len);\n extern ssize_t xwrite(int fd, const void *buf, size_t len);\n+extern ssize_t writef(int fd, const char *fmt, ...)\n+\t__attribute__((format (printf, 2, 3)));\n+extern ssize_t sendtof(int sockfd, int flags, const void *dest_addr,\n+\t\t socklen_t addrlen, const char *fmt, ...)\n+\t__attribute__((format (printf, 5, 6)));\n extern int xdup(int fd);\n extern FILE *xfdopen(int fd, const char *mode);\n extern int xmkstemp(char *template);\ndiff --git a/read-cache.c b/read-cache.c\nindex 098d3b6..506d488 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1443,6 +1443,49 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static void connect_watcher(struct index_state *istate, const char *path)\n+{\n+\tstruct stat st;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint i;\n+\n+\tstrbuf_addf(&sb, \"%s.watcher\", path);\n+\tif (!stat(sb.buf, &st) && S_ISSOCK(st.st_mode)) {\n+\t\tstruct sockaddr_un sun;\n+\t\tistate->watcher = socket(AF_UNIX, SOCK_DGRAM, 0);\n+\t\tsun.sun_family = AF_UNIX;\n+\t\tstrlcpy(sun.sun_path, sb.buf, sizeof(sun.sun_path));\n+\t\tif (connect(istate->watcher, (struct sockaddr *)&sun, sizeof(sun))) {\n+\t\t\tperror(\"connect\");\n+\t\t\tclose(istate->watcher);\n+\t\t\tistate->watcher = -1;\n+\t\t}\n+\t\tsprintf(sun.sun_path, \"%c%\"PRIuMAX, 0, (uintmax_t)getpid());\n+\t\tbind(istate->watcher, (struct sockaddr *)&sun, sizeof(sun));\n+\t} else\n+\t\tistate->watcher = -1;\n+\tstrbuf_release(&sb);\n+\tif (istate->watcher != -1) {\n+\t\tchar line[1024];\n+\t\tint len;\n+\t\tstrbuf_addf(&sb, \"hello %s\", sha1_to_hex(istate->sha1));\n+\t\twrite(istate->watcher, sb.buf, sb.len);\n+\t\tlen = read(istate->watcher, line, sizeof(line) - 1);\n+\t\tif (len > 0) {\n+\t\t\tline[len] = '\\0';\n+\t\t\tif (!strcmp(sb.buf, line))\n+\t\t\t\treturn; /* good */\n+\t\t}\n+\t}\n+\n+\t/* No the file watcher is out of date, clear everything */\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHED) {\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_WATCHED;\n+\t\t\tistate->cache_changed = 1;\n+\t\t}\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n@@ -1528,6 +1571,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsrc_offset += extsize;\n \t}\n \tmunmap(mmap, mmap_size);\n+\tconnect_watcher(istate, path);\n \treturn istate->cache_nr;\n \n unmap:\n@@ -1557,6 +1601,10 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->watcher != -1) {\n+\t\tclose(istate->watcher);\n+\t\tistate->watcher = -1;\n+\t}\n \treturn 0;\n }\n \ndiff --git a/wrapper.c b/wrapper.c\nindex 0cc5636..29e3b35 100644\n--- a/wrapper.c\n+++ b/wrapper.c\n@@ -455,3 +455,30 @@ struct passwd *xgetpwuid_self(void)\n \t\t errno ? strerror(errno) : _(\"no such user\"));\n \treturn pw;\n }\n+\n+ssize_t writef(int fd, const char *fmt, ...)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\tva_list ap;\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tva_end(ap);\n+\tret = write(fd, sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+ssize_t sendtof(int sockfd, int flags, const void *dest_addr, socklen_t addrlen,\n+\t\tconst char *fmt, ...)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\tva_list ap;\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tva_end(ap);\n+\tret = sendto(sockfd, sb.buf, sb.len, flags, dest_addr, addrlen);\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"233035","messageId":"1389524622-6702-5-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389524622-6702-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 4/6] read-cache: get \"updated\" path list from file watcher","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-12T11:03:40Z","receivedAt":"2014-01-12T11:03:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"A new command is added to file watcher to send back the list of\nupdated files to git. These entries will have CE_WATCHED removed. The\nremaining CE_WATCHED entries will have CE_VALID set (i.e. no changes\nand no lstat either).\n\nThe file watcher keeps reporting the same \"updated\" list until it\nreceives \"forget\" commands, which should only be issued after the\nupdated index is written down. This ensures that if git crashes half\nway before it could update the index (or multiple processes is reading\nthe same index), \"updated\" info is not lost. After the index is\nupdated (e.g. in this case because of toggling CE_WATCHED bits), git\nsends the new index signature to the file watcher.\n\nThe file watcher does not cache stat info and send back to git. Its\nmain purpose is to reduce lstat on most untouched files, not to\ncompletely eliminate lstat.\n\nOne can see that, assuming CE_WATCHED is magically set in some\nentries, they will be all cleared over the time and we need to do\nlstat on all entries. We haven't talked about how CE_WATCHED is set\nyet. More to come later.\n\nTODO: get as many paths as possible in one packet to reduce round\ntrips\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n file-watcher.c | 35 +++++++++++++++++++---\n read-cache.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n 3 files changed, 119 insertions(+), 8 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 6a182b5..2eddc1e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -282,6 +282,7 @@ struct index_state {\n \tstruct hash_table dir_hash;\n \tunsigned char sha1[20];\n \tint watcher;\n+\tstruct string_list *updated_entries;\n };\n \n extern struct index_state the_index;\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 66b44e5..6aeed4d 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -1,7 +1,16 @@\n #include \"cache.h\"\n #include \"sigchain.h\"\n+#include \"string-list.h\"\n \n static char index_signature[41];\n+static struct string_list updated = STRING_LIST_INIT_DUP;\n+static int updated_sorted;\n+\n+static void reset(const char *sig)\n+{\n+\tstring_list_clear(&updated, 0);\n+\tstrlcpy(index_signature, sig, sizeof(index_signature));\n+}\n \n static int handle_command(int fd, char *msg, int msgsize)\n {\n@@ -22,10 +31,28 @@ static int handle_command(int fd, char *msg, int msgsize)\n \t\tsendtof(fd, 0, &sun, socklen, \"hello %s\", index_signature);\n \t\tif (!strcmp(index_signature, arg))\n \t\t\treturn 0;\n-\t\t/*\n-\t\t * Index SHA-1 mismatch, something has gone\n-\t\t * wrong. Clean up and start over.\n-\t\t */\n+\t\treset(arg);\n+\t} else if ((arg = skip_prefix(msg, \"reset \"))) {\n+\t\treset(arg);\n+\t} else if (!strcmp(msg, \"status\")) {\n+\t\tint i;\n+\t\tfor (i = 0; i < updated.nr; i++)\n+\t\t\tsendto(fd, updated.items[i].string,\n+\t\t\t strlen(updated.items[i].string),\n+\t\t\t 0, &sun, socklen);\n+\t\tsendtof(fd, 0, &sun, socklen, \"%c\", 0);\n+\t} else if ((arg = skip_prefix(msg, \"forget \"))) {\n+\t\tstruct string_list_item *item;\n+\t\tif (!updated_sorted) {\n+\t\t\tsort_string_list(&updated);\n+\t\t\tupdated_sorted = 1;\n+\t\t}\n+\t\titem = string_list_lookup(&updated, arg);\n+\t\tif (item)\n+\t\t\tunsorted_string_list_delete_item(&updated,\n+\t\t\t\t\t\t\t item - updated.items,\n+\t\t\t\t\t\t\t 0);\n+\t} else if ((arg = skip_prefix(msg, \"bye \"))) {\n \t\tstrlcpy(index_signature, arg, sizeof(index_signature));\n \t} else {\n \t\tdie(\"unrecognized command %s\", msg);\ndiff --git a/read-cache.c b/read-cache.c\nindex 506d488..caa2298 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1443,6 +1443,55 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static void update_watched_files(struct index_state *istate)\n+{\n+\tint i;\n+\tif (istate->watcher == -1)\n+\t\treturn;\n+\tif (writef(istate->watcher, \"status\") < 0)\n+\t\tgoto failed;\n+\tfor (;;) {\n+\t\tchar line[1024];\n+\t\tint len;\n+\t\tlen = read(istate->watcher, line, sizeof(line) - 1);\n+\t\tif (len <= 0)\n+\t\t\tgoto failed;\n+\t\tline[len] = '\\0';\n+\t\tif (len == 1 && line[0] == '\\0')\n+\t\t\tbreak;\n+\t\ti = index_name_pos(istate, line, len);\n+\t\tif (i < 0)\n+\t\t\tcontinue;\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHED) {\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_WATCHED;\n+\t\t\tistate->cache_changed = 1;\n+\t\t}\n+\t\tif (!istate->updated_entries) {\n+\t\t\tstruct string_list *sl;\n+\t\t\tsl = xmalloc(sizeof(*sl));\n+\t\t\tmemset(sl, 0, sizeof(*sl));\n+\t\t\tsl->strdup_strings = 1;\n+\t\t\tistate->updated_entries = sl;\n+\t\t}\n+\t\tstring_list_append(istate->updated_entries, line);\n+\t}\n+\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHED)\n+\t\t\tistate->cache[i]->ce_flags |= CE_VALID;\n+\treturn;\n+failed:\n+\tif (istate->updated_entries) {\n+\t\tstring_list_clear(istate->updated_entries, 0);\n+\t\tfree(istate->updated_entries);\n+\t\tistate->updated_entries = NULL;\n+\t}\n+\twritef(istate->watcher, \"reset %s\", sha1_to_hex(istate->sha1));\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tistate->cache[i]->ce_flags &= ~CE_WATCHED;\n+\tistate->cache_changed = 1;\n+}\n+\n static void connect_watcher(struct index_state *istate, const char *path)\n {\n \tstruct stat st;\n@@ -1473,8 +1522,10 @@ static void connect_watcher(struct index_state *istate, const char *path)\n \t\tlen = read(istate->watcher, line, sizeof(line) - 1);\n \t\tif (len > 0) {\n \t\t\tline[len] = '\\0';\n-\t\t\tif (!strcmp(sb.buf, line))\n+\t\t\tif (!strcmp(sb.buf, line)) {\n+\t\t\t\tupdate_watched_files(istate);\n \t\t\t\treturn; /* good */\n+\t\t\t}\n \t\t}\n \t}\n \n@@ -1486,6 +1537,20 @@ static void connect_watcher(struct index_state *istate, const char *path)\n \t\t}\n }\n \n+static void farewell_watcher(struct index_state *istate,\n+\t\t\t const unsigned char *sha1)\n+{\n+\tif (istate->watcher == -1)\n+\t\treturn;\n+\tif (istate->updated_entries) {\n+\t\tint i;\n+\t\tfor (i = 0; i < istate->updated_entries->nr; i++)\n+\t\t\twritef(istate->watcher, \"forget %s\",\n+\t\t\t istate->updated_entries->items[i].string);\n+\t}\n+\twritef(istate->watcher, \"bye %s\", sha1_to_hex(sha1));\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n@@ -1605,6 +1670,11 @@ int discard_index(struct index_state *istate)\n \t\tclose(istate->watcher);\n \t\tistate->watcher = -1;\n \t}\n+\tif (istate->updated_entries) {\n+\t\tstring_list_clear(istate->updated_entries, 0);\n+\t\tfree(istate->updated_entries);\n+\t\tistate->updated_entries = NULL;\n+\t}\n \treturn 0;\n }\n \n@@ -1665,7 +1735,7 @@ static int write_index_ext_header(git_SHA_CTX *context, int fd,\n \t\t(ce_write(context, fd, &sz, 4) < 0)) ? -1 : 0;\n }\n \n-static int ce_flush(git_SHA_CTX *context, int fd)\n+static int ce_flush(git_SHA_CTX *context, int fd, unsigned char *sha1)\n {\n \tunsigned int left = write_buffer_len;\n \n@@ -1683,6 +1753,8 @@ static int ce_flush(git_SHA_CTX *context, int fd)\n \n \t/* Append the SHA1 signature at the end */\n \tgit_SHA1_Final(write_buffer + left, context);\n+\tif (sha1)\n+\t\thashcpy(sha1, write_buffer + left);\n \tleft += 20;\n \treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n }\n@@ -1847,12 +1919,22 @@ int write_index(struct index_state *istate, int newfd)\n \tint entries = istate->cache_nr;\n \tstruct stat st;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n+\tunsigned char sha1[20];\n \n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n \t\t\tremoved++;\n-\t\telse if (cache[i]->ce_flags & CE_WATCHED)\n+\t\telse if (cache[i]->ce_flags & CE_WATCHED) {\n+\t\t\t/*\n+\t\t\t * CE_VALID when used with CE_WATCHED is not\n+\t\t\t * supposed to be persistent. Next time git\n+\t\t\t * runs, if this entry is still watched and\n+\t\t\t * nothing has changed, CE_VALID will be\n+\t\t\t * reinstated.\n+\t\t\t */\n+\t\t\tcache[i]->ce_flags &= ~CE_VALID;\n \t\t\thas_watches++;\n+\t\t}\n \n \t\t/* reduce extended entries if possible */\n \t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n@@ -1945,8 +2027,9 @@ int write_index(struct index_state *istate, int newfd)\n \t\t\treturn -1;\n \t}\n \n-\tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n+\tif (ce_flush(&c, newfd, sha1) || fstat(newfd, &st))\n \t\treturn -1;\n+\tfarewell_watcher(istate, sha1);\n \tistate->timestamp.sec = (unsigned int)st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \treturn 0;\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"233036","messageId":"1389524622-6702-6-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389524622-6702-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 5/6] read-cache: ask file watcher to watch files","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-12T11:03:41Z","receivedAt":"2014-01-12T11:03:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We want to watch files that are never changed because lstat() on those\nfiles is a wasted effort. So we sort unwatched files by date and start\nadding them to the file watcher until it barfs (e.g. hits inotify\nlimit). Recently updated entries are also excluded from watch list.\nCE_VALID is used in combination with CE_WATCHED. Those entries that\nhave CE_VALID already set will never be watched.\n\nWe send as many paths as possible in one packet in pkt-line\nformat. For small projects like git, all entries can be packed in one\npacket. For large projects like webkit (182k entries) it takes two\npackets. We may do prefix compression as well to send more in fewer\npackets..\n\nThe file watcher replies how many entries it can watch (because at\nleast inotify has system limits).\n\nNote that we still do lstat() on these new watched files because they\ncould have changed before the file watcher could watch them. Watched\nfiles may only skip lstat() at the next git run.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher.c | 27 ++++++++++++++++\n pkt-line.c | 2 +-\n pkt-line.h | 2 ++\n read-cache.c | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 127 insertions(+), 1 deletion(-)\n\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 6aeed4d..35781fa 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -1,17 +1,41 @@\n #include \"cache.h\"\n #include \"sigchain.h\"\n #include \"string-list.h\"\n+#include \"pkt-line.h\"\n \n static char index_signature[41];\n static struct string_list updated = STRING_LIST_INIT_DUP;\n static int updated_sorted;\n \n+static int watch_path(char *path)\n+{\n+\treturn -1;\n+}\n+\n static void reset(const char *sig)\n {\n \tstring_list_clear(&updated, 0);\n \tstrlcpy(index_signature, sig, sizeof(index_signature));\n }\n \n+static void watch_paths(char *buf, int maxlen,\n+\t\t\tint fd, struct sockaddr *sock,\n+\t\t\tsocklen_t socklen)\n+{\n+\tchar *end = buf + maxlen;\n+\tint n, ret, len;\n+\tfor (n = ret = 0; buf < end && !ret; buf += len) {\n+\t\tchar ch;\n+\t\tlen = packet_length(buf);\n+\t\tch = buf[len];\n+\t\tbuf[len] = '\\0';\n+\t\tif (!(ret = watch_path(buf + 4)))\n+\t\t\tn++;\n+\t\tbuf[len] = ch;\n+\t}\n+\tsendtof(fd, 0, sock, socklen, \"fine %d\", n);\n+}\n+\n static int handle_command(int fd, char *msg, int msgsize)\n {\n \tstruct sockaddr_un sun;\n@@ -41,6 +65,9 @@ static int handle_command(int fd, char *msg, int msgsize)\n \t\t\t strlen(updated.items[i].string),\n \t\t\t 0, &sun, socklen);\n \t\tsendtof(fd, 0, &sun, socklen, \"%c\", 0);\n+\t} else if (starts_with(msg, \"watch \")) {\n+\t\twatch_paths(msg + 6, len - 6,\n+\t\t\t fd, (struct sockaddr *)&sun, socklen);\n \t} else if ((arg = skip_prefix(msg, \"forget \"))) {\n \t\tstruct string_list_item *item;\n \t\tif (!updated_sorted) {\ndiff --git a/pkt-line.c b/pkt-line.c\nindex bc63b3b..b5af84e 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -135,7 +135,7 @@ static int get_packet_data(int fd, char **src_buf, size_t *src_size,\n \treturn ret;\n }\n \n-static int packet_length(const char *linelen)\n+int packet_length(const char *linelen)\n {\n \tint n;\n \tint len = 0;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 0a838d1..40470b9 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -75,6 +75,8 @@ char *packet_read_line(int fd, int *size);\n */\n char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n \n+int packet_length(const char *linelen);\n+\n #define DEFAULT_PACKET_MAX 1000\n #define LARGE_PACKET_MAX 65520\n extern char packet_buffer[LARGE_PACKET_MAX];\ndiff --git a/read-cache.c b/read-cache.c\nindex caa2298..839fd7c 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -14,6 +14,7 @@\n #include \"resolve-undo.h\"\n #include \"strbuf.h\"\n #include \"varint.h\"\n+#include \"pkt-line.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int really);\n \n@@ -1537,6 +1538,90 @@ static void connect_watcher(struct index_state *istate, const char *path)\n \t\t}\n }\n \n+static int sort_by_date(const void *a_, const void *b_)\n+{\n+\tconst struct cache_entry *a = *(const struct cache_entry **)a_;\n+\tconst struct cache_entry *b = *(const struct cache_entry **)b_;\n+\tuint32_t seca = a->ce_stat_data.sd_mtime.sec;\n+\tuint32_t secb = b->ce_stat_data.sd_mtime.sec;\n+\treturn seca - secb;\n+}\n+\n+static inline int ce_watchable(struct cache_entry *ce, time_t now)\n+{\n+\treturn !(ce->ce_flags & CE_WATCHED) &&\n+\t\t!(ce->ce_flags & CE_VALID) &&\n+\t\t(ce->ce_stat_data.sd_mtime.sec + 1800 < now);\n+}\n+\n+static int do_watch_entries(struct index_state *istate,\n+\t\t\t struct cache_entry **cache,\n+\t\t\t struct strbuf *sb, int start, int now)\n+{\n+\tchar line[1024];\n+\tint i, len;\n+\n+\twrite(istate->watcher, sb->buf, sb->len);\n+\tlen = read(istate->watcher, line, sizeof(line) - 1);\n+\tif (len <= 0)\n+\t\treturn -1;\n+\tline[len] = '\\0';\n+\tif (starts_with(line, \"fine \")) {\n+\t\tchar *end;\n+\t\tlong n = strtoul(line + 5, &end, 10);\n+\t\tif (end != line + len)\n+\t\t\treturn -1;\n+\t\tfor (i = 0; i < n; i++)\n+\t\t\tcache[start + i]->ce_flags |= CE_WATCHED;\n+\t\tistate->cache_changed = 1;\n+\t\tif (i != now)\n+\t\t\treturn -1;\n+\t} else\n+\t\treturn -1;\n+\tstart = i;\n+\tstrbuf_reset(sb);\n+\tstrbuf_addstr(sb, \"watch \");\n+\treturn 0;\n+}\n+\n+static void watch_entries(struct index_state *istate)\n+{\n+\tint i, start, nr;\n+\tstruct cache_entry **sorted;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint val;\n+\tsocklen_t vallen = sizeof(val);\n+\ttime_t now = time(NULL);\n+\n+\tif (istate->watcher == -1)\n+\t\treturn;\n+\tfor (i = nr = 0; i < istate->cache_nr; i++)\n+\t\tif (ce_watchable(istate->cache[i], now))\n+\t\t\tnr++;\n+\tif (nr < 50)\n+\t\treturn;\n+\tsorted = xmalloc(sizeof(*sorted) * nr);\n+\tfor (i = nr = 0; i < istate->cache_nr; i++)\n+\t\tif (ce_watchable(istate->cache[i], now))\n+\t\t\tsorted[nr++] = istate->cache[i];\n+\n+\tgetsockopt(istate->watcher, SOL_SOCKET, SO_SNDBUF, &val, &vallen);\n+\tstrbuf_grow(&sb, val);\n+\tstrbuf_addstr(&sb, \"watch \");\n+\n+\tqsort(sorted, nr, sizeof(*sorted), sort_by_date);\n+\tfor (i = start = 0; i < nr; i++) {\n+\t\tif (sb.len + 4 + ce_namelen(sorted[i]) >= val &&\n+\t\t do_watch_entries(istate, sorted, &sb, start, i))\n+\t\t\tbreak;\n+\t\tpacket_buf_write(&sb, \"%s\", sorted[i]->name);\n+\t}\n+\tif (i == nr && start < i)\n+\t\tdo_watch_entries(istate, sorted, &sb, start, i);\n+\tstrbuf_release(&sb);\n+\tfree(sorted);\n+}\n+\n static void farewell_watcher(struct index_state *istate,\n \t\t\t const unsigned char *sha1)\n {\n@@ -1637,6 +1722,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t}\n \tmunmap(mmap, mmap_size);\n \tconnect_watcher(istate, path);\n+\twatch_entries(istate);\n \treturn istate->cache_nr;\n \n unmap:\n@@ -1933,6 +2019,17 @@ int write_index(struct index_state *istate, int newfd)\n \t\t\t * reinstated.\n \t\t\t */\n \t\t\tcache[i]->ce_flags &= ~CE_VALID;\n+\t\t\t/*\n+\t\t\t * We may set CE_WATCHED (but not CE_VALID)\n+\t\t\t * early when refresh has not been done\n+\t\t\t * yet. At that time we had no idea if the\n+\t\t\t * entry may have been updated. If it has\n+\t\t\t * been, remove CE_WATCHED so CE_VALID won't\n+\t\t\t * incorrectly be set next time if the file\n+\t\t\t * watcher reports no changes.\n+\t\t\t */\n+\t\t\tif (!ce_uptodate(cache[i]))\n+\t\t\t\tcache[i]->ce_flags &= ~CE_WATCHED;\n \t\t\thas_watches++;\n \t\t}\n \n-- \n1.8.5.2.240.g8478abd\n"},{"id":"233037","messageId":"1389524622-6702-7-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389524622-6702-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 6/6] file-watcher: support inotify","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-12T11:03:42Z","receivedAt":"2014-01-12T11:03:42Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\"git diff\" on webkit:\n\n no file watcher 1st run subsequent runs\nreal 0m1.361s 0m1.445s 0m0.691s\nuser 0m0.889s 0m0.940s 0m0.649s\nsys 0m0.469s 0m0.495s 0m0.040s\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n config.mak.uname | 1 +\n file-watcher.c | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 140 insertions(+)\n\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 82d549e..603890d 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -33,6 +33,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_PATHS_H = YesPlease\n \tLIBC_CONTAINS_LIBINTL = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tBASIC_CFLAGS += -DHAVE_INOTIFY\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tNO_STRLCPY = YesPlease\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 35781fa..1512b46 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -3,17 +3,140 @@\n #include \"string-list.h\"\n #include \"pkt-line.h\"\n \n+#ifdef HAVE_INOTIFY\n+#include \n+#endif\n+\n static char index_signature[41];\n static struct string_list updated = STRING_LIST_INIT_DUP;\n static int updated_sorted;\n \n+#ifdef HAVE_INOTIFY\n+\n+static struct string_list watched_dirs = STRING_LIST_INIT_DUP;\n+static int watched_dirs_sorted;\n+static int inotify_fd;\n+\n+struct dir_info {\n+\tint wd;\n+\tstruct string_list names;\n+\tint names_sorted;\n+};\n+\n+static int handle_inotify(int fd)\n+{\n+\tchar buf[sizeof(struct inotify_event) + NAME_MAX + 1];\n+\tstruct inotify_event *event;\n+\tstruct dir_info *dir;\n+\tstruct string_list_item *item;\n+\tint i;\n+\tint len = read(fd, buf, sizeof(buf));\n+\tif (len < 0)\n+\t\treturn -1;\n+\tevent = (struct inotify_event *)buf;\n+\n+\tif (len <= sizeof(struct inotify_event))\n+\t\treturn 0;\n+\n+\tfor (i = 0; i < watched_dirs.nr; i++) {\n+\t\tstruct dir_info *dir = watched_dirs.items[i].util;\n+\t\tif (dir->wd == event->wd)\n+\t\t\tbreak;\n+\t}\n+\tif (i == watched_dirs.nr)\n+\t\treturn 0;\n+\tdir = watched_dirs.items[i].util;\n+\n+\tif (!dir->names_sorted) {\n+\t\tsort_string_list(&dir->names);\n+\t\tdir->names_sorted = 1;\n+\t}\n+\titem = string_list_lookup(&dir->names, event->name);\n+\tif (item) {\n+\t\tif (!strcmp(watched_dirs.items[i].string, \".\"))\n+\t\t\tstring_list_append(&updated, event->name);\n+\t\telse {\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\t\tstrbuf_addf(&sb, \"%s/%s\", watched_dirs.items[i].string,\n+\t\t\t\t item->string);\n+\t\t\tstring_list_append(&updated, sb.buf);\n+\t\t\tupdated_sorted = 0;\n+\t\t\tstrbuf_release(&sb);\n+\t\t}\n+\n+\t\tunsorted_string_list_delete_item(&dir->names,\n+\t\t\t\t\t\t item - dir->names.items, 0);\n+\t\tif (dir->names.nr == 0) {\n+\t\t\tinotify_rm_watch(inotify_fd, dir->wd);\n+\t\t\tunsorted_string_list_delete_item(&watched_dirs, i, 1);\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int watch_path(char *path)\n+{\n+\tstruct string_list_item *item;\n+\tchar *sep = strrchr(path, '/');\n+\tstruct dir_info *dir;\n+\tconst char *dirname = \".\";\n+\n+\tif (sep) {\n+\t\t*sep = '\\0';\n+\t\tdirname = path;\n+\t}\n+\n+\tif (!watched_dirs_sorted) {\n+\t\tsort_string_list(&watched_dirs);\n+\t\twatched_dirs_sorted = 1;\n+\t}\n+\titem = string_list_lookup(&watched_dirs, dirname);\n+\tif (!item) {\n+\t\tint ret = inotify_add_watch(inotify_fd, dirname,\n+\t\t\t\t\t IN_ATTRIB | IN_DELETE | IN_MODIFY |\n+\t\t\t\t\t IN_MOVED_FROM | IN_MOVED_TO);\n+\t\tif (ret < 0)\n+\t\t\treturn -1;\n+\t\tdir = xmalloc(sizeof(*dir));\n+\t\tmemset(dir, 0, sizeof(*dir));\n+\t\tdir->wd = ret;\n+\t\tdir->names.strdup_strings = 1;\n+\t\titem = string_list_append(&watched_dirs, dirname);\n+\t\titem->util = dir;\n+\t}\n+\tdir = item->util;\n+\tstring_list_append(&dir->names, sep ? sep + 1 : path);\n+\tdir->names_sorted = 0;\n+\treturn 0;\n+}\n+\n+static void reset_watches(void)\n+{\n+\tint i;\n+\tfor (i = 0; i < watched_dirs.nr; i++) {\n+\t\tstruct dir_info *dir = watched_dirs.items[i].util;\n+\t\tinotify_rm_watch(inotify_fd, dir->wd);\n+\t\tstring_list_clear(&dir->names, 0);\n+\t}\n+\tstring_list_clear(&watched_dirs, 1);\n+}\n+\n+#else\n+\n static int watch_path(char *path)\n {\n \treturn -1;\n }\n \n+static void reset_watches(void)\n+{\n+}\n+\n+#endif\n+\n static void reset(const char *sig)\n {\n+\treset_watches();\n \tstring_list_clear(&updated, 0);\n \tstrlcpy(index_signature, sig, sizeof(index_signature));\n }\n@@ -155,6 +278,14 @@ int main(int argc, char **argv)\n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n \n+#ifdef HAVE_INOTIFY\n+\tinotify_fd = inotify_init();\n+\tif (inotify_fd < 0)\n+\t\tdie_errno(\"unable to initialize inotify\");\n+#else\n+\tdie(\"no file watching mechanism is supported\");\n+#endif\n+\n \tif (getsockopt(fd, SOL_SOCKET, SO_SNDBUF, &msgsize, &vallen))\n \t\tdie_errno(\"could not get SO_SNDBUF\");\n \tmsg = xmalloc(msgsize + 1);\n@@ -173,6 +304,10 @@ int main(int argc, char **argv)\n \tnr = 0;\n \tpfd[nr].fd = fd;\n \tpfd[nr++].events = POLLIN;\n+#ifdef HAVE_INOTIFY\n+\tpfd[nr].fd = inotify_fd;\n+\tpfd[nr++].events = POLLIN;\n+#endif\n \n \tfor (;;) {\n \t\tif (poll(pfd, nr, -1) < 0) {\n@@ -185,6 +320,10 @@ int main(int argc, char **argv)\n \n \t\tif ((pfd[0].revents & POLLIN) && handle_command(fd, msg, msgsize))\n \t\t\tbreak;\n+#ifdef HAVE_INOTIFY\n+\t\tif ((pfd[1].revents & POLLIN) && handle_inotify(inotify_fd))\n+\t\t\tbreak;\n+#endif\n \t}\n \treturn 0;\n }\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"233054","messageId":"20140113170222.GA18964@google.com","threadId":"35642","inReplyTo":"1389524622-6702-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 2/6] read-cache: new extension to mark what file is watched","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-01-13T17:02:22Z","receivedAt":"2014-01-13T17:02:22Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nNguyễn Thái Ngọc Duy wrote:\n\n> If an entry is \"watched\", git lets an external program decide if the\n> entry is modified or not. It's more like --assume-unchanged, but\n> designed to be controlled by machine.\n>\n> We are running out of on-disk ce_flags, so instead of extending\n> on-disk entry format again, \"watched\" flags are in-core only and\n> stored as extension instead.\n\nMakes sense.\n\nCare to add a brief description of the on-disk format for\nDocumetnation/technical/index-format.txt as well?\n\n[...]\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -168,6 +168,7 @@ struct cache_entry {\n> \n> /* used to temporarily mark paths matched by pathspecs */\n> #define CE_MATCHED (1 << 26)\n> +#define CE_WATCHED (1 << 27)\n\nNit: I'd add a blank line before the definition of CE_WATCHED to make\nit clear that the comment doesn't apply to it.\n\nMaybe it belongs with one of the groups before (e.g., UNPACKED +\nNEW_SKIP_WORKTREE). I dunno.\n\n> --- a/read-cache.c\n> +++ b/read-cache.c\n[...]\n> @@ -1289,6 +1290,19 @@ static int verify_hdr(struct cache_header *hdr,\n> \treturn 0;\n> }\n> \n> +static void read_watch_extension(struct index_state *istate, uint8_t *data,\n> +\t\t\t\t unsigned long sz)\n> +{\n> +\tint i;\n> +\tif ((istate->cache_nr + 7) / 8 != sz) {\n> +\t\terror(\"invalid 'WATC' extension\");\n> +\t\treturn;\n> +\t}\n> +\tfor (i = 0; i < istate->cache_nr; i++)\n> +\t\tif (data[i / 8] & (1 << (i % 8)))\n> +\t\t\tistate->cache[i]->ce_flags |= CE_WATCHED;\n> +}\n\nSo the WATC section has one bit per index entry, encoding whether that\nentry is WATCHED. Makes sense.\n\nDo I understand correctly that this patch just takes care of the\nbookkeeping for the CE_WATCHED bit and the actual semantics will\ncome in a later patch?\n\nThanks,\nJonathan\n"},{"id":"233075","messageId":"CACsJy8D7F9TK_1sPLS0kN_wCi971iZoUeznQZ=H_oXjiWkq=Gg@mail.gmail.com","threadId":"35642","inReplyTo":"20140113170222.GA18964@google.com","subject":"Re: [PATCH 2/6] read-cache: new extension to mark what file is watched","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-14T01:25:33Z","receivedAt":"2014-01-14T01:25:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jan 14, 2014 at 12:02 AM, Jonathan Nieder wrote:\n> Hi,\n>\n> Nguyễn Thái Ngọc Duy wrote:\n>\n>> If an entry is \"watched\", git lets an external program decide if the\n>> entry is modified or not. It's more like --assume-unchanged, but\n>> designed to be controlled by machine.\n>>\n>> We are running out of on-disk ce_flags, so instead of extending\n>> on-disk entry format again, \"watched\" flags are in-core only and\n>> stored as extension instead.\n>\n> Makes sense.\n>\n> Care to add a brief description of the on-disk format for\n> Documetnation/technical/index-format.txt as well?\n\nSure, in the reroll after I fix inotify bugs that make the test suite fail.\n\n>> +static void read_watch_extension(struct index_state *istate, uint8_t *data,\n>> + unsigned long sz)\n>> +{\n>> + int i;\n>> + if ((istate->cache_nr + 7) / 8 != sz) {\n>> + error(\"invalid 'WATC' extension\");\n>> + return;\n>> + }\n>> + for (i = 0; i < istate->cache_nr; i++)\n>> + if (data[i / 8] & (1 << (i % 8)))\n>> + istate->cache[i]->ce_flags |= CE_WATCHED;\n>> +}\n>\n> So the WATC section has one bit per index entry, encoding whether that\n> entry is WATCHED. Makes sense.\n>\n> Do I understand correctly that this patch just takes care of the\n> bookkeeping for the CE_WATCHED bit and the actual semantics will\n> come in a later patch?\n\nCorrect.\n-- \nDuy\n"},{"id":"233076","messageId":"CACsJy8C9uWkbeHNeTp6WBaj2dh98hXe6q3_V_Ys067jAn1QFbQ@mail.gmail.com","threadId":"35642","inReplyTo":"1389524622-6702-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 2/6] read-cache: new extension to mark what file is watched","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-14T01:39:31Z","receivedAt":"2014-01-14T01:39:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jan 12, 2014 at 6:03 PM, Nguyễn Thái Ngọc Duy wrote:\n> We are running out of on-disk ce_flags,\n\nCorrection, we're not. I saw\n\n/*\n * Extended on-disk flags\n */\n#define CE_INTENT_TO_ADD (1 << 29)\n#define CE_SKIP_WORKTREE (1 << 30)\n\nfollowed by\n\n/* CE_EXTENDED2 is for future extension */\n#define CE_EXTENDED2 (1 << 31)\n\nand panicked, but on-disk flags could be added backward (e.g. bit 28,\n27...). Anyway using extended flags means 2 extra bytes per entry for\nalmost every entry in this case (and for index v5 it means redoing\ncrc32 for almost every entry too when the bit is updated) so it may\nstill be a good idea to keep the new flag separate.\n\n> so instead of extending\n> on-disk entry format again, \"watched\" flags are in-core only and\n> stored as extension instead.\n-- \nDuy\n"},{"id":"233143","messageId":"20140115105842.GG14335@sigill.intra.peff.net","threadId":"35642","inReplyTo":"1389524622-6702-4-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 3/6] read-cache: connect to file watcher","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-01-15T10:58:42Z","receivedAt":"2014-01-15T10:58:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 12, 2014 at 06:03:39PM +0700, Nguyễn Thái Ngọc Duy wrote:\n\n> This patch establishes a connection between a new file watcher daemon\n> and git. Each index file may have at most one file watcher attached to\n> it. The file watcher maintains a UNIX socket at\n> $GIT_DIR/index.watcher. Any process that has write access to $GIT_DIR\n> can talk to the file watcher.\n\nIIRC, this is not portable. Some systems (not Linux) will allow anyone\nto connect to the socket if it the file is accessible to them (so\nanybody with read access to $GIT_DIR can talk to the file watcher). The\nusual trick is to put it in a sub-directory that only the connectors can\naccess (e.g., put it in \"$GIT_DIR/watcher/index\", and create \"watcher\"\nmode 0700).\n\n-Peff\n"},{"id":"233272","messageId":"1389952060-12297-1-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389524622-6702-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 00/14] inotify support","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:26Z","receivedAt":"2014-01-17T09:47:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is getting in better shape. Still wondering if the design is\nright, so documentation, tests and some error cases are still\nneglected. I have not addressed Jonathan's and Jeff's comments in this\nreroll, but I haven't forgotten them yet. The test suite seems to be\nfine when file-watcher is forced on with GIT_TEST_FORCE_WATCHER set..\n\nThomas, you were a proponent of per-user daemon last time. I agree\nthat is a better solution when you need to support submodules. So if\nyou have time, have a look and see if anything I did may prevent\nper-user daemon changes later (hint, I have a few unfriendly exit() in\nfile-watcher.c). You also worked with inotify before maybe you can\nhelp spot some mishandling too as I'm totally new to inotify.\n\nNguyễn Thái Ngọc Duy (14):\n read-cache: save trailing sha-1\n read-cache: new extension to mark what file is watched\n read-cache: connect to file watcher\n read-cache: ask file watcher to watch files\n read-cache: put some limits on file watching\n read-cache: get modified file list from file watcher\n read-cache: add config to start file watcher automatically\n read-cache: add GIT_TEST_FORCE_WATCHER for testing\n file-watcher: add --shutdown and --log options\n file-watcher: automatically quit\n file-watcher: support inotify\n file-watcher: exit when cwd is gone\n pkt-line.c: increase buffer size to 8192\n t1301: exclude sockets from file permission check\n\n .gitignore | 1 +\n Documentation/config.txt | 14 ++\n Makefile | 2 +\n cache.h | 8 +\n config.mak.uname | 1 +\n file-watcher-lib.c (new) | 109 +++++++++++\n file-watcher-lib.h (new) | 9 +\n file-watcher.c (new) | 483 +++++++++++++++++++++++++++++++++++++++++++++++\n git-compat-util.h | 3 +\n pkt-line.c | 4 +-\n pkt-line.h | 2 +\n read-cache.c | 338 ++++++++++++++++++++++++++++++++-\n t/t1301-shared-repo.sh | 3 +-\n trace.c | 3 +-\n 14 files changed, 969 insertions(+), 11 deletions(-)\n create mode 100644 file-watcher-lib.c\n create mode 100644 file-watcher-lib.h\n create mode 100644 file-watcher.c\n\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233273","messageId":"1389952060-12297-2-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 01/14] read-cache: save trailing sha-1","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:27Z","receivedAt":"2014-01-17T09:47:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This will be used as signature to know if the index has changed.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n read-cache.c | 7 ++++---\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 323481c..a09d622 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -279,6 +279,7 @@ struct index_state {\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n+\tunsigned char sha1[20];\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 3f735f3..fe1d153 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1273,10 +1273,11 @@ struct ondisk_cache_entry_extended {\n \t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr(struct cache_header *hdr, unsigned long size)\n+static int verify_hdr(struct cache_header *hdr,\n+\t\t unsigned long size,\n+\t\t unsigned char *sha1)\n {\n \tgit_SHA_CTX c;\n-\tunsigned char sha1[20];\n \tint hdr_version;\n \n \tif (hdr->hdr_signature != htonl(CACHE_SIGNATURE))\n@@ -1465,7 +1466,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tclose(fd);\n \n \thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n+\tif (verify_hdr(hdr, mmap_size, istate->sha1) < 0)\n \t\tgoto unmap;\n \n \tistate->version = ntohl(hdr->hdr_version);\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233274","messageId":"1389952060-12297-3-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 02/14] read-cache: new extension to mark what file is watched","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:28Z","receivedAt":"2014-01-17T09:47:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If an entry is \"watched\", git lets an external program decide if the\nentry is modified or not. It's more like --assume-unchanged, but\ndesigned to be controlled by machine.\n\nWe are running out of on-disk ce_flags, so instead of extending\non-disk entry format again, \"watched\" flags are in-core only and\nstored as extension instead.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 2 ++\n read-cache.c | 41 ++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 42 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex a09d622..069dce7 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -168,6 +168,8 @@ struct cache_entry {\n /* used to temporarily mark paths matched by pathspecs */\n #define CE_MATCHED (1 << 26)\n \n+#define CE_WATCHED (1 << 27)\n+\n /*\n * Extended on-disk flags\n */\ndiff --git a/read-cache.c b/read-cache.c\nindex fe1d153..6f21e3f 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -33,6 +33,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n #define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n #define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n+#define CACHE_EXT_WATCH 0x57415443\t /* \"WATC\" */\n \n struct index_state the_index;\n \n@@ -1293,6 +1294,19 @@ static int verify_hdr(struct cache_header *hdr,\n \treturn 0;\n }\n \n+static void read_watch_extension(struct index_state *istate, uint8_t *data,\n+\t\t\t\t unsigned long sz)\n+{\n+\tint i;\n+\tif ((istate->cache_nr + 7) / 8 != sz) {\n+\t\terror(\"invalid 'WATC' extension\");\n+\t\treturn;\n+\t}\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (data[i / 8] & (1 << (i % 8)))\n+\t\t\tistate->cache[i]->ce_flags |= CE_WATCHED;\n+}\n+\n static int read_index_extension(struct index_state *istate,\n \t\t\t\tconst char *ext, void *data, unsigned long sz)\n {\n@@ -1303,6 +1317,9 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_RESOLVE_UNDO:\n \t\tistate->resolve_undo = resolve_undo_read(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_WATCH:\n+\t\tread_watch_extension(istate, data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1781,7 +1798,7 @@ int write_index(struct index_state *istate, int newfd)\n {\n \tgit_SHA_CTX c;\n \tstruct cache_header hdr;\n-\tint i, err, removed, extended, hdr_version;\n+\tint i, err, removed, extended, hdr_version, has_watches = 0;\n \tstruct cache_entry **cache = istate->cache;\n \tint entries = istate->cache_nr;\n \tstruct stat st;\n@@ -1790,6 +1807,8 @@ int write_index(struct index_state *istate, int newfd)\n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n \t\t\tremoved++;\n+\t\telse if (cache[i]->ce_flags & CE_WATCHED)\n+\t\t\thas_watches++;\n \n \t\t/* reduce extended entries if possible */\n \t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n@@ -1861,6 +1880,26 @@ int write_index(struct index_state *istate, int newfd)\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (has_watches) {\n+\t\tint id, sz = (entries - removed + 7) / 8;\n+\t\tuint8_t *data = xmalloc(sz);\n+\t\tmemset(data, 0, sz);\n+\t\tfor (i = 0, id = 0; i < entries && has_watches; i++) {\n+\t\t\tstruct cache_entry *ce = cache[i];\n+\t\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\t\tcontinue;\n+\t\t\tif (ce->ce_flags & CE_WATCHED) {\n+\t\t\t\tdata[id / 8] |= 1 << (id % 8);\n+\t\t\t\thas_watches--;\n+\t\t\t}\n+\t\t\tid++;\n+\t\t}\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCH, sz) < 0\n+\t\t\t|| ce_write(&c, newfd, data, sz) < 0;\n+\t\tfree(data);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n \t\treturn -1;\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233283","messageId":"1389952060-12297-4-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 03/14] read-cache: connect to file watcher","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:29Z","receivedAt":"2014-01-17T09:47:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This patch establishes a connection between a new file watcher daemon\nand git. Each index file may have at most one file watcher attached to\nit. The file watcher maintains a UNIX socket at\n$GIT_DIR/index.watcher. Any process that has write access to $GIT_DIR\ncan talk to the file watcher.\n\nA validation is performed after git connects to the file watcher to\nmake sure both sides have the same view. This is done by exchanging\nthe index signature (*) The file watcher keeps a copy of the signature\nlocally while git computes the signature from the index. If the\nsignatures do not match, something has gone wrong so both sides\nreinitialize wrt. to file watching: the file watcher clears all\nwatches while git clears CE_WATCHED flags.\n\nIf the signatures match, we can trust the file watcher and git can\nstart asking questions that are not important to this patch.\n\nThis file watching thing is all about speed. So if the daemon is not\nresponding within 20ms (or even hanging), git moves on without it.\n\nA note about per-repo vs global (or per-user) daemon approach. While I\nimplement per-repo daemon, this is actually implementation\ndetails. Nothing can stop you from writing a global daemon that opens\nunix sockets to many repos, e.g. to avoid hitting inotify's 128 user\ninstances limit.\n\nIf env variable GIT_NO_FILE_WATCHER is set, the file watcher is\nignored. 'WATC' extension is kept, but if the index is updated\n(likely), it'll become invalid at the next connection.\n\n(*) for current index versions, the signature is the index SHA-1\ntrailer. But it could be something else (e.g. v5 does not have SHA-1\ntrailer)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n .gitignore | 1 +\n Makefile | 2 +\n cache.h | 3 +\n file-watcher-lib.c (new) | 97 ++++++++++++++++++++++++++++++++\n file-watcher-lib.h (new) | 9 +++\n file-watcher.c (new) | 142 +++++++++++++++++++++++++++++++++++++++++++++++\n read-cache.c | 37 ++++++++++++\n trace.c | 3 +-\n 8 files changed, 292 insertions(+), 2 deletions(-)\n create mode 100644 file-watcher-lib.c\n create mode 100644 file-watcher-lib.h\n create mode 100644 file-watcher.c\n\ndiff --git a/.gitignore b/.gitignore\nindex dc600f9..dc870cc 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -56,6 +56,7 @@\n /git-fast-import\n /git-fetch\n /git-fetch-pack\n+/git-file-watcher\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\ndiff --git a/Makefile b/Makefile\nindex 287e6f8..4369b3b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -536,6 +536,7 @@ PROGRAMS += $(EXTRA_PROGRAMS)\n PROGRAM_OBJS += credential-store.o\n PROGRAM_OBJS += daemon.o\n PROGRAM_OBJS += fast-import.o\n+PROGRAM_OBJS += file-watcher.o\n PROGRAM_OBJS += http-backend.o\n PROGRAM_OBJS += imap-send.o\n PROGRAM_OBJS += sh-i18n--envsubst.o\n@@ -798,6 +799,7 @@ LIB_OBJS += entry.o\n LIB_OBJS += environment.o\n LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-pack.o\n+LIB_OBJS += file-watcher-lib.o\n LIB_OBJS += fsck.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\ndiff --git a/cache.h b/cache.h\nindex 069dce7..0d55551 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -282,6 +282,7 @@ struct index_state {\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n+\tint watcher;\n };\n \n extern struct index_state the_index;\n@@ -1241,6 +1242,8 @@ extern void alloc_report(void);\n __attribute__((format (printf, 1, 2)))\n extern void trace_printf(const char *format, ...);\n __attribute__((format (printf, 2, 3)))\n+extern void trace_printf_key(const char *key, const char *fmt, ...);\n+__attribute__((format (printf, 2, 3)))\n extern void trace_argv_printf(const char **argv, const char *format, ...);\n extern void trace_repo_setup(const char *prefix);\n extern int trace_want(const char *key);\ndiff --git a/file-watcher-lib.c b/file-watcher-lib.c\nnew file mode 100644\nindex 0000000..ed14ef9\n--- /dev/null\n+++ b/file-watcher-lib.c\n@@ -0,0 +1,97 @@\n+#include \"cache.h\"\n+\n+#define WAIT_TIME 20\t\t/* in ms */\n+#define TRACE_KEY \"GIT_TRACE_WATCHER\"\n+\n+int connect_watcher(const char *path)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct stat st;\n+\tint fd = -1;\n+\n+\tstrbuf_addf(&sb, \"%s.watcher\", path);\n+\tif (!stat(sb.buf, &st) && S_ISSOCK(st.st_mode)) {\n+\t\tstruct sockaddr_un sun;\n+\t\tfd = socket(AF_UNIX, SOCK_DGRAM, 0);\n+\t\tsun.sun_family = AF_UNIX;\n+\t\tstrlcpy(sun.sun_path, sb.buf, sizeof(sun.sun_path));\n+\t\tif (connect(fd, (struct sockaddr *)&sun, sizeof(sun))) {\n+\t\t\terror(_(\"unable to connect to file watcher: %s\"),\n+\t\t\t strerror(errno));\n+\t\t\tclose(fd);\n+\t\t\tfd = -1;\n+\t\t} else {\n+\t\t\tsprintf(sun.sun_path, \"%c%\"PRIuMAX, 0, (uintmax_t)getpid());\n+\t\t\tif (bind(fd, (struct sockaddr *)&sun, sizeof(sun))) {\n+\t\t\t\terror(_(\"unable to bind socket: %s\"),\n+\t\t\t\t strerror(errno));\n+\t\t\t\tclose(fd);\n+\t\t\t\tfd = -1;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tstrbuf_release(&sb);\n+\treturn fd;\n+}\n+\n+ssize_t send_watcher(int sockfd, struct sockaddr_un *dest,\n+\t\t const char *fmt, ...)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct pollfd pfd;\n+\tint ret;\n+\n+\tva_list ap;\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\tva_end(ap);\n+\n+\tpfd.fd = sockfd;\n+\tpfd.events = POLLOUT;\n+\tret = poll(&pfd, 1, WAIT_TIME);\n+\tif (ret > 0 && pfd.revents & POLLOUT) {\n+\t\ttrace_printf_key(TRACE_KEY, \"< %s\\n\", sb.buf);\n+\t\tif (dest)\n+\t\t\tret = sendto(sockfd, sb.buf, sb.len, 0,\n+\t\t\t\t (struct sockaddr *)dest,\n+\t\t\t\t sizeof(struct sockaddr_un));\n+\t\telse\n+\t\t\tret = write(sockfd, sb.buf, sb.len);\n+\t}\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+char *read_watcher(int fd, ssize_t *size, struct sockaddr_un *sun)\n+{\n+\tstatic char *buf;\n+\tstatic int buf_size;\n+\tstruct pollfd pfd;\n+\tssize_t len;\n+\n+\tif (!buf_size) {\n+\t\tsocklen_t vallen = sizeof(buf_size);\n+\t\tif (getsockopt(fd, SOL_SOCKET, SO_SNDBUF, &buf_size, &vallen))\n+\t\t\tdie_errno(\"could not get SO_SNDBUF from socket %d\", fd);\n+\t\tbuf = xmalloc(buf_size + 1);\n+\t}\n+\n+\tpfd.fd = fd;\n+\tpfd.events = POLLIN;\n+\tif (poll(&pfd, 1, WAIT_TIME) > 0 &&\n+\t (pfd.revents & POLLIN)) {\n+\t\tif (sun) {\n+\t\t\tsocklen_t socklen = sizeof(*sun);\n+\t\t\tlen = recvfrom(fd, buf, buf_size, 0, sun, &socklen);\n+\t\t} else\n+\t\t\tlen = read(fd, buf, buf_size);\n+\t\tif (len > 0)\n+\t\t\tbuf[len] = '\\0';\n+\t\tif (size)\n+\t\t\t*size = len;\n+\t\ttrace_printf_key(TRACE_KEY, \"> %s\\n\", buf);\n+\t\treturn buf;\n+\t} else if (size)\n+\t\t*size = 0;\n+\treturn NULL;\n+}\ndiff --git a/file-watcher-lib.h b/file-watcher-lib.h\nnew file mode 100644\nindex 0000000..0fe9399\n--- /dev/null\n+++ b/file-watcher-lib.h\n@@ -0,0 +1,9 @@\n+#ifndef __FILE_WATCHER_LIB__\n+#define __FILE_WATCHER_LIB__\n+\n+int connect_watcher(const char *path);\n+ssize_t send_watcher(int sockfd, struct sockaddr_un *dest,\n+\t\t const char *fmt, ...)\n+\t__attribute__((format (printf, 3, 4)));\n+char *read_watcher(int fd, ssize_t *size, struct sockaddr_un *sun);\n+#endif\ndiff --git a/file-watcher.c b/file-watcher.c\nnew file mode 100644\nindex 0000000..36a9a8d\n--- /dev/null\n+++ b/file-watcher.c\n@@ -0,0 +1,142 @@\n+#include \"cache.h\"\n+#include \"sigchain.h\"\n+#include \"parse-options.h\"\n+#include \"exec_cmd.h\"\n+#include \"file-watcher-lib.h\"\n+\n+static const char *const file_watcher_usage[] = {\n+\tN_(\"git file-watcher [options]\"),\n+\tNULL\n+};\n+\n+static char index_signature[41];\n+\n+static int handle_command(int fd)\n+{\n+\tstruct sockaddr_un sun;\n+\tssize_t len;\n+\tconst char *arg;\n+\tchar *msg;\n+\n+\tif (!(msg = read_watcher(fd, &len, &sun)))\n+\t\tdie_errno(\"read from socket\");\n+\n+\tif ((arg = skip_prefix(msg, \"hello \"))) {\n+\t\tsend_watcher(fd, &sun, \"hello %s\", index_signature);\n+\t\tif (strcmp(arg, index_signature))\n+\t\t\t/*\n+\t\t\t * Index SHA-1 mismatch, something has gone\n+\t\t\t * wrong. Clean up and start over.\n+\t\t\t */\n+\t\t\tindex_signature[0] = '\\0';\n+\t} else if (!strcmp(msg, \"die\")) {\n+\t\texit(0);\n+\t} else {\n+\t\tdie(\"unrecognized command %s\", msg);\n+\t}\n+\treturn 0;\n+}\n+\n+static const char *socket_path;\n+static int do_not_clean_up;\n+\n+static void cleanup(void)\n+{\n+\tif (do_not_clean_up)\n+\t\treturn;\n+\tunlink(socket_path);\n+}\n+\n+static void cleanup_on_signal(int signo)\n+{\n+\tcleanup();\n+\tsigchain_pop(signo);\n+\traise(signo);\n+}\n+\n+static void daemonize(void)\n+{\n+#ifdef NO_POSIX_GOODIES\n+\tdie(\"fork not supported on this platform\");\n+#else\n+\tswitch (fork()) {\n+\t\tcase 0:\n+\t\t\tbreak;\n+\t\tcase -1:\n+\t\t\tdie_errno(\"fork failed\");\n+\t\tdefault:\n+\t\t\tdo_not_clean_up = 1;\n+\t\t\texit(0);\n+\t}\n+\tif (setsid() == -1)\n+\t\tdie_errno(\"setsid failed\");\n+\tclose(0);\n+\tclose(1);\n+\tclose(2);\n+\tsanitize_stdfds();\n+#endif\n+}\n+\n+int main(int argc, const char **argv)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct sockaddr_un sun;\n+\tstruct pollfd pfd[2];\n+\tint fd, err, nr;\n+\tconst char *prefix;\n+\tint daemon = 0;\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"daemon\", &daemon,\n+\t\t\t N_(\"run in background\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_extract_argv0_path(argv[0]);\n+\tgit_setup_gettext();\n+\tprefix = setup_git_directory();\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t file_watcher_usage, 0);\n+\tif (argc)\n+\t\tdie(\"too many arguments\");\n+\n+\tstrbuf_addf(&sb, \"%s.watcher\", get_index_file());\n+\tsocket_path = strbuf_detach(&sb, NULL);\n+\tmemset(index_signature, 0, sizeof(index_signature));\n+\tfd = socket(AF_UNIX, SOCK_DGRAM, 0);\n+\tsun.sun_family = AF_UNIX;\n+\tstrlcpy(sun.sun_path, socket_path, sizeof(sun.sun_path));\n+\tif (bind(fd, (struct sockaddr *)&sun, sizeof(sun)))\n+\t\tdie_errno(\"unable to bind to %s\", socket_path);\n+\tatexit(cleanup);\n+\tsigchain_push_common(cleanup_on_signal);\n+\n+\tif (daemon) {\n+\t\tstrbuf_addf(&sb, \"%s.log\", socket_path);\n+\t\terr = open(sb.buf, O_CREAT | O_TRUNC | O_WRONLY, 0600);\n+\t\tadjust_shared_perm(sb.buf);\n+\t\tif (err == -1)\n+\t\t\tdie_errno(\"unable to create %s\", sb.buf);\n+\t\tdaemonize();\n+\t\tdup2(err, 1);\n+\t\tdup2(err, 2);\n+\t\tclose(err);\n+\t}\n+\n+\tnr = 0;\n+\tpfd[nr].fd = fd;\n+\tpfd[nr++].events = POLLIN;\n+\n+\tfor (;;) {\n+\t\tif (poll(pfd, nr, -1) < 0) {\n+\t\t\tif (errno != EINTR) {\n+\t\t\t\terror(\"Poll failed, resuming: %s\", strerror(errno));\n+\t\t\t\tsleep(1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif ((pfd[0].revents & POLLIN) && handle_command(fd))\n+\t\t\tbreak;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/read-cache.c b/read-cache.c\nindex 6f21e3f..76cf0e3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -14,6 +14,7 @@\n #include \"resolve-undo.h\"\n #include \"strbuf.h\"\n #include \"varint.h\"\n+#include \"file-watcher-lib.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int really);\n \n@@ -1447,6 +1448,37 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static void validate_watcher(struct index_state *istate, const char *path)\n+{\n+\tint i;\n+\n+\tif (getenv(\"GIT_NO_FILE_WATCHER\")) {\n+\t\tistate->watcher = -1;\n+\t\treturn;\n+\t}\n+\n+\tistate->watcher = connect_watcher(path);\n+\tif (istate->watcher != -1) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tchar *msg;\n+\t\tstrbuf_addf(&sb, \"hello %s\", sha1_to_hex(istate->sha1));\n+\t\tif (send_watcher(istate->watcher, NULL, \"%s\", sb.buf) > 0 &&\n+\t\t (msg = read_watcher(istate->watcher, NULL, NULL)) != NULL &&\n+\t\t !strcmp(msg, sb.buf)) { /* good */\n+\t\t\tstrbuf_release(&sb);\n+\t\t\treturn;\n+\t\t}\n+\t\tstrbuf_release(&sb);\n+\t}\n+\n+\t/* No the file watcher is out of date, clear everything */\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHED) {\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_WATCHED;\n+\t\t\tistate->cache_changed = 1;\n+\t\t}\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n@@ -1532,6 +1564,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsrc_offset += extsize;\n \t}\n \tmunmap(mmap, mmap_size);\n+\tvalidate_watcher(istate, path);\n \treturn istate->cache_nr;\n \n unmap:\n@@ -1557,6 +1590,10 @@ int discard_index(struct index_state *istate)\n \tistate->timestamp.nsec = 0;\n \tfree_name_hash(istate);\n \tcache_tree_free(&(istate->cache_tree));\n+\tif (istate->watcher > 0) {\n+\t\tclose(istate->watcher);\n+\t\tistate->watcher = -1;\n+\t}\n \tistate->initialized = 0;\n \tfree(istate->cache);\n \tistate->cache = NULL;\ndiff --git a/trace.c b/trace.c\nindex 3d744d1..0b8ebe0 100644\n--- a/trace.c\n+++ b/trace.c\n@@ -75,8 +75,7 @@ static void trace_vprintf(const char *key, const char *fmt, va_list ap)\n \tstrbuf_release(&buf);\n }\n \n-__attribute__((format (printf, 2, 3)))\n-static void trace_printf_key(const char *key, const char *fmt, ...)\n+void trace_printf_key(const char *key, const char *fmt, ...)\n {\n \tva_list ap;\n \tva_start(ap, fmt);\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233275","messageId":"1389952060-12297-5-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 04/14] read-cache: ask file watcher to watch files","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:30Z","receivedAt":"2014-01-17T09:47:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We want to watch files that are never changed because lstat() on those\nfiles is a wasted effort. So we sort unwatched files by date and start\nadding them to the file watcher until it barfs (e.g. hits inotify\nlimit). Recently updated entries are also excluded from watch list.\nCE_VALID is used in combination with CE_WATCHED. Those entries that\nhave CE_VALID already set will never be watched.\n\nWe send as many paths as possible in one packet in pkt-line format to\nreduce roundtrips. For small projects like git, all entries can be\npacked in one packet. For large projects like webkit (182k entries) it\ntakes two packets. We may do prefix compression as well to send more\nin fewer packets..\n\nThe file watcher replies how many entries it can watch (because at\nleast inotify has system limits).\n\nNote that we still do lstat() on these new watched files because they\ncould have changed before the file watcher could watch them. Watched\nfiles may only skip lstat() at the next git run.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher.c | 31 ++++++++++++++++\n pkt-line.c | 2 +-\n pkt-line.h | 2 ++\n read-cache.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 4 files changed, 143 insertions(+), 3 deletions(-)\n\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 36a9a8d..3a54168 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"exec_cmd.h\"\n #include \"file-watcher-lib.h\"\n+#include \"pkt-line.h\"\n \n static const char *const file_watcher_usage[] = {\n \tN_(\"git file-watcher [options]\"),\n@@ -11,6 +12,34 @@ static const char *const file_watcher_usage[] = {\n \n static char index_signature[41];\n \n+static int watch_path(char *path)\n+{\n+\t/*\n+\t * Consider send \"wait\" every 10ms or so, in case there are\n+\t * many paths to process that takes more than 20ms or the\n+\t * sender won't keep waiting. This is usually one-time cost,\n+\t * waiting a bit is ok.\n+\t */\n+\treturn -1;\n+}\n+\n+static void watch_paths(char *buf, int maxlen,\n+\t\t\tint fd, struct sockaddr_un *sock)\n+{\n+\tchar *end = buf + maxlen;\n+\tint n, ret, len;\n+\tfor (n = ret = 0; buf < end && !ret; buf += len) {\n+\t\tchar ch;\n+\t\tlen = packet_length(buf);\n+\t\tch = buf[len];\n+\t\tbuf[len] = '\\0';\n+\t\tif (!(ret = watch_path(buf + 4)))\n+\t\t\tn++;\n+\t\tbuf[len] = ch;\n+\t}\n+\tsend_watcher(fd, sock, \"fine %d\", n);\n+}\n+\n static int handle_command(int fd)\n {\n \tstruct sockaddr_un sun;\n@@ -29,6 +58,8 @@ static int handle_command(int fd)\n \t\t\t * wrong. Clean up and start over.\n \t\t\t */\n \t\t\tindex_signature[0] = '\\0';\n+\t} else if (starts_with(msg, \"watch \")) {\n+\t\twatch_paths(msg + 6, len - 6, fd, &sun);\n \t} else if (!strcmp(msg, \"die\")) {\n \t\texit(0);\n \t} else {\ndiff --git a/pkt-line.c b/pkt-line.c\nindex bc63b3b..b5af84e 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -135,7 +135,7 @@ static int get_packet_data(int fd, char **src_buf, size_t *src_size,\n \treturn ret;\n }\n \n-static int packet_length(const char *linelen)\n+int packet_length(const char *linelen)\n {\n \tint n;\n \tint len = 0;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 0a838d1..40470b9 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -75,6 +75,8 @@ char *packet_read_line(int fd, int *size);\n */\n char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n \n+int packet_length(const char *linelen);\n+\n #define DEFAULT_PACKET_MAX 1000\n #define LARGE_PACKET_MAX 65520\n extern char packet_buffer[LARGE_PACKET_MAX];\ndiff --git a/read-cache.c b/read-cache.c\nindex 76cf0e3..21c3207 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -15,6 +15,7 @@\n #include \"strbuf.h\"\n #include \"varint.h\"\n #include \"file-watcher-lib.h\"\n+#include \"pkt-line.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int really);\n \n@@ -1479,6 +1480,98 @@ static void validate_watcher(struct index_state *istate, const char *path)\n \t\t}\n }\n \n+static int sort_by_date(const void *a_, const void *b_)\n+{\n+\tconst struct cache_entry *a = *(const struct cache_entry **)a_;\n+\tconst struct cache_entry *b = *(const struct cache_entry **)b_;\n+\tuint32_t seca = a->ce_stat_data.sd_mtime.sec;\n+\tuint32_t secb = b->ce_stat_data.sd_mtime.sec;\n+\treturn seca - secb;\n+}\n+\n+static int do_watch_entries(struct index_state *istate,\n+\t\t\t struct cache_entry **cache,\n+\t\t\t struct strbuf *sb, int start, int now)\n+{\n+\tchar *line;\n+\tint i;\n+\tssize_t len;\n+\n+\tsend_watcher(istate->watcher, NULL, \"%s\", sb->buf);\n+\tline = read_watcher(istate->watcher, &len, NULL);\n+\tif (!line) {\n+\t\tif (!len) {\n+\t\t\tclose(istate->watcher);\n+\t\t\tistate->watcher = -1;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\tif (starts_with(line, \"fine \")) {\n+\t\tchar *end;\n+\t\tlong n = strtoul(line + 5, &end, 10);\n+\t\tif (end != line + len)\n+\t\t\treturn -1;\n+\t\tfor (i = 0; i < n; i++)\n+\t\t\tcache[start + i]->ce_flags |= CE_WATCHED;\n+\t\tistate->cache_changed = 1;\n+\t\tif (i != now)\n+\t\t\treturn -1;\n+\t} else\n+\t\treturn -1;\n+\tstart = i;\n+\tstrbuf_reset(sb);\n+\tstrbuf_addstr(sb, \"watch \");\n+\treturn 0;\n+}\n+\n+static inline int ce_watchable(struct cache_entry *ce)\n+{\n+\treturn ce_uptodate(ce) && /* write_index will catch late ce_uptodate bits */\n+\t\t!(ce->ce_flags & CE_WATCHED) &&\n+\t\t!(ce->ce_flags & CE_VALID) &&\n+\t\t/*\n+\t\t * S_IFGITLINK should not be watched\n+\t\t * obviously. S_IFLNK could be problematic because\n+\t\t * inotify may follow symlinks without IN_DONT_FOLLOW\n+\t\t */\n+\t\tS_ISREG(ce->ce_mode);\n+}\n+\n+static void watch_entries(struct index_state *istate)\n+{\n+\tint i, start, nr;\n+\tstruct cache_entry **sorted;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint val;\n+\tsocklen_t vallen = sizeof(val);\n+\n+\tif (istate->watcher <= 0)\n+\t\treturn;\n+\tfor (i = nr = 0; i < istate->cache_nr; i++)\n+\t\tif (ce_watchable(istate->cache[i]))\n+\t\t\tnr++;\n+\tsorted = xmalloc(sizeof(*sorted) * nr);\n+\tfor (i = nr = 0; i < istate->cache_nr; i++)\n+\t\tif (ce_watchable(istate->cache[i]))\n+\t\t\tsorted[nr++] = istate->cache[i];\n+\n+\tgetsockopt(istate->watcher, SOL_SOCKET, SO_SNDBUF, &val, &vallen);\n+\tstrbuf_grow(&sb, val);\n+\tstrbuf_addstr(&sb, \"watch \");\n+\n+\tqsort(sorted, nr, sizeof(*sorted), sort_by_date);\n+\tfor (i = start = 0; i < nr; i++) {\n+\t\tif (sb.len + 4 + ce_namelen(sorted[i]) >= val &&\n+\t\t do_watch_entries(istate, sorted, &sb, start, i))\n+\t\t\tbreak;\n+\t\tpacket_buf_write(&sb, \"%s\", sorted[i]->name);\n+\t}\n+\tif (i == nr && start < i)\n+\t\tdo_watch_entries(istate, sorted, &sb, start, i);\n+\tstrbuf_release(&sb);\n+\tfree(sorted);\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n@@ -1565,6 +1658,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t}\n \tmunmap(mmap, mmap_size);\n \tvalidate_watcher(istate, path);\n+\twatch_entries(istate);\n \treturn istate->cache_nr;\n \n unmap:\n@@ -1844,8 +1938,21 @@ int write_index(struct index_state *istate, int newfd)\n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n \t\t\tremoved++;\n-\t\telse if (cache[i]->ce_flags & CE_WATCHED)\n-\t\t\thas_watches++;\n+\t\telse if (cache[i]->ce_flags & CE_WATCHED) {\n+\t\t\t/*\n+\t\t\t * We may set CE_WATCHED (but not CE_VALID)\n+\t\t\t * early when refresh has not been done\n+\t\t\t * yet. At that time we had no idea if the\n+\t\t\t * entry may have been updated. If it has\n+\t\t\t * been, remove CE_WATCHED so CE_VALID won't\n+\t\t\t * incorrectly be set next time if the file\n+\t\t\t * watcher reports no changes.\n+\t\t\t */\n+\t\t\tif (!ce_uptodate(cache[i]))\n+\t\t\t\tcache[i]->ce_flags &= ~CE_WATCHED;\n+\t\t\telse\n+\t\t\t\thas_watches++;\n+\t\t}\n \n \t\t/* reduce extended entries if possible */\n \t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233276","messageId":"1389952060-12297-6-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 05/14] read-cache: put some limits on file watching","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:31Z","receivedAt":"2014-01-17T09:47:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"watch_entries() is a lot of computation and could trigger a lot more\nlookups in file-watcher. Normally after the first set of watches are\nin place, we do not need to update often. Moreover if the number of\nentries is small, the overhead of file watcher may actually slow git\ndown.\n\nThis patch only allows to update watches if the number of watchable\nfiles is over a limit (and there are new files added if this is not\nthe first time). Measurements on Core i5-2520M and Linux 3.7.6, about\n920 lstat() take 1ms. Somewhere between 2^16 and 2^17 lstat calls that\nit starts to take longer than 100ms. 2^16 is chosen at the minimum\nlimit to start using file watcher.\n\nRecently updated files are not considered watchable because they are\nlikely to be updated again soon, not worth the ping-pong game with\nfile watcher. The default limit 30min is just a random value.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 9 +++++++++\n cache.h | 1 +\n read-cache.c | 44 ++++++++++++++++++++++++++++++++++++--------\n 3 files changed, 46 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex a405806..e394399 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1038,6 +1038,15 @@ difftool..cmd::\n difftool.prompt::\n \tPrompt before each invocation of the diff tool.\n \n+filewatcher.minfiles::\n+\tStart watching files if the number of watchable files are\n+\tabove this limit. Default value is 65536.\n+\n+filewatcher.recentlimit::\n+\tFiles that are last updated within filewatcher.recentlimit\n+\tseconds from now are not considered watchable. Default value\n+\tis 1800 (30 minutes).\n+\n fetch.recurseSubmodules::\n \tThis option can be either set to a boolean value or to 'on-demand'.\n \tSetting it to a boolean changes the behavior of fetch and pull to\ndiff --git a/cache.h b/cache.h\nindex 0d55551..bcec29b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -278,6 +278,7 @@ struct index_state {\n \tstruct cache_tree *cache_tree;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n+\t\t update_watches : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\ndiff --git a/read-cache.c b/read-cache.c\nindex 21c3207..406834a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -38,6 +38,8 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n #define CACHE_EXT_WATCH 0x57415443\t /* \"WATC\" */\n \n struct index_state the_index;\n+static int watch_lowerlimit = 65536;\n+static int recent_limit = 1800;\n \n static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n {\n@@ -1014,6 +1016,7 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n \t\t\t(istate->cache_nr - pos - 1) * sizeof(ce));\n \tset_index_entry(istate, pos, ce);\n \tistate->cache_changed = 1;\n+\tistate->update_watches = 1;\n \treturn 0;\n }\n \n@@ -1300,13 +1303,14 @@ static void read_watch_extension(struct index_state *istate, uint8_t *data,\n \t\t\t\t unsigned long sz)\n {\n \tint i;\n-\tif ((istate->cache_nr + 7) / 8 != sz) {\n+\tif ((istate->cache_nr + 7) / 8 + 1 != sz) {\n \t\terror(\"invalid 'WATC' extension\");\n \t\treturn;\n \t}\n \tfor (i = 0; i < istate->cache_nr; i++)\n \t\tif (data[i / 8] & (1 << (i % 8)))\n \t\t\tistate->cache[i]->ce_flags |= CE_WATCHED;\n+\tistate->update_watches = data[sz - 1];\n }\n \n static int read_index_extension(struct index_state *istate,\n@@ -1449,6 +1453,19 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static int watcher_config(const char *var, const char *value, void *data)\n+{\n+\tif (!strcmp(var, \"filewatcher.minfiles\")) {\n+\t\twatch_lowerlimit = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(var, \"filewatcher.recentlimit\")) {\n+\t\trecent_limit = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\treturn 0;\n+}\n+\n static void validate_watcher(struct index_state *istate, const char *path)\n {\n \tint i;\n@@ -1458,6 +1475,7 @@ static void validate_watcher(struct index_state *istate, const char *path)\n \t\treturn;\n \t}\n \n+\tgit_config(watcher_config, NULL);\n \tistate->watcher = connect_watcher(path);\n \tif (istate->watcher != -1) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n@@ -1478,6 +1496,7 @@ static void validate_watcher(struct index_state *istate, const char *path)\n \t\t\tistate->cache[i]->ce_flags &= ~CE_WATCHED;\n \t\t\tistate->cache_changed = 1;\n \t\t}\n+\tistate->update_watches = 1;\n }\n \n static int sort_by_date(const void *a_, const void *b_)\n@@ -1524,7 +1543,7 @@ static int do_watch_entries(struct index_state *istate,\n \treturn 0;\n }\n \n-static inline int ce_watchable(struct cache_entry *ce)\n+static inline int ce_watchable(struct cache_entry *ce, time_t now)\n {\n \treturn ce_uptodate(ce) && /* write_index will catch late ce_uptodate bits */\n \t\t!(ce->ce_flags & CE_WATCHED) &&\n@@ -1534,7 +1553,8 @@ static inline int ce_watchable(struct cache_entry *ce)\n \t\t * obviously. S_IFLNK could be problematic because\n \t\t * inotify may follow symlinks without IN_DONT_FOLLOW\n \t\t */\n-\t\tS_ISREG(ce->ce_mode);\n+\t\tS_ISREG(ce->ce_mode) &&\n+\t\t(ce->ce_stat_data.sd_mtime.sec + recent_limit < now);\n }\n \n static void watch_entries(struct index_state *istate)\n@@ -1544,15 +1564,20 @@ static void watch_entries(struct index_state *istate)\n \tstruct strbuf sb = STRBUF_INIT;\n \tint val;\n \tsocklen_t vallen = sizeof(val);\n+\ttime_t now = time(NULL);\n \n-\tif (istate->watcher <= 0)\n+\tif (istate->watcher <= 0 || !istate->update_watches)\n \t\treturn;\n+\tistate->update_watches = 0;\n+\tistate->cache_changed = 1;\n \tfor (i = nr = 0; i < istate->cache_nr; i++)\n-\t\tif (ce_watchable(istate->cache[i]))\n+\t\tif (ce_watchable(istate->cache[i], now))\n \t\t\tnr++;\n+\tif (nr < watch_lowerlimit)\n+\t\treturn;\n \tsorted = xmalloc(sizeof(*sorted) * nr);\n \tfor (i = nr = 0; i < istate->cache_nr; i++)\n-\t\tif (ce_watchable(istate->cache[i]))\n+\t\tif (ce_watchable(istate->cache[i], now))\n \t\t\tsorted[nr++] = istate->cache[i];\n \n \tgetsockopt(istate->watcher, SOL_SOCKET, SO_SNDBUF, &val, &vallen);\n@@ -1616,6 +1641,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tistate->cache_alloc = alloc_nr(istate->cache_nr);\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n+\tistate->update_watches = 1;\n \n \tif (istate->version == 4)\n \t\tprevious_name = &previous_name_buf;\n@@ -2024,8 +2050,9 @@ int write_index(struct index_state *istate, int newfd)\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n-\tif (has_watches) {\n-\t\tint id, sz = (entries - removed + 7) / 8;\n+\tif (has_watches ||\n+\t (istate->watcher != -1 && !istate->update_watches)) {\n+\t\tint id, sz = (entries - removed + 7) / 8 + 1;\n \t\tuint8_t *data = xmalloc(sz);\n \t\tmemset(data, 0, sz);\n \t\tfor (i = 0, id = 0; i < entries && has_watches; i++) {\n@@ -2038,6 +2065,7 @@ int write_index(struct index_state *istate, int newfd)\n \t\t\t}\n \t\t\tid++;\n \t\t}\n+\t\tdata[sz - 1] = istate->update_watches;\n \t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCH, sz) < 0\n \t\t\t|| ce_write(&c, newfd, data, sz) < 0;\n \t\tfree(data);\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233280","messageId":"1389952060-12297-7-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 06/14] read-cache: get modified file list from file watcher","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:32Z","receivedAt":"2014-01-17T09:47:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"A new command is added to file watcher to send back the list of\nupdated files to git. These entries will have CE_WATCHED removed. The\nremaining CE_WATCHED entries will have CE_VALID set (i.e. no changes\nand no lstat either).\n\nThe file watcher does not cache stat info and send back to git. Its\nmain purpose is to reduce lstat on most untouched files, not to\ncompletely eliminate lstat.\n\nThe file watcher keeps reporting the same \"updated\" list until it\nreceives \"forget\" commands, which should only be issued after the\nupdated index is written down. This ensures that if git crashes half\nway before it could update the index (or multiple processes is reading\nthe same index), \"updated\" info is not lost.\n\nAfter the index is updated (e.g. in this case because of toggling\nCE_WATCHED bits), git sends the new index signature to the file\nwatcher.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n file-watcher.c | 63 +++++++++++++++++++++++++++++++++---\n read-cache.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 3 files changed, 157 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex bcec29b..8f065ed 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -284,6 +284,7 @@ struct index_state {\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n \tint watcher;\n+\tstruct string_list *updated_entries;\n };\n \n extern struct index_state the_index;\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 3a54168..369af37 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"exec_cmd.h\"\n #include \"file-watcher-lib.h\"\n+#include \"string-list.h\"\n #include \"pkt-line.h\"\n \n static const char *const file_watcher_usage[] = {\n@@ -11,6 +12,8 @@ static const char *const file_watcher_usage[] = {\n };\n \n static char index_signature[41];\n+static struct string_list updated = STRING_LIST_INIT_DUP;\n+static int updated_sorted;\n \n static int watch_path(char *path)\n {\n@@ -23,6 +26,37 @@ static int watch_path(char *path)\n \treturn -1;\n }\n \n+static void reset(void)\n+{\n+\tstring_list_clear(&updated, 0);\n+\tindex_signature[0] = '\\0';\n+}\n+\n+static void send_status(int fd, struct sockaddr_un *sun)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint i, size;\n+\tsocklen_t vallen = sizeof(size);\n+\tif (getsockopt(fd, SOL_SOCKET, SO_SNDBUF, &size, &vallen))\n+\t\tdie_errno(\"could not get SO_SNDBUF from socket %d\", fd);\n+\n+\tstrbuf_grow(&sb, size);\n+\tstrbuf_addstr(&sb, \"new \");\n+\n+\tfor (i = 0; i < updated.nr; i++) {\n+\t\tint len = strlen(updated.items[i].string) + 4;\n+\t\tif (sb.len + len >= size) {\n+\t\t\tsend_watcher(fd, sun, \"%s\", sb.buf);\n+\t\t\tstrbuf_reset(&sb);\n+\t\t\tstrbuf_addstr(&sb, \"new \");\n+\t\t}\n+\t\tpacket_buf_write(&sb, \"%s\", updated.items[i].string);\n+\t}\n+\tstrbuf_addstr(&sb, \"0000\");\n+\tsend_watcher(fd, sun, \"%s\", sb.buf);\n+\tstrbuf_release(&sb);\n+}\n+\n static void watch_paths(char *buf, int maxlen,\n \t\t\tint fd, struct sockaddr_un *sock)\n {\n@@ -40,6 +74,19 @@ static void watch_paths(char *buf, int maxlen,\n \tsend_watcher(fd, sock, \"fine %d\", n);\n }\n \n+static void remove_updated(const char *path)\n+{\n+\tstruct string_list_item *item;\n+\tif (!updated_sorted) {\n+\t\tsort_string_list(&updated);\n+\t\tupdated_sorted = 1;\n+\t}\n+\titem = string_list_lookup(&updated, path);\n+\tif (!item)\n+\t\treturn;\n+\tunsorted_string_list_delete_item(&updated, item - updated.items, 0);\n+}\n+\n static int handle_command(int fd)\n {\n \tstruct sockaddr_un sun;\n@@ -53,11 +100,17 @@ static int handle_command(int fd)\n \tif ((arg = skip_prefix(msg, \"hello \"))) {\n \t\tsend_watcher(fd, &sun, \"hello %s\", index_signature);\n \t\tif (strcmp(arg, index_signature))\n-\t\t\t/*\n-\t\t\t * Index SHA-1 mismatch, something has gone\n-\t\t\t * wrong. Clean up and start over.\n-\t\t\t */\n-\t\t\tindex_signature[0] = '\\0';\n+\t\t\treset();\n+\t} else if ((arg = skip_prefix(msg, \"clear\"))) {\n+\t\treset();\n+\t} else if (!strcmp(msg, \"status\")) {\n+\t\tsend_status(fd, &sun);\n+\t} else if ((arg = skip_prefix(msg, \"bye \"))) {\n+\t\tstrlcpy(index_signature, arg, sizeof(index_signature));\n+\t} else if ((arg = skip_prefix(msg, \"forget \"))) {\n+\t\tint len = strlen(index_signature);\n+\t\tif (!strncmp(arg, index_signature, len) && arg[len] == ' ')\n+\t\t\tremove_updated(arg + len + 1);\n \t} else if (starts_with(msg, \"watch \")) {\n \t\twatch_paths(msg + 6, len - 6, fd, &sun);\n \t} else if (!strcmp(msg, \"die\")) {\ndiff --git a/read-cache.c b/read-cache.c\nindex 406834a..3aa541d 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1453,6 +1453,69 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static void update_watched_files(struct index_state *istate)\n+{\n+\tint i;\n+\tif (istate->watcher <= 0)\n+\t\treturn;\n+\tif (send_watcher(istate->watcher, NULL, \"status\") < 0)\n+\t\tgoto failed;\n+\tfor (;;) {\n+\t\tchar *line, *end;\n+\t\tssize_t len;\n+\t\tint ch;\n+\t\tline = read_watcher(istate->watcher, &len, NULL);\n+\t\tif (!line || !starts_with(line, \"new \")) {\n+\t\t\tif (!len) {\n+\t\t\t\tclose(istate->watcher);\n+\t\t\t\tistate->watcher = -1;\n+\t\t\t}\n+\t\t\tgoto failed;\n+\t\t}\n+\t\tend = line + len;\n+\t\tline += 4;\n+\t\tfor (; line < end; line[len] = ch, line += len) {\n+\t\t\tlen = packet_length(line);\n+\t\t\tif (!len)\n+\t\t\t\tbreak;\n+\t\t\tch = line[len];\n+\t\t\tline[len] = '\\0';\n+\t\t\ti = index_name_pos(istate, line + 4, len - 4);\n+\t\t\tif (i < 0)\n+\t\t\t\tcontinue;\n+\t\t\tif (istate->cache[i]->ce_flags & CE_WATCHED) {\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_WATCHED;\n+\t\t\t\tistate->cache_changed = 1;\n+\t\t\t}\n+\t\t\tif (!istate->updated_entries) {\n+\t\t\t\tstruct string_list *sl;\n+\t\t\t\tsl = xmalloc(sizeof(*sl));\n+\t\t\t\tmemset(sl, 0, sizeof(*sl));\n+\t\t\t\tsl->strdup_strings = 1;\n+\t\t\t\tistate->updated_entries = sl;\n+\t\t\t}\n+\t\t\tstring_list_append(istate->updated_entries, line + 4);\n+\t\t}\n+\t\tif (!len)\n+\t\t\tbreak;\n+\t}\n+\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHED)\n+\t\t\tistate->cache[i]->ce_flags |= CE_VALID;\n+\treturn;\n+failed:\n+\tif (istate->updated_entries) {\n+\t\tstring_list_clear(istate->updated_entries, 0);\n+\t\tfree(istate->updated_entries);\n+\t\tistate->updated_entries = NULL;\n+\t}\n+\tsend_watcher(istate->watcher, NULL, \"clear\");\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tistate->cache[i]->ce_flags &= ~CE_WATCHED;\n+\tistate->cache_changed = 1;\n+}\n+\n static int watcher_config(const char *var, const char *value, void *data)\n {\n \tif (!strcmp(var, \"filewatcher.minfiles\")) {\n@@ -1484,6 +1547,7 @@ static void validate_watcher(struct index_state *istate, const char *path)\n \t\tif (send_watcher(istate->watcher, NULL, \"%s\", sb.buf) > 0 &&\n \t\t (msg = read_watcher(istate->watcher, NULL, NULL)) != NULL &&\n \t\t !strcmp(msg, sb.buf)) { /* good */\n+\t\t\tupdate_watched_files(istate);\n \t\t\tstrbuf_release(&sb);\n \t\t\treturn;\n \t\t}\n@@ -1597,6 +1661,21 @@ static void watch_entries(struct index_state *istate)\n \tfree(sorted);\n }\n \n+static void farewell_watcher(struct index_state *istate,\n+\t\t\t const unsigned char *sha1)\n+{\n+\tint i;\n+\tif (istate->watcher <= 0)\n+\t\treturn;\n+\tsend_watcher(istate->watcher, NULL, \"bye %s\", sha1_to_hex(sha1));\n+\tif (!istate->updated_entries)\n+\t\treturn;\n+\tfor (i = 0; i < istate->updated_entries->nr; i++)\n+\t\tsend_watcher(istate->watcher, NULL, \"forget %s %s\",\n+\t\t\t sha1_to_hex(sha1),\n+\t\t\t istate->updated_entries->items[i].string);\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n@@ -1718,6 +1797,11 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->updated_entries) {\n+\t\tstring_list_clear(istate->updated_entries, 0);\n+\t\tfree(istate->updated_entries);\n+\t\tistate->updated_entries = NULL;\n+\t}\n \treturn 0;\n }\n \n@@ -1778,7 +1862,7 @@ static int write_index_ext_header(git_SHA_CTX *context, int fd,\n \t\t(ce_write(context, fd, &sz, 4) < 0)) ? -1 : 0;\n }\n \n-static int ce_flush(git_SHA_CTX *context, int fd)\n+static int ce_flush(git_SHA_CTX *context, int fd, unsigned char *sha1)\n {\n \tunsigned int left = write_buffer_len;\n \n@@ -1796,6 +1880,8 @@ static int ce_flush(git_SHA_CTX *context, int fd)\n \n \t/* Append the SHA1 signature at the end */\n \tgit_SHA1_Final(write_buffer + left, context);\n+\tif (sha1)\n+\t\thashcpy(sha1, write_buffer + left);\n \tleft += 20;\n \treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n }\n@@ -1960,12 +2046,21 @@ int write_index(struct index_state *istate, int newfd)\n \tint entries = istate->cache_nr;\n \tstruct stat st;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n+\tunsigned char sha1[20];\n \n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n \t\t\tremoved++;\n \t\telse if (cache[i]->ce_flags & CE_WATCHED) {\n \t\t\t/*\n+\t\t\t * CE_VALID when used with CE_WATCHED is not\n+\t\t\t * supposed to be persistent. Next time git\n+\t\t\t * runs, if this entry is still watched and\n+\t\t\t * nothing has changed, CE_VALID will be\n+\t\t\t * reinstated.\n+\t\t\t */\n+\t\t\tcache[i]->ce_flags &= ~CE_VALID;\n+\t\t\t/*\n \t\t\t * We may set CE_WATCHED (but not CE_VALID)\n \t\t\t * early when refresh has not been done\n \t\t\t * yet. At that time we had no idea if the\n@@ -2073,8 +2168,9 @@ int write_index(struct index_state *istate, int newfd)\n \t\t\treturn -1;\n \t}\n \n-\tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n+\tif (ce_flush(&c, newfd, sha1) || fstat(newfd, &st))\n \t\treturn -1;\n+\tfarewell_watcher(istate, sha1);\n \tistate->timestamp.sec = (unsigned int)st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \treturn 0;\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233277","messageId":"1389952060-12297-8-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 07/14] read-cache: add config to start file watcher automatically","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:33Z","receivedAt":"2014-01-17T09:47:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 5 +++++\n file-watcher-lib.c | 18 +++++++++++++++---\n file-watcher-lib.h | 2 +-\n file-watcher.c | 8 ++++++--\n read-cache.c | 17 +++++++++++++++--\n 5 files changed, 42 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex e394399..3316b69 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1038,6 +1038,11 @@ difftool..cmd::\n difftool.prompt::\n \tPrompt before each invocation of the diff tool.\n \n+filewatcher.autorun::\n+\tRun `git file-watcher` automatically if the number of cached\n+\tentries is greater than this limit. Zero means no running\n+\tfile-watcher automatically. Default value is zero.\n+\n filewatcher.minfiles::\n \tStart watching files if the number of watchable files are\n \tabove this limit. Default value is 65536.\ndiff --git a/file-watcher-lib.c b/file-watcher-lib.c\nindex ed14ef9..71c8545 100644\n--- a/file-watcher-lib.c\n+++ b/file-watcher-lib.c\n@@ -1,16 +1,28 @@\n #include \"cache.h\"\n+#include \"run-command.h\"\n \n #define WAIT_TIME 20\t\t/* in ms */\n #define TRACE_KEY \"GIT_TRACE_WATCHER\"\n \n-int connect_watcher(const char *path)\n+int connect_watcher(const char *path, int autorun)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct stat st;\n-\tint fd = -1;\n+\tint fd = -1, ret;\n \n \tstrbuf_addf(&sb, \"%s.watcher\", path);\n-\tif (!stat(sb.buf, &st) && S_ISSOCK(st.st_mode)) {\n+\tret = stat(sb.buf, &st);\n+\tif (autorun && ret && errno == ENOENT) {\n+\t\tconst char *av[] = { \"file-watcher\", \"--daemon\", \"--quiet\", NULL };\n+\t\tstruct child_process cp;\n+\t\tmemset(&cp, 0, sizeof(cp));\n+\t\tcp.git_cmd = 1;\n+\t\tcp.argv = av;\n+\t\tif (run_command(&cp))\n+\t\t\treturn -1;\n+\t\tret = stat(sb.buf, &st);\n+\t}\n+\tif (!ret && S_ISSOCK(st.st_mode)) {\n \t\tstruct sockaddr_un sun;\n \t\tfd = socket(AF_UNIX, SOCK_DGRAM, 0);\n \t\tsun.sun_family = AF_UNIX;\ndiff --git a/file-watcher-lib.h b/file-watcher-lib.h\nindex 0fe9399..ef3d196 100644\n--- a/file-watcher-lib.h\n+++ b/file-watcher-lib.h\n@@ -1,7 +1,7 @@\n #ifndef __FILE_WATCHER_LIB__\n #define __FILE_WATCHER_LIB__\n \n-int connect_watcher(const char *path);\n+int connect_watcher(const char *path, int autorun);\n ssize_t send_watcher(int sockfd, struct sockaddr_un *dest,\n \t\t const char *fmt, ...)\n \t__attribute__((format (printf, 3, 4)));\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 369af37..1b4ac0a 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -168,8 +168,9 @@ int main(int argc, const char **argv)\n \tstruct pollfd pfd[2];\n \tint fd, err, nr;\n \tconst char *prefix;\n-\tint daemon = 0;\n+\tint daemon = 0, quiet = 0;\n \tstruct option options[] = {\n+\t\tOPT__QUIET(&quiet, N_(\"be quiet\")),\n \t\tOPT_BOOL(0, \"daemon\", &daemon,\n \t\t\t N_(\"run in background\")),\n \t\tOPT_END()\n@@ -189,8 +190,11 @@ int main(int argc, const char **argv)\n \tfd = socket(AF_UNIX, SOCK_DGRAM, 0);\n \tsun.sun_family = AF_UNIX;\n \tstrlcpy(sun.sun_path, socket_path, sizeof(sun.sun_path));\n-\tif (bind(fd, (struct sockaddr *)&sun, sizeof(sun)))\n+\tif (bind(fd, (struct sockaddr *)&sun, sizeof(sun))) {\n+\t\tif (quiet)\n+\t\t\texit(128);\n \t\tdie_errno(\"unable to bind to %s\", socket_path);\n+\t}\n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 3aa541d..5dae9eb 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -40,6 +40,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n struct index_state the_index;\n static int watch_lowerlimit = 65536;\n static int recent_limit = 1800;\n+static int autorun_watcher = -1;\n \n static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n {\n@@ -1518,6 +1519,10 @@ failed:\n \n static int watcher_config(const char *var, const char *value, void *data)\n {\n+\tif (!strcmp(var, \"filewatcher.autorun\")) {\n+\t\tautorun_watcher = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(var, \"filewatcher.minfiles\")) {\n \t\twatch_lowerlimit = git_config_int(var, value);\n \t\treturn 0;\n@@ -1538,8 +1543,16 @@ static void validate_watcher(struct index_state *istate, const char *path)\n \t\treturn;\n \t}\n \n-\tgit_config(watcher_config, NULL);\n-\tistate->watcher = connect_watcher(path);\n+\tif (autorun_watcher == -1) {\n+\t\tgit_config(watcher_config, NULL);\n+\t\tif (autorun_watcher == -1)\n+\t\t\tautorun_watcher = 0;\n+\t}\n+\n+\tistate->watcher = connect_watcher(path,\n+\t\t\t\t\t autorun_watcher &&\n+\t\t\t\t\t istate->cache_nr >= autorun_watcher);\n+\tautorun_watcher = 0;\n \tif (istate->watcher != -1) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n \t\tchar *msg;\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233279","messageId":"1389952060-12297-9-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 08/14] read-cache: add GIT_TEST_FORCE_WATCHER for testing","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:34Z","receivedAt":"2014-01-17T09:47:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This can be used to force watcher on when running the test\nsuite.\n\ngit-file-watcher processes are not automatically cleaned up after each\ntest. So after running the test suite you'll be left with plenty\ngit-file-watcher processes that should all end after about a minute.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 5dae9eb..a1245d4 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1544,7 +1544,12 @@ static void validate_watcher(struct index_state *istate, const char *path)\n \t}\n \n \tif (autorun_watcher == -1) {\n-\t\tgit_config(watcher_config, NULL);\n+\t\tif (getenv(\"GIT_TEST_FORCE_WATCHER\")) {\n+\t\t\twatch_lowerlimit = 0;\n+\t\t\trecent_limit = 0;\n+\t\t\tautorun_watcher = 1;\n+\t\t} else\n+\t\t\tgit_config(watcher_config, NULL);\n \t\tif (autorun_watcher == -1)\n \t\t\tautorun_watcher = 0;\n \t}\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233278","messageId":"1389952060-12297-10-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 09/14] file-watcher: add --shutdown and --log options","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:35Z","receivedAt":"2014-01-17T09:47:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher.c | 24 ++++++++++++++++++++++--\n 1 file changed, 22 insertions(+), 2 deletions(-)\n\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 1b4ac0a..df06529 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -113,6 +113,8 @@ static int handle_command(int fd)\n \t\t\tremove_updated(arg + len + 1);\n \t} else if (starts_with(msg, \"watch \")) {\n \t\twatch_paths(msg + 6, len - 6, fd, &sun);\n+\t} else if ((arg = skip_prefix(msg, \"log \"))) {\n+\t\tfprintf(stderr, \"log %s\\n\", arg);\n \t} else if (!strcmp(msg, \"die\")) {\n \t\texit(0);\n \t} else {\n@@ -168,11 +170,16 @@ int main(int argc, const char **argv)\n \tstruct pollfd pfd[2];\n \tint fd, err, nr;\n \tconst char *prefix;\n-\tint daemon = 0, quiet = 0;\n+\tint daemon = 0, quiet = 0, shutdown = 0;\n+\tconst char *log_string = NULL;\n \tstruct option options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"be quiet\")),\n \t\tOPT_BOOL(0, \"daemon\", &daemon,\n-\t\t\t N_(\"run in background\")),\n+\t\t\t N_(\"run in background (default)\")),\n+\t\tOPT_BOOL(0, \"shutdown\", &shutdown,\n+\t\t\t N_(\"shut down running file-watcher daemon\")),\n+\t\tOPT_STRING(0, \"log\", &log_string, \"string\",\n+\t\t\t N_(\"string to log to index.watcher.log\")),\n \t\tOPT_END()\n \t};\n \n@@ -190,11 +197,24 @@ int main(int argc, const char **argv)\n \tfd = socket(AF_UNIX, SOCK_DGRAM, 0);\n \tsun.sun_family = AF_UNIX;\n \tstrlcpy(sun.sun_path, socket_path, sizeof(sun.sun_path));\n+\n+\tif (shutdown || log_string) {\n+\t\tstruct stat st;\n+\t\tif (stat(socket_path, &st) || !S_ISSOCK(st.st_mode))\n+\t\t\treturn 0;\n+\t\tif (log_string && send_watcher(fd, &sun, \"log %s\", log_string) < 0)\n+\t\t\tdie_errno(\"failed to shut file-watcher down\");\n+\t\tif (shutdown && send_watcher(fd, &sun, \"die\") < 0)\n+\t\t\tdie_errno(\"failed to shut file-watcher down\");\n+\t\treturn 0;\n+\t}\n+\n \tif (bind(fd, (struct sockaddr *)&sun, sizeof(sun))) {\n \t\tif (quiet)\n \t\t\texit(128);\n \t\tdie_errno(\"unable to bind to %s\", socket_path);\n \t}\n+\n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n \n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233282","messageId":"1389952060-12297-11-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 10/14] file-watcher: automatically quit","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:36Z","receivedAt":"2014-01-17T09:47:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If $GIT_DIR/index.watcher or $GIT_DIR/index is gone, exit. We could\nwatch this path too, but we'll waste precious resources (at least with\ninotify). And with inotify, it seems to miss the case when $GIT_DIR is\nmoved. Just check if the socket path still exists every minute.\n\nAs the last resort, if we do not receive any commands in the last 6\nhours, exit. The code is structured this way because later on inotify\nis also polled. On an busy watched directory, the timeout may never\nhappen for us to kil the watcher, even if index.watcher is already\ngone.\n\nFor mass cleanup, \"killall -USR1 git-file-watcher\" asks every watcher\nprocess to question the purpose of its existence.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher.c | 38 +++++++++++++++++++++++++++++++++++---\n 1 file changed, 35 insertions(+), 3 deletions(-)\n\ndiff --git a/file-watcher.c b/file-watcher.c\nindex df06529..f334e23 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -163,6 +163,12 @@ static void daemonize(void)\n #endif\n }\n \n+static int check_exit_please;\n+static void check_exit_signal(int signo)\n+{\n+\tcheck_exit_please = 1;\n+}\n+\n int main(int argc, const char **argv)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -172,6 +178,8 @@ int main(int argc, const char **argv)\n \tconst char *prefix;\n \tint daemon = 0, quiet = 0, shutdown = 0;\n \tconst char *log_string = NULL;\n+\tstruct stat socket_st;\n+\tstruct timeval tv_last_command;\n \tstruct option options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"be quiet\")),\n \t\tOPT_BOOL(0, \"daemon\", &daemon,\n@@ -217,6 +225,10 @@ int main(int argc, const char **argv)\n \n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n+\tsigchain_push(SIGUSR1, check_exit_signal);\n+\n+\tif (stat(socket_path, &socket_st))\n+\t\tdie_errno(\"failed to stat %s\", socket_path);\n \n \tif (daemon) {\n \t\tstrbuf_addf(&sb, \"%s.log\", socket_path);\n@@ -234,17 +246,37 @@ int main(int argc, const char **argv)\n \tpfd[nr].fd = fd;\n \tpfd[nr++].events = POLLIN;\n \n+\tgettimeofday(&tv_last_command, NULL);\n \tfor (;;) {\n-\t\tif (poll(pfd, nr, -1) < 0) {\n+\t\tint check_exit = check_exit_please;\n+\t\tint ret = poll(pfd, nr, check_exit ? 0 : 60 * 1000);\n+\t\tif (ret < 0) {\n \t\t\tif (errno != EINTR) {\n \t\t\t\terror(\"Poll failed, resuming: %s\", strerror(errno));\n \t\t\t\tsleep(1);\n \t\t\t}\n \t\t\tcontinue;\n+\t\t} else if (!ret)\n+\t\t\tcheck_exit = 1;\n+\n+\t\tif ((pfd[0].revents & POLLIN)) {\n+\t\t\tif (handle_command(fd))\n+\t\t\t\tbreak;\n+\t\t\tgettimeofday(&tv_last_command, NULL);\n \t\t}\n \n-\t\tif ((pfd[0].revents & POLLIN) && handle_command(fd))\n-\t\t\tbreak;\n+\t\tif (check_exit) {\n+\t\t\tstruct stat st;\n+\t\t\tstruct timeval now;\n+\t\t\tgettimeofday(&now, NULL);\n+\t\t\tif (tv_last_command.tv_sec + 6 * 60 < now.tv_sec)\n+\t\t\t\tbreak;\n+\t\t\tif (stat(socket_path, &st) ||\n+\t\t\t st.st_ino != socket_st.st_ino ||\n+\t\t\t stat(get_index_file(), &st))\n+\t\t\t\tbreak;\n+\t\t\tcheck_exit_please = 0;\n+\t\t}\n \t}\n \treturn 0;\n }\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233281","messageId":"1389952060-12297-12-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH/WIP v2 11/14] file-watcher: support inotify","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T09:47:37Z","receivedAt":"2014-01-17T09:47:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\"git diff\" on webkit:\n\n no file watcher 1st run subsequent runs\nreal 0m1.361s 0m1.445s 0m0.691s\nuser 0m0.889s 0m0.940s 0m0.649s\nsys 0m0.469s 0m0.495s 0m0.040s\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n config.mak.uname | 1 +\n file-watcher.c | 194 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n git-compat-util.h | 3 +\n 3 files changed, 198 insertions(+)\n\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 7d31fad..ee548f5 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -33,6 +33,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_PATHS_H = YesPlease\n \tLIBC_CONTAINS_LIBINTL = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tBASIC_CFLAGS += -DHAVE_INOTIFY\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tNO_STRLCPY = YesPlease\ndiff --git a/file-watcher.c b/file-watcher.c\nindex f334e23..356b58a 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -15,6 +15,166 @@ static char index_signature[41];\n static struct string_list updated = STRING_LIST_INIT_DUP;\n static int updated_sorted;\n \n+#ifdef HAVE_INOTIFY\n+\n+static struct string_list watched_dirs = STRING_LIST_INIT_DUP;\n+static int watched_dirs_sorted;\n+static int inotify_fd;\n+\n+struct dir_info {\n+\tint wd;\n+\tstruct string_list names;\n+\tint names_sorted;\n+};\n+\n+static void reset_watches(void)\n+{\n+\tint i;\n+\tfor (i = 0; i < watched_dirs.nr; i++) {\n+\t\tstruct dir_info *dir = watched_dirs.items[i].util;\n+\t\tinotify_rm_watch(inotify_fd, dir->wd);\n+\t\tstring_list_clear(&dir->names, 0);\n+\t}\n+\tstring_list_clear(&watched_dirs, 1);\n+}\n+\n+static void update(const char *base, const char *name)\n+{\n+\tif (!strcmp(base, \".\"))\n+\t\tstring_list_append(&updated, name);\n+\telse {\n+\t\tstatic struct strbuf sb = STRBUF_INIT;\n+\t\tstrbuf_addf(&sb, \"%s/%s\", base, name);\n+\t\tstring_list_append(&updated, sb.buf);\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\tupdated_sorted = 0;\n+}\n+\n+static int do_handle_inotify(const struct inotify_event *event)\n+{\n+\tstruct dir_info *dir;\n+\tstruct string_list_item *item;\n+\tint i;\n+\n+\tif (event->mask & (IN_Q_OVERFLOW | IN_UNMOUNT)) {\n+\t\t/*\n+\t\t * The connectionless nature of file watcher means we\n+\t\t * can never tell we are reset in the middle of a\n+\t\t * \"session\" because there are no \"sessions\". Close\n+\t\t * the socket so all clients can react on it.\n+\t\t */\n+\t\texit(0);\n+\t}\n+\n+\t/* Should have indexed them for faster access like trast's watch */\n+\tfor (i = 0; i < watched_dirs.nr; i++) {\n+\t\tstruct dir_info *dir = watched_dirs.items[i].util;\n+\t\tif (dir->wd == event->wd)\n+\t\t\tbreak;\n+\t}\n+\tif (i == watched_dirs.nr)\n+\t\treturn 0;\n+\tdir = watched_dirs.items[i].util;\n+\n+\t/*\n+\t * If something happened to the watched directory, consider\n+\t * everything inside modified\n+\t */\n+\tif (event->mask & (IN_DELETE_SELF | IN_MOVE_SELF)) {\n+\t\tint dir_idx = i;\n+\t\tfor (i = 0; i < dir->names.nr; i++)\n+\t\t\tupdate(watched_dirs.items[dir_idx].string,\n+\t\t\t dir->names.items[i].string);\n+\t\tinotify_rm_watch(inotify_fd, dir->wd);\n+\t\tunsorted_string_list_delete_item(&watched_dirs, dir_idx, 1);\n+\t\treturn 0;\n+\t}\n+\n+\tif (!dir->names_sorted) {\n+\t\tsort_string_list(&dir->names);\n+\t\tdir->names_sorted = 1;\n+\t}\n+\titem = string_list_lookup(&dir->names, event->name);\n+\tif (item) {\n+\t\tupdate(watched_dirs.items[i].string, item->string);\n+\t\tunsorted_string_list_delete_item(&dir->names,\n+\t\t\t\t\t\t item - dir->names.items, 0);\n+\t\tif (dir->names.nr == 0) {\n+\t\t\tinotify_rm_watch(inotify_fd, dir->wd);\n+\t\t\tunsorted_string_list_delete_item(&watched_dirs, i, 1);\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int handle_inotify(int fd)\n+{\n+\tstatic char buf[10 * (sizeof(struct inotify_event) + NAME_MAX + 1)];\n+\tstruct inotify_event *event;\n+\tint offset = 0;\n+\tint len = read(fd, buf, sizeof(buf));\n+\tif (len <= 0)\n+\t\treturn -1;\n+\tfor (event = (struct inotify_event *)(buf + offset);\n+\t offset < len;\n+\t offset += sizeof(struct inotify_event) + event->len) {\n+\t\tif (do_handle_inotify(event))\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int watch_path(char *path)\n+{\n+\tstruct string_list_item *item;\n+\tchar *sep = strrchr(path, '/');\n+\tstruct dir_info *dir;\n+\tconst char *dirname = \".\";\n+\n+\tif (sep) {\n+\t\t*sep = '\\0';\n+\t\tdirname = path;\n+\t}\n+\n+\tif (!watched_dirs_sorted) {\n+\t\tsort_string_list(&watched_dirs);\n+\t\twatched_dirs_sorted = 1;\n+\t}\n+\titem = string_list_lookup(&watched_dirs, dirname);\n+\tif (!item) {\n+\t\t/*\n+\t\t * IN_CREATE is not included because we're targetting\n+\t\t * lstat() for index vs worktree. If a file is not\n+\t\t * tracked in index, it's not worth watching. If the\n+\t\t * index has it, but the worktree is already gone\n+\t\t * before watching, the file has already been marked\n+\t\t * modified and should _not_ be watched.\n+\t\t *\n+\t\t * Problematic: IN_DONT_FOLLOW\n+\t\t */\n+\t\tint ret = inotify_add_watch(inotify_fd, dirname,\n+\t\t\t\t\t IN_DELETE_SELF | IN_MOVE_SELF |\n+\t\t\t\t\t IN_ATTRIB | IN_DELETE | IN_MODIFY |\n+\t\t\t\t\t IN_MOVED_FROM | IN_MOVED_TO);\n+\t\tif (ret < 0)\n+\t\t\treturn -1;\n+\t\tdir = xmalloc(sizeof(*dir));\n+\t\tmemset(dir, 0, sizeof(*dir));\n+\t\tdir->wd = ret;\n+\t\tdir->names.strdup_strings = 1;\n+\t\titem = string_list_append(&watched_dirs, dirname);\n+\t\titem->util = dir;\n+\t\twatched_dirs_sorted = 0;\n+\t}\n+\tdir = item->util;\n+\tstring_list_append(&dir->names, sep ? sep + 1 : path);\n+\tdir->names_sorted = 0;\n+\treturn 0;\n+}\n+\n+#else\n+\n static int watch_path(char *path)\n {\n \t/*\n@@ -26,8 +186,15 @@ static int watch_path(char *path)\n \treturn -1;\n }\n \n+static void reset_watches(void)\n+{\n+}\n+\n+#endif\n+\n static void reset(void)\n {\n+\treset_watches();\n \tstring_list_clear(&updated, 0);\n \tindex_signature[0] = '\\0';\n }\n@@ -180,6 +347,7 @@ int main(int argc, const char **argv)\n \tconst char *log_string = NULL;\n \tstruct stat socket_st;\n \tstruct timeval tv_last_command;\n+\tstruct timeval tv_last_inotify;\n \tstruct option options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"be quiet\")),\n \t\tOPT_BOOL(0, \"daemon\", &daemon,\n@@ -193,6 +361,15 @@ int main(int argc, const char **argv)\n \n \tgit_extract_argv0_path(argv[0]);\n \tgit_setup_gettext();\n+\n+#ifdef HAVE_INOTIFY\n+\tinotify_fd = inotify_init();\n+\tif (inotify_fd < 0)\n+\t\tdie_errno(\"unable to initialize inotify\");\n+#else\n+\tdie(\"no file watching mechanism is supported\");\n+#endif\n+\n \tprefix = setup_git_directory();\n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t file_watcher_usage, 0);\n@@ -245,8 +422,13 @@ int main(int argc, const char **argv)\n \tnr = 0;\n \tpfd[nr].fd = fd;\n \tpfd[nr++].events = POLLIN;\n+#ifdef HAVE_INOTIFY\n+\tpfd[nr].fd = inotify_fd;\n+\tpfd[nr++].events = POLLIN;\n+#endif\n \n \tgettimeofday(&tv_last_command, NULL);\n+\tgettimeofday(&tv_last_inotify, NULL);\n \tfor (;;) {\n \t\tint check_exit = check_exit_please;\n \t\tint ret = poll(pfd, nr, check_exit ? 0 : 60 * 1000);\n@@ -265,6 +447,18 @@ int main(int argc, const char **argv)\n \t\t\tgettimeofday(&tv_last_command, NULL);\n \t\t}\n \n+#ifdef HAVE_INOTIFY\n+\t\tif ((pfd[1].revents & POLLIN)) {\n+\t\t\tstruct timeval now;\n+\t\t\tif (handle_inotify(inotify_fd))\n+\t\t\t\tbreak;\n+\t\t\tgettimeofday(&now, NULL);\n+\t\t\tif (tv_last_inotify.tv_sec + 60 < now.tv_sec) {\n+\t\t\t\tcheck_exit = 1;\n+\t\t\t\ttv_last_inotify = now;\n+\t\t\t}\n+\t\t}\n+#endif\n \t\tif (check_exit) {\n \t\t\tstruct stat st;\n \t\t\tstruct timeval now;\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex cbd86c3..de5996a 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -128,6 +128,9 @@\n #else\n #include \n #endif\n+#ifdef HAVE_INOTIFY\n+#include \n+#endif\n \n #if defined(__MINGW32__)\n /* pull in Windows compatibility stuff */\n-- \n1.8.5.1.208.g05b12ea\n"},{"id":"233284","messageId":"87ha92c0uu.fsf@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH/WIP v2 02/14] read-cache: new extension to mark what file is watched","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2014-01-17T11:19:05Z","receivedAt":"2014-01-17T11:19:05Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> If an entry is \"watched\", git lets an external program decide if the\n> entry is modified or not. It's more like --assume-unchanged, but\n> designed to be controlled by machine.\n>\n> We are running out of on-disk ce_flags, so instead of extending\n> on-disk entry format again, \"watched\" flags are in-core only and\n> stored as extension instead.\n\nAs you said yourself in http://thread.gmane.org/gmane.comp.version-control.git/240339/focus=240385\nthis is not quite true. As for your explanation there,\n\n> Anyway using extended flags means 2 extra bytes per entry for\n> almost every entry in this case (and for index v5 it means redoing\n> crc32 for almost every entry too when the bit is updated) so it may\n> still be a good idea to keep the new flag separate.\n\nI don't think adding 2 extra bytes would be too bad, since we are\nalready using 62 bytes plus the bytes for the filename for each index\nentry, so it would be a less than 3% increase in the index file size.\n(And the extended flags may be used anyway in some cases)\n\nAs for index-v5 (if that's ever going to happen), it depends mostly on\nhow often the CE_WATCHED is going to be updated, to decide whether it\nmakes sense to store this as extension.\n\nThat said, I don't care too deeply if it's stored one way or another,\nbut I think it would be good to update the commit message with a better\nrationale for the choice.\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> cache.h | 2 ++\n> read-cache.c | 41 ++++++++++++++++++++++++++++++++++++++++-\n> 2 files changed, 42 insertions(+), 1 deletion(-)\n>\n> diff --git a/cache.h b/cache.h\n> index a09d622..069dce7 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -168,6 +168,8 @@ struct cache_entry {\n> /* used to temporarily mark paths matched by pathspecs */\n> #define CE_MATCHED (1 << 26)\n>\n> +#define CE_WATCHED (1 << 27)\n> +\n> /*\n> * Extended on-disk flags\n> */\n> diff --git a/read-cache.c b/read-cache.c\n> index fe1d153..6f21e3f 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -33,6 +33,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n> #define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n> #define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n> #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n> +#define CACHE_EXT_WATCH 0x57415443\t /* \"WATC\" */\n>\n> struct index_state the_index;\n>\n> @@ -1293,6 +1294,19 @@ static int verify_hdr(struct cache_header *hdr,\n> \treturn 0;\n> }\n>\n> +static void read_watch_extension(struct index_state *istate, uint8_t *data,\n> +\t\t\t\t unsigned long sz)\n> +{\n> +\tint i;\n> +\tif ((istate->cache_nr + 7) / 8 != sz) {\n> +\t\terror(\"invalid 'WATC' extension\");\n> +\t\treturn;\n> +\t}\n> +\tfor (i = 0; i < istate->cache_nr; i++)\n> +\t\tif (data[i / 8] & (1 << (i % 8)))\n> +\t\t\tistate->cache[i]->ce_flags |= CE_WATCHED;\n> +}\n> +\n> static int read_index_extension(struct index_state *istate,\n> \t\t\t\tconst char *ext, void *data, unsigned long sz)\n> {\n> @@ -1303,6 +1317,9 @@ static int read_index_extension(struct index_state *istate,\n> \tcase CACHE_EXT_RESOLVE_UNDO:\n> \t\tistate->resolve_undo = resolve_undo_read(data, sz);\n> \t\tbreak;\n> +\tcase CACHE_EXT_WATCH:\n> +\t\tread_watch_extension(istate, data, sz);\n> +\t\tbreak;\n> \tdefault:\n> \t\tif (*ext < 'A' || 'Z' < *ext)\n> \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n> @@ -1781,7 +1798,7 @@ int write_index(struct index_state *istate, int newfd)\n> {\n> \tgit_SHA_CTX c;\n> \tstruct cache_header hdr;\n> -\tint i, err, removed, extended, hdr_version;\n> +\tint i, err, removed, extended, hdr_version, has_watches = 0;\n> \tstruct cache_entry **cache = istate->cache;\n> \tint entries = istate->cache_nr;\n> \tstruct stat st;\n> @@ -1790,6 +1807,8 @@ int write_index(struct index_state *istate, int newfd)\n> \tfor (i = removed = extended = 0; i < entries; i++) {\n> \t\tif (cache[i]->ce_flags & CE_REMOVE)\n> \t\t\tremoved++;\n> +\t\telse if (cache[i]->ce_flags & CE_WATCHED)\n> +\t\t\thas_watches++;\n>\n> \t\t/* reduce extended entries if possible */\n> \t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n> @@ -1861,6 +1880,26 @@ int write_index(struct index_state *istate, int newfd)\n> \t\tif (err)\n> \t\t\treturn -1;\n> \t}\n> +\tif (has_watches) {\n> +\t\tint id, sz = (entries - removed + 7) / 8;\n> +\t\tuint8_t *data = xmalloc(sz);\n> +\t\tmemset(data, 0, sz);\n> +\t\tfor (i = 0, id = 0; i < entries && has_watches; i++) {\n> +\t\t\tstruct cache_entry *ce = cache[i];\n> +\t\t\tif (ce->ce_flags & CE_REMOVE)\n> +\t\t\t\tcontinue;\n> +\t\t\tif (ce->ce_flags & CE_WATCHED) {\n> +\t\t\t\tdata[id / 8] |= 1 << (id % 8);\n> +\t\t\t\thas_watches--;\n> +\t\t\t}\n> +\t\t\tid++;\n> +\t\t}\n> +\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCH, sz) < 0\n> +\t\t\t|| ce_write(&c, newfd, data, sz) < 0;\n> +\t\tfree(data);\n> +\t\tif (err)\n> +\t\t\treturn -1;\n> +\t}\n>\n> \tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n> \t\treturn -1;\n> --\n> 1.8.5.1.208.g05b12ea\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n\n--\nThomas\n"},{"id":"233290","messageId":"52D94B2D.2@web.de","threadId":"35642","inReplyTo":"1389952060-12297-4-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH/WIP v2 03/14] read-cache: connect to file watcher","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-01-17T15:24:29Z","receivedAt":"2014-01-17T15:24:29Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2014-01-17 10.47, Nguyễn Thái Ngọc Duy wrote:\n[snip[\n> diff --git a/file-watcher-lib.c b/file-watcher-lib.c\n\n\n> +int connect_watcher(const char *path)\nCould it be worth to check if we can use some code from unix-socket.c ?\n\nEspecially important could be that unix_sockaddr_init() wotks around a problem\nwhen \"long\" path names are used. \n"},{"id":"233292","messageId":"CACsJy8DHDd6AFF3tiEgs+TcVdcB4GZQQ2rgMTB-bU+wdKFH0yQ@mail.gmail.com","threadId":"35642","inReplyTo":"52D94B2D.2@web.de","subject":"Re: [PATCH/WIP v2 03/14] read-cache: connect to file watcher","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-17T16:21:24Z","receivedAt":"2014-01-17T16:21:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jan 17, 2014 at 10:24 PM, Torsten Bögershausen wrote:\n> On 2014-01-17 10.47, Nguyễn Thái Ngọc Duy wrote:\n> [snip[\n>> diff --git a/file-watcher-lib.c b/file-watcher-lib.c\n>\n>\n>> +int connect_watcher(const char *path)\n> Could it be worth to check if we can use some code from unix-socket.c ?\n>\n> Especially important could be that unix_sockaddr_init() wotks around a problem\n> when \"long\" path names are used.\n>\n\nThanks! I did not even know about unix-socket.c. Well, I never paid\nattention to credential-cache.c :(\n-- \nDuy\n"},{"id":"233382","messageId":"87mwirewd7.fsf@thomasrast.ch","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH/WIP v2 00/14] inotify support","fromName":"Thomas Rast","fromEmail":"tr@thomasrast.ch","sentAt":"2014-01-19T17:04:36Z","receivedAt":"2014-01-19T17:04:36Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n> read-cache: save trailing sha-1\n> read-cache: new extension to mark what file is watched\n> read-cache: connect to file watcher\n> read-cache: ask file watcher to watch files\n> read-cache: put some limits on file watching\n> read-cache: get modified file list from file watcher\n> read-cache: add config to start file watcher automatically\n> read-cache: add GIT_TEST_FORCE_WATCHER for testing\n> file-watcher: add --shutdown and --log options\n> file-watcher: automatically quit\n> file-watcher: support inotify\n> file-watcher: exit when cwd is gone\n> pkt-line.c: increase buffer size to 8192\n> t1301: exclude sockets from file permission check\n\nI never got the last three patches, did you send them?\n\nAlso, this doesn't cleanly apply anywhere at my end. Can you push it\nsomewhere for easier experimentation?\n\n> This is getting in better shape. Still wondering if the design is\n> right, so documentation, tests and some error cases are still\n> neglected. I have not addressed Jonathan's and Jeff's comments in this\n> reroll, but I haven't forgotten them yet. The test suite seems to be\n> fine when file-watcher is forced on with GIT_TEST_FORCE_WATCHER set..\n\nI tried to figure out whether there were any corners you are painting it\ninto, but doing so without a clear overview of the protocol makes my\nhead spin.\n\nSo here's what I gather from the patches (I would have started from the\nfinal result, but see above).\n\nThe slow path, before the watcher is ready, works like:\n\n spawn watcher\n send \"hello $index_sha1\"\n receive \"hello \"\n mark most files CE_WATCHED\n send \"watch $paths\" for each CE_WATCHED\n the paths are actually a pkt-line-encoded series of paths\n get back \"fine $n\" (n = number of watches processed)\n do work as usual (including lstat())\n\nThe fast path:\n\n load CE_WATCHED from index\n send \"hello $index_sha1\"\n receive \"hello $index_sha1\"\n send \"status\"\n get back \"new $path\" for each changed file\n again aggregate\n mark files that are \"new\" as ~CE_WATCHED\n files that are CE_WATCHED can skip their lstat(), since they are unchanged\n\nOn index writing (both paths):\n\n save CE_WATCHED in index extension\n send \"bye $index_sha1\"\n send \"forget $index_sha1 $path\" for every path that is known to be changed\n\nSo I think the watcher protocol can be specified roughly as:\n\n Definitions: The watcher is a separate process that maintains a set of\n _watched paths_. Git uses the commands below to add or remove paths\n from this set.\n\n In addition it maintains a set of _changed paths_, which is a subset\n of the watched paths. Git occasionally queries the changed paths. If\n at any time after a path P was added to the \"watched\" set, P has or\n may have changed, it MUST be added to the \"changed\" set.\n\n Note that a path being \"unchanged\" under this definition does NOT mean\n that it is unchanged in the index as per `git diff`. It may have been\n changed before the watch was issued! Therefore, Git MUST test whether\n the path started out unchanged using lstat() AFTER it was added to the\n \"watched\" set.\n\n Handshake::\n On connecting, git sends \"hello \". is an opaque\n string that identifies the state of the index. The watcher MUST\n respond with \"hello \", where previous_magic is\n obtained from the \"bye\" command (below). If !=\n , the watcher MUST reset state for this repository.\n\n Watch::\n Git sends \"watch \" where the consists of a\n series of 4-digit lengths and literal pathnames (as in the pkt-line\n format) for each path it is interested in. The watcher MUST respond\n with \"fine \" after processing a message that contained paths.\n The watcher MUST add each path to the \"watched\" set and remove it\n from the \"changed\" set (no error if it was already watched, or not\n changed).\n\n Status::\n Git sends \"status\". The watcher MUST respond with its current\n \"changed\" set. This uses the format \"new \", with\n formatted as for the \"watch\" command.\n\n Bye::\n Git sends \"bye \" to indicate it has finished writing the\n index. The watcher must store so as to use it as the\n in a \"hello\" response.\n\n Forget::\n Git sends \"forget \", with the formatted as for\n the \"watch\" command. The watcher SHOULD remove each path in\n from its watched set.\n\nDid I get that approximately straight? Perhaps you can fix up as needed\nand then turn it into the documentation for the protocol.\n\nThere are several points about this that I don't like:\n\n* What does the datagram socket buy us? You already do a lot of\n pkt-line work for the really big messages, so why not just use\n pkt-line everywhere and a streaming socket?\n\n* The handshake should have room for capabilities, so that we can extend\n the protocol.\n\n* The hello/hello handshake is confusing, and probably would allow you\n to point two watchers at each other without them noticing. Make it\n hello/ready, or some other unambiguous choice.\n\n* I took some liberty and tried to fully define the transitions between\n the sets. So even though I'm not sure this is currently handled, I\n made it well-defined to issue \"watch\" for a path that is in the\n changed set.\n\n* \"bye\" is confusing, because in practice git issues \"forget\"s after\n \"bye\". The best I can come up with is \"setstate\", I'm sure you have\n better ideas.\n\nThere's also the problem of ordering guarantees between the socket and\ninotify. I haven't found any, so I would conservatively assume that the\nsocket messages may in fact arrive before inotify, which is a race in\nthe current code. E.g., in the sequence 'touch foo; git status' the\ndaemon may see\n\n socket inotify \n < hello...\n < status\n > new \n touch foo\n\nI think a clever way to handle this would be to add a new command:\n\n Wait::\n This command serves synchronization. Git creates a file of its\n choice in $GIT_DIR/watch (say, `.git/watch/wait.`). Then it\n sends \"wait \". The watcher MUST block until it has processed\n all change notifications up to and including .\n\nThis assumes the FS notification API to be ordered, which appears to be\nthe case for inotify; from inotify(7):\n\n The events returned by reading from an inotify file descriptor form an\n ordered queue. Thus, for example, it is guaranteed that when renaming\n from one directory to another, events will be produced in the correct\n order on the inotify file descriptor.\n\nAs a corollary, from watching .git/watch you get free notification of\n'rm .git/watch/socket' as a termination signal.\n\nWe also need to think about how other OS's APIs work with this. From\nwhat I've heard about the Windows API, it should work well, so that's\ngreat. I suspect the *bsd/darwin API corresponding to inotify won't be\ntoo different, but it's better to work this out now.\n\n> Thomas, you were a proponent of per-user daemon last time. I agree\n> that is a better solution when you need to support submodules. So if\n> you have time, have a look and see if anything I did may prevent\n> per-user daemon changes later (hint, I have a few unfriendly exit() in\n> file-watcher.c). You also worked with inotify before maybe you can\n> help spot some mishandling too as I'm totally new to inotify.\n\nAs you note, the protocol for \"help, I don't know what's true any more\"\n(i.e., it got an inotify buffer overflow event) needs to be a special\n\"reset\" message, not exit().\n\nThe per-repo approach does require keeping open an extra FD per repo\nthough, regardless of whether the daemon is actually for all of the\nuser's repos. The linuxen I run as desktop systems usually default\nulimit -n to 1024, so that's not a massive restriction, but still.\n\nAs a reminder to other reviewers, the reason I wanted a per-user watcher\nis that /proc/sys/fs/inotify/max_user_instances defaults to 128 on my\nsystems. We need one inotify FD per watcher process, and given that a\nfull android tree had something on the order of 300 repos last I looked,\nthat just won't fly.\n\nAs far as inotify corner-cases go, the only one I'm aware of is\ndirectory renames. I suspect we'll have to watch directories all the\nway up to the repository root to reliably detect when this happens. Not\nsure how to best handle this. Perhaps we should declare Git completely\nagnostic wrt such issues, and behind the scenes issue all watches up to\nthe root even if we don't need them for anything other than directory\nrenames.\n\n\nOk, that's probably a confused sum of rambles. Let me know if you can\nmake any sense of it.\n\n-- \nThomas Rast\ntr@thomasrast.ch\n"},{"id":"233383","messageId":"87lhybewcs.fsf@thomasrast.ch","threadId":"35642","inReplyTo":"1389952060-12297-9-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH/WIP v2 08/14] read-cache: add GIT_TEST_FORCE_WATCHER for testing","fromName":"Thomas Rast","fromEmail":"tr@thomasrast.ch","sentAt":"2014-01-19T17:04:51Z","receivedAt":"2014-01-19T17:04:51Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> This can be used to force watcher on when running the test\n> suite.\n>\n> git-file-watcher processes are not automatically cleaned up after each\n> test. So after running the test suite you'll be left with plenty\n> git-file-watcher processes that should all end after about a minute.\n\nProbably not a very good idea, especially in noninteractive use? E.g.,\na bisection through the test suite or parallel test runs on different\ncommits may exhaust the available processes and/or memory.\n\nEach test should make an effort to clean up all watchers before\nterminating.\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> read-cache.c | 7 ++++++-\n> 1 file changed, 6 insertions(+), 1 deletion(-)\n>\n> diff --git a/read-cache.c b/read-cache.c\n> index 5dae9eb..a1245d4 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1544,7 +1544,12 @@ static void validate_watcher(struct index_state *istate, const char *path)\n> \t}\n> \n> \tif (autorun_watcher == -1) {\n> -\t\tgit_config(watcher_config, NULL);\n> +\t\tif (getenv(\"GIT_TEST_FORCE_WATCHER\")) {\n> +\t\t\twatch_lowerlimit = 0;\n> +\t\t\trecent_limit = 0;\n> +\t\t\tautorun_watcher = 1;\n> +\t\t} else\n> +\t\t\tgit_config(watcher_config, NULL);\n> \t\tif (autorun_watcher == -1)\n> \t\t\tautorun_watcher = 0;\n> \t}\n\n-- \nThomas Rast\ntr@thomasrast.ch\n"},{"id":"233384","messageId":"87ha8zewas.fsf@thomasrast.ch","threadId":"35642","inReplyTo":"1389952060-12297-6-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH/WIP v2 05/14] read-cache: put some limits on file watching","fromName":"Thomas Rast","fromEmail":"tr@thomasrast.ch","sentAt":"2014-01-19T17:06:03Z","receivedAt":"2014-01-19T17:06:03Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> watch_entries() is a lot of computation and could trigger a lot more\n> lookups in file-watcher. Normally after the first set of watches are\n> in place, we do not need to update often. Moreover if the number of\n> entries is small, the overhead of file watcher may actually slow git\n> down.\n>\n> This patch only allows to update watches if the number of watchable\n> files is over a limit (and there are new files added if this is not\n> the first time). Measurements on Core i5-2520M and Linux 3.7.6, about\n> 920 lstat() take 1ms. Somewhere between 2^16 and 2^17 lstat calls that\n> it starts to take longer than 100ms. 2^16 is chosen at the minimum\n> limit to start using file watcher.\n>\n> Recently updated files are not considered watchable because they are\n> likely to be updated again soon, not worth the ping-pong game with\n> file watcher. The default limit 30min is just a random value.\n\nBut then a fresh clone of a big repository would not get any benefit\nfrom the watcher?\n\nNot yet sure how to best handle this.\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> Documentation/config.txt | 9 +++++++++\n> cache.h | 1 +\n> read-cache.c | 44 ++++++++++++++++++++++++++++++++++++--------\n> 3 files changed, 46 insertions(+), 8 deletions(-)\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index a405806..e394399 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1038,6 +1038,15 @@ difftool..cmd::\n> difftool.prompt::\n> \tPrompt before each invocation of the diff tool.\n> \n> +filewatcher.minfiles::\n> +\tStart watching files if the number of watchable files are\n> +\tabove this limit. Default value is 65536.\n> +\n> +filewatcher.recentlimit::\n> +\tFiles that are last updated within filewatcher.recentlimit\n> +\tseconds from now are not considered watchable. Default value\n> +\tis 1800 (30 minutes).\n> +\n> fetch.recurseSubmodules::\n> \tThis option can be either set to a boolean value or to 'on-demand'.\n> \tSetting it to a boolean changes the behavior of fetch and pull to\n> diff --git a/cache.h b/cache.h\n> index 0d55551..bcec29b 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -278,6 +278,7 @@ struct index_state {\n> \tstruct cache_tree *cache_tree;\n> \tstruct cache_time timestamp;\n> \tunsigned name_hash_initialized : 1,\n> +\t\t update_watches : 1,\n> \t\t initialized : 1;\n> \tstruct hashmap name_hash;\n> \tstruct hashmap dir_hash;\n> diff --git a/read-cache.c b/read-cache.c\n> index 21c3207..406834a 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -38,6 +38,8 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n> #define CACHE_EXT_WATCH 0x57415443\t /* \"WATC\" */\n> \n> struct index_state the_index;\n> +static int watch_lowerlimit = 65536;\n> +static int recent_limit = 1800;\n> \n> static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n> {\n> @@ -1014,6 +1016,7 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n> \t\t\t(istate->cache_nr - pos - 1) * sizeof(ce));\n> \tset_index_entry(istate, pos, ce);\n> \tistate->cache_changed = 1;\n> +\tistate->update_watches = 1;\n> \treturn 0;\n> }\n> \n> @@ -1300,13 +1303,14 @@ static void read_watch_extension(struct index_state *istate, uint8_t *data,\n> \t\t\t\t unsigned long sz)\n> {\n> \tint i;\n> -\tif ((istate->cache_nr + 7) / 8 != sz) {\n> +\tif ((istate->cache_nr + 7) / 8 + 1 != sz) {\n> \t\terror(\"invalid 'WATC' extension\");\n> \t\treturn;\n> \t}\n> \tfor (i = 0; i < istate->cache_nr; i++)\n> \t\tif (data[i / 8] & (1 << (i % 8)))\n> \t\t\tistate->cache[i]->ce_flags |= CE_WATCHED;\n> +\tistate->update_watches = data[sz - 1];\n> }\n> \n> static int read_index_extension(struct index_state *istate,\n> @@ -1449,6 +1453,19 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n> \treturn ce;\n> }\n> \n> +static int watcher_config(const char *var, const char *value, void *data)\n> +{\n> +\tif (!strcmp(var, \"filewatcher.minfiles\")) {\n> +\t\twatch_lowerlimit = git_config_int(var, value);\n> +\t\treturn 0;\n> +\t}\n> +\tif (!strcmp(var, \"filewatcher.recentlimit\")) {\n> +\t\trecent_limit = git_config_int(var, value);\n> +\t\treturn 0;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> static void validate_watcher(struct index_state *istate, const char *path)\n> {\n> \tint i;\n> @@ -1458,6 +1475,7 @@ static void validate_watcher(struct index_state *istate, const char *path)\n> \t\treturn;\n> \t}\n> \n> +\tgit_config(watcher_config, NULL);\n> \tistate->watcher = connect_watcher(path);\n> \tif (istate->watcher != -1) {\n> \t\tstruct strbuf sb = STRBUF_INIT;\n> @@ -1478,6 +1496,7 @@ static void validate_watcher(struct index_state *istate, const char *path)\n> \t\t\tistate->cache[i]->ce_flags &= ~CE_WATCHED;\n> \t\t\tistate->cache_changed = 1;\n> \t\t}\n> +\tistate->update_watches = 1;\n> }\n> \n> static int sort_by_date(const void *a_, const void *b_)\n> @@ -1524,7 +1543,7 @@ static int do_watch_entries(struct index_state *istate,\n> \treturn 0;\n> }\n> \n> -static inline int ce_watchable(struct cache_entry *ce)\n> +static inline int ce_watchable(struct cache_entry *ce, time_t now)\n> {\n> \treturn ce_uptodate(ce) && /* write_index will catch late ce_uptodate bits */\n> \t\t!(ce->ce_flags & CE_WATCHED) &&\n> @@ -1534,7 +1553,8 @@ static inline int ce_watchable(struct cache_entry *ce)\n> \t\t * obviously. S_IFLNK could be problematic because\n> \t\t * inotify may follow symlinks without IN_DONT_FOLLOW\n> \t\t */\n> -\t\tS_ISREG(ce->ce_mode);\n> +\t\tS_ISREG(ce->ce_mode) &&\n> +\t\t(ce->ce_stat_data.sd_mtime.sec + recent_limit < now);\n> }\n> \n> static void watch_entries(struct index_state *istate)\n> @@ -1544,15 +1564,20 @@ static void watch_entries(struct index_state *istate)\n> \tstruct strbuf sb = STRBUF_INIT;\n> \tint val;\n> \tsocklen_t vallen = sizeof(val);\n> +\ttime_t now = time(NULL);\n> \n> -\tif (istate->watcher <= 0)\n> +\tif (istate->watcher <= 0 || !istate->update_watches)\n> \t\treturn;\n> +\tistate->update_watches = 0;\n> +\tistate->cache_changed = 1;\n> \tfor (i = nr = 0; i < istate->cache_nr; i++)\n> -\t\tif (ce_watchable(istate->cache[i]))\n> +\t\tif (ce_watchable(istate->cache[i], now))\n> \t\t\tnr++;\n> +\tif (nr < watch_lowerlimit)\n> +\t\treturn;\n> \tsorted = xmalloc(sizeof(*sorted) * nr);\n> \tfor (i = nr = 0; i < istate->cache_nr; i++)\n> -\t\tif (ce_watchable(istate->cache[i]))\n> +\t\tif (ce_watchable(istate->cache[i], now))\n> \t\t\tsorted[nr++] = istate->cache[i];\n> \n> \tgetsockopt(istate->watcher, SOL_SOCKET, SO_SNDBUF, &val, &vallen);\n> @@ -1616,6 +1641,7 @@ int read_index_from(struct index_state *istate, const char *path)\n> \tistate->cache_alloc = alloc_nr(istate->cache_nr);\n> \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n> \tistate->initialized = 1;\n> +\tistate->update_watches = 1;\n> \n> \tif (istate->version == 4)\n> \t\tprevious_name = &previous_name_buf;\n> @@ -2024,8 +2050,9 @@ int write_index(struct index_state *istate, int newfd)\n> \t\tif (err)\n> \t\t\treturn -1;\n> \t}\n> -\tif (has_watches) {\n> -\t\tint id, sz = (entries - removed + 7) / 8;\n> +\tif (has_watches ||\n> +\t (istate->watcher != -1 && !istate->update_watches)) {\n> +\t\tint id, sz = (entries - removed + 7) / 8 + 1;\n> \t\tuint8_t *data = xmalloc(sz);\n> \t\tmemset(data, 0, sz);\n> \t\tfor (i = 0, id = 0; i < entries && has_watches; i++) {\n> @@ -2038,6 +2065,7 @@ int write_index(struct index_state *istate, int newfd)\n> \t\t\t}\n> \t\t\tid++;\n> \t\t}\n> +\t\tdata[sz - 1] = istate->update_watches;\n> \t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCH, sz) < 0\n> \t\t\t|| ce_write(&c, newfd, data, sz) < 0;\n> \t\tfree(data);\n\n-- \nThomas Rast\ntr@thomasrast.ch\n"},{"id":"233385","messageId":"87fvojewaf.fsf@thomasrast.ch","threadId":"35642","inReplyTo":"1389952060-12297-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH/WIP v2 02/14] read-cache: new extension to mark what file is watched","fromName":"Thomas Rast","fromEmail":"tr@thomasrast.ch","sentAt":"2014-01-19T17:06:16Z","receivedAt":"2014-01-19T17:06:16Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> If an entry is \"watched\", git lets an external program decide if the\n> entry is modified or not. It's more like --assume-unchanged, but\n> designed to be controlled by machine.\n>\n> We are running out of on-disk ce_flags, so instead of extending\n> on-disk entry format again, \"watched\" flags are in-core only and\n> stored as extension instead.\n\nI wonder if this would be a good use-case for EWAH bitmaps? Presumably\nmost users would end up having only a few large ranges of files that are\nbeing watched. Quite possibly most users would watch *all* files.\n\n-- \nThomas Rast\ntr@thomasrast.ch\n"},{"id":"233391","messageId":"CACsJy8AbsC8m5ju8pj2r3GzyUEiYSp5Kye=d+OhhTDaOhte8hg@mail.gmail.com","threadId":"35642","inReplyTo":"87mwirewd7.fsf@thomasrast.ch","subject":"Re: [PATCH/WIP v2 00/14] inotify support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-20T01:28:26Z","receivedAt":"2014-01-20T01:28:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jan 20, 2014 at 12:04 AM, Thomas Rast wrote:\n> I never got the last three patches, did you send them?\n>\n> Also, this doesn't cleanly apply anywhere at my end. Can you push it\n> somewhere for easier experimentation?\n\nSorry I rebased it on top of kb/fast-hashmap but never got around to\nactually using hash tables. The code is here (and on top of master)\n\nhttps://github.com/pclouds/git.git file-watcher\n\n> So I think the watcher protocol can be specified roughly as:\n>\n> Definitions: The watcher is a separate process that maintains a set of\n> _watched paths_. Git uses the commands below to add or remove paths\n> from this set.\n>\n> In addition it maintains a set of _changed paths_, which is a subset\n> of the watched paths. Git occasionally queries the changed paths. If\n> at any time after a path P was added to the \"watched\" set, P has or\n> may have changed, it MUST be added to the \"changed\" set.\n>\n> Note that a path being \"unchanged\" under this definition does NOT mean\n> that it is unchanged in the index as per `git diff`. It may have been\n> changed before the watch was issued! Therefore, Git MUST test whether\n> the path started out unchanged using lstat() AFTER it was added to the\n> \"watched\" set.\n\nCorrect.\n\n> Handshake::\n> On connecting, git sends \"hello \". is an opaque\n> string that identifies the state of the index. The watcher MUST\n> respond with \"hello \", where previous_magic is\n> obtained from the \"bye\" command (below). If !=\n> , the watcher MUST reset state for this repository.\n\nIn addition, git must reset itself (i.e. clear all CE_WATCHED flags)\nbecause nothing can be trusted at this point.\n\n...\n>\n> Did I get that approximately straight? Perhaps you can fix up as needed\n> and then turn it into the documentation for the protocol.\n\nWill do (and probably steal some of your text above).\n\n> There are several points about this that I don't like:\n>\n> * What does the datagram socket buy us? You already do a lot of\n> pkt-line work for the really big messages, so why not just use\n> pkt-line everywhere and a streaming socket?\n\nWith datagram sockets I did not have to maintain the connected\nsockets, which made it somewhat simpler to handle so far.\n\nThe default SO_SNDBUF goes up to 212k, so we can send up to that\namount without blocking. With current pkt-line we send up to 64k in\n\"watch\" message then we have to wait for \"fine\", which results in more\ncontext switches. But I think we can extend pkt-line's length field to\n5 hex digit to cover this.\n\nStreaming sockets are probably the way to go for per-user daemon, so\nwe can identify a socket with a repo.\n\n> * The handshake should have room for capabilities, so that we can extend\n> the protocol.\n\nYeah. One more point for streaming sockets. With datagram sockets it's\nharder to define a \"session\" and thus hard to add capabilities.\n\n> * The hello/hello handshake is confusing, and probably would allow you\n> to point two watchers at each other without them noticing. Make it\n> hello/ready, or some other unambiguous choice.\n\nOK\n\n> * I took some liberty and tried to fully define the transitions between\n> the sets. So even though I'm not sure this is currently handled, I\n> made it well-defined to issue \"watch\" for a path that is in the\n> changed set.\n\nYes that should avoid races. The path can be removed from \"watched\"\nset later after git acknowledges it.\n\n> * \"bye\" is confusing, because in practice git issues \"forget\"s after\n> \"bye\". The best I can come up with is \"setstate\", I'm sure you have\n> better ideas.\n\nOriginally it was \"forget\", \"forget\", \"forget\" then \"bye\". But with\nthat order, if git crashes before sending \"bye\" we could lose info in\n\"changed\" set so the order was changed but I did not update the\ncommand name.\n\n> There's also the problem of ordering guarantees between the socket and\n> inotify. I haven't found any, so I would conservatively assume that the\n> socket messages may in fact arrive before inotify, which is a race in\n> the current code. E.g., in the sequence 'touch foo; git status' the\n> daemon may see\n>\n> socket inotify\n> < hello...\n> < status\n> > new \n> touch foo\n>\n> I think a clever way to handle this would be to add a new command:\n>\n> Wait::\n> This command serves synchronization. Git creates a file of its\n> choice in $GIT_DIR/watch (say, `.git/watch/wait.`). Then it\n> sends \"wait \". The watcher MUST block until it has processed\n> all change notifications up to and including .\n\nSo wait. inotify event functions as a barrier. Nice.\n\n> As far as inotify corner-cases go, the only one I'm aware of is\n> directory renames. I suspect we'll have to watch directories all the\n> way up to the repository root to reliably detect when this happens. Not\n> sure how to best handle this. Perhaps we should declare Git completely\n> agnostic wrt such issues, and behind the scenes issue all watches up to\n> the root even if we don't need them for anything other than directory\n> renames.\n\nUnder normal circumstances we would watch all directories in the\nworktree anyway. I'll need to write some tests for inotify..\n\n> Ok, that's probably a confused sum of rambles. Let me know if you can\n> make any sense of it.\n\nThank you for your input. Now I'm back to the white board (or paper).\n-- \nDuy\n"},{"id":"233392","messageId":"CACsJy8BQ-Z5NtmxDhLw98q_rianK7Ny_pJLubQ8=KtmTA9g8kg@mail.gmail.com","threadId":"35642","inReplyTo":"87lhybewcs.fsf@thomasrast.ch","subject":"Re: [PATCH/WIP v2 08/14] read-cache: add GIT_TEST_FORCE_WATCHER for testing","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-20T01:32:38Z","receivedAt":"2014-01-20T01:32:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jan 20, 2014 at 12:04 AM, Thomas Rast wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> This can be used to force watcher on when running the test\n>> suite.\n>>\n>> git-file-watcher processes are not automatically cleaned up after each\n>> test. So after running the test suite you'll be left with plenty\n>> git-file-watcher processes that should all end after about a minute.\n>\n> Probably not a very good idea, especially in noninteractive use? E.g.,\n> a bisection through the test suite or parallel test runs on different\n> commits may exhaust the available processes and/or memory.\n\nI think we run out of inotify resources before hitting process/memory\nlimits. At least that's the case when running tests in parallel.\n\n> Each test should make an effort to clean up all watchers before\n> terminating.\n\nFor now it's hard to do this correctly. Maybe once we get the\nmulti-repo file watcher, we could launch one watcher per trash\ndirectory and cleaning up would be easier.\n-- \nDuy\n"},{"id":"233393","messageId":"CACsJy8Cpno0-z5sgZRsRz6H6Oo+V5qa4UtKqNLT+Cs3Cj420Nw@mail.gmail.com","threadId":"35642","inReplyTo":"87ha8zewas.fsf@thomasrast.ch","subject":"Re: [PATCH/WIP v2 05/14] read-cache: put some limits on file watching","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-20T01:36:48Z","receivedAt":"2014-01-20T01:36:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jan 20, 2014 at 12:06 AM, Thomas Rast wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> watch_entries() is a lot of computation and could trigger a lot more\n>> lookups in file-watcher. Normally after the first set of watches are\n>> in place, we do not need to update often. Moreover if the number of\n>> entries is small, the overhead of file watcher may actually slow git\n>> down.\n>>\n>> This patch only allows to update watches if the number of watchable\n>> files is over a limit (and there are new files added if this is not\n>> the first time). Measurements on Core i5-2520M and Linux 3.7.6, about\n>> 920 lstat() take 1ms. Somewhere between 2^16 and 2^17 lstat calls that\n>> it starts to take longer than 100ms. 2^16 is chosen at the minimum\n>> limit to start using file watcher.\n>>\n>> Recently updated files are not considered watchable because they are\n>> likely to be updated again soon, not worth the ping-pong game with\n>> file watcher. The default limit 30min is just a random value.\n>\n> But then a fresh clone of a big repository would not get any benefit\n> from the watcher?\n>\n> Not yet sure how to best handle this.\n\nGaahh, perhaps limit the number of unwatchable recent files to a\nhundred or so in addition to time limit.\n-- \nDuy\n"},{"id":"233394","messageId":"CACsJy8C5z1AoVKCD-w967O+TbXk0-dUJ2AP34JTAsjfeqWhmrg@mail.gmail.com","threadId":"35642","inReplyTo":"87fvojewaf.fsf@thomasrast.ch","subject":"Re: [PATCH/WIP v2 02/14] read-cache: new extension to mark what file is watched","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-20T01:38:25Z","receivedAt":"2014-01-20T01:38:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jan 20, 2014 at 12:06 AM, Thomas Rast wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> If an entry is \"watched\", git lets an external program decide if the\n>> entry is modified or not. It's more like --assume-unchanged, but\n>> designed to be controlled by machine.\n>>\n>> We are running out of on-disk ce_flags, so instead of extending\n>> on-disk entry format again, \"watched\" flags are in-core only and\n>> stored as extension instead.\n>\n> I wonder if this would be a good use-case for EWAH bitmaps? Presumably\n> most users would end up having only a few large ranges of files that are\n> being watched. Quite possibly most users would watch *all* files.\n\nOh yeah. I edited my commit message locally to this a while ago\n\n On webkit.git with\n 182k entries, that's 364k more to be SHA-1'd on current index\n versions, compared to 22k in this format (and even less when\n jk/pack-bitmap ewah graduates and we can use ewah compression)\n-- \nDuy\n"},{"id":"233416","messageId":"87iotecof7.fsf@thomasrast.ch","threadId":"35642","inReplyTo":"CACsJy8AbsC8m5ju8pj2r3GzyUEiYSp5Kye=d+OhhTDaOhte8hg@mail.gmail.com","subject":"Re: [PATCH/WIP v2 00/14] inotify support","fromName":"Thomas Rast","fromEmail":"tr@thomasrast.ch","sentAt":"2014-01-20T21:51:24Z","receivedAt":"2014-01-20T21:51:24Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Duy Nguyen writes:\n\n>> I think a clever way to handle this would be to add a new command:\n>>\n>> Wait::\n>> This command serves synchronization. Git creates a file of its\n>> choice in $GIT_DIR/watch (say, `.git/watch/wait.`). Then it\n>> sends \"wait \". The watcher MUST block until it has processed\n>> all change notifications up to and including .\n>\n> So wait. inotify event functions as a barrier. Nice.\n\nI forgot to specify a return for \"wait\". Not sure you need one, though\ncorrectly handling the timeout (that you apply for all select()) may be\nsomewhat tricky without it.\n\n>> Ok, that's probably a confused sum of rambles. Let me know if you can\n>> make any sense of it.\n>\n> Thank you for your input. Now I'm back to the white board (or paper).\n\nDon't go too far ;-)\n\nThanks a lot for doing this! It's good that you picked it up, and I\nthink your design strikes a good balance in the complexity of the\nprotocol and the daemon's state.\n\n-- \nThomas Rast\ntr@thomasrast.ch\n"},{"id":"233864","messageId":"CACsJy8AuJQTYdHX0080DXWOWBr7MiDhYkNQ4mMAiPjLi3OTfQQ@mail.gmail.com","threadId":"35642","inReplyTo":"87mwirewd7.fsf@thomasrast.ch","subject":"Re: [PATCH/WIP v2 00/14] inotify support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-28T10:46:56Z","receivedAt":"2014-01-28T10:46:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jan 20, 2014 at 12:04 AM, Thomas Rast wrote:\n> There's also the problem of ordering guarantees between the socket and\n> inotify. I haven't found any, so I would conservatively assume that the\n> socket messages may in fact arrive before inotify, which is a race in\n> the current code. E.g., in the sequence 'touch foo; git status' the\n> daemon may see\n>\n> socket inotify\n> < hello...\n> < status\n> > new \n> touch foo\n>\n> I think a clever way to handle this would be to add a new command:\n>\n> Wait::\n> This command serves synchronization. Git creates a file of its\n> choice in $GIT_DIR/watch (say, `.git/watch/wait.`). Then it\n> sends \"wait \". The watcher MUST block until it has processed\n> all change notifications up to and including .\n\nAssuming that the time between foo is touched and the time an event is\nput in the daemon's queue is reasonably small, would emptying the\nevent queue at \"hello\" be enough? To my innocent eyes (at the kernel),\nit seems inotify handling happens immediately after an fs event, and\nit's uninterruptable (or at least not interruptable by another user\nspace process, I don't think we need to care about true interrupts).\nIf that's true, by the time the \"touch syscall\" is finished, the event\nis already sitting in the daemon's queue.\n\nThe problem with wait. is we need to tell the daemon to expect\nit. Otherwise if the daemon processes the \"wait.\" even before\n\"wait\" is sent, it would try to wait for the (lost) \"wait.\"\nevent forever. An extension is git touch wait. regularly. Or\nto keep a queue of processed \"wait.*\" events. Both look ugly imo.\nAnother option is send \"expect wait.\" first, wait for ack,\ntouch wait., then send \"wait\", which is too much.\n-- \nDuy\n"},{"id":"234071","messageId":"1391401754-15347-1-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1389952060-12297-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 00/26] inotify support","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:48Z","receivedAt":"2014-02-03T04:28:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"I'm happy with this now. The only things left are applying ewah on the\nwatch index extension and maybe improve lookup performance a bit. The\nformer needs jk/pack-bitmap graduated. The latter is not urgent. Oh\nand maybe address \"BUGS\" section (more like known limitations) in\ngit-file-watcher.txt.\n\nFor early adopters that fear a buggy file-watcher may cause update\nloss, set GIT_TEST_WATCHED=1 (or 2). It'll do lstat() to verify\nfile-watcher results (so no perf. gain). Beware of race condition\nthat may lead to false positives, mentioned in 20/26 (maybe I should\ndo something about it).\n\nThe series can also be fetched from\n\nhttps://github.com/pclouds/git.git file-watcher\n\nNguyễn Thái Ngọc Duy (26):\n pkt-line.c: rename global variable buffer[] to something less generic\n pkt-line.c: add packet_write_timeout()\n pkt-line.c: add packet_read_line_timeout()\n unix-socket: make unlink() optional in unix_stream_listen()\n Add git-file-watcher and basic connection handling logic\n file-watcher: check socket directory permission\n file-watcher: remove socket on exit\n file-watcher: add --detach\n read-cache: save trailing sha-1\n read-cache: new flag CE_WATCHED to mark what file is watched\n Clear CE_WATCHED when set CE_VALID alone\n read-cache: basic hand shaking to the file watcher\n read-cache: ask file watcher to watch files\n read-cache: put some limits on file watching\n read-cache: get changed file list from file watcher\n git-compat-util.h: add inotify stubs on non-Linux platforms\n file-watcher: inotify support, watching part\n file-watcher: inotify support, notification part\n Wrap CE_VALID test with ce_valid()\n read-cache: new variable to verify file-watcher results\n Support running file watcher with the test suite\n file-watcher: quit if $WATCHER/socket is gone\n file-watcher: tests for the daemon\n ls-files: print CE_WATCHED as W (or \"w\" with CE_VALID)\n file-watcher: tests for the client side\n Disable file-watcher with system inotify on some tests\n\n .gitignore | 2 +\n Documentation/config.txt | 19 +\n Documentation/git-file-watcher.txt (new) | 54 ++\n Documentation/git-ls-files.txt | 1 +\n Documentation/technical/index-format.txt | 9 +\n Makefile | 3 +\n builtin/grep.c | 2 +-\n builtin/ls-files.c | 14 +-\n builtin/update-index.c | 12 +-\n cache.h | 17 +\n config.mak.uname | 1 +\n credential-cache--daemon.c | 2 +-\n daemon.c | 30 +-\n diff-lib.c | 4 +-\n diff.c | 2 +-\n file-watcher-lib.c (new) | 321 +++++++++\n file-watcher-lib.h (new) | 8 +\n file-watcher.c (new) | 1149 ++++++++++++++++++++++++++++++\n git-compat-util.h | 43 ++\n pkt-line.c | 61 +-\n pkt-line.h | 2 +\n read-cache.c | 119 +++-\n setup.c | 25 +\n t/t1011-read-tree-sparse-checkout.sh | 2 +\n t/t2104-update-index-skip-worktree.sh | 2 +\n t/t7011-skip-worktree-reading.sh | 2 +\n t/t7012-skip-worktree-writing.sh | 2 +\n t/t7513-file-watcher.sh (new +x) | 382 ++++++++++\n t/t7514-file-watcher-lib.sh (new +x) | 190 +++++\n test-file-watcher.c (new) | 111 +++\n unix-socket.c | 5 +-\n unix-socket.h | 2 +-\n unpack-trees.c | 2 +-\n wrapper.c | 47 ++\n 34 files changed, 2591 insertions(+), 56 deletions(-)\n create mode 100644 Documentation/git-file-watcher.txt\n create mode 100644 file-watcher-lib.c\n create mode 100644 file-watcher-lib.h\n create mode 100644 file-watcher.c\n create mode 100755 t/t7513-file-watcher.sh\n create mode 100755 t/t7514-file-watcher-lib.sh\n create mode 100644 test-file-watcher.c\n\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234072","messageId":"1391401754-15347-2-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 01/26] pkt-line.c: rename global variable buffer[] to something less generic","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:49Z","receivedAt":"2014-02-03T04:28:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\"buffer\" is a local variable in some other functions. Rename the\nglobal one to make it less confusing.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n pkt-line.c | 11 ++++++-----\n 1 file changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex bc63b3b..eac45ad 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -64,14 +64,15 @@ void packet_buf_flush(struct strbuf *buf)\n }\n \n #define hex(a) (hexchar[(a) & 15])\n-static char buffer[1000];\n+static char write_buffer[1000];\n static unsigned format_packet(const char *fmt, va_list args)\n {\n \tstatic char hexchar[] = \"0123456789abcdef\";\n+\tchar *buffer = write_buffer;\n \tunsigned n;\n \n-\tn = vsnprintf(buffer + 4, sizeof(buffer) - 4, fmt, args);\n-\tif (n >= sizeof(buffer)-4)\n+\tn = vsnprintf(buffer + 4, sizeof(write_buffer) - 4, fmt, args);\n+\tif (n >= sizeof(write_buffer)-4)\n \t\tdie(\"protocol error: impossibly long line\");\n \tn += 4;\n \tbuffer[0] = hex(n >> 12);\n@@ -90,7 +91,7 @@ void packet_write(int fd, const char *fmt, ...)\n \tva_start(args, fmt);\n \tn = format_packet(fmt, args);\n \tva_end(args);\n-\twrite_or_die(fd, buffer, n);\n+\twrite_or_die(fd, write_buffer, n);\n }\n \n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n@@ -101,7 +102,7 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n \tva_start(args, fmt);\n \tn = format_packet(fmt, args);\n \tva_end(args);\n-\tstrbuf_add(buf, buffer, n);\n+\tstrbuf_add(buf, write_buffer, n);\n }\n \n static int get_packet_data(int fd, char **src_buf, size_t *src_size,\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234073","messageId":"1391401754-15347-3-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 02/26] pkt-line.c: add packet_write_timeout()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:50Z","receivedAt":"2014-02-03T04:28:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n pkt-line.c | 15 +++++++++++++++\n pkt-line.h | 1 +\n wrapper.c | 26 ++++++++++++++++++++++++++\n 4 files changed, 43 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex dc040fb..718e32b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1231,6 +1231,7 @@ extern void fsync_or_die(int fd, const char *);\n \n extern ssize_t read_in_full(int fd, void *buf, size_t count);\n extern ssize_t write_in_full(int fd, const void *buf, size_t count);\n+extern ssize_t write_in_full_timeout(int fd, const void *buf, size_t count, int timeout);\n static inline ssize_t write_str_in_full(int fd, const char *str)\n {\n \treturn write_in_full(fd, str, strlen(str));\ndiff --git a/pkt-line.c b/pkt-line.c\nindex eac45ad..cf681e9 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -94,6 +94,21 @@ void packet_write(int fd, const char *fmt, ...)\n \twrite_or_die(fd, write_buffer, n);\n }\n \n+int packet_write_timeout(int fd, int timeout, const char *fmt, ...)\n+{\n+\tstatic struct strbuf sb = STRBUF_INIT;\n+\tva_list args;\n+\tunsigned n;\n+\n+\tif (fd == -1)\n+\t\treturn -1;\n+\tva_start(args, fmt);\n+\tstrbuf_reset(&sb);\n+\tn = format_packet(fmt, args);\n+\tva_end(args);\n+\treturn write_in_full_timeout(fd, write_buffer, n, timeout);\n+}\n+\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n {\n \tva_list args;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 0a838d1..4b93a0c 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -21,6 +21,7 @@\n */\n void packet_flush(int fd);\n void packet_write(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+int packet_write_timeout(int fd, int timeout, const char *fmt, ...) __attribute__((format (printf, 3, 4)));\n void packet_buf_flush(struct strbuf *buf);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n \ndiff --git a/wrapper.c b/wrapper.c\nindex 0cc5636..9a0e289 100644\n--- a/wrapper.c\n+++ b/wrapper.c\n@@ -214,6 +214,32 @@ ssize_t write_in_full(int fd, const void *buf, size_t count)\n \treturn total;\n }\n \n+ssize_t write_in_full_timeout(int fd, const void *buf,\n+\t\t\t size_t count, int timeout)\n+{\n+\tstruct pollfd pfd;\n+\tconst char *p = buf;\n+\tssize_t total = 0;\n+\n+\tpfd.fd = fd;\n+\tpfd.events = POLLOUT;\n+\twhile (count > 0 && poll(&pfd, 1, timeout) > 0 &&\n+\t (pfd.revents & POLLOUT)) {\n+\t\tssize_t written = xwrite(fd, p, count);\n+\t\tif (written < 0)\n+\t\t\treturn -1;\n+\t\tif (!written) {\n+\t\t\terrno = ENOSPC;\n+\t\t\treturn -1;\n+\t\t}\n+\t\tcount -= written;\n+\t\tp += written;\n+\t\ttotal += written;\n+\t}\n+\n+\treturn count ? -1 : total;\n+}\n+\n int xdup(int fd)\n {\n \tint ret = dup(fd);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234074","messageId":"1391401754-15347-4-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 03/26] pkt-line.c: add packet_read_line_timeout()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:51Z","receivedAt":"2014-02-03T04:28:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This version is also gentler than its friend packet_read_line()\nbecause it's designed for side channel I/O that should not abort the\nprogram even if the channel is broken.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n pkt-line.c | 35 +++++++++++++++++++++++++++++++++++\n pkt-line.h | 1 +\n wrapper.c | 21 +++++++++++++++++++++\n 4 files changed, 58 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 718e32b..939db46 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1230,6 +1230,7 @@ extern int write_or_whine_pipe(int fd, const void *buf, size_t count, const char\n extern void fsync_or_die(int fd, const char *);\n \n extern ssize_t read_in_full(int fd, void *buf, size_t count);\n+extern ssize_t read_in_full_timeout(int fd, void *buf, size_t count, int timeout);\n extern ssize_t write_in_full(int fd, const void *buf, size_t count);\n extern ssize_t write_in_full_timeout(int fd, const void *buf, size_t count, int timeout);\n static inline ssize_t write_str_in_full(int fd, const char *str)\ndiff --git a/pkt-line.c b/pkt-line.c\nindex cf681e9..5a07e97 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -229,3 +229,38 @@ char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)\n {\n \treturn packet_read_line_generic(-1, src, src_len, dst_len);\n }\n+\n+char *packet_read_line_timeout(int fd, int timeout, int *len_p)\n+{\n+\tchar *buf = packet_buffer;\n+\tint ret, len, buf_len = sizeof(packet_buffer);\n+\tchar linelen[4];\n+\n+\tif (fd == -1)\n+\t\treturn NULL;\n+\tif ((ret = read_in_full_timeout(fd, linelen, 4, timeout)) < 0)\n+\t\treturn NULL;\n+\tlen = packet_length(linelen);\n+\tif (len < 0) {\n+\t\terror(\"protocol error: bad line length character: %.4s\", linelen);\n+\t\treturn NULL;\n+\t}\n+\tif (!len) {\n+\t\tpacket_trace(\"0000\", 4, 0);\n+\t\tif (len_p)\n+\t\t\t*len_p = 0;\n+\t\treturn \"\";\n+\t}\n+\tlen -= 4;\n+\tif (len >= buf_len) {\n+\t\terror(\"protocol error: bad line length %d\", len);\n+\t\treturn NULL;\n+\t}\n+\tif ((ret = read_in_full_timeout(fd, buf, len, timeout)) < 0)\n+\t\treturn NULL;\n+\tbuf[len] = '\\0';\n+\tif (len_p)\n+\t\t*len_p = len;\n+\tpacket_trace(buf, len, 0);\n+\treturn buf;\n+}\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 4b93a0c..d47dca5 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -69,6 +69,7 @@ int packet_read(int fd, char **src_buffer, size_t *src_len, char\n * packet is written to it.\n */\n char *packet_read_line(int fd, int *size);\n+char *packet_read_line_timeout(int fd, int timeout, int *size);\n \n /*\n * Same as packet_read_line, but read from a buf rather than a descriptor;\ndiff --git a/wrapper.c b/wrapper.c\nindex 9a0e289..9cf10b2 100644\n--- a/wrapper.c\n+++ b/wrapper.c\n@@ -193,6 +193,27 @@ ssize_t read_in_full(int fd, void *buf, size_t count)\n \treturn total;\n }\n \n+ssize_t read_in_full_timeout(int fd, void *buf, size_t count, int timeout)\n+{\n+\tchar *p = buf;\n+\tssize_t total = 0;\n+\tstruct pollfd pfd;\n+\n+\tpfd.fd = fd;\n+\tpfd.events = POLLIN;\n+\twhile (count > 0 && poll(&pfd, 1, timeout) > 0 &&\n+\t (pfd.revents & POLLIN)) {\n+\t\tssize_t loaded = xread(fd, p, count);\n+\t\tif (loaded <= 0)\n+\t\t\treturn -1;\n+\t\tcount -= loaded;\n+\t\tp += loaded;\n+\t\ttotal += loaded;\n+\t}\n+\n+\treturn count ? -1 : total;\n+}\n+\n ssize_t write_in_full(int fd, const void *buf, size_t count)\n {\n \tconst char *p = buf;\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234075","messageId":"1391401754-15347-5-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 04/26] unix-socket: make unlink() optional in unix_stream_listen()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:52Z","receivedAt":"2014-02-03T04:28:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n credential-cache--daemon.c | 2 +-\n unix-socket.c | 5 +++--\n unix-socket.h | 2 +-\n 3 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/credential-cache--daemon.c b/credential-cache--daemon.c\nindex 390f194..1b995a9 100644\n--- a/credential-cache--daemon.c\n+++ b/credential-cache--daemon.c\n@@ -207,7 +207,7 @@ static void serve_cache(const char *socket_path)\n {\n \tint fd;\n \n-\tfd = unix_stream_listen(socket_path);\n+\tfd = unix_stream_listen(socket_path, 1);\n \tif (fd < 0)\n \t\tdie_errno(\"unable to bind to '%s'\", socket_path);\n \ndiff --git a/unix-socket.c b/unix-socket.c\nindex 01f119f..2be1af6 100644\n--- a/unix-socket.c\n+++ b/unix-socket.c\n@@ -93,7 +93,7 @@ fail:\n \treturn -1;\n }\n \n-int unix_stream_listen(const char *path)\n+int unix_stream_listen(const char *path, int replace)\n {\n \tint fd, saved_errno;\n \tstruct sockaddr_un sa;\n@@ -103,7 +103,8 @@ int unix_stream_listen(const char *path)\n \t\treturn -1;\n \tfd = unix_stream_socket();\n \n-\tunlink(path);\n+\tif (replace)\n+\t\tunlink(path);\n \tif (bind(fd, (struct sockaddr *)&sa, sizeof(sa)) < 0)\n \t\tgoto fail;\n \ndiff --git a/unix-socket.h b/unix-socket.h\nindex e271aee..18ee290 100644\n--- a/unix-socket.h\n+++ b/unix-socket.h\n@@ -2,6 +2,6 @@\n #define UNIX_SOCKET_H\n \n int unix_stream_connect(const char *path);\n-int unix_stream_listen(const char *path);\n+int unix_stream_listen(const char *path, int replace);\n \n #endif /* UNIX_SOCKET_H */\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234076","messageId":"1391401754-15347-6-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 05/26] Add git-file-watcher and basic connection handling logic","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:53Z","receivedAt":"2014-02-03T04:28:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"git-file-watcher is a daemon (*) that watches file changes and tells\ngit about them. The intent is to avoid lstat() calls at index refresh\ntime, which could be a lot on big working directory.\n\nThe actual monitoring needs support from OS (inotify, FSEvents,\nFindFirstChangeNotification or kqueue) and is not part of this patch\nor the next few yet. This patch only provides a UNIX socket server.\n\n(*) it will be a a daemon in this end, but in this patch it runs in\nthe foreground.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n .gitignore | 1 +\n Documentation/git-file-watcher.txt (new) | 29 +++++++\n Makefile | 1 +\n file-watcher.c (new) | 140 +++++++++++++++++++++++++++++++\n 4 files changed, 171 insertions(+)\n create mode 100644 Documentation/git-file-watcher.txt\n create mode 100644 file-watcher.c\n\ndiff --git a/.gitignore b/.gitignore\nindex b5f9def..12c78f0 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -56,6 +56,7 @@\n /git-fast-import\n /git-fetch\n /git-fetch-pack\n+/git-file-watcher\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\ndiff --git a/Documentation/git-file-watcher.txt b/Documentation/git-file-watcher.txt\nnew file mode 100644\nindex 0000000..625a389\n--- /dev/null\n+++ b/Documentation/git-file-watcher.txt\n@@ -0,0 +1,29 @@\n+git-file-watcher(1)\n+===================\n+\n+NAME\n+----\n+git-file-watcher - File update notification daemon\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git file-watcher' [options] \n+\n+DESCRIPTION\n+-----------\n+This program watches file changes in a git working directory and let\n+Git now what files have been changed so that Git does not have to call\n+lstat(2) to detect that itself.\n+\n+OPTIONS\n+-------\n+\n+SEE ALSO\n+--------\n+linkgit:git-update-index[1],\n+linkgit:git-config[1]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex dddaf4f..8eef0d6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -536,6 +536,7 @@ PROGRAMS += $(EXTRA_PROGRAMS)\n PROGRAM_OBJS += credential-store.o\n PROGRAM_OBJS += daemon.o\n PROGRAM_OBJS += fast-import.o\n+PROGRAM_OBJS += file-watcher.o\n PROGRAM_OBJS += http-backend.o\n PROGRAM_OBJS += imap-send.o\n PROGRAM_OBJS += sh-i18n--envsubst.o\ndiff --git a/file-watcher.c b/file-watcher.c\nnew file mode 100644\nindex 0000000..a6d7f17\n--- /dev/null\n+++ b/file-watcher.c\n@@ -0,0 +1,140 @@\n+#include \"cache.h\"\n+#include \"sigchain.h\"\n+#include \"parse-options.h\"\n+#include \"exec_cmd.h\"\n+#include \"unix-socket.h\"\n+\n+static const char *const file_watcher_usage[] = {\n+\tN_(\"git file-watcher [options] \"),\n+\tNULL\n+};\n+\n+struct connection {\n+\tint sock;\n+};\n+\n+static struct connection **conns;\n+static struct pollfd *pfd;\n+static int conns_alloc, pfd_nr, pfd_alloc;\n+\n+static int shutdown_connection(int id)\n+{\n+\tstruct connection *conn = conns[id];\n+\tconns[id] = NULL;\n+\tpfd[id].fd = -1;\n+\tif (!conn)\n+\t\treturn 0;\n+\tclose(conn->sock);\n+\tfree(conn);\n+\treturn 0;\n+}\n+\n+static int handle_command(int conn_id)\n+{\n+\treturn 0;\n+}\n+\n+static void accept_connection(int fd)\n+{\n+\tstruct connection *conn;\n+\tint client = accept(fd, NULL, NULL);\n+\tif (client < 0) {\n+\t\twarning(_(\"accept failed: %s\"), strerror(errno));\n+\t\treturn;\n+\t}\n+\n+\tALLOC_GROW(pfd, pfd_nr + 1, pfd_alloc);\n+\tpfd[pfd_nr].fd = client;\n+\tpfd[pfd_nr].events = POLLIN;\n+\tpfd[pfd_nr].revents = 0;\n+\n+\tALLOC_GROW(conns, pfd_nr + 1, conns_alloc);\n+\tconn = xmalloc(sizeof(*conn));\n+\tmemset(conn, 0, sizeof(*conn));\n+\tconn->sock = client;\n+\tconns[pfd_nr] = conn;\n+\tpfd_nr++;\n+}\n+\n+int main(int argc, const char **argv)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint i, new_nr, fd, quit = 0, nr_common;\n+\tconst char *socket_path = NULL;\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_extract_argv0_path(argv[0]);\n+\tgit_setup_gettext();\n+\targc = parse_options(argc, argv, NULL, options,\n+\t\t\t file_watcher_usage, 0);\n+\tif (argc < 1)\n+\t\tdie(_(\"socket path missing\"));\n+\telse if (argc > 1)\n+\t\tdie(_(\"too many arguments\"));\n+\n+\tsocket_path = argv[0];\n+\tstrbuf_addf(&sb, \"%s/socket\", socket_path);\n+\tfd = unix_stream_listen(sb.buf, 0);\n+\tif (fd == -1)\n+\t\tdie_errno(_(\"unable to listen at %s\"), sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\tnr_common = 1;\n+\tpfd_alloc = pfd_nr = nr_common;\n+\tpfd = xmalloc(sizeof(*pfd) * pfd_alloc);\n+\tpfd[0].fd = fd;\n+\tpfd[0].events = POLLIN;\n+\n+\twhile (!quit) {\n+\t\tif (poll(pfd, pfd_nr, -1) < 0) {\n+\t\t\tif (errno != EINTR) {\n+\t\t\t\terror(\"Poll failed, resuming: %s\",\n+\t\t\t\t strerror(errno));\n+\t\t\t\tsleep(1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (new_nr = i = nr_common; i < pfd_nr; i++) {\n+\t\t\tif (pfd[i].revents & (POLLERR | POLLNVAL))\n+\t\t\t\tshutdown_connection(i);\n+\t\t\telse if ((pfd[i].revents & POLLIN) && conns[i]) {\n+\t\t\t\tunsigned int avail = 1;\n+\t\t\t\t/*\n+\t\t\t\t * pkt-line is not gentle with eof, at\n+\t\t\t\t * least not with\n+\t\t\t\t * packet_read_line(). Avoid feeding\n+\t\t\t\t * eof to it.\n+\t\t\t\t */\n+\t\t\t\tif ((pfd[i].revents & POLLHUP) &&\n+\t\t\t\t ioctl(pfd[i].fd, FIONREAD, &avail))\n+\t\t\t\t\tdie_errno(\"unable to FIONREAD inotify handle\");\n+\t\t\t\t/*\n+\t\t\t\t * We better have a way to handle all\n+\t\t\t\t * packets in one go...\n+\t\t\t\t */\n+\t\t\t\tif (avail)\n+\t\t\t\t\thandle_command(i);\n+\t\t\t\telse\n+\t\t\t\t\tshutdown_connection(i);\n+\t\t\t} else if (pfd[i].revents & POLLHUP)\n+\t\t\t\tshutdown_connection(i);\n+\t\t\tif (!conns[i])\n+\t\t\t\tcontinue;\n+\t\t\tif (i != new_nr) { /* pfd[] is shrunk, move pfd[i] up */\n+\t\t\t\tconns[new_nr] = conns[i];\n+\t\t\t\tpfd[new_nr] = pfd[i];\n+\t\t\t}\n+\t\t\tnew_nr++; /* keep the good socket */\n+\t\t}\n+\t\tpfd_nr = new_nr;\n+\n+\t\tif (pfd[0].revents & POLLIN)\n+\t\t\taccept_connection(pfd[0].fd);\n+\t\tif (pfd[0].revents & (POLLHUP | POLLERR | POLLNVAL))\n+\t\t\tdie(_(\"error on listening socket\"));\n+\t}\n+\treturn 0;\n+}\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234077","messageId":"1391401754-15347-7-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 06/26] file-watcher: check socket directory permission","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:54Z","receivedAt":"2014-02-03T04:28:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Access to the unix socket $WATCHER/socket is covered by $WATCHER's\npermission. While the file watcher does not carry much information,\nrepo file listing is sometimes not something you want other users to\nsee. Make sure $WATCHER has 0700 permission to stop unwanted access.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher.c | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/file-watcher.c b/file-watcher.c\nindex a6d7f17..91f4cfe 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -56,6 +56,37 @@ static void accept_connection(int fd)\n \tpfd_nr++;\n }\n \n+static const char permissions_advice[] =\n+N_(\"The permissions on your socket directory are too loose; other\\n\"\n+ \"processes may be able to read your file listing. Consider running:\\n\"\n+ \"\\n\"\n+ \"\tchmod 0700 %s\");\n+static void check_socket_directory(const char *path)\n+{\n+\tstruct stat st;\n+\tchar *path_copy = xstrdup(path);\n+\tchar *dir = dirname(path_copy);\n+\n+\tif (!stat(dir, &st)) {\n+\t\tif (st.st_mode & 077)\n+\t\t\tdie(_(permissions_advice), dir);\n+\t\tfree(path_copy);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * We must be sure to create the directory with the correct mode,\n+\t * not just chmod it after the fact; otherwise, there is a race\n+\t * condition in which somebody can chdir to it, sleep, then try to open\n+\t * our protected socket.\n+\t */\n+\tif (safe_create_leading_directories_const(dir) < 0)\n+\t\tdie_errno(_(\"unable to create directories for '%s'\"), dir);\n+\tif (mkdir(dir, 0700) < 0)\n+\t\tdie_errno(_(\"unable to mkdir '%s'\"), dir);\n+\tfree(path_copy);\n+}\n+\n int main(int argc, const char **argv)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -76,6 +107,7 @@ int main(int argc, const char **argv)\n \n \tsocket_path = argv[0];\n \tstrbuf_addf(&sb, \"%s/socket\", socket_path);\n+\tcheck_socket_directory(sb.buf);\n \tfd = unix_stream_listen(sb.buf, 0);\n \tif (fd == -1)\n \t\tdie_errno(_(\"unable to listen at %s\"), sb.buf);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234078","messageId":"1391401754-15347-8-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 07/26] file-watcher: remove socket on exit","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:55Z","receivedAt":"2014-02-03T04:28:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher.c | 24 +++++++++++++++++++++++-\n 1 file changed, 23 insertions(+), 1 deletion(-)\n\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 91f4cfe..9c639ef 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -56,6 +56,26 @@ static void accept_connection(int fd)\n \tpfd_nr++;\n }\n \n+static const char *socket_path;\n+static int do_not_clean_up;\n+\n+static void cleanup(void)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tif (do_not_clean_up)\n+\t\treturn;\n+\tstrbuf_addf(&sb, \"%s/socket\", socket_path);\n+\tunlink(sb.buf);\n+\tstrbuf_release(&sb);\n+}\n+\n+static void cleanup_on_signal(int signo)\n+{\n+\tcleanup();\n+\tsigchain_pop(signo);\n+\traise(signo);\n+}\n+\n static const char permissions_advice[] =\n N_(\"The permissions on your socket directory are too loose; other\\n\"\n \"processes may be able to read your file listing. Consider running:\\n\"\n@@ -91,7 +111,6 @@ int main(int argc, const char **argv)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint i, new_nr, fd, quit = 0, nr_common;\n-\tconst char *socket_path = NULL;\n \tstruct option options[] = {\n \t\tOPT_END()\n \t};\n@@ -113,6 +132,9 @@ int main(int argc, const char **argv)\n \t\tdie_errno(_(\"unable to listen at %s\"), sb.buf);\n \tstrbuf_reset(&sb);\n \n+\tatexit(cleanup);\n+\tsigchain_push_common(cleanup_on_signal);\n+\n \tnr_common = 1;\n \tpfd_alloc = pfd_nr = nr_common;\n \tpfd = xmalloc(sizeof(*pfd) * pfd_alloc);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234079","messageId":"1391401754-15347-9-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 08/26] file-watcher: add --detach","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:56Z","receivedAt":"2014-02-03T04:28:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In daemon mode, stdout and stderr are saved in $WATCHER/log.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-file-watcher.txt | 2 ++\n cache.h | 1 +\n daemon.c | 30 ++++--------------------------\n file-watcher.c | 17 +++++++++++++++++\n setup.c | 25 +++++++++++++++++++++++++\n 5 files changed, 49 insertions(+), 26 deletions(-)\n\ndiff --git a/Documentation/git-file-watcher.txt b/Documentation/git-file-watcher.txt\nindex 625a389..ec81f18 100644\n--- a/Documentation/git-file-watcher.txt\n+++ b/Documentation/git-file-watcher.txt\n@@ -18,6 +18,8 @@ lstat(2) to detect that itself.\n \n OPTIONS\n -------\n+--detach::\n+\tRun in background.\n \n SEE ALSO\n --------\ndiff --git a/cache.h b/cache.h\nindex 939db46..7a836b1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -434,6 +434,7 @@ extern int set_git_dir_init(const char *git_dir, const char *real_git_dir, int);\n extern int init_db(const char *template_dir, unsigned int flags);\n \n extern void sanitize_stdfds(void);\n+extern int daemonize(int *);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \ndiff --git a/daemon.c b/daemon.c\nindex 503e039..2650504 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -1056,11 +1056,6 @@ static void drop_privileges(struct credentials *cred)\n \t/* nothing */\n }\n \n-static void daemonize(void)\n-{\n-\tdie(\"--detach not supported on this platform\");\n-}\n-\n static struct credentials *prepare_credentials(const char *user_name,\n const char *group_name)\n {\n@@ -1102,24 +1097,6 @@ static struct credentials *prepare_credentials(const char *user_name,\n \n \treturn &c;\n }\n-\n-static void daemonize(void)\n-{\n-\tswitch (fork()) {\n-\t\tcase 0:\n-\t\t\tbreak;\n-\t\tcase -1:\n-\t\t\tdie_errno(\"fork failed\");\n-\t\tdefault:\n-\t\t\texit(0);\n-\t}\n-\tif (setsid() == -1)\n-\t\tdie_errno(\"setsid failed\");\n-\tclose(0);\n-\tclose(1);\n-\tclose(2);\n-\tsanitize_stdfds();\n-}\n #endif\n \n static void store_pid(const char *path)\n@@ -1333,9 +1310,10 @@ int main(int argc, char **argv)\n \tif (inetd_mode || serve_mode)\n \t\treturn execute();\n \n-\tif (detach)\n-\t\tdaemonize();\n-\telse\n+\tif (detach) {\n+\t\tif (daemonize(NULL))\n+\t\t\tdie(\"--detach not supported on this platform\");\n+\t} else\n \t\tsanitize_stdfds();\n \n \tif (pid_file)\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 9c639ef..1e1ccad 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -111,7 +111,10 @@ int main(int argc, const char **argv)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint i, new_nr, fd, quit = 0, nr_common;\n+\tint daemon = 0;\n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"detach\", &daemon,\n+\t\t\t N_(\"run in background\")),\n \t\tOPT_END()\n \t};\n \n@@ -135,6 +138,20 @@ int main(int argc, const char **argv)\n \tatexit(cleanup);\n \tsigchain_push_common(cleanup_on_signal);\n \n+\tif (daemon) {\n+\t\tint err;\n+\t\tstrbuf_addf(&sb, \"%s/log\", socket_path);\n+\t\terr = open(sb.buf, O_CREAT | O_TRUNC | O_WRONLY, 0600);\n+\t\tadjust_shared_perm(sb.buf);\n+\t\tif (err == -1)\n+\t\t\tdie_errno(_(\"unable to create %s\"), sb.buf);\n+\t\tif (daemonize(&do_not_clean_up))\n+\t\t\tdie(_(\"--detach not supported on this platform\"));\n+\t\tdup2(err, 1);\n+\t\tdup2(err, 2);\n+\t\tclose(err);\n+\t}\n+\n \tnr_common = 1;\n \tpfd_alloc = pfd_nr = nr_common;\n \tpfd = xmalloc(sizeof(*pfd) * pfd_alloc);\ndiff --git a/setup.c b/setup.c\nindex 6c3f85f..757c45f 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -787,3 +787,28 @@ void sanitize_stdfds(void)\n \tif (fd > 2)\n \t\tclose(fd);\n }\n+\n+int daemonize(int *flag)\n+{\n+#ifndef NO_POSIX_GOODIES\n+\tswitch (fork()) {\n+\t\tcase 0:\n+\t\t\tbreak;\n+\t\tcase -1:\n+\t\t\tdie_errno(\"fork failed\");\n+\t\tdefault:\n+\t\t\tif (flag)\n+\t\t\t\t*flag = 1;\n+\t\t\texit(0);\n+\t}\n+\tif (setsid() == -1)\n+\t\tdie_errno(\"setsid failed\");\n+\tclose(0);\n+\tclose(1);\n+\tclose(2);\n+\tsanitize_stdfds();\n+\treturn 0;\n+#else\n+\treturn -1;\n+#endif\n+}\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234080","messageId":"1391401754-15347-10-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 09/26] read-cache: save trailing sha-1","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:57Z","receivedAt":"2014-02-03T04:28:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This will be used as signature to know if the index has changed.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n read-cache.c | 7 ++++---\n 2 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7a836b1..f14d535 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -279,6 +279,7 @@ struct index_state {\n \t\t initialized : 1;\n \tstruct hash_table name_hash;\n \tstruct hash_table dir_hash;\n+\tunsigned char sha1[20];\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 33dd676..3b6daf1 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1269,10 +1269,11 @@ struct ondisk_cache_entry_extended {\n \t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr(struct cache_header *hdr, unsigned long size)\n+static int verify_hdr(struct cache_header *hdr,\n+\t\t unsigned long size,\n+\t\t unsigned char *sha1)\n {\n \tgit_SHA_CTX c;\n-\tunsigned char sha1[20];\n \tint hdr_version;\n \n \tif (hdr->hdr_signature != htonl(CACHE_SIGNATURE))\n@@ -1461,7 +1462,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tclose(fd);\n \n \thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n+\tif (verify_hdr(hdr, mmap_size, istate->sha1) < 0)\n \t\tgoto unmap;\n \n \tistate->version = ntohl(hdr->hdr_version);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234081","messageId":"1391401754-15347-11-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 10/26] read-cache: new flag CE_WATCHED to mark what file is watched","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:58Z","receivedAt":"2014-02-03T04:28:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This bit is basically \"dynamic CE_VALID\". It marks entries that are\nbeing watched by the incoming file-watcher. When an index is loaded,\nfile watcher is contacted and the list of updated paths is retrieved.\n\nThese paths will have CE_WATCHED cleared and lstat() will be called on\nthem. Those that have CE_WATCHED and not in the list will have\nCE_VALID turn on to skip lstat(). The setting is temporarily, CE_VALID\nis not saved to disk if CE_WATCHED is also set.\n\nWe keep the CE_WATCHED in a new extension, separated from the entries\nto save some space because extended ce_flags adds 2 bytes per entry\nand this flag would be present in the majority of entries. When stored\nas bitmap, this extension could compress very well with ewah algorithm.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/index-format.txt | 6 +++++\n cache.h | 3 +++\n read-cache.c | 41 +++++++++++++++++++++++++++++++-\n 3 files changed, 49 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex f352a9b..24fd0ae 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -198,3 +198,9 @@ Git index format\n - At most three 160-bit object names of the entry in stages from 1 to 3\n (nothing is written for a missing stage).\n \n+=== File watcher\n+\n+ The signature of this extension is { 'W', 'A', 'T', 'C' }.\n+\n+ - A bit map of all entries in the index, n-th bit of m-th byte\n+ corresponds to CE_WATCHED of the -th index entry.\ndiff --git a/cache.h b/cache.h\nindex f14d535..a0af2a5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -169,6 +169,9 @@ struct cache_entry {\n /* used to temporarily mark paths matched by pathspecs */\n #define CE_MATCHED (1 << 26)\n \n+/* set CE_VALID at runtime if the entry is guaranteed not updated */\n+#define CE_WATCHED (1 << 27)\n+\n /*\n * Extended on-disk flags\n */\ndiff --git a/read-cache.c b/read-cache.c\nindex 3b6daf1..098d3b6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -33,6 +33,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n #define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n #define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n+#define CACHE_EXT_WATCH 0x57415443\t /* \"WATC\" */\n \n struct index_state the_index;\n \n@@ -1289,6 +1290,19 @@ static int verify_hdr(struct cache_header *hdr,\n \treturn 0;\n }\n \n+static void read_watch_extension(struct index_state *istate, uint8_t *data,\n+\t\t\t\t unsigned long sz)\n+{\n+\tint i;\n+\tif ((istate->cache_nr + 7) / 8 != sz) {\n+\t\terror(\"invalid 'WATC' extension\");\n+\t\treturn;\n+\t}\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (data[i / 8] & (1 << (i % 8)))\n+\t\t\tistate->cache[i]->ce_flags |= CE_WATCHED;\n+}\n+\n static int read_index_extension(struct index_state *istate,\n \t\t\t\tconst char *ext, void *data, unsigned long sz)\n {\n@@ -1299,6 +1313,9 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_RESOLVE_UNDO:\n \t\tistate->resolve_undo = resolve_undo_read(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_WATCH:\n+\t\tread_watch_extension(istate, data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1777,7 +1794,7 @@ int write_index(struct index_state *istate, int newfd)\n {\n \tgit_SHA_CTX c;\n \tstruct cache_header hdr;\n-\tint i, err, removed, extended, hdr_version;\n+\tint i, err, removed, extended, hdr_version, has_watches = 0;\n \tstruct cache_entry **cache = istate->cache;\n \tint entries = istate->cache_nr;\n \tstruct stat st;\n@@ -1786,6 +1803,8 @@ int write_index(struct index_state *istate, int newfd)\n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n \t\t\tremoved++;\n+\t\telse if (cache[i]->ce_flags & CE_WATCHED)\n+\t\t\thas_watches++;\n \n \t\t/* reduce extended entries if possible */\n \t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n@@ -1857,6 +1876,26 @@ int write_index(struct index_state *istate, int newfd)\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (has_watches) {\n+\t\tint id, sz = (entries - removed + 7) / 8;\n+\t\tuint8_t *data = xmalloc(sz);\n+\t\tmemset(data, 0, sz);\n+\t\tfor (i = 0, id = 0; i < entries && has_watches; i++) {\n+\t\t\tstruct cache_entry *ce = cache[i];\n+\t\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\t\tcontinue;\n+\t\t\tif (ce->ce_flags & CE_WATCHED) {\n+\t\t\t\tdata[id / 8] |= 1 << (id % 8);\n+\t\t\t\thas_watches--;\n+\t\t\t}\n+\t\t\tid++;\n+\t\t}\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCH, sz) < 0\n+\t\t\t|| ce_write(&c, newfd, data, sz) < 0;\n+\t\tfree(data);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n \t\treturn -1;\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234083","messageId":"1391401754-15347-12-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 11/26] Clear CE_WATCHED when set CE_VALID alone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:28:59Z","receivedAt":"2014-02-03T04:28:59Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If CE_WATCHED is set, CE_VALID is controlled by CE_WATCHED and will be\ncleared bfore writing to disk. Users of --assume-unchanged therefore\nneed to clear CE_WATCHED to avoid this side effect.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/update-index.c | 12 ++++++++----\n read-cache.c | 2 +-\n 2 files changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex e3a10d7..9283fd6 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -50,9 +50,13 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \tint namelen = strlen(path);\n \tint pos = cache_name_pos(path, namelen);\n \tif (0 <= pos) {\n-\t\tif (mark)\n-\t\t\tactive_cache[pos]->ce_flags |= flag;\n-\t\telse\n+\t\tif (mark) {\n+\t\t\tstruct cache_entry *ce = active_cache[pos];\n+\t\t\tif (flag == CE_VALID)\n+\t\t\t\tce->ce_flags = (ce->ce_flags & ~CE_WATCHED) | CE_VALID;\n+\t\t\telse\n+\t\t\t\tce->ce_flags |= flag;\n+\t\t} else\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tcache_tree_invalidate_path(active_cache_tree, path);\n \t\tactive_cache_changed = 1;\n@@ -235,7 +239,7 @@ static int add_cacheinfo(unsigned int mode, const unsigned char *sha1,\n \tce->ce_namelen = len;\n \tce->ce_mode = create_ce_mode(mode);\n \tif (assume_unchanged)\n-\t\tce->ce_flags |= CE_VALID;\n+\t\tce->ce_flags = (ce->ce_flags & ~CE_WATCHED) | CE_VALID;\n \toption = allow_add ? ADD_CACHE_OK_TO_ADD : 0;\n \toption |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;\n \tif (add_cache_entry(ce, option))\ndiff --git a/read-cache.c b/read-cache.c\nindex 098d3b6..8961864 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -133,7 +133,7 @@ void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n \tfill_stat_data(&ce->ce_stat_data, st);\n \n \tif (assume_unchanged)\n-\t\tce->ce_flags |= CE_VALID;\n+\t\tce->ce_flags = (ce->ce_flags & ~CE_WATCHED) | CE_VALID;\n \n \tif (S_ISREG(st->st_mode))\n \t\tce_mark_uptodate(ce);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234082","messageId":"1391401754-15347-13-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 12/26] read-cache: basic hand shaking to the file watcher","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:00Z","receivedAt":"2014-02-03T04:29:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"read_cache() connects to the file watcher, specified by\nfilewatcher.path config, and performs basic hand shaking. CE_WATCHED\nis cleared if git and file watcher have different views on the index\nstate.\n\nAll send/receive calls must be complete within a limited time to avoid\na buggy file-watcher hang \"git status\" forever. And the whole point of\ndoing this is speed. If file watcher can't respond fast enough, for\nwhatever reason, then it should not be used.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 10 +++\n Documentation/git-file-watcher.txt | 4 +-\n Makefile | 1 +\n cache.h | 1 +\n file-watcher-lib.c (new) | 91 ++++++++++++++++++++++\n file-watcher-lib.h (new) | 6 ++\n file-watcher.c | 152 ++++++++++++++++++++++++++++++++++++-\n read-cache.c | 6 ++\n 8 files changed, 269 insertions(+), 2 deletions(-)\n create mode 100644 file-watcher-lib.c\n create mode 100644 file-watcher-lib.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 5f4d793..6ad653a 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1042,6 +1042,16 @@ difftool..cmd::\n difftool.prompt::\n \tPrompt before each invocation of the diff tool.\n \n+filewatcher.path::\n+\tThe directory that contains the socket of `git\tfile-watcher`.\n+\tIf it's not an absolute path, it's relative to $GIT_DIR. An\n+\tempty path means no connection to file watcher.\n+\n+filewatcher.timeout::\n+\tThis is the maximum time in milliseconds that Git waits for\n+\tthe file watcher to respond before giving up. Default value is\n+\t50. Setting to -1 makes Git wait forever.\n+\n fetch.recurseSubmodules::\n \tThis option can be either set to a boolean value or to 'on-demand'.\n \tSetting it to a boolean changes the behavior of fetch and pull to\ndiff --git a/Documentation/git-file-watcher.txt b/Documentation/git-file-watcher.txt\nindex ec81f18..d91caf3 100644\n--- a/Documentation/git-file-watcher.txt\n+++ b/Documentation/git-file-watcher.txt\n@@ -14,7 +14,9 @@ DESCRIPTION\n -----------\n This program watches file changes in a git working directory and let\n Git now what files have been changed so that Git does not have to call\n-lstat(2) to detect that itself.\n+lstat(2) to detect that itself. Config key filewatcher.path needs to\n+be set to `` so Git knows how to contact to the file\n+watcher.\n \n OPTIONS\n -------\ndiff --git a/Makefile b/Makefile\nindex 8eef0d6..1c4d659 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -798,6 +798,7 @@ LIB_OBJS += entry.o\n LIB_OBJS += environment.o\n LIB_OBJS += exec_cmd.o\n LIB_OBJS += fetch-pack.o\n+LIB_OBJS += file-watcher-lib.o\n LIB_OBJS += fsck.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\ndiff --git a/cache.h b/cache.h\nindex a0af2a5..b3ea574 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -283,6 +283,7 @@ struct index_state {\n \tstruct hash_table name_hash;\n \tstruct hash_table dir_hash;\n \tunsigned char sha1[20];\n+\tint watcher;\n };\n \n extern struct index_state the_index;\ndiff --git a/file-watcher-lib.c b/file-watcher-lib.c\nnew file mode 100644\nindex 0000000..d0636cc\n--- /dev/null\n+++ b/file-watcher-lib.c\n@@ -0,0 +1,91 @@\n+#include \"cache.h\"\n+#include \"file-watcher-lib.h\"\n+#include \"pkt-line.h\"\n+#include \"unix-socket.h\"\n+\n+static char *watcher_path;\n+static int WAIT_TIME = 50;\t/* in ms */\n+\n+static int connect_watcher(const char *path)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint fd;\n+\n+\tif (!path || !*path)\n+\t\treturn -1;\n+\n+\tstrbuf_addf(&sb, \"%s/socket\", path);\n+\tfd = unix_stream_connect(sb.buf);\n+\tstrbuf_release(&sb);\n+\treturn fd;\n+}\n+\n+static void reset_watches(struct index_state *istate, int disconnect)\n+{\n+\tint i;\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHED) {\n+\t\t\tistate->cache[i]->ce_flags &= ~(CE_WATCHED | CE_VALID);\n+\t\t\tistate->cache_changed = 1;\n+\t\t}\n+\tif (disconnect && istate->watcher > 0) {\n+\t\tclose(istate->watcher);\n+\t\tistate->watcher = -1;\n+\t}\n+}\n+\n+static int watcher_config(const char *var, const char *value, void *data)\n+{\n+\tif (!strcmp(var, \"filewatcher.path\")) {\n+\t\tif (is_absolute_path(value))\n+\t\t\twatcher_path = xstrdup(value);\n+\t\telse if (*value == '~')\n+\t\t\twatcher_path = expand_user_path(value);\n+\t\telse\n+\t\t\twatcher_path = git_pathdup(\"%s\", value);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(var, \"filewatcher.timeout\")) {\n+\t\tWAIT_TIME = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\treturn 0;\n+}\n+\n+void open_watcher(struct index_state *istate)\n+{\n+\tstatic int read_config = 0;\n+\tchar *msg;\n+\n+\tif (!get_git_work_tree()) {\n+\t\treset_watches(istate, 1);\n+\t\treturn;\n+\t}\n+\n+\tif (!read_config) {\n+\t\t/*\n+\t\t * can't hook into git_default_config because\n+\t\t * read_cache() may be called even before git_config()\n+\t\t * call.\n+\t\t */\n+\t\tgit_config(watcher_config, NULL);\n+\t\tread_config = 1;\n+\t}\n+\n+\tistate->watcher = connect_watcher(watcher_path);\n+\tif (packet_write_timeout(istate->watcher, WAIT_TIME, \"hello\") <= 0 ||\n+\t (msg = packet_read_line_timeout(istate->watcher, WAIT_TIME, NULL)) == NULL ||\n+\t strcmp(msg, \"hello\")) {\n+\t\treset_watches(istate, 1);\n+\t\treturn;\n+\t}\n+\n+\tif (packet_write_timeout(istate->watcher, WAIT_TIME, \"index %s %s\",\n+\t\t\t\t sha1_to_hex(istate->sha1),\n+\t\t\t\t get_git_work_tree()) <= 0 ||\n+\t (msg = packet_read_line_timeout(istate->watcher, WAIT_TIME, NULL)) == NULL ||\n+\t strcmp(msg, \"ok\")) {\n+\t\treset_watches(istate, 0);\n+\t\treturn;\n+\t}\n+}\ndiff --git a/file-watcher-lib.h b/file-watcher-lib.h\nnew file mode 100644\nindex 0000000..eb6edf5\n--- /dev/null\n+++ b/file-watcher-lib.h\n@@ -0,0 +1,6 @@\n+#ifndef __FILE_WATCHER_LIB__\n+#define __FILE_WATCHER_LIB__\n+\n+void open_watcher(struct index_state *istate);\n+\n+#endif\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 1e1ccad..6df3a48 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -3,20 +3,78 @@\n #include \"parse-options.h\"\n #include \"exec_cmd.h\"\n #include \"unix-socket.h\"\n+#include \"pkt-line.h\"\n \n static const char *const file_watcher_usage[] = {\n \tN_(\"git file-watcher [options] \"),\n \tNULL\n };\n \n+struct repository {\n+\tchar *work_tree;\n+\tchar index_signature[41];\n+\t/*\n+\t * At least with inotify we don't keep track down to \"/\". So\n+\t * if worktree is /abc/def and someone moves /abc to /ghi, and\n+\t * /jlk to /abc (and /jlk/def exists before the move), we\n+\t * cant' detect that /abc/def is totally new. Checking inode\n+\t * is probably enough for this case.\n+\t */\n+\tino_t inode;\n+};\n+\n+const char *invalid_signature = \"0000000000000000000000000000000000000000\";\n+\n+static struct repository **repos;\n+static int nr_repos;\n+\n struct connection {\n-\tint sock;\n+\tint sock, polite;\n+\tstruct repository *repo;\n };\n \n static struct connection **conns;\n static struct pollfd *pfd;\n static int conns_alloc, pfd_nr, pfd_alloc;\n \n+static struct repository *get_repo(const char *work_tree)\n+{\n+\tint first, last;\n+\tstruct repository *repo;\n+\n+\tfirst = 0;\n+\tlast = nr_repos;\n+\twhile (last > first) {\n+\t\tint next = (last + first) >> 1;\n+\t\tint cmp = strcmp(work_tree, repos[next]->work_tree);\n+\t\tif (!cmp)\n+\t\t\treturn repos[next];\n+\t\tif (cmp < 0) {\n+\t\t\tlast = next;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfirst = next+1;\n+\t}\n+\n+\tnr_repos++;\n+\trepos = xrealloc(repos, sizeof(*repos) * nr_repos);\n+\tif (nr_repos > first + 1)\n+\t\tmemmove(repos + first + 1, repos + first,\n+\t\t\t(nr_repos - first - 1) * sizeof(*repos));\n+\trepo = xmalloc(sizeof(*repo));\n+\tmemset(repo, 0, sizeof(*repo));\n+\trepo->work_tree = xstrdup(work_tree);\n+\tmemset(repo->index_signature, '0', 40);\n+\trepos[first] = repo;\n+\treturn repo;\n+}\n+\n+static void reset_repo(struct repository *repo, ino_t inode)\n+{\n+\tmemcpy(repo->index_signature, invalid_signature, 40);\n+\trepo->inode = inode;\n+}\n+\n static int shutdown_connection(int id)\n {\n \tstruct connection *conn = conns[id];\n@@ -31,6 +89,98 @@ static int shutdown_connection(int id)\n \n static int handle_command(int conn_id)\n {\n+\tint fd = conns[conn_id]->sock;\n+\tint len;\n+\tconst char *arg;\n+\tchar *msg;\n+\n+\t/*\n+\t * \">\" denotes an incoming packet, \"<\" outgoing. The lack of\n+\t * \"<\" means no reply expected.\n+\t *\n+\t * < \"error\" SP ERROR-STRING\n+\t *\n+\t * This can be sent whenever the client violates the protocol.\n+\t */\n+\n+\tmsg = packet_read_line(fd, &len);\n+\tif (!msg) {\n+\t\tpacket_write(fd, \"error invalid pkt-line\");\n+\t\treturn shutdown_connection(conn_id);\n+\t}\n+\n+\t/*\n+\t * > \"hello\" [SP CAP [SP CAP..]]\n+\t * < \"hello\" [SP CAP [SP CAP..]]\n+\t *\n+\t * Advertise capabilities of both sides. File watcher may\n+\t * disconnect if the client does not advertise the required\n+\t * capabilities. Capabilities in uppercase MUST be\n+\t * supported. If any side does not understand any of the\n+\t * advertised uppercase capabilities, it must disconnect.\n+\t */\n+\tif ((arg = skip_prefix(msg, \"hello\"))) {\n+\t\tif (*arg) {\t/* no capabilities supported yet */\n+\t\t\tpacket_write(fd, \"error capabilities not supported\");\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\tpacket_write(fd, \"hello\");\n+\t\tconns[conn_id]->polite = 1;\n+\t}\n+\n+\t/*\n+\t * > \"index\" SP INDEX-SIGNATURE SP WORK-TREE-PATH\n+\t * < \"ok\" | \"inconsistent\"\n+\t *\n+\t * INDEX-SIGNATURE consists of 40 hexadecimal letters\n+\t * WORK-TREE-PATH must be absolute and normalized path\n+\t *\n+\t * Watch file changes in index. The client sends the index and\n+\t * work tree info. File watcher validates that it holds the\n+\t * same info. If so it sends \"ok\" back indicating both sides\n+\t * are on the same page and CE_WATCHED bits can be ketpt.\n+\t *\n+\t * Otherwise it sends \"inconsistent\" and both sides must reset\n+\t * back to initial state. File watcher keeps its index\n+\t * signature all-zero until the client has updated the index\n+\t * ondisk and request to update index signature.\n+\t *\n+\t * \"hello\" must be exchanged first. After this command the\n+\t * connection is associated with a worktree/index. Many\n+\t * commands may require this to proceed.\n+\t */\n+\telse if (starts_with(msg, \"index \")) {\n+\t\tstruct repository *repo;\n+\t\tstruct stat st;\n+\t\tif (!conns[conn_id]->polite) {\n+\t\t\tpacket_write(fd, \"error why did you not greet me? go away\");\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\tif (len < 47 || msg[46] != ' ' || !is_absolute_path(msg + 47)) {\n+\t\t\tpacket_write(fd, \"error invalid index line %s\", msg);\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\n+\t\tif (lstat(msg + 47, &st) || !S_ISDIR(st.st_mode)) {\n+\t\t\tpacket_write(fd, \"error work tree does not exist: %s\",\n+\t\t\t\t strerror(errno));\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\trepo = get_repo(msg + 47);\n+\t\tconns[conn_id]->repo = repo;\n+\t\tif (memcmp(msg + 6, repo->index_signature, 40) ||\n+\t\t !memcmp(msg + 6, invalid_signature, 40) ||\n+\t\t repo->inode != st.st_ino) {\n+\t\t\tpacket_write(fd, \"inconsistent\");\n+\t\t\treset_repo(repo, st.st_ino);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tpacket_write(fd, \"ok\");\n+\t}\n+\telse {\n+\t\tpacket_write(fd, \"error unrecognized command %s\", msg);\n+\t\treturn shutdown_connection(conn_id);\n+\t}\n \treturn 0;\n }\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 8961864..a7e5735 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -14,6 +14,7 @@\n #include \"resolve-undo.h\"\n #include \"strbuf.h\"\n #include \"varint.h\"\n+#include \"file-watcher-lib.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int really);\n \n@@ -1528,6 +1529,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsrc_offset += extsize;\n \t}\n \tmunmap(mmap, mmap_size);\n+\topen_watcher(istate);\n \treturn istate->cache_nr;\n \n unmap:\n@@ -1553,6 +1555,10 @@ int discard_index(struct index_state *istate)\n \tistate->timestamp.nsec = 0;\n \tfree_name_hash(istate);\n \tcache_tree_free(&(istate->cache_tree));\n+\tif (istate->watcher > 0) {\n+\t\tclose(istate->watcher);\n+\t\tistate->watcher = -1;\n+\t}\n \tistate->initialized = 0;\n \tfree(istate->cache);\n \tistate->cache = NULL;\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234084","messageId":"1391401754-15347-14-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 13/26] read-cache: ask file watcher to watch files","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:01Z","receivedAt":"2014-02-03T04:29:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We want to watch files that are never changed because lstat() on those\nfiles is a wasted effort. So we sort unwatched files by date and start\nadding them to the file watcher until it barfs (e.g. hits inotify\nlimit).\n\nNote that we still do lstat() on these new watched files because they\ncould have changed before the file watcher could watch them. Watched\nfiles may only skip lstat() at the next run.\n\nAlso, at this early in the index loading process, we don't know what\nfiles are dirty and thus can skip watching (we do clear CE_WATCHED on\nentries that are not verified clean before writing index). So the\nwatches are set, but git ignores its results. Maybe in future we could\nstore the list of dirty files in WATC extension and use it as a hint\nto skip watching.\n\nIn the future, file watcher should figure out what paths are\nwatchable, what not (e.g. network filesystems) and reject them. For\nnow it's the user resposibility to set (or unset) filewatcher.path\nproperly.\n\nThe previous attempt sends paths in batch, 64k per pkt-line, then wait\nfor response. It's designed to stop short in case file watcher is out\nof resources. But that's a rare case, and send/wait cycles increase\nlatency.\n\nInstead we now send everything in one packet, and not in pkt-line to\navoid the 64k limit. Then we wait for the response. On webkit.git,\nnormal \"status -uno\" takes 0m1.138s. The sending 14M (of 182k paths)\ntakes 52ms extra. Previous approach takes 213ms extra. Of course in\nthe end, extra time is longer because file watcher is basically no-op\nso far.\n\nThere is not much room for improvement. If we compress paths to reduce\npayload, zlib time costs about 300ms (so how small the end result is\nno longer matters). Even simple prefix compressing (index v4 style)\nwould cost 76ms on processing time alone (reducing payload to 3M).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher-lib.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++\n file-watcher-lib.h | 1 +\n file-watcher.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n read-cache.c | 18 +++++++++--\n 4 files changed, 191 insertions(+), 2 deletions(-)\n\ndiff --git a/file-watcher-lib.c b/file-watcher-lib.c\nindex d0636cc..791faae 100644\n--- a/file-watcher-lib.c\n+++ b/file-watcher-lib.c\n@@ -89,3 +89,86 @@ void open_watcher(struct index_state *istate)\n \t\treturn;\n \t}\n }\n+\n+static int sort_by_date(const void *a_, const void *b_)\n+{\n+\tconst struct cache_entry *a = *(const struct cache_entry **)a_;\n+\tconst struct cache_entry *b = *(const struct cache_entry **)b_;\n+\tuint32_t seca = a->ce_stat_data.sd_mtime.sec;\n+\tuint32_t secb = b->ce_stat_data.sd_mtime.sec;\n+\treturn seca - secb;\n+}\n+\n+static inline int ce_watchable(struct cache_entry *ce)\n+{\n+\treturn\n+\t\t!(ce->ce_flags & CE_WATCHED) &&\n+\t\t!(ce->ce_flags & CE_VALID) &&\n+\t\t/*\n+\t\t * S_IFGITLINK should not be watched\n+\t\t * obviously. S_IFLNK could be problematic because\n+\t\t * inotify may follow symlinks without IN_DONT_FOLLOW\n+\t\t */\n+\t\tS_ISREG(ce->ce_mode);\n+}\n+\n+static void send_watches(struct index_state *istate,\n+\t\t\t struct cache_entry **sorted, int nr)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint i, len = 0;\n+\n+\tfor (i = 0; i < nr; i++)\n+\t\tlen += ce_namelen(sorted[i]) + 1;\n+\n+\tif (packet_write_timeout(istate->watcher, WAIT_TIME, \"watch %d\", len) <= 0)\n+\t\treturn;\n+\n+\tstrbuf_grow(&sb, len);\n+\tfor (i = 0; i < nr; i++)\n+\t\tstrbuf_add(&sb, sorted[i]->name, ce_namelen(sorted[i]) + 1);\n+\n+\tif (write_in_full_timeout(istate->watcher, sb.buf,\n+\t\t\t\t sb.len, WAIT_TIME) != sb.len) {\n+\t\tstrbuf_release(&sb);\n+\t\treturn;\n+\t}\n+\tstrbuf_release(&sb);\n+\n+\tfor (;;) {\n+\t\tchar *line, *end;\n+\t\tunsigned long n;\n+\n+\t\tif (!(line = packet_read_line_timeout(istate->watcher,\n+\t\t\t\t\t\t WAIT_TIME, &len)))\n+\t\t\treturn;\n+\t\tif (starts_with(line, \"watching \"))\n+\t\t\tcontinue;\n+\t\tif (!starts_with(line, \"watched \"))\n+\t\t\treturn;\n+\t\tn = strtoul(line + 8, &end, 10);\n+\t\tfor (i = 0; i < n; i++)\n+\t\t\tsorted[i]->ce_flags |= CE_WATCHED;\n+\t\tistate->cache_changed = 1;\n+\t\tbreak;\n+\t}\n+}\n+\n+void watch_entries(struct index_state *istate)\n+{\n+\tint i, nr;\n+\tstruct cache_entry **sorted;\n+\n+\tif (istate->watcher <= 0)\n+\t\treturn;\n+\tfor (i = nr = 0; i < istate->cache_nr; i++)\n+\t\tif (ce_watchable(istate->cache[i]))\n+\t\t\tnr++;\n+\tsorted = xmalloc(sizeof(*sorted) * nr);\n+\tfor (i = nr = 0; i < istate->cache_nr; i++)\n+\t\tif (ce_watchable(istate->cache[i]))\n+\t\t\tsorted[nr++] = istate->cache[i];\n+\tqsort(sorted, nr, sizeof(*sorted), sort_by_date);\n+\tsend_watches(istate, sorted, nr);\n+\tfree(sorted);\n+}\ndiff --git a/file-watcher-lib.h b/file-watcher-lib.h\nindex eb6edf5..1641024 100644\n--- a/file-watcher-lib.h\n+++ b/file-watcher-lib.h\n@@ -2,5 +2,6 @@\n #define __FILE_WATCHER_LIB__\n \n void open_watcher(struct index_state *istate);\n+void watch_entries(struct index_state *istate);\n \n #endif\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 6df3a48..c257414 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -37,6 +37,70 @@ static struct connection **conns;\n static struct pollfd *pfd;\n static int conns_alloc, pfd_nr, pfd_alloc;\n \n+static int watch_path(struct repository *repo, char *path)\n+{\n+\treturn -1;\n+}\n+\n+static inline uint64_t stamp(void)\n+{\n+\tstruct timeval tv;\n+\tgettimeofday(&tv, NULL);\n+\treturn (uint64_t)tv.tv_sec * 1000000 + tv.tv_usec;\n+}\n+\n+static int shutdown_connection(int id);\n+static void watch_paths(int conn_id, char *buf, int maxlen)\n+{\n+\tint ret, len, n;\n+\tuint64_t start, now;\n+\tchar *end;\n+\n+\tn = strtol(buf, &end, 10);\n+\tif (end != buf + maxlen) {\n+\t\tpacket_write(conns[conn_id]->sock,\n+\t\t\t \"error invalid watch number %s\", buf);\n+\t\tshutdown_connection(conn_id);\n+\t\treturn;\n+\t}\n+\n+\tbuf = xmallocz(n);\n+\tend = buf + n;\n+\t/*\n+\t * Careful if this takes longer than 50ms, it'll upset other\n+\t * connections\n+\t */\n+\tif (read_in_full(conns[conn_id]->sock, buf, n) != n) {\n+\t\tshutdown_connection(conn_id);\n+\t\treturn;\n+\t}\n+\tif (chdir(conns[conn_id]->repo->work_tree)) {\n+\t\tpacket_write(conns[conn_id]->sock,\n+\t\t\t \"error chdir %s\", strerror(errno));\n+\t\treturn;\n+\t}\n+\tstart = stamp();\n+\tfor (n = ret = 0; buf < end; buf += len + 1) {\n+\t\tlen = strlen(buf);\n+\t\tif (watch_path(conns[conn_id]->repo, buf))\n+\t\t\tbreak;\n+\t\tn++;\n+\t\tif (n & 0x3ff)\n+\t\t\tcontinue;\n+\t\tnow = stamp();\n+\t\t/*\n+\t\t * If we process for too long, the client may timeout\n+\t\t * and give up. Let the client know we're not dead\n+\t\t * yet, every 30ms.\n+\t\t */\n+\t\tif (start + 30000 < now) {\n+\t\t\tpacket_write(conns[conn_id]->sock, \"watching %d\", n);\n+\t\t\tstart = now;\n+\t\t}\n+\t}\n+\tpacket_write(conns[conn_id]->sock, \"watched %u\", n);\n+}\n+\n static struct repository *get_repo(const char *work_tree)\n {\n \tint first, last;\n@@ -177,6 +241,33 @@ static int handle_command(int conn_id)\n \t\t}\n \t\tpacket_write(fd, \"ok\");\n \t}\n+\n+\t/*\n+\t * > \"watch\" SP LENGTH\n+\t * > PATH-LIST\n+\t * < \"watching\" SP NUM\n+\t * < \"watched\" SP NUM\n+\t *\n+\t * PATH-LIST is the list of paths, each terminated with\n+\t * NUL. PATH-LIST is not wrapped in pkt-line format. LENGTH is\n+\t * the size of PATH-LIST in bytes.\n+\t *\n+\t * The client asks file watcher to watcher a number of\n+\t * paths. File watcher starts to process from path by path in\n+\t * received order. File watcher returns the actual number of\n+\t * watched paths with \"watched\" command.\n+\t *\n+\t * File watcher may send any number of \"watching\" messages\n+\t * before \"watched\". This packet is to keep the connection\n+\t * alive and has no other values.\n+\t */\n+\telse if (starts_with(msg, \"watch \")) {\n+\t\tif (!conns[conn_id]->repo) {\n+\t\t\tpacket_write(fd, \"error have not received index command\");\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\twatch_paths(conn_id, msg + 6, len - 6);\n+\t}\n \telse {\n \t\tpacket_write(fd, \"error unrecognized command %s\", msg);\n \t\treturn shutdown_connection(conn_id);\ndiff --git a/read-cache.c b/read-cache.c\nindex a7e5735..cb2188f 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1530,6 +1530,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t}\n \tmunmap(mmap, mmap_size);\n \topen_watcher(istate);\n+\twatch_entries(istate);\n \treturn istate->cache_nr;\n \n unmap:\n@@ -1809,8 +1810,21 @@ int write_index(struct index_state *istate, int newfd)\n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n \t\t\tremoved++;\n-\t\telse if (cache[i]->ce_flags & CE_WATCHED)\n-\t\t\thas_watches++;\n+\t\telse if (cache[i]->ce_flags & CE_WATCHED) {\n+\t\t\t/*\n+\t\t\t * We may set CE_WATCHED (but not CE_VALID)\n+\t\t\t * early when refresh has not been done\n+\t\t\t * yet. At that time we had no idea if the\n+\t\t\t * entry may have been updated. If it has\n+\t\t\t * been, remove CE_WATCHED so CE_VALID won't\n+\t\t\t * incorrectly be set next time if the file\n+\t\t\t * watcher reports no changes.\n+\t\t\t */\n+\t\t\tif (!ce_uptodate(cache[i]))\n+\t\t\t\tcache[i]->ce_flags &= ~CE_WATCHED;\n+\t\t\telse\n+\t\t\t\thas_watches++;\n+\t\t}\n \n \t\t/* reduce extended entries if possible */\n \t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234085","messageId":"1391401754-15347-15-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 14/26] read-cache: put some limits on file watching","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:02Z","receivedAt":"2014-02-03T04:29:02Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"watch_entries() is a lot of computation and could trigger a lot more\nlookups in file-watcher. Normally after the first set of watches are\nin place, we do not need to update often. Moreover if the number of\nentries is small, the overhead of file watcher may actually slow git\ndown.\n\nThis patch only allows to update watches if the number of watchable\nfiles is over a limit (and there are new files added if this is not\nthe first time). Measurements on Core i5-2520M and Linux 3.7.6, about\n920 lstat() take 1ms. Somewhere between 2^16 and 2^17 lstat calls that\nit starts to take longer than 100ms. 2^16 is chosen at the minimum\nlimit to start using file watcher.\n\nOf course this is only sensible default for single-repo use\ncase. Lower it when you need to work with many small repos.\n\nRecently updated files are not considered watchable because they are\nlikely to be updated again soon, not worth the ping-pong game with\nfile watcher. The default limit 10min is just a random value. Recent\nlimit is ignored if there are no watched files (e.g. a fresh clone, or\nafter a bad hand shake with file watcher).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 9 +++++++\n Documentation/technical/index-format.txt | 3 +++\n cache.h | 1 +\n file-watcher-lib.c | 42 ++++++++++++++++++++++++++------\n read-cache.c | 11 ++++++---\n 5 files changed, 56 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 6ad653a..451c100 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1052,6 +1052,15 @@ filewatcher.timeout::\n \tthe file watcher to respond before giving up. Default value is\n \t50. Setting to -1 makes Git wait forever.\n \n+filewatcher.minfiles::\n+\tStart watching files if the number of watchable files are\n+\tabove this limit. Default value is 65536.\n+\n+filewatcher.recentlimit::\n+\tFiles that are last updated within filewatcher.recentlimit\n+\tseconds from now are not considered watchable. Default value\n+\tis 600 (5 minutes).\n+\n fetch.recurseSubmodules::\n \tThis option can be either set to a boolean value or to 'on-demand'.\n \tSetting it to a boolean changes the behavior of fetch and pull to\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex 24fd0ae..7081e55 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -204,3 +204,6 @@ Git index format\n \n - A bit map of all entries in the index, n-th bit of m-th byte\n corresponds to CE_WATCHED of the -th index entry.\n+\n+ - 1-byte, non-zero indicates the index should be scanned for new\n+ watched entries.\ndiff --git a/cache.h b/cache.h\nindex b3ea574..10ff33e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -279,6 +279,7 @@ struct index_state {\n \tstruct cache_tree *cache_tree;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n+\t\t update_watches : 1,\n \t\t initialized : 1;\n \tstruct hash_table name_hash;\n \tstruct hash_table dir_hash;\ndiff --git a/file-watcher-lib.c b/file-watcher-lib.c\nindex 791faae..d4949a5 100644\n--- a/file-watcher-lib.c\n+++ b/file-watcher-lib.c\n@@ -5,6 +5,8 @@\n \n static char *watcher_path;\n static int WAIT_TIME = 50;\t/* in ms */\n+static int watch_lowerlimit = 65536;\n+static int recent_limit = 600;\n \n static int connect_watcher(const char *path)\n {\n@@ -22,12 +24,17 @@ static int connect_watcher(const char *path)\n \n static void reset_watches(struct index_state *istate, int disconnect)\n {\n-\tint i;\n+\tint i, changed = 0;\n \tfor (i = 0; i < istate->cache_nr; i++)\n \t\tif (istate->cache[i]->ce_flags & CE_WATCHED) {\n \t\t\tistate->cache[i]->ce_flags &= ~(CE_WATCHED | CE_VALID);\n-\t\t\tistate->cache_changed = 1;\n+\t\t\tchanged = 1;\n \t\t}\n+\trecent_limit = 0;\n+\tif (changed) {\n+\t\tistate->update_watches = 1;\n+\t\tistate->cache_changed = 1;\n+\t}\n \tif (disconnect && istate->watcher > 0) {\n \t\tclose(istate->watcher);\n \t\tistate->watcher = -1;\n@@ -49,6 +56,14 @@ static int watcher_config(const char *var, const char *value, void *data)\n \t\tWAIT_TIME = git_config_int(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"filewatcher.minfiles\")) {\n+\t\twatch_lowerlimit = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(var, \"filewatcher.recentlimit\")) {\n+\t\trecent_limit = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n \treturn 0;\n }\n \n@@ -63,12 +78,18 @@ void open_watcher(struct index_state *istate)\n \t}\n \n \tif (!read_config) {\n+\t\tint i;\n \t\t/*\n \t\t * can't hook into git_default_config because\n \t\t * read_cache() may be called even before git_config()\n \t\t * call.\n \t\t */\n \t\tgit_config(watcher_config, NULL);\n+\t\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\t\tif (istate->cache[i]->ce_flags & CE_WATCHED)\n+\t\t\t\tbreak;\n+\t\tif (i == istate->cache_nr)\n+\t\t\trecent_limit = 0;\n \t\tread_config = 1;\n \t}\n \n@@ -86,6 +107,7 @@ void open_watcher(struct index_state *istate)\n \t (msg = packet_read_line_timeout(istate->watcher, WAIT_TIME, NULL)) == NULL ||\n \t strcmp(msg, \"ok\")) {\n \t\treset_watches(istate, 0);\n+\t\tistate->update_watches = 1;\n \t\treturn;\n \t}\n }\n@@ -99,7 +121,7 @@ static int sort_by_date(const void *a_, const void *b_)\n \treturn seca - secb;\n }\n \n-static inline int ce_watchable(struct cache_entry *ce)\n+static inline int ce_watchable(struct cache_entry *ce, time_t now)\n {\n \treturn\n \t\t!(ce->ce_flags & CE_WATCHED) &&\n@@ -109,7 +131,8 @@ static inline int ce_watchable(struct cache_entry *ce)\n \t\t * obviously. S_IFLNK could be problematic because\n \t\t * inotify may follow symlinks without IN_DONT_FOLLOW\n \t\t */\n-\t\tS_ISREG(ce->ce_mode);\n+\t\tS_ISREG(ce->ce_mode) &&\n+\t\t(ce->ce_stat_data.sd_mtime.sec + recent_limit <= now);\n }\n \n static void send_watches(struct index_state *istate,\n@@ -158,15 +181,20 @@ void watch_entries(struct index_state *istate)\n {\n \tint i, nr;\n \tstruct cache_entry **sorted;\n+\ttime_t now = time(NULL);\n \n-\tif (istate->watcher <= 0)\n+\tif (istate->watcher <= 0 || !istate->update_watches)\n \t\treturn;\n+\tistate->update_watches = 0;\n+\tistate->cache_changed = 1;\n \tfor (i = nr = 0; i < istate->cache_nr; i++)\n-\t\tif (ce_watchable(istate->cache[i]))\n+\t\tif (ce_watchable(istate->cache[i], now))\n \t\t\tnr++;\n+\tif (nr < watch_lowerlimit)\n+\t\treturn;\n \tsorted = xmalloc(sizeof(*sorted) * nr);\n \tfor (i = nr = 0; i < istate->cache_nr; i++)\n-\t\tif (ce_watchable(istate->cache[i]))\n+\t\tif (ce_watchable(istate->cache[i], now))\n \t\t\tsorted[nr++] = istate->cache[i];\n \tqsort(sorted, nr, sizeof(*sorted), sort_by_date);\n \tsend_watches(istate, sorted, nr);\ndiff --git a/read-cache.c b/read-cache.c\nindex cb2188f..dc49858 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1009,6 +1009,7 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n \t\t\t(istate->cache_nr - pos - 1) * sizeof(ce));\n \tset_index_entry(istate, pos, ce);\n \tistate->cache_changed = 1;\n+\tistate->update_watches = 1;\n \treturn 0;\n }\n \n@@ -1295,13 +1296,14 @@ static void read_watch_extension(struct index_state *istate, uint8_t *data,\n \t\t\t\t unsigned long sz)\n {\n \tint i;\n-\tif ((istate->cache_nr + 7) / 8 != sz) {\n+\tif ((istate->cache_nr + 7) / 8 + 1 != sz) {\n \t\terror(\"invalid 'WATC' extension\");\n \t\treturn;\n \t}\n \tfor (i = 0; i < istate->cache_nr; i++)\n \t\tif (data[i / 8] & (1 << (i % 8)))\n \t\t\tistate->cache[i]->ce_flags |= CE_WATCHED;\n+\tistate->update_watches = data[sz - 1];\n }\n \n static int read_index_extension(struct index_state *istate,\n@@ -1488,6 +1490,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tistate->cache_alloc = alloc_nr(istate->cache_nr);\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n+\tistate->update_watches = 1;\n \n \tif (istate->version == 4)\n \t\tprevious_name = &previous_name_buf;\n@@ -1896,8 +1899,9 @@ int write_index(struct index_state *istate, int newfd)\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n-\tif (has_watches) {\n-\t\tint id, sz = (entries - removed + 7) / 8;\n+\tif (has_watches ||\n+\t (istate->watcher != -1 && !istate->update_watches)) {\n+\t\tint id, sz = (entries - removed + 7) / 8 + 1;\n \t\tuint8_t *data = xmalloc(sz);\n \t\tmemset(data, 0, sz);\n \t\tfor (i = 0, id = 0; i < entries && has_watches; i++) {\n@@ -1910,6 +1914,7 @@ int write_index(struct index_state *istate, int newfd)\n \t\t\t}\n \t\t\tid++;\n \t\t}\n+\t\tdata[sz - 1] = istate->update_watches;\n \t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_WATCH, sz) < 0\n \t\t\t|| ce_write(&c, newfd, data, sz) < 0;\n \t\tfree(data);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234086","messageId":"1391401754-15347-16-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 15/26] read-cache: get changed file list from file watcher","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:03Z","receivedAt":"2014-02-03T04:29:03Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When some paths are watched, they are added to the \"watched\" list in\nfile watcher. When a path in this list is updated, the path is moved\nto \"changed\" list and no longer watched.\n\nWith this patch we have a complete path exchanging picture between git\nand file-watcher:\n\n1) Hand shake\n\n2) Get the list of changed paths, clear CE_WATCHED on these paths. Set\n CE_VALID on the remaining CE_WATCHED paths\n\n3) (Optionally) Ask to watch more paths. Set CE_WATCHED on\n them. CE_VALID is not set so these are still lstat'd\n\n4) Refresh as usual. lstat is skipped on CE_VALID paths. If one of\n those paths at step 3 are found modified, CE_WATCHED is removed.\n\n5) Write index to disk. Notify file-watcher about new index\n signature. Ask file watcher to remove the \"changed paths\".\n\nA few points:\n\n - Changed list remains until step 5. If git crashes or do not write\n index down, next time it starts, it'll fed the same changed list.\n\n - If git writes index down without telling file-watcher about it,\n next time it starts, hand shake should fail and git should clear\n all CE_WATCHED.\n\n - There's a buffer between starting watch at #3 and saving watch at\n #5. We do verify paths are clean at #4. But that time all watches\n should have been active for a while. No chance for race conditions.\n\n - #5 is sort of atomic. If git crashes half way through step 5, file\n watcher should not update its index signature. Which means next\n time git starts, hand shake fails (because new index's written) so\n we'll start over.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n file-watcher-lib.c | 99 ++++++++++++++++++++++++++++++++++++++\n file-watcher-lib.h | 1 +\n file-watcher.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n read-cache.c | 21 +++++++-\n 5 files changed, 258 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 10ff33e..9f7d952 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -285,6 +285,7 @@ struct index_state {\n \tstruct hash_table dir_hash;\n \tunsigned char sha1[20];\n \tint watcher;\n+\tstruct string_list *updated_entries;\n };\n \n extern struct index_state the_index;\ndiff --git a/file-watcher-lib.c b/file-watcher-lib.c\nindex d4949a5..b6b0848 100644\n--- a/file-watcher-lib.c\n+++ b/file-watcher-lib.c\n@@ -2,6 +2,7 @@\n #include \"file-watcher-lib.h\"\n #include \"pkt-line.h\"\n #include \"unix-socket.h\"\n+#include \"string-list.h\"\n \n static char *watcher_path;\n static int WAIT_TIME = 50;\t/* in ms */\n@@ -25,6 +26,11 @@ static int connect_watcher(const char *path)\n static void reset_watches(struct index_state *istate, int disconnect)\n {\n \tint i, changed = 0;\n+\tif (istate->updated_entries) {\n+\t\tstring_list_clear(istate->updated_entries, 0);\n+\t\tfree(istate->updated_entries);\n+\t\tistate->updated_entries = NULL;\n+\t}\n \tfor (i = 0; i < istate->cache_nr; i++)\n \t\tif (istate->cache[i]->ce_flags & CE_WATCHED) {\n \t\t\tistate->cache[i]->ce_flags &= ~(CE_WATCHED | CE_VALID);\n@@ -41,6 +47,58 @@ static void reset_watches(struct index_state *istate, int disconnect)\n \t}\n }\n \n+static void mark_ce_valid(struct index_state *istate)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tchar *line, *end;\n+\tint i, len;\n+\tunsigned long n;\n+\tif (packet_write_timeout(istate->watcher, WAIT_TIME, \"get-changed\") <= 0 ||\n+\t !(line = packet_read_line_timeout(istate->watcher, WAIT_TIME, &len)) ||\n+\t !starts_with(line, \"changed \")) {\n+\t\treset_watches(istate, 1);\n+\t\treturn;\n+\t}\n+\tn = strtoul(line + 8, &end, 10);\n+\tif (end != line + len) {\n+\t\treset_watches(istate, 1);\n+\t\treturn;\n+\t}\n+\tif (!n)\n+\t\tgoto done;\n+\tstrbuf_grow(&sb, n);\n+\tif (read_in_full_timeout(istate->watcher, sb.buf, n, WAIT_TIME) != n) {\n+\t\tstrbuf_release(&sb);\n+\t\treset_watches(istate, 1);\n+\t\treturn;\n+\t}\n+\tline = sb.buf;\n+\tend = line + n;\n+\tfor (; line < end; line += len + 1) {\n+\t\tlen = strlen(line);\n+\t\ti = index_name_pos(istate, line, len);\n+\t\tif (i < 0)\n+\t\t\tcontinue;\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHED) {\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_WATCHED;\n+\t\t\tistate->cache_changed = 1;\n+\t\t}\n+\t\tif (!istate->updated_entries) {\n+\t\t\tstruct string_list *sl;\n+\t\t\tsl = xmalloc(sizeof(*sl));\n+\t\t\tmemset(sl, 0, sizeof(*sl));\n+\t\t\tsl->strdup_strings = 1;\n+\t\t\tistate->updated_entries = sl;\n+\t\t}\n+\t\tstring_list_append(istate->updated_entries, line);\n+\t}\n+\tstrbuf_release(&sb);\n+done:\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tif (istate->cache[i]->ce_flags & CE_WATCHED)\n+\t\t\tistate->cache[i]->ce_flags |= CE_VALID;\n+}\n+\n static int watcher_config(const char *var, const char *value, void *data)\n {\n \tif (!strcmp(var, \"filewatcher.path\")) {\n@@ -110,6 +168,8 @@ void open_watcher(struct index_state *istate)\n \t\tistate->update_watches = 1;\n \t\treturn;\n \t}\n+\n+\tmark_ce_valid(istate);\n }\n \n static int sort_by_date(const void *a_, const void *b_)\n@@ -200,3 +260,42 @@ void watch_entries(struct index_state *istate)\n \tsend_watches(istate, sorted, nr);\n \tfree(sorted);\n }\n+\n+void close_watcher(struct index_state *istate, const unsigned char *sha1)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint len, i, nr;\n+\tif (istate->watcher <= 0)\n+\t\treturn;\n+\tif (packet_write_timeout(istate->watcher, WAIT_TIME,\n+\t\t\t\t \"new-index %s\", sha1_to_hex(sha1)) <= 0)\n+\t\tgoto done;\n+\tnr = istate->updated_entries ? istate->updated_entries->nr : 0;\n+\tif (!nr) {\n+\t\tpacket_write_timeout(istate->watcher, WAIT_TIME, \"unchange 0\");\n+\t\tgoto done;\n+\t}\n+\tfor (i = len = 0; i < nr; i++) {\n+\t\tconst char *s = istate->updated_entries->items[i].string;\n+\t\tlen += strlen(s) + 1;\n+\t}\n+\tif (packet_write_timeout(istate->watcher, WAIT_TIME,\n+\t\t\t\t \"unchange %d\", len) <= 0)\n+\t goto done;\n+\tstrbuf_grow(&sb, len);\n+\tfor (i = 0; i < nr; i++) {\n+\t\tconst char *s = istate->updated_entries->items[i].string;\n+\t\tint len = strlen(s);\n+\t\tstrbuf_add(&sb, s, len + 1);\n+\t}\n+\t/*\n+\t * it does not matter if it fails anymore, we're closing\n+\t * down. If it only gets through partially, file watcher\n+\t * should ignore it.\n+\t */\n+\twrite_in_full_timeout(istate->watcher, sb.buf, sb.len, WAIT_TIME);\n+\tstrbuf_release(&sb);\n+done:\n+\tclose(istate->watcher);\n+\tistate->watcher = -1;\n+}\ndiff --git a/file-watcher-lib.h b/file-watcher-lib.h\nindex 1641024..df68a73 100644\n--- a/file-watcher-lib.h\n+++ b/file-watcher-lib.h\n@@ -3,5 +3,6 @@\n \n void open_watcher(struct index_state *istate);\n void watch_entries(struct index_state *istate);\n+void close_watcher(struct index_state *istate, const unsigned char *sha1);\n \n #endif\ndiff --git a/file-watcher.c b/file-watcher.c\nindex c257414..aa2daf6 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"exec_cmd.h\"\n #include \"unix-socket.h\"\n+#include \"string-list.h\"\n #include \"pkt-line.h\"\n \n static const char *const file_watcher_usage[] = {\n@@ -21,6 +22,9 @@ struct repository {\n \t * is probably enough for this case.\n \t */\n \tino_t inode;\n+\tstruct string_list updated;\n+\tint updated_sorted;\n+\tint updating;\n };\n \n const char *invalid_signature = \"0000000000000000000000000000000000000000\";\n@@ -31,6 +35,8 @@ static int nr_repos;\n struct connection {\n \tint sock, polite;\n \tstruct repository *repo;\n+\n+\tchar new_index[41];\n };\n \n static struct connection **conns;\n@@ -42,6 +48,24 @@ static int watch_path(struct repository *repo, char *path)\n \treturn -1;\n }\n \n+static void get_changed_list(int conn_id)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint i, size, fd = conns[conn_id]->sock;\n+\tstruct repository *repo = conns[conn_id]->repo;\n+\n+\tfor (i = size = 0; i < repo->updated.nr; i++)\n+\t\tsize += strlen(repo->updated.items[i].string) + 1;\n+\tpacket_write(fd, \"changed %d\", size);\n+\tif (!size)\n+\t\treturn;\n+\tstrbuf_grow(&sb, size);\n+\tfor (i = 0; i < repo->updated.nr; i++)\n+\t\tstrbuf_add(&sb, repo->updated.items[i].string,\n+\t\t\t strlen(repo->updated.items[i].string) + 1);\n+\twrite_in_full(fd, sb.buf, sb.len);\n+}\n+\n static inline uint64_t stamp(void)\n {\n \tstruct timeval tv;\n@@ -101,6 +125,43 @@ static void watch_paths(int conn_id, char *buf, int maxlen)\n \tpacket_write(conns[conn_id]->sock, \"watched %u\", n);\n }\n \n+static void unchange(int conn_id, unsigned long size)\n+{\n+\tstruct connection *conn = conns[conn_id];\n+\tstruct repository *repo = conn->repo;\n+\tif (size) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tchar *p;\n+\t\tint len;\n+\t\tstrbuf_grow(&sb, size);\n+\t\tif (read_in_full(conn->sock, sb.buf, size) <= 0)\n+\t\t\treturn;\n+\t\tif (!repo->updated_sorted) {\n+\t\t\tsort_string_list(&repo->updated);\n+\t\t\trepo->updated_sorted = 1;\n+\t\t}\n+\t\tfor (p = sb.buf; p - sb.buf < size; p += len + 1) {\n+\t\t\tstruct string_list_item *item;\n+\t\t\tlen = strlen(p);\n+\t\t\titem = string_list_lookup(&repo->updated, p);\n+\t\t\tif (!item)\n+\t\t\t\tcontinue;\n+\t\t\tunsorted_string_list_delete_item(&repo->updated,\n+\t\t\t\t\t\t\t item - repo->updated.items, 0);\n+\t\t}\n+\t\tstrbuf_release(&sb);\n+\t}\n+\tmemcpy(repo->index_signature, conn->new_index, 40);\n+\t/*\n+\t * If other connections on this repo are in some sort of\n+\t * session that depend on the previous repository state, we\n+\t * may need to disconnect them to be safe.\n+\t */\n+\n+\t/* pfd[0] is the listening socket, can't be a connection */\n+\trepo->updating = 0;\n+}\n+\n static struct repository *get_repo(const char *work_tree)\n {\n \tint first, last;\n@@ -129,12 +190,14 @@ static struct repository *get_repo(const char *work_tree)\n \tmemset(repo, 0, sizeof(*repo));\n \trepo->work_tree = xstrdup(work_tree);\n \tmemset(repo->index_signature, '0', 40);\n+\trepo->updated.strdup_strings = 1;\n \trepos[first] = repo;\n \treturn repo;\n }\n \n static void reset_repo(struct repository *repo, ino_t inode)\n {\n+\tstring_list_clear(&repo->updated, 0);\n \tmemcpy(repo->index_signature, invalid_signature, 40);\n \trepo->inode = inode;\n }\n@@ -147,6 +210,8 @@ static int shutdown_connection(int id)\n \tif (!conn)\n \t\treturn 0;\n \tclose(conn->sock);\n+\tif (conn->repo && conn->repo->updating == id)\n+\t\tconn->repo->updating = 0;\n \tfree(conn);\n \treturn 0;\n }\n@@ -268,6 +333,77 @@ static int handle_command(int conn_id)\n \t\t}\n \t\twatch_paths(conn_id, msg + 6, len - 6);\n \t}\n+\n+\t/*\n+\t * > \"get-changed\"\n+\t * < changed SP LENGTH\n+\t * < PATH-LIST\n+\t *\n+\t * When watched path gets updated, the path is moved from\n+\t * \"watched\" list to \"changed\" list and is no longer watched.\n+\t * This command get the list of changed paths. PATH-LIST is\n+\t * also sent if LENGTH is non-zero.\n+\t */\n+\telse if (!strcmp(msg, \"get-changed\")) {\n+\t\tif (!conns[conn_id]->repo) {\n+\t\t\tpacket_write(fd, \"error have not received index command\");\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\tget_changed_list(conn_id);\n+\t}\n+\n+\t/*\n+\t * > \"new-index\" INDEX-SIGNATURE\n+\t * > \"unchange\" SP LENGTH\n+\t * > PATH-LIST\n+\t *\n+\t * \"new-index\" passes new index signature from the\n+\t * client. \"unchange\" sends the list of paths to be removed\n+\t * from \"changed\" list.\n+\t *\n+\t * \"new-index\" must be sent before \"unchange\". File watcher\n+\t * waits until the last \"unchange\" line, then update its index\n+\t * signature as well as \"changed\" list.\n+\t */\n+\telse if (starts_with(msg, \"new-index \")) {\n+\t\tif (len != 50) {\n+\t\t\tpacket_write(fd, \"error invalid new-index line %s\", msg);\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\tif (!conns[conn_id]->repo) {\n+\t\t\tpacket_write(fd, \"error have not received index command\");\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\tif (conns[conn_id]->repo->updating == conn_id) {\n+\t\t\tpacket_write(fd, \"error received new-index command more than once\");\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\tmemcpy(conns[conn_id]->new_index, msg + 10, 40);\n+\t\t/*\n+\t\t * if updating is non-zero the other client will get\n+\t\t * disconnected at the next \"unchange\" command because\n+\t\t * \"updating\" no longer points to its connection.\n+\t\t */\n+\t\tconns[conn_id]->repo->updating = conn_id;\n+\t}\n+\telse if (skip_prefix(msg, \"unchange \")) {\n+\t\tunsigned long n;\n+\t\tchar *end;\n+\t\tn = strtoul(msg + 9, &end, 10);\n+\t\tif (end != msg + len) {\n+\t\t\tpacket_write(fd, \"error invalid unchange line %s\", msg);\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\tif (!conns[conn_id]->repo) {\n+\t\t\tpacket_write(fd, \"error have not received index command\");\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\tif (conns[conn_id]->repo->updating != conn_id) {\n+\t\t\tpacket_write(fd, \"error have not received new-index command\");\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\tunchange(conn_id, n);\n+\t}\n \telse {\n \t\tpacket_write(fd, \"error unrecognized command %s\", msg);\n \t\treturn shutdown_connection(conn_id);\n@@ -436,6 +572,8 @@ int main(int argc, const char **argv)\n \t\t\tif (!conns[i])\n \t\t\t\tcontinue;\n \t\t\tif (i != new_nr) { /* pfd[] is shrunk, move pfd[i] up */\n+\t\t\t\tif (conns[i]->repo && conns[i]->repo->updating == i)\n+\t\t\t\t\tconns[i]->repo->updating = new_nr;\n \t\t\t\tconns[new_nr] = conns[i];\n \t\t\t\tpfd[new_nr] = pfd[i];\n \t\t\t}\ndiff --git a/read-cache.c b/read-cache.c\nindex dc49858..5540b06 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1567,6 +1567,11 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->updated_entries) {\n+\t\tstring_list_clear(istate->updated_entries, 0);\n+\t\tfree(istate->updated_entries);\n+\t\tistate->updated_entries = NULL;\n+\t}\n \treturn 0;\n }\n \n@@ -1627,7 +1632,7 @@ static int write_index_ext_header(git_SHA_CTX *context, int fd,\n \t\t(ce_write(context, fd, &sz, 4) < 0)) ? -1 : 0;\n }\n \n-static int ce_flush(git_SHA_CTX *context, int fd)\n+static int ce_flush(git_SHA_CTX *context, int fd, unsigned char *sha1)\n {\n \tunsigned int left = write_buffer_len;\n \n@@ -1645,6 +1650,8 @@ static int ce_flush(git_SHA_CTX *context, int fd)\n \n \t/* Append the SHA1 signature at the end */\n \tgit_SHA1_Final(write_buffer + left, context);\n+\tif (sha1)\n+\t\thashcpy(sha1, write_buffer + left);\n \tleft += 20;\n \treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n }\n@@ -1809,12 +1816,21 @@ int write_index(struct index_state *istate, int newfd)\n \tint entries = istate->cache_nr;\n \tstruct stat st;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n+\tunsigned char sha1[20];\n \n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n \t\t\tremoved++;\n \t\telse if (cache[i]->ce_flags & CE_WATCHED) {\n \t\t\t/*\n+\t\t\t * CE_VALID when used with CE_WATCHED is not\n+\t\t\t * supposed to be persistent. Next time git\n+\t\t\t * runs, if this entry is still watched and\n+\t\t\t * nothing has changed, CE_VALID will be\n+\t\t\t * reinstated.\n+\t\t\t */\n+\t\t\tcache[i]->ce_flags &= ~CE_VALID;\n+\t\t\t/*\n \t\t\t * We may set CE_WATCHED (but not CE_VALID)\n \t\t\t * early when refresh has not been done\n \t\t\t * yet. At that time we had no idea if the\n@@ -1922,8 +1938,9 @@ int write_index(struct index_state *istate, int newfd)\n \t\t\treturn -1;\n \t}\n \n-\tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n+\tif (ce_flush(&c, newfd, sha1) || fstat(newfd, &st))\n \t\treturn -1;\n+\tclose_watcher(istate, sha1);\n \tistate->timestamp.sec = (unsigned int)st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \treturn 0;\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234087","messageId":"1391401754-15347-17-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 16/26] git-compat-util.h: add inotify stubs on non-Linux platforms","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:04Z","receivedAt":"2014-02-03T04:29:04Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is to avoid spreading #ifdef HAVE_INOTIFY around and keep most\ncode compiled even if it's never active.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n config.mak.uname | 1 +\n git-compat-util.h | 43 +++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 44 insertions(+)\n\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 7d31fad..ee548f5 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -33,6 +33,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_PATHS_H = YesPlease\n \tLIBC_CONTAINS_LIBINTL = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tBASIC_CFLAGS += -DHAVE_INOTIFY\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tNO_STRLCPY = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex cbd86c3..8b55dd0 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -128,6 +128,9 @@\n #else\n #include \n #endif\n+#ifdef HAVE_INOTIFY\n+#include \n+#endif\n \n #if defined(__MINGW32__)\n /* pull in Windows compatibility stuff */\n@@ -721,4 +724,44 @@ void warn_on_inaccessible(const char *path);\n /* Get the passwd entry for the UID of the current process. */\n struct passwd *xgetpwuid_self(void);\n \n+#ifndef HAVE_INOTIFY\n+/* Keep inotify-specific code build, even if it's not used */\n+\n+#define IN_DELETE_SELF\t1\n+#define IN_MOVE_SELF\t2\n+#define IN_ATTRIB\t4\n+#define IN_DELETE\t8\n+#define IN_MODIFY\t16\n+#define IN_MOVED_FROM\t32\n+#define IN_MOVED_TO\t64\n+#define IN_Q_OVERFLOW\t128\n+#define IN_UNMOUNT\t256\n+#define IN_CREATE\t512\n+#define IN_ISDIR\t1024\n+#define IN_IGNORED\t2048\n+\n+struct inotify_event {\n+\tint event, mask, wd, len;\n+\tchar name[FLEX_ARRAY];\n+};\n+\n+static inline int inotify_init()\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+\n+static inline int inotify_add_watch(int fd, const char *path, int options)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+\n+static inline int inotify_rm_watch(int fd, int wd)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+#endif\n+\n #endif\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234088","messageId":"1391401754-15347-18-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 17/26] file-watcher: inotify support, watching part","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:05Z","receivedAt":"2014-02-03T04:29:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\"struct dir\" manages inotify file descriptor and forms a tree. \"struct\nfile\" manages a file. When a file is watched, all dirs up to the file\nis watched. Any changes on a directory impacts all subdirs and files.\n\nThe way data structure is made might be inotify-specific. I haven't\nthought of how other file notification mechanisms may be\nimplemented. So there may be some refactoring later when a new OS is\nsupported.\n\nRoom for improvement: consecutive watched paths likely share the same\ndirectory part (even though they are sorted by mtime, not name). Try\nremember the last \"dir\" sequence and reduce lookups.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-file-watcher.txt | 13 +++\n file-watcher.c | 226 ++++++++++++++++++++++++++++++++++++-\n 2 files changed, 238 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-file-watcher.txt b/Documentation/git-file-watcher.txt\nindex d91caf3..d694fea 100644\n--- a/Documentation/git-file-watcher.txt\n+++ b/Documentation/git-file-watcher.txt\n@@ -18,11 +18,24 @@ lstat(2) to detect that itself. Config key filewatcher.path needs to\n be set to `` so Git knows how to contact to the file\n watcher.\n \n+This program is only supported under Linux with inotify(7) support.\n+\n OPTIONS\n -------\n --detach::\n \tRun in background.\n \n+BUGS\n+----\n+On Linux, file watcher may fail to detect changes if you move the work\n+tree from outside. For example if you have work tree at\n+`/tmp/foo/work`, you move `/tmp/foo` to `/tmp/bar`, make some changes\n+in `/tmp/bar/work` and move `/tmp/bar` back to `/tmp/foo`, changes\n+won't get noticed. Moving `/tmp/foo/work` to something else is fine.\n+\n+inotify may not work well with network filesystems and a few special\n+others. Check inotify documents.\n+\n SEE ALSO\n --------\n linkgit:git-update-index[1],\ndiff --git a/file-watcher.c b/file-watcher.c\nindex aa2daf6..d0762e6 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -11,6 +11,28 @@ static const char *const file_watcher_usage[] = {\n \tNULL\n };\n \n+struct dir;\n+struct repository;\n+\n+struct file {\n+\tchar *name;\n+\tstruct dir *parent;\n+\tstruct repository *repo;\n+\tstruct file *next;\n+};\n+\n+struct dir {\n+\tchar *name;\n+\tstruct dir *parent;\n+\tstruct dir **subdirs;\n+\tstruct file **files;\n+\tstruct repository *repo; /* only for root note */\n+\tint wd, nr_subdirs, nr_files;\n+};\n+\n+static struct dir **wds;\n+static int wds_alloc;\n+\n struct repository {\n \tchar *work_tree;\n \tchar index_signature[41];\n@@ -25,6 +47,7 @@ struct repository {\n \tstruct string_list updated;\n \tint updated_sorted;\n \tint updating;\n+\tstruct dir *root;\n };\n \n const char *invalid_signature = \"0000000000000000000000000000000000000000\";\n@@ -42,10 +65,199 @@ struct connection {\n static struct connection **conns;\n static struct pollfd *pfd;\n static int conns_alloc, pfd_nr, pfd_alloc;\n+static int inotify_fd;\n+\n+/*\n+ * IN_DONT_FOLLOW does not matter now as we do not monitor\n+ * symlinks. See ce_watchable().\n+ */\n+#define INOTIFY_MASKS (IN_DELETE_SELF | IN_MOVE_SELF | \\\n+\t\t IN_CREATE | IN_ATTRIB | IN_DELETE | IN_MODIFY |\t\\\n+\t\t IN_MOVED_FROM | IN_MOVED_TO)\n+static struct dir *create_dir(struct dir *parent, const char *path,\n+\t\t\t const char *basename)\n+{\n+\tstruct dir *d;\n+\tint wd = inotify_add_watch(inotify_fd, path, INOTIFY_MASKS);\n+\tif (wd < 0)\n+\t\treturn NULL;\n+\n+\td = xmalloc(sizeof(*d));\n+\tmemset(d, 0, sizeof(*d));\n+\td->wd = wd;\n+\td->parent = parent;\n+\td->name = xstrdup(basename);\n+\n+\tALLOC_GROW(wds, wd + 1, wds_alloc);\n+\twds[wd] = d;\n+\treturn d;\n+}\n+\n+static int get_dir_pos(struct dir *d, const char *name)\n+{\n+\tint first, last;\n+\n+\tfirst = 0;\n+\tlast = d->nr_subdirs;\n+\twhile (last > first) {\n+\t\tint next = (last + first) >> 1;\n+\t\tint cmp = strcmp(name, d->subdirs[next]->name);\n+\t\tif (!cmp)\n+\t\t\treturn next;\n+\t\tif (cmp < 0) {\n+\t\t\tlast = next;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfirst = next+1;\n+\t}\n+\n+\treturn -first-1;\n+}\n+\n+static void free_file(struct dir *d, int pos, int topdown);\n+static void free_dir(struct dir *d, int topdown)\n+{\n+\tstruct dir *p = d->parent;\n+\tint pos;\n+\tif (!topdown && p && (pos = get_dir_pos(p, d->name)) < 0)\n+\t\tdie(\"How come this directory is not registered in its parent?\");\n+\tif (d->repo)\n+\t\td->repo->root = NULL;\n+\twds[d->wd] = NULL;\n+\tinotify_rm_watch(inotify_fd, d->wd);\n+\tif (topdown) {\n+\t\tint i;\n+\t\tfor (i = 0; i < d->nr_subdirs; i++)\n+\t\t\tfree_dir(d->subdirs[i], topdown);\n+\t\tfor (i = 0; i < d->nr_files; i++)\n+\t\t\tfree_file(d, i, topdown);\n+\t}\n+\tfree(d->name);\n+\tfree(d->subdirs);\n+\tfree(d->files);\n+\tfree(d);\n+\tif (p && !topdown) {\n+\t\tp->nr_subdirs--;\n+\t\tmemmove(p->subdirs + pos, p->subdirs + pos + 1,\n+\t\t\t(p->nr_subdirs - pos) * sizeof(*p->subdirs));\n+\t\tif (!p->nr_subdirs && !p->nr_files)\n+\t\t\tfree_dir(p, topdown);\n+\t}\n+}\n+\n+static int get_file_pos(struct dir *d, const char *name)\n+{\n+\tint first, last;\n+\n+\tfirst = 0;\n+\tlast = d->nr_files;\n+\twhile (last > first) {\n+\t\tint next = (last + first) >> 1;\n+\t\tint cmp = strcmp(name, d->files[next]->name);\n+\t\tif (!cmp)\n+\t\t\treturn next;\n+\t\tif (cmp < 0) {\n+\t\t\tlast = next;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfirst = next+1;\n+\t}\n+\n+\treturn -first-1;\n+}\n+\n+static void free_file(struct dir *d, int pos, int topdown)\n+{\n+\tstruct file *f = d->files[pos];\n+\tfree(f->name);\n+\tfree(f);\n+\tif (!topdown) {\n+\t\td->nr_files--;\n+\t\tmemmove(d->files + pos, d->files + pos + 1,\n+\t\t\t(d->nr_files - pos) * sizeof(*d->files));\n+\t\tif (!d->nr_subdirs && !d->nr_files)\n+\t\t\tfree_dir(d, topdown);\n+\t}\n+}\n+\n+static struct dir *add_dir(struct dir *d,\n+\t\t\t const char *path, const char *basename)\n+{\n+\tstruct dir *new;\n+\tint pos = get_dir_pos(d, basename);\n+\tif (pos >= 0)\n+\t\treturn d->subdirs[pos];\n+\tpos = -pos-1;\n+\n+\tnew = create_dir(d, path, basename);\n+\tif (!new)\n+\t\treturn NULL;\n+\n+\td->nr_subdirs++;\n+\td->subdirs = xrealloc(d->subdirs, sizeof(*d->subdirs) * d->nr_subdirs);\n+\tif (d->nr_subdirs > pos + 1)\n+\t\tmemmove(d->subdirs + pos + 1, d->subdirs + pos,\n+\t\t\t(d->nr_subdirs - pos - 1) * sizeof(*d->subdirs));\n+\td->subdirs[pos] = new;\n+\treturn new;\n+}\n+\n+static struct file *add_file(struct dir *d, const char *name)\n+{\n+\tstruct file *new;\n+\tint pos = get_file_pos(d, name);\n+\tif (pos >= 0)\n+\t\treturn d->files[pos];\n+\tpos = -pos-1;\n+\n+\tnew = xmalloc(sizeof(*new));\n+\tmemset(new, 0, sizeof(*new));\n+\tnew->parent = d;\n+\tnew->name = xstrdup(name);\n+\n+\td->nr_files++;\n+\td->files = xrealloc(d->files, sizeof(*d->files) * d->nr_files);\n+\tif (d->nr_files > pos + 1)\n+\t\tmemmove(d->files + pos + 1, d->files + pos,\n+\t\t\t(d->nr_files - pos - 1) * sizeof(*d->files));\n+\td->files[pos] = new;\n+\treturn new;\n+}\n \n static int watch_path(struct repository *repo, char *path)\n {\n-\treturn -1;\n+\tstruct dir *d = repo->root;\n+\tchar *p = path;\n+\n+\tif (!d) {\n+\t\td = create_dir(NULL, \".\", \"\");\n+\t\tif (!d)\n+\t\t\treturn -1;\n+\t\trepo->root = d;\n+\t\td->repo = repo;\n+\t}\n+\n+\tfor (;;) {\n+\t\tchar *next, *sep;\n+\t\tsep = strchr(p, '/');\n+\t\tif (!sep) {\n+\t\t\tstruct file *file;\n+\t\t\tfile = add_file(d, p);\n+\t\t\tif (!file->repo)\n+\t\t\t\tfile->repo = repo;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tnext = sep + 1;\n+\t\t*sep = '\\0';\n+\t\td = add_dir(d, path, p);\n+\t\tif (!d)\n+\t\t\t/* we could free oldest watches and try again */\n+\t\t\treturn -1;\n+\t\t*sep = '/';\n+\t\tp = next;\n+\t}\n+\treturn 0;\n }\n \n static void get_changed_list(int conn_id)\n@@ -195,8 +407,15 @@ static struct repository *get_repo(const char *work_tree)\n \treturn repo;\n }\n \n+static void reset_watches(struct repository *repo)\n+{\n+\tif (repo->root)\n+\t\tfree_dir(repo->root, 1);\n+}\n+\n static void reset_repo(struct repository *repo, ino_t inode)\n {\n+\treset_watches(repo);\n \tstring_list_clear(&repo->updated, 0);\n \tmemcpy(repo->index_signature, invalid_signature, 40);\n \trepo->inode = inode;\n@@ -497,6 +716,11 @@ int main(int argc, const char **argv)\n \n \tgit_extract_argv0_path(argv[0]);\n \tgit_setup_gettext();\n+\n+\tinotify_fd = inotify_init();\n+\tif (inotify_fd < 0)\n+\t\tdie_errno(\"unable to initialize inotify\");\n+\n \targc = parse_options(argc, argv, NULL, options,\n \t\t\t file_watcher_usage, 0);\n \tif (argc < 1)\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234089","messageId":"1391401754-15347-19-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 18/26] file-watcher: inotify support, notification part","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:06Z","receivedAt":"2014-02-03T04:29:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 141 insertions(+), 1 deletion(-)\n\ndiff --git a/file-watcher.c b/file-watcher.c\nindex d0762e6..5867942 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -260,6 +260,131 @@ static int watch_path(struct repository *repo, char *path)\n \treturn 0;\n }\n \n+static inline void queue_file_changed(struct file *f, struct strbuf *sb)\n+{\n+\tint len = sb->len;\n+\tstrbuf_addf(sb, \"%s%s\", f->parent->parent ? \"/\" : \"\", f->name);\n+\tstring_list_append(&f->repo->updated, sb->buf);\n+\tf->repo->updated_sorted = 0;\n+\tstrbuf_setlen(sb, len);\n+}\n+\n+static void construct_path(struct dir *d, struct strbuf *sb)\n+{\n+\tif (!d->parent)\n+\t\treturn;\n+\tif (!d->parent->parent) {\n+\t\tstrbuf_addstr(sb, d->name);\n+\t\treturn;\n+\t}\n+\tconstruct_path(d->parent, sb);\n+\tstrbuf_addf(sb, \"/%s\", d->name);\n+}\n+\n+static void file_changed(const struct inotify_event *event,\n+\t\t\t struct dir *d, int pos)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconstruct_path(d, &sb);\n+\tqueue_file_changed(d->files[pos], &sb);\n+\tstrbuf_release(&sb);\n+\tfree_file(d, pos, 0);\n+}\n+\n+static void dir_changed(const struct inotify_event *event, struct dir *d,\n+\t\t\tconst char *base)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint i;\n+\n+\tif (!base)\t\t/* top call -> base == NULL */\n+\t\tconstruct_path(d, &sb);\n+\telse {\n+\t\tstrbuf_addstr(&sb, base);\n+\t\tif (sb.len)\n+\t\t\tstrbuf_addch(&sb, '/');\n+\t\tstrbuf_addstr(&sb, d->name);\n+\t}\n+\n+\tfor (i = 0; i < d->nr_files; i++)\n+\t\tqueue_file_changed(d->files[i], &sb);\n+\tfor (i = 0; i < d->nr_subdirs; i++) {\n+\t\tdir_changed(event, d->subdirs[i], sb.buf);\n+\t\tif (!base)\n+\t\t\tfree_dir(d->subdirs[i], 1);\n+\t}\n+\tstrbuf_release(&sb);\n+\tif (!base)\n+\t\tfree_dir(d, 0);\n+}\n+\n+static void reset_repo(struct repository *repo, ino_t inode);\n+static int do_handle_inotify(const struct inotify_event *event)\n+{\n+\tstruct dir *d;\n+\tint pos;\n+\n+\tif (event->mask & (IN_Q_OVERFLOW | IN_UNMOUNT)) {\n+\t\tint i;\n+\t\tfor (i = 0; i < nr_repos; i++)\n+\t\t\treset_repo(repos[i], 0);\n+\t\treturn 0;\n+\t}\n+\n+\tif ((event->mask & IN_IGNORED) ||\n+\t /*\n+\t * Perhaps left over events that we have not consumed\n+\t * before the watch descriptor is removed.\n+\t */\n+\t event->wd >= wds_alloc || wds[event->wd] == NULL)\n+\t\treturn 0;\n+\n+\td = wds[event->wd];\n+\n+\t/*\n+\t * If something happened to the watched directory, consider\n+\t * everything inside modified\n+\t */\n+\tif (event->mask & (IN_DELETE_SELF | IN_MOVE_SELF)) {\n+\t\tdir_changed(event, d, NULL);\n+\t\treturn 0;\n+\t}\n+\n+\tif (!(event->mask & IN_ISDIR)) {\n+\t\tpos = get_file_pos(d, event->name);\n+\t\tif (pos >= 0)\n+\t\t\tfile_changed(event, d, pos);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int handle_inotify(int fd)\n+{\n+\tstatic char *buf;\n+\tstatic unsigned int buf_len = 0;\n+\tunsigned int avail, offset;\n+\tint ret, len;\n+\n+\t/* drain the event queue */\n+\tif (ioctl(fd, FIONREAD, &avail))\n+\t\tdie_errno(\"unable to FIONREAD inotify handle\");\n+\tif (buf_len < avail) {\n+\t\tbuf = xrealloc(buf, avail);\n+\t\tbuf_len = avail;\n+\t}\n+\tlen = read(fd, buf, avail);\n+\tif (len <= 0)\n+\t\treturn -1;\n+\tret = offset = 0;\n+\twhile (offset < len) {\n+\t\tstruct inotify_event *event = (void *)(buf + offset);\n+\t\tret += do_handle_inotify(event);\n+\t\toffset += sizeof(struct inotify_event) + event->len;\n+\t}\n+\treturn ret;\n+}\n+\n static void get_changed_list(int conn_id)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -466,6 +591,12 @@ static int handle_command(int conn_id)\n \t * capabilities. Capabilities in uppercase MUST be\n \t * supported. If any side does not understand any of the\n \t * advertised uppercase capabilities, it must disconnect.\n+\t *\n+\t * The way the main event loop is structured, we should get at\n+\t * least one handle_inotify() before receiving the next\n+\t * command. And handle_inotify() should process all events by\n+\t * this point of time. This guarantees our reports won't miss\n+\t * anything by the time get-changed is called.\n \t */\n \tif ((arg = skip_prefix(msg, \"hello\"))) {\n \t\tif (*arg) {\t/* no capabilities supported yet */\n@@ -753,11 +884,15 @@ int main(int argc, const char **argv)\n \t\tclose(err);\n \t}\n \n-\tnr_common = 1;\n+\tnr_common = 1 + !!inotify_fd;\n \tpfd_alloc = pfd_nr = nr_common;\n \tpfd = xmalloc(sizeof(*pfd) * pfd_alloc);\n \tpfd[0].fd = fd;\n \tpfd[0].events = POLLIN;\n+\tif (inotify_fd) {\n+\t\tpfd[1].fd = inotify_fd;\n+\t\tpfd[1].events = POLLIN;\n+\t}\n \n \twhile (!quit) {\n \t\tif (poll(pfd, pfd_nr, -1) < 0) {\n@@ -769,6 +904,11 @@ int main(int argc, const char **argv)\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (inotify_fd && (pfd[1].revents & POLLIN)) {\n+\t\t\tif (handle_inotify(inotify_fd))\n+\t\t\t\tbreak;\n+\t\t}\n+\n \t\tfor (new_nr = i = nr_common; i < pfd_nr; i++) {\n \t\t\tif (pfd[i].revents & (POLLERR | POLLNVAL))\n \t\t\t\tshutdown_connection(i);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234090","messageId":"1391401754-15347-20-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 19/26] Wrap CE_VALID test with ce_valid()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:07Z","receivedAt":"2014-02-03T04:29:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The next patch wants to ignore CE_VALID under some condition but not\nreally clears it. Centralizing its access makes such a change easier.\n\nNot all \"ce_flags & CE_VALID\" is converted though. The tests that\nreally mean _bit_ CE_VALID remains so. The tests that mean \"ignore\nworktree\" are converted.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/grep.c | 2 +-\n cache.h | 4 ++++\n diff-lib.c | 4 ++--\n diff.c | 2 +-\n read-cache.c | 6 +++---\n unpack-trees.c | 2 +-\n 6 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 63f8603..00526a1 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -386,7 +386,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int\n \t\t * are identical, even if worktree file has been modified, so use\n \t\t * cache version instead\n \t\t */\n-\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n+\t\tif (cached || ce_valid(ce) || ce_skip_worktree(ce)) {\n \t\t\tif (ce_stage(ce))\n \t\t\t\tcontinue;\n \t\t\thit |= grep_sha1(opt, ce->sha1, ce->name, 0, ce->name);\ndiff --git a/cache.h b/cache.h\nindex 9f7d952..c229bf9 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -222,6 +222,10 @@ static inline unsigned create_ce_flags(unsigned stage)\n #define ce_uptodate(ce) ((ce)->ce_flags & CE_UPTODATE)\n #define ce_skip_worktree(ce) ((ce)->ce_flags & CE_SKIP_WORKTREE)\n #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)\n+static inline int ce_valid(const struct cache_entry *ce)\n+{\n+\treturn ce->ce_flags & CE_VALID;\n+}\n \n #define ce_permissions(mode) (((mode) & 0100) ? 0755 : 0644)\n static inline unsigned int create_ce_mode(unsigned int mode)\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 346cac6..dcda7f3 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -198,7 +198,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tcontinue;\n \n \t\t/* If CE_VALID is set, don't look at workdir for file removal */\n-\t\tchanged = (ce->ce_flags & CE_VALID) ? 0 : check_removed(ce, &st);\n+\t\tchanged = ce_valid(ce) ? 0 : check_removed(ce, &st);\n \t\tif (changed) {\n \t\t\tif (changed < 0) {\n \t\t\t\tperror(ce->name);\n@@ -369,7 +369,7 @@ static void do_oneway_diff(struct unpack_trees_options *o,\n \n \t/* if the entry is not checked out, don't examine work tree */\n \tcached = o->index_only ||\n-\t\t(idx && ((idx->ce_flags & CE_VALID) || ce_skip_worktree(idx)));\n+\t\t(idx && (ce_valid(idx) || ce_skip_worktree(idx)));\n \t/*\n \t * Backward compatibility wart - \"diff-index -m\" does\n \t * not mean \"do not ignore merges\", but \"match_missing\".\ndiff --git a/diff.c b/diff.c\nindex 8e4a6a9..22c73fe 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -2636,7 +2636,7 @@ static int reuse_worktree_file(const char *name, const unsigned char *sha1, int\n \t * If ce is marked as \"assume unchanged\", there is no\n \t * guarantee that work tree matches what we are looking for.\n \t */\n-\tif ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce))\n+\tif (ce_valid(ce) || ce_skip_worktree(ce))\n \t\treturn 0;\n \n \t/*\ndiff --git a/read-cache.c b/read-cache.c\nindex 5540b06..95c9ccb 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -289,8 +289,8 @@ int ie_match_stat(const struct index_state *istate,\n \t */\n \tif (!ignore_skip_worktree && ce_skip_worktree(ce))\n \t\treturn 0;\n-\tif (!ignore_valid && (ce->ce_flags & CE_VALID))\n-\t\treturn 0;\n+\tif (!ignore_valid && ce_valid(ce))\n+\t\t\treturn 0;\n \n \t/*\n \t * Intent-to-add entries have not been added, so the index entry\n@@ -1047,7 +1047,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\tce_mark_uptodate(ce);\n \t\treturn ce;\n \t}\n-\tif (!ignore_valid && (ce->ce_flags & CE_VALID)) {\n+\tif (!ignore_valid && ce_valid(ce)) {\n \t\tce_mark_uptodate(ce);\n \t\treturn ce;\n \t}\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 164354d..61c3f35 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1215,7 +1215,7 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n \t * if this entry is truly up-to-date because this file may be\n \t * overwritten.\n \t */\n-\tif ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce))\n+\tif (ce_valid(ce) || ce_skip_worktree(ce))\n \t\t; /* keep checking */\n \telse if (o->reset || ce_uptodate(ce))\n \t\treturn 0;\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234091","messageId":"1391401754-15347-21-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 20/26] read-cache: new variable to verify file-watcher results","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:08Z","receivedAt":"2014-02-03T04:29:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If GIT_TEST_WATCHED is set to a non-zero value, Git still uses file\nwatcher if configured. But it does lstat() anyway and notifies the\nuser if a file is changed but the file watcher said otherwise.\n\nNote that there is a race condition. Changed paths are retrieved at\ntime X, then refresh and validation at time Y. Even if X and Y are\nvery close, an update can happen between X and Y, causing a false\nreport.\n\nIf GIT_TEST_WATCHED is set greater than 1, git will abort instead of\njust warn and move on.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-file-watcher.txt | 8 ++++++++\n cache.h | 5 ++++-\n read-cache.c | 17 +++++++++++++++++\n 3 files changed, 29 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-file-watcher.txt b/Documentation/git-file-watcher.txt\nindex d694fea..dd09e30 100644\n--- a/Documentation/git-file-watcher.txt\n+++ b/Documentation/git-file-watcher.txt\n@@ -25,6 +25,14 @@ OPTIONS\n --detach::\n \tRun in background.\n \n+TROUBLESHOOTING\n+---------------\n+Setting environment variable `GIT_TEST_WATCHED` to a non-zero number\n+makes Git communicate with the file watcher, but do lstat anyway to\n+verify that the file watcher results. Setting to 1 prints warning when\n+file watcher fails to monitor files correctly. Setting to 2 aborts Git\n+when it happens.\n+\n BUGS\n ----\n On Linux, file watcher may fail to detect changes if you move the work\ndiff --git a/cache.h b/cache.h\nindex c229bf9..806c886 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -224,7 +224,10 @@ static inline unsigned create_ce_flags(unsigned stage)\n #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)\n static inline int ce_valid(const struct cache_entry *ce)\n {\n-\treturn ce->ce_flags & CE_VALID;\n+\textern int test_watched;\n+\tif (!test_watched)\n+\t\treturn ce->ce_flags & CE_VALID;\n+\treturn (ce->ce_flags & CE_VALID) && !(ce->ce_flags & CE_WATCHED);\n }\n \n #define ce_permissions(mode) (((mode) & 0100) ? 0755 : 0644)\ndiff --git a/read-cache.c b/read-cache.c\nindex 95c9ccb..d5f084a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -37,6 +37,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n #define CACHE_EXT_WATCH 0x57415443\t /* \"WATC\" */\n \n struct index_state the_index;\n+int test_watched;\n \n static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n {\n@@ -1117,6 +1118,16 @@ static void show_file(const char * fmt, const char * name, int in_porcelain,\n \tprintf(fmt, name);\n }\n \n+static void report_bad_watcher(struct index_state *istate,\n+\t\t\t struct cache_entry *ce)\n+{\n+\tif (test_watched > 1)\n+\t\tdie(\"%s is updated but file-watcher said no\",\n+\t\t ce->name);\n+\twarning(\"%s is updated but file-watcher said no\",\n+\t\tce->name);\n+}\n+\n int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t const struct pathspec *pathspec,\n \t\t char *seen, const char *header_msg)\n@@ -1188,6 +1199,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\t\tce->ce_flags &= ~CE_VALID;\n \t\t\t\tistate->cache_changed = 1;\n \t\t\t}\n+\t\t\tif (test_watched &&\n+\t\t\t (ce->ce_flags & CE_WATCHED) && (ce->ce_flags & CE_VALID))\n+\t\t\t\treport_bad_watcher(istate, ce);\n \t\t\tif (quiet)\n \t\t\t\tcontinue;\n \n@@ -1460,6 +1474,9 @@ int read_index_from(struct index_state *istate, const char *path)\n \tif (istate->initialized)\n \t\treturn istate->cache_nr;\n \n+\tif (getenv(\"GIT_TEST_WATCHED\"))\n+\t\ttest_watched = atoi(getenv(\"GIT_TEST_WATCHED\"));\n+\n \tistate->timestamp.sec = 0;\n \tistate->timestamp.nsec = 0;\n \tfd = open(path, O_RDONLY);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234092","messageId":"1391401754-15347-22-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 21/26] Support running file watcher with the test suite","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:09Z","receivedAt":"2014-02-03T04:29:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is to force running the test suite with file-watcher with\n\n $ mkdir /tmp/watcher\n $ chmod 700 /tmp/watcher\n $ git-file-watcher /tmp/watcher/\n\nthen open another terminal and run\n\n $ export GIT_TEST_WATCHED=2 GIT_TEST_WATCHER=2\n $ export GIT_TEST_WATCHER_PATH=/tmp/watcher\n $ make test\n\nTIME_WAIT set set to unlimited by GIT_TEST_WATCHER=2 so the test suite\ncould hang up indefinitely due to a file-watcher bug. Luckily\neverything passes.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher-lib.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/file-watcher-lib.c b/file-watcher-lib.c\nindex b6b0848..93afb52 100644\n--- a/file-watcher-lib.c\n+++ b/file-watcher-lib.c\n@@ -136,6 +136,7 @@ void open_watcher(struct index_state *istate)\n \t}\n \n \tif (!read_config) {\n+\t\tconst char *s;\n \t\tint i;\n \t\t/*\n \t\t * can't hook into git_default_config because\n@@ -149,6 +150,18 @@ void open_watcher(struct index_state *istate)\n \t\tif (i == istate->cache_nr)\n \t\t\trecent_limit = 0;\n \t\tread_config = 1;\n+\n+\t\ts = getenv(\"GIT_TEST_WATCHER\");\n+\t\tif (s) {\n+\t\t\twatch_lowerlimit = 1;\n+\t\t\trecent_limit = 0;\n+\t\t\tWAIT_TIME = -1;\n+\t\t\tif (atoi(s) > 1)\n+\t\t\t\tistate->update_watches = 1;\n+\t\t\ts = getenv(\"GIT_TEST_WATCHER_PATH\");\n+\t\t\tif (s)\n+\t\t\t\twatcher_path = xstrdup(s);\n+\t\t}\n \t}\n \n \tistate->watcher = connect_watcher(watcher_path);\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234093","messageId":"1391401754-15347-23-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 22/26] file-watcher: quit if $WATCHER/socket is gone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:10Z","receivedAt":"2014-02-03T04:29:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is more of an issue in development than in production. When a\nfile-watcher related test fails, the daemon may be left hanging. When\nyou rerun the same test, old $TRASH_DIRECTORY is wiped out and no one\ncan communicate with the old daemon any more. Make the old daemon quit\nafter 5 minutes in such cases.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher.c | 26 ++++++++++++++++++++++++--\n 1 file changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 5867942..1e45b25 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -834,11 +834,22 @@ static void check_socket_directory(const char *path)\n \tfree(path_copy);\n }\n \n+static void run_housekeeping(void)\n+{\n+\tstruct stat st;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstrbuf_addf(&sb, \"%s/socket\", socket_path);\n+\tif (stat(sb.buf, &st) || !S_ISSOCK(st.st_mode))\n+\t\texit(0);\n+\tstrbuf_release(&sb);\n+}\n+\n int main(int argc, const char **argv)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint i, new_nr, fd, quit = 0, nr_common;\n \tint daemon = 0;\n+\ttime_t last_checked;\n \tstruct option options[] = {\n \t\tOPT_BOOL(0, \"detach\", &daemon,\n \t\t\t N_(\"run in background\")),\n@@ -894,19 +905,25 @@ int main(int argc, const char **argv)\n \t\tpfd[1].events = POLLIN;\n \t}\n \n+\tlast_checked = time(NULL);\n \twhile (!quit) {\n-\t\tif (poll(pfd, pfd_nr, -1) < 0) {\n+\t\tint ret = poll(pfd, pfd_nr, 300000);\n+\t\tint time_for_housekeeping = 0;\n+\t\tif (ret < 0) {\n \t\t\tif (errno != EINTR) {\n \t\t\t\terror(\"Poll failed, resuming: %s\",\n \t\t\t\t strerror(errno));\n \t\t\t\tsleep(1);\n \t\t\t}\n \t\t\tcontinue;\n-\t\t}\n+\t\t} else if (ret == 0)\n+\t\t\ttime_for_housekeeping = 1;\n \n \t\tif (inotify_fd && (pfd[1].revents & POLLIN)) {\n \t\t\tif (handle_inotify(inotify_fd))\n \t\t\t\tbreak;\n+\t\t\tif (last_checked + 300 < time(NULL))\n+\t\t\t\ttime_for_housekeeping = 1;\n \t\t}\n \n \t\tfor (new_nr = i = nr_common; i < pfd_nr; i++) {\n@@ -949,6 +966,11 @@ int main(int argc, const char **argv)\n \t\t\taccept_connection(pfd[0].fd);\n \t\tif (pfd[0].revents & (POLLHUP | POLLERR | POLLNVAL))\n \t\t\tdie(_(\"error on listening socket\"));\n+\n+\t\tif (time_for_housekeeping) {\n+\t\t\trun_housekeeping();\n+\t\t\tlast_checked = time(NULL);\n+\t\t}\n \t}\n \treturn 0;\n }\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234094","messageId":"1391401754-15347-24-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 23/26] file-watcher: tests for the daemon","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:11Z","receivedAt":"2014-02-03T04:29:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"test-file-watcher is a simple chat program to talk to file\nwatcher. Typically you would write something like this\n\n cat >expect <'\n >hello\n ok\n EOF\n test-file-watcher . actual\n\nand test-file-watcher will execute the commands and get responses. If\nall go well, \"actual\" should be the same as \"expect\". '<' and '>'\ndenote send and receive packets respectively. '<<' and '>>' can be\nused to send and receive a list of NUL-terminated paths.\n\n$GIT_TEST_WATCHER enables a few more commands for testing purposes.\nThe most important one is 'test-mode' where system inotify is taken\nout and inotify events could be injected via test-file-watcher.\n\nThere are two debug commands in file-watcher that's not used by the\ntest suite, but would help debugging: setenv and log. They can be used\nto turn on GIT_TRACE_PACKET then any \"log\" command will show, which\nfunctions as barrier between events file watcher.\n\nGIT_TRACE_WATCHER can also be enabled (dynamically or at startup) to\ntrack inotify events.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n .gitignore | 1 +\n Makefile | 1 +\n file-watcher.c | 181 ++++++++++++++++++-\n t/t7513-file-watcher.sh (new +x) | 382 +++++++++++++++++++++++++++++++++++++++\n test-file-watcher.c (new) | 96 ++++++++++\n 5 files changed, 657 insertions(+), 4 deletions(-)\n create mode 100755 t/t7513-file-watcher.sh\n create mode 100644 test-file-watcher.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 12c78f0..277f929 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -181,6 +181,7 @@\n /test-date\n /test-delta\n /test-dump-cache-tree\n+/test-file-watcher\n /test-scrap-cache-tree\n /test-genrandom\n /test-index-version\ndiff --git a/Makefile b/Makefile\nindex 1c4d659..f0dc2cc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -555,6 +555,7 @@ TEST_PROGRAMS_NEED_X += test-ctype\n TEST_PROGRAMS_NEED_X += test-date\n TEST_PROGRAMS_NEED_X += test-delta\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n+TEST_PROGRAMS_NEED_X += test-file-watcher\n TEST_PROGRAMS_NEED_X += test-genrandom\n TEST_PROGRAMS_NEED_X += test-index-version\n TEST_PROGRAMS_NEED_X += test-line-buffer\ndiff --git a/file-watcher.c b/file-watcher.c\nindex 1e45b25..3ab0a11 100644\n--- a/file-watcher.c\n+++ b/file-watcher.c\n@@ -65,7 +65,8 @@ struct connection {\n static struct connection **conns;\n static struct pollfd *pfd;\n static int conns_alloc, pfd_nr, pfd_alloc;\n-static int inotify_fd;\n+static int inotify_fd, test_mode;\n+static int wd_counter = 1;\n \n /*\n * IN_DONT_FOLLOW does not matter now as we do not monitor\n@@ -78,10 +79,19 @@ static struct dir *create_dir(struct dir *parent, const char *path,\n \t\t\t const char *basename)\n {\n \tstruct dir *d;\n-\tint wd = inotify_add_watch(inotify_fd, path, INOTIFY_MASKS);\n+\tint wd;\n+\tif (!test_mode)\n+\t\twd = inotify_add_watch(inotify_fd, path, INOTIFY_MASKS);\n+\telse {\n+\t\twd = wd_counter++;\n+\t\tif (wd > 8)\n+\t\t\twd = -1;\n+\t}\n \tif (wd < 0)\n \t\treturn NULL;\n \n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"inotify: watch %d %s\\n\",\n+\t\t\t wd, path);\n \td = xmalloc(sizeof(*d));\n \tmemset(d, 0, sizeof(*d));\n \td->wd = wd;\n@@ -124,7 +134,9 @@ static void free_dir(struct dir *d, int topdown)\n \tif (d->repo)\n \t\td->repo->root = NULL;\n \twds[d->wd] = NULL;\n-\tinotify_rm_watch(inotify_fd, d->wd);\n+\tif (!test_mode)\n+\t\tinotify_rm_watch(inotify_fd, d->wd);\n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"inotify: unwatch %d\\n\", d->wd);\n \tif (topdown) {\n \t\tint i;\n \t\tfor (i = 0; i < d->nr_subdirs; i++)\n@@ -265,6 +277,7 @@ static inline void queue_file_changed(struct file *f, struct strbuf *sb)\n \tint len = sb->len;\n \tstrbuf_addf(sb, \"%s%s\", f->parent->parent ? \"/\" : \"\", f->name);\n \tstring_list_append(&f->repo->updated, sb->buf);\n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"watcher: changed %s\\n\", sb->buf);\n \tf->repo->updated_sorted = 0;\n \tstrbuf_setlen(sb, len);\n }\n@@ -324,6 +337,10 @@ static int do_handle_inotify(const struct inotify_event *event)\n \tstruct dir *d;\n \tint pos;\n \n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"inotify: event %08x wd %d %s\\n\",\n+\t\t\t event->mask, event->wd,\n+\t\t\t event->len ? event->name : \"N/A\");\n+\n \tif (event->mask & (IN_Q_OVERFLOW | IN_UNMOUNT)) {\n \t\tint i;\n \t\tfor (i = 0; i < nr_repos; i++)\n@@ -385,6 +402,81 @@ static int handle_inotify(int fd)\n \treturn ret;\n }\n \n+struct constant {\n+\tconst char *name;\n+\tint value;\n+};\n+\n+#define CONSTANT(x) { #x, x }\n+static const struct constant inotify_masks[] = {\n+\tCONSTANT(IN_DELETE_SELF),\n+\tCONSTANT(IN_MOVE_SELF),\n+\tCONSTANT(IN_ATTRIB),\n+\tCONSTANT(IN_DELETE),\n+\tCONSTANT(IN_MODIFY),\n+\tCONSTANT(IN_MOVED_FROM),\n+\tCONSTANT(IN_MOVED_TO),\n+\tCONSTANT(IN_Q_OVERFLOW),\n+\tCONSTANT(IN_UNMOUNT),\n+\t{ NULL, 0 },\n+};\n+\n+static void inject_inotify(const char *msg)\n+{\n+\tchar buf[sizeof(struct inotify_event) + NAME_MAX + 1];\n+\tstruct inotify_event *event = (struct inotify_event *)buf;\n+\tchar *end, *p;\n+\tint i;\n+\tmemset(event, 0, sizeof(*event));\n+\tevent->wd = strtol(msg, &end, 0);\n+\tif (*end++ != ' ')\n+\t\tdie(\"expect a space after watch descriptor\");\n+\tp = end;\n+\tend = strchrnul(p, ' ');\n+\tif (*end)\n+\t\tstrcpy(event->name, end + 1);\n+\twhile (p < end) {\n+\t\tchar *sep = strchrnul(p, '|');\n+\t\tif (sep > end)\n+\t\t\tsep = end;\n+\t\t*sep = '\\0';\n+\t\tfor (i = 0; inotify_masks[i].name; i++)\n+\t\t\tif (!strcmp(inotify_masks[i].name, p))\n+\t\t\t\tbreak;\n+\t\tif (!inotify_masks[i].name)\n+\t\t\tdie(\"unrecognize event mask %s\", p);\n+\t\tevent->mask |= inotify_masks[i].value;\n+\t\tp = sep + 1;\n+\t}\n+\tdo_handle_inotify(event);\n+}\n+\n+static void dump_watches(struct dir *d, struct strbuf *sb, struct strbuf *out)\n+{\n+\tint i, len = sb->len;\n+\tstrbuf_addstr(sb, d->name);\n+\tstrbuf_addf(out, \"%s %d%c\", sb->buf[0] ? sb->buf : \".\", d->wd, '\\0');\n+\tif (d->name[0])\n+\t\tstrbuf_addch(sb, '/');\n+\tfor (i = 0; i < d->nr_subdirs; i++)\n+\t\tdump_watches(d->subdirs[i], sb, out);\n+\tfor (i = 0; i < d->nr_files; i++)\n+\t\tstrbuf_addf(out, \"%s%s%c\", sb->buf, d->files[i]->name, '\\0');\n+\tstrbuf_setlen(sb, len);\n+}\n+\n+static void dump_changes(struct repository *repo, struct strbuf *sb)\n+{\n+\tint i;\n+\tif (!repo->updated_sorted) {\n+\t\tsort_string_list(&repo->updated);\n+\t\trepo->updated_sorted = 1;\n+\t}\n+\tfor (i = 0; i < repo->updated.nr; i++)\n+\t\tstrbuf_add(sb, repo->updated.items[i].string,\n+\t\t\t strlen(repo->updated.items[i].string) + 1);\n+}\n+\n static void get_changed_list(int conn_id)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -483,11 +575,13 @@ static void unchange(int conn_id, unsigned long size)\n \t\t\titem = string_list_lookup(&repo->updated, p);\n \t\t\tif (!item)\n \t\t\t\tcontinue;\n+\t\t\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"watcher: unchange %s\\n\", p);\n \t\t\tunsorted_string_list_delete_item(&repo->updated,\n \t\t\t\t\t\t\t item - repo->updated.items, 0);\n \t\t}\n \t\tstrbuf_release(&sb);\n \t}\n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"watcher: unchange complete\\n\");\n \tmemcpy(repo->index_signature, conn->new_index, 40);\n \t/*\n \t * If other connections on this repo are in some sort of\n@@ -540,6 +634,13 @@ static void reset_watches(struct repository *repo)\n \n static void reset_repo(struct repository *repo, ino_t inode)\n {\n+\tif (test_mode)\n+\t\t/*\n+\t\t * test-mode is designed for single repo, we can\n+\t\t * safely reset wd counter because all wd should be\n+\t\t * deleted\n+\t\t */\n+\t\twd_counter = 1;\n \treset_watches(repo);\n \tstring_list_clear(&repo->updated, 0);\n \tmemcpy(repo->index_signature, invalid_signature, 40);\n@@ -560,6 +661,7 @@ static int shutdown_connection(int id)\n \treturn 0;\n }\n \n+static void cleanup(void);\n static int handle_command(int conn_id)\n {\n \tint fd = conns[conn_id]->sock;\n@@ -754,6 +856,71 @@ static int handle_command(int conn_id)\n \t\t}\n \t\tunchange(conn_id, n);\n \t}\n+\n+\t/*\n+\t * Testing and debugging support\n+\t */\n+\telse if (!strcmp(msg, \"test-mode\") && getenv(\"GIT_TEST_WATCHER\")) {\n+\t\ttest_mode = 1;\n+\t\tpacket_write(fd, \"test mode on\");\n+\t}\n+\telse if (starts_with(msg, \"setenv \")) {\n+\t\t/* useful for setting GIT_TRACE_WATCHER or GIT_TRACE_PACKET */\n+\t\tchar *sep = strchr(msg + 7, ' ');\n+\t\tif (!sep) {\n+\t\t\tpacket_write(fd, \"error invalid setenv line %s\", msg);\n+\t\t\treturn shutdown_connection(conn_id);\n+\t\t}\n+\t\t*sep = '\\0';\n+\t\tsetenv(msg + 7, sep + 1, 1);\n+\t}\n+\telse if (starts_with(msg, \"log \")) {\n+\t\t; /* do nothing, if GIT_TRACE_PACKET is on, it's already logged */\n+\t}\n+\telse if (!strcmp(msg, \"die\") && getenv(\"GIT_TEST_WATCHER\")) {\n+\t\t/*\n+\t\t * The client will wait for \"see you\" before it may\n+\t\t * run another daemon with the same path. So there's\n+\t\t * no racing on unlink() and listen() on the same\n+\t\t * socket path.\n+\t\t */\n+\t\tcleanup();\n+\t\tpacket_write(fd, \"see you\");\n+\t\tclose(fd);\n+\t\texit(0);\n+\t}\n+\telse if (starts_with(msg, \"dump \") && getenv(\"GIT_TEST_WATCHER\")) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tstruct strbuf out = STRBUF_INIT;\n+\t\tconst char *reply = NULL;\n+\t\tif (!strcmp(msg + 5, \"watches\")) {\n+\t\t\tif (conns[conn_id]->repo) {\n+\t\t\t\tif (conns[conn_id]->repo->root)\n+\t\t\t\t\tdump_watches(conns[conn_id]->repo->root, &sb, &out);\n+\t\t\t} else {\n+\t\t\t\tint i;\n+\t\t\t\tfor (i = 0; i < nr_repos; i++) {\n+\t\t\t\t\tstrbuf_addf(&out, \"%s%c\", repos[i]->work_tree, '\\0');\n+\t\t\t\t\tif (repos[i]->root)\n+\t\t\t\t\t\tdump_watches(repos[i]->root, &sb, &out);\n+\t\t\t\t\tstrbuf_reset(&out);\n+\t\t\t\t\tstrbuf_reset(&sb);\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\treply = \"watching\";\n+\t\t} else if (!strcmp(msg + 5, \"changes\")) {\n+\t\t\tdump_changes(conns[conn_id]->repo, &out);\n+\t\t\treply = \"changed\";\n+\t\t}\n+\t\tpacket_write(fd, \"%s %d\", reply, (int)out.len);\n+\t\tif (out.len)\n+\t\t\twrite_in_full(fd, out.buf, out.len);\n+\t\tstrbuf_release(&out);\n+\t\tstrbuf_release(&sb);\n+\t}\n+\telse if (starts_with(msg, \"inotify \") && getenv(\"GIT_TEST_WATCHER\")) {\n+\t\tinject_inotify(msg + 8);\n+\t}\n \telse {\n \t\tpacket_write(fd, \"error unrecognized command %s\", msg);\n \t\treturn shutdown_connection(conn_id);\n@@ -848,11 +1015,13 @@ int main(int argc, const char **argv)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint i, new_nr, fd, quit = 0, nr_common;\n-\tint daemon = 0;\n+\tint daemon = 0, check_support = 0;\n \ttime_t last_checked;\n \tstruct option options[] = {\n \t\tOPT_BOOL(0, \"detach\", &daemon,\n \t\t\t N_(\"run in background\")),\n+\t\tOPT_BOOL(0, \"check-support\", &check_support,\n+\t\t\t N_(\"return zero file watcher is available\")),\n \t\tOPT_END()\n \t};\n \n@@ -865,6 +1034,10 @@ int main(int argc, const char **argv)\n \n \targc = parse_options(argc, argv, NULL, options,\n \t\t\t file_watcher_usage, 0);\n+\n+\tif (check_support)\n+\t\treturn 0;\n+\n \tif (argc < 1)\n \t\tdie(_(\"socket path missing\"));\n \telse if (argc > 1)\ndiff --git a/t/t7513-file-watcher.sh b/t/t7513-file-watcher.sh\nnew file mode 100755\nindex 0000000..bf64fc4\n--- /dev/null\n+++ b/t/t7513-file-watcher.sh\n@@ -0,0 +1,382 @@\n+#!/bin/sh\n+\n+test_description='File watcher daemon tests'\n+\n+. ./test-lib.sh\n+\n+if git file-watcher --check-support && test_have_prereq POSIXPERM; then\n+\t:\t\t\t\t# good\n+else\n+\tskip_all=\"file-watcher not supported on this system\"\n+\ttest_done\n+fi\n+\n+kill_it() {\n+\ttest-file-watcher \"$1\" </dev/null\n+see you\n+EOF\n+}\n+\n+GIT_TEST_WATCHER=1\n+export GIT_TEST_WATCHER\n+\n+test_expect_success 'test-file-watcher can kill the daemon' '\n+\tchmod 700 . &&\n+\tgit file-watcher --detach . &&\n+\tcat >expect <see you\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+see you\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+inconsistent\n+EOF\n+\ttest-file-watcher . >actual expect2 <hello\n+inconsistent\n+actual2 expect3 <hello\n+ok\n+see you\n+EOF\n+\ttest-file-watcher . >actual3 expect <error why did you not greet me? go away\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+error invalid index line index 1234 $TRASH_DIRECTORY\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+error work tree does not exist: No such file or directory\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+inconsistent\n+test mode on\n+<watched 2\n+>watching\n+>>. 1\n+>>abc 2\n+>>abc/bar\n+>>foo\n+actual expect <hello\n+ok\n+>watching\n+>>. 1\n+>>foo\n+>changed\n+>>abc/bar\n+see you\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+inconsistent\n+test mode on\n+<watched 1\n+>watching\n+>>. 1\n+>>foo 2\n+>>foo/abc 3\n+>>foo/abc/bar\n+>watching\n+>changed\n+>>foo/abc/bar\n+actual expect <hello\n+ok\n+>changed\n+>>foo/abc/bar\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+ok\n+>changed\n+>>foo/abc/bar\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+ok\n+>changed\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+ok\n+see you\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+inconsistent\n+# Do not call inotify_add_watch()\n+test mode on\n+# First batch should be all ok\n+<watched 4\n+# Second batch hits the limit\n+<watched 3\n+# The third batch is already registered, should accept too\n+<watched 3\n+# Try again, see if it still rejects\n+<watched 0\n+>watching\n+>>. 1\n+>>dir1 2\n+>>dir1/foo\n+>>dir2 3\n+>>dir2/foo\n+>>dir3 4\n+>>dir3/foo\n+>>dir4 5\n+>>dir4/foo\n+>>dir5 6\n+>>dir5/foo\n+>>dir6 7\n+>>dir6/foo\n+>>dir7 8\n+>>dir7/foo\n+see you\n+EOF\n+\ttest-file-watcher . >actual expect <hello\n+inconsistent\n+test mode on\n+<watched 2\n+>watching\n+>>. 1\n+>>foo\n+>changed\n+>>abc/bar\n+>watching\n+>changed\n+EOF\n+\ttest-file-watcher . >actual expect2 <hello\n+inconsistent\n+see you\n+EOF\n+\ttest-file-watcher . >actual2 ') {\n+\t\t\tif (last_command_is_reply)\n+\t\t\t\tcontinue;\n+\t\t\tlast_command_is_reply = 1;\n+\t\t} else\n+\t\t\tlast_command_is_reply = 0;\n+\n+\t\tif (sb.buf[0] == '<' && sb.buf[1] == '<') {\n+\t\t\tputs(sb.buf);\n+\t\t\tif (!packing) {\n+\t\t\t\tpacking = xstrdup(sb.buf + 2);\n+\t\t\t\tstrbuf_reset(&packed);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!sb.buf[2]) {\n+\t\t\t\tpacket_write(fd, \"%s %d\", packing, (int)packed.len);\n+\t\t\t\tif (packed.len)\n+\t\t\t\t\twrite_in_full(fd, packed.buf, packed.len);\n+\t\t\t\tfree(packing);\n+\t\t\t\tpacking = NULL;\n+\t\t\t} else\n+\t\t\t\tstrbuf_add(&packed, sb.buf + 2, sb.len - 2 + 1);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (sb.buf[0] == '<') {\n+\t\t\tpacket_write(fd, \"%s\", sb.buf + 1);\n+\t\t\tputs(sb.buf);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (sb.buf[0] == '>' && sb.buf[1] == '>') {\n+\t\t\tint len;\n+\t\t\tchar *p, *reply = packet_read_line(fd, &len);\n+\t\t\tif (!starts_with(reply, sb.buf + 2) ||\n+\t\t\t reply[sb.len - 2] != ' ') {\n+\t\t\t\tprintf(\">%s\\n\", reply);\n+\t\t\t\tcontinue;\n+\t\t\t} else {\n+\t\t\t\tp = reply + sb.len - 2;\n+\t\t\t\tprintf(\">>%.*s\\n\", (int)(p - reply), reply);\n+\t\t\t\tlen = atoi(p + 1);\n+\t\t\t\tif (!len)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tstrbuf_reset(&packed);\n+\t\t\tstrbuf_grow(&packed, len);\n+\t\t\tif (read_in_full(fd, packed.buf, len) <= 0)\n+\t\t\t\treturn 1;\n+\t\t\tstrbuf_setlen(&packed, len);\n+\t\t\tfor (p = packed.buf; p - packed.buf < packed.len; p += len + 1) {\n+\t\t\t\tlen = strlen(p);\n+\t\t\t\tprintf(\">>%s\\n\", p);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (sb.buf[0] == '>') {\n+\t\t\tint len;\n+\t\t\tchar *reply = packet_read_line(fd, &len);\n+\t\t\tif (!reply)\n+\t\t\t\tputs(\">\");\n+\t\t\telse\n+\t\t\t\tprintf(\">%s\\n\", reply);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tdie(\"unrecognize command %s\", sb.buf);\n+\t}\n+\treturn 0;\n+}\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234095","messageId":"1391401754-15347-25-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 24/26] ls-files: print CE_WATCHED as W (or \"w\" with CE_VALID)","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:12Z","receivedAt":"2014-02-03T04:29:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-ls-files.txt | 1 +\n builtin/ls-files.c | 14 ++++++++++++--\n 2 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex c0856a6..bdb17a5 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -123,6 +123,7 @@ a space) at the start of each line:\n \tR::\tremoved/deleted\n \tC::\tmodified/changed\n \tK::\tto be killed\n+\tW::\tbeing watched by `git file-watcher`\n \t?::\tother\n \n -v::\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex e1cf6d8..f1f7c07 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -46,6 +46,7 @@ static const char *tag_killed = \"\";\n static const char *tag_modified = \"\";\n static const char *tag_skip_worktree = \"\";\n static const char *tag_resolve_undo = \"\";\n+static const char *tag_watched = \"\";\n \n static void write_name(const char *name)\n {\n@@ -231,6 +232,7 @@ static void show_files(struct dir_struct *dir)\n \tif (show_cached || show_stage) {\n \t\tfor (i = 0; i < active_nr; i++) {\n \t\t\tconst struct cache_entry *ce = active_cache[i];\n+\t\t\tconst char *tag;\n \t\t\tif ((dir->flags & DIR_SHOW_IGNORED) &&\n \t\t\t !ce_excluded(dir, ce))\n \t\t\t\tcontinue;\n@@ -238,8 +240,15 @@ static void show_files(struct dir_struct *dir)\n \t\t\t\tcontinue;\n \t\t\tif (ce->ce_flags & CE_UPDATE)\n \t\t\t\tcontinue;\n-\t\t\tshow_ce_entry(ce_stage(ce) ? tag_unmerged :\n-\t\t\t\t(ce_skip_worktree(ce) ? tag_skip_worktree : tag_cached), ce);\n+\t\t\tif (ce_stage(ce))\n+\t\t\t\ttag = tag_unmerged;\n+\t\t\telse if (ce_skip_worktree(ce))\n+\t\t\t\ttag = tag_skip_worktree;\n+\t\t\telse if (ce->ce_flags & CE_WATCHED)\n+\t\t\t\ttag = tag_watched;\n+\t\t\telse\n+\t\t\t\ttag = tag_cached;\n+\t\t\tshow_ce_entry(tag, ce);\n \t\t}\n \t}\n \tif (show_deleted || show_modified) {\n@@ -530,6 +539,7 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\ttag_killed = \"K \";\n \t\ttag_skip_worktree = \"S \";\n \t\ttag_resolve_undo = \"U \";\n+\t\ttag_watched = \"W \";\n \t}\n \tif (show_modified || show_others || show_deleted || (dir.flags & DIR_SHOW_IGNORED) || show_killed)\n \t\trequire_work_tree = 1;\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234096","messageId":"1391401754-15347-26-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 25/26] file-watcher: tests for the client side","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:13Z","receivedAt":"2014-02-03T04:29:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to t7513, system inotify is taken out to give us better\ncontrolled environment.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n file-watcher-lib.c | 21 ++--\n t/t7514-file-watcher-lib.sh (new +x) | 190 +++++++++++++++++++++++++++++++++++\n test-file-watcher.c | 15 +++\n 3 files changed, 219 insertions(+), 7 deletions(-)\n create mode 100755 t/t7514-file-watcher-lib.sh\n\ndiff --git a/file-watcher-lib.c b/file-watcher-lib.c\nindex 93afb52..8cc3b73 100644\n--- a/file-watcher-lib.c\n+++ b/file-watcher-lib.c\n@@ -23,7 +23,8 @@ static int connect_watcher(const char *path)\n \treturn fd;\n }\n \n-static void reset_watches(struct index_state *istate, int disconnect)\n+static void reset_watches(struct index_state *istate, int disconnect,\n+\t\t\t const char *reason)\n {\n \tint i, changed = 0;\n \tif (istate->updated_entries) {\n@@ -45,6 +46,8 @@ static void reset_watches(struct index_state *istate, int disconnect)\n \t\tclose(istate->watcher);\n \t\tistate->watcher = -1;\n \t}\n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"reset%s: %s\\n\",\n+\t\t\t disconnect ? \"/disconnect\" : \"\", reason);\n }\n \n static void mark_ce_valid(struct index_state *istate)\n@@ -53,15 +56,16 @@ static void mark_ce_valid(struct index_state *istate)\n \tchar *line, *end;\n \tint i, len;\n \tunsigned long n;\n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"mark_ce_valid\\n\");\n \tif (packet_write_timeout(istate->watcher, WAIT_TIME, \"get-changed\") <= 0 ||\n \t !(line = packet_read_line_timeout(istate->watcher, WAIT_TIME, &len)) ||\n \t !starts_with(line, \"changed \")) {\n-\t\treset_watches(istate, 1);\n+\t\treset_watches(istate, 1, \"invalid get-changed response\");\n \t\treturn;\n \t}\n \tn = strtoul(line + 8, &end, 10);\n \tif (end != line + len) {\n-\t\treset_watches(istate, 1);\n+\t\treset_watches(istate, 1, \"invalid get-changed response\");\n \t\treturn;\n \t}\n \tif (!n)\n@@ -69,7 +73,7 @@ static void mark_ce_valid(struct index_state *istate)\n \tstrbuf_grow(&sb, n);\n \tif (read_in_full_timeout(istate->watcher, sb.buf, n, WAIT_TIME) != n) {\n \t\tstrbuf_release(&sb);\n-\t\treset_watches(istate, 1);\n+\t\treset_watches(istate, 1, \"invalid get-changed payload\");\n \t\treturn;\n \t}\n \tline = sb.buf;\n@@ -131,7 +135,7 @@ void open_watcher(struct index_state *istate)\n \tchar *msg;\n \n \tif (!get_git_work_tree()) {\n-\t\treset_watches(istate, 1);\n+\t\treset_watches(istate, 1, \"no worktree\");\n \t\treturn;\n \t}\n \n@@ -165,10 +169,11 @@ void open_watcher(struct index_state *istate)\n \t}\n \n \tistate->watcher = connect_watcher(watcher_path);\n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"open watcher %d\\n\", istate->watcher);\n \tif (packet_write_timeout(istate->watcher, WAIT_TIME, \"hello\") <= 0 ||\n \t (msg = packet_read_line_timeout(istate->watcher, WAIT_TIME, NULL)) == NULL ||\n \t strcmp(msg, \"hello\")) {\n-\t\treset_watches(istate, 1);\n+\t\treset_watches(istate, 1, \"invalid hello response\");\n \t\treturn;\n \t}\n \n@@ -177,7 +182,7 @@ void open_watcher(struct index_state *istate)\n \t\t\t\t get_git_work_tree()) <= 0 ||\n \t (msg = packet_read_line_timeout(istate->watcher, WAIT_TIME, NULL)) == NULL ||\n \t strcmp(msg, \"ok\")) {\n-\t\treset_watches(istate, 0);\n+\t\treset_watches(istate, 0, \"inconsistent\");\n \t\tistate->update_watches = 1;\n \t\treturn;\n \t}\n@@ -265,6 +270,7 @@ void watch_entries(struct index_state *istate)\n \t\t\tnr++;\n \tif (nr < watch_lowerlimit)\n \t\treturn;\n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"watch %d\\n\", nr);\n \tsorted = xmalloc(sizeof(*sorted) * nr);\n \tfor (i = nr = 0; i < istate->cache_nr; i++)\n \t\tif (ce_watchable(istate->cache[i], now))\n@@ -280,6 +286,7 @@ void close_watcher(struct index_state *istate, const unsigned char *sha1)\n \tint len, i, nr;\n \tif (istate->watcher <= 0)\n \t\treturn;\n+\ttrace_printf_key(\"GIT_TRACE_WATCHER\", \"close watcher\\n\");\n \tif (packet_write_timeout(istate->watcher, WAIT_TIME,\n \t\t\t\t \"new-index %s\", sha1_to_hex(sha1)) <= 0)\n \t\tgoto done;\ndiff --git a/t/t7514-file-watcher-lib.sh b/t/t7514-file-watcher-lib.sh\nnew file mode 100755\nindex 0000000..8dabb13\n--- /dev/null\n+++ b/t/t7514-file-watcher-lib.sh\n@@ -0,0 +1,190 @@\n+#!/bin/sh\n+\n+test_description='File watcher daemon and client tests'\n+\n+. ./test-lib.sh\n+\n+if git file-watcher --check-support && test_have_prereq POSIXPERM; then\n+\t: # good\n+else\n+\tskip_all=\"file-watcher not supported on this system\"\n+\ttest_done\n+fi\n+\n+kill_it() {\n+\ttest-file-watcher \"$1\" </dev/null\n+see you\n+EOF\n+}\n+\n+GIT_TEST_WATCHER=2\n+GIT_TEST_WATCHER_PATH=\"$TRASH_DIRECTORY\"\n+export GIT_TEST_WATCHER GIT_TEST_WATCHER_PATH\n+\n+test_expect_success 'setup' '\n+\tchmod 700 . &&\n+\tmkdir foo bar &&\n+\ttouch abc foo/def bar/ghi &&\n+\tgit add . &&\n+\tgit file-watcher --detach . &&\n+\tcat <expect &&\n+test mode on\n+EOF\n+\ttest-file-watcher . actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'initial opening sequence' '\n+\tSIG=`test-file-watcher --index-signature .git/index` &&\n+\trm actual &&\n+\tGIT_TRACE_PACKET=\"$TRASH_DIRECTORY/actual\" git ls-files >/dev/null &&\n+\tcat <expect &&\n+packet: git> hello\n+packet: git< hello\n+packet: git> index $SIG $TRASH_DIRECTORY\n+packet: git< inconsistent\n+packet: git> watch 20\n+packet: git< watched 3\n+EOF\n+\ttest_cmp expect actual &&\n+\n+\t# second time gives the same result because file-watch has not\n+\t# received new-index\n+\tGIT_TRACE_PACKET=\"$TRASH_DIRECTORY/actual2\" git ls-files >/dev/null &&\n+\ttest_cmp expect actual2\n+'\n+\n+test_expect_success 'full sequence' '\n+\tSIG=`test-file-watcher --index-signature .git/index` &&\n+\trm actual &&\n+\tGIT_TRACE_PACKET=\"$TRASH_DIRECTORY/actual\" git status >/dev/null &&\n+\tSIG2=`test-file-watcher --index-signature .git/index` &&\n+\tcat <expect &&\n+packet: git> hello\n+packet: git< hello\n+packet: git> index $SIG $TRASH_DIRECTORY\n+packet: git< inconsistent\n+packet: git> watch 20\n+packet: git< watched 3\n+packet: git> new-index $SIG2\n+packet: git> unchange 0\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'full sequence after file-watcher is active' '\n+\tSIG=`test-file-watcher --index-signature .git/index` &&\n+\trm actual &&\n+\tGIT_TRACE_PACKET=\"$TRASH_DIRECTORY/actual\" git ls-files -v >paths.actual &&\n+\tcat <expect &&\n+packet: git> hello\n+packet: git< hello\n+packet: git> index $SIG $TRASH_DIRECTORY\n+packet: git< ok\n+packet: git> get-changed\n+packet: git< changed 0\n+EOF\n+\ttest_cmp expect actual &&\n+\tcat <paths.expect &&\n+w abc\n+w bar/ghi\n+w foo/def\n+EOF\n+\ttest_cmp paths.expect paths.actual\n+'\n+\n+test_expect_success 'inject a file change' '\n+\techo modified >bar/ghi &&\n+\tSIG=`test-file-watcher --index-signature .git/index` &&\n+\tcat >expect <hello\n+ok\n+>changed\n+>>bar/ghi\n+EOF\n+\ttest-file-watcher . >actual paths.actual &&\n+\tcat <expect &&\n+packet: git> hello\n+packet: git< hello\n+packet: git> index $SIG $TRASH_DIRECTORY\n+packet: git< ok\n+packet: git> get-changed\n+packet: git< changed 8\n+EOF\n+\ttest_cmp expect actual &&\n+\tcat <paths.expect &&\n+w abc\n+H bar/ghi\n+w foo/def\n+EOF\n+\ttest_cmp paths.expect paths.actual\n+'\n+\n+test_expect_success 'sync file-watcher after index update' '\n+\tSIG=`test-file-watcher --index-signature .git/index` &&\n+\trm actual &&\n+\tGIT_TRACE_PACKET=\"$TRASH_DIRECTORY/actual\" git status --porcelain | grep -vF \"??\" >paths.actual &&\n+\tSIG2=`test-file-watcher --index-signature .git/index` &&\n+\tcat <expect &&\n+packet: git> hello\n+packet: git< hello\n+packet: git> index $SIG $TRASH_DIRECTORY\n+packet: git< ok\n+packet: git> get-changed\n+packet: git< changed 8\n+packet: git> watch 8\n+packet: git< watched 1\n+packet: git> new-index $SIG2\n+packet: git> unchange 8\n+EOF\n+\ttest_cmp expect actual &&\n+\tcat <paths.expect &&\n+A abc\n+AM bar/ghi\n+A foo/def\n+EOF\n+\ttest_cmp paths.expect paths.actual\n+'\n+\n+test_expect_success 'make sure file-watcher cleans its changed list' '\n+\tSIG=`test-file-watcher --index-signature .git/index` &&\n+\trm actual &&\n+\tGIT_TEST_WATCHER=1 GIT_TRACE_PACKET=\"$TRASH_DIRECTORY/actual\" git ls-files -v >paths.actual &&\n+\tcat <expect &&\n+packet: git> hello\n+packet: git< hello\n+packet: git> index $SIG $TRASH_DIRECTORY\n+packet: git< ok\n+packet: git> get-changed\n+packet: git< changed 0\n+EOF\n+\ttest_cmp expect actual &&\n+\tcat <paths.expect &&\n+w abc\n+H bar/ghi\n+w foo/def\n+EOF\n+\ttest_cmp paths.expect paths.actual\n+'\n+\n+test_expect_success 'closing the daemon' '\n+\ttest-file-watcher . </dev/null\n+see you\n+EOF\n+'\n+\n+test_done\ndiff --git a/test-file-watcher.c b/test-file-watcher.c\nindex ffff198..77037e1 100644\n--- a/test-file-watcher.c\n+++ b/test-file-watcher.c\n@@ -11,6 +11,21 @@ int main(int ac, char **av)\n \tint last_command_is_reply = 0;\n \tint fd;\n \n+\tif (!strcmp(av[1], \"--index-signature\")) {\n+\t\tstruct stat st;\n+\t\tvoid *mmap;\n+\t\tif (lstat(av[2], &st))\n+\t\t\tdie_errno(\"lstat\");\n+\t\tfd = open(av[2], O_RDONLY);\n+\t\tif (fd < 0)\n+\t\t\tdie_errno(\"open\");\n+\t\tmmap = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\t\tif (mmap == MAP_FAILED)\n+\t\t\tdie_errno(\"mmap\");\n+\t\tprintf(\"%s\\n\", sha1_to_hex((unsigned char*)mmap + st.st_size - 20));\n+\t\treturn 0;\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/socket\", av[1]);\n \tfd = unix_stream_connect(sb.buf);\n \tif (fd < 0)\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234097","messageId":"1391401754-15347-27-git-send-email-pclouds@gmail.com","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 26/26] Disable file-watcher with system inotify on some tests","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-03T04:29:14Z","receivedAt":"2014-02-03T04:29:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When file watcher is active via GIT_TEST_WATCHER_PATH, `ls-files -v`\noutput may be different (e.g. \"H\" becomes \"W\"). Disable file watcher\nin those cases.\n\nWe could make ls-files turn 'W' back to 'H' for these cases, but not\nsure it's worth the effort. The intention of running with\nGIT_TEST_WATCHER_PATH is to exercise file watcher in somewhat real\nscenarios and the remaining tests are more than enough for that.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n t/t1011-read-tree-sparse-checkout.sh | 2 ++\n t/t2104-update-index-skip-worktree.sh | 2 ++\n t/t7011-skip-worktree-reading.sh | 2 ++\n t/t7012-skip-worktree-writing.sh | 2 ++\n 4 files changed, 8 insertions(+)\n\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 0c74bee..00e92bc 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -14,6 +14,8 @@ test_description='sparse checkout tests\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-read-tree.sh\n \n+unset GIT_TEST_WATCHER_PATH\n+\n test_expect_success 'setup' '\n \tcat >expected <<-\\EOF &&\n \t100644 77f0ba1734ed79d12881f81b36ee134de6a3327b 0\tinit.t\ndiff --git a/t/t2104-update-index-skip-worktree.sh b/t/t2104-update-index-skip-worktree.sh\nindex 1d0879b..2b93c58 100755\n--- a/t/t2104-update-index-skip-worktree.sh\n+++ b/t/t2104-update-index-skip-worktree.sh\n@@ -7,6 +7,8 @@ test_description='skip-worktree bit test'\n \n . ./test-lib.sh\n \n+unset GIT_TEST_WATCHER_PATH\n+\n cat >expect.full <expect.full <> init.t &&\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"234494","messageId":"52F5E521.4090707@web.de","threadId":"35642","inReplyTo":"1391401754-15347-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 00/26] inotify support","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-02-08T08:04:49Z","receivedAt":"2014-02-08T08:04:49Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 03.02.14 05:28, Nguyễn Thái Ngọc Duy wrote:\n\nI managed to review the code 0..12/26, so some parts are missing.\nThe list below became longer than what I intended,\nmy comments may be hard to read,\nand there is a mixture of minor and major remarks.\n\nI would appreciate if we could have an outline of the protocol\nas a seperate \"document\" somewhere, to be able to have a look at the protocol\nfirst, before looking into the code.\n\n(Am I using wireshark too much to dream about a dissector ?)\n\nAll in all I like the concept, thanks for the work.\n\n\n1)\n write_in_full_timeout()\n packet_read_line_timeout()\n At other places we handle EINTR after calling poll().\n Looking at the code, it could be easier to introduce\n a new function xpoll() in wrapper.c, and use that instead\n of poll().\n\n2)\n Similar for the usage of accept().\n I like the idea of xread() xwrite() and all the x functions\n so it coud make sense to establish xpoll() and xaccept()\n before inotify suppport.\n\n\n3)\n> -int unix_stream_listen(const char *path)\n> +int unix_stream_listen(const char *path, int replace)\n> {\n> \tint fd, saved_errno;\n> \tstruct sockaddr_un sa;\n> @@ -103,7 +103,8 @@ int unix_stream_listen(const char *path)\n> \t\treturn -1;\n> \tfd = unix_stream_socket();\n> \n> -\tunlink(path);\n> +\tif (replace)\n> +\t\tunlink(path);\n\nMinor remark:\nAs we do not do the replace here:\ns/replace/un_link/ may be ?\n\n\n4)\n> +++ b/file-watcher.c\n{}\n> +static const char *const file_watcher_usage[] = {\n> +\tN_(\"git file-watcher [options] \"),\n> +\tNULL\n> +};\nDo we already have options here?\nI can think about having \n-d daemoniye\n-u uses Unix doain socket\n(And later -t to use a TCP/IP socket, when the repo\n is on a mounted NFS (or SAMBA) drive, and the daemon is on a \n different machine.\n I don't say this patch should include this logic in first round,\n But I can see a gain for this kind of setup)\n\n\n5)\n> +++ b/file-watcher.c\n[]\n> +static int shutdown_connection(int id)\n> +{\n> +\tstruct connection *conn = conns[id];\n> +\tconns[id] = NULL;\n> +\tpfd[id].fd = -1;\n> +\tif (!conn)\n> +\t\treturn 0;\n> +\tclose(conn->sock);\n> +\tfree(conn);\n> +\treturn 0;\n> +}\nThe function is called shutdown_connection(), but it does a close()\nCould it be named close_connection() ?\n\n6) \n> +++ b/file-watcher.c\n[]\nDo we have a sequence chart about the command flow between the watcher\ndaemon and the client ?\n\n------------\n7)\n\nin 03/26:\n>This version is also gentler than its friend packet_read_line()\ngentler, what does this mean?\n\n>because it's designed for side channel I/O that should not abort the\n>program even if the channel is broken.\nI'm not so familar with side-channel I/O. How does it fit in here?\n\nDoes this make sense:\nIn opposite to packet_read_line() which can call die()\nto abort the program, read_in_full_timeout() will keep the program running.\n(or something like this)\n\n>\n>Signed-off-by: Nguyễn Thái Ngọc Duy \n>---\n> cache.h | 1 +\n> pkt-line.c | 35 +++++++++++++++++++++++++++++++++++\n> pkt-line.h | 1 +\n> wrapper.c | 21 +++++++++++++++++++++\n> 4 files changed, 58 insertions(+)\n>\n>diff --git a/cache.h b/cache.h\n>index 718e32b..939db46 100644\n>--- a/cache.h\n>+++ b/cache.h\n>@@ -1230,6 +1230,7 @@ extern int write_or_whine_pipe(int fd, const void *buf, size_t count, const char\n> extern void fsync_or_die(int fd, const char *);\n> \n> extern ssize_t read_in_full(int fd, void *buf, size_t count);\n>+extern ssize_t read_in_full_timeout(int fd, void *buf, size_t count, int timeout);\n> extern ssize_t write_in_full(int fd, const void *buf, size_t count);\n> extern ssize_t write_in_full_timeout(int fd, const void *buf, size_t count, int timeout);\n> static inline ssize_t write_str_in_full(int fd, const char *str)\n>diff --git a/pkt-line.c b/pkt-line.c\n>index cf681e9..5a07e97 100644\n>--- a/pkt-line.c\n>+++ b/pkt-line.c\n>@@ -229,3 +229,38 @@ char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)\n> {\n> \treturn packet_read_line_generic(-1, src, src_len, dst_len);\n> }\n>+\n\nIn what is the timeout measured ?\nseconds, milli years?\nAs we use poll() I think it is milli seconds.\n(I like the idea of naming timeout timeout_ms)\n\n[]\n>+\tlen -= 4;\n>+\tif (len >= buf_len) {\n>+\t\terror(\"protocol error: bad line length %d\", len);\n>+\t\treturn NULL;\n>+\t}\n>+\tif ((ret = read_in_full_timeout(fd, buf, len, timeout)) < 0)\n>+\t\treturn NULL;\nDo we want a packet_trace here?\n\nWhen a timeout occurs, do we want to close the connection,\nmarking it as dead?\nOr need to look at errno?\n\n\n>+\tbuf[len] = '\\0';\n>+\tif (len_p)\n>+\t\t*len_p = len;\n>+\tpacket_trace(buf, len, 0);\n>+\treturn buf;\n>+}\n\n>diff --git a/pkt-line.h b/pkt-line.h\n>index 4b93a0c..d47dca5 100644\n>--- a/pkt-line.h\n>+++ b/pkt-line.h\n>@@ -69,6 +69,7 @@ int packet_read(int fd, char **src_buffer, size_t *src_len, char\n> * packet is written to it.\n> */\n> char *packet_read_line(int fd, int *size);\n>+char *packet_read_line_timeout(int fd, int timeout, int *size);\n> \n> /*\n> * Same as packet_read_line, but read from a buf rather than a descriptor;\n>diff --git a/wrapper.c b/wrapper.c\n>index 9a0e289..9cf10b2 100644\n>--- a/wrapper.c\n>+++ b/wrapper.c\n>@@ -193,6 +193,27 @@ ssize_t read_in_full(int fd, void *buf, size_t count)\n> \treturn total;\n> }\n> \n>+ssize_t read_in_full_timeout(int fd, void *buf, size_t count, int timeout)\n>+{\n>+\tchar *p = buf;\n>+\tssize_t total = 0;\n>+\tstruct pollfd pfd;\n>+\n>+\tpfd.fd = fd;\n>+\tpfd.events = POLLIN;\n>+\twhile (count > 0 && poll(&pfd, 1, timeout) > 0 &&\n>+\t (pfd.revents & POLLIN)) {\n>+\t\tssize_t loaded = xread(fd, p, count);\n>+\t\tif (loaded <= 0)\n>+\t\t\treturn -1;\n>+\t\tcount -= loaded;\n>+\t\tp += loaded;\n>+\t\ttotal += loaded;\n>+\t}\n>+\n>+\treturn count ? -1 : total;\nIsn't it that \nret < 0 means \"error of some kind\"\nret == 0 means EOF,\nret > 0 means some data\nWhy do we turn 0 into -1?\n\n--------------\n8)\n+++ b/unix-socket.c\n@@ -93,7 +93,7 @@ fail:\n \treturn -1;\n }\n \n-int unix_stream_listen(const char *path)\n+int unix_stream_listen(const char *path, int replace)\n {\n \tint fd, saved_errno;\n \tstruct sockaddr_un sa;\n@@ -103,7 +103,8 @@ int unix_stream_listen(const char *path)\n \t\treturn -1;\n \tfd = unix_stream_socket();\n \n-\tunlink(path);\n+\tif (replace)\n+\t\tunlink(path);\n \tif (bind(fd, (struct sockaddr *)&sa, sizeof(sa)) < 0)\nWhy do we call the parameter replace, when it does an\nunlink() ?\ns/replace/un_link/ ? \n\n\n9)\n>Access to the unix socket $WATCHER/socket is covered by $WATCHER's\n>permission. While the file watcher does not carry much information,\n>repo file listing is sometimes not something you want other users to\n>see. Make sure $WATCHER has 0700 permission to stop unwanted access.\n\n>Signed-off-by: Nguyễn Thái Ngọc Duy \n>---\n> file-watcher.c | 32 ++++++++++++++++++++++++++++++++\n> 1 file changed, 32 insertions(+)\n\nI feel a little bit unsure about the 700.\nMost often Git does not care about permissions,\nand relies on umask being set appropriatly.\n(Please correct me if I'm wrong)\n\nMy spontanous feeling is that adjust_shared_perm() could be used.\n\n10)\nAn other thing:\n>strbuf_addf(&sb, \"%s/socket\", socket_path);\nDoes it make sense to name the socket \"%s/watcher\" ?\n\n\n11)\n>In daemon mode, stdout and stderr are saved in $WATCHER/log.\n\n[]\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-file-watcher.txt | 2 ++\n cache.h | 1 +\n daemon.c | 30 ++++--------------------------\n file-watcher.c | 17 +++++++++++++++++\n setup.c | 25 +++++++++++++++++++++++++\n 5 files changed, 49 insertions(+), 26 deletions(-)\n\ndiff --git a/Documentation/git-file-watcher.txt b/Documentation/git-file-watcher.txt\nindex 625a389..ec81f18 100644\n--- a/Documentation/git-file-watcher.txt\n+++ b/Documentation/git-file-watcher.txt\n@@ -18,6 +18,8 @@ lstat(2) to detect that itself.\n \n OPTIONS\n -------\n+--detach::\n+\tRun in background.\n\nShouldn't that be named --daemon ?\n\n\n12)\n>read_cache() connects to the file watcher, specified by\n>filewatcher.path config, and performs basic hand shaking. CE_WATCHED\n>is cleared if git and file watcher have different views on the index\n>state.\n>\n>All send/receive calls must be complete within a limited time to avoid\n>a buggy file-watcher hang \"git status\" forever. And the whole point of\n>doing this is speed. If file watcher can't respond fast enough, for\n>whatever reason, then it should not be used.\n>\n>Signed-off-by: Nguyễn Thái Ngọc Duy \n>---\n> Documentation/config.txt | 10 +++\n> Documentation/git-file-watcher.txt | 4 +-\n> Makefile | 1 +\n> cache.h | 1 +\n> file-watcher-lib.c (new) | 91 ++++++++++++++++++++++\n> file-watcher-lib.h (new) | 6 ++\n> file-watcher.c | 152 ++++++++++++++++++++++++++++++++++++-\n> read-cache.c | 6 ++\n> 8 files changed, 269 insertions(+), 2 deletions(-)\n> create mode 100644 file-watcher-lib.c\n> create mode 100644 file-watcher-lib.h\n>\n>diff --git a/Documentation/config.txt b/Documentation/config.txt\n>index 5f4d793..6ad653a 100644\n>--- a/Documentation/config.txt\n>+++ b/Documentation/config.txt\n>@@ -1042,6 +1042,16 @@ difftool..cmd::\n> difftool.prompt::\n> \tPrompt before each invocation of the diff tool.\n> \n>+filewatcher.path::\n>+\tThe directory that contains the socket of `git\tfile-watcher`.\n>+\tIf it's not an absolute path, it's relative to $GIT_DIR. An\n>+\tempty path means no connection to file watcher.\nI would really like to be able to have different tranport mechanismen\nusing the same config.\nToday this is only a matter of documentation and naming things:\n\nHow about this idea:\nfilewatcher.url::\nThe directory that contains the socket of `git\tfile-watcher`.\nIt can be an absolute path, meaning a unix domain socket in the\nfile system.\nIt can be a path relative to $GIT_DIR.\n\nIt must start with either \n\"/\" (absolut path) or \n\"./\" (relative path to $GIT_DIR) or \n\"~/\" (relaive path in your $HOME directoy.\nAn empty url means no connection to file watcher.\n(Later we can add url schemes like \"tcp://\" or \"pipe://\")\n\n\n>+\n>+filewatcher.timeout::\n>+\tThis is the maximum time in milliseconds that Git waits for\n>+\tthe file watcher to respond before giving up. Default value is\n>+\t50. Setting to -1 makes Git wait forever.\n50 feels low, especially on \"older/slower machines\"\n200 is probably acceptable even for the very impatient user,\nI could think of 500 to make the user aware of it.\n\n[]\n>+\t/*\n>+\t * \">\" denotes an incoming packet, \"<\" outgoing. The lack of\n>+\t * \"<\" means no reply expected.\n>+\t *\n>+\t * < \"error\" SP ERROR-STRING\n>+\t *\n>+\t * This can be sent whenever the client violates the protocol.\n>+\t */\n>+\n>+\tmsg = packet_read_line(fd, &len);\n>+\tif (!msg) {\n>+\t\tpacket_write(fd, \"error invalid pkt-line\");\n>+\t\treturn shutdown_connection(conn_id);\n>+\t}\n>+\n>+\t/*\n>+\t * > \"hello\" [SP CAP [SP CAP..]]\n>+\t * < \"hello\" [SP CAP [SP CAP..]]\n>+\t *\n>+\t * Advertise capabilities of both sides. File watcher may\n>+\t * disconnect if the client does not advertise the required\n>+\t * capabilities. Capabilities in uppercase MUST be\n>+\t * supported. If any side does not understand any of the\n>+\t * advertised uppercase capabilities, it must disconnect.\n>+\t */\n>+\tif ((arg = skip_prefix(msg, \"hello\"))) {\n>+\t\tif (*arg) {\t/* no capabilities supported yet */\n>+\t\t\tpacket_write(fd, \"error capabilities not supported\");\n>+\t\t\treturn shutdown_connection(conn_id);\n>+\t\t}\n>+\t\tpacket_write(fd, \"hello\");\n>+\t\tconns[conn_id]->polite = 1;\n>+\t}\n>+\n>+\t/*\n>+\t * > \"index\" SP INDEX-SIGNATURE SP WORK-TREE-PATH\n>+\t * < \"ok\" | \"inconsistent\"\n>+\t *\n>+\t * INDEX-SIGNATURE consists of 40 hexadecimal letters\n>+\t * WORK-TREE-PATH must be absolute and normalized path\n>+\t *\n>+\t * Watch file changes in index. The client sends the index and\n>+\t * work tree info. File watcher validates that it holds the\n>+\t * same info. If so it sends \"ok\" back indicating both sides\n>+\t * are on the same page and CE_WATCHED bits can be ketpt.\n>+\t *\n>+\t * Otherwise it sends \"inconsistent\" and both sides must reset\n>+\t * back to initial state. File watcher keeps its index\n>+\t * signature all-zero until the client has updated the index\n>+\t * ondisk and request to update index signature.\n>+\t *\n>+\t * \"hello\" must be exchanged first. After this command the\n>+\t * connection is associated with a worktree/index. Many\n>+\t * commands may require this to proceed.\n>+\t */\n>+\telse if (starts_with(msg, \"index \")) {\n>+\t\tstruct repository *repo;\n>+\t\tstruct stat st;\n>+\t\tif (!conns[conn_id]->polite) {\n>+\t\t\tpacket_write(fd, \"error why did you not greet me? go away\");\n>+\t\t\treturn shutdown_connection(conn_id);\n>+\t\t}\n>+\t\tif (len < 47 || msg[46] != ' ' || !is_absolute_path(msg + 47)) {\n>+\t\t\tpacket_write(fd, \"error invalid index line %s\", msg);\n>+\t\t\treturn shutdown_connection(conn_id);\n>+\t\t}\n>+\n>+\t\tif (lstat(msg + 47, &st) || !S_ISDIR(st.st_mode)) {\n>+\t\t\tpacket_write(fd, \"error work tree does not exist: %s\",\n>+\t\t\t\t strerror(errno));\n>+\t\t\treturn shutdown_connection(conn_id);\n>+\t\t}\n>+\t\trepo = get_repo(msg + 47);\n>+\t\tconns[conn_id]->repo = repo;\n>+\t\tif (memcmp(msg + 6, repo->index_signature, 40) ||\n>+\t\t !memcmp(msg + 6, invalid_signature, 40) ||\n>+\t\t repo->inode != st.st_ino) {\n>+\t\t\tpacket_write(fd, \"inconsistent\");\n>+\t\t\treset_repo(repo, st.st_ino);\n>+\t\t\treturn 0;\n>+\t\t}\n>+\t\tpacket_write(fd, \"ok\");\n>+\t}\n>+\telse {\n>+\t\tpacket_write(fd, \"error unrecognized command %s\", msg);\n>+\t\treturn shutdown_connection(conn_id);\nI feel a little bit unsure about this.\nWe establish a protocol which is not extendable.\nDo we need to call shutdown_connection() ?\n(And as I noticed earlier, close_connection() could be a better name)\n"},{"id":"234499","messageId":"CACsJy8C0x1FL_6KtYj4MMnhtR6RrPSxR6_yysnZykubdotT9GA@mail.gmail.com","threadId":"35642","inReplyTo":"52F5E521.4090707@web.de","subject":"Re: [PATCH v3 00/26] inotify support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-08T08:53:05Z","receivedAt":"2014-02-08T08:53:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Thanks for the comments. I can see I now have some work to do in the\ncoming weeks :)\n\nOn Sat, Feb 8, 2014 at 3:04 PM, Torsten Bögershausen wrote:\n> I would appreciate if we could have an outline of the protocol\n> as a seperate \"document\" somewhere, to be able to have a look at the protocol\n> first, before looking into the code.\n\nMy fear is document becomes outdated because people don't always\nremember to update doc when they change code. Which is why I embed the\nprotocol as comments in handle_command() function. If the final\nversion [1] is still not easy to read, I'll split the protocol out\ninto a separate document.\n\n[1] https://github.com/pclouds/git/blob/file-watcher/file-watcher.c#L672\n\n> (Am I using wireshark too much to dream about a dissector ?)\n\nTry GIT_TRACE_PACKET=2\n\n> 1)\n> write_in_full_timeout()\n> packet_read_line_timeout()\n> At other places we handle EINTR after calling poll().\n> Looking at the code, it could be easier to introduce\n> a new function xpoll() in wrapper.c, and use that instead\n> of poll().\n\nYeah there are already 4 poll call sites before file-watcher jumps in. Will do.\n\n> 2)\n> Similar for the usage of accept().\n> I like the idea of xread() xwrite() and all the x functions\n> so it coud make sense to establish xpoll() and xaccept()\n> before inotify suppport.\n\nOK\n\n> 3)\n>> -int unix_stream_listen(const char *path)\n>> +int unix_stream_listen(const char *path, int replace)\n>> {\n>> int fd, saved_errno;\n>> struct sockaddr_un sa;\n>> @@ -103,7 +103,8 @@ int unix_stream_listen(const char *path)\n>> return -1;\n>> fd = unix_stream_socket();\n>>\n>> - unlink(path);\n>> + if (replace)\n>> + unlink(path);\n>\n> Minor remark:\n> As we do not do the replace here:\n> s/replace/un_link/ may be ?\n\nHeh, I thought of using the name \"unlink\" but it's taken so I chose\n\"replace\" and did not think of underscore.Will do.\n\n> 4)\n>> +++ b/file-watcher.c\n> {}\n>> +static const char *const file_watcher_usage[] = {\n>> + N_(\"git file-watcher [options] \"),\n>> + NULL\n>> +};\n> Do we already have options here?\n> I can think about having\n> -d daemoniye\n> -u uses Unix doain socket\n> (And later -t to use a TCP/IP socket, when the repo\n> is on a mounted NFS (or SAMBA) drive, and the daemon is on a\n> different machine.\n> I don't say this patch should include this logic in first round,\n> But I can see a gain for this kind of setup)\n\nLater on we have two, --detach (i.e. daemonize, I reuse the same name\nfrom git-daemon) and --check-support. Transport settings (like unix vs\ntcp/ip...) should be config options though. You don't want to specify\nit here, then again when you run \"git status\". Actually I should add\n--default, that will retrieve from config key\nfilewatcher.path, so the user does not have to specify it..\n\n> 5)\n>> +++ b/file-watcher.c\n> []\n>> +static int shutdown_connection(int id)\n>> +{\n>> + struct connection *conn = conns[id];\n>> + conns[id] = NULL;\n>> + pfd[id].fd = -1;\n>> + if (!conn)\n>> + return 0;\n>> + close(conn->sock);\n>> + free(conn);\n>> + return 0;\n>> +}\n> The function is called shutdown_connection(), but it does a close()\n> Could it be named close_connection() ?\n\nYes, there was a close_connection() which did something similar, and\nthen it was killed off. Will rename.\n\n> 6)\n>> +++ b/file-watcher.c\n> []\n> Do we have a sequence chart about the command flow between the watcher\n> daemon and the client ?\n\nI suggest you have a look at the file-watcher test. For example, from\n[2] we have this sequence\n\nhello\ninconsistent\n# Do not call inotify_add_watch()\ntest mode on\n# First batch should be all ok\n<\" the opposite\ndirection. But you can always obtain a real flow with\nGIT_TRACE_PACKET=2 (path lists not available though, so really just a\nflow).\n\n[2] https://github.com/pclouds/git/blob/file-watcher/t/t7513-file-watcher.sh#L273\n\n> in 03/26:\n>>This version is also gentler than its friend packet_read_line()\n> gentler, what does this mean?\n\nNo dying for whatever error. packet_read_line is designed for git\nprotocol. It's the main job so if there's an error, dying is the right\nthing to do. file-watcher on the other hand is a side job and should\nnot stop whatever command from doing. Will update commit message.\n\n>>because it's designed for side channel I/O that should not abort the\n>>program even if the channel is broken.\n> I'm not so familar with side-channel I/O. How does it fit in here?\n\nTo sum up, whatever error in communication with file-watcher should\nnot stop you from doing whatever you're doing. file-watcher is\ncontacted whenever $GIT_DIR/index is read, so \"whatever you're doing\"\nis basically all git commands that involve worktree or index.\n\n> Does this make sense:\n> In opposite to packet_read_line() which can call die()\n> to abort the program, read_in_full_timeout() will keep the program running.\n> (or something like this)\n\nExactly!\n\n>>diff --git a/cache.h b/cache.h\n>>index 718e32b..939db46 100644\n>>--- a/cache.h\n>>+++ b/cache.h\n>>@@ -1230,6 +1230,7 @@ extern int write_or_whine_pipe(int fd, const void *buf, size_t count, const char\n>> extern void fsync_or_die(int fd, const char *);\n>>\n>> extern ssize_t read_in_full(int fd, void *buf, size_t count);\n>>+extern ssize_t read_in_full_timeout(int fd, void *buf, size_t count, int timeout);\n>> extern ssize_t write_in_full(int fd, const void *buf, size_t count);\n>> extern ssize_t write_in_full_timeout(int fd, const void *buf, size_t count, int timeout);\n>> static inline ssize_t write_str_in_full(int fd, const char *str)\n>>diff --git a/pkt-line.c b/pkt-line.c\n>>index cf681e9..5a07e97 100644\n>>--- a/pkt-line.c\n>>+++ b/pkt-line.c\n>>@@ -229,3 +229,38 @@ char *packet_read_line_buf(char **src, size_t *src_len, int *dst_len)\n>> {\n>> return packet_read_line_generic(-1, src, src_len, dst_len);\n>> }\n>>+\n>\n> In what is the timeout measured ?\n> seconds, milli years?\n> As we use poll() I think it is milli seconds.\n> (I like the idea of naming timeout timeout_ms)\n\nYes in ms. Will rename.\n\n>>+ len -= 4;\n>>+ if (len >= buf_len) {\n>>+ error(\"protocol error: bad line length %d\", len);\n>>+ return NULL;\n>>+ }\n>>+ if ((ret = read_in_full_timeout(fd, buf, len, timeout)) < 0)\n>>+ return NULL;\n> Do we want a packet_trace here?\n\nCompared to packet_read_line(), probably not.\n\n> When a timeout occurs, do we want to close the connection,\n> marking it as dead?\n> Or need to look at errno?\n\nWe do want to close the connection, but the caller should do that and\nclear the file handle variable at the same time. Or we risk race\ncondition.\n\n>>diff --git a/wrapper.c b/wrapper.c\n>>index 9a0e289..9cf10b2 100644\n>>--- a/wrapper.c\n>>+++ b/wrapper.c\n>>@@ -193,6 +193,27 @@ ssize_t read_in_full(int fd, void *buf, size_t count)\n>> return total;\n>> }\n>>\n>>+ssize_t read_in_full_timeout(int fd, void *buf, size_t count, int timeout)\n>>+{\n>>+ char *p = buf;\n>>+ ssize_t total = 0;\n>>+ struct pollfd pfd;\n>>+\n>>+ pfd.fd = fd;\n>>+ pfd.events = POLLIN;\n>>+ while (count > 0 && poll(&pfd, 1, timeout) > 0 &&\n>>+ (pfd.revents & POLLIN)) {\n>>+ ssize_t loaded = xread(fd, p, count);\n>>+ if (loaded <= 0)\n>>+ return -1;\n>>+ count -= loaded;\n>>+ p += loaded;\n>>+ total += loaded;\n>>+ }\n>>+\n>>+ return count ? -1 : total;\n> Isn't it that\n> ret < 0 means \"error of some kind\"\n> ret == 0 means EOF,\n> ret > 0 means some data\n> Why do we turn 0 into -1?\n\nBecause this is read_in_full. The caller expects either all is read, or errors.\n\n> 9)\n>>Access to the unix socket $WATCHER/socket is covered by $WATCHER's\n>>permission. While the file watcher does not carry much information,\n>>repo file listing is sometimes not something you want other users to\n>>see. Make sure $WATCHER has 0700 permission to stop unwanted access.\n>\n>>Signed-off-by: Nguyễn Thái Ngọc Duy \n>>---\n>> file-watcher.c | 32 ++++++++++++++++++++++++++++++++\n>> 1 file changed, 32 insertions(+)\n>\n> I feel a little bit unsure about the 700.\n> Most often Git does not care about permissions,\n> and relies on umask being set appropriatly.\n> (Please correct me if I'm wrong)\n\nGit does care. See credential-cache--daemon.c. In fact this function\nis a copy of check_socket_directory() from that file.\n\n> My spontanous feeling is that adjust_shared_perm() could be used.\n>\n> 10)\n> An other thing:\n>>strbuf_addf(&sb, \"%s/socket\", socket_path);\n> Does it make sense to name the socket \"%s/watcher\" ?\n\nNo strong opinions. Whatever name works for me. Will change unless\nsomebody else raises their voice.\n\n> 11)\n>>In daemon mode, stdout and stderr are saved in $WATCHER/log.\n\nMostly for debugging purposes.\n\n> diff --git a/Documentation/git-file-watcher.txt b/Documentation/git-file-watcher.txt\n> index 625a389..ec81f18 100644\n> --- a/Documentation/git-file-watcher.txt\n> +++ b/Documentation/git-file-watcher.txt\n> @@ -18,6 +18,8 @@ lstat(2) to detect that itself.\n>\n> OPTIONS\n> -------\n> +--detach::\n> + Run in background.\n>\n> Shouldn't that be named --daemon ?\n\nI just followed the naming convention in git-daemon. I think detach is\nmore friendly then daemon anyway.\n\n> 12)\n>>+filewatcher.path::\n>>+ The directory that contains the socket of `git file-watcher`.\n>>+ If it's not an absolute path, it's relative to $GIT_DIR. An\n>>+ empty path means no connection to file watcher.\n> I would really like to be able to have different tranport mechanismen\n> using the same config.\n> Today this is only a matter of documentation and naming things:\n>\n> How about this idea:\n> filewatcher.url::\n> The directory that contains the socket of `git file-watcher`.\n> It can be an absolute path, meaning a unix domain socket in the\n> file system.\n> It can be a path relative to $GIT_DIR.\n>\n> It must start with either\n> \"/\" (absolut path) or\n> \"./\" (relative path to $GIT_DIR) or\n> \"~/\" (relaive path in your $HOME directoy.\n> An empty url means no connection to file watcher.\n> (Later we can add url schemes like \"tcp://\" or \"pipe://\")\n\nMakes sense.\n\n>>+filewatcher.timeout::\n>>+ This is the maximum time in milliseconds that Git waits for\n>>+ the file watcher to respond before giving up. Default value is\n>>+ 50. Setting to -1 makes Git wait forever.\n> 50 feels low, especially on \"older/slower machines\"\n> 200 is probably acceptable even for the very impatient user,\n> I could think of 500 to make the user aware of it.\n\nNote that this timeout should never kick in unless there is an error\nin communication. So in practice it should not matter much. OK I lied\na bit because file-watcher hard codes to \"ping\" the client every 30ms\nto avoid this timeout. So it does matter, but very little.\n\n>>+ /*\n>>+ * \">\" denotes an incoming packet, \"<\" outgoing. The lack of\n>>+ * \"<\" means no reply expected.\n>>+ *\n>>+ * < \"error\" SP ERROR-STRING\n>>+ *\n>>+ * This can be sent whenever the client violates the protocol.\n>>+ */\n>>+\n>>+ msg = packet_read_line(fd, &len);\n>>+ if (!msg) {\n>>+ packet_write(fd, \"error invalid pkt-line\");\n>>+ return shutdown_connection(conn_id);\n>>+ }\n>>+\n>>+ /*\n>>+ * > \"hello\" [SP CAP [SP CAP..]]\n>>+ * < \"hello\" [SP CAP [SP CAP..]]\n>>+ *\n>>+ * Advertise capabilities of both sides. File watcher may\n>>+ * disconnect if the client does not advertise the required\n>>+ * capabilities. Capabilities in uppercase MUST be\n>>+ * supported. If any side does not understand any of the\n>>+ * advertised uppercase capabilities, it must disconnect.\n>>+ */\n>>+ if ((arg = skip_prefix(msg, \"hello\"))) {\n>>+ if (*arg) { /* no capabilities supported yet */\n>>+ packet_write(fd, \"error capabilities not supported\");\n>>+ return shutdown_connection(conn_id);\n>>+ }\n>>+ packet_write(fd, \"hello\");\n>>+ conns[conn_id]->polite = 1;\n>>+ }\n>>+\n>>+ /*\n>>+ * > \"index\" SP INDEX-SIGNATURE SP WORK-TREE-PATH\n>>+ * < \"ok\" | \"inconsistent\"\n>>+ *\n>>+ * INDEX-SIGNATURE consists of 40 hexadecimal letters\n>>+ * WORK-TREE-PATH must be absolute and normalized path\n>>+ *\n>>+ * Watch file changes in index. The client sends the index and\n>>+ * work tree info. File watcher validates that it holds the\n>>+ * same info. If so it sends \"ok\" back indicating both sides\n>>+ * are on the same page and CE_WATCHED bits can be ketpt.\n>>+ *\n>>+ * Otherwise it sends \"inconsistent\" and both sides must reset\n>>+ * back to initial state. File watcher keeps its index\n>>+ * signature all-zero until the client has updated the index\n>>+ * ondisk and request to update index signature.\n>>+ *\n>>+ * \"hello\" must be exchanged first. After this command the\n>>+ * connection is associated with a worktree/index. Many\n>>+ * commands may require this to proceed.\n>>+ */\n>>+ else if (starts_with(msg, \"index \")) {\n>>+ struct repository *repo;\n>>+ struct stat st;\n>>+ if (!conns[conn_id]->polite) {\n>>+ packet_write(fd, \"error why did you not greet me? go away\");\n>>+ return shutdown_connection(conn_id);\n>>+ }\n>>+ if (len < 47 || msg[46] != ' ' || !is_absolute_path(msg + 47)) {\n>>+ packet_write(fd, \"error invalid index line %s\", msg);\n>>+ return shutdown_connection(conn_id);\n>>+ }\n>>+\n>>+ if (lstat(msg + 47, &st) || !S_ISDIR(st.st_mode)) {\n>>+ packet_write(fd, \"error work tree does not exist: %s\",\n>>+ strerror(errno));\n>>+ return shutdown_connection(conn_id);\n>>+ }\n>>+ repo = get_repo(msg + 47);\n>>+ conns[conn_id]->repo = repo;\n>>+ if (memcmp(msg + 6, repo->index_signature, 40) ||\n>>+ !memcmp(msg + 6, invalid_signature, 40) ||\n>>+ repo->inode != st.st_ino) {\n>>+ packet_write(fd, \"inconsistent\");\n>>+ reset_repo(repo, st.st_ino);\n>>+ return 0;\n>>+ }\n>>+ packet_write(fd, \"ok\");\n>>+ }\n>>+ else {\n>>+ packet_write(fd, \"error unrecognized command %s\", msg);\n>>+ return shutdown_connection(conn_id);\n> I feel a little bit unsure about this.\n> We establish a protocol which is not extendable.\n\nIt should be extensible via \"hello\" message, as described in the\ncomments above it.\n\n> Do we need to call shutdown_connection() ?\n\nIf you receive an unrecognized command, there's not much you can do\nbut shut the connection down.\n-- \nDuy\n"},{"id":"234553","messageId":"52F7E2BC.5030905@web.de","threadId":"35642","inReplyTo":"CACsJy8C0x1FL_6KtYj4MMnhtR6RrPSxR6_yysnZykubdotT9GA@mail.gmail.com","subject":"Re: [PATCH v3 00/26] inotify support","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-02-09T20:19:08Z","receivedAt":"2014-02-09T20:19:08Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\nOn 2014-02-08 09.53, Duy Nguyen wrote:\n> Thanks for the comments. I can see I now have some work to do in the\n> coming weeks :)\n> \n\n\n--------------------\n>>> file-watcher.c | 32 ++++++++++++++++++++++++++++++++\n>>> 1 file changed, 32 insertions(+)\n>>\n>> I feel a little bit unsure about the 700.\n>> Most often Git does not care about permissions,\n>> and relies on umask being set appropriatly.\n>> (Please correct me if I'm wrong)\n>\n>Git does care. See credential-cache--daemon.c. In fact this function\n>is a copy of check_socket_directory() from that file.\n>\nI was probably a little bit unclear.\nOf course credentials should be protected well and stored with 700.\nThe rest of the repo could be more loose by using adjust_shared_perm().\nBecause the whole repo can be shared (or not) and data is visible\nto the group or everyone.\n(this is a minor issue)\n\nPlease see filewatcher.c:\n+\tif (daemon) {\n+\t\tint err;\n+\t\tstrbuf_addf(&sb, \"%s/log\", socket_path);\n+\t\terr = open(sb.buf, O_CREAT | O_TRUNC | O_WRONLY, 0600);\n+\t\tadjust_shared_perm(sb.buf);\n(And now we talk about the logfile:\n\"In daemon mode, stdout and stderr are saved in $WATCHER/log.\"\nIt could be nice to make this feature configrable,\neither XXX/log or /dev/null.\nOn the long run we may eat to much disc space on a machine.\nThe other thing is that we may want to seperate stdout\nfrom stderr, but even this is a low prio comment.\n\n\n----------------\nThere is a small issue when I tested on a machine,\nwhere the \"data directory\" called \"daten\" is softlinked to another disk:\ndaten -> /disk3/home2/tb/daten\n\nand the \"projects\" directory is softlinked to \"daten/projects\"\nprojects -> daten/projects/\n\nt7514 fails like this:\n--- expect 2014-02-08 14:37:07.000000000 +0000\n+++ actual 2014-02-08 14:37:07.000000000 +0000\n@@ -1,6 +1,6 @@\n packet: git> hello\n packet: git< hello\n-packet: git> index 6cb9741eee29ca02c5b79e9c0bc647bcf47ce948 /home/tb/projects/git/tb/t/trash directory.t7514-file-watcher-lib\n+packet: git> index 6cb9741eee29ca02c5b79e9c0bc647bcf47ce948 /disk3/home2/tb/daten/projects/git/tb/t/trash directory.t7514-file-watcher-lib\n\nCould we use relative path names internally, relative to $GIT_DIR ?\n\n\n-------------------\nAnother thing:\nCompiling under Mingw gives this:\n LINK git-credential-store.exe\nlibgit.a(file-watcher-lib.o): In function `connect_watcher':\nc:\\Dokumente und Einstellungen\\tb\\projects\\git\\tb/file-watcher-lib.c:21: undefined reference to `unix_stream_connect'\ncollect2: ld returned 1 exit status\nmake: *** [git-credential-store.exe] Error 1\n\nWe may need a compiler option like HAS_UNIX_SOCKETS or so.\n\n--------------------------\n+++ b/file-watcher.c\n\n+#define INOTIFY_MASKS (IN_DELETE_SELF | IN_MOVE_SELF | \\\n+\t\t IN_CREATE | IN_ATTRIB | IN_DELETE | IN_MODIFY |\t\\\n+\t\t IN_MOVED_FROM | IN_MOVED_TO)\nThis feels confusing:\na) we have inotify_masks with lower case below.\nb) how about INOTIFY_NEEDED_BITS ?\n---------------\n\n\n\n\nI'm OK with having the protocol having specified in the\ntest cases.\nOne thing that I have on the wish list is to make the\ncommands/responses more unique, to be able to run grep\non the code base.\n\nOne idea could be to use a prefix\n\"fwr\" for \"file watcher request\" or\n\"fwr\" for \"file watcher response\".\nThis does not work, hihi, so\n\n\"fwq\" for \"file watcher reQuest\" and\n\"fwe\" for \"file watcher rEsponse\".\nOr \n\"ffw\" as \"from file watcher\" and\n\"tfw\" as \"to file watcher\" for the people who have problems\nwith left and right, < and > could work.\n\nThis is all for today.\nI will have a look at different error scenarios, what happens\nwhen the watcher crashes and needs to be restarted,\nor when Git itself dies with a segfault and doesn't tell the\nwatcher.\n\nThe easiest way to simulate this would be in terms of test cases.\nSo I will try to write some\n/Torsten\n \n"},{"id":"234574","messageId":"CACsJy8A9JxDWhEWpdUUL_6tTJZvmf93Ga_nPt09yUzG44mc-Qg@mail.gmail.com","threadId":"35642","inReplyTo":"52F7E2BC.5030905@web.de","subject":"Re: [PATCH v3 00/26] inotify support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-10T10:37:16Z","receivedAt":"2014-02-10T10:37:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Feb 10, 2014 at 3:19 AM, Torsten Bögershausen wrote:\n> Please see filewatcher.c:\n> + if (daemon) {\n> + int err;\n> + strbuf_addf(&sb, \"%s/log\", socket_path);\n> + err = open(sb.buf, O_CREAT | O_TRUNC | O_WRONLY, 0600);\n> + adjust_shared_perm(sb.buf);\n> (And now we talk about the logfile:\n> \"In daemon mode, stdout and stderr are saved in $WATCHER/log.\"\n> It could be nice to make this feature configrable,\n> either XXX/log or /dev/null.\n> On the long run we may eat to much disc space on a machine.\n> The other thing is that we may want to seperate stdout\n> from stderr, but even this is a low prio comment.\n\nI probably should follow git-daemon and put these to syslog.\n\n> ----------------\n> There is a small issue when I tested on a machine,\n> where the \"data directory\" called \"daten\" is softlinked to another disk:\n> daten -> /disk3/home2/tb/daten\n>\n> and the \"projects\" directory is softlinked to \"daten/projects\"\n> projects -> daten/projects/\n>\n> t7514 fails like this:\n> --- expect 2014-02-08 14:37:07.000000000 +0000\n> +++ actual 2014-02-08 14:37:07.000000000 +0000\n> @@ -1,6 +1,6 @@\n> packet: git> hello\n> packet: git< hello\n> -packet: git> index 6cb9741eee29ca02c5b79e9c0bc647bcf47ce948 /home/tb/projects/git/tb/t/trash directory.t7514-file-watcher-lib\n> +packet: git> index 6cb9741eee29ca02c5b79e9c0bc647bcf47ce948 /disk3/home2/tb/daten/projects/git/tb/t/trash directory.t7514-file-watcher-lib\n>\n> Could we use relative path names internally, relative to $GIT_DIR ?\n\nNo because this is when the client tell the server about $GIT_DIR. I\nguess we can use realpath(1) here.\n\n> -------------------\n> Another thing:\n> Compiling under Mingw gives this:\n> LINK git-credential-store.exe\n> libgit.a(file-watcher-lib.o): In function `connect_watcher':\n> c:\\Dokumente und Einstellungen\\tb\\projects\\git\\tb/file-watcher-lib.c:21: undefined reference to `unix_stream_connect'\n> collect2: ld returned 1 exit status\n> make: *** [git-credential-store.exe] Error 1\n>\n> We may need a compiler option like HAS_UNIX_SOCKETS or so.\n\nI'll make unix-socket.o build unconditionally and return error at runtime.\n\n> --------------------------\n> +++ b/file-watcher.c\n>\n> +#define INOTIFY_MASKS (IN_DELETE_SELF | IN_MOVE_SELF | \\\n> + IN_CREATE | IN_ATTRIB | IN_DELETE | IN_MODIFY | \\\n> + IN_MOVED_FROM | IN_MOVED_TO)\n> This feels confusing:\n> a) we have inotify_masks with lower case below.\n> b) how about INOTIFY_NEEDED_BITS ?\n> ---------------\n\nOK\n\n> I'm OK with having the protocol having specified in the\n> test cases.\n> One thing that I have on the wish list is to make the\n> commands/responses more unique, to be able to run grep\n> on the code base.\n>\n> One idea could be to use a prefix\n> \"fwr\" for \"file watcher request\" or\n> \"fwr\" for \"file watcher response\".\n> This does not work, hihi, so\n>\n> \"fwq\" for \"file watcher reQuest\" and\n> \"fwe\" for \"file watcher rEsponse\".\n> Or\n> \"ffw\" as \"from file watcher\" and\n> \"tfw\" as \"to file watcher\" for the people who have problems\n> with left and right, < and > could work.\n\nIf you want I can update test-file-watcher to accept \"send<\" and\n\"recv>\" instead of \"<\" and \">\", respectively. The only command with\nthe same name for response and request is \"hello\". I can make it\n\"hello\" and \"helloack\" (or \"bonjour\" as response?).\n-- \nDuy\n"},{"id":"234583","messageId":"52F90483.7050206@web.de","threadId":"35642","inReplyTo":"CACsJy8A9JxDWhEWpdUUL_6tTJZvmf93Ga_nPt09yUzG44mc-Qg@mail.gmail.com","subject":"Re: [PATCH v3 00/26] inotify support","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-02-10T16:55:31Z","receivedAt":"2014-02-10T16:55:31Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2014-02-10 11.37, Duy Nguyen wrote:\n>>\n>> Could we use relative path names internally, relative to $GIT_DIR ?\n> \n> No because this is when the client tell the server about $GIT_DIR. I\n> guess we can use realpath(1) here.\nGood.\n\nI realized that the watcher can watch several repos at the same time.\n\nHowever, we could allow relative path names, which will be relative to $SOCKET_DIR,\nand loosen the demand for an absolut path name a little bit.\nAnd $SOCKET_DIR can be the same as $GIT_DIR, when we are watching only one repo.\n> If you want I can update test-file-watcher to accept \"send<\" and\n> \"recv>\" instead of \"<\" and \">\", respectively. The only command with\n> the same name for response and request is \"hello\". I can make it\n> \"hello\" and \"helloack\" (or \"bonjour\" as response?).\n\nhelloack looks good. \n"},{"id":"234602","messageId":"CACsJy8ACvd6QaqUySiZSuJmMkJiM1QWr5+jsutPKuajPY_spEg@mail.gmail.com","threadId":"35642","inReplyTo":"52F90483.7050206@web.de","subject":"Re: [PATCH v3 00/26] inotify support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-10T23:34:52Z","receivedAt":"2014-02-10T23:34:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Feb 10, 2014 at 11:55 PM, Torsten Bögershausen wrote:\n> On 2014-02-10 11.37, Duy Nguyen wrote:\n>>>\n>>> Could we use relative path names internally, relative to $GIT_DIR ?\n>>\n>> No because this is when the client tell the server about $GIT_DIR. I\n>> guess we can use realpath(1) here.\n> Good.\n>\n> I realized that the watcher can watch several repos at the same time.\n>\n> However, we could allow relative path names, which will be relative to $SOCKET_DIR,\n> and loosen the demand for an absolut path name a little bit.\n> And $SOCKET_DIR can be the same as $GIT_DIR, when we are watching only one repo.\n\nIt does not help much anyway because file-watcher-lib.c sends\nget_git_work_tree(), which is absolute/normalized path, to\nfile-watcher. There's no sources of sending $GIT_DIR relative to\n$SOCKET_DIR (and I don't think we want to make get_git_work_tree()\nrelative before sending, it's more work on both sides we no benefits,\nexcept for tracing).\n-- \nDuy\n"},{"id":"234924","messageId":"CACsJy8Ck_Bx6kYOgx+ZMcihXPW4hd38k2U7gsGvhyG-PJhz9cQ@mail.gmail.com","threadId":"35642","inReplyTo":"52F7E2BC.5030905@web.de","subject":"Re: [PATCH v3 00/26] inotify support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-17T12:36:55Z","receivedAt":"2014-02-17T12:36:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Feb 10, 2014 at 3:19 AM, Torsten Bögershausen wrote:\n>\n> On 2014-02-08 09.53, Duy Nguyen wrote:\n>>>> file-watcher.c | 32 ++++++++++++++++++++++++++++++++\n>>>> 1 file changed, 32 insertions(+)\n>>>\n>>> I feel a little bit unsure about the 700.\n>>> Most often Git does not care about permissions,\n>>> and relies on umask being set appropriatly.\n>>> (Please correct me if I'm wrong)\n>>\n>>Git does care. See credential-cache--daemon.c. In fact this function\n>>is a copy of check_socket_directory() from that file.\n>>\n> I was probably a little bit unclear.\n> Of course credentials should be protected well and stored with 700.\n> The rest of the repo could be more loose by using adjust_shared_perm().\n> Because the whole repo can be shared (or not) and data is visible\n> to the group or everyone.\n> (this is a minor issue)\n\nSo how about a check whenever a worktree is connected to the daemon,\nif that worktree has stricter permission, e.g. 0700 vs 0770 of the\ndaemon socket directory, then the daemon refuses the worktree (maybe\nwith a warning)?\n-- \nDuy\n"},{"id":"235072","messageId":"CAJo=hJuu8-F_=XyhPVqqK4JPQabK2iZJS=K+=mQ9dnu4Vm1uaQ@mail.gmail.com","threadId":"35642","inReplyTo":"1389524622-6702-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 0/6] inotify support","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2014-02-19T20:35:28Z","receivedAt":"2014-02-19T20:35:28Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sun, Jan 12, 2014 at 3:03 AM, Nguyễn Thái Ngọc Duy wrote:\n> It's been 37 weeks since Robert Zeh's attempt to bring inotify support\n> to Git [1] and unless I missed some mails, no updates since. So here's\n> another attempt with my preferred approach (can't help it, playing\n> with your own ideas is more fun than improving other people's code)\n>\n> To compare to Robert's approach:\n>\n> - This one uses UNIX datagram socket. If I read its man page right,\n> unix socket respects the containing directory's permission. Which\n> means on normal repos, only the user process can access. On shared\n> repos, multiple users can access it. This should work on Mac.\n> Windows will need a different transport.\n>\n> - The daemon is dumb. It passes the paths around and that's it.\n> lstat() is done by git. If I design it right, there's should not be\n> any race conditions that make git miss file updates.\n>\n> - CE_VALID is reused to avoid mass changes (granted there's other\n> neat ways as well). I quite like the idea of machine-controlled\n> CE_VALID.\n>\n> inotify support has the potential of reducing syscalls in\n> read_directory() as well. I wrote about using lstat() to reduce\n> readdir() a while back, if that's implemented then inotify will fit in\n> nicely.\n>\n> This is just a proof of concept. I'm sure I haven't handled all error\n> cases very well. The first five patches show the protocol and git\n> side's changes. The last one fills inotify in.\n>\n> [1] http://thread.gmane.org/gmane.comp.version-control.git/215820/focus=222278\n\nWhy a new daemon? Why don't we reuse the stable\nhttps://github.com/facebook/watchman project Facebook built to make\nHg's status system fast?\n"},{"id":"235081","messageId":"CACsJy8CaTe+disAm8xNPd2T=Z986quL14WVDCVn-q_rUd21A-Q@mail.gmail.com","threadId":"35642","inReplyTo":"CAJo=hJuu8-F_=XyhPVqqK4JPQabK2iZJS=K+=mQ9dnu4Vm1uaQ@mail.gmail.com","subject":"Re: [PATCH 0/6] inotify support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-19T23:45:17Z","receivedAt":"2014-02-19T23:45:17Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Feb 20, 2014 at 3:35 AM, Shawn Pearce wrote:\n> Why a new daemon? Why don't we reuse the stable\n> https://github.com/facebook/watchman project Facebook built to make\n> Hg's status system fast?\n\nI did look briefly through its readme before but there were a few\noff-factors like CLA, JSON.. that made me go away. I agree that\nreusing the same daemon would save us maintenance code and buy the\nstability (and maybe sharing some inotify handles for users that use\nboth git and hg). I'll need to have a closer look at it and compare\nwith what I've got, now that I have a better picture of how things\nshould work.\n-- \nDuy\n"}]}