{"thread":{"id":"56518","subject":"[PATCH 0/5] Builtin FSMonitor Part 2","startedAt":"2021-09-16T19:54:28Z","lastAt":"2022-03-28T16:28:07Z","messageCount":298,"participants":["Jeff Hostetler via GitGitGadget","Bagas Sanjaya","Junio C Hamano","Jeff Hostetler","Ævar Arnfjörð Bjarmason","Eric Sunshine","Johannes Schindelin","Tao Klerks","rsbecker@nexbridge.com"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"436151","messageId":"pull.1041.git.1631822063.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":null,"subject":"[PATCH 0/5] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-16T19:54:18Z","receivedAt":"2021-09-16T19:54:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is Part 2 of my Builtin FSMonitor series. Part 2 requires\njh/builtin-fsmonitor-part1 be present.\n\nPart 2 contains:\n\n * Client-side code to connect to a (yet to be developed) FSMonitor daemon\n over simple IPC (using Named Pipes or Unix Domain Sockets).\n * Updated config settings to allow the client to decide whether to use\n Simple IPC or the existing FSMonitor Hook API.\n\nThe original FSMonitor feature \"e05336bdda (Merge branch 'bp/fsmonitor',\n2017-11-21)\" consisted of two parts:\n\n 1. Client code: 1.1. Called by commands like git status 1.2. Uses the Hook\n API to spawn and talk to a shell or perl script (usually) 1.3. Receives\n a list of recently modified files and/or directories 1.4. Updates flags\n and/or loops in various algorithms to avoid unnecessary lstat() calls on\n unchanged ones.\n\n 2. Hook code: 2.1. A hook process, spawned by each client command, talks to\n a long-running FS monitoring process. 2.2. A long-running FS monitoring\n process, usually Watchman. 2.3. The hook process acts as a relay to\n translate requests responses from Watchman back to the client command.\n\nThe Builtin FSMonitor feature extends step 1.2 and allows clients to\ncommunicate directly with a FS daemon process using Simple IPC. The daemon\nresponse is then processed as before in step 1.3.\n\nPart 2 contains these client-side changes for IPC. A builtin FSMonitor\ndaemon will be added in a later patch series.\n\nJeff Hostetler (5):\n fsmonitor: enhance existing comments\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor: update fsmonitor config documentation\n\n Documentation/config/core.txt | 56 ++++++---\n Documentation/git-update-index.txt | 27 +++--\n Documentation/githooks.txt | 3 +-\n Makefile | 2 +\n builtin/update-index.c | 19 +++-\n cache.h | 1 -\n config.c | 14 ---\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-ipc.c | 176 +++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 ++++++++\n fsmonitor-settings.c | 97 ++++++++++++++++\n fsmonitor-settings.h | 21 ++++\n fsmonitor.c | 130 +++++++++++++++------\n fsmonitor.h | 18 ++-\n repo-settings.c | 2 +\n repository.h | 3 +\n t/README | 4 +-\n 18 files changed, 538 insertions(+), 85 deletions(-)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\n\nbase-commit: 8b7c11b8668b4e774f81a9f0b4c30144b818f1d1\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1041\n-- \ngitgitgadget\n"},{"id":"436152","messageId":"6be687ba39d49f56a0729d3197ecbf3a64041357.1631822063.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.git.1631822063.gitgitgadget@gmail.com","subject":"[PATCH 1/5] fsmonitor: enhance existing comments","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-16T19:54:19Z","receivedAt":"2021-09-16T19:54:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 37 ++++++++++++++++++++++++++++++-------\n 1 file changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..ec4c46407c5 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -301,9 +301,25 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n \tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -318,11 +334,15 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We received a trivial response, so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -330,7 +350,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"436153","messageId":"b584f133d280e002a1c05860168c275abb184119.1631822063.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.git.1631822063.gitgitgadget@gmail.com","subject":"[PATCH 2/5] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-16T19:54:20Z","receivedAt":"2021-09-16T19:54:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 +++++++++++++\n 3 files changed, 225 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 429c276058d..79e72ca9977 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -901,6 +901,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..ccc32d2a17e\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,176 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\n+\t\tif (fsmonitor_is_trivial_response(answer))\n+\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(\"fsmonitor--daemon is not running\");\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\", c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..b6a7067c3af\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"436154","messageId":"f1266c99adfc45a0570f08682b93ca43a3b7e4e4.1631822063.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.git.1631822063.gitgitgadget@gmail.com","subject":"[PATCH 3/5] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-16T19:54:21Z","receivedAt":"2021-09-16T19:54:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove FSMonitor config settings to a new `struct fsmonitor_settings`\nstructure. Add a lazily-loaded pointer to `struct repo_settings`.\nCreate `fsm_settings__get_*()` getters to lazily look up fsmonitor-\nrelated config settings.\n\nGet rid of the `core_fsmonitor` global variable, and add support for\nthe new `core.useBuiltinFSMonitor` config setting. Move config code\nto lookup the existing `core.fsmonitor` value to `fsmonitor-settings.[ch]`.\n\nThe `core_fsmonitor` global variable was used to store the pathname to\nthe FSMonitor hook and it was used as a boolean to see if FSMonitor\nwas enabled. This dual usage will lead to confusion when we add\nsupport for a builtin FSMonitor based on IPC, since the builtin\nFSMonitor doesn't need the hook pathname.\n\nReplace the boolean usage with an `enum fsmonitor_mode` to represent\nthe state of FSMonitor. And only set the pathname when in HOOK mode.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n builtin/update-index.c | 19 +++++++--\n cache.h | 1 -\n config.c | 14 ------\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-settings.c | 97 ++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 21 +++++++++\n fsmonitor.c | 63 ++++++++++++++++-----------\n fsmonitor.h | 18 ++++++--\n repo-settings.c | 2 +\n repository.h | 3 ++\n t/README | 4 +-\n 13 files changed, 194 insertions(+), 51 deletions(-)\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\ndiff --git a/Makefile b/Makefile\nindex 79e72ca9977..8b5d241df72 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -902,6 +902,7 @@ LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += fsmonitor-ipc.o\n+LIB_OBJS += fsmonitor-settings.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 187203e8bb5..79db3ff37e2 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1214,14 +1214,25 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is unset; \"\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"builtin fsmonitor\"));\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n-\t\t\t\t\"set it if you really want to \"\n-\t\t\t\t\"enable fsmonitor\"));\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"hook-based fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode == FSMONITOR_MODE_IPC)\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is set; \"\n+\t\t\t\t\"remove it if you really want to \"\n+\t\t\t\t\"disable fsmonitor\"));\n+\t\tif (fsm_mode == FSMONITOR_MODE_HOOK)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex d23de693680..fa83dc2aa84 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -990,7 +990,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex cb4a8058bff..7389ddf18f6 100644\n--- a/config.c\n+++ b/config.c\n@@ -2517,20 +2517,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex a2200f31115..110c426b082 100644\n--- a/config.h\n+++ b/config.h\n@@ -609,7 +609,6 @@ int git_config_get_index_threads(int *dest);\n int git_config_get_untracked_cache(void);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex d6b22ede7ea..e6b66315284 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nnew file mode 100644\nindex 00000000000..2770266f5ee\n--- /dev/null\n+++ b/fsmonitor-settings.c\n@@ -0,0 +1,97 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+/*\n+ * We keep this structure defintion private and have getters\n+ * for all fields so that we can lazy load it as needed.\n+ */\n+struct fsmonitor_settings {\n+\tenum fsmonitor_mode mode;\n+\tchar *hook_path;\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_IPC;\n+}\n+\n+void fsm_settings__set_hook(struct repository *r, const char *path)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_HOOK;\n+\ts->hook_path = strdup(path);\n+}\n+\n+void fsm_settings__set_disabled(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\tFREE_AND_NULL(s->hook_path);\n+}\n+\n+static int check_for_ipc(struct repository *r)\n+{\n+\tint value;\n+\n+\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) &&\n+\t value) {\n+\t\tfsm_settings__set_ipc(r);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int check_for_hook(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n+\n+\tif (const_str && *const_str) {\n+\t\tfsm_settings__set_hook(r, const_str);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\n+\tr->settings.fsmonitor = s;\n+\n+\tif (check_for_ipc(r))\n+\t\treturn;\n+\n+\tif (check_for_hook(r))\n+\t\treturn;\n+\n+\tfsm_settings__set_disabled(r);\n+}\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n+{\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->mode;\n+}\n+\n+const char *fsm_settings__get_hook_path(struct repository *r)\n+{\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->hook_path;\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nnew file mode 100644\nindex 00000000000..50b29234616\n--- /dev/null\n+++ b/fsmonitor-settings.h\n@@ -0,0 +1,21 @@\n+#ifndef FSMONITOR_SETTINGS_H\n+#define FSMONITOR_SETTINGS_H\n+\n+struct repository;\n+\n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n+\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r);\n+void fsm_settings__set_hook(struct repository *r, const char *path);\n+void fsm_settings__set_disabled(struct repository *r);\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+const char *fsm_settings__get_hook_path(struct repository *r);\n+\n+struct fsmonitor_settings;\n+\n+#endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ec4c46407c5..63174630c0e 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tstrvec_push(&cp.args, fsm_settings__get_hook_path(r));\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -238,17 +242,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (fsm_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -256,13 +271,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -292,13 +308,17 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t}\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tfsm_settings__get_hook_path(r));\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t fsm_settings__get_hook_path(r),\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/*\n@@ -434,7 +454,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tint fsmonitor_enabled = (fsm_settings__get_mode(r) > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -454,16 +475,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..f9201411aa7 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -3,6 +3,7 @@\n \n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"fsmonitor-settings.h\"\n \n extern struct trace_key trace_fsmonitor;\n \n@@ -57,7 +58,11 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +72,11 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +92,10 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 0cfe8b787db..e69dad1e776 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -26,6 +26,8 @@ void prepare_repo_settings(struct repository *r)\n \tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n \n+\tr->settings.fsmonitor = NULL; /* lazy loaded */\n+\n \tif (!repo_config_get_int(r, \"index.version\", &value))\n \t\tr->settings.index_version = value;\n \tif (!repo_config_get_maybe_bool(r, \"core.untrackedcache\", &value)) {\ndiff --git a/repository.h b/repository.h\nindex 3740c93bc0f..fdc0a818b83 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n #include \"path.h\"\n \n struct config_set;\n+struct fsmonitor_settings;\n struct git_hash_algo;\n struct index_state;\n struct lock_file;\n@@ -35,6 +36,8 @@ struct repo_settings {\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \n+\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex 9e701223020..7be3662e038 100644\n--- a/t/README\n+++ b/t/README\n@@ -398,8 +398,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code path for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\n-- \ngitgitgadget\n\n"},{"id":"436155","messageId":"f2e5da9e5c622d9142baafee47c3e1cdffbef666.1631822063.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.git.1631822063.gitgitgadget@gmail.com","subject":"[PATCH 4/5] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-16T19:54:22Z","receivedAt":"2021-09-16T19:54:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.useBuiltinFSMonitor` is set.\n\nThe `core.fsmonitor` setting has already been defined as a HOOK\npathname. Historically, this has been set to a HOOK script that will\ntalk with Watchman. For compatibility reasons, we do not want to\noverload that definition (and cause problems if users have multiple\nversions of Git installed).\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 34 ++++++++++++++++++++++++++++++++--\n 1 file changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 63174630c0e..695fb0ce4e7 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -254,8 +254,37 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (fsm_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\tgoto apply_results;\n \t}\n \n \tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n@@ -321,6 +350,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/*\n \t * The response from FSMonitor (excluding the header token) is\n \t * either:\n-- \ngitgitgadget\n\n"},{"id":"436156","messageId":"fb5251e47ae6c8132d4a5a3c01bdfc33525f625d.1631822063.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.git.1631822063.gitgitgadget@gmail.com","subject":"[PATCH 5/5] fsmonitor: update fsmonitor config documentation","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-16T19:54:23Z","receivedAt":"2021-09-16T19:54:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to mention the new `core.useBuiltinFSMonitor`\nvalue.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/config/core.txt | 56 ++++++++++++++++++++++--------\n Documentation/git-update-index.txt | 27 +++++++-------\n Documentation/githooks.txt | 3 +-\n 3 files changed, 59 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..4f6e519bc02 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,50 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n+\n+core.useBuiltinFSMonitor::\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+Note: if this config setting is set to `true`, the values of\n+`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..c7c31b3fcf9 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory\n looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n-the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n-across all repositories you use, because you can set the configuration\n-variable in your `$HOME/.gitconfig` just once and have it affect all\n-repositories you touch.\n-\n-When the `core.fsmonitor` configuration variable is changed, the\n-file system monitor is added to or removed from the index the next time\n-a command reads the index. When `--[no-]fsmonitor` are used, the file\n-system monitor is immediately added to or removed from the index.\n+the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable (see linkgit:git-config[1]) than using the `--fsmonitor`\n+option to `git update-index` in each repository, especially if you\n+want to do so across all repositories you use, because you can set the\n+configuration variable in your `$HOME/.gitconfig` just once and have\n+it affect all repositories you touch.\n+\n+When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable is changed, the file system monitor is added to or removed\n+from the index the next time a command reads the index. When\n+`--[no-]fsmonitor` are used, the file system monitor is immediately\n+added to or removed from the index.\n \n CONFIGURATION\n -------------\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex b51959ff941..b7d5e926f7b 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -593,7 +593,8 @@ fsmonitor-watchman\n \n This hook is invoked when the configuration option `core.fsmonitor` is\n set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n-depending on the version of the hook to use.\n+depending on the version of the hook to use, unless overridden via\n+`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n \n Version 1 takes two arguments, a version (1) and the time in elapsed\n nanoseconds since midnight, January 1, 1970.\n-- \ngitgitgadget\n"},{"id":"436209","messageId":"2b7f0885-bb76-d695-941d-3160a8d62ccc@gmail.com","threadId":"56518","inReplyTo":"6be687ba39d49f56a0729d3197ecbf3a64041357.1631822063.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/5] fsmonitor: enhance existing comments","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-09-17T06:31:54Z","receivedAt":"2021-09-17T06:31:59Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 17/09/21 02.54, Jeff Hostetler via GitGitGadget wrote:\n> -\t\t/* If we're going to check every file, ensure we save the results */\n> +\t\t/*\n> +\t\t * If we're going to check every file, ensure we save\n> +\t\t * the results.\n> +\t\t */\n\nWhy did you split the comment above?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"436211","messageId":"xmqqr1dnitve.fsf@gitster.g","threadId":"56518","inReplyTo":"2b7f0885-bb76-d695-941d-3160a8d62ccc@gmail.com","subject":"Re: [PATCH 1/5] fsmonitor: enhance existing comments","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-17T06:44:21Z","receivedAt":"2021-09-17T06:44:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Bagas Sanjaya writes:\n\n> On 17/09/21 02.54, Jeff Hostetler via GitGitGadget wrote:\n>> -\t\t/* If we're going to check every file, ensure we save the results */\n>> +\t\t/*\n>> +\t\t * If we're going to check every file, ensure we save\n>> +\t\t * the results.\n>> +\t\t */\n>\n> Why did you split the comment above?\n\nI would guess that the reason why it is done is because the original\nline is overly long it (extends to 84 columns, if I am counting\ncorrectly).\n"},{"id":"436857","messageId":"f24512e5-2828-2bf1-67c8-493ecb3d73b6@jeffhostetler.com","threadId":"56518","inReplyTo":"xmqqr1dnitve.fsf@gitster.g","subject":"Re: [PATCH 1/5] fsmonitor: enhance existing comments","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-09-23T14:11:33Z","receivedAt":"2021-09-23T14:11:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 9/17/21 2:44 AM, Junio C Hamano wrote:\n> Bagas Sanjaya writes:\n> \n>> On 17/09/21 02.54, Jeff Hostetler via GitGitGadget wrote:\n>>> -\t\t/* If we're going to check every file, ensure we save the results */\n>>> +\t\t/*\n>>> +\t\t * If we're going to check every file, ensure we save\n>>> +\t\t * the results.\n>>> +\t\t */\n>>\n>> Why did you split the comment above?\n> \n> I would guess that the reason why it is done is because the original\n> line is overly long it (extends to 84 columns, if I am counting\n> correctly).\n> \n\nYes, I just wrapped it because it was too long and the commit\nwas focused on improving other nearby comments (and no code),\nso it seemed like a good opportunity to cleanup this one too.\n\nJeff\n"},{"id":"438163","messageId":"pull.1041.v2.git.1633614772.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.git.1631822063.gitgitgadget@gmail.com","subject":"[PATCH v2 0/5] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T13:52:47Z","receivedAt":"2021-10-07T13:53:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V2 of Part 2 of my Builtin FSMonitor series.\n\nThis version has been rebased onto \"next\" because it collided with a change\n\"repo-settings.c\" from \"ab/repo-settings-cleanup\". And the third commit was\nupdated to resolve those conflicts.\n\nIt also requires \"jh/builtin-fsmonitor-part1\" which is now in \"next\".\n\nPart 2 contains:\n\nClient-side code to connect to a (yet to be developed) FSMonitor daemon over\nSimple IPC (using Named Pipes or Unix Domain Sockets).\n\nUpdated config settings to allow the client to decide whether to use Simple\nIPC or the existing FSMonitor Hook API. This includes retiring the existing\n\"core_fsmonitor\" global variable that had confusing double-duty.\n\nPart 2 essentially enables client-side code to talk to different types of\nproviders of FS change data. Processing of that data is unchanged.\n\nA builtin FSMonitor daemon will be added in a later patch series.\n\nJeff Hostetler (5):\n fsmonitor: enhance existing comments\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor: update fsmonitor config documentation\n\n Documentation/config/core.txt | 56 ++++++---\n Documentation/git-update-index.txt | 27 +++--\n Documentation/githooks.txt | 3 +-\n Makefile | 2 +\n builtin/update-index.c | 19 +++-\n cache.h | 1 -\n config.c | 14 ---\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-ipc.c | 176 +++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 ++++++++\n fsmonitor-settings.c | 97 ++++++++++++++++\n fsmonitor-settings.h | 21 ++++\n fsmonitor.c | 130 +++++++++++++++------\n fsmonitor.h | 18 ++-\n repo-settings.c | 2 +\n repository.h | 3 +\n t/README | 4 +-\n 18 files changed, 538 insertions(+), 85 deletions(-)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\n\nbase-commit: 6e70778dc91e2139466c15ff15a02a22a2ada2d1\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1041\n\nRange-diff vs v1:\n\n 1: 6be687ba39d = 1: cb25eeaf72d fsmonitor: enhance existing comments\n 2: b584f133d28 = 2: df81a63acee fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n 3: f1266c99adf ! 3: 7d5a353e74d fsmonitor: config settings are repository-specific\n @@ config.c: int git_config_get_max_percent_split_change(void)\n \tint is_bool, val;\n \n ## config.h ##\n -@@ config.h: int git_config_get_index_threads(int *dest);\n - int git_config_get_untracked_cache(void);\n +@@ config.h: int git_config_get_pathname(const char *key, const char **dest);\n + int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n -int git_config_get_fsmonitor(void);\n @@ fsmonitor.h: static inline void mark_fsmonitor_valid(struct index_state *istate,\n \n ## repo-settings.c ##\n @@ repo-settings.c: void prepare_repo_settings(struct repository *r)\n - \tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n - \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n + \tif (r->settings.initialized++)\n + \t\treturn;\n \n +\tr->settings.fsmonitor = NULL; /* lazy loaded */\n +\n - \tif (!repo_config_get_int(r, \"index.version\", &value))\n - \t\tr->settings.index_version = value;\n - \tif (!repo_config_get_maybe_bool(r, \"core.untrackedcache\", &value)) {\n + \t/* Defaults */\n + \tr->settings.index_version = -1;\n + \tr->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;\n \n ## repository.h ##\n @@\n @@ repository.h\n struct index_state;\n struct lock_file;\n @@ repository.h: struct repo_settings {\n - \tint gc_write_commit_graph;\n - \tint fetch_write_commit_graph;\n + \tint command_requires_full_index;\n + \tint sparse_index;\n \n +\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n +\n 4: f2e5da9e5c6 = 4: 8608c8718d8 fsmonitor: use IPC to query the builtin FSMonitor daemon\n 5: fb5251e47ae = 5: 7c22ce53377 fsmonitor: update fsmonitor config documentation\n\n-- \ngitgitgadget\n"},{"id":"438164","messageId":"cb25eeaf72dfe25bfa62bd550a0cabf8817bf914.1633614772.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v2.git.1633614772.gitgitgadget@gmail.com","subject":"[PATCH v2 1/5] fsmonitor: enhance existing comments","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T13:52:48Z","receivedAt":"2021-10-07T13:53:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 37 ++++++++++++++++++++++++++++++-------\n 1 file changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..ec4c46407c5 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -301,9 +301,25 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n \tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -318,11 +334,15 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We received a trivial response, so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -330,7 +350,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"438165","messageId":"df81a63aceec1b27efce7d7646afe9b0a48b8e59.1633614772.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v2.git.1633614772.gitgitgadget@gmail.com","subject":"[PATCH v2 2/5] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T13:52:49Z","receivedAt":"2021-10-07T13:53:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 +++++++++++++\n 3 files changed, 225 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 381bed2c1d2..d51fd8b33ce 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -897,6 +897,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..ccc32d2a17e\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,176 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\n+\t\tif (fsmonitor_is_trivial_response(answer))\n+\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(\"fsmonitor--daemon is not running\");\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\", c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..b6a7067c3af\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"438166","messageId":"7d5a353e74d2f2a1ed3f9275425bb564265323d6.1633614772.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v2.git.1633614772.gitgitgadget@gmail.com","subject":"[PATCH v2 3/5] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T13:52:50Z","receivedAt":"2021-10-07T13:53:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove FSMonitor config settings to a new `struct fsmonitor_settings`\nstructure. Add a lazily-loaded pointer to `struct repo_settings`.\nCreate `fsm_settings__get_*()` getters to lazily look up fsmonitor-\nrelated config settings.\n\nGet rid of the `core_fsmonitor` global variable, and add support for\nthe new `core.useBuiltinFSMonitor` config setting. Move config code\nto lookup the existing `core.fsmonitor` value to `fsmonitor-settings.[ch]`.\n\nThe `core_fsmonitor` global variable was used to store the pathname to\nthe FSMonitor hook and it was used as a boolean to see if FSMonitor\nwas enabled. This dual usage will lead to confusion when we add\nsupport for a builtin FSMonitor based on IPC, since the builtin\nFSMonitor doesn't need the hook pathname.\n\nReplace the boolean usage with an `enum fsmonitor_mode` to represent\nthe state of FSMonitor. And only set the pathname when in HOOK mode.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n builtin/update-index.c | 19 +++++++--\n cache.h | 1 -\n config.c | 14 ------\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-settings.c | 97 ++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 21 +++++++++\n fsmonitor.c | 63 ++++++++++++++++-----------\n fsmonitor.h | 18 ++++++--\n repo-settings.c | 2 +\n repository.h | 3 ++\n t/README | 4 +-\n 13 files changed, 194 insertions(+), 51 deletions(-)\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\ndiff --git a/Makefile b/Makefile\nindex d51fd8b33ce..29ed7c4aba6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -898,6 +898,7 @@ LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += fsmonitor-ipc.o\n+LIB_OBJS += fsmonitor-settings.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 187203e8bb5..79db3ff37e2 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1214,14 +1214,25 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is unset; \"\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"builtin fsmonitor\"));\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n-\t\t\t\t\"set it if you really want to \"\n-\t\t\t\t\"enable fsmonitor\"));\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"hook-based fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode == FSMONITOR_MODE_IPC)\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is set; \"\n+\t\t\t\t\"remove it if you really want to \"\n+\t\t\t\t\"disable fsmonitor\"));\n+\t\tif (fsm_mode == FSMONITOR_MODE_HOOK)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex d092820c943..8f4e3c8bd1d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -989,7 +989,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex 2dcbe901b6b..6b6e9cacac3 100644\n--- a/config.c\n+++ b/config.c\n@@ -2502,20 +2502,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex f119de01309..69d733824a0 100644\n--- a/config.h\n+++ b/config.h\n@@ -610,7 +610,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex 9da7f3c1a19..68f90632245 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -82,7 +82,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nnew file mode 100644\nindex 00000000000..2770266f5ee\n--- /dev/null\n+++ b/fsmonitor-settings.c\n@@ -0,0 +1,97 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+/*\n+ * We keep this structure defintion private and have getters\n+ * for all fields so that we can lazy load it as needed.\n+ */\n+struct fsmonitor_settings {\n+\tenum fsmonitor_mode mode;\n+\tchar *hook_path;\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_IPC;\n+}\n+\n+void fsm_settings__set_hook(struct repository *r, const char *path)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_HOOK;\n+\ts->hook_path = strdup(path);\n+}\n+\n+void fsm_settings__set_disabled(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\tFREE_AND_NULL(s->hook_path);\n+}\n+\n+static int check_for_ipc(struct repository *r)\n+{\n+\tint value;\n+\n+\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) &&\n+\t value) {\n+\t\tfsm_settings__set_ipc(r);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int check_for_hook(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n+\n+\tif (const_str && *const_str) {\n+\t\tfsm_settings__set_hook(r, const_str);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\n+\tr->settings.fsmonitor = s;\n+\n+\tif (check_for_ipc(r))\n+\t\treturn;\n+\n+\tif (check_for_hook(r))\n+\t\treturn;\n+\n+\tfsm_settings__set_disabled(r);\n+}\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n+{\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->mode;\n+}\n+\n+const char *fsm_settings__get_hook_path(struct repository *r)\n+{\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->hook_path;\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nnew file mode 100644\nindex 00000000000..50b29234616\n--- /dev/null\n+++ b/fsmonitor-settings.h\n@@ -0,0 +1,21 @@\n+#ifndef FSMONITOR_SETTINGS_H\n+#define FSMONITOR_SETTINGS_H\n+\n+struct repository;\n+\n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n+\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r);\n+void fsm_settings__set_hook(struct repository *r, const char *path);\n+void fsm_settings__set_disabled(struct repository *r);\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+const char *fsm_settings__get_hook_path(struct repository *r);\n+\n+struct fsmonitor_settings;\n+\n+#endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ec4c46407c5..63174630c0e 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tstrvec_push(&cp.args, fsm_settings__get_hook_path(r));\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -238,17 +242,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (fsm_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -256,13 +271,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -292,13 +308,17 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t}\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tfsm_settings__get_hook_path(r));\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t fsm_settings__get_hook_path(r),\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/*\n@@ -434,7 +454,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tint fsmonitor_enabled = (fsm_settings__get_mode(r) > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -454,16 +475,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..f9201411aa7 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -3,6 +3,7 @@\n \n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"fsmonitor-settings.h\"\n \n extern struct trace_key trace_fsmonitor;\n \n@@ -57,7 +58,11 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +72,11 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +92,10 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex b93e91a212e..0ca85f54bb8 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -20,6 +20,8 @@ void prepare_repo_settings(struct repository *r)\n \tif (r->settings.initialized++)\n \t\treturn;\n \n+\tr->settings.fsmonitor = NULL; /* lazy loaded */\n+\n \t/* Defaults */\n \tr->settings.index_version = -1;\n \tr->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;\ndiff --git a/repository.h b/repository.h\nindex a057653981c..89a1873ade7 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n #include \"path.h\"\n \n struct config_set;\n+struct fsmonitor_settings;\n struct git_hash_algo;\n struct index_state;\n struct lock_file;\n@@ -34,6 +35,8 @@ struct repo_settings {\n \tint command_requires_full_index;\n \tint sparse_index;\n \n+\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex b92155a822e..6dc4a1d10cf 100644\n--- a/t/README\n+++ b/t/README\n@@ -405,8 +405,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code path for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\n-- \ngitgitgadget\n\n"},{"id":"438167","messageId":"8608c8718d8f839a1ae3e6d4f8f8cc2247c26111.1633614772.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v2.git.1633614772.gitgitgadget@gmail.com","subject":"[PATCH v2 4/5] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T13:52:51Z","receivedAt":"2021-10-07T13:53:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.useBuiltinFSMonitor` is set.\n\nThe `core.fsmonitor` setting has already been defined as a HOOK\npathname. Historically, this has been set to a HOOK script that will\ntalk with Watchman. For compatibility reasons, we do not want to\noverload that definition (and cause problems if users have multiple\nversions of Git installed).\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 34 ++++++++++++++++++++++++++++++++--\n 1 file changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 63174630c0e..695fb0ce4e7 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -254,8 +254,37 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (fsm_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\tgoto apply_results;\n \t}\n \n \tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n@@ -321,6 +350,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/*\n \t * The response from FSMonitor (excluding the header token) is\n \t * either:\n-- \ngitgitgadget\n\n"},{"id":"438168","messageId":"7c22ce533776ad0549551024cb9fa87c8b10947e.1633614772.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v2.git.1633614772.gitgitgadget@gmail.com","subject":"[PATCH v2 5/5] fsmonitor: update fsmonitor config documentation","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T13:52:52Z","receivedAt":"2021-10-07T13:53:14Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to mention the new `core.useBuiltinFSMonitor`\nvalue.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/config/core.txt | 56 ++++++++++++++++++++++--------\n Documentation/git-update-index.txt | 27 +++++++-------\n Documentation/githooks.txt | 3 +-\n 3 files changed, 59 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..4f6e519bc02 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,50 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n+\n+core.useBuiltinFSMonitor::\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+Note: if this config setting is set to `true`, the values of\n+`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..c7c31b3fcf9 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory\n looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n-the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n-across all repositories you use, because you can set the configuration\n-variable in your `$HOME/.gitconfig` just once and have it affect all\n-repositories you touch.\n-\n-When the `core.fsmonitor` configuration variable is changed, the\n-file system monitor is added to or removed from the index the next time\n-a command reads the index. When `--[no-]fsmonitor` are used, the file\n-system monitor is immediately added to or removed from the index.\n+the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable (see linkgit:git-config[1]) than using the `--fsmonitor`\n+option to `git update-index` in each repository, especially if you\n+want to do so across all repositories you use, because you can set the\n+configuration variable in your `$HOME/.gitconfig` just once and have\n+it affect all repositories you touch.\n+\n+When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable is changed, the file system monitor is added to or removed\n+from the index the next time a command reads the index. When\n+`--[no-]fsmonitor` are used, the file system monitor is immediately\n+added to or removed from the index.\n \n CONFIGURATION\n -------------\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex b51959ff941..b7d5e926f7b 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -593,7 +593,8 @@ fsmonitor-watchman\n \n This hook is invoked when the configuration option `core.fsmonitor` is\n set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n-depending on the version of the hook to use.\n+depending on the version of the hook to use, unless overridden via\n+`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n \n Version 1 takes two arguments, a version (1) and the time in elapsed\n nanoseconds since midnight, January 1, 1970.\n-- \ngitgitgadget\n"},{"id":"438180","messageId":"877deohiad.fsf@evledraar.gmail.com","threadId":"56518","inReplyTo":"7d5a353e74d2f2a1ed3f9275425bb564265323d6.1633614772.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/5] fsmonitor: config settings are repository-specific","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-07T16:59:07Z","receivedAt":"2021-10-07T17:11:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 07 2021, Jeff Hostetler via GitGitGadget wrote:\n\nGood to see this move forward!\n\nThis bit:\n\n> --- a/repo-settings.c\n> +++ b/repo-settings.c\n> @@ -20,6 +20,8 @@ void prepare_repo_settings(struct repository *r)\n> \tif (r->settings.initialized++)\n> \t\treturn;\n> \n> +\tr->settings.fsmonitor = NULL; /* lazy loaded */\n> +\n> \t/* Defaults */\n> \tr->settings.index_version = -1;\n> \tr->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;\n> diff --git a/repository.h b/repository.h\n\nIs carried forward from v1, but with 3050b6dfc75 (repo-settings.c:\nsimplify the setup, 2021-09-21) isn't needed. It's init'd to 0/NULL\nalready, but was -1 before.\n\nSo this hunk can go, and its presence makes for confusing reading\nwithout that history, is it set before somehow? No, just working around\nolder code that's no longer there.\n\nBut also: For untracked_cache_setting and fetch_negotiation_setting\nwe've got an embedded enum in the struct, but this...\n\n> index a057653981c..89a1873ade7 100644\n> --- a/repository.h\n> +++ b/repository.h\n> @@ -4,6 +4,7 @@\n> #include \"path.h\"\n> \n> struct config_set;\n> +struct fsmonitor_settings;\n> struct git_hash_algo;\n> struct index_state;\n> struct lock_file;\n> @@ -34,6 +35,8 @@ struct repo_settings {\n> \tint command_requires_full_index;\n> \tint sparse_index;\n> \n> +\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n> +\n> \tint index_version;\n> \tenum untracked_cache_setting core_untracked_cache;\n> \n\nIs a pointer to a struct that has an \"enum fsmonitor_mode mode\", and the\ncode in fsmonitor-settings.c seems to be repeating the patterns we had\nin repo-settings.c pre-3050b6dfc75, e.g. checking whether a bool config\nvariable exists *and* is true, v.s. checking if it exists (presumably an\nexplicit false wants to override something).\n\nI haven't looked carefully, but between that & the \"char *hook_path\"\nbeing something that'll need to be made to use Emily's hook config\nseries sooner than later, can't we read/setup the initial config in\n\"repo_cfg_bool\"?\n\nThe relevant commit message just says:\n\n Move FSMonitor config settings to a new `struct fsmonitor_settings`\n structure. Add a lazily-loaded pointer to `struct repo_settings`.\n Create `fsm_settings__get_*()` getters to lazily look up fsmonitor-\n related config settings.[...]\n\nWhich I think can be paraphrased as \"Add scaffolding to repo-settings.c\nbut do config loading differently than everything there (lazily),\nbecause...\", except the \"because\" is missing :)\n\n\n"},{"id":"438347","messageId":"492b6dcb-dfae-f57b-6467-ba5a6808fc92@jeffhostetler.com","threadId":"56518","inReplyTo":"877deohiad.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 3/5] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-10-08T20:36:32Z","receivedAt":"2021-10-08T20:36:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/7/21 12:59 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Oct 07 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n> Good to see this move forward!\n> \n> This bit:\n> \n>> --- a/repo-settings.c\n>> +++ b/repo-settings.c\n>> @@ -20,6 +20,8 @@ void prepare_repo_settings(struct repository *r)\n>> \tif (r->settings.initialized++)\n>> \t\treturn;\n>> \n>> +\tr->settings.fsmonitor = NULL; /* lazy loaded */\n>> +\n>> \t/* Defaults */\n>> \tr->settings.index_version = -1;\n>> \tr->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;\n>> diff --git a/repository.h b/repository.h\n> \n> Is carried forward from v1, but with 3050b6dfc75 (repo-settings.c:\n> simplify the setup, 2021-09-21) isn't needed. It's init'd to 0/NULL\n> already, but was -1 before.\n> \n> So this hunk can go, and its presence makes for confusing reading\n> without that history, is it set before somehow? No, just working around\n> older code that's no longer there.\n\nUm, yeah, if the structure is now zero-filled on init\nrather than the weird -1, then we don't need this here.\nI'll remove in my next version. Thanks!\n\n\n> \n> But also: For untracked_cache_setting and fetch_negotiation_setting\n> we've got an embedded enum in the struct, but this...\n> \n>> index a057653981c..89a1873ade7 100644\n>> --- a/repository.h\n>> +++ b/repository.h\n>> @@ -4,6 +4,7 @@\n>> #include \"path.h\"\n>> \n>> struct config_set;\n>> +struct fsmonitor_settings;\n>> struct git_hash_algo;\n>> struct index_state;\n>> struct lock_file;\n>> @@ -34,6 +35,8 @@ struct repo_settings {\n>> \tint command_requires_full_index;\n>> \tint sparse_index;\n>> \n>> +\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n>> +\n>> \tint index_version;\n>> \tenum untracked_cache_setting core_untracked_cache;\n>> \n> \n> Is a pointer to a struct that has an \"enum fsmonitor_mode mode\", and the\n> code in fsmonitor-settings.c seems to be repeating the patterns we had\n> in repo-settings.c pre-3050b6dfc75, e.g. checking whether a bool config\n> variable exists *and* is true, v.s. checking if it exists (presumably an\n> explicit false wants to override something).\n\nMy usage is a little more complicated because the historical value\nhad double duty. And I only want to set the hook value when not\nusing the builtin and when the repo itself is compatible with being\nmonitored. Later in the series we'll disallow remote worktrees,\nfor example. I wanted to hide some of those details in my\nfsmonitor-settings.c and keep them out of repo-settings.c. And\nto avoid paying for that when not needed, I made is a lazy-load\nand hid the values behind an opaque type with access functions.\n\n\n> \n> I haven't looked carefully, but between that & the \"char *hook_path\"\n> being something that'll need to be made to use Emily's hook config\n> series sooner than later, can't we read/setup the initial config in\n> \"repo_cfg_bool\"?\n\nIt's been a while since I looked at Emily's series. I'll have to\nrevisit that later. I do wonder if the existing fsmonitor hook\nis in the same class as the other hooks or whether it is a one-off\nand should be separate, but again I need to review that work first.\n\n> \n> The relevant commit message just says:\n> \n> Move FSMonitor config settings to a new `struct fsmonitor_settings`\n> structure. Add a lazily-loaded pointer to `struct repo_settings`.\n> Create `fsm_settings__get_*()` getters to lazily look up fsmonitor-\n> related config settings.[...]\n> \n> Which I think can be paraphrased as \"Add scaffolding to repo-settings.c\n> but do config loading differently than everything there (lazily),\n> because...\", except the \"because\" is missing :)\n> \n\nLet me reword the commit message and try to explain what I was\ngoing for. Thanks!\n\nJeff\n\n\n"},{"id":"438692","messageId":"pull.1041.v3.git.1634157107.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v2.git.1633614772.gitgitgadget@gmail.com","subject":"[PATCH v3 0/5] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-13T20:31:42Z","receivedAt":"2021-10-13T20:31:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V3 of Part 2 of my Builtin FSMonitor series. Like V2, it is built\nupon \"next\" because it requires \"ab/repo-settings-cleanup\" and\n\"jh/builtin-fsmonitor-part1\" series.\n\nV3 removes the explicit initialization of r->repo_settings->fsmonitor in\nrepo-settings.c as requested. It also includes a more detailed commit\nmessage for the 3 commit to explain the rationale for putting fsmonitor\nsettings in its own source file rather than adding it repo-settings.c\n\nThere was a comment on the V2 series about integrating the fsmonitor hook\npath with the on-going \"hook config\" effort that I did not address. I think\nit would be best to do that in a follow-on if it makes sense to do so.\n\ncc: Bagas Sanjaya bagasdotme@gmail.com cc: Jeff Hostetler\ngit@jeffhostetler.com\n\nJeff Hostetler (5):\n fsmonitor: enhance existing comments\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor: update fsmonitor config documentation\n\n Documentation/config/core.txt | 56 ++++++---\n Documentation/git-update-index.txt | 27 +++--\n Documentation/githooks.txt | 3 +-\n Makefile | 2 +\n builtin/update-index.c | 19 +++-\n cache.h | 1 -\n config.c | 14 ---\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-ipc.c | 176 +++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 ++++++++\n fsmonitor-settings.c | 97 ++++++++++++++++\n fsmonitor-settings.h | 21 ++++\n fsmonitor.c | 130 +++++++++++++++------\n fsmonitor.h | 18 ++-\n repository.h | 3 +\n t/README | 4 +-\n 17 files changed, 536 insertions(+), 85 deletions(-)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\n\nbase-commit: 6e70778dc91e2139466c15ff15a02a22a2ada2d1\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1041\n\nRange-diff vs v2:\n\n 1: cb25eeaf72d = 1: cb25eeaf72d fsmonitor: enhance existing comments\n 2: df81a63acee = 2: df81a63acee fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n 3: 7d5a353e74d ! 3: a1d606aa622 fsmonitor: config settings are repository-specific\n @@ Metadata\n ## Commit message ##\n fsmonitor: config settings are repository-specific\n \n - Move FSMonitor config settings to a new `struct fsmonitor_settings`\n - structure. Add a lazily-loaded pointer to `struct repo_settings`.\n + Move fsmonitor config settings to a new and opaque\n + `struct fsmonitor_settings` structure. Add a lazily-loaded pointer\n + to this into `struct repo_settings`\n +\n + Create an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\n + represent the state of fsmonitor. This lets us represent which, if\n + any, fsmonitor provider (hook or IPC) is enabled.\n +\n Create `fsm_settings__get_*()` getters to lazily look up fsmonitor-\n related config settings.\n \n - Get rid of the `core_fsmonitor` global variable, and add support for\n - the new `core.useBuiltinFSMonitor` config setting. Move config code\n - to lookup the existing `core.fsmonitor` value to `fsmonitor-settings.[ch]`.\n + Add support for the new `core.useBuiltinFSMonitor` config setting.\n +\n + Get rid of the `core_fsmonitor` global variable. Move the code to\n + lookup the existing `core.fsmonitor` config value into the fsmonitor\n + settings.\n +\n + Create a hook pathname variable in `struct fsmonitor-settings` and\n + only set it when in hook mode.\n \n - The `core_fsmonitor` global variable was used to store the pathname to\n - the FSMonitor hook and it was used as a boolean to see if FSMonitor\n - was enabled. This dual usage will lead to confusion when we add\n - support for a builtin FSMonitor based on IPC, since the builtin\n - FSMonitor doesn't need the hook pathname.\n + The existing `core_fsmonitor` global variable was used to store the\n + pathname to the fsmonitor hook *and* it was used as a boolean to see\n + if fsmonitor was enabled. This dual usage and global visibility leads\n + to confusion when we add the IPC-based provider. So lets hide the\n + details in fsmonitor-settings.c and let it decide which provider to\n + use in the case of multiple settings. This avoids cluttering up\n + repo-settings.c with these private details.\n \n - Replace the boolean usage with an `enum fsmonitor_mode` to represent\n - the state of FSMonitor. And only set the pathname when in HOOK mode.\n + A future commit in builtin-fsmonitor series will add the ability to\n + disqualify worktrees for various reasons, such as being mounted from a\n + remote volume, where fsmonitor should not be started. Having the\n + config settings hidden in fsmonitor-settings.c allows such worktree\n + restrictions to override the config values used.\n \n Signed-off-by: Jeff Hostetler \n \n @@ fsmonitor.h: static inline void mark_fsmonitor_valid(struct index_state *istate,\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\n \n - ## repo-settings.c ##\n -@@ repo-settings.c: void prepare_repo_settings(struct repository *r)\n - \tif (r->settings.initialized++)\n - \t\treturn;\n - \n -+\tr->settings.fsmonitor = NULL; /* lazy loaded */\n -+\n - \t/* Defaults */\n - \tr->settings.index_version = -1;\n - \tr->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;\n -\n ## repository.h ##\n @@\n #include \"path.h\"\n 4: 8608c8718d8 = 4: 4d8d812be08 fsmonitor: use IPC to query the builtin FSMonitor daemon\n 5: 7c22ce53377 = 5: 45a86cef8d7 fsmonitor: update fsmonitor config documentation\n\n-- \ngitgitgadget\n"},{"id":"438693","messageId":"df81a63aceec1b27efce7d7646afe9b0a48b8e59.1634157107.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v3.git.1634157107.gitgitgadget@gmail.com","subject":"[PATCH v3 2/5] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-13T20:31:44Z","receivedAt":"2021-10-13T20:31:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 +++++++++++++\n 3 files changed, 225 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 381bed2c1d2..d51fd8b33ce 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -897,6 +897,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..ccc32d2a17e\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,176 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\n+\t\tif (fsmonitor_is_trivial_response(answer))\n+\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(\"fsmonitor--daemon is not running\");\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\", c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..b6a7067c3af\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"438694","messageId":"cb25eeaf72dfe25bfa62bd550a0cabf8817bf914.1634157107.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v3.git.1634157107.gitgitgadget@gmail.com","subject":"[PATCH v3 1/5] fsmonitor: enhance existing comments","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-13T20:31:43Z","receivedAt":"2021-10-13T20:31:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 37 ++++++++++++++++++++++++++++++-------\n 1 file changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..ec4c46407c5 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -301,9 +301,25 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n \tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -318,11 +334,15 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We received a trivial response, so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -330,7 +350,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"438695","messageId":"a1d606aa6222eb8d6c4735a6e8779dc2a910ede9.1634157108.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v3.git.1634157107.gitgitgadget@gmail.com","subject":"[PATCH v3 3/5] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-13T20:31:45Z","receivedAt":"2021-10-13T20:31:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove fsmonitor config settings to a new and opaque\n`struct fsmonitor_settings` structure. Add a lazily-loaded pointer\nto this into `struct repo_settings`\n\nCreate an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\nrepresent the state of fsmonitor. This lets us represent which, if\nany, fsmonitor provider (hook or IPC) is enabled.\n\nCreate `fsm_settings__get_*()` getters to lazily look up fsmonitor-\nrelated config settings.\n\nAdd support for the new `core.useBuiltinFSMonitor` config setting.\n\nGet rid of the `core_fsmonitor` global variable. Move the code to\nlookup the existing `core.fsmonitor` config value into the fsmonitor\nsettings.\n\nCreate a hook pathname variable in `struct fsmonitor-settings` and\nonly set it when in hook mode.\n\nThe existing `core_fsmonitor` global variable was used to store the\npathname to the fsmonitor hook *and* it was used as a boolean to see\nif fsmonitor was enabled. This dual usage and global visibility leads\nto confusion when we add the IPC-based provider. So lets hide the\ndetails in fsmonitor-settings.c and let it decide which provider to\nuse in the case of multiple settings. This avoids cluttering up\nrepo-settings.c with these private details.\n\nA future commit in builtin-fsmonitor series will add the ability to\ndisqualify worktrees for various reasons, such as being mounted from a\nremote volume, where fsmonitor should not be started. Having the\nconfig settings hidden in fsmonitor-settings.c allows such worktree\nrestrictions to override the config values used.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n builtin/update-index.c | 19 +++++++--\n cache.h | 1 -\n config.c | 14 ------\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-settings.c | 97 ++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 21 +++++++++\n fsmonitor.c | 63 ++++++++++++++++-----------\n fsmonitor.h | 18 ++++++--\n repository.h | 3 ++\n t/README | 4 +-\n 12 files changed, 192 insertions(+), 51 deletions(-)\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\ndiff --git a/Makefile b/Makefile\nindex d51fd8b33ce..29ed7c4aba6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -898,6 +898,7 @@ LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += fsmonitor-ipc.o\n+LIB_OBJS += fsmonitor-settings.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 187203e8bb5..79db3ff37e2 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1214,14 +1214,25 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is unset; \"\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"builtin fsmonitor\"));\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n-\t\t\t\t\"set it if you really want to \"\n-\t\t\t\t\"enable fsmonitor\"));\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"hook-based fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode == FSMONITOR_MODE_IPC)\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is set; \"\n+\t\t\t\t\"remove it if you really want to \"\n+\t\t\t\t\"disable fsmonitor\"));\n+\t\tif (fsm_mode == FSMONITOR_MODE_HOOK)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex d092820c943..8f4e3c8bd1d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -989,7 +989,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex 2dcbe901b6b..6b6e9cacac3 100644\n--- a/config.c\n+++ b/config.c\n@@ -2502,20 +2502,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex f119de01309..69d733824a0 100644\n--- a/config.h\n+++ b/config.h\n@@ -610,7 +610,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex 9da7f3c1a19..68f90632245 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -82,7 +82,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nnew file mode 100644\nindex 00000000000..2770266f5ee\n--- /dev/null\n+++ b/fsmonitor-settings.c\n@@ -0,0 +1,97 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+/*\n+ * We keep this structure defintion private and have getters\n+ * for all fields so that we can lazy load it as needed.\n+ */\n+struct fsmonitor_settings {\n+\tenum fsmonitor_mode mode;\n+\tchar *hook_path;\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_IPC;\n+}\n+\n+void fsm_settings__set_hook(struct repository *r, const char *path)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_HOOK;\n+\ts->hook_path = strdup(path);\n+}\n+\n+void fsm_settings__set_disabled(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\tFREE_AND_NULL(s->hook_path);\n+}\n+\n+static int check_for_ipc(struct repository *r)\n+{\n+\tint value;\n+\n+\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) &&\n+\t value) {\n+\t\tfsm_settings__set_ipc(r);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int check_for_hook(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n+\n+\tif (const_str && *const_str) {\n+\t\tfsm_settings__set_hook(r, const_str);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\n+\tr->settings.fsmonitor = s;\n+\n+\tif (check_for_ipc(r))\n+\t\treturn;\n+\n+\tif (check_for_hook(r))\n+\t\treturn;\n+\n+\tfsm_settings__set_disabled(r);\n+}\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n+{\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->mode;\n+}\n+\n+const char *fsm_settings__get_hook_path(struct repository *r)\n+{\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->hook_path;\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nnew file mode 100644\nindex 00000000000..50b29234616\n--- /dev/null\n+++ b/fsmonitor-settings.h\n@@ -0,0 +1,21 @@\n+#ifndef FSMONITOR_SETTINGS_H\n+#define FSMONITOR_SETTINGS_H\n+\n+struct repository;\n+\n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n+\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r);\n+void fsm_settings__set_hook(struct repository *r, const char *path);\n+void fsm_settings__set_disabled(struct repository *r);\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+const char *fsm_settings__get_hook_path(struct repository *r);\n+\n+struct fsmonitor_settings;\n+\n+#endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ec4c46407c5..63174630c0e 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tstrvec_push(&cp.args, fsm_settings__get_hook_path(r));\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -238,17 +242,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (fsm_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -256,13 +271,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -292,13 +308,17 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t}\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tfsm_settings__get_hook_path(r));\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t fsm_settings__get_hook_path(r),\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/*\n@@ -434,7 +454,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tint fsmonitor_enabled = (fsm_settings__get_mode(r) > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -454,16 +475,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..f9201411aa7 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -3,6 +3,7 @@\n \n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"fsmonitor-settings.h\"\n \n extern struct trace_key trace_fsmonitor;\n \n@@ -57,7 +58,11 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +72,11 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +92,10 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repository.h b/repository.h\nindex a057653981c..89a1873ade7 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n #include \"path.h\"\n \n struct config_set;\n+struct fsmonitor_settings;\n struct git_hash_algo;\n struct index_state;\n struct lock_file;\n@@ -34,6 +35,8 @@ struct repo_settings {\n \tint command_requires_full_index;\n \tint sparse_index;\n \n+\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex b92155a822e..6dc4a1d10cf 100644\n--- a/t/README\n+++ b/t/README\n@@ -405,8 +405,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code path for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\n-- \ngitgitgadget\n\n"},{"id":"438696","messageId":"4d8d812be084459b850f11c811bbfc02c4c6f8c4.1634157108.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v3.git.1634157107.gitgitgadget@gmail.com","subject":"[PATCH v3 4/5] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-13T20:31:46Z","receivedAt":"2021-10-13T20:31:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.useBuiltinFSMonitor` is set.\n\nThe `core.fsmonitor` setting has already been defined as a HOOK\npathname. Historically, this has been set to a HOOK script that will\ntalk with Watchman. For compatibility reasons, we do not want to\noverload that definition (and cause problems if users have multiple\nversions of Git installed).\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 34 ++++++++++++++++++++++++++++++++--\n 1 file changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 63174630c0e..695fb0ce4e7 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -254,8 +254,37 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (fsm_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\tgoto apply_results;\n \t}\n \n \tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n@@ -321,6 +350,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/*\n \t * The response from FSMonitor (excluding the header token) is\n \t * either:\n-- \ngitgitgadget\n\n"},{"id":"438697","messageId":"45a86cef8d75911e3448dc9e3cb3d4f7c0ee76a0.1634157108.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v3.git.1634157107.gitgitgadget@gmail.com","subject":"[PATCH v3 5/5] fsmonitor: update fsmonitor config documentation","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-13T20:31:47Z","receivedAt":"2021-10-13T20:31:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to mention the new `core.useBuiltinFSMonitor`\nvalue.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/config/core.txt | 56 ++++++++++++++++++++++--------\n Documentation/git-update-index.txt | 27 +++++++-------\n Documentation/githooks.txt | 3 +-\n 3 files changed, 59 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..4f6e519bc02 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,50 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n+\n+core.useBuiltinFSMonitor::\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+Note: if this config setting is set to `true`, the values of\n+`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..c7c31b3fcf9 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory\n looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n-the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n-across all repositories you use, because you can set the configuration\n-variable in your `$HOME/.gitconfig` just once and have it affect all\n-repositories you touch.\n-\n-When the `core.fsmonitor` configuration variable is changed, the\n-file system monitor is added to or removed from the index the next time\n-a command reads the index. When `--[no-]fsmonitor` are used, the file\n-system monitor is immediately added to or removed from the index.\n+the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable (see linkgit:git-config[1]) than using the `--fsmonitor`\n+option to `git update-index` in each repository, especially if you\n+want to do so across all repositories you use, because you can set the\n+configuration variable in your `$HOME/.gitconfig` just once and have\n+it affect all repositories you touch.\n+\n+When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable is changed, the file system monitor is added to or removed\n+from the index the next time a command reads the index. When\n+`--[no-]fsmonitor` are used, the file system monitor is immediately\n+added to or removed from the index.\n \n CONFIGURATION\n -------------\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex b51959ff941..b7d5e926f7b 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -593,7 +593,8 @@ fsmonitor-watchman\n \n This hook is invoked when the configuration option `core.fsmonitor` is\n set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n-depending on the version of the hook to use.\n+depending on the version of the hook to use, unless overridden via\n+`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n \n Version 1 takes two arguments, a version (1) and the time in elapsed\n nanoseconds since midnight, January 1, 1970.\n-- \ngitgitgadget\n"},{"id":"438840","messageId":"87lf2ueafl.fsf@evledraar.gmail.com","threadId":"56518","inReplyTo":"pull.1041.v3.git.1634157107.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/5] Builtin FSMonitor Part 2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-15T12:03:50Z","receivedAt":"2021-10-15T12:31:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 13 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> Here is V3 of Part 2 of my Builtin FSMonitor series. Like V2, it is built\n> upon \"next\" because it requires \"ab/repo-settings-cleanup\" and\n> \"jh/builtin-fsmonitor-part1\" series.\n\nFYI: Both of those have landed, so a next iteration of this can be built\non \"master\".\n\n> V3 removes the explicit initialization of r->repo_settings->fsmonitor in\n> repo-settings.c as requested. It also includes a more detailed commit\n> message for the 3 commit to explain the rationale for putting fsmonitor\n> settings in its own source file rather than adding it repo-settings.c\n\nThis series breaks with the \"make check-docs\" target, which as an aside\nis broken because of recent changes of mine, so CI didn't catch this\n(I'll submit a series to fix it):\n\n config/core.txt:95: error: git-fsmonitor--daemon[1]: link outside of our own docs, shown with 'HERE' below:\n config/core.txt:95: 'daemon for this working directory (linkgit:git-fsmonitor--daemon[1]' <-- HERE\n git-update-index.txt:502: error: git-fsmonitor--daemon[1]: link outside of our own docs, shown with 'HERE' below:\n git-update-index.txt:502: 'linkgit:git-fsmonitor--daemon[1]' <-- HERE\n\nBut that broken-ness points to a more general issue, which is that it's\nnot just a broken link, but that docs in this series are referring to a\nmanpage that doesn't exist yet.\n\nI was going to check out some of the semantics of\ncore.useBuiltinfsMonitor that I commented on in earlier rounds, but I\nsee that there's no tests for it, probably for similar reasons as there\nnot being a git-fsmonitor--daemon yet: It's all (hopefully) in some\nyet-to-come series.\n\nI'm all for this being split up in steps from the 30-something patches\nit was before, but if we're making forward-references to docs that don't\nexist yet, adding ~100 line *.c files that seemingly aren't used at all\nyet etc., the split-up seems to be a bit too aggressive.\n\nE.g. the below diff seems to have all tests passing, so\ncore.usebuiltinfsmonitor is unused still? There's also large amounts of\ncode ifdef'd away under HAVE_FSMONITOR_DAEMON_BACKEND, but no other\nin-tree reference to it?\n\nAFAICT that's going to be used in the future. So seemingly something\nthat'll only be used in the series after this? I.e. this seems to have\nstopped at around part 7/28 of a previous submission, but digging in the\narchive e.g. HAVE_FSMONITOR_DAEMON_BACKEND semes to be first used in\nstep 11/28 of that[1].\n\nSorry to be party pooper again, I really did try to review this, but\njust found that I couldn't, since so much of it seems to be a\nforward-reference to some future state.\n\nIsn't there a way to re-arrange this so that the \"teach daemon XYZ\ncommand\" around steps 13-14/28 of your previous series comes before\nreferences in the docs to the daemon that doesn't exist yet (5/5 here),\nor there's perhaps some of the config scaffolding, but\ne.g. \"core.usebuiltinfsmonitor\" comes along with the later change that\nuses it?\n\nOr just make this \"part 2\" series larger than 5 patches, so that it\nmanages to tell some holistic story, and the various bits are used at\nthe end?\n\nEven so it would be a lot easier to follow if e.g. a patch using a new\n\"core.usebuiltinfsmonitor\" variable doesn't come at say step 11/20, with\nthat variable having been added at a 3/20. I.e. maybe 3/20 can have the\nscaffolding, but the part that adds the \"core.usebuiltinfsmonitor\"\nsupport either immediately precedes its use, or is squashed into the\nrelevant commit that needs it. Ditto docs for so-far-not-existing things\netc.\n\nThanks! And hopefully this helps.\n\n1. https://lore.kernel.org/git/49f9e2e3d49ce6e7b56839bf44535f271216abeb.1621691828.git.gitgitgadget@gmail.com/\n\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 2770266f5ee..2c479d4f36b 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -35,19 +35,6 @@ void fsm_settings__set_disabled(struct repository *r)\n \tFREE_AND_NULL(s->hook_path);\n }\n \n-static int check_for_ipc(struct repository *r)\n-{\n-\tint value;\n-\n-\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) &&\n-\t value) {\n-\t\tfsm_settings__set_ipc(r);\n-\t\treturn 1;\n-\t}\n-\n-\treturn 0;\n-}\n-\n static int check_for_hook(struct repository *r)\n {\n \tconst char *const_str;\n@@ -71,9 +58,6 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \n \tr->settings.fsmonitor = s;\n \n-\tif (check_for_ipc(r))\n-\t\treturn;\n-\n \tif (check_for_hook(r))\n \t\treturn;\n \n"},{"id":"439145","messageId":"bfe8089a-1cc0-17b3-e80c-a3539ced6742@jeffhostetler.com","threadId":"56518","inReplyTo":"87lf2ueafl.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 0/5] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-10-20T21:43:59Z","receivedAt":"2021-10-20T21:44:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/15/21 8:03 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Oct 13 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n>> Here is V3 of Part 2 of my Builtin FSMonitor series. Like V2, it is built\n>> upon \"next\" because it requires \"ab/repo-settings-cleanup\" and\n>> \"jh/builtin-fsmonitor-part1\" series.\n> \n> FYI: Both of those have landed, so a next iteration of this can be built\n> on \"master\".\n> \n>> V3 removes the explicit initialization of r->repo_settings->fsmonitor in\n>> repo-settings.c as requested. It also includes a more detailed commit\n>> message for the 3 commit to explain the rationale for putting fsmonitor\n>> settings in its own source file rather than adding it repo-settings.c\n> \n> This series breaks with the \"make check-docs\" target, which as an aside\n> is broken because of recent changes of mine, so CI didn't catch this\n> (I'll submit a series to fix it):\n> \n> config/core.txt:95: error: git-fsmonitor--daemon[1]: link outside of our own docs, shown with 'HERE' below:\n> config/core.txt:95: 'daemon for this working directory (linkgit:git-fsmonitor--daemon[1]' <-- HERE\n> git-update-index.txt:502: error: git-fsmonitor--daemon[1]: link outside of our own docs, shown with 'HERE' below:\n> git-update-index.txt:502: 'linkgit:git-fsmonitor--daemon[1]' <-- HERE\n> \n> But that broken-ness points to a more general issue, which is that it's\n> not just a broken link, but that docs in this series are referring to a\n> manpage that doesn't exist yet.\n> \n> I was going to check out some of the semantics of\n> core.useBuiltinfsMonitor that I commented on in earlier rounds, but I\n> see that there's no tests for it, probably for similar reasons as there\n> not being a git-fsmonitor--daemon yet: It's all (hopefully) in some\n> yet-to-come series.\n> \n> I'm all for this being split up in steps from the 30-something patches\n> it was before, but if we're making forward-references to docs that don't\n> exist yet, adding ~100 line *.c files that seemingly aren't used at all\n> yet etc., the split-up seems to be a bit too aggressive.\n> \n> E.g. the below diff seems to have all tests passing, so\n> core.usebuiltinfsmonitor is unused still? There's also large amounts of\n> code ifdef'd away under HAVE_FSMONITOR_DAEMON_BACKEND, but no other\n> in-tree reference to it?\n> \n> AFAICT that's going to be used in the future. So seemingly something\n> that'll only be used in the series after this? I.e. this seems to have\n> stopped at around part 7/28 of a previous submission, but digging in the\n> archive e.g. HAVE_FSMONITOR_DAEMON_BACKEND semes to be first used in\n> step 11/28 of that[1].\n> \n> Sorry to be party pooper again, I really did try to review this, but\n> just found that I couldn't, since so much of it seems to be a\n> forward-reference to some future state.\n> \n> Isn't there a way to re-arrange this so that the \"teach daemon XYZ\n> command\" around steps 13-14/28 of your previous series comes before\n> references in the docs to the daemon that doesn't exist yet (5/5 here),\n> or there's perhaps some of the config scaffolding, but\n> e.g. \"core.usebuiltinfsmonitor\" comes along with the later change that\n> uses it?\n> \n> Or just make this \"part 2\" series larger than 5 patches, so that it\n> manages to tell some holistic story, and the various bits are used at\n> the end?\n> \n> Even so it would be a lot easier to follow if e.g. a patch using a new\n> \"core.usebuiltinfsmonitor\" variable doesn't come at say step 11/20, with\n> that variable having been added at a 3/20. I.e. maybe 3/20 can have the\n> scaffolding, but the part that adds the \"core.usebuiltinfsmonitor\"\n> support either immediately precedes its use, or is squashed into the\n> relevant commit that needs it. Ditto docs for so-far-not-existing things\n> etc.\n> \n> Thanks! And hopefully this helps.\n> \n> 1. https://lore.kernel.org/git/49f9e2e3d49ce6e7b56839bf44535f271216abeb.1621691828.git.gitgitgadget@gmail.com/\n> \n> diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\n> index 2770266f5ee..2c479d4f36b 100644\n> --- a/fsmonitor-settings.c\n> +++ b/fsmonitor-settings.c\n> @@ -35,19 +35,6 @@ void fsm_settings__set_disabled(struct repository *r)\n> \tFREE_AND_NULL(s->hook_path);\n> }\n> \n> -static int check_for_ipc(struct repository *r)\n> -{\n> -\tint value;\n> -\n> -\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) &&\n> -\t value) {\n> -\t\tfsm_settings__set_ipc(r);\n> -\t\treturn 1;\n> -\t}\n> -\n> -\treturn 0;\n> -}\n> -\n> static int check_for_hook(struct repository *r)\n> {\n> \tconst char *const_str;\n> @@ -71,9 +58,6 @@ static void lookup_fsmonitor_settings(struct repository *r)\n> \n> \tr->settings.fsmonitor = s;\n> \n> -\tif (check_for_ipc(r))\n> -\t\treturn;\n> -\n> \tif (check_for_hook(r))\n> \t\treturn;\n> \n> \n\nBecause of the size -- I'm up to ~60 commits now -- I was trying\nto break it up into logical pieces. And yes, there's a bit of a\nchicken-n-egg problem with introducing pieces. I was trying to make\nPart 2 be the client side -- everything needed to talk to a daemon\n(even if it doesn't exist yet) and then have Part 3 be the daemon\nproper (and build it up without worrying about when the corresponding\nclient peer code appears because the client side is already present).\nThe daemon in Part 3 would be a little rough around the edges, but\nsufficient to do end-to-end testing with both t/ and t/perf tests.\nThen Part 4 would clean up some of the details and add more tests.\n\nThat was the idea anyway.\n\nI'll take a stab at refactoring this and send a new Part 2 that\ncontains both the client and MVP daemon parts. This will let\nlet basic end-to-end testing happen, fix the doc breakage, and\nhopefully make it easier to see the big picture in a single patch\nseries. This should be about 30 commits. Once that has settled\na bit, I'll followup with a new Part 3 that contains the refinements\nand additional tests.\n\n\nThanks,\nJeff\n\n\n"},{"id":"439205","messageId":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v3.git.1634157107.gitgitgadget@gmail.com","subject":"[PATCH v4 00/29] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:40Z","receivedAt":"2021-10-21T14:25:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V4 of Part 2 of my Builtin FSMonitor series. I have rebased it onto\nthe current \"master\" branch to address conflicts and errors with recent\nchanges already in master.\n\nIn response to comments on V3, I have pulled in more commits from complete\nseries. Part 2 now includes both the client-side code to talk to via a\ndaemon via IPC and an MVP implementation of the fsmonitor--daemon itself.\n\nThis version of the daemon is functional and can be tested using t/t7527 and\nt/perf/p7519.\n\nA followup Part 3 will contain additional refinements to the daemon and\nadditional tests. I drew the line here between Part 2 and 3 to make it\neasier to review.\n\ncc: Bagas Sanjaya bagasdotme@gmail.com cc: Ævar Arnfjörð Bjarmason\navarab@gmail.com cc: Jeff Hostetler git@jeffhostetler.com\n\nJeff Hostetler (29):\n fsmonitor: enhance existing comments\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor: document builtin fsmonitor\n fsmonitor--daemon: add a built-in fsmonitor daemon\n fsmonitor--daemon: implement 'stop' and 'status' commands\n compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n fsmonitor--daemon: implement 'run' command\n fsmonitor--daemon: implement 'start' command\n fsmonitor--daemon: add pathname classification\n fsmonitor--daemon: define token-ids\n fsmonitor--daemon: create token-based changed path cache\n compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on\n Windows\n compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent\n compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on\n MacOS\n fsmonitor--daemon: implement handle_client callback\n help: include fsmonitor--daemon feature flag in version info\n t/helper/fsmonitor-client: create IPC client to talk to FSMonitor\n Daemon\n t7527: create test for fsmonitor--daemon\n t/perf: avoid copying builtin fsmonitor files into test repo\n t/helper/test-chmtime: skip directories on Windows\n t/perf/p7519: speed up test on Windows\n t/perf/p7519: add fsmonitor--daemon test cases\n fsmonitor--daemon: periodically truncate list of modified files\n fsmonitor--daemon: use a cookie file to sync with file system\n fsmonitor: force update index after large responses\n t7527: test status with untracked-cache and fsmonitor--daemon\n\n .gitignore | 1 +\n Documentation/config/core.txt | 56 +-\n Documentation/git-fsmonitor--daemon.txt | 75 ++\n Documentation/git-update-index.txt | 27 +-\n Documentation/githooks.txt | 3 +-\n Makefile | 17 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 1454 +++++++++++++++++++++++\n builtin/update-index.c | 19 +-\n cache.h | 1 -\n compat/fsmonitor/fsm-listen-darwin.c | 496 ++++++++\n compat/fsmonitor/fsm-listen-win32.c | 586 +++++++++\n compat/fsmonitor/fsm-listen.h | 49 +\n config.c | 14 -\n config.h | 1 -\n config.mak.uname | 20 +\n contrib/buildsystems/CMakeLists.txt | 10 +\n environment.c | 1 -\n fsmonitor--daemon.h | 140 +++\n fsmonitor-ipc.c | 176 +++\n fsmonitor-ipc.h | 48 +\n fsmonitor-settings.c | 97 ++\n fsmonitor-settings.h | 21 +\n fsmonitor.c | 187 ++-\n fsmonitor.h | 18 +-\n git.c | 1 +\n help.c | 4 +\n repo-settings.c | 1 +\n repository.h | 3 +\n t/README | 4 +-\n t/helper/test-chmtime.c | 15 +\n t/helper/test-fsmonitor-client.c | 121 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/perf/p7519-fsmonitor.sh | 61 +-\n t/perf/perf-lib.sh | 2 +-\n t/t7527-builtin-fsmonitor.sh | 607 ++++++++++\n t/test-lib.sh | 6 +\n 38 files changed, 4247 insertions(+), 98 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n create mode 100644 builtin/fsmonitor--daemon.c\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n create mode 100644 fsmonitor--daemon.h\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n create mode 100644 t/helper/test-fsmonitor-client.c\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\n\nbase-commit: 9d530dc0024503ab4218fe6c4395b8a0aa245478\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1041\n\nRange-diff vs v3:\n\n 1: cb25eeaf72d = 1: ecc40795fa2 fsmonitor: enhance existing comments\n 2: df81a63acee = 2: 82f17692128 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n 3: a1d606aa622 = 3: 882789b4dfe fsmonitor: config settings are repository-specific\n 4: 4d8d812be08 = 4: de82c726182 fsmonitor: use IPC to query the builtin FSMonitor daemon\n 5: 45a86cef8d7 ! 5: d365704d551 fsmonitor: update fsmonitor config documentation\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: update fsmonitor config documentation\n + fsmonitor: document builtin fsmonitor\n +\n + Document the new `core.useBuiltinFSMonitor` config value.\n \n Update references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\n - pointers to `Watchman` to mention the new `core.useBuiltinFSMonitor`\n - value.\n + pointers to `Watchman` to refer to it.\n +\n + Create `git-fsmonitor--daemon` manual page and describe its features.\n \n Signed-off-by: Jeff Hostetler \n \n @@ Documentation/config/core.txt: core.protectNTFS::\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \n + ## Documentation/git-fsmonitor--daemon.txt (new) ##\n +@@\n ++git-fsmonitor--daemon(1)\n ++========================\n ++\n ++NAME\n ++----\n ++git-fsmonitor--daemon - A Built-in File System Monitor\n ++\n ++SYNOPSIS\n ++--------\n ++[verse]\n ++'git fsmonitor--daemon' start\n ++'git fsmonitor--daemon' run\n ++'git fsmonitor--daemon' stop\n ++'git fsmonitor--daemon' status\n ++\n ++DESCRIPTION\n ++-----------\n ++\n ++A daemon to watch the working directory for file and directory\n ++changes using platform-specific file system notification facilities.\n ++\n ++This daemon communicates directly with commands like `git status`\n ++using the link:technical/api-simple-ipc.html[simple IPC] interface\n ++instead of the slower linkgit:githooks[5] interface.\n ++\n ++This daemon is built into Git so that no third-party tools are\n ++required.\n ++\n ++OPTIONS\n ++-------\n ++\n ++start::\n ++\tStarts a daemon in the background.\n ++\n ++run::\n ++\tRuns a daemon in the foreground.\n ++\n ++stop::\n ++\tStops the daemon running in the current working\n ++\tdirectory, if present.\n ++\n ++status::\n ++\tExits with zero status if a daemon is watching the\n ++\tcurrent working directory.\n ++\n ++REMARKS\n ++-------\n ++\n ++This daemon is a long running process used to watch a single working\n ++directory and maintain a list of the recently changed files and\n ++directories. Performance of commands such as `git status` can be\n ++increased if they just ask for a summary of changes to the working\n ++directory and can avoid scanning the disk.\n ++\n ++When `core.useBuiltinFSMonitor` is set to `true` (see\n ++linkgit:git-config[1]) commands, such as `git status`, will ask the\n ++daemon for changes and automatically start it (if necessary).\n ++\n ++For more information see the \"File System Monitor\" section in\n ++linkgit:git-update-index[1].\n ++\n ++CAVEATS\n ++-------\n ++\n ++The fsmonitor daemon does not currently know about submodules and does\n ++not know to filter out file system events that happen within a\n ++submodule. If fsmonitor daemon is watching a super repo and a file is\n ++modified within the working directory of a submodule, it will report\n ++the change (as happening against the super repo). However, the client\n ++will properly ignore these extra events, so performance may be affected\n ++but it will not cause an incorrect result.\n ++\n ++GIT\n ++---\n ++Part of the linkgit:git[1] suite\n +\n ## Documentation/git-update-index.txt ##\n @@ Documentation/git-update-index.txt: FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n -: ----------- > 6: 78e682fc530 fsmonitor--daemon: add a built-in fsmonitor daemon\n -: ----------- > 7: ea64b5c9753 fsmonitor--daemon: implement 'stop' and 'status' commands\n -: ----------- > 8: 5a40b33a00c compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n -: ----------- > 9: ed5819e29f8 compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n -: ----------- > 10: d3ac973a5f1 fsmonitor--daemon: implement 'run' command\n -: ----------- > 11: d08c28b549c fsmonitor--daemon: implement 'start' command\n -: ----------- > 12: 6fa71fdc825 fsmonitor--daemon: add pathname classification\n -: ----------- > 13: 65821da5b03 fsmonitor--daemon: define token-ids\n -: ----------- > 14: 429c48a5bad fsmonitor--daemon: create token-based changed path cache\n -: ----------- > 15: b04c460c619 compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows\n -: ----------- > 16: 862bbfcc32e compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent\n -: ----------- > 17: 40d9a816b52 compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS\n -: ----------- > 18: 241962894f1 fsmonitor--daemon: implement handle_client callback\n -: ----------- > 19: 704d37d2033 help: include fsmonitor--daemon feature flag in version info\n -: ----------- > 20: de6c72a9ce0 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n -: ----------- > 21: eedaa787c2e t7527: create test for fsmonitor--daemon\n -: ----------- > 22: 4e96e0667ba t/perf: avoid copying builtin fsmonitor files into test repo\n -: ----------- > 23: de9c015d78c t/helper/test-chmtime: skip directories on Windows\n -: ----------- > 24: 1c2eccacff6 t/perf/p7519: speed up test on Windows\n -: ----------- > 25: 236b5966257 t/perf/p7519: add fsmonitor--daemon test cases\n -: ----------- > 26: 54710a4830d fsmonitor--daemon: periodically truncate list of modified files\n -: ----------- > 27: 1e2bd77fcea fsmonitor--daemon: use a cookie file to sync with file system\n -: ----------- > 28: 30e61b6d1ad fsmonitor: force update index after large responses\n -: ----------- > 29: 507020bbef0 t7527: test status with untracked-cache and fsmonitor--daemon\n\n-- \ngitgitgadget\n"},{"id":"439206","messageId":"82f176921282d13daee08bde7244df5d22e01ce6.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 02/29] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:42Z","receivedAt":"2021-10-21T14:25:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 +++++++++++++\n 3 files changed, 225 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 381bed2c1d2..d51fd8b33ce 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -897,6 +897,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..ccc32d2a17e\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,176 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\n+\t\tif (fsmonitor_is_trivial_response(answer))\n+\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(\"fsmonitor--daemon is not running\");\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\", c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..b6a7067c3af\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"439207","messageId":"ecc40795fa26ea86525421682303449f70132216.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 01/29] fsmonitor: enhance existing comments","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:41Z","receivedAt":"2021-10-21T14:25:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 37 ++++++++++++++++++++++++++++++-------\n 1 file changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..ec4c46407c5 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -301,9 +301,25 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n \tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -318,11 +334,15 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We received a trivial response, so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -330,7 +350,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"439208","messageId":"882789b4dfebddb059f62b0b2edb95b92f3c69ee.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 03/29] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:43Z","receivedAt":"2021-10-21T14:25:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove fsmonitor config settings to a new and opaque\n`struct fsmonitor_settings` structure. Add a lazily-loaded pointer\nto this into `struct repo_settings`\n\nCreate an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\nrepresent the state of fsmonitor. This lets us represent which, if\nany, fsmonitor provider (hook or IPC) is enabled.\n\nCreate `fsm_settings__get_*()` getters to lazily look up fsmonitor-\nrelated config settings.\n\nAdd support for the new `core.useBuiltinFSMonitor` config setting.\n\nGet rid of the `core_fsmonitor` global variable. Move the code to\nlookup the existing `core.fsmonitor` config value into the fsmonitor\nsettings.\n\nCreate a hook pathname variable in `struct fsmonitor-settings` and\nonly set it when in hook mode.\n\nThe existing `core_fsmonitor` global variable was used to store the\npathname to the fsmonitor hook *and* it was used as a boolean to see\nif fsmonitor was enabled. This dual usage and global visibility leads\nto confusion when we add the IPC-based provider. So lets hide the\ndetails in fsmonitor-settings.c and let it decide which provider to\nuse in the case of multiple settings. This avoids cluttering up\nrepo-settings.c with these private details.\n\nA future commit in builtin-fsmonitor series will add the ability to\ndisqualify worktrees for various reasons, such as being mounted from a\nremote volume, where fsmonitor should not be started. Having the\nconfig settings hidden in fsmonitor-settings.c allows such worktree\nrestrictions to override the config values used.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n builtin/update-index.c | 19 +++++++--\n cache.h | 1 -\n config.c | 14 ------\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-settings.c | 97 ++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 21 +++++++++\n fsmonitor.c | 63 ++++++++++++++++-----------\n fsmonitor.h | 18 ++++++--\n repository.h | 3 ++\n t/README | 4 +-\n 12 files changed, 192 insertions(+), 51 deletions(-)\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\ndiff --git a/Makefile b/Makefile\nindex d51fd8b33ce..29ed7c4aba6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -898,6 +898,7 @@ LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += fsmonitor-ipc.o\n+LIB_OBJS += fsmonitor-settings.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 187203e8bb5..79db3ff37e2 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1214,14 +1214,25 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is unset; \"\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"builtin fsmonitor\"));\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n-\t\t\t\t\"set it if you really want to \"\n-\t\t\t\t\"enable fsmonitor\"));\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"hook-based fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode == FSMONITOR_MODE_IPC)\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is set; \"\n+\t\t\t\t\"remove it if you really want to \"\n+\t\t\t\t\"disable fsmonitor\"));\n+\t\tif (fsm_mode == FSMONITOR_MODE_HOOK)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex d092820c943..8f4e3c8bd1d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -989,7 +989,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex 2dcbe901b6b..6b6e9cacac3 100644\n--- a/config.c\n+++ b/config.c\n@@ -2502,20 +2502,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex f119de01309..69d733824a0 100644\n--- a/config.h\n+++ b/config.h\n@@ -610,7 +610,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex 9da7f3c1a19..68f90632245 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -82,7 +82,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nnew file mode 100644\nindex 00000000000..2770266f5ee\n--- /dev/null\n+++ b/fsmonitor-settings.c\n@@ -0,0 +1,97 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+/*\n+ * We keep this structure defintion private and have getters\n+ * for all fields so that we can lazy load it as needed.\n+ */\n+struct fsmonitor_settings {\n+\tenum fsmonitor_mode mode;\n+\tchar *hook_path;\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_IPC;\n+}\n+\n+void fsm_settings__set_hook(struct repository *r, const char *path)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_HOOK;\n+\ts->hook_path = strdup(path);\n+}\n+\n+void fsm_settings__set_disabled(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\tFREE_AND_NULL(s->hook_path);\n+}\n+\n+static int check_for_ipc(struct repository *r)\n+{\n+\tint value;\n+\n+\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) &&\n+\t value) {\n+\t\tfsm_settings__set_ipc(r);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int check_for_hook(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n+\n+\tif (const_str && *const_str) {\n+\t\tfsm_settings__set_hook(r, const_str);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\n+\tr->settings.fsmonitor = s;\n+\n+\tif (check_for_ipc(r))\n+\t\treturn;\n+\n+\tif (check_for_hook(r))\n+\t\treturn;\n+\n+\tfsm_settings__set_disabled(r);\n+}\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n+{\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->mode;\n+}\n+\n+const char *fsm_settings__get_hook_path(struct repository *r)\n+{\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->hook_path;\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nnew file mode 100644\nindex 00000000000..50b29234616\n--- /dev/null\n+++ b/fsmonitor-settings.h\n@@ -0,0 +1,21 @@\n+#ifndef FSMONITOR_SETTINGS_H\n+#define FSMONITOR_SETTINGS_H\n+\n+struct repository;\n+\n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n+\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r);\n+void fsm_settings__set_hook(struct repository *r, const char *path);\n+void fsm_settings__set_disabled(struct repository *r);\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+const char *fsm_settings__get_hook_path(struct repository *r);\n+\n+struct fsmonitor_settings;\n+\n+#endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ec4c46407c5..63174630c0e 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tstrvec_push(&cp.args, fsm_settings__get_hook_path(r));\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -238,17 +242,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (fsm_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -256,13 +271,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -292,13 +308,17 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t}\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tfsm_settings__get_hook_path(r));\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t fsm_settings__get_hook_path(r),\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/*\n@@ -434,7 +454,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tint fsmonitor_enabled = (fsm_settings__get_mode(r) > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -454,16 +475,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..f9201411aa7 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -3,6 +3,7 @@\n \n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"fsmonitor-settings.h\"\n \n extern struct trace_key trace_fsmonitor;\n \n@@ -57,7 +58,11 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +72,11 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +92,10 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repository.h b/repository.h\nindex a057653981c..89a1873ade7 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n #include \"path.h\"\n \n struct config_set;\n+struct fsmonitor_settings;\n struct git_hash_algo;\n struct index_state;\n struct lock_file;\n@@ -34,6 +35,8 @@ struct repo_settings {\n \tint command_requires_full_index;\n \tint sparse_index;\n \n+\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex b92155a822e..6dc4a1d10cf 100644\n--- a/t/README\n+++ b/t/README\n@@ -405,8 +405,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code path for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\n-- \ngitgitgadget\n\n"},{"id":"439209","messageId":"de82c72618265c644c6ef65fedbf95e056b420a7.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 04/29] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:44Z","receivedAt":"2021-10-21T14:25:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.useBuiltinFSMonitor` is set.\n\nThe `core.fsmonitor` setting has already been defined as a HOOK\npathname. Historically, this has been set to a HOOK script that will\ntalk with Watchman. For compatibility reasons, we do not want to\noverload that definition (and cause problems if users have multiple\nversions of Git installed).\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 34 ++++++++++++++++++++++++++++++++--\n 1 file changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 63174630c0e..695fb0ce4e7 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -254,8 +254,37 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (fsm_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\tgoto apply_results;\n \t}\n \n \tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n@@ -321,6 +350,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/*\n \t * The response from FSMonitor (excluding the header token) is\n \t * either:\n-- \ngitgitgadget\n\n"},{"id":"439210","messageId":"d365704d5517d15a1ad74c5b2a6e6d6a7dcaffb1.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 05/29] fsmonitor: document builtin fsmonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:45Z","receivedAt":"2021-10-21T14:25:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDocument the new `core.useBuiltinFSMonitor` config value.\n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to refer to it.\n\nCreate `git-fsmonitor--daemon` manual page and describe its features.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/config/core.txt | 56 +++++++++++++-----\n Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n Documentation/git-update-index.txt | 27 +++++----\n Documentation/githooks.txt | 3 +-\n 4 files changed, 134 insertions(+), 27 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..4f6e519bc02 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,50 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n+\n+core.useBuiltinFSMonitor::\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+Note: if this config setting is set to `true`, the values of\n+`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\nnew file mode 100644\nindex 00000000000..154e7684daa\n--- /dev/null\n+++ b/Documentation/git-fsmonitor--daemon.txt\n@@ -0,0 +1,75 @@\n+git-fsmonitor--daemon(1)\n+========================\n+\n+NAME\n+----\n+git-fsmonitor--daemon - A Built-in File System Monitor\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git fsmonitor--daemon' start\n+'git fsmonitor--daemon' run\n+'git fsmonitor--daemon' stop\n+'git fsmonitor--daemon' status\n+\n+DESCRIPTION\n+-----------\n+\n+A daemon to watch the working directory for file and directory\n+changes using platform-specific file system notification facilities.\n+\n+This daemon communicates directly with commands like `git status`\n+using the link:technical/api-simple-ipc.html[simple IPC] interface\n+instead of the slower linkgit:githooks[5] interface.\n+\n+This daemon is built into Git so that no third-party tools are\n+required.\n+\n+OPTIONS\n+-------\n+\n+start::\n+\tStarts a daemon in the background.\n+\n+run::\n+\tRuns a daemon in the foreground.\n+\n+stop::\n+\tStops the daemon running in the current working\n+\tdirectory, if present.\n+\n+status::\n+\tExits with zero status if a daemon is watching the\n+\tcurrent working directory.\n+\n+REMARKS\n+-------\n+\n+This daemon is a long running process used to watch a single working\n+directory and maintain a list of the recently changed files and\n+directories. Performance of commands such as `git status` can be\n+increased if they just ask for a summary of changes to the working\n+directory and can avoid scanning the disk.\n+\n+When `core.useBuiltinFSMonitor` is set to `true` (see\n+linkgit:git-config[1]) commands, such as `git status`, will ask the\n+daemon for changes and automatically start it (if necessary).\n+\n+For more information see the \"File System Monitor\" section in\n+linkgit:git-update-index[1].\n+\n+CAVEATS\n+-------\n+\n+The fsmonitor daemon does not currently know about submodules and does\n+not know to filter out file system events that happen within a\n+submodule. If fsmonitor daemon is watching a super repo and a file is\n+modified within the working directory of a submodule, it will report\n+the change (as happening against the super repo). However, the client\n+will properly ignore these extra events, so performance may be affected\n+but it will not cause an incorrect result.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..c7c31b3fcf9 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory\n looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n-the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n-across all repositories you use, because you can set the configuration\n-variable in your `$HOME/.gitconfig` just once and have it affect all\n-repositories you touch.\n-\n-When the `core.fsmonitor` configuration variable is changed, the\n-file system monitor is added to or removed from the index the next time\n-a command reads the index. When `--[no-]fsmonitor` are used, the file\n-system monitor is immediately added to or removed from the index.\n+the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable (see linkgit:git-config[1]) than using the `--fsmonitor`\n+option to `git update-index` in each repository, especially if you\n+want to do so across all repositories you use, because you can set the\n+configuration variable in your `$HOME/.gitconfig` just once and have\n+it affect all repositories you touch.\n+\n+When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable is changed, the file system monitor is added to or removed\n+from the index the next time a command reads the index. When\n+`--[no-]fsmonitor` are used, the file system monitor is immediately\n+added to or removed from the index.\n \n CONFIGURATION\n -------------\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex b51959ff941..b7d5e926f7b 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -593,7 +593,8 @@ fsmonitor-watchman\n \n This hook is invoked when the configuration option `core.fsmonitor` is\n set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n-depending on the version of the hook to use.\n+depending on the version of the hook to use, unless overridden via\n+`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n \n Version 1 takes two arguments, a version (1) and the time in elapsed\n nanoseconds since midnight, January 1, 1970.\n-- \ngitgitgadget\n\n"},{"id":"439211","messageId":"78e682fc5307087157ecb46848c01389885404d3.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 06/29] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:46Z","receivedAt":"2021-10-21T14:25:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a built-in file system monitoring daemon that can be used by\nthe existing `fsmonitor` feature (protocol API and index extension)\nto improve the performance of various Git commands, such as `status`.\n\nThe `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\nprovides an alternative to hook access to existing fsmonitors such\nas `watchman`.\n\nThis commit merely adds the new command without any functionality.\n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 46 +++++++++++++++++++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 50 insertions(+)\n create mode 100644 builtin/fsmonitor--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 054249b20a8..e863add420a 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -72,6 +72,7 @@\n /git-format-patch\n /git-fsck\n /git-fsck-objects\n+/git-fsmonitor--daemon\n /git-gc\n /git-get-tar-commit-id\n /git-grep\ndiff --git a/Makefile b/Makefile\nindex 29ed7c4aba6..95ac224bc8d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1104,6 +1104,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n+BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex 8a58743ed63..837b9ede598 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\n int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nnew file mode 100644\nindex 00000000000..f0498793379\n--- /dev/null\n+++ b/builtin/fsmonitor--daemon.c\n@@ -0,0 +1,46 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"simple-ipc.h\"\n+#include \"khash.h\"\n+\n+static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tNULL\n+};\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *subcmd;\n+\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\tif (argc != 1)\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\tsubcmd = argv[0];\n+\n+\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n+}\n+\n+#else\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n+}\n+#endif\ndiff --git a/git.c b/git.c\nindex 5ff21be21f3..28c8af6b795 100644\n--- a/git.c\n+++ b/git.c\n@@ -533,6 +533,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"439212","messageId":"d08c28b549c4835514375b1c457a564eeb336c7d.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 11/29] fsmonitor--daemon: implement 'start' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:51Z","receivedAt":"2021-10-21T14:25:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement 'git fsmonitor--daemon start' command. This command starts\nan instance of 'git fsmonitor--daemon run' in the background using\nthe new 'start_bg_command()' function.\n\nWe avoid the fork-and-call technique on Unix systems in favor of a\nfork-and-exec technique. This gives us more uniform Trace2 child-*\nevents. It also makes our usage more consistent with Windows usage.\n\nOn Windows, teach 'git fsmonitor--daemon run' to optionally call\n'FreeConsole()' to release handles to the inherited Win32 console\n(despite being passed invalid handles for stdin/out/err). Without\nthis, command prompts and powershell terminal windows could hang\nin \"exit\" until the last background child process exited or released\ntheir Win32 console handle. (This was not seen with git-bash shells\nbecause they don't have a Win32 console attached to them.)\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 107 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 105 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex b5ebd1eca33..8988440b7c1 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -9,6 +9,7 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon start []\"),\n \tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n@@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n #define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n static int fsmonitor__ipc_threads = 8;\n \n+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n+static int fsmonitor__start_timeout_sec = 60;\n+\n static int fsmonitor_config(const char *var, const char *value, void *cb)\n {\n \tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n@@ -33,6 +37,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n+\t\tfsmonitor__start_timeout_sec = i;\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -237,7 +250,7 @@ done:\n \treturn err;\n }\n \n-static int try_to_run_foreground_daemon(void)\n+static int try_to_run_foreground_daemon(int free_console)\n {\n \t/*\n \t * Technically, we don't need to probe for an existing daemon\n@@ -255,17 +268,104 @@ static int try_to_run_foreground_daemon(void)\n \t the_repository->worktree);\n \tfflush(stdout);\n \n+#ifdef GIT_WINDOWS_NATIVE\n+\tif (free_console)\n+\t\tFreeConsole();\n+#endif\n+\n \treturn !!fsmonitor_run_daemon();\n }\n \n+static start_bg_wait_cb bg_wait_cb;\n+\n+static int bg_wait_cb(const struct child_process *cp, void *cb_data)\n+{\n+\tenum ipc_active_state s = fsmonitor_ipc__get_state();\n+\n+\tswitch (s) {\n+\tcase IPC_STATE__LISTENING:\n+\t\t/* child is \"ready\" */\n+\t\treturn 0;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\t/* give child more time */\n+\t\treturn 1;\n+\n+\tdefault:\n+\tcase IPC_STATE__INVALID_PATH:\n+\tcase IPC_STATE__OTHER_ERROR:\n+\t\t/* all the time in world won't help */\n+\t\treturn -1;\n+\t}\n+}\n+\n+static int try_to_start_background_daemon(void)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tenum start_bg_result sbgr;\n+\n+\t/*\n+\t * Before we try to create a background daemon process, see\n+\t * if a daemon process is already listening. This makes it\n+\t * easier for us to report an already-listening error to the\n+\t * console, since our spawn/daemon can only report the success\n+\t * of creating the background process (and not whether it\n+\t * immediately exited).\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n+\t\t the_repository->worktree);\n+\n+\tprintf(_(\"starting fsmonitor-daemon in '%s'\\n\"),\n+\t the_repository->worktree);\n+\tfflush(stdout);\n+\n+\tcp.git_cmd = 1;\n+\n+\tstrvec_push(&cp.args, \"fsmonitor--daemon\");\n+\tstrvec_push(&cp.args, \"run\");\n+\tstrvec_push(&cp.args, \"--free-console\");\n+\tstrvec_pushf(&cp.args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n+\n+\tcp.no_stdin = 1;\n+\tcp.no_stdout = 1;\n+\tcp.no_stderr = 1;\n+\n+\tsbgr = start_bg_command(&cp, bg_wait_cb, NULL,\n+\t\t\t\tfsmonitor__start_timeout_sec);\n+\n+\tswitch (sbgr) {\n+\tcase SBGR_READY:\n+\t\treturn 0;\n+\n+\tdefault:\n+\tcase SBGR_ERROR:\n+\tcase SBGR_CB_ERROR:\n+\t\treturn error(\"daemon failed to start\");\n+\n+\tcase SBGR_TIMEOUT:\n+\t\treturn error(\"daemon not online yet\");\n+\n+\tcase SBGR_DIED:\n+\t\treturn error(\"daemon terminated\");\n+\t}\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tint free_console = 0;\n \n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"free-console\", &free_console, N_(\"free console\")),\n \t\tOPT_INTEGER(0, \"ipc-threads\",\n \t\t\t &fsmonitor__ipc_threads,\n \t\t\t N_(\"use ipc worker threads\")),\n+\t\tOPT_INTEGER(0, \"start-timeout\",\n+\t\t\t &fsmonitor__start_timeout_sec,\n+\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -281,8 +381,11 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tif (!strcmp(subcmd, \"start\"))\n+\t\treturn !!try_to_start_background_daemon();\n+\n \tif (!strcmp(subcmd, \"run\"))\n-\t\treturn !!try_to_run_foreground_daemon();\n+\t\treturn !!try_to_run_foreground_daemon(free_console);\n \n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n-- \ngitgitgadget\n\n"},{"id":"439213","messageId":"5a40b33a00c927373c64536c87fba76a676f8a77.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 08/29] compat/fsmonitor/fsm-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:48Z","receivedAt":"2021-10-21T14:25:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty filesystem listener backend for fsmonitor--daemon on Windows.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 ++++++++\n compat/fsmonitor/fsm-listen-win32.c | 21 +++++++++++++\n compat/fsmonitor/fsm-listen.h | 49 +++++++++++++++++++++++++++++\n config.mak.uname | 10 ++++++\n contrib/buildsystems/CMakeLists.txt | 7 +++++\n repo-settings.c | 1 +\n 6 files changed, 101 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n\ndiff --git a/Makefile b/Makefile\nindex 95ac224bc8d..8a26696c9ed 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -465,6 +465,11 @@ all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n+# If your platform supports a built-in fsmonitor backend, set\n+# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n+# `compat/fsmonitor/fsm-listen-.c` that implements the\n+# `fsm_listen__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1930,6 +1935,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n \tCOMPAT_OBJS += compat/access.o\n endif\n \n+ifdef FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2805,6 +2815,9 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n \t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n+ifdef FSMONITOR_DAEMON_BACKEND\n+\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nnew file mode 100644\nindex 00000000000..916cbea254f\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nnew file mode 100644\nindex 00000000000..f0539349baf\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -0,0 +1,49 @@\n+#ifndef FSM_LISTEN_H\n+#define FSM_LISTEN_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * fsmonitor_fs_listener thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread AFTER joining the listener.\n+ */\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to watch for\n+ * filesystem events. This will run in the fsmonitor_fs_listen thread.\n+ *\n+ * It should call `ipc_server_stop_async()` if the listener thread\n+ * prematurely terminates (because of a filesystem error or if it\n+ * detects that the .git directory has been deleted). (It should NOT\n+ * do so if the listener thread receives a normal shutdown signal from\n+ * the IPC layer.)\n+ *\n+ * It should set `state->error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the fsmonitor listener thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_LISTEN_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 3236a4918a3..e6a900c6fd4 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -426,6 +426,11 @@ ifeq ($(uname_S),Windows)\n \t# so we don't need this:\n \t#\n \t# SNPRINTF_RETURNS_BOGUS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -609,6 +614,11 @@ ifeq ($(uname_S),MINGW)\n \tNO_STRTOUMAX = YesPlease\n \tNO_MKDTEMP = YesPlease\n \tNO_SVN_TESTS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex fd1399c440f..24a26743dcd 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -285,6 +285,13 @@ else()\n \tendif()\n endif()\n \n+if(SUPPORTS_SIMPLE_IPC)\n+\tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\tendif()\n+endif()\n+\n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n \n #header checks\ndiff --git a/repo-settings.c b/repo-settings.c\nindex b93e91a212e..3ed763ad9c6 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"midx.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n \n static void repo_cfg_bool(struct repository *r, const char *key, int *dest,\n \t\t\t int def)\n-- \ngitgitgadget\n\n"},{"id":"439214","messageId":"ea64b5c9753ebba0d0f7b4e3d4a2475597f207fa.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 07/29] fsmonitor--daemon: implement 'stop' and 'status' commands","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:47Z","receivedAt":"2021-10-21T14:25:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `stop` and `status` client commands to control and query the\nstatus of a `fsmonitor--daemon` server process (and implicitly start a\nserver process if necessary).\n\nLater commits will implement the actual server and monitor the file\nsystem.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 51 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex f0498793379..5e3178b8bdd 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,10 +7,55 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon stop\"),\n+\tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n+ * and wait for it to shutdown.\n+ */\n+static int do_as_client__send_stop(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"quit\", &answer);\n+\n+\t/* The quit command does not return any response data. */\n+\tstrbuf_release(&answer);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\twhile (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tsleep_millisec(50);\n+\ttrace2_region_leave(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\n+\treturn 0;\n+}\n+\n+static int do_as_client__status(void)\n+{\n+\tenum ipc_active_state state = fsmonitor_ipc__get_state();\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tprintf(_(\"fsmonitor-daemon is watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tprintf(_(\"fsmonitor-daemon is not watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 1;\n+\t}\n+}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n@@ -28,6 +73,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (!strcmp(subcmd, \"stop\"))\n+\t\treturn !!do_as_client__send_stop();\n+\n+\tif (!strcmp(subcmd, \"status\"))\n+\t\treturn !!do_as_client__status();\n+\n \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"439215","messageId":"ed5819e29f8743abb8812c21649a785bd451d78a.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 09/29] compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:49Z","receivedAt":"2021-10-21T14:25:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty implementation of fsmonitor--daemon\nbackend for Darwin (aka MacOS).\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 20 ++++++++++++++++++++\n config.mak.uname | 10 ++++++++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 33 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nnew file mode 100644\nindex 00000000000..c84e3344ab9\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -0,0 +1,20 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex e6a900c6fd4..81f21d9e201 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -153,6 +153,16 @@ ifeq ($(uname_S),Darwin)\n \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n \t\tendif\n \tendif\n+\n+\t# The builtin FSMonitor on MacOS builds upon Simple-IPC. Both require\n+\t# Unix domain sockets and PThreads.\n+\tifndef NO_PTHREADS\n+\tifndef NO_UNIX_SOCKETS\n+\tFSMONITOR_DAEMON_BACKEND = darwin\n+\tendif\n+\tendif\n+\n+\tBASIC_LDFLAGS += -framework CoreServices\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 24a26743dcd..41395705ac6 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"439216","messageId":"d3ac973a5f1dbf99b2f4f915c8644f6c302bef1a.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 10/29] fsmonitor--daemon: implement 'run' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:50Z","receivedAt":"2021-10-21T14:25:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `run` command to try to begin listening for file system events.\n\nThis version defines the thread structure with a single fsmonitor_fs_listen\nthread to watch for file system events and a simple IPC thread pool to\nwatch for connection from Git clients over a well-known named pipe or\nUnix domain socket.\n\nThis commit does not actually do anything yet because the platform\nbackends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 213 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 34 ++++++\n 2 files changed, 246 insertions(+), 1 deletion(-)\n create mode 100644 fsmonitor--daemon.h\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 5e3178b8bdd..b5ebd1eca33 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,16 +3,39 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Global state loaded from config.\n+ */\n+#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n+static int fsmonitor__ipc_threads = 8;\n+\n+static int fsmonitor_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 1)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__IPC_THREADS, i);\n+\t\tfsmonitor__ipc_threads = i;\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n /*\n * Acting as a CLIENT.\n *\n@@ -57,15 +80,196 @@ static int do_as_client__status(void)\n \t}\n }\n \n+static ipc_server_application_cb handle_client;\n+\n+static int handle_client(void *data,\n+\t\t\t const char *command, size_t command_len,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\t/* struct fsmonitor_daemon_state *state = data; */\n+\tint result;\n+\n+\t/*\n+\t * The Simple IPC API now supports {char*, len} arguments, but\n+\t * FSMonitor always uses proper null-terminated strings, so\n+\t * we can ignore the command_len argument. (Trust, but verify.)\n+\t */\n+\tif (command_len != strlen(command))\n+\t\tBUG(\"FSMonitor assumes text messages\");\n+\n+\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n+\n+\tresult = 0; /* TODO Do something here. */\n+\n+\ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n+\n+\treturn result;\n+}\n+\n+static void *fsm_listen__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-listen\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Watching: worktree '%s'\",\n+\t\t\t state->path_worktree_watch.buf);\n+\tif (state->nr_paths_watching > 1)\n+\t\ttrace_printf_key(&trace_fsmonitor, \"Watching: gitdir '%s'\",\n+\t\t\t\t state->path_gitdir_watch.buf);\n+\n+\tfsm_listen__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct ipc_server_opts ipc_opts = {\n+\t\t.nr_threads = fsmonitor__ipc_threads,\n+\n+\t\t/*\n+\t\t * We know that there are no other active threads yet,\n+\t\t * so we can let the IPC layer temporarily chdir() if\n+\t\t * it needs to when creating the server side of the\n+\t\t * Unix domain socket.\n+\t\t */\n+\t\t.uds_disallow_chdir = 0\n+\t};\n+\n+\t/*\n+\t * Start the IPC thread pool before the we've started the file\n+\t * system event listener thread so that we have the IPC handle\n+\t * before we need it.\n+\t */\n+\tif (ipc_server_run_async(&state->ipc_server_data,\n+\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t handle_client, state))\n+\t\treturn error_errno(\n+\t\t\t_(\"could not start IPC thread pool on '%s'\"),\n+\t\t\tfsmonitor_ipc__get_path());\n+\n+\t/*\n+\t * Start the fsmonitor listener thread to collect filesystem\n+\t * events.\n+\t */\n+\tif (pthread_create(&state->listener_thread, NULL,\n+\t\t\t fsm_listen__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\tipc_server_await(state->ipc_server_data);\n+\n+\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t}\n+\n+\t/*\n+\t * The daemon is now fully functional in background threads.\n+\t * Wait for the IPC thread pool to shutdown (whether by client\n+\t * request or from filesystem activity).\n+\t */\n+\tipc_server_await(state->ipc_server_data);\n+\n+\t/*\n+\t * The fsmonitor listener thread may have received a shutdown\n+\t * event from the IPC thread pool, but it doesn't hurt to tell\n+\t * it again. And wait for it to shutdown.\n+\t */\n+\tfsm_listen__stop_async(state);\n+\tpthread_join(state->listener_thread, NULL);\n+\n+\treturn state->error_code;\n+}\n+\n+static int fsmonitor_run_daemon(void)\n+{\n+\tstruct fsmonitor_daemon_state state;\n+\tint err;\n+\n+\tmemset(&state, 0, sizeof(state));\n+\n+\tpthread_mutex_init(&state.main_lock, NULL);\n+\tstate.error_code = 0;\n+\tstate.current_token_data = NULL;\n+\n+\t/* Prepare to (recursively) watch the directory. */\n+\tstrbuf_init(&state.path_worktree_watch, 0);\n+\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n+\tstate.nr_paths_watching = 1;\n+\n+\t/*\n+\t * We create and delete cookie files somewhere inside the .git\n+\t * directory to help us keep sync with the file system. If\n+\t * \".git\" is not a directory, then is not inside the\n+\t * cone of , so set up a second watch to watch\n+\t * the so that we get events for the cookie files.\n+\t */\n+\tstrbuf_init(&state.path_gitdir_watch, 0);\n+\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n+\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n+\tif (!is_directory(state.path_gitdir_watch.buf)) {\n+\t\tstrbuf_reset(&state.path_gitdir_watch);\n+\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n+\t\tstate.nr_paths_watching = 2;\n+\t}\n+\n+\t/*\n+\t * Confirm that we can create platform-specific resources for the\n+\t * filesystem listener before we bother starting all the threads.\n+\t */\n+\tif (fsm_listen__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize listener thread\"));\n+\t\tgoto done;\n+\t}\n+\n+\terr = fsmonitor_run_daemon_1(&state);\n+\n+done:\n+\tpthread_mutex_destroy(&state.main_lock);\n+\tfsm_listen__dtor(&state);\n+\n+\tipc_server_free(state.ipc_server_data);\n+\n+\tstrbuf_release(&state.path_worktree_watch);\n+\tstrbuf_release(&state.path_gitdir_watch);\n+\n+\treturn err;\n+}\n+\n+static int try_to_run_foreground_daemon(void)\n+{\n+\t/*\n+\t * Technically, we don't need to probe for an existing daemon\n+\t * process, since we could just call `fsmonitor_run_daemon()`\n+\t * and let it fail if the pipe/socket is busy.\n+\t *\n+\t * However, this method gives us a nicer error message for a\n+\t * common error case.\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n+\t\t the_repository->worktree);\n+\n+\tprintf(_(\"running fsmonitor-daemon in '%s'\\n\"),\n+\t the_repository->worktree);\n+\tfflush(stdout);\n+\n+\treturn !!fsmonitor_run_daemon();\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n \n \tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"ipc-threads\",\n+\t\t\t &fsmonitor__ipc_threads,\n+\t\t\t N_(\"use ipc worker threads\")),\n \t\tOPT_END()\n \t};\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(fsmonitor_config, NULL);\n \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n@@ -73,6 +277,13 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (fsmonitor__ipc_threads < 1)\n+\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n+\t\t fsmonitor__ipc_threads);\n+\n+\tif (!strcmp(subcmd, \"run\"))\n+\t\treturn !!try_to_run_foreground_daemon();\n+\n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n \ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nnew file mode 100644\nindex 00000000000..3009c1a83de\n--- /dev/null\n+++ b/fsmonitor--daemon.h\n@@ -0,0 +1,34 @@\n+#ifndef FSMONITOR_DAEMON_H\n+#define FSMONITOR_DAEMON_H\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+#include \"thread-utils.h\"\n+\n+struct fsmonitor_batch;\n+struct fsmonitor_token_data;\n+\n+struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+\n+struct fsmonitor_daemon_state {\n+\tpthread_t listener_thread;\n+\tpthread_mutex_t main_lock;\n+\n+\tstruct strbuf path_worktree_watch;\n+\tstruct strbuf path_gitdir_watch;\n+\tint nr_paths_watching;\n+\n+\tstruct fsmonitor_token_data *current_token_data;\n+\n+\tint error_code;\n+\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\n+\tstruct ipc_server_data *ipc_server_data;\n+};\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"439217","messageId":"6fa71fdc8255e6e58fe391671a980f1e75887b45.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 12/29] fsmonitor--daemon: add pathname classification","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:52Z","receivedAt":"2021-10-21T14:25:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to classify relative and absolute\npathnames and decide how they should be handled. This will\nbe used by the platform-specific backend to respond to each\nfilesystem event.\n\nWhen we register for filesystem notifications on a directory,\nwe get events for everything (recursively) in the directory.\nWe want to report to clients changes to tracked and untracked\npaths within the working directory. We do not want to report\nchanges within the .git directory, for example.\n\nThis classification will be used in a later commit by the\ndifferent backends to classify paths as events are received.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 81 +++++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 61 ++++++++++++++++++++++++++++\n 2 files changed, 142 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 8988440b7c1..1618f1aa27f 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -121,6 +121,87 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n+#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *rel)\n+{\n+\tif (fspathncmp(rel, \".git\", 4))\n+\t\treturn IS_WORKDIR_PATH;\n+\trel += 4;\n+\n+\tif (!*rel)\n+\t\treturn IS_DOT_GIT;\n+\tif (*rel != '/')\n+\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n+\trel++;\n+\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_DOT_GIT;\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *rel)\n+{\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_GITDIR;\n+}\n+\n+static enum fsmonitor_path_type try_classify_workdir_abs_path(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\n+\tif (fspathncmp(path, state->path_worktree_watch.buf,\n+\t\t state->path_worktree_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_worktree_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_workdir_relative(rel);\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\tenum fsmonitor_path_type t;\n+\n+\tt = try_classify_workdir_abs_path(state, path);\n+\tif (state->nr_paths_watching == 1)\n+\t\treturn t;\n+\tif (t != IS_OUTSIDE_CONE)\n+\t\treturn t;\n+\n+\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n+\t\t state->path_gitdir_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_gitdir_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_GITDIR; /* it is the exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_gitdir_relative(rel);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 3009c1a83de..7bbb3a27a1c 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -30,5 +30,66 @@ struct fsmonitor_daemon_state {\n \tstruct ipc_server_data *ipc_server_data;\n };\n \n+/*\n+ * Pathname classifications.\n+ *\n+ * The daemon classifies the pathnames that it receives from file\n+ * system notification events into the following categories and uses\n+ * that to decide whether clients are told about them. (And to watch\n+ * for file system synchronization events.)\n+ *\n+ * The client should only care about paths within the working\n+ * directory proper (inside the working directory and not \".git\" nor\n+ * inside of \".git/\"). That is, the client has read the index and is\n+ * asking for a list of any paths in the working directory that have\n+ * been modified since the last token. The client does not care about\n+ * file system changes within the .git directory (such as new loose\n+ * objects or packfiles). So the client will only receive paths that\n+ * are classified as IS_WORKDIR_PATH.\n+ *\n+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n+ * exact \".git\" directory or GITDIR. If the daemon receives a delete\n+ * event for either of these directories, it will automatically\n+ * shutdown, for example.\n+ *\n+ * Note that the daemon DOES NOT explicitly watch nor special case the\n+ * \".git/index\" file. The daemon does not read the index and does not\n+ * have any internal index-relative state. The daemon only collects\n+ * the set of modified paths within the working directory.\n+ */\n+enum fsmonitor_path_type {\n+\tIS_WORKDIR_PATH = 0,\n+\n+\tIS_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n+\n+\tIS_GITDIR,\n+\tIS_INSIDE_GITDIR,\n+\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n+\n+\tIS_OUTSIDE_CONE,\n+};\n+\n+/*\n+ * Classify a pathname relative to the root of the working directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify a pathname relative to a that is external to the\n+ * worktree directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify an absolute pathname received from a filesystem event.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"439218","messageId":"429c48a5badf783ec5e1838d7de8e27fc3fa0f47.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 14/29] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:54Z","receivedAt":"2021-10-21T14:25:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to build a list of changed paths and associate\nthem with a token-id. This will be used by the platform-specific\nbackends to accumulate changed paths in response to filesystem events.\n\nThe platform-specific file system listener thread receives file system\nevents containing one or more changed pathnames (with whatever bucketing\nor grouping that is convenient for the file system). These paths are\naccumulated (without locking) by the file system layer into a `fsmonitor_batch`.\n\nWhen the file system layer has drained the kernel event queue, it will\n\"publish\" them to our token queue and make them visible to concurrent\nclient worker threads. The token layer is free to combine and/or de-dup\npaths within these batches for efficient presentation to clients.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 230 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 40 +++++++\n 2 files changed, 268 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 2997c2cba8c..34603f23053 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -168,17 +168,27 @@ struct fsmonitor_token_data {\n \tuint64_t client_ref_count;\n };\n \n+struct fsmonitor_batch {\n+\tstruct fsmonitor_batch *next;\n+\tuint64_t batch_seq_nr;\n+\tconst char **interned_paths;\n+\tsize_t nr, alloc;\n+\ttime_t pinned_time;\n+};\n+\n static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n {\n \tstatic int test_env_value = -1;\n \tstatic uint64_t flush_count = 0;\n \tstruct fsmonitor_token_data *token;\n+\tstruct fsmonitor_batch *batch;\n \n \tCALLOC_ARRAY(token, 1);\n+\tbatch = fsmonitor_batch__new();\n \n \tstrbuf_init(&token->token_id, 0);\n-\ttoken->batch_head = NULL;\n-\ttoken->batch_tail = NULL;\n+\ttoken->batch_head = batch;\n+\ttoken->batch_tail = batch;\n \ttoken->client_ref_count = 0;\n \n \tif (test_env_value < 0)\n@@ -204,9 +214,143 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n \t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n \t}\n \n+\t/*\n+\t * We created a new and are starting a new series\n+\t * of tokens with a zero .\n+\t *\n+\t * Since clients cannot guess our new (non test) \n+\t * they will always receive a trivial response (because of the\n+\t * mismatch on the ). The trivial response will\n+\t * tell them our new so that subsequent requests\n+\t * will be relative to our new series. (And when sending that\n+\t * response, we pin the current head of the batch list.)\n+\t *\n+\t * Even if the client correctly guesses the , their\n+\t * request of \"builtin::0\" asks for all changes MORE\n+\t * RECENT than batch/bin 0.\n+\t *\n+\t * This implies that it is a waste to accumulate paths in the\n+\t * initial batch/bin (because they will never be transmitted).\n+\t *\n+\t * So the daemon could be running for days and watching the\n+\t * file system, but doesn't need to actually accumulate any\n+\t * paths UNTIL we need to set a reference point for a later\n+\t * relative request.\n+\t *\n+\t * However, it is very useful for testing to always have a\n+\t * reference point set. Pin batch 0 to force early file system\n+\t * events to accumulate.\n+\t */\n+\tif (test_env_value)\n+\t\tbatch->pinned_time = time(NULL);\n+\n \treturn token;\n }\n \n+struct fsmonitor_batch *fsmonitor_batch__new(void)\n+{\n+\tstruct fsmonitor_batch *batch;\n+\n+\tCALLOC_ARRAY(batch, 1);\n+\n+\treturn batch;\n+}\n+\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch)\n+{\n+\twhile (batch) {\n+\t\tstruct fsmonitor_batch *next = batch->next;\n+\n+\t\t/*\n+\t\t * The actual strings within the array of this batch\n+\t\t * are interned, so we don't own them. We only own\n+\t\t * the array.\n+\t\t */\n+\t\tfree(batch->interned_paths);\n+\t\tfree(batch);\n+\n+\t\tbatch = next;\n+\t}\n+}\n+\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n+\t\t\t const char *path)\n+{\n+\tconst char *interned_path = strintern(path);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n+\n+\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n+\tbatch->interned_paths[batch->nr++] = interned_path;\n+}\n+\n+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n+\t\t\t\t const struct fsmonitor_batch *batch_src)\n+{\n+\tsize_t k;\n+\n+\tALLOC_GROW(batch_dest->interned_paths,\n+\t\t batch_dest->nr + batch_src->nr + 1,\n+\t\t batch_dest->alloc);\n+\n+\tfor (k = 0; k < batch_src->nr; k++)\n+\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n+\t\t\tbatch_src->interned_paths[k];\n+}\n+\n+static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n+{\n+\tif (!token)\n+\t\treturn;\n+\n+\tassert(token->client_ref_count == 0);\n+\n+\tstrbuf_release(&token->token_id);\n+\n+\tfsmonitor_batch__free_list(token->batch_head);\n+\n+\tfree(token);\n+}\n+\n+/*\n+ * Flush all of our cached data about the filesystem. Call this if we\n+ * lose sync with the filesystem and miss some notification events.\n+ *\n+ * [1] If we are missing events, then we no longer have a complete\n+ * history of the directory (relative to our current start token).\n+ * We should create a new token and start fresh (as if we just\n+ * booted up).\n+ *\n+ * If there are no concurrent threads readering the current token data\n+ * series, we can free it now. Otherwise, let the last reader free\n+ * it.\n+ *\n+ * Either way, the old token data series is no longer associated with\n+ * our state data.\n+ */\n+static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct fsmonitor_token_data *free_me = NULL;\n+\tstruct fsmonitor_token_data *new_one = NULL;\n+\n+\tnew_one = fsmonitor_new_token_data();\n+\n+\tif (state->current_token_data->client_ref_count == 0)\n+\t\tfree_me = state->current_token_data;\n+\tstate->current_token_data = new_one;\n+\n+\tfsmonitor_free_token_data(free_me);\n+}\n+\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\tpthread_mutex_lock(&state->main_lock);\n+\twith_lock__do_force_resync(state);\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -316,6 +460,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n \n+/*\n+ * We try to combine small batches at the front of the batch-list to avoid\n+ * having a long list. This hopefully makes it a little easier when we want\n+ * to truncate and maintain the list. However, we don't want the paths array\n+ * to just keep growing and growing with realloc, so we insert an arbitrary\n+ * limit.\n+ */\n+#define MY_COMBINE_LIMIT (1024)\n+\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names)\n+{\n+\tif (!batch && !cookie_names->nr)\n+\t\treturn;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (batch) {\n+\t\tstruct fsmonitor_batch *head;\n+\n+\t\thead = state->current_token_data->batch_head;\n+\t\tif (!head) {\n+\t\t\tBUG(\"token does not have batch\");\n+\t\t} else if (head->pinned_time) {\n+\t\t\t/*\n+\t\t\t * We cannot alter the current batch list\n+\t\t\t * because:\n+\t\t\t *\n+\t\t\t * [a] it is being transmitted to at least one\n+\t\t\t * client and the handle_client() thread has a\n+\t\t\t * ref-count, but not a lock on the batch list\n+\t\t\t * starting with this item.\n+\t\t\t *\n+\t\t\t * [b] it has been transmitted in the past to\n+\t\t\t * at least one client such that future\n+\t\t\t * requests are relative to this head batch.\n+\t\t\t *\n+\t\t\t * So, we can only prepend a new batch onto\n+\t\t\t * the front of the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else if (!head->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * Batch 0 is unpinned. See the note in\n+\t\t\t * `fsmonitor_new_token_data()` about why we\n+\t\t\t * don't need to accumulate these paths.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n+\t\t\t/*\n+\t\t\t * The head batch in the list has never been\n+\t\t\t * transmitted to a client, but folding the\n+\t\t\t * contents of the new batch onto it would\n+\t\t\t * exceed our arbitrary limit, so just prepend\n+\t\t\t * the new batch onto the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We are free to add the paths in the given\n+\t\t\t * batch onto the end of the current head batch.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__combine(head, batch);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -330,6 +549,13 @@ static void *fsm_listen__thread_proc(void *_state)\n \n \tfsm_listen__loop(state);\n \n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (state->current_token_data &&\n+\t state->current_token_data->client_ref_count == 0)\n+\t\tfsmonitor_free_token_data(state->current_token_data);\n+\tstate->current_token_data = NULL;\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n \ttrace2_thread_exit();\n \treturn NULL;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 7bbb3a27a1c..20a815d80f8 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -12,6 +12,27 @@\n struct fsmonitor_batch;\n struct fsmonitor_token_data;\n \n+/*\n+ * Create a new batch of path(s). The returned batch is considered\n+ * private and not linked into the fsmonitor daemon state. The caller\n+ * should fill this batch with one or more paths and then publish it.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__new(void);\n+\n+/*\n+ * Free the list of batches starting with this one.\n+ */\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n+\n+/*\n+ * Add this path to this batch of modified files.\n+ *\n+ * The batch should be private and NOT (yet) linked into the fsmonitor\n+ * daemon state and therefore not yet visible to worker threads and so\n+ * no locking is required.\n+ */\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n+\n struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n \n struct fsmonitor_daemon_state {\n@@ -91,5 +112,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \tstruct fsmonitor_daemon_state *state,\n \tconst char *path);\n \n+/*\n+ * Prepend the this batch of path(s) onto the list of batches associated\n+ * with the current token. This makes the batch visible to worker threads.\n+ *\n+ * The caller no longer owns the batch and must not free it.\n+ *\n+ * Wake up the client threads waiting on these cookies.\n+ */\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names);\n+\n+/*\n+ * If the platform-specific layer loses sync with the filesystem,\n+ * it should call this to invalidate cached data and abort waiting\n+ * threads.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"439219","messageId":"b04c460c6199ad58211356e9f92587cae8d1fa6a.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 15/29] compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:55Z","receivedAt":"2021-10-21T14:25:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the win32 backend to register a watch on the working tree\nroot directory (recursively). Also watch the if it is\nnot inside the working tree. And to collect path change notifications\ninto batches and publish.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 565 ++++++++++++++++++++++++++++\n 1 file changed, 565 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 916cbea254f..c2d11acbc1e 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -2,20 +2,585 @@\n #include \"config.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+/*\n+ * The documentation of ReadDirectoryChangesW() states that the maximum\n+ * buffer size is 64K when the monitored directory is remote.\n+ *\n+ * Larger buffers may be used when the monitored directory is local and\n+ * will help us receive events faster from the kernel and avoid dropped\n+ * events.\n+ *\n+ * So we try to use a very large buffer and silently fallback to 64K if\n+ * we get an error.\n+ */\n+#define MAX_RDCW_BUF_FALLBACK (65536)\n+#define MAX_RDCW_BUF (65536 * 8)\n+\n+struct one_watch\n+{\n+\tchar buffer[MAX_RDCW_BUF];\n+\tDWORD buf_len;\n+\tDWORD count;\n+\n+\tstruct strbuf path;\n+\tHANDLE hDir;\n+\tHANDLE hEvent;\n+\tOVERLAPPED overlapped;\n+\n+\t/*\n+\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n+\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n+\t */\n+\tBOOL is_active;\n+};\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tstruct one_watch *watch_worktree;\n+\tstruct one_watch *watch_gitdir;\n+\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hListener[3]; /* we don't own these handles */\n+#define LISTENER_SHUTDOWN 0\n+#define LISTENER_HAVE_DATA_WORKTREE 1\n+#define LISTENER_HAVE_DATA_GITDIR 2\n+\tint nr_listener_handles;\n+};\n+\n+/*\n+ * Convert the WCHAR path from the notification into UTF8 and\n+ * then normalize it.\n+ */\n+static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+\t\t\t\t struct strbuf *normalized_path)\n+{\n+\tint reserve;\n+\tint len = 0;\n+\n+\tstrbuf_reset(normalized_path);\n+\tif (!info->FileNameLength)\n+\t\tgoto normalize;\n+\n+\t/*\n+\t * Pre-reserve enough space in the UTF8 buffer for\n+\t * each Unicode WCHAR character to be mapped into a\n+\t * sequence of 2 UTF8 characters. That should let us\n+\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n+\t */\n+\treserve = info->FileNameLength + 1;\n+\tstrbuf_grow(normalized_path, reserve);\n+\n+\tfor (;;) {\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n+\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t\t\t normalized_path->buf,\n+\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n+\t\t\t\t\t NULL, NULL);\n+\t\tif (len > 0)\n+\t\t\tgoto normalize;\n+\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n+\t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n+\t\t\t GetLastError(),\n+\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n+\t\t\t info->FileName);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tstrbuf_grow(normalized_path,\n+\t\t\t strbuf_avail(normalized_path) + reserve);\n+\t}\n+\n+normalize:\n+\tstrbuf_setlen(normalized_path, len);\n+\treturn strbuf_normalize_path(normalized_path);\n+}\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+}\n+\n+static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const char *path)\n+{\n+\tstruct one_watch *watch = NULL;\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\twchar_t wpath[MAX_PATH];\n+\n+\tif (xutftowcs_path(wpath, path) < 0) {\n+\t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n+\t\treturn NULL;\n+\t}\n+\n+\thDir = CreateFileW(wpath,\n+\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n+\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n+\t\t\t NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n+\t\t GetLastError(), path);\n+\t\treturn NULL;\n+\t}\n+\n+\tCALLOC_ARRAY(watch, 1);\n+\n+\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n+\n+\tstrbuf_init(&watch->path, 0);\n+\tstrbuf_addstr(&watch->path, path);\n+\n+\twatch->hDir = hDir;\n+\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\treturn watch;\n+}\n+\n+static void destroy_watch(struct one_watch *watch)\n+{\n+\tif (!watch)\n+\t\treturn;\n+\n+\tstrbuf_release(&watch->path);\n+\tif (watch->hDir != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hDir);\n+\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hEvent);\n+\n+\tfree(watch);\n+}\n+\n+static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+\t\t\t struct one_watch *watch)\n+{\n+\tDWORD dwNotifyFilter =\n+\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n+\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n+\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n+\t\tFILE_NOTIFY_CHANGE_SIZE |\n+\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n+\t\tFILE_NOTIFY_CHANGE_CREATION;\n+\n+\tResetEvent(watch->hEvent);\n+\n+\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n+\twatch->overlapped.hEvent = watch->hEvent;\n+\n+\t/*\n+\t * Queue an async call using Overlapped IO. This returns immediately.\n+\t * Our event handle will be signalled when the real result is available.\n+\t *\n+\t * The return value here just means that we successfully queued it.\n+\t * We won't know if the Read...() actually produces data until later.\n+\t */\n+\twatch->is_active = ReadDirectoryChangesW(\n+\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n+\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n+\n+\tif (watch->is_active)\n+\t\treturn 0;\n+\n+\terror(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static int recv_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD gle;\n+\n+\twatch->is_active = FALSE;\n+\n+\t/*\n+\t * The overlapped result is ready. If the Read...() was successful\n+\t * we finally receive the actual result into our buffer.\n+\t */\n+\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n+\t\t\t\tTRUE))\n+\t\treturn 0;\n+\n+\tgle = GetLastError();\n+\tif (gle == ERROR_INVALID_PARAMETER &&\n+\t /*\n+\t * The kernel throws an invalid parameter error when our\n+\t * buffer is too big and we are pointed at a remote\n+\t * directory (and possibly for other reasons). Quietly\n+\t * set it down and try again.\n+\t *\n+\t * See note about MAX_RDCW_BUF at the top.\n+\t */\n+\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n+\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n+\t\treturn -2;\n+\t}\n+\n+\t/*\n+\t * NEEDSWORK: If an external is deleted, the above\n+\t * returns an error. I'm not sure that there's anything that\n+\t * we can do here other than failing -- the /.git\n+\t * link file would be broken anyway. We might try to check\n+\t * for that and return a better error message, but I'm not\n+\t * sure it is worth it.\n+\t */\n+\n+\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, gle);\n+\treturn -1;\n+}\n+\n+static void cancel_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD count;\n+\n+\tif (!watch || !watch->is_active)\n+\t\treturn;\n+\n+\t/*\n+\t * The calls to ReadDirectoryChangesW() and GetOverlappedResult()\n+\t * form a \"pair\" (my term) where we queue an IO and promise to\n+\t * hang around and wait for the kernel to give us the result.\n+\t *\n+\t * If for some reason after we queue the IO, we have to quit\n+\t * or otherwise not stick around for the second half, we must\n+\t * tell the kernel to abort the IO. This prevents the kernel\n+\t * from writing to our buffer and/or signalling our event\n+\t * after we free them.\n+\t *\n+\t * (Ask me how much fun it was to track that one down).\n+\t */\n+\tCancelIoEx(watch->hDir, &watch->overlapped);\n+\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n+\twatch->is_active = FALSE;\n+}\n+\n+/*\n+ * Process filesystem events that happen anywhere (recursively) under the\n+ * root directory. For a normal working directory, this includes\n+ * both version controlled files and the contents of the .git/ directory.\n+ *\n+ * If /.git is a file, then we only see events for the file\n+ * itself.\n+ */\n+static int process_worktree_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_worktree;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tconst char *p = watch->buffer;\n+\n+\t/*\n+\t * If the kernel gets more events than will fit in the kernel\n+\t * buffer associated with our RDCW handle, it drops them and\n+\t * returns a count of zero.\n+\t *\n+\t * Yes, the call returns WITHOUT error and with length zero.\n+\t * This is the documented behavior. (My testing has confirmed\n+\t * that it also sets the last error to ERROR_NOTIFY_ENUM_DIR,\n+\t * but we do not rely on that since the function did not\n+\t * return an error and it is not documented.)\n+\t *\n+\t * (The \"overflow\" case is not ambiguous with the \"no data\" case\n+\t * because we did an INFINITE wait.)\n+\t *\n+\t * This means we have a gap in coverage. Tell the daemon layer\n+\t * to resync.\n+\t */\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\t}\n+\n+\t/*\n+\t * On Windows, `info` contains an \"array\" of paths that are\n+\t * relative to the root of whichever directory handle received\n+\t * the event.\n+\t */\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\t\t/* ignore everything inside of \"/.git/\" */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* queue normal pathname */\n+\t\t\tif (!batch)\n+\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tbatch = NULL;\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_SHUTDOWN;\n+}\n+\n+/*\n+ * Process filesystem events that happened anywhere (recursively) under the\n+ * external (such as non-primary worktrees or submodules).\n+ * We only care about cookie files that our client threads created here.\n+ *\n+ * Note that we DO NOT get filesystem events on the external \n+ * itself (it is not inside something that we are watching). In particular,\n+ * we do not get an event if the external is deleted.\n+ */\n+static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_gitdir;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *p = watch->buffer;\n+\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_GITDIR;\n+\t}\n+\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\tgoto skip_this_path;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, NULL, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_GITDIR;\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tDWORD dwWait;\n+\tint result;\n+\n+\tstate->error_code = 0;\n+\n+\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tif (data->watch_gitdir &&\n+\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tfor (;;) {\n+\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n+\t\t\t\t\t\tdata->hListener,\n+\t\t\t\t\t\tFALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_worktree);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_gitdir);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->error_code = -1;\n+\n+force_shutdown:\n+\t/*\n+\t * Tell the IPC thead pool to stop (which completes the await\n+\t * in the main thread (which will also signal this thread (if\n+\t * we are still alive))).\n+\t */\n+\tipc_server_stop_async(state->ipc_server_data);\n+\n+clean_shutdown:\n+\tcancel_rdcw_watch(data->watch_worktree);\n+\tcancel_rdcw_watch(data->watch_gitdir);\n }\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->watch_worktree = create_watch(state,\n+\t\t\t\t\t state->path_worktree_watch.buf);\n+\tif (!data->watch_worktree)\n+\t\tgoto failed;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->watch_gitdir = create_watch(state,\n+\t\t\t\t\t\t state->path_gitdir_watch.buf);\n+\t\tif (!data->watch_gitdir)\n+\t\t\tgoto failed;\n+\t}\n+\n+\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_listener_handles++;\n+\n+\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n+\t\tdata->watch_worktree->hEvent;\n+\tdata->nr_listener_handles++;\n+\n+\tif (data->watch_gitdir) {\n+\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n+\t\t\tdata->watch_gitdir->hEvent;\n+\t\tdata->nr_listener_handles++;\n+\t}\n+\n+\tstate->backend_data = data;\n+\treturn 0;\n+\n+failed:\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n-- \ngitgitgadget\n\n"},{"id":"439220","messageId":"65821da5b03d9e2f849da376e6ecbff12c9e8f69.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 13/29] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:53Z","receivedAt":"2021-10-21T14:25:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to create token-ids and define the\noverall token naming scheme.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 115 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1618f1aa27f..2997c2cba8c 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -93,6 +93,120 @@ static int do_as_client__status(void)\n \t}\n }\n \n+/*\n+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n+ * \"token\" as a virtual timestamp. Clients can request a summary of all\n+ * created/deleted/modified files relative to a token. In the response,\n+ * clients receive a new token for the next (relative) request.\n+ *\n+ *\n+ * Token Format\n+ * ============\n+ *\n+ * The contents of the token are private and provider-specific.\n+ *\n+ * For the built-in fsmonitor--daemon, we define a token as follows:\n+ *\n+ * \"builtin\" \":\" \":\" \n+ *\n+ * The \"builtin\" prefix is used as a namespace to avoid conflicts\n+ * with other providers (such as Watchman).\n+ *\n+ * The is an arbitrary OPAQUE string, such as a GUID,\n+ * UUID, or {timestamp,pid}. It is used to group all filesystem\n+ * events that happened while the daemon was monitoring (and in-sync\n+ * with the filesystem).\n+ *\n+ * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n+ * and does not define less-than/greater-than relationships.\n+ * (There are too many race conditions to rely on file system\n+ * event timestamps.)\n+ *\n+ * The is a simple integer incremented whenever the\n+ * daemon needs to make its state public. For example, if 1000 file\n+ * system events come in, but no clients have requested the data,\n+ * the daemon can continue to accumulate file changes in the same\n+ * bin and does not need to advance the sequence number. However,\n+ * as soon as a client does arrive, the daemon needs to start a new\n+ * bin and increment the sequence number.\n+ *\n+ * The sequence number serves as the boundary between 2 sets\n+ * of bins -- the older ones that the client has already seen\n+ * and the newer ones that it hasn't.\n+ *\n+ * When a new is created, the is reset to\n+ * zero.\n+ *\n+ *\n+ * About Token Ids\n+ * ===============\n+ *\n+ * A new token_id is created:\n+ *\n+ * [1] each time the daemon is started.\n+ *\n+ * [2] any time that the daemon must re-sync with the filesystem\n+ * (such as when the kernel drops or we miss events on a very\n+ * active volume).\n+ *\n+ * [3] in response to a client \"flush\" command (for dropped event\n+ * testing).\n+ *\n+ * When a new token_id is created, the daemon is free to discard all\n+ * cached filesystem events associated with any previous token_ids.\n+ * Events associated with a non-current token_id will never be sent\n+ * to a client. A token_id change implicitly means that the daemon\n+ * has gap in its event history.\n+ *\n+ * Therefore, clients that present a token with a stale (non-current)\n+ * token_id will always be given a trivial response.\n+ */\n+struct fsmonitor_token_data {\n+\tstruct strbuf token_id;\n+\tstruct fsmonitor_batch *batch_head;\n+\tstruct fsmonitor_batch *batch_tail;\n+\tuint64_t client_ref_count;\n+};\n+\n+static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n+{\n+\tstatic int test_env_value = -1;\n+\tstatic uint64_t flush_count = 0;\n+\tstruct fsmonitor_token_data *token;\n+\n+\tCALLOC_ARRAY(token, 1);\n+\n+\tstrbuf_init(&token->token_id, 0);\n+\ttoken->batch_head = NULL;\n+\ttoken->batch_tail = NULL;\n+\ttoken->client_ref_count = 0;\n+\n+\tif (test_env_value < 0)\n+\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n+\n+\tif (!test_env_value) {\n+\t\tstruct timeval tv;\n+\t\tstruct tm tm;\n+\t\ttime_t secs;\n+\n+\t\tgettimeofday(&tv, NULL);\n+\t\tsecs = tv.tv_sec;\n+\t\tgmtime_r(&secs, &tm);\n+\n+\t\tstrbuf_addf(&token->token_id,\n+\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n+\t\t\t flush_count++,\n+\t\t\t getpid(),\n+\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n+\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n+\t\t\t (long)tv.tv_usec);\n+\t} else {\n+\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n+\t}\n+\n+\treturn token;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -285,7 +399,7 @@ static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n-\tstate.current_token_data = NULL;\n+\tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n \tstrbuf_init(&state.path_worktree_watch, 0);\n-- \ngitgitgadget\n\n"},{"id":"439221","messageId":"862bbfcc32ef5d4136abed3bbb64a51595a5d60b.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 16/29] compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:56Z","receivedAt":"2021-10-21T14:25:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInclude MacOS system declarations to allow us to use FSEvent and\nCoreFoundation APIs. We need GCC and clang versions because of\ncompiler and header file conflicts.\n\nWhile it is quite possible to #include Apple's CoreServices.h when\ncompiling C source code with clang, trying to build it with GCC\ncurrently fails with this error:\n\nIn file included\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n\nThe underlying reason is that GCC (rightfully) objects that an `enum`\nvalue such as `kAuthorizationExternalFormLength` is not a constant\n(because it is not, the preprocessor has no knowledge of it, only the\nactual C compiler does) and can therefore not be used to define the size\nof a C array.\n\nThis is a known problem and tracked in GCC's bug tracker:\nhttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nIn the meantime, let's not block things and go the slightly ugly route\nof declaring/defining the FSEvents constants, data structures and\nfunctions that we need, so that we can avoid above-mentioned issue.\n\nLet's do this _only_ for GCC, though, so that the CI/PR builds (which\nbuild both with clang and with GCC) can guarantee that we _are_ using\nthe correct data types.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 96 ++++++++++++++++++++++++++++\n 1 file changed, 96 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex c84e3344ab9..f424253d3eb 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -1,3 +1,99 @@\n+#if defined(__GNUC__)\n+/*\n+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n+ * with clang, but not with GCC as of time of writing.\n+ *\n+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details.\n+ */\n+typedef unsigned int FSEventStreamCreateFlags;\n+#define kFSEventStreamEventFlagNone 0x00000000\n+#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n+#define kFSEventStreamEventFlagUserDropped 0x00000002\n+#define kFSEventStreamEventFlagKernelDropped 0x00000004\n+#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n+#define kFSEventStreamEventFlagHistoryDone 0x00000010\n+#define kFSEventStreamEventFlagRootChanged 0x00000020\n+#define kFSEventStreamEventFlagMount 0x00000040\n+#define kFSEventStreamEventFlagUnmount 0x00000080\n+#define kFSEventStreamEventFlagItemCreated 0x00000100\n+#define kFSEventStreamEventFlagItemRemoved 0x00000200\n+#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n+#define kFSEventStreamEventFlagItemRenamed 0x00000800\n+#define kFSEventStreamEventFlagItemModified 0x00001000\n+#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n+#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n+#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n+#define kFSEventStreamEventFlagItemIsFile 0x00010000\n+#define kFSEventStreamEventFlagItemIsDir 0x00020000\n+#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n+#define kFSEventStreamEventFlagOwnEvent 0x00080000\n+#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n+#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+\n+typedef struct __FSEventStream *FSEventStreamRef;\n+typedef const FSEventStreamRef ConstFSEventStreamRef;\n+\n+typedef unsigned int CFStringEncoding;\n+#define kCFStringEncodingUTF8 0x08000100\n+\n+typedef const struct __CFString *CFStringRef;\n+typedef const struct __CFArray *CFArrayRef;\n+typedef const struct __CFRunLoop *CFRunLoopRef;\n+\n+struct FSEventStreamContext {\n+ long long version;\n+ void *cb_data, *retain, *release, *copy_description;\n+};\n+\n+typedef struct FSEventStreamContext FSEventStreamContext;\n+typedef unsigned int FSEventStreamEventFlags;\n+#define kFSEventStreamCreateFlagNoDefer 0x02\n+#define kFSEventStreamCreateFlagWatchRoot 0x04\n+#define kFSEventStreamCreateFlagFileEvents 0x10\n+\n+typedef unsigned long long FSEventStreamEventId;\n+#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL\n+\n+typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef,\n+\t\t\t\t void *context,\n+\t\t\t\t __SIZE_TYPE__ num_of_events,\n+\t\t\t\t void *event_paths,\n+\t\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t\t const FSEventStreamEventId event_ids[]);\n+typedef double CFTimeInterval;\n+FSEventStreamRef FSEventStreamCreate(void *allocator,\n+\t\t\t\t FSEventStreamCallback callback,\n+\t\t\t\t FSEventStreamContext *context,\n+\t\t\t\t CFArrayRef paths_to_watch,\n+\t\t\t\t FSEventStreamEventId since_when,\n+\t\t\t\t CFTimeInterval latency,\n+\t\t\t\t FSEventStreamCreateFlags flags);\n+CFStringRef CFStringCreateWithCString(void *allocator, const char *string,\n+\t\t\t\t CFStringEncoding encoding);\n+CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count,\n+\t\t\t void *callbacks);\n+void CFRunLoopRun(void);\n+void CFRunLoopStop(CFRunLoopRef run_loop);\n+CFRunLoopRef CFRunLoopGetCurrent(void);\n+extern CFStringRef kCFRunLoopDefaultMode;\n+void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream,\n+\t\t\t\t CFRunLoopRef run_loop,\n+\t\t\t\t CFStringRef run_loop_mode);\n+unsigned char FSEventStreamStart(FSEventStreamRef stream);\n+void FSEventStreamStop(FSEventStreamRef stream);\n+void FSEventStreamInvalidate(FSEventStreamRef stream);\n+void FSEventStreamRelease(FSEventStreamRef stream);\n+#else\n+/*\n+ * Let Apple's headers declare `isalnum()` first, before\n+ * Git's headers override it via a constant\n+ */\n+#include \n+#include \n+#include \n+#endif\n+\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n-- \ngitgitgadget\n\n"},{"id":"439222","messageId":"40d9a816b52e1e3c389eb3239a65800b90fabd82.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 17/29] compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:57Z","receivedAt":"2021-10-21T14:25:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement file system event listener on MacOS using FSEvent,\nCoreFoundation, and CoreServices.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 382 ++++++++++++++++++++++++++-\n 1 file changed, 381 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex f424253d3eb..2aefdc14d89 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -1,4 +1,4 @@\n-#if defined(__GNUC__)\n+#ifndef __clang__\n /*\n * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n * with clang, but not with GCC as of time of writing.\n@@ -97,20 +97,400 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tCFStringRef cfsr_worktree_path;\n+\tCFStringRef cfsr_gitdir_path;\n+\n+\tCFArrayRef cfar_paths_to_watch;\n+\tint nr_paths_watching;\n+\n+\tFSEventStreamRef stream;\n+\n+\tCFRunLoopRef rl;\n+\n+\tenum shutdown_style {\n+\t\tSHUTDOWN_EVENT = 0,\n+\t\tFORCE_SHUTDOWN,\n+\t\tFORCE_ERROR_STOP,\n+\t} shutdown_style;\n+\n+\tunsigned int stream_scheduled:1;\n+\tunsigned int stream_started:1;\n+};\n+\n+static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n+\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n+\tif (flag & kFSEventStreamEventFlagUserDropped)\n+\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n+\tif (flag & kFSEventStreamEventFlagKernelDropped)\n+\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n+\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n+\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n+\tif (flag & kFSEventStreamEventFlagHistoryDone)\n+\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n+\tif (flag & kFSEventStreamEventFlagRootChanged)\n+\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n+\tif (flag & kFSEventStreamEventFlagMount)\n+\t\tstrbuf_addstr(&msg, \"Mount|\");\n+\tif (flag & kFSEventStreamEventFlagUnmount)\n+\t\tstrbuf_addstr(&msg, \"Unmount|\");\n+\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n+\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n+\tif (flag & kFSEventStreamEventFlagItemCreated)\n+\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n+\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n+\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n+\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsDir)\n+\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsFile)\n+\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemModified)\n+\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n+\tif (flag & kFSEventStreamEventFlagItemRemoved)\n+\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n+\tif (flag & kFSEventStreamEventFlagItemRenamed)\n+\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n+\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n+\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n+\tif (flag & kFSEventStreamEventFlagOwnEvent)\n+\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n+\tif (flag & kFSEventStreamEventFlagItemCloned)\n+\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\t\t\t path, flag, msg.buf);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRemoved);\n+}\n+\n+static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRenamed);\n+}\n+\n+static int ef_is_dropped(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagMustScanSubDirs ||\n+\t\tef & kFSEventStreamEventFlagKernelDropped ||\n+\t\tef & kFSEventStreamEventFlagUserDropped);\n+}\n+\n+static void fsevent_callback(ConstFSEventStreamRef streamRef,\n+\t\t\t void *ctx,\n+\t\t\t size_t num_of_events,\n+\t\t\t void *event_paths,\n+\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t const FSEventStreamEventId event_ids[])\n+{\n+\tstruct fsmonitor_daemon_state *state = ctx;\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tchar **paths = (char **)event_paths;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *path_k;\n+\tconst char *slash;\n+\tint k;\n+\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t/*\n+\t * Build a list of all filesystem changes into a private/local\n+\t * list and without holding any locks.\n+\t */\n+\tfor (k = 0; k < num_of_events; k++) {\n+\t\t/*\n+\t\t * On Mac, we receive an array of absolute paths.\n+\t\t */\n+\t\tpath_k = paths[k];\n+\n+\t\t/*\n+\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n+\t\t * Please don't log them to Trace2.\n+\t\t *\n+\t\t * trace_printf_key(&trace_fsmonitor, \"Path: '%s'\", path_k);\n+\t\t */\n+\n+\t\t/*\n+\t\t * If event[k] is marked as dropped, we assume that we have\n+\t\t * lost sync with the filesystem and should flush our cached\n+\t\t * data. We need to:\n+\t\t *\n+\t\t * [1] Abort/wake any client threads waiting for a cookie and\n+\t\t * flush the cached state data (the current token), and\n+\t\t * create a new token.\n+\t\t *\n+\t\t * [2] Discard the batch that we were locally building (since\n+\t\t * they are conceptually relative to the just flushed\n+\t\t * token).\n+\t\t */\n+\t\tif (ef_is_dropped(event_flags[k])) {\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\tfsmonitor_force_resync(state);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t\tstring_list_clear(&cookie_list, 0);\n+\n+\t\t\t/*\n+\t\t\t * We assume that any events that we received\n+\t\t\t * in this callback after this dropped event\n+\t\t\t * may still be valid, so we continue rather\n+\t\t\t * than break. (And just in case there is a\n+\t\t\t * delete of \".git\" hiding in there.)\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n+\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git or gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path_k);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path_k);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\t/* ignore all other paths inside of .git or gitdir */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\tcase IS_GITDIR:\n+\t\t\t/*\n+\t\t\t * If .git directory is deleted or renamed away,\n+\t\t\t * we have to quit.\n+\t\t\t */\n+\t\t\tif (ef_is_root_delete(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir renamed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* try to queue normal pathnames */\n+\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\t/*\n+\t\t\t * Because of the implicit \"binning\" (the\n+\t\t\t * kernel calls us at a given frequency) and\n+\t\t\t * de-duping (the kernel is free to combine\n+\t\t\t * multiple events for a given pathname), an\n+\t\t\t * individual fsevent could be marked as both\n+\t\t\t * a file and directory. Add it to the queue\n+\t\t\t * with both spellings so that the client will\n+\t\t\t * know how much to invalidate/refresh.\n+\t\t\t */\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t}\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tstrbuf_reset(&tmp);\n+\t\t\t\tstrbuf_addstr(&tmp, rel);\n+\t\t\t\tstrbuf_addch(&tmp, '/');\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\n+\t\tcase IS_OUTSIDE_CONE:\n+\t\tdefault:\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignoring '%s'\", path_k);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\n+\tdata->shutdown_style = FORCE_SHUTDOWN;\n+\tCFRunLoopStop(data->rl);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+}\n+\n+/*\n+ * NEEDSWORK: Investigate the proper value for the `latency` argument\n+ * in the call to `FSEventStreamCreate()`. I'm not sure that this\n+ * needs to be a config setting or just something that we tune after\n+ * some testing.\n+ *\n+ * With a latency of 0.1, I was seeing lots of dropped events during\n+ * the \"touch 100000\" files test within t/perf/p7519, but with a\n+ * latency of 0.001 I did not see any dropped events. So the\n+ * \"correct\" value may be somewhere in between.\n+ *\n+ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n+ */\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n+\t\tkFSEventStreamCreateFlagWatchRoot |\n+\t\tkFSEventStreamCreateFlagFileEvents;\n+\tFSEventStreamContext ctx = {\n+\t\t0,\n+\t\tstate,\n+\t\tNULL,\n+\t\tNULL,\n+\t\tNULL\n+\t};\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\tconst void *dir_array[2];\n+\n+\tCALLOC_ARRAY(data, 1);\n+\tstate->backend_data = data;\n+\n+\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n+\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n+\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n+\t\t\tNULL, state->path_gitdir_watch.buf,\n+\t\t\tkCFStringEncodingUTF8);\n+\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n+\t}\n+\n+\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n+\t\t\t\t\t\t data->nr_paths_watching,\n+\t\t\t\t\t\t NULL);\n+\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n+\t\t\t\t\t data->cfar_paths_to_watch,\n+\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n+\t\t\t\t\t 0.001, flags);\n+\tif (data->stream == NULL)\n+\t\tgoto failed;\n+\n+\t/*\n+\t * `data->rl` needs to be set inside the listener thread.\n+\t */\n+\n+\treturn 0;\n+\n+failed:\n+\terror(\"Unable to create FSEventStream.\");\n+\n+\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tif (data->stream) {\n+\t\tif (data->stream_started)\n+\t\t\tFSEventStreamStop(data->stream);\n+\t\tif (data->stream_scheduled)\n+\t\t\tFSEventStreamInvalidate(data->stream);\n+\t\tFSEventStreamRelease(data->stream);\n+\t}\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\tdata->shutdown_style = SHUTDOWN_EVENT;\n+\n+\tCFRunLoopStop(data->rl);\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\n+\tdata->rl = CFRunLoopGetCurrent();\n+\n+\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n+\tdata->stream_scheduled = 1;\n+\n+\tif (!FSEventStreamStart(data->stream)) {\n+\t\terror(\"Failed to start the FSEventStream\");\n+\t\tgoto force_error_stop_without_loop;\n+\t}\n+\tdata->stream_started = 1;\n+\n+\tCFRunLoopRun();\n+\n+\tswitch (data->shutdown_style) {\n+\tcase FORCE_ERROR_STOP:\n+\t\tstate->error_code = -1;\n+\t\t/* fall thru */\n+\tcase FORCE_SHUTDOWN:\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\t/* fall thru */\n+\tcase SHUTDOWN_EVENT:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn;\n+\n+force_error_stop_without_loop:\n+\tstate->error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+\treturn;\n }\n-- \ngitgitgadget\n\n"},{"id":"439223","messageId":"241962894f1a25e9422621dc80baf42d70ea1433.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 18/29] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:58Z","receivedAt":"2021-10-21T14:25:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to respond to IPC requests from client\nGit processes and respond with a list of modified pathnames\nrelative to the provided token.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 312 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 310 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 34603f23053..4c88171e06c 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,6 +7,7 @@\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n+#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon start []\"),\n@@ -351,6 +352,311 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+/*\n+ * Format an opaque token string to send to the client.\n+ */\n+static void with_lock__format_response_token(\n+\tstruct strbuf *response_token,\n+\tconst struct strbuf *response_token_id,\n+\tconst struct fsmonitor_batch *batch)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstrbuf_reset(response_token);\n+\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n+\t\t response_token_id->buf, batch->batch_seq_nr);\n+}\n+\n+/*\n+ * Parse an opaque token from the client.\n+ * Returns -1 on error.\n+ */\n+static int fsmonitor_parse_client_token(const char *buf_token,\n+\t\t\t\t\tstruct strbuf *requested_token_id,\n+\t\t\t\t\tuint64_t *seq_nr)\n+{\n+\tconst char *p;\n+\tchar *p_end;\n+\n+\tstrbuf_reset(requested_token_id);\n+\t*seq_nr = 0;\n+\n+\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n+\t\treturn -1;\n+\n+\twhile (*p && *p != ':')\n+\t\tstrbuf_addch(requested_token_id, *p++);\n+\tif (!*p++)\n+\t\treturn -1;\n+\n+\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n+\tif (*p_end)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal)\n+\n+static int do_handle_client(struct fsmonitor_daemon_state *state,\n+\t\t\t const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\tstruct fsmonitor_token_data *token_data = NULL;\n+\tstruct strbuf response_token = STRBUF_INIT;\n+\tstruct strbuf requested_token_id = STRBUF_INIT;\n+\tstruct strbuf payload = STRBUF_INIT;\n+\tuint64_t requested_oldest_seq_nr = 0;\n+\tuint64_t total_response_len = 0;\n+\tconst char *p;\n+\tconst struct fsmonitor_batch *batch_head;\n+\tconst struct fsmonitor_batch *batch;\n+\tintmax_t count = 0, duplicates = 0;\n+\tkh_str_t *shown;\n+\tint hash_ret;\n+\tint do_trivial = 0;\n+\tint do_flush = 0;\n+\n+\t/*\n+\t * We expect `command` to be of the form:\n+\t *\n+\t * := quit NUL\n+\t * | flush NUL\n+\t * | NUL\n+\t * | NUL\n+\t */\n+\n+\tif (!strcmp(command, \"quit\")) {\n+\t\t/*\n+\t\t * A client has requested over the socket/pipe that the\n+\t\t * daemon shutdown.\n+\t\t *\n+\t\t * Tell the IPC thread pool to shutdown (which completes\n+\t\t * the await in the main thread (which can stop the\n+\t\t * fsmonitor listener thread)).\n+\t\t *\n+\t\t * There is no reply to the client.\n+\t\t */\n+\t\treturn SIMPLE_IPC_QUIT;\n+\n+\t} else if (!strcmp(command, \"flush\")) {\n+\t\t/*\n+\t\t * Flush all of our cached data and generate a new token\n+\t\t * just like if we lost sync with the filesystem.\n+\t\t *\n+\t\t * Then send a trivial response using the new token.\n+\t\t */\n+\t\tdo_flush = 1;\n+\t\tdo_trivial = 1;\n+\n+\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n+\t\t/* assume V1 timestamp or garbage */\n+\n+\t\tchar *p_end;\n+\n+\t\tstrtoumax(command, &p_end, 10);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t ((*p_end) ?\n+\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n+\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n+\t\t\t\t command);\n+\t\tdo_trivial = 1;\n+\n+\t} else {\n+\t\t/* We have \"builtin:*\" */\n+\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n+\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n+\t\t\t\t\t command);\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We have a V2 valid token:\n+\t\t\t * \"builtin::\"\n+\t\t\t */\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (!state->current_token_data)\n+\t\tBUG(\"fsmonitor state does not have a current token\");\n+\n+\tif (do_flush)\n+\t\twith_lock__do_force_resync(state);\n+\n+\t/*\n+\t * We mark the current head of the batch list as \"pinned\" so\n+\t * that the listener thread will treat this item as read-only\n+\t * (and prevent any more paths from being added to it) from\n+\t * now on.\n+\t */\n+\ttoken_data = state->current_token_data;\n+\tbatch_head = token_data->batch_head;\n+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n+\n+\t/*\n+\t * FSMonitor Protocol V2 requires that we send a response header\n+\t * with a \"new current token\" and then all of the paths that changed\n+\t * since the \"requested token\". We send the seq_nr of the just-pinned\n+\t * head batch so that future requests from a client will be relative\n+\t * to it.\n+\t */\n+\twith_lock__format_response_token(&response_token,\n+\t\t\t\t\t &token_data->token_id, batch_head);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttotal_response_len += response_token.len + 1;\n+\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n+\t\t\t response_token.buf);\n+\n+\tif (!do_trivial) {\n+\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n+\t\t\t/*\n+\t\t\t * The client last spoke to a different daemon\n+\t\t\t * instance -OR- the daemon had to resync with\n+\t\t\t * the filesystem (and lost events), so reject.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n+\t\t\t\t\t \"response/token\", \"different\");\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else if (requested_oldest_seq_nr <\n+\t\t\t token_data->batch_tail->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * The client wants older events than we have for\n+\t\t\t * this token_id. This means that the end of our\n+\t\t\t * batch list was truncated and we cannot give the\n+\t\t\t * client a complete snapshot relative to their\n+\t\t\t * request.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"client requested truncated data\");\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n+\tif (do_trivial) {\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\treply(reply_data, \"/\", 2);\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n+\t\t\t\t \"response/trivial\", 1);\n+\n+\t\tstrbuf_release(&response_token);\n+\t\tstrbuf_release(&requested_token_id);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * We're going to hold onto a pointer to the current\n+\t * token-data while we walk the list of batches of files.\n+\t * During this time, we will NOT be under the lock.\n+\t * So we ref-count it.\n+\t *\n+\t * This allows the listener thread to continue prepending\n+\t * new batches of items to the token-data (which we'll ignore).\n+\t *\n+\t * AND it allows the listener thread to do a token-reset\n+\t * (and install a new `current_token_data`).\n+\t */\n+\ttoken_data->client_ref_count++;\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * The client request is relative to the token that they sent,\n+\t * so walk the batch list backwards from the current head back\n+\t * to the batch (sequence number) they named.\n+\t *\n+\t * We use khash to de-dup the list of pathnames.\n+\t *\n+\t * NEEDSWORK: each batch contains a list of interned strings,\n+\t * so we only need to do pointer comparisons here to build the\n+\t * hash table. Currently, we're still comparing the string\n+\t * values.\n+\t */\n+\tshown = kh_init_str();\n+\tfor (batch = batch_head;\n+\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n+\t batch = batch->next) {\n+\t\tsize_t k;\n+\n+\t\tfor (k = 0; k < batch->nr; k++) {\n+\t\t\tconst char *s = batch->interned_paths[k];\n+\t\t\tsize_t s_len;\n+\n+\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n+\t\t\t\tduplicates++;\n+\t\t\telse {\n+\t\t\t\tkh_put_str(shown, s, &hash_ret);\n+\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n+\t\t\t\t\t\t count, s);\n+\n+\t\t\t\t/* Each path gets written with a trailing NUL */\n+\t\t\t\ts_len = strlen(s) + 1;\n+\n+\t\t\t\tif (payload.len + s_len >=\n+\t\t\t\t LARGE_PACKET_DATA_MAX) {\n+\t\t\t\t\treply(reply_data, payload.buf,\n+\t\t\t\t\t payload.len);\n+\t\t\t\t\ttotal_response_len += payload.len;\n+\t\t\t\t\tstrbuf_reset(&payload);\n+\t\t\t\t}\n+\n+\t\t\t\tstrbuf_add(&payload, s, s_len);\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (payload.len) {\n+\t\treply(reply_data, payload.buf, payload.len);\n+\t\ttotal_response_len += payload.len;\n+\t}\n+\n+\tkh_release_str(shown);\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (token_data->client_ref_count > 0)\n+\t\ttoken_data->client_ref_count--;\n+\n+\tif (token_data->client_ref_count == 0) {\n+\t\tif (token_data != state->current_token_data) {\n+\t\t\t/*\n+\t\t\t * The listener thread did a token-reset while we were\n+\t\t\t * walking the batch list. Therefore, this token is\n+\t\t\t * stale and can be discarded completely. If we are\n+\t\t\t * the last reader thread using this token, we own\n+\t\t\t * that work.\n+\t\t\t */\n+\t\t\tfsmonitor_free_token_data(token_data);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n+\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\tstrbuf_release(&payload);\n+\n+\treturn 0;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -358,7 +664,7 @@ static int handle_client(void *data,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n-\t/* struct fsmonitor_daemon_state *state = data; */\n+\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n \t/*\n@@ -369,10 +675,12 @@ static int handle_client(void *data,\n \tif (command_len != strlen(command))\n \t\tBUG(\"FSMonitor assumes text messages\");\n \n+\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n+\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n \n-\tresult = 0; /* TODO Do something here. */\n+\tresult = do_handle_client(state, command, reply, reply_data);\n \n \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n \n-- \ngitgitgadget\n\n"},{"id":"439224","messageId":"de6c72a9ce0ab09ea7b030852e2e894d6c6ec9b2.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 20/29] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:00Z","receivedAt":"2021-10-21T14:25:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate an IPC client to send query and flush commands to the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n t/helper/test-fsmonitor-client.c | 121 +++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 124 insertions(+)\n create mode 100644 t/helper/test-fsmonitor-client.c\n\ndiff --git a/Makefile b/Makefile\nindex 8a26696c9ed..5ad0cef69f4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -710,6 +710,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\n TEST_BUILTINS_OBJS += test-getcwd.o\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nnew file mode 100644\nindex 00000000000..f7a5b3a32fa\n--- /dev/null\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -0,0 +1,121 @@\n+/*\n+ * test-fsmonitor-client.c: client code to send commands/requests to\n+ * a `git fsmonitor--daemon` daemon.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tdie(\"fsmonitor--daemon not available on this platform\");\n+}\n+#else\n+\n+/*\n+ * Read the `.git/index` to get the last token written to the\n+ * FSMonitor Index Extension.\n+ */\n+static const char *get_token_from_index(void)\n+{\n+\tstruct index_state *istate = the_repository->index;\n+\n+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update)\n+\t\tdie(\"index file does not have fsmonitor extension\");\n+\n+\treturn istate->fsmonitor_last_update;\n+}\n+\n+/*\n+ * Send an IPC query to a `git-fsmonitor--daemon` daemon and\n+ * ask for the changes since the given token or from the last\n+ * token in the index extension.\n+ *\n+ * This will implicitly start a daemon process if necessary. The\n+ * daemon process will persist after we exit.\n+ */\n+static int do_send_query(const char *token)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\n+\tret = fsmonitor_ipc__send_query(token, &answer);\n+\tif (ret < 0)\n+\t\tdie(_(\"could not query fsmonitor--daemon\"));\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n+ * and tell it to flush its cache.\n+ *\n+ * This feature is primarily used by the test suite to simulate a loss of\n+ * sync with the filesystem where we miss kernel events.\n+ */\n+static int do_send_flush(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tconst char *subcmd;\n+\tconst char *token = NULL;\n+\n+\tconst char * const fsmonitor_client_usage[] = {\n+\t\tN_(\"test-helper fsmonitor-client query []\"),\n+\t\tN_(\"test-helper fsmonitor-client flush\"),\n+\t\tNULL,\n+\t};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING(0, \"token\", &token, N_(\"token\"),\n+\t\t\t N_(\"command token to send to the server\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc < 2)\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tsubcmd = argv[1];\n+\targv--;\n+\targc++;\n+\n+\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n+\n+\tsetup_git_directory();\n+\n+\tif (!strcmp(subcmd, \"query\"))\n+\t\treturn !!do_send_query(token);\n+\n+\tif (!strcmp(subcmd, \"flush\"))\n+\t\treturn !!do_send_flush();\n+\n+\tdie(\"Unhandled subcommand: '%s'\", subcmd);\n+}\n+#endif\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 3ce5585e53a..39119844693 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\n \t{ \"getcwd\", cmd__getcwd },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9f0f5228508..0f1074a32a6 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -21,6 +21,7 @@ int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"439225","messageId":"4e96e0667ba51fafa7da110c55f691fd39a15d56.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 22/29] t/perf: avoid copying builtin fsmonitor files into test repo","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:02Z","receivedAt":"2021-10-21T14:25:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDo not copy any of the various fsmonitor--daemon files from the .git\ndirectory of the (GIT_PREF_REPO or GIT_PERF_LARGE_REPO) source repo\ninto the test's trash directory.\n\nWhen perf tests start, they copy the contents of the source repo into\nthe test's trash directory. If fsmonitor is running in the source repo,\nthere may be control files, such as the IPC socket and/or fsmonitor\ncookie files. These should not be copied into the test repo.\n\nUnix domain sockets cannot be copied in the manner used by the test\nsetup, so if present, the test setup fails.\n\nCookie files are harmless, but we should avoid them.\n\nThe builtin fsmonitor keeps all such control files/sockets in\n.git/fsmonitor--daemon*, so it is simple to exclude them.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/perf-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex f5ed092ee59..dc29df12fd5 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -74,7 +74,7 @@ test_perf_copy_repo_contents () {\n \tfor stuff in \"$1\"/*\n \tdo\n \t\tcase \"$stuff\" in\n-\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees)\n+\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)\n \t\t\t;;\n \t\t*)\n \t\t\tcp -R \"$stuff\" \"$repo/.git/\" || exit 1\n-- \ngitgitgadget\n\n"},{"id":"439226","messageId":"704d37d2033d423c5eecda6cd5b97aacaca130c1.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 19/29] help: include fsmonitor--daemon feature flag in version info","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:24:59Z","receivedAt":"2021-10-21T14:25:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the \"feature: fsmonitor--daemon\" message to the output of\n`git version --build-options`.\n\nThe builtin FSMonitor is only available on certain platforms and\neven then only when certain Makefile flags are enabled, so print\na message in the verbose version output when it is available.\n\nThis can be used by test scripts for prereq testing. Granted, tests\ncould just try `git fsmonitor--daemon status` and look for a 128 exit\ncode or grep for a \"not supported\" message on stderr, but this is\nrather obscure.\n\nThe main advantage is that the feature message will automatically\nappear in bug reports and other support requests.\n\nSigned-off-by: Jeff Hostetler \n---\n help.c | 4 ++++\n t/test-lib.sh | 6 ++++++\n 2 files changed, 10 insertions(+)\n\ndiff --git a/help.c b/help.c\nindex 973e47cdc30..708eed5d9ad 100644\n--- a/help.c\n+++ b/help.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"parse-options.h\"\n #include \"prompt.h\"\n+#include \"fsmonitor-ipc.h\"\n \n struct category_description {\n \tuint32_t category;\n@@ -695,6 +696,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n+\n+\t\tif (fsmonitor_ipc__is_supported())\n+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n \t}\n }\n \ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 151da80c561..4013ef8906e 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1736,3 +1736,9 @@ test_lazy_prereq SHA1 '\n # Tests that verify the scheduler integration must set this locally\n # to avoid errors.\n GIT_TEST_MAINT_SCHEDULER=\"none:exit 1\"\n+\n+# Does this platform support `git fsmonitor--daemon`\n+#\n+test_lazy_prereq FSMONITOR_DAEMON '\n+\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n+'\n-- \ngitgitgadget\n\n"},{"id":"439227","messageId":"eedaa787c2ebeb56fd745ca23199b1810dd5245e.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 21/29] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:01Z","receivedAt":"2021-10-21T14:26:01Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 511 +++++++++++++++++++++++++++++++++++\n 1 file changed, 511 insertions(+)\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..eea9ca1a309\n--- /dev/null\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -0,0 +1,511 @@\n+#!/bin/sh\n+\n+test_description='built-in file system watcher'\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+stop_daemon_delete_repo () {\n+\tr=$1\n+\tgit -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n+\trm -rf $1\n+\treturn 0\n+}\n+\n+start_daemon () {\n+\tcase \"$#\" in\n+\t\t1) r=\"-C $1\";;\n+\t\t*) r=\"\";\n+\tesac\n+\n+\tgit $r fsmonitor--daemon start || return $?\n+\tgit $r fsmonitor--daemon status || return $?\n+\n+\treturn 0\n+}\n+\n+# Is a Trace2 data event present with the given catetory and key?\n+# We do not care what the value is.\n+#\n+have_t2_data_event () {\n+\tc=$1\n+\tk=$2\n+\n+\tgrep -e '\"event\":\"data\".*\"category\":\"'\"$c\"'\".*\"key\":\"'\"$k\"'\"'\n+}\n+\n+test_expect_success 'explicit daemon start and stop' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n+\n+\tgit init test_explicit &&\n+\tstart_daemon test_explicit &&\n+\n+\tgit -C test_explicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon start' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit\" &&\n+\n+\tgit init test_implicit &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n+\n+\t# query will implicitly start the daemon.\n+\t#\n+\t# for test-script simplicity, we send a V1 timestamp rather than\n+\t# a V2 token. either way, the daemon response to any query contains\n+\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n+\t# but this test case is only concerned with whether the daemon was\n+\t# implicitly started.)\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace\" \\\n+\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n+\tnul_to_q actual.filtered &&\n+\tgrep \"builtin:\" actual.filtered &&\n+\n+\t# confirm that a daemon was started in the background.\n+\t#\n+\t# since the mechanism for starting the background daemon is platform\n+\t# dependent, just confirm that the foreground command received a\n+\t# response from the daemon.\n+\n+\thave_t2_data_event fsm_client query/response-length <.git/trace &&\n+\n+\tgit -C test_implicit fsmonitor--daemon status &&\n+\tgit -C test_implicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (delete .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n+\n+\tgit init test_implicit_1 &&\n+\n+\tstart_daemon test_implicit_1 &&\n+\n+\t# deleting the .git directory will implicitly stop the daemon.\n+\trm -rf test_implicit_1/.git &&\n+\n+\t# [1] Create an empty .git directory so that the following Git\n+\t# command will stay relative to the `-C` directory.\n+\t#\n+\t# Without this, the Git command will override the requested\n+\t# -C argument and crawl out to the containing Git source tree.\n+\t# This would make the test result dependent upon whether we\n+\t# were using fsmonitor on our development worktree.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_1/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (rename .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_2\" &&\n+\n+\tgit init test_implicit_2 &&\n+\n+\tstart_daemon test_implicit_2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n+\n+\t# See [1] above.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'cannot start multiple daemons' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n+\n+\tgit init test_multiple &&\n+\n+\tstart_daemon test_multiple &&\n+\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n+\tgrep \"fsmonitor--daemon is already running\" actual &&\n+\n+\tgit -C test_multiple fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n+'\n+\n+# These tests use the main repo in the trash directory\n+\n+test_expect_success 'setup' '\n+\t>tracked &&\n+\t>modified &&\n+\t>delete &&\n+\t>rename &&\n+\tmkdir dir1 &&\n+\t>dir1/tracked &&\n+\t>dir1/modified &&\n+\t>dir1/delete &&\n+\t>dir1/rename &&\n+\tmkdir dir2 &&\n+\t>dir2/tracked &&\n+\t>dir2/modified &&\n+\t>dir2/delete &&\n+\t>dir2/rename &&\n+\tmkdir dirtorename &&\n+\t>dirtorename/a &&\n+\t>dirtorename/b &&\n+\n+\tcat >.gitignore <<-\\EOF &&\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tEOF\n+\n+\tgit -c core.useBuiltinFSMonitor= add . &&\n+\ttest_tick &&\n+\tgit -c core.useBuiltinFSMonitor= commit -m initial &&\n+\n+\tgit config core.useBuiltinFSMonitor true\n+'\n+\n+# The test already explicitly stopped (or tried to stop) the daemon.\n+# This is here in case something else fails first.\n+#\n+redundant_stop_daemon () {\n+\tgit fsmonitor--daemon stop\n+\treturn 0\n+}\n+\n+test_expect_success 'update-index implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_1\" \\\n+\t\tgit update-index --fsmonitor &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_1\n+'\n+\n+test_expect_success 'status implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_2\" \\\n+\t\tgit status >actual &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_2\n+'\n+\n+edit_files() {\n+\techo 1 >modified\n+\techo 2 >dir1/modified\n+\techo 3 >dir2/modified\n+\t>dir1/untracked\n+}\n+\n+delete_files() {\n+\trm -f delete\n+\trm -f dir1/delete\n+\trm -f dir2/delete\n+}\n+\n+create_files() {\n+\techo 1 >new\n+\techo 2 >dir1/new\n+\techo 3 >dir2/new\n+}\n+\n+rename_files() {\n+\tmv rename renamed\n+\tmv dir1/rename dir1/renamed\n+\tmv dir2/rename dir2/renamed\n+}\n+\n+file_to_directory() {\n+\trm -f delete\n+\tmkdir delete\n+\techo 1 >delete/new\n+}\n+\n+directory_to_file() {\n+\trm -rf dir1\n+\techo 1 >dir1\n+}\n+\n+verify_status() {\n+\tgit status >actual &&\n+\tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n+\tGIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect &&\n+\ttest_cmp expect actual &&\n+\techo HELLO AFTER &&\n+\tcat .git/trace &&\n+\techo HELLO AFTER\n+}\n+\n+# The next few test cases confirm that our fsmonitor daemon sees each type\n+# of OS filesystem notification that we care about. At this layer we just\n+# ensure we are getting the OS notifications and do not try to confirm what\n+# is reported by `git status`.\n+#\n+# We run a simple query after modifying the filesystem just to introduce\n+# a bit of a delay so that the trace logging from the daemon has time to\n+# get flushed to disk.\n+#\n+# We `reset` and `clean` at the bottom of each test (and before stopping the\n+# daemon) because these commands might implicitly restart the daemon.\n+\n+clean_up_repo_and_stop_daemon () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+\tgit fsmonitor--daemon stop\n+\trm -f .git/trace\n+}\n+\n+test_expect_success 'edit some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tedit_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/modified$\" .git/trace &&\n+\tgrep \"^event: dir2/modified$\" .git/trace &&\n+\tgrep \"^event: modified$\" .git/trace &&\n+\tgrep \"^event: dir1/untracked$\" .git/trace\n+'\n+\n+test_expect_success 'create some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tcreate_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/new$\" .git/trace &&\n+\tgrep \"^event: dir2/new$\" .git/trace &&\n+\tgrep \"^event: new$\" .git/trace\n+'\n+\n+test_expect_success 'delete some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdelete_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/delete$\" .git/trace &&\n+\tgrep \"^event: dir2/delete$\" .git/trace &&\n+\tgrep \"^event: delete$\" .git/trace\n+'\n+\n+test_expect_success 'rename some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\trename_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/rename$\" .git/trace &&\n+\tgrep \"^event: dir2/rename$\" .git/trace &&\n+\tgrep \"^event: rename$\" .git/trace &&\n+\tgrep \"^event: dir1/renamed$\" .git/trace &&\n+\tgrep \"^event: dir2/renamed$\" .git/trace &&\n+\tgrep \"^event: renamed$\" .git/trace\n+'\n+\n+test_expect_success 'rename directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tmv dirtorename dirrenamed &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dirtorename/*$\" .git/trace &&\n+\tgrep \"^event: dirrenamed/*$\" .git/trace\n+'\n+\n+test_expect_success 'file changes to directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tfile_to_directory &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: delete$\" .git/trace &&\n+\tgrep \"^event: delete/new$\" .git/trace\n+'\n+\n+test_expect_success 'directory changes to a file' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdirectory_to_file &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1$\" .git/trace\n+'\n+\n+# The next few test cases exercise the token-resync code. When filesystem\n+# drops events (because of filesystem velocity or because the daemon isn't\n+# polling fast enough), we need to discard the cached data (relative to the\n+# current token) and start collecting events under a new token.\n+#\n+# the 'test-tool fsmonitor-client flush' command can be used to send a\n+# \"flush\" message to a running daemon and ask it to do a flush/resync.\n+\n+test_expect_success 'flush cached data' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_flush\" &&\n+\n+\tgit init test_flush &&\n+\n+\t(\n+\t\tGIT_TEST_FSMONITOR_TOKEN=true &&\n+\t\texport GIT_TEST_FSMONITOR_TOKEN &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace_daemon\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon test_flush\n+\t) &&\n+\n+\t# The daemon should have an initial token with no events in _0 and\n+\t# then a few (probably platform-specific number of) events in _1.\n+\t# These should both have the same .\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n+\tnul_to_q actual_q0 &&\n+\n+\ttouch test_flush/file_1 &&\n+\ttouch test_flush/file_2 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n+\tnul_to_q actual_q1 &&\n+\n+\tgrep \"file_1\" actual_q1 &&\n+\n+\t# Force a flush. This will change the , reset the , and\n+\t# flush the file data. Then create some events and ensure that the file\n+\t# again appears in the cache. It should have the new .\n+\n+\ttest-tool -C test_flush fsmonitor-client flush >flush_0 &&\n+\tnul_to_q flush_q0 &&\n+\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_2 &&\n+\tnul_to_q actual_q2 &&\n+\n+\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n+\n+\ttouch test_flush/file_3 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n+\tnul_to_q actual_q3 &&\n+\n+\tgrep \"file_3\" actual_q3\n+'\n+\n+# The next few test cases create repos where the .git directory is NOT\n+# inside the one of the working directory. That is, where .git is a file\n+# that points to a directory elsewhere. This happens for submodules and\n+# non-primary worktrees.\n+\n+test_expect_success 'setup worktree base' '\n+\tgit init wt-base &&\n+\techo 1 >wt-base/file1 &&\n+\tgit -C wt-base add file1 &&\n+\tgit -C wt-base commit -m \"c1\"\n+'\n+\n+test_expect_success 'worktree with .git file' '\n+\tgit -C wt-base worktree add ../wt-secondary &&\n+\n+\t(\n+\t\tGIT_TRACE2_PERF=\"$(pwd)/trace2_wt_secondary\" &&\n+\t\texport GIT_TRACE2_PERF &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/trace_wt_secondary\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon wt-secondary\n+\t) &&\n+\n+\tgit -C wt-secondary fsmonitor--daemon stop &&\n+\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n+'\n+\n+# NEEDSWORK: Repeat one of the \"edit\" tests on wt-secondary and\n+# confirm that we get the same events and behavior -- that is, that\n+# fsmonitor--daemon correctly watches BOTH the working directory and\n+# the external GITDIR directory and behaves the same as when \".git\"\n+# is a directory inside the working directory.\n+\n+test_expect_success 'cleanup worktrees' '\n+\tstop_daemon_delete_repo wt-secondary &&\n+\tstop_daemon_delete_repo wt-base\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"439228","messageId":"1c2eccacff64edc0aba1d60a120f71cb94532e8b.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 24/29] t/perf/p7519: speed up test on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:04Z","receivedAt":"2021-10-21T14:26:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nChange p7519 to use `test_seq` and `xargs` rather than a `for` loop\nto touch thousands of files. This takes minutes off of test runs\non Windows because of process creation overhead.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 5eb5044a103..171644ffc90 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -98,6 +98,13 @@ trace_stop() {\n \tfi\n }\n \n+touch_files() {\n+\tn=$1\n+\td=\"$n\"_files\n+\n+\t(cd $d ; test_seq 1 $n | xargs touch )\n+}\n+\n test_expect_success \"one time repo setup\" '\n \t# set untrackedCache depending on the environment\n \tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n@@ -119,10 +126,11 @@ test_expect_success \"one time repo setup\" '\n \tfi &&\n \n \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n-\tfor i in $(test_seq 1 10); do touch 10_files/$i; done &&\n-\tfor i in $(test_seq 1 100); do touch 100_files/$i; done &&\n-\tfor i in $(test_seq 1 1000); do touch 1000_files/$i; done &&\n-\tfor i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n+\ttouch_files 1 &&\n+\ttouch_files 10 &&\n+\ttouch_files 100 &&\n+\ttouch_files 1000 &&\n+\ttouch_files 10000 &&\n \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n \tgit commit -qm \"Add files\" &&\n \n@@ -199,15 +207,15 @@ test_fsmonitor_suite() {\n \n \t# Update the mtimes on upto 100k files to make status think\n \t# that they are dirty. For simplicity, omit any files with\n-\t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n-\t# Then fully backslash-quote the paths to capture any\n-\t# whitespace so that they pass thru xargs properly.\n+\t# LFs (i.e. anything that ls-files thinks it needs to dquote)\n+\t# and any files with whitespace so that they pass thru xargs\n+\t# properly.\n \t#\n \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n \t\tgit ls-files | \\\n \t\t\thead -100000 | \\\n \t\t\tgrep -v \\\" | \\\n-\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n+\t\t\tegrep -v \" .\" | \\\n \t\t\txargs test-tool chmtime -300 &&\n \t\tgit status\n \t'\n-- \ngitgitgadget\n\n"},{"id":"439229","messageId":"de9c015d78c3ad87e84a1d424ebeacaf2491d1db.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 23/29] t/helper/test-chmtime: skip directories on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:03Z","receivedAt":"2021-10-21T14:26:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach `test-tool.exe chmtime` to ignore errors when setting the mtime\non a directory on Windows.\n\nNEEDSWORK: The Windows version of `utime()` (aka `mingw_utime()`) does\nnot properly handle directories because it uses `_wopen()`. It should\nbe converted to using `CreateFileW()` and backup semantics at a minimum.\nSince I'm already in the middle of a large patch series, I did not want\nto destabilize other callers of `utime()` right now. The problem has\nonly been observed in the t/perf/p7519 test when the test repo contains\nan empty directory on disk.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-chmtime.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/t/helper/test-chmtime.c b/t/helper/test-chmtime.c\nindex 524b55ca496..dc28890a183 100644\n--- a/t/helper/test-chmtime.c\n+++ b/t/helper/test-chmtime.c\n@@ -134,6 +134,21 @@ int cmd__chmtime(int argc, const char **argv)\n \t\t}\n \n \t\tif (utb.modtime != sb.st_mtime && utime(argv[i], &utb) < 0) {\n+#ifdef GIT_WINDOWS_NATIVE\n+\t\t\tif (S_ISDIR(sb.st_mode)) {\n+\t\t\t\t/*\n+\t\t\t\t * NEEDSWORK: The Windows version of `utime()`\n+\t\t\t\t * (aka `mingw_utime()`) does not correctly\n+\t\t\t\t * handle directory arguments, since it uses\n+\t\t\t\t * `_wopen()`. Ignore it for now since this\n+\t\t\t\t * is just a test.\n+\t\t\t\t */\n+\t\t\t\tfprintf(stderr,\n+\t\t\t\t\t(\"Failed to modify time on directory %s. \"\n+\t\t\t\t\t \"Skipping\\n\"), argv[i]);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+#endif\n \t\t\tfprintf(stderr, \"Failed to modify time on %s: %s\\n\",\n \t\t\t argv[i], strerror(errno));\n \t\t\treturn 1;\n-- \ngitgitgadget\n\n"},{"id":"439230","messageId":"236b5966257da68c14087b807cb4feef3f1a806f.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 25/29] t/perf/p7519: add fsmonitor--daemon test cases","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:05Z","receivedAt":"2021-10-21T14:26:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRepeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\nthe \"Simple IPC\" interface.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 37 ++++++++++++++++++++++++++++++++++---\n 1 file changed, 34 insertions(+), 3 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 171644ffc90..e70252ed65a 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -24,7 +24,8 @@ test_description=\"Test core.fsmonitor\"\n # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\n # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an\n # absolute path to an integration. May be a space delimited list of\n-# absolute paths to integrations.\n+# absolute paths to integrations. (This hook or list of hooks does not\n+# include the built-in fsmonitor--daemon.)\n #\n # The big win for using fsmonitor is the elimination of the need to scan the\n # working directory looking for changed and untracked files. If the file\n@@ -143,10 +144,16 @@ test_expect_success \"one time repo setup\" '\n \n setup_for_fsmonitor() {\n \t# set INTEGRATION_SCRIPT depending on the environment\n-\tif test -n \"$INTEGRATION_PATH\"\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n \tthen\n+\t\tgit config core.useBuiltinFSMonitor true &&\n+\t\tINTEGRATION_SCRIPT=false\n+\telif test -n \"$INTEGRATION_PATH\"\n+\tthen\n+\t\tgit config core.useBuiltinFSMonitor false &&\n \t\tINTEGRATION_SCRIPT=\"$INTEGRATION_PATH\"\n \telse\n+\t\tgit config core.useBuiltinFSMonitor false &&\n \t\t#\n \t\t# Choose integration script based on existence of Watchman.\n \t\t# Fall back to an empty integration script.\n@@ -182,7 +189,10 @@ test_perf_w_drop_caches () {\n }\n \n test_fsmonitor_suite() {\n-\tif test -n \"$INTEGRATION_SCRIPT\"; then\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n+\tthen\n+\t\tDESC=\"builtin fsmonitor--daemon\"\n+\telif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n \t\tDESC=\"fsmonitor=disabled\"\n@@ -293,4 +303,25 @@ test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n \n+#\n+# Run a full set of perf tests using the built-in fsmonitor--daemon.\n+# It does not use the Hook API, so it has a different setup.\n+# Explicitly start the daemon here and before we start client commands\n+# so that we can later add custom tracing.\n+#\n+if test_have_prereq FSMONITOR_DAEMON\n+then\n+\tUSE_FSMONITOR_DAEMON=t\n+\n+\ttrace_start fsmonitor--daemon--server\n+\tgit fsmonitor--daemon start\n+\n+\ttrace_start fsmonitor--daemon--client\n+\ttest_expect_success \"setup for fsmonitor--daemon\" 'setup_for_fsmonitor'\n+\ttest_fsmonitor_suite\n+\n+\tgit fsmonitor--daemon stop\n+\ttrace_stop\n+fi\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"439231","messageId":"54710a4830dc317797430da1ce210c5fb48d7198.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 26/29] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:06Z","receivedAt":"2021-10-21T14:26:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to periodically truncate the list of\nmodified files to save some memory.\n\nClients will ask for the set of changes relative to a token that they\nfound in the FSMN index extension in the index. (This token is like a\npoint in time, but different). Clients will then update the index to\ncontain the response token (so that subsequent commands will be\nrelative to this new token).\n\nTherefore, the daemon can gradually truncate the in-memory list of\nchanged paths as they become obsolete (older than the previous token).\nSince we may have multiple clients making concurrent requests with a\nskew of tokens and clients may be racing to the talk to the daemon,\nwe lazily truncate the list.\n\nWe introduce a 5 minute delay and truncate batches 5 minutes after\nthey are considered obsolete.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 88 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 88 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 4c88171e06c..962b24569e1 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -299,6 +299,75 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n \t\t\tbatch_src->interned_paths[k];\n }\n \n+/*\n+ * To keep the batch list from growing unbounded in response to filesystem\n+ * activity, we try to truncate old batches from the end of the list as\n+ * they become irrelevant.\n+ *\n+ * We assume that the .git/index will be updated with the most recent token\n+ * any time the index is updated. And future commands will only ask for\n+ * recent changes *since* that new token. So as tokens advance into the\n+ * future, older batch items will never be requested/needed. So we can\n+ * truncate them without loss of functionality.\n+ *\n+ * However, multiple commands may be talking to the daemon concurrently\n+ * or perform a slow command, so a little \"token skew\" is possible.\n+ * Therefore, we want this to be a little bit lazy and have a generous\n+ * delay.\n+ *\n+ * The current reader thread walked backwards in time from `token->batch_head`\n+ * back to `batch_marker` somewhere in the middle of the batch list.\n+ *\n+ * Let's walk backwards in time from that marker an arbitrary delay\n+ * and truncate the list there. Note that these timestamps are completely\n+ * artificial (based on when we pinned the batch item) and not on any\n+ * filesystem activity.\n+ *\n+ * Return the obsolete portion of the list after we have removed it from\n+ * the official list so that the caller can free it after leaving the lock.\n+ */\n+#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */\n+\n+static struct fsmonitor_batch *with_lock__truncate_old_batches(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst struct fsmonitor_batch *batch_marker)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder;\n+\n+\tif (!batch_marker)\n+\t\treturn NULL;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Truncate: mark (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t batch_marker->batch_seq_nr,\n+\t\t\t (uint64_t)batch_marker->pinned_time);\n+\n+\tfor (batch = batch_marker; batch; batch = batch->next) {\n+\t\ttime_t t;\n+\n+\t\tif (!batch->pinned_time) /* an overflow batch */\n+\t\t\tcontinue;\n+\n+\t\tt = batch->pinned_time + MY_TIME_DELAY_SECONDS;\n+\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n+\t\t\tcontinue;\n+\n+\t\tgoto truncate_past_here;\n+\t}\n+\n+\treturn NULL;\n+\n+truncate_past_here:\n+\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n+\n+\tremainder = ((struct fsmonitor_batch *)batch)->next;\n+\t((struct fsmonitor_batch *)batch)->next = NULL;\n+\n+\treturn remainder;\n+}\n+\n static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n {\n \tif (!token)\n@@ -412,6 +481,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tconst char *p;\n \tconst struct fsmonitor_batch *batch_head;\n \tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder = NULL;\n \tintmax_t count = 0, duplicates = 0;\n \tkh_str_t *shown;\n \tint hash_ret;\n@@ -641,11 +711,29 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * that work.\n \t\t\t */\n \t\t\tfsmonitor_free_token_data(token_data);\n+\t\t} else if (batch) {\n+\t\t\t/*\n+\t\t\t * We are holding the lock and are the only\n+\t\t\t * reader of the ref-counted portion of the\n+\t\t\t * list, so we get the honor of seeing if the\n+\t\t\t * list can be truncated to save memory.\n+\t\t\t *\n+\t\t\t * The main loop did not walk to the end of the\n+\t\t\t * list, so this batch is the first item in the\n+\t\t\t * batch-list that is older than the requested\n+\t\t\t * end-point sequence number. See if the tail\n+\t\t\t * end of the list is obsolete.\n+\t\t\t */\n+\t\t\tremainder = with_lock__truncate_old_batches(state,\n+\t\t\t\t\t\t\t\t batch);\n \t\t}\n \t}\n \n \tpthread_mutex_unlock(&state->main_lock);\n \n+\tif (remainder)\n+\t\tfsmonitor_batch__free_list(remainder);\n+\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n-- \ngitgitgadget\n\n"},{"id":"439232","messageId":"1e2bd77fcea7beab17eeec8ea68c4ffbc4541655.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 27/29] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:07Z","receivedAt":"2021-10-21T14:26:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon client threads to create a cookie file\ninside the .git directory and then wait until FS events for the\ncookie are observed by the FS listener thread.\n\nThis helps address the racy nature of file system events by\nblocking the client response until the kernel has drained any\nevent backlog.\n\nThis is especially important on MacOS where kernel events are\nonly issued with a limited frequency. See the `latency` argument\nof `FSeventStreamCreate()`. The kernel only signals every `latency`\nseconds, but does not guarantee that the kernel queue is completely\ndrained, so we may have to wait more than one interval. If we\nincrease the frequency, the system is more likely to drop events.\nWe avoid these issues by having each client thread create a unique\ncookie file and then wait until it is seen in the event stream.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 5 +\n 2 files changed, 232 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 962b24569e1..6011fe42ee0 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -94,6 +94,149 @@ static int do_as_client__status(void)\n \t}\n }\n \n+enum fsmonitor_cookie_item_result {\n+\tFCIR_ERROR = -1, /* could not create cookie file ? */\n+\tFCIR_INIT = 0,\n+\tFCIR_SEEN,\n+\tFCIR_ABORT,\n+};\n+\n+struct fsmonitor_cookie_item {\n+\tstruct hashmap_entry entry;\n+\tconst char *name;\n+\tenum fsmonitor_cookie_item_result result;\n+};\n+\n+static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n+\t\t const struct hashmap_entry *he2, const void *keydata)\n+{\n+\tconst struct fsmonitor_cookie_item *a =\n+\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n+\tconst struct fsmonitor_cookie_item *b =\n+\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n+\n+\treturn strcmp(a->name, keydata ? keydata : b->name);\n+}\n+\n+static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(\n+\tstruct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint fd;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tstruct strbuf cookie_pathname = STRBUF_INIT;\n+\tstruct strbuf cookie_filename = STRBUF_INIT;\n+\tenum fsmonitor_cookie_item_result result;\n+\tint my_cookie_seq;\n+\n+\tCALLOC_ARRAY(cookie, 1);\n+\n+\tmy_cookie_seq = state->cookie_seq++;\n+\n+\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n+\n+\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n+\tstrbuf_addbuf(&cookie_pathname, &cookie_filename);\n+\n+\tcookie->name = strbuf_detach(&cookie_filename, NULL);\n+\tcookie->result = FCIR_INIT;\n+\thashmap_entry_init(&cookie->entry, strhash(cookie->name));\n+\n+\thashmap_add(&state->cookies, &cookie->entry);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n+\t\t\t cookie->name, cookie_pathname.buf);\n+\n+\t/*\n+\t * Create the cookie file on disk and then wait for a notification\n+\t * that the listener thread has seen it.\n+\t */\n+\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n+\tif (fd >= 0) {\n+\t\tclose(fd);\n+\t\tunlink(cookie_pathname.buf);\n+\n+\t\t/*\n+\t\t * NEEDSWORK: This is an infinite wait (well, unless another\n+\t\t * thread sends us an abort). I'd like to change this to\n+\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n+\t\t * and let the caller do the trivial response thing.\n+\t\t */\n+\t\twhile (cookie->result == FCIR_INIT)\n+\t\t\tpthread_cond_wait(&state->cookies_cond,\n+\t\t\t\t\t &state->main_lock);\n+\t} else {\n+\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n+\t\t\t cookie->name);\n+\n+\t\tcookie->result = FCIR_ERROR;\n+\t}\n+\n+\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n+\n+\tresult = cookie->result;\n+\n+\tfree((char*)cookie->name);\n+\tfree(cookie);\n+\tstrbuf_release(&cookie_pathname);\n+\n+\treturn result;\n+}\n+\n+/*\n+ * Mark these cookies as _SEEN and wake up the corresponding client threads.\n+ */\n+static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,\n+\t\t\t\t\t const struct string_list *cookie_names)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint k;\n+\tint nr_seen = 0;\n+\n+\tfor (k = 0; k < cookie_names->nr; k++) {\n+\t\tstruct fsmonitor_cookie_item key;\n+\t\tstruct fsmonitor_cookie_item *cookie;\n+\n+\t\tkey.name = cookie_names->items[k].string;\n+\t\thashmap_entry_init(&key.entry, strhash(key.name));\n+\n+\t\tcookie = hashmap_get_entry(&state->cookies, &key, entry, NULL);\n+\t\tif (cookie) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-seen: '%s'\",\n+\t\t\t\t\t cookie->name);\n+\t\t\tcookie->result = FCIR_SEEN;\n+\t\t\tnr_seen++;\n+\t\t}\n+\t}\n+\n+\tif (nr_seen)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n+/*\n+ * Set _ABORT on all pending cookies and wake up all client threads.\n+ */\n+static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct hashmap_iter iter;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tint nr_aborted = 0;\n+\n+\thashmap_for_each_entry(&state->cookies, &iter, cookie, entry) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-abort: '%s'\",\n+\t\t\t\t cookie->name);\n+\t\tcookie->result = FCIR_ABORT;\n+\t\tnr_aborted++;\n+\t}\n+\n+\tif (nr_aborted)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n /*\n * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n * \"token\" as a virtual timestamp. Clients can request a summary of all\n@@ -391,6 +534,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n *\n+ * [2] Some of those lost events may have been for cookie files. We\n+ * should assume the worst and abort them rather letting them starve.\n+ *\n * If there are no concurrent threads readering the current token data\n * series, we can free it now. Otherwise, let the last reader free\n * it.\n@@ -412,6 +558,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n \tstate->current_token_data = new_one;\n \n \tfsmonitor_free_token_data(free_me);\n+\n+\twith_lock__abort_all_cookies(state);\n }\n \n void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n@@ -487,6 +635,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tint hash_ret;\n \tint do_trivial = 0;\n \tint do_flush = 0;\n+\tint do_cookie = 0;\n+\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n@@ -547,6 +697,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * We have a V2 valid token:\n \t\t\t * \"builtin::\"\n \t\t\t */\n+\t\t\tdo_cookie = 1;\n \t\t}\n \t}\n \n@@ -555,6 +706,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tif (!state->current_token_data)\n \t\tBUG(\"fsmonitor state does not have a current token\");\n \n+\t/*\n+\t * Write a cookie file inside the directory being watched in\n+\t * an effort to flush out existing filesystem events that we\n+\t * actually care about. Suspend this client thread until we\n+\t * see the filesystem events for this cookie file.\n+\t *\n+\t * Creating the cookie lets us guarantee that our FS listener\n+\t * thread has drained the kernel queue and we are caught up\n+\t * with the kernel.\n+\t *\n+\t * If we cannot create the cookie (or otherwise guarantee that\n+\t * we are caught up), we send a trivial response. We have to\n+\t * assume that there might be some very, very recent activity\n+\t * on the FS still in flight.\n+\t */\n+\tif (do_cookie) {\n+\t\tcookie_result = with_lock__wait_for_cookie(state);\n+\t\tif (cookie_result != FCIR_SEEN) {\n+\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n+\t\t\t cookie_result);\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n \tif (do_flush)\n \t\twith_lock__do_force_resync(state);\n \n@@ -775,7 +950,9 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n-#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+#define FSMONITOR_DIR \"fsmonitor--daemon\"\n+#define FSMONITOR_COOKIE_DIR \"cookies\"\n+#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n \n enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n \tconst char *rel)\n@@ -928,6 +1105,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n+\tif (cookie_names->nr)\n+\t\twith_lock__mark_cookies_seen(state, cookie_names);\n+\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n@@ -1019,7 +1199,9 @@ static int fsmonitor_run_daemon(void)\n \n \tmemset(&state, 0, sizeof(state));\n \n+\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n+\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n@@ -1044,6 +1226,44 @@ static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n+\t/*\n+\t * We will write filesystem syncing cookie files into\n+\t * ///-.\n+\t *\n+\t * The extra layers of subdirectories here keep us from\n+\t * changing the mtime on \".git/\" or \".git/foo/\" when we create\n+\t * or delete cookie files.\n+\t *\n+\t * There have been problems with some IDEs that do a\n+\t * non-recursive watch of the \".git/\" directory and run a\n+\t * series of commands any time something happens.\n+\t *\n+\t * For example, if we place our cookie files directly in\n+\t * \".git/\" or \".git/foo/\" then a `git status` (or similar\n+\t * command) from the IDE will cause a cookie file to be\n+\t * created in one of those dirs. This causes the mtime of\n+\t * those dirs to change. This triggers the IDE's watch\n+\t * notification. This triggers the IDE to run those commands\n+\t * again. And the process repeats and the machine never goes\n+\t * idle.\n+\t *\n+\t * Adding the extra layers of subdirectories prevents the\n+\t * mtime of \".git/\" and \".git/foo\" from changing when a\n+\t * cookie file is created.\n+\t */\n+\tstrbuf_init(&state.path_cookie_prefix, 0);\n+\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1056,6 +1276,7 @@ static int fsmonitor_run_daemon(void)\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n+\tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n \n@@ -1063,6 +1284,11 @@ done:\n \n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n+\tstrbuf_release(&state.path_cookie_prefix);\n+\n+\t/*\n+\t * NEEDSWORK: Consider \"rm -rf /\"\n+\t */\n \n \treturn err;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 20a815d80f8..c16ef095688 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n \n \tstruct fsmonitor_token_data *current_token_data;\n \n+\tstruct strbuf path_cookie_prefix;\n+\tpthread_cond_t cookies_cond;\n+\tint cookie_seq;\n+\tstruct hashmap cookies;\n+\n \tint error_code;\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n-- \ngitgitgadget\n\n"},{"id":"439233","messageId":"30e61b6d1adc9e0f4a515e98cb75be06c493e711.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 28/29] fsmonitor: force update index after large responses","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:08Z","receivedAt":"2021-10-21T14:26:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMeasure the time taken to apply the FSMonitor query result\nto the index and the untracked-cache.\n\nSet the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when\nFSMonitor returns a very large repsonse to ensure that the index is\nwritten to disk.\n\nNormally, when the FSMonitor response includes a tracked file, the\nindex is always updated. Similarly, the index might be updated when\nthe response alters the untracked-cache (when enabled). However, in\ncases where neither of those cause the index to be considered changed,\nthe FSMonitor response is wasted. Subsequent Git commands will make\nrequests with the same token and receive the same response.\n\nIf that response is very large, performance may suffer. It would be\nmore efficient to force update the index now (and the token in the\nindex extension) in order to reduce the size of the response received\nby future commands.\n\nThis was observed on Windows after a large checkout. On Windows, the\nkernel emits events for the files that are changed as they are\nchanged. However, it might delay events for the containing\ndirectories until the system is more idle (or someone scans the\ndirectory (so it seems)). The first status following a checkout would\nget the list of files. The subsequent status commands would get the\nlist of directories as the events trickled out. But they would never\ncatch up because the token was not advanced because the index wasn't\nupdated.\n\nThis list of directories caused `wt_status_collect_untracked()` to\nunnecessarily spend time actually scanning them during each command.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 56 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 695fb0ce4e7..2befde1ffd2 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -233,6 +233,45 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n+/*\n+ * The number of pathnames that we need to receive from FSMonitor\n+ * before we force the index to be updated.\n+ *\n+ * Note that any pathname within the set of received paths MAY cause\n+ * cache-entry or istate flag bits to be updated and thus cause the\n+ * index to be updated on disk.\n+ *\n+ * However, the response may contain many paths (such as ignored\n+ * paths) that will not update any flag bits. And thus not force the\n+ * index to be updated. (This is fine and normal.) It also means\n+ * that the token will not be updated in the FSMonitor index\n+ * extension. So the next Git command will find the same token in the\n+ * index, make the same token-relative request, and receive the same\n+ * response (plus any newly changed paths). If this response is large\n+ * (and continues to grow), performance could be impacted.\n+ *\n+ * For example, if the user runs a build and it writes 100K object\n+ * files but doesn't modify any source files, the index would not need\n+ * to be updated. The FSMonitor response (after the build and\n+ * relative to a pre-build token) might be 5MB. Each subsequent Git\n+ * command will receive that same 100K/5MB response until something\n+ * causes the index to be updated. And `refresh_fsmonitor()` will\n+ * have to iterate over those 100K paths each time.\n+ *\n+ * Performance could be improved if we optionally force update the\n+ * index after a very large response and get an updated token into\n+ * the FSMonitor index extension. This should allow subsequent\n+ * commands to get smaller and more current responses.\n+ *\n+ * The value chosen here does not need to be precise. The index\n+ * will be updated automatically the first time the user touches\n+ * a tracked file and causes a command like `git status` to\n+ * update an mtime to be updated and/or set a flag bit.\n+ *\n+ * NEEDSWORK: Does this need to be a config value?\n+ */\n+static int fsmonitor_force_update_threshold = 100;\n+\n void refresh_fsmonitor(struct index_state *istate)\n {\n \tstruct strbuf query_result = STRBUF_INIT;\n@@ -364,25 +403,39 @@ apply_results:\n \t * information and that we should consider everything\n \t * invalid. We call this a trivial response.\n \t */\n+\ttrace2_region_enter(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tif (query_success && query_result.buf[bol] != '/') {\n \t\t/*\n \t\t * Mark all pathnames returned by the monitor as dirty.\n \t\t *\n \t\t * This updates both the cache-entries and the untracked-cache.\n \t\t */\n+\t\tint count = 0;\n+\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n \t\t\t\tcontinue;\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n \t\t\tbol = i + 1;\n+\t\t\tcount++;\n \t\t}\n-\t\tif (bol < query_result.len)\n+\t\tif (bol < query_result.len) {\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tcount++;\n+\t\t}\n \n \t\t/* Now mark the untracked cache for fsmonitor usage */\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n+\n+\t\tif (count > fsmonitor_force_update_threshold)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", istate->repo, \"apply_count\",\n+\t\t\t\t count);\n+\n \t} else {\n \t\t/*\n \t\t * We received a trivial response, so invalidate everything.\n@@ -410,6 +463,8 @@ apply_results:\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 0;\n \t}\n+\ttrace2_region_leave(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tstrbuf_release(&query_result);\n \n \t/* Now that we've updated istate, save the last_update_token */\n-- \ngitgitgadget\n\n"},{"id":"439234","messageId":"507020bbef08a02afc53815754cc999c390eb7c7.1634826309.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v4 29/29] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-21T14:25:09Z","receivedAt":"2021-10-21T14:26:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate 2x2 test matrix with the untracked-cache and fsmonitor--daemon\nfeatures and a series of edits and verify that status output is\nidentical.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 96 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 96 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex eea9ca1a309..e62ec9aa3ca 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -162,6 +162,8 @@ test_expect_success 'setup' '\n \t.gitignore\n \texpect*\n \tactual*\n+\tflush*\n+\ttrace*\n \tEOF\n \n \tgit -c core.useBuiltinFSMonitor= add . &&\n@@ -508,4 +510,98 @@ test_expect_success 'cleanup worktrees' '\n \tstop_daemon_delete_repo wt-base\n '\n \n+# The next few tests perform arbitrary/contrived file operations and\n+# confirm that status is correct. That is, that the data (or lack of\n+# data) from fsmonitor doesn't cause incorrect results. And doesn't\n+# cause incorrect results when the untracked-cache is enabled.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n+\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n+\tgit update-index --no-fsmonitor &&\n+\ttest_might_fail git fsmonitor--daemon stop\n+'\n+\n+matrix_clean_up_repo () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+}\n+\n+matrix_try () {\n+\tuc=$1\n+\tfsm=$2\n+\tfn=$3\n+\n+\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n+\t\tmatrix_clean_up_repo &&\n+\t\t$fn &&\n+\t\tif test $uc = false -a $fsm = false\n+\t\tthen\n+\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n+\t\telse\n+\t\t\tgit status --porcelain=v1 >.git/actual.$fn\n+\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n+\t\tfi\n+\t'\n+\n+\treturn $?\n+}\n+\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+for uc_val in $uc_values\n+do\n+\tif test $uc_val = false\n+\tthen\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n+\t\t\tgit config core.untrackedcache false &&\n+\t\t\tgit update-index --no-untracked-cache\n+\t\t'\n+\telse\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n+\t\t\tgit config core.untrackedcache true &&\n+\t\t\tgit update-index --untracked-cache\n+\t\t'\n+\tfi\n+\n+\tfsm_values=\"false true\"\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tif test $fsm_val = false\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n+\t\t\t\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop 2>/dev/null\n+\t\t\t'\n+\t\telse\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n+\t\t\t\tgit config core.useBuiltinFSMonitor true &&\n+\t\t\t\tgit fsmonitor--daemon start &&\n+\t\t\t\tgit update-index --fsmonitor\n+\t\t\t'\n+\t\tfi\n+\n+\t\tmatrix_try $uc_val $fsm_val edit_files\n+\t\tmatrix_try $uc_val $fsm_val delete_files\n+\t\tmatrix_try $uc_val $fsm_val create_files\n+\t\tmatrix_try $uc_val $fsm_val rename_files\n+\t\tmatrix_try $uc_val $fsm_val file_to_directory\n+\t\tmatrix_try $uc_val $fsm_val directory_to_file\n+\n+\t\tif test $fsm_val = true\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n+\t\t\t\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop 2>/dev/null\n+\t\t\t'\n+\t\tfi\n+\tdone\n+done\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"439280","messageId":"xmqq1r4ekt7d.fsf@gitster.g","threadId":"56518","inReplyTo":"ecc40795fa26ea86525421682303449f70132216.1634826309.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 01/29] fsmonitor: enhance existing comments","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-21T20:40:06Z","receivedAt":"2021-10-21T20:40:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 37 ++++++++++++++++++++++++++++++-------\n> 1 file changed, 30 insertions(+), 7 deletions(-)\n>\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index ab9bfc60b34..ec4c46407c5 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -301,9 +301,25 @@ void refresh_fsmonitor(struct index_state *istate)\n> \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n> \t}\n> \n> -\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n> +\t/*\n> +\t * The response from FSMonitor (excluding the header token) is\n> +\t * either:\n> +\t *\n> +\t * [a] a (possibly empty) list of NUL delimited relative\n> +\t * pathnames of changed paths. This list can contain\n> +\t * files and directories. Directories have a trailing\n> +\t * slash.\n> +\t *\n> +\t * [b] a single '/' to indicate the provider had no\n> +\t * information and that we should consider everything\n> +\t * invalid. We call this a trivial response.\n> +\t */\n> \tif (query_success && query_result.buf[bol] != '/') {\n> -\t\t/* Mark all entries returned by the monitor as dirty */\n> +\t\t/*\n> +\t\t * Mark all pathnames returned by the monitor as dirty.\n> +\t\t *\n> +\t\t * This updates both the cache-entries and the untracked-cache.\n> +\t\t */\n\nNot a problem this patch introduces, but we only checked that the\nquery result begins with a slash, not \"we did receive a trivial\nresponse\", but the \"else\" clause of this statement pretends as if we\ndid.\n\nIt is a shame that we do have fsmonitor_is_trivial_response()\nfunction defined, but its interface is not capable of helping us\nhere.\n\nOr is fsmonitor_is_trivial_response() already good to do this, and\nreliance of [bol] this code has is the source of confusion? I\nnotice that when we have last update token and makes a call to\nquery_fsmonitor() with HOOK_INTERFACE_VERSION1, nobody updates bol\n(hence it stays 0), and with HOOK_INTERFACE_VERSION2, bol is at the\nNUL that terminates the initial string of the query result, after\nwhich presumably has either '/' NUL (trivial) or list of paths.\n\nI am not sure about the VERSION1 case, but at least in the VERSION2\ncase, making sure that the last three bytes are NUL slash NUL like\nfsmonitor_is_trivial_response() does and the half check the above is\ndoing (i.e. the byte after the NUL is slash, without making sure\nabout the length of the whole response or what follows the slash is\nNUL), seems \"close enough\" (meaning: the check in this code is a\nsloppy attempt to reinvent what _is_trivial_response() function\nalready does).\n\nSo, would it make sense to rewrite the condition to\n\n\tif (query_success &&\n\t !fsmonitor_is_trivial_response(&query_result)) {\n\nhere? Or perhaps\n\n\tif (query_success &&\n\t !(query_result.len == bol + 3 &&\n\t query_result[bol] == '/' && !query_result[bol+1])) {\n\nwhich would be open coding the _is_trivial_response() function.\n\n> \t\tbuf = query_result.buf;\n> \t\tfor (i = bol; i < query_result.len; i++) {\n> \t\t\tif (buf[i] != '\\0')\n> @@ -318,11 +334,15 @@ void refresh_fsmonitor(struct index_state *istate)\n> \t\tif (istate->untracked)\n> \t\t\tistate->untracked->use_fsmonitor = 1;\n> \t} else {\n> -\n> -\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n> -\t\t * if we actually changed entries or not */\n> +\t\t/*\n> +\t\t * We received a trivial response, so invalidate everything.\n> +\t\t *\n\n\n> +\t\t * We only want to run the post index changed hook if\n> +\t\t * we've actually changed entries, so keep track if we\n> +\t\t * actually changed entries or not.\n> +\t\t */\n> \t\tint is_cache_changed = 0;\n> -\t\t/* Mark all entries invalid */\n> +\n> \t\tfor (i = 0; i < istate->cache_nr; i++) {\n> \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n> \t\t\t\tis_cache_changed = 1;\n> @@ -330,7 +350,10 @@ void refresh_fsmonitor(struct index_state *istate)\n> \t\t\t}\n> \t\t}\n> \n> -\t\t/* If we're going to check every file, ensure we save the results */\n> +\t\t/*\n> +\t\t * If we're going to check every file, ensure we save\n> +\t\t * the results.\n> +\t\t */\n> \t\tif (is_cache_changed)\n> \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n"},{"id":"439285","messageId":"xmqqlf2mjdgf.fsf@gitster.g","threadId":"56518","inReplyTo":"882789b4dfebddb059f62b0b2edb95b92f3c69ee.1634826309.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 03/29] fsmonitor: config settings are repository-specific","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-21T21:05:36Z","receivedAt":"2021-10-21T21:05:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> \tif (fsmonitor > 0) {\n> -\t\tif (git_config_get_fsmonitor() == 0)\n> +\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n> +\n> +\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n> +\t\t\twarning(_(\"core.useBuiltinFSMonitor is unset; \"\n> +\t\t\t\t\"set it if you really want to enable the \"\n> +\t\t\t\t\"builtin fsmonitor\"));\n> \t\t\twarning(_(\"core.fsmonitor is unset; \"\n> -\t\t\t\t\"set it if you really want to \"\n> -\t\t\t\t\"enable fsmonitor\"));\n> +\t\t\t\t\"set it if you really want to enable the \"\n> +\t\t\t\t\"hook-based fsmonitor\"));\n> +\t\t}\n> \t\tadd_fsmonitor(&the_index);\n> \t\treport(_(\"fsmonitor enabled\"));\n> \t} else if (!fsmonitor) {\n> -\t\tif (git_config_get_fsmonitor() == 1)\n> +\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n> +\t\tif (fsm_mode == FSMONITOR_MODE_IPC)\n> +\t\t\twarning(_(\"core.useBuiltinFSMonitor is set; \"\n> +\t\t\t\t\"remove it if you really want to \"\n> +\t\t\t\t\"disable fsmonitor\"));\n> +\t\tif (fsm_mode == FSMONITOR_MODE_HOOK)\n> \t\t\twarning(_(\"core.fsmonitor is set; \"\n> \t\t\t\t\"remove it if you really want to \"\n> \t\t\t\t\"disable fsmonitor\"));\n\nHmph. This does not change the behaviour per-se, but what are we\ntrying to achieve with these warning messages? \n\nThe user uses --fsmonitor or --no-fsmonitor option from the command\nline, presumably as a one-shot \"this time I'd operate the command\ndifferently from the configured way\", so it seems unlikely that the\nuser is doing so because \"... really want to enable/disable\". The\n\"report()\" calls in these if/else cases seem sufficient reminder of\nwhat is going on---perhaps these warnings should be made silenceable\nby turning them into advice messages?\n\n> -int git_config_get_fsmonitor(void)\n> -{\n> -\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n> -\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n> -\n> -\tif (core_fsmonitor && !*core_fsmonitor)\n> -\t\tcore_fsmonitor = NULL;\n> -\n> -\tif (core_fsmonitor)\n> -\t\treturn 1;\n> -\n> -\treturn 0;\n> -}\n\nThis used to be how we got the configuration.\n\n> --- a/config.h\n> +++ b/config.h\n> @@ -610,7 +610,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n> int git_config_get_index_threads(int *dest);\n> int git_config_get_split_index(void);\n> int git_config_get_max_percent_split_change(void);\n> -int git_config_get_fsmonitor(void);\n\nAnd that is removed so any in-flight topic that adds new caller will\nbe caught by the compiler. OK.\n\n> diff --git a/environment.c b/environment.c\n> index 9da7f3c1a19..68f90632245 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -82,7 +82,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n> #define PROTECT_NTFS_DEFAULT 1\n> #endif\n> int protect_ntfs = PROTECT_NTFS_DEFAULT;\n> -const char *core_fsmonitor;\n\nSo is this.\n\nAll nice.\n\n> +static void lookup_fsmonitor_settings(struct repository *r)\n> +{\n> +\tstruct fsmonitor_settings *s;\n> +\n> +\tCALLOC_ARRAY(s, 1);\n> +\n> +\tr->settings.fsmonitor = s;\n> +\n> +\tif (check_for_ipc(r))\n> +\t\treturn;\n> +\n> +\tif (check_for_hook(r))\n> +\t\treturn;\n> +\n> +\tfsm_settings__set_disabled(r);\n> +}\n> +\n> +enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n> +{\n> +\tif (!r->settings.fsmonitor)\n> +\t\tlookup_fsmonitor_settings(r);\n\nOK, and these \"lookup\" calls are what make this field \"lazily\nloaded\". A helper\n\nstatic inline void lazily_load_fsmonitor_settings(struct repository *r)\n{\n\tif (!r->settings.fsmonitor)\n\t\tlookup_fsmonitor_settings(r);\n}\n\nmight be handy. Also an assert to ensure nobody calls lookup() on a\nrepository that already has lazily loaded the settings would be\nnecessary.\n\n\tstatic void lookup_fsmonitor_settings(struct repository *r)\n\t{\n\t\tif (r->settings.fsmonitor)\n\t\t\tBUG(\"...\");\n\t\tCALLOC_ARRAY(r->settings.fsmonitor, 1);\n\n> +enum fsmonitor_mode {\n> +\tFSMONITOR_MODE_DISABLED = 0,\n> +\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n> +\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n> +};\n\nPlease remind me why we need a new separate variable, instead of\nturning the core.fsmonitor variable into an extended bool ? The compatibility issues during transition is the\nsame either way. Old clients will ignore the request silently when\nyou set core.useBuiltinFSMonitor, or they will barf if you set\ncore.fsmonitor to 'builtin', so in a sense, extending the existing\nvariable may be a safer option.\n\n> diff --git a/repository.h b/repository.h\n> index a057653981c..89a1873ade7 100644\n> --- a/repository.h\n> +++ b/repository.h\n> @@ -4,6 +4,7 @@\n> #include \"path.h\"\n> \n> struct config_set;\n> +struct fsmonitor_settings;\n> struct git_hash_algo;\n> struct index_state;\n> struct lock_file;\n> @@ -34,6 +35,8 @@ struct repo_settings {\n> \tint command_requires_full_index;\n> \tint sparse_index;\n> \n> +\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n\n\"lazily\" loaded, I think.\n\n> GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n> -code path for utilizing a file system monitor to speed up detecting\n> -new or changed files.\n> +code path for utilizing a (hook based) file system monitor to speed up\n> +detecting new or changed files.\n\nNice attention to the detail here.\n\nThanks.\n"},{"id":"439286","messageId":"xmqqh7dajcxr.fsf@gitster.g","threadId":"56518","inReplyTo":"xmqqlf2mjdgf.fsf@gitster.g","subject":"Re: [PATCH v4 03/29] fsmonitor: config settings are repository-specific","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-21T21:16:48Z","receivedAt":"2021-10-21T21:16:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> +enum fsmonitor_mode {\n>> +\tFSMONITOR_MODE_DISABLED = 0,\n>> +\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n>> +\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n>> +};\n>\n> Please remind me why we need a new separate variable, instead of\n> turning the core.fsmonitor variable into an extended bool true, builtin>?\n\nAh, I see.\n\nThe vocabulary of the value for the existing variable is between\n\"unset means disabled\" and \"the path-to-hook means enabled\", so\nunless we forbid a bareword path \"builtin\" (which I do not think is\nsuch a bad idea, by the way), it becomes a bit fuzzy what a\nnon-empty token means.\n\nIn any case, the \"set to path to enable, leave unset to leave\ndisabled\" is a cumbersome to use and may want to be rethought. It\nis unclear how one would override a configured path-to-hook, for\nexample.\n\nConsidering that we need to reserve a special word, say, \"disabled\",\nthat has to be distinguishable from a normal \"here is a path to the\nhook script\" ANYWAY, in order to allow such a \"last one wins\"\nconfiguration override (or \"git -c core.fsmonitor=disabled cmd\"), it\nstarts to sound more and more reasonable to reserve yet another word\n\"builtin\" as a special value of core.fsmonitor, without having to\nintroduce a new configuration variable, no?\n\n"},{"id":"439341","messageId":"CAPig+cT1j6NsZaSM+_JgGksmH4XGsnh7Pe=dbELoBU+FisHGag@mail.gmail.com","threadId":"56518","inReplyTo":"507020bbef08a02afc53815754cc999c390eb7c7.1634826309.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 29/29] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-10-22T05:23:07Z","receivedAt":"2021-10-22T05:23:21Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Oct 21, 2021 at 10:26 AM Jeff Hostetler via GitGitGadget\n wrote:\n> Create 2x2 test matrix with the untracked-cache and fsmonitor--daemon\n> features and a series of edits and verify that status output is\n> identical.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n> @@ -508,4 +510,98 @@ test_expect_success 'cleanup worktrees' '\n> +test_lazy_prereq UNTRACKED_CACHE '\n> + { git update-index --test-untracked-cache; ret=$?; } &&\n> + test $ret -ne 1\n> +'\n\nI may be missing something obvious, but can't this be expressed more simply as:\n\n test_lazy_prereq UNTRACKED_CACHE '\n git update-index --test-untracked-cache\n test $? -ne 1\n '\n\nHow significant is it to test specifically against 1? If not, then\neven simpler would be:\n\n test_lazy_prereq UNTRACKED_CACHE '\n git update-index --test-untracked-cache\n '\n\nwhich is also more robust since it won't be fooled by die() or crashes.\n\n> +test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n> + test_might_fail git config --unset core.useBuiltinFSMonitor &&\n\nMore idiomatic:\n\n test_unconfig core.useBuiltinFSMonitor &&\n\n> + git update-index --no-fsmonitor &&\n> + test_might_fail git fsmonitor--daemon stop\n> +'\n> +\n> +matrix_clean_up_repo () {\n> + git reset --hard HEAD\n> + git clean -fd\n> +}\n\nSince calls to this function are part of the &&-chain in tests, it\nprobably would be a good idea to maintain the &&-chain within the body\nof the function, as well.\n\n> +matrix_try () {\n> + test_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n> + matrix_clean_up_repo &&\n> + $fn &&\n> + if test $uc = false -a $fsm = false\n\nWe avoid -a and -o with `test` and instead chain them with &&:\n\n if test $uc = false && test $fsm = false\n\nDocumentation/CodingGuidelines mentions this. Also see [1] & [2].\n\n[1]: https://lore.kernel.org/git/xmqqa6qkb5fi.fsf@gitster.g/\n[2]: https://lore.kernel.org/git/CAPig+cQFFsLeE921WpzTxVnBMnNRiKs4N=hUQ2UQi1VznNEQwg@mail.gmail.com/\n\n> + then\n> + git status --porcelain=v1 >.git/expect.$fn\n> + else\n> + git status --porcelain=v1 >.git/actual.$fn\n> + test_cmp .git/expect.$fn .git/actual.$fn\n> + fi\n> + '\n\nBroken &&-chain in the `else` arm.\n\n> + return $?\n> +}\n\nNo callers care about the return value of this function, so the\n`return $?` can be dropped.\n\n> +uc_values=\"false\"\n> +test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n> +for uc_val in $uc_values\n> +do\n> + if test $uc_val = false\n> + then\n> + test_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n> + git config core.untrackedcache false &&\n> + git update-index --no-untracked-cache\n> + '\n> + else\n> + test_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n> + git config core.untrackedcache true &&\n> + git update-index --untracked-cache\n> + '\n> + fi\n> +\n> + fsm_values=\"false true\"\n> + for fsm_val in $fsm_values\n> + do\n> + if test $fsm_val = false\n> + then\n> + test_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n> + test_might_fail git config --unset core.useBuiltinFSMonitor &&\n\nDitto: test_unconfig()\n\n> + git update-index --no-fsmonitor &&\n> + test_might_fail git fsmonitor--daemon stop 2>/dev/null\n> + '\n\nstderr is redirected within tests anyhow, so we normally don't\nsuppress it manually like this (especially since it may come in handy\nwhen debugging a failing test).\n\n> + else\n> + test_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n> + git config core.useBuiltinFSMonitor true &&\n> + git fsmonitor--daemon start &&\n> + git update-index --fsmonitor\n> + '\n> + fi\n> +\n> + matrix_try $uc_val $fsm_val edit_files\n> + matrix_try $uc_val $fsm_val delete_files\n> + matrix_try $uc_val $fsm_val create_files\n> + matrix_try $uc_val $fsm_val rename_files\n> + matrix_try $uc_val $fsm_val file_to_directory\n> + matrix_try $uc_val $fsm_val directory_to_file\n> +\n> + if test $fsm_val = true\n> + then\n> + test_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n> + test_might_fail git config --unset core.useBuiltinFSMonitor &&\n\nDitto: test_unconfig()\n\n> + git update-index --no-fsmonitor &&\n> + test_might_fail git fsmonitor--daemon stop 2>/dev/null\n\nDitto: stderr\n\n> + '\n> + fi\n> + done\n> +done\n"},{"id":"439807","messageId":"295e9e70-8c56-1e77-d878-dd8031b6c2c6@jeffhostetler.com","threadId":"56518","inReplyTo":"xmqq1r4ekt7d.fsf@gitster.g","subject":"Re: [PATCH v4 01/29] fsmonitor: enhance existing comments","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-10-27T18:46:02Z","receivedAt":"2021-10-27T18:46:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/21/21 4:40 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> From: Jeff Hostetler \n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> fsmonitor.c | 37 ++++++++++++++++++++++++++++++-------\n>> 1 file changed, 30 insertions(+), 7 deletions(-)\n>>\n>> diff --git a/fsmonitor.c b/fsmonitor.c\n>> index ab9bfc60b34..ec4c46407c5 100644\n>> --- a/fsmonitor.c\n>> +++ b/fsmonitor.c\n>> @@ -301,9 +301,25 @@ void refresh_fsmonitor(struct index_state *istate)\n>> \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n>> \t}\n>> \n>> -\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n>> +\t/*\n>> +\t * The response from FSMonitor (excluding the header token) is\n>> +\t * either:\n>> +\t *\n>> +\t * [a] a (possibly empty) list of NUL delimited relative\n>> +\t * pathnames of changed paths. This list can contain\n>> +\t * files and directories. Directories have a trailing\n>> +\t * slash.\n>> +\t *\n>> +\t * [b] a single '/' to indicate the provider had no\n>> +\t * information and that we should consider everything\n>> +\t * invalid. We call this a trivial response.\n>> +\t */\n>> \tif (query_success && query_result.buf[bol] != '/') {\n>> -\t\t/* Mark all entries returned by the monitor as dirty */\n>> +\t\t/*\n>> +\t\t * Mark all pathnames returned by the monitor as dirty.\n>> +\t\t *\n>> +\t\t * This updates both the cache-entries and the untracked-cache.\n>> +\t\t */\n> \n> Not a problem this patch introduces, but we only checked that the\n> query result begins with a slash, not \"we did receive a trivial\n> response\", but the \"else\" clause of this statement pretends as if we\n> did.\n> \n> It is a shame that we do have fsmonitor_is_trivial_response()\n> function defined, but its interface is not capable of helping us\n> here.\n> \n> Or is fsmonitor_is_trivial_response() already good to do this, and\n> reliance of [bol] this code has is the source of confusion? I\n> notice that when we have last update token and makes a call to\n> query_fsmonitor() with HOOK_INTERFACE_VERSION1, nobody updates bol\n> (hence it stays 0), and with HOOK_INTERFACE_VERSION2, bol is at the\n> NUL that terminates the initial string of the query result, after\n> which presumably has either '/' NUL (trivial) or list of paths.\n> \n> I am not sure about the VERSION1 case, but at least in the VERSION2\n> case, making sure that the last three bytes are NUL slash NUL like\n> fsmonitor_is_trivial_response() does and the half check the above is\n> doing (i.e. the byte after the NUL is slash, without making sure\n> about the length of the whole response or what follows the slash is\n> NUL), seems \"close enough\" (meaning: the check in this code is a\n> sloppy attempt to reinvent what _is_trivial_response() function\n> already does).\n> \n> So, would it make sense to rewrite the condition to\n> \n> \tif (query_success &&\n> \t !fsmonitor_is_trivial_response(&query_result)) {\n> \n> here? Or perhaps\n> \n> \tif (query_success &&\n> \t !(query_result.len == bol + 3 &&\n> \t query_result[bol] == '/' && !query_result[bol+1])) {\n> \n> which would be open coding the _is_trivial_response() function.\n> \n\nYes, there is an opportunity here to clean this up.\n\nThe original code was a little sloppy -- just testing buf[bol] for\nslash and assuming that that was sufficient. Technically, no response\nshould begin with slash (since the set of reported modified files are\nrelative to the root directory), so my _is_trivial_ function could be\nsimplified a little.\n\nI'll refactor this.\n\nThanks\nJeff\n"},{"id":"439811","messageId":"d7c7d96c-06c2-d330-03b6-040b7bb4a23b@jeffhostetler.com","threadId":"56518","inReplyTo":"xmqqlf2mjdgf.fsf@gitster.g","subject":"Re: [PATCH v4 03/29] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-10-27T19:03:38Z","receivedAt":"2021-10-27T19:03:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/21/21 5:05 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> \tif (fsmonitor > 0) {\n>> -\t\tif (git_config_get_fsmonitor() == 0)\n>> +\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n>> +\n>> +\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n>> +\t\t\twarning(_(\"core.useBuiltinFSMonitor is unset; \"\n>> +\t\t\t\t\"set it if you really want to enable the \"\n>> +\t\t\t\t\"builtin fsmonitor\"));\n>> \t\t\twarning(_(\"core.fsmonitor is unset; \"\n>> -\t\t\t\t\"set it if you really want to \"\n>> -\t\t\t\t\"enable fsmonitor\"));\n>> +\t\t\t\t\"set it if you really want to enable the \"\n>> +\t\t\t\t\"hook-based fsmonitor\"));\n>> +\t\t}\n>> \t\tadd_fsmonitor(&the_index);\n>> \t\treport(_(\"fsmonitor enabled\"));\n>> \t} else if (!fsmonitor) {\n>> -\t\tif (git_config_get_fsmonitor() == 1)\n>> +\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n>> +\t\tif (fsm_mode == FSMONITOR_MODE_IPC)\n>> +\t\t\twarning(_(\"core.useBuiltinFSMonitor is set; \"\n>> +\t\t\t\t\"remove it if you really want to \"\n>> +\t\t\t\t\"disable fsmonitor\"));\n>> +\t\tif (fsm_mode == FSMONITOR_MODE_HOOK)\n>> \t\t\twarning(_(\"core.fsmonitor is set; \"\n>> \t\t\t\t\"remove it if you really want to \"\n>> \t\t\t\t\"disable fsmonitor\"));\n> \n> Hmph. This does not change the behaviour per-se, but what are we\n> trying to achieve with these warning messages?\n> \n> The user uses --fsmonitor or --no-fsmonitor option from the command\n> line, presumably as a one-shot \"this time I'd operate the command\n> differently from the configured way\", so it seems unlikely that the\n> user is doing so because \"... really want to enable/disable\". The\n> \"report()\" calls in these if/else cases seem sufficient reminder of\n> what is going on---perhaps these warnings should be made silenceable\n> by turning them into advice messages?\n>\n\nThe original code had the basic warning for `core.fsmonitor`\nwhich becomes ambiguous/confusing when we have two different\nconfig values.\n\nI was really wanting to just get rid of the warnings. They\nonly appear if the users passes a `--[no-]fsmonitor` on the\ncommand line to temporarily override the config setting.\n\nI'll see about making them advice messages.\n\n\n>> -int git_config_get_fsmonitor(void)\n>> -{\n>> -\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n>> -\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n>> -\n>> -\tif (core_fsmonitor && !*core_fsmonitor)\n>> -\t\tcore_fsmonitor = NULL;\n>> -\n>> -\tif (core_fsmonitor)\n>> -\t\treturn 1;\n>> -\n>> -\treturn 0;\n>> -}\n> \n> This used to be how we got the configuration.\n> \n>> --- a/config.h\n>> +++ b/config.h\n>> @@ -610,7 +610,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n>> int git_config_get_index_threads(int *dest);\n>> int git_config_get_split_index(void);\n>> int git_config_get_max_percent_split_change(void);\n>> -int git_config_get_fsmonitor(void);\n> \n> And that is removed so any in-flight topic that adds new caller will\n> be caught by the compiler. OK.\n> \n>> diff --git a/environment.c b/environment.c\n>> index 9da7f3c1a19..68f90632245 100644\n>> --- a/environment.c\n>> +++ b/environment.c\n>> @@ -82,7 +82,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n>> #define PROTECT_NTFS_DEFAULT 1\n>> #endif\n>> int protect_ntfs = PROTECT_NTFS_DEFAULT;\n>> -const char *core_fsmonitor;\n> \n> So is this.\n> \n> All nice.\n> \n>> +static void lookup_fsmonitor_settings(struct repository *r)\n>> +{\n>> +\tstruct fsmonitor_settings *s;\n>> +\n>> +\tCALLOC_ARRAY(s, 1);\n>> +\n>> +\tr->settings.fsmonitor = s;\n>> +\n>> +\tif (check_for_ipc(r))\n>> +\t\treturn;\n>> +\n>> +\tif (check_for_hook(r))\n>> +\t\treturn;\n>> +\n>> +\tfsm_settings__set_disabled(r);\n>> +}\n>> +\n>> +enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n>> +{\n>> +\tif (!r->settings.fsmonitor)\n>> +\t\tlookup_fsmonitor_settings(r);\n> \n> OK, and these \"lookup\" calls are what make this field \"lazily\n> loaded\". A helper\n> \n> static inline void lazily_load_fsmonitor_settings(struct repository *r)\n> {\n> \tif (!r->settings.fsmonitor)\n> \t\tlookup_fsmonitor_settings(r);\n> }\n> \n> might be handy. Also an assert to ensure nobody calls lookup() on a\n> repository that already has lazily loaded the settings would be\n> necessary.\n> \n> \tstatic void lookup_fsmonitor_settings(struct repository *r)\n> \t{\n> \t\tif (r->settings.fsmonitor)\n> \t\t\tBUG(\"...\");\n> \t\tCALLOC_ARRAY(r->settings.fsmonitor, 1);\n\ngood point.\n\n\n> \n>> +enum fsmonitor_mode {\n>> +\tFSMONITOR_MODE_DISABLED = 0,\n>> +\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n>> +\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n>> +};\n> \n> Please remind me why we need a new separate variable, instead of\n> turning the core.fsmonitor variable into an extended bool true, builtin>? The compatibility issues during transition is the\n> same either way. Old clients will ignore the request silently when\n> you set core.useBuiltinFSMonitor, or they will barf if you set\n> core.fsmonitor to 'builtin', so in a sense, extending the existing\n> variable may be a safer option.\n> \n>> diff --git a/repository.h b/repository.h\n>> index a057653981c..89a1873ade7 100644\n>> --- a/repository.h\n>> +++ b/repository.h\n>> @@ -4,6 +4,7 @@\n>> #include \"path.h\"\n>> \n>> struct config_set;\n>> +struct fsmonitor_settings;\n>> struct git_hash_algo;\n>> struct index_state;\n>> struct lock_file;\n>> @@ -34,6 +35,8 @@ struct repo_settings {\n>> \tint command_requires_full_index;\n>> \tint sparse_index;\n>> \n>> +\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n> \n> \"lazily\" loaded, I think.\n> \n>> GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n>> -code path for utilizing a file system monitor to speed up detecting\n>> -new or changed files.\n>> +code path for utilizing a (hook based) file system monitor to speed up\n>> +detecting new or changed files.\n> \n> Nice attention to the detail here.\n> \n> Thanks.\n> \n"},{"id":"439820","messageId":"74282d08-aaeb-0a1e-cad3-1de17d59b4d1@jeffhostetler.com","threadId":"56518","inReplyTo":"xmqqh7dajcxr.fsf@gitster.g","subject":"Re: [PATCH v4 03/29] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-10-27T19:53:07Z","receivedAt":"2021-10-27T19:53:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/21/21 5:16 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n>>> +enum fsmonitor_mode {\n>>> +\tFSMONITOR_MODE_DISABLED = 0,\n>>> +\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n>>> +\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n>>> +};\n>>\n>> Please remind me why we need a new separate variable, instead of\n>> turning the core.fsmonitor variable into an extended bool > true, builtin>?\n> \n> Ah, I see.\n> \n> The vocabulary of the value for the existing variable is between\n> \"unset means disabled\" and \"the path-to-hook means enabled\", so\n> unless we forbid a bareword path \"builtin\" (which I do not think is\n> such a bad idea, by the way), it becomes a bit fuzzy what a\n> non-empty token means.\n> \n> In any case, the \"set to path to enable, leave unset to leave\n> disabled\" is a cumbersome to use and may want to be rethought. It\n> is unclear how one would override a configured path-to-hook, for\n> example.\n> \n> Considering that we need to reserve a special word, say, \"disabled\",\n> that has to be distinguishable from a normal \"here is a path to the\n> hook script\" ANYWAY, in order to allow such a \"last one wins\"\n> configuration override (or \"git -c core.fsmonitor=disabled cmd\"), it\n> starts to sound more and more reasonable to reserve yet another word\n> \"builtin\" as a special value of core.fsmonitor, without having to\n> introduce a new configuration variable, no?\n> \n\nFor a while we were using a \":builtin:\" reserved value (which isn't\na valid pathname on Windows) but thought it better to split it into\ntwo different config values to avoid the confusion (since it is a\nvalid path on Mac/Linux). But having 2 config vars is also confusing.\n\nAnd yes, I'm not sure there is a way for a local fsmonitor hook\nconfig to override a global hook value unless we add a \"disabled\"\nor \"off\" value.\n\nLet me revisit this. We could have:\n\n [] unset, , \"disabled\"\n [] \n [] \"builtin\", \"ipc\"\n\nand just make the literals special reserved values.\n\n\nI've not kept up on the configurable hooks series, but I have to\nwonder if this usage (pathname or reserved word) will cause problems\nwith the changes being planned for hooks. The existing fsmonitor\nusage doesn't use find_hook() nor run_hook*(), so I don't expect\nan immediate conflict. But again, I haven't kept up with that\neffort.\n\nJeff\n"},{"id":"439822","messageId":"d9c3ef61-768c-3560-2858-3438c355a742@jeffhostetler.com","threadId":"56518","inReplyTo":"CAPig+cT1j6NsZaSM+_JgGksmH4XGsnh7Pe=dbELoBU+FisHGag@mail.gmail.com","subject":"Re: [PATCH v4 29/29] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-10-27T20:06:18Z","receivedAt":"2021-10-27T20:06:21Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/22/21 1:23 AM, Eric Sunshine wrote:\n> On Thu, Oct 21, 2021 at 10:26 AM Jeff Hostetler via GitGitGadget\n> wrote:\n>> Create 2x2 test matrix with the untracked-cache and fsmonitor--daemon\n>> features and a series of edits and verify that status output is\n>> identical.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n>> @@ -508,4 +510,98 @@ test_expect_success 'cleanup worktrees' '\n>> +test_lazy_prereq UNTRACKED_CACHE '\n>> + { git update-index --test-untracked-cache; ret=$?; } &&\n>> + test $ret -ne 1\n>> +'\n> \n> I may be missing something obvious, but can't this be expressed more simply as:\n> \n> test_lazy_prereq UNTRACKED_CACHE '\n> git update-index --test-untracked-cache\n> test $? -ne 1\n> '\n> \n> How significant is it to test specifically against 1? If not, then\n> even simpler would be:\n> \n> test_lazy_prereq UNTRACKED_CACHE '\n> git update-index --test-untracked-cache\n> '\n> \n> which is also more robust since it won't be fooled by die() or crashes.\n> \n\nIIRC I stole that from t7063. It didn't occur to me to try to\nsimplify it.\n\n\n>> +test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n>> + test_might_fail git config --unset core.useBuiltinFSMonitor &&\n> \n> More idiomatic:\n> \n> test_unconfig core.useBuiltinFSMonitor &&\n> \n\nGood to know, thanks!\nI'll take a look at this and the rest of your comments below.\n\nthanks\nJeff\n\n\n>> + git update-index --no-fsmonitor &&\n>> + test_might_fail git fsmonitor--daemon stop\n>> +'\n>> +\n>> +matrix_clean_up_repo () {\n>> + git reset --hard HEAD\n>> + git clean -fd\n>> +}\n> \n> Since calls to this function are part of the &&-chain in tests, it\n> probably would be a good idea to maintain the &&-chain within the body\n> of the function, as well.\n> \n>> +matrix_try () {\n>> + test_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n>> + matrix_clean_up_repo &&\n>> + $fn &&\n>> + if test $uc = false -a $fsm = false\n> \n> We avoid -a and -o with `test` and instead chain them with &&:\n> \n> if test $uc = false && test $fsm = false\n> \n> Documentation/CodingGuidelines mentions this. Also see [1] & [2].\n> \n> [1]: https://lore.kernel.org/git/xmqqa6qkb5fi.fsf@gitster.g/\n> [2]: https://lore.kernel.org/git/CAPig+cQFFsLeE921WpzTxVnBMnNRiKs4N=hUQ2UQi1VznNEQwg@mail.gmail.com/\n> \n>> + then\n>> + git status --porcelain=v1 >.git/expect.$fn\n>> + else\n>> + git status --porcelain=v1 >.git/actual.$fn\n>> + test_cmp .git/expect.$fn .git/actual.$fn\n>> + fi\n>> + '\n> \n> Broken &&-chain in the `else` arm.\n> \n>> + return $?\n>> +}\n> \n> No callers care about the return value of this function, so the\n> `return $?` can be dropped.\n> \n>> +uc_values=\"false\"\n>> +test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n>> +for uc_val in $uc_values\n>> +do\n>> + if test $uc_val = false\n>> + then\n>> + test_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n>> + git config core.untrackedcache false &&\n>> + git update-index --no-untracked-cache\n>> + '\n>> + else\n>> + test_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n>> + git config core.untrackedcache true &&\n>> + git update-index --untracked-cache\n>> + '\n>> + fi\n>> +\n>> + fsm_values=\"false true\"\n>> + for fsm_val in $fsm_values\n>> + do\n>> + if test $fsm_val = false\n>> + then\n>> + test_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n>> + test_might_fail git config --unset core.useBuiltinFSMonitor &&\n> \n> Ditto: test_unconfig()\n> \n>> + git update-index --no-fsmonitor &&\n>> + test_might_fail git fsmonitor--daemon stop 2>/dev/null\n>> + '\n> \n> stderr is redirected within tests anyhow, so we normally don't\n> suppress it manually like this (especially since it may come in handy\n> when debugging a failing test).\n> \n>> + else\n>> + test_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n>> + git config core.useBuiltinFSMonitor true &&\n>> + git fsmonitor--daemon start &&\n>> + git update-index --fsmonitor\n>> + '\n>> + fi\n>> +\n>> + matrix_try $uc_val $fsm_val edit_files\n>> + matrix_try $uc_val $fsm_val delete_files\n>> + matrix_try $uc_val $fsm_val create_files\n>> + matrix_try $uc_val $fsm_val rename_files\n>> + matrix_try $uc_val $fsm_val file_to_directory\n>> + matrix_try $uc_val $fsm_val directory_to_file\n>> +\n>> + if test $fsm_val = true\n>> + then\n>> + test_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n>> + test_might_fail git config --unset core.useBuiltinFSMonitor &&\n> \n> Ditto: test_unconfig()\n> \n>> + git update-index --no-fsmonitor &&\n>> + test_might_fail git fsmonitor--daemon stop 2>/dev/null\n> \n> Ditto: stderr\n> \n>> + '\n>> + fi\n>> + done\n>> +done\n"},{"id":"444960","messageId":"xmqq1r203tjq.fsf@gitster.g","threadId":"56518","inReplyTo":"eedaa787c2ebeb56fd745ca23199b1810dd5245e.1634826309.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 21/29] t7527: create test for fsmonitor--daemon","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-26T03:59:37Z","receivedAt":"2021-12-26T03:59:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"While I was looking at the differences between next..seen to catch\nnew topics that try to sneak style violations into the codebase, I\nfound this one (there was another in the same series).\n\n----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----\nSubject: [PATCH] fixup! t7527: create test for fsmonitor--daemon\n\nFix style violation of a shell-function definition\n\nSigned-off-by: Junio C Hamano \n---\n t/t7527-builtin-fsmonitor.sh | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex e62ec9aa3c..313c4fad34 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -213,43 +213,43 @@ test_expect_success 'status implicitly starts daemon' '\n \ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_2\n '\n \n-edit_files() {\n+edit_files () {\n \techo 1 >modified\n \techo 2 >dir1/modified\n \techo 3 >dir2/modified\n \t>dir1/untracked\n }\n \n-delete_files() {\n+delete_files () {\n \trm -f delete\n \trm -f dir1/delete\n \trm -f dir2/delete\n }\n \n-create_files() {\n+create_files () {\n \techo 1 >new\n \techo 2 >dir1/new\n \techo 3 >dir2/new\n }\n \n-rename_files() {\n+rename_files () {\n \tmv rename renamed\n \tmv dir1/rename dir1/renamed\n \tmv dir2/rename dir2/renamed\n }\n \n-file_to_directory() {\n+file_to_directory () {\n \trm -f delete\n \tmkdir delete\n \techo 1 >delete/new\n }\n \n-directory_to_file() {\n+directory_to_file () {\n \trm -rf dir1\n \techo 1 >dir1\n }\n \n-verify_status() {\n+verify_status () {\n \tgit status >actual &&\n \tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n \tGIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect &&\n-- \n2.34.1-568-g69e9fd72b5\n\n"},{"id":"444961","messageId":"xmqqv8zc2eim.fsf@gitster.g","threadId":"56518","inReplyTo":"1c2eccacff64edc0aba1d60a120f71cb94532e8b.1634826309.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 24/29] t/perf/p7519: speed up test on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-26T04:09:37Z","receivedAt":"2021-12-26T04:09:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ditto.\n\n----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----\nSubject: [PATCH] fixup! t/perf/p7519: speed up test on Windows\n\nFix style violation of a shell-function definition introduced by\nthis step, and correct existing ones while at it.\n\nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 10 +++++-----\n 1 file changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex e70252ed65..9e08fc169a 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -73,7 +73,7 @@ then\n \tfi\n fi\n \n-trace_start() {\n+trace_start () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tname=\"$1\"\n@@ -92,14 +92,14 @@ trace_start() {\n \tfi\n }\n \n-trace_stop() {\n+trace_stop () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tunset GIT_TRACE2_PERF\n \tfi\n }\n \n-touch_files() {\n+touch_files () {\n \tn=$1\n \td=\"$n\"_files\n \n@@ -142,7 +142,7 @@ test_expect_success \"one time repo setup\" '\n \tfi\n '\n \n-setup_for_fsmonitor() {\n+setup_for_fsmonitor () {\n \t# set INTEGRATION_SCRIPT depending on the environment\n \tif test -n \"$USE_FSMONITOR_DAEMON\"\n \tthen\n@@ -188,7 +188,7 @@ test_perf_w_drop_caches () {\n \ttest_perf \"$@\"\n }\n \n-test_fsmonitor_suite() {\n+test_fsmonitor_suite () {\n \tif test -n \"$USE_FSMONITOR_DAEMON\"\n \tthen\n \t\tDESC=\"builtin fsmonitor--daemon\"\n-- \n2.34.1-568-g69e9fd72b5\n\n"},{"id":"448252","messageId":"a5ecabb4d02eb0eb7167f2b4e6be2d4f89d2bba5.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 01/30] fsmonitor: enhance existing comments, clarify trivial response handling","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:50Z","receivedAt":"2022-02-11T20:56:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 64 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 41 insertions(+), 23 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..448d0ee33f5 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -168,29 +168,15 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf *\n \n \tif (result)\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/failed\", result);\n-\telse {\n+\telse\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/response-length\",\n \t\t\t\t query_result->len);\n \n-\t\tif (fsmonitor_is_trivial_response(query_result))\n-\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n-\t\t\t\t\t \"query/trivial-response\", 1);\n-\t}\n-\n \ttrace2_region_leave(\"fsm_hook\", \"query\", NULL);\n \n \treturn result;\n }\n \n-int fsmonitor_is_trivial_response(const struct strbuf *query_result)\n-{\n-\tstatic char trivial_response[3] = { '\\0', '/', '\\0' };\n-\n-\treturn query_result->len >= 3 &&\n-\t\t!memcmp(trivial_response,\n-\t\t\t&query_result->buf[query_result->len - 3], 3);\n-}\n-\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n@@ -238,6 +224,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tint is_trivial = 0;\n \n \tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n \t\treturn;\n@@ -283,6 +270,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t\tquery_success = 0;\n \t\t\t\t} else {\n \t\t\t\t\tbol = last_update_token.len + 1;\n+\t\t\t\t\tis_trivial = query_result.buf[bol] == '/';\n \t\t\t\t}\n \t\t\t} else if (hook_version < 0) {\n \t\t\t\thook_version = HOOK_INTERFACE_VERSION1;\n@@ -294,16 +282,38 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\t\tif (query_success)\n+\t\t\t\tis_trivial = query_result.buf[0] == '/';\n \t\t}\n \n+\t\tif (is_trivial)\n+\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n \t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n \t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n-\tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n+\tif (query_success && !is_trivial) {\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -318,11 +328,16 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We failed to get a response or received a trivial response,\n+\t\t * so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -330,7 +345,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"448253","messageId":"365964b766400654a9978092f8fe5cfaa525fd3b.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 02/30] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:51Z","receivedAt":"2022-02-11T20:56:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 ++++++++++++++\n 3 files changed, 220 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 8e07003840b..dee3ba4a0b1 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -899,6 +899,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..01c8c25bf50\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,171 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(\"fsmonitor--daemon is not running\");\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\", c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..b6a7067c3af\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"448254","messageId":"384516ce1a190d836324b2c2262d611090207357.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 03/30] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:52Z","receivedAt":"2022-02-11T20:56:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove fsmonitor config settings to a new and opaque\n`struct fsmonitor_settings` structure. Add a lazily-loaded pointer\nto this into `struct repo_settings`\n\nCreate an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\nrepresent the state of fsmonitor. This lets us represent which, if\nany, fsmonitor provider (hook or IPC) is enabled.\n\nCreate `fsm_settings__get_*()` getters to lazily look up fsmonitor-\nrelated config settings.\n\nGet rid of the `core_fsmonitor` global variable. Move the code to\nlookup the existing `core.fsmonitor` config value into the fsmonitor\nsettings.\n\nCreate a hook pathname variable in `struct fsmonitor-settings` and\nonly set it when in hook mode.\n\nExtend the definition of `core.fsmonitor` to be either a boolean\nor a hook pathname. When true, the builtin FSMonitor is used.\nWhen false or unset, no FSMonitor (neither builtin nor hook) is\nused.\n\nThe existing `core_fsmonitor` global variable was used to store the\npathname to the fsmonitor hook *and* it was used as a boolean to see\nif fsmonitor was enabled. This dual usage and global visibility leads\nto confusion when we add the IPC-based provider. So lets hide the\ndetails in fsmonitor-settings.c and let it decide which provider to\nuse in the case of multiple settings. This avoids cluttering up\nrepo-settings.c with these private details.\n\nA future commit in builtin-fsmonitor series will add the ability to\ndisqualify worktrees for various reasons, such as being mounted from a\nremote volume, where fsmonitor should not be started. Having the\nconfig settings hidden in fsmonitor-settings.c allows such worktree\nrestrictions to override the config values used.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n builtin/update-index.c | 7 ++-\n cache.h | 1 -\n config.c | 14 ------\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-settings.c | 100 +++++++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 21 +++++++++\n fsmonitor.c | 63 +++++++++++++++-----------\n fsmonitor.h | 18 ++++++--\n repository.h | 3 ++\n t/README | 4 +-\n 12 files changed, 185 insertions(+), 49 deletions(-)\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\ndiff --git a/Makefile b/Makefile\nindex dee3ba4a0b1..9943f0f7c11 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -900,6 +900,7 @@ LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += fsmonitor-ipc.o\n+LIB_OBJS += fsmonitor-settings.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 7e0a0d9bf80..fed24ea1fb6 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1225,14 +1225,17 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\n \t\t\t\t\"enable fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode > FSMONITOR_MODE_DISABLED)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex 4148b6322d5..c67c426a9dd 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1000,7 +1000,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex e0c03d154c9..0494d8256d0 100644\n--- a/config.c\n+++ b/config.c\n@@ -2624,20 +2624,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex ab0106d2875..03294d340d1 100644\n--- a/config.h\n+++ b/config.h\n@@ -590,7 +590,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a5..00682e638d7 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nnew file mode 100644\nindex 00000000000..eb4d661c81e\n--- /dev/null\n+++ b/fsmonitor-settings.c\n@@ -0,0 +1,100 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+/*\n+ * We keep this structure defintion private and have getters\n+ * for all fields so that we can lazy load it as needed.\n+ */\n+struct fsmonitor_settings {\n+\tenum fsmonitor_mode mode;\n+\tchar *hook_path;\n+};\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s;\n+\tconst char *const_str;\n+\tint bool_value;\n+\n+\tif (r->settings.fsmonitor)\n+\t\treturn;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\n+\tr->settings.fsmonitor = s;\n+\n+\tfsm_settings__set_disabled(r);\n+\n+\t/*\n+\t * Overload the existing \"core.fsmonitor\" config setting (which\n+\t * has historically been either unset or a hook pathname) to\n+\t * now allow a boolean value to enable the builtin FSMonitor\n+\t * or to turn everything off. (This does imply that you can't\n+\t * use a hook script named \"true\" or \"false\", but that's OK.)\n+\t */\n+\tswitch (repo_config_get_maybe_bool(r, \"core.fsmonitor\", &bool_value)) {\n+\n+\tcase 0: /* config value was set to */\n+\t\tif (bool_value)\n+\t\t\tfsm_settings__set_ipc(r);\n+\t\treturn;\n+\n+\tcase 1: /* config value was unset */\n+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n+\t\tbreak;\n+\n+\tcase -1: /* config value set to an arbitrary string */\n+\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n+\t\t\treturn; /* should not happen */\n+\t\tbreak;\n+\n+\tdefault: /* should not happen */\n+\t\treturn;\n+\t}\n+\n+\tif (!const_str || !*const_str)\n+\t\treturn;\n+\n+\tfsm_settings__set_hook(r, const_str);\n+}\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n+{\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->mode;\n+}\n+\n+const char *fsm_settings__get_hook_path(struct repository *r)\n+{\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->hook_path;\n+}\n+\n+void fsm_settings__set_ipc(struct repository *r)\n+{\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\n+\n+void fsm_settings__set_hook(struct repository *r, const char *path)\n+{\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+\tr->settings.fsmonitor->hook_path = strdup(path);\n+}\n+\n+void fsm_settings__set_disabled(struct repository *r)\n+{\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nnew file mode 100644\nindex 00000000000..a4c5d7b4889\n--- /dev/null\n+++ b/fsmonitor-settings.h\n@@ -0,0 +1,21 @@\n+#ifndef FSMONITOR_SETTINGS_H\n+#define FSMONITOR_SETTINGS_H\n+\n+struct repository;\n+\n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n+\tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r);\n+void fsm_settings__set_hook(struct repository *r, const char *path);\n+void fsm_settings__set_disabled(struct repository *r);\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+const char *fsm_settings__get_hook_path(struct repository *r);\n+\n+struct fsmonitor_settings;\n+\n+#endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 448d0ee33f5..17cba68235a 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tstrvec_push(&cp.args, fsm_settings__get_hook_path(r));\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -225,17 +229,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \tchar *buf;\n \tunsigned int i;\n \tint is_trivial = 0;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (fsm_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -243,13 +258,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -280,7 +296,8 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t\tif (query_success)\n \t\t\t\tis_trivial = query_result.buf[0] == '/';\n@@ -290,9 +307,12 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n \t\t\t\t\t \"query/trivial-response\", 1);\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tfsm_settings__get_hook_path(r));\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t fsm_settings__get_hook_path(r),\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/*\n@@ -429,7 +449,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tint fsmonitor_enabled = (fsm_settings__get_mode(r) > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -449,16 +470,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..f9201411aa7 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -3,6 +3,7 @@\n \n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"fsmonitor-settings.h\"\n \n extern struct trace_key trace_fsmonitor;\n \n@@ -57,7 +58,11 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +72,11 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +92,10 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repository.h b/repository.h\nindex 2b5cf97f31e..6708207cd4c 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n #include \"path.h\"\n \n struct config_set;\n+struct fsmonitor_settings;\n struct git_hash_algo;\n struct index_state;\n struct lock_file;\n@@ -35,6 +36,8 @@ struct repo_settings {\n \tint command_requires_full_index;\n \tint sparse_index;\n \n+\tstruct fsmonitor_settings *fsmonitor; /* lazily loaded */\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex f48e0542cdc..9ffea1d3147 100644\n--- a/t/README\n+++ b/t/README\n@@ -405,8 +405,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code paths for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\n-- \ngitgitgadget\n\n"},{"id":"448255","messageId":"8e738a83bc5beb1f222d5ed8591fe2c8aa9ff6f6.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 04/30] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:53Z","receivedAt":"2022-02-11T20:56:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.fsmonitor` is set to true.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 38 ++++++++++++++++++++++++++++++++++++--\n 1 file changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 17cba68235a..4287aad6bbb 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -241,8 +241,41 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (fsm_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t\tis_trivial = query_result.buf[bol] == '/';\n+\t\t\tif (is_trivial)\n+\t\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t\t \"query/trivial-response\", 1);\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\tgoto apply_results;\n \t}\n \n \tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n@@ -315,6 +348,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/*\n \t * The response from FSMonitor (excluding the header token) is\n \t * either:\n-- \ngitgitgadget\n\n"},{"id":"448257","messageId":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v4.git.1634826309.gitgitgadget@gmail.com","subject":"[PATCH v5 00/30] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:49Z","receivedAt":"2022-02-11T20:56:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V5 of Part 2 of my Builtin FSMonitor series. I apologize for the\ndelay since V4 that I submitted back in October. (Insert the usual $DayJob\nexcuse...)\n\nI have rebased this branch onto the current \"master\" branch.\n\nIn this version I removed the core.useBuiltinFSMonitor config setting and\ninstead extended the existing core.fsmonitor. Previously it was defined as\ncontaining the pathname to the FSMonitor hook script/application. Now it can\nalso have a boolean value to select the builtin FSMonitor daemon. This does\nsimplify a few things in the documentation and in the code.\n\nThere were other changes, but they are mostly just general cleanup and\ncoding style items. I also squashed in Junio's Dec 25 commits related to\ncoding style in the test scripts.\n\nThis version contains the client code and a MVP version of the daemon. It\ncan be be tested using t/t7527 and t/perf/p7519.\n\nA followup Part 3 will contain additional refinements to the daemon and\nadditional tests. I drew the line here between Part 2 and 3 to make it\neasier to review.\n\nJeff Hostetler (30):\n fsmonitor: enhance existing comments, clarify trivial response\n handling\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor: document builtin fsmonitor\n fsmonitor--daemon: add a built-in fsmonitor daemon\n fsmonitor--daemon: implement 'stop' and 'status' commands\n compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n fsmonitor--daemon: implement 'run' command\n fsmonitor--daemon: implement 'start' command\n fsmonitor--daemon: add pathname classification\n fsmonitor--daemon: define token-ids\n fsmonitor--daemon: create token-based changed path cache\n compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on\n Windows\n compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent\n compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on\n MacOS\n fsmonitor--daemon: implement handle_client callback\n help: include fsmonitor--daemon feature flag in version info\n t/helper/fsmonitor-client: create IPC client to talk to FSMonitor\n Daemon\n t7527: create test for fsmonitor--daemon\n t/perf: avoid copying builtin fsmonitor files into test repo\n t/helper/test-chmtime: skip directories on Windows\n t/perf/p7519: speed up test on Windows\n t/perf/p7519: add fsmonitor--daemon test cases\n fsmonitor--daemon: periodically truncate list of modified files\n fsmonitor--daemon: use a cookie file to sync with file system\n fsmonitor: force update index after large responses\n t7527: test status with untracked-cache and fsmonitor--daemon\n update-index: convert fsmonitor warnings to advise\n\n .gitignore | 1 +\n Documentation/config/core.txt | 46 +-\n Documentation/git-fsmonitor--daemon.txt | 75 ++\n Documentation/git-update-index.txt | 8 +-\n Makefile | 17 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 1454 +++++++++++++++++++++++\n builtin/update-index.c | 19 +-\n cache.h | 1 -\n compat/fsmonitor/fsm-listen-darwin.c | 496 ++++++++\n compat/fsmonitor/fsm-listen-win32.c | 586 +++++++++\n compat/fsmonitor/fsm-listen.h | 49 +\n config.c | 14 -\n config.h | 1 -\n config.mak.uname | 20 +\n contrib/buildsystems/CMakeLists.txt | 10 +\n environment.c | 1 -\n fsmonitor--daemon.h | 140 +++\n fsmonitor-ipc.c | 171 +++\n fsmonitor-ipc.h | 48 +\n fsmonitor-settings.c | 100 ++\n fsmonitor-settings.h | 21 +\n fsmonitor.c | 218 +++-\n fsmonitor.h | 18 +-\n git.c | 1 +\n help.c | 4 +\n repo-settings.c | 1 +\n repository.h | 3 +\n t/README | 4 +-\n t/helper/test-chmtime.c | 15 +\n t/helper/test-fsmonitor-client.c | 121 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/perf/p7519-fsmonitor.sh | 61 +-\n t/perf/perf-lib.sh | 2 +-\n t/t7527-builtin-fsmonitor.sh | 604 ++++++++++\n t/test-lib.sh | 6 +\n 37 files changed, 4229 insertions(+), 110 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n create mode 100644 builtin/fsmonitor--daemon.c\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n create mode 100644 fsmonitor--daemon.h\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n create mode 100644 t/helper/test-fsmonitor-client.c\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\n\nbase-commit: 2b9c1209706bc2ef0ab09fb0bdc7d405e225ce8b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/1041\n\nRange-diff vs v4:\n\n 1: ecc40795fa2 ! 1: a5ecabb4d02 fsmonitor: enhance existing comments\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: enhance existing comments\n + fsmonitor: enhance existing comments, clarify trivial response handling\n \n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n +@@ fsmonitor.c: static int query_fsmonitor(int version, const char *last_update, struct strbuf *\n + \n + \tif (result)\n + \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/failed\", result);\n +-\telse {\n ++\telse\n + \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/response-length\",\n + \t\t\t\t query_result->len);\n + \n +-\t\tif (fsmonitor_is_trivial_response(query_result))\n +-\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n +-\t\t\t\t\t \"query/trivial-response\", 1);\n +-\t}\n +-\n + \ttrace2_region_leave(\"fsm_hook\", \"query\", NULL);\n + \n + \treturn result;\n + }\n + \n +-int fsmonitor_is_trivial_response(const struct strbuf *query_result)\n +-{\n +-\tstatic char trivial_response[3] = { '\\0', '/', '\\0' };\n +-\n +-\treturn query_result->len >= 3 &&\n +-\t\t!memcmp(trivial_response,\n +-\t\t\t&query_result->buf[query_result->len - 3], 3);\n +-}\n +-\n + static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n + {\n + \tint i, len = strlen(name);\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \tstruct strbuf last_update_token = STRBUF_INIT;\n + \tchar *buf;\n + \tunsigned int i;\n ++\tint is_trivial = 0;\n + \n + \tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n + \t\treturn;\n +@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \t\t\t\t\tquery_success = 0;\n + \t\t\t\t} else {\n + \t\t\t\t\tbol = last_update_token.len + 1;\n ++\t\t\t\t\tis_trivial = query_result.buf[bol] == '/';\n + \t\t\t\t}\n + \t\t\t} else if (hook_version < 0) {\n + \t\t\t\thook_version = HOOK_INTERFACE_VERSION1;\n +@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n + \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n + \t\t\t\tistate->fsmonitor_last_update, &query_result);\n ++\t\t\tif (query_success)\n ++\t\t\t\tis_trivial = query_result.buf[0] == '/';\n + \t\t}\n + \n ++\t\tif (is_trivial)\n ++\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n ++\t\t\t\t\t \"query/trivial-response\", 1);\n ++\n + \t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n + \t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n -\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n +-\tif (query_success && query_result.buf[bol] != '/') {\n +-\t\t/* Mark all entries returned by the monitor as dirty */\n +\t/*\n +\t * The response from FSMonitor (excluding the header token) is\n +\t * either:\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n +\t * information and that we should consider everything\n +\t * invalid. We call this a trivial response.\n +\t */\n - \tif (query_success && query_result.buf[bol] != '/') {\n --\t\t/* Mark all entries returned by the monitor as dirty */\n ++\tif (query_success && !is_trivial) {\n +\t\t/*\n +\t\t * Mark all pathnames returned by the monitor as dirty.\n +\t\t *\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n -\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n -\t\t * if we actually changed entries or not */\n +\t\t/*\n -+\t\t * We received a trivial response, so invalidate everything.\n ++\t\t * We failed to get a response or received a trivial response,\n ++\t\t * so invalidate everything.\n +\t\t *\n +\t\t * We only want to run the post index changed hook if\n +\t\t * we've actually changed entries, so keep track if we\n 2: 82f17692128 ! 2: 365964b7664 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n @@ fsmonitor-ipc.c (new)\n +\n +\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n +\t\t\t\t \"query/response-length\", answer->len);\n -+\n -+\t\tif (fsmonitor_is_trivial_response(answer))\n -+\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n -+\t\t\t\t\t \"query/trivial-response\", 1);\n -+\n +\t\tgoto done;\n +\n +\tcase IPC_STATE__NOT_LISTENING:\n 3: 882789b4dfe ! 3: 384516ce1a1 fsmonitor: config settings are repository-specific\n @@ Commit message\n Create `fsm_settings__get_*()` getters to lazily look up fsmonitor-\n related config settings.\n \n - Add support for the new `core.useBuiltinFSMonitor` config setting.\n -\n Get rid of the `core_fsmonitor` global variable. Move the code to\n lookup the existing `core.fsmonitor` config value into the fsmonitor\n settings.\n @@ Commit message\n Create a hook pathname variable in `struct fsmonitor-settings` and\n only set it when in hook mode.\n \n + Extend the definition of `core.fsmonitor` to be either a boolean\n + or a hook pathname. When true, the builtin FSMonitor is used.\n + When false or unset, no FSMonitor (neither builtin nor hook) is\n + used.\n +\n The existing `core_fsmonitor` global variable was used to store the\n pathname to the fsmonitor hook *and* it was used as a boolean to see\n if fsmonitor was enabled. This dual usage and global visibility leads\n @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const\n \tif (fsmonitor > 0) {\n -\t\tif (git_config_get_fsmonitor() == 0)\n +\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n -+\n +\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n -+\t\t\twarning(_(\"core.useBuiltinFSMonitor is unset; \"\n -+\t\t\t\t\"set it if you really want to enable the \"\n -+\t\t\t\t\"builtin fsmonitor\"));\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n --\t\t\t\t\"set it if you really want to \"\n --\t\t\t\t\"enable fsmonitor\"));\n -+\t\t\t\t\"set it if you really want to enable the \"\n -+\t\t\t\t\"hook-based fsmonitor\"));\n + \t\t\t\t\"set it if you really want to \"\n + \t\t\t\t\"enable fsmonitor\"));\n +\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n -\t\tif (git_config_get_fsmonitor() == 1)\n +\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n -+\t\tif (fsm_mode == FSMONITOR_MODE_IPC)\n -+\t\t\twarning(_(\"core.useBuiltinFSMonitor is set; \"\n -+\t\t\t\t\"remove it if you really want to \"\n -+\t\t\t\t\"disable fsmonitor\"));\n -+\t\tif (fsm_mode == FSMONITOR_MODE_HOOK)\n ++\t\tif (fsm_mode > FSMONITOR_MODE_DISABLED)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\n @@ fsmonitor-settings.c (new)\n +\tchar *hook_path;\n +};\n +\n -+void fsm_settings__set_ipc(struct repository *r)\n ++static void lookup_fsmonitor_settings(struct repository *r)\n +{\n -+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n ++\tstruct fsmonitor_settings *s;\n ++\tconst char *const_str;\n ++\tint bool_value;\n +\n -+\ts->mode = FSMONITOR_MODE_IPC;\n -+}\n ++\tif (r->settings.fsmonitor)\n ++\t\treturn;\n +\n -+void fsm_settings__set_hook(struct repository *r, const char *path)\n -+{\n -+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n ++\tCALLOC_ARRAY(s, 1);\n +\n -+\ts->mode = FSMONITOR_MODE_HOOK;\n -+\ts->hook_path = strdup(path);\n -+}\n ++\tr->settings.fsmonitor = s;\n +\n -+void fsm_settings__set_disabled(struct repository *r)\n -+{\n -+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n ++\tfsm_settings__set_disabled(r);\n +\n -+\ts->mode = FSMONITOR_MODE_DISABLED;\n -+\tFREE_AND_NULL(s->hook_path);\n -+}\n ++\t/*\n ++\t * Overload the existing \"core.fsmonitor\" config setting (which\n ++\t * has historically been either unset or a hook pathname) to\n ++\t * now allow a boolean value to enable the builtin FSMonitor\n ++\t * or to turn everything off. (This does imply that you can't\n ++\t * use a hook script named \"true\" or \"false\", but that's OK.)\n ++\t */\n ++\tswitch (repo_config_get_maybe_bool(r, \"core.fsmonitor\", &bool_value)) {\n ++\n ++\tcase 0: /* config value was set to */\n ++\t\tif (bool_value)\n ++\t\t\tfsm_settings__set_ipc(r);\n ++\t\treturn;\n +\n -+static int check_for_ipc(struct repository *r)\n -+{\n -+\tint value;\n ++\tcase 1: /* config value was unset */\n ++\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n ++\t\tbreak;\n +\n -+\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) &&\n -+\t value) {\n -+\t\tfsm_settings__set_ipc(r);\n -+\t\treturn 1;\n ++\tcase -1: /* config value set to an arbitrary string */\n ++\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n ++\t\t\treturn; /* should not happen */\n ++\t\tbreak;\n ++\n ++\tdefault: /* should not happen */\n ++\t\treturn;\n +\t}\n +\n -+\treturn 0;\n ++\tif (!const_str || !*const_str)\n ++\t\treturn;\n ++\n ++\tfsm_settings__set_hook(r, const_str);\n +}\n +\n -+static int check_for_hook(struct repository *r)\n ++enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n +{\n -+\tconst char *const_str;\n ++\tlookup_fsmonitor_settings(r);\n +\n -+\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n -+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n -+\n -+\tif (const_str && *const_str) {\n -+\t\tfsm_settings__set_hook(r, const_str);\n -+\t\treturn 1;\n -+\t}\n -+\n -+\treturn 0;\n ++\treturn r->settings.fsmonitor->mode;\n +}\n +\n -+static void lookup_fsmonitor_settings(struct repository *r)\n ++const char *fsm_settings__get_hook_path(struct repository *r)\n +{\n -+\tstruct fsmonitor_settings *s;\n -+\n -+\tCALLOC_ARRAY(s, 1);\n ++\tlookup_fsmonitor_settings(r);\n +\n -+\tr->settings.fsmonitor = s;\n -+\n -+\tif (check_for_ipc(r))\n -+\t\treturn;\n ++\treturn r->settings.fsmonitor->hook_path;\n ++}\n +\n -+\tif (check_for_hook(r))\n -+\t\treturn;\n ++void fsm_settings__set_ipc(struct repository *r)\n ++{\n ++\tlookup_fsmonitor_settings(r);\n +\n -+\tfsm_settings__set_disabled(r);\n ++\tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n ++\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n +}\n +\n -+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n ++void fsm_settings__set_hook(struct repository *r, const char *path)\n +{\n -+\tif (!r->settings.fsmonitor)\n -+\t\tlookup_fsmonitor_settings(r);\n ++\tlookup_fsmonitor_settings(r);\n +\n -+\treturn r->settings.fsmonitor->mode;\n ++\tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n ++\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n ++\tr->settings.fsmonitor->hook_path = strdup(path);\n +}\n +\n -+const char *fsm_settings__get_hook_path(struct repository *r)\n ++void fsm_settings__set_disabled(struct repository *r)\n +{\n -+\tif (!r->settings.fsmonitor)\n -+\t\tlookup_fsmonitor_settings(r);\n ++\tlookup_fsmonitor_settings(r);\n +\n -+\treturn r->settings.fsmonitor->hook_path;\n ++\tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n ++\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n +}\n \n ## fsmonitor-settings.h (new) ##\n @@ fsmonitor-settings.h (new)\n +\n +enum fsmonitor_mode {\n +\tFSMONITOR_MODE_DISABLED = 0,\n -+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n -+\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n ++\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n ++\tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n +};\n +\n +void fsm_settings__set_ipc(struct repository *r);\n @@ fsmonitor.c: void write_fsmonitor_extension(struct strbuf *sb, struct index_stat\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n - \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n + \tint is_trivial = 0;\n +\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n +\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n +\t\t\tquery_success = !query_fsmonitor_hook(\n +\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n - \t\t}\n + \t\t\tif (query_success)\n + \t\t\t\tis_trivial = query_result.buf[0] == '/';\n +@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n + \t\t\t\t\t \"query/trivial-response\", 1);\n \n -\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n -\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n @@ repository.h: struct repo_settings {\n \tint command_requires_full_index;\n \tint sparse_index;\n \n -+\tstruct fsmonitor_settings *fsmonitor; /* lazy loaded */\n ++\tstruct fsmonitor_settings *fsmonitor; /* lazily loaded */\n +\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n @@ t/README: every 'git commit-graph write', as if the `--changed-paths` option was\n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n -code path for utilizing a file system monitor to speed up detecting\n -new or changed files.\n -+code path for utilizing a (hook based) file system monitor to speed up\n ++code paths for utilizing a (hook based) file system monitor to speed up\n +detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n 4: de82c726182 ! 4: 8e738a83bc5 fsmonitor: use IPC to query the builtin FSMonitor daemon\n @@ Commit message\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n \n Use simple IPC to directly communicate with the new builtin file\n - system monitor daemon when `core.useBuiltinFSMonitor` is set.\n -\n - The `core.fsmonitor` setting has already been defined as a HOOK\n - pathname. Historically, this has been set to a HOOK script that will\n - talk with Watchman. For compatibility reasons, we do not want to\n - overload that definition (and cause problems if users have multiple\n - versions of Git installed).\n + system monitor daemon when `core.fsmonitor` is set to true.\n \n Signed-off-by: Johannes Schindelin \n Signed-off-by: Jeff Hostetler \n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n +\t\t\tbuf = query_result.buf;\n +\t\t\tstrbuf_addstr(&last_update_token, buf);\n +\t\t\tbol = last_update_token.len + 1;\n ++\t\t\tis_trivial = query_result.buf[bol] == '/';\n ++\t\t\tif (is_trivial)\n ++\t\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n ++\t\t\t\t\t\t \"query/trivial-response\", 1);\n +\t\t} else {\n +\t\t\t/*\n +\t\t\t * The builtin daemon is not available on this\n 5: d365704d551 ! 5: 49e4c146e02 fsmonitor: document builtin fsmonitor\n @@ Metadata\n ## Commit message ##\n fsmonitor: document builtin fsmonitor\n \n - Document the new `core.useBuiltinFSMonitor` config value.\n + Document how `core.fsmonitor` can be set to a boolean to enable\n + or disable the builtin FSMonitor.\n \n Update references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\n pointers to `Watchman` to refer to it.\n @@ Documentation/config/core.txt: core.protectNTFS::\n -\trequested date/time. This information is used to speed up git by\n -\tavoiding unnecessary processing of files that have not changed.\n -\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n -+\tIf set, this variable contains the pathname of the \"fsmonitor\"\n ++\tIf set to true, enable the built-in file system monitor\n ++\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n +++\n ++Like hook-based file system monitors, the built-in file system monitor\n ++can speed up Git commands that need to refresh the Git index\n ++(e.g. `git status`) in a working directory with many files. The\n ++built-in monitor eliminates the need to install and maintain an\n ++external third-party tool.\n +++\n ++The built-in file system monitor is currently available only on a\n ++limited set of supported platforms. Currently, this includes Windows\n ++and MacOS.\n +++\n ++\tOtherwise, this variable contains the pathname of the \"fsmonitor\"\n +\thook command.\n ++\n +This hook command is used to identify all files that may have changed\n @@ Documentation/config/core.txt: core.protectNTFS::\n +git by avoiding unnecessary scanning of files that have not changed.\n ++\n +See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n -++\n -+Note: The value of this config setting is ignored if the\n -+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n \n core.fsmonitorHookVersion::\n -\tSets the version of hook that is to be used when calling fsmonitor.\n @@ Documentation/config/core.txt: core.protectNTFS::\n +Version 2 uses an opaque string so that the monitor can return\n +something that can be used to determine what files have changed\n +without race conditions.\n -++\n -+Note: The value of this config setting is ignored if the\n -+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n -+\n -+core.useBuiltinFSMonitor::\n -+\tIf set to true, enable the built-in file system monitor\n -+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n -++\n -+Like hook-based file system monitors, the built-in file system monitor\n -+can speed up Git commands that need to refresh the Git index\n -+(e.g. `git status`) in a working directory with many files. The\n -+built-in monitor eliminates the need to install and maintain an\n -+external third-party tool.\n -++\n -+The built-in file system monitor is currently available only on a\n -+limited set of supported platforms. Currently, this includes Windows\n -+and MacOS.\n -++\n -+Note: if this config setting is set to `true`, the values of\n -+`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\n @@ Documentation/git-fsmonitor--daemon.txt (new)\n +increased if they just ask for a summary of changes to the working\n +directory and can avoid scanning the disk.\n +\n -+When `core.useBuiltinFSMonitor` is set to `true` (see\n -+linkgit:git-config[1]) commands, such as `git status`, will ask the\n -+daemon for changes and automatically start it (if necessary).\n ++When `core.fsmonitor` is set to `true` (see linkgit:git-config[1])\n ++commands, such as `git status`, will ask the daemon for changes and\n ++automatically start it (if necessary).\n +\n +For more information see the \"File System Monitor\" section in\n +linkgit:git-update-index[1].\n @@ Documentation/git-update-index.txt: FILE SYSTEM MONITOR\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n -@@ Documentation/git-update-index.txt: performance by avoiding the cost of scanning the entire working directory\n - looking for new files.\n +@@ Documentation/git-update-index.txt: looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n --the `core.fsmonitor` configuration variable (see\n + the `core.fsmonitor` configuration variable (see\n -linkgit:git-config[1]) than using the `--fsmonitor` option to\n -`git update-index` in each repository, especially if you want to do so\n --across all repositories you use, because you can set the configuration\n --variable in your `$HOME/.gitconfig` just once and have it affect all\n --repositories you touch.\n --\n --When the `core.fsmonitor` configuration variable is changed, the\n --file system monitor is added to or removed from the index the next time\n --a command reads the index. When `--[no-]fsmonitor` are used, the file\n --system monitor is immediately added to or removed from the index.\n -+the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n -+variable (see linkgit:git-config[1]) than using the `--fsmonitor`\n -+option to `git update-index` in each repository, especially if you\n -+want to do so across all repositories you use, because you can set the\n -+configuration variable in your `$HOME/.gitconfig` just once and have\n -+it affect all repositories you touch.\n -+\n -+When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n -+variable is changed, the file system monitor is added to or removed\n -+from the index the next time a command reads the index. When\n -+`--[no-]fsmonitor` are used, the file system monitor is immediately\n -+added to or removed from the index.\n - \n - CONFIGURATION\n - -------------\n -\n - ## Documentation/githooks.txt ##\n -@@ Documentation/githooks.txt: fsmonitor-watchman\n - \n - This hook is invoked when the configuration option `core.fsmonitor` is\n - set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n --depending on the version of the hook to use.\n -+depending on the version of the hook to use, unless overridden via\n -+`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n - \n - Version 1 takes two arguments, a version (1) and the time in elapsed\n - nanoseconds since midnight, January 1, 1970.\n ++linkgit:git-config[1]) than using the `--fsmonitor` option to `git\n ++update-index` in each repository, especially if you want to do so\n + across all repositories you use, because you can set the configuration\n + variable in your `$HOME/.gitconfig` just once and have it affect all\n + repositories you touch.\n 6: 78e682fc530 = 6: bdd7334da31 fsmonitor--daemon: add a built-in fsmonitor daemon\n 7: ea64b5c9753 = 7: 9f00ada3dd3 fsmonitor--daemon: implement 'stop' and 'status' commands\n 8: 5a40b33a00c = 8: d6819bdad66 compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n 9: ed5819e29f8 = 9: 7de3d01cccc compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n 10: d3ac973a5f1 = 10: 6fe5a2bc79e fsmonitor--daemon: implement 'run' command\n 11: d08c28b549c = 11: 69fc0998286 fsmonitor--daemon: implement 'start' command\n 12: 6fa71fdc825 = 12: 21c099c5197 fsmonitor--daemon: add pathname classification\n 13: 65821da5b03 = 13: 72979c35ceb fsmonitor--daemon: define token-ids\n 14: 429c48a5bad ! 14: 8d1db644409 fsmonitor--daemon: create token-based changed path cache\n @@ Commit message\n backends to accumulate changed paths in response to filesystem events.\n \n The platform-specific file system listener thread receives file system\n - events containing one or more changed pathnames (with whatever bucketing\n - or grouping that is convenient for the file system). These paths are\n - accumulated (without locking) by the file system layer into a `fsmonitor_batch`.\n + events containing one or more changed pathnames (with whatever\n + bucketing or grouping that is convenient for the file system). These\n + paths are accumulated (without locking) by the file system layer into\n + a `fsmonitor_batch`.\n \n When the file system layer has drained the kernel event queue, it will\n \"publish\" them to our token queue and make them visible to concurrent\n 15: b04c460c619 = 15: 98c5adf8ca0 compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows\n 16: 862bbfcc32e ! 16: a3b881315fa compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent\n @@ Commit message\n currently fails with this error:\n \n In file included\n - from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n - from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42,\n - from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43,\n - from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29,\n - from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23,\n - from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23,\n - from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n - /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope\n + from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n + ...Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n + from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n + ...Library/Frameworks/Security.framework/Headers/Security.h:42,\n + from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n + ...Library/Frameworks/CoreServices.framework/Frameworks/...\n + ...OSServices.framework/Headers/CSIdentity.h:43,\n + from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n + ...Library/Frameworks/CoreServices.framework/Frameworks/...\n + ...OSServices.framework/Headers/OSServices.h:29,\n + from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n + ...Library/Frameworks/CoreServices.framework/Frameworks/...\n + ...LaunchServices.framework/Headers/IconsCore.h:23,\n + from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n + ...Library/Frameworks/CoreServices.framework/Frameworks/...\n + ...LaunchServices.framework/Headers/LaunchServices.h:23,\n + from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n + ...Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n +\n + /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n + ...Library/Frameworks/Security.framework/Headers/Authorization.h:193:7:\n + error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n \n 17: 40d9a816b52 = 17: 162e357db72 compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS\n 18: 241962894f1 = 18: 3de1c43beaf fsmonitor--daemon: implement handle_client callback\n 19: 704d37d2033 = 19: 3517c4a3c13 help: include fsmonitor--daemon feature flag in version info\n 20: de6c72a9ce0 ! 20: 4ffc2ddf516 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n @@ t/helper/test-tool.c: static struct test_cmd cmds[] = {\n \t{ \"getcwd\", cmd__getcwd },\n \n ## t/helper/test-tool.h ##\n -@@ t/helper/test-tool.h: int cmd__dump_split_index(int argc, const char **argv);\n - int cmd__dump_untracked_cache(int argc, const char **argv);\n +@@ t/helper/test-tool.h: int cmd__dump_untracked_cache(int argc, const char **argv);\n + int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n +int cmd__fsmonitor_client(int argc, const char **argv);\n 21: eedaa787c2e ! 21: d93310a7c64 t7527: create test for fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\tactual*\n +\tEOF\n +\n -+\tgit -c core.useBuiltinFSMonitor= add . &&\n ++\tgit -c core.fsmonitor=false add . &&\n +\ttest_tick &&\n -+\tgit -c core.useBuiltinFSMonitor= commit -m initial &&\n ++\tgit -c core.fsmonitor=false commit -m initial &&\n +\n -+\tgit config core.useBuiltinFSMonitor true\n ++\tgit config core.fsmonitor true\n +'\n +\n +# The test already explicitly stopped (or tried to stop) the daemon.\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_2\n +'\n +\n -+edit_files() {\n ++edit_files () {\n +\techo 1 >modified\n +\techo 2 >dir1/modified\n +\techo 3 >dir2/modified\n +\t>dir1/untracked\n +}\n +\n -+delete_files() {\n ++delete_files () {\n +\trm -f delete\n +\trm -f dir1/delete\n +\trm -f dir2/delete\n +}\n +\n -+create_files() {\n ++create_files () {\n +\techo 1 >new\n +\techo 2 >dir1/new\n +\techo 3 >dir2/new\n +}\n +\n -+rename_files() {\n ++rename_files () {\n +\tmv rename renamed\n +\tmv dir1/rename dir1/renamed\n +\tmv dir2/rename dir2/renamed\n +}\n +\n -+file_to_directory() {\n ++file_to_directory () {\n +\trm -f delete\n +\tmkdir delete\n +\techo 1 >delete/new\n +}\n +\n -+directory_to_file() {\n ++directory_to_file () {\n +\trm -rf dir1\n +\techo 1 >dir1\n +}\n +\n -+verify_status() {\n ++verify_status () {\n +\tgit status >actual &&\n +\tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n -+\tGIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect &&\n ++\tGIT_INDEX_FILE=.git/fresh-index git -c core.fsmonitor=false status >expect &&\n +\ttest_cmp expect actual &&\n +\techo HELLO AFTER &&\n +\tcat .git/trace &&\n 22: 4e96e0667ba = 22: 27e47108908 t/perf: avoid copying builtin fsmonitor files into test repo\n 23: de9c015d78c = 23: 6cba1d950b0 t/helper/test-chmtime: skip directories on Windows\n 24: 1c2eccacff6 ! 24: fcf843a0d42 t/perf/p7519: speed up test on Windows\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## t/perf/p7519-fsmonitor.sh ##\n -@@ t/perf/p7519-fsmonitor.sh: trace_stop() {\n +@@ t/perf/p7519-fsmonitor.sh: then\n + \tfi\n + fi\n + \n +-trace_start() {\n ++trace_start () {\n + \tif test -n \"$GIT_PERF_7519_TRACE\"\n + \tthen\n + \t\tname=\"$1\"\n +@@ t/perf/p7519-fsmonitor.sh: trace_start() {\n + \tfi\n + }\n + \n +-trace_stop() {\n ++trace_stop () {\n + \tif test -n \"$GIT_PERF_7519_TRACE\"\n + \tthen\n + \t\tunset GIT_TRACE2_PERF\n \tfi\n }\n \n -+touch_files() {\n ++touch_files () {\n +\tn=$1\n +\td=\"$n\"_files\n +\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n \tfi &&\n \n \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n --\tfor i in $(test_seq 1 10); do touch 10_files/$i; done &&\n --\tfor i in $(test_seq 1 100); do touch 100_files/$i; done &&\n --\tfor i in $(test_seq 1 1000); do touch 1000_files/$i; done &&\n --\tfor i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n +-\tfor i in $(test_seq 1 10); do touch 10_files/$i || return 1; done &&\n +-\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n +-\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n +-\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n +\ttouch_files 1 &&\n +\ttouch_files 10 &&\n +\ttouch_files 100 &&\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n \tgit commit -qm \"Add files\" &&\n \n +@@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n + \tfi\n + '\n + \n +-setup_for_fsmonitor() {\n ++setup_for_fsmonitor () {\n + \t# set INTEGRATION_SCRIPT depending on the environment\n + \tif test -n \"$INTEGRATION_PATH\"\n + \tthen\n +@@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n + \ttest_perf \"$@\"\n + }\n + \n +-test_fsmonitor_suite() {\n ++test_fsmonitor_suite () {\n + \tif test -n \"$INTEGRATION_SCRIPT\"; then\n + \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n + \telse\n @@ t/perf/p7519-fsmonitor.sh: test_fsmonitor_suite() {\n \n \t# Update the mtimes on upto 100k files to make status think\n 25: 236b5966257 ! 25: 198f47bda5a t/perf/p7519: add fsmonitor--daemon test cases\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## t/perf/p7519-fsmonitor.sh ##\n -@@ t/perf/p7519-fsmonitor.sh: test_description=\"Test core.fsmonitor\"\n - # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\n - # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an\n - # absolute path to an integration. May be a space delimited list of\n --# absolute paths to integrations.\n -+# absolute paths to integrations. (This hook or list of hooks does not\n -+# include the built-in fsmonitor--daemon.)\n - #\n - # The big win for using fsmonitor is the elimination of the need to scan the\n - # working directory looking for changed and untracked files. If the file\n -@@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n - \n - setup_for_fsmonitor() {\n - \t# set INTEGRATION_SCRIPT depending on the environment\n --\tif test -n \"$INTEGRATION_PATH\"\n -+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n - \tthen\n -+\t\tgit config core.useBuiltinFSMonitor true &&\n -+\t\tINTEGRATION_SCRIPT=false\n -+\telif test -n \"$INTEGRATION_PATH\"\n -+\tthen\n -+\t\tgit config core.useBuiltinFSMonitor false &&\n - \t\tINTEGRATION_SCRIPT=\"$INTEGRATION_PATH\"\n - \telse\n -+\t\tgit config core.useBuiltinFSMonitor false &&\n - \t\t#\n - \t\t# Choose integration script based on existence of Watchman.\n - \t\t# Fall back to an empty integration script.\n @@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n }\n \n - test_fsmonitor_suite() {\n + test_fsmonitor_suite () {\n -\tif test -n \"$INTEGRATION_SCRIPT\"; then\n +\tif test -n \"$USE_FSMONITOR_DAEMON\"\n +\tthen\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"setup without fsmonitor\" '\n +\tgit fsmonitor--daemon start\n +\n +\ttrace_start fsmonitor--daemon--client\n -+\ttest_expect_success \"setup for fsmonitor--daemon\" 'setup_for_fsmonitor'\n ++\n ++\tgit config core.fsmonitor true\n ++\tgit update-index --fsmonitor\n ++\n +\ttest_fsmonitor_suite\n +\n +\tgit fsmonitor--daemon stop\n 26: 54710a4830d = 26: 19993c130d2 fsmonitor--daemon: periodically truncate list of modified files\n 27: 1e2bd77fcea = 27: f47a763dc26 fsmonitor--daemon: use a cookie file to sync with file system\n 28: 30e61b6d1ad ! 28: aec44a21afd fsmonitor: force update index after large responses\n @@ fsmonitor.c: apply_results:\n \t */\n +\ttrace2_region_enter(\"fsmonitor\", \"apply_results\", istate->repo);\n +\n - \tif (query_success && query_result.buf[bol] != '/') {\n + \tif (query_success && !is_trivial) {\n \t\t/*\n \t\t * Mark all pathnames returned by the monitor as dirty.\n \t\t *\n @@ fsmonitor.c: apply_results:\n +\n \t} else {\n \t\t/*\n - \t\t * We received a trivial response, so invalidate everything.\n + \t\t * We failed to get a response or received a trivial response,\n @@ fsmonitor.c: apply_results:\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 0;\n 29: 507020bbef0 ! 29: d6039987df8 t7527: test status with untracked-cache and fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '\n +\ttrace*\n \tEOF\n \n - \tgit -c core.useBuiltinFSMonitor= add . &&\n + \tgit -c core.fsmonitor=false add . &&\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n \tstop_daemon_delete_repo wt-base\n '\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n +# cause incorrect results when the untracked-cache is enabled.\n +\n +test_lazy_prereq UNTRACKED_CACHE '\n -+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n -+\ttest $ret -ne 1\n ++\tgit update-index --test-untracked-cache\n +'\n +\n +test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n -+\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n ++\ttest_unconfig core.fsmonitor &&\n +\tgit update-index --no-fsmonitor &&\n +\ttest_might_fail git fsmonitor--daemon stop\n +'\n +\n +matrix_clean_up_repo () {\n -+\tgit reset --hard HEAD\n ++\tgit reset --hard HEAD &&\n +\tgit clean -fd\n +}\n +\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n +\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n +\t\tmatrix_clean_up_repo &&\n +\t\t$fn &&\n -+\t\tif test $uc = false -a $fsm = false\n ++\t\tif test $uc = false && test $fsm = false\n +\t\tthen\n +\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n +\t\telse\n -+\t\t\tgit status --porcelain=v1 >.git/actual.$fn\n ++\t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n +\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n +\t\tfi\n +\t'\n -+\n -+\treturn $?\n +}\n +\n +uc_values=\"false\"\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n +\t\tif test $fsm_val = false\n +\t\tthen\n +\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n -+\t\t\t\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n ++\t\t\t\ttest_unconfig core.fsmonitor &&\n +\t\t\t\tgit update-index --no-fsmonitor &&\n -+\t\t\t\ttest_might_fail git fsmonitor--daemon stop 2>/dev/null\n ++\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n +\t\t\t'\n +\t\telse\n +\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n -+\t\t\t\tgit config core.useBuiltinFSMonitor true &&\n ++\t\t\t\tgit config core.fsmonitor true &&\n +\t\t\t\tgit fsmonitor--daemon start &&\n +\t\t\t\tgit update-index --fsmonitor\n +\t\t\t'\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n +\t\tif test $fsm_val = true\n +\t\tthen\n +\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n -+\t\t\t\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n ++\t\t\t\ttest_unconfig core.fsmonitor &&\n +\t\t\t\tgit update-index --no-fsmonitor &&\n -+\t\t\t\ttest_might_fail git fsmonitor--daemon stop 2>/dev/null\n ++\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n +\t\t\t'\n +\t\tfi\n +\tdone\n -: ----------- > 30: 5117fbdfc63 update-index: convert fsmonitor warnings to advise\n\n-- \ngitgitgadget\n"},{"id":"448256","messageId":"49e4c146e02002f9b0af8c3f4db15706041e7b95.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 05/30] fsmonitor: document builtin fsmonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:54Z","receivedAt":"2022-02-11T20:56:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDocument how `core.fsmonitor` can be set to a boolean to enable\nor disable the builtin FSMonitor.\n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to refer to it.\n\nCreate `git-fsmonitor--daemon` manual page and describe its features.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/config/core.txt | 46 ++++++++++-----\n Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n Documentation/git-update-index.txt | 8 ++-\n 3 files changed, 112 insertions(+), 17 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..86059dc7b88 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,40 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+\tOtherwise, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\nnew file mode 100644\nindex 00000000000..0fedf5a4565\n--- /dev/null\n+++ b/Documentation/git-fsmonitor--daemon.txt\n@@ -0,0 +1,75 @@\n+git-fsmonitor--daemon(1)\n+========================\n+\n+NAME\n+----\n+git-fsmonitor--daemon - A Built-in File System Monitor\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git fsmonitor--daemon' start\n+'git fsmonitor--daemon' run\n+'git fsmonitor--daemon' stop\n+'git fsmonitor--daemon' status\n+\n+DESCRIPTION\n+-----------\n+\n+A daemon to watch the working directory for file and directory\n+changes using platform-specific file system notification facilities.\n+\n+This daemon communicates directly with commands like `git status`\n+using the link:technical/api-simple-ipc.html[simple IPC] interface\n+instead of the slower linkgit:githooks[5] interface.\n+\n+This daemon is built into Git so that no third-party tools are\n+required.\n+\n+OPTIONS\n+-------\n+\n+start::\n+\tStarts a daemon in the background.\n+\n+run::\n+\tRuns a daemon in the foreground.\n+\n+stop::\n+\tStops the daemon running in the current working\n+\tdirectory, if present.\n+\n+status::\n+\tExits with zero status if a daemon is watching the\n+\tcurrent working directory.\n+\n+REMARKS\n+-------\n+\n+This daemon is a long running process used to watch a single working\n+directory and maintain a list of the recently changed files and\n+directories. Performance of commands such as `git status` can be\n+increased if they just ask for a summary of changes to the working\n+directory and can avoid scanning the disk.\n+\n+When `core.fsmonitor` is set to `true` (see linkgit:git-config[1])\n+commands, such as `git status`, will ask the daemon for changes and\n+automatically start it (if necessary).\n+\n+For more information see the \"File System Monitor\" section in\n+linkgit:git-update-index[1].\n+\n+CAVEATS\n+-------\n+\n+The fsmonitor daemon does not currently know about submodules and does\n+not know to filter out file system events that happen within a\n+submodule. If fsmonitor daemon is watching a super repo and a file is\n+modified within the working directory of a submodule, it will report\n+the change (as happening against the super repo). However, the client\n+will properly ignore these extra events, so performance may be affected\n+but it will not cause an incorrect result.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..53ea48a04e2 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -509,8 +511,8 @@ looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n+linkgit:git-config[1]) than using the `--fsmonitor` option to `git\n+update-index` in each repository, especially if you want to do so\n across all repositories you use, because you can set the configuration\n variable in your `$HOME/.gitconfig` just once and have it affect all\n repositories you touch.\n-- \ngitgitgadget\n\n"},{"id":"448258","messageId":"bdd7334da3162ce77c216d61ce9d979f12637ac5.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 06/30] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:55Z","receivedAt":"2022-02-11T20:56:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a built-in file system monitoring daemon that can be used by\nthe existing `fsmonitor` feature (protocol API and index extension)\nto improve the performance of various Git commands, such as `status`.\n\nThe `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\nprovides an alternative to hook access to existing fsmonitors such\nas `watchman`.\n\nThis commit merely adds the new command without any functionality.\n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 46 +++++++++++++++++++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 50 insertions(+)\n create mode 100644 builtin/fsmonitor--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex f817c509ec0..e81de1063a4 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -72,6 +72,7 @@\n /git-format-patch\n /git-fsck\n /git-fsck-objects\n+/git-fsmonitor--daemon\n /git-gc\n /git-get-tar-commit-id\n /git-grep\ndiff --git a/Makefile b/Makefile\nindex 9943f0f7c11..3b7a3f88b50 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1106,6 +1106,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n+BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex 83379f3832c..40e9ecc8485 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\n int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nnew file mode 100644\nindex 00000000000..f0498793379\n--- /dev/null\n+++ b/builtin/fsmonitor--daemon.c\n@@ -0,0 +1,46 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"simple-ipc.h\"\n+#include \"khash.h\"\n+\n+static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tNULL\n+};\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *subcmd;\n+\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\tif (argc != 1)\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\tsubcmd = argv[0];\n+\n+\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n+}\n+\n+#else\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n+}\n+#endif\ndiff --git a/git.c b/git.c\nindex 340665d4a04..a8b44d9b587 100644\n--- a/git.c\n+++ b/git.c\n@@ -536,6 +536,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"448259","messageId":"d6819bdad66628b125f163b62344ef90e720b495.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 08/30] compat/fsmonitor/fsm-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:57Z","receivedAt":"2022-02-11T20:56:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty filesystem listener backend for fsmonitor--daemon on Windows.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 ++++++++\n compat/fsmonitor/fsm-listen-win32.c | 21 +++++++++++++\n compat/fsmonitor/fsm-listen.h | 49 +++++++++++++++++++++++++++++\n config.mak.uname | 10 ++++++\n contrib/buildsystems/CMakeLists.txt | 7 +++++\n repo-settings.c | 1 +\n 6 files changed, 101 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n\ndiff --git a/Makefile b/Makefile\nindex 3b7a3f88b50..b98f4899ac0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -464,6 +464,11 @@ all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n+# If your platform supports a built-in fsmonitor backend, set\n+# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n+# `compat/fsmonitor/fsm-listen-.c` that implements the\n+# `fsm_listen__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1939,6 +1944,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n \tCOMPAT_OBJS += compat/access.o\n endif\n \n+ifdef FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2858,6 +2868,9 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n \t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n+ifdef FSMONITOR_DAEMON_BACKEND\n+\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nnew file mode 100644\nindex 00000000000..916cbea254f\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nnew file mode 100644\nindex 00000000000..f0539349baf\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -0,0 +1,49 @@\n+#ifndef FSM_LISTEN_H\n+#define FSM_LISTEN_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * fsmonitor_fs_listener thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread AFTER joining the listener.\n+ */\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to watch for\n+ * filesystem events. This will run in the fsmonitor_fs_listen thread.\n+ *\n+ * It should call `ipc_server_stop_async()` if the listener thread\n+ * prematurely terminates (because of a filesystem error or if it\n+ * detects that the .git directory has been deleted). (It should NOT\n+ * do so if the listener thread receives a normal shutdown signal from\n+ * the IPC layer.)\n+ *\n+ * It should set `state->error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the fsmonitor listener thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_LISTEN_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex c48db45106c..9c88e1b244e 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -434,6 +434,11 @@ ifeq ($(uname_S),Windows)\n \t# so we don't need this:\n \t#\n \t# SNPRINTF_RETURNS_BOGUS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -619,6 +624,11 @@ ifeq ($(uname_S),MINGW)\n \tNO_STRTOUMAX = YesPlease\n \tNO_MKDTEMP = YesPlease\n \tNO_SVN_TESTS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 5100f56bb37..1525a81109c 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -285,6 +285,13 @@ else()\n \tendif()\n endif()\n \n+if(SUPPORTS_SIMPLE_IPC)\n+\tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\tendif()\n+endif()\n+\n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n \n #header checks\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 00ca5571a1a..373a341ad2b 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"midx.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n \n static void repo_cfg_bool(struct repository *r, const char *key, int *dest,\n \t\t\t int def)\n-- \ngitgitgadget\n\n"},{"id":"448260","messageId":"9f00ada3dd35a5dde09dbf1527f0a8987392b05b.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 07/30] fsmonitor--daemon: implement 'stop' and 'status' commands","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:56Z","receivedAt":"2022-02-11T20:56:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `stop` and `status` client commands to control and query the\nstatus of a `fsmonitor--daemon` server process (and implicitly start a\nserver process if necessary).\n\nLater commits will implement the actual server and monitor the file\nsystem.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 51 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex f0498793379..5e3178b8bdd 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,10 +7,55 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon stop\"),\n+\tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n+ * and wait for it to shutdown.\n+ */\n+static int do_as_client__send_stop(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"quit\", &answer);\n+\n+\t/* The quit command does not return any response data. */\n+\tstrbuf_release(&answer);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\twhile (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tsleep_millisec(50);\n+\ttrace2_region_leave(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\n+\treturn 0;\n+}\n+\n+static int do_as_client__status(void)\n+{\n+\tenum ipc_active_state state = fsmonitor_ipc__get_state();\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tprintf(_(\"fsmonitor-daemon is watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tprintf(_(\"fsmonitor-daemon is not watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 1;\n+\t}\n+}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n@@ -28,6 +73,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (!strcmp(subcmd, \"stop\"))\n+\t\treturn !!do_as_client__send_stop();\n+\n+\tif (!strcmp(subcmd, \"status\"))\n+\t\treturn !!do_as_client__status();\n+\n \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"448261","messageId":"7de3d01ccccb7afce6a9b66bb834402541d1ae1c.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 09/30] compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:58Z","receivedAt":"2022-02-11T20:56:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty implementation of fsmonitor--daemon\nbackend for Darwin (aka MacOS).\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 20 ++++++++++++++++++++\n config.mak.uname | 10 ++++++++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 33 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nnew file mode 100644\nindex 00000000000..c84e3344ab9\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -0,0 +1,20 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 9c88e1b244e..d65a5eb807c 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -157,6 +157,16 @@ ifeq ($(uname_S),Darwin)\n \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n \t\tendif\n \tendif\n+\n+\t# The builtin FSMonitor on MacOS builds upon Simple-IPC. Both require\n+\t# Unix domain sockets and PThreads.\n+\tifndef NO_PTHREADS\n+\tifndef NO_UNIX_SOCKETS\n+\tFSMONITOR_DAEMON_BACKEND = darwin\n+\tendif\n+\tendif\n+\n+\tBASIC_LDFLAGS += -framework CoreServices\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 1525a81109c..7da31c38f48 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"448262","messageId":"6fe5a2bc79eec132b6455d53196906036d0f4a80.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 10/30] fsmonitor--daemon: implement 'run' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:55:59Z","receivedAt":"2022-02-11T20:57:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `run` command to try to begin listening for file system events.\n\nThis version defines the thread structure with a single fsmonitor_fs_listen\nthread to watch for file system events and a simple IPC thread pool to\nwatch for connection from Git clients over a well-known named pipe or\nUnix domain socket.\n\nThis commit does not actually do anything yet because the platform\nbackends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 213 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 34 ++++++\n 2 files changed, 246 insertions(+), 1 deletion(-)\n create mode 100644 fsmonitor--daemon.h\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 5e3178b8bdd..b5ebd1eca33 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,16 +3,39 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Global state loaded from config.\n+ */\n+#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n+static int fsmonitor__ipc_threads = 8;\n+\n+static int fsmonitor_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 1)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__IPC_THREADS, i);\n+\t\tfsmonitor__ipc_threads = i;\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n /*\n * Acting as a CLIENT.\n *\n@@ -57,15 +80,196 @@ static int do_as_client__status(void)\n \t}\n }\n \n+static ipc_server_application_cb handle_client;\n+\n+static int handle_client(void *data,\n+\t\t\t const char *command, size_t command_len,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\t/* struct fsmonitor_daemon_state *state = data; */\n+\tint result;\n+\n+\t/*\n+\t * The Simple IPC API now supports {char*, len} arguments, but\n+\t * FSMonitor always uses proper null-terminated strings, so\n+\t * we can ignore the command_len argument. (Trust, but verify.)\n+\t */\n+\tif (command_len != strlen(command))\n+\t\tBUG(\"FSMonitor assumes text messages\");\n+\n+\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n+\n+\tresult = 0; /* TODO Do something here. */\n+\n+\ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n+\n+\treturn result;\n+}\n+\n+static void *fsm_listen__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-listen\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Watching: worktree '%s'\",\n+\t\t\t state->path_worktree_watch.buf);\n+\tif (state->nr_paths_watching > 1)\n+\t\ttrace_printf_key(&trace_fsmonitor, \"Watching: gitdir '%s'\",\n+\t\t\t\t state->path_gitdir_watch.buf);\n+\n+\tfsm_listen__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct ipc_server_opts ipc_opts = {\n+\t\t.nr_threads = fsmonitor__ipc_threads,\n+\n+\t\t/*\n+\t\t * We know that there are no other active threads yet,\n+\t\t * so we can let the IPC layer temporarily chdir() if\n+\t\t * it needs to when creating the server side of the\n+\t\t * Unix domain socket.\n+\t\t */\n+\t\t.uds_disallow_chdir = 0\n+\t};\n+\n+\t/*\n+\t * Start the IPC thread pool before the we've started the file\n+\t * system event listener thread so that we have the IPC handle\n+\t * before we need it.\n+\t */\n+\tif (ipc_server_run_async(&state->ipc_server_data,\n+\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t handle_client, state))\n+\t\treturn error_errno(\n+\t\t\t_(\"could not start IPC thread pool on '%s'\"),\n+\t\t\tfsmonitor_ipc__get_path());\n+\n+\t/*\n+\t * Start the fsmonitor listener thread to collect filesystem\n+\t * events.\n+\t */\n+\tif (pthread_create(&state->listener_thread, NULL,\n+\t\t\t fsm_listen__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\tipc_server_await(state->ipc_server_data);\n+\n+\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t}\n+\n+\t/*\n+\t * The daemon is now fully functional in background threads.\n+\t * Wait for the IPC thread pool to shutdown (whether by client\n+\t * request or from filesystem activity).\n+\t */\n+\tipc_server_await(state->ipc_server_data);\n+\n+\t/*\n+\t * The fsmonitor listener thread may have received a shutdown\n+\t * event from the IPC thread pool, but it doesn't hurt to tell\n+\t * it again. And wait for it to shutdown.\n+\t */\n+\tfsm_listen__stop_async(state);\n+\tpthread_join(state->listener_thread, NULL);\n+\n+\treturn state->error_code;\n+}\n+\n+static int fsmonitor_run_daemon(void)\n+{\n+\tstruct fsmonitor_daemon_state state;\n+\tint err;\n+\n+\tmemset(&state, 0, sizeof(state));\n+\n+\tpthread_mutex_init(&state.main_lock, NULL);\n+\tstate.error_code = 0;\n+\tstate.current_token_data = NULL;\n+\n+\t/* Prepare to (recursively) watch the directory. */\n+\tstrbuf_init(&state.path_worktree_watch, 0);\n+\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n+\tstate.nr_paths_watching = 1;\n+\n+\t/*\n+\t * We create and delete cookie files somewhere inside the .git\n+\t * directory to help us keep sync with the file system. If\n+\t * \".git\" is not a directory, then is not inside the\n+\t * cone of , so set up a second watch to watch\n+\t * the so that we get events for the cookie files.\n+\t */\n+\tstrbuf_init(&state.path_gitdir_watch, 0);\n+\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n+\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n+\tif (!is_directory(state.path_gitdir_watch.buf)) {\n+\t\tstrbuf_reset(&state.path_gitdir_watch);\n+\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n+\t\tstate.nr_paths_watching = 2;\n+\t}\n+\n+\t/*\n+\t * Confirm that we can create platform-specific resources for the\n+\t * filesystem listener before we bother starting all the threads.\n+\t */\n+\tif (fsm_listen__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize listener thread\"));\n+\t\tgoto done;\n+\t}\n+\n+\terr = fsmonitor_run_daemon_1(&state);\n+\n+done:\n+\tpthread_mutex_destroy(&state.main_lock);\n+\tfsm_listen__dtor(&state);\n+\n+\tipc_server_free(state.ipc_server_data);\n+\n+\tstrbuf_release(&state.path_worktree_watch);\n+\tstrbuf_release(&state.path_gitdir_watch);\n+\n+\treturn err;\n+}\n+\n+static int try_to_run_foreground_daemon(void)\n+{\n+\t/*\n+\t * Technically, we don't need to probe for an existing daemon\n+\t * process, since we could just call `fsmonitor_run_daemon()`\n+\t * and let it fail if the pipe/socket is busy.\n+\t *\n+\t * However, this method gives us a nicer error message for a\n+\t * common error case.\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n+\t\t the_repository->worktree);\n+\n+\tprintf(_(\"running fsmonitor-daemon in '%s'\\n\"),\n+\t the_repository->worktree);\n+\tfflush(stdout);\n+\n+\treturn !!fsmonitor_run_daemon();\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n \n \tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"ipc-threads\",\n+\t\t\t &fsmonitor__ipc_threads,\n+\t\t\t N_(\"use ipc worker threads\")),\n \t\tOPT_END()\n \t};\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(fsmonitor_config, NULL);\n \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n@@ -73,6 +277,13 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (fsmonitor__ipc_threads < 1)\n+\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n+\t\t fsmonitor__ipc_threads);\n+\n+\tif (!strcmp(subcmd, \"run\"))\n+\t\treturn !!try_to_run_foreground_daemon();\n+\n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n \ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nnew file mode 100644\nindex 00000000000..3009c1a83de\n--- /dev/null\n+++ b/fsmonitor--daemon.h\n@@ -0,0 +1,34 @@\n+#ifndef FSMONITOR_DAEMON_H\n+#define FSMONITOR_DAEMON_H\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+#include \"thread-utils.h\"\n+\n+struct fsmonitor_batch;\n+struct fsmonitor_token_data;\n+\n+struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+\n+struct fsmonitor_daemon_state {\n+\tpthread_t listener_thread;\n+\tpthread_mutex_t main_lock;\n+\n+\tstruct strbuf path_worktree_watch;\n+\tstruct strbuf path_gitdir_watch;\n+\tint nr_paths_watching;\n+\n+\tstruct fsmonitor_token_data *current_token_data;\n+\n+\tint error_code;\n+\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\n+\tstruct ipc_server_data *ipc_server_data;\n+};\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"448263","messageId":"21c099c5197168ffba5861d2b7a992fce84a7252.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 12/30] fsmonitor--daemon: add pathname classification","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:01Z","receivedAt":"2022-02-11T20:57:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to classify relative and absolute\npathnames and decide how they should be handled. This will\nbe used by the platform-specific backend to respond to each\nfilesystem event.\n\nWhen we register for filesystem notifications on a directory,\nwe get events for everything (recursively) in the directory.\nWe want to report to clients changes to tracked and untracked\npaths within the working directory. We do not want to report\nchanges within the .git directory, for example.\n\nThis classification will be used in a later commit by the\ndifferent backends to classify paths as events are received.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 81 +++++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 61 ++++++++++++++++++++++++++++\n 2 files changed, 142 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 8988440b7c1..1618f1aa27f 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -121,6 +121,87 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n+#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *rel)\n+{\n+\tif (fspathncmp(rel, \".git\", 4))\n+\t\treturn IS_WORKDIR_PATH;\n+\trel += 4;\n+\n+\tif (!*rel)\n+\t\treturn IS_DOT_GIT;\n+\tif (*rel != '/')\n+\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n+\trel++;\n+\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_DOT_GIT;\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *rel)\n+{\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_GITDIR;\n+}\n+\n+static enum fsmonitor_path_type try_classify_workdir_abs_path(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\n+\tif (fspathncmp(path, state->path_worktree_watch.buf,\n+\t\t state->path_worktree_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_worktree_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_workdir_relative(rel);\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\tenum fsmonitor_path_type t;\n+\n+\tt = try_classify_workdir_abs_path(state, path);\n+\tif (state->nr_paths_watching == 1)\n+\t\treturn t;\n+\tif (t != IS_OUTSIDE_CONE)\n+\t\treturn t;\n+\n+\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n+\t\t state->path_gitdir_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_gitdir_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_GITDIR; /* it is the exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_gitdir_relative(rel);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 3009c1a83de..7bbb3a27a1c 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -30,5 +30,66 @@ struct fsmonitor_daemon_state {\n \tstruct ipc_server_data *ipc_server_data;\n };\n \n+/*\n+ * Pathname classifications.\n+ *\n+ * The daemon classifies the pathnames that it receives from file\n+ * system notification events into the following categories and uses\n+ * that to decide whether clients are told about them. (And to watch\n+ * for file system synchronization events.)\n+ *\n+ * The client should only care about paths within the working\n+ * directory proper (inside the working directory and not \".git\" nor\n+ * inside of \".git/\"). That is, the client has read the index and is\n+ * asking for a list of any paths in the working directory that have\n+ * been modified since the last token. The client does not care about\n+ * file system changes within the .git directory (such as new loose\n+ * objects or packfiles). So the client will only receive paths that\n+ * are classified as IS_WORKDIR_PATH.\n+ *\n+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n+ * exact \".git\" directory or GITDIR. If the daemon receives a delete\n+ * event for either of these directories, it will automatically\n+ * shutdown, for example.\n+ *\n+ * Note that the daemon DOES NOT explicitly watch nor special case the\n+ * \".git/index\" file. The daemon does not read the index and does not\n+ * have any internal index-relative state. The daemon only collects\n+ * the set of modified paths within the working directory.\n+ */\n+enum fsmonitor_path_type {\n+\tIS_WORKDIR_PATH = 0,\n+\n+\tIS_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n+\n+\tIS_GITDIR,\n+\tIS_INSIDE_GITDIR,\n+\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n+\n+\tIS_OUTSIDE_CONE,\n+};\n+\n+/*\n+ * Classify a pathname relative to the root of the working directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify a pathname relative to a that is external to the\n+ * worktree directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify an absolute pathname received from a filesystem event.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"448264","messageId":"8d1db6444091e17041f6a649131c2e8705365c95.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 14/30] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:03Z","receivedAt":"2022-02-11T20:57:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to build a list of changed paths and associate\nthem with a token-id. This will be used by the platform-specific\nbackends to accumulate changed paths in response to filesystem events.\n\nThe platform-specific file system listener thread receives file system\nevents containing one or more changed pathnames (with whatever\nbucketing or grouping that is convenient for the file system). These\npaths are accumulated (without locking) by the file system layer into\na `fsmonitor_batch`.\n\nWhen the file system layer has drained the kernel event queue, it will\n\"publish\" them to our token queue and make them visible to concurrent\nclient worker threads. The token layer is free to combine and/or de-dup\npaths within these batches for efficient presentation to clients.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 230 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 40 +++++++\n 2 files changed, 268 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 2997c2cba8c..34603f23053 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -168,17 +168,27 @@ struct fsmonitor_token_data {\n \tuint64_t client_ref_count;\n };\n \n+struct fsmonitor_batch {\n+\tstruct fsmonitor_batch *next;\n+\tuint64_t batch_seq_nr;\n+\tconst char **interned_paths;\n+\tsize_t nr, alloc;\n+\ttime_t pinned_time;\n+};\n+\n static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n {\n \tstatic int test_env_value = -1;\n \tstatic uint64_t flush_count = 0;\n \tstruct fsmonitor_token_data *token;\n+\tstruct fsmonitor_batch *batch;\n \n \tCALLOC_ARRAY(token, 1);\n+\tbatch = fsmonitor_batch__new();\n \n \tstrbuf_init(&token->token_id, 0);\n-\ttoken->batch_head = NULL;\n-\ttoken->batch_tail = NULL;\n+\ttoken->batch_head = batch;\n+\ttoken->batch_tail = batch;\n \ttoken->client_ref_count = 0;\n \n \tif (test_env_value < 0)\n@@ -204,9 +214,143 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n \t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n \t}\n \n+\t/*\n+\t * We created a new and are starting a new series\n+\t * of tokens with a zero .\n+\t *\n+\t * Since clients cannot guess our new (non test) \n+\t * they will always receive a trivial response (because of the\n+\t * mismatch on the ). The trivial response will\n+\t * tell them our new so that subsequent requests\n+\t * will be relative to our new series. (And when sending that\n+\t * response, we pin the current head of the batch list.)\n+\t *\n+\t * Even if the client correctly guesses the , their\n+\t * request of \"builtin::0\" asks for all changes MORE\n+\t * RECENT than batch/bin 0.\n+\t *\n+\t * This implies that it is a waste to accumulate paths in the\n+\t * initial batch/bin (because they will never be transmitted).\n+\t *\n+\t * So the daemon could be running for days and watching the\n+\t * file system, but doesn't need to actually accumulate any\n+\t * paths UNTIL we need to set a reference point for a later\n+\t * relative request.\n+\t *\n+\t * However, it is very useful for testing to always have a\n+\t * reference point set. Pin batch 0 to force early file system\n+\t * events to accumulate.\n+\t */\n+\tif (test_env_value)\n+\t\tbatch->pinned_time = time(NULL);\n+\n \treturn token;\n }\n \n+struct fsmonitor_batch *fsmonitor_batch__new(void)\n+{\n+\tstruct fsmonitor_batch *batch;\n+\n+\tCALLOC_ARRAY(batch, 1);\n+\n+\treturn batch;\n+}\n+\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch)\n+{\n+\twhile (batch) {\n+\t\tstruct fsmonitor_batch *next = batch->next;\n+\n+\t\t/*\n+\t\t * The actual strings within the array of this batch\n+\t\t * are interned, so we don't own them. We only own\n+\t\t * the array.\n+\t\t */\n+\t\tfree(batch->interned_paths);\n+\t\tfree(batch);\n+\n+\t\tbatch = next;\n+\t}\n+}\n+\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n+\t\t\t const char *path)\n+{\n+\tconst char *interned_path = strintern(path);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n+\n+\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n+\tbatch->interned_paths[batch->nr++] = interned_path;\n+}\n+\n+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n+\t\t\t\t const struct fsmonitor_batch *batch_src)\n+{\n+\tsize_t k;\n+\n+\tALLOC_GROW(batch_dest->interned_paths,\n+\t\t batch_dest->nr + batch_src->nr + 1,\n+\t\t batch_dest->alloc);\n+\n+\tfor (k = 0; k < batch_src->nr; k++)\n+\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n+\t\t\tbatch_src->interned_paths[k];\n+}\n+\n+static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n+{\n+\tif (!token)\n+\t\treturn;\n+\n+\tassert(token->client_ref_count == 0);\n+\n+\tstrbuf_release(&token->token_id);\n+\n+\tfsmonitor_batch__free_list(token->batch_head);\n+\n+\tfree(token);\n+}\n+\n+/*\n+ * Flush all of our cached data about the filesystem. Call this if we\n+ * lose sync with the filesystem and miss some notification events.\n+ *\n+ * [1] If we are missing events, then we no longer have a complete\n+ * history of the directory (relative to our current start token).\n+ * We should create a new token and start fresh (as if we just\n+ * booted up).\n+ *\n+ * If there are no concurrent threads readering the current token data\n+ * series, we can free it now. Otherwise, let the last reader free\n+ * it.\n+ *\n+ * Either way, the old token data series is no longer associated with\n+ * our state data.\n+ */\n+static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct fsmonitor_token_data *free_me = NULL;\n+\tstruct fsmonitor_token_data *new_one = NULL;\n+\n+\tnew_one = fsmonitor_new_token_data();\n+\n+\tif (state->current_token_data->client_ref_count == 0)\n+\t\tfree_me = state->current_token_data;\n+\tstate->current_token_data = new_one;\n+\n+\tfsmonitor_free_token_data(free_me);\n+}\n+\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\tpthread_mutex_lock(&state->main_lock);\n+\twith_lock__do_force_resync(state);\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -316,6 +460,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n \n+/*\n+ * We try to combine small batches at the front of the batch-list to avoid\n+ * having a long list. This hopefully makes it a little easier when we want\n+ * to truncate and maintain the list. However, we don't want the paths array\n+ * to just keep growing and growing with realloc, so we insert an arbitrary\n+ * limit.\n+ */\n+#define MY_COMBINE_LIMIT (1024)\n+\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names)\n+{\n+\tif (!batch && !cookie_names->nr)\n+\t\treturn;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (batch) {\n+\t\tstruct fsmonitor_batch *head;\n+\n+\t\thead = state->current_token_data->batch_head;\n+\t\tif (!head) {\n+\t\t\tBUG(\"token does not have batch\");\n+\t\t} else if (head->pinned_time) {\n+\t\t\t/*\n+\t\t\t * We cannot alter the current batch list\n+\t\t\t * because:\n+\t\t\t *\n+\t\t\t * [a] it is being transmitted to at least one\n+\t\t\t * client and the handle_client() thread has a\n+\t\t\t * ref-count, but not a lock on the batch list\n+\t\t\t * starting with this item.\n+\t\t\t *\n+\t\t\t * [b] it has been transmitted in the past to\n+\t\t\t * at least one client such that future\n+\t\t\t * requests are relative to this head batch.\n+\t\t\t *\n+\t\t\t * So, we can only prepend a new batch onto\n+\t\t\t * the front of the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else if (!head->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * Batch 0 is unpinned. See the note in\n+\t\t\t * `fsmonitor_new_token_data()` about why we\n+\t\t\t * don't need to accumulate these paths.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n+\t\t\t/*\n+\t\t\t * The head batch in the list has never been\n+\t\t\t * transmitted to a client, but folding the\n+\t\t\t * contents of the new batch onto it would\n+\t\t\t * exceed our arbitrary limit, so just prepend\n+\t\t\t * the new batch onto the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We are free to add the paths in the given\n+\t\t\t * batch onto the end of the current head batch.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__combine(head, batch);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -330,6 +549,13 @@ static void *fsm_listen__thread_proc(void *_state)\n \n \tfsm_listen__loop(state);\n \n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (state->current_token_data &&\n+\t state->current_token_data->client_ref_count == 0)\n+\t\tfsmonitor_free_token_data(state->current_token_data);\n+\tstate->current_token_data = NULL;\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n \ttrace2_thread_exit();\n \treturn NULL;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 7bbb3a27a1c..20a815d80f8 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -12,6 +12,27 @@\n struct fsmonitor_batch;\n struct fsmonitor_token_data;\n \n+/*\n+ * Create a new batch of path(s). The returned batch is considered\n+ * private and not linked into the fsmonitor daemon state. The caller\n+ * should fill this batch with one or more paths and then publish it.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__new(void);\n+\n+/*\n+ * Free the list of batches starting with this one.\n+ */\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n+\n+/*\n+ * Add this path to this batch of modified files.\n+ *\n+ * The batch should be private and NOT (yet) linked into the fsmonitor\n+ * daemon state and therefore not yet visible to worker threads and so\n+ * no locking is required.\n+ */\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n+\n struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n \n struct fsmonitor_daemon_state {\n@@ -91,5 +112,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \tstruct fsmonitor_daemon_state *state,\n \tconst char *path);\n \n+/*\n+ * Prepend the this batch of path(s) onto the list of batches associated\n+ * with the current token. This makes the batch visible to worker threads.\n+ *\n+ * The caller no longer owns the batch and must not free it.\n+ *\n+ * Wake up the client threads waiting on these cookies.\n+ */\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names);\n+\n+/*\n+ * If the platform-specific layer loses sync with the filesystem,\n+ * it should call this to invalidate cached data and abort waiting\n+ * threads.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"448265","messageId":"72979c35ceb1a4a3d95412834ddff002aaac9b16.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 13/30] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:02Z","receivedAt":"2022-02-11T20:57:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to create token-ids and define the\noverall token naming scheme.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 115 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1618f1aa27f..2997c2cba8c 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -93,6 +93,120 @@ static int do_as_client__status(void)\n \t}\n }\n \n+/*\n+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n+ * \"token\" as a virtual timestamp. Clients can request a summary of all\n+ * created/deleted/modified files relative to a token. In the response,\n+ * clients receive a new token for the next (relative) request.\n+ *\n+ *\n+ * Token Format\n+ * ============\n+ *\n+ * The contents of the token are private and provider-specific.\n+ *\n+ * For the built-in fsmonitor--daemon, we define a token as follows:\n+ *\n+ * \"builtin\" \":\" \":\" \n+ *\n+ * The \"builtin\" prefix is used as a namespace to avoid conflicts\n+ * with other providers (such as Watchman).\n+ *\n+ * The is an arbitrary OPAQUE string, such as a GUID,\n+ * UUID, or {timestamp,pid}. It is used to group all filesystem\n+ * events that happened while the daemon was monitoring (and in-sync\n+ * with the filesystem).\n+ *\n+ * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n+ * and does not define less-than/greater-than relationships.\n+ * (There are too many race conditions to rely on file system\n+ * event timestamps.)\n+ *\n+ * The is a simple integer incremented whenever the\n+ * daemon needs to make its state public. For example, if 1000 file\n+ * system events come in, but no clients have requested the data,\n+ * the daemon can continue to accumulate file changes in the same\n+ * bin and does not need to advance the sequence number. However,\n+ * as soon as a client does arrive, the daemon needs to start a new\n+ * bin and increment the sequence number.\n+ *\n+ * The sequence number serves as the boundary between 2 sets\n+ * of bins -- the older ones that the client has already seen\n+ * and the newer ones that it hasn't.\n+ *\n+ * When a new is created, the is reset to\n+ * zero.\n+ *\n+ *\n+ * About Token Ids\n+ * ===============\n+ *\n+ * A new token_id is created:\n+ *\n+ * [1] each time the daemon is started.\n+ *\n+ * [2] any time that the daemon must re-sync with the filesystem\n+ * (such as when the kernel drops or we miss events on a very\n+ * active volume).\n+ *\n+ * [3] in response to a client \"flush\" command (for dropped event\n+ * testing).\n+ *\n+ * When a new token_id is created, the daemon is free to discard all\n+ * cached filesystem events associated with any previous token_ids.\n+ * Events associated with a non-current token_id will never be sent\n+ * to a client. A token_id change implicitly means that the daemon\n+ * has gap in its event history.\n+ *\n+ * Therefore, clients that present a token with a stale (non-current)\n+ * token_id will always be given a trivial response.\n+ */\n+struct fsmonitor_token_data {\n+\tstruct strbuf token_id;\n+\tstruct fsmonitor_batch *batch_head;\n+\tstruct fsmonitor_batch *batch_tail;\n+\tuint64_t client_ref_count;\n+};\n+\n+static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n+{\n+\tstatic int test_env_value = -1;\n+\tstatic uint64_t flush_count = 0;\n+\tstruct fsmonitor_token_data *token;\n+\n+\tCALLOC_ARRAY(token, 1);\n+\n+\tstrbuf_init(&token->token_id, 0);\n+\ttoken->batch_head = NULL;\n+\ttoken->batch_tail = NULL;\n+\ttoken->client_ref_count = 0;\n+\n+\tif (test_env_value < 0)\n+\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n+\n+\tif (!test_env_value) {\n+\t\tstruct timeval tv;\n+\t\tstruct tm tm;\n+\t\ttime_t secs;\n+\n+\t\tgettimeofday(&tv, NULL);\n+\t\tsecs = tv.tv_sec;\n+\t\tgmtime_r(&secs, &tm);\n+\n+\t\tstrbuf_addf(&token->token_id,\n+\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n+\t\t\t flush_count++,\n+\t\t\t getpid(),\n+\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n+\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n+\t\t\t (long)tv.tv_usec);\n+\t} else {\n+\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n+\t}\n+\n+\treturn token;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -285,7 +399,7 @@ static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n-\tstate.current_token_data = NULL;\n+\tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n \tstrbuf_init(&state.path_worktree_watch, 0);\n-- \ngitgitgadget\n\n"},{"id":"448266","messageId":"69fc0998286cbc791f199710a68a2028080e1632.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 11/30] fsmonitor--daemon: implement 'start' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:00Z","receivedAt":"2022-02-11T20:57:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement 'git fsmonitor--daemon start' command. This command starts\nan instance of 'git fsmonitor--daemon run' in the background using\nthe new 'start_bg_command()' function.\n\nWe avoid the fork-and-call technique on Unix systems in favor of a\nfork-and-exec technique. This gives us more uniform Trace2 child-*\nevents. It also makes our usage more consistent with Windows usage.\n\nOn Windows, teach 'git fsmonitor--daemon run' to optionally call\n'FreeConsole()' to release handles to the inherited Win32 console\n(despite being passed invalid handles for stdin/out/err). Without\nthis, command prompts and powershell terminal windows could hang\nin \"exit\" until the last background child process exited or released\ntheir Win32 console handle. (This was not seen with git-bash shells\nbecause they don't have a Win32 console attached to them.)\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 107 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 105 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex b5ebd1eca33..8988440b7c1 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -9,6 +9,7 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon start []\"),\n \tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n@@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n #define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n static int fsmonitor__ipc_threads = 8;\n \n+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n+static int fsmonitor__start_timeout_sec = 60;\n+\n static int fsmonitor_config(const char *var, const char *value, void *cb)\n {\n \tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n@@ -33,6 +37,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n+\t\tfsmonitor__start_timeout_sec = i;\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -237,7 +250,7 @@ done:\n \treturn err;\n }\n \n-static int try_to_run_foreground_daemon(void)\n+static int try_to_run_foreground_daemon(int free_console)\n {\n \t/*\n \t * Technically, we don't need to probe for an existing daemon\n@@ -255,17 +268,104 @@ static int try_to_run_foreground_daemon(void)\n \t the_repository->worktree);\n \tfflush(stdout);\n \n+#ifdef GIT_WINDOWS_NATIVE\n+\tif (free_console)\n+\t\tFreeConsole();\n+#endif\n+\n \treturn !!fsmonitor_run_daemon();\n }\n \n+static start_bg_wait_cb bg_wait_cb;\n+\n+static int bg_wait_cb(const struct child_process *cp, void *cb_data)\n+{\n+\tenum ipc_active_state s = fsmonitor_ipc__get_state();\n+\n+\tswitch (s) {\n+\tcase IPC_STATE__LISTENING:\n+\t\t/* child is \"ready\" */\n+\t\treturn 0;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\t/* give child more time */\n+\t\treturn 1;\n+\n+\tdefault:\n+\tcase IPC_STATE__INVALID_PATH:\n+\tcase IPC_STATE__OTHER_ERROR:\n+\t\t/* all the time in world won't help */\n+\t\treturn -1;\n+\t}\n+}\n+\n+static int try_to_start_background_daemon(void)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tenum start_bg_result sbgr;\n+\n+\t/*\n+\t * Before we try to create a background daemon process, see\n+\t * if a daemon process is already listening. This makes it\n+\t * easier for us to report an already-listening error to the\n+\t * console, since our spawn/daemon can only report the success\n+\t * of creating the background process (and not whether it\n+\t * immediately exited).\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n+\t\t the_repository->worktree);\n+\n+\tprintf(_(\"starting fsmonitor-daemon in '%s'\\n\"),\n+\t the_repository->worktree);\n+\tfflush(stdout);\n+\n+\tcp.git_cmd = 1;\n+\n+\tstrvec_push(&cp.args, \"fsmonitor--daemon\");\n+\tstrvec_push(&cp.args, \"run\");\n+\tstrvec_push(&cp.args, \"--free-console\");\n+\tstrvec_pushf(&cp.args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n+\n+\tcp.no_stdin = 1;\n+\tcp.no_stdout = 1;\n+\tcp.no_stderr = 1;\n+\n+\tsbgr = start_bg_command(&cp, bg_wait_cb, NULL,\n+\t\t\t\tfsmonitor__start_timeout_sec);\n+\n+\tswitch (sbgr) {\n+\tcase SBGR_READY:\n+\t\treturn 0;\n+\n+\tdefault:\n+\tcase SBGR_ERROR:\n+\tcase SBGR_CB_ERROR:\n+\t\treturn error(\"daemon failed to start\");\n+\n+\tcase SBGR_TIMEOUT:\n+\t\treturn error(\"daemon not online yet\");\n+\n+\tcase SBGR_DIED:\n+\t\treturn error(\"daemon terminated\");\n+\t}\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tint free_console = 0;\n \n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"free-console\", &free_console, N_(\"free console\")),\n \t\tOPT_INTEGER(0, \"ipc-threads\",\n \t\t\t &fsmonitor__ipc_threads,\n \t\t\t N_(\"use ipc worker threads\")),\n+\t\tOPT_INTEGER(0, \"start-timeout\",\n+\t\t\t &fsmonitor__start_timeout_sec,\n+\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -281,8 +381,11 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tif (!strcmp(subcmd, \"start\"))\n+\t\treturn !!try_to_start_background_daemon();\n+\n \tif (!strcmp(subcmd, \"run\"))\n-\t\treturn !!try_to_run_foreground_daemon();\n+\t\treturn !!try_to_run_foreground_daemon(free_console);\n \n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n-- \ngitgitgadget\n\n"},{"id":"448267","messageId":"3517c4a3c1322f07252f63204d60ace9cbb6b1b0.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 19/30] help: include fsmonitor--daemon feature flag in version info","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:08Z","receivedAt":"2022-02-11T20:57:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the \"feature: fsmonitor--daemon\" message to the output of\n`git version --build-options`.\n\nThe builtin FSMonitor is only available on certain platforms and\neven then only when certain Makefile flags are enabled, so print\na message in the verbose version output when it is available.\n\nThis can be used by test scripts for prereq testing. Granted, tests\ncould just try `git fsmonitor--daemon status` and look for a 128 exit\ncode or grep for a \"not supported\" message on stderr, but this is\nrather obscure.\n\nThe main advantage is that the feature message will automatically\nappear in bug reports and other support requests.\n\nSigned-off-by: Jeff Hostetler \n---\n help.c | 4 ++++\n t/test-lib.sh | 6 ++++++\n 2 files changed, 10 insertions(+)\n\ndiff --git a/help.c b/help.c\nindex 71444906ddf..9112a51e84b 100644\n--- a/help.c\n+++ b/help.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"parse-options.h\"\n #include \"prompt.h\"\n+#include \"fsmonitor-ipc.h\"\n \n struct category_description {\n \tuint32_t category;\n@@ -695,6 +696,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n+\n+\t\tif (fsmonitor_ipc__is_supported())\n+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n \t}\n }\n \ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e4716b0b867..46cd596e7f5 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1799,3 +1799,9 @@ test_lazy_prereq SHA1 '\n # Tests that verify the scheduler integration must set this locally\n # to avoid errors.\n GIT_TEST_MAINT_SCHEDULER=\"none:exit 1\"\n+\n+# Does this platform support `git fsmonitor--daemon`\n+#\n+test_lazy_prereq FSMONITOR_DAEMON '\n+\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n+'\n-- \ngitgitgadget\n\n"},{"id":"448268","messageId":"3de1c43beaf0c558572e1eaefbf1e266753f20a1.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 18/30] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:07Z","receivedAt":"2022-02-11T20:57:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to respond to IPC requests from client\nGit processes and respond with a list of modified pathnames\nrelative to the provided token.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 312 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 310 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 34603f23053..4c88171e06c 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,6 +7,7 @@\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n+#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon start []\"),\n@@ -351,6 +352,311 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+/*\n+ * Format an opaque token string to send to the client.\n+ */\n+static void with_lock__format_response_token(\n+\tstruct strbuf *response_token,\n+\tconst struct strbuf *response_token_id,\n+\tconst struct fsmonitor_batch *batch)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstrbuf_reset(response_token);\n+\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n+\t\t response_token_id->buf, batch->batch_seq_nr);\n+}\n+\n+/*\n+ * Parse an opaque token from the client.\n+ * Returns -1 on error.\n+ */\n+static int fsmonitor_parse_client_token(const char *buf_token,\n+\t\t\t\t\tstruct strbuf *requested_token_id,\n+\t\t\t\t\tuint64_t *seq_nr)\n+{\n+\tconst char *p;\n+\tchar *p_end;\n+\n+\tstrbuf_reset(requested_token_id);\n+\t*seq_nr = 0;\n+\n+\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n+\t\treturn -1;\n+\n+\twhile (*p && *p != ':')\n+\t\tstrbuf_addch(requested_token_id, *p++);\n+\tif (!*p++)\n+\t\treturn -1;\n+\n+\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n+\tif (*p_end)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal)\n+\n+static int do_handle_client(struct fsmonitor_daemon_state *state,\n+\t\t\t const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\tstruct fsmonitor_token_data *token_data = NULL;\n+\tstruct strbuf response_token = STRBUF_INIT;\n+\tstruct strbuf requested_token_id = STRBUF_INIT;\n+\tstruct strbuf payload = STRBUF_INIT;\n+\tuint64_t requested_oldest_seq_nr = 0;\n+\tuint64_t total_response_len = 0;\n+\tconst char *p;\n+\tconst struct fsmonitor_batch *batch_head;\n+\tconst struct fsmonitor_batch *batch;\n+\tintmax_t count = 0, duplicates = 0;\n+\tkh_str_t *shown;\n+\tint hash_ret;\n+\tint do_trivial = 0;\n+\tint do_flush = 0;\n+\n+\t/*\n+\t * We expect `command` to be of the form:\n+\t *\n+\t * := quit NUL\n+\t * | flush NUL\n+\t * | NUL\n+\t * | NUL\n+\t */\n+\n+\tif (!strcmp(command, \"quit\")) {\n+\t\t/*\n+\t\t * A client has requested over the socket/pipe that the\n+\t\t * daemon shutdown.\n+\t\t *\n+\t\t * Tell the IPC thread pool to shutdown (which completes\n+\t\t * the await in the main thread (which can stop the\n+\t\t * fsmonitor listener thread)).\n+\t\t *\n+\t\t * There is no reply to the client.\n+\t\t */\n+\t\treturn SIMPLE_IPC_QUIT;\n+\n+\t} else if (!strcmp(command, \"flush\")) {\n+\t\t/*\n+\t\t * Flush all of our cached data and generate a new token\n+\t\t * just like if we lost sync with the filesystem.\n+\t\t *\n+\t\t * Then send a trivial response using the new token.\n+\t\t */\n+\t\tdo_flush = 1;\n+\t\tdo_trivial = 1;\n+\n+\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n+\t\t/* assume V1 timestamp or garbage */\n+\n+\t\tchar *p_end;\n+\n+\t\tstrtoumax(command, &p_end, 10);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t ((*p_end) ?\n+\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n+\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n+\t\t\t\t command);\n+\t\tdo_trivial = 1;\n+\n+\t} else {\n+\t\t/* We have \"builtin:*\" */\n+\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n+\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n+\t\t\t\t\t command);\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We have a V2 valid token:\n+\t\t\t * \"builtin::\"\n+\t\t\t */\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (!state->current_token_data)\n+\t\tBUG(\"fsmonitor state does not have a current token\");\n+\n+\tif (do_flush)\n+\t\twith_lock__do_force_resync(state);\n+\n+\t/*\n+\t * We mark the current head of the batch list as \"pinned\" so\n+\t * that the listener thread will treat this item as read-only\n+\t * (and prevent any more paths from being added to it) from\n+\t * now on.\n+\t */\n+\ttoken_data = state->current_token_data;\n+\tbatch_head = token_data->batch_head;\n+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n+\n+\t/*\n+\t * FSMonitor Protocol V2 requires that we send a response header\n+\t * with a \"new current token\" and then all of the paths that changed\n+\t * since the \"requested token\". We send the seq_nr of the just-pinned\n+\t * head batch so that future requests from a client will be relative\n+\t * to it.\n+\t */\n+\twith_lock__format_response_token(&response_token,\n+\t\t\t\t\t &token_data->token_id, batch_head);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttotal_response_len += response_token.len + 1;\n+\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n+\t\t\t response_token.buf);\n+\n+\tif (!do_trivial) {\n+\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n+\t\t\t/*\n+\t\t\t * The client last spoke to a different daemon\n+\t\t\t * instance -OR- the daemon had to resync with\n+\t\t\t * the filesystem (and lost events), so reject.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n+\t\t\t\t\t \"response/token\", \"different\");\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else if (requested_oldest_seq_nr <\n+\t\t\t token_data->batch_tail->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * The client wants older events than we have for\n+\t\t\t * this token_id. This means that the end of our\n+\t\t\t * batch list was truncated and we cannot give the\n+\t\t\t * client a complete snapshot relative to their\n+\t\t\t * request.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"client requested truncated data\");\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n+\tif (do_trivial) {\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\treply(reply_data, \"/\", 2);\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n+\t\t\t\t \"response/trivial\", 1);\n+\n+\t\tstrbuf_release(&response_token);\n+\t\tstrbuf_release(&requested_token_id);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * We're going to hold onto a pointer to the current\n+\t * token-data while we walk the list of batches of files.\n+\t * During this time, we will NOT be under the lock.\n+\t * So we ref-count it.\n+\t *\n+\t * This allows the listener thread to continue prepending\n+\t * new batches of items to the token-data (which we'll ignore).\n+\t *\n+\t * AND it allows the listener thread to do a token-reset\n+\t * (and install a new `current_token_data`).\n+\t */\n+\ttoken_data->client_ref_count++;\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * The client request is relative to the token that they sent,\n+\t * so walk the batch list backwards from the current head back\n+\t * to the batch (sequence number) they named.\n+\t *\n+\t * We use khash to de-dup the list of pathnames.\n+\t *\n+\t * NEEDSWORK: each batch contains a list of interned strings,\n+\t * so we only need to do pointer comparisons here to build the\n+\t * hash table. Currently, we're still comparing the string\n+\t * values.\n+\t */\n+\tshown = kh_init_str();\n+\tfor (batch = batch_head;\n+\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n+\t batch = batch->next) {\n+\t\tsize_t k;\n+\n+\t\tfor (k = 0; k < batch->nr; k++) {\n+\t\t\tconst char *s = batch->interned_paths[k];\n+\t\t\tsize_t s_len;\n+\n+\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n+\t\t\t\tduplicates++;\n+\t\t\telse {\n+\t\t\t\tkh_put_str(shown, s, &hash_ret);\n+\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n+\t\t\t\t\t\t count, s);\n+\n+\t\t\t\t/* Each path gets written with a trailing NUL */\n+\t\t\t\ts_len = strlen(s) + 1;\n+\n+\t\t\t\tif (payload.len + s_len >=\n+\t\t\t\t LARGE_PACKET_DATA_MAX) {\n+\t\t\t\t\treply(reply_data, payload.buf,\n+\t\t\t\t\t payload.len);\n+\t\t\t\t\ttotal_response_len += payload.len;\n+\t\t\t\t\tstrbuf_reset(&payload);\n+\t\t\t\t}\n+\n+\t\t\t\tstrbuf_add(&payload, s, s_len);\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (payload.len) {\n+\t\treply(reply_data, payload.buf, payload.len);\n+\t\ttotal_response_len += payload.len;\n+\t}\n+\n+\tkh_release_str(shown);\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (token_data->client_ref_count > 0)\n+\t\ttoken_data->client_ref_count--;\n+\n+\tif (token_data->client_ref_count == 0) {\n+\t\tif (token_data != state->current_token_data) {\n+\t\t\t/*\n+\t\t\t * The listener thread did a token-reset while we were\n+\t\t\t * walking the batch list. Therefore, this token is\n+\t\t\t * stale and can be discarded completely. If we are\n+\t\t\t * the last reader thread using this token, we own\n+\t\t\t * that work.\n+\t\t\t */\n+\t\t\tfsmonitor_free_token_data(token_data);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n+\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\tstrbuf_release(&payload);\n+\n+\treturn 0;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -358,7 +664,7 @@ static int handle_client(void *data,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n-\t/* struct fsmonitor_daemon_state *state = data; */\n+\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n \t/*\n@@ -369,10 +675,12 @@ static int handle_client(void *data,\n \tif (command_len != strlen(command))\n \t\tBUG(\"FSMonitor assumes text messages\");\n \n+\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n+\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n \n-\tresult = 0; /* TODO Do something here. */\n+\tresult = do_handle_client(state, command, reply, reply_data);\n \n \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n \n-- \ngitgitgadget\n\n"},{"id":"448269","messageId":"162e357db7211e72c8a39e722e8432baf710c70d.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 17/30] compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:06Z","receivedAt":"2022-02-11T20:57:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement file system event listener on MacOS using FSEvent,\nCoreFoundation, and CoreServices.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 382 ++++++++++++++++++++++++++-\n 1 file changed, 381 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex f424253d3eb..2aefdc14d89 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -1,4 +1,4 @@\n-#if defined(__GNUC__)\n+#ifndef __clang__\n /*\n * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n * with clang, but not with GCC as of time of writing.\n@@ -97,20 +97,400 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tCFStringRef cfsr_worktree_path;\n+\tCFStringRef cfsr_gitdir_path;\n+\n+\tCFArrayRef cfar_paths_to_watch;\n+\tint nr_paths_watching;\n+\n+\tFSEventStreamRef stream;\n+\n+\tCFRunLoopRef rl;\n+\n+\tenum shutdown_style {\n+\t\tSHUTDOWN_EVENT = 0,\n+\t\tFORCE_SHUTDOWN,\n+\t\tFORCE_ERROR_STOP,\n+\t} shutdown_style;\n+\n+\tunsigned int stream_scheduled:1;\n+\tunsigned int stream_started:1;\n+};\n+\n+static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n+\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n+\tif (flag & kFSEventStreamEventFlagUserDropped)\n+\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n+\tif (flag & kFSEventStreamEventFlagKernelDropped)\n+\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n+\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n+\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n+\tif (flag & kFSEventStreamEventFlagHistoryDone)\n+\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n+\tif (flag & kFSEventStreamEventFlagRootChanged)\n+\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n+\tif (flag & kFSEventStreamEventFlagMount)\n+\t\tstrbuf_addstr(&msg, \"Mount|\");\n+\tif (flag & kFSEventStreamEventFlagUnmount)\n+\t\tstrbuf_addstr(&msg, \"Unmount|\");\n+\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n+\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n+\tif (flag & kFSEventStreamEventFlagItemCreated)\n+\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n+\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n+\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n+\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsDir)\n+\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsFile)\n+\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemModified)\n+\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n+\tif (flag & kFSEventStreamEventFlagItemRemoved)\n+\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n+\tif (flag & kFSEventStreamEventFlagItemRenamed)\n+\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n+\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n+\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n+\tif (flag & kFSEventStreamEventFlagOwnEvent)\n+\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n+\tif (flag & kFSEventStreamEventFlagItemCloned)\n+\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\t\t\t path, flag, msg.buf);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRemoved);\n+}\n+\n+static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRenamed);\n+}\n+\n+static int ef_is_dropped(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagMustScanSubDirs ||\n+\t\tef & kFSEventStreamEventFlagKernelDropped ||\n+\t\tef & kFSEventStreamEventFlagUserDropped);\n+}\n+\n+static void fsevent_callback(ConstFSEventStreamRef streamRef,\n+\t\t\t void *ctx,\n+\t\t\t size_t num_of_events,\n+\t\t\t void *event_paths,\n+\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t const FSEventStreamEventId event_ids[])\n+{\n+\tstruct fsmonitor_daemon_state *state = ctx;\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tchar **paths = (char **)event_paths;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *path_k;\n+\tconst char *slash;\n+\tint k;\n+\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t/*\n+\t * Build a list of all filesystem changes into a private/local\n+\t * list and without holding any locks.\n+\t */\n+\tfor (k = 0; k < num_of_events; k++) {\n+\t\t/*\n+\t\t * On Mac, we receive an array of absolute paths.\n+\t\t */\n+\t\tpath_k = paths[k];\n+\n+\t\t/*\n+\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n+\t\t * Please don't log them to Trace2.\n+\t\t *\n+\t\t * trace_printf_key(&trace_fsmonitor, \"Path: '%s'\", path_k);\n+\t\t */\n+\n+\t\t/*\n+\t\t * If event[k] is marked as dropped, we assume that we have\n+\t\t * lost sync with the filesystem and should flush our cached\n+\t\t * data. We need to:\n+\t\t *\n+\t\t * [1] Abort/wake any client threads waiting for a cookie and\n+\t\t * flush the cached state data (the current token), and\n+\t\t * create a new token.\n+\t\t *\n+\t\t * [2] Discard the batch that we were locally building (since\n+\t\t * they are conceptually relative to the just flushed\n+\t\t * token).\n+\t\t */\n+\t\tif (ef_is_dropped(event_flags[k])) {\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\tfsmonitor_force_resync(state);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t\tstring_list_clear(&cookie_list, 0);\n+\n+\t\t\t/*\n+\t\t\t * We assume that any events that we received\n+\t\t\t * in this callback after this dropped event\n+\t\t\t * may still be valid, so we continue rather\n+\t\t\t * than break. (And just in case there is a\n+\t\t\t * delete of \".git\" hiding in there.)\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n+\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git or gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path_k);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path_k);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\t/* ignore all other paths inside of .git or gitdir */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\tcase IS_GITDIR:\n+\t\t\t/*\n+\t\t\t * If .git directory is deleted or renamed away,\n+\t\t\t * we have to quit.\n+\t\t\t */\n+\t\t\tif (ef_is_root_delete(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir renamed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* try to queue normal pathnames */\n+\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\t/*\n+\t\t\t * Because of the implicit \"binning\" (the\n+\t\t\t * kernel calls us at a given frequency) and\n+\t\t\t * de-duping (the kernel is free to combine\n+\t\t\t * multiple events for a given pathname), an\n+\t\t\t * individual fsevent could be marked as both\n+\t\t\t * a file and directory. Add it to the queue\n+\t\t\t * with both spellings so that the client will\n+\t\t\t * know how much to invalidate/refresh.\n+\t\t\t */\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t}\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tstrbuf_reset(&tmp);\n+\t\t\t\tstrbuf_addstr(&tmp, rel);\n+\t\t\t\tstrbuf_addch(&tmp, '/');\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\n+\t\tcase IS_OUTSIDE_CONE:\n+\t\tdefault:\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignoring '%s'\", path_k);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\n+\tdata->shutdown_style = FORCE_SHUTDOWN;\n+\tCFRunLoopStop(data->rl);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+}\n+\n+/*\n+ * NEEDSWORK: Investigate the proper value for the `latency` argument\n+ * in the call to `FSEventStreamCreate()`. I'm not sure that this\n+ * needs to be a config setting or just something that we tune after\n+ * some testing.\n+ *\n+ * With a latency of 0.1, I was seeing lots of dropped events during\n+ * the \"touch 100000\" files test within t/perf/p7519, but with a\n+ * latency of 0.001 I did not see any dropped events. So the\n+ * \"correct\" value may be somewhere in between.\n+ *\n+ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n+ */\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n+\t\tkFSEventStreamCreateFlagWatchRoot |\n+\t\tkFSEventStreamCreateFlagFileEvents;\n+\tFSEventStreamContext ctx = {\n+\t\t0,\n+\t\tstate,\n+\t\tNULL,\n+\t\tNULL,\n+\t\tNULL\n+\t};\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\tconst void *dir_array[2];\n+\n+\tCALLOC_ARRAY(data, 1);\n+\tstate->backend_data = data;\n+\n+\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n+\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n+\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n+\t\t\tNULL, state->path_gitdir_watch.buf,\n+\t\t\tkCFStringEncodingUTF8);\n+\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n+\t}\n+\n+\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n+\t\t\t\t\t\t data->nr_paths_watching,\n+\t\t\t\t\t\t NULL);\n+\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n+\t\t\t\t\t data->cfar_paths_to_watch,\n+\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n+\t\t\t\t\t 0.001, flags);\n+\tif (data->stream == NULL)\n+\t\tgoto failed;\n+\n+\t/*\n+\t * `data->rl` needs to be set inside the listener thread.\n+\t */\n+\n+\treturn 0;\n+\n+failed:\n+\terror(\"Unable to create FSEventStream.\");\n+\n+\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tif (data->stream) {\n+\t\tif (data->stream_started)\n+\t\t\tFSEventStreamStop(data->stream);\n+\t\tif (data->stream_scheduled)\n+\t\t\tFSEventStreamInvalidate(data->stream);\n+\t\tFSEventStreamRelease(data->stream);\n+\t}\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\tdata->shutdown_style = SHUTDOWN_EVENT;\n+\n+\tCFRunLoopStop(data->rl);\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\n+\tdata->rl = CFRunLoopGetCurrent();\n+\n+\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n+\tdata->stream_scheduled = 1;\n+\n+\tif (!FSEventStreamStart(data->stream)) {\n+\t\terror(\"Failed to start the FSEventStream\");\n+\t\tgoto force_error_stop_without_loop;\n+\t}\n+\tdata->stream_started = 1;\n+\n+\tCFRunLoopRun();\n+\n+\tswitch (data->shutdown_style) {\n+\tcase FORCE_ERROR_STOP:\n+\t\tstate->error_code = -1;\n+\t\t/* fall thru */\n+\tcase FORCE_SHUTDOWN:\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\t/* fall thru */\n+\tcase SHUTDOWN_EVENT:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn;\n+\n+force_error_stop_without_loop:\n+\tstate->error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+\treturn;\n }\n-- \ngitgitgadget\n\n"},{"id":"448270","messageId":"4ffc2ddf516f248c35a0ca155f758a455722d7a9.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 20/30] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:09Z","receivedAt":"2022-02-11T20:57:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate an IPC client to send query and flush commands to the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n t/helper/test-fsmonitor-client.c | 121 +++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 124 insertions(+)\n create mode 100644 t/helper/test-fsmonitor-client.c\n\ndiff --git a/Makefile b/Makefile\nindex b98f4899ac0..6846d91c37b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\n TEST_BUILTINS_OBJS += test-getcwd.o\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nnew file mode 100644\nindex 00000000000..f7a5b3a32fa\n--- /dev/null\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -0,0 +1,121 @@\n+/*\n+ * test-fsmonitor-client.c: client code to send commands/requests to\n+ * a `git fsmonitor--daemon` daemon.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tdie(\"fsmonitor--daemon not available on this platform\");\n+}\n+#else\n+\n+/*\n+ * Read the `.git/index` to get the last token written to the\n+ * FSMonitor Index Extension.\n+ */\n+static const char *get_token_from_index(void)\n+{\n+\tstruct index_state *istate = the_repository->index;\n+\n+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update)\n+\t\tdie(\"index file does not have fsmonitor extension\");\n+\n+\treturn istate->fsmonitor_last_update;\n+}\n+\n+/*\n+ * Send an IPC query to a `git-fsmonitor--daemon` daemon and\n+ * ask for the changes since the given token or from the last\n+ * token in the index extension.\n+ *\n+ * This will implicitly start a daemon process if necessary. The\n+ * daemon process will persist after we exit.\n+ */\n+static int do_send_query(const char *token)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\n+\tret = fsmonitor_ipc__send_query(token, &answer);\n+\tif (ret < 0)\n+\t\tdie(_(\"could not query fsmonitor--daemon\"));\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n+ * and tell it to flush its cache.\n+ *\n+ * This feature is primarily used by the test suite to simulate a loss of\n+ * sync with the filesystem where we miss kernel events.\n+ */\n+static int do_send_flush(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tconst char *subcmd;\n+\tconst char *token = NULL;\n+\n+\tconst char * const fsmonitor_client_usage[] = {\n+\t\tN_(\"test-helper fsmonitor-client query []\"),\n+\t\tN_(\"test-helper fsmonitor-client flush\"),\n+\t\tNULL,\n+\t};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING(0, \"token\", &token, N_(\"token\"),\n+\t\t\t N_(\"command token to send to the server\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc < 2)\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tsubcmd = argv[1];\n+\targv--;\n+\targc++;\n+\n+\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n+\n+\tsetup_git_directory();\n+\n+\tif (!strcmp(subcmd, \"query\"))\n+\t\treturn !!do_send_query(token);\n+\n+\tif (!strcmp(subcmd, \"flush\"))\n+\t\treturn !!do_send_flush();\n+\n+\tdie(\"Unhandled subcommand: '%s'\", subcmd);\n+}\n+#endif\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 338a57b104d..a4c5647b2f1 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\n \t{ \"getcwd\", cmd__getcwd },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 48cee1f4a2d..fa4c936310c 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -22,6 +22,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"448271","messageId":"27e471089083dca302e3e160f7b1a0290046a49b.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 22/30] t/perf: avoid copying builtin fsmonitor files into test repo","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:11Z","receivedAt":"2022-02-11T20:57:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDo not copy any of the various fsmonitor--daemon files from the .git\ndirectory of the (GIT_PREF_REPO or GIT_PERF_LARGE_REPO) source repo\ninto the test's trash directory.\n\nWhen perf tests start, they copy the contents of the source repo into\nthe test's trash directory. If fsmonitor is running in the source repo,\nthere may be control files, such as the IPC socket and/or fsmonitor\ncookie files. These should not be copied into the test repo.\n\nUnix domain sockets cannot be copied in the manner used by the test\nsetup, so if present, the test setup fails.\n\nCookie files are harmless, but we should avoid them.\n\nThe builtin fsmonitor keeps all such control files/sockets in\n.git/fsmonitor--daemon*, so it is simple to exclude them.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/perf-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 407252bac70..932105cd12c 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -78,7 +78,7 @@ test_perf_copy_repo_contents () {\n \tfor stuff in \"$1\"/*\n \tdo\n \t\tcase \"$stuff\" in\n-\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees)\n+\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)\n \t\t\t;;\n \t\t*)\n \t\t\tcp -R \"$stuff\" \"$repo/.git/\" || exit 1\n-- \ngitgitgadget\n\n"},{"id":"448272","messageId":"d93310a7c64b4b02e05c3da3c7d52fca39be759b.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 21/30] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:10Z","receivedAt":"2022-02-11T20:57:21Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 511 +++++++++++++++++++++++++++++++++++\n 1 file changed, 511 insertions(+)\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..5f7b8e54233\n--- /dev/null\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -0,0 +1,511 @@\n+#!/bin/sh\n+\n+test_description='built-in file system watcher'\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+stop_daemon_delete_repo () {\n+\tr=$1\n+\tgit -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n+\trm -rf $1\n+\treturn 0\n+}\n+\n+start_daemon () {\n+\tcase \"$#\" in\n+\t\t1) r=\"-C $1\";;\n+\t\t*) r=\"\";\n+\tesac\n+\n+\tgit $r fsmonitor--daemon start || return $?\n+\tgit $r fsmonitor--daemon status || return $?\n+\n+\treturn 0\n+}\n+\n+# Is a Trace2 data event present with the given catetory and key?\n+# We do not care what the value is.\n+#\n+have_t2_data_event () {\n+\tc=$1\n+\tk=$2\n+\n+\tgrep -e '\"event\":\"data\".*\"category\":\"'\"$c\"'\".*\"key\":\"'\"$k\"'\"'\n+}\n+\n+test_expect_success 'explicit daemon start and stop' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n+\n+\tgit init test_explicit &&\n+\tstart_daemon test_explicit &&\n+\n+\tgit -C test_explicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon start' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit\" &&\n+\n+\tgit init test_implicit &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n+\n+\t# query will implicitly start the daemon.\n+\t#\n+\t# for test-script simplicity, we send a V1 timestamp rather than\n+\t# a V2 token. either way, the daemon response to any query contains\n+\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n+\t# but this test case is only concerned with whether the daemon was\n+\t# implicitly started.)\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace\" \\\n+\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n+\tnul_to_q actual.filtered &&\n+\tgrep \"builtin:\" actual.filtered &&\n+\n+\t# confirm that a daemon was started in the background.\n+\t#\n+\t# since the mechanism for starting the background daemon is platform\n+\t# dependent, just confirm that the foreground command received a\n+\t# response from the daemon.\n+\n+\thave_t2_data_event fsm_client query/response-length <.git/trace &&\n+\n+\tgit -C test_implicit fsmonitor--daemon status &&\n+\tgit -C test_implicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (delete .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n+\n+\tgit init test_implicit_1 &&\n+\n+\tstart_daemon test_implicit_1 &&\n+\n+\t# deleting the .git directory will implicitly stop the daemon.\n+\trm -rf test_implicit_1/.git &&\n+\n+\t# [1] Create an empty .git directory so that the following Git\n+\t# command will stay relative to the `-C` directory.\n+\t#\n+\t# Without this, the Git command will override the requested\n+\t# -C argument and crawl out to the containing Git source tree.\n+\t# This would make the test result dependent upon whether we\n+\t# were using fsmonitor on our development worktree.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_1/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (rename .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_2\" &&\n+\n+\tgit init test_implicit_2 &&\n+\n+\tstart_daemon test_implicit_2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n+\n+\t# See [1] above.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'cannot start multiple daemons' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n+\n+\tgit init test_multiple &&\n+\n+\tstart_daemon test_multiple &&\n+\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n+\tgrep \"fsmonitor--daemon is already running\" actual &&\n+\n+\tgit -C test_multiple fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n+'\n+\n+# These tests use the main repo in the trash directory\n+\n+test_expect_success 'setup' '\n+\t>tracked &&\n+\t>modified &&\n+\t>delete &&\n+\t>rename &&\n+\tmkdir dir1 &&\n+\t>dir1/tracked &&\n+\t>dir1/modified &&\n+\t>dir1/delete &&\n+\t>dir1/rename &&\n+\tmkdir dir2 &&\n+\t>dir2/tracked &&\n+\t>dir2/modified &&\n+\t>dir2/delete &&\n+\t>dir2/rename &&\n+\tmkdir dirtorename &&\n+\t>dirtorename/a &&\n+\t>dirtorename/b &&\n+\n+\tcat >.gitignore <<-\\EOF &&\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tEOF\n+\n+\tgit -c core.fsmonitor=false add . &&\n+\ttest_tick &&\n+\tgit -c core.fsmonitor=false commit -m initial &&\n+\n+\tgit config core.fsmonitor true\n+'\n+\n+# The test already explicitly stopped (or tried to stop) the daemon.\n+# This is here in case something else fails first.\n+#\n+redundant_stop_daemon () {\n+\tgit fsmonitor--daemon stop\n+\treturn 0\n+}\n+\n+test_expect_success 'update-index implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_1\" \\\n+\t\tgit update-index --fsmonitor &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_1\n+'\n+\n+test_expect_success 'status implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_2\" \\\n+\t\tgit status >actual &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_2\n+'\n+\n+edit_files () {\n+\techo 1 >modified\n+\techo 2 >dir1/modified\n+\techo 3 >dir2/modified\n+\t>dir1/untracked\n+}\n+\n+delete_files () {\n+\trm -f delete\n+\trm -f dir1/delete\n+\trm -f dir2/delete\n+}\n+\n+create_files () {\n+\techo 1 >new\n+\techo 2 >dir1/new\n+\techo 3 >dir2/new\n+}\n+\n+rename_files () {\n+\tmv rename renamed\n+\tmv dir1/rename dir1/renamed\n+\tmv dir2/rename dir2/renamed\n+}\n+\n+file_to_directory () {\n+\trm -f delete\n+\tmkdir delete\n+\techo 1 >delete/new\n+}\n+\n+directory_to_file () {\n+\trm -rf dir1\n+\techo 1 >dir1\n+}\n+\n+verify_status () {\n+\tgit status >actual &&\n+\tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n+\tGIT_INDEX_FILE=.git/fresh-index git -c core.fsmonitor=false status >expect &&\n+\ttest_cmp expect actual &&\n+\techo HELLO AFTER &&\n+\tcat .git/trace &&\n+\techo HELLO AFTER\n+}\n+\n+# The next few test cases confirm that our fsmonitor daemon sees each type\n+# of OS filesystem notification that we care about. At this layer we just\n+# ensure we are getting the OS notifications and do not try to confirm what\n+# is reported by `git status`.\n+#\n+# We run a simple query after modifying the filesystem just to introduce\n+# a bit of a delay so that the trace logging from the daemon has time to\n+# get flushed to disk.\n+#\n+# We `reset` and `clean` at the bottom of each test (and before stopping the\n+# daemon) because these commands might implicitly restart the daemon.\n+\n+clean_up_repo_and_stop_daemon () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+\tgit fsmonitor--daemon stop\n+\trm -f .git/trace\n+}\n+\n+test_expect_success 'edit some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tedit_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/modified$\" .git/trace &&\n+\tgrep \"^event: dir2/modified$\" .git/trace &&\n+\tgrep \"^event: modified$\" .git/trace &&\n+\tgrep \"^event: dir1/untracked$\" .git/trace\n+'\n+\n+test_expect_success 'create some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tcreate_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/new$\" .git/trace &&\n+\tgrep \"^event: dir2/new$\" .git/trace &&\n+\tgrep \"^event: new$\" .git/trace\n+'\n+\n+test_expect_success 'delete some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdelete_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/delete$\" .git/trace &&\n+\tgrep \"^event: dir2/delete$\" .git/trace &&\n+\tgrep \"^event: delete$\" .git/trace\n+'\n+\n+test_expect_success 'rename some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\trename_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/rename$\" .git/trace &&\n+\tgrep \"^event: dir2/rename$\" .git/trace &&\n+\tgrep \"^event: rename$\" .git/trace &&\n+\tgrep \"^event: dir1/renamed$\" .git/trace &&\n+\tgrep \"^event: dir2/renamed$\" .git/trace &&\n+\tgrep \"^event: renamed$\" .git/trace\n+'\n+\n+test_expect_success 'rename directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tmv dirtorename dirrenamed &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dirtorename/*$\" .git/trace &&\n+\tgrep \"^event: dirrenamed/*$\" .git/trace\n+'\n+\n+test_expect_success 'file changes to directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tfile_to_directory &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: delete$\" .git/trace &&\n+\tgrep \"^event: delete/new$\" .git/trace\n+'\n+\n+test_expect_success 'directory changes to a file' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdirectory_to_file &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1$\" .git/trace\n+'\n+\n+# The next few test cases exercise the token-resync code. When filesystem\n+# drops events (because of filesystem velocity or because the daemon isn't\n+# polling fast enough), we need to discard the cached data (relative to the\n+# current token) and start collecting events under a new token.\n+#\n+# the 'test-tool fsmonitor-client flush' command can be used to send a\n+# \"flush\" message to a running daemon and ask it to do a flush/resync.\n+\n+test_expect_success 'flush cached data' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_flush\" &&\n+\n+\tgit init test_flush &&\n+\n+\t(\n+\t\tGIT_TEST_FSMONITOR_TOKEN=true &&\n+\t\texport GIT_TEST_FSMONITOR_TOKEN &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace_daemon\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon test_flush\n+\t) &&\n+\n+\t# The daemon should have an initial token with no events in _0 and\n+\t# then a few (probably platform-specific number of) events in _1.\n+\t# These should both have the same .\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n+\tnul_to_q actual_q0 &&\n+\n+\ttouch test_flush/file_1 &&\n+\ttouch test_flush/file_2 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n+\tnul_to_q actual_q1 &&\n+\n+\tgrep \"file_1\" actual_q1 &&\n+\n+\t# Force a flush. This will change the , reset the , and\n+\t# flush the file data. Then create some events and ensure that the file\n+\t# again appears in the cache. It should have the new .\n+\n+\ttest-tool -C test_flush fsmonitor-client flush >flush_0 &&\n+\tnul_to_q flush_q0 &&\n+\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_2 &&\n+\tnul_to_q actual_q2 &&\n+\n+\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n+\n+\ttouch test_flush/file_3 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n+\tnul_to_q actual_q3 &&\n+\n+\tgrep \"file_3\" actual_q3\n+'\n+\n+# The next few test cases create repos where the .git directory is NOT\n+# inside the one of the working directory. That is, where .git is a file\n+# that points to a directory elsewhere. This happens for submodules and\n+# non-primary worktrees.\n+\n+test_expect_success 'setup worktree base' '\n+\tgit init wt-base &&\n+\techo 1 >wt-base/file1 &&\n+\tgit -C wt-base add file1 &&\n+\tgit -C wt-base commit -m \"c1\"\n+'\n+\n+test_expect_success 'worktree with .git file' '\n+\tgit -C wt-base worktree add ../wt-secondary &&\n+\n+\t(\n+\t\tGIT_TRACE2_PERF=\"$(pwd)/trace2_wt_secondary\" &&\n+\t\texport GIT_TRACE2_PERF &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/trace_wt_secondary\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon wt-secondary\n+\t) &&\n+\n+\tgit -C wt-secondary fsmonitor--daemon stop &&\n+\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n+'\n+\n+# NEEDSWORK: Repeat one of the \"edit\" tests on wt-secondary and\n+# confirm that we get the same events and behavior -- that is, that\n+# fsmonitor--daemon correctly watches BOTH the working directory and\n+# the external GITDIR directory and behaves the same as when \".git\"\n+# is a directory inside the working directory.\n+\n+test_expect_success 'cleanup worktrees' '\n+\tstop_daemon_delete_repo wt-secondary &&\n+\tstop_daemon_delete_repo wt-base\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"448273","messageId":"6cba1d950b013410ecc6ffc15bfcba02c51d6de2.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 23/30] t/helper/test-chmtime: skip directories on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:12Z","receivedAt":"2022-02-11T20:57:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach `test-tool.exe chmtime` to ignore errors when setting the mtime\non a directory on Windows.\n\nNEEDSWORK: The Windows version of `utime()` (aka `mingw_utime()`) does\nnot properly handle directories because it uses `_wopen()`. It should\nbe converted to using `CreateFileW()` and backup semantics at a minimum.\nSince I'm already in the middle of a large patch series, I did not want\nto destabilize other callers of `utime()` right now. The problem has\nonly been observed in the t/perf/p7519 test when the test repo contains\nan empty directory on disk.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-chmtime.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/t/helper/test-chmtime.c b/t/helper/test-chmtime.c\nindex 524b55ca496..dc28890a183 100644\n--- a/t/helper/test-chmtime.c\n+++ b/t/helper/test-chmtime.c\n@@ -134,6 +134,21 @@ int cmd__chmtime(int argc, const char **argv)\n \t\t}\n \n \t\tif (utb.modtime != sb.st_mtime && utime(argv[i], &utb) < 0) {\n+#ifdef GIT_WINDOWS_NATIVE\n+\t\t\tif (S_ISDIR(sb.st_mode)) {\n+\t\t\t\t/*\n+\t\t\t\t * NEEDSWORK: The Windows version of `utime()`\n+\t\t\t\t * (aka `mingw_utime()`) does not correctly\n+\t\t\t\t * handle directory arguments, since it uses\n+\t\t\t\t * `_wopen()`. Ignore it for now since this\n+\t\t\t\t * is just a test.\n+\t\t\t\t */\n+\t\t\t\tfprintf(stderr,\n+\t\t\t\t\t(\"Failed to modify time on directory %s. \"\n+\t\t\t\t\t \"Skipping\\n\"), argv[i]);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+#endif\n \t\t\tfprintf(stderr, \"Failed to modify time on %s: %s\\n\",\n \t\t\t argv[i], strerror(errno));\n \t\t\treturn 1;\n-- \ngitgitgadget\n\n"},{"id":"448274","messageId":"19993c130d22b1f5f7beaee1a23ac2d6b377fec3.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 26/30] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:15Z","receivedAt":"2022-02-11T20:57:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to periodically truncate the list of\nmodified files to save some memory.\n\nClients will ask for the set of changes relative to a token that they\nfound in the FSMN index extension in the index. (This token is like a\npoint in time, but different). Clients will then update the index to\ncontain the response token (so that subsequent commands will be\nrelative to this new token).\n\nTherefore, the daemon can gradually truncate the in-memory list of\nchanged paths as they become obsolete (older than the previous token).\nSince we may have multiple clients making concurrent requests with a\nskew of tokens and clients may be racing to the talk to the daemon,\nwe lazily truncate the list.\n\nWe introduce a 5 minute delay and truncate batches 5 minutes after\nthey are considered obsolete.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 88 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 88 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 4c88171e06c..962b24569e1 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -299,6 +299,75 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n \t\t\tbatch_src->interned_paths[k];\n }\n \n+/*\n+ * To keep the batch list from growing unbounded in response to filesystem\n+ * activity, we try to truncate old batches from the end of the list as\n+ * they become irrelevant.\n+ *\n+ * We assume that the .git/index will be updated with the most recent token\n+ * any time the index is updated. And future commands will only ask for\n+ * recent changes *since* that new token. So as tokens advance into the\n+ * future, older batch items will never be requested/needed. So we can\n+ * truncate them without loss of functionality.\n+ *\n+ * However, multiple commands may be talking to the daemon concurrently\n+ * or perform a slow command, so a little \"token skew\" is possible.\n+ * Therefore, we want this to be a little bit lazy and have a generous\n+ * delay.\n+ *\n+ * The current reader thread walked backwards in time from `token->batch_head`\n+ * back to `batch_marker` somewhere in the middle of the batch list.\n+ *\n+ * Let's walk backwards in time from that marker an arbitrary delay\n+ * and truncate the list there. Note that these timestamps are completely\n+ * artificial (based on when we pinned the batch item) and not on any\n+ * filesystem activity.\n+ *\n+ * Return the obsolete portion of the list after we have removed it from\n+ * the official list so that the caller can free it after leaving the lock.\n+ */\n+#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */\n+\n+static struct fsmonitor_batch *with_lock__truncate_old_batches(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst struct fsmonitor_batch *batch_marker)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder;\n+\n+\tif (!batch_marker)\n+\t\treturn NULL;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Truncate: mark (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t batch_marker->batch_seq_nr,\n+\t\t\t (uint64_t)batch_marker->pinned_time);\n+\n+\tfor (batch = batch_marker; batch; batch = batch->next) {\n+\t\ttime_t t;\n+\n+\t\tif (!batch->pinned_time) /* an overflow batch */\n+\t\t\tcontinue;\n+\n+\t\tt = batch->pinned_time + MY_TIME_DELAY_SECONDS;\n+\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n+\t\t\tcontinue;\n+\n+\t\tgoto truncate_past_here;\n+\t}\n+\n+\treturn NULL;\n+\n+truncate_past_here:\n+\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n+\n+\tremainder = ((struct fsmonitor_batch *)batch)->next;\n+\t((struct fsmonitor_batch *)batch)->next = NULL;\n+\n+\treturn remainder;\n+}\n+\n static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n {\n \tif (!token)\n@@ -412,6 +481,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tconst char *p;\n \tconst struct fsmonitor_batch *batch_head;\n \tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder = NULL;\n \tintmax_t count = 0, duplicates = 0;\n \tkh_str_t *shown;\n \tint hash_ret;\n@@ -641,11 +711,29 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * that work.\n \t\t\t */\n \t\t\tfsmonitor_free_token_data(token_data);\n+\t\t} else if (batch) {\n+\t\t\t/*\n+\t\t\t * We are holding the lock and are the only\n+\t\t\t * reader of the ref-counted portion of the\n+\t\t\t * list, so we get the honor of seeing if the\n+\t\t\t * list can be truncated to save memory.\n+\t\t\t *\n+\t\t\t * The main loop did not walk to the end of the\n+\t\t\t * list, so this batch is the first item in the\n+\t\t\t * batch-list that is older than the requested\n+\t\t\t * end-point sequence number. See if the tail\n+\t\t\t * end of the list is obsolete.\n+\t\t\t */\n+\t\t\tremainder = with_lock__truncate_old_batches(state,\n+\t\t\t\t\t\t\t\t batch);\n \t\t}\n \t}\n \n \tpthread_mutex_unlock(&state->main_lock);\n \n+\tif (remainder)\n+\t\tfsmonitor_batch__free_list(remainder);\n+\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n-- \ngitgitgadget\n\n"},{"id":"448275","messageId":"f47a763dc260c57c6f411cde7b4b8826732a8c7a.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 27/30] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:16Z","receivedAt":"2022-02-11T20:57:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon client threads to create a cookie file\ninside the .git directory and then wait until FS events for the\ncookie are observed by the FS listener thread.\n\nThis helps address the racy nature of file system events by\nblocking the client response until the kernel has drained any\nevent backlog.\n\nThis is especially important on MacOS where kernel events are\nonly issued with a limited frequency. See the `latency` argument\nof `FSeventStreamCreate()`. The kernel only signals every `latency`\nseconds, but does not guarantee that the kernel queue is completely\ndrained, so we may have to wait more than one interval. If we\nincrease the frequency, the system is more likely to drop events.\nWe avoid these issues by having each client thread create a unique\ncookie file and then wait until it is seen in the event stream.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 5 +\n 2 files changed, 232 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 962b24569e1..6011fe42ee0 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -94,6 +94,149 @@ static int do_as_client__status(void)\n \t}\n }\n \n+enum fsmonitor_cookie_item_result {\n+\tFCIR_ERROR = -1, /* could not create cookie file ? */\n+\tFCIR_INIT = 0,\n+\tFCIR_SEEN,\n+\tFCIR_ABORT,\n+};\n+\n+struct fsmonitor_cookie_item {\n+\tstruct hashmap_entry entry;\n+\tconst char *name;\n+\tenum fsmonitor_cookie_item_result result;\n+};\n+\n+static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n+\t\t const struct hashmap_entry *he2, const void *keydata)\n+{\n+\tconst struct fsmonitor_cookie_item *a =\n+\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n+\tconst struct fsmonitor_cookie_item *b =\n+\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n+\n+\treturn strcmp(a->name, keydata ? keydata : b->name);\n+}\n+\n+static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(\n+\tstruct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint fd;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tstruct strbuf cookie_pathname = STRBUF_INIT;\n+\tstruct strbuf cookie_filename = STRBUF_INIT;\n+\tenum fsmonitor_cookie_item_result result;\n+\tint my_cookie_seq;\n+\n+\tCALLOC_ARRAY(cookie, 1);\n+\n+\tmy_cookie_seq = state->cookie_seq++;\n+\n+\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n+\n+\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n+\tstrbuf_addbuf(&cookie_pathname, &cookie_filename);\n+\n+\tcookie->name = strbuf_detach(&cookie_filename, NULL);\n+\tcookie->result = FCIR_INIT;\n+\thashmap_entry_init(&cookie->entry, strhash(cookie->name));\n+\n+\thashmap_add(&state->cookies, &cookie->entry);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n+\t\t\t cookie->name, cookie_pathname.buf);\n+\n+\t/*\n+\t * Create the cookie file on disk and then wait for a notification\n+\t * that the listener thread has seen it.\n+\t */\n+\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n+\tif (fd >= 0) {\n+\t\tclose(fd);\n+\t\tunlink(cookie_pathname.buf);\n+\n+\t\t/*\n+\t\t * NEEDSWORK: This is an infinite wait (well, unless another\n+\t\t * thread sends us an abort). I'd like to change this to\n+\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n+\t\t * and let the caller do the trivial response thing.\n+\t\t */\n+\t\twhile (cookie->result == FCIR_INIT)\n+\t\t\tpthread_cond_wait(&state->cookies_cond,\n+\t\t\t\t\t &state->main_lock);\n+\t} else {\n+\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n+\t\t\t cookie->name);\n+\n+\t\tcookie->result = FCIR_ERROR;\n+\t}\n+\n+\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n+\n+\tresult = cookie->result;\n+\n+\tfree((char*)cookie->name);\n+\tfree(cookie);\n+\tstrbuf_release(&cookie_pathname);\n+\n+\treturn result;\n+}\n+\n+/*\n+ * Mark these cookies as _SEEN and wake up the corresponding client threads.\n+ */\n+static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,\n+\t\t\t\t\t const struct string_list *cookie_names)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint k;\n+\tint nr_seen = 0;\n+\n+\tfor (k = 0; k < cookie_names->nr; k++) {\n+\t\tstruct fsmonitor_cookie_item key;\n+\t\tstruct fsmonitor_cookie_item *cookie;\n+\n+\t\tkey.name = cookie_names->items[k].string;\n+\t\thashmap_entry_init(&key.entry, strhash(key.name));\n+\n+\t\tcookie = hashmap_get_entry(&state->cookies, &key, entry, NULL);\n+\t\tif (cookie) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-seen: '%s'\",\n+\t\t\t\t\t cookie->name);\n+\t\t\tcookie->result = FCIR_SEEN;\n+\t\t\tnr_seen++;\n+\t\t}\n+\t}\n+\n+\tif (nr_seen)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n+/*\n+ * Set _ABORT on all pending cookies and wake up all client threads.\n+ */\n+static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct hashmap_iter iter;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tint nr_aborted = 0;\n+\n+\thashmap_for_each_entry(&state->cookies, &iter, cookie, entry) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-abort: '%s'\",\n+\t\t\t\t cookie->name);\n+\t\tcookie->result = FCIR_ABORT;\n+\t\tnr_aborted++;\n+\t}\n+\n+\tif (nr_aborted)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n /*\n * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n * \"token\" as a virtual timestamp. Clients can request a summary of all\n@@ -391,6 +534,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n *\n+ * [2] Some of those lost events may have been for cookie files. We\n+ * should assume the worst and abort them rather letting them starve.\n+ *\n * If there are no concurrent threads readering the current token data\n * series, we can free it now. Otherwise, let the last reader free\n * it.\n@@ -412,6 +558,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n \tstate->current_token_data = new_one;\n \n \tfsmonitor_free_token_data(free_me);\n+\n+\twith_lock__abort_all_cookies(state);\n }\n \n void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n@@ -487,6 +635,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tint hash_ret;\n \tint do_trivial = 0;\n \tint do_flush = 0;\n+\tint do_cookie = 0;\n+\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n@@ -547,6 +697,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * We have a V2 valid token:\n \t\t\t * \"builtin::\"\n \t\t\t */\n+\t\t\tdo_cookie = 1;\n \t\t}\n \t}\n \n@@ -555,6 +706,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tif (!state->current_token_data)\n \t\tBUG(\"fsmonitor state does not have a current token\");\n \n+\t/*\n+\t * Write a cookie file inside the directory being watched in\n+\t * an effort to flush out existing filesystem events that we\n+\t * actually care about. Suspend this client thread until we\n+\t * see the filesystem events for this cookie file.\n+\t *\n+\t * Creating the cookie lets us guarantee that our FS listener\n+\t * thread has drained the kernel queue and we are caught up\n+\t * with the kernel.\n+\t *\n+\t * If we cannot create the cookie (or otherwise guarantee that\n+\t * we are caught up), we send a trivial response. We have to\n+\t * assume that there might be some very, very recent activity\n+\t * on the FS still in flight.\n+\t */\n+\tif (do_cookie) {\n+\t\tcookie_result = with_lock__wait_for_cookie(state);\n+\t\tif (cookie_result != FCIR_SEEN) {\n+\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n+\t\t\t cookie_result);\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n \tif (do_flush)\n \t\twith_lock__do_force_resync(state);\n \n@@ -775,7 +950,9 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n-#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+#define FSMONITOR_DIR \"fsmonitor--daemon\"\n+#define FSMONITOR_COOKIE_DIR \"cookies\"\n+#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n \n enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n \tconst char *rel)\n@@ -928,6 +1105,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n+\tif (cookie_names->nr)\n+\t\twith_lock__mark_cookies_seen(state, cookie_names);\n+\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n@@ -1019,7 +1199,9 @@ static int fsmonitor_run_daemon(void)\n \n \tmemset(&state, 0, sizeof(state));\n \n+\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n+\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n@@ -1044,6 +1226,44 @@ static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n+\t/*\n+\t * We will write filesystem syncing cookie files into\n+\t * ///-.\n+\t *\n+\t * The extra layers of subdirectories here keep us from\n+\t * changing the mtime on \".git/\" or \".git/foo/\" when we create\n+\t * or delete cookie files.\n+\t *\n+\t * There have been problems with some IDEs that do a\n+\t * non-recursive watch of the \".git/\" directory and run a\n+\t * series of commands any time something happens.\n+\t *\n+\t * For example, if we place our cookie files directly in\n+\t * \".git/\" or \".git/foo/\" then a `git status` (or similar\n+\t * command) from the IDE will cause a cookie file to be\n+\t * created in one of those dirs. This causes the mtime of\n+\t * those dirs to change. This triggers the IDE's watch\n+\t * notification. This triggers the IDE to run those commands\n+\t * again. And the process repeats and the machine never goes\n+\t * idle.\n+\t *\n+\t * Adding the extra layers of subdirectories prevents the\n+\t * mtime of \".git/\" and \".git/foo\" from changing when a\n+\t * cookie file is created.\n+\t */\n+\tstrbuf_init(&state.path_cookie_prefix, 0);\n+\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1056,6 +1276,7 @@ static int fsmonitor_run_daemon(void)\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n+\tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n \n@@ -1063,6 +1284,11 @@ done:\n \n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n+\tstrbuf_release(&state.path_cookie_prefix);\n+\n+\t/*\n+\t * NEEDSWORK: Consider \"rm -rf /\"\n+\t */\n \n \treturn err;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 20a815d80f8..c16ef095688 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n \n \tstruct fsmonitor_token_data *current_token_data;\n \n+\tstruct strbuf path_cookie_prefix;\n+\tpthread_cond_t cookies_cond;\n+\tint cookie_seq;\n+\tstruct hashmap cookies;\n+\n \tint error_code;\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n-- \ngitgitgadget\n\n"},{"id":"448276","messageId":"aec44a21afda7f058aca274dcb93c1eec442784a.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 28/30] fsmonitor: force update index after large responses","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:17Z","receivedAt":"2022-02-11T20:57:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMeasure the time taken to apply the FSMonitor query result\nto the index and the untracked-cache.\n\nSet the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when\nFSMonitor returns a very large repsonse to ensure that the index is\nwritten to disk.\n\nNormally, when the FSMonitor response includes a tracked file, the\nindex is always updated. Similarly, the index might be updated when\nthe response alters the untracked-cache (when enabled). However, in\ncases where neither of those cause the index to be considered changed,\nthe FSMonitor response is wasted. Subsequent Git commands will make\nrequests with the same token and receive the same response.\n\nIf that response is very large, performance may suffer. It would be\nmore efficient to force update the index now (and the token in the\nindex extension) in order to reduce the size of the response received\nby future commands.\n\nThis was observed on Windows after a large checkout. On Windows, the\nkernel emits events for the files that are changed as they are\nchanged. However, it might delay events for the containing\ndirectories until the system is more idle (or someone scans the\ndirectory (so it seems)). The first status following a checkout would\nget the list of files. The subsequent status commands would get the\nlist of directories as the events trickled out. But they would never\ncatch up because the token was not advanced because the index wasn't\nupdated.\n\nThis list of directories caused `wt_status_collect_untracked()` to\nunnecessarily spend time actually scanning them during each command.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 56 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 4287aad6bbb..8e3499d0667 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -219,6 +219,45 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n+/*\n+ * The number of pathnames that we need to receive from FSMonitor\n+ * before we force the index to be updated.\n+ *\n+ * Note that any pathname within the set of received paths MAY cause\n+ * cache-entry or istate flag bits to be updated and thus cause the\n+ * index to be updated on disk.\n+ *\n+ * However, the response may contain many paths (such as ignored\n+ * paths) that will not update any flag bits. And thus not force the\n+ * index to be updated. (This is fine and normal.) It also means\n+ * that the token will not be updated in the FSMonitor index\n+ * extension. So the next Git command will find the same token in the\n+ * index, make the same token-relative request, and receive the same\n+ * response (plus any newly changed paths). If this response is large\n+ * (and continues to grow), performance could be impacted.\n+ *\n+ * For example, if the user runs a build and it writes 100K object\n+ * files but doesn't modify any source files, the index would not need\n+ * to be updated. The FSMonitor response (after the build and\n+ * relative to a pre-build token) might be 5MB. Each subsequent Git\n+ * command will receive that same 100K/5MB response until something\n+ * causes the index to be updated. And `refresh_fsmonitor()` will\n+ * have to iterate over those 100K paths each time.\n+ *\n+ * Performance could be improved if we optionally force update the\n+ * index after a very large response and get an updated token into\n+ * the FSMonitor index extension. This should allow subsequent\n+ * commands to get smaller and more current responses.\n+ *\n+ * The value chosen here does not need to be precise. The index\n+ * will be updated automatically the first time the user touches\n+ * a tracked file and causes a command like `git status` to\n+ * update an mtime to be updated and/or set a flag bit.\n+ *\n+ * NEEDSWORK: Does this need to be a config value?\n+ */\n+static int fsmonitor_force_update_threshold = 100;\n+\n void refresh_fsmonitor(struct index_state *istate)\n {\n \tstruct strbuf query_result = STRBUF_INIT;\n@@ -362,25 +401,39 @@ apply_results:\n \t * information and that we should consider everything\n \t * invalid. We call this a trivial response.\n \t */\n+\ttrace2_region_enter(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tif (query_success && !is_trivial) {\n \t\t/*\n \t\t * Mark all pathnames returned by the monitor as dirty.\n \t\t *\n \t\t * This updates both the cache-entries and the untracked-cache.\n \t\t */\n+\t\tint count = 0;\n+\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n \t\t\t\tcontinue;\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n \t\t\tbol = i + 1;\n+\t\t\tcount++;\n \t\t}\n-\t\tif (bol < query_result.len)\n+\t\tif (bol < query_result.len) {\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tcount++;\n+\t\t}\n \n \t\t/* Now mark the untracked cache for fsmonitor usage */\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n+\n+\t\tif (count > fsmonitor_force_update_threshold)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", istate->repo, \"apply_count\",\n+\t\t\t\t count);\n+\n \t} else {\n \t\t/*\n \t\t * We failed to get a response or received a trivial response,\n@@ -409,6 +462,8 @@ apply_results:\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 0;\n \t}\n+\ttrace2_region_leave(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tstrbuf_release(&query_result);\n \n \t/* Now that we've updated istate, save the last_update_token */\n-- \ngitgitgadget\n\n"},{"id":"448277","messageId":"198f47bda5a03460532581a14528df52a3b90a83.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 25/30] t/perf/p7519: add fsmonitor--daemon test cases","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:14Z","receivedAt":"2022-02-11T20:57:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRepeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\nthe \"Simple IPC\" interface.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 29 ++++++++++++++++++++++++++++-\n 1 file changed, 28 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 054fc8d5d1d..9a2288a622d 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -182,7 +182,10 @@ test_perf_w_drop_caches () {\n }\n \n test_fsmonitor_suite () {\n-\tif test -n \"$INTEGRATION_SCRIPT\"; then\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n+\tthen\n+\t\tDESC=\"builtin fsmonitor--daemon\"\n+\telif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n \t\tDESC=\"fsmonitor=disabled\"\n@@ -293,4 +296,28 @@ test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n \n+#\n+# Run a full set of perf tests using the built-in fsmonitor--daemon.\n+# It does not use the Hook API, so it has a different setup.\n+# Explicitly start the daemon here and before we start client commands\n+# so that we can later add custom tracing.\n+#\n+if test_have_prereq FSMONITOR_DAEMON\n+then\n+\tUSE_FSMONITOR_DAEMON=t\n+\n+\ttrace_start fsmonitor--daemon--server\n+\tgit fsmonitor--daemon start\n+\n+\ttrace_start fsmonitor--daemon--client\n+\n+\tgit config core.fsmonitor true\n+\tgit update-index --fsmonitor\n+\n+\ttest_fsmonitor_suite\n+\n+\tgit fsmonitor--daemon stop\n+\ttrace_stop\n+fi\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"448278","messageId":"d6039987df80c217cf798e59bc88cb46f5a22707.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 29/30] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:18Z","receivedAt":"2022-02-11T20:57:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate 2x2 test matrix with the untracked-cache and fsmonitor--daemon\nfeatures and a series of edits and verify that status output is\nidentical.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 93 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 5f7b8e54233..0ccbfb9616f 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -162,6 +162,8 @@ test_expect_success 'setup' '\n \t.gitignore\n \texpect*\n \tactual*\n+\tflush*\n+\ttrace*\n \tEOF\n \n \tgit -c core.fsmonitor=false add . &&\n@@ -508,4 +510,95 @@ test_expect_success 'cleanup worktrees' '\n \tstop_daemon_delete_repo wt-base\n '\n \n+# The next few tests perform arbitrary/contrived file operations and\n+# confirm that status is correct. That is, that the data (or lack of\n+# data) from fsmonitor doesn't cause incorrect results. And doesn't\n+# cause incorrect results when the untracked-cache is enabled.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\tgit update-index --test-untracked-cache\n+'\n+\n+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n+\ttest_unconfig core.fsmonitor &&\n+\tgit update-index --no-fsmonitor &&\n+\ttest_might_fail git fsmonitor--daemon stop\n+'\n+\n+matrix_clean_up_repo () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd\n+}\n+\n+matrix_try () {\n+\tuc=$1\n+\tfsm=$2\n+\tfn=$3\n+\n+\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n+\t\tmatrix_clean_up_repo &&\n+\t\t$fn &&\n+\t\tif test $uc = false && test $fsm = false\n+\t\tthen\n+\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n+\t\telse\n+\t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n+\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n+\t\tfi\n+\t'\n+}\n+\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+for uc_val in $uc_values\n+do\n+\tif test $uc_val = false\n+\tthen\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n+\t\t\tgit config core.untrackedcache false &&\n+\t\t\tgit update-index --no-untracked-cache\n+\t\t'\n+\telse\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n+\t\t\tgit config core.untrackedcache true &&\n+\t\t\tgit update-index --untracked-cache\n+\t\t'\n+\tfi\n+\n+\tfsm_values=\"false true\"\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tif test $fsm_val = false\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\telse\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n+\t\t\t\tgit config core.fsmonitor true &&\n+\t\t\t\tgit fsmonitor--daemon start &&\n+\t\t\t\tgit update-index --fsmonitor\n+\t\t\t'\n+\t\tfi\n+\n+\t\tmatrix_try $uc_val $fsm_val edit_files\n+\t\tmatrix_try $uc_val $fsm_val delete_files\n+\t\tmatrix_try $uc_val $fsm_val create_files\n+\t\tmatrix_try $uc_val $fsm_val rename_files\n+\t\tmatrix_try $uc_val $fsm_val file_to_directory\n+\t\tmatrix_try $uc_val $fsm_val directory_to_file\n+\n+\t\tif test $fsm_val = true\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\tfi\n+\tdone\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"448279","messageId":"5117fbdfc6391b80ef25eb7553af1a7a86042048.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 30/30] update-index: convert fsmonitor warnings to advise","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:19Z","receivedAt":"2022-02-11T20:57:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n builtin/update-index.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex fed24ea1fb6..f94a89d30c8 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1227,18 +1227,18 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n-\t\t\twarning(_(\"core.fsmonitor is unset; \"\n-\t\t\t\t\"set it if you really want to \"\n-\t\t\t\t\"enable fsmonitor\"));\n+\t\t\tadvise(_(\"core.fsmonitor is unset; \"\n+\t\t\t\t \"set it if you really want to \"\n+\t\t\t\t \"enable fsmonitor\"));\n \t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \t\tif (fsm_mode > FSMONITOR_MODE_DISABLED)\n-\t\t\twarning(_(\"core.fsmonitor is set; \"\n-\t\t\t\t\"remove it if you really want to \"\n-\t\t\t\t\"disable fsmonitor\"));\n+\t\t\tadvise(_(\"core.fsmonitor is set; \"\n+\t\t\t\t \"remove it if you really want to \"\n+\t\t\t\t \"disable fsmonitor\"));\n \t\tremove_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor disabled\"));\n \t}\n-- \ngitgitgadget\n"},{"id":"448280","messageId":"fcf843a0d42f3a6b1d226b42014835c3e410fc7d.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 24/30] t/perf/p7519: speed up test on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:13Z","receivedAt":"2022-02-11T20:57:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nChange p7519 to use `test_seq` and `xargs` rather than a `for` loop\nto touch thousands of files. This takes minutes off of test runs\non Windows because of process creation overhead.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 32 ++++++++++++++++++++------------\n 1 file changed, 20 insertions(+), 12 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex c8be58f3c76..054fc8d5d1d 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -72,7 +72,7 @@ then\n \tfi\n fi\n \n-trace_start() {\n+trace_start () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tname=\"$1\"\n@@ -91,13 +91,20 @@ trace_start() {\n \tfi\n }\n \n-trace_stop() {\n+trace_stop () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tunset GIT_TRACE2_PERF\n \tfi\n }\n \n+touch_files () {\n+\tn=$1\n+\td=\"$n\"_files\n+\n+\t(cd $d ; test_seq 1 $n | xargs touch )\n+}\n+\n test_expect_success \"one time repo setup\" '\n \t# set untrackedCache depending on the environment\n \tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n@@ -119,10 +126,11 @@ test_expect_success \"one time repo setup\" '\n \tfi &&\n \n \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n-\tfor i in $(test_seq 1 10); do touch 10_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n+\ttouch_files 1 &&\n+\ttouch_files 10 &&\n+\ttouch_files 100 &&\n+\ttouch_files 1000 &&\n+\ttouch_files 10000 &&\n \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n \tgit commit -qm \"Add files\" &&\n \n@@ -133,7 +141,7 @@ test_expect_success \"one time repo setup\" '\n \tfi\n '\n \n-setup_for_fsmonitor() {\n+setup_for_fsmonitor () {\n \t# set INTEGRATION_SCRIPT depending on the environment\n \tif test -n \"$INTEGRATION_PATH\"\n \tthen\n@@ -173,7 +181,7 @@ test_perf_w_drop_caches () {\n \ttest_perf \"$@\"\n }\n \n-test_fsmonitor_suite() {\n+test_fsmonitor_suite () {\n \tif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n@@ -199,15 +207,15 @@ test_fsmonitor_suite() {\n \n \t# Update the mtimes on upto 100k files to make status think\n \t# that they are dirty. For simplicity, omit any files with\n-\t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n-\t# Then fully backslash-quote the paths to capture any\n-\t# whitespace so that they pass thru xargs properly.\n+\t# LFs (i.e. anything that ls-files thinks it needs to dquote)\n+\t# and any files with whitespace so that they pass thru xargs\n+\t# properly.\n \t#\n \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n \t\tgit ls-files | \\\n \t\t\thead -100000 | \\\n \t\t\tgrep -v \\\" | \\\n-\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n+\t\t\tegrep -v \" .\" | \\\n \t\t\txargs test-tool chmtime -300 &&\n \t\tgit status\n \t'\n-- \ngitgitgadget\n\n"},{"id":"448281","messageId":"a3b881315faea153fb6f4839d7077caf6fc8899c.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 16/30] compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:05Z","receivedAt":"2022-02-11T20:57:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInclude MacOS system declarations to allow us to use FSEvent and\nCoreFoundation APIs. We need GCC and clang versions because of\ncompiler and header file conflicts.\n\nWhile it is quite possible to #include Apple's CoreServices.h when\ncompiling C source code with clang, trying to build it with GCC\ncurrently fails with this error:\n\nIn file included\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Security.h:42,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/CSIdentity.h:43,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/OSServices.h:29,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/IconsCore.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/LaunchServices.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n\n /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Authorization.h:193:7:\n error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n\nThe underlying reason is that GCC (rightfully) objects that an `enum`\nvalue such as `kAuthorizationExternalFormLength` is not a constant\n(because it is not, the preprocessor has no knowledge of it, only the\nactual C compiler does) and can therefore not be used to define the size\nof a C array.\n\nThis is a known problem and tracked in GCC's bug tracker:\nhttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nIn the meantime, let's not block things and go the slightly ugly route\nof declaring/defining the FSEvents constants, data structures and\nfunctions that we need, so that we can avoid above-mentioned issue.\n\nLet's do this _only_ for GCC, though, so that the CI/PR builds (which\nbuild both with clang and with GCC) can guarantee that we _are_ using\nthe correct data types.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 96 ++++++++++++++++++++++++++++\n 1 file changed, 96 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex c84e3344ab9..f424253d3eb 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -1,3 +1,99 @@\n+#if defined(__GNUC__)\n+/*\n+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n+ * with clang, but not with GCC as of time of writing.\n+ *\n+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details.\n+ */\n+typedef unsigned int FSEventStreamCreateFlags;\n+#define kFSEventStreamEventFlagNone 0x00000000\n+#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n+#define kFSEventStreamEventFlagUserDropped 0x00000002\n+#define kFSEventStreamEventFlagKernelDropped 0x00000004\n+#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n+#define kFSEventStreamEventFlagHistoryDone 0x00000010\n+#define kFSEventStreamEventFlagRootChanged 0x00000020\n+#define kFSEventStreamEventFlagMount 0x00000040\n+#define kFSEventStreamEventFlagUnmount 0x00000080\n+#define kFSEventStreamEventFlagItemCreated 0x00000100\n+#define kFSEventStreamEventFlagItemRemoved 0x00000200\n+#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n+#define kFSEventStreamEventFlagItemRenamed 0x00000800\n+#define kFSEventStreamEventFlagItemModified 0x00001000\n+#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n+#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n+#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n+#define kFSEventStreamEventFlagItemIsFile 0x00010000\n+#define kFSEventStreamEventFlagItemIsDir 0x00020000\n+#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n+#define kFSEventStreamEventFlagOwnEvent 0x00080000\n+#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n+#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+\n+typedef struct __FSEventStream *FSEventStreamRef;\n+typedef const FSEventStreamRef ConstFSEventStreamRef;\n+\n+typedef unsigned int CFStringEncoding;\n+#define kCFStringEncodingUTF8 0x08000100\n+\n+typedef const struct __CFString *CFStringRef;\n+typedef const struct __CFArray *CFArrayRef;\n+typedef const struct __CFRunLoop *CFRunLoopRef;\n+\n+struct FSEventStreamContext {\n+ long long version;\n+ void *cb_data, *retain, *release, *copy_description;\n+};\n+\n+typedef struct FSEventStreamContext FSEventStreamContext;\n+typedef unsigned int FSEventStreamEventFlags;\n+#define kFSEventStreamCreateFlagNoDefer 0x02\n+#define kFSEventStreamCreateFlagWatchRoot 0x04\n+#define kFSEventStreamCreateFlagFileEvents 0x10\n+\n+typedef unsigned long long FSEventStreamEventId;\n+#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL\n+\n+typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef,\n+\t\t\t\t void *context,\n+\t\t\t\t __SIZE_TYPE__ num_of_events,\n+\t\t\t\t void *event_paths,\n+\t\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t\t const FSEventStreamEventId event_ids[]);\n+typedef double CFTimeInterval;\n+FSEventStreamRef FSEventStreamCreate(void *allocator,\n+\t\t\t\t FSEventStreamCallback callback,\n+\t\t\t\t FSEventStreamContext *context,\n+\t\t\t\t CFArrayRef paths_to_watch,\n+\t\t\t\t FSEventStreamEventId since_when,\n+\t\t\t\t CFTimeInterval latency,\n+\t\t\t\t FSEventStreamCreateFlags flags);\n+CFStringRef CFStringCreateWithCString(void *allocator, const char *string,\n+\t\t\t\t CFStringEncoding encoding);\n+CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count,\n+\t\t\t void *callbacks);\n+void CFRunLoopRun(void);\n+void CFRunLoopStop(CFRunLoopRef run_loop);\n+CFRunLoopRef CFRunLoopGetCurrent(void);\n+extern CFStringRef kCFRunLoopDefaultMode;\n+void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream,\n+\t\t\t\t CFRunLoopRef run_loop,\n+\t\t\t\t CFStringRef run_loop_mode);\n+unsigned char FSEventStreamStart(FSEventStreamRef stream);\n+void FSEventStreamStop(FSEventStreamRef stream);\n+void FSEventStreamInvalidate(FSEventStreamRef stream);\n+void FSEventStreamRelease(FSEventStreamRef stream);\n+#else\n+/*\n+ * Let Apple's headers declare `isalnum()` first, before\n+ * Git's headers override it via a constant\n+ */\n+#include \n+#include \n+#include \n+#endif\n+\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n-- \ngitgitgadget\n\n"},{"id":"448282","messageId":"98c5adf8ca0112ebf729970a0f15302d55806bd2.1644612979.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v5 15/30] compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-11T20:56:04Z","receivedAt":"2022-02-11T20:57:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the win32 backend to register a watch on the working tree\nroot directory (recursively). Also watch the if it is\nnot inside the working tree. And to collect path change notifications\ninto batches and publish.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 565 ++++++++++++++++++++++++++++\n 1 file changed, 565 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 916cbea254f..c2d11acbc1e 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -2,20 +2,585 @@\n #include \"config.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+/*\n+ * The documentation of ReadDirectoryChangesW() states that the maximum\n+ * buffer size is 64K when the monitored directory is remote.\n+ *\n+ * Larger buffers may be used when the monitored directory is local and\n+ * will help us receive events faster from the kernel and avoid dropped\n+ * events.\n+ *\n+ * So we try to use a very large buffer and silently fallback to 64K if\n+ * we get an error.\n+ */\n+#define MAX_RDCW_BUF_FALLBACK (65536)\n+#define MAX_RDCW_BUF (65536 * 8)\n+\n+struct one_watch\n+{\n+\tchar buffer[MAX_RDCW_BUF];\n+\tDWORD buf_len;\n+\tDWORD count;\n+\n+\tstruct strbuf path;\n+\tHANDLE hDir;\n+\tHANDLE hEvent;\n+\tOVERLAPPED overlapped;\n+\n+\t/*\n+\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n+\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n+\t */\n+\tBOOL is_active;\n+};\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tstruct one_watch *watch_worktree;\n+\tstruct one_watch *watch_gitdir;\n+\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hListener[3]; /* we don't own these handles */\n+#define LISTENER_SHUTDOWN 0\n+#define LISTENER_HAVE_DATA_WORKTREE 1\n+#define LISTENER_HAVE_DATA_GITDIR 2\n+\tint nr_listener_handles;\n+};\n+\n+/*\n+ * Convert the WCHAR path from the notification into UTF8 and\n+ * then normalize it.\n+ */\n+static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+\t\t\t\t struct strbuf *normalized_path)\n+{\n+\tint reserve;\n+\tint len = 0;\n+\n+\tstrbuf_reset(normalized_path);\n+\tif (!info->FileNameLength)\n+\t\tgoto normalize;\n+\n+\t/*\n+\t * Pre-reserve enough space in the UTF8 buffer for\n+\t * each Unicode WCHAR character to be mapped into a\n+\t * sequence of 2 UTF8 characters. That should let us\n+\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n+\t */\n+\treserve = info->FileNameLength + 1;\n+\tstrbuf_grow(normalized_path, reserve);\n+\n+\tfor (;;) {\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n+\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t\t\t normalized_path->buf,\n+\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n+\t\t\t\t\t NULL, NULL);\n+\t\tif (len > 0)\n+\t\t\tgoto normalize;\n+\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n+\t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n+\t\t\t GetLastError(),\n+\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n+\t\t\t info->FileName);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tstrbuf_grow(normalized_path,\n+\t\t\t strbuf_avail(normalized_path) + reserve);\n+\t}\n+\n+normalize:\n+\tstrbuf_setlen(normalized_path, len);\n+\treturn strbuf_normalize_path(normalized_path);\n+}\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+}\n+\n+static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const char *path)\n+{\n+\tstruct one_watch *watch = NULL;\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\twchar_t wpath[MAX_PATH];\n+\n+\tif (xutftowcs_path(wpath, path) < 0) {\n+\t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n+\t\treturn NULL;\n+\t}\n+\n+\thDir = CreateFileW(wpath,\n+\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n+\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n+\t\t\t NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n+\t\t GetLastError(), path);\n+\t\treturn NULL;\n+\t}\n+\n+\tCALLOC_ARRAY(watch, 1);\n+\n+\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n+\n+\tstrbuf_init(&watch->path, 0);\n+\tstrbuf_addstr(&watch->path, path);\n+\n+\twatch->hDir = hDir;\n+\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\treturn watch;\n+}\n+\n+static void destroy_watch(struct one_watch *watch)\n+{\n+\tif (!watch)\n+\t\treturn;\n+\n+\tstrbuf_release(&watch->path);\n+\tif (watch->hDir != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hDir);\n+\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hEvent);\n+\n+\tfree(watch);\n+}\n+\n+static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+\t\t\t struct one_watch *watch)\n+{\n+\tDWORD dwNotifyFilter =\n+\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n+\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n+\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n+\t\tFILE_NOTIFY_CHANGE_SIZE |\n+\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n+\t\tFILE_NOTIFY_CHANGE_CREATION;\n+\n+\tResetEvent(watch->hEvent);\n+\n+\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n+\twatch->overlapped.hEvent = watch->hEvent;\n+\n+\t/*\n+\t * Queue an async call using Overlapped IO. This returns immediately.\n+\t * Our event handle will be signalled when the real result is available.\n+\t *\n+\t * The return value here just means that we successfully queued it.\n+\t * We won't know if the Read...() actually produces data until later.\n+\t */\n+\twatch->is_active = ReadDirectoryChangesW(\n+\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n+\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n+\n+\tif (watch->is_active)\n+\t\treturn 0;\n+\n+\terror(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static int recv_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD gle;\n+\n+\twatch->is_active = FALSE;\n+\n+\t/*\n+\t * The overlapped result is ready. If the Read...() was successful\n+\t * we finally receive the actual result into our buffer.\n+\t */\n+\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n+\t\t\t\tTRUE))\n+\t\treturn 0;\n+\n+\tgle = GetLastError();\n+\tif (gle == ERROR_INVALID_PARAMETER &&\n+\t /*\n+\t * The kernel throws an invalid parameter error when our\n+\t * buffer is too big and we are pointed at a remote\n+\t * directory (and possibly for other reasons). Quietly\n+\t * set it down and try again.\n+\t *\n+\t * See note about MAX_RDCW_BUF at the top.\n+\t */\n+\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n+\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n+\t\treturn -2;\n+\t}\n+\n+\t/*\n+\t * NEEDSWORK: If an external is deleted, the above\n+\t * returns an error. I'm not sure that there's anything that\n+\t * we can do here other than failing -- the /.git\n+\t * link file would be broken anyway. We might try to check\n+\t * for that and return a better error message, but I'm not\n+\t * sure it is worth it.\n+\t */\n+\n+\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, gle);\n+\treturn -1;\n+}\n+\n+static void cancel_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD count;\n+\n+\tif (!watch || !watch->is_active)\n+\t\treturn;\n+\n+\t/*\n+\t * The calls to ReadDirectoryChangesW() and GetOverlappedResult()\n+\t * form a \"pair\" (my term) where we queue an IO and promise to\n+\t * hang around and wait for the kernel to give us the result.\n+\t *\n+\t * If for some reason after we queue the IO, we have to quit\n+\t * or otherwise not stick around for the second half, we must\n+\t * tell the kernel to abort the IO. This prevents the kernel\n+\t * from writing to our buffer and/or signalling our event\n+\t * after we free them.\n+\t *\n+\t * (Ask me how much fun it was to track that one down).\n+\t */\n+\tCancelIoEx(watch->hDir, &watch->overlapped);\n+\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n+\twatch->is_active = FALSE;\n+}\n+\n+/*\n+ * Process filesystem events that happen anywhere (recursively) under the\n+ * root directory. For a normal working directory, this includes\n+ * both version controlled files and the contents of the .git/ directory.\n+ *\n+ * If /.git is a file, then we only see events for the file\n+ * itself.\n+ */\n+static int process_worktree_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_worktree;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tconst char *p = watch->buffer;\n+\n+\t/*\n+\t * If the kernel gets more events than will fit in the kernel\n+\t * buffer associated with our RDCW handle, it drops them and\n+\t * returns a count of zero.\n+\t *\n+\t * Yes, the call returns WITHOUT error and with length zero.\n+\t * This is the documented behavior. (My testing has confirmed\n+\t * that it also sets the last error to ERROR_NOTIFY_ENUM_DIR,\n+\t * but we do not rely on that since the function did not\n+\t * return an error and it is not documented.)\n+\t *\n+\t * (The \"overflow\" case is not ambiguous with the \"no data\" case\n+\t * because we did an INFINITE wait.)\n+\t *\n+\t * This means we have a gap in coverage. Tell the daemon layer\n+\t * to resync.\n+\t */\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\t}\n+\n+\t/*\n+\t * On Windows, `info` contains an \"array\" of paths that are\n+\t * relative to the root of whichever directory handle received\n+\t * the event.\n+\t */\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\t\t/* ignore everything inside of \"/.git/\" */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* queue normal pathname */\n+\t\t\tif (!batch)\n+\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tbatch = NULL;\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_SHUTDOWN;\n+}\n+\n+/*\n+ * Process filesystem events that happened anywhere (recursively) under the\n+ * external (such as non-primary worktrees or submodules).\n+ * We only care about cookie files that our client threads created here.\n+ *\n+ * Note that we DO NOT get filesystem events on the external \n+ * itself (it is not inside something that we are watching). In particular,\n+ * we do not get an event if the external is deleted.\n+ */\n+static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_gitdir;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *p = watch->buffer;\n+\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_GITDIR;\n+\t}\n+\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\tgoto skip_this_path;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, NULL, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_GITDIR;\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tDWORD dwWait;\n+\tint result;\n+\n+\tstate->error_code = 0;\n+\n+\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tif (data->watch_gitdir &&\n+\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tfor (;;) {\n+\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n+\t\t\t\t\t\tdata->hListener,\n+\t\t\t\t\t\tFALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_worktree);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_gitdir);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->error_code = -1;\n+\n+force_shutdown:\n+\t/*\n+\t * Tell the IPC thead pool to stop (which completes the await\n+\t * in the main thread (which will also signal this thread (if\n+\t * we are still alive))).\n+\t */\n+\tipc_server_stop_async(state->ipc_server_data);\n+\n+clean_shutdown:\n+\tcancel_rdcw_watch(data->watch_worktree);\n+\tcancel_rdcw_watch(data->watch_gitdir);\n }\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->watch_worktree = create_watch(state,\n+\t\t\t\t\t state->path_worktree_watch.buf);\n+\tif (!data->watch_worktree)\n+\t\tgoto failed;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->watch_gitdir = create_watch(state,\n+\t\t\t\t\t\t state->path_gitdir_watch.buf);\n+\t\tif (!data->watch_gitdir)\n+\t\t\tgoto failed;\n+\t}\n+\n+\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_listener_handles++;\n+\n+\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n+\t\tdata->watch_worktree->hEvent;\n+\tdata->nr_listener_handles++;\n+\n+\tif (data->watch_gitdir) {\n+\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n+\t\t\tdata->watch_gitdir->hEvent;\n+\t\tdata->nr_listener_handles++;\n+\t}\n+\n+\tstate->backend_data = data;\n+\treturn 0;\n+\n+failed:\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n-- \ngitgitgadget\n\n"},{"id":"448650","messageId":"xmqqk0dub8ex.fsf@gitster.g","threadId":"56518","inReplyTo":"fcf843a0d42f3a6b1d226b42014835c3e410fc7d.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 24/30] t/perf/p7519: speed up test on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-17T01:15:18Z","receivedAt":"2022-02-17T01:15:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> +touch_files () {\n> +\tn=$1\n> +\td=\"$n\"_files\n> +\n> +\t(cd $d ; test_seq 1 $n | xargs touch )\n> +}\n> +\n> test_expect_success \"one time repo setup\" '\n> \t# set untrackedCache depending on the environment\n> \tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n> @@ -119,10 +126,11 @@ test_expect_success \"one time repo setup\" '\n> \tfi &&\n> \n> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n> -\tfor i in $(test_seq 1 10); do touch 10_files/$i || return 1; done &&\n> -\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n> -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n> -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n> +\ttouch_files 1 &&\n\nThis causes touch_files to chdir to 1_files and run \"touch 1\" in\nthere, but because there is no such directory (we have 1_file/\ndirectory, but not 1_files/ directory), it would fail.\n\n> +\ttouch_files 10 &&\n> +\ttouch_files 100 &&\n> +\ttouch_files 1000 &&\n> +\ttouch_files 10000 &&\n\nApparently nobody has run this perf script recently since part #2\nwas posted.\n\n> \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n\nThe original introduced at bb7cc7e7 (t/perf/fsmonitor: separate one\ntime repo initialization, 2020-10-26) created an empty directory 1_file\nand without creating anything in it, ran \"git add\" on it.\n\nIf we are not doing anything to 1_file directory anyway, perhaps we\ncan get rid of it to avoid the breakage in \"make perf\"?\n\nIf we have a chance to reroll this series, we can squash in\nsomething like this, perhaps (it does not deserve to be a separate\nstep).\n\n--- >8 ---\nSubject: [PATCH] p7519: leave 1_file directory empty\n\nThe step \"t/perf/p7519: speed up test on Windows\" in the topic\nbuiltin-fsmonitor-part-2 (not in 'next' yet) attempts to create one\nfile in 1_files directory, but the original introduced at bb7cc7e7\n(t/perf/fsmonitor: separate one time repo initialization,\n2020-10-26):\n\n (1) created 1_file directory,\n\n (2) left the directory empty, and\n\n (3) a later test expected (and still expects) that there is nothing\n in the directory.\n\nRevert the behaviour back to what the original wanted to do.\n\nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 9a2288a622..a1c552129c 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -126,7 +126,7 @@ test_expect_success \"one time repo setup\" '\n \tfi &&\n \n \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n-\ttouch_files 1 &&\n+\t: 1_file directory should be left empty &&\n \ttouch_files 10 &&\n \ttouch_files 100 &&\n \ttouch_files 1000 &&\n-- \n2.35.1-193-g45fe28c951\n\n"},{"id":"448710","messageId":"nycvar.QRO.7.76.6.2202171655390.348@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 00/30] Builtin FSMonitor Part 2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-17T16:06:26Z","receivedAt":"2022-02-17T16:06:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> Here is V5 of Part 2 of my Builtin FSMonitor series. I apologize for the\n> delay since V4 that I submitted back in October. (Insert the usual $DayJob\n> excuse...)\n>\n> I have rebased this branch onto the current \"master\" branch.\n\nThank you for your tireless work on this. I do see that it requires a ton\nof effort on your part and just wanted to let you know that I appreciate\nit very much!\n\n> In this version I removed the core.useBuiltinFSMonitor config setting and\n> instead extended the existing core.fsmonitor.\n\nI am somewhat surprised that a reviewer suggested this, as it breaks the\ncommon paradigm we use to allow using several Git versions on the same\nworktree.\n\nImagine, for example, that you run a Git version that understands\n`core.fsmonitor=true` to imply the built-in FSMonitor, while you _also_\nuse an IDE that bundles a slightly older Git version that mistakes the\n`true` for meaning the executable `true` (which is not a FSMonitor at all,\nbut its exit code suggests that everything's fine and dandy). The result\nwould be that the IDE does not see _any_ updates anymore, but nothing\nwould suggest that anything is wrong.\n\nWe can probably warn users about this, and we can also work around the\nfact that Git for Windows already uses `core.useBuiltinFSMonitor`, but it\nmakes me somewhat uneasy nevertheless.\n\nThank you,\nDscho\n"},{"id":"448711","messageId":"nycvar.QRO.7.76.6.2202171710270.348@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"365964b766400654a9978092f8fe5cfaa525fd3b.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 02/30] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-17T16:13:25Z","receivedAt":"2022-02-17T16:13:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Create fsmonitor_ipc__*() client routines to spawn the built-in file\n> system monitor daemon and send it an IPC request using the `Simple\n> IPC` API.\n>\n> Stub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nThis looks very straight-forward, thanks to your design of the Simple IPC.\n\nJust one thing:\n\n> diff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\n> new file mode 100644\n> index 00000000000..01c8c25bf50\n> --- /dev/null\n> +++ b/fsmonitor-ipc.c\n> @@ -0,0 +1,171 @@\n> +#include \"cache.h\"\n> +#include \"fsmonitor.h\"\n> +#include \"simple-ipc.h\"\n> +#include \"fsmonitor-ipc.h\"\n> +#include \"run-command.h\"\n> +#include \"strbuf.h\"\n> +#include \"trace2.h\"\n> +\n> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> +\n> [...]\n> +\n> +#else\n> +\n> +/*\n> + * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n> + * platforms.\n> + */\n> +\n> +int fsmonitor_ipc__is_supported(void)\n> +{\n> +\treturn 0;\n> +}\n> +\n> +const char *fsmonitor_ipc__get_path(void)\n> +{\n> +\treturn NULL;\n> +}\n> +\n> +enum ipc_active_state fsmonitor_ipc__get_state(void)\n> +{\n> +\treturn IPC_STATE__OTHER_ERROR;\n> +}\n> +\n> +int fsmonitor_ipc__send_query(const char *since_token,\n> +\t\t\t struct strbuf *answer)\n> +{\n> +\treturn -1;\n> +}\n> +\n> +int fsmonitor_ipc__send_command(const char *command,\n> +\t\t\t\tstruct strbuf *answer)\n> +{\n> +\treturn -1;\n> +}\n> +\n> +#endif\n\nNot a big deal (read: I won't mind if you leave the code as-is), but I\nusually find it easier to read the shorter, more trivial arm of\nconditional if/else arms first. In this instance, the dummy implementation\nfor platforms that are not (yet) supported looks more trivial to me.\n\nThanks,\nDscho\n"},{"id":"448715","messageId":"nycvar.QRO.7.76.6.2202171725060.348@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"384516ce1a190d836324b2c2262d611090207357.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 03/30] fsmonitor: config settings are repository-specific","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-17T16:27:37Z","receivedAt":"2022-02-17T16:27:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Move fsmonitor config settings to a new and opaque\n> `struct fsmonitor_settings` structure. Add a lazily-loaded pointer\n> to this into `struct repo_settings`\n>\n> Create an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\n> represent the state of fsmonitor. This lets us represent which, if\n> any, fsmonitor provider (hook or IPC) is enabled.\n>\n> Create `fsm_settings__get_*()` getters to lazily look up fsmonitor-\n> related config settings.\n>\n> Get rid of the `core_fsmonitor` global variable. Move the code to\n> lookup the existing `core.fsmonitor` config value into the fsmonitor\n> settings.\n>\n> Create a hook pathname variable in `struct fsmonitor-settings` and\n> only set it when in hook mode.\n>\n> Extend the definition of `core.fsmonitor` to be either a boolean\n> or a hook pathname. When true, the builtin FSMonitor is used.\n> When false or unset, no FSMonitor (neither builtin nor hook) is\n> used.\n>\n> The existing `core_fsmonitor` global variable was used to store the\n> pathname to the fsmonitor hook *and* it was used as a boolean to see\n> if fsmonitor was enabled. This dual usage and global visibility leads\n> to confusion when we add the IPC-based provider. So lets hide the\n> details in fsmonitor-settings.c and let it decide which provider to\n> use in the case of multiple settings. This avoids cluttering up\n> repo-settings.c with these private details.\n>\n> A future commit in builtin-fsmonitor series will add the ability to\n> disqualify worktrees for various reasons, such as being mounted from a\n> remote volume, where fsmonitor should not be started. Having the\n> config settings hidden in fsmonitor-settings.c allows such worktree\n> restrictions to override the config values used.\n\nApart from my forward-compatibility concern regarding interpreting\n`core.fsmonitor` as a Boolean, this looks good. Just one thing:\n\n> diff --git a/fsmonitor.h b/fsmonitor.h\n> index f20d72631d7..f9201411aa7 100644\n> --- a/fsmonitor.h\n> +++ b/fsmonitor.h\n> @@ -3,6 +3,7 @@\n>\n> #include \"cache.h\"\n> #include \"dir.h\"\n> +#include \"fsmonitor-settings.h\"\n>\n> extern struct trace_key trace_fsmonitor;\n>\n> @@ -57,7 +58,11 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n> */\n> static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n> {\n> -\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n> +\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n\nI see this repeated a few times. Would it maybe make sense to change the\nsignature of the `fsm_settings__*()` functions to accept an index instead\nof a repository?\n\nCiao,\nDscho\n"},{"id":"448716","messageId":"nycvar.QRO.7.76.6.2202171744410.348@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"6fe5a2bc79eec132b6455d53196906036d0f4a80.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 10/30] fsmonitor--daemon: implement 'run' command","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-17T16:46:02Z","receivedAt":"2022-02-17T16:46:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> +static int try_to_run_foreground_daemon(void)\n> +{\n> +\t/*\n> +\t * Technically, we don't need to probe for an existing daemon\n> +\t * process, since we could just call `fsmonitor_run_daemon()`\n> +\t * and let it fail if the pipe/socket is busy.\n> +\t *\n> +\t * However, this method gives us a nicer error message for a\n> +\t * common error case.\n> +\t */\n> +\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n> +\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n> +\t\t the_repository->worktree);\n> +\n> +\tprintf(_(\"running fsmonitor-daemon in '%s'\\n\"),\n> +\t the_repository->worktree);\n> +\tfflush(stdout);\n\nDo we want to print this to `stderr` instead?\n\nCiao,\nDscho\n"},{"id":"448717","messageId":"nycvar.QRO.7.76.6.2202171748010.348@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"69fc0998286cbc791f199710a68a2028080e1632.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 11/30] fsmonitor--daemon: implement 'start' command","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-17T16:50:05Z","receivedAt":"2022-02-17T16:50:18Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> +static int try_to_start_background_daemon(void)\n> +{\n> +\tstruct child_process cp = CHILD_PROCESS_INIT;\n> +\tenum start_bg_result sbgr;\n> +\n> +\t/*\n> +\t * Before we try to create a background daemon process, see\n> +\t * if a daemon process is already listening. This makes it\n> +\t * easier for us to report an already-listening error to the\n> +\t * console, since our spawn/daemon can only report the success\n> +\t * of creating the background process (and not whether it\n> +\t * immediately exited).\n> +\t */\n> +\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n> +\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n> +\t\t the_repository->worktree);\n> +\n> +\tprintf(_(\"starting fsmonitor-daemon in '%s'\\n\"),\n> +\t the_repository->worktree);\n> +\tfflush(stdout);\n\nJust like for the patch before, my question whether `stdout` or `stderr`\nis preferable here?\n\n> +\tcp.git_cmd = 1;\n> +\n> +\tstrvec_push(&cp.args, \"fsmonitor--daemon\");\n> +\tstrvec_push(&cp.args, \"run\");\n> +\tstrvec_push(&cp.args, \"--free-console\");\n\nWe could call this function `--detached` or `--detach`, too, to indicate\nthe intention.\n\nI am fine with the code as-is, just wanted to make sure that these\nquestions are on record ;-)\n\nCiao,\nDscho\n"},{"id":"448726","messageId":"66663b27-952e-5072-3123-6a091c56d315@jeffhostetler.com","threadId":"56518","inReplyTo":"xmqqk0dub8ex.fsf@gitster.g","subject":"Re: [PATCH v5 24/30] t/perf/p7519: speed up test on Windows","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-02-17T19:03:53Z","receivedAt":"2022-02-17T19:03:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/16/22 8:15 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> +touch_files () {\n>> +\tn=$1\n>> +\td=\"$n\"_files\n>> +\n>> +\t(cd $d ; test_seq 1 $n | xargs touch )\n>> +}\n>> +\n>> test_expect_success \"one time repo setup\" '\n>> \t# set untrackedCache depending on the environment\n>> \tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n>> @@ -119,10 +126,11 @@ test_expect_success \"one time repo setup\" '\n>> \tfi &&\n>> \n>> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n>> -\tfor i in $(test_seq 1 10); do touch 10_files/$i || return 1; done &&\n>> -\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n>> -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n>> -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n>> +\ttouch_files 1 &&\n> \n> This causes touch_files to chdir to 1_files and run \"touch 1\" in\n> there, but because there is no such directory (we have 1_file/\n> directory, but not 1_files/ directory), it would fail.\n> \n>> +\ttouch_files 10 &&\n>> +\ttouch_files 100 &&\n>> +\ttouch_files 1000 &&\n>> +\ttouch_files 10000 &&\n> \n> Apparently nobody has run this perf script recently since part #2\n> was posted.\n> \n>> \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n> \n> The original introduced at bb7cc7e7 (t/perf/fsmonitor: separate one\n> time repo initialization, 2020-10-26) created an empty directory 1_file\n> and without creating anything in it, ran \"git add\" on it.\n> \n> If we are not doing anything to 1_file directory anyway, perhaps we\n> can get rid of it to avoid the breakage in \"make perf\"?\n> \n> If we have a chance to reroll this series, we can squash in\n> something like this, perhaps (it does not deserve to be a separate\n> step).\n\nGood catch!\n\nIt looks to me like there was an oversight/typo in the original\n89afd5f5ad (t/perf: add fsmonitor perf test for git diff, 2020-10-20).\nThey created the \"1_file\" directory and didn't put anything in it.\nThen later when they test it, they say \"0_files\" in the test name\nand \"1_file\" in the \"git diff\" command.\n\nI don't think it's worth keeping an empty directory here, since there\nwon't be anything in the index after the add and since the directory\nis empty the untracked cache won't have anything to scan.\n\nMy version (without the typo) would have created 1 file in the directory\nbut I don't think that's worth keeping either, since we create thousands\nof files in steps right after it.\n\nI'll make a note to remove it.\n\nJeff\n\n\n\n> \n> --- >8 ---\n> Subject: [PATCH] p7519: leave 1_file directory empty\n> \n> The step \"t/perf/p7519: speed up test on Windows\" in the topic\n> builtin-fsmonitor-part-2 (not in 'next' yet) attempts to create one\n> file in 1_files directory, but the original introduced at bb7cc7e7\n> (t/perf/fsmonitor: separate one time repo initialization,\n> 2020-10-26):\n> \n> (1) created 1_file directory,\n> \n> (2) left the directory empty, and\n> \n> (3) a later test expected (and still expects) that there is nothing\n> in the directory.\n> \n> Revert the behaviour back to what the original wanted to do.\n> \n> Signed-off-by: Junio C Hamano \n> ---\n> t/perf/p7519-fsmonitor.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n> index 9a2288a622..a1c552129c 100755\n> --- a/t/perf/p7519-fsmonitor.sh\n> +++ b/t/perf/p7519-fsmonitor.sh\n> @@ -126,7 +126,7 @@ test_expect_success \"one time repo setup\" '\n> \tfi &&\n> \n> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n> -\ttouch_files 1 &&\n> +\t: 1_file directory should be left empty &&\n> \ttouch_files 10 &&\n> \ttouch_files 100 &&\n> \ttouch_files 1000 &&\n> \n"},{"id":"448728","messageId":"456172c1-9648-c01b-c747-c6d77ae570bf@jeffhostetler.com","threadId":"56518","inReplyTo":"nycvar.QRO.7.76.6.2202171725060.348@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v5 03/30] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-02-17T19:21:05Z","receivedAt":"2022-02-17T19:21:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/17/22 11:27 AM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> \n> On Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Move fsmonitor config settings to a new and opaque\n>> `struct fsmonitor_settings` structure. Add a lazily-loaded pointer\n>> to this into `struct repo_settings`\n>>\n>> Create an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\n>> represent the state of fsmonitor. This lets us represent which, if\n>> any, fsmonitor provider (hook or IPC) is enabled.\n>>\n>> Create `fsm_settings__get_*()` getters to lazily look up fsmonitor-\n>> related config settings.\n>>\n>> Get rid of the `core_fsmonitor` global variable. Move the code to\n>> lookup the existing `core.fsmonitor` config value into the fsmonitor\n>> settings.\n>>\n>> Create a hook pathname variable in `struct fsmonitor-settings` and\n>> only set it when in hook mode.\n>>\n>> Extend the definition of `core.fsmonitor` to be either a boolean\n>> or a hook pathname. When true, the builtin FSMonitor is used.\n>> When false or unset, no FSMonitor (neither builtin nor hook) is\n>> used.\n>>\n>> The existing `core_fsmonitor` global variable was used to store the\n>> pathname to the fsmonitor hook *and* it was used as a boolean to see\n>> if fsmonitor was enabled. This dual usage and global visibility leads\n>> to confusion when we add the IPC-based provider. So lets hide the\n>> details in fsmonitor-settings.c and let it decide which provider to\n>> use in the case of multiple settings. This avoids cluttering up\n>> repo-settings.c with these private details.\n>>\n>> A future commit in builtin-fsmonitor series will add the ability to\n>> disqualify worktrees for various reasons, such as being mounted from a\n>> remote volume, where fsmonitor should not be started. Having the\n>> config settings hidden in fsmonitor-settings.c allows such worktree\n>> restrictions to override the config values used.\n> \n> Apart from my forward-compatibility concern regarding interpreting\n> `core.fsmonitor` as a Boolean, this looks good. Just one thing:\n> \n>> diff --git a/fsmonitor.h b/fsmonitor.h\n>> index f20d72631d7..f9201411aa7 100644\n>> --- a/fsmonitor.h\n>> +++ b/fsmonitor.h\n>> @@ -3,6 +3,7 @@\n>>\n>> #include \"cache.h\"\n>> #include \"dir.h\"\n>> +#include \"fsmonitor-settings.h\"\n>>\n>> extern struct trace_key trace_fsmonitor;\n>>\n>> @@ -57,7 +58,11 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n>> */\n>> static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n>> {\n>> -\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n>> +\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n> \n> I see this repeated a few times. Would it maybe make sense to change the\n> signature of the `fsm_settings__*()` functions to accept an index instead\n> of a repository?\n\nI think is just me being paranoid -- testing istate->repo for null\nand assuming the_repository if necessary. I'm wondering if it is\nalways safe to just do\n\n\tfsm_mode = fsm_settings__get_mode(istate->repo);\n\n(or maybe put the null check inside the fsm_settings__*() functions.\n\n\nchanging the signature of those fsm_* functions seems wrong since they\nare associated with a repo and not an index.\n\nJeff\n\n\n"},{"id":"448729","messageId":"6e03d1e0-8765-a474-f69f-b1ce63146b6a@jeffhostetler.com","threadId":"56518","inReplyTo":"nycvar.QRO.7.76.6.2202171744410.348@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v5 10/30] fsmonitor--daemon: implement 'run' command","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-02-17T19:26:42Z","receivedAt":"2022-02-17T19:26:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/17/22 11:46 AM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> On Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> +static int try_to_run_foreground_daemon(void)\n>> +{\n>> +\t/*\n>> +\t * Technically, we don't need to probe for an existing daemon\n>> +\t * process, since we could just call `fsmonitor_run_daemon()`\n>> +\t * and let it fail if the pipe/socket is busy.\n>> +\t *\n>> +\t * However, this method gives us a nicer error message for a\n>> +\t * common error case.\n>> +\t */\n>> +\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n>> +\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n>> +\t\t the_repository->worktree);\n>> +\n>> +\tprintf(_(\"running fsmonitor-daemon in '%s'\\n\"),\n>> +\t the_repository->worktree);\n>> +\tfflush(stdout);\n> \n> Do we want to print this to `stderr` instead?\n\n\nYes. I have a patch in part 3 that moves it to stderr\nand only emits it if a config setting is present.\n\nfsmonitor--daemon: print start message only if fsmonitor.announceStartup\n\nWondering if I should squash that in here instead.\n\nJeff\n"},{"id":"448730","messageId":"xmqqpmnl8evc.fsf@gitster.g","threadId":"56518","inReplyTo":"nycvar.QRO.7.76.6.2202171655390.348@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v5 00/30] Builtin FSMonitor Part 2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-17T19:36:23Z","receivedAt":"2022-02-17T19:36:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> In this version I removed the core.useBuiltinFSMonitor config setting and\n>> instead extended the existing core.fsmonitor.\n>\n> I am somewhat surprised that a reviewer suggested this, as it breaks the\n> common paradigm we use to allow using several Git versions on the same\n> worktree.\n\nI do not think sharing the same repository with different versions\nof Git was considered as a possible source of problems during the\nreview discussion.\n\nhttps://lore.kernel.org/git/74282d08-aaeb-0a1e-cad3-1de17d59b4d1@jeffhostetler.com/\n\nI am not saying that we should not consider it; I am just stating\nthe fact that there was nobody who raised as a potential issue\nduring the discussion that lead to the cited message.\n"},{"id":"449186","messageId":"37f54cd9-3e53-7d38-2c23-2fc245dc1132@jeffhostetler.com","threadId":"56518","inReplyTo":"nycvar.QRO.7.76.6.2202171655390.348@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v5 00/30] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-02-22T18:53:49Z","receivedAt":"2022-02-22T18:53:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/17/22 11:06 AM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> On Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> Here is V5 of Part 2 of my Builtin FSMonitor series. I apologize for the\n>> delay since V4 that I submitted back in October. (Insert the usual $DayJob\n>> excuse...)\n>>\n>> I have rebased this branch onto the current \"master\" branch.\n> \n> Thank you for your tireless work on this. I do see that it requires a ton\n> of effort on your part and just wanted to let you know that I appreciate\n> it very much!\n> \n>> In this version I removed the core.useBuiltinFSMonitor config setting and\n>> instead extended the existing core.fsmonitor.\n> \n> I am somewhat surprised that a reviewer suggested this, as it breaks the\n> common paradigm we use to allow using several Git versions on the same\n> worktree.\n> \n> Imagine, for example, that you run a Git version that understands\n> `core.fsmonitor=true` to imply the built-in FSMonitor, while you _also_\n> use an IDE that bundles a slightly older Git version that mistakes the\n> `true` for meaning the executable `true` (which is not a FSMonitor at all,\n> but its exit code suggests that everything's fine and dandy). The result\n> would be that the IDE does not see _any_ updates anymore, but nothing\n> would suggest that anything is wrong.\n> \n> We can probably warn users about this, and we can also work around the\n> fact that Git for Windows already uses `core.useBuiltinFSMonitor`, but it\n> makes me somewhat uneasy nevertheless.\n> \n> Thank you,\n> Dscho\n> \n\nThis is a valid concern and I should have thought to mention it when\nthe suggestion came up on the list. Yes, extending `core.fsmonitor` to\ntake a boolean or a path could confuse older clients (like ones bundled\nwith an IDE, like VS).\n\nMy assumption was that since we shipped `core.useBuiltinFSMonitor`\nin GFW with an experimental label, that normal users would not be\nusing it at all and especially not from their IDEs, so it wouldn't\nmatter. And experimental features are just that -- experimental\nand subject to change.\n\nBut your point is valid -- if someone does have the odd hook called\n\"true\" or \"1\", they'll get an unexpected result.\n\nJeff\n\n\n\n"},{"id":"449440","messageId":"nycvar.QRO.7.76.6.2202241532560.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"21c099c5197168ffba5861d2b7a992fce84a7252.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 12/30] fsmonitor--daemon: add pathname classification","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T14:36:33Z","receivedAt":"2022-02-24T14:36:48Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n> +\tconst char *rel)\n> +{\n> +\tif (fspathncmp(rel, \".git\", 4))\n> +\t\treturn IS_WORKDIR_PATH;\n> +\trel += 4;\n> +\n> +\tif (!*rel)\n> +\t\treturn IS_DOT_GIT;\n> +\tif (*rel != '/')\n> +\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n> +\trel++;\n> +\n> +\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n> +\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n> +\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n> +\n> +\treturn IS_INSIDE_DOT_GIT;\n> +}\n> +\n> +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n> +\tconst char *rel)\n> +{\n> +\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n> +\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n> +\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n> +\n> +\treturn IS_INSIDE_GITDIR;\n\nAt first, I was puzzled why this is not `IS_INSIDE_DOT_GIT` as above, but\nthen...\n\n> [...]\n> diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\n> index 3009c1a83de..7bbb3a27a1c 100644\n> --- a/fsmonitor--daemon.h\n> +++ b/fsmonitor--daemon.h\n> @@ -30,5 +30,66 @@ struct fsmonitor_daemon_state {\n> \tstruct ipc_server_data *ipc_server_data;\n> };\n>\n> +/*\n> + * Pathname classifications.\n> + *\n> + * The daemon classifies the pathnames that it receives from file\n> + * system notification events into the following categories and uses\n> + * that to decide whether clients are told about them. (And to watch\n> + * for file system synchronization events.)\n> + *\n> + * The client should only care about paths within the working\n> + * directory proper (inside the working directory and not \".git\" nor\n> + * inside of \".git/\"). That is, the client has read the index and is\n> + * asking for a list of any paths in the working directory that have\n> + * been modified since the last token. The client does not care about\n> + * file system changes within the .git directory (such as new loose\n> + * objects or packfiles). So the client will only receive paths that\n> + * are classified as IS_WORKDIR_PATH.\n> + *\n> + * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n> + * exact \".git\" directory or GITDIR. If the daemon receives a delete\n\n... I read this, and it started to click.\n\nHowever, may I suggest to point out that \".git\" can also be a _file_, not\nonly a directory? Otherwise it would not make sense to distinguish between\nthe `.git` directory/file and `GITDIR`, I'd say.\n\nThe rest of the patch looks good to me.\n\nThanks!\nDscho\n"},{"id":"449446","messageId":"nycvar.QRO.7.76.6.2202241605140.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"98c5adf8ca0112ebf729970a0f15302d55806bd2.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 15/30] compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T15:10:54Z","receivedAt":"2022-02-24T15:11:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nmerely a comment:\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> +/*\n> + * Convert the WCHAR path from the notification into UTF8 and\n> + * then normalize it.\n> + */\n> +static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n> +\t\t\t\t struct strbuf *normalized_path)\n> +{\n> +\tint reserve;\n> +\tint len = 0;\n> +\n> +\tstrbuf_reset(normalized_path);\n> +\tif (!info->FileNameLength)\n> +\t\tgoto normalize;\n> +\n> +\t/*\n> +\t * Pre-reserve enough space in the UTF8 buffer for\n> +\t * each Unicode WCHAR character to be mapped into a\n> +\t * sequence of 2 UTF8 characters. That should let us\n> +\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n> +\t */\n> +\treserve = info->FileNameLength + 1;\n> +\tstrbuf_grow(normalized_path, reserve);\n> +\n> +\tfor (;;) {\n> +\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n> +\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n> +\t\t\t\t\t normalized_path->buf,\n> +\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n> +\t\t\t\t\t NULL, NULL);\n> +\t\tif (len > 0)\n> +\t\t\tgoto normalize;\n> +\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n> +\t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n> +\t\t\t GetLastError(),\n> +\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n> +\t\t\t info->FileName);\n> +\t\t\treturn -1;\n> +\t\t}\n> +\n> +\t\tstrbuf_grow(normalized_path,\n> +\t\t\t strbuf_avail(normalized_path) + reserve);\n> +\t}\n> +\n> +normalize:\n> +\tstrbuf_setlen(normalized_path, len);\n> +\treturn strbuf_normalize_path(normalized_path);\n> +}\n\nThere are Unicode pages that require quite a few more bytes per wide\ncharacter (IIRC it can blow up to six bytes), but it should be good enough\nfor now, and we can always revisit this easily enough at a later stage.\n\nI really like this patch, as it makes the complexities of handling two\nwatches and the complexities of handling overlapped (and asynchronous)\nresults look easy.\n\nThank you!\nDscho\n"},{"id":"449448","messageId":"nycvar.QRO.7.76.6.2202241612340.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"a3b881315faea153fb6f4839d7077caf6fc8899c.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 16/30] compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T15:13:45Z","receivedAt":"2022-02-24T15:13:59Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\njust a quick note:\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> [...]\n> The underlying reason is that GCC (rightfully) objects that an `enum`\n> value such as `kAuthorizationExternalFormLength` is not a constant\n> (because it is not, the preprocessor has no knowledge of it, only the\n> actual C compiler does) and can therefore not be used to define the size\n> of a C array.\n>\n> This is a known problem and tracked in GCC's bug tracker:\n> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nI just checked and the bug is still marked as \"NEW\". Therefore we still\nneed this patch.\n\nCiao,\nDscho\n"},{"id":"449449","messageId":"nycvar.QRO.7.76.6.2202241614040.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"162e357db7211e72c8a39e722e8432baf710c70d.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 17/30] compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T15:21:57Z","receivedAt":"2022-02-24T15:23:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\n> index f424253d3eb..2aefdc14d89 100644\n> --- a/compat/fsmonitor/fsm-listen-darwin.c\n> +++ b/compat/fsmonitor/fsm-listen-darwin.c\n> @@ -1,4 +1,4 @@\n> -#if defined(__GNUC__)\n> +#ifndef __clang__\n\nThis probably wants to be squashed into the previous patch, and while at\nit, the corresponding commit message should probably be adjusted:\n\n\tWe need GCC and clang versions because of compiler and header file\n\tconflicts.\n\nThis should probably say something along the lines \"This is only needed\nfor GCC, clang is fine\" instead.\n\n> +static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n> +{\n> +\tstruct strbuf msg = STRBUF_INIT;\n> +\n> +\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n> +\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n> +\tif (flag & kFSEventStreamEventFlagUserDropped)\n> +\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n> +\tif (flag & kFSEventStreamEventFlagKernelDropped)\n> +\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n> +\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n> +\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n> +\tif (flag & kFSEventStreamEventFlagHistoryDone)\n> +\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n> +\tif (flag & kFSEventStreamEventFlagRootChanged)\n> +\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n> +\tif (flag & kFSEventStreamEventFlagMount)\n> +\t\tstrbuf_addstr(&msg, \"Mount|\");\n> +\tif (flag & kFSEventStreamEventFlagUnmount)\n> +\t\tstrbuf_addstr(&msg, \"Unmount|\");\n> +\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n> +\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n> +\tif (flag & kFSEventStreamEventFlagItemCreated)\n> +\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n> +\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n> +\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n> +\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n> +\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsDir)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsFile)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n> +\tif (flag & kFSEventStreamEventFlagItemModified)\n> +\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n> +\tif (flag & kFSEventStreamEventFlagItemRemoved)\n> +\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n> +\tif (flag & kFSEventStreamEventFlagItemRenamed)\n> +\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n> +\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n> +\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n> +\tif (flag & kFSEventStreamEventFlagOwnEvent)\n> +\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n> +\tif (flag & kFSEventStreamEventFlagItemCloned)\n> +\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n\nI cannot think of any more elegant way to do this, either, but I wish\nthere was a way...\n\n> +/*\n> + * NEEDSWORK: Investigate the proper value for the `latency` argument\n> + * in the call to `FSEventStreamCreate()`. I'm not sure that this\n> + * needs to be a config setting or just something that we tune after\n> + * some testing.\n\nSince this was written, we had ample time to investigate the proper value,\nand I think you actually did. Therefore this comment is probably no longer\nneeded.\n\n> + *\n> + * With a latency of 0.1, I was seeing lots of dropped events during\n> + * the \"touch 100000\" files test within t/perf/p7519, but with a\n> + * latency of 0.001 I did not see any dropped events. So the\n> + * \"correct\" value may be somewhere in between.\n> + *\n> + * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n> + */\n\nThe rest of the patch looks good to me, and since we saw this code working\nnicely in practice, I have high confidence in it.\n\nCiao,\nDscho\n"},{"id":"449451","messageId":"nycvar.QRO.7.76.6.2202241627150.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"69fc0998286cbc791f199710a68a2028080e1632.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 11/30] fsmonitor--daemon: implement 'start' command","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T15:30:12Z","receivedAt":"2022-02-24T15:30:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nsomething I missed in my review, but which causes failures in `seen`\nbecause of the interplay with `ac/usage-string-fixups`:\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char *subcmd;\n> +\tint free_console = 0;\n>\n> \tstruct option options[] = {\n> +\t\tOPT_BOOL(0, \"free-console\", &free_console, N_(\"free console\")),\n> \t\tOPT_INTEGER(0, \"ipc-threads\",\n> \t\t\t &fsmonitor__ipc_threads,\n> \t\t\t N_(\"use ipc worker threads\")),\n> +\t\tOPT_INTEGER(0, \"start-timeout\",\n> +\t\t\t &fsmonitor__start_timeout_sec,\n> +\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n\nGit is about to be stricter about these option usage strings: they are no\nlonger allowed to start with an upper-case letter. This diff fixes it for me:\n\n-- snip --\nFrom: Johannes Schindelin \nDate: Thu, 24 Feb 2022 15:48:01 +0100\nSubject: [PATCH] fixup??? fsmonitor--daemon: implement 'start' command\n\nThere is a patch series in `seen` that errors out on option usage\nstrings starting with a capital letter. Let's avoid that preemptively.\n\nSigned-off-by: Johannes Schindelin \n---\n builtin/fsmonitor--daemon.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 591433e897df..775e4de5584d 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1504,7 +1504,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"use ipc worker threads\")),\n \t\tOPT_INTEGER(0, \"start-timeout\",\n \t\t\t &fsmonitor__start_timeout_sec,\n-\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n+\t\t\t N_(\"max seconds to wait for background daemon startup\")),\n\n \t\tOPT_END()\n \t};\n--\n2.35.1.windows.2\n-- snap --\n\nCould I ask you to squash this in before you send a new iteration?\n\nThank you,\nDscho\n\n\n> +\n> \t\tOPT_END()\n> \t};\n>\n"},{"id":"449452","messageId":"nycvar.QRO.7.76.6.2202241630450.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"6e03d1e0-8765-a474-f69f-b1ce63146b6a@jeffhostetler.com","subject":"Re: [PATCH v5 10/30] fsmonitor--daemon: implement 'run' command","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T15:31:04Z","receivedAt":"2022-02-24T15:31:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Thu, 17 Feb 2022, Jeff Hostetler wrote:\n\n> On 2/17/22 11:46 AM, Johannes Schindelin wrote:\n>\n> > On Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n> >\n> > > +static int try_to_run_foreground_daemon(void)\n> > > +{\n> > > +\t/*\n> > > +\t * Technically, we don't need to probe for an existing daemon\n> > > +\t * process, since we could just call `fsmonitor_run_daemon()`\n> > > +\t * and let it fail if the pipe/socket is busy.\n> > > +\t *\n> > > +\t * However, this method gives us a nicer error message for a\n> > > +\t * common error case.\n> > > +\t */\n> > > +\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n> > > +\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n> > > +\t\t the_repository->worktree);\n> > > +\n> > > +\tprintf(_(\"running fsmonitor-daemon in '%s'\\n\"),\n> > > +\t the_repository->worktree);\n> > > +\tfflush(stdout);\n> >\n> > Do we want to print this to `stderr` instead?\n>\n> Yes. I have a patch in part 3 that moves it to stderr\n> and only emits it if a config setting is present.\n>\n> fsmonitor--daemon: print start message only if fsmonitor.announceStartup\n>\n> Wondering if I should squash that in here instead.\n\nI would like that.\n\nThanks,\nDscho\n"},{"id":"449453","messageId":"nycvar.QRO.7.76.6.2202241638460.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"3517c4a3c1322f07252f63204d60ace9cbb6b1b0.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 19/30] help: include fsmonitor--daemon feature flag in version info","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T15:39:41Z","receivedAt":"2022-02-24T15:39:53Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Add the \"feature: fsmonitor--daemon\" message to the output of\n> `git version --build-options`.\n>\n> The builtin FSMonitor is only available on certain platforms and\n> even then only when certain Makefile flags are enabled, so print\n> a message in the verbose version output when it is available.\n>\n> This can be used by test scripts for prereq testing. Granted, tests\n> could just try `git fsmonitor--daemon status` and look for a 128 exit\n> code or grep for a \"not supported\" message on stderr, but this is\n> rather obscure.\n>\n> The main advantage is that the feature message will automatically\n> appear in bug reports and other support requests.\n\nIt was also used by Scalar before we started the journey to integrate it\ninto core Git ;-)\n\nMaybe include that, for history buffs like me?\n\nCiao,\nDscho\n"},{"id":"449454","messageId":"nycvar.QRO.7.76.6.2202241646430.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"xmqqpmnl8evc.fsf@gitster.g","subject":"Re: [PATCH v5 00/30] Builtin FSMonitor Part 2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T15:47:17Z","receivedAt":"2022-02-24T15:47:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 17 Feb 2022, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> >> In this version I removed the core.useBuiltinFSMonitor config setting and\n> >> instead extended the existing core.fsmonitor.\n> >\n> > I am somewhat surprised that a reviewer suggested this, as it breaks the\n> > common paradigm we use to allow using several Git versions on the same\n> > worktree.\n>\n> I do not think sharing the same repository with different versions\n> of Git was considered as a possible source of problems during the\n> review discussion.\n>\n> https://lore.kernel.org/git/74282d08-aaeb-0a1e-cad3-1de17d59b4d1@jeffhostetler.com/\n>\n> I am not saying that we should not consider it; I am just stating\n> the fact that there was nobody who raised as a potential issue\n> during the discussion that lead to the cited message.\n\nJust to make sure: I did not intend to insult anyone (and in hindsight I\nwish that I had made that clearer).\n\nCiao,\nDscho\n"},{"id":"449455","messageId":"nycvar.QRO.7.76.6.2202241647440.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"456172c1-9648-c01b-c747-c6d77ae570bf@jeffhostetler.com","subject":"Re: [PATCH v5 03/30] fsmonitor: config settings are repository-specific","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T15:50:07Z","receivedAt":"2022-02-24T15:50:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Thu, 17 Feb 2022, Jeff Hostetler wrote:\n\n> On 2/17/22 11:27 AM, Johannes Schindelin wrote:\n>\n> > On Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n> >\n> > > diff --git a/fsmonitor.h b/fsmonitor.h\n> > > index f20d72631d7..f9201411aa7 100644\n> > > --- a/fsmonitor.h\n> > > +++ b/fsmonitor.h\n> > > @@ -3,6 +3,7 @@\n> > >\n> > > #include \"cache.h\"\n> > > #include \"dir.h\"\n> > > +#include \"fsmonitor-settings.h\"\n> > >\n> > > extern struct trace_key trace_fsmonitor;\n> > >\n> > > @@ -57,7 +58,11 @@ int fsmonitor_is_trivial_response(const struct strbuf\n> > > *query_result);\n> > > */\n> > > static inline int is_fsmonitor_refreshed(const struct index_state\n> > > *istate)\n> > > {\n> > > -\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n> > > +\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n> >\n> > I see this repeated a few times. Would it maybe make sense to change\n> > the signature of the `fsm_settings__*()` functions to accept an index\n> > instead of a repository?\n>\n> I think is just me being paranoid -- testing istate->repo for null\n> and assuming the_repository if necessary.\n\nIndeed, this might be a bit too careful here. There is a pretty strong\nassumption built into the Git index that there is a repository, after all.\n\n> I'm wondering if it is always safe to just do\n>\n> \tfsm_mode = fsm_settings__get_mode(istate->repo);\n>\n> (or maybe put the null check inside the fsm_settings__*() functions.\n\nI would like to try. The test suite should give us enough confidence that\nwe have dashed all Ts and dotted all Is on the question whether\n`istate->repo` is always non-NULL.\n\n> changing the signature of those fsm_* functions seems wrong since they\n> are associated with a repo and not an index.\n\nTrue.\n\nThank you for the sanity check,\nDscho\n"},{"id":"449467","messageId":"nycvar.QRO.7.76.6.2202241650200.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"37f54cd9-3e53-7d38-2c23-2fc245dc1132@jeffhostetler.com","subject":"Re: [PATCH v5 00/30] Builtin FSMonitor Part 2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T16:22:10Z","receivedAt":"2022-02-24T16:27:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Tue, 22 Feb 2022, Jeff Hostetler wrote:\n\n> On 2/17/22 11:06 AM, Johannes Schindelin wrote:\n>\n> > On Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n> >\n> > > In this version I removed the core.useBuiltinFSMonitor config\n> > > setting and instead extended the existing core.fsmonitor.\n> >\n> > I am somewhat surprised that a reviewer suggested this, as it breaks\n> > the common paradigm we use to allow using several Git versions on the\n> > same worktree.\n> >\n> > Imagine, for example, that you run a Git version that understands\n> > `core.fsmonitor=true` to imply the built-in FSMonitor, while you\n> > _also_ use an IDE that bundles a slightly older Git version that\n> > mistakes the `true` for meaning the executable `true` (which is not a\n> > FSMonitor at all, but its exit code suggests that everything's fine\n> > and dandy). The result would be that the IDE does not see _any_\n> > updates anymore, but nothing would suggest that anything is wrong.\n> >\n> > We can probably warn users about this, and we can also work around the\n> > fact that Git for Windows already uses `core.useBuiltinFSMonitor`, but\n> > it makes me somewhat uneasy nevertheless.\n>\n> This is a valid concern and I should have thought to mention it when\n> the suggestion came up on the list. Yes, extending `core.fsmonitor` to\n> take a boolean or a path could confuse older clients (like ones bundled\n> with an IDE, like VS).\n>\n> My assumption was that since we shipped `core.useBuiltinFSMonitor`\n> in GFW with an experimental label, that normal users would not be\n> using it at all and especially not from their IDEs, so it wouldn't\n> matter. And experimental features are just that -- experimental\n> and subject to change.\n>\n> But your point is valid -- if someone does have the odd hook called\n> \"true\" or \"1\", they'll get an unexpected result.\n\nI wondered about that for a while, and put that to a test last night. I\nset `core.fsmonitor = true` and then modified a file and ran `git status`.\nSomething I did not expect happened: it picked up on the modified file!\n\nIt also printed out a warning:\n\n\twarning: Empty last update token.\n\nThis is the reason why it works: by default, current Git versions assume\nthat the FSMonitor hook understands the FSMonitor protocol v2, which\nstarts by the client sending out a token, receiving a new token and then\nthe paths of the files/directories/symlinks to inspect. Since the program\n`true` does _not_ write that token, Git warns that it did not receive a\ntoken and continues as if no FSMonitor had been configured.\n\nSo that's good news!\n\nThe less good news is that prior to v2.26.0, Git did not support v2 of the\nFSMonitor protocol, but only v1. And v1 does not expect such a token. Git\nversions between v2.16.0 and v2.26.0 will interpret a successful run of\nthe `true` executable with an empty output to mean that no files have been\nmodified.\n\nAnd indeed, in my tests, after making sure that the Git index had been\nrefreshed explicitly and then modifying a file and then running `git\nstatus` with v2.16.0, Git did not pick up on the modification.\n\nThat's the less good news.\n\nAt first I thought that we're pretty safe because nobody should use older\nGit versions and enable FSMonitor because FSMonitor protocol v1 is known\nto be subject to racy behavior. But then, Git users sometimes do not\ncompletely control which Git versions they use. Take for example Visual\nStudio users who also use the Git Bash to work on their worktree. While\ntheir Git Bash might be reasonably recent, Visual Studio comes with its\nown embedded Git version. Therefore, a user might want to play with the\nbuilt-in FSMonitor in Git Bash, find that it dramatically speeds up\neverything (as it does for me, thank you so much!), and not realize that\nthe Git executable used by Visual Studio totally misinterprets\n`core.fsmonitor` to refer to `/usr/bin/true.exe` and then miss any\nmodifications.\n\nAs long as the embedded Git version is at least v2.26.0, Visual Studio\nwill at least work correctly (because it ignores `true.exe`'s output and\ncontinue as if no FSMonitor had been configured). But as soon as an older\nversion is used, Git would work incorrectly, without any indication what\nis going wrong.\n\nI tried to come up with alternatives (because I _really_ dislike being a\nreviewer who only points out what's wrong without any constructive\nsuggestion how to do it better), and the best alternatives I came up were:\n\n- stick with `core.useBuiltinFSMonitor` as before, or\n\n- use a special value of `core.fsmonitor` that simply is not a valid\n executable name. In 2019, when I worked on the original precursor of the\n built-in FSMonitor (before I had to drop working on FSMonitor\n because of all the security work that went into v2.24.1), I had picked\n `:builtin:` because colons are illegal on Windows, but of _course_ they\n are legal everywhere else. But one thing is not possible, even on Linux:\n to have a trailing slash in an executable name. So something like\n `/builtin-fsmonitor/` would work.\n\nHowever, after seeing how nicely your latest iteration cleans up the code\nby simply interpreting a Boolean value to refer to the built-in FSMonitor,\nI _really_ would like to make it work.\n\nMaybe we can declare that it is \"safe enough\" to rely on new enough Git\nversions to be used by users who use multiple Git versions on the same\nworktree? They should use _at least_ v2.26.1 anyway, because that one\nfixed a rather important vulnerability (CVE-2020-5260)? At least for\nVisual Studio, this is true: it ships with Git version 2.33.0.windows.2.\n\nWhat do you think? Can we somehow make `core.fsmonitor = true` work?\n\nCiao,\nDscho\n"},{"id":"449473","messageId":"nycvar.QRO.7.76.6.2202241736560.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"19993c130d22b1f5f7beaee1a23ac2d6b377fec3.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 26/30] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T16:38:17Z","receivedAt":"2022-02-24T16:38:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Teach fsmonitor--daemon to periodically truncate the list of\n> modified files to save some memory.\n>\n> Clients will ask for the set of changes relative to a token that they\n> found in the FSMN index extension in the index. (This token is like a\n> point in time, but different). Clients will then update the index to\n> contain the response token (so that subsequent commands will be\n> relative to this new token).\n>\n> Therefore, the daemon can gradually truncate the in-memory list of\n> changed paths as they become obsolete (older than the previous token).\n> Since we may have multiple clients making concurrent requests with a\n> skew of tokens and clients may be racing to the talk to the daemon,\n> we lazily truncate the list.\n>\n> We introduce a 5 minute delay and truncate batches 5 minutes after\n> they are considered obsolete.\n\nI tried to poke holes into this strategy for a while, but failed whichever\nway I looked. Meaning: I am now convinced that this strategy is sound.\n\nThanks,\nDscho\n"},{"id":"449476","messageId":"nycvar.QRO.7.76.6.2202241742500.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"f47a763dc260c57c6f411cde7b4b8826732a8c7a.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 27/30] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T16:45:55Z","receivedAt":"2022-02-24T16:46:06Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Teach fsmonitor--daemon client threads to create a cookie file\n> inside the .git directory and then wait until FS events for the\n> cookie are observed by the FS listener thread.\n>\n> This helps address the racy nature of file system events by\n> blocking the client response until the kernel has drained any\n> event backlog.\n>\n> This is especially important on MacOS where kernel events are\n> only issued with a limited frequency. See the `latency` argument\n> of `FSeventStreamCreate()`. The kernel only signals every `latency`\n> seconds, but does not guarantee that the kernel queue is completely\n> drained, so we may have to wait more than one interval. If we\n> increase the frequency, the system is more likely to drop events.\n> We avoid these issues by having each client thread create a unique\n> cookie file and then wait until it is seen in the event stream.\n\nIt took a couple iterations of this cookie file business to become\nrobust... ;-)\n\nAbout these NEEDSWORKs:\n\n> +\t/*\n> +\t * Create the cookie file on disk and then wait for a notification\n> +\t * that the listener thread has seen it.\n> +\t */\n> +\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n> +\tif (fd >= 0) {\n> +\t\tclose(fd);\n> +\t\tunlink(cookie_pathname.buf);\n> +\n> +\t\t/*\n> +\t\t * NEEDSWORK: This is an infinite wait (well, unless another\n> +\t\t * thread sends us an abort). I'd like to change this to\n> +\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n> +\t\t * and let the caller do the trivial response thing.\n> +\t\t */\n> +\t\twhile (cookie->result == FCIR_INIT)\n> +\t\t\tpthread_cond_wait(&state->cookies_cond,\n> +\t\t\t\t\t &state->main_lock);\n\nIt would probably make sense to do this at some stage, but since we have\ncode that has seen quite a bit of real-world testing, I am in favor of\npostponing this change to a later date.\n\n> @@ -1063,6 +1284,11 @@ done:\n>\n> \tstrbuf_release(&state.path_worktree_watch);\n> \tstrbuf_release(&state.path_gitdir_watch);\n> +\tstrbuf_release(&state.path_cookie_prefix);\n> +\n> +\t/*\n> +\t * NEEDSWORK: Consider \"rm -rf /\"\n> +\t */\n>\n> \treturn err;\n> }\n\nIn this instance, I think we can just drop the `NEEDSWORK`: it makes sense\nto keep around these directories rather than destroying and re-creating\nthem over and over again.\n\nCiao,\nDscho\n"},{"id":"449480","messageId":"nycvar.QRO.7.76.6.2202241749340.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"aec44a21afda7f058aca274dcb93c1eec442784a.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 28/30] fsmonitor: force update index after large responses","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T16:51:08Z","receivedAt":"2022-02-24T16:51:28Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> [...]\n> + *\n> + * NEEDSWORK: Does this need to be a config value?\n> + */\n> +static int fsmonitor_force_update_threshold = 100;\n> +\n> void refresh_fsmonitor(struct index_state *istate)\n> {\n> \tstruct strbuf query_result = STRBUF_INIT;\n\nWe gave this ample opportunity to rise to a need, and no need arose, to\nintroduce a config option.\n\nTherefore I am in favor of simply dropping the comment.\n\nCiao,\nDscho\n"},{"id":"449482","messageId":"xmqqo82wnq14.fsf@gitster.g","threadId":"56518","inReplyTo":"nycvar.QRO.7.76.6.2202241646430.11118@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v5 00/30] Builtin FSMonitor Part 2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-24T17:16:39Z","receivedAt":"2022-02-24T17:16:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Just to make sure: I did not intend to insult anyone (and in hindsight I\n> wish that I had made that clearer).\n\nIt is OK that you are wiser in hindsight. We all are, and we try to\ndo better the next time ;-)\n\nThanks for reminding of the topic.\n\nAs a general principle, when introducing a new feature that achieves\nthe same goal in a new and improved way, it is safer to introduce it\nin such a way that users of older implementations that lack the\nfeature cannot choose it by mistake.\n\nOne way to do so is to we reuse the same configuration knob by\nadding a new settings value that older implementations would choke\non, the users will be forced to ensure that the older and proven way\nis used consistently everywhere, until every tool the user uses are\nready.\n\nFor this instance, I think it is OK to split and allow two to\noperate on the same data at the same time, because I believe that\nboth old and new implementation will leave a permanent difference to\nthe on-disk data that cannot later be reused by the other [*].\n\nBut it is an exception than a norm when adding a new thing that\nextends an existing feature (as opposed to inventing totally a new\nthing that won't overlap with any existing one). As a general\nprinciple, it is much safer to make sure it breaks (and have users\nhold off) when the new setting is given to an old implementation.\n\n * Side note. For example, if we introduce the index-v5 feature\n by not reusing the index.version but with index.usev5 variable,\n new implementations that know about the knob would write out v5\n data that existing implementations will not work with.\n\nAlso, from the point of view of the longer-term maintenance, of\ncourse not having to deal with orthogonal looking different\nconfiguration variables where newer ones override the older ones\nwill induce more pain over time.\n"},{"id":"449486","messageId":"4aa1293e-00b6-b9ef-efd4-cdf605db37a1@jeffhostetler.com","threadId":"56518","inReplyTo":"nycvar.QRO.7.76.6.2202241650200.11118@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v5 00/30] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-02-24T18:13:09Z","receivedAt":"2022-02-24T18:13:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 11:22 AM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> On Tue, 22 Feb 2022, Jeff Hostetler wrote:\n> \n>> On 2/17/22 11:06 AM, Johannes Schindelin wrote:\n>>\n>>> On Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n>>>\n>>>> In this version I removed the core.useBuiltinFSMonitor config\n>>>> setting and instead extended the existing core.fsmonitor.\n>>>\n>>> I am somewhat surprised that a reviewer suggested this, as it breaks\n>>> the common paradigm we use to allow using several Git versions on the\n>>> same worktree.\n>>>\n>>> Imagine, for example, that you run a Git version that understands\n>>> `core.fsmonitor=true` to imply the built-in FSMonitor, while you\n>>> _also_ use an IDE that bundles a slightly older Git version that\n>>> mistakes the `true` for meaning the executable `true` (which is not a\n>>> FSMonitor at all, but its exit code suggests that everything's fine\n>>> and dandy). The result would be that the IDE does not see _any_\n>>> updates anymore, but nothing would suggest that anything is wrong.\n>>>\n>>> We can probably warn users about this, and we can also work around the\n>>> fact that Git for Windows already uses `core.useBuiltinFSMonitor`, but\n>>> it makes me somewhat uneasy nevertheless.\n>>\n>> This is a valid concern and I should have thought to mention it when\n>> the suggestion came up on the list. Yes, extending `core.fsmonitor` to\n>> take a boolean or a path could confuse older clients (like ones bundled\n>> with an IDE, like VS).\n>>\n>> My assumption was that since we shipped `core.useBuiltinFSMonitor`\n>> in GFW with an experimental label, that normal users would not be\n>> using it at all and especially not from their IDEs, so it wouldn't\n>> matter. And experimental features are just that -- experimental\n>> and subject to change.\n>>\n>> But your point is valid -- if someone does have the odd hook called\n>> \"true\" or \"1\", they'll get an unexpected result.\n> \n> I wondered about that for a while, and put that to a test last night. I\n> set `core.fsmonitor = true` and then modified a file and ran `git status`.\n> Something I did not expect happened: it picked up on the modified file!\n> \n> It also printed out a warning:\n> \n> \twarning: Empty last update token.\n> \n> This is the reason why it works: by default, current Git versions assume\n> that the FSMonitor hook understands the FSMonitor protocol v2, which\n> starts by the client sending out a token, receiving a new token and then\n> the paths of the files/directories/symlinks to inspect. Since the program\n> `true` does _not_ write that token, Git warns that it did not receive a\n> token and continues as if no FSMonitor had been configured.\n> \n> So that's good news!\n> \n> The less good news is that prior to v2.26.0, Git did not support v2 of the\n> FSMonitor protocol, but only v1. And v1 does not expect such a token. Git\n> versions between v2.16.0 and v2.26.0 will interpret a successful run of\n> the `true` executable with an empty output to mean that no files have been\n> modified.\n> \n> And indeed, in my tests, after making sure that the Git index had been\n> refreshed explicitly and then modifying a file and then running `git\n> status` with v2.16.0, Git did not pick up on the modification.\n> \n> That's the less good news.\n> \n> At first I thought that we're pretty safe because nobody should use older\n> Git versions and enable FSMonitor because FSMonitor protocol v1 is known\n> to be subject to racy behavior. But then, Git users sometimes do not\n> completely control which Git versions they use. Take for example Visual\n> Studio users who also use the Git Bash to work on their worktree. While\n> their Git Bash might be reasonably recent, Visual Studio comes with its\n> own embedded Git version. Therefore, a user might want to play with the\n> built-in FSMonitor in Git Bash, find that it dramatically speeds up\n> everything (as it does for me, thank you so much!), and not realize that\n> the Git executable used by Visual Studio totally misinterprets\n> `core.fsmonitor` to refer to `/usr/bin/true.exe` and then miss any\n> modifications.\n> \n> As long as the embedded Git version is at least v2.26.0, Visual Studio\n> will at least work correctly (because it ignores `true.exe`'s output and\n> continue as if no FSMonitor had been configured). But as soon as an older\n> version is used, Git would work incorrectly, without any indication what\n> is going wrong.\n> \n> I tried to come up with alternatives (because I _really_ dislike being a\n> reviewer who only points out what's wrong without any constructive\n> suggestion how to do it better), and the best alternatives I came up were:\n> \n> - stick with `core.useBuiltinFSMonitor` as before, or\n> \n> - use a special value of `core.fsmonitor` that simply is not a valid\n> executable name. In 2019, when I worked on the original precursor of the\n> built-in FSMonitor (before I had to drop working on FSMonitor\n> because of all the security work that went into v2.24.1), I had picked\n> `:builtin:` because colons are illegal on Windows, but of _course_ they\n> are legal everywhere else. But one thing is not possible, even on Linux:\n> to have a trailing slash in an executable name. So something like\n> `/builtin-fsmonitor/` would work.\n> \n> However, after seeing how nicely your latest iteration cleans up the code\n> by simply interpreting a Boolean value to refer to the built-in FSMonitor,\n> I _really_ would like to make it work.\n> \n> Maybe we can declare that it is \"safe enough\" to rely on new enough Git\n> versions to be used by users who use multiple Git versions on the same\n> worktree? They should use _at least_ v2.26.1 anyway, because that one\n> fixed a rather important vulnerability (CVE-2020-5260)? At least for\n> Visual Studio, this is true: it ships with Git version 2.33.0.windows.2.\n> \n> What do you think? Can we somehow make `core.fsmonitor = true` work?\n> \n> Ciao,\n> Dscho\n> \n\nThanks for the testing and background information here!\n\nI agree. I would like to keep the current\n\n \"core.fsmonitor = | \"\n\nusage that I have in V5.\n\nIt cleaned up things very nicely and it got rid of the somewhat awkward\nusage of having \"core.useBuiltinFSMonitor\" override the existing\n\"core.fsmonitor\" setting.\n\nIt is unfortunate that it might cause a breakage for users who are\n*also* running a Git version between 2.16 ... 2.26. I have to wonder\nif it wouldn't be better to spend our energy documenting that users\nshould upgrade, rather than trying to support interop with them.\n\nThe 2.16 ... 2.26 versions don't have the security fix referenced\nabove.\n\nThe 2.16 ... 2.26 versions are based upon the V1 FSMonitor protocol\nand don't have fixes for several serious bugs or races in the original\nimplementation:\n\n* 398a3b0899 (fsmonitor: force a refresh after the index was discarded, \n2019-05-07)\n* 679f2f9fdd (unpack-trees: skip stat on fsmonitor-valid files, 2019-11-20)\n* dfaed02862 (fsmonitor: update documentation for hook version and \nwatchman hooks, 2020-01-23)\n\nThat last fix added the V2 FSMonitor protocol. This is used by both\nthe hook- and the IPC-based providers.\n\nSo anything still using the V1 FSMonitor protocol is going to unreliable\nand racy and users should not use it, so I don't think it is worth the \neffort to complicate our current solution to maintain compatibility.\n(I hate to say that, but they just shouldn't be using V1 any more.)\n\n\nOn a slight tangent, the current code (before my patch series) does\nsupport a \"core.fsmonitorhookversion\" to allow the client to talk to\na V1 or V2 provider explicitly (vs the default of trying V2 and then\ntrying V1). The IPC implementation does not use this config setting,\nbut I could see adding something to emit a warning if it was set to\n1 when using the builtin FSMonitor. This might help users who are\n*also* running a Git version between 2.26 and 2.35 to understand the\nfallback after the true.exe warning that Johannes described.\n\n\nOn another slight tangent, I'm wondering if we want to officially\ndeprecate the V1 hook code and/or remove support for it from the code.\n\n\nJeff\n"},{"id":"449495","messageId":"nycvar.QRO.7.76.6.2202242006020.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"4aa1293e-00b6-b9ef-efd4-cdf605db37a1@jeffhostetler.com","subject":"Re: [PATCH v5 00/30] Builtin FSMonitor Part 2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-24T19:16:54Z","receivedAt":"2022-02-24T19:17:11Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Thu, 24 Feb 2022, Jeff Hostetler wrote:\n\n> On 2/24/22 11:22 AM, Johannes Schindelin wrote:\n>\n> > On Tue, 22 Feb 2022, Jeff Hostetler wrote:\n> >\n> > > On 2/17/22 11:06 AM, Johannes Schindelin wrote:\n> > >\n> > > > On Fri, 11 Feb 2022, Jeff Hostetler via GitGitGadget wrote:\n> > > >\n> > > > > In this version I removed the core.useBuiltinFSMonitor config\n> > > > > setting and instead extended the existing core.fsmonitor.\n> > > >\n> > > > I am somewhat surprised that a reviewer suggested this, as it\n> > > > breaks the common paradigm we use to allow using several Git\n> > > > versions on the same worktree.\n> > [...]\n> >\n> > I wondered about that for a while, and put that to a test last night.\n> > I set `core.fsmonitor = true` and then modified a file and ran `git\n> > status`. Something I did not expect happened: it picked up on the\n> > modified file!\n> >\n> > [... describing how FSMonitor protocol v1 is affected...]\n> >\n> > However, after seeing how nicely your latest iteration cleans up the\n> > code by simply interpreting a Boolean value to refer to the built-in\n> > FSMonitor, I _really_ would like to make it work.\n> >\n> > Maybe we can declare that it is \"safe enough\" to rely on new enough\n> > Git versions to be used by users who use multiple Git versions on the\n> > same worktree? They should use _at least_ v2.26.1 anyway, because that\n> > one fixed a rather important vulnerability (CVE-2020-5260)? At least\n> > for Visual Studio, this is true: it ships with Git version\n> > 2.33.0.windows.2.\n> >\n> > What do you think? Can we somehow make `core.fsmonitor = true` work?\n>\n> [...]\n>\n> I agree. I would like to keep the current\n>\n> \"core.fsmonitor = | \"\n>\n> usage that I have in V5.\n>\n> It cleaned up things very nicely and it got rid of the somewhat awkward\n> usage of having \"core.useBuiltinFSMonitor\" override the existing\n> \"core.fsmonitor\" setting.\n\nYes!\n\n> It is unfortunate that it might cause a breakage for users who are\n> *also* running a Git version between 2.16 ... 2.26. I have to wonder\n> if it wouldn't be better to spend our energy documenting that users\n> should upgrade, rather than trying to support interop with them.\n\nThat's a good point: I guess if you added a comment to the documentation\nof `core.fsmonitor = true`, that should be good enough.\n\n> [...]\n>\n> So anything still using the V1 FSMonitor protocol is going to unreliable\n> and racy and users should not use it, so I don't think it is worth the effort\n> to complicate our current solution to maintain compatibility.\n> (I hate to say that, but they just shouldn't be using V1 any more.)\n\nThat's a really good point.\n\n> On a slight tangent, the current code (before my patch series) does\n> support a \"core.fsmonitorhookversion\" to allow the client to talk to\n> a V1 or V2 provider explicitly (vs the default of trying V2 and then\n> trying V1). The IPC implementation does not use this config setting,\n> but I could see adding something to emit a warning if it was set to\n> 1 when using the builtin FSMonitor. This might help users who are\n> *also* running a Git version between 2.26 and 2.35 to understand the\n> fallback after the true.exe warning that Johannes described.\n\nHow about making it an error instead? That should really be helpful: if\n`core.fsmonitor = true` and `core.fsmonitorHookVersion = 1`, just error\nout. That way, users will more likely fall into the pit of success.\n\n> On another slight tangent, I'm wondering if we want to officially\n> deprecate the V1 hook code and/or remove support for it from the code.\n\nOooh! That's _also_ a good point.\n\nMaybe we can keep this deprecation out of this here patch series, though.\nIt would be good to get this finished and into `next`, I think.\n\nI spent some time (in two separate thrusts) to review the entire patch\nseries, and hope that my feedback was useful to you.\n\nIn particular with your idea to document the incompatibilities of\n`core.fsmonitor = true` with Git v2.16.0..v2.26.0, I am really eager to\nsee (the next iteration of) this patch series advance to the `next`\nbranch, and then into an official Git version so that more users can\nbenefit from it.\n\nThank you for all your work on this,\nDscho\n"},{"id":"449639","messageId":"220225.865yp27e1c.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"bdd7334da3162ce77c216d61ce9d979f12637ac5.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 06/30] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-25T22:46:49Z","receivedAt":"2022-02-25T22:54:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Feb 11 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Create a built-in file system monitoring daemon that can be used by\n> the existing `fsmonitor` feature (protocol API and index extension)\n> to improve the performance of various Git commands, such as `status`.\n>\n> The `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\n> provides an alternative to hook access to existing fsmonitors such\n> as `watchman`.\n>\n> This commit merely adds the new command without any functionality.\n>\n> Co-authored-by: Johannes Schindelin \n> Signed-off-by: Jeff Hostetler \n> ---\n> .gitignore | 1 +\n> Makefile | 1 +\n> builtin.h | 1 +\n> builtin/fsmonitor--daemon.c | 46 +++++++++++++++++++++++++++++++++++++\n> git.c | 1 +\n> 5 files changed, 50 insertions(+)\n> create mode 100644 builtin/fsmonitor--daemon.c\n>\n> diff --git a/.gitignore b/.gitignore\n> index f817c509ec0..e81de1063a4 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -72,6 +72,7 @@\n> /git-format-patch\n> /git-fsck\n> /git-fsck-objects\n> +/git-fsmonitor--daemon\n> /git-gc\n> /git-get-tar-commit-id\n> /git-grep\n> diff --git a/Makefile b/Makefile\n> index 9943f0f7c11..3b7a3f88b50 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1106,6 +1106,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n> BUILTIN_OBJS += builtin/for-each-ref.o\n> BUILTIN_OBJS += builtin/for-each-repo.o\n> BUILTIN_OBJS += builtin/fsck.o\n> +BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n> BUILTIN_OBJS += builtin/gc.o\n> BUILTIN_OBJS += builtin/get-tar-commit-id.o\n> BUILTIN_OBJS += builtin/grep.o\n> diff --git a/builtin.h b/builtin.h\n> index 83379f3832c..40e9ecc8485 100644\n> --- a/builtin.h\n> +++ b/builtin.h\n> @@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n> int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n> int cmd_format_patch(int argc, const char **argv, const char *prefix);\n> int cmd_fsck(int argc, const char **argv, const char *prefix);\n> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n> int cmd_gc(int argc, const char **argv, const char *prefix);\n> int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n> int cmd_grep(int argc, const char **argv, const char *prefix);\n> diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n> new file mode 100644\n> index 00000000000..f0498793379\n> --- /dev/null\n> +++ b/builtin/fsmonitor--daemon.c\n> @@ -0,0 +1,46 @@\n> +#include \"builtin.h\"\n> +#include \"config.h\"\n> +#include \"parse-options.h\"\n> +#include \"fsmonitor.h\"\n> +#include \"fsmonitor-ipc.h\"\n> +#include \"simple-ipc.h\"\n> +#include \"khash.h\"\n> +\n> +static const char * const builtin_fsmonitor__daemon_usage[] = {\n> +\tNULL\n> +};\n> +\n> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> +\n> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> +{\n> +\tconst char *subcmd;\n> +\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\tgit_config(git_default_config, NULL);\n> +\n> +\targc = parse_options(argc, argv, prefix, options,\n> +\t\t\t builtin_fsmonitor__daemon_usage, 0);\n> +\tif (argc != 1)\n> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n> +\tsubcmd = argv[0];\n> +\n> +\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n> +}\n> +\n> +#else\n> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n> +\n> +\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n> +}\n> +#endif\n> diff --git a/git.c b/git.c\n> index 340665d4a04..a8b44d9b587 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -536,6 +536,7 @@ static struct cmd_struct commands[] = {\n> \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n> \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n> \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n> +\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n> \t{ \"gc\", cmd_gc, RUN_SETUP },\n> \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n> \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n\n\nI brought this up in another thread in how this series interacts with\nanother, but this patch below on top of \"seen\" would allow you to catch\nparse_options() BUGs on Linux, even if you don't have a no-OSX\nnon-Windows backend yet:\n\t\n\tdiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n\tindex 591433e897d..62c0b1d486b 100644\n\t--- a/builtin/fsmonitor--daemon.c\n\t+++ b/builtin/fsmonitor--daemon.c\n\t@@ -18,7 +18,6 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n\t \tNULL\n\t };\n\t \n\t-#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n\t /*\n\t * Global state loaded from config.\n\t */\n\t@@ -63,6 +62,7 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n\t \n\t \treturn git_default_config(var, value, cb);\n\t }\n\t+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n\t \n\t /*\n\t * Acting as a CLIENT.\n\t@@ -1492,6 +1492,8 @@ static int try_to_start_background_daemon(void)\n\t \t}\n\t }\n\t \n\t+#endif\n\t+\n\t int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n\t {\n\t \tconst char *subcmd;\n\t@@ -1532,6 +1534,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n\t \t\treturn -1;\n\t \t}\n\t \n\t+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n\t \tif (!strcmp(subcmd, \"start\"))\n\t \t\treturn !!try_to_start_background_daemon();\n\t \n\t@@ -1543,20 +1546,8 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n\t \n\t \tif (!strcmp(subcmd, \"status\"))\n\t \t\treturn !!do_as_client__status();\n\t-\n\t \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n\t-}\n\t-\n\t #else\n\t-int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n\t-{\n\t-\tstruct option options[] = {\n\t-\t\tOPT_END()\n\t-\t};\n\t-\n\t-\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n\t-\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n\t-\n\t \tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n\t-}\n\t #endif\n\t+}\n\nI.e. we can be a less zealous when setting the ifdef boundaries, and\nit's actually less code as well.\n"},{"id":"449736","messageId":"CAPMMpogycjDdoFR2kzLypb48LSQW_vU=yaeaMKdM1Thd+J3DUg@mail.gmail.com","threadId":"56518","inReplyTo":"6cba1d950b013410ecc6ffc15bfcba02c51d6de2.1644612979.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 23/30] t/helper/test-chmtime: skip directories on Windows","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2022-02-28T09:43:07Z","receivedAt":"2022-02-28T09:43:33Z","isPatch":true,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Random follow-up on this: in message/patch\n76b6216281e3463821e650495f3090c677905f73.1646041236.git.gitgitgadget@gmail.com\nI propose a fix to this issue. If accepted, that fix (and related\nchanges to rely\nmore heavily on chmtime in t7063) will cause *this* change to cause test\nfailures.\n\nIf that patch is accepted, this commit will simply need to be dropped\nas far as I understand.\n\n\n\nOn Fri, Feb 11, 2022 at 9:57 PM Jeff Hostetler via GitGitGadget\n wrote:\n>\n> From: Jeff Hostetler \n>\n> Teach `test-tool.exe chmtime` to ignore errors when setting the mtime\n> on a directory on Windows.\n>\n> NEEDSWORK: The Windows version of `utime()` (aka `mingw_utime()`) does\n> not properly handle directories because it uses `_wopen()`. It should\n> be converted to using `CreateFileW()` and backup semantics at a minimum.\n> Since I'm already in the middle of a large patch series, I did not want\n> to destabilize other callers of `utime()` right now. The problem has\n> only been observed in the t/perf/p7519 test when the test repo contains\n> an empty directory on disk.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> t/helper/test-chmtime.c | 15 +++++++++++++++\n> 1 file changed, 15 insertions(+)\n>\n> diff --git a/t/helper/test-chmtime.c b/t/helper/test-chmtime.c\n> index 524b55ca496..dc28890a183 100644\n> --- a/t/helper/test-chmtime.c\n> +++ b/t/helper/test-chmtime.c\n> @@ -134,6 +134,21 @@ int cmd__chmtime(int argc, const char **argv)\n> }\n>\n> if (utb.modtime != sb.st_mtime && utime(argv[i], &utb) < 0) {\n> +#ifdef GIT_WINDOWS_NATIVE\n> + if (S_ISDIR(sb.st_mode)) {\n> + /*\n> + * NEEDSWORK: The Windows version of `utime()`\n> + * (aka `mingw_utime()`) does not correctly\n> + * handle directory arguments, since it uses\n> + * `_wopen()`. Ignore it for now since this\n> + * is just a test.\n> + */\n> + fprintf(stderr,\n> + (\"Failed to modify time on directory %s. \"\n> + \"Skipping\\n\"), argv[i]);\n> + continue;\n> + }\n> +#endif\n> fprintf(stderr, \"Failed to modify time on %s: %s\\n\",\n> argv[i], strerror(errno));\n> return 1;\n> --\n> gitgitgadget\n>\n"},{"id":"449789","messageId":"73a47838-2f9e-dccb-9aaf-6eb143906d36@jeffhostetler.com","threadId":"56518","inReplyTo":"CAPMMpogycjDdoFR2kzLypb48LSQW_vU=yaeaMKdM1Thd+J3DUg@mail.gmail.com","subject":"Re: [PATCH v5 23/30] t/helper/test-chmtime: skip directories on Windows","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-02-28T17:49:58Z","receivedAt":"2022-02-28T18:10:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/28/22 4:43 AM, Tao Klerks wrote:\n> Random follow-up on this: in message/patch\n> 76b6216281e3463821e650495f3090c677905f73.1646041236.git.gitgitgadget@gmail.com\n> I propose a fix to this issue. If accepted, that fix (and related\n> changes to rely\n> more heavily on chmtime in t7063) will cause *this* change to cause test\n> failures.\n> \n> If that patch is accepted, this commit will simply need to be dropped\n> as far as I understand.\n> \n\nThanks for the note. Yes, with your change we should be able to\ndrop or revert this commit.\n\nThanks\nJeff\n\n"},{"id":"449796","messageId":"xmqqv8wy25un.fsf@gitster.g","threadId":"56518","inReplyTo":"73a47838-2f9e-dccb-9aaf-6eb143906d36@jeffhostetler.com","subject":"Re: [PATCH v5 23/30] t/helper/test-chmtime: skip directories on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-28T18:39:28Z","receivedAt":"2022-02-28T18:43:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> On 2/28/22 4:43 AM, Tao Klerks wrote:\n>> Random follow-up on this: in message/patch\n>> 76b6216281e3463821e650495f3090c677905f73.1646041236.git.gitgitgadget@gmail.com\n>> I propose a fix to this issue. If accepted, that fix (and related\n>> changes to rely\n>> more heavily on chmtime in t7063) will cause *this* change to cause test\n>> failures.\n>> If that patch is accepted, this commit will simply need to be\n>> dropped\n>> as far as I understand.\n>> \n>\n> Thanks for the note. Yes, with your change we should be able to\n> drop or revert this commit.\n\nThat sounds nice. The fewer differences among platforms we have to\nmaintain, the easier.\n"},{"id":"449929","messageId":"ffcd037b-256c-2e4b-79b7-ae725ad17c04@jeffhostetler.com","threadId":"56518","inReplyTo":"220225.865yp27e1c.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v5 06/30] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-01T14:58:42Z","receivedAt":"2022-03-01T14:59:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/25/22 5:46 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Feb 11 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Create a built-in file system monitoring daemon that can be used by\n>> the existing `fsmonitor` feature (protocol API and index extension)\n>> to improve the performance of various Git commands, such as `status`.\n>>\n>> The `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\n>> provides an alternative to hook access to existing fsmonitors such\n>> as `watchman`.\n>>\n>> This commit merely adds the new command without any functionality.\n>>\n[...]\n>> +\n>> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n>> +\n>> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n>> +{\n>> +\tconst char *subcmd;\n>> +\n>> +\tstruct option options[] = {\n>> +\t\tOPT_END()\n>> +\t};\n>> +\n>> +\tgit_config(git_default_config, NULL);\n>> +\n>> +\targc = parse_options(argc, argv, prefix, options,\n>> +\t\t\t builtin_fsmonitor__daemon_usage, 0);\n>> +\tif (argc != 1)\n>> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n>> +\tsubcmd = argv[0];\n>> +\n>> +\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n>> +}\n>> +\n>> +#else\n>> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n>> +{\n>> +\tstruct option options[] = {\n>> +\t\tOPT_END()\n>> +\t};\n>> +\n>> +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n>> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n>> +\n>> +\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n>> +}\n>> +#endif\n> \n> \n> I brought this up in another thread in how this series interacts with\n> another, but this patch below on top of \"seen\" would allow you to catch\n> parse_options() BUGs on Linux, even if you don't have a no-OSX\n> non-Windows backend yet:\n> \t\n> \tdiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n> \tindex 591433e897d..62c0b1d486b 100644\n> \t--- a/builtin/fsmonitor--daemon.c\n> \t+++ b/builtin/fsmonitor--daemon.c\n> \t@@ -18,7 +18,6 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n> \t \tNULL\n> \t };\n> \t\n> \t-#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> \t /*\n> \t * Global state loaded from config.\n> \t */\n> \t@@ -63,6 +62,7 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n> \t\n> \t \treturn git_default_config(var, value, cb);\n> \t }\n> \t+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> \t\n> \t /*\n> \t * Acting as a CLIENT.\n> \t@@ -1492,6 +1492,8 @@ static int try_to_start_background_daemon(void)\n> \t \t}\n> \t }\n> \t\n> \t+#endif\n> \t+\n> \t int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> \t {\n> \t \tconst char *subcmd;\n> \t@@ -1532,6 +1534,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> \t \t\treturn -1;\n> \t \t}\n> \t\n> \t+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> \t \tif (!strcmp(subcmd, \"start\"))\n> \t \t\treturn !!try_to_start_background_daemon();\n> \t\n> \t@@ -1543,20 +1546,8 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> \t\n> \t \tif (!strcmp(subcmd, \"status\"))\n> \t \t\treturn !!do_as_client__status();\n> \t-\n> \t \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n> \t-}\n> \t-\n> \t #else\n> \t-int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> \t-{\n> \t-\tstruct option options[] = {\n> \t-\t\tOPT_END()\n> \t-\t};\n> \t-\n> \t-\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n> \t-\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n> \t-\n> \t \tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n> \t-}\n> \t #endif\n> \t+}\n> \n> I.e. we can be a less zealous when setting the ifdef boundaries, and\n> it's actually less code as well.\n> \n\nYes, it would be possible to distribute the ifdef throughout the file\nand avoid duplicating the function declaration, but I'm not sure that\nthat adds any clarity or readability.\n\nIn my version, I have a stub version of the cmd_fsmonitor__daemon()\nfunction and it is very clear that it does nothing when the feature\nis not supported on a platform. The rest of the source file is\nconcerned with supporting the feature. And no interweaving of ifdefs\nthroughout the file is required.\n\nYour version sets us up for future problems inside the body of the\ncmd_ function. For example, any static function called in the\nsupported portion of the function would also need to be ifdef'd\n(as you have indicated). But any local variables needed by the\nsupported portion would need to be declared at the top of the\nfunction and also ifdef'd -- or we'd need to indent the entire body\nof the supported portion inside another level of { }. None of this\nadds clarity. (Just to avoid an 11 line stub function.)\n\nFinally, I'm not sure how much value there is in being able to catch\nparse_options() BUGs on Linux (or any other yet-to-be-supported\nplatform). The daemon isn't supported and dies immediately. I'm not\nsure that forcing the user to properly compose any arguments before\nwe just call die() is helpful.\n\nSo, I'd rather leave this as is.\n\nThanks,\nJeff\n"},{"id":"449940","messageId":"xmqq35k1shf8.fsf@gitster.g","threadId":"56518","inReplyTo":"ffcd037b-256c-2e4b-79b7-ae725ad17c04@jeffhostetler.com","subject":"Re: [PATCH v5 06/30] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-01T17:37:15Z","receivedAt":"2022-03-01T17:37:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> Finally, I'm not sure how much value there is in being able to catch\n> parse_options() BUGs on Linux (or any other yet-to-be-supported\n> platform).\n\nI don't either. Even though I am not 100% happy with the current\nimplementation of the embedded sanity checker in parse-options API,\nas long as it is made available on all platforms, developers of a\nplatform specific part can also take advantage of it to catch any\nissues while they develop, without waiting for Linux (or other\nplatforms) users to help them do so.\n\n\n"},{"id":"449955","messageId":"10a34082fcc7ddf693b4a50fb2ce05905ee2986f.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 01/30] fsmonitor: enhance existing comments, clarify trivial response handling","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:03Z","receivedAt":"2022-03-01T18:43:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 64 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 41 insertions(+), 23 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..448d0ee33f5 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -168,29 +168,15 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf *\n \n \tif (result)\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/failed\", result);\n-\telse {\n+\telse\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/response-length\",\n \t\t\t\t query_result->len);\n \n-\t\tif (fsmonitor_is_trivial_response(query_result))\n-\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n-\t\t\t\t\t \"query/trivial-response\", 1);\n-\t}\n-\n \ttrace2_region_leave(\"fsm_hook\", \"query\", NULL);\n \n \treturn result;\n }\n \n-int fsmonitor_is_trivial_response(const struct strbuf *query_result)\n-{\n-\tstatic char trivial_response[3] = { '\\0', '/', '\\0' };\n-\n-\treturn query_result->len >= 3 &&\n-\t\t!memcmp(trivial_response,\n-\t\t\t&query_result->buf[query_result->len - 3], 3);\n-}\n-\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n@@ -238,6 +224,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tint is_trivial = 0;\n \n \tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n \t\treturn;\n@@ -283,6 +270,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t\tquery_success = 0;\n \t\t\t\t} else {\n \t\t\t\t\tbol = last_update_token.len + 1;\n+\t\t\t\t\tis_trivial = query_result.buf[bol] == '/';\n \t\t\t\t}\n \t\t\t} else if (hook_version < 0) {\n \t\t\t\thook_version = HOOK_INTERFACE_VERSION1;\n@@ -294,16 +282,38 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\t\tif (query_success)\n+\t\t\t\tis_trivial = query_result.buf[0] == '/';\n \t\t}\n \n+\t\tif (is_trivial)\n+\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n \t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n \t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n-\tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n+\tif (query_success && !is_trivial) {\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -318,11 +328,16 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We failed to get a response or received a trivial response,\n+\t\t * so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -330,7 +345,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"449956","messageId":"e1c946af2c54146dcc333b862544d8ba94a53b3e.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 02/30] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:04Z","receivedAt":"2022-03-01T18:43:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 ++++++++++++++\n 3 files changed, 220 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 6f0b4b775fe..a19d850e716 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -907,6 +907,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..91a535f1219\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,171 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#else\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(\"fsmonitor--daemon is not running\");\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\", c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..b6a7067c3af\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"449957","messageId":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v5.git.1644612979.gitgitgadget@gmail.com","subject":"[PATCH v6 00/30] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:02Z","receivedAt":"2022-03-01T18:43:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V6 of Part 2 of my builtin FSmonitor series.\n\nThis version contains mostly cleanup based on feedback from V5. Of note:\n\n * I squashed in the 1_file fix for p7519.\n * I squashed in a commit from part 3 to optionally print the \"running\n daemon on...\" message on stderr.\n * I added a note to the documentation about incompatible changes around\n core.fsmonitor.\n * Removed/rephrased some obsolete NEEDSWORK items.\n\nTao has an ongoing parallel series to fix test-chmtime on Windows.\nhttps://lore.kernel.org/all/pull.1166.git.1646041236.gitgitgadget@gmail.com/\n\nIf that lands first, we should be able to drop my 't/helper/test-chmtime:\nskip directories on Windows' commit.\n\nA followup Part 3 will contain additional refinements to the daemon and\nadditional tests. I drew the line here between Part 2 and 3 to make it\neasier to review.\n\nJeff Hostetler (30):\n fsmonitor: enhance existing comments, clarify trivial response\n handling\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor: document builtin fsmonitor\n fsmonitor--daemon: add a built-in fsmonitor daemon\n fsmonitor--daemon: implement 'stop' and 'status' commands\n compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n fsmonitor--daemon: implement 'run' command\n fsmonitor--daemon: implement 'start' command\n fsmonitor--daemon: add pathname classification\n fsmonitor--daemon: define token-ids\n fsmonitor--daemon: create token-based changed path cache\n compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on\n Windows\n compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on\n MacOS\n fsmonitor--daemon: implement handle_client callback\n help: include fsmonitor--daemon feature flag in version info\n t/helper/fsmonitor-client: create IPC client to talk to FSMonitor\n Daemon\n t7527: create test for fsmonitor--daemon\n t/perf: avoid copying builtin fsmonitor files into test repo\n t/helper/test-chmtime: skip directories on Windows\n t/perf/p7519: speed up test on Windows\n t/perf/p7519: add fsmonitor--daemon test cases\n fsmonitor--daemon: periodically truncate list of modified files\n fsmonitor--daemon: use a cookie file to sync with file system\n fsmonitor: force update index after large responses\n t7527: test status with untracked-cache and fsmonitor--daemon\n update-index: convert fsmonitor warnings to advise\n\n .gitignore | 1 +\n Documentation/config/core.txt | 60 +-\n Documentation/git-fsmonitor--daemon.txt | 75 ++\n Documentation/git-update-index.txt | 8 +-\n Makefile | 17 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 1473 +++++++++++++++++++++++\n builtin/update-index.c | 19 +-\n cache.h | 1 -\n compat/fsmonitor/fsm-listen-darwin.c | 499 ++++++++\n compat/fsmonitor/fsm-listen-win32.c | 586 +++++++++\n compat/fsmonitor/fsm-listen.h | 49 +\n config.c | 14 -\n config.h | 1 -\n config.mak.uname | 20 +\n contrib/buildsystems/CMakeLists.txt | 10 +\n environment.c | 1 -\n fsmonitor--daemon.h | 166 +++\n fsmonitor-ipc.c | 171 +++\n fsmonitor-ipc.h | 48 +\n fsmonitor-settings.c | 115 ++\n fsmonitor-settings.h | 21 +\n fsmonitor.c | 216 +++-\n fsmonitor.h | 15 +-\n git.c | 1 +\n help.c | 4 +\n repo-settings.c | 1 +\n repository.h | 3 +\n t/README | 4 +-\n t/helper/test-chmtime.c | 15 +\n t/helper/test-fsmonitor-client.c | 121 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/perf/p7519-fsmonitor.sh | 61 +-\n t/perf/perf-lib.sh | 2 +-\n t/t7527-builtin-fsmonitor.sh | 604 ++++++++++\n t/test-lib.sh | 6 +\n 37 files changed, 4301 insertions(+), 110 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n create mode 100644 builtin/fsmonitor--daemon.c\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n create mode 100644 fsmonitor--daemon.h\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n create mode 100644 t/helper/test-fsmonitor-client.c\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\n\nbase-commit: 715d08a9e51251ad8290b181b6ac3b9e1f9719d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/1041\n\nRange-diff vs v5:\n\n 1: a5ecabb4d02 ! 1: 10a34082fcc fsmonitor: enhance existing comments, clarify trivial response handling\n @@ Commit message\n fsmonitor: enhance existing comments, clarify trivial response handling\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## fsmonitor.c ##\n @@ fsmonitor.c: static int query_fsmonitor(int version, const char *last_update, struct strbuf *\n 2: 365964b7664 ! 2: e1c946af2c5 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n @@ Commit message\n Stub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## Makefile ##\n @@ Makefile: LIB_OBJS += fetch-pack.o\n @@ fsmonitor-ipc.c (new)\n +#include \"strbuf.h\"\n +#include \"trace2.h\"\n +\n -+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n ++#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n ++\n ++/*\n ++ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n ++ * platforms.\n ++ */\n ++\n ++int fsmonitor_ipc__is_supported(void)\n ++{\n ++\treturn 0;\n ++}\n ++\n ++const char *fsmonitor_ipc__get_path(void)\n ++{\n ++\treturn NULL;\n ++}\n ++\n ++enum ipc_active_state fsmonitor_ipc__get_state(void)\n ++{\n ++\treturn IPC_STATE__OTHER_ERROR;\n ++}\n ++\n ++int fsmonitor_ipc__send_query(const char *since_token,\n ++\t\t\t struct strbuf *answer)\n ++{\n ++\treturn -1;\n ++}\n ++\n ++int fsmonitor_ipc__send_command(const char *command,\n ++\t\t\t\tstruct strbuf *answer)\n ++{\n ++\treturn -1;\n ++}\n ++\n ++#else\n +\n +int fsmonitor_ipc__is_supported(void)\n +{\n @@ fsmonitor-ipc.c (new)\n +\treturn 0;\n +}\n +\n -+#else\n -+\n -+/*\n -+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n -+ * platforms.\n -+ */\n -+\n -+int fsmonitor_ipc__is_supported(void)\n -+{\n -+\treturn 0;\n -+}\n -+\n -+const char *fsmonitor_ipc__get_path(void)\n -+{\n -+\treturn NULL;\n -+}\n -+\n -+enum ipc_active_state fsmonitor_ipc__get_state(void)\n -+{\n -+\treturn IPC_STATE__OTHER_ERROR;\n -+}\n -+\n -+int fsmonitor_ipc__send_query(const char *since_token,\n -+\t\t\t struct strbuf *answer)\n -+{\n -+\treturn -1;\n -+}\n -+\n -+int fsmonitor_ipc__send_command(const char *command,\n -+\t\t\t\tstruct strbuf *answer)\n -+{\n -+\treturn -1;\n -+}\n -+\n +#endif\n \n ## fsmonitor-ipc.h (new) ##\n 3: 384516ce1a1 ! 3: ae622a517cf fsmonitor: config settings are repository-specific\n @@ Commit message\n restrictions to override the config values used.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## Makefile ##\n @@ Makefile: LIB_OBJS += fmt-merge-msg.o\n @@ fsmonitor-settings.c (new)\n +\n +enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n +{\n ++\tif (!r)\n ++\t\tr = the_repository;\n ++\n +\tlookup_fsmonitor_settings(r);\n +\n +\treturn r->settings.fsmonitor->mode;\n @@ fsmonitor-settings.c (new)\n +\n +const char *fsm_settings__get_hook_path(struct repository *r)\n +{\n ++\tif (!r)\n ++\t\tr = the_repository;\n ++\n +\tlookup_fsmonitor_settings(r);\n +\n +\treturn r->settings.fsmonitor->hook_path;\n @@ fsmonitor-settings.c (new)\n +\n +void fsm_settings__set_ipc(struct repository *r)\n +{\n ++\tif (!r)\n ++\t\tr = the_repository;\n ++\n +\tlookup_fsmonitor_settings(r);\n +\n +\tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n @@ fsmonitor-settings.c (new)\n +\n +void fsm_settings__set_hook(struct repository *r, const char *path)\n +{\n ++\tif (!r)\n ++\t\tr = the_repository;\n ++\n +\tlookup_fsmonitor_settings(r);\n +\n +\tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n @@ fsmonitor-settings.c (new)\n +\n +void fsm_settings__set_disabled(struct repository *r)\n +{\n ++\tif (!r)\n ++\t\tr = the_repository;\n ++\n +\tlookup_fsmonitor_settings(r);\n +\n +\tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n @@ fsmonitor.c: void remove_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n -\tint fsmonitor_enabled = git_config_get_fsmonitor();\n -+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n -+\tint fsmonitor_enabled = (fsm_settings__get_mode(r) > FSMONITOR_MODE_DISABLED);\n ++\tint fsmonitor_enabled = (fsm_settings__get_mode(istate->repo)\n ++\t\t\t\t > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n @@ fsmonitor.h: int fsmonitor_is_trivial_response(const struct strbuf *query_result\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n -\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n -+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n -+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n ++\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n +\n +\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n +\t\tistate->fsmonitor_has_run_once;\n @@ fsmonitor.h: static inline int is_fsmonitor_refreshed(const struct index_state *\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n -\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n -+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n -+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n ++\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n +\n +\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n +\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n @@ fsmonitor.h: static inline void mark_fsmonitor_valid(struct index_state *istate,\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n -\tif (core_fsmonitor) {\n -+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n -+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n ++\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n +\n +\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n 4: 8e738a83bc5 ! 4: 55974867da5 fsmonitor: use IPC to query the builtin FSMonitor daemon\n @@ Commit message\n \n Signed-off-by: Johannes Schindelin \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## fsmonitor.c ##\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n 5: 49e4c146e02 ! 5: ce42d5bbaf6 fsmonitor: document builtin fsmonitor\n @@ Commit message\n Create `git-fsmonitor--daemon` manual page and describe its features.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## Documentation/config/core.txt ##\n @@ Documentation/config/core.txt: core.protectNTFS::\n @@ Documentation/config/core.txt: core.protectNTFS::\n +git by avoiding unnecessary scanning of files that have not changed.\n ++\n +See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n +++\n ++Note that if you concurrently use multiple versions of Git, such\n ++as one version on the command line and another version in an IDE\n ++tool, that the definition of `core.fsmonitor` was extended to\n ++allow boolean values in addition to hook pathnames. Git versions\n ++2.35.1 and prior will not understand the boolean values and will\n ++consider the \"true\" or \"false\" values as hook pathnames to be\n ++invoked. Git versions 2.26 thru 2.35.1 default to hook protocol\n ++V2 and will fall back to no fsmonitor (full scan). Git versions\n ++prior to 2.26 default to hook protocol V1 and will silently\n ++assume there were no changes to report (no scan), so status\n ++commands may report incomplete results. For this reason, it is\n ++best to upgrade all of your Git versions before using the built-in\n ++file system monitor.\n \n core.fsmonitorHookVersion::\n -\tSets the version of hook that is to be used when calling fsmonitor.\n 6: bdd7334da31 ! 6: 9ce938c69b5 fsmonitor--daemon: add a built-in fsmonitor daemon\n @@ Commit message\n \n Co-authored-by: Johannes Schindelin \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## .gitignore ##\n @@\n 7: 9f00ada3dd3 ! 7: 3bd7e6a051e fsmonitor--daemon: implement 'stop' and 'status' commands\n @@ Commit message\n system.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/fsmonitor--daemon.c ##\n @@\n 8: d6819bdad66 ! 8: 92d3e54da7f compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n @@ Commit message\n Stub in empty filesystem listener backend for fsmonitor--daemon on Windows.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## Makefile ##\n @@ Makefile: all::\n 9: 7de3d01cccc ! 9: f1219221270 compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n @@ Commit message\n backend for Darwin (aka MacOS).\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## compat/fsmonitor/fsm-listen-darwin.c (new) ##\n @@\n 10: 6fe5a2bc79e ! 10: a3f94627cf6 fsmonitor--daemon: implement 'run' command\n @@ Commit message\n backends are still just stubs.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/fsmonitor--daemon.c ##\n @@\n @@ builtin/fsmonitor--daemon.c\n +#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n +static int fsmonitor__ipc_threads = 8;\n +\n ++#define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n ++static int fsmonitor__announce_startup = 0;\n ++\n +static int fsmonitor_config(const char *var, const char *value, void *cb)\n +{\n +\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n @@ builtin/fsmonitor--daemon.c\n +\t\treturn 0;\n +\t}\n +\n ++\tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n ++\t\tint is_bool;\n ++\t\tint i = git_config_bool_or_int(var, value, &is_bool);\n ++\t\tif (i < 0)\n ++\t\t\treturn error(_(\"value of '%s' not bool or int: %d\"),\n ++\t\t\t\t var, i);\n ++\t\tfsmonitor__announce_startup = i;\n ++\t\treturn 0;\n ++\t}\n ++\n +\treturn git_default_config(var, value, cb);\n +}\n +\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n +\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n +\t\t the_repository->worktree);\n +\n -+\tprintf(_(\"running fsmonitor-daemon in '%s'\\n\"),\n -+\t the_repository->worktree);\n -+\tfflush(stdout);\n ++\tif (fsmonitor__announce_startup) {\n ++\t\tfprintf(stderr, _(\"running fsmonitor-daemon in '%s'\\n\"),\n ++\t\t\tthe_repository->worktree);\n ++\t\tfflush(stderr);\n ++\t}\n +\n +\treturn !!fsmonitor_run_daemon();\n +}\n 11: 69fc0998286 ! 11: 8de40b0fe8b fsmonitor--daemon: implement 'start' command\n @@ Commit message\n because they don't have a Win32 console attached to them.)\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/fsmonitor--daemon.c ##\n @@\n @@ builtin/fsmonitor--daemon.c: static const char * const builtin_fsmonitor__daemon\n +#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n +static int fsmonitor__start_timeout_sec = 60;\n +\n - static int fsmonitor_config(const char *var, const char *value, void *cb)\n - {\n - \tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n + #define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n + static int fsmonitor__announce_startup = 0;\n + \n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_config(const char *var, const\n +\t\treturn 0;\n +\t}\n +\n - \treturn git_default_config(var, value, cb);\n - }\n - \n + \tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n + \t\tint is_bool;\n + \t\tint i = git_config_bool_or_int(var, value, &is_bool);\n @@ builtin/fsmonitor--daemon.c: done:\n \treturn err;\n }\n \n -static int try_to_run_foreground_daemon(void)\n -+static int try_to_run_foreground_daemon(int free_console)\n ++static int try_to_run_foreground_daemon(int detach_console)\n {\n \t/*\n \t * Technically, we don't need to probe for an existing daemon\n @@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)\n - \t the_repository->worktree);\n - \tfflush(stdout);\n + \t\tfflush(stderr);\n + \t}\n \n +#ifdef GIT_WINDOWS_NATIVE\n -+\tif (free_console)\n ++\tif (detach_console)\n +\t\tFreeConsole();\n +#endif\n +\n @@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)\n +\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n +\t\t the_repository->worktree);\n +\n -+\tprintf(_(\"starting fsmonitor-daemon in '%s'\\n\"),\n -+\t the_repository->worktree);\n -+\tfflush(stdout);\n ++\tif (fsmonitor__announce_startup) {\n ++\t\tfprintf(stderr, _(\"starting fsmonitor-daemon in '%s'\\n\"),\n ++\t\t\tthe_repository->worktree);\n ++\t\tfflush(stderr);\n ++\t}\n +\n +\tcp.git_cmd = 1;\n +\n +\tstrvec_push(&cp.args, \"fsmonitor--daemon\");\n +\tstrvec_push(&cp.args, \"run\");\n -+\tstrvec_push(&cp.args, \"--free-console\");\n ++\tstrvec_push(&cp.args, \"--detach\");\n +\tstrvec_pushf(&cp.args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n +\n +\tcp.no_stdin = 1;\n @@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n -+\tint free_console = 0;\n ++\tint detach_console = 0;\n \n \tstruct option options[] = {\n -+\t\tOPT_BOOL(0, \"free-console\", &free_console, N_(\"free console\")),\n ++\t\tOPT_BOOL(0, \"detach\", &detach_console, N_(\"detach from console\")),\n \t\tOPT_INTEGER(0, \"ipc-threads\",\n \t\t\t &fsmonitor__ipc_threads,\n \t\t\t N_(\"use ipc worker threads\")),\n +\t\tOPT_INTEGER(0, \"start-timeout\",\n +\t\t\t &fsmonitor__start_timeout_sec,\n -+\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n ++\t\t\t N_(\"max seconds to wait for background daemon startup\")),\n +\n \t\tOPT_END()\n \t};\n @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar\n +\n \tif (!strcmp(subcmd, \"run\"))\n -\t\treturn !!try_to_run_foreground_daemon();\n -+\t\treturn !!try_to_run_foreground_daemon(free_console);\n ++\t\treturn !!try_to_run_foreground_daemon(detach_console);\n \n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n 12: 21c099c5197 ! 12: e5d419b1ea0 fsmonitor--daemon: add pathname classification\n @@ Commit message\n When we register for filesystem notifications on a directory,\n we get events for everything (recursively) in the directory.\n We want to report to clients changes to tracked and untracked\n - paths within the working directory. We do not want to report\n - changes within the .git directory, for example.\n + paths within the working directory proper. We do not want to\n + report changes within the .git directory, for example.\n \n This classification will be used in a later commit by the\n different backends to classify paths as events are received.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/fsmonitor--daemon.c ##\n @@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,\n @@ fsmonitor--daemon.h: struct fsmonitor_daemon_state {\n + * that to decide whether clients are told about them. (And to watch\n + * for file system synchronization events.)\n + *\n ++ * The daemon only collects and reports on the set of modified paths\n ++ * within the working directory (proper).\n ++ *\n + * The client should only care about paths within the working\n + * directory proper (inside the working directory and not \".git\" nor\n + * inside of \".git/\"). That is, the client has read the index and is\n + * asking for a list of any paths in the working directory that have\n + * been modified since the last token. The client does not care about\n -+ * file system changes within the .git directory (such as new loose\n ++ * file system changes within the \".git/\" directory (such as new loose\n + * objects or packfiles). So the client will only receive paths that\n + * are classified as IS_WORKDIR_PATH.\n + *\n ++ * Note that \".git\" is usually a directory and is therefore inside\n ++ * the cone of the FS watch that we have on the working directory root,\n ++ * so we will also get FS events for disk activity on and within \".git/\"\n ++ * that we need to respond to or filter from the client.\n ++ *\n ++ * But Git also allows \".git\" to be a *file* that points to a GITDIR\n ++ * outside of the working directory. When this happens, we need to\n ++ * create FS watches on both the working directory root *and* on the\n ++ * (external) GITDIR root. (The latter is required because we put\n ++ * cookie files inside it and use them to sync with the FS event\n ++ * stream.)\n ++ *\n ++ * Note that in the context of this discussion, I'm using \"GITDIR\"\n ++ * to only mean an external GITDIR referenced by a \".git\" file.\n ++ *\n ++ * The platform FS event backends will receive watch-specific\n ++ * relative paths (except for those OS's that always emit absolute\n ++ * paths). We use the following enum and routines to classify each\n ++ * path so that we know how to handle it. There is a slight asymmetry\n ++ * here because \".git/\" is inside the working directory and the\n ++ * (external) GITDIR is not, and therefore how we handle events may\n ++ * vary slightly, so I have different enums for \"IS...DOT_GIT...\" and\n ++ * \"IS...GITDIR...\".\n ++ *\n + * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n -+ * exact \".git\" directory or GITDIR. If the daemon receives a delete\n -+ * event for either of these directories, it will automatically\n -+ * shutdown, for example.\n ++ * exact \".git\" file/directory or GITDIR directory. If the daemon\n ++ * receives a delete event for either of these paths, it will\n ++ * automatically shutdown, for example.\n + *\n + * Note that the daemon DOES NOT explicitly watch nor special case the\n -+ * \".git/index\" file. The daemon does not read the index and does not\n -+ * have any internal index-relative state. The daemon only collects\n -+ * the set of modified paths within the working directory.\n ++ * index. The daemon does not read the index nor have any internal\n ++ * index-relative state, so there are no \"IS...INDEX...\" enum values.\n + */\n +enum fsmonitor_path_type {\n +\tIS_WORKDIR_PATH = 0,\n 13: 72979c35ceb ! 13: c2f3668a866 fsmonitor--daemon: define token-ids\n @@ Commit message\n overall token naming scheme.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/fsmonitor--daemon.c ##\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n 14: 8d1db644409 ! 14: bd492d13fb1 fsmonitor--daemon: create token-based changed path cache\n @@ Commit message\n paths within these batches for efficient presentation to clients.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/fsmonitor--daemon.c ##\n @@ builtin/fsmonitor--daemon.c: struct fsmonitor_token_data {\n @@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t\n + * We should create a new token and start fresh (as if we just\n + * booted up).\n + *\n -+ * If there are no concurrent threads readering the current token data\n ++ * If there are no concurrent threads reading the current token data\n + * series, we can free it now. Otherwise, let the last reader free\n + * it.\n + *\n 15: 98c5adf8ca0 ! 15: 6b5b3d08227 compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows\n @@ Commit message\n into batches and publish.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## compat/fsmonitor/fsm-listen-win32.c ##\n @@\n 16: a3b881315fa ! 16: cdef9730b3f compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent\n + compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n \n Include MacOS system declarations to allow us to use FSEvent and\n - CoreFoundation APIs. We need GCC and clang versions because of\n - compiler and header file conflicts.\n + CoreFoundation APIs. We need different versions of the declarations\n + for GCC vs. clang because of compiler and header file conflicts.\n \n While it is quite possible to #include Apple's CoreServices.h when\n compiling C source code with clang, trying to build it with GCC\n @@ Commit message\n \n Signed-off-by: Johannes Schindelin \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## compat/fsmonitor/fsm-listen-darwin.c ##\n @@\n -+#if defined(__GNUC__)\n ++#ifndef __clang__\n +/*\n + * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n + * with clang, but not with GCC as of time of writing.\n 17: 162e357db72 ! 17: aabfebd3a41 compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS\n @@ Commit message\n Co-authored-by: Kevin Willford \n Co-authored-by: Johannes Schindelin \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## compat/fsmonitor/fsm-listen-darwin.c ##\n -@@\n --#if defined(__GNUC__)\n -+#ifndef __clang__\n - /*\n - * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n - * with clang, but not with GCC as of time of writing.\n @@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef stream);\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n @@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef\n +}\n +\n +/*\n -+ * NEEDSWORK: Investigate the proper value for the `latency` argument\n -+ * in the call to `FSEventStreamCreate()`. I'm not sure that this\n -+ * needs to be a config setting or just something that we tune after\n -+ * some testing.\n ++ * In the call to `FSEventStreamCreate()` to setup our watch, the\n ++ * `latency` argument determines the frequency of calls to our callback\n ++ * with new FS events. Too slow and events get dropped; too fast and\n ++ * we burn CPU unnecessarily. Since it is rather obscure, I don't\n ++ * think this needs to be a config setting. I've done extensive\n ++ * testing on my systems and chosen the value below. It gives good\n ++ * results and I've not seen any dropped events.\n + *\n + * With a latency of 0.1, I was seeing lots of dropped events during\n + * the \"touch 100000\" files test within t/perf/p7519, but with a\n -+ * latency of 0.001 I did not see any dropped events. So the\n -+ * \"correct\" value may be somewhere in between.\n ++ * latency of 0.001 I did not see any dropped events. So I'm going\n ++ * to assume that this is the \"correct\" value.\n + *\n + * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n + */\n 18: 3de1c43beaf ! 18: 0896e72e2b8 fsmonitor--daemon: implement handle_client callback\n @@ Commit message\n relative to the provided token.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/fsmonitor--daemon.c ##\n @@\n 19: 3517c4a3c13 ! 19: bc240a9e665 help: include fsmonitor--daemon feature flag in version info\n @@ Commit message\n \n This can be used by test scripts for prereq testing. Granted, tests\n could just try `git fsmonitor--daemon status` and look for a 128 exit\n - code or grep for a \"not supported\" message on stderr, but this is\n - rather obscure.\n + code or grep for a \"not supported\" message on stderr, but these\n + methods are rather obscure.\n \n The main advantage is that the feature message will automatically\n appear in bug reports and other support requests.\n \n + This concept was also used during the development of Scalar for\n + similar reasons.\n +\n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## help.c ##\n @@\n 20: 4ffc2ddf516 ! 20: 2b563c240eb t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n @@ Commit message\n Create an IPC client to send query and flush commands to the daemon.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## Makefile ##\n @@ Makefile: TEST_BUILTINS_OBJS += test-dump-split-index.o\n 21: d93310a7c64 ! 21: db8809da089 t7527: create test for fsmonitor--daemon\n @@ Commit message\n t7527: create test for fsmonitor--daemon\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## t/t7527-builtin-fsmonitor.sh (new) ##\n @@\n 22: 27e47108908 ! 22: 15bd5aaff36 t/perf: avoid copying builtin fsmonitor files into test repo\n @@ Commit message\n .git/fsmonitor--daemon*, so it is simple to exclude them.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## t/perf/perf-lib.sh ##\n @@ t/perf/perf-lib.sh: test_perf_copy_repo_contents () {\n 23: 6cba1d950b0 ! 23: 4cb97918d20 t/helper/test-chmtime: skip directories on Windows\n @@ Commit message\n an empty directory on disk.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## t/helper/test-chmtime.c ##\n @@ t/helper/test-chmtime.c: int cmd__chmtime(int argc, const char **argv)\n 24: fcf843a0d42 ! 24: a70748b4640 t/perf/p7519: speed up test on Windows\n @@ Commit message\n on Windows because of process creation overhead.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## t/perf/p7519-fsmonitor.sh ##\n @@ t/perf/p7519-fsmonitor.sh: then\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n -\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n -+\ttouch_files 1 &&\n ++\t: 1_file directory should be left empty &&\n +\ttouch_files 10 &&\n +\ttouch_files 100 &&\n +\ttouch_files 1000 &&\n 25: 198f47bda5a ! 25: b0c9c9c7a40 t/perf/p7519: add fsmonitor--daemon test cases\n @@ Commit message\n the \"Simple IPC\" interface.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## t/perf/p7519-fsmonitor.sh ##\n @@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n 26: 19993c130d2 ! 26: 64a5b741670 fsmonitor--daemon: periodically truncate list of modified files\n @@ Commit message\n they are considered obsolete.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/fsmonitor--daemon.c ##\n @@ builtin/fsmonitor--daemon.c: static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n 27: f47a763dc26 ! 27: 5b06eb5d0e6 fsmonitor--daemon: use a cookie file to sync with file system\n @@ Commit message\n of `FSeventStreamCreate()`. The kernel only signals every `latency`\n seconds, but does not guarantee that the kernel queue is completely\n drained, so we may have to wait more than one interval. If we\n - increase the frequency, the system is more likely to drop events.\n + increase the latency, the system is more likely to drop events.\n We avoid these issues by having each client thread create a unique\n cookie file and then wait until it is seen in the event stream.\n \n Co-authored-by: Kevin Willford \n Co-authored-by: Johannes Schindelin \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/fsmonitor--daemon.c ##\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n +\t\tunlink(cookie_pathname.buf);\n +\n +\t\t/*\n -+\t\t * NEEDSWORK: This is an infinite wait (well, unless another\n ++\t\t * Technically, this is an infinite wait (well, unless another\n +\t\t * thread sends us an abort). I'd like to change this to\n +\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n -+\t\t * and let the caller do the trivial response thing.\n ++\t\t * and let the caller do the trivial response thing, but we\n ++\t\t * don't have that routine in our thread-utils.\n ++\t\t *\n ++\t\t * After extensive beta testing I'm not really worried about\n ++\t\t * this. Also note that the above open() and unlink() calls\n ++\t\t * will cause at least two FS events on that path, so the odds\n ++\t\t * of getting stuck are pretty slim.\n +\t\t */\n +\t\twhile (cookie->result == FCIR_INIT)\n +\t\t\tpthread_cond_wait(&state->cookies_cond,\n @@ builtin/fsmonitor--daemon.c: static void fsmonitor_free_token_data(struct fsmoni\n + * [2] Some of those lost events may have been for cookie files. We\n + * should assume the worst and abort them rather letting them starve.\n + *\n - * If there are no concurrent threads readering the current token data\n + * If there are no concurrent threads reading the current token data\n * series, we can free it now. Otherwise, let the last reader free\n * it.\n @@ builtin/fsmonitor--daemon.c: static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n @@ builtin/fsmonitor--daemon.c: done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n +\tstrbuf_release(&state.path_cookie_prefix);\n -+\n -+\t/*\n -+\t * NEEDSWORK: Consider \"rm -rf /\"\n -+\t */\n \n \treturn err;\n }\n 28: aec44a21afd ! 28: 1fd5439de03 fsmonitor: force update index after large responses\n @@ Commit message\n unnecessarily spend time actually scanning them during each command.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## fsmonitor.c ##\n @@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n @@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate,\n + * will be updated automatically the first time the user touches\n + * a tracked file and causes a command like `git status` to\n + * update an mtime to be updated and/or set a flag bit.\n -+ *\n -+ * NEEDSWORK: Does this need to be a config value?\n + */\n +static int fsmonitor_force_update_threshold = 100;\n +\n 29: d6039987df8 ! 29: 6fc1430285f t7527: test status with untracked-cache and fsmonitor--daemon\n @@ Commit message\n identical.\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## t/t7527-builtin-fsmonitor.sh ##\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '\n 30: 5117fbdfc63 ! 30: b915b95cc2f update-index: convert fsmonitor warnings to advise\n @@ Commit message\n update-index: convert fsmonitor warnings to advise\n \n Signed-off-by: Jeff Hostetler \n + Signed-off-by: Junio C Hamano \n \n ## builtin/update-index.c ##\n @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const char *prefix)\n\n-- \ngitgitgadget\n"},{"id":"449958","messageId":"55974867da5f791fe736221ffa1ae4cf3e204c17.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 04/30] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:06Z","receivedAt":"2022-03-01T18:43:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.fsmonitor` is set to true.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 38 ++++++++++++++++++++++++++++++++++++--\n 1 file changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 0e961b74d82..a38b5710eb3 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -241,8 +241,41 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (fsm_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t\tis_trivial = query_result.buf[bol] == '/';\n+\t\t\tif (is_trivial)\n+\t\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t\t \"query/trivial-response\", 1);\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\tgoto apply_results;\n \t}\n \n \tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n@@ -315,6 +348,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/*\n \t * The response from FSMonitor (excluding the header token) is\n \t * either:\n-- \ngitgitgadget\n\n"},{"id":"449959","messageId":"ae622a517cf500d5b068871434e33f98e9cf586b.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 03/30] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:05Z","receivedAt":"2022-03-01T18:43:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove fsmonitor config settings to a new and opaque\n`struct fsmonitor_settings` structure. Add a lazily-loaded pointer\nto this into `struct repo_settings`\n\nCreate an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\nrepresent the state of fsmonitor. This lets us represent which, if\nany, fsmonitor provider (hook or IPC) is enabled.\n\nCreate `fsm_settings__get_*()` getters to lazily look up fsmonitor-\nrelated config settings.\n\nGet rid of the `core_fsmonitor` global variable. Move the code to\nlookup the existing `core.fsmonitor` config value into the fsmonitor\nsettings.\n\nCreate a hook pathname variable in `struct fsmonitor-settings` and\nonly set it when in hook mode.\n\nExtend the definition of `core.fsmonitor` to be either a boolean\nor a hook pathname. When true, the builtin FSMonitor is used.\nWhen false or unset, no FSMonitor (neither builtin nor hook) is\nused.\n\nThe existing `core_fsmonitor` global variable was used to store the\npathname to the fsmonitor hook *and* it was used as a boolean to see\nif fsmonitor was enabled. This dual usage and global visibility leads\nto confusion when we add the IPC-based provider. So lets hide the\ndetails in fsmonitor-settings.c and let it decide which provider to\nuse in the case of multiple settings. This avoids cluttering up\nrepo-settings.c with these private details.\n\nA future commit in builtin-fsmonitor series will add the ability to\ndisqualify worktrees for various reasons, such as being mounted from a\nremote volume, where fsmonitor should not be started. Having the\nconfig settings hidden in fsmonitor-settings.c allows such worktree\nrestrictions to override the config values used.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n builtin/update-index.c | 7 ++-\n cache.h | 1 -\n config.c | 14 -----\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-settings.c | 115 +++++++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 21 ++++++++\n fsmonitor.c | 63 +++++++++++++---------\n fsmonitor.h | 15 ++++--\n repository.h | 3 ++\n t/README | 4 +-\n 12 files changed, 197 insertions(+), 49 deletions(-)\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\ndiff --git a/Makefile b/Makefile\nindex a19d850e716..707a56d4c11 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -908,6 +908,7 @@ LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += fsmonitor-ipc.o\n+LIB_OBJS += fsmonitor-settings.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex aafe7eeac2a..876112abb21 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1236,14 +1236,17 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\n \t\t\t\t\"enable fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode > FSMONITOR_MODE_DISABLED)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex 04d4d2db25c..aaf334e2aa4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -999,7 +999,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex 383b1a4885b..3f9b0739a78 100644\n--- a/config.c\n+++ b/config.c\n@@ -2626,20 +2626,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex bb49baf1ee0..7654f61c634 100644\n--- a/config.h\n+++ b/config.h\n@@ -597,7 +597,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a5..00682e638d7 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nnew file mode 100644\nindex 00000000000..3b54e7a51f6\n--- /dev/null\n+++ b/fsmonitor-settings.c\n@@ -0,0 +1,115 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+/*\n+ * We keep this structure defintion private and have getters\n+ * for all fields so that we can lazy load it as needed.\n+ */\n+struct fsmonitor_settings {\n+\tenum fsmonitor_mode mode;\n+\tchar *hook_path;\n+};\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s;\n+\tconst char *const_str;\n+\tint bool_value;\n+\n+\tif (r->settings.fsmonitor)\n+\t\treturn;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\n+\tr->settings.fsmonitor = s;\n+\n+\tfsm_settings__set_disabled(r);\n+\n+\t/*\n+\t * Overload the existing \"core.fsmonitor\" config setting (which\n+\t * has historically been either unset or a hook pathname) to\n+\t * now allow a boolean value to enable the builtin FSMonitor\n+\t * or to turn everything off. (This does imply that you can't\n+\t * use a hook script named \"true\" or \"false\", but that's OK.)\n+\t */\n+\tswitch (repo_config_get_maybe_bool(r, \"core.fsmonitor\", &bool_value)) {\n+\n+\tcase 0: /* config value was set to */\n+\t\tif (bool_value)\n+\t\t\tfsm_settings__set_ipc(r);\n+\t\treturn;\n+\n+\tcase 1: /* config value was unset */\n+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n+\t\tbreak;\n+\n+\tcase -1: /* config value set to an arbitrary string */\n+\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n+\t\t\treturn; /* should not happen */\n+\t\tbreak;\n+\n+\tdefault: /* should not happen */\n+\t\treturn;\n+\t}\n+\n+\tif (!const_str || !*const_str)\n+\t\treturn;\n+\n+\tfsm_settings__set_hook(r, const_str);\n+}\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->mode;\n+}\n+\n+const char *fsm_settings__get_hook_path(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->hook_path;\n+}\n+\n+void fsm_settings__set_ipc(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\n+\n+void fsm_settings__set_hook(struct repository *r, const char *path)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+\tr->settings.fsmonitor->hook_path = strdup(path);\n+}\n+\n+void fsm_settings__set_disabled(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nnew file mode 100644\nindex 00000000000..a4c5d7b4889\n--- /dev/null\n+++ b/fsmonitor-settings.h\n@@ -0,0 +1,21 @@\n+#ifndef FSMONITOR_SETTINGS_H\n+#define FSMONITOR_SETTINGS_H\n+\n+struct repository;\n+\n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n+\tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r);\n+void fsm_settings__set_hook(struct repository *r, const char *path);\n+void fsm_settings__set_disabled(struct repository *r);\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+const char *fsm_settings__get_hook_path(struct repository *r);\n+\n+struct fsmonitor_settings;\n+\n+#endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 448d0ee33f5..0e961b74d82 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tstrvec_push(&cp.args, fsm_settings__get_hook_path(r));\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -225,17 +229,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \tchar *buf;\n \tunsigned int i;\n \tint is_trivial = 0;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (fsm_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -243,13 +258,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -280,7 +296,8 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t\tif (query_success)\n \t\t\t\tis_trivial = query_result.buf[0] == '/';\n@@ -290,9 +307,12 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n \t\t\t\t\t \"query/trivial-response\", 1);\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tfsm_settings__get_hook_path(r));\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t fsm_settings__get_hook_path(r),\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/*\n@@ -429,7 +449,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tint fsmonitor_enabled = (fsm_settings__get_mode(istate->repo)\n+\t\t\t\t > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -449,16 +470,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..3f41f653691 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -3,6 +3,7 @@\n \n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"fsmonitor-settings.h\"\n \n extern struct trace_key trace_fsmonitor;\n \n@@ -57,7 +58,10 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +71,10 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +90,9 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repository.h b/repository.h\nindex ca837cb9e91..9bbb4659cc8 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n #include \"path.h\"\n \n struct config_set;\n+struct fsmonitor_settings;\n struct git_hash_algo;\n struct index_state;\n struct lock_file;\n@@ -35,6 +36,8 @@ struct repo_settings {\n \tint command_requires_full_index;\n \tint sparse_index;\n \n+\tstruct fsmonitor_settings *fsmonitor; /* lazily loaded */\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex f48e0542cdc..9ffea1d3147 100644\n--- a/t/README\n+++ b/t/README\n@@ -405,8 +405,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code paths for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\n-- \ngitgitgadget\n\n"},{"id":"449960","messageId":"ce42d5bbaf605cd39c2a704bafbf9f6c567004f1.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 05/30] fsmonitor: document builtin fsmonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:07Z","receivedAt":"2022-03-01T18:43:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDocument how `core.fsmonitor` can be set to a boolean to enable\nor disable the builtin FSMonitor.\n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to refer to it.\n\nCreate `git-fsmonitor--daemon` manual page and describe its features.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Documentation/config/core.txt | 60 +++++++++++++++-----\n Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n Documentation/git-update-index.txt | 8 ++-\n 3 files changed, 126 insertions(+), 17 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..6303c36c7ed 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,54 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+\tOtherwise, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note that if you concurrently use multiple versions of Git, such\n+as one version on the command line and another version in an IDE\n+tool, that the definition of `core.fsmonitor` was extended to\n+allow boolean values in addition to hook pathnames. Git versions\n+2.35.1 and prior will not understand the boolean values and will\n+consider the \"true\" or \"false\" values as hook pathnames to be\n+invoked. Git versions 2.26 thru 2.35.1 default to hook protocol\n+V2 and will fall back to no fsmonitor (full scan). Git versions\n+prior to 2.26 default to hook protocol V1 and will silently\n+assume there were no changes to report (no scan), so status\n+commands may report incomplete results. For this reason, it is\n+best to upgrade all of your Git versions before using the built-in\n+file system monitor.\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\nnew file mode 100644\nindex 00000000000..0fedf5a4565\n--- /dev/null\n+++ b/Documentation/git-fsmonitor--daemon.txt\n@@ -0,0 +1,75 @@\n+git-fsmonitor--daemon(1)\n+========================\n+\n+NAME\n+----\n+git-fsmonitor--daemon - A Built-in File System Monitor\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git fsmonitor--daemon' start\n+'git fsmonitor--daemon' run\n+'git fsmonitor--daemon' stop\n+'git fsmonitor--daemon' status\n+\n+DESCRIPTION\n+-----------\n+\n+A daemon to watch the working directory for file and directory\n+changes using platform-specific file system notification facilities.\n+\n+This daemon communicates directly with commands like `git status`\n+using the link:technical/api-simple-ipc.html[simple IPC] interface\n+instead of the slower linkgit:githooks[5] interface.\n+\n+This daemon is built into Git so that no third-party tools are\n+required.\n+\n+OPTIONS\n+-------\n+\n+start::\n+\tStarts a daemon in the background.\n+\n+run::\n+\tRuns a daemon in the foreground.\n+\n+stop::\n+\tStops the daemon running in the current working\n+\tdirectory, if present.\n+\n+status::\n+\tExits with zero status if a daemon is watching the\n+\tcurrent working directory.\n+\n+REMARKS\n+-------\n+\n+This daemon is a long running process used to watch a single working\n+directory and maintain a list of the recently changed files and\n+directories. Performance of commands such as `git status` can be\n+increased if they just ask for a summary of changes to the working\n+directory and can avoid scanning the disk.\n+\n+When `core.fsmonitor` is set to `true` (see linkgit:git-config[1])\n+commands, such as `git status`, will ask the daemon for changes and\n+automatically start it (if necessary).\n+\n+For more information see the \"File System Monitor\" section in\n+linkgit:git-update-index[1].\n+\n+CAVEATS\n+-------\n+\n+The fsmonitor daemon does not currently know about submodules and does\n+not know to filter out file system events that happen within a\n+submodule. If fsmonitor daemon is watching a super repo and a file is\n+modified within the working directory of a submodule, it will report\n+the change (as happening against the super repo). However, the client\n+will properly ignore these extra events, so performance may be affected\n+but it will not cause an incorrect result.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..53ea48a04e2 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -509,8 +511,8 @@ looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n+linkgit:git-config[1]) than using the `--fsmonitor` option to `git\n+update-index` in each repository, especially if you want to do so\n across all repositories you use, because you can set the configuration\n variable in your `$HOME/.gitconfig` just once and have it affect all\n repositories you touch.\n-- \ngitgitgadget\n\n"},{"id":"449961","messageId":"9ce938c69b5a3e47b8c0a68379892955c2d7da88.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 06/30] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:08Z","receivedAt":"2022-03-01T18:43:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a built-in file system monitoring daemon that can be used by\nthe existing `fsmonitor` feature (protocol API and index extension)\nto improve the performance of various Git commands, such as `status`.\n\nThe `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\nprovides an alternative to hook access to existing fsmonitors such\nas `watchman`.\n\nThis commit merely adds the new command without any functionality.\n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 46 +++++++++++++++++++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 50 insertions(+)\n create mode 100644 builtin/fsmonitor--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex f817c509ec0..e81de1063a4 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -72,6 +72,7 @@\n /git-format-patch\n /git-fsck\n /git-fsck-objects\n+/git-fsmonitor--daemon\n /git-gc\n /git-get-tar-commit-id\n /git-grep\ndiff --git a/Makefile b/Makefile\nindex 707a56d4c11..5af1d5b112e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1114,6 +1114,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n+BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex 83379f3832c..40e9ecc8485 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\n int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nnew file mode 100644\nindex 00000000000..f0498793379\n--- /dev/null\n+++ b/builtin/fsmonitor--daemon.c\n@@ -0,0 +1,46 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"simple-ipc.h\"\n+#include \"khash.h\"\n+\n+static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tNULL\n+};\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *subcmd;\n+\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\tif (argc != 1)\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\tsubcmd = argv[0];\n+\n+\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n+}\n+\n+#else\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n+}\n+#endif\ndiff --git a/git.c b/git.c\nindex a25940d72e8..3d8e48cf555 100644\n--- a/git.c\n+++ b/git.c\n@@ -537,6 +537,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"449962","messageId":"92d3e54da7f1e6148ba521b5377af17900f7d42d.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 08/30] compat/fsmonitor/fsm-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:10Z","receivedAt":"2022-03-01T18:44:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty filesystem listener backend for fsmonitor--daemon on Windows.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 13 ++++++++\n compat/fsmonitor/fsm-listen-win32.c | 21 +++++++++++++\n compat/fsmonitor/fsm-listen.h | 49 +++++++++++++++++++++++++++++\n config.mak.uname | 10 ++++++\n contrib/buildsystems/CMakeLists.txt | 7 +++++\n repo-settings.c | 1 +\n 6 files changed, 101 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n\ndiff --git a/Makefile b/Makefile\nindex 5af1d5b112e..26567d4f772 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -470,6 +470,11 @@ all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n+# If your platform supports a built-in fsmonitor backend, set\n+# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n+# `compat/fsmonitor/fsm-listen-.c` that implements the\n+# `fsm_listen__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1968,6 +1973,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n \tCOMPAT_OBJS += compat/access.o\n endif\n \n+ifdef FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2887,6 +2897,9 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n \t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n+ifdef FSMONITOR_DAEMON_BACKEND\n+\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nnew file mode 100644\nindex 00000000000..916cbea254f\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nnew file mode 100644\nindex 00000000000..f0539349baf\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -0,0 +1,49 @@\n+#ifndef FSM_LISTEN_H\n+#define FSM_LISTEN_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * fsmonitor_fs_listener thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread AFTER joining the listener.\n+ */\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to watch for\n+ * filesystem events. This will run in the fsmonitor_fs_listen thread.\n+ *\n+ * It should call `ipc_server_stop_async()` if the listener thread\n+ * prematurely terminates (because of a filesystem error or if it\n+ * detects that the .git directory has been deleted). (It should NOT\n+ * do so if the listener thread receives a normal shutdown signal from\n+ * the IPC layer.)\n+ *\n+ * It should set `state->error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the fsmonitor listener thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_LISTEN_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 4352ea39e9b..26074f56bed 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -435,6 +435,11 @@ ifeq ($(uname_S),Windows)\n \t# so we don't need this:\n \t#\n \t# SNPRINTF_RETURNS_BOGUS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -619,6 +624,11 @@ ifeq ($(uname_S),MINGW)\n \tNO_STRTOUMAX = YesPlease\n \tNO_MKDTEMP = YesPlease\n \tNO_SVN_TESTS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex e44232f85d3..0963629db7f 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -285,6 +285,13 @@ else()\n \tendif()\n endif()\n \n+if(SUPPORTS_SIMPLE_IPC)\n+\tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\tendif()\n+endif()\n+\n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n \n #header checks\ndiff --git a/repo-settings.c b/repo-settings.c\nindex b4fbd16cdcc..2dfcb2b6542 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"midx.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n \n static void repo_cfg_bool(struct repository *r, const char *key, int *dest,\n \t\t\t int def)\n-- \ngitgitgadget\n\n"},{"id":"449963","messageId":"3bd7e6a051e7fca1ac2a78e038587de10fd9401a.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 07/30] fsmonitor--daemon: implement 'stop' and 'status' commands","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:09Z","receivedAt":"2022-03-01T18:44:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `stop` and `status` client commands to control and query the\nstatus of a `fsmonitor--daemon` server process (and implicitly start a\nserver process if necessary).\n\nLater commits will implement the actual server and monitor the file\nsystem.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 51 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex f0498793379..5e3178b8bdd 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,10 +7,55 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon stop\"),\n+\tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n+ * and wait for it to shutdown.\n+ */\n+static int do_as_client__send_stop(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"quit\", &answer);\n+\n+\t/* The quit command does not return any response data. */\n+\tstrbuf_release(&answer);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\twhile (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tsleep_millisec(50);\n+\ttrace2_region_leave(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\n+\treturn 0;\n+}\n+\n+static int do_as_client__status(void)\n+{\n+\tenum ipc_active_state state = fsmonitor_ipc__get_state();\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tprintf(_(\"fsmonitor-daemon is watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tprintf(_(\"fsmonitor-daemon is not watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 1;\n+\t}\n+}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n@@ -28,6 +73,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (!strcmp(subcmd, \"stop\"))\n+\t\treturn !!do_as_client__send_stop();\n+\n+\tif (!strcmp(subcmd, \"status\"))\n+\t\treturn !!do_as_client__status();\n+\n \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"449964","messageId":"a3f94627cf6bd5a66df4fd09586cbeb7b764e2d1.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 10/30] fsmonitor--daemon: implement 'run' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:12Z","receivedAt":"2022-03-01T18:44:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `run` command to try to begin listening for file system events.\n\nThis version defines the thread structure with a single fsmonitor_fs_listen\nthread to watch for file system events and a simple IPC thread pool to\nwatch for connection from Git clients over a well-known named pipe or\nUnix domain socket.\n\nThis commit does not actually do anything yet because the platform\nbackends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 34 ++++++\n 2 files changed, 261 insertions(+), 1 deletion(-)\n create mode 100644 fsmonitor--daemon.h\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 5e3178b8bdd..950614f82f6 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,16 +3,52 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Global state loaded from config.\n+ */\n+#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n+static int fsmonitor__ipc_threads = 8;\n+\n+#define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n+static int fsmonitor__announce_startup = 0;\n+\n+static int fsmonitor_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 1)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__IPC_THREADS, i);\n+\t\tfsmonitor__ipc_threads = i;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n+\t\tint is_bool;\n+\t\tint i = git_config_bool_or_int(var, value, &is_bool);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' not bool or int: %d\"),\n+\t\t\t\t var, i);\n+\t\tfsmonitor__announce_startup = i;\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n /*\n * Acting as a CLIENT.\n *\n@@ -57,15 +93,198 @@ static int do_as_client__status(void)\n \t}\n }\n \n+static ipc_server_application_cb handle_client;\n+\n+static int handle_client(void *data,\n+\t\t\t const char *command, size_t command_len,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\t/* struct fsmonitor_daemon_state *state = data; */\n+\tint result;\n+\n+\t/*\n+\t * The Simple IPC API now supports {char*, len} arguments, but\n+\t * FSMonitor always uses proper null-terminated strings, so\n+\t * we can ignore the command_len argument. (Trust, but verify.)\n+\t */\n+\tif (command_len != strlen(command))\n+\t\tBUG(\"FSMonitor assumes text messages\");\n+\n+\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n+\n+\tresult = 0; /* TODO Do something here. */\n+\n+\ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n+\n+\treturn result;\n+}\n+\n+static void *fsm_listen__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-listen\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Watching: worktree '%s'\",\n+\t\t\t state->path_worktree_watch.buf);\n+\tif (state->nr_paths_watching > 1)\n+\t\ttrace_printf_key(&trace_fsmonitor, \"Watching: gitdir '%s'\",\n+\t\t\t\t state->path_gitdir_watch.buf);\n+\n+\tfsm_listen__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct ipc_server_opts ipc_opts = {\n+\t\t.nr_threads = fsmonitor__ipc_threads,\n+\n+\t\t/*\n+\t\t * We know that there are no other active threads yet,\n+\t\t * so we can let the IPC layer temporarily chdir() if\n+\t\t * it needs to when creating the server side of the\n+\t\t * Unix domain socket.\n+\t\t */\n+\t\t.uds_disallow_chdir = 0\n+\t};\n+\n+\t/*\n+\t * Start the IPC thread pool before the we've started the file\n+\t * system event listener thread so that we have the IPC handle\n+\t * before we need it.\n+\t */\n+\tif (ipc_server_run_async(&state->ipc_server_data,\n+\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t handle_client, state))\n+\t\treturn error_errno(\n+\t\t\t_(\"could not start IPC thread pool on '%s'\"),\n+\t\t\tfsmonitor_ipc__get_path());\n+\n+\t/*\n+\t * Start the fsmonitor listener thread to collect filesystem\n+\t * events.\n+\t */\n+\tif (pthread_create(&state->listener_thread, NULL,\n+\t\t\t fsm_listen__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\tipc_server_await(state->ipc_server_data);\n+\n+\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t}\n+\n+\t/*\n+\t * The daemon is now fully functional in background threads.\n+\t * Wait for the IPC thread pool to shutdown (whether by client\n+\t * request or from filesystem activity).\n+\t */\n+\tipc_server_await(state->ipc_server_data);\n+\n+\t/*\n+\t * The fsmonitor listener thread may have received a shutdown\n+\t * event from the IPC thread pool, but it doesn't hurt to tell\n+\t * it again. And wait for it to shutdown.\n+\t */\n+\tfsm_listen__stop_async(state);\n+\tpthread_join(state->listener_thread, NULL);\n+\n+\treturn state->error_code;\n+}\n+\n+static int fsmonitor_run_daemon(void)\n+{\n+\tstruct fsmonitor_daemon_state state;\n+\tint err;\n+\n+\tmemset(&state, 0, sizeof(state));\n+\n+\tpthread_mutex_init(&state.main_lock, NULL);\n+\tstate.error_code = 0;\n+\tstate.current_token_data = NULL;\n+\n+\t/* Prepare to (recursively) watch the directory. */\n+\tstrbuf_init(&state.path_worktree_watch, 0);\n+\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n+\tstate.nr_paths_watching = 1;\n+\n+\t/*\n+\t * We create and delete cookie files somewhere inside the .git\n+\t * directory to help us keep sync with the file system. If\n+\t * \".git\" is not a directory, then is not inside the\n+\t * cone of , so set up a second watch to watch\n+\t * the so that we get events for the cookie files.\n+\t */\n+\tstrbuf_init(&state.path_gitdir_watch, 0);\n+\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n+\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n+\tif (!is_directory(state.path_gitdir_watch.buf)) {\n+\t\tstrbuf_reset(&state.path_gitdir_watch);\n+\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n+\t\tstate.nr_paths_watching = 2;\n+\t}\n+\n+\t/*\n+\t * Confirm that we can create platform-specific resources for the\n+\t * filesystem listener before we bother starting all the threads.\n+\t */\n+\tif (fsm_listen__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize listener thread\"));\n+\t\tgoto done;\n+\t}\n+\n+\terr = fsmonitor_run_daemon_1(&state);\n+\n+done:\n+\tpthread_mutex_destroy(&state.main_lock);\n+\tfsm_listen__dtor(&state);\n+\n+\tipc_server_free(state.ipc_server_data);\n+\n+\tstrbuf_release(&state.path_worktree_watch);\n+\tstrbuf_release(&state.path_gitdir_watch);\n+\n+\treturn err;\n+}\n+\n+static int try_to_run_foreground_daemon(void)\n+{\n+\t/*\n+\t * Technically, we don't need to probe for an existing daemon\n+\t * process, since we could just call `fsmonitor_run_daemon()`\n+\t * and let it fail if the pipe/socket is busy.\n+\t *\n+\t * However, this method gives us a nicer error message for a\n+\t * common error case.\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n+\t\t the_repository->worktree);\n+\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"running fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n+\n+\treturn !!fsmonitor_run_daemon();\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n \n \tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"ipc-threads\",\n+\t\t\t &fsmonitor__ipc_threads,\n+\t\t\t N_(\"use ipc worker threads\")),\n \t\tOPT_END()\n \t};\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(fsmonitor_config, NULL);\n \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n@@ -73,6 +292,13 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (fsmonitor__ipc_threads < 1)\n+\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n+\t\t fsmonitor__ipc_threads);\n+\n+\tif (!strcmp(subcmd, \"run\"))\n+\t\treturn !!try_to_run_foreground_daemon();\n+\n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n \ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nnew file mode 100644\nindex 00000000000..3009c1a83de\n--- /dev/null\n+++ b/fsmonitor--daemon.h\n@@ -0,0 +1,34 @@\n+#ifndef FSMONITOR_DAEMON_H\n+#define FSMONITOR_DAEMON_H\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+#include \"thread-utils.h\"\n+\n+struct fsmonitor_batch;\n+struct fsmonitor_token_data;\n+\n+struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+\n+struct fsmonitor_daemon_state {\n+\tpthread_t listener_thread;\n+\tpthread_mutex_t main_lock;\n+\n+\tstruct strbuf path_worktree_watch;\n+\tstruct strbuf path_gitdir_watch;\n+\tint nr_paths_watching;\n+\n+\tstruct fsmonitor_token_data *current_token_data;\n+\n+\tint error_code;\n+\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\n+\tstruct ipc_server_data *ipc_server_data;\n+};\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"449965","messageId":"f12192212702b2ee4543b666136143cf18522a3a.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 09/30] compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:11Z","receivedAt":"2022-03-01T18:44:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty implementation of fsmonitor--daemon\nbackend for Darwin (aka MacOS).\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-darwin.c | 20 ++++++++++++++++++++\n config.mak.uname | 10 ++++++++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 33 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nnew file mode 100644\nindex 00000000000..c84e3344ab9\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -0,0 +1,20 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 26074f56bed..501970902da 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -157,6 +157,16 @@ ifeq ($(uname_S),Darwin)\n \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n \t\tendif\n \tendif\n+\n+\t# The builtin FSMonitor on MacOS builds upon Simple-IPC. Both require\n+\t# Unix domain sockets and PThreads.\n+\tifndef NO_PTHREADS\n+\tifndef NO_UNIX_SOCKETS\n+\tFSMONITOR_DAEMON_BACKEND = darwin\n+\tendif\n+\tendif\n+\n+\tBASIC_LDFLAGS += -framework CoreServices\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 0963629db7f..ee0d7257b77 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"449966","messageId":"8de40b0fe8b097534630cfac28fc7c84dd304463.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 11/30] fsmonitor--daemon: implement 'start' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:13Z","receivedAt":"2022-03-01T18:44:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement 'git fsmonitor--daemon start' command. This command starts\nan instance of 'git fsmonitor--daemon run' in the background using\nthe new 'start_bg_command()' function.\n\nWe avoid the fork-and-call technique on Unix systems in favor of a\nfork-and-exec technique. This gives us more uniform Trace2 child-*\nevents. It also makes our usage more consistent with Windows usage.\n\nOn Windows, teach 'git fsmonitor--daemon run' to optionally call\n'FreeConsole()' to release handles to the inherited Win32 console\n(despite being passed invalid handles for stdin/out/err). Without\nthis, command prompts and powershell terminal windows could hang\nin \"exit\" until the last background child process exited or released\ntheir Win32 console handle. (This was not seen with git-bash shells\nbecause they don't have a Win32 console attached to them.)\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 109 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 107 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 950614f82f6..2f721aae016 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -9,6 +9,7 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon start []\"),\n \tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n@@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n #define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n static int fsmonitor__ipc_threads = 8;\n \n+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n+static int fsmonitor__start_timeout_sec = 60;\n+\n #define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n static int fsmonitor__announce_startup = 0;\n \n@@ -36,6 +40,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n+\t\tfsmonitor__start_timeout_sec = i;\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n \t\tint is_bool;\n \t\tint i = git_config_bool_or_int(var, value, &is_bool);\n@@ -250,7 +263,7 @@ done:\n \treturn err;\n }\n \n-static int try_to_run_foreground_daemon(void)\n+static int try_to_run_foreground_daemon(int detach_console)\n {\n \t/*\n \t * Technically, we don't need to probe for an existing daemon\n@@ -270,17 +283,106 @@ static int try_to_run_foreground_daemon(void)\n \t\tfflush(stderr);\n \t}\n \n+#ifdef GIT_WINDOWS_NATIVE\n+\tif (detach_console)\n+\t\tFreeConsole();\n+#endif\n+\n \treturn !!fsmonitor_run_daemon();\n }\n \n+static start_bg_wait_cb bg_wait_cb;\n+\n+static int bg_wait_cb(const struct child_process *cp, void *cb_data)\n+{\n+\tenum ipc_active_state s = fsmonitor_ipc__get_state();\n+\n+\tswitch (s) {\n+\tcase IPC_STATE__LISTENING:\n+\t\t/* child is \"ready\" */\n+\t\treturn 0;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\t/* give child more time */\n+\t\treturn 1;\n+\n+\tdefault:\n+\tcase IPC_STATE__INVALID_PATH:\n+\tcase IPC_STATE__OTHER_ERROR:\n+\t\t/* all the time in world won't help */\n+\t\treturn -1;\n+\t}\n+}\n+\n+static int try_to_start_background_daemon(void)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tenum start_bg_result sbgr;\n+\n+\t/*\n+\t * Before we try to create a background daemon process, see\n+\t * if a daemon process is already listening. This makes it\n+\t * easier for us to report an already-listening error to the\n+\t * console, since our spawn/daemon can only report the success\n+\t * of creating the background process (and not whether it\n+\t * immediately exited).\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n+\t\t the_repository->worktree);\n+\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"starting fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n+\n+\tcp.git_cmd = 1;\n+\n+\tstrvec_push(&cp.args, \"fsmonitor--daemon\");\n+\tstrvec_push(&cp.args, \"run\");\n+\tstrvec_push(&cp.args, \"--detach\");\n+\tstrvec_pushf(&cp.args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n+\n+\tcp.no_stdin = 1;\n+\tcp.no_stdout = 1;\n+\tcp.no_stderr = 1;\n+\n+\tsbgr = start_bg_command(&cp, bg_wait_cb, NULL,\n+\t\t\t\tfsmonitor__start_timeout_sec);\n+\n+\tswitch (sbgr) {\n+\tcase SBGR_READY:\n+\t\treturn 0;\n+\n+\tdefault:\n+\tcase SBGR_ERROR:\n+\tcase SBGR_CB_ERROR:\n+\t\treturn error(\"daemon failed to start\");\n+\n+\tcase SBGR_TIMEOUT:\n+\t\treturn error(\"daemon not online yet\");\n+\n+\tcase SBGR_DIED:\n+\t\treturn error(\"daemon terminated\");\n+\t}\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tint detach_console = 0;\n \n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"detach\", &detach_console, N_(\"detach from console\")),\n \t\tOPT_INTEGER(0, \"ipc-threads\",\n \t\t\t &fsmonitor__ipc_threads,\n \t\t\t N_(\"use ipc worker threads\")),\n+\t\tOPT_INTEGER(0, \"start-timeout\",\n+\t\t\t &fsmonitor__start_timeout_sec,\n+\t\t\t N_(\"max seconds to wait for background daemon startup\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -296,8 +398,11 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tif (!strcmp(subcmd, \"start\"))\n+\t\treturn !!try_to_start_background_daemon();\n+\n \tif (!strcmp(subcmd, \"run\"))\n-\t\treturn !!try_to_run_foreground_daemon();\n+\t\treturn !!try_to_run_foreground_daemon(detach_console);\n \n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n-- \ngitgitgadget\n\n"},{"id":"449967","messageId":"bd492d13fb154dd1dcae4e85d088afc0c68e93f6.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 14/30] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:16Z","receivedAt":"2022-03-01T18:44:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to build a list of changed paths and associate\nthem with a token-id. This will be used by the platform-specific\nbackends to accumulate changed paths in response to filesystem events.\n\nThe platform-specific file system listener thread receives file system\nevents containing one or more changed pathnames (with whatever\nbucketing or grouping that is convenient for the file system). These\npaths are accumulated (without locking) by the file system layer into\na `fsmonitor_batch`.\n\nWhen the file system layer has drained the kernel event queue, it will\n\"publish\" them to our token queue and make them visible to concurrent\nclient worker threads. The token layer is free to combine and/or de-dup\npaths within these batches for efficient presentation to clients.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 230 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 40 +++++++\n 2 files changed, 268 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex c8d1509d87d..7c44b979035 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -181,17 +181,27 @@ struct fsmonitor_token_data {\n \tuint64_t client_ref_count;\n };\n \n+struct fsmonitor_batch {\n+\tstruct fsmonitor_batch *next;\n+\tuint64_t batch_seq_nr;\n+\tconst char **interned_paths;\n+\tsize_t nr, alloc;\n+\ttime_t pinned_time;\n+};\n+\n static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n {\n \tstatic int test_env_value = -1;\n \tstatic uint64_t flush_count = 0;\n \tstruct fsmonitor_token_data *token;\n+\tstruct fsmonitor_batch *batch;\n \n \tCALLOC_ARRAY(token, 1);\n+\tbatch = fsmonitor_batch__new();\n \n \tstrbuf_init(&token->token_id, 0);\n-\ttoken->batch_head = NULL;\n-\ttoken->batch_tail = NULL;\n+\ttoken->batch_head = batch;\n+\ttoken->batch_tail = batch;\n \ttoken->client_ref_count = 0;\n \n \tif (test_env_value < 0)\n@@ -217,9 +227,143 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n \t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n \t}\n \n+\t/*\n+\t * We created a new and are starting a new series\n+\t * of tokens with a zero .\n+\t *\n+\t * Since clients cannot guess our new (non test) \n+\t * they will always receive a trivial response (because of the\n+\t * mismatch on the ). The trivial response will\n+\t * tell them our new so that subsequent requests\n+\t * will be relative to our new series. (And when sending that\n+\t * response, we pin the current head of the batch list.)\n+\t *\n+\t * Even if the client correctly guesses the , their\n+\t * request of \"builtin::0\" asks for all changes MORE\n+\t * RECENT than batch/bin 0.\n+\t *\n+\t * This implies that it is a waste to accumulate paths in the\n+\t * initial batch/bin (because they will never be transmitted).\n+\t *\n+\t * So the daemon could be running for days and watching the\n+\t * file system, but doesn't need to actually accumulate any\n+\t * paths UNTIL we need to set a reference point for a later\n+\t * relative request.\n+\t *\n+\t * However, it is very useful for testing to always have a\n+\t * reference point set. Pin batch 0 to force early file system\n+\t * events to accumulate.\n+\t */\n+\tif (test_env_value)\n+\t\tbatch->pinned_time = time(NULL);\n+\n \treturn token;\n }\n \n+struct fsmonitor_batch *fsmonitor_batch__new(void)\n+{\n+\tstruct fsmonitor_batch *batch;\n+\n+\tCALLOC_ARRAY(batch, 1);\n+\n+\treturn batch;\n+}\n+\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch)\n+{\n+\twhile (batch) {\n+\t\tstruct fsmonitor_batch *next = batch->next;\n+\n+\t\t/*\n+\t\t * The actual strings within the array of this batch\n+\t\t * are interned, so we don't own them. We only own\n+\t\t * the array.\n+\t\t */\n+\t\tfree(batch->interned_paths);\n+\t\tfree(batch);\n+\n+\t\tbatch = next;\n+\t}\n+}\n+\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n+\t\t\t const char *path)\n+{\n+\tconst char *interned_path = strintern(path);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n+\n+\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n+\tbatch->interned_paths[batch->nr++] = interned_path;\n+}\n+\n+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n+\t\t\t\t const struct fsmonitor_batch *batch_src)\n+{\n+\tsize_t k;\n+\n+\tALLOC_GROW(batch_dest->interned_paths,\n+\t\t batch_dest->nr + batch_src->nr + 1,\n+\t\t batch_dest->alloc);\n+\n+\tfor (k = 0; k < batch_src->nr; k++)\n+\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n+\t\t\tbatch_src->interned_paths[k];\n+}\n+\n+static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n+{\n+\tif (!token)\n+\t\treturn;\n+\n+\tassert(token->client_ref_count == 0);\n+\n+\tstrbuf_release(&token->token_id);\n+\n+\tfsmonitor_batch__free_list(token->batch_head);\n+\n+\tfree(token);\n+}\n+\n+/*\n+ * Flush all of our cached data about the filesystem. Call this if we\n+ * lose sync with the filesystem and miss some notification events.\n+ *\n+ * [1] If we are missing events, then we no longer have a complete\n+ * history of the directory (relative to our current start token).\n+ * We should create a new token and start fresh (as if we just\n+ * booted up).\n+ *\n+ * If there are no concurrent threads reading the current token data\n+ * series, we can free it now. Otherwise, let the last reader free\n+ * it.\n+ *\n+ * Either way, the old token data series is no longer associated with\n+ * our state data.\n+ */\n+static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct fsmonitor_token_data *free_me = NULL;\n+\tstruct fsmonitor_token_data *new_one = NULL;\n+\n+\tnew_one = fsmonitor_new_token_data();\n+\n+\tif (state->current_token_data->client_ref_count == 0)\n+\t\tfree_me = state->current_token_data;\n+\tstate->current_token_data = new_one;\n+\n+\tfsmonitor_free_token_data(free_me);\n+}\n+\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\tpthread_mutex_lock(&state->main_lock);\n+\twith_lock__do_force_resync(state);\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -329,6 +473,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n \n+/*\n+ * We try to combine small batches at the front of the batch-list to avoid\n+ * having a long list. This hopefully makes it a little easier when we want\n+ * to truncate and maintain the list. However, we don't want the paths array\n+ * to just keep growing and growing with realloc, so we insert an arbitrary\n+ * limit.\n+ */\n+#define MY_COMBINE_LIMIT (1024)\n+\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names)\n+{\n+\tif (!batch && !cookie_names->nr)\n+\t\treturn;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (batch) {\n+\t\tstruct fsmonitor_batch *head;\n+\n+\t\thead = state->current_token_data->batch_head;\n+\t\tif (!head) {\n+\t\t\tBUG(\"token does not have batch\");\n+\t\t} else if (head->pinned_time) {\n+\t\t\t/*\n+\t\t\t * We cannot alter the current batch list\n+\t\t\t * because:\n+\t\t\t *\n+\t\t\t * [a] it is being transmitted to at least one\n+\t\t\t * client and the handle_client() thread has a\n+\t\t\t * ref-count, but not a lock on the batch list\n+\t\t\t * starting with this item.\n+\t\t\t *\n+\t\t\t * [b] it has been transmitted in the past to\n+\t\t\t * at least one client such that future\n+\t\t\t * requests are relative to this head batch.\n+\t\t\t *\n+\t\t\t * So, we can only prepend a new batch onto\n+\t\t\t * the front of the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else if (!head->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * Batch 0 is unpinned. See the note in\n+\t\t\t * `fsmonitor_new_token_data()` about why we\n+\t\t\t * don't need to accumulate these paths.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n+\t\t\t/*\n+\t\t\t * The head batch in the list has never been\n+\t\t\t * transmitted to a client, but folding the\n+\t\t\t * contents of the new batch onto it would\n+\t\t\t * exceed our arbitrary limit, so just prepend\n+\t\t\t * the new batch onto the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We are free to add the paths in the given\n+\t\t\t * batch onto the end of the current head batch.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__combine(head, batch);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -343,6 +562,13 @@ static void *fsm_listen__thread_proc(void *_state)\n \n \tfsm_listen__loop(state);\n \n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (state->current_token_data &&\n+\t state->current_token_data->client_ref_count == 0)\n+\t\tfsmonitor_free_token_data(state->current_token_data);\n+\tstate->current_token_data = NULL;\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n \ttrace2_thread_exit();\n \treturn NULL;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 8c3a71a48bd..010fbfe60e9 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -12,6 +12,27 @@\n struct fsmonitor_batch;\n struct fsmonitor_token_data;\n \n+/*\n+ * Create a new batch of path(s). The returned batch is considered\n+ * private and not linked into the fsmonitor daemon state. The caller\n+ * should fill this batch with one or more paths and then publish it.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__new(void);\n+\n+/*\n+ * Free the list of batches starting with this one.\n+ */\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n+\n+/*\n+ * Add this path to this batch of modified files.\n+ *\n+ * The batch should be private and NOT (yet) linked into the fsmonitor\n+ * daemon state and therefore not yet visible to worker threads and so\n+ * no locking is required.\n+ */\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n+\n struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n \n struct fsmonitor_daemon_state {\n@@ -117,5 +138,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \tstruct fsmonitor_daemon_state *state,\n \tconst char *path);\n \n+/*\n+ * Prepend the this batch of path(s) onto the list of batches associated\n+ * with the current token. This makes the batch visible to worker threads.\n+ *\n+ * The caller no longer owns the batch and must not free it.\n+ *\n+ * Wake up the client threads waiting on these cookies.\n+ */\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names);\n+\n+/*\n+ * If the platform-specific layer loses sync with the filesystem,\n+ * it should call this to invalidate cached data and abort waiting\n+ * threads.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"449968","messageId":"2b563c240eb1b592e95e89400aa5d57df1bf29b8.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 20/30] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:22Z","receivedAt":"2022-03-01T18:44:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate an IPC client to send query and flush commands to the daemon.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n t/helper/test-fsmonitor-client.c | 121 +++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 124 insertions(+)\n create mode 100644 t/helper/test-fsmonitor-client.c\n\ndiff --git a/Makefile b/Makefile\nindex 26567d4f772..daa21bed6c3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -716,6 +716,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\n TEST_BUILTINS_OBJS += test-getcwd.o\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nnew file mode 100644\nindex 00000000000..f7a5b3a32fa\n--- /dev/null\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -0,0 +1,121 @@\n+/*\n+ * test-fsmonitor-client.c: client code to send commands/requests to\n+ * a `git fsmonitor--daemon` daemon.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tdie(\"fsmonitor--daemon not available on this platform\");\n+}\n+#else\n+\n+/*\n+ * Read the `.git/index` to get the last token written to the\n+ * FSMonitor Index Extension.\n+ */\n+static const char *get_token_from_index(void)\n+{\n+\tstruct index_state *istate = the_repository->index;\n+\n+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update)\n+\t\tdie(\"index file does not have fsmonitor extension\");\n+\n+\treturn istate->fsmonitor_last_update;\n+}\n+\n+/*\n+ * Send an IPC query to a `git-fsmonitor--daemon` daemon and\n+ * ask for the changes since the given token or from the last\n+ * token in the index extension.\n+ *\n+ * This will implicitly start a daemon process if necessary. The\n+ * daemon process will persist after we exit.\n+ */\n+static int do_send_query(const char *token)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\n+\tret = fsmonitor_ipc__send_query(token, &answer);\n+\tif (ret < 0)\n+\t\tdie(_(\"could not query fsmonitor--daemon\"));\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n+ * and tell it to flush its cache.\n+ *\n+ * This feature is primarily used by the test suite to simulate a loss of\n+ * sync with the filesystem where we miss kernel events.\n+ */\n+static int do_send_flush(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tconst char *subcmd;\n+\tconst char *token = NULL;\n+\n+\tconst char * const fsmonitor_client_usage[] = {\n+\t\tN_(\"test-helper fsmonitor-client query []\"),\n+\t\tN_(\"test-helper fsmonitor-client flush\"),\n+\t\tNULL,\n+\t};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING(0, \"token\", &token, N_(\"token\"),\n+\t\t\t N_(\"command token to send to the server\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc < 2)\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tsubcmd = argv[1];\n+\targv--;\n+\targc++;\n+\n+\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n+\n+\tsetup_git_directory();\n+\n+\tif (!strcmp(subcmd, \"query\"))\n+\t\treturn !!do_send_query(token);\n+\n+\tif (!strcmp(subcmd, \"flush\"))\n+\t\treturn !!do_send_flush();\n+\n+\tdie(\"Unhandled subcommand: '%s'\", subcmd);\n+}\n+#endif\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex e6ec69cf326..0424f7adf5d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -32,6 +32,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\n \t{ \"getcwd\", cmd__getcwd },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 20756eefdda..c876e8246fb 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"449969","messageId":"15bd5aaff366169a22ad3fb94b412a7be38d1737.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 22/30] t/perf: avoid copying builtin fsmonitor files into test repo","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:24Z","receivedAt":"2022-03-01T18:44:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDo not copy any of the various fsmonitor--daemon files from the .git\ndirectory of the (GIT_PREF_REPO or GIT_PERF_LARGE_REPO) source repo\ninto the test's trash directory.\n\nWhen perf tests start, they copy the contents of the source repo into\nthe test's trash directory. If fsmonitor is running in the source repo,\nthere may be control files, such as the IPC socket and/or fsmonitor\ncookie files. These should not be copied into the test repo.\n\nUnix domain sockets cannot be copied in the manner used by the test\nsetup, so if present, the test setup fails.\n\nCookie files are harmless, but we should avoid them.\n\nThe builtin fsmonitor keeps all such control files/sockets in\n.git/fsmonitor--daemon*, so it is simple to exclude them.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/perf-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 407252bac70..932105cd12c 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -78,7 +78,7 @@ test_perf_copy_repo_contents () {\n \tfor stuff in \"$1\"/*\n \tdo\n \t\tcase \"$stuff\" in\n-\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees)\n+\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)\n \t\t\t;;\n \t\t*)\n \t\t\tcp -R \"$stuff\" \"$repo/.git/\" || exit 1\n-- \ngitgitgadget\n\n"},{"id":"449970","messageId":"4cb97918d20e19cf57c69e6a2a66d289ae873a7e.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 23/30] t/helper/test-chmtime: skip directories on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:25Z","receivedAt":"2022-03-01T18:44:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach `test-tool.exe chmtime` to ignore errors when setting the mtime\non a directory on Windows.\n\nNEEDSWORK: The Windows version of `utime()` (aka `mingw_utime()`) does\nnot properly handle directories because it uses `_wopen()`. It should\nbe converted to using `CreateFileW()` and backup semantics at a minimum.\nSince I'm already in the middle of a large patch series, I did not want\nto destabilize other callers of `utime()` right now. The problem has\nonly been observed in the t/perf/p7519 test when the test repo contains\nan empty directory on disk.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/helper/test-chmtime.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/t/helper/test-chmtime.c b/t/helper/test-chmtime.c\nindex 524b55ca496..dc28890a183 100644\n--- a/t/helper/test-chmtime.c\n+++ b/t/helper/test-chmtime.c\n@@ -134,6 +134,21 @@ int cmd__chmtime(int argc, const char **argv)\n \t\t}\n \n \t\tif (utb.modtime != sb.st_mtime && utime(argv[i], &utb) < 0) {\n+#ifdef GIT_WINDOWS_NATIVE\n+\t\t\tif (S_ISDIR(sb.st_mode)) {\n+\t\t\t\t/*\n+\t\t\t\t * NEEDSWORK: The Windows version of `utime()`\n+\t\t\t\t * (aka `mingw_utime()`) does not correctly\n+\t\t\t\t * handle directory arguments, since it uses\n+\t\t\t\t * `_wopen()`. Ignore it for now since this\n+\t\t\t\t * is just a test.\n+\t\t\t\t */\n+\t\t\t\tfprintf(stderr,\n+\t\t\t\t\t(\"Failed to modify time on directory %s. \"\n+\t\t\t\t\t \"Skipping\\n\"), argv[i]);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+#endif\n \t\t\tfprintf(stderr, \"Failed to modify time on %s: %s\\n\",\n \t\t\t argv[i], strerror(errno));\n \t\t\treturn 1;\n-- \ngitgitgadget\n\n"},{"id":"449971","messageId":"64a5b7416702c8bec948a1c3e289e18f9b6f87d5.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 26/30] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:28Z","receivedAt":"2022-03-01T18:44:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to periodically truncate the list of\nmodified files to save some memory.\n\nClients will ask for the set of changes relative to a token that they\nfound in the FSMN index extension in the index. (This token is like a\npoint in time, but different). Clients will then update the index to\ncontain the response token (so that subsequent commands will be\nrelative to this new token).\n\nTherefore, the daemon can gradually truncate the in-memory list of\nchanged paths as they become obsolete (older than the previous token).\nSince we may have multiple clients making concurrent requests with a\nskew of tokens and clients may be racing to the talk to the daemon,\nwe lazily truncate the list.\n\nWe introduce a 5 minute delay and truncate batches 5 minutes after\nthey are considered obsolete.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 88 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 88 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 65c1ef7d4bf..6ca4f5959e6 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -312,6 +312,75 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n \t\t\tbatch_src->interned_paths[k];\n }\n \n+/*\n+ * To keep the batch list from growing unbounded in response to filesystem\n+ * activity, we try to truncate old batches from the end of the list as\n+ * they become irrelevant.\n+ *\n+ * We assume that the .git/index will be updated with the most recent token\n+ * any time the index is updated. And future commands will only ask for\n+ * recent changes *since* that new token. So as tokens advance into the\n+ * future, older batch items will never be requested/needed. So we can\n+ * truncate them without loss of functionality.\n+ *\n+ * However, multiple commands may be talking to the daemon concurrently\n+ * or perform a slow command, so a little \"token skew\" is possible.\n+ * Therefore, we want this to be a little bit lazy and have a generous\n+ * delay.\n+ *\n+ * The current reader thread walked backwards in time from `token->batch_head`\n+ * back to `batch_marker` somewhere in the middle of the batch list.\n+ *\n+ * Let's walk backwards in time from that marker an arbitrary delay\n+ * and truncate the list there. Note that these timestamps are completely\n+ * artificial (based on when we pinned the batch item) and not on any\n+ * filesystem activity.\n+ *\n+ * Return the obsolete portion of the list after we have removed it from\n+ * the official list so that the caller can free it after leaving the lock.\n+ */\n+#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */\n+\n+static struct fsmonitor_batch *with_lock__truncate_old_batches(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst struct fsmonitor_batch *batch_marker)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder;\n+\n+\tif (!batch_marker)\n+\t\treturn NULL;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Truncate: mark (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t batch_marker->batch_seq_nr,\n+\t\t\t (uint64_t)batch_marker->pinned_time);\n+\n+\tfor (batch = batch_marker; batch; batch = batch->next) {\n+\t\ttime_t t;\n+\n+\t\tif (!batch->pinned_time) /* an overflow batch */\n+\t\t\tcontinue;\n+\n+\t\tt = batch->pinned_time + MY_TIME_DELAY_SECONDS;\n+\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n+\t\t\tcontinue;\n+\n+\t\tgoto truncate_past_here;\n+\t}\n+\n+\treturn NULL;\n+\n+truncate_past_here:\n+\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n+\n+\tremainder = ((struct fsmonitor_batch *)batch)->next;\n+\t((struct fsmonitor_batch *)batch)->next = NULL;\n+\n+\treturn remainder;\n+}\n+\n static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n {\n \tif (!token)\n@@ -425,6 +494,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tconst char *p;\n \tconst struct fsmonitor_batch *batch_head;\n \tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder = NULL;\n \tintmax_t count = 0, duplicates = 0;\n \tkh_str_t *shown;\n \tint hash_ret;\n@@ -654,11 +724,29 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * that work.\n \t\t\t */\n \t\t\tfsmonitor_free_token_data(token_data);\n+\t\t} else if (batch) {\n+\t\t\t/*\n+\t\t\t * We are holding the lock and are the only\n+\t\t\t * reader of the ref-counted portion of the\n+\t\t\t * list, so we get the honor of seeing if the\n+\t\t\t * list can be truncated to save memory.\n+\t\t\t *\n+\t\t\t * The main loop did not walk to the end of the\n+\t\t\t * list, so this batch is the first item in the\n+\t\t\t * batch-list that is older than the requested\n+\t\t\t * end-point sequence number. See if the tail\n+\t\t\t * end of the list is obsolete.\n+\t\t\t */\n+\t\t\tremainder = with_lock__truncate_old_batches(state,\n+\t\t\t\t\t\t\t\t batch);\n \t\t}\n \t}\n \n \tpthread_mutex_unlock(&state->main_lock);\n \n+\tif (remainder)\n+\t\tfsmonitor_batch__free_list(remainder);\n+\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n-- \ngitgitgadget\n\n"},{"id":"449972","messageId":"a70748b4640e673de32e24a77080c27da580a1df.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 24/30] t/perf/p7519: speed up test on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:26Z","receivedAt":"2022-03-01T18:44:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nChange p7519 to use `test_seq` and `xargs` rather than a `for` loop\nto touch thousands of files. This takes minutes off of test runs\non Windows because of process creation overhead.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 32 ++++++++++++++++++++------------\n 1 file changed, 20 insertions(+), 12 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex c8be58f3c76..aed7b1146b0 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -72,7 +72,7 @@ then\n \tfi\n fi\n \n-trace_start() {\n+trace_start () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tname=\"$1\"\n@@ -91,13 +91,20 @@ trace_start() {\n \tfi\n }\n \n-trace_stop() {\n+trace_stop () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tunset GIT_TRACE2_PERF\n \tfi\n }\n \n+touch_files () {\n+\tn=$1\n+\td=\"$n\"_files\n+\n+\t(cd $d ; test_seq 1 $n | xargs touch )\n+}\n+\n test_expect_success \"one time repo setup\" '\n \t# set untrackedCache depending on the environment\n \tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n@@ -119,10 +126,11 @@ test_expect_success \"one time repo setup\" '\n \tfi &&\n \n \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n-\tfor i in $(test_seq 1 10); do touch 10_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n+\t: 1_file directory should be left empty &&\n+\ttouch_files 10 &&\n+\ttouch_files 100 &&\n+\ttouch_files 1000 &&\n+\ttouch_files 10000 &&\n \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n \tgit commit -qm \"Add files\" &&\n \n@@ -133,7 +141,7 @@ test_expect_success \"one time repo setup\" '\n \tfi\n '\n \n-setup_for_fsmonitor() {\n+setup_for_fsmonitor () {\n \t# set INTEGRATION_SCRIPT depending on the environment\n \tif test -n \"$INTEGRATION_PATH\"\n \tthen\n@@ -173,7 +181,7 @@ test_perf_w_drop_caches () {\n \ttest_perf \"$@\"\n }\n \n-test_fsmonitor_suite() {\n+test_fsmonitor_suite () {\n \tif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n@@ -199,15 +207,15 @@ test_fsmonitor_suite() {\n \n \t# Update the mtimes on upto 100k files to make status think\n \t# that they are dirty. For simplicity, omit any files with\n-\t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n-\t# Then fully backslash-quote the paths to capture any\n-\t# whitespace so that they pass thru xargs properly.\n+\t# LFs (i.e. anything that ls-files thinks it needs to dquote)\n+\t# and any files with whitespace so that they pass thru xargs\n+\t# properly.\n \t#\n \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n \t\tgit ls-files | \\\n \t\t\thead -100000 | \\\n \t\t\tgrep -v \\\" | \\\n-\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n+\t\t\tegrep -v \" .\" | \\\n \t\t\txargs test-tool chmtime -300 &&\n \t\tgit status\n \t'\n-- \ngitgitgadget\n\n"},{"id":"449973","messageId":"db8809da089f28282c6bce1752a25423811a374a.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 21/30] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:23Z","receivedAt":"2022-03-01T18:44:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/t7527-builtin-fsmonitor.sh | 511 +++++++++++++++++++++++++++++++++++\n 1 file changed, 511 insertions(+)\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..5f7b8e54233\n--- /dev/null\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -0,0 +1,511 @@\n+#!/bin/sh\n+\n+test_description='built-in file system watcher'\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+stop_daemon_delete_repo () {\n+\tr=$1\n+\tgit -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n+\trm -rf $1\n+\treturn 0\n+}\n+\n+start_daemon () {\n+\tcase \"$#\" in\n+\t\t1) r=\"-C $1\";;\n+\t\t*) r=\"\";\n+\tesac\n+\n+\tgit $r fsmonitor--daemon start || return $?\n+\tgit $r fsmonitor--daemon status || return $?\n+\n+\treturn 0\n+}\n+\n+# Is a Trace2 data event present with the given catetory and key?\n+# We do not care what the value is.\n+#\n+have_t2_data_event () {\n+\tc=$1\n+\tk=$2\n+\n+\tgrep -e '\"event\":\"data\".*\"category\":\"'\"$c\"'\".*\"key\":\"'\"$k\"'\"'\n+}\n+\n+test_expect_success 'explicit daemon start and stop' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n+\n+\tgit init test_explicit &&\n+\tstart_daemon test_explicit &&\n+\n+\tgit -C test_explicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon start' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit\" &&\n+\n+\tgit init test_implicit &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n+\n+\t# query will implicitly start the daemon.\n+\t#\n+\t# for test-script simplicity, we send a V1 timestamp rather than\n+\t# a V2 token. either way, the daemon response to any query contains\n+\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n+\t# but this test case is only concerned with whether the daemon was\n+\t# implicitly started.)\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace\" \\\n+\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n+\tnul_to_q actual.filtered &&\n+\tgrep \"builtin:\" actual.filtered &&\n+\n+\t# confirm that a daemon was started in the background.\n+\t#\n+\t# since the mechanism for starting the background daemon is platform\n+\t# dependent, just confirm that the foreground command received a\n+\t# response from the daemon.\n+\n+\thave_t2_data_event fsm_client query/response-length <.git/trace &&\n+\n+\tgit -C test_implicit fsmonitor--daemon status &&\n+\tgit -C test_implicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (delete .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n+\n+\tgit init test_implicit_1 &&\n+\n+\tstart_daemon test_implicit_1 &&\n+\n+\t# deleting the .git directory will implicitly stop the daemon.\n+\trm -rf test_implicit_1/.git &&\n+\n+\t# [1] Create an empty .git directory so that the following Git\n+\t# command will stay relative to the `-C` directory.\n+\t#\n+\t# Without this, the Git command will override the requested\n+\t# -C argument and crawl out to the containing Git source tree.\n+\t# This would make the test result dependent upon whether we\n+\t# were using fsmonitor on our development worktree.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_1/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (rename .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_2\" &&\n+\n+\tgit init test_implicit_2 &&\n+\n+\tstart_daemon test_implicit_2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n+\n+\t# See [1] above.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'cannot start multiple daemons' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n+\n+\tgit init test_multiple &&\n+\n+\tstart_daemon test_multiple &&\n+\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n+\tgrep \"fsmonitor--daemon is already running\" actual &&\n+\n+\tgit -C test_multiple fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n+'\n+\n+# These tests use the main repo in the trash directory\n+\n+test_expect_success 'setup' '\n+\t>tracked &&\n+\t>modified &&\n+\t>delete &&\n+\t>rename &&\n+\tmkdir dir1 &&\n+\t>dir1/tracked &&\n+\t>dir1/modified &&\n+\t>dir1/delete &&\n+\t>dir1/rename &&\n+\tmkdir dir2 &&\n+\t>dir2/tracked &&\n+\t>dir2/modified &&\n+\t>dir2/delete &&\n+\t>dir2/rename &&\n+\tmkdir dirtorename &&\n+\t>dirtorename/a &&\n+\t>dirtorename/b &&\n+\n+\tcat >.gitignore <<-\\EOF &&\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tEOF\n+\n+\tgit -c core.fsmonitor=false add . &&\n+\ttest_tick &&\n+\tgit -c core.fsmonitor=false commit -m initial &&\n+\n+\tgit config core.fsmonitor true\n+'\n+\n+# The test already explicitly stopped (or tried to stop) the daemon.\n+# This is here in case something else fails first.\n+#\n+redundant_stop_daemon () {\n+\tgit fsmonitor--daemon stop\n+\treturn 0\n+}\n+\n+test_expect_success 'update-index implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_1\" \\\n+\t\tgit update-index --fsmonitor &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_1\n+'\n+\n+test_expect_success 'status implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_2\" \\\n+\t\tgit status >actual &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_2\n+'\n+\n+edit_files () {\n+\techo 1 >modified\n+\techo 2 >dir1/modified\n+\techo 3 >dir2/modified\n+\t>dir1/untracked\n+}\n+\n+delete_files () {\n+\trm -f delete\n+\trm -f dir1/delete\n+\trm -f dir2/delete\n+}\n+\n+create_files () {\n+\techo 1 >new\n+\techo 2 >dir1/new\n+\techo 3 >dir2/new\n+}\n+\n+rename_files () {\n+\tmv rename renamed\n+\tmv dir1/rename dir1/renamed\n+\tmv dir2/rename dir2/renamed\n+}\n+\n+file_to_directory () {\n+\trm -f delete\n+\tmkdir delete\n+\techo 1 >delete/new\n+}\n+\n+directory_to_file () {\n+\trm -rf dir1\n+\techo 1 >dir1\n+}\n+\n+verify_status () {\n+\tgit status >actual &&\n+\tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n+\tGIT_INDEX_FILE=.git/fresh-index git -c core.fsmonitor=false status >expect &&\n+\ttest_cmp expect actual &&\n+\techo HELLO AFTER &&\n+\tcat .git/trace &&\n+\techo HELLO AFTER\n+}\n+\n+# The next few test cases confirm that our fsmonitor daemon sees each type\n+# of OS filesystem notification that we care about. At this layer we just\n+# ensure we are getting the OS notifications and do not try to confirm what\n+# is reported by `git status`.\n+#\n+# We run a simple query after modifying the filesystem just to introduce\n+# a bit of a delay so that the trace logging from the daemon has time to\n+# get flushed to disk.\n+#\n+# We `reset` and `clean` at the bottom of each test (and before stopping the\n+# daemon) because these commands might implicitly restart the daemon.\n+\n+clean_up_repo_and_stop_daemon () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+\tgit fsmonitor--daemon stop\n+\trm -f .git/trace\n+}\n+\n+test_expect_success 'edit some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tedit_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/modified$\" .git/trace &&\n+\tgrep \"^event: dir2/modified$\" .git/trace &&\n+\tgrep \"^event: modified$\" .git/trace &&\n+\tgrep \"^event: dir1/untracked$\" .git/trace\n+'\n+\n+test_expect_success 'create some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tcreate_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/new$\" .git/trace &&\n+\tgrep \"^event: dir2/new$\" .git/trace &&\n+\tgrep \"^event: new$\" .git/trace\n+'\n+\n+test_expect_success 'delete some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdelete_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/delete$\" .git/trace &&\n+\tgrep \"^event: dir2/delete$\" .git/trace &&\n+\tgrep \"^event: delete$\" .git/trace\n+'\n+\n+test_expect_success 'rename some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\trename_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/rename$\" .git/trace &&\n+\tgrep \"^event: dir2/rename$\" .git/trace &&\n+\tgrep \"^event: rename$\" .git/trace &&\n+\tgrep \"^event: dir1/renamed$\" .git/trace &&\n+\tgrep \"^event: dir2/renamed$\" .git/trace &&\n+\tgrep \"^event: renamed$\" .git/trace\n+'\n+\n+test_expect_success 'rename directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tmv dirtorename dirrenamed &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dirtorename/*$\" .git/trace &&\n+\tgrep \"^event: dirrenamed/*$\" .git/trace\n+'\n+\n+test_expect_success 'file changes to directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tfile_to_directory &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: delete$\" .git/trace &&\n+\tgrep \"^event: delete/new$\" .git/trace\n+'\n+\n+test_expect_success 'directory changes to a file' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdirectory_to_file &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1$\" .git/trace\n+'\n+\n+# The next few test cases exercise the token-resync code. When filesystem\n+# drops events (because of filesystem velocity or because the daemon isn't\n+# polling fast enough), we need to discard the cached data (relative to the\n+# current token) and start collecting events under a new token.\n+#\n+# the 'test-tool fsmonitor-client flush' command can be used to send a\n+# \"flush\" message to a running daemon and ask it to do a flush/resync.\n+\n+test_expect_success 'flush cached data' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_flush\" &&\n+\n+\tgit init test_flush &&\n+\n+\t(\n+\t\tGIT_TEST_FSMONITOR_TOKEN=true &&\n+\t\texport GIT_TEST_FSMONITOR_TOKEN &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace_daemon\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon test_flush\n+\t) &&\n+\n+\t# The daemon should have an initial token with no events in _0 and\n+\t# then a few (probably platform-specific number of) events in _1.\n+\t# These should both have the same .\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n+\tnul_to_q actual_q0 &&\n+\n+\ttouch test_flush/file_1 &&\n+\ttouch test_flush/file_2 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n+\tnul_to_q actual_q1 &&\n+\n+\tgrep \"file_1\" actual_q1 &&\n+\n+\t# Force a flush. This will change the , reset the , and\n+\t# flush the file data. Then create some events and ensure that the file\n+\t# again appears in the cache. It should have the new .\n+\n+\ttest-tool -C test_flush fsmonitor-client flush >flush_0 &&\n+\tnul_to_q flush_q0 &&\n+\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_2 &&\n+\tnul_to_q actual_q2 &&\n+\n+\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n+\n+\ttouch test_flush/file_3 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n+\tnul_to_q actual_q3 &&\n+\n+\tgrep \"file_3\" actual_q3\n+'\n+\n+# The next few test cases create repos where the .git directory is NOT\n+# inside the one of the working directory. That is, where .git is a file\n+# that points to a directory elsewhere. This happens for submodules and\n+# non-primary worktrees.\n+\n+test_expect_success 'setup worktree base' '\n+\tgit init wt-base &&\n+\techo 1 >wt-base/file1 &&\n+\tgit -C wt-base add file1 &&\n+\tgit -C wt-base commit -m \"c1\"\n+'\n+\n+test_expect_success 'worktree with .git file' '\n+\tgit -C wt-base worktree add ../wt-secondary &&\n+\n+\t(\n+\t\tGIT_TRACE2_PERF=\"$(pwd)/trace2_wt_secondary\" &&\n+\t\texport GIT_TRACE2_PERF &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/trace_wt_secondary\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon wt-secondary\n+\t) &&\n+\n+\tgit -C wt-secondary fsmonitor--daemon stop &&\n+\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n+'\n+\n+# NEEDSWORK: Repeat one of the \"edit\" tests on wt-secondary and\n+# confirm that we get the same events and behavior -- that is, that\n+# fsmonitor--daemon correctly watches BOTH the working directory and\n+# the external GITDIR directory and behaves the same as when \".git\"\n+# is a directory inside the working directory.\n+\n+test_expect_success 'cleanup worktrees' '\n+\tstop_daemon_delete_repo wt-secondary &&\n+\tstop_daemon_delete_repo wt-base\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"449974","messageId":"b0c9c9c7a4005e2aa2a587fe5bc5588bf5a32d65.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 25/30] t/perf/p7519: add fsmonitor--daemon test cases","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:27Z","receivedAt":"2022-03-01T18:44:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRepeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\nthe \"Simple IPC\" interface.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 29 ++++++++++++++++++++++++++++-\n 1 file changed, 28 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex aed7b1146b0..a1c552129cc 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -182,7 +182,10 @@ test_perf_w_drop_caches () {\n }\n \n test_fsmonitor_suite () {\n-\tif test -n \"$INTEGRATION_SCRIPT\"; then\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n+\tthen\n+\t\tDESC=\"builtin fsmonitor--daemon\"\n+\telif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n \t\tDESC=\"fsmonitor=disabled\"\n@@ -293,4 +296,28 @@ test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n \n+#\n+# Run a full set of perf tests using the built-in fsmonitor--daemon.\n+# It does not use the Hook API, so it has a different setup.\n+# Explicitly start the daemon here and before we start client commands\n+# so that we can later add custom tracing.\n+#\n+if test_have_prereq FSMONITOR_DAEMON\n+then\n+\tUSE_FSMONITOR_DAEMON=t\n+\n+\ttrace_start fsmonitor--daemon--server\n+\tgit fsmonitor--daemon start\n+\n+\ttrace_start fsmonitor--daemon--client\n+\n+\tgit config core.fsmonitor true\n+\tgit update-index --fsmonitor\n+\n+\ttest_fsmonitor_suite\n+\n+\tgit fsmonitor--daemon stop\n+\ttrace_stop\n+fi\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"449975","messageId":"bc240a9e665841a622c96b8a245ce033684394f6.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 19/30] help: include fsmonitor--daemon feature flag in version info","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:21Z","receivedAt":"2022-03-01T18:44:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the \"feature: fsmonitor--daemon\" message to the output of\n`git version --build-options`.\n\nThe builtin FSMonitor is only available on certain platforms and\neven then only when certain Makefile flags are enabled, so print\na message in the verbose version output when it is available.\n\nThis can be used by test scripts for prereq testing. Granted, tests\ncould just try `git fsmonitor--daemon status` and look for a 128 exit\ncode or grep for a \"not supported\" message on stderr, but these\nmethods are rather obscure.\n\nThe main advantage is that the feature message will automatically\nappear in bug reports and other support requests.\n\nThis concept was also used during the development of Scalar for\nsimilar reasons.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n help.c | 4 ++++\n t/test-lib.sh | 6 ++++++\n 2 files changed, 10 insertions(+)\n\ndiff --git a/help.c b/help.c\nindex 71444906ddf..9112a51e84b 100644\n--- a/help.c\n+++ b/help.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"parse-options.h\"\n #include \"prompt.h\"\n+#include \"fsmonitor-ipc.h\"\n \n struct category_description {\n \tuint32_t category;\n@@ -695,6 +696,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n+\n+\t\tif (fsmonitor_ipc__is_supported())\n+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n \t}\n }\n \ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e4716b0b867..46cd596e7f5 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1799,3 +1799,9 @@ test_lazy_prereq SHA1 '\n # Tests that verify the scheduler integration must set this locally\n # to avoid errors.\n GIT_TEST_MAINT_SCHEDULER=\"none:exit 1\"\n+\n+# Does this platform support `git fsmonitor--daemon`\n+#\n+test_lazy_prereq FSMONITOR_DAEMON '\n+\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n+'\n-- \ngitgitgadget\n\n"},{"id":"449976","messageId":"0896e72e2b8e23354de9d476fbda55008a11de68.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 18/30] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:20Z","receivedAt":"2022-03-01T18:44:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to respond to IPC requests from client\nGit processes and respond with a list of modified pathnames\nrelative to the provided token.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 312 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 310 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 7c44b979035..65c1ef7d4bf 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,6 +7,7 @@\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n+#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon start []\"),\n@@ -364,6 +365,311 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+/*\n+ * Format an opaque token string to send to the client.\n+ */\n+static void with_lock__format_response_token(\n+\tstruct strbuf *response_token,\n+\tconst struct strbuf *response_token_id,\n+\tconst struct fsmonitor_batch *batch)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstrbuf_reset(response_token);\n+\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n+\t\t response_token_id->buf, batch->batch_seq_nr);\n+}\n+\n+/*\n+ * Parse an opaque token from the client.\n+ * Returns -1 on error.\n+ */\n+static int fsmonitor_parse_client_token(const char *buf_token,\n+\t\t\t\t\tstruct strbuf *requested_token_id,\n+\t\t\t\t\tuint64_t *seq_nr)\n+{\n+\tconst char *p;\n+\tchar *p_end;\n+\n+\tstrbuf_reset(requested_token_id);\n+\t*seq_nr = 0;\n+\n+\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n+\t\treturn -1;\n+\n+\twhile (*p && *p != ':')\n+\t\tstrbuf_addch(requested_token_id, *p++);\n+\tif (!*p++)\n+\t\treturn -1;\n+\n+\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n+\tif (*p_end)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal)\n+\n+static int do_handle_client(struct fsmonitor_daemon_state *state,\n+\t\t\t const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\tstruct fsmonitor_token_data *token_data = NULL;\n+\tstruct strbuf response_token = STRBUF_INIT;\n+\tstruct strbuf requested_token_id = STRBUF_INIT;\n+\tstruct strbuf payload = STRBUF_INIT;\n+\tuint64_t requested_oldest_seq_nr = 0;\n+\tuint64_t total_response_len = 0;\n+\tconst char *p;\n+\tconst struct fsmonitor_batch *batch_head;\n+\tconst struct fsmonitor_batch *batch;\n+\tintmax_t count = 0, duplicates = 0;\n+\tkh_str_t *shown;\n+\tint hash_ret;\n+\tint do_trivial = 0;\n+\tint do_flush = 0;\n+\n+\t/*\n+\t * We expect `command` to be of the form:\n+\t *\n+\t * := quit NUL\n+\t * | flush NUL\n+\t * | NUL\n+\t * | NUL\n+\t */\n+\n+\tif (!strcmp(command, \"quit\")) {\n+\t\t/*\n+\t\t * A client has requested over the socket/pipe that the\n+\t\t * daemon shutdown.\n+\t\t *\n+\t\t * Tell the IPC thread pool to shutdown (which completes\n+\t\t * the await in the main thread (which can stop the\n+\t\t * fsmonitor listener thread)).\n+\t\t *\n+\t\t * There is no reply to the client.\n+\t\t */\n+\t\treturn SIMPLE_IPC_QUIT;\n+\n+\t} else if (!strcmp(command, \"flush\")) {\n+\t\t/*\n+\t\t * Flush all of our cached data and generate a new token\n+\t\t * just like if we lost sync with the filesystem.\n+\t\t *\n+\t\t * Then send a trivial response using the new token.\n+\t\t */\n+\t\tdo_flush = 1;\n+\t\tdo_trivial = 1;\n+\n+\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n+\t\t/* assume V1 timestamp or garbage */\n+\n+\t\tchar *p_end;\n+\n+\t\tstrtoumax(command, &p_end, 10);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t ((*p_end) ?\n+\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n+\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n+\t\t\t\t command);\n+\t\tdo_trivial = 1;\n+\n+\t} else {\n+\t\t/* We have \"builtin:*\" */\n+\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n+\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n+\t\t\t\t\t command);\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We have a V2 valid token:\n+\t\t\t * \"builtin::\"\n+\t\t\t */\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (!state->current_token_data)\n+\t\tBUG(\"fsmonitor state does not have a current token\");\n+\n+\tif (do_flush)\n+\t\twith_lock__do_force_resync(state);\n+\n+\t/*\n+\t * We mark the current head of the batch list as \"pinned\" so\n+\t * that the listener thread will treat this item as read-only\n+\t * (and prevent any more paths from being added to it) from\n+\t * now on.\n+\t */\n+\ttoken_data = state->current_token_data;\n+\tbatch_head = token_data->batch_head;\n+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n+\n+\t/*\n+\t * FSMonitor Protocol V2 requires that we send a response header\n+\t * with a \"new current token\" and then all of the paths that changed\n+\t * since the \"requested token\". We send the seq_nr of the just-pinned\n+\t * head batch so that future requests from a client will be relative\n+\t * to it.\n+\t */\n+\twith_lock__format_response_token(&response_token,\n+\t\t\t\t\t &token_data->token_id, batch_head);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttotal_response_len += response_token.len + 1;\n+\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n+\t\t\t response_token.buf);\n+\n+\tif (!do_trivial) {\n+\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n+\t\t\t/*\n+\t\t\t * The client last spoke to a different daemon\n+\t\t\t * instance -OR- the daemon had to resync with\n+\t\t\t * the filesystem (and lost events), so reject.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n+\t\t\t\t\t \"response/token\", \"different\");\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else if (requested_oldest_seq_nr <\n+\t\t\t token_data->batch_tail->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * The client wants older events than we have for\n+\t\t\t * this token_id. This means that the end of our\n+\t\t\t * batch list was truncated and we cannot give the\n+\t\t\t * client a complete snapshot relative to their\n+\t\t\t * request.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"client requested truncated data\");\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n+\tif (do_trivial) {\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\treply(reply_data, \"/\", 2);\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n+\t\t\t\t \"response/trivial\", 1);\n+\n+\t\tstrbuf_release(&response_token);\n+\t\tstrbuf_release(&requested_token_id);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * We're going to hold onto a pointer to the current\n+\t * token-data while we walk the list of batches of files.\n+\t * During this time, we will NOT be under the lock.\n+\t * So we ref-count it.\n+\t *\n+\t * This allows the listener thread to continue prepending\n+\t * new batches of items to the token-data (which we'll ignore).\n+\t *\n+\t * AND it allows the listener thread to do a token-reset\n+\t * (and install a new `current_token_data`).\n+\t */\n+\ttoken_data->client_ref_count++;\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * The client request is relative to the token that they sent,\n+\t * so walk the batch list backwards from the current head back\n+\t * to the batch (sequence number) they named.\n+\t *\n+\t * We use khash to de-dup the list of pathnames.\n+\t *\n+\t * NEEDSWORK: each batch contains a list of interned strings,\n+\t * so we only need to do pointer comparisons here to build the\n+\t * hash table. Currently, we're still comparing the string\n+\t * values.\n+\t */\n+\tshown = kh_init_str();\n+\tfor (batch = batch_head;\n+\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n+\t batch = batch->next) {\n+\t\tsize_t k;\n+\n+\t\tfor (k = 0; k < batch->nr; k++) {\n+\t\t\tconst char *s = batch->interned_paths[k];\n+\t\t\tsize_t s_len;\n+\n+\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n+\t\t\t\tduplicates++;\n+\t\t\telse {\n+\t\t\t\tkh_put_str(shown, s, &hash_ret);\n+\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n+\t\t\t\t\t\t count, s);\n+\n+\t\t\t\t/* Each path gets written with a trailing NUL */\n+\t\t\t\ts_len = strlen(s) + 1;\n+\n+\t\t\t\tif (payload.len + s_len >=\n+\t\t\t\t LARGE_PACKET_DATA_MAX) {\n+\t\t\t\t\treply(reply_data, payload.buf,\n+\t\t\t\t\t payload.len);\n+\t\t\t\t\ttotal_response_len += payload.len;\n+\t\t\t\t\tstrbuf_reset(&payload);\n+\t\t\t\t}\n+\n+\t\t\t\tstrbuf_add(&payload, s, s_len);\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (payload.len) {\n+\t\treply(reply_data, payload.buf, payload.len);\n+\t\ttotal_response_len += payload.len;\n+\t}\n+\n+\tkh_release_str(shown);\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (token_data->client_ref_count > 0)\n+\t\ttoken_data->client_ref_count--;\n+\n+\tif (token_data->client_ref_count == 0) {\n+\t\tif (token_data != state->current_token_data) {\n+\t\t\t/*\n+\t\t\t * The listener thread did a token-reset while we were\n+\t\t\t * walking the batch list. Therefore, this token is\n+\t\t\t * stale and can be discarded completely. If we are\n+\t\t\t * the last reader thread using this token, we own\n+\t\t\t * that work.\n+\t\t\t */\n+\t\t\tfsmonitor_free_token_data(token_data);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n+\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\tstrbuf_release(&payload);\n+\n+\treturn 0;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -371,7 +677,7 @@ static int handle_client(void *data,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n-\t/* struct fsmonitor_daemon_state *state = data; */\n+\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n \t/*\n@@ -382,10 +688,12 @@ static int handle_client(void *data,\n \tif (command_len != strlen(command))\n \t\tBUG(\"FSMonitor assumes text messages\");\n \n+\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n+\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n \n-\tresult = 0; /* TODO Do something here. */\n+\tresult = do_handle_client(state, command, reply, reply_data);\n \n \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n \n-- \ngitgitgadget\n\n"},{"id":"449977","messageId":"6b5b3d08227df88a54084860557cc1895acb7202.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 15/30] compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:17Z","receivedAt":"2022-03-01T18:44:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the win32 backend to register a watch on the working tree\nroot directory (recursively). Also watch the if it is\nnot inside the working tree. And to collect path change notifications\ninto batches and publish.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-win32.c | 565 ++++++++++++++++++++++++++++\n 1 file changed, 565 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 916cbea254f..c2d11acbc1e 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -2,20 +2,585 @@\n #include \"config.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+/*\n+ * The documentation of ReadDirectoryChangesW() states that the maximum\n+ * buffer size is 64K when the monitored directory is remote.\n+ *\n+ * Larger buffers may be used when the monitored directory is local and\n+ * will help us receive events faster from the kernel and avoid dropped\n+ * events.\n+ *\n+ * So we try to use a very large buffer and silently fallback to 64K if\n+ * we get an error.\n+ */\n+#define MAX_RDCW_BUF_FALLBACK (65536)\n+#define MAX_RDCW_BUF (65536 * 8)\n+\n+struct one_watch\n+{\n+\tchar buffer[MAX_RDCW_BUF];\n+\tDWORD buf_len;\n+\tDWORD count;\n+\n+\tstruct strbuf path;\n+\tHANDLE hDir;\n+\tHANDLE hEvent;\n+\tOVERLAPPED overlapped;\n+\n+\t/*\n+\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n+\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n+\t */\n+\tBOOL is_active;\n+};\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tstruct one_watch *watch_worktree;\n+\tstruct one_watch *watch_gitdir;\n+\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hListener[3]; /* we don't own these handles */\n+#define LISTENER_SHUTDOWN 0\n+#define LISTENER_HAVE_DATA_WORKTREE 1\n+#define LISTENER_HAVE_DATA_GITDIR 2\n+\tint nr_listener_handles;\n+};\n+\n+/*\n+ * Convert the WCHAR path from the notification into UTF8 and\n+ * then normalize it.\n+ */\n+static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+\t\t\t\t struct strbuf *normalized_path)\n+{\n+\tint reserve;\n+\tint len = 0;\n+\n+\tstrbuf_reset(normalized_path);\n+\tif (!info->FileNameLength)\n+\t\tgoto normalize;\n+\n+\t/*\n+\t * Pre-reserve enough space in the UTF8 buffer for\n+\t * each Unicode WCHAR character to be mapped into a\n+\t * sequence of 2 UTF8 characters. That should let us\n+\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n+\t */\n+\treserve = info->FileNameLength + 1;\n+\tstrbuf_grow(normalized_path, reserve);\n+\n+\tfor (;;) {\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n+\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t\t\t normalized_path->buf,\n+\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n+\t\t\t\t\t NULL, NULL);\n+\t\tif (len > 0)\n+\t\t\tgoto normalize;\n+\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n+\t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n+\t\t\t GetLastError(),\n+\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n+\t\t\t info->FileName);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tstrbuf_grow(normalized_path,\n+\t\t\t strbuf_avail(normalized_path) + reserve);\n+\t}\n+\n+normalize:\n+\tstrbuf_setlen(normalized_path, len);\n+\treturn strbuf_normalize_path(normalized_path);\n+}\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+}\n+\n+static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const char *path)\n+{\n+\tstruct one_watch *watch = NULL;\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\twchar_t wpath[MAX_PATH];\n+\n+\tif (xutftowcs_path(wpath, path) < 0) {\n+\t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n+\t\treturn NULL;\n+\t}\n+\n+\thDir = CreateFileW(wpath,\n+\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n+\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n+\t\t\t NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n+\t\t GetLastError(), path);\n+\t\treturn NULL;\n+\t}\n+\n+\tCALLOC_ARRAY(watch, 1);\n+\n+\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n+\n+\tstrbuf_init(&watch->path, 0);\n+\tstrbuf_addstr(&watch->path, path);\n+\n+\twatch->hDir = hDir;\n+\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\treturn watch;\n+}\n+\n+static void destroy_watch(struct one_watch *watch)\n+{\n+\tif (!watch)\n+\t\treturn;\n+\n+\tstrbuf_release(&watch->path);\n+\tif (watch->hDir != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hDir);\n+\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hEvent);\n+\n+\tfree(watch);\n+}\n+\n+static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+\t\t\t struct one_watch *watch)\n+{\n+\tDWORD dwNotifyFilter =\n+\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n+\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n+\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n+\t\tFILE_NOTIFY_CHANGE_SIZE |\n+\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n+\t\tFILE_NOTIFY_CHANGE_CREATION;\n+\n+\tResetEvent(watch->hEvent);\n+\n+\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n+\twatch->overlapped.hEvent = watch->hEvent;\n+\n+\t/*\n+\t * Queue an async call using Overlapped IO. This returns immediately.\n+\t * Our event handle will be signalled when the real result is available.\n+\t *\n+\t * The return value here just means that we successfully queued it.\n+\t * We won't know if the Read...() actually produces data until later.\n+\t */\n+\twatch->is_active = ReadDirectoryChangesW(\n+\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n+\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n+\n+\tif (watch->is_active)\n+\t\treturn 0;\n+\n+\terror(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static int recv_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD gle;\n+\n+\twatch->is_active = FALSE;\n+\n+\t/*\n+\t * The overlapped result is ready. If the Read...() was successful\n+\t * we finally receive the actual result into our buffer.\n+\t */\n+\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n+\t\t\t\tTRUE))\n+\t\treturn 0;\n+\n+\tgle = GetLastError();\n+\tif (gle == ERROR_INVALID_PARAMETER &&\n+\t /*\n+\t * The kernel throws an invalid parameter error when our\n+\t * buffer is too big and we are pointed at a remote\n+\t * directory (and possibly for other reasons). Quietly\n+\t * set it down and try again.\n+\t *\n+\t * See note about MAX_RDCW_BUF at the top.\n+\t */\n+\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n+\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n+\t\treturn -2;\n+\t}\n+\n+\t/*\n+\t * NEEDSWORK: If an external is deleted, the above\n+\t * returns an error. I'm not sure that there's anything that\n+\t * we can do here other than failing -- the /.git\n+\t * link file would be broken anyway. We might try to check\n+\t * for that and return a better error message, but I'm not\n+\t * sure it is worth it.\n+\t */\n+\n+\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, gle);\n+\treturn -1;\n+}\n+\n+static void cancel_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD count;\n+\n+\tif (!watch || !watch->is_active)\n+\t\treturn;\n+\n+\t/*\n+\t * The calls to ReadDirectoryChangesW() and GetOverlappedResult()\n+\t * form a \"pair\" (my term) where we queue an IO and promise to\n+\t * hang around and wait for the kernel to give us the result.\n+\t *\n+\t * If for some reason after we queue the IO, we have to quit\n+\t * or otherwise not stick around for the second half, we must\n+\t * tell the kernel to abort the IO. This prevents the kernel\n+\t * from writing to our buffer and/or signalling our event\n+\t * after we free them.\n+\t *\n+\t * (Ask me how much fun it was to track that one down).\n+\t */\n+\tCancelIoEx(watch->hDir, &watch->overlapped);\n+\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n+\twatch->is_active = FALSE;\n+}\n+\n+/*\n+ * Process filesystem events that happen anywhere (recursively) under the\n+ * root directory. For a normal working directory, this includes\n+ * both version controlled files and the contents of the .git/ directory.\n+ *\n+ * If /.git is a file, then we only see events for the file\n+ * itself.\n+ */\n+static int process_worktree_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_worktree;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tconst char *p = watch->buffer;\n+\n+\t/*\n+\t * If the kernel gets more events than will fit in the kernel\n+\t * buffer associated with our RDCW handle, it drops them and\n+\t * returns a count of zero.\n+\t *\n+\t * Yes, the call returns WITHOUT error and with length zero.\n+\t * This is the documented behavior. (My testing has confirmed\n+\t * that it also sets the last error to ERROR_NOTIFY_ENUM_DIR,\n+\t * but we do not rely on that since the function did not\n+\t * return an error and it is not documented.)\n+\t *\n+\t * (The \"overflow\" case is not ambiguous with the \"no data\" case\n+\t * because we did an INFINITE wait.)\n+\t *\n+\t * This means we have a gap in coverage. Tell the daemon layer\n+\t * to resync.\n+\t */\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\t}\n+\n+\t/*\n+\t * On Windows, `info` contains an \"array\" of paths that are\n+\t * relative to the root of whichever directory handle received\n+\t * the event.\n+\t */\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\t\t/* ignore everything inside of \"/.git/\" */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* queue normal pathname */\n+\t\t\tif (!batch)\n+\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tbatch = NULL;\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_SHUTDOWN;\n+}\n+\n+/*\n+ * Process filesystem events that happened anywhere (recursively) under the\n+ * external (such as non-primary worktrees or submodules).\n+ * We only care about cookie files that our client threads created here.\n+ *\n+ * Note that we DO NOT get filesystem events on the external \n+ * itself (it is not inside something that we are watching). In particular,\n+ * we do not get an event if the external is deleted.\n+ */\n+static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_gitdir;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *p = watch->buffer;\n+\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_GITDIR;\n+\t}\n+\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\tgoto skip_this_path;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, NULL, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_GITDIR;\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tDWORD dwWait;\n+\tint result;\n+\n+\tstate->error_code = 0;\n+\n+\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tif (data->watch_gitdir &&\n+\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tfor (;;) {\n+\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n+\t\t\t\t\t\tdata->hListener,\n+\t\t\t\t\t\tFALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_worktree);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_gitdir);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->error_code = -1;\n+\n+force_shutdown:\n+\t/*\n+\t * Tell the IPC thead pool to stop (which completes the await\n+\t * in the main thread (which will also signal this thread (if\n+\t * we are still alive))).\n+\t */\n+\tipc_server_stop_async(state->ipc_server_data);\n+\n+clean_shutdown:\n+\tcancel_rdcw_watch(data->watch_worktree);\n+\tcancel_rdcw_watch(data->watch_gitdir);\n }\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->watch_worktree = create_watch(state,\n+\t\t\t\t\t state->path_worktree_watch.buf);\n+\tif (!data->watch_worktree)\n+\t\tgoto failed;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->watch_gitdir = create_watch(state,\n+\t\t\t\t\t\t state->path_gitdir_watch.buf);\n+\t\tif (!data->watch_gitdir)\n+\t\t\tgoto failed;\n+\t}\n+\n+\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_listener_handles++;\n+\n+\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n+\t\tdata->watch_worktree->hEvent;\n+\tdata->nr_listener_handles++;\n+\n+\tif (data->watch_gitdir) {\n+\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n+\t\t\tdata->watch_gitdir->hEvent;\n+\t\tdata->nr_listener_handles++;\n+\t}\n+\n+\tstate->backend_data = data;\n+\treturn 0;\n+\n+failed:\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n-- \ngitgitgadget\n\n"},{"id":"449978","messageId":"aabfebd3a41208091d9738c60ffd27573f4faa28.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 17/30] compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:19Z","receivedAt":"2022-03-01T18:44:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement file system event listener on MacOS using FSEvent,\nCoreFoundation, and CoreServices.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-darwin.c | 383 +++++++++++++++++++++++++++\n 1 file changed, 383 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex f76341317dd..5c5de1ae702 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -97,20 +97,403 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tCFStringRef cfsr_worktree_path;\n+\tCFStringRef cfsr_gitdir_path;\n+\n+\tCFArrayRef cfar_paths_to_watch;\n+\tint nr_paths_watching;\n+\n+\tFSEventStreamRef stream;\n+\n+\tCFRunLoopRef rl;\n+\n+\tenum shutdown_style {\n+\t\tSHUTDOWN_EVENT = 0,\n+\t\tFORCE_SHUTDOWN,\n+\t\tFORCE_ERROR_STOP,\n+\t} shutdown_style;\n+\n+\tunsigned int stream_scheduled:1;\n+\tunsigned int stream_started:1;\n+};\n+\n+static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n+\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n+\tif (flag & kFSEventStreamEventFlagUserDropped)\n+\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n+\tif (flag & kFSEventStreamEventFlagKernelDropped)\n+\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n+\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n+\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n+\tif (flag & kFSEventStreamEventFlagHistoryDone)\n+\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n+\tif (flag & kFSEventStreamEventFlagRootChanged)\n+\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n+\tif (flag & kFSEventStreamEventFlagMount)\n+\t\tstrbuf_addstr(&msg, \"Mount|\");\n+\tif (flag & kFSEventStreamEventFlagUnmount)\n+\t\tstrbuf_addstr(&msg, \"Unmount|\");\n+\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n+\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n+\tif (flag & kFSEventStreamEventFlagItemCreated)\n+\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n+\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n+\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n+\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsDir)\n+\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsFile)\n+\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemModified)\n+\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n+\tif (flag & kFSEventStreamEventFlagItemRemoved)\n+\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n+\tif (flag & kFSEventStreamEventFlagItemRenamed)\n+\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n+\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n+\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n+\tif (flag & kFSEventStreamEventFlagOwnEvent)\n+\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n+\tif (flag & kFSEventStreamEventFlagItemCloned)\n+\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\t\t\t path, flag, msg.buf);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRemoved);\n+}\n+\n+static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRenamed);\n+}\n+\n+static int ef_is_dropped(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagMustScanSubDirs ||\n+\t\tef & kFSEventStreamEventFlagKernelDropped ||\n+\t\tef & kFSEventStreamEventFlagUserDropped);\n+}\n+\n+static void fsevent_callback(ConstFSEventStreamRef streamRef,\n+\t\t\t void *ctx,\n+\t\t\t size_t num_of_events,\n+\t\t\t void *event_paths,\n+\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t const FSEventStreamEventId event_ids[])\n+{\n+\tstruct fsmonitor_daemon_state *state = ctx;\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tchar **paths = (char **)event_paths;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *path_k;\n+\tconst char *slash;\n+\tint k;\n+\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t/*\n+\t * Build a list of all filesystem changes into a private/local\n+\t * list and without holding any locks.\n+\t */\n+\tfor (k = 0; k < num_of_events; k++) {\n+\t\t/*\n+\t\t * On Mac, we receive an array of absolute paths.\n+\t\t */\n+\t\tpath_k = paths[k];\n+\n+\t\t/*\n+\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n+\t\t * Please don't log them to Trace2.\n+\t\t *\n+\t\t * trace_printf_key(&trace_fsmonitor, \"Path: '%s'\", path_k);\n+\t\t */\n+\n+\t\t/*\n+\t\t * If event[k] is marked as dropped, we assume that we have\n+\t\t * lost sync with the filesystem and should flush our cached\n+\t\t * data. We need to:\n+\t\t *\n+\t\t * [1] Abort/wake any client threads waiting for a cookie and\n+\t\t * flush the cached state data (the current token), and\n+\t\t * create a new token.\n+\t\t *\n+\t\t * [2] Discard the batch that we were locally building (since\n+\t\t * they are conceptually relative to the just flushed\n+\t\t * token).\n+\t\t */\n+\t\tif (ef_is_dropped(event_flags[k])) {\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\tfsmonitor_force_resync(state);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t\tstring_list_clear(&cookie_list, 0);\n+\n+\t\t\t/*\n+\t\t\t * We assume that any events that we received\n+\t\t\t * in this callback after this dropped event\n+\t\t\t * may still be valid, so we continue rather\n+\t\t\t * than break. (And just in case there is a\n+\t\t\t * delete of \".git\" hiding in there.)\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n+\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git or gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path_k);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path_k);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\t/* ignore all other paths inside of .git or gitdir */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\tcase IS_GITDIR:\n+\t\t\t/*\n+\t\t\t * If .git directory is deleted or renamed away,\n+\t\t\t * we have to quit.\n+\t\t\t */\n+\t\t\tif (ef_is_root_delete(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir renamed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* try to queue normal pathnames */\n+\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\t/*\n+\t\t\t * Because of the implicit \"binning\" (the\n+\t\t\t * kernel calls us at a given frequency) and\n+\t\t\t * de-duping (the kernel is free to combine\n+\t\t\t * multiple events for a given pathname), an\n+\t\t\t * individual fsevent could be marked as both\n+\t\t\t * a file and directory. Add it to the queue\n+\t\t\t * with both spellings so that the client will\n+\t\t\t * know how much to invalidate/refresh.\n+\t\t\t */\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t}\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tstrbuf_reset(&tmp);\n+\t\t\t\tstrbuf_addstr(&tmp, rel);\n+\t\t\t\tstrbuf_addch(&tmp, '/');\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\n+\t\tcase IS_OUTSIDE_CONE:\n+\t\tdefault:\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignoring '%s'\", path_k);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\n+\tdata->shutdown_style = FORCE_SHUTDOWN;\n+\tCFRunLoopStop(data->rl);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+}\n+\n+/*\n+ * In the call to `FSEventStreamCreate()` to setup our watch, the\n+ * `latency` argument determines the frequency of calls to our callback\n+ * with new FS events. Too slow and events get dropped; too fast and\n+ * we burn CPU unnecessarily. Since it is rather obscure, I don't\n+ * think this needs to be a config setting. I've done extensive\n+ * testing on my systems and chosen the value below. It gives good\n+ * results and I've not seen any dropped events.\n+ *\n+ * With a latency of 0.1, I was seeing lots of dropped events during\n+ * the \"touch 100000\" files test within t/perf/p7519, but with a\n+ * latency of 0.001 I did not see any dropped events. So I'm going\n+ * to assume that this is the \"correct\" value.\n+ *\n+ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n+ */\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n+\t\tkFSEventStreamCreateFlagWatchRoot |\n+\t\tkFSEventStreamCreateFlagFileEvents;\n+\tFSEventStreamContext ctx = {\n+\t\t0,\n+\t\tstate,\n+\t\tNULL,\n+\t\tNULL,\n+\t\tNULL\n+\t};\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\tconst void *dir_array[2];\n+\n+\tCALLOC_ARRAY(data, 1);\n+\tstate->backend_data = data;\n+\n+\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n+\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n+\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n+\t\t\tNULL, state->path_gitdir_watch.buf,\n+\t\t\tkCFStringEncodingUTF8);\n+\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n+\t}\n+\n+\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n+\t\t\t\t\t\t data->nr_paths_watching,\n+\t\t\t\t\t\t NULL);\n+\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n+\t\t\t\t\t data->cfar_paths_to_watch,\n+\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n+\t\t\t\t\t 0.001, flags);\n+\tif (data->stream == NULL)\n+\t\tgoto failed;\n+\n+\t/*\n+\t * `data->rl` needs to be set inside the listener thread.\n+\t */\n+\n+\treturn 0;\n+\n+failed:\n+\terror(\"Unable to create FSEventStream.\");\n+\n+\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tif (data->stream) {\n+\t\tif (data->stream_started)\n+\t\t\tFSEventStreamStop(data->stream);\n+\t\tif (data->stream_scheduled)\n+\t\t\tFSEventStreamInvalidate(data->stream);\n+\t\tFSEventStreamRelease(data->stream);\n+\t}\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\tdata->shutdown_style = SHUTDOWN_EVENT;\n+\n+\tCFRunLoopStop(data->rl);\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\n+\tdata->rl = CFRunLoopGetCurrent();\n+\n+\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n+\tdata->stream_scheduled = 1;\n+\n+\tif (!FSEventStreamStart(data->stream)) {\n+\t\terror(\"Failed to start the FSEventStream\");\n+\t\tgoto force_error_stop_without_loop;\n+\t}\n+\tdata->stream_started = 1;\n+\n+\tCFRunLoopRun();\n+\n+\tswitch (data->shutdown_style) {\n+\tcase FORCE_ERROR_STOP:\n+\t\tstate->error_code = -1;\n+\t\t/* fall thru */\n+\tcase FORCE_SHUTDOWN:\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\t/* fall thru */\n+\tcase SHUTDOWN_EVENT:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn;\n+\n+force_error_stop_without_loop:\n+\tstate->error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+\treturn;\n }\n-- \ngitgitgadget\n\n"},{"id":"449979","messageId":"cdef9730b3f93a6f0f98d68ffb81bcb89d6e698e.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:18Z","receivedAt":"2022-03-01T18:44:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInclude MacOS system declarations to allow us to use FSEvent and\nCoreFoundation APIs. We need different versions of the declarations\nfor GCC vs. clang because of compiler and header file conflicts.\n\nWhile it is quite possible to #include Apple's CoreServices.h when\ncompiling C source code with clang, trying to build it with GCC\ncurrently fails with this error:\n\nIn file included\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Security.h:42,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/CSIdentity.h:43,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/OSServices.h:29,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/IconsCore.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/LaunchServices.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n\n /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Authorization.h:193:7:\n error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n\nThe underlying reason is that GCC (rightfully) objects that an `enum`\nvalue such as `kAuthorizationExternalFormLength` is not a constant\n(because it is not, the preprocessor has no knowledge of it, only the\nactual C compiler does) and can therefore not be used to define the size\nof a C array.\n\nThis is a known problem and tracked in GCC's bug tracker:\nhttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nIn the meantime, let's not block things and go the slightly ugly route\nof declaring/defining the FSEvents constants, data structures and\nfunctions that we need, so that we can avoid above-mentioned issue.\n\nLet's do this _only_ for GCC, though, so that the CI/PR builds (which\nbuild both with clang and with GCC) can guarantee that we _are_ using\nthe correct data types.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-darwin.c | 96 ++++++++++++++++++++++++++++\n 1 file changed, 96 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex c84e3344ab9..f76341317dd 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -1,3 +1,99 @@\n+#ifndef __clang__\n+/*\n+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n+ * with clang, but not with GCC as of time of writing.\n+ *\n+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details.\n+ */\n+typedef unsigned int FSEventStreamCreateFlags;\n+#define kFSEventStreamEventFlagNone 0x00000000\n+#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n+#define kFSEventStreamEventFlagUserDropped 0x00000002\n+#define kFSEventStreamEventFlagKernelDropped 0x00000004\n+#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n+#define kFSEventStreamEventFlagHistoryDone 0x00000010\n+#define kFSEventStreamEventFlagRootChanged 0x00000020\n+#define kFSEventStreamEventFlagMount 0x00000040\n+#define kFSEventStreamEventFlagUnmount 0x00000080\n+#define kFSEventStreamEventFlagItemCreated 0x00000100\n+#define kFSEventStreamEventFlagItemRemoved 0x00000200\n+#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n+#define kFSEventStreamEventFlagItemRenamed 0x00000800\n+#define kFSEventStreamEventFlagItemModified 0x00001000\n+#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n+#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n+#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n+#define kFSEventStreamEventFlagItemIsFile 0x00010000\n+#define kFSEventStreamEventFlagItemIsDir 0x00020000\n+#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n+#define kFSEventStreamEventFlagOwnEvent 0x00080000\n+#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n+#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+\n+typedef struct __FSEventStream *FSEventStreamRef;\n+typedef const FSEventStreamRef ConstFSEventStreamRef;\n+\n+typedef unsigned int CFStringEncoding;\n+#define kCFStringEncodingUTF8 0x08000100\n+\n+typedef const struct __CFString *CFStringRef;\n+typedef const struct __CFArray *CFArrayRef;\n+typedef const struct __CFRunLoop *CFRunLoopRef;\n+\n+struct FSEventStreamContext {\n+ long long version;\n+ void *cb_data, *retain, *release, *copy_description;\n+};\n+\n+typedef struct FSEventStreamContext FSEventStreamContext;\n+typedef unsigned int FSEventStreamEventFlags;\n+#define kFSEventStreamCreateFlagNoDefer 0x02\n+#define kFSEventStreamCreateFlagWatchRoot 0x04\n+#define kFSEventStreamCreateFlagFileEvents 0x10\n+\n+typedef unsigned long long FSEventStreamEventId;\n+#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL\n+\n+typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef,\n+\t\t\t\t void *context,\n+\t\t\t\t __SIZE_TYPE__ num_of_events,\n+\t\t\t\t void *event_paths,\n+\t\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t\t const FSEventStreamEventId event_ids[]);\n+typedef double CFTimeInterval;\n+FSEventStreamRef FSEventStreamCreate(void *allocator,\n+\t\t\t\t FSEventStreamCallback callback,\n+\t\t\t\t FSEventStreamContext *context,\n+\t\t\t\t CFArrayRef paths_to_watch,\n+\t\t\t\t FSEventStreamEventId since_when,\n+\t\t\t\t CFTimeInterval latency,\n+\t\t\t\t FSEventStreamCreateFlags flags);\n+CFStringRef CFStringCreateWithCString(void *allocator, const char *string,\n+\t\t\t\t CFStringEncoding encoding);\n+CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count,\n+\t\t\t void *callbacks);\n+void CFRunLoopRun(void);\n+void CFRunLoopStop(CFRunLoopRef run_loop);\n+CFRunLoopRef CFRunLoopGetCurrent(void);\n+extern CFStringRef kCFRunLoopDefaultMode;\n+void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream,\n+\t\t\t\t CFRunLoopRef run_loop,\n+\t\t\t\t CFStringRef run_loop_mode);\n+unsigned char FSEventStreamStart(FSEventStreamRef stream);\n+void FSEventStreamStop(FSEventStreamRef stream);\n+void FSEventStreamInvalidate(FSEventStreamRef stream);\n+void FSEventStreamRelease(FSEventStreamRef stream);\n+#else\n+/*\n+ * Let Apple's headers declare `isalnum()` first, before\n+ * Git's headers override it via a constant\n+ */\n+#include \n+#include \n+#include \n+#endif\n+\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n-- \ngitgitgadget\n\n"},{"id":"449980","messageId":"c2f3668a866c3a795e15d8155aa9dcf2e7401658.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 13/30] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:15Z","receivedAt":"2022-03-01T18:44:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to create token-ids and define the\noverall token naming scheme.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 115 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex cb126883832..c8d1509d87d 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -106,6 +106,120 @@ static int do_as_client__status(void)\n \t}\n }\n \n+/*\n+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n+ * \"token\" as a virtual timestamp. Clients can request a summary of all\n+ * created/deleted/modified files relative to a token. In the response,\n+ * clients receive a new token for the next (relative) request.\n+ *\n+ *\n+ * Token Format\n+ * ============\n+ *\n+ * The contents of the token are private and provider-specific.\n+ *\n+ * For the built-in fsmonitor--daemon, we define a token as follows:\n+ *\n+ * \"builtin\" \":\" \":\" \n+ *\n+ * The \"builtin\" prefix is used as a namespace to avoid conflicts\n+ * with other providers (such as Watchman).\n+ *\n+ * The is an arbitrary OPAQUE string, such as a GUID,\n+ * UUID, or {timestamp,pid}. It is used to group all filesystem\n+ * events that happened while the daemon was monitoring (and in-sync\n+ * with the filesystem).\n+ *\n+ * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n+ * and does not define less-than/greater-than relationships.\n+ * (There are too many race conditions to rely on file system\n+ * event timestamps.)\n+ *\n+ * The is a simple integer incremented whenever the\n+ * daemon needs to make its state public. For example, if 1000 file\n+ * system events come in, but no clients have requested the data,\n+ * the daemon can continue to accumulate file changes in the same\n+ * bin and does not need to advance the sequence number. However,\n+ * as soon as a client does arrive, the daemon needs to start a new\n+ * bin and increment the sequence number.\n+ *\n+ * The sequence number serves as the boundary between 2 sets\n+ * of bins -- the older ones that the client has already seen\n+ * and the newer ones that it hasn't.\n+ *\n+ * When a new is created, the is reset to\n+ * zero.\n+ *\n+ *\n+ * About Token Ids\n+ * ===============\n+ *\n+ * A new token_id is created:\n+ *\n+ * [1] each time the daemon is started.\n+ *\n+ * [2] any time that the daemon must re-sync with the filesystem\n+ * (such as when the kernel drops or we miss events on a very\n+ * active volume).\n+ *\n+ * [3] in response to a client \"flush\" command (for dropped event\n+ * testing).\n+ *\n+ * When a new token_id is created, the daemon is free to discard all\n+ * cached filesystem events associated with any previous token_ids.\n+ * Events associated with a non-current token_id will never be sent\n+ * to a client. A token_id change implicitly means that the daemon\n+ * has gap in its event history.\n+ *\n+ * Therefore, clients that present a token with a stale (non-current)\n+ * token_id will always be given a trivial response.\n+ */\n+struct fsmonitor_token_data {\n+\tstruct strbuf token_id;\n+\tstruct fsmonitor_batch *batch_head;\n+\tstruct fsmonitor_batch *batch_tail;\n+\tuint64_t client_ref_count;\n+};\n+\n+static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n+{\n+\tstatic int test_env_value = -1;\n+\tstatic uint64_t flush_count = 0;\n+\tstruct fsmonitor_token_data *token;\n+\n+\tCALLOC_ARRAY(token, 1);\n+\n+\tstrbuf_init(&token->token_id, 0);\n+\ttoken->batch_head = NULL;\n+\ttoken->batch_tail = NULL;\n+\ttoken->client_ref_count = 0;\n+\n+\tif (test_env_value < 0)\n+\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n+\n+\tif (!test_env_value) {\n+\t\tstruct timeval tv;\n+\t\tstruct tm tm;\n+\t\ttime_t secs;\n+\n+\t\tgettimeofday(&tv, NULL);\n+\t\tsecs = tv.tv_sec;\n+\t\tgmtime_r(&secs, &tm);\n+\n+\t\tstrbuf_addf(&token->token_id,\n+\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n+\t\t\t flush_count++,\n+\t\t\t getpid(),\n+\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n+\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n+\t\t\t (long)tv.tv_usec);\n+\t} else {\n+\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n+\t}\n+\n+\treturn token;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -298,7 +412,7 @@ static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n-\tstate.current_token_data = NULL;\n+\tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n \tstrbuf_init(&state.path_worktree_watch, 0);\n-- \ngitgitgadget\n\n"},{"id":"449981","messageId":"e5d419b1ea0ffe57a364c09fa24912d30eb19085.1646160212.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 12/30] fsmonitor--daemon: add pathname classification","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:14Z","receivedAt":"2022-03-01T18:44:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to classify relative and absolute\npathnames and decide how they should be handled. This will\nbe used by the platform-specific backend to respond to each\nfilesystem event.\n\nWhen we register for filesystem notifications on a directory,\nwe get events for everything (recursively) in the directory.\nWe want to report to clients changes to tracked and untracked\npaths within the working directory proper. We do not want to\nreport changes within the .git directory, for example.\n\nThis classification will be used in a later commit by the\ndifferent backends to classify paths as events are received.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 81 ++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 87 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 168 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 2f721aae016..cb126883832 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -134,6 +134,87 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n+#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *rel)\n+{\n+\tif (fspathncmp(rel, \".git\", 4))\n+\t\treturn IS_WORKDIR_PATH;\n+\trel += 4;\n+\n+\tif (!*rel)\n+\t\treturn IS_DOT_GIT;\n+\tif (*rel != '/')\n+\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n+\trel++;\n+\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_DOT_GIT;\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *rel)\n+{\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_GITDIR;\n+}\n+\n+static enum fsmonitor_path_type try_classify_workdir_abs_path(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\n+\tif (fspathncmp(path, state->path_worktree_watch.buf,\n+\t\t state->path_worktree_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_worktree_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_workdir_relative(rel);\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\tenum fsmonitor_path_type t;\n+\n+\tt = try_classify_workdir_abs_path(state, path);\n+\tif (state->nr_paths_watching == 1)\n+\t\treturn t;\n+\tif (t != IS_OUTSIDE_CONE)\n+\t\treturn t;\n+\n+\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n+\t\t state->path_gitdir_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_gitdir_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_GITDIR; /* it is the exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_gitdir_relative(rel);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 3009c1a83de..8c3a71a48bd 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -30,5 +30,92 @@ struct fsmonitor_daemon_state {\n \tstruct ipc_server_data *ipc_server_data;\n };\n \n+/*\n+ * Pathname classifications.\n+ *\n+ * The daemon classifies the pathnames that it receives from file\n+ * system notification events into the following categories and uses\n+ * that to decide whether clients are told about them. (And to watch\n+ * for file system synchronization events.)\n+ *\n+ * The daemon only collects and reports on the set of modified paths\n+ * within the working directory (proper).\n+ *\n+ * The client should only care about paths within the working\n+ * directory proper (inside the working directory and not \".git\" nor\n+ * inside of \".git/\"). That is, the client has read the index and is\n+ * asking for a list of any paths in the working directory that have\n+ * been modified since the last token. The client does not care about\n+ * file system changes within the \".git/\" directory (such as new loose\n+ * objects or packfiles). So the client will only receive paths that\n+ * are classified as IS_WORKDIR_PATH.\n+ *\n+ * Note that \".git\" is usually a directory and is therefore inside\n+ * the cone of the FS watch that we have on the working directory root,\n+ * so we will also get FS events for disk activity on and within \".git/\"\n+ * that we need to respond to or filter from the client.\n+ *\n+ * But Git also allows \".git\" to be a *file* that points to a GITDIR\n+ * outside of the working directory. When this happens, we need to\n+ * create FS watches on both the working directory root *and* on the\n+ * (external) GITDIR root. (The latter is required because we put\n+ * cookie files inside it and use them to sync with the FS event\n+ * stream.)\n+ *\n+ * Note that in the context of this discussion, I'm using \"GITDIR\"\n+ * to only mean an external GITDIR referenced by a \".git\" file.\n+ *\n+ * The platform FS event backends will receive watch-specific\n+ * relative paths (except for those OS's that always emit absolute\n+ * paths). We use the following enum and routines to classify each\n+ * path so that we know how to handle it. There is a slight asymmetry\n+ * here because \".git/\" is inside the working directory and the\n+ * (external) GITDIR is not, and therefore how we handle events may\n+ * vary slightly, so I have different enums for \"IS...DOT_GIT...\" and\n+ * \"IS...GITDIR...\".\n+ *\n+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n+ * exact \".git\" file/directory or GITDIR directory. If the daemon\n+ * receives a delete event for either of these paths, it will\n+ * automatically shutdown, for example.\n+ *\n+ * Note that the daemon DOES NOT explicitly watch nor special case the\n+ * index. The daemon does not read the index nor have any internal\n+ * index-relative state, so there are no \"IS...INDEX...\" enum values.\n+ */\n+enum fsmonitor_path_type {\n+\tIS_WORKDIR_PATH = 0,\n+\n+\tIS_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n+\n+\tIS_GITDIR,\n+\tIS_INSIDE_GITDIR,\n+\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n+\n+\tIS_OUTSIDE_CONE,\n+};\n+\n+/*\n+ * Classify a pathname relative to the root of the working directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify a pathname relative to a that is external to the\n+ * worktree directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify an absolute pathname received from a filesystem event.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"449982","messageId":"5b06eb5d0e60c7d23987d915609c07364e4f7372.1646160213.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 27/30] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:29Z","receivedAt":"2022-03-01T18:45:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon client threads to create a cookie file\ninside the .git directory and then wait until FS events for the\ncookie are observed by the FS listener thread.\n\nThis helps address the racy nature of file system events by\nblocking the client response until the kernel has drained any\nevent backlog.\n\nThis is especially important on MacOS where kernel events are\nonly issued with a limited frequency. See the `latency` argument\nof `FSeventStreamCreate()`. The kernel only signals every `latency`\nseconds, but does not guarantee that the kernel queue is completely\ndrained, so we may have to wait more than one interval. If we\nincrease the latency, the system is more likely to drop events.\nWe avoid these issues by having each client thread create a unique\ncookie file and then wait until it is seen in the event stream.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 230 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 5 +\n 2 files changed, 234 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 6ca4f5959e6..97ca2a356e5 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -107,6 +107,155 @@ static int do_as_client__status(void)\n \t}\n }\n \n+enum fsmonitor_cookie_item_result {\n+\tFCIR_ERROR = -1, /* could not create cookie file ? */\n+\tFCIR_INIT = 0,\n+\tFCIR_SEEN,\n+\tFCIR_ABORT,\n+};\n+\n+struct fsmonitor_cookie_item {\n+\tstruct hashmap_entry entry;\n+\tconst char *name;\n+\tenum fsmonitor_cookie_item_result result;\n+};\n+\n+static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n+\t\t const struct hashmap_entry *he2, const void *keydata)\n+{\n+\tconst struct fsmonitor_cookie_item *a =\n+\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n+\tconst struct fsmonitor_cookie_item *b =\n+\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n+\n+\treturn strcmp(a->name, keydata ? keydata : b->name);\n+}\n+\n+static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(\n+\tstruct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint fd;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tstruct strbuf cookie_pathname = STRBUF_INIT;\n+\tstruct strbuf cookie_filename = STRBUF_INIT;\n+\tenum fsmonitor_cookie_item_result result;\n+\tint my_cookie_seq;\n+\n+\tCALLOC_ARRAY(cookie, 1);\n+\n+\tmy_cookie_seq = state->cookie_seq++;\n+\n+\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n+\n+\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n+\tstrbuf_addbuf(&cookie_pathname, &cookie_filename);\n+\n+\tcookie->name = strbuf_detach(&cookie_filename, NULL);\n+\tcookie->result = FCIR_INIT;\n+\thashmap_entry_init(&cookie->entry, strhash(cookie->name));\n+\n+\thashmap_add(&state->cookies, &cookie->entry);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n+\t\t\t cookie->name, cookie_pathname.buf);\n+\n+\t/*\n+\t * Create the cookie file on disk and then wait for a notification\n+\t * that the listener thread has seen it.\n+\t */\n+\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n+\tif (fd >= 0) {\n+\t\tclose(fd);\n+\t\tunlink(cookie_pathname.buf);\n+\n+\t\t/*\n+\t\t * Technically, this is an infinite wait (well, unless another\n+\t\t * thread sends us an abort). I'd like to change this to\n+\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n+\t\t * and let the caller do the trivial response thing, but we\n+\t\t * don't have that routine in our thread-utils.\n+\t\t *\n+\t\t * After extensive beta testing I'm not really worried about\n+\t\t * this. Also note that the above open() and unlink() calls\n+\t\t * will cause at least two FS events on that path, so the odds\n+\t\t * of getting stuck are pretty slim.\n+\t\t */\n+\t\twhile (cookie->result == FCIR_INIT)\n+\t\t\tpthread_cond_wait(&state->cookies_cond,\n+\t\t\t\t\t &state->main_lock);\n+\t} else {\n+\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n+\t\t\t cookie->name);\n+\n+\t\tcookie->result = FCIR_ERROR;\n+\t}\n+\n+\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n+\n+\tresult = cookie->result;\n+\n+\tfree((char*)cookie->name);\n+\tfree(cookie);\n+\tstrbuf_release(&cookie_pathname);\n+\n+\treturn result;\n+}\n+\n+/*\n+ * Mark these cookies as _SEEN and wake up the corresponding client threads.\n+ */\n+static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,\n+\t\t\t\t\t const struct string_list *cookie_names)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint k;\n+\tint nr_seen = 0;\n+\n+\tfor (k = 0; k < cookie_names->nr; k++) {\n+\t\tstruct fsmonitor_cookie_item key;\n+\t\tstruct fsmonitor_cookie_item *cookie;\n+\n+\t\tkey.name = cookie_names->items[k].string;\n+\t\thashmap_entry_init(&key.entry, strhash(key.name));\n+\n+\t\tcookie = hashmap_get_entry(&state->cookies, &key, entry, NULL);\n+\t\tif (cookie) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-seen: '%s'\",\n+\t\t\t\t\t cookie->name);\n+\t\t\tcookie->result = FCIR_SEEN;\n+\t\t\tnr_seen++;\n+\t\t}\n+\t}\n+\n+\tif (nr_seen)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n+/*\n+ * Set _ABORT on all pending cookies and wake up all client threads.\n+ */\n+static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct hashmap_iter iter;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tint nr_aborted = 0;\n+\n+\thashmap_for_each_entry(&state->cookies, &iter, cookie, entry) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-abort: '%s'\",\n+\t\t\t\t cookie->name);\n+\t\tcookie->result = FCIR_ABORT;\n+\t\tnr_aborted++;\n+\t}\n+\n+\tif (nr_aborted)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n /*\n * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n * \"token\" as a virtual timestamp. Clients can request a summary of all\n@@ -404,6 +553,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n *\n+ * [2] Some of those lost events may have been for cookie files. We\n+ * should assume the worst and abort them rather letting them starve.\n+ *\n * If there are no concurrent threads reading the current token data\n * series, we can free it now. Otherwise, let the last reader free\n * it.\n@@ -425,6 +577,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n \tstate->current_token_data = new_one;\n \n \tfsmonitor_free_token_data(free_me);\n+\n+\twith_lock__abort_all_cookies(state);\n }\n \n void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n@@ -500,6 +654,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tint hash_ret;\n \tint do_trivial = 0;\n \tint do_flush = 0;\n+\tint do_cookie = 0;\n+\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n@@ -560,6 +716,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * We have a V2 valid token:\n \t\t\t * \"builtin::\"\n \t\t\t */\n+\t\t\tdo_cookie = 1;\n \t\t}\n \t}\n \n@@ -568,6 +725,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tif (!state->current_token_data)\n \t\tBUG(\"fsmonitor state does not have a current token\");\n \n+\t/*\n+\t * Write a cookie file inside the directory being watched in\n+\t * an effort to flush out existing filesystem events that we\n+\t * actually care about. Suspend this client thread until we\n+\t * see the filesystem events for this cookie file.\n+\t *\n+\t * Creating the cookie lets us guarantee that our FS listener\n+\t * thread has drained the kernel queue and we are caught up\n+\t * with the kernel.\n+\t *\n+\t * If we cannot create the cookie (or otherwise guarantee that\n+\t * we are caught up), we send a trivial response. We have to\n+\t * assume that there might be some very, very recent activity\n+\t * on the FS still in flight.\n+\t */\n+\tif (do_cookie) {\n+\t\tcookie_result = with_lock__wait_for_cookie(state);\n+\t\tif (cookie_result != FCIR_SEEN) {\n+\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n+\t\t\t cookie_result);\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n \tif (do_flush)\n \t\twith_lock__do_force_resync(state);\n \n@@ -788,7 +969,9 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n-#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+#define FSMONITOR_DIR \"fsmonitor--daemon\"\n+#define FSMONITOR_COOKIE_DIR \"cookies\"\n+#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n \n enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n \tconst char *rel)\n@@ -941,6 +1124,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n+\tif (cookie_names->nr)\n+\t\twith_lock__mark_cookies_seen(state, cookie_names);\n+\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n@@ -1032,7 +1218,9 @@ static int fsmonitor_run_daemon(void)\n \n \tmemset(&state, 0, sizeof(state));\n \n+\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n+\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n@@ -1057,6 +1245,44 @@ static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n+\t/*\n+\t * We will write filesystem syncing cookie files into\n+\t * ///-.\n+\t *\n+\t * The extra layers of subdirectories here keep us from\n+\t * changing the mtime on \".git/\" or \".git/foo/\" when we create\n+\t * or delete cookie files.\n+\t *\n+\t * There have been problems with some IDEs that do a\n+\t * non-recursive watch of the \".git/\" directory and run a\n+\t * series of commands any time something happens.\n+\t *\n+\t * For example, if we place our cookie files directly in\n+\t * \".git/\" or \".git/foo/\" then a `git status` (or similar\n+\t * command) from the IDE will cause a cookie file to be\n+\t * created in one of those dirs. This causes the mtime of\n+\t * those dirs to change. This triggers the IDE's watch\n+\t * notification. This triggers the IDE to run those commands\n+\t * again. And the process repeats and the machine never goes\n+\t * idle.\n+\t *\n+\t * Adding the extra layers of subdirectories prevents the\n+\t * mtime of \".git/\" and \".git/foo\" from changing when a\n+\t * cookie file is created.\n+\t */\n+\tstrbuf_init(&state.path_cookie_prefix, 0);\n+\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1069,6 +1295,7 @@ static int fsmonitor_run_daemon(void)\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n+\tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n \n@@ -1076,6 +1303,7 @@ done:\n \n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n+\tstrbuf_release(&state.path_cookie_prefix);\n \n \treturn err;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 010fbfe60e9..bd09fffc176 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n \n \tstruct fsmonitor_token_data *current_token_data;\n \n+\tstruct strbuf path_cookie_prefix;\n+\tpthread_cond_t cookies_cond;\n+\tint cookie_seq;\n+\tstruct hashmap cookies;\n+\n \tint error_code;\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n-- \ngitgitgadget\n\n"},{"id":"449983","messageId":"1fd5439de0353a10240fac123d64109d8ddf00cd.1646160213.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 28/30] fsmonitor: force update index after large responses","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:30Z","receivedAt":"2022-03-01T18:45:01Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMeasure the time taken to apply the FSMonitor query result\nto the index and the untracked-cache.\n\nSet the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when\nFSMonitor returns a very large repsonse to ensure that the index is\nwritten to disk.\n\nNormally, when the FSMonitor response includes a tracked file, the\nindex is always updated. Similarly, the index might be updated when\nthe response alters the untracked-cache (when enabled). However, in\ncases where neither of those cause the index to be considered changed,\nthe FSMonitor response is wasted. Subsequent Git commands will make\nrequests with the same token and receive the same response.\n\nIf that response is very large, performance may suffer. It would be\nmore efficient to force update the index now (and the token in the\nindex extension) in order to reduce the size of the response received\nby future commands.\n\nThis was observed on Windows after a large checkout. On Windows, the\nkernel emits events for the files that are changed as they are\nchanged. However, it might delay events for the containing\ndirectories until the system is more idle (or someone scans the\ndirectory (so it seems)). The first status following a checkout would\nget the list of files. The subsequent status commands would get the\nlist of directories as the events trickled out. But they would never\ncatch up because the token was not advanced because the index wasn't\nupdated.\n\nThis list of directories caused `wt_status_collect_untracked()` to\nunnecessarily spend time actually scanning them during each command.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 54 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex a38b5710eb3..292a6742b4f 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -219,6 +219,43 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n+/*\n+ * The number of pathnames that we need to receive from FSMonitor\n+ * before we force the index to be updated.\n+ *\n+ * Note that any pathname within the set of received paths MAY cause\n+ * cache-entry or istate flag bits to be updated and thus cause the\n+ * index to be updated on disk.\n+ *\n+ * However, the response may contain many paths (such as ignored\n+ * paths) that will not update any flag bits. And thus not force the\n+ * index to be updated. (This is fine and normal.) It also means\n+ * that the token will not be updated in the FSMonitor index\n+ * extension. So the next Git command will find the same token in the\n+ * index, make the same token-relative request, and receive the same\n+ * response (plus any newly changed paths). If this response is large\n+ * (and continues to grow), performance could be impacted.\n+ *\n+ * For example, if the user runs a build and it writes 100K object\n+ * files but doesn't modify any source files, the index would not need\n+ * to be updated. The FSMonitor response (after the build and\n+ * relative to a pre-build token) might be 5MB. Each subsequent Git\n+ * command will receive that same 100K/5MB response until something\n+ * causes the index to be updated. And `refresh_fsmonitor()` will\n+ * have to iterate over those 100K paths each time.\n+ *\n+ * Performance could be improved if we optionally force update the\n+ * index after a very large response and get an updated token into\n+ * the FSMonitor index extension. This should allow subsequent\n+ * commands to get smaller and more current responses.\n+ *\n+ * The value chosen here does not need to be precise. The index\n+ * will be updated automatically the first time the user touches\n+ * a tracked file and causes a command like `git status` to\n+ * update an mtime to be updated and/or set a flag bit.\n+ */\n+static int fsmonitor_force_update_threshold = 100;\n+\n void refresh_fsmonitor(struct index_state *istate)\n {\n \tstruct strbuf query_result = STRBUF_INIT;\n@@ -362,25 +399,39 @@ apply_results:\n \t * information and that we should consider everything\n \t * invalid. We call this a trivial response.\n \t */\n+\ttrace2_region_enter(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tif (query_success && !is_trivial) {\n \t\t/*\n \t\t * Mark all pathnames returned by the monitor as dirty.\n \t\t *\n \t\t * This updates both the cache-entries and the untracked-cache.\n \t\t */\n+\t\tint count = 0;\n+\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n \t\t\t\tcontinue;\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n \t\t\tbol = i + 1;\n+\t\t\tcount++;\n \t\t}\n-\t\tif (bol < query_result.len)\n+\t\tif (bol < query_result.len) {\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tcount++;\n+\t\t}\n \n \t\t/* Now mark the untracked cache for fsmonitor usage */\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n+\n+\t\tif (count > fsmonitor_force_update_threshold)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", istate->repo, \"apply_count\",\n+\t\t\t\t count);\n+\n \t} else {\n \t\t/*\n \t\t * We failed to get a response or received a trivial response,\n@@ -409,6 +460,8 @@ apply_results:\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 0;\n \t}\n+\ttrace2_region_leave(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tstrbuf_release(&query_result);\n \n \t/* Now that we've updated istate, save the last_update_token */\n-- \ngitgitgadget\n\n"},{"id":"449984","messageId":"b915b95cc2f82728e3f5f7135f041a709a74ce5d.1646160213.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 30/30] update-index: convert fsmonitor warnings to advise","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:32Z","receivedAt":"2022-03-01T18:45:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/update-index.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 876112abb21..d335f1ac72a 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1238,18 +1238,18 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n-\t\t\twarning(_(\"core.fsmonitor is unset; \"\n-\t\t\t\t\"set it if you really want to \"\n-\t\t\t\t\"enable fsmonitor\"));\n+\t\t\tadvise(_(\"core.fsmonitor is unset; \"\n+\t\t\t\t \"set it if you really want to \"\n+\t\t\t\t \"enable fsmonitor\"));\n \t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \t\tif (fsm_mode > FSMONITOR_MODE_DISABLED)\n-\t\t\twarning(_(\"core.fsmonitor is set; \"\n-\t\t\t\t\"remove it if you really want to \"\n-\t\t\t\t\"disable fsmonitor\"));\n+\t\t\tadvise(_(\"core.fsmonitor is set; \"\n+\t\t\t\t \"remove it if you really want to \"\n+\t\t\t\t \"disable fsmonitor\"));\n \t\tremove_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor disabled\"));\n \t}\n-- \ngitgitgadget\n"},{"id":"449985","messageId":"6fc1430285ff814ef972ad3f3128f09e751cdb1a.1646160213.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v6 29/30] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T18:43:31Z","receivedAt":"2022-03-01T18:45:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate 2x2 test matrix with the untracked-cache and fsmonitor--daemon\nfeatures and a series of edits and verify that status output is\nidentical.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 93 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 5f7b8e54233..0ccbfb9616f 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -162,6 +162,8 @@ test_expect_success 'setup' '\n \t.gitignore\n \texpect*\n \tactual*\n+\tflush*\n+\ttrace*\n \tEOF\n \n \tgit -c core.fsmonitor=false add . &&\n@@ -508,4 +510,95 @@ test_expect_success 'cleanup worktrees' '\n \tstop_daemon_delete_repo wt-base\n '\n \n+# The next few tests perform arbitrary/contrived file operations and\n+# confirm that status is correct. That is, that the data (or lack of\n+# data) from fsmonitor doesn't cause incorrect results. And doesn't\n+# cause incorrect results when the untracked-cache is enabled.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\tgit update-index --test-untracked-cache\n+'\n+\n+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n+\ttest_unconfig core.fsmonitor &&\n+\tgit update-index --no-fsmonitor &&\n+\ttest_might_fail git fsmonitor--daemon stop\n+'\n+\n+matrix_clean_up_repo () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd\n+}\n+\n+matrix_try () {\n+\tuc=$1\n+\tfsm=$2\n+\tfn=$3\n+\n+\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n+\t\tmatrix_clean_up_repo &&\n+\t\t$fn &&\n+\t\tif test $uc = false && test $fsm = false\n+\t\tthen\n+\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n+\t\telse\n+\t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n+\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n+\t\tfi\n+\t'\n+}\n+\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+for uc_val in $uc_values\n+do\n+\tif test $uc_val = false\n+\tthen\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n+\t\t\tgit config core.untrackedcache false &&\n+\t\t\tgit update-index --no-untracked-cache\n+\t\t'\n+\telse\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n+\t\t\tgit config core.untrackedcache true &&\n+\t\t\tgit update-index --untracked-cache\n+\t\t'\n+\tfi\n+\n+\tfsm_values=\"false true\"\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tif test $fsm_val = false\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\telse\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n+\t\t\t\tgit config core.fsmonitor true &&\n+\t\t\t\tgit fsmonitor--daemon start &&\n+\t\t\t\tgit update-index --fsmonitor\n+\t\t\t'\n+\t\tfi\n+\n+\t\tmatrix_try $uc_val $fsm_val edit_files\n+\t\tmatrix_try $uc_val $fsm_val delete_files\n+\t\tmatrix_try $uc_val $fsm_val create_files\n+\t\tmatrix_try $uc_val $fsm_val rename_files\n+\t\tmatrix_try $uc_val $fsm_val file_to_directory\n+\t\tmatrix_try $uc_val $fsm_val directory_to_file\n+\n+\t\tif test $fsm_val = true\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\tfi\n+\tdone\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"449986","messageId":"xmqq35k1qyrw.fsf@gitster.g","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 00/30] Builtin FSMonitor Part 2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-01T19:05:23Z","receivedAt":"2022-03-01T19:05:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> Here is V6 of Part 2 of my builtin FSmonitor series.\n>\n> This version contains mostly cleanup based on feedback from V5. Of note:\n>\n> * I squashed in the 1_file fix for p7519.\n> * I squashed in a commit from part 3 to optionally print the \"running\n> daemon on...\" message on stderr.\n> * I added a note to the documentation about incompatible changes around\n> core.fsmonitor.\n> * Removed/rephrased some obsolete NEEDSWORK items.\n>\n> Tao has an ongoing parallel series to fix test-chmtime on Windows.\n> https://lore.kernel.org/all/pull.1166.git.1646041236.gitgitgadget@gmail.com/\n>\n> If that lands first, we should be able to drop my 't/helper/test-chmtime:\n> skip directories on Windows' commit.\n>\n> A followup Part 3 will contain additional refinements to the daemon and\n> additional tests. I drew the line here between Part 2 and 3 to make it\n> easier to review.\n\nHopefully this round will quickly be reviewed solidly to merge it\ndown to 'next' and lower.\n\nWill queue.\n\nThanks.\n"},{"id":"449988","messageId":"nycvar.QRO.7.76.6.2203012014290.11118@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 00/30] Builtin FSMonitor Part 2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-03-01T19:20:13Z","receivedAt":"2022-03-01T19:20:26Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Tue, 1 Mar 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> Here is V6 of Part 2 of my builtin FSmonitor series.\n>\n> This version contains mostly cleanup based on feedback from V5. Of note:\n>\n> * I squashed in the 1_file fix for p7519.\n> * I squashed in a commit from part 3 to optionally print the \"running\n> daemon on...\" message on stderr.\n> * I added a note to the documentation about incompatible changes around\n> core.fsmonitor.\n> * Removed/rephrased some obsolete NEEDSWORK items.\n\nThank you! I also saw a grammar fix ;-)\n\n> Tao has an ongoing parallel series to fix test-chmtime on Windows.\n> https://lore.kernel.org/all/pull.1166.git.1646041236.gitgitgadget@gmail.com/\n>\n> If that lands first, we should be able to drop my 't/helper/test-chmtime:\n> skip directories on Windows' commit.\n\nThank you for this note so that this overwhelmed reviewer knows about\nthis.\n\n> A followup Part 3 will contain additional refinements to the daemon and\n> additional tests. I drew the line here between Part 2 and 3 to make it\n> easier to review.\n\nExcellent!\n\nI only found one slight issue with the range-diff: I _assume_ that Junio\nlikes to add his own Sign-off (but I am being told frequently that my\nassumptions are typically incorrect, so take that with a grain of salt\n;-)).\n\nApart from that, I saw that you addressed all of my concerns, and I liked\nin particular this elegant change:\n\n> Range-diff vs v5:\n>\n> [...]\n> 3: 384516ce1a1 ! 3: ae622a517cf fsmonitor: config settings are repository-specific\n> [...]\n> @@ fsmonitor-settings.c (new)\n> +\n> +enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n> +{\n> ++\tif (!r)\n> ++\t\tr = the_repository;\n> ++\n> +\tlookup_fsmonitor_settings(r);\n> +\n> +\treturn r->settings.fsmonitor->mode;\n> [...]\n> @@ fsmonitor.c: void remove_fsmonitor(struct index_state *istate)\n> {\n> \tunsigned int i;\n> -\tint fsmonitor_enabled = git_config_get_fsmonitor();\n> -+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n> -+\tint fsmonitor_enabled = (fsm_settings__get_mode(r) > FSMONITOR_MODE_DISABLED);\n> ++\tint fsmonitor_enabled = (fsm_settings__get_mode(istate->repo)\n> ++\t\t\t\t > FSMONITOR_MODE_DISABLED);\n>\n> \tif (istate->fsmonitor_dirty) {\n> \t\tif (fsmonitor_enabled) {\n\nFrom my side, this patch series is all clear to advance to `next`.\n\nThank you,\nDscho\n"},{"id":"450518","messageId":"220307.86h78a2gcn.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"cdef9730b3f93a6f0f98d68ffb81bcb89d6e698e.1646160212.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T10:37:25Z","receivedAt":"2022-03-07T11:18:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n> [...]\n> +#define kFSEventStreamEventFlagNone 0x00000000\n> +#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n> +#define kFSEventStreamEventFlagUserDropped 0x00000002\n> +#define kFSEventStreamEventFlagKernelDropped 0x00000004\n> +#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n> +#define kFSEventStreamEventFlagHistoryDone 0x00000010\n> +#define kFSEventStreamEventFlagRootChanged 0x00000020\n> +#define kFSEventStreamEventFlagMount 0x00000040\n> +#define kFSEventStreamEventFlagUnmount 0x00000080\n> +#define kFSEventStreamEventFlagItemCreated 0x00000100\n> +#define kFSEventStreamEventFlagItemRemoved 0x00000200\n> +#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n> +#define kFSEventStreamEventFlagItemRenamed 0x00000800\n> +#define kFSEventStreamEventFlagItemModified 0x00001000\n> +#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n> +#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n> +#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n> +#define kFSEventStreamEventFlagItemIsFile 0x00010000\n> +#define kFSEventStreamEventFlagItemIsDir 0x00020000\n> +#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n> +#define kFSEventStreamEventFlagOwnEvent 0x00080000\n> +#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n> +#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n> +#define kFSEventStreamEventFlagItemCloned 0x00400000\n\nCan we define these as 1<<0, 1<<1, 1<<2 etc.? We do that in most other\nplaces, and it helps to quickly eyeball these and see that they don't\nhave gaps.\n\n> +#define kCFStringEncodingUTF8 0x08000100\n\nShould this be an OR of some of the above, or is it unrelated?\n\n> +typedef struct FSEventStreamContext FSEventStreamContext;\n> +typedef unsigned int FSEventStreamEventFlags;\n> +#define kFSEventStreamCreateFlagNoDefer 0x02\n> +#define kFSEventStreamCreateFlagWatchRoot 0x04\n> +#define kFSEventStreamCreateFlagFileEvents 0x10\n\nDitto 1<<0 etc.\n\n> +#else\n> +/*\n> + * Let Apple's headers declare `isalnum()` first, before\n> + * Git's headers override it via a constant\n> + */\n\n\n\n\n> +#include \n> +#include \n> +#include \n> +#endif\n\nIn cache.h which you'rejust about to include we don't include string.h,\nbut we do in git-compat-util.h, but that one includes string.h before\ndoing those overrides.\n\nThis either isn't needed, or really should be some addition to\ngit-compat-util.h instead. I.e. if we've missed some edge case with\nstring.h and ctype.h on OSX we should handle that in git-compat-util.h\nrather than needing a portability workaround.\n\n> +\n> #include \"cache.h\"\n> #include \"fsmonitor.h\"\n> #include \"fsm-listen.h\"\n\n"},{"id":"450520","messageId":"220307.86cziy2fvv.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"bc240a9e665841a622c96b8a245ce033684394f6.1646160212.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 19/30] help: include fsmonitor--daemon feature flag in version info","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T10:51:04Z","receivedAt":"2022-03-07T11:24:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Add the \"feature: fsmonitor--daemon\" message to the output of\n> `git version --build-options`.\n>\n> The builtin FSMonitor is only available on certain platforms and\n> even then only when certain Makefile flags are enabled, so print\n> a message in the verbose version output when it is available.\n>\n> This can be used by test scripts for prereq testing. Granted, tests\n> could just try `git fsmonitor--daemon status` and look for a 128 exit\n> code or grep for a \"not supported\" message on stderr, but these\n> methods are rather obscure.\n>\n> The main advantage is that the feature message will automatically\n> appear in bug reports and other support requests.\n>\n> This concept was also used during the development of Scalar for\n> similar reasons.\n>\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Junio C Hamano \n> ---\n> help.c | 4 ++++\n> t/test-lib.sh | 6 ++++++\n> 2 files changed, 10 insertions(+)\n>\n> diff --git a/help.c b/help.c\n> index 71444906ddf..9112a51e84b 100644\n> --- a/help.c\n> +++ b/help.c\n> @@ -12,6 +12,7 @@\n> #include \"refs.h\"\n> #include \"parse-options.h\"\n> #include \"prompt.h\"\n> +#include \"fsmonitor-ipc.h\"\n> \n> struct category_description {\n> \tuint32_t category;\n> @@ -695,6 +696,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n> \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n> \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n> \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n> +\n> +\t\tif (fsmonitor_ipc__is_supported())\n> +\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n> \t}\n> }\n> \n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index e4716b0b867..46cd596e7f5 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -1799,3 +1799,9 @@ test_lazy_prereq SHA1 '\n> # Tests that verify the scheduler integration must set this locally\n> # to avoid errors.\n> GIT_TEST_MAINT_SCHEDULER=\"none:exit 1\"\n> +\n> +# Does this platform support `git fsmonitor--daemon`\n> +#\n> +test_lazy_prereq FSMONITOR_DAEMON '\n> +\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n> +'\n\nAs I found recently (referenced in another series) the test_lazy_prereq\ndoesn't currently catch segfaults etc. in git even if test_must_fail and\nfriends are used.\n\nBut it's still better to future-proof things and not add more cases of\ngit on the LHS of a pipe. So instead:\n\n git version .. >out &&\n grep ...\n\nThe prereqs are run in their own temporary directory, so creating those\nfiles is OK.\n\nAlso: You run \"grep\" here twice, but as the code context shown we could\njust run it once.\n\n"},{"id":"450522","messageId":"220307.868rtm2fme.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"2b563c240eb1b592e95e89400aa5d57df1bf29b8.1646160212.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 20/30] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T10:53:27Z","receivedAt":"2022-03-07T11:25:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n> [...]\n> +/*\n> + * test-fsmonitor-client.c: client code to send commands/requests to\n> + * a `git fsmonitor--daemon` daemon.\n> + */\n> +\n> +#include \"test-tool.h\"\n> +#include \"cache.h\"\n> +#include \"parse-options.h\"\n> +#include \"fsmonitor-ipc.h\"\n> +\n> +#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n> +int cmd__fsmonitor_client(int argc, const char **argv)\n> +{\n> +\tdie(\"fsmonitor--daemon not available on this platform\");\n\nUX strings added in the series should be marked for translation with\n_(), but in this case this is correct as this is a test helper, so only\ngit developers will see it. We shouldn't waste translator time on it.\n\n> +\t\tdie(\"unable to read index file\");\n\n...ditto\n\n> +\t\tdie(_(\"could not query fsmonitor--daemon\"));\n\n\n...but this one is marked for translation, but shouldn't be.\n\nThere are other such cases in this series, found by e.g. diffing\norigin/master.. and grepping for '\"', not noting all of those in review,\nI trust you can eyeball them & see what's missing ... :)\n\n> +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n> +\t\tusage_with_options(fsmonitor_client_usage, options);\n\nI don't think you need this special handling for -h...\n\n> +\n> +\tsubcmd = argv[1];\n> +\targv--;\n> +\targc++;\n> +\n> +\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n\n...because this will do it for you.\n"},{"id":"450524","messageId":"220307.86zgm210fw.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"a70748b4640e673de32e24a77080c27da580a1df.1646160212.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 24/30] t/perf/p7519: speed up test on Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T11:09:36Z","receivedAt":"2022-03-07T11:28:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Change p7519 to use `test_seq` and `xargs` rather than a `for` loop\n> to touch thousands of files. This takes minutes off of test runs\n> on Windows because of process creation overhead.\n>\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Junio C Hamano \n> ---\n> t/perf/p7519-fsmonitor.sh | 32 ++++++++++++++++++++------------\n> 1 file changed, 20 insertions(+), 12 deletions(-)\n>\n> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n> index c8be58f3c76..aed7b1146b0 100755\n> --- a/t/perf/p7519-fsmonitor.sh\n> +++ b/t/perf/p7519-fsmonitor.sh\n> @@ -72,7 +72,7 @@ then\n> \tfi\n> fi\n> \n> -trace_start() {\n> +trace_start () {\n> \tif test -n \"$GIT_PERF_7519_TRACE\"\n> \tthen\n> \t\tname=\"$1\"\n> @@ -91,13 +91,20 @@ trace_start() {\n> \tfi\n> }\n> \n> -trace_stop() {\n> +trace_stop () {\n> \tif test -n \"$GIT_PERF_7519_TRACE\"\n> \tthen\n> \t\tunset GIT_TRACE2_PERF\n> \tfi\n> }\n\nThese minor unrelated style fixups could be split up / sent seperately?\nEspecially as the seem not to conflict hunk-wise with the actual changes\nhere.\n\n> +touch_files () {\n> +\tn=$1\n> +\td=\"$n\"_files\n> +\n> +\t(cd $d ; test_seq 1 $n | xargs touch )\n\nMissing &&-chaining for \"cd\"\n\n> -setup_for_fsmonitor() {\n> +setup_for_fsmonitor () {\n> \t# set INTEGRATION_SCRIPT depending on the environment\n> \tif test -n \"$INTEGRATION_PATH\"\n> \tthen\n> @@ -173,7 +181,7 @@ test_perf_w_drop_caches () {\n> \ttest_perf \"$@\"\n> }\n> \n> -test_fsmonitor_suite() {\n> +test_fsmonitor_suite () {\n\nditto unrelated style changes.\n\n> \tif test -n \"$INTEGRATION_SCRIPT\"; then\n> \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n> \telse\n> @@ -199,15 +207,15 @@ test_fsmonitor_suite() {\n> \n> \t# Update the mtimes on upto 100k files to make status think\n> \t# that they are dirty. For simplicity, omit any files with\n> -\t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n> -\t# Then fully backslash-quote the paths to capture any\n> -\t# whitespace so that they pass thru xargs properly.\n> +\t# LFs (i.e. anything that ls-files thinks it needs to dquote)\n> +\t# and any files with whitespace so that they pass thru xargs\n> +\t# properly.\n> \t#\n> \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n> \t\tgit ls-files | \\\n> \t\t\thead -100000 | \\\n> \t\t\tgrep -v \\\" | \\\n> -\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n> +\t\t\tegrep -v \" .\" | \\\n\nPer dcf9a748cab (t7700: replace egrep with grep, 2019-12-04) should we\nbe adding more egrep?\n\nAlso, even if we did want that, why does a the \" .\" regex need ERE over\nBRE here?\n"},{"id":"450525","messageId":"220307.864k4a2f6m.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"db8809da089f28282c6bce1752a25423811a374a.1646160212.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 21/30] t7527: create test for fsmonitor--daemon","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T10:58:58Z","receivedAt":"2022-03-07T11:29:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Junio C Hamano \n> ---\n> t/t7527-builtin-fsmonitor.sh | 511 +++++++++++++++++++++++++++++++++++\n> 1 file changed, 511 insertions(+)\n> create mode 100755 t/t7527-builtin-fsmonitor.sh\n>\n> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n> new file mode 100755\n> index 00000000000..5f7b8e54233\n> --- /dev/null\n> +++ b/t/t7527-builtin-fsmonitor.sh\n> @@ -0,0 +1,511 @@\n> +#!/bin/sh\n> +\n> +test_description='built-in file system watcher'\n> +\n> +. ./test-lib.sh\n> +\n> +if ! test_have_prereq FSMONITOR_DAEMON\n> +then\n> +\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n> +\ttest_done\n> +fi\n> +\n> +stop_daemon_delete_repo () {\n> +\tr=$1\n> +\tgit -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n\nDo we really need to quiet all its output? Why not just have the\n--verbose option do its thing?\n\n> +\trm -rf $1\n> +\treturn 0\n\nMissing &&-chaining\n> +}\n> +\n> +start_daemon () {\n> +\tcase \"$#\" in\n> +\t\t1) r=\"-C $1\";;\n> +\t\t*) r=\"\";\n\nOur usual style is not to indent these.\n\nBut maybe just use the same pattern as test_commit et al use? It's a bit\nmore verbose, but IMO clearer.\n\n> +\tesac\n> +\n> +\tgit $r fsmonitor--daemon start || return $?\n> +\tgit $r fsmonitor--daemon status || return $?\n\nMaybe don't do all this \"return\" and just &&-chain these instead (including the case/esac)?\n\n> +\n> +\treturn 0\n> +}\n> +\n> +# Is a Trace2 data event present with the given catetory and key?\n> +# We do not care what the value is.\n> +#\n> +have_t2_data_event () {\n> +\tc=$1\n> +\tk=$2\n> +\n> +\tgrep -e '\"event\":\"data\".*\"category\":\"'\"$c\"'\".*\"key\":\"'\"$k\"'\"'\n> +}\n\nOptional & aside: But it would be really nice to just have this amend\nthe test_region function into something more general, so this could be:\n\n test_trace2 --event data --category \"$c\" --key \"$k\"\n\nAnd have \"test_region\" then be a thin wrapper for that, and do the\nappropriate \"maybe only the start one, not the end one?\" logic there.\n\n> +test_expect_success 'explicit daemon start and stop' '\n> +\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n> +\n> +\tgit init test_explicit &&\n> +\tstart_daemon test_explicit &&\n> +\n> +\tgit -C test_explicit fsmonitor--daemon stop &&\n> +\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n> +'\n> +\n> +test_expect_success 'implicit daemon start' '\n> +\ttest_when_finished \"stop_daemon_delete_repo test_implicit\" &&\n> +\n> +\tgit init test_implicit &&\n> +\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n> +\n> +\t# query will implicitly start the daemon.\n> +\t#\n> +\t# for test-script simplicity, we send a V1 timestamp rather than\n> +\t# a V2 token. either way, the daemon response to any query contains\n> +\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n> +\t# but this test case is only concerned with whether the daemon was\n> +\t# implicitly started.)\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace\" \\\n\nBetter to use $PWD than $(pwd)\n\n> +delete_files () {\n> +\trm -f delete\n> +\trm -f dir1/delete\n> +\trm -f dir2/delete\n\nMore missing &&-chaining.\n\n> +}\n> +\n> +create_files () {\n> +\techo 1 >new\n> +\techo 2 >dir1/new\n> +\techo 3 >dir2/new\n> +}\n> +\n> +rename_files () {\n> +\tmv rename renamed\n> +\tmv dir1/rename dir1/renamed\n> +\tmv dir2/rename dir2/renamed\n\nditto.\n\n\n> +}\n> +\n> +file_to_directory () {\n> +\trm -f delete\n> +\tmkdir delete\n> +\techo 1 >delete/new\n\nditto.\n\n\n> +}\n> +\n> +directory_to_file () {\n> +\trm -rf dir1\n> +\techo 1 >dir1\n\nditto.\n> +}\n> +\n> +verify_status () {\n\nThis is used by nothing? Maybe it'll be used later, but that commit\ncould/should add it then?\n\nHrm, nope, just read ahead and nothing uses it at all.\n\n> +clean_up_repo_and_stop_daemon () {\n> +\tgit reset --hard HEAD\n> +\tgit clean -fd\n> +\tgit fsmonitor--daemon stop\n> +\trm -f .git/trace\n\nMissing &&-chaining (will stop noting these now, please look through the rest...)\n\n> +}\n> +\n> +test_expect_success 'edit some files' '\n> +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n> +\n> +\t(\n> +\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n> +\t\texport GIT_TRACE_FSMONITOR &&\n> +\n> +\t\tstart_daemon\n\nMaybe have this \"start_daemon\" take an optional --trace argument or\nsomething, allowing us to skip all these subsequent subshells.\n\n> +\t) &&\n> +\n> +\tedit_files &&\n> +\n> +\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n\nFor these & the rest: just skip the quieting of the output? I.e. let the\ntest's --verbose do its job?\n\n\n> +test_expect_success 'flush cached data' '\n> +\ttest_when_finished \"stop_daemon_delete_repo test_flush\" &&\n> +\n> +\tgit init test_flush &&\n> +\n> +\t(\n> +\t\tGIT_TEST_FSMONITOR_TOKEN=true &&\n> +\t\texport GIT_TEST_FSMONITOR_TOKEN &&\n> +\n> +\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace_daemon\" &&\n> +\t\texport GIT_TRACE_FSMONITOR &&\n> +\n> +\t\tstart_daemon test_flush\n> +\t) &&\n> +\n> +\t# The daemon should have an initial token with no events in _0 and\n> +\t# then a few (probably platform-specific number of) events in _1.\n> +\t# These should both have the same .\n> +\n> +\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n> +\tnul_to_q actual_q0 &&\n> +\n> +\ttouch test_flush/file_1 &&\n> +\ttouch test_flush/file_2 &&\n\nI may be missinga subtlety here, but is \"touch\" needed v.s. \">\", i.e. we\njust created \"test_flush\", so if we create a new file it'll have the\ncurrent timestamp.\n\nOr did it get created by the helpers?\n"},{"id":"450526","messageId":"220307.86v8wq10cb.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"b0c9c9c7a4005e2aa2a587fe5bc5588bf5a32d65.1646160212.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 25/30] t/perf/p7519: add fsmonitor--daemon test cases","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T11:12:20Z","receivedAt":"2022-03-07T11:29:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Repeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\n> the \"Simple IPC\" interface.\n>\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Junio C Hamano \n> ---\n> t/perf/p7519-fsmonitor.sh | 29 ++++++++++++++++++++++++++++-\n> 1 file changed, 28 insertions(+), 1 deletion(-)\n>\n> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n> index aed7b1146b0..a1c552129cc 100755\n> --- a/t/perf/p7519-fsmonitor.sh\n> +++ b/t/perf/p7519-fsmonitor.sh\n> @@ -182,7 +182,10 @@ test_perf_w_drop_caches () {\n> }\n> \n> test_fsmonitor_suite () {\n> -\tif test -n \"$INTEGRATION_SCRIPT\"; then\n> +\tif test -n \"$USE_FSMONITOR_DAEMON\"\n> +\tthen\n\n...another candidate for \"shell script style change\" re my comment on\n25/30.\n\n> +\t\tDESC=\"builtin fsmonitor--daemon\"\n> +\telif test -n \"$INTEGRATION_SCRIPT\"; then\n\n\n...but it's especially odd in this case, becuse here the added code\n*doesn't* follow our usual style... :)\n\nSince this hunk-wise would conflict with the meaningful change I don't\nmind if we \"fix it while we're at it\", but it really should use the\npreferred style for new code then.\n\n> \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n> \telse\n> \t\tDESC=\"fsmonitor=disabled\"\n> @@ -293,4 +296,28 @@ test_expect_success \"setup without fsmonitor\" '\n> test_fsmonitor_suite\n> trace_stop\n> \n> +#\n> +# Run a full set of perf tests using the built-in fsmonitor--daemon.\n> +# It does not use the Hook API, so it has a different setup.\n> +# Explicitly start the daemon here and before we start client commands\n> +# so that we can later add custom tracing.\n> +#\n> +if test_have_prereq FSMONITOR_DAEMON\n> +then\n> +\tUSE_FSMONITOR_DAEMON=t\n> +\n> +\ttrace_start fsmonitor--daemon--server\n> +\tgit fsmonitor--daemon start\n> +\n> +\ttrace_start fsmonitor--daemon--client\n> +\n> +\tgit config core.fsmonitor true\n> +\tgit update-index --fsmonitor\n> +\n> +\ttest_fsmonitor_suite\n> +\n> +\tgit fsmonitor--daemon stop\n> +\ttrace_stop\n> +fi\n\nUrm, shouldn't this be in a test_perf or test_expect_success?\n\n> test_done\n\n"},{"id":"450527","messageId":"220307.86r17e0zv0.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"5b06eb5d0e60c7d23987d915609c07364e4f7372.1646160213.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 27/30] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T11:15:26Z","receivedAt":"2022-03-07T11:31:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n> [...]\n> +enum fsmonitor_cookie_item_result {\n> +\tFCIR_ERROR = -1, /* could not create cookie file ? */\n> +\tFCIR_INIT = 0,\n\nnit: redundant = 0 assignment, i.e....\n\n> +\tFCIR_SEEN,\n> +\tFCIR_ABORT,\n\nIf we're going to make these implicit (which we usually do) let's do the\nsame for FCIR_INIT.\n\n> +\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n\nFor trace2 we encoded getpid() in the SID using a fixed-width encoding\nformat, maybe do the same here?\n\n> +\t/*\n> +\t * Create the cookie file on disk and then wait for a notification\n> +\t * that the listener thread has seen it.\n> +\t */\n> +\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n> +\tif (fd >= 0) {\n> +\t\tclose(fd);\n\nAlso check the return value of close()...\n\n> +\t\tunlink(cookie_pathname.buf);\n\n...and unlink()? If we explicitly prefer not to, note why?\n> +\n> +\t\t/*\n> +\t\t * Technically, this is an infinite wait (well, unless another\n> +\t\t * thread sends us an abort). I'd like to change this to\n> +\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n> +\t\t * and let the caller do the trivial response thing, but we\n> +\t\t * don't have that routine in our thread-utils.\n> +\t\t *\n> +\t\t * After extensive beta testing I'm not really worried about\n> +\t\t * this. Also note that the above open() and unlink() calls\n> +\t\t * will cause at least two FS events on that path, so the odds\n> +\t\t * of getting stuck are pretty slim.\n> +\t\t */\n> +\t\twhile (cookie->result == FCIR_INIT)\n> +\t\t\tpthread_cond_wait(&state->cookies_cond,\n> +\t\t\t\t\t &state->main_lock);\n> +\t} else {\n> +\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n> +\t\t\t cookie->name);\n> +\n> +\t\tcookie->result = FCIR_ERROR;\n\n...more readable if we just:\n\n if (open(...) < 0) {\n ...;\n goto cleanup;\n\nSo no need to indent the comment/main logic of the function above?\n\n> +\t}\n> +\n> +\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n> +\n> +\tresult = cookie->result;\n> +\n> +\tfree((char*)cookie->name);\n\nSince the \"name\" here is always an allocated string we should just make\nit \"char *\" in the struct and avoid these casts.\n\nIn other places we play this casting game because the struct is a public\nAPI, but in this case it's all ours.\n"},{"id":"450529","messageId":"220307.86mti20zc3.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"b915b95cc2f82728e3f5f7135f041a709a74ce5d.1646160213.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 30/30] update-index: convert fsmonitor warnings to advise","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T11:29:34Z","receivedAt":"2022-03-07T11:36:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Junio C Hamano \n> ---\n> builtin/update-index.c | 12 ++++++------\n> 1 file changed, 6 insertions(+), 6 deletions(-)\n>\n> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> index 876112abb21..d335f1ac72a 100644\n> --- a/builtin/update-index.c\n> +++ b/builtin/update-index.c\n> @@ -1238,18 +1238,18 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n> \tif (fsmonitor > 0) {\n> \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n> \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n> -\t\t\twarning(_(\"core.fsmonitor is unset; \"\n> -\t\t\t\t\"set it if you really want to \"\n> -\t\t\t\t\"enable fsmonitor\"));\n> +\t\t\tadvise(_(\"core.fsmonitor is unset; \"\n> +\t\t\t\t \"set it if you really want to \"\n> +\t\t\t\t \"enable fsmonitor\"));\n> \t\t}\n> \t\tadd_fsmonitor(&the_index);\n> \t\treport(_(\"fsmonitor enabled\"));\n> \t} else if (!fsmonitor) {\n> \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n> \t\tif (fsm_mode > FSMONITOR_MODE_DISABLED)\n> -\t\t\twarning(_(\"core.fsmonitor is set; \"\n> -\t\t\t\t\"remove it if you really want to \"\n> -\t\t\t\t\"disable fsmonitor\"));\n> +\t\t\tadvise(_(\"core.fsmonitor is set; \"\n> +\t\t\t\t \"remove it if you really want to \"\n> +\t\t\t\t \"disable fsmonitor\"));\n> \t\tremove_fsmonitor(&the_index);\n> \t\treport(_(\"fsmonitor disabled\"));\n> \t}\n\nFor any new advice you should:\n\n * Add it to advice.[ch]\n * Add it to Documentation/config/advise.txt\n * This should use advise_if_enabled() with the relevant enum you add\n (which should also be documented in the *.txt)\n\nI'm not oppposed to this change per-se, but I think this narrow change\nmakes the post-image of builtin/update-index.c worse.\n\nWhy are we issuing an advice() only for this config, but not changing\nthe other two uses of warning()? In the post-image:\n \n 6 matches for \"\\(advise\\|warning\\)\" in buffer: update-index.c 1195: warning(_(\"core.splitIndex is set to false; \"\n 1204: warning(_(\"core.splitIndex is set to true; \"\n 1216: warning(_(\"core.untrackedCache is set to true; \"\n 1228: warning(_(\"core.untrackedCache is set to false; \"\n 1241: advise(_(\"core.fsmonitor is unset; \"\n 1250: advise(_(\"core.fsmonitor is set; \"\n\nWe should either leave these alone, or give them all the treatment of\nadding a new advise.updateIndexConfig or whatever, and have them all use\nadvise_if_enabled(ADVICE_UPDATE_INDEX_CONFIG, ...).\n\nAlso these existing messages seem really confusing both for the\nfsmonitor & others. We're telling the user \"do XYZ to disable ABC\", but\nthen right afterwards we say \"ABC disabled\" via report()?\n\n"},{"id":"450771","messageId":"9bc32c17-7249-303f-e579-8992c5bc8497@jeffhostetler.com","threadId":"56518","inReplyTo":"220307.86h78a2gcn.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-08T20:26:13Z","receivedAt":"2022-03-08T20:26:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>> [...]\n>> +#define kFSEventStreamEventFlagNone 0x00000000\n[...]\n>> +#define kFSEventStreamEventFlagItemCloned 0x00400000\n> \n> Can we define these as 1<<0, 1<<1, 1<<2 etc.? We do that in most other\n> places, and it helps to quickly eyeball these and see that they don't\n> have gaps.\n\nAll of these constants are defined by Apple in their header\nfiles and system documentation. For example, see:\nhttps://developer.apple.com/documentation/coreservices/1455361-fseventstreameventflags/kfseventstreameventflagitemcloned\n\nThe set is relatively fixed by Apple and we won't be adding any\n(since they define the bits in a FS event from the kernel).\n\nChanging them to shifts would be wrong.\n\n\nthanks\nJeff\n"},{"id":"450772","messageId":"aa6276f9-8d10-22f9-bfc0-2aa718d002e1@jeffhostetler.com","threadId":"56518","inReplyTo":"220307.86h78a2gcn.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-08T21:09:37Z","receivedAt":"2022-03-08T21:09:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>> [...]\n[...]\n> \n>> +#else\n>> +/*\n>> + * Let Apple's headers declare `isalnum()` first, before\n>> + * Git's headers override it via a constant\n>> + */\n> \n> \n> \n> \n>> +#include \n>> +#include \n>> +#include \n>> +#endif\n> \n> In cache.h which you'rejust about to include we don't include string.h,\n> but we do in git-compat-util.h, but that one includes string.h before\n> doing those overrides.\n> \n> This either isn't needed, or really should be some addition to\n> git-compat-util.h instead. I.e. if we've missed some edge case with\n> string.h and ctype.h on OSX we should handle that in git-compat-util.h\n> rather than needing a portability workaround.\n> \n>> +\n>> #include \"cache.h\"\n>> #include \"fsmonitor.h\"\n>> #include \"fsm-listen.h\"\n> \n\nYou may be right here. I commented out the and\nthe <...CoreFoundation.h> lines and everything still compiled\nand t7527 passed.\n\nI'm not sure why was added here (I inherited that\nfile when I took over the feature). It may be that recent SDK\nupdates have eliminated the need for it. Or it may be that it\nwas never necessary. (However, the comment above it suggests\nthat there was a problem in the past.)\n\nWhile it may not (now at least) be necessary, it's not doing\nany harm, so I'd rather leave it and not interrupt things.\nWe can always revisit it later if we want.\n\nThanks,\nJeff\n"},{"id":"450773","messageId":"1442043e-d8d4-715a-7534-32f76536eb41@jeffhostetler.com","threadId":"56518","inReplyTo":"220307.86cziy2fvv.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 19/30] help: include fsmonitor--daemon feature flag in version info","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-08T21:19:59Z","receivedAt":"2022-03-08T21:20:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/7/22 5:51 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n[...]\n>> +# Does this platform support `git fsmonitor--daemon`\n>> +#\n>> +test_lazy_prereq FSMONITOR_DAEMON '\n>> +\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n>> +'\n> \n> As I found recently (referenced in another series) the test_lazy_prereq\n> doesn't currently catch segfaults etc. in git even if test_must_fail and\n> friends are used.\n> \n> But it's still better to future-proof things and not add more cases of\n> git on the LHS of a pipe. So instead:\n> \n> git version .. >out &&\n> grep ...\n> \n> The prereqs are run in their own temporary directory, so creating those\n> files is OK.\n\nYes, I think I saw a series on this topic in my inbox recently.\n\nThis command is very safe, so I'd rather not reroll just for this.\n\n\n> \n> Also: You run \"grep\" here twice, but as the code context shown we could\n> just run it once.\n> \n\nTrue, but I think this can wait too.\n\nThanks,\nJeff\n"},{"id":"450867","messageId":"1a060357-3296-81d5-bf23-a55263ef6d10@jeffhostetler.com","threadId":"56518","inReplyTo":"220307.86h78a2gcn.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-09T13:37:41Z","receivedAt":"2022-03-09T13:37:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>> [...]\n[...]\n\nÆvar sent feedback (thanks!) on 8 commits in the V6 version\non March 7. I started responding to each as I got to them\nin my inbox yesterday, but I'd like to take a break from\nresponding individually to each of them inside of Part 2.\n\nSince most of the feedback is for \"general cleanup\" and since\nPart 2 V6 is already in \"next\", I'd like to revisit these\nissues with a few \"cleanup\" commits on top of Part 3 (which\nis still in active review), rather than re-rolling or\nappending \"fixup\" commits onto Part 2.\n\nI think this would be quicker and easier in the long run\nand give us the same net result.\n\nThanks,\nJeff\n\n"},{"id":"450870","messageId":"220309.86fsnrmdv2.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"9bc32c17-7249-303f-e579-8992c5bc8497@jeffhostetler.com","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-09T13:52:17Z","receivedAt":"2022-03-09T13:53:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 08 2022, Jeff Hostetler wrote:\n\n> On 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n>> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> From: Jeff Hostetler \n>>> [...]\n>>> +#define kFSEventStreamEventFlagNone 0x00000000\n> [...]\n>>> +#define kFSEventStreamEventFlagItemCloned 0x00400000\n>> Can we define these as 1<<0, 1<<1, 1<<2 etc.? We do that in most\n>> other\n>> places, and it helps to quickly eyeball these and see that they don't\n>> have gaps.\n>\n> All of these constants are defined by Apple in their header\n> files and system documentation. For example, see:\n> https://developer.apple.com/documentation/coreservices/1455361-fseventstreameventflags/kfseventstreameventflagitemcloned\n>\n> The set is relatively fixed by Apple and we won't be adding any\n> (since they define the bits in a FS event from the kernel).\n>\n> Changing them to shifts would be wrong.\n\nAh, I missed the \"ifndef __clang__\" at the start, so most of it is not\nneeded except with gcc.\n\nFWIW I think having that whole part just split into\ncompat/fsmonitor/darwin-gcc.h would make it obvious where all the\nGCC-specific hackery is.\n"},{"id":"450871","messageId":"220309.86bkyfmctk.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"aa6276f9-8d10-22f9-bfc0-2aa718d002e1@jeffhostetler.com","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-09T14:14:40Z","receivedAt":"2022-03-09T14:15:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 08 2022, Jeff Hostetler wrote:\n\n> On 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n>> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> From: Jeff Hostetler \n>>> [...]\n> [...]\n>> \n>>> +#else\n>>> +/*\n>>> + * Let Apple's headers declare `isalnum()` first, before\n>>> + * Git's headers override it via a constant\n>>> + */\n>> \n>> \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#endif\n>> In cache.h which you'rejust about to include we don't include\n>> string.h,\n>> but we do in git-compat-util.h, but that one includes string.h before\n>> doing those overrides.\n>> This either isn't needed, or really should be some addition to\n>> git-compat-util.h instead. I.e. if we've missed some edge case with\n>> string.h and ctype.h on OSX we should handle that in git-compat-util.h\n>> rather than needing a portability workaround.\n>> \n>>> +\n>>> #include \"cache.h\"\n>>> #include \"fsmonitor.h\"\n>>> #include \"fsm-listen.h\"\n>> \n>\n> You may be right here. I commented out the and\n> the <...CoreFoundation.h> lines and everything still compiled\n> and t7527 passed.\n>\n> I'm not sure why was added here (I inherited that\n> file when I took over the feature). It may be that recent SDK\n> updates have eliminated the need for it. Or it may be that it\n> was never necessary. (However, the comment above it suggests\n> that there was a problem in the past.)\n>\n> While it may not (now at least) be necessary, it's not doing\n> any harm, so I'd rather leave it and not interrupt things.\n> We can always revisit it later if we want.\n\nIn terms of figuring out some mysterious portability issue, I think the\nright time is now rather than later.\n\nI.e. now this doesn't have anyone relying on it, so we can easily\n(re)discover whatever issue this was trying to solve.\n\nWhereas anyone who'd need to figure out why we include string.h on OSX\nearly in this case later would be left with this otherwise dead-end\nthread, and a change at that point would possibly break existing code...\n"},{"id":"450906","messageId":"xmqqk0d3kl73.fsf@gitster.g","threadId":"56518","inReplyTo":"1a060357-3296-81d5-bf23-a55263ef6d10@jeffhostetler.com","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-09T18:57:52Z","receivedAt":"2022-03-09T18:57:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> On 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n>> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> From: Jeff Hostetler \n>>> [...]\n> [...]\n>\n> Ævar sent feedback (thanks!) on 8 commits in the V6 version\n> on March 7. I started responding to each as I got to them\n> in my inbox yesterday, but I'd like to take a break from\n> responding individually to each of them inside of Part 2.\n>\n> Since most of the feedback is for \"general cleanup\" and since\n> Part 2 V6 is already in \"next\", I'd like to revisit these\n> issues with a few \"cleanup\" commits on top of Part 3 (which\n> is still in active review), rather than re-rolling or\n> appending \"fixup\" commits onto Part 2.\n\nSounds good. Prepending \"preliminary clean-up\" before part 3 would\nbe even cleaner, I would suspect.\n\nIn any case, let's consider part 2 \"more or less done\" unless we see\na glaring mistake that requires us to revert and redo it from\nscratch.\n\nThanks.\n"},{"id":"450919","messageId":"220309.8635jqncfr.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"xmqqk0d3kl73.fsf@gitster.g","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-09T19:37:32Z","receivedAt":"2022-03-09T19:38:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Mar 09 2022, Junio C Hamano wrote:\n\n> Jeff Hostetler writes:\n>\n>> On 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n>>> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n>>> \n>>>> From: Jeff Hostetler \n>>>> [...]\n>> [...]\n>>\n>> Ævar sent feedback (thanks!) on 8 commits in the V6 version\n>> on March 7. I started responding to each as I got to them\n>> in my inbox yesterday, but I'd like to take a break from\n>> responding individually to each of them inside of Part 2.\n>>\n>> Since most of the feedback is for \"general cleanup\" and since\n>> Part 2 V6 is already in \"next\", I'd like to revisit these\n>> issues with a few \"cleanup\" commits on top of Part 3 (which\n>> is still in active review), rather than re-rolling or\n>> appending \"fixup\" commits onto Part 2.\n>\n> Sounds good. Prepending \"preliminary clean-up\" before part 3 would\n> be even cleaner, I would suspect.\n>\n> In any case, let's consider part 2 \"more or less done\" unless we see\n> a glaring mistake that requires us to revert and redo it from\n> scratch.\n\nSounds good, my comments on v6 today were before I'd noticed that it was\nin \"next\", I think all of those can (well, it would be that way either\nway at this point...) be left for some potential follow-up.\n\nThanks both, especially Jeff for sticking with this fsmonitor topic for\nso long & keeping it advancing.\n"},{"id":"451021","messageId":"nycvar.QRO.7.76.6.2203101445360.357@tvgsbejvaqbjf.bet","threadId":"56518","inReplyTo":"220309.86bkyfmctk.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-03-10T14:32:28Z","receivedAt":"2022-03-10T14:53:58Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Wed, 9 Mar 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> On Tue, Mar 08 2022, Jeff Hostetler wrote:\n>\n> > On 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n> >> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n> >>\n> >>> From: Jeff Hostetler \n> >>> [...]\n> > [...]\n> >>\n> >>> +#else\n> >>> +/*\n> >>> + * Let Apple's headers declare `isalnum()` first, before\n> >>> + * Git's headers override it via a constant\n> >>> + */\n> >>\n> >>\n> >>> +#include \n> >>> +#include \n> >>> +#include \n> >>> +#endif\n> >>\n> >> In cache.h which you'rejust about to include we don't include\n> >> string.h, but we do in git-compat-util.h, but that one includes\n> >> string.h before doing those overrides.\n> >>\n> >> This either isn't needed, or really should be some addition to\n> >> git-compat-util.h instead. I.e. if we've missed some edge case with\n> >> string.h and ctype.h on OSX we should handle that in git-compat-util.h\n> >> rather than needing a portability workaround.\n> >\n> > [...]\n> >\n> > While it may not (now at least) be necessary, it's not doing\n> > any harm, so I'd rather leave it and not interrupt things.\n> > We can always revisit it later if we want.\n>\n> In terms of figuring out some mysterious portability issue, I think the\n> right time is now rather than later.\n\nI do not see that.\n\nIn FSMonitor, this was clearly a problem that needed to be solved (and if\nyou try to compile on an older macOS, you will run into those problems\nagain).\n\nIf you are talking about the mysterious portability issue with an eye on\n`git-compat-util.h`, well... you can successfully compile Git's source\ncode without this hack in `git-compat-util.h`. That's why the hack is not\nneeded. Problem solved. Actually, there was not even a problem.\n\n> I.e. now this doesn't have anyone relying on it, so we can easily\n> (re)discover whatever issue this was trying to solve.\n>\n> Whereas anyone who'd need to figure out why we include string.h on OSX\n> early in this case later would be left with this otherwise dead-end\n> thread, and a change at that point would possibly break existing code...\n\nAnyone who would need to figure out why we `#include` this header early\nwould read the comment about `isalnum()`, I would hope, and understand\nthat there are circumstances when Git's `isalnum()` macro interferes with\nApple's, and that this `#include` order addresses that problem.\n\nThey might even get to the point where they find\nhttps://github.com/dscho/git/commit/0f89c726a1912dce2bdab14aff8ebfec8550340d,\nmaybe even the \"original original\" commit at\nhttps://github.com/kewillford/git/commit/d11ee4698c63347f40a8993ab86ee4e97f695d9b,\nwhich was a still-experimental version of the macOS backend, and where the\n`#include` order clearly mattered, else why would Kevin have bothered.\n\nFurther, I strongly suspect that it had something to do with\n`CoreFoundation.h` or with `CoreServices.h` being `#include`d, and if you\ncare to check the code above the quoted lines, you will see that you\ncannot even `#include` those headers using GCC, it only works with clang:\nhttps://github.com/jeffhostetler/git/commit/cdef9730b3f93a6f0f98d68ffb81bcb89d6e698e#diff-4e865160016fe490b80ad11273a10daca8bc412a75f2da4c6b08fb9e5e3b5e18R3\n\nAt this stage, anybody investigating this issue who is a little bit like\nme would then be a bit bored with the investigation because there is\nactually no breakage here, just a curious `#include` order, and nothing\nelse. So they might then drop it and move along.\n\nEven you might find it a more satisfying use of your time to implement,\nsay, a Linux backend for FSMonitor on top of Jeff's work, instead of\nworrying about non-issues ;-)\n\nReally, there are so many more interesting issues to discuss than this\n`#include` non-issue. And on this note, I will steer my attention to\nprecisely such more interesting issues.\n\nCiao,\nJohannes\n"},{"id":"451024","messageId":"220310.86a6dxkgjr.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"nycvar.QRO.7.76.6.2203101445360.357@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-10T14:42:00Z","receivedAt":"2022-03-10T14:54:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 10 2022, Johannes Schindelin wrote:\n\n> Hi Ævar,\n>\n> On Wed, 9 Mar 2022, Ævar Arnfjörð Bjarmason wrote:\n>\n>> On Tue, Mar 08 2022, Jeff Hostetler wrote:\n>>\n>> > On 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n>> >> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n>> >>\n>> >>> From: Jeff Hostetler \n>> >>> [...]\n>> > [...]\n>> >>\n>> >>> +#else\n>> >>> +/*\n>> >>> + * Let Apple's headers declare `isalnum()` first, before\n>> >>> + * Git's headers override it via a constant\n>> >>> + */\n>> >>\n>> >>\n>> >>> +#include \n>> >>> +#include \n>> >>> +#include \n>> >>> +#endif\n>> >>\n>> >> In cache.h which you'rejust about to include we don't include\n>> >> string.h, but we do in git-compat-util.h, but that one includes\n>> >> string.h before doing those overrides.\n>> >>\n>> >> This either isn't needed, or really should be some addition to\n>> >> git-compat-util.h instead. I.e. if we've missed some edge case with\n>> >> string.h and ctype.h on OSX we should handle that in git-compat-util.h\n>> >> rather than needing a portability workaround.\n>> >\n>> > [...]\n>> >\n>> > While it may not (now at least) be necessary, it's not doing\n>> > any harm, so I'd rather leave it and not interrupt things.\n>> > We can always revisit it later if we want.\n>>\n>> In terms of figuring out some mysterious portability issue, I think the\n>> right time is now rather than later.\n>\n> I do not see that.\n>\n> In FSMonitor, this was clearly a problem that needed to be solved (and if\n> you try to compile on an older macOS, you will run into those problems\n> again).\n\nSo you can reproduce an issue where removing the \"#include \"\nfrom compat/fsmonitor/fsm-listen-darwin.c has an effect? Does swaping it\nfor \"ctype.h\" also solve that issue?\n\nI was asking why the non-obvious portability hack was needed, and it\nseems Jeff suggested it might not be upthread in\n, but here you\nseem to have a reproduction of in being needed, without more of the\nrelevant details (e.g. what OSX version(s))?\n\n> If you are talking about the mysterious portability issue with an eye on\n> `git-compat-util.h`, well... you can successfully compile Git's source\n> code without this hack in `git-compat-util.h`. That's why the hack is not\n> needed. Problem solved. Actually, there was not even a problem.\n\nDo you mean we don't need the ctype.h overrides in git-compat-util.h at\nall? I haven't looked into it, but needing to\n\n>> I.e. now this doesn't have anyone relying on it, so we can easily\n>> (re)discover whatever issue this was trying to solve.\n>>\n>> Whereas anyone who'd need to figure out why we include string.h on OSX\n>> early in this case later would be left with this otherwise dead-end\n>> thread, and a change at that point would possibly break existing code...\n>\n> Anyone who would need to figure out why we `#include` this header early\n> would read the comment about `isalnum()`, I would hope, and understand\n> that there are circumstances when Git's `isalnum()` macro interferes with\n> Apple's, and that this `#include` order addresses that problem.\n>\n> They might even get to the point where they find\n> https://github.com/dscho/git/commit/0f89c726a1912dce2bdab14aff8ebfec8550340d,\n> maybe even the \"original original\" commit at\n> https://github.com/kewillford/git/commit/d11ee4698c63347f40a8993ab86ee4e97f695d9b,\n> which was a still-experimental version of the macOS backend, and where the\n> `#include` order clearly mattered, else why would Kevin have bothered.\n>\n> Further, I strongly suspect that it had something to do with\n> `CoreFoundation.h` or with `CoreServices.h` being `#include`d, and if you\n> care to check the code above the quoted lines, you will see that you\n> cannot even `#include` those headers using GCC, it only works with clang:\n> https://github.com/jeffhostetler/git/commit/cdef9730b3f93a6f0f98d68ffb81bcb89d6e698e#diff-4e865160016fe490b80ad11273a10daca8bc412a75f2da4c6b08fb9e5e3b5e18R3\n>\n> At this stage, anybody investigating this issue who is a little bit like\n> me would then be a bit bored with the investigation because there is\n> actually no breakage here, just a curious `#include` order, and nothing\n> else. So they might then drop it and move along.\n\nMy implicit observation upthread is that those sorts of details would\nideally be included in a comment or the commit message. I.e. I didn't\nquite see why it was needed, and neither could the person submitting the\npatch series when asked.\n\nSure, it's a minor issue, but patch review is also meant to uncover such\nsmall issues.\n"},{"id":"451032","messageId":"4fec0300-ff6b-f981-49f0-f8002847385f@jeffhostetler.com","threadId":"56518","inReplyTo":"220310.86a6dxkgjr.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-10T15:42:27Z","receivedAt":"2022-03-10T15:42:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/10/22 9:42 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Mar 10 2022, Johannes Schindelin wrote:\n> \n>> Hi Ævar,\n>>\n>> On Wed, 9 Mar 2022, Ævar Arnfjörð Bjarmason wrote:\n>>\n>>> On Tue, Mar 08 2022, Jeff Hostetler wrote:\n>>>\n>>>> On 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n>>>>> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n>>>>>\n>>>>>> From: Jeff Hostetler \n>>>>>> [...]\n>>>> [...]\n>>>>>\n>>>>>> +#else\n>>>>>> +/*\n>>>>>> + * Let Apple's headers declare `isalnum()` first, before\n>>>>>> + * Git's headers override it via a constant\n>>>>>> + */\n>>>>>\n>>>>>\n>>>>>> +#include \n>>>>>> +#include \n>>>>>> +#include \n>>>>>> +#endif\n>>>>>\n>>>>> In cache.h which you'rejust about to include we don't include\n>>>>> string.h, but we do in git-compat-util.h, but that one includes\n>>>>> string.h before doing those overrides.\n>>>>>\n>>>>> This either isn't needed, or really should be some addition to\n>>>>> git-compat-util.h instead. I.e. if we've missed some edge case with\n>>>>> string.h and ctype.h on OSX we should handle that in git-compat-util.h\n>>>>> rather than needing a portability workaround.\n>>>>\n>>>> [...]\n>>>>\n>>>> While it may not (now at least) be necessary, it's not doing\n>>>> any harm, so I'd rather leave it and not interrupt things.\n>>>> We can always revisit it later if we want.\n>>>\n>>> In terms of figuring out some mysterious portability issue, I think the\n>>> right time is now rather than later.\n>>\n>> I do not see that.\n>>\n>> In FSMonitor, this was clearly a problem that needed to be solved (and if\n>> you try to compile on an older macOS, you will run into those problems\n>> again).\n> \n> So you can reproduce an issue where removing the \"#include \"\n> from compat/fsmonitor/fsm-listen-darwin.c has an effect? Does swaping it\n> for \"ctype.h\" also solve that issue?\n> \n> I was asking why the non-obvious portability hack was needed, and it\n> seems Jeff suggested it might not be upthread in\n> , but here you\n> seem to have a reproduction of in being needed, without more of the\n> relevant details (e.g. what OSX version(s))?\n> \n>> If you are talking about the mysterious portability issue with an eye on\n>> `git-compat-util.h`, well... you can successfully compile Git's source\n>> code without this hack in `git-compat-util.h`. That's why the hack is not\n>> needed. Problem solved. Actually, there was not even a problem.\n> \n> Do you mean we don't need the ctype.h overrides in git-compat-util.h at\n> all? I haven't looked into it, but needing to\n> \n>>> I.e. now this doesn't have anyone relying on it, so we can easily\n>>> (re)discover whatever issue this was trying to solve.\n>>>\n>>> Whereas anyone who'd need to figure out why we include string.h on OSX\n>>> early in this case later would be left with this otherwise dead-end\n>>> thread, and a change at that point would possibly break existing code...\n>>\n>> Anyone who would need to figure out why we `#include` this header early\n>> would read the comment about `isalnum()`, I would hope, and understand\n>> that there are circumstances when Git's `isalnum()` macro interferes with\n>> Apple's, and that this `#include` order addresses that problem.\n>>\n>> They might even get to the point where they find\n>> https://github.com/dscho/git/commit/0f89c726a1912dce2bdab14aff8ebfec8550340d,\n>> maybe even the \"original original\" commit at\n>> https://github.com/kewillford/git/commit/d11ee4698c63347f40a8993ab86ee4e97f695d9b,\n>> which was a still-experimental version of the macOS backend, and where the\n>> `#include` order clearly mattered, else why would Kevin have bothered.\n>>\n>> Further, I strongly suspect that it had something to do with\n>> `CoreFoundation.h` or with `CoreServices.h` being `#include`d, and if you\n>> care to check the code above the quoted lines, you will see that you\n>> cannot even `#include` those headers using GCC, it only works with clang:\n>> https://github.com/jeffhostetler/git/commit/cdef9730b3f93a6f0f98d68ffb81bcb89d6e698e#diff-4e865160016fe490b80ad11273a10daca8bc412a75f2da4c6b08fb9e5e3b5e18R3\n>>\n>> At this stage, anybody investigating this issue who is a little bit like\n>> me would then be a bit bored with the investigation because there is\n>> actually no breakage here, just a curious `#include` order, and nothing\n>> else. So they might then drop it and move along.\n> \n> My implicit observation upthread is that those sorts of details would\n> ideally be included in a comment or the commit message. I.e. I didn't\n> quite see why it was needed, and neither could the person submitting the\n> patch series when asked.\n> \n> Sure, it's a minor issue, but patch review is also meant to uncover such\n> small issues.\n> \n\nThere are two independent issues here.\n(1) compiling something that includes with GCC.\n(2) the need for the #include when compiling with clang.\n\nTo address (1), we've #ifdef'd the top of the file and insert just\nthe essential typedefs and prototypes. (I'll pull them into a separate\nlocal header file as you suggested earlier to make that easier to see.)\nBut otherwise, GCC is not an issue.\n\nWRT (2) I have tried clang-11 on macOS 10.15 and clang-13 on 11.6 both\nwith and without the and it doesn't matter. Everything\ncompiles and t7527 passes in all [2x2] cases. So I have to assume that\nsomething has changed in the Apple/clang SDK or runtime libraries or\nour source code in that single source file in the ~2 years since Kevin\nneeded to add it. I do not have access to an older Mac (Apple makes it\nhard to test back-compat with older OS's), so I cannot reproduce the\nerror. But I don't doubt that there was an error at one point -- I just\ndon't know what it was. So that's my context for saying that I don't\nthink it is needed now, but I was willing to carry it forward in case\nit is still helpful for people with older Macs. FWIW, it really seems\npretty isolated and trivial and would only affect code in this single\nsource file -- which from a quick scan, doesn't actually reference any\nof the functions in , so there shouldn't be any need to think\nabout git-compat-util.h or ctype.h, right?\n\nJeff\n"},{"id":"451106","messageId":"220311.86fsnpi73l.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"ae622a517cf500d5b068871434e33f98e9cf586b.1646160212.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 03/30] fsmonitor: config settings are repository-specific","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-11T01:47:09Z","receivedAt":"2022-03-11T01:57:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n\nI know this is in \"next\", just looking over this code again...\n\n> +static void lookup_fsmonitor_settings(struct repository *r)\n\nHere we'll start loading the settings...\n\n> +{\n> +\tstruct fsmonitor_settings *s;\n> +\tconst char *const_str;\n> +\tint bool_value;\n> +\n> +\tif (r->settings.fsmonitor)\n> +\t\treturn;\n\nMARK\n\n> +\tCALLOC_ARRAY(s, 1);\n> +\n> +\tr->settings.fsmonitor = s;\n\nAnd right after we alloc the r->settings.fsmonitor we'll ...\n\n> +\tfsm_settings__set_disabled(r);\n\n...call this method...\n> +\n> +\t/*\n> +\t * Overload the existing \"core.fsmonitor\" config setting (which\n> +\t * has historically been either unset or a hook pathname) to\n> +\t * now allow a boolean value to enable the builtin FSMonitor\n> +\t * or to turn everything off. (This does imply that you can't\n> +\t * use a hook script named \"true\" or \"false\", but that's OK.)\n> +\t */\n> +\tswitch (repo_config_get_maybe_bool(r, \"core.fsmonitor\", &bool_value)) {\n> +\n> +\tcase 0: /* config value was set to */\n> +\t\tif (bool_value)\n> +\t\t\tfsm_settings__set_ipc(r);\n> +\t\treturn;\n> +\n> +\tcase 1: /* config value was unset */\n> +\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n> +\t\tbreak;\n> +\n> +\tcase -1: /* config value set to an arbitrary string */\n> +\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n> +\t\t\treturn; /* should not happen */\n> +\t\tbreak;\n> +\n> +\tdefault: /* should not happen */\n> +\t\treturn;\n> +\t}\n> +\n> +\tif (!const_str || !*const_str)\n> +\t\treturn;\n> +\n> +\tfsm_settings__set_hook(r, const_str);\n> +}\n> [...]\n> +void fsm_settings__set_disabled(struct repository *r)\n> +{\n> +\tif (!r)\n> +\t\tr = the_repository;\n> +\n> +\tlookup_fsmonitor_settings(r);\n\n...which here will recurse into lookup_fsmonitor_settings and hit\n\"MARK\".\n\nSo isn't that fsm_settings__set_disabled() within that method pointless?\n\n> +\tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n> +\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n\nIt seems as though the intent was to reach this, but these all happen to\nbe the same thing you'd get with CALLOC_ARRAY(), so I think this just\nhappened to work out...\n\n> +enum fsmonitor_mode {\n> +\tFSMONITOR_MODE_DISABLED = 0,\n\n...I.e. this is luckily zero.\n"},{"id":"451131","messageId":"dd9a1847-7163-49e6-4a8b-bc5c8c381f0d@jeffhostetler.com","threadId":"56518","inReplyTo":"220311.86fsnpi73l.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 03/30] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-11T14:07:47Z","receivedAt":"2022-03-11T14:08:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/10/22 8:47 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n> \n> I know this is in \"next\", just looking over this code again...\n> \n>> +static void lookup_fsmonitor_settings(struct repository *r)\n> \n> Here we'll start loading the settings...\n> \n>> +{\n>> +\tstruct fsmonitor_settings *s;\n>> +\tconst char *const_str;\n>> +\tint bool_value;\n>> +\n>> +\tif (r->settings.fsmonitor)\n>> +\t\treturn;\n> \n> MARK\n> \n>> +\tCALLOC_ARRAY(s, 1);\n>> +\n>> +\tr->settings.fsmonitor = s;\n> \n> And right after we alloc the r->settings.fsmonitor we'll ...\n> \n>> +\tfsm_settings__set_disabled(r);\n> \n> ...call this method...\n>> +\n>> +\t/*\n>> +\t * Overload the existing \"core.fsmonitor\" config setting (which\n>> +\t * has historically been either unset or a hook pathname) to\n>> +\t * now allow a boolean value to enable the builtin FSMonitor\n>> +\t * or to turn everything off. (This does imply that you can't\n>> +\t * use a hook script named \"true\" or \"false\", but that's OK.)\n>> +\t */\n>> +\tswitch (repo_config_get_maybe_bool(r, \"core.fsmonitor\", &bool_value)) {\n>> +\n>> +\tcase 0: /* config value was set to */\n>> +\t\tif (bool_value)\n>> +\t\t\tfsm_settings__set_ipc(r);\n>> +\t\treturn;\n>> +\n>> +\tcase 1: /* config value was unset */\n>> +\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n>> +\t\tbreak;\n>> +\n>> +\tcase -1: /* config value set to an arbitrary string */\n>> +\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n>> +\t\t\treturn; /* should not happen */\n>> +\t\tbreak;\n>> +\n>> +\tdefault: /* should not happen */\n>> +\t\treturn;\n>> +\t}\n>> +\n>> +\tif (!const_str || !*const_str)\n>> +\t\treturn;\n>> +\n>> +\tfsm_settings__set_hook(r, const_str);\n>> +}\n>> [...]\n>> +void fsm_settings__set_disabled(struct repository *r)\n>> +{\n>> +\tif (!r)\n>> +\t\tr = the_repository;\n>> +\n>> +\tlookup_fsmonitor_settings(r);\n> \n> ...which here will recurse into lookup_fsmonitor_settings and hit\n> \"MARK\".\n> \n> So isn't that fsm_settings__set_disabled() within that method pointless?\n\nYeah, that got rolled up in one too many rebase/refactorings...\nI'll simplify.\n\nthanks\njeff\n\n> \n>> +\tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n>> +\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n> \n> It seems as though the intent was to reach this, but these all happen to\n> be the same thing you'd get with CALLOC_ARRAY(), so I think this just\n> happened to work out...\n> \n>> +enum fsmonitor_mode {\n>> +\tFSMONITOR_MODE_DISABLED = 0,\n> \n> ...I.e. this is luckily zero.\n> \n"},{"id":"451182","messageId":"0e28579e-4ba8-eb7e-b979-f9e93b7c1e51@jeffhostetler.com","threadId":"56518","inReplyTo":"xmqqk0d3kl73.fsf@gitster.g","subject":"Re: [PATCH v6 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-11T21:01:56Z","receivedAt":"2022-03-11T23:02:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/9/22 1:57 PM, Junio C Hamano wrote:\n> Jeff Hostetler writes:\n> \n>> On 3/7/22 5:37 AM, Ævar Arnfjörð Bjarmason wrote:\n>>> On Tue, Mar 01 2022, Jeff Hostetler via GitGitGadget wrote:\n>>>\n>>>> From: Jeff Hostetler \n>>>> [...]\n>> [...]\n>>\n>> Ævar sent feedback (thanks!) on 8 commits in the V6 version\n>> on March 7. I started responding to each as I got to them\n>> in my inbox yesterday, but I'd like to take a break from\n>> responding individually to each of them inside of Part 2.\n>>\n>> Since most of the feedback is for \"general cleanup\" and since\n>> Part 2 V6 is already in \"next\", I'd like to revisit these\n>> issues with a few \"cleanup\" commits on top of Part 3 (which\n>> is still in active review), rather than re-rolling or\n>> appending \"fixup\" commits onto Part 2.\n> \n> Sounds good. Prepending \"preliminary clean-up\" before part 3 would\n> be even cleaner, I would suspect.\n> \n> In any case, let's consider part 2 \"more or less done\" unless we see\n> a glaring mistake that requires us to revert and redo it from\n> scratch.\n> \n> Thanks.\n> \n\nThe cleanup here turned into 16 small commits. I'll send them as\na \"Part 2.5\" so that they stand alone and can either be appended\nto part 2 or treated as a new branch.\n\nGGG wouldn't let me send fixup! commits, so inside of each\ncommit message is a \"fixup! ...\" line which you can use if\nyou want to squash them into part 2. But otherwise they\nhave a normal (non-fixup) commit summary.\n\nAfter I send that I'll send a new version of part 3 that builds\nupon them.\n\nThanks\nJeff\n"},{"id":"451859","messageId":"e98373f997f5994a08423b3f42b6ddadb3a2a511.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 01/29] fsmonitor: enhance existing comments, clarify trivial response handling","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:42Z","receivedAt":"2022-03-22T18:00:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 64 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 41 insertions(+), 23 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..448d0ee33f5 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -168,29 +168,15 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf *\n \n \tif (result)\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/failed\", result);\n-\telse {\n+\telse\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/response-length\",\n \t\t\t\t query_result->len);\n \n-\t\tif (fsmonitor_is_trivial_response(query_result))\n-\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n-\t\t\t\t\t \"query/trivial-response\", 1);\n-\t}\n-\n \ttrace2_region_leave(\"fsm_hook\", \"query\", NULL);\n \n \treturn result;\n }\n \n-int fsmonitor_is_trivial_response(const struct strbuf *query_result)\n-{\n-\tstatic char trivial_response[3] = { '\\0', '/', '\\0' };\n-\n-\treturn query_result->len >= 3 &&\n-\t\t!memcmp(trivial_response,\n-\t\t\t&query_result->buf[query_result->len - 3], 3);\n-}\n-\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n@@ -238,6 +224,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tint is_trivial = 0;\n \n \tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n \t\treturn;\n@@ -283,6 +270,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t\tquery_success = 0;\n \t\t\t\t} else {\n \t\t\t\t\tbol = last_update_token.len + 1;\n+\t\t\t\t\tis_trivial = query_result.buf[bol] == '/';\n \t\t\t\t}\n \t\t\t} else if (hook_version < 0) {\n \t\t\t\thook_version = HOOK_INTERFACE_VERSION1;\n@@ -294,16 +282,38 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\t\tif (query_success)\n+\t\t\t\tis_trivial = query_result.buf[0] == '/';\n \t\t}\n \n+\t\tif (is_trivial)\n+\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n \t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n \t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n-\tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n+\tif (query_success && !is_trivial) {\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -318,11 +328,16 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We failed to get a response or received a trivial response,\n+\t\t * so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -330,7 +345,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"451860","messageId":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v6.git.1646160212.gitgitgadget@gmail.com","subject":"[PATCH v7 00/29] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:41Z","receivedAt":"2022-03-22T18:00:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V7 of Part 2 of my builtin FSmonitor series. This version squashes\nin the fixups from Part 2.5 and any other last minute suggestions. I'll send\nan updated version of Part 3 that builds upon this version.\n\nHere is a range-diff from V6 to V7 relative to 715d08a9e5 (The eighth batch,\n2022-02-25). Changes since V6 can be summarized as: (1) Adding/removing\n\"_()\" on die() and error() messages. (2) Splitting GCC-specific declarations\nfor MacOS FSEvents into a separate header file. (3) Enhancing the\n\"start_daemon()\" function in t7527 to take keyword args and reduce some code\nduplication in unit tests. (4) Fixing \"&&\" chains in the tests. (5) Other\nminor style cleanups.\n\n$ git range-diff eight..builtin-fsmonitor-part2-v6 eight..builtin-fsmonitor-part2\n 1: 10a34082fc = 1: e98373f997 fsmonitor: enhance existing comments, clarify trivial response handling\n 2: e1c946af2c ! 2: ab68b94417 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n @@ fsmonitor-ipc.c (new)\n + state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n + &connection);\n + if (state != IPC_STATE__LISTENING) {\n -+ die(\"fsmonitor--daemon is not running\");\n ++ die(_(\"fsmonitor--daemon is not running\"));\n + return -1;\n + }\n +\n @@ fsmonitor-ipc.c (new)\n + ipc_client_close_connection(connection);\n +\n + if (ret == -1) {\n -+ die(\"could not send '%s' command to fsmonitor--daemon\", c);\n ++ die(_(\"could not send '%s' command to fsmonitor--daemon\"), c);\n + return -1;\n + }\n +\n 3: ae622a517c ! 3: e04c7301f2 fsmonitor: config settings are repository-specific\n @@ fsmonitor-settings.c (new)\n + return;\n +\n + CALLOC_ARRAY(s, 1);\n ++ s->mode = FSMONITOR_MODE_DISABLED;\n +\n + r->settings.fsmonitor = s;\n +\n -+ fsm_settings__set_disabled(r);\n -+\n + /*\n + * Overload the existing \"core.fsmonitor\" config setting (which\n + * has historically been either unset or a hook pathname) to\n 4: 55974867da = 4: ea02ba25d8 fsmonitor: use IPC to query the builtin FSMonitor daemon\n 5: ce42d5bbaf = 5: 6ab7db9cb7 fsmonitor: document builtin fsmonitor\n 6: 9ce938c69b = 6: 0ce8ae3f2c fsmonitor--daemon: add a built-in fsmonitor daemon\n 7: 3bd7e6a051 = 7: 4624ce2fa4 fsmonitor--daemon: implement 'stop' and 'status' commands\n 8: 92d3e54da7 = 8: a29fe7266a compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n 9: f121922127 = 9: 2f8a42fdb9 compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n10: a3f94627cf ! 10: f07800690e fsmonitor--daemon: implement 'run' command\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n + * common error case.\n + */\n + if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n -+ die(\"fsmonitor--daemon is already running '%s'\",\n ++ die(_(\"fsmonitor--daemon is already running '%s'\"),\n + the_repository->worktree);\n +\n + if (fsmonitor__announce_startup) {\n11: 8de40b0fe8 ! 11: a6a39a3306 fsmonitor--daemon: implement 'start' command\n @@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)\n + * immediately exited).\n + */\n + if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n -+ die(\"fsmonitor--daemon is already running '%s'\",\n ++ die(_(\"fsmonitor--daemon is already running '%s'\"),\n + the_repository->worktree);\n +\n + if (fsmonitor__announce_startup) {\n @@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)\n + default:\n + case SBGR_ERROR:\n + case SBGR_CB_ERROR:\n -+ return error(\"daemon failed to start\");\n ++ return error(_(\"daemon failed to start\"));\n +\n + case SBGR_TIMEOUT:\n -+ return error(\"daemon not online yet\");\n ++ return error(_(\"daemon not online yet\"));\n +\n + case SBGR_DIED:\n -+ return error(\"daemon terminated\");\n ++ return error(_(\"daemon terminated\"));\n + }\n +}\n +\n12: e5d419b1ea = 12: d62e338d00 fsmonitor--daemon: add pathname classification\n13: c2f3668a86 = 13: 53e06b4ae5 fsmonitor--daemon: define token-ids\n14: bd492d13fb = 14: 39f43fabe0 fsmonitor--daemon: create token-based changed path cache\n15: 6b5b3d0822 ! 15: 239558e34f compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows\n @@ compat/fsmonitor/fsm-listen-win32.c\n + if (len > 0)\n + goto normalize;\n + if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n -+ error(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n ++ error(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n + GetLastError(),\n + (int)(info->FileNameLength / sizeof(WCHAR)),\n + info->FileName);\n @@ compat/fsmonitor/fsm-listen-win32.c\n + if (watch->is_active)\n + return 0;\n +\n -+ error(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\",\n ++ error(_(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\"),\n + watch->path.buf, GetLastError());\n + return -1;\n +}\n @@ compat/fsmonitor/fsm-listen-win32.c\n + * sure it is worth it.\n + */\n +\n -+ error(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n ++ error(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\n + watch->path.buf, gle);\n + return -1;\n +}\n16: cdef9730b3 ! 16: 14b775e9d8 compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n Signed-off-by: Junio C Hamano \n \n - ## compat/fsmonitor/fsm-listen-darwin.c ##\n + ## compat/fsmonitor/fsm-darwin-gcc.h (new) ##\n @@\n ++#ifndef FSM_DARWIN_GCC_H\n ++#define FSM_DARWIN_GCC_H\n ++\n +#ifndef __clang__\n +/*\n + * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n @@ compat/fsmonitor/fsm-listen-darwin.c\n +void FSEventStreamStop(FSEventStreamRef stream);\n +void FSEventStreamInvalidate(FSEventStreamRef stream);\n +void FSEventStreamRelease(FSEventStreamRef stream);\n ++\n ++#endif /* !clang */\n ++#endif /* FSM_DARWIN_GCC_H */\n +\n + ## compat/fsmonitor/fsm-listen-darwin.c ##\n +@@\n ++#ifndef __clang__\n ++#include \"fsm-darwin-gcc.h\"\n +#else\n -+/*\n -+ * Let Apple's headers declare `isalnum()` first, before\n -+ * Git's headers override it via a constant\n -+ */\n -+#include \n +#include \n +#include \n ++\n ++#ifndef AVAILABLE_MAC_OS_X_VERSION_10_13_AND_LATER\n ++/*\n ++ * This enum value was added in 10.13 to:\n ++ *\n ++ * /Applications/Xcode.app/Contents/Developer/Platforms/ \\\n ++ * MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/ \\\n ++ * Library/Frameworks/CoreServices.framework/Frameworks/ \\\n ++ * FSEvents.framework/Versions/Current/Headers/FSEvents.h\n ++ *\n ++ * If we're compiling against an older SDK, this symbol won't be\n ++ * present. Silently define it here so that we don't have to ifdef\n ++ * the logging or masking below. This should be harmless since older\n ++ * versions of macOS won't ever emit this FS event anyway.\n ++ */\n ++#define kFSEventStreamEventFlagItemCloned 0x00400000\n ++#endif\n +#endif\n +\n #include \"cache.h\"\n17: aabfebd3a4 ! 17: 55bd7aee06 compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS\n @@ Commit message\n Signed-off-by: Junio C Hamano \n \n ## compat/fsmonitor/fsm-listen-darwin.c ##\n -@@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef stream);\n +@@\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n @@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef\n + return 0;\n +\n +failed:\n -+ error(\"Unable to create FSEventStream.\");\n ++ error(_(\"Unable to create FSEventStream.\"));\n +\n + FREE_AND_NULL(state->backend_data);\n return -1;\n @@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef\n + data->stream_scheduled = 1;\n +\n + if (!FSEventStreamStart(data->stream)) {\n -+ error(\"Failed to start the FSEventStream\");\n ++ error(_(\"Failed to start the FSEventStream\"));\n + goto force_error_stop_without_loop;\n + }\n + data->stream_started = 1;\n18: 0896e72e2b = 18: 1f4b5209bf fsmonitor--daemon: implement handle_client callback\n19: bc240a9e66 ! 19: 8cf62c9fc6 help: include fsmonitor--daemon feature flag in version info\n @@ t/test-lib.sh: test_lazy_prereq SHA1 '\n +# Does this platform support `git fsmonitor--daemon`\n +#\n +test_lazy_prereq FSMONITOR_DAEMON '\n -+ git version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n ++ git version --build-options >output &&\n ++ grep \"feature: fsmonitor--daemon\" output\n +'\n20: 2b563c240e ! 20: 1bd74a8159 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n @@ t/helper/test-fsmonitor-client.c (new)\n +\n + ret = fsmonitor_ipc__send_query(token, &answer);\n + if (ret < 0)\n -+ die(_(\"could not query fsmonitor--daemon\"));\n ++ die(\"could not query fsmonitor--daemon\");\n +\n + write_in_full(1, answer.buf, answer.len);\n + strbuf_release(&answer);\n @@ t/helper/test-fsmonitor-client.c (new)\n + const char *token = NULL;\n +\n + const char * const fsmonitor_client_usage[] = {\n -+ N_(\"test-helper fsmonitor-client query []\"),\n -+ N_(\"test-helper fsmonitor-client flush\"),\n ++ \"test-tool fsmonitor-client query []\",\n ++ \"test-tool fsmonitor-client flush\",\n + NULL,\n + };\n +\n + struct option options[] = {\n -+ OPT_STRING(0, \"token\", &token, N_(\"token\"),\n -+ N_(\"command token to send to the server\")),\n ++ OPT_STRING(0, \"token\", &token, \"token\",\n ++ \"command token to send to the server\"),\n + OPT_END()\n + };\n +\n -+ if (argc < 2)\n -+ usage_with_options(fsmonitor_client_usage, options);\n ++ argc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n +\n -+ if (argc == 2 && !strcmp(argv[1], \"-h\"))\n ++ if (argc != 1)\n + usage_with_options(fsmonitor_client_usage, options);\n +\n -+ subcmd = argv[1];\n -+ argv--;\n -+ argc++;\n -+\n -+ argc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n ++ subcmd = argv[0];\n +\n + setup_git_directory();\n +\n21: db8809da08 ! 21: 4a920d0b54 t7527: create test for fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +fi\n +\n +stop_daemon_delete_repo () {\n -+ r=$1\n -+ git -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n ++ r=$1 &&\n ++ test_might_fail git -C $r fsmonitor--daemon stop &&\n + rm -rf $1\n -+ return 0\n +}\n +\n +start_daemon () {\n -+ case \"$#\" in\n -+ 1) r=\"-C $1\";;\n -+ *) r=\"\";\n -+ esac\n ++ r= &&\n ++ tf= &&\n ++ t2= &&\n ++ tk= &&\n ++\n ++ while test \"$#\" -ne 0\n ++ do\n ++ case \"$1\" in\n ++ -C)\n ++ shift;\n ++ test \"$#\" -ne 0 || BUG \"error: -C requires arg\"\n ++ r=\"-C $1\"\n ++ shift\n ++ ;;\n ++ -tf)\n ++ shift;\n ++ test \"$#\" -ne 0 || BUG \"error: -tf requires arg\"\n ++ tf=\"$1\"\n ++ shift\n ++ ;;\n ++ -t2)\n ++ shift;\n ++ test \"$#\" -ne 0 || BUG \"error: -t2 requires arg\"\n ++ t2=\"$1\"\n ++ shift\n ++ ;;\n ++ -tk)\n ++ shift;\n ++ test \"$#\" -ne 0 || BUG \"error: -tk requires arg\"\n ++ tk=\"$1\"\n ++ shift\n ++ ;;\n ++ *)\n ++ BUG \"error: unknown option: '$1'\"\n ++ ;;\n ++ esac\n ++ done &&\n +\n -+ git $r fsmonitor--daemon start || return $?\n -+ git $r fsmonitor--daemon status || return $?\n -+\n -+ return 0\n ++ (\n ++ if test -n \"$tf\"\n ++ then\n ++ GIT_TRACE_FSMONITOR=\"$tf\"\n ++ export GIT_TRACE_FSMONITOR\n ++ fi &&\n ++\n ++ if test -n \"$t2\"\n ++ then\n ++ GIT_TRACE2_PERF=\"$t2\"\n ++ export GIT_TRACE2_PERF\n ++ fi &&\n ++\n ++ if test -n \"$tk\"\n ++ then\n ++ GIT_TEST_FSMONITOR_TOKEN=\"$tk\"\n ++ export GIT_TEST_FSMONITOR_TOKEN\n ++ fi &&\n ++\n ++ git $r fsmonitor--daemon start &&\n ++ git $r fsmonitor--daemon status\n ++ )\n +}\n +\n +# Is a Trace2 data event present with the given catetory and key?\n +# We do not care what the value is.\n +#\n +have_t2_data_event () {\n -+ c=$1\n -+ k=$2\n ++ c=$1 &&\n ++ k=$2 &&\n +\n + grep -e '\"event\":\"data\".*\"category\":\"'\"$c\"'\".*\"key\":\"'\"$k\"'\"'\n +}\n @@ t/t7527-builtin-fsmonitor.sh (new)\n + test_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n +\n + git init test_explicit &&\n -+ start_daemon test_explicit &&\n ++ start_daemon -C test_explicit &&\n +\n + git -C test_explicit fsmonitor--daemon stop &&\n + test_must_fail git -C test_explicit fsmonitor--daemon status\n @@ t/t7527-builtin-fsmonitor.sh (new)\n + # but this test case is only concerned with whether the daemon was\n + # implicitly started.)\n +\n -+ GIT_TRACE2_EVENT=\"$(pwd)/.git/trace\" \\\n ++ GIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n + test-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n + nul_to_q actual.filtered &&\n + grep \"builtin:\" actual.filtered &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n + git init test_implicit_1 &&\n +\n -+ start_daemon test_implicit_1 &&\n ++ start_daemon -C test_implicit_1 &&\n +\n + # deleting the .git directory will implicitly stop the daemon.\n + rm -rf test_implicit_1/.git &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n + git init test_implicit_2 &&\n +\n -+ start_daemon test_implicit_2 &&\n ++ start_daemon -C test_implicit_2 &&\n +\n + # renaming the .git directory will implicitly stop the daemon.\n + mv test_implicit_2/.git test_implicit_2/.xxx &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n + git init test_multiple &&\n +\n -+ start_daemon test_multiple &&\n ++ start_daemon -C test_multiple &&\n +\n + test_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n + grep \"fsmonitor--daemon is already running\" actual &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +# This is here in case something else fails first.\n +#\n +redundant_stop_daemon () {\n -+ git fsmonitor--daemon stop\n -+ return 0\n ++ test_might_fail git fsmonitor--daemon stop\n +}\n +\n +test_expect_success 'update-index implicitly starts daemon' '\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n + test_must_fail git fsmonitor--daemon status &&\n +\n -+ GIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_1\" \\\n ++ GIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n + git update-index --fsmonitor &&\n +\n + git fsmonitor--daemon status &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n + test_must_fail git fsmonitor--daemon status &&\n +\n -+ GIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_2\" \\\n ++ GIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n + git status >actual &&\n +\n + git fsmonitor--daemon status &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +edit_files () {\n -+ echo 1 >modified\n -+ echo 2 >dir1/modified\n -+ echo 3 >dir2/modified\n ++ echo 1 >modified &&\n ++ echo 2 >dir1/modified &&\n ++ echo 3 >dir2/modified &&\n + >dir1/untracked\n +}\n +\n +delete_files () {\n -+ rm -f delete\n -+ rm -f dir1/delete\n ++ rm -f delete &&\n ++ rm -f dir1/delete &&\n + rm -f dir2/delete\n +}\n +\n +create_files () {\n -+ echo 1 >new\n -+ echo 2 >dir1/new\n ++ echo 1 >new &&\n ++ echo 2 >dir1/new &&\n + echo 3 >dir2/new\n +}\n +\n +rename_files () {\n -+ mv rename renamed\n -+ mv dir1/rename dir1/renamed\n ++ mv rename renamed &&\n ++ mv dir1/rename dir1/renamed &&\n + mv dir2/rename dir2/renamed\n +}\n +\n +file_to_directory () {\n -+ rm -f delete\n -+ mkdir delete\n ++ rm -f delete &&\n ++ mkdir delete &&\n + echo 1 >delete/new\n +}\n +\n +directory_to_file () {\n -+ rm -rf dir1\n ++ rm -rf dir1 &&\n + echo 1 >dir1\n +}\n +\n -+verify_status () {\n -+ git status >actual &&\n -+ GIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n -+ GIT_INDEX_FILE=.git/fresh-index git -c core.fsmonitor=false status >expect &&\n -+ test_cmp expect actual &&\n -+ echo HELLO AFTER &&\n -+ cat .git/trace &&\n -+ echo HELLO AFTER\n -+}\n -+\n +# The next few test cases confirm that our fsmonitor daemon sees each type\n +# of OS filesystem notification that we care about. At this layer we just\n +# ensure we are getting the OS notifications and do not try to confirm what\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +# daemon) because these commands might implicitly restart the daemon.\n +\n +clean_up_repo_and_stop_daemon () {\n -+ git reset --hard HEAD\n -+ git clean -fd\n -+ git fsmonitor--daemon stop\n ++ git reset --hard HEAD &&\n ++ git clean -fd &&\n ++ test_might_fail git fsmonitor--daemon stop &&\n + rm -f .git/trace\n +}\n +\n +test_expect_success 'edit some files' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ (\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n -+\n -+ start_daemon\n -+ ) &&\n ++ start_daemon -tf \"$PWD/.git/trace\" &&\n +\n + edit_files &&\n +\n -+ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++ test-tool fsmonitor-client query --token 0 &&\n +\n + grep \"^event: dir1/modified$\" .git/trace &&\n + grep \"^event: dir2/modified$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'create some files' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ (\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n -+\n -+ start_daemon\n -+ ) &&\n ++ start_daemon -tf \"$PWD/.git/trace\" &&\n +\n + create_files &&\n +\n -+ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++ test-tool fsmonitor-client query --token 0 &&\n +\n + grep \"^event: dir1/new$\" .git/trace &&\n + grep \"^event: dir2/new$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'delete some files' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ (\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n -+\n -+ start_daemon\n -+ ) &&\n ++ start_daemon -tf \"$PWD/.git/trace\" &&\n +\n + delete_files &&\n +\n -+ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++ test-tool fsmonitor-client query --token 0 &&\n +\n + grep \"^event: dir1/delete$\" .git/trace &&\n + grep \"^event: dir2/delete$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'rename some files' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ (\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n -+\n -+ start_daemon\n -+ ) &&\n ++ start_daemon -tf \"$PWD/.git/trace\" &&\n +\n + rename_files &&\n +\n -+ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++ test-tool fsmonitor-client query --token 0 &&\n +\n + grep \"^event: dir1/rename$\" .git/trace &&\n + grep \"^event: dir2/rename$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'rename directory' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ (\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n -+\n -+ start_daemon\n -+ ) &&\n ++ start_daemon -tf \"$PWD/.git/trace\" &&\n +\n + mv dirtorename dirrenamed &&\n +\n -+ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++ test-tool fsmonitor-client query --token 0 &&\n +\n + grep \"^event: dirtorename/*$\" .git/trace &&\n + grep \"^event: dirrenamed/*$\" .git/trace\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'file changes to directory' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ (\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n -+\n -+ start_daemon\n -+ ) &&\n ++ start_daemon -tf \"$PWD/.git/trace\" &&\n +\n + file_to_directory &&\n +\n -+ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++ test-tool fsmonitor-client query --token 0 &&\n +\n + grep \"^event: delete$\" .git/trace &&\n + grep \"^event: delete/new$\" .git/trace\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'directory changes to a file' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ (\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n -+\n -+ start_daemon\n -+ ) &&\n ++ start_daemon -tf \"$PWD/.git/trace\" &&\n +\n + directory_to_file &&\n +\n -+ test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++ test-tool fsmonitor-client query --token 0 &&\n +\n + grep \"^event: dir1$\" .git/trace\n +'\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n + git init test_flush &&\n +\n -+ (\n -+ GIT_TEST_FSMONITOR_TOKEN=true &&\n -+ export GIT_TEST_FSMONITOR_TOKEN &&\n -+\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace_daemon\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n -+\n -+ start_daemon test_flush\n -+ ) &&\n ++ start_daemon -C test_flush -tf \"$PWD/.git/trace_daemon\" -tk true &&\n +\n + # The daemon should have an initial token with no events in _0 and\n + # then a few (probably platform-specific number of) events in _1.\n @@ t/t7527-builtin-fsmonitor.sh (new)\n + test-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n + nul_to_q actual_q0 &&\n +\n -+ touch test_flush/file_1 &&\n -+ touch test_flush/file_2 &&\n ++ >test_flush/file_1 &&\n ++ >test_flush/file_2 &&\n +\n + test-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n + nul_to_q actual_q1 &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n + grep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n +\n -+ touch test_flush/file_3 &&\n ++ >test_flush/file_3 &&\n +\n + test-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n + nul_to_q actual_q3 &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'worktree with .git file' '\n + git -C wt-base worktree add ../wt-secondary &&\n +\n -+ (\n -+ GIT_TRACE2_PERF=\"$(pwd)/trace2_wt_secondary\" &&\n -+ export GIT_TRACE2_PERF &&\n -+\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/trace_wt_secondary\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n -+\n -+ start_daemon wt-secondary\n -+ ) &&\n ++ start_daemon -C wt-secondary \\\n ++ -tf \"$PWD/trace_wt_secondary\" \\\n ++ -t2 \"$PWD/trace2_wt_secondary\" &&\n +\n + git -C wt-secondary fsmonitor--daemon stop &&\n + test_must_fail git -C wt-secondary fsmonitor--daemon status\n22: 15bd5aaff3 = 22: c925a9a745 t/perf: avoid copying builtin fsmonitor files into test repo\n23: 4cb97918d2 = 23: 5b3381c223 t/helper/test-chmtime: skip directories on Windows\n24: a70748b464 ! 24: 803a540cc0 t/perf/p7519: speed up test on Windows\n @@ t/perf/p7519-fsmonitor.sh: test_fsmonitor_suite() {\n head -100000 | \\\n grep -v \\\" | \\\n - sed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n -+ egrep -v \" .\" | \\\n ++ grep -v \" .\" | \\\n xargs test-tool chmtime -300 &&\n git status\n '\n25: b0c9c9c7a4 ! 25: d5ca2df31c t/perf/p7519: add fsmonitor--daemon test cases\n @@ Commit message\n Signed-off-by: Junio C Hamano \n \n ## t/perf/p7519-fsmonitor.sh ##\n +@@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n + fi\n + '\n + \n +-setup_for_fsmonitor () {\n ++setup_for_fsmonitor_hook () {\n + # set INTEGRATION_SCRIPT depending on the environment\n + if test -n \"$INTEGRATION_PATH\"\n + then\n @@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n }\n \n @@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n + if test -n \"$USE_FSMONITOR_DAEMON\"\n + then\n + DESC=\"builtin fsmonitor--daemon\"\n -+ elif test -n \"$INTEGRATION_SCRIPT\"; then\n ++ elif test -n \"$INTEGRATION_SCRIPT\"\n ++ then\n DESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n else\n DESC=\"fsmonitor=disabled\"\n +@@ t/perf/p7519-fsmonitor.sh: test_fsmonitor_suite () {\n + trace_start fsmonitor-watchman\n + if test -n \"$GIT_PERF_7519_FSMONITOR\"; then\n + for INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do\n +- test_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor'\n ++ test_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor_hook'\n + test_fsmonitor_suite\n + done\n + else\n +- test_expect_success \"setup for fsmonitor\" 'setup_for_fsmonitor'\n ++ test_expect_success \"setup for fsmonitor hook\" 'setup_for_fsmonitor_hook'\n + test_fsmonitor_suite\n + fi\n + \n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"setup without fsmonitor\" '\n +then\n + USE_FSMONITOR_DAEMON=t\n +\n -+ trace_start fsmonitor--daemon--server\n -+ git fsmonitor--daemon start\n ++ test_expect_success \"setup for builtin fsmonitor\" '\n ++ trace_start fsmonitor--daemon--server &&\n ++ git fsmonitor--daemon start &&\n +\n -+ trace_start fsmonitor--daemon--client\n ++ trace_start fsmonitor--daemon--client &&\n +\n -+ git config core.fsmonitor true\n -+ git update-index --fsmonitor\n ++ git config core.fsmonitor true &&\n ++ git update-index --fsmonitor\n ++ '\n +\n + test_fsmonitor_suite\n +\n26: 64a5b74167 = 26: 42631259e8 fsmonitor--daemon: periodically truncate list of modified files\n27: 5b06eb5d0e ! 27: f256c3cbe8 fsmonitor--daemon: use a cookie file to sync with file system\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n \n +enum fsmonitor_cookie_item_result {\n + FCIR_ERROR = -1, /* could not create cookie file ? */\n -+ FCIR_INIT = 0,\n ++ FCIR_INIT,\n + FCIR_SEEN,\n + FCIR_ABORT,\n +};\n +\n +struct fsmonitor_cookie_item {\n + struct hashmap_entry entry;\n -+ const char *name;\n ++ char *name;\n + enum fsmonitor_cookie_item_result result;\n +};\n +\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n + * that the listener thread has seen it.\n + */\n + fd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n -+ if (fd >= 0) {\n -+ close(fd);\n -+ unlink(cookie_pathname.buf);\n -+\n -+ /*\n -+ * Technically, this is an infinite wait (well, unless another\n -+ * thread sends us an abort). I'd like to change this to\n -+ * use `pthread_cond_timedwait()` and return an error/timeout\n -+ * and let the caller do the trivial response thing, but we\n -+ * don't have that routine in our thread-utils.\n -+ *\n -+ * After extensive beta testing I'm not really worried about\n -+ * this. Also note that the above open() and unlink() calls\n -+ * will cause at least two FS events on that path, so the odds\n -+ * of getting stuck are pretty slim.\n -+ */\n -+ while (cookie->result == FCIR_INIT)\n -+ pthread_cond_wait(&state->cookies_cond,\n -+ &state->main_lock);\n -+ } else {\n ++ if (fd < 0) {\n + error_errno(_(\"could not create fsmonitor cookie '%s'\"),\n + cookie->name);\n +\n + cookie->result = FCIR_ERROR;\n ++ goto done;\n + }\n +\n ++ /*\n ++ * Technically, close() and unlink() can fail, but we don't\n ++ * care here. We only created the file to trigger a watch\n ++ * event from the FS to know that when we're up to date.\n ++ */\n ++ close(fd);\n ++ unlink(cookie_pathname.buf);\n ++\n ++ /*\n ++ * Technically, this is an infinite wait (well, unless another\n ++ * thread sends us an abort). I'd like to change this to\n ++ * use `pthread_cond_timedwait()` and return an error/timeout\n ++ * and let the caller do the trivial response thing, but we\n ++ * don't have that routine in our thread-utils.\n ++ *\n ++ * After extensive beta testing I'm not really worried about\n ++ * this. Also note that the above open() and unlink() calls\n ++ * will cause at least two FS events on that path, so the odds\n ++ * of getting stuck are pretty slim.\n ++ */\n ++ while (cookie->result == FCIR_INIT)\n ++ pthread_cond_wait(&state->cookies_cond,\n ++ &state->main_lock);\n ++\n ++done:\n + hashmap_remove(&state->cookies, &cookie->entry, NULL);\n +\n + result = cookie->result;\n +\n -+ free((char*)cookie->name);\n ++ free(cookie->name);\n + free(cookie);\n + strbuf_release(&cookie_pathname);\n +\n28: 1fd5439de0 = 28: 08af8296f9 fsmonitor: force update index after large responses\n29: 6fc1430285 ! 29: e6cf84dc8e t7527: test status with untracked-cache and fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n +}\n +\n +matrix_try () {\n -+ uc=$1\n -+ fsm=$2\n -+ fn=$3\n ++ uc=$1 &&\n ++ fsm=$2 &&\n ++ fn=$3 &&\n +\n + test_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n + matrix_clean_up_repo &&\n30: b915b95cc2 < -: ---------- update-index: convert fsmonitor warnings to advise\n\n\nJeff Hostetler (29):\n fsmonitor: enhance existing comments, clarify trivial response\n handling\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor: document builtin fsmonitor\n fsmonitor--daemon: add a built-in fsmonitor daemon\n fsmonitor--daemon: implement 'stop' and 'status' commands\n compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n fsmonitor--daemon: implement 'run' command\n fsmonitor--daemon: implement 'start' command\n fsmonitor--daemon: add pathname classification\n fsmonitor--daemon: define token-ids\n fsmonitor--daemon: create token-based changed path cache\n compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on\n Windows\n compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on\n MacOS\n fsmonitor--daemon: implement handle_client callback\n help: include fsmonitor--daemon feature flag in version info\n t/helper/fsmonitor-client: create IPC client to talk to FSMonitor\n Daemon\n t7527: create test for fsmonitor--daemon\n t/perf: avoid copying builtin fsmonitor files into test repo\n t/helper/test-chmtime: skip directories on Windows\n t/perf/p7519: speed up test on Windows\n t/perf/p7519: add fsmonitor--daemon test cases\n fsmonitor--daemon: periodically truncate list of modified files\n fsmonitor--daemon: use a cookie file to sync with file system\n fsmonitor: force update index after large responses\n t7527: test status with untracked-cache and fsmonitor--daemon\n\n .gitignore | 1 +\n Documentation/config/core.txt | 60 +-\n Documentation/git-fsmonitor--daemon.txt | 75 ++\n Documentation/git-update-index.txt | 8 +-\n Makefile | 17 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 1480 +++++++++++++++++++++++\n builtin/update-index.c | 7 +-\n cache.h | 1 -\n compat/fsmonitor/fsm-darwin-gcc.h | 92 ++\n compat/fsmonitor/fsm-listen-darwin.c | 427 +++++++\n compat/fsmonitor/fsm-listen-win32.c | 586 +++++++++\n compat/fsmonitor/fsm-listen.h | 49 +\n config.c | 14 -\n config.h | 1 -\n config.mak.uname | 20 +\n contrib/buildsystems/CMakeLists.txt | 10 +\n environment.c | 1 -\n fsmonitor--daemon.h | 166 +++\n fsmonitor-ipc.c | 171 +++\n fsmonitor-ipc.h | 48 +\n fsmonitor-settings.c | 114 ++\n fsmonitor-settings.h | 21 +\n fsmonitor.c | 216 +++-\n fsmonitor.h | 15 +-\n git.c | 1 +\n help.c | 4 +\n repo-settings.c | 1 +\n repository.h | 3 +\n t/README | 4 +-\n t/helper/test-chmtime.c | 15 +\n t/helper/test-fsmonitor-client.c | 116 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/perf/p7519-fsmonitor.sh | 68 +-\n t/perf/perf-lib.sh | 2 +-\n t/t7527-builtin-fsmonitor.sh | 594 +++++++++\n t/test-lib.sh | 7 +\n 38 files changed, 4312 insertions(+), 106 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n create mode 100644 builtin/fsmonitor--daemon.c\n create mode 100644 compat/fsmonitor/fsm-darwin-gcc.h\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n create mode 100644 fsmonitor--daemon.h\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n create mode 100644 t/helper/test-fsmonitor-client.c\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\n\nbase-commit: 715d08a9e51251ad8290b181b6ac3b9e1f9719d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v7\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v7\nPull-Request: https://github.com/gitgitgadget/git/pull/1041\n\nRange-diff vs v6:\n\n 1: 10a34082fcc = 1: e98373f997f fsmonitor: enhance existing comments, clarify trivial response handling\n 2: e1c946af2c5 ! 2: ab68b944173 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n @@ fsmonitor-ipc.c (new)\n +\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n +\t\t\t\t &connection);\n +\tif (state != IPC_STATE__LISTENING) {\n -+\t\tdie(\"fsmonitor--daemon is not running\");\n ++\t\tdie(_(\"fsmonitor--daemon is not running\"));\n +\t\treturn -1;\n +\t}\n +\n @@ fsmonitor-ipc.c (new)\n +\tipc_client_close_connection(connection);\n +\n +\tif (ret == -1) {\n -+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\", c);\n ++\t\tdie(_(\"could not send '%s' command to fsmonitor--daemon\"), c);\n +\t\treturn -1;\n +\t}\n +\n 3: ae622a517cf ! 3: e04c7301f24 fsmonitor: config settings are repository-specific\n @@ fsmonitor-settings.c (new)\n +\t\treturn;\n +\n +\tCALLOC_ARRAY(s, 1);\n ++\ts->mode = FSMONITOR_MODE_DISABLED;\n +\n +\tr->settings.fsmonitor = s;\n +\n -+\tfsm_settings__set_disabled(r);\n -+\n +\t/*\n +\t * Overload the existing \"core.fsmonitor\" config setting (which\n +\t * has historically been either unset or a hook pathname) to\n 4: 55974867da5 = 4: ea02ba25d8f fsmonitor: use IPC to query the builtin FSMonitor daemon\n 5: ce42d5bbaf6 = 5: 6ab7db9cb76 fsmonitor: document builtin fsmonitor\n 6: 9ce938c69b5 = 6: 0ce8ae3f2cf fsmonitor--daemon: add a built-in fsmonitor daemon\n 7: 3bd7e6a051e = 7: 4624ce2fa47 fsmonitor--daemon: implement 'stop' and 'status' commands\n 8: 92d3e54da7f = 8: a29fe7266a4 compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n 9: f1219221270 = 9: 2f8a42fdb93 compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n 10: a3f94627cf6 ! 10: f07800690ee fsmonitor--daemon: implement 'run' command\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n +\t * common error case.\n +\t */\n +\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n -+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n ++\t\tdie(_(\"fsmonitor--daemon is already running '%s'\"),\n +\t\t the_repository->worktree);\n +\n +\tif (fsmonitor__announce_startup) {\n 11: 8de40b0fe8b ! 11: a6a39a3306d fsmonitor--daemon: implement 'start' command\n @@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)\n +\t * immediately exited).\n +\t */\n +\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n -+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n ++\t\tdie(_(\"fsmonitor--daemon is already running '%s'\"),\n +\t\t the_repository->worktree);\n +\n +\tif (fsmonitor__announce_startup) {\n @@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)\n +\tdefault:\n +\tcase SBGR_ERROR:\n +\tcase SBGR_CB_ERROR:\n -+\t\treturn error(\"daemon failed to start\");\n ++\t\treturn error(_(\"daemon failed to start\"));\n +\n +\tcase SBGR_TIMEOUT:\n -+\t\treturn error(\"daemon not online yet\");\n ++\t\treturn error(_(\"daemon not online yet\"));\n +\n +\tcase SBGR_DIED:\n -+\t\treturn error(\"daemon terminated\");\n ++\t\treturn error(_(\"daemon terminated\"));\n +\t}\n +}\n +\n 12: e5d419b1ea0 = 12: d62e338d008 fsmonitor--daemon: add pathname classification\n 13: c2f3668a866 = 13: 53e06b4ae5d fsmonitor--daemon: define token-ids\n 14: bd492d13fb1 = 14: 39f43fabe02 fsmonitor--daemon: create token-based changed path cache\n 15: 6b5b3d08227 ! 15: 239558e34ff compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows\n @@ compat/fsmonitor/fsm-listen-win32.c\n +\t\tif (len > 0)\n +\t\t\tgoto normalize;\n +\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n -+\t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n ++\t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n +\t\t\t GetLastError(),\n +\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n +\t\t\t info->FileName);\n @@ compat/fsmonitor/fsm-listen-win32.c\n +\tif (watch->is_active)\n +\t\treturn 0;\n +\n -+\terror(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\",\n ++\terror(_(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\"),\n +\t watch->path.buf, GetLastError());\n +\treturn -1;\n +}\n @@ compat/fsmonitor/fsm-listen-win32.c\n +\t * sure it is worth it.\n +\t */\n +\n -+\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n ++\terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\n +\t watch->path.buf, gle);\n +\treturn -1;\n +}\n 16: cdef9730b3f ! 16: 14b775e9d8b compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n Signed-off-by: Junio C Hamano \n \n - ## compat/fsmonitor/fsm-listen-darwin.c ##\n + ## compat/fsmonitor/fsm-darwin-gcc.h (new) ##\n @@\n ++#ifndef FSM_DARWIN_GCC_H\n ++#define FSM_DARWIN_GCC_H\n ++\n +#ifndef __clang__\n +/*\n + * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n @@ compat/fsmonitor/fsm-listen-darwin.c\n +void FSEventStreamStop(FSEventStreamRef stream);\n +void FSEventStreamInvalidate(FSEventStreamRef stream);\n +void FSEventStreamRelease(FSEventStreamRef stream);\n ++\n ++#endif /* !clang */\n ++#endif /* FSM_DARWIN_GCC_H */\n +\n + ## compat/fsmonitor/fsm-listen-darwin.c ##\n +@@\n ++#ifndef __clang__\n ++#include \"fsm-darwin-gcc.h\"\n +#else\n -+/*\n -+ * Let Apple's headers declare `isalnum()` first, before\n -+ * Git's headers override it via a constant\n -+ */\n -+#include \n +#include \n +#include \n ++\n ++#ifndef AVAILABLE_MAC_OS_X_VERSION_10_13_AND_LATER\n ++/*\n ++ * This enum value was added in 10.13 to:\n ++ *\n ++ * /Applications/Xcode.app/Contents/Developer/Platforms/ \\\n ++ * MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/ \\\n ++ * Library/Frameworks/CoreServices.framework/Frameworks/ \\\n ++ * FSEvents.framework/Versions/Current/Headers/FSEvents.h\n ++ *\n ++ * If we're compiling against an older SDK, this symbol won't be\n ++ * present. Silently define it here so that we don't have to ifdef\n ++ * the logging or masking below. This should be harmless since older\n ++ * versions of macOS won't ever emit this FS event anyway.\n ++ */\n ++#define kFSEventStreamEventFlagItemCloned 0x00400000\n ++#endif\n +#endif\n +\n #include \"cache.h\"\n 17: aabfebd3a41 ! 17: 55bd7aee06c compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS\n @@ Commit message\n Signed-off-by: Junio C Hamano \n \n ## compat/fsmonitor/fsm-listen-darwin.c ##\n -@@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef stream);\n +@@\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n @@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef\n +\treturn 0;\n +\n +failed:\n -+\terror(\"Unable to create FSEventStream.\");\n ++\terror(_(\"Unable to create FSEventStream.\"));\n +\n +\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n @@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef\n +\tdata->stream_scheduled = 1;\n +\n +\tif (!FSEventStreamStart(data->stream)) {\n -+\t\terror(\"Failed to start the FSEventStream\");\n ++\t\terror(_(\"Failed to start the FSEventStream\"));\n +\t\tgoto force_error_stop_without_loop;\n +\t}\n +\tdata->stream_started = 1;\n 18: 0896e72e2b8 = 18: 1f4b5209bf6 fsmonitor--daemon: implement handle_client callback\n 19: bc240a9e665 ! 19: 8cf62c9fc6f help: include fsmonitor--daemon feature flag in version info\n @@ t/test-lib.sh: test_lazy_prereq SHA1 '\n +# Does this platform support `git fsmonitor--daemon`\n +#\n +test_lazy_prereq FSMONITOR_DAEMON '\n -+\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n ++\tgit version --build-options >output &&\n ++\tgrep \"feature: fsmonitor--daemon\" output\n +'\n 20: 2b563c240eb ! 20: 1bd74a81593 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n @@ t/helper/test-fsmonitor-client.c (new)\n +\n +\tret = fsmonitor_ipc__send_query(token, &answer);\n +\tif (ret < 0)\n -+\t\tdie(_(\"could not query fsmonitor--daemon\"));\n ++\t\tdie(\"could not query fsmonitor--daemon\");\n +\n +\twrite_in_full(1, answer.buf, answer.len);\n +\tstrbuf_release(&answer);\n @@ t/helper/test-fsmonitor-client.c (new)\n +\tconst char *token = NULL;\n +\n +\tconst char * const fsmonitor_client_usage[] = {\n -+\t\tN_(\"test-helper fsmonitor-client query []\"),\n -+\t\tN_(\"test-helper fsmonitor-client flush\"),\n ++\t\t\"test-tool fsmonitor-client query []\",\n ++\t\t\"test-tool fsmonitor-client flush\",\n +\t\tNULL,\n +\t};\n +\n +\tstruct option options[] = {\n -+\t\tOPT_STRING(0, \"token\", &token, N_(\"token\"),\n -+\t\t\t N_(\"command token to send to the server\")),\n ++\t\tOPT_STRING(0, \"token\", &token, \"token\",\n ++\t\t\t \"command token to send to the server\"),\n +\t\tOPT_END()\n +\t};\n +\n -+\tif (argc < 2)\n -+\t\tusage_with_options(fsmonitor_client_usage, options);\n ++\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n +\n -+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n ++\tif (argc != 1)\n +\t\tusage_with_options(fsmonitor_client_usage, options);\n +\n -+\tsubcmd = argv[1];\n -+\targv--;\n -+\targc++;\n -+\n -+\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n ++\tsubcmd = argv[0];\n +\n +\tsetup_git_directory();\n +\n 21: db8809da089 ! 21: 4a920d0b54a t7527: create test for fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +fi\n +\n +stop_daemon_delete_repo () {\n -+\tr=$1\n -+\tgit -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n ++\tr=$1 &&\n ++\ttest_might_fail git -C $r fsmonitor--daemon stop &&\n +\trm -rf $1\n -+\treturn 0\n +}\n +\n +start_daemon () {\n -+\tcase \"$#\" in\n -+\t\t1) r=\"-C $1\";;\n -+\t\t*) r=\"\";\n -+\tesac\n ++\tr= &&\n ++\ttf= &&\n ++\tt2= &&\n ++\ttk= &&\n ++\n ++\twhile test \"$#\" -ne 0\n ++\tdo\n ++\t\tcase \"$1\" in\n ++\t\t-C)\n ++\t\t\tshift;\n ++\t\t\ttest \"$#\" -ne 0 || BUG \"error: -C requires arg\"\n ++\t\t\tr=\"-C $1\"\n ++\t\t\tshift\n ++\t\t\t;;\n ++\t\t-tf)\n ++\t\t\tshift;\n ++\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tf requires arg\"\n ++\t\t\ttf=\"$1\"\n ++\t\t\tshift\n ++\t\t\t;;\n ++\t\t-t2)\n ++\t\t\tshift;\n ++\t\t\ttest \"$#\" -ne 0 || BUG \"error: -t2 requires arg\"\n ++\t\t\tt2=\"$1\"\n ++\t\t\tshift\n ++\t\t\t;;\n ++\t\t-tk)\n ++\t\t\tshift;\n ++\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tk requires arg\"\n ++\t\t\ttk=\"$1\"\n ++\t\t\tshift\n ++\t\t\t;;\n ++\t\t*)\n ++\t\t\tBUG \"error: unknown option: '$1'\"\n ++\t\t\t;;\n ++\t\tesac\n ++\tdone &&\n +\n -+\tgit $r fsmonitor--daemon start || return $?\n -+\tgit $r fsmonitor--daemon status || return $?\n -+\n -+\treturn 0\n ++\t(\n ++\t\tif test -n \"$tf\"\n ++\t\tthen\n ++\t\t\tGIT_TRACE_FSMONITOR=\"$tf\"\n ++\t\t\texport GIT_TRACE_FSMONITOR\n ++\t\tfi &&\n ++\n ++\t\tif test -n \"$t2\"\n ++\t\tthen\n ++\t\t\tGIT_TRACE2_PERF=\"$t2\"\n ++\t\t\texport GIT_TRACE2_PERF\n ++\t\tfi &&\n ++\n ++\t\tif test -n \"$tk\"\n ++\t\tthen\n ++\t\t\tGIT_TEST_FSMONITOR_TOKEN=\"$tk\"\n ++\t\t\texport GIT_TEST_FSMONITOR_TOKEN\n ++\t\tfi &&\n ++\n ++\t\tgit $r fsmonitor--daemon start &&\n ++\t\tgit $r fsmonitor--daemon status\n ++\t)\n +}\n +\n +# Is a Trace2 data event present with the given catetory and key?\n +# We do not care what the value is.\n +#\n +have_t2_data_event () {\n -+\tc=$1\n -+\tk=$2\n ++\tc=$1 &&\n ++\tk=$2 &&\n +\n +\tgrep -e '\"event\":\"data\".*\"category\":\"'\"$c\"'\".*\"key\":\"'\"$k\"'\"'\n +}\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n +\n +\tgit init test_explicit &&\n -+\tstart_daemon test_explicit &&\n ++\tstart_daemon -C test_explicit &&\n +\n +\tgit -C test_explicit fsmonitor--daemon stop &&\n +\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t# but this test case is only concerned with whether the daemon was\n +\t# implicitly started.)\n +\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace\" \\\n ++\tGIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n +\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n +\tnul_to_q actual.filtered &&\n +\tgrep \"builtin:\" actual.filtered &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tgit init test_implicit_1 &&\n +\n -+\tstart_daemon test_implicit_1 &&\n ++\tstart_daemon -C test_implicit_1 &&\n +\n +\t# deleting the .git directory will implicitly stop the daemon.\n +\trm -rf test_implicit_1/.git &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tgit init test_implicit_2 &&\n +\n -+\tstart_daemon test_implicit_2 &&\n ++\tstart_daemon -C test_implicit_2 &&\n +\n +\t# renaming the .git directory will implicitly stop the daemon.\n +\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tgit init test_multiple &&\n +\n -+\tstart_daemon test_multiple &&\n ++\tstart_daemon -C test_multiple &&\n +\n +\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n +\tgrep \"fsmonitor--daemon is already running\" actual &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +# This is here in case something else fails first.\n +#\n +redundant_stop_daemon () {\n -+\tgit fsmonitor--daemon stop\n -+\treturn 0\n ++\ttest_might_fail git fsmonitor--daemon stop\n +}\n +\n +test_expect_success 'update-index implicitly starts daemon' '\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\ttest_must_fail git fsmonitor--daemon status &&\n +\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_1\" \\\n ++\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n +\t\tgit update-index --fsmonitor &&\n +\n +\tgit fsmonitor--daemon status &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\ttest_must_fail git fsmonitor--daemon status &&\n +\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_2\" \\\n ++\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n +\t\tgit status >actual &&\n +\n +\tgit fsmonitor--daemon status &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +edit_files () {\n -+\techo 1 >modified\n -+\techo 2 >dir1/modified\n -+\techo 3 >dir2/modified\n ++\techo 1 >modified &&\n ++\techo 2 >dir1/modified &&\n ++\techo 3 >dir2/modified &&\n +\t>dir1/untracked\n +}\n +\n +delete_files () {\n -+\trm -f delete\n -+\trm -f dir1/delete\n ++\trm -f delete &&\n ++\trm -f dir1/delete &&\n +\trm -f dir2/delete\n +}\n +\n +create_files () {\n -+\techo 1 >new\n -+\techo 2 >dir1/new\n ++\techo 1 >new &&\n ++\techo 2 >dir1/new &&\n +\techo 3 >dir2/new\n +}\n +\n +rename_files () {\n -+\tmv rename renamed\n -+\tmv dir1/rename dir1/renamed\n ++\tmv rename renamed &&\n ++\tmv dir1/rename dir1/renamed &&\n +\tmv dir2/rename dir2/renamed\n +}\n +\n +file_to_directory () {\n -+\trm -f delete\n -+\tmkdir delete\n ++\trm -f delete &&\n ++\tmkdir delete &&\n +\techo 1 >delete/new\n +}\n +\n +directory_to_file () {\n -+\trm -rf dir1\n ++\trm -rf dir1 &&\n +\techo 1 >dir1\n +}\n +\n -+verify_status () {\n -+\tgit status >actual &&\n -+\tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n -+\tGIT_INDEX_FILE=.git/fresh-index git -c core.fsmonitor=false status >expect &&\n -+\ttest_cmp expect actual &&\n -+\techo HELLO AFTER &&\n -+\tcat .git/trace &&\n -+\techo HELLO AFTER\n -+}\n -+\n +# The next few test cases confirm that our fsmonitor daemon sees each type\n +# of OS filesystem notification that we care about. At this layer we just\n +# ensure we are getting the OS notifications and do not try to confirm what\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +# daemon) because these commands might implicitly restart the daemon.\n +\n +clean_up_repo_and_stop_daemon () {\n -+\tgit reset --hard HEAD\n -+\tgit clean -fd\n -+\tgit fsmonitor--daemon stop\n ++\tgit reset --hard HEAD &&\n ++\tgit clean -fd &&\n ++\ttest_might_fail git fsmonitor--daemon stop &&\n +\trm -f .git/trace\n +}\n +\n +test_expect_success 'edit some files' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n -+\n -+\t\tstart_daemon\n -+\t) &&\n ++\tstart_daemon -tf \"$PWD/.git/trace\" &&\n +\n +\tedit_files &&\n +\n -+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 &&\n +\n +\tgrep \"^event: dir1/modified$\" .git/trace &&\n +\tgrep \"^event: dir2/modified$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'create some files' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n -+\n -+\t\tstart_daemon\n -+\t) &&\n ++\tstart_daemon -tf \"$PWD/.git/trace\" &&\n +\n +\tcreate_files &&\n +\n -+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 &&\n +\n +\tgrep \"^event: dir1/new$\" .git/trace &&\n +\tgrep \"^event: dir2/new$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'delete some files' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n -+\n -+\t\tstart_daemon\n -+\t) &&\n ++\tstart_daemon -tf \"$PWD/.git/trace\" &&\n +\n +\tdelete_files &&\n +\n -+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 &&\n +\n +\tgrep \"^event: dir1/delete$\" .git/trace &&\n +\tgrep \"^event: dir2/delete$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'rename some files' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n -+\n -+\t\tstart_daemon\n -+\t) &&\n ++\tstart_daemon -tf \"$PWD/.git/trace\" &&\n +\n +\trename_files &&\n +\n -+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 &&\n +\n +\tgrep \"^event: dir1/rename$\" .git/trace &&\n +\tgrep \"^event: dir2/rename$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'rename directory' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n -+\n -+\t\tstart_daemon\n -+\t) &&\n ++\tstart_daemon -tf \"$PWD/.git/trace\" &&\n +\n +\tmv dirtorename dirrenamed &&\n +\n -+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 &&\n +\n +\tgrep \"^event: dirtorename/*$\" .git/trace &&\n +\tgrep \"^event: dirrenamed/*$\" .git/trace\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'file changes to directory' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n -+\n -+\t\tstart_daemon\n -+\t) &&\n ++\tstart_daemon -tf \"$PWD/.git/trace\" &&\n +\n +\tfile_to_directory &&\n +\n -+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 &&\n +\n +\tgrep \"^event: delete$\" .git/trace &&\n +\tgrep \"^event: delete/new$\" .git/trace\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'directory changes to a file' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n -+\n -+\t\tstart_daemon\n -+\t) &&\n ++\tstart_daemon -tf \"$PWD/.git/trace\" &&\n +\n +\tdirectory_to_file &&\n +\n -+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 &&\n +\n +\tgrep \"^event: dir1$\" .git/trace\n +'\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tgit init test_flush &&\n +\n -+\t(\n -+\t\tGIT_TEST_FSMONITOR_TOKEN=true &&\n -+\t\texport GIT_TEST_FSMONITOR_TOKEN &&\n -+\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace_daemon\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n -+\n -+\t\tstart_daemon test_flush\n -+\t) &&\n ++\tstart_daemon -C test_flush -tf \"$PWD/.git/trace_daemon\" -tk true &&\n +\n +\t# The daemon should have an initial token with no events in _0 and\n +\t# then a few (probably platform-specific number of) events in _1.\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n +\tnul_to_q actual_q0 &&\n +\n -+\ttouch test_flush/file_1 &&\n -+\ttouch test_flush/file_2 &&\n ++\t>test_flush/file_1 &&\n ++\t>test_flush/file_2 &&\n +\n +\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n +\tnul_to_q actual_q1 &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n +\n -+\ttouch test_flush/file_3 &&\n ++\t>test_flush/file_3 &&\n +\n +\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n +\tnul_to_q actual_q3 &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'worktree with .git file' '\n +\tgit -C wt-base worktree add ../wt-secondary &&\n +\n -+\t(\n -+\t\tGIT_TRACE2_PERF=\"$(pwd)/trace2_wt_secondary\" &&\n -+\t\texport GIT_TRACE2_PERF &&\n -+\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/trace_wt_secondary\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n -+\n -+\t\tstart_daemon wt-secondary\n -+\t) &&\n ++\tstart_daemon -C wt-secondary \\\n ++\t\t-tf \"$PWD/trace_wt_secondary\" \\\n ++\t\t-t2 \"$PWD/trace2_wt_secondary\" &&\n +\n +\tgit -C wt-secondary fsmonitor--daemon stop &&\n +\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n 22: 15bd5aaff36 = 22: c925a9a7459 t/perf: avoid copying builtin fsmonitor files into test repo\n 23: 4cb97918d20 = 23: 5b3381c223e t/helper/test-chmtime: skip directories on Windows\n 24: a70748b4640 ! 24: 803a540cc00 t/perf/p7519: speed up test on Windows\n @@ t/perf/p7519-fsmonitor.sh: test_fsmonitor_suite() {\n \t\t\thead -100000 | \\\n \t\t\tgrep -v \\\" | \\\n -\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n -+\t\t\tegrep -v \" .\" | \\\n ++\t\t\tgrep -v \" .\" | \\\n \t\t\txargs test-tool chmtime -300 &&\n \t\tgit status\n \t'\n 25: b0c9c9c7a40 ! 25: d5ca2df31c8 t/perf/p7519: add fsmonitor--daemon test cases\n @@ Commit message\n Signed-off-by: Junio C Hamano \n \n ## t/perf/p7519-fsmonitor.sh ##\n +@@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n + \tfi\n + '\n + \n +-setup_for_fsmonitor () {\n ++setup_for_fsmonitor_hook () {\n + \t# set INTEGRATION_SCRIPT depending on the environment\n + \tif test -n \"$INTEGRATION_PATH\"\n + \tthen\n @@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n }\n \n @@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n +\tif test -n \"$USE_FSMONITOR_DAEMON\"\n +\tthen\n +\t\tDESC=\"builtin fsmonitor--daemon\"\n -+\telif test -n \"$INTEGRATION_SCRIPT\"; then\n ++\telif test -n \"$INTEGRATION_SCRIPT\"\n ++\tthen\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n \t\tDESC=\"fsmonitor=disabled\"\n +@@ t/perf/p7519-fsmonitor.sh: test_fsmonitor_suite () {\n + trace_start fsmonitor-watchman\n + if test -n \"$GIT_PERF_7519_FSMONITOR\"; then\n + \tfor INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do\n +-\t\ttest_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor'\n ++\t\ttest_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor_hook'\n + \t\ttest_fsmonitor_suite\n + \tdone\n + else\n +-\ttest_expect_success \"setup for fsmonitor\" 'setup_for_fsmonitor'\n ++\ttest_expect_success \"setup for fsmonitor hook\" 'setup_for_fsmonitor_hook'\n + \ttest_fsmonitor_suite\n + fi\n + \n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"setup without fsmonitor\" '\n +then\n +\tUSE_FSMONITOR_DAEMON=t\n +\n -+\ttrace_start fsmonitor--daemon--server\n -+\tgit fsmonitor--daemon start\n ++\ttest_expect_success \"setup for builtin fsmonitor\" '\n ++\t\ttrace_start fsmonitor--daemon--server &&\n ++\t\tgit fsmonitor--daemon start &&\n +\n -+\ttrace_start fsmonitor--daemon--client\n ++\t\ttrace_start fsmonitor--daemon--client &&\n +\n -+\tgit config core.fsmonitor true\n -+\tgit update-index --fsmonitor\n ++\t\tgit config core.fsmonitor true &&\n ++\t\tgit update-index --fsmonitor\n ++\t'\n +\n +\ttest_fsmonitor_suite\n +\n 26: 64a5b741670 = 26: 42631259e89 fsmonitor--daemon: periodically truncate list of modified files\n 27: 5b06eb5d0e6 ! 27: f256c3cbe8b fsmonitor--daemon: use a cookie file to sync with file system\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n \n +enum fsmonitor_cookie_item_result {\n +\tFCIR_ERROR = -1, /* could not create cookie file ? */\n -+\tFCIR_INIT = 0,\n ++\tFCIR_INIT,\n +\tFCIR_SEEN,\n +\tFCIR_ABORT,\n +};\n +\n +struct fsmonitor_cookie_item {\n +\tstruct hashmap_entry entry;\n -+\tconst char *name;\n ++\tchar *name;\n +\tenum fsmonitor_cookie_item_result result;\n +};\n +\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n +\t * that the listener thread has seen it.\n +\t */\n +\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n -+\tif (fd >= 0) {\n -+\t\tclose(fd);\n -+\t\tunlink(cookie_pathname.buf);\n -+\n -+\t\t/*\n -+\t\t * Technically, this is an infinite wait (well, unless another\n -+\t\t * thread sends us an abort). I'd like to change this to\n -+\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n -+\t\t * and let the caller do the trivial response thing, but we\n -+\t\t * don't have that routine in our thread-utils.\n -+\t\t *\n -+\t\t * After extensive beta testing I'm not really worried about\n -+\t\t * this. Also note that the above open() and unlink() calls\n -+\t\t * will cause at least two FS events on that path, so the odds\n -+\t\t * of getting stuck are pretty slim.\n -+\t\t */\n -+\t\twhile (cookie->result == FCIR_INIT)\n -+\t\t\tpthread_cond_wait(&state->cookies_cond,\n -+\t\t\t\t\t &state->main_lock);\n -+\t} else {\n ++\tif (fd < 0) {\n +\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n +\t\t\t cookie->name);\n +\n +\t\tcookie->result = FCIR_ERROR;\n ++\t\tgoto done;\n +\t}\n +\n ++\t/*\n ++\t * Technically, close() and unlink() can fail, but we don't\n ++\t * care here. We only created the file to trigger a watch\n ++\t * event from the FS to know that when we're up to date.\n ++\t */\n ++\tclose(fd);\n ++\tunlink(cookie_pathname.buf);\n ++\n ++\t/*\n ++\t * Technically, this is an infinite wait (well, unless another\n ++\t * thread sends us an abort). I'd like to change this to\n ++\t * use `pthread_cond_timedwait()` and return an error/timeout\n ++\t * and let the caller do the trivial response thing, but we\n ++\t * don't have that routine in our thread-utils.\n ++\t *\n ++\t * After extensive beta testing I'm not really worried about\n ++\t * this. Also note that the above open() and unlink() calls\n ++\t * will cause at least two FS events on that path, so the odds\n ++\t * of getting stuck are pretty slim.\n ++\t */\n ++\twhile (cookie->result == FCIR_INIT)\n ++\t\tpthread_cond_wait(&state->cookies_cond,\n ++\t\t\t\t &state->main_lock);\n ++\n ++done:\n +\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n +\n +\tresult = cookie->result;\n +\n -+\tfree((char*)cookie->name);\n ++\tfree(cookie->name);\n +\tfree(cookie);\n +\tstrbuf_release(&cookie_pathname);\n +\n 28: 1fd5439de03 = 28: 08af8296f96 fsmonitor: force update index after large responses\n 29: 6fc1430285f ! 29: e6cf84dc8eb t7527: test status with untracked-cache and fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n +}\n +\n +matrix_try () {\n -+\tuc=$1\n -+\tfsm=$2\n -+\tfn=$3\n ++\tuc=$1 &&\n ++\tfsm=$2 &&\n ++\tfn=$3 &&\n +\n +\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n +\t\tmatrix_clean_up_repo &&\n 30: b915b95cc2f < -: ----------- update-index: convert fsmonitor warnings to advise\n\n-- \ngitgitgadget\n"},{"id":"451861","messageId":"ab68b944173cb8d4fdc0d7583a3d1dd11e5aef78.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 02/29] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:43Z","receivedAt":"2022-03-22T18:00:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 ++++++++++++++\n 3 files changed, 220 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 6f0b4b775fe..a19d850e716 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -907,6 +907,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..789e7397baa\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,171 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#else\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(_(\"fsmonitor--daemon is not running\"));\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(_(\"could not send '%s' command to fsmonitor--daemon\"), c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..b6a7067c3af\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"451862","messageId":"e04c7301f249a75a33e5498258a0dc628f3ebe66.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 03/29] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:44Z","receivedAt":"2022-03-22T18:00:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove fsmonitor config settings to a new and opaque\n`struct fsmonitor_settings` structure. Add a lazily-loaded pointer\nto this into `struct repo_settings`\n\nCreate an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\nrepresent the state of fsmonitor. This lets us represent which, if\nany, fsmonitor provider (hook or IPC) is enabled.\n\nCreate `fsm_settings__get_*()` getters to lazily look up fsmonitor-\nrelated config settings.\n\nGet rid of the `core_fsmonitor` global variable. Move the code to\nlookup the existing `core.fsmonitor` config value into the fsmonitor\nsettings.\n\nCreate a hook pathname variable in `struct fsmonitor-settings` and\nonly set it when in hook mode.\n\nExtend the definition of `core.fsmonitor` to be either a boolean\nor a hook pathname. When true, the builtin FSMonitor is used.\nWhen false or unset, no FSMonitor (neither builtin nor hook) is\nused.\n\nThe existing `core_fsmonitor` global variable was used to store the\npathname to the fsmonitor hook *and* it was used as a boolean to see\nif fsmonitor was enabled. This dual usage and global visibility leads\nto confusion when we add the IPC-based provider. So lets hide the\ndetails in fsmonitor-settings.c and let it decide which provider to\nuse in the case of multiple settings. This avoids cluttering up\nrepo-settings.c with these private details.\n\nA future commit in builtin-fsmonitor series will add the ability to\ndisqualify worktrees for various reasons, such as being mounted from a\nremote volume, where fsmonitor should not be started. Having the\nconfig settings hidden in fsmonitor-settings.c allows such worktree\nrestrictions to override the config values used.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n builtin/update-index.c | 7 ++-\n cache.h | 1 -\n config.c | 14 -----\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-settings.c | 114 +++++++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 21 ++++++++\n fsmonitor.c | 63 ++++++++++++++---------\n fsmonitor.h | 15 ++++--\n repository.h | 3 ++\n t/README | 4 +-\n 12 files changed, 196 insertions(+), 49 deletions(-)\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\ndiff --git a/Makefile b/Makefile\nindex a19d850e716..707a56d4c11 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -908,6 +908,7 @@ LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += fsmonitor-ipc.o\n+LIB_OBJS += fsmonitor-settings.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex aafe7eeac2a..876112abb21 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1236,14 +1236,17 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\n \t\t\t\t\"enable fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode > FSMONITOR_MODE_DISABLED)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex 04d4d2db25c..aaf334e2aa4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -999,7 +999,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex 383b1a4885b..3f9b0739a78 100644\n--- a/config.c\n+++ b/config.c\n@@ -2626,20 +2626,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex bb49baf1ee0..7654f61c634 100644\n--- a/config.h\n+++ b/config.h\n@@ -597,7 +597,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a5..00682e638d7 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nnew file mode 100644\nindex 00000000000..757d230d538\n--- /dev/null\n+++ b/fsmonitor-settings.c\n@@ -0,0 +1,114 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+/*\n+ * We keep this structure defintion private and have getters\n+ * for all fields so that we can lazy load it as needed.\n+ */\n+struct fsmonitor_settings {\n+\tenum fsmonitor_mode mode;\n+\tchar *hook_path;\n+};\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s;\n+\tconst char *const_str;\n+\tint bool_value;\n+\n+\tif (r->settings.fsmonitor)\n+\t\treturn;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\n+\tr->settings.fsmonitor = s;\n+\n+\t/*\n+\t * Overload the existing \"core.fsmonitor\" config setting (which\n+\t * has historically been either unset or a hook pathname) to\n+\t * now allow a boolean value to enable the builtin FSMonitor\n+\t * or to turn everything off. (This does imply that you can't\n+\t * use a hook script named \"true\" or \"false\", but that's OK.)\n+\t */\n+\tswitch (repo_config_get_maybe_bool(r, \"core.fsmonitor\", &bool_value)) {\n+\n+\tcase 0: /* config value was set to */\n+\t\tif (bool_value)\n+\t\t\tfsm_settings__set_ipc(r);\n+\t\treturn;\n+\n+\tcase 1: /* config value was unset */\n+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n+\t\tbreak;\n+\n+\tcase -1: /* config value set to an arbitrary string */\n+\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n+\t\t\treturn; /* should not happen */\n+\t\tbreak;\n+\n+\tdefault: /* should not happen */\n+\t\treturn;\n+\t}\n+\n+\tif (!const_str || !*const_str)\n+\t\treturn;\n+\n+\tfsm_settings__set_hook(r, const_str);\n+}\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->mode;\n+}\n+\n+const char *fsm_settings__get_hook_path(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->hook_path;\n+}\n+\n+void fsm_settings__set_ipc(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\n+\n+void fsm_settings__set_hook(struct repository *r, const char *path)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+\tr->settings.fsmonitor->hook_path = strdup(path);\n+}\n+\n+void fsm_settings__set_disabled(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nnew file mode 100644\nindex 00000000000..a4c5d7b4889\n--- /dev/null\n+++ b/fsmonitor-settings.h\n@@ -0,0 +1,21 @@\n+#ifndef FSMONITOR_SETTINGS_H\n+#define FSMONITOR_SETTINGS_H\n+\n+struct repository;\n+\n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n+\tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r);\n+void fsm_settings__set_hook(struct repository *r, const char *path);\n+void fsm_settings__set_disabled(struct repository *r);\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+const char *fsm_settings__get_hook_path(struct repository *r);\n+\n+struct fsmonitor_settings;\n+\n+#endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 448d0ee33f5..0e961b74d82 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tstrvec_push(&cp.args, fsm_settings__get_hook_path(r));\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -225,17 +229,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \tchar *buf;\n \tunsigned int i;\n \tint is_trivial = 0;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (fsm_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -243,13 +258,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -280,7 +296,8 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t\tif (query_success)\n \t\t\t\tis_trivial = query_result.buf[0] == '/';\n@@ -290,9 +307,12 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n \t\t\t\t\t \"query/trivial-response\", 1);\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tfsm_settings__get_hook_path(r));\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t fsm_settings__get_hook_path(r),\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/*\n@@ -429,7 +449,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tint fsmonitor_enabled = (fsm_settings__get_mode(istate->repo)\n+\t\t\t\t > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -449,16 +470,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..3f41f653691 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -3,6 +3,7 @@\n \n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"fsmonitor-settings.h\"\n \n extern struct trace_key trace_fsmonitor;\n \n@@ -57,7 +58,10 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +71,10 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +90,9 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repository.h b/repository.h\nindex ca837cb9e91..9bbb4659cc8 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n #include \"path.h\"\n \n struct config_set;\n+struct fsmonitor_settings;\n struct git_hash_algo;\n struct index_state;\n struct lock_file;\n@@ -35,6 +36,8 @@ struct repo_settings {\n \tint command_requires_full_index;\n \tint sparse_index;\n \n+\tstruct fsmonitor_settings *fsmonitor; /* lazily loaded */\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex f48e0542cdc..9ffea1d3147 100644\n--- a/t/README\n+++ b/t/README\n@@ -405,8 +405,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code paths for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\n-- \ngitgitgadget\n\n"},{"id":"451863","messageId":"ea02ba25d8f0e5e76a646d23306eb6a2b312ebe2.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 04/29] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:45Z","receivedAt":"2022-03-22T18:00:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.fsmonitor` is set to true.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 38 ++++++++++++++++++++++++++++++++++++--\n 1 file changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 0e961b74d82..a38b5710eb3 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -241,8 +241,41 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (fsm_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t\tis_trivial = query_result.buf[bol] == '/';\n+\t\t\tif (is_trivial)\n+\t\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t\t \"query/trivial-response\", 1);\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\tgoto apply_results;\n \t}\n \n \tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n@@ -315,6 +348,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/*\n \t * The response from FSMonitor (excluding the header token) is\n \t * either:\n-- \ngitgitgadget\n\n"},{"id":"451864","messageId":"6ab7db9cb769fe44db2c76e82da0b5d898538ec0.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 05/29] fsmonitor: document builtin fsmonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:46Z","receivedAt":"2022-03-22T18:00:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDocument how `core.fsmonitor` can be set to a boolean to enable\nor disable the builtin FSMonitor.\n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to refer to it.\n\nCreate `git-fsmonitor--daemon` manual page and describe its features.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Documentation/config/core.txt | 60 +++++++++++++++-----\n Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n Documentation/git-update-index.txt | 8 ++-\n 3 files changed, 126 insertions(+), 17 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..6303c36c7ed 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,54 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+\tOtherwise, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note that if you concurrently use multiple versions of Git, such\n+as one version on the command line and another version in an IDE\n+tool, that the definition of `core.fsmonitor` was extended to\n+allow boolean values in addition to hook pathnames. Git versions\n+2.35.1 and prior will not understand the boolean values and will\n+consider the \"true\" or \"false\" values as hook pathnames to be\n+invoked. Git versions 2.26 thru 2.35.1 default to hook protocol\n+V2 and will fall back to no fsmonitor (full scan). Git versions\n+prior to 2.26 default to hook protocol V1 and will silently\n+assume there were no changes to report (no scan), so status\n+commands may report incomplete results. For this reason, it is\n+best to upgrade all of your Git versions before using the built-in\n+file system monitor.\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\nnew file mode 100644\nindex 00000000000..0fedf5a4565\n--- /dev/null\n+++ b/Documentation/git-fsmonitor--daemon.txt\n@@ -0,0 +1,75 @@\n+git-fsmonitor--daemon(1)\n+========================\n+\n+NAME\n+----\n+git-fsmonitor--daemon - A Built-in File System Monitor\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git fsmonitor--daemon' start\n+'git fsmonitor--daemon' run\n+'git fsmonitor--daemon' stop\n+'git fsmonitor--daemon' status\n+\n+DESCRIPTION\n+-----------\n+\n+A daemon to watch the working directory for file and directory\n+changes using platform-specific file system notification facilities.\n+\n+This daemon communicates directly with commands like `git status`\n+using the link:technical/api-simple-ipc.html[simple IPC] interface\n+instead of the slower linkgit:githooks[5] interface.\n+\n+This daemon is built into Git so that no third-party tools are\n+required.\n+\n+OPTIONS\n+-------\n+\n+start::\n+\tStarts a daemon in the background.\n+\n+run::\n+\tRuns a daemon in the foreground.\n+\n+stop::\n+\tStops the daemon running in the current working\n+\tdirectory, if present.\n+\n+status::\n+\tExits with zero status if a daemon is watching the\n+\tcurrent working directory.\n+\n+REMARKS\n+-------\n+\n+This daemon is a long running process used to watch a single working\n+directory and maintain a list of the recently changed files and\n+directories. Performance of commands such as `git status` can be\n+increased if they just ask for a summary of changes to the working\n+directory and can avoid scanning the disk.\n+\n+When `core.fsmonitor` is set to `true` (see linkgit:git-config[1])\n+commands, such as `git status`, will ask the daemon for changes and\n+automatically start it (if necessary).\n+\n+For more information see the \"File System Monitor\" section in\n+linkgit:git-update-index[1].\n+\n+CAVEATS\n+-------\n+\n+The fsmonitor daemon does not currently know about submodules and does\n+not know to filter out file system events that happen within a\n+submodule. If fsmonitor daemon is watching a super repo and a file is\n+modified within the working directory of a submodule, it will report\n+the change (as happening against the super repo). However, the client\n+will properly ignore these extra events, so performance may be affected\n+but it will not cause an incorrect result.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..53ea48a04e2 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -509,8 +511,8 @@ looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n+linkgit:git-config[1]) than using the `--fsmonitor` option to `git\n+update-index` in each repository, especially if you want to do so\n across all repositories you use, because you can set the configuration\n variable in your `$HOME/.gitconfig` just once and have it affect all\n repositories you touch.\n-- \ngitgitgadget\n\n"},{"id":"451865","messageId":"0ce8ae3f2cf07bfffdb1cb14454768d2b6d84bf2.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 06/29] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:47Z","receivedAt":"2022-03-22T18:00:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a built-in file system monitoring daemon that can be used by\nthe existing `fsmonitor` feature (protocol API and index extension)\nto improve the performance of various Git commands, such as `status`.\n\nThe `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\nprovides an alternative to hook access to existing fsmonitors such\nas `watchman`.\n\nThis commit merely adds the new command without any functionality.\n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 46 +++++++++++++++++++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 50 insertions(+)\n create mode 100644 builtin/fsmonitor--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex f817c509ec0..e81de1063a4 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -72,6 +72,7 @@\n /git-format-patch\n /git-fsck\n /git-fsck-objects\n+/git-fsmonitor--daemon\n /git-gc\n /git-get-tar-commit-id\n /git-grep\ndiff --git a/Makefile b/Makefile\nindex 707a56d4c11..5af1d5b112e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1114,6 +1114,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n+BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex 83379f3832c..40e9ecc8485 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\n int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nnew file mode 100644\nindex 00000000000..f0498793379\n--- /dev/null\n+++ b/builtin/fsmonitor--daemon.c\n@@ -0,0 +1,46 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"simple-ipc.h\"\n+#include \"khash.h\"\n+\n+static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tNULL\n+};\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *subcmd;\n+\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\tif (argc != 1)\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\tsubcmd = argv[0];\n+\n+\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n+}\n+\n+#else\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n+}\n+#endif\ndiff --git a/git.c b/git.c\nindex a25940d72e8..3d8e48cf555 100644\n--- a/git.c\n+++ b/git.c\n@@ -537,6 +537,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"451866","messageId":"4624ce2fa471b3ebb3a08baf0de2609447eb9fe3.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 07/29] fsmonitor--daemon: implement 'stop' and 'status' commands","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:48Z","receivedAt":"2022-03-22T18:00:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `stop` and `status` client commands to control and query the\nstatus of a `fsmonitor--daemon` server process (and implicitly start a\nserver process if necessary).\n\nLater commits will implement the actual server and monitor the file\nsystem.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 51 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex f0498793379..5e3178b8bdd 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,10 +7,55 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon stop\"),\n+\tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n+ * and wait for it to shutdown.\n+ */\n+static int do_as_client__send_stop(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"quit\", &answer);\n+\n+\t/* The quit command does not return any response data. */\n+\tstrbuf_release(&answer);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\twhile (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tsleep_millisec(50);\n+\ttrace2_region_leave(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\n+\treturn 0;\n+}\n+\n+static int do_as_client__status(void)\n+{\n+\tenum ipc_active_state state = fsmonitor_ipc__get_state();\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tprintf(_(\"fsmonitor-daemon is watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tprintf(_(\"fsmonitor-daemon is not watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 1;\n+\t}\n+}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n@@ -28,6 +73,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (!strcmp(subcmd, \"stop\"))\n+\t\treturn !!do_as_client__send_stop();\n+\n+\tif (!strcmp(subcmd, \"status\"))\n+\t\treturn !!do_as_client__status();\n+\n \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"451867","messageId":"a29fe7266a456d88125ca1f87d4b16370caf486f.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 08/29] compat/fsmonitor/fsm-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:49Z","receivedAt":"2022-03-22T18:00:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty filesystem listener backend for fsmonitor--daemon on Windows.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 13 ++++++++\n compat/fsmonitor/fsm-listen-win32.c | 21 +++++++++++++\n compat/fsmonitor/fsm-listen.h | 49 +++++++++++++++++++++++++++++\n config.mak.uname | 10 ++++++\n contrib/buildsystems/CMakeLists.txt | 7 +++++\n repo-settings.c | 1 +\n 6 files changed, 101 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n\ndiff --git a/Makefile b/Makefile\nindex 5af1d5b112e..26567d4f772 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -470,6 +470,11 @@ all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n+# If your platform supports a built-in fsmonitor backend, set\n+# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n+# `compat/fsmonitor/fsm-listen-.c` that implements the\n+# `fsm_listen__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1968,6 +1973,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n \tCOMPAT_OBJS += compat/access.o\n endif\n \n+ifdef FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2887,6 +2897,9 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n \t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n+ifdef FSMONITOR_DAEMON_BACKEND\n+\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nnew file mode 100644\nindex 00000000000..916cbea254f\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nnew file mode 100644\nindex 00000000000..f0539349baf\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -0,0 +1,49 @@\n+#ifndef FSM_LISTEN_H\n+#define FSM_LISTEN_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * fsmonitor_fs_listener thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread AFTER joining the listener.\n+ */\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to watch for\n+ * filesystem events. This will run in the fsmonitor_fs_listen thread.\n+ *\n+ * It should call `ipc_server_stop_async()` if the listener thread\n+ * prematurely terminates (because of a filesystem error or if it\n+ * detects that the .git directory has been deleted). (It should NOT\n+ * do so if the listener thread receives a normal shutdown signal from\n+ * the IPC layer.)\n+ *\n+ * It should set `state->error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the fsmonitor listener thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_LISTEN_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 4352ea39e9b..26074f56bed 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -435,6 +435,11 @@ ifeq ($(uname_S),Windows)\n \t# so we don't need this:\n \t#\n \t# SNPRINTF_RETURNS_BOGUS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -619,6 +624,11 @@ ifeq ($(uname_S),MINGW)\n \tNO_STRTOUMAX = YesPlease\n \tNO_MKDTEMP = YesPlease\n \tNO_SVN_TESTS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex e44232f85d3..0963629db7f 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -285,6 +285,13 @@ else()\n \tendif()\n endif()\n \n+if(SUPPORTS_SIMPLE_IPC)\n+\tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\tendif()\n+endif()\n+\n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n \n #header checks\ndiff --git a/repo-settings.c b/repo-settings.c\nindex b4fbd16cdcc..2dfcb2b6542 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"midx.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n \n static void repo_cfg_bool(struct repository *r, const char *key, int *dest,\n \t\t\t int def)\n-- \ngitgitgadget\n\n"},{"id":"451868","messageId":"a6a39a3306df2602e06cc3d9a755fa247f8d41b0.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 11/29] fsmonitor--daemon: implement 'start' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:52Z","receivedAt":"2022-03-22T18:00:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement 'git fsmonitor--daemon start' command. This command starts\nan instance of 'git fsmonitor--daemon run' in the background using\nthe new 'start_bg_command()' function.\n\nWe avoid the fork-and-call technique on Unix systems in favor of a\nfork-and-exec technique. This gives us more uniform Trace2 child-*\nevents. It also makes our usage more consistent with Windows usage.\n\nOn Windows, teach 'git fsmonitor--daemon run' to optionally call\n'FreeConsole()' to release handles to the inherited Win32 console\n(despite being passed invalid handles for stdin/out/err). Without\nthis, command prompts and powershell terminal windows could hang\nin \"exit\" until the last background child process exited or released\ntheir Win32 console handle. (This was not seen with git-bash shells\nbecause they don't have a Win32 console attached to them.)\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 109 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 107 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 5591339399a..69dd39121a3 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -9,6 +9,7 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon start []\"),\n \tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n@@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n #define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n static int fsmonitor__ipc_threads = 8;\n \n+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n+static int fsmonitor__start_timeout_sec = 60;\n+\n #define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n static int fsmonitor__announce_startup = 0;\n \n@@ -36,6 +40,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n+\t\tfsmonitor__start_timeout_sec = i;\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n \t\tint is_bool;\n \t\tint i = git_config_bool_or_int(var, value, &is_bool);\n@@ -250,7 +263,7 @@ done:\n \treturn err;\n }\n \n-static int try_to_run_foreground_daemon(void)\n+static int try_to_run_foreground_daemon(int detach_console)\n {\n \t/*\n \t * Technically, we don't need to probe for an existing daemon\n@@ -270,17 +283,106 @@ static int try_to_run_foreground_daemon(void)\n \t\tfflush(stderr);\n \t}\n \n+#ifdef GIT_WINDOWS_NATIVE\n+\tif (detach_console)\n+\t\tFreeConsole();\n+#endif\n+\n \treturn !!fsmonitor_run_daemon();\n }\n \n+static start_bg_wait_cb bg_wait_cb;\n+\n+static int bg_wait_cb(const struct child_process *cp, void *cb_data)\n+{\n+\tenum ipc_active_state s = fsmonitor_ipc__get_state();\n+\n+\tswitch (s) {\n+\tcase IPC_STATE__LISTENING:\n+\t\t/* child is \"ready\" */\n+\t\treturn 0;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\t/* give child more time */\n+\t\treturn 1;\n+\n+\tdefault:\n+\tcase IPC_STATE__INVALID_PATH:\n+\tcase IPC_STATE__OTHER_ERROR:\n+\t\t/* all the time in world won't help */\n+\t\treturn -1;\n+\t}\n+}\n+\n+static int try_to_start_background_daemon(void)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tenum start_bg_result sbgr;\n+\n+\t/*\n+\t * Before we try to create a background daemon process, see\n+\t * if a daemon process is already listening. This makes it\n+\t * easier for us to report an already-listening error to the\n+\t * console, since our spawn/daemon can only report the success\n+\t * of creating the background process (and not whether it\n+\t * immediately exited).\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(_(\"fsmonitor--daemon is already running '%s'\"),\n+\t\t the_repository->worktree);\n+\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"starting fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n+\n+\tcp.git_cmd = 1;\n+\n+\tstrvec_push(&cp.args, \"fsmonitor--daemon\");\n+\tstrvec_push(&cp.args, \"run\");\n+\tstrvec_push(&cp.args, \"--detach\");\n+\tstrvec_pushf(&cp.args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n+\n+\tcp.no_stdin = 1;\n+\tcp.no_stdout = 1;\n+\tcp.no_stderr = 1;\n+\n+\tsbgr = start_bg_command(&cp, bg_wait_cb, NULL,\n+\t\t\t\tfsmonitor__start_timeout_sec);\n+\n+\tswitch (sbgr) {\n+\tcase SBGR_READY:\n+\t\treturn 0;\n+\n+\tdefault:\n+\tcase SBGR_ERROR:\n+\tcase SBGR_CB_ERROR:\n+\t\treturn error(_(\"daemon failed to start\"));\n+\n+\tcase SBGR_TIMEOUT:\n+\t\treturn error(_(\"daemon not online yet\"));\n+\n+\tcase SBGR_DIED:\n+\t\treturn error(_(\"daemon terminated\"));\n+\t}\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tint detach_console = 0;\n \n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"detach\", &detach_console, N_(\"detach from console\")),\n \t\tOPT_INTEGER(0, \"ipc-threads\",\n \t\t\t &fsmonitor__ipc_threads,\n \t\t\t N_(\"use ipc worker threads\")),\n+\t\tOPT_INTEGER(0, \"start-timeout\",\n+\t\t\t &fsmonitor__start_timeout_sec,\n+\t\t\t N_(\"max seconds to wait for background daemon startup\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -296,8 +398,11 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tif (!strcmp(subcmd, \"start\"))\n+\t\treturn !!try_to_start_background_daemon();\n+\n \tif (!strcmp(subcmd, \"run\"))\n-\t\treturn !!try_to_run_foreground_daemon();\n+\t\treturn !!try_to_run_foreground_daemon(detach_console);\n \n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n-- \ngitgitgadget\n\n"},{"id":"451869","messageId":"f07800690eeda000271b05435aa032ba17b35086.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 10/29] fsmonitor--daemon: implement 'run' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:51Z","receivedAt":"2022-03-22T18:00:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `run` command to try to begin listening for file system events.\n\nThis version defines the thread structure with a single fsmonitor_fs_listen\nthread to watch for file system events and a simple IPC thread pool to\nwatch for connection from Git clients over a well-known named pipe or\nUnix domain socket.\n\nThis commit does not actually do anything yet because the platform\nbackends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 34 ++++++\n 2 files changed, 261 insertions(+), 1 deletion(-)\n create mode 100644 fsmonitor--daemon.h\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 5e3178b8bdd..5591339399a 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,16 +3,52 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Global state loaded from config.\n+ */\n+#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n+static int fsmonitor__ipc_threads = 8;\n+\n+#define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n+static int fsmonitor__announce_startup = 0;\n+\n+static int fsmonitor_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 1)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__IPC_THREADS, i);\n+\t\tfsmonitor__ipc_threads = i;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n+\t\tint is_bool;\n+\t\tint i = git_config_bool_or_int(var, value, &is_bool);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' not bool or int: %d\"),\n+\t\t\t\t var, i);\n+\t\tfsmonitor__announce_startup = i;\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n /*\n * Acting as a CLIENT.\n *\n@@ -57,15 +93,198 @@ static int do_as_client__status(void)\n \t}\n }\n \n+static ipc_server_application_cb handle_client;\n+\n+static int handle_client(void *data,\n+\t\t\t const char *command, size_t command_len,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\t/* struct fsmonitor_daemon_state *state = data; */\n+\tint result;\n+\n+\t/*\n+\t * The Simple IPC API now supports {char*, len} arguments, but\n+\t * FSMonitor always uses proper null-terminated strings, so\n+\t * we can ignore the command_len argument. (Trust, but verify.)\n+\t */\n+\tif (command_len != strlen(command))\n+\t\tBUG(\"FSMonitor assumes text messages\");\n+\n+\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n+\n+\tresult = 0; /* TODO Do something here. */\n+\n+\ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n+\n+\treturn result;\n+}\n+\n+static void *fsm_listen__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-listen\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Watching: worktree '%s'\",\n+\t\t\t state->path_worktree_watch.buf);\n+\tif (state->nr_paths_watching > 1)\n+\t\ttrace_printf_key(&trace_fsmonitor, \"Watching: gitdir '%s'\",\n+\t\t\t\t state->path_gitdir_watch.buf);\n+\n+\tfsm_listen__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct ipc_server_opts ipc_opts = {\n+\t\t.nr_threads = fsmonitor__ipc_threads,\n+\n+\t\t/*\n+\t\t * We know that there are no other active threads yet,\n+\t\t * so we can let the IPC layer temporarily chdir() if\n+\t\t * it needs to when creating the server side of the\n+\t\t * Unix domain socket.\n+\t\t */\n+\t\t.uds_disallow_chdir = 0\n+\t};\n+\n+\t/*\n+\t * Start the IPC thread pool before the we've started the file\n+\t * system event listener thread so that we have the IPC handle\n+\t * before we need it.\n+\t */\n+\tif (ipc_server_run_async(&state->ipc_server_data,\n+\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t handle_client, state))\n+\t\treturn error_errno(\n+\t\t\t_(\"could not start IPC thread pool on '%s'\"),\n+\t\t\tfsmonitor_ipc__get_path());\n+\n+\t/*\n+\t * Start the fsmonitor listener thread to collect filesystem\n+\t * events.\n+\t */\n+\tif (pthread_create(&state->listener_thread, NULL,\n+\t\t\t fsm_listen__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\tipc_server_await(state->ipc_server_data);\n+\n+\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t}\n+\n+\t/*\n+\t * The daemon is now fully functional in background threads.\n+\t * Wait for the IPC thread pool to shutdown (whether by client\n+\t * request or from filesystem activity).\n+\t */\n+\tipc_server_await(state->ipc_server_data);\n+\n+\t/*\n+\t * The fsmonitor listener thread may have received a shutdown\n+\t * event from the IPC thread pool, but it doesn't hurt to tell\n+\t * it again. And wait for it to shutdown.\n+\t */\n+\tfsm_listen__stop_async(state);\n+\tpthread_join(state->listener_thread, NULL);\n+\n+\treturn state->error_code;\n+}\n+\n+static int fsmonitor_run_daemon(void)\n+{\n+\tstruct fsmonitor_daemon_state state;\n+\tint err;\n+\n+\tmemset(&state, 0, sizeof(state));\n+\n+\tpthread_mutex_init(&state.main_lock, NULL);\n+\tstate.error_code = 0;\n+\tstate.current_token_data = NULL;\n+\n+\t/* Prepare to (recursively) watch the directory. */\n+\tstrbuf_init(&state.path_worktree_watch, 0);\n+\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n+\tstate.nr_paths_watching = 1;\n+\n+\t/*\n+\t * We create and delete cookie files somewhere inside the .git\n+\t * directory to help us keep sync with the file system. If\n+\t * \".git\" is not a directory, then is not inside the\n+\t * cone of , so set up a second watch to watch\n+\t * the so that we get events for the cookie files.\n+\t */\n+\tstrbuf_init(&state.path_gitdir_watch, 0);\n+\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n+\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n+\tif (!is_directory(state.path_gitdir_watch.buf)) {\n+\t\tstrbuf_reset(&state.path_gitdir_watch);\n+\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n+\t\tstate.nr_paths_watching = 2;\n+\t}\n+\n+\t/*\n+\t * Confirm that we can create platform-specific resources for the\n+\t * filesystem listener before we bother starting all the threads.\n+\t */\n+\tif (fsm_listen__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize listener thread\"));\n+\t\tgoto done;\n+\t}\n+\n+\terr = fsmonitor_run_daemon_1(&state);\n+\n+done:\n+\tpthread_mutex_destroy(&state.main_lock);\n+\tfsm_listen__dtor(&state);\n+\n+\tipc_server_free(state.ipc_server_data);\n+\n+\tstrbuf_release(&state.path_worktree_watch);\n+\tstrbuf_release(&state.path_gitdir_watch);\n+\n+\treturn err;\n+}\n+\n+static int try_to_run_foreground_daemon(void)\n+{\n+\t/*\n+\t * Technically, we don't need to probe for an existing daemon\n+\t * process, since we could just call `fsmonitor_run_daemon()`\n+\t * and let it fail if the pipe/socket is busy.\n+\t *\n+\t * However, this method gives us a nicer error message for a\n+\t * common error case.\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(_(\"fsmonitor--daemon is already running '%s'\"),\n+\t\t the_repository->worktree);\n+\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"running fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n+\n+\treturn !!fsmonitor_run_daemon();\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n \n \tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"ipc-threads\",\n+\t\t\t &fsmonitor__ipc_threads,\n+\t\t\t N_(\"use ipc worker threads\")),\n \t\tOPT_END()\n \t};\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(fsmonitor_config, NULL);\n \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n@@ -73,6 +292,13 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (fsmonitor__ipc_threads < 1)\n+\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n+\t\t fsmonitor__ipc_threads);\n+\n+\tif (!strcmp(subcmd, \"run\"))\n+\t\treturn !!try_to_run_foreground_daemon();\n+\n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n \ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nnew file mode 100644\nindex 00000000000..3009c1a83de\n--- /dev/null\n+++ b/fsmonitor--daemon.h\n@@ -0,0 +1,34 @@\n+#ifndef FSMONITOR_DAEMON_H\n+#define FSMONITOR_DAEMON_H\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+#include \"thread-utils.h\"\n+\n+struct fsmonitor_batch;\n+struct fsmonitor_token_data;\n+\n+struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+\n+struct fsmonitor_daemon_state {\n+\tpthread_t listener_thread;\n+\tpthread_mutex_t main_lock;\n+\n+\tstruct strbuf path_worktree_watch;\n+\tstruct strbuf path_gitdir_watch;\n+\tint nr_paths_watching;\n+\n+\tstruct fsmonitor_token_data *current_token_data;\n+\n+\tint error_code;\n+\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\n+\tstruct ipc_server_data *ipc_server_data;\n+};\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"451870","messageId":"2f8a42fdb93623053256939fdbb45cf02fb1c5c9.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 09/29] compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:50Z","receivedAt":"2022-03-22T18:00:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty implementation of fsmonitor--daemon\nbackend for Darwin (aka MacOS).\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-darwin.c | 20 ++++++++++++++++++++\n config.mak.uname | 10 ++++++++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 33 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nnew file mode 100644\nindex 00000000000..c84e3344ab9\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -0,0 +1,20 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 26074f56bed..501970902da 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -157,6 +157,16 @@ ifeq ($(uname_S),Darwin)\n \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n \t\tendif\n \tendif\n+\n+\t# The builtin FSMonitor on MacOS builds upon Simple-IPC. Both require\n+\t# Unix domain sockets and PThreads.\n+\tifndef NO_PTHREADS\n+\tifndef NO_UNIX_SOCKETS\n+\tFSMONITOR_DAEMON_BACKEND = darwin\n+\tendif\n+\tendif\n+\n+\tBASIC_LDFLAGS += -framework CoreServices\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 0963629db7f..ee0d7257b77 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"451871","messageId":"d62e338d00829bf3fe08e649ffa298c087a354a9.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 12/29] fsmonitor--daemon: add pathname classification","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:53Z","receivedAt":"2022-03-22T18:00:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to classify relative and absolute\npathnames and decide how they should be handled. This will\nbe used by the platform-specific backend to respond to each\nfilesystem event.\n\nWhen we register for filesystem notifications on a directory,\nwe get events for everything (recursively) in the directory.\nWe want to report to clients changes to tracked and untracked\npaths within the working directory proper. We do not want to\nreport changes within the .git directory, for example.\n\nThis classification will be used in a later commit by the\ndifferent backends to classify paths as events are received.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 81 ++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 87 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 168 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 69dd39121a3..1ce00b7c150 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -134,6 +134,87 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n+#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *rel)\n+{\n+\tif (fspathncmp(rel, \".git\", 4))\n+\t\treturn IS_WORKDIR_PATH;\n+\trel += 4;\n+\n+\tif (!*rel)\n+\t\treturn IS_DOT_GIT;\n+\tif (*rel != '/')\n+\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n+\trel++;\n+\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_DOT_GIT;\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *rel)\n+{\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_GITDIR;\n+}\n+\n+static enum fsmonitor_path_type try_classify_workdir_abs_path(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\n+\tif (fspathncmp(path, state->path_worktree_watch.buf,\n+\t\t state->path_worktree_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_worktree_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_workdir_relative(rel);\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\tenum fsmonitor_path_type t;\n+\n+\tt = try_classify_workdir_abs_path(state, path);\n+\tif (state->nr_paths_watching == 1)\n+\t\treturn t;\n+\tif (t != IS_OUTSIDE_CONE)\n+\t\treturn t;\n+\n+\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n+\t\t state->path_gitdir_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_gitdir_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_GITDIR; /* it is the exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_gitdir_relative(rel);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 3009c1a83de..8c3a71a48bd 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -30,5 +30,92 @@ struct fsmonitor_daemon_state {\n \tstruct ipc_server_data *ipc_server_data;\n };\n \n+/*\n+ * Pathname classifications.\n+ *\n+ * The daemon classifies the pathnames that it receives from file\n+ * system notification events into the following categories and uses\n+ * that to decide whether clients are told about them. (And to watch\n+ * for file system synchronization events.)\n+ *\n+ * The daemon only collects and reports on the set of modified paths\n+ * within the working directory (proper).\n+ *\n+ * The client should only care about paths within the working\n+ * directory proper (inside the working directory and not \".git\" nor\n+ * inside of \".git/\"). That is, the client has read the index and is\n+ * asking for a list of any paths in the working directory that have\n+ * been modified since the last token. The client does not care about\n+ * file system changes within the \".git/\" directory (such as new loose\n+ * objects or packfiles). So the client will only receive paths that\n+ * are classified as IS_WORKDIR_PATH.\n+ *\n+ * Note that \".git\" is usually a directory and is therefore inside\n+ * the cone of the FS watch that we have on the working directory root,\n+ * so we will also get FS events for disk activity on and within \".git/\"\n+ * that we need to respond to or filter from the client.\n+ *\n+ * But Git also allows \".git\" to be a *file* that points to a GITDIR\n+ * outside of the working directory. When this happens, we need to\n+ * create FS watches on both the working directory root *and* on the\n+ * (external) GITDIR root. (The latter is required because we put\n+ * cookie files inside it and use them to sync with the FS event\n+ * stream.)\n+ *\n+ * Note that in the context of this discussion, I'm using \"GITDIR\"\n+ * to only mean an external GITDIR referenced by a \".git\" file.\n+ *\n+ * The platform FS event backends will receive watch-specific\n+ * relative paths (except for those OS's that always emit absolute\n+ * paths). We use the following enum and routines to classify each\n+ * path so that we know how to handle it. There is a slight asymmetry\n+ * here because \".git/\" is inside the working directory and the\n+ * (external) GITDIR is not, and therefore how we handle events may\n+ * vary slightly, so I have different enums for \"IS...DOT_GIT...\" and\n+ * \"IS...GITDIR...\".\n+ *\n+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n+ * exact \".git\" file/directory or GITDIR directory. If the daemon\n+ * receives a delete event for either of these paths, it will\n+ * automatically shutdown, for example.\n+ *\n+ * Note that the daemon DOES NOT explicitly watch nor special case the\n+ * index. The daemon does not read the index nor have any internal\n+ * index-relative state, so there are no \"IS...INDEX...\" enum values.\n+ */\n+enum fsmonitor_path_type {\n+\tIS_WORKDIR_PATH = 0,\n+\n+\tIS_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n+\n+\tIS_GITDIR,\n+\tIS_INSIDE_GITDIR,\n+\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n+\n+\tIS_OUTSIDE_CONE,\n+};\n+\n+/*\n+ * Classify a pathname relative to the root of the working directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify a pathname relative to a that is external to the\n+ * worktree directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify an absolute pathname received from a filesystem event.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"451872","messageId":"53e06b4ae5d0eb1d3c2cb6a61ffd14abc36f6534.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 13/29] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:54Z","receivedAt":"2022-03-22T18:00:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to create token-ids and define the\noverall token naming scheme.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 115 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1ce00b7c150..1c7c156288d 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -106,6 +106,120 @@ static int do_as_client__status(void)\n \t}\n }\n \n+/*\n+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n+ * \"token\" as a virtual timestamp. Clients can request a summary of all\n+ * created/deleted/modified files relative to a token. In the response,\n+ * clients receive a new token for the next (relative) request.\n+ *\n+ *\n+ * Token Format\n+ * ============\n+ *\n+ * The contents of the token are private and provider-specific.\n+ *\n+ * For the built-in fsmonitor--daemon, we define a token as follows:\n+ *\n+ * \"builtin\" \":\" \":\" \n+ *\n+ * The \"builtin\" prefix is used as a namespace to avoid conflicts\n+ * with other providers (such as Watchman).\n+ *\n+ * The is an arbitrary OPAQUE string, such as a GUID,\n+ * UUID, or {timestamp,pid}. It is used to group all filesystem\n+ * events that happened while the daemon was monitoring (and in-sync\n+ * with the filesystem).\n+ *\n+ * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n+ * and does not define less-than/greater-than relationships.\n+ * (There are too many race conditions to rely on file system\n+ * event timestamps.)\n+ *\n+ * The is a simple integer incremented whenever the\n+ * daemon needs to make its state public. For example, if 1000 file\n+ * system events come in, but no clients have requested the data,\n+ * the daemon can continue to accumulate file changes in the same\n+ * bin and does not need to advance the sequence number. However,\n+ * as soon as a client does arrive, the daemon needs to start a new\n+ * bin and increment the sequence number.\n+ *\n+ * The sequence number serves as the boundary between 2 sets\n+ * of bins -- the older ones that the client has already seen\n+ * and the newer ones that it hasn't.\n+ *\n+ * When a new is created, the is reset to\n+ * zero.\n+ *\n+ *\n+ * About Token Ids\n+ * ===============\n+ *\n+ * A new token_id is created:\n+ *\n+ * [1] each time the daemon is started.\n+ *\n+ * [2] any time that the daemon must re-sync with the filesystem\n+ * (such as when the kernel drops or we miss events on a very\n+ * active volume).\n+ *\n+ * [3] in response to a client \"flush\" command (for dropped event\n+ * testing).\n+ *\n+ * When a new token_id is created, the daemon is free to discard all\n+ * cached filesystem events associated with any previous token_ids.\n+ * Events associated with a non-current token_id will never be sent\n+ * to a client. A token_id change implicitly means that the daemon\n+ * has gap in its event history.\n+ *\n+ * Therefore, clients that present a token with a stale (non-current)\n+ * token_id will always be given a trivial response.\n+ */\n+struct fsmonitor_token_data {\n+\tstruct strbuf token_id;\n+\tstruct fsmonitor_batch *batch_head;\n+\tstruct fsmonitor_batch *batch_tail;\n+\tuint64_t client_ref_count;\n+};\n+\n+static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n+{\n+\tstatic int test_env_value = -1;\n+\tstatic uint64_t flush_count = 0;\n+\tstruct fsmonitor_token_data *token;\n+\n+\tCALLOC_ARRAY(token, 1);\n+\n+\tstrbuf_init(&token->token_id, 0);\n+\ttoken->batch_head = NULL;\n+\ttoken->batch_tail = NULL;\n+\ttoken->client_ref_count = 0;\n+\n+\tif (test_env_value < 0)\n+\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n+\n+\tif (!test_env_value) {\n+\t\tstruct timeval tv;\n+\t\tstruct tm tm;\n+\t\ttime_t secs;\n+\n+\t\tgettimeofday(&tv, NULL);\n+\t\tsecs = tv.tv_sec;\n+\t\tgmtime_r(&secs, &tm);\n+\n+\t\tstrbuf_addf(&token->token_id,\n+\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n+\t\t\t flush_count++,\n+\t\t\t getpid(),\n+\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n+\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n+\t\t\t (long)tv.tv_usec);\n+\t} else {\n+\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n+\t}\n+\n+\treturn token;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -298,7 +412,7 @@ static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n-\tstate.current_token_data = NULL;\n+\tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n \tstrbuf_init(&state.path_worktree_watch, 0);\n-- \ngitgitgadget\n\n"},{"id":"451873","messageId":"39f43fabe024917e6c84b8916197b0dc0add97fc.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 14/29] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:55Z","receivedAt":"2022-03-22T18:01:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to build a list of changed paths and associate\nthem with a token-id. This will be used by the platform-specific\nbackends to accumulate changed paths in response to filesystem events.\n\nThe platform-specific file system listener thread receives file system\nevents containing one or more changed pathnames (with whatever\nbucketing or grouping that is convenient for the file system). These\npaths are accumulated (without locking) by the file system layer into\na `fsmonitor_batch`.\n\nWhen the file system layer has drained the kernel event queue, it will\n\"publish\" them to our token queue and make them visible to concurrent\nclient worker threads. The token layer is free to combine and/or de-dup\npaths within these batches for efficient presentation to clients.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 230 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 40 +++++++\n 2 files changed, 268 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1c7c156288d..69312119b07 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -181,17 +181,27 @@ struct fsmonitor_token_data {\n \tuint64_t client_ref_count;\n };\n \n+struct fsmonitor_batch {\n+\tstruct fsmonitor_batch *next;\n+\tuint64_t batch_seq_nr;\n+\tconst char **interned_paths;\n+\tsize_t nr, alloc;\n+\ttime_t pinned_time;\n+};\n+\n static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n {\n \tstatic int test_env_value = -1;\n \tstatic uint64_t flush_count = 0;\n \tstruct fsmonitor_token_data *token;\n+\tstruct fsmonitor_batch *batch;\n \n \tCALLOC_ARRAY(token, 1);\n+\tbatch = fsmonitor_batch__new();\n \n \tstrbuf_init(&token->token_id, 0);\n-\ttoken->batch_head = NULL;\n-\ttoken->batch_tail = NULL;\n+\ttoken->batch_head = batch;\n+\ttoken->batch_tail = batch;\n \ttoken->client_ref_count = 0;\n \n \tif (test_env_value < 0)\n@@ -217,9 +227,143 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n \t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n \t}\n \n+\t/*\n+\t * We created a new and are starting a new series\n+\t * of tokens with a zero .\n+\t *\n+\t * Since clients cannot guess our new (non test) \n+\t * they will always receive a trivial response (because of the\n+\t * mismatch on the ). The trivial response will\n+\t * tell them our new so that subsequent requests\n+\t * will be relative to our new series. (And when sending that\n+\t * response, we pin the current head of the batch list.)\n+\t *\n+\t * Even if the client correctly guesses the , their\n+\t * request of \"builtin::0\" asks for all changes MORE\n+\t * RECENT than batch/bin 0.\n+\t *\n+\t * This implies that it is a waste to accumulate paths in the\n+\t * initial batch/bin (because they will never be transmitted).\n+\t *\n+\t * So the daemon could be running for days and watching the\n+\t * file system, but doesn't need to actually accumulate any\n+\t * paths UNTIL we need to set a reference point for a later\n+\t * relative request.\n+\t *\n+\t * However, it is very useful for testing to always have a\n+\t * reference point set. Pin batch 0 to force early file system\n+\t * events to accumulate.\n+\t */\n+\tif (test_env_value)\n+\t\tbatch->pinned_time = time(NULL);\n+\n \treturn token;\n }\n \n+struct fsmonitor_batch *fsmonitor_batch__new(void)\n+{\n+\tstruct fsmonitor_batch *batch;\n+\n+\tCALLOC_ARRAY(batch, 1);\n+\n+\treturn batch;\n+}\n+\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch)\n+{\n+\twhile (batch) {\n+\t\tstruct fsmonitor_batch *next = batch->next;\n+\n+\t\t/*\n+\t\t * The actual strings within the array of this batch\n+\t\t * are interned, so we don't own them. We only own\n+\t\t * the array.\n+\t\t */\n+\t\tfree(batch->interned_paths);\n+\t\tfree(batch);\n+\n+\t\tbatch = next;\n+\t}\n+}\n+\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n+\t\t\t const char *path)\n+{\n+\tconst char *interned_path = strintern(path);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n+\n+\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n+\tbatch->interned_paths[batch->nr++] = interned_path;\n+}\n+\n+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n+\t\t\t\t const struct fsmonitor_batch *batch_src)\n+{\n+\tsize_t k;\n+\n+\tALLOC_GROW(batch_dest->interned_paths,\n+\t\t batch_dest->nr + batch_src->nr + 1,\n+\t\t batch_dest->alloc);\n+\n+\tfor (k = 0; k < batch_src->nr; k++)\n+\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n+\t\t\tbatch_src->interned_paths[k];\n+}\n+\n+static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n+{\n+\tif (!token)\n+\t\treturn;\n+\n+\tassert(token->client_ref_count == 0);\n+\n+\tstrbuf_release(&token->token_id);\n+\n+\tfsmonitor_batch__free_list(token->batch_head);\n+\n+\tfree(token);\n+}\n+\n+/*\n+ * Flush all of our cached data about the filesystem. Call this if we\n+ * lose sync with the filesystem and miss some notification events.\n+ *\n+ * [1] If we are missing events, then we no longer have a complete\n+ * history of the directory (relative to our current start token).\n+ * We should create a new token and start fresh (as if we just\n+ * booted up).\n+ *\n+ * If there are no concurrent threads reading the current token data\n+ * series, we can free it now. Otherwise, let the last reader free\n+ * it.\n+ *\n+ * Either way, the old token data series is no longer associated with\n+ * our state data.\n+ */\n+static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct fsmonitor_token_data *free_me = NULL;\n+\tstruct fsmonitor_token_data *new_one = NULL;\n+\n+\tnew_one = fsmonitor_new_token_data();\n+\n+\tif (state->current_token_data->client_ref_count == 0)\n+\t\tfree_me = state->current_token_data;\n+\tstate->current_token_data = new_one;\n+\n+\tfsmonitor_free_token_data(free_me);\n+}\n+\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\tpthread_mutex_lock(&state->main_lock);\n+\twith_lock__do_force_resync(state);\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -329,6 +473,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n \n+/*\n+ * We try to combine small batches at the front of the batch-list to avoid\n+ * having a long list. This hopefully makes it a little easier when we want\n+ * to truncate and maintain the list. However, we don't want the paths array\n+ * to just keep growing and growing with realloc, so we insert an arbitrary\n+ * limit.\n+ */\n+#define MY_COMBINE_LIMIT (1024)\n+\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names)\n+{\n+\tif (!batch && !cookie_names->nr)\n+\t\treturn;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (batch) {\n+\t\tstruct fsmonitor_batch *head;\n+\n+\t\thead = state->current_token_data->batch_head;\n+\t\tif (!head) {\n+\t\t\tBUG(\"token does not have batch\");\n+\t\t} else if (head->pinned_time) {\n+\t\t\t/*\n+\t\t\t * We cannot alter the current batch list\n+\t\t\t * because:\n+\t\t\t *\n+\t\t\t * [a] it is being transmitted to at least one\n+\t\t\t * client and the handle_client() thread has a\n+\t\t\t * ref-count, but not a lock on the batch list\n+\t\t\t * starting with this item.\n+\t\t\t *\n+\t\t\t * [b] it has been transmitted in the past to\n+\t\t\t * at least one client such that future\n+\t\t\t * requests are relative to this head batch.\n+\t\t\t *\n+\t\t\t * So, we can only prepend a new batch onto\n+\t\t\t * the front of the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else if (!head->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * Batch 0 is unpinned. See the note in\n+\t\t\t * `fsmonitor_new_token_data()` about why we\n+\t\t\t * don't need to accumulate these paths.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n+\t\t\t/*\n+\t\t\t * The head batch in the list has never been\n+\t\t\t * transmitted to a client, but folding the\n+\t\t\t * contents of the new batch onto it would\n+\t\t\t * exceed our arbitrary limit, so just prepend\n+\t\t\t * the new batch onto the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We are free to add the paths in the given\n+\t\t\t * batch onto the end of the current head batch.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__combine(head, batch);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -343,6 +562,13 @@ static void *fsm_listen__thread_proc(void *_state)\n \n \tfsm_listen__loop(state);\n \n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (state->current_token_data &&\n+\t state->current_token_data->client_ref_count == 0)\n+\t\tfsmonitor_free_token_data(state->current_token_data);\n+\tstate->current_token_data = NULL;\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n \ttrace2_thread_exit();\n \treturn NULL;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 8c3a71a48bd..010fbfe60e9 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -12,6 +12,27 @@\n struct fsmonitor_batch;\n struct fsmonitor_token_data;\n \n+/*\n+ * Create a new batch of path(s). The returned batch is considered\n+ * private and not linked into the fsmonitor daemon state. The caller\n+ * should fill this batch with one or more paths and then publish it.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__new(void);\n+\n+/*\n+ * Free the list of batches starting with this one.\n+ */\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n+\n+/*\n+ * Add this path to this batch of modified files.\n+ *\n+ * The batch should be private and NOT (yet) linked into the fsmonitor\n+ * daemon state and therefore not yet visible to worker threads and so\n+ * no locking is required.\n+ */\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n+\n struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n \n struct fsmonitor_daemon_state {\n@@ -117,5 +138,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \tstruct fsmonitor_daemon_state *state,\n \tconst char *path);\n \n+/*\n+ * Prepend the this batch of path(s) onto the list of batches associated\n+ * with the current token. This makes the batch visible to worker threads.\n+ *\n+ * The caller no longer owns the batch and must not free it.\n+ *\n+ * Wake up the client threads waiting on these cookies.\n+ */\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names);\n+\n+/*\n+ * If the platform-specific layer loses sync with the filesystem,\n+ * it should call this to invalidate cached data and abort waiting\n+ * threads.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"451874","messageId":"239558e34ffea4240de8699707cd3b404b682488.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 15/29] compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:56Z","receivedAt":"2022-03-22T18:01:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the win32 backend to register a watch on the working tree\nroot directory (recursively). Also watch the if it is\nnot inside the working tree. And to collect path change notifications\ninto batches and publish.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-win32.c | 565 ++++++++++++++++++++++++++++\n 1 file changed, 565 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 916cbea254f..5b928ab66e5 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -2,20 +2,585 @@\n #include \"config.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+/*\n+ * The documentation of ReadDirectoryChangesW() states that the maximum\n+ * buffer size is 64K when the monitored directory is remote.\n+ *\n+ * Larger buffers may be used when the monitored directory is local and\n+ * will help us receive events faster from the kernel and avoid dropped\n+ * events.\n+ *\n+ * So we try to use a very large buffer and silently fallback to 64K if\n+ * we get an error.\n+ */\n+#define MAX_RDCW_BUF_FALLBACK (65536)\n+#define MAX_RDCW_BUF (65536 * 8)\n+\n+struct one_watch\n+{\n+\tchar buffer[MAX_RDCW_BUF];\n+\tDWORD buf_len;\n+\tDWORD count;\n+\n+\tstruct strbuf path;\n+\tHANDLE hDir;\n+\tHANDLE hEvent;\n+\tOVERLAPPED overlapped;\n+\n+\t/*\n+\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n+\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n+\t */\n+\tBOOL is_active;\n+};\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tstruct one_watch *watch_worktree;\n+\tstruct one_watch *watch_gitdir;\n+\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hListener[3]; /* we don't own these handles */\n+#define LISTENER_SHUTDOWN 0\n+#define LISTENER_HAVE_DATA_WORKTREE 1\n+#define LISTENER_HAVE_DATA_GITDIR 2\n+\tint nr_listener_handles;\n+};\n+\n+/*\n+ * Convert the WCHAR path from the notification into UTF8 and\n+ * then normalize it.\n+ */\n+static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+\t\t\t\t struct strbuf *normalized_path)\n+{\n+\tint reserve;\n+\tint len = 0;\n+\n+\tstrbuf_reset(normalized_path);\n+\tif (!info->FileNameLength)\n+\t\tgoto normalize;\n+\n+\t/*\n+\t * Pre-reserve enough space in the UTF8 buffer for\n+\t * each Unicode WCHAR character to be mapped into a\n+\t * sequence of 2 UTF8 characters. That should let us\n+\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n+\t */\n+\treserve = info->FileNameLength + 1;\n+\tstrbuf_grow(normalized_path, reserve);\n+\n+\tfor (;;) {\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n+\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t\t\t normalized_path->buf,\n+\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n+\t\t\t\t\t NULL, NULL);\n+\t\tif (len > 0)\n+\t\t\tgoto normalize;\n+\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n+\t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n+\t\t\t GetLastError(),\n+\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n+\t\t\t info->FileName);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tstrbuf_grow(normalized_path,\n+\t\t\t strbuf_avail(normalized_path) + reserve);\n+\t}\n+\n+normalize:\n+\tstrbuf_setlen(normalized_path, len);\n+\treturn strbuf_normalize_path(normalized_path);\n+}\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+}\n+\n+static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const char *path)\n+{\n+\tstruct one_watch *watch = NULL;\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\twchar_t wpath[MAX_PATH];\n+\n+\tif (xutftowcs_path(wpath, path) < 0) {\n+\t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n+\t\treturn NULL;\n+\t}\n+\n+\thDir = CreateFileW(wpath,\n+\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n+\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n+\t\t\t NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n+\t\t GetLastError(), path);\n+\t\treturn NULL;\n+\t}\n+\n+\tCALLOC_ARRAY(watch, 1);\n+\n+\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n+\n+\tstrbuf_init(&watch->path, 0);\n+\tstrbuf_addstr(&watch->path, path);\n+\n+\twatch->hDir = hDir;\n+\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\treturn watch;\n+}\n+\n+static void destroy_watch(struct one_watch *watch)\n+{\n+\tif (!watch)\n+\t\treturn;\n+\n+\tstrbuf_release(&watch->path);\n+\tif (watch->hDir != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hDir);\n+\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hEvent);\n+\n+\tfree(watch);\n+}\n+\n+static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+\t\t\t struct one_watch *watch)\n+{\n+\tDWORD dwNotifyFilter =\n+\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n+\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n+\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n+\t\tFILE_NOTIFY_CHANGE_SIZE |\n+\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n+\t\tFILE_NOTIFY_CHANGE_CREATION;\n+\n+\tResetEvent(watch->hEvent);\n+\n+\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n+\twatch->overlapped.hEvent = watch->hEvent;\n+\n+\t/*\n+\t * Queue an async call using Overlapped IO. This returns immediately.\n+\t * Our event handle will be signalled when the real result is available.\n+\t *\n+\t * The return value here just means that we successfully queued it.\n+\t * We won't know if the Read...() actually produces data until later.\n+\t */\n+\twatch->is_active = ReadDirectoryChangesW(\n+\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n+\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n+\n+\tif (watch->is_active)\n+\t\treturn 0;\n+\n+\terror(_(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\"),\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static int recv_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD gle;\n+\n+\twatch->is_active = FALSE;\n+\n+\t/*\n+\t * The overlapped result is ready. If the Read...() was successful\n+\t * we finally receive the actual result into our buffer.\n+\t */\n+\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n+\t\t\t\tTRUE))\n+\t\treturn 0;\n+\n+\tgle = GetLastError();\n+\tif (gle == ERROR_INVALID_PARAMETER &&\n+\t /*\n+\t * The kernel throws an invalid parameter error when our\n+\t * buffer is too big and we are pointed at a remote\n+\t * directory (and possibly for other reasons). Quietly\n+\t * set it down and try again.\n+\t *\n+\t * See note about MAX_RDCW_BUF at the top.\n+\t */\n+\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n+\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n+\t\treturn -2;\n+\t}\n+\n+\t/*\n+\t * NEEDSWORK: If an external is deleted, the above\n+\t * returns an error. I'm not sure that there's anything that\n+\t * we can do here other than failing -- the /.git\n+\t * link file would be broken anyway. We might try to check\n+\t * for that and return a better error message, but I'm not\n+\t * sure it is worth it.\n+\t */\n+\n+\terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\n+\t watch->path.buf, gle);\n+\treturn -1;\n+}\n+\n+static void cancel_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD count;\n+\n+\tif (!watch || !watch->is_active)\n+\t\treturn;\n+\n+\t/*\n+\t * The calls to ReadDirectoryChangesW() and GetOverlappedResult()\n+\t * form a \"pair\" (my term) where we queue an IO and promise to\n+\t * hang around and wait for the kernel to give us the result.\n+\t *\n+\t * If for some reason after we queue the IO, we have to quit\n+\t * or otherwise not stick around for the second half, we must\n+\t * tell the kernel to abort the IO. This prevents the kernel\n+\t * from writing to our buffer and/or signalling our event\n+\t * after we free them.\n+\t *\n+\t * (Ask me how much fun it was to track that one down).\n+\t */\n+\tCancelIoEx(watch->hDir, &watch->overlapped);\n+\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n+\twatch->is_active = FALSE;\n+}\n+\n+/*\n+ * Process filesystem events that happen anywhere (recursively) under the\n+ * root directory. For a normal working directory, this includes\n+ * both version controlled files and the contents of the .git/ directory.\n+ *\n+ * If /.git is a file, then we only see events for the file\n+ * itself.\n+ */\n+static int process_worktree_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_worktree;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tconst char *p = watch->buffer;\n+\n+\t/*\n+\t * If the kernel gets more events than will fit in the kernel\n+\t * buffer associated with our RDCW handle, it drops them and\n+\t * returns a count of zero.\n+\t *\n+\t * Yes, the call returns WITHOUT error and with length zero.\n+\t * This is the documented behavior. (My testing has confirmed\n+\t * that it also sets the last error to ERROR_NOTIFY_ENUM_DIR,\n+\t * but we do not rely on that since the function did not\n+\t * return an error and it is not documented.)\n+\t *\n+\t * (The \"overflow\" case is not ambiguous with the \"no data\" case\n+\t * because we did an INFINITE wait.)\n+\t *\n+\t * This means we have a gap in coverage. Tell the daemon layer\n+\t * to resync.\n+\t */\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\t}\n+\n+\t/*\n+\t * On Windows, `info` contains an \"array\" of paths that are\n+\t * relative to the root of whichever directory handle received\n+\t * the event.\n+\t */\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\t\t/* ignore everything inside of \"/.git/\" */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* queue normal pathname */\n+\t\t\tif (!batch)\n+\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tbatch = NULL;\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_SHUTDOWN;\n+}\n+\n+/*\n+ * Process filesystem events that happened anywhere (recursively) under the\n+ * external (such as non-primary worktrees or submodules).\n+ * We only care about cookie files that our client threads created here.\n+ *\n+ * Note that we DO NOT get filesystem events on the external \n+ * itself (it is not inside something that we are watching). In particular,\n+ * we do not get an event if the external is deleted.\n+ */\n+static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_gitdir;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *p = watch->buffer;\n+\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_GITDIR;\n+\t}\n+\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\tgoto skip_this_path;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, NULL, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_GITDIR;\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tDWORD dwWait;\n+\tint result;\n+\n+\tstate->error_code = 0;\n+\n+\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tif (data->watch_gitdir &&\n+\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tfor (;;) {\n+\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n+\t\t\t\t\t\tdata->hListener,\n+\t\t\t\t\t\tFALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_worktree);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_gitdir);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->error_code = -1;\n+\n+force_shutdown:\n+\t/*\n+\t * Tell the IPC thead pool to stop (which completes the await\n+\t * in the main thread (which will also signal this thread (if\n+\t * we are still alive))).\n+\t */\n+\tipc_server_stop_async(state->ipc_server_data);\n+\n+clean_shutdown:\n+\tcancel_rdcw_watch(data->watch_worktree);\n+\tcancel_rdcw_watch(data->watch_gitdir);\n }\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->watch_worktree = create_watch(state,\n+\t\t\t\t\t state->path_worktree_watch.buf);\n+\tif (!data->watch_worktree)\n+\t\tgoto failed;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->watch_gitdir = create_watch(state,\n+\t\t\t\t\t\t state->path_gitdir_watch.buf);\n+\t\tif (!data->watch_gitdir)\n+\t\t\tgoto failed;\n+\t}\n+\n+\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_listener_handles++;\n+\n+\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n+\t\tdata->watch_worktree->hEvent;\n+\tdata->nr_listener_handles++;\n+\n+\tif (data->watch_gitdir) {\n+\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n+\t\t\tdata->watch_gitdir->hEvent;\n+\t\tdata->nr_listener_handles++;\n+\t}\n+\n+\tstate->backend_data = data;\n+\treturn 0;\n+\n+failed:\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n-- \ngitgitgadget\n\n"},{"id":"451875","messageId":"14b775e9d8b1a4672f8175a546eb70e2790c1b23.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 16/29] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:57Z","receivedAt":"2022-03-22T18:01:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInclude MacOS system declarations to allow us to use FSEvent and\nCoreFoundation APIs. We need different versions of the declarations\nfor GCC vs. clang because of compiler and header file conflicts.\n\nWhile it is quite possible to #include Apple's CoreServices.h when\ncompiling C source code with clang, trying to build it with GCC\ncurrently fails with this error:\n\nIn file included\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Security.h:42,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/CSIdentity.h:43,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/OSServices.h:29,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/IconsCore.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/LaunchServices.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n\n /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Authorization.h:193:7:\n error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n\nThe underlying reason is that GCC (rightfully) objects that an `enum`\nvalue such as `kAuthorizationExternalFormLength` is not a constant\n(because it is not, the preprocessor has no knowledge of it, only the\nactual C compiler does) and can therefore not be used to define the size\nof a C array.\n\nThis is a known problem and tracked in GCC's bug tracker:\nhttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nIn the meantime, let's not block things and go the slightly ugly route\nof declaring/defining the FSEvents constants, data structures and\nfunctions that we need, so that we can avoid above-mentioned issue.\n\nLet's do this _only_ for GCC, though, so that the CI/PR builds (which\nbuild both with clang and with GCC) can guarantee that we _are_ using\nthe correct data types.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-darwin-gcc.h | 92 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-listen-darwin.c | 24 ++++++++\n 2 files changed, 116 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-darwin-gcc.h\n\ndiff --git a/compat/fsmonitor/fsm-darwin-gcc.h b/compat/fsmonitor/fsm-darwin-gcc.h\nnew file mode 100644\nindex 00000000000..1c75c3d48e7\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-darwin-gcc.h\n@@ -0,0 +1,92 @@\n+#ifndef FSM_DARWIN_GCC_H\n+#define FSM_DARWIN_GCC_H\n+\n+#ifndef __clang__\n+/*\n+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n+ * with clang, but not with GCC as of time of writing.\n+ *\n+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details.\n+ */\n+typedef unsigned int FSEventStreamCreateFlags;\n+#define kFSEventStreamEventFlagNone 0x00000000\n+#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n+#define kFSEventStreamEventFlagUserDropped 0x00000002\n+#define kFSEventStreamEventFlagKernelDropped 0x00000004\n+#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n+#define kFSEventStreamEventFlagHistoryDone 0x00000010\n+#define kFSEventStreamEventFlagRootChanged 0x00000020\n+#define kFSEventStreamEventFlagMount 0x00000040\n+#define kFSEventStreamEventFlagUnmount 0x00000080\n+#define kFSEventStreamEventFlagItemCreated 0x00000100\n+#define kFSEventStreamEventFlagItemRemoved 0x00000200\n+#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n+#define kFSEventStreamEventFlagItemRenamed 0x00000800\n+#define kFSEventStreamEventFlagItemModified 0x00001000\n+#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n+#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n+#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n+#define kFSEventStreamEventFlagItemIsFile 0x00010000\n+#define kFSEventStreamEventFlagItemIsDir 0x00020000\n+#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n+#define kFSEventStreamEventFlagOwnEvent 0x00080000\n+#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n+#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+\n+typedef struct __FSEventStream *FSEventStreamRef;\n+typedef const FSEventStreamRef ConstFSEventStreamRef;\n+\n+typedef unsigned int CFStringEncoding;\n+#define kCFStringEncodingUTF8 0x08000100\n+\n+typedef const struct __CFString *CFStringRef;\n+typedef const struct __CFArray *CFArrayRef;\n+typedef const struct __CFRunLoop *CFRunLoopRef;\n+\n+struct FSEventStreamContext {\n+ long long version;\n+ void *cb_data, *retain, *release, *copy_description;\n+};\n+\n+typedef struct FSEventStreamContext FSEventStreamContext;\n+typedef unsigned int FSEventStreamEventFlags;\n+#define kFSEventStreamCreateFlagNoDefer 0x02\n+#define kFSEventStreamCreateFlagWatchRoot 0x04\n+#define kFSEventStreamCreateFlagFileEvents 0x10\n+\n+typedef unsigned long long FSEventStreamEventId;\n+#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL\n+\n+typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef,\n+\t\t\t\t void *context,\n+\t\t\t\t __SIZE_TYPE__ num_of_events,\n+\t\t\t\t void *event_paths,\n+\t\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t\t const FSEventStreamEventId event_ids[]);\n+typedef double CFTimeInterval;\n+FSEventStreamRef FSEventStreamCreate(void *allocator,\n+\t\t\t\t FSEventStreamCallback callback,\n+\t\t\t\t FSEventStreamContext *context,\n+\t\t\t\t CFArrayRef paths_to_watch,\n+\t\t\t\t FSEventStreamEventId since_when,\n+\t\t\t\t CFTimeInterval latency,\n+\t\t\t\t FSEventStreamCreateFlags flags);\n+CFStringRef CFStringCreateWithCString(void *allocator, const char *string,\n+\t\t\t\t CFStringEncoding encoding);\n+CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count,\n+\t\t\t void *callbacks);\n+void CFRunLoopRun(void);\n+void CFRunLoopStop(CFRunLoopRef run_loop);\n+CFRunLoopRef CFRunLoopGetCurrent(void);\n+extern CFStringRef kCFRunLoopDefaultMode;\n+void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream,\n+\t\t\t\t CFRunLoopRef run_loop,\n+\t\t\t\t CFStringRef run_loop_mode);\n+unsigned char FSEventStreamStart(FSEventStreamRef stream);\n+void FSEventStreamStop(FSEventStreamRef stream);\n+void FSEventStreamInvalidate(FSEventStreamRef stream);\n+void FSEventStreamRelease(FSEventStreamRef stream);\n+\n+#endif /* !clang */\n+#endif /* FSM_DARWIN_GCC_H */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex c84e3344ab9..d2ce942cade 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -1,3 +1,27 @@\n+#ifndef __clang__\n+#include \"fsm-darwin-gcc.h\"\n+#else\n+#include \n+#include \n+\n+#ifndef AVAILABLE_MAC_OS_X_VERSION_10_13_AND_LATER\n+/*\n+ * This enum value was added in 10.13 to:\n+ *\n+ * /Applications/Xcode.app/Contents/Developer/Platforms/ \\\n+ * MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/ \\\n+ * Library/Frameworks/CoreServices.framework/Frameworks/ \\\n+ * FSEvents.framework/Versions/Current/Headers/FSEvents.h\n+ *\n+ * If we're compiling against an older SDK, this symbol won't be\n+ * present. Silently define it here so that we don't have to ifdef\n+ * the logging or masking below. This should be harmless since older\n+ * versions of macOS won't ever emit this FS event anyway.\n+ */\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+#endif\n+#endif\n+\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n-- \ngitgitgadget\n\n"},{"id":"451876","messageId":"55bd7aee06ca04e6ae9010700103a261d1255113.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 17/29] compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:58Z","receivedAt":"2022-03-22T18:01:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement file system event listener on MacOS using FSEvent,\nCoreFoundation, and CoreServices.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-darwin.c | 383 +++++++++++++++++++++++++++\n 1 file changed, 383 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex d2ce942cade..0741fe834c3 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -25,20 +25,403 @@\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tCFStringRef cfsr_worktree_path;\n+\tCFStringRef cfsr_gitdir_path;\n+\n+\tCFArrayRef cfar_paths_to_watch;\n+\tint nr_paths_watching;\n+\n+\tFSEventStreamRef stream;\n+\n+\tCFRunLoopRef rl;\n+\n+\tenum shutdown_style {\n+\t\tSHUTDOWN_EVENT = 0,\n+\t\tFORCE_SHUTDOWN,\n+\t\tFORCE_ERROR_STOP,\n+\t} shutdown_style;\n+\n+\tunsigned int stream_scheduled:1;\n+\tunsigned int stream_started:1;\n+};\n+\n+static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n+\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n+\tif (flag & kFSEventStreamEventFlagUserDropped)\n+\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n+\tif (flag & kFSEventStreamEventFlagKernelDropped)\n+\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n+\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n+\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n+\tif (flag & kFSEventStreamEventFlagHistoryDone)\n+\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n+\tif (flag & kFSEventStreamEventFlagRootChanged)\n+\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n+\tif (flag & kFSEventStreamEventFlagMount)\n+\t\tstrbuf_addstr(&msg, \"Mount|\");\n+\tif (flag & kFSEventStreamEventFlagUnmount)\n+\t\tstrbuf_addstr(&msg, \"Unmount|\");\n+\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n+\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n+\tif (flag & kFSEventStreamEventFlagItemCreated)\n+\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n+\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n+\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n+\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsDir)\n+\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsFile)\n+\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemModified)\n+\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n+\tif (flag & kFSEventStreamEventFlagItemRemoved)\n+\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n+\tif (flag & kFSEventStreamEventFlagItemRenamed)\n+\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n+\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n+\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n+\tif (flag & kFSEventStreamEventFlagOwnEvent)\n+\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n+\tif (flag & kFSEventStreamEventFlagItemCloned)\n+\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\t\t\t path, flag, msg.buf);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRemoved);\n+}\n+\n+static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRenamed);\n+}\n+\n+static int ef_is_dropped(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagMustScanSubDirs ||\n+\t\tef & kFSEventStreamEventFlagKernelDropped ||\n+\t\tef & kFSEventStreamEventFlagUserDropped);\n+}\n+\n+static void fsevent_callback(ConstFSEventStreamRef streamRef,\n+\t\t\t void *ctx,\n+\t\t\t size_t num_of_events,\n+\t\t\t void *event_paths,\n+\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t const FSEventStreamEventId event_ids[])\n+{\n+\tstruct fsmonitor_daemon_state *state = ctx;\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tchar **paths = (char **)event_paths;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *path_k;\n+\tconst char *slash;\n+\tint k;\n+\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t/*\n+\t * Build a list of all filesystem changes into a private/local\n+\t * list and without holding any locks.\n+\t */\n+\tfor (k = 0; k < num_of_events; k++) {\n+\t\t/*\n+\t\t * On Mac, we receive an array of absolute paths.\n+\t\t */\n+\t\tpath_k = paths[k];\n+\n+\t\t/*\n+\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n+\t\t * Please don't log them to Trace2.\n+\t\t *\n+\t\t * trace_printf_key(&trace_fsmonitor, \"Path: '%s'\", path_k);\n+\t\t */\n+\n+\t\t/*\n+\t\t * If event[k] is marked as dropped, we assume that we have\n+\t\t * lost sync with the filesystem and should flush our cached\n+\t\t * data. We need to:\n+\t\t *\n+\t\t * [1] Abort/wake any client threads waiting for a cookie and\n+\t\t * flush the cached state data (the current token), and\n+\t\t * create a new token.\n+\t\t *\n+\t\t * [2] Discard the batch that we were locally building (since\n+\t\t * they are conceptually relative to the just flushed\n+\t\t * token).\n+\t\t */\n+\t\tif (ef_is_dropped(event_flags[k])) {\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\tfsmonitor_force_resync(state);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t\tstring_list_clear(&cookie_list, 0);\n+\n+\t\t\t/*\n+\t\t\t * We assume that any events that we received\n+\t\t\t * in this callback after this dropped event\n+\t\t\t * may still be valid, so we continue rather\n+\t\t\t * than break. (And just in case there is a\n+\t\t\t * delete of \".git\" hiding in there.)\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n+\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git or gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path_k);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path_k);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\t/* ignore all other paths inside of .git or gitdir */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\tcase IS_GITDIR:\n+\t\t\t/*\n+\t\t\t * If .git directory is deleted or renamed away,\n+\t\t\t * we have to quit.\n+\t\t\t */\n+\t\t\tif (ef_is_root_delete(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir renamed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* try to queue normal pathnames */\n+\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\t/*\n+\t\t\t * Because of the implicit \"binning\" (the\n+\t\t\t * kernel calls us at a given frequency) and\n+\t\t\t * de-duping (the kernel is free to combine\n+\t\t\t * multiple events for a given pathname), an\n+\t\t\t * individual fsevent could be marked as both\n+\t\t\t * a file and directory. Add it to the queue\n+\t\t\t * with both spellings so that the client will\n+\t\t\t * know how much to invalidate/refresh.\n+\t\t\t */\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t}\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tstrbuf_reset(&tmp);\n+\t\t\t\tstrbuf_addstr(&tmp, rel);\n+\t\t\t\tstrbuf_addch(&tmp, '/');\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\n+\t\tcase IS_OUTSIDE_CONE:\n+\t\tdefault:\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignoring '%s'\", path_k);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\n+\tdata->shutdown_style = FORCE_SHUTDOWN;\n+\tCFRunLoopStop(data->rl);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+}\n+\n+/*\n+ * In the call to `FSEventStreamCreate()` to setup our watch, the\n+ * `latency` argument determines the frequency of calls to our callback\n+ * with new FS events. Too slow and events get dropped; too fast and\n+ * we burn CPU unnecessarily. Since it is rather obscure, I don't\n+ * think this needs to be a config setting. I've done extensive\n+ * testing on my systems and chosen the value below. It gives good\n+ * results and I've not seen any dropped events.\n+ *\n+ * With a latency of 0.1, I was seeing lots of dropped events during\n+ * the \"touch 100000\" files test within t/perf/p7519, but with a\n+ * latency of 0.001 I did not see any dropped events. So I'm going\n+ * to assume that this is the \"correct\" value.\n+ *\n+ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n+ */\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n+\t\tkFSEventStreamCreateFlagWatchRoot |\n+\t\tkFSEventStreamCreateFlagFileEvents;\n+\tFSEventStreamContext ctx = {\n+\t\t0,\n+\t\tstate,\n+\t\tNULL,\n+\t\tNULL,\n+\t\tNULL\n+\t};\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\tconst void *dir_array[2];\n+\n+\tCALLOC_ARRAY(data, 1);\n+\tstate->backend_data = data;\n+\n+\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n+\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n+\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n+\t\t\tNULL, state->path_gitdir_watch.buf,\n+\t\t\tkCFStringEncodingUTF8);\n+\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n+\t}\n+\n+\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n+\t\t\t\t\t\t data->nr_paths_watching,\n+\t\t\t\t\t\t NULL);\n+\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n+\t\t\t\t\t data->cfar_paths_to_watch,\n+\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n+\t\t\t\t\t 0.001, flags);\n+\tif (data->stream == NULL)\n+\t\tgoto failed;\n+\n+\t/*\n+\t * `data->rl` needs to be set inside the listener thread.\n+\t */\n+\n+\treturn 0;\n+\n+failed:\n+\terror(_(\"Unable to create FSEventStream.\"));\n+\n+\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tif (data->stream) {\n+\t\tif (data->stream_started)\n+\t\t\tFSEventStreamStop(data->stream);\n+\t\tif (data->stream_scheduled)\n+\t\t\tFSEventStreamInvalidate(data->stream);\n+\t\tFSEventStreamRelease(data->stream);\n+\t}\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\tdata->shutdown_style = SHUTDOWN_EVENT;\n+\n+\tCFRunLoopStop(data->rl);\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\n+\tdata->rl = CFRunLoopGetCurrent();\n+\n+\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n+\tdata->stream_scheduled = 1;\n+\n+\tif (!FSEventStreamStart(data->stream)) {\n+\t\terror(_(\"Failed to start the FSEventStream\"));\n+\t\tgoto force_error_stop_without_loop;\n+\t}\n+\tdata->stream_started = 1;\n+\n+\tCFRunLoopRun();\n+\n+\tswitch (data->shutdown_style) {\n+\tcase FORCE_ERROR_STOP:\n+\t\tstate->error_code = -1;\n+\t\t/* fall thru */\n+\tcase FORCE_SHUTDOWN:\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\t/* fall thru */\n+\tcase SHUTDOWN_EVENT:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn;\n+\n+force_error_stop_without_loop:\n+\tstate->error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+\treturn;\n }\n-- \ngitgitgadget\n\n"},{"id":"451877","messageId":"803a540cc0022e893a75eae8815b3275a7fac3af.1647972011.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 24/29] t/perf/p7519: speed up test on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:05Z","receivedAt":"2022-03-22T18:01:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nChange p7519 to use `test_seq` and `xargs` rather than a `for` loop\nto touch thousands of files. This takes minutes off of test runs\non Windows because of process creation overhead.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 32 ++++++++++++++++++++------------\n 1 file changed, 20 insertions(+), 12 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex c8be58f3c76..0611e533951 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -72,7 +72,7 @@ then\n \tfi\n fi\n \n-trace_start() {\n+trace_start () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tname=\"$1\"\n@@ -91,13 +91,20 @@ trace_start() {\n \tfi\n }\n \n-trace_stop() {\n+trace_stop () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tunset GIT_TRACE2_PERF\n \tfi\n }\n \n+touch_files () {\n+\tn=$1\n+\td=\"$n\"_files\n+\n+\t(cd $d ; test_seq 1 $n | xargs touch )\n+}\n+\n test_expect_success \"one time repo setup\" '\n \t# set untrackedCache depending on the environment\n \tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n@@ -119,10 +126,11 @@ test_expect_success \"one time repo setup\" '\n \tfi &&\n \n \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n-\tfor i in $(test_seq 1 10); do touch 10_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n+\t: 1_file directory should be left empty &&\n+\ttouch_files 10 &&\n+\ttouch_files 100 &&\n+\ttouch_files 1000 &&\n+\ttouch_files 10000 &&\n \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n \tgit commit -qm \"Add files\" &&\n \n@@ -133,7 +141,7 @@ test_expect_success \"one time repo setup\" '\n \tfi\n '\n \n-setup_for_fsmonitor() {\n+setup_for_fsmonitor () {\n \t# set INTEGRATION_SCRIPT depending on the environment\n \tif test -n \"$INTEGRATION_PATH\"\n \tthen\n@@ -173,7 +181,7 @@ test_perf_w_drop_caches () {\n \ttest_perf \"$@\"\n }\n \n-test_fsmonitor_suite() {\n+test_fsmonitor_suite () {\n \tif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n@@ -199,15 +207,15 @@ test_fsmonitor_suite() {\n \n \t# Update the mtimes on upto 100k files to make status think\n \t# that they are dirty. For simplicity, omit any files with\n-\t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n-\t# Then fully backslash-quote the paths to capture any\n-\t# whitespace so that they pass thru xargs properly.\n+\t# LFs (i.e. anything that ls-files thinks it needs to dquote)\n+\t# and any files with whitespace so that they pass thru xargs\n+\t# properly.\n \t#\n \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n \t\tgit ls-files | \\\n \t\t\thead -100000 | \\\n \t\t\tgrep -v \\\" | \\\n-\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n+\t\t\tgrep -v \" .\" | \\\n \t\t\txargs test-tool chmtime -300 &&\n \t\tgit status\n \t'\n-- \ngitgitgadget\n\n"},{"id":"451878","messageId":"8cf62c9fc6f07980d35138b4be9cacbf007cd32a.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 19/29] help: include fsmonitor--daemon feature flag in version info","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:00Z","receivedAt":"2022-03-22T18:01:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the \"feature: fsmonitor--daemon\" message to the output of\n`git version --build-options`.\n\nThe builtin FSMonitor is only available on certain platforms and\neven then only when certain Makefile flags are enabled, so print\na message in the verbose version output when it is available.\n\nThis can be used by test scripts for prereq testing. Granted, tests\ncould just try `git fsmonitor--daemon status` and look for a 128 exit\ncode or grep for a \"not supported\" message on stderr, but these\nmethods are rather obscure.\n\nThe main advantage is that the feature message will automatically\nappear in bug reports and other support requests.\n\nThis concept was also used during the development of Scalar for\nsimilar reasons.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n help.c | 4 ++++\n t/test-lib.sh | 7 +++++++\n 2 files changed, 11 insertions(+)\n\ndiff --git a/help.c b/help.c\nindex 71444906ddf..9112a51e84b 100644\n--- a/help.c\n+++ b/help.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"parse-options.h\"\n #include \"prompt.h\"\n+#include \"fsmonitor-ipc.h\"\n \n struct category_description {\n \tuint32_t category;\n@@ -695,6 +696,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n+\n+\t\tif (fsmonitor_ipc__is_supported())\n+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n \t}\n }\n \ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e4716b0b867..5d819c1bc11 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1799,3 +1799,10 @@ test_lazy_prereq SHA1 '\n # Tests that verify the scheduler integration must set this locally\n # to avoid errors.\n GIT_TEST_MAINT_SCHEDULER=\"none:exit 1\"\n+\n+# Does this platform support `git fsmonitor--daemon`\n+#\n+test_lazy_prereq FSMONITOR_DAEMON '\n+\tgit version --build-options >output &&\n+\tgrep \"feature: fsmonitor--daemon\" output\n+'\n-- \ngitgitgadget\n\n"},{"id":"451879","messageId":"1f4b5209bf6092dc389db4046e0bdedc61cc4581.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 18/29] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T17:59:59Z","receivedAt":"2022-03-22T18:01:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to respond to IPC requests from client\nGit processes and respond with a list of modified pathnames\nrelative to the provided token.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 312 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 310 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 69312119b07..fc3aee0ada0 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,6 +7,7 @@\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n+#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon start []\"),\n@@ -364,6 +365,311 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+/*\n+ * Format an opaque token string to send to the client.\n+ */\n+static void with_lock__format_response_token(\n+\tstruct strbuf *response_token,\n+\tconst struct strbuf *response_token_id,\n+\tconst struct fsmonitor_batch *batch)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstrbuf_reset(response_token);\n+\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n+\t\t response_token_id->buf, batch->batch_seq_nr);\n+}\n+\n+/*\n+ * Parse an opaque token from the client.\n+ * Returns -1 on error.\n+ */\n+static int fsmonitor_parse_client_token(const char *buf_token,\n+\t\t\t\t\tstruct strbuf *requested_token_id,\n+\t\t\t\t\tuint64_t *seq_nr)\n+{\n+\tconst char *p;\n+\tchar *p_end;\n+\n+\tstrbuf_reset(requested_token_id);\n+\t*seq_nr = 0;\n+\n+\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n+\t\treturn -1;\n+\n+\twhile (*p && *p != ':')\n+\t\tstrbuf_addch(requested_token_id, *p++);\n+\tif (!*p++)\n+\t\treturn -1;\n+\n+\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n+\tif (*p_end)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal)\n+\n+static int do_handle_client(struct fsmonitor_daemon_state *state,\n+\t\t\t const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\tstruct fsmonitor_token_data *token_data = NULL;\n+\tstruct strbuf response_token = STRBUF_INIT;\n+\tstruct strbuf requested_token_id = STRBUF_INIT;\n+\tstruct strbuf payload = STRBUF_INIT;\n+\tuint64_t requested_oldest_seq_nr = 0;\n+\tuint64_t total_response_len = 0;\n+\tconst char *p;\n+\tconst struct fsmonitor_batch *batch_head;\n+\tconst struct fsmonitor_batch *batch;\n+\tintmax_t count = 0, duplicates = 0;\n+\tkh_str_t *shown;\n+\tint hash_ret;\n+\tint do_trivial = 0;\n+\tint do_flush = 0;\n+\n+\t/*\n+\t * We expect `command` to be of the form:\n+\t *\n+\t * := quit NUL\n+\t * | flush NUL\n+\t * | NUL\n+\t * | NUL\n+\t */\n+\n+\tif (!strcmp(command, \"quit\")) {\n+\t\t/*\n+\t\t * A client has requested over the socket/pipe that the\n+\t\t * daemon shutdown.\n+\t\t *\n+\t\t * Tell the IPC thread pool to shutdown (which completes\n+\t\t * the await in the main thread (which can stop the\n+\t\t * fsmonitor listener thread)).\n+\t\t *\n+\t\t * There is no reply to the client.\n+\t\t */\n+\t\treturn SIMPLE_IPC_QUIT;\n+\n+\t} else if (!strcmp(command, \"flush\")) {\n+\t\t/*\n+\t\t * Flush all of our cached data and generate a new token\n+\t\t * just like if we lost sync with the filesystem.\n+\t\t *\n+\t\t * Then send a trivial response using the new token.\n+\t\t */\n+\t\tdo_flush = 1;\n+\t\tdo_trivial = 1;\n+\n+\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n+\t\t/* assume V1 timestamp or garbage */\n+\n+\t\tchar *p_end;\n+\n+\t\tstrtoumax(command, &p_end, 10);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t ((*p_end) ?\n+\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n+\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n+\t\t\t\t command);\n+\t\tdo_trivial = 1;\n+\n+\t} else {\n+\t\t/* We have \"builtin:*\" */\n+\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n+\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n+\t\t\t\t\t command);\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We have a V2 valid token:\n+\t\t\t * \"builtin::\"\n+\t\t\t */\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (!state->current_token_data)\n+\t\tBUG(\"fsmonitor state does not have a current token\");\n+\n+\tif (do_flush)\n+\t\twith_lock__do_force_resync(state);\n+\n+\t/*\n+\t * We mark the current head of the batch list as \"pinned\" so\n+\t * that the listener thread will treat this item as read-only\n+\t * (and prevent any more paths from being added to it) from\n+\t * now on.\n+\t */\n+\ttoken_data = state->current_token_data;\n+\tbatch_head = token_data->batch_head;\n+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n+\n+\t/*\n+\t * FSMonitor Protocol V2 requires that we send a response header\n+\t * with a \"new current token\" and then all of the paths that changed\n+\t * since the \"requested token\". We send the seq_nr of the just-pinned\n+\t * head batch so that future requests from a client will be relative\n+\t * to it.\n+\t */\n+\twith_lock__format_response_token(&response_token,\n+\t\t\t\t\t &token_data->token_id, batch_head);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttotal_response_len += response_token.len + 1;\n+\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n+\t\t\t response_token.buf);\n+\n+\tif (!do_trivial) {\n+\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n+\t\t\t/*\n+\t\t\t * The client last spoke to a different daemon\n+\t\t\t * instance -OR- the daemon had to resync with\n+\t\t\t * the filesystem (and lost events), so reject.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n+\t\t\t\t\t \"response/token\", \"different\");\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else if (requested_oldest_seq_nr <\n+\t\t\t token_data->batch_tail->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * The client wants older events than we have for\n+\t\t\t * this token_id. This means that the end of our\n+\t\t\t * batch list was truncated and we cannot give the\n+\t\t\t * client a complete snapshot relative to their\n+\t\t\t * request.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"client requested truncated data\");\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n+\tif (do_trivial) {\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\treply(reply_data, \"/\", 2);\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n+\t\t\t\t \"response/trivial\", 1);\n+\n+\t\tstrbuf_release(&response_token);\n+\t\tstrbuf_release(&requested_token_id);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * We're going to hold onto a pointer to the current\n+\t * token-data while we walk the list of batches of files.\n+\t * During this time, we will NOT be under the lock.\n+\t * So we ref-count it.\n+\t *\n+\t * This allows the listener thread to continue prepending\n+\t * new batches of items to the token-data (which we'll ignore).\n+\t *\n+\t * AND it allows the listener thread to do a token-reset\n+\t * (and install a new `current_token_data`).\n+\t */\n+\ttoken_data->client_ref_count++;\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * The client request is relative to the token that they sent,\n+\t * so walk the batch list backwards from the current head back\n+\t * to the batch (sequence number) they named.\n+\t *\n+\t * We use khash to de-dup the list of pathnames.\n+\t *\n+\t * NEEDSWORK: each batch contains a list of interned strings,\n+\t * so we only need to do pointer comparisons here to build the\n+\t * hash table. Currently, we're still comparing the string\n+\t * values.\n+\t */\n+\tshown = kh_init_str();\n+\tfor (batch = batch_head;\n+\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n+\t batch = batch->next) {\n+\t\tsize_t k;\n+\n+\t\tfor (k = 0; k < batch->nr; k++) {\n+\t\t\tconst char *s = batch->interned_paths[k];\n+\t\t\tsize_t s_len;\n+\n+\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n+\t\t\t\tduplicates++;\n+\t\t\telse {\n+\t\t\t\tkh_put_str(shown, s, &hash_ret);\n+\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n+\t\t\t\t\t\t count, s);\n+\n+\t\t\t\t/* Each path gets written with a trailing NUL */\n+\t\t\t\ts_len = strlen(s) + 1;\n+\n+\t\t\t\tif (payload.len + s_len >=\n+\t\t\t\t LARGE_PACKET_DATA_MAX) {\n+\t\t\t\t\treply(reply_data, payload.buf,\n+\t\t\t\t\t payload.len);\n+\t\t\t\t\ttotal_response_len += payload.len;\n+\t\t\t\t\tstrbuf_reset(&payload);\n+\t\t\t\t}\n+\n+\t\t\t\tstrbuf_add(&payload, s, s_len);\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (payload.len) {\n+\t\treply(reply_data, payload.buf, payload.len);\n+\t\ttotal_response_len += payload.len;\n+\t}\n+\n+\tkh_release_str(shown);\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (token_data->client_ref_count > 0)\n+\t\ttoken_data->client_ref_count--;\n+\n+\tif (token_data->client_ref_count == 0) {\n+\t\tif (token_data != state->current_token_data) {\n+\t\t\t/*\n+\t\t\t * The listener thread did a token-reset while we were\n+\t\t\t * walking the batch list. Therefore, this token is\n+\t\t\t * stale and can be discarded completely. If we are\n+\t\t\t * the last reader thread using this token, we own\n+\t\t\t * that work.\n+\t\t\t */\n+\t\t\tfsmonitor_free_token_data(token_data);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n+\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\tstrbuf_release(&payload);\n+\n+\treturn 0;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -371,7 +677,7 @@ static int handle_client(void *data,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n-\t/* struct fsmonitor_daemon_state *state = data; */\n+\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n \t/*\n@@ -382,10 +688,12 @@ static int handle_client(void *data,\n \tif (command_len != strlen(command))\n \t\tBUG(\"FSMonitor assumes text messages\");\n \n+\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n+\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n \n-\tresult = 0; /* TODO Do something here. */\n+\tresult = do_handle_client(state, command, reply, reply_data);\n \n \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n \n-- \ngitgitgadget\n\n"},{"id":"451880","messageId":"5b3381c223e272ee939cba7b1be8617b6eb597f8.1647972011.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 23/29] t/helper/test-chmtime: skip directories on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:04Z","receivedAt":"2022-03-22T18:01:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach `test-tool.exe chmtime` to ignore errors when setting the mtime\non a directory on Windows.\n\nNEEDSWORK: The Windows version of `utime()` (aka `mingw_utime()`) does\nnot properly handle directories because it uses `_wopen()`. It should\nbe converted to using `CreateFileW()` and backup semantics at a minimum.\nSince I'm already in the middle of a large patch series, I did not want\nto destabilize other callers of `utime()` right now. The problem has\nonly been observed in the t/perf/p7519 test when the test repo contains\nan empty directory on disk.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/helper/test-chmtime.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/t/helper/test-chmtime.c b/t/helper/test-chmtime.c\nindex 524b55ca496..dc28890a183 100644\n--- a/t/helper/test-chmtime.c\n+++ b/t/helper/test-chmtime.c\n@@ -134,6 +134,21 @@ int cmd__chmtime(int argc, const char **argv)\n \t\t}\n \n \t\tif (utb.modtime != sb.st_mtime && utime(argv[i], &utb) < 0) {\n+#ifdef GIT_WINDOWS_NATIVE\n+\t\t\tif (S_ISDIR(sb.st_mode)) {\n+\t\t\t\t/*\n+\t\t\t\t * NEEDSWORK: The Windows version of `utime()`\n+\t\t\t\t * (aka `mingw_utime()`) does not correctly\n+\t\t\t\t * handle directory arguments, since it uses\n+\t\t\t\t * `_wopen()`. Ignore it for now since this\n+\t\t\t\t * is just a test.\n+\t\t\t\t */\n+\t\t\t\tfprintf(stderr,\n+\t\t\t\t\t(\"Failed to modify time on directory %s. \"\n+\t\t\t\t\t \"Skipping\\n\"), argv[i]);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+#endif\n \t\t\tfprintf(stderr, \"Failed to modify time on %s: %s\\n\",\n \t\t\t argv[i], strerror(errno));\n \t\t\treturn 1;\n-- \ngitgitgadget\n\n"},{"id":"451881","messageId":"08af8296f962498066bdb90c4203b84e52373fbc.1647972011.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 28/29] fsmonitor: force update index after large responses","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:09Z","receivedAt":"2022-03-22T18:02:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMeasure the time taken to apply the FSMonitor query result\nto the index and the untracked-cache.\n\nSet the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when\nFSMonitor returns a very large repsonse to ensure that the index is\nwritten to disk.\n\nNormally, when the FSMonitor response includes a tracked file, the\nindex is always updated. Similarly, the index might be updated when\nthe response alters the untracked-cache (when enabled). However, in\ncases where neither of those cause the index to be considered changed,\nthe FSMonitor response is wasted. Subsequent Git commands will make\nrequests with the same token and receive the same response.\n\nIf that response is very large, performance may suffer. It would be\nmore efficient to force update the index now (and the token in the\nindex extension) in order to reduce the size of the response received\nby future commands.\n\nThis was observed on Windows after a large checkout. On Windows, the\nkernel emits events for the files that are changed as they are\nchanged. However, it might delay events for the containing\ndirectories until the system is more idle (or someone scans the\ndirectory (so it seems)). The first status following a checkout would\nget the list of files. The subsequent status commands would get the\nlist of directories as the events trickled out. But they would never\ncatch up because the token was not advanced because the index wasn't\nupdated.\n\nThis list of directories caused `wt_status_collect_untracked()` to\nunnecessarily spend time actually scanning them during each command.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 54 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex a38b5710eb3..292a6742b4f 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -219,6 +219,43 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n+/*\n+ * The number of pathnames that we need to receive from FSMonitor\n+ * before we force the index to be updated.\n+ *\n+ * Note that any pathname within the set of received paths MAY cause\n+ * cache-entry or istate flag bits to be updated and thus cause the\n+ * index to be updated on disk.\n+ *\n+ * However, the response may contain many paths (such as ignored\n+ * paths) that will not update any flag bits. And thus not force the\n+ * index to be updated. (This is fine and normal.) It also means\n+ * that the token will not be updated in the FSMonitor index\n+ * extension. So the next Git command will find the same token in the\n+ * index, make the same token-relative request, and receive the same\n+ * response (plus any newly changed paths). If this response is large\n+ * (and continues to grow), performance could be impacted.\n+ *\n+ * For example, if the user runs a build and it writes 100K object\n+ * files but doesn't modify any source files, the index would not need\n+ * to be updated. The FSMonitor response (after the build and\n+ * relative to a pre-build token) might be 5MB. Each subsequent Git\n+ * command will receive that same 100K/5MB response until something\n+ * causes the index to be updated. And `refresh_fsmonitor()` will\n+ * have to iterate over those 100K paths each time.\n+ *\n+ * Performance could be improved if we optionally force update the\n+ * index after a very large response and get an updated token into\n+ * the FSMonitor index extension. This should allow subsequent\n+ * commands to get smaller and more current responses.\n+ *\n+ * The value chosen here does not need to be precise. The index\n+ * will be updated automatically the first time the user touches\n+ * a tracked file and causes a command like `git status` to\n+ * update an mtime to be updated and/or set a flag bit.\n+ */\n+static int fsmonitor_force_update_threshold = 100;\n+\n void refresh_fsmonitor(struct index_state *istate)\n {\n \tstruct strbuf query_result = STRBUF_INIT;\n@@ -362,25 +399,39 @@ apply_results:\n \t * information and that we should consider everything\n \t * invalid. We call this a trivial response.\n \t */\n+\ttrace2_region_enter(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tif (query_success && !is_trivial) {\n \t\t/*\n \t\t * Mark all pathnames returned by the monitor as dirty.\n \t\t *\n \t\t * This updates both the cache-entries and the untracked-cache.\n \t\t */\n+\t\tint count = 0;\n+\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n \t\t\t\tcontinue;\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n \t\t\tbol = i + 1;\n+\t\t\tcount++;\n \t\t}\n-\t\tif (bol < query_result.len)\n+\t\tif (bol < query_result.len) {\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tcount++;\n+\t\t}\n \n \t\t/* Now mark the untracked cache for fsmonitor usage */\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n+\n+\t\tif (count > fsmonitor_force_update_threshold)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", istate->repo, \"apply_count\",\n+\t\t\t\t count);\n+\n \t} else {\n \t\t/*\n \t\t * We failed to get a response or received a trivial response,\n@@ -409,6 +460,8 @@ apply_results:\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 0;\n \t}\n+\ttrace2_region_leave(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tstrbuf_release(&query_result);\n \n \t/* Now that we've updated istate, save the last_update_token */\n-- \ngitgitgadget\n\n"},{"id":"451882","messageId":"4a920d0b54a25a442bf52efc171139c698d59dc7.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 21/29] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:02Z","receivedAt":"2022-03-22T18:02:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/t7527-builtin-fsmonitor.sh | 501 +++++++++++++++++++++++++++++++++++\n 1 file changed, 501 insertions(+)\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..d79635e7596\n--- /dev/null\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -0,0 +1,501 @@\n+#!/bin/sh\n+\n+test_description='built-in file system watcher'\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+stop_daemon_delete_repo () {\n+\tr=$1 &&\n+\ttest_might_fail git -C $r fsmonitor--daemon stop &&\n+\trm -rf $1\n+}\n+\n+start_daemon () {\n+\tr= &&\n+\ttf= &&\n+\tt2= &&\n+\ttk= &&\n+\n+\twhile test \"$#\" -ne 0\n+\tdo\n+\t\tcase \"$1\" in\n+\t\t-C)\n+\t\t\tshift;\n+\t\t\ttest \"$#\" -ne 0 || BUG \"error: -C requires arg\"\n+\t\t\tr=\"-C $1\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t-tf)\n+\t\t\tshift;\n+\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tf requires arg\"\n+\t\t\ttf=\"$1\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t-t2)\n+\t\t\tshift;\n+\t\t\ttest \"$#\" -ne 0 || BUG \"error: -t2 requires arg\"\n+\t\t\tt2=\"$1\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t-tk)\n+\t\t\tshift;\n+\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tk requires arg\"\n+\t\t\ttk=\"$1\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t*)\n+\t\t\tBUG \"error: unknown option: '$1'\"\n+\t\t\t;;\n+\t\tesac\n+\tdone &&\n+\n+\t(\n+\t\tif test -n \"$tf\"\n+\t\tthen\n+\t\t\tGIT_TRACE_FSMONITOR=\"$tf\"\n+\t\t\texport GIT_TRACE_FSMONITOR\n+\t\tfi &&\n+\n+\t\tif test -n \"$t2\"\n+\t\tthen\n+\t\t\tGIT_TRACE2_PERF=\"$t2\"\n+\t\t\texport GIT_TRACE2_PERF\n+\t\tfi &&\n+\n+\t\tif test -n \"$tk\"\n+\t\tthen\n+\t\t\tGIT_TEST_FSMONITOR_TOKEN=\"$tk\"\n+\t\t\texport GIT_TEST_FSMONITOR_TOKEN\n+\t\tfi &&\n+\n+\t\tgit $r fsmonitor--daemon start &&\n+\t\tgit $r fsmonitor--daemon status\n+\t)\n+}\n+\n+# Is a Trace2 data event present with the given catetory and key?\n+# We do not care what the value is.\n+#\n+have_t2_data_event () {\n+\tc=$1 &&\n+\tk=$2 &&\n+\n+\tgrep -e '\"event\":\"data\".*\"category\":\"'\"$c\"'\".*\"key\":\"'\"$k\"'\"'\n+}\n+\n+test_expect_success 'explicit daemon start and stop' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n+\n+\tgit init test_explicit &&\n+\tstart_daemon -C test_explicit &&\n+\n+\tgit -C test_explicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon start' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit\" &&\n+\n+\tgit init test_implicit &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n+\n+\t# query will implicitly start the daemon.\n+\t#\n+\t# for test-script simplicity, we send a V1 timestamp rather than\n+\t# a V2 token. either way, the daemon response to any query contains\n+\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n+\t# but this test case is only concerned with whether the daemon was\n+\t# implicitly started.)\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n+\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n+\tnul_to_q actual.filtered &&\n+\tgrep \"builtin:\" actual.filtered &&\n+\n+\t# confirm that a daemon was started in the background.\n+\t#\n+\t# since the mechanism for starting the background daemon is platform\n+\t# dependent, just confirm that the foreground command received a\n+\t# response from the daemon.\n+\n+\thave_t2_data_event fsm_client query/response-length <.git/trace &&\n+\n+\tgit -C test_implicit fsmonitor--daemon status &&\n+\tgit -C test_implicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (delete .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n+\n+\tgit init test_implicit_1 &&\n+\n+\tstart_daemon -C test_implicit_1 &&\n+\n+\t# deleting the .git directory will implicitly stop the daemon.\n+\trm -rf test_implicit_1/.git &&\n+\n+\t# [1] Create an empty .git directory so that the following Git\n+\t# command will stay relative to the `-C` directory.\n+\t#\n+\t# Without this, the Git command will override the requested\n+\t# -C argument and crawl out to the containing Git source tree.\n+\t# This would make the test result dependent upon whether we\n+\t# were using fsmonitor on our development worktree.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_1/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (rename .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_2\" &&\n+\n+\tgit init test_implicit_2 &&\n+\n+\tstart_daemon -C test_implicit_2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n+\n+\t# See [1] above.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'cannot start multiple daemons' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n+\n+\tgit init test_multiple &&\n+\n+\tstart_daemon -C test_multiple &&\n+\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n+\tgrep \"fsmonitor--daemon is already running\" actual &&\n+\n+\tgit -C test_multiple fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n+'\n+\n+# These tests use the main repo in the trash directory\n+\n+test_expect_success 'setup' '\n+\t>tracked &&\n+\t>modified &&\n+\t>delete &&\n+\t>rename &&\n+\tmkdir dir1 &&\n+\t>dir1/tracked &&\n+\t>dir1/modified &&\n+\t>dir1/delete &&\n+\t>dir1/rename &&\n+\tmkdir dir2 &&\n+\t>dir2/tracked &&\n+\t>dir2/modified &&\n+\t>dir2/delete &&\n+\t>dir2/rename &&\n+\tmkdir dirtorename &&\n+\t>dirtorename/a &&\n+\t>dirtorename/b &&\n+\n+\tcat >.gitignore <<-\\EOF &&\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tEOF\n+\n+\tgit -c core.fsmonitor=false add . &&\n+\ttest_tick &&\n+\tgit -c core.fsmonitor=false commit -m initial &&\n+\n+\tgit config core.fsmonitor true\n+'\n+\n+# The test already explicitly stopped (or tried to stop) the daemon.\n+# This is here in case something else fails first.\n+#\n+redundant_stop_daemon () {\n+\ttest_might_fail git fsmonitor--daemon stop\n+}\n+\n+test_expect_success 'update-index implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n+\t\tgit update-index --fsmonitor &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_1\n+'\n+\n+test_expect_success 'status implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n+\t\tgit status >actual &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_2\n+'\n+\n+edit_files () {\n+\techo 1 >modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\t>dir1/untracked\n+}\n+\n+delete_files () {\n+\trm -f delete &&\n+\trm -f dir1/delete &&\n+\trm -f dir2/delete\n+}\n+\n+create_files () {\n+\techo 1 >new &&\n+\techo 2 >dir1/new &&\n+\techo 3 >dir2/new\n+}\n+\n+rename_files () {\n+\tmv rename renamed &&\n+\tmv dir1/rename dir1/renamed &&\n+\tmv dir2/rename dir2/renamed\n+}\n+\n+file_to_directory () {\n+\trm -f delete &&\n+\tmkdir delete &&\n+\techo 1 >delete/new\n+}\n+\n+directory_to_file () {\n+\trm -rf dir1 &&\n+\techo 1 >dir1\n+}\n+\n+# The next few test cases confirm that our fsmonitor daemon sees each type\n+# of OS filesystem notification that we care about. At this layer we just\n+# ensure we are getting the OS notifications and do not try to confirm what\n+# is reported by `git status`.\n+#\n+# We run a simple query after modifying the filesystem just to introduce\n+# a bit of a delay so that the trace logging from the daemon has time to\n+# get flushed to disk.\n+#\n+# We `reset` and `clean` at the bottom of each test (and before stopping the\n+# daemon) because these commands might implicitly restart the daemon.\n+\n+clean_up_repo_and_stop_daemon () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\trm -f .git/trace\n+}\n+\n+test_expect_success 'edit some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n+\n+\tedit_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/modified$\" .git/trace &&\n+\tgrep \"^event: dir2/modified$\" .git/trace &&\n+\tgrep \"^event: modified$\" .git/trace &&\n+\tgrep \"^event: dir1/untracked$\" .git/trace\n+'\n+\n+test_expect_success 'create some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n+\n+\tcreate_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/new$\" .git/trace &&\n+\tgrep \"^event: dir2/new$\" .git/trace &&\n+\tgrep \"^event: new$\" .git/trace\n+'\n+\n+test_expect_success 'delete some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n+\n+\tdelete_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/delete$\" .git/trace &&\n+\tgrep \"^event: dir2/delete$\" .git/trace &&\n+\tgrep \"^event: delete$\" .git/trace\n+'\n+\n+test_expect_success 'rename some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n+\n+\trename_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/rename$\" .git/trace &&\n+\tgrep \"^event: dir2/rename$\" .git/trace &&\n+\tgrep \"^event: rename$\" .git/trace &&\n+\tgrep \"^event: dir1/renamed$\" .git/trace &&\n+\tgrep \"^event: dir2/renamed$\" .git/trace &&\n+\tgrep \"^event: renamed$\" .git/trace\n+'\n+\n+test_expect_success 'rename directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n+\n+\tmv dirtorename dirrenamed &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dirtorename/*$\" .git/trace &&\n+\tgrep \"^event: dirrenamed/*$\" .git/trace\n+'\n+\n+test_expect_success 'file changes to directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n+\n+\tfile_to_directory &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: delete$\" .git/trace &&\n+\tgrep \"^event: delete/new$\" .git/trace\n+'\n+\n+test_expect_success 'directory changes to a file' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n+\n+\tdirectory_to_file &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1$\" .git/trace\n+'\n+\n+# The next few test cases exercise the token-resync code. When filesystem\n+# drops events (because of filesystem velocity or because the daemon isn't\n+# polling fast enough), we need to discard the cached data (relative to the\n+# current token) and start collecting events under a new token.\n+#\n+# the 'test-tool fsmonitor-client flush' command can be used to send a\n+# \"flush\" message to a running daemon and ask it to do a flush/resync.\n+\n+test_expect_success 'flush cached data' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_flush\" &&\n+\n+\tgit init test_flush &&\n+\n+\tstart_daemon -C test_flush -tf \"$PWD/.git/trace_daemon\" -tk true &&\n+\n+\t# The daemon should have an initial token with no events in _0 and\n+\t# then a few (probably platform-specific number of) events in _1.\n+\t# These should both have the same .\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n+\tnul_to_q actual_q0 &&\n+\n+\t>test_flush/file_1 &&\n+\t>test_flush/file_2 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n+\tnul_to_q actual_q1 &&\n+\n+\tgrep \"file_1\" actual_q1 &&\n+\n+\t# Force a flush. This will change the , reset the , and\n+\t# flush the file data. Then create some events and ensure that the file\n+\t# again appears in the cache. It should have the new .\n+\n+\ttest-tool -C test_flush fsmonitor-client flush >flush_0 &&\n+\tnul_to_q flush_q0 &&\n+\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_2 &&\n+\tnul_to_q actual_q2 &&\n+\n+\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n+\n+\t>test_flush/file_3 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n+\tnul_to_q actual_q3 &&\n+\n+\tgrep \"file_3\" actual_q3\n+'\n+\n+# The next few test cases create repos where the .git directory is NOT\n+# inside the one of the working directory. That is, where .git is a file\n+# that points to a directory elsewhere. This happens for submodules and\n+# non-primary worktrees.\n+\n+test_expect_success 'setup worktree base' '\n+\tgit init wt-base &&\n+\techo 1 >wt-base/file1 &&\n+\tgit -C wt-base add file1 &&\n+\tgit -C wt-base commit -m \"c1\"\n+'\n+\n+test_expect_success 'worktree with .git file' '\n+\tgit -C wt-base worktree add ../wt-secondary &&\n+\n+\tstart_daemon -C wt-secondary \\\n+\t\t-tf \"$PWD/trace_wt_secondary\" \\\n+\t\t-t2 \"$PWD/trace2_wt_secondary\" &&\n+\n+\tgit -C wt-secondary fsmonitor--daemon stop &&\n+\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n+'\n+\n+# NEEDSWORK: Repeat one of the \"edit\" tests on wt-secondary and\n+# confirm that we get the same events and behavior -- that is, that\n+# fsmonitor--daemon correctly watches BOTH the working directory and\n+# the external GITDIR directory and behaves the same as when \".git\"\n+# is a directory inside the working directory.\n+\n+test_expect_success 'cleanup worktrees' '\n+\tstop_daemon_delete_repo wt-secondary &&\n+\tstop_daemon_delete_repo wt-base\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"451883","messageId":"1bd74a815937f6db8fae42afc0d0b780053087b8.1647972010.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 20/29] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:01Z","receivedAt":"2022-03-22T18:02:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate an IPC client to send query and flush commands to the daemon.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n t/helper/test-fsmonitor-client.c | 116 +++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 119 insertions(+)\n create mode 100644 t/helper/test-fsmonitor-client.c\n\ndiff --git a/Makefile b/Makefile\nindex 26567d4f772..daa21bed6c3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -716,6 +716,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\n TEST_BUILTINS_OBJS += test-getcwd.o\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nnew file mode 100644\nindex 00000000000..3062c8a3c2b\n--- /dev/null\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -0,0 +1,116 @@\n+/*\n+ * test-fsmonitor-client.c: client code to send commands/requests to\n+ * a `git fsmonitor--daemon` daemon.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tdie(\"fsmonitor--daemon not available on this platform\");\n+}\n+#else\n+\n+/*\n+ * Read the `.git/index` to get the last token written to the\n+ * FSMonitor Index Extension.\n+ */\n+static const char *get_token_from_index(void)\n+{\n+\tstruct index_state *istate = the_repository->index;\n+\n+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update)\n+\t\tdie(\"index file does not have fsmonitor extension\");\n+\n+\treturn istate->fsmonitor_last_update;\n+}\n+\n+/*\n+ * Send an IPC query to a `git-fsmonitor--daemon` daemon and\n+ * ask for the changes since the given token or from the last\n+ * token in the index extension.\n+ *\n+ * This will implicitly start a daemon process if necessary. The\n+ * daemon process will persist after we exit.\n+ */\n+static int do_send_query(const char *token)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\n+\tret = fsmonitor_ipc__send_query(token, &answer);\n+\tif (ret < 0)\n+\t\tdie(\"could not query fsmonitor--daemon\");\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n+ * and tell it to flush its cache.\n+ *\n+ * This feature is primarily used by the test suite to simulate a loss of\n+ * sync with the filesystem where we miss kernel events.\n+ */\n+static int do_send_flush(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tconst char *subcmd;\n+\tconst char *token = NULL;\n+\n+\tconst char * const fsmonitor_client_usage[] = {\n+\t\t\"test-tool fsmonitor-client query []\",\n+\t\t\"test-tool fsmonitor-client flush\",\n+\t\tNULL,\n+\t};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING(0, \"token\", &token, \"token\",\n+\t\t\t \"command token to send to the server\"),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n+\n+\tif (argc != 1)\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tsubcmd = argv[0];\n+\n+\tsetup_git_directory();\n+\n+\tif (!strcmp(subcmd, \"query\"))\n+\t\treturn !!do_send_query(token);\n+\n+\tif (!strcmp(subcmd, \"flush\"))\n+\t\treturn !!do_send_flush();\n+\n+\tdie(\"Unhandled subcommand: '%s'\", subcmd);\n+}\n+#endif\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex e6ec69cf326..0424f7adf5d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -32,6 +32,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\n \t{ \"getcwd\", cmd__getcwd },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 20756eefdda..c876e8246fb 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"451884","messageId":"c925a9a7459a0642180282b41b210defe934e38f.1647972011.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 22/29] t/perf: avoid copying builtin fsmonitor files into test repo","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:03Z","receivedAt":"2022-03-22T18:02:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDo not copy any of the various fsmonitor--daemon files from the .git\ndirectory of the (GIT_PREF_REPO or GIT_PERF_LARGE_REPO) source repo\ninto the test's trash directory.\n\nWhen perf tests start, they copy the contents of the source repo into\nthe test's trash directory. If fsmonitor is running in the source repo,\nthere may be control files, such as the IPC socket and/or fsmonitor\ncookie files. These should not be copied into the test repo.\n\nUnix domain sockets cannot be copied in the manner used by the test\nsetup, so if present, the test setup fails.\n\nCookie files are harmless, but we should avoid them.\n\nThe builtin fsmonitor keeps all such control files/sockets in\n.git/fsmonitor--daemon*, so it is simple to exclude them.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/perf-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 407252bac70..932105cd12c 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -78,7 +78,7 @@ test_perf_copy_repo_contents () {\n \tfor stuff in \"$1\"/*\n \tdo\n \t\tcase \"$stuff\" in\n-\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees)\n+\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)\n \t\t\t;;\n \t\t*)\n \t\t\tcp -R \"$stuff\" \"$repo/.git/\" || exit 1\n-- \ngitgitgadget\n\n"},{"id":"451885","messageId":"d5ca2df31c835ab802b6ebfb26685449bfb0e9f2.1647972011.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 25/29] t/perf/p7519: add fsmonitor--daemon test cases","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:06Z","receivedAt":"2022-03-22T18:02:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRepeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\nthe \"Simple IPC\" interface.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 38 ++++++++++++++++++++++++++++++++++----\n 1 file changed, 34 insertions(+), 4 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 0611e533951..3b232e386b9 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -141,7 +141,7 @@ test_expect_success \"one time repo setup\" '\n \tfi\n '\n \n-setup_for_fsmonitor () {\n+setup_for_fsmonitor_hook () {\n \t# set INTEGRATION_SCRIPT depending on the environment\n \tif test -n \"$INTEGRATION_PATH\"\n \tthen\n@@ -182,7 +182,11 @@ test_perf_w_drop_caches () {\n }\n \n test_fsmonitor_suite () {\n-\tif test -n \"$INTEGRATION_SCRIPT\"; then\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n+\tthen\n+\t\tDESC=\"builtin fsmonitor--daemon\"\n+\telif test -n \"$INTEGRATION_SCRIPT\"\n+\tthen\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n \t\tDESC=\"fsmonitor=disabled\"\n@@ -261,11 +265,11 @@ test_fsmonitor_suite () {\n trace_start fsmonitor-watchman\n if test -n \"$GIT_PERF_7519_FSMONITOR\"; then\n \tfor INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do\n-\t\ttest_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor'\n+\t\ttest_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor_hook'\n \t\ttest_fsmonitor_suite\n \tdone\n else\n-\ttest_expect_success \"setup for fsmonitor\" 'setup_for_fsmonitor'\n+\ttest_expect_success \"setup for fsmonitor hook\" 'setup_for_fsmonitor_hook'\n \ttest_fsmonitor_suite\n fi\n \n@@ -293,4 +297,30 @@ test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n \n+#\n+# Run a full set of perf tests using the built-in fsmonitor--daemon.\n+# It does not use the Hook API, so it has a different setup.\n+# Explicitly start the daemon here and before we start client commands\n+# so that we can later add custom tracing.\n+#\n+if test_have_prereq FSMONITOR_DAEMON\n+then\n+\tUSE_FSMONITOR_DAEMON=t\n+\n+\ttest_expect_success \"setup for builtin fsmonitor\" '\n+\t\ttrace_start fsmonitor--daemon--server &&\n+\t\tgit fsmonitor--daemon start &&\n+\n+\t\ttrace_start fsmonitor--daemon--client &&\n+\n+\t\tgit config core.fsmonitor true &&\n+\t\tgit update-index --fsmonitor\n+\t'\n+\n+\ttest_fsmonitor_suite\n+\n+\tgit fsmonitor--daemon stop\n+\ttrace_stop\n+fi\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"451886","messageId":"42631259e89e0f5306befe17da519b2c34cdd207.1647972011.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 26/29] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:07Z","receivedAt":"2022-03-22T18:02:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to periodically truncate the list of\nmodified files to save some memory.\n\nClients will ask for the set of changes relative to a token that they\nfound in the FSMN index extension in the index. (This token is like a\npoint in time, but different). Clients will then update the index to\ncontain the response token (so that subsequent commands will be\nrelative to this new token).\n\nTherefore, the daemon can gradually truncate the in-memory list of\nchanged paths as they become obsolete (older than the previous token).\nSince we may have multiple clients making concurrent requests with a\nskew of tokens and clients may be racing to the talk to the daemon,\nwe lazily truncate the list.\n\nWe introduce a 5 minute delay and truncate batches 5 minutes after\nthey are considered obsolete.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 88 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 88 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex fc3aee0ada0..edd00379cca 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -312,6 +312,75 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n \t\t\tbatch_src->interned_paths[k];\n }\n \n+/*\n+ * To keep the batch list from growing unbounded in response to filesystem\n+ * activity, we try to truncate old batches from the end of the list as\n+ * they become irrelevant.\n+ *\n+ * We assume that the .git/index will be updated with the most recent token\n+ * any time the index is updated. And future commands will only ask for\n+ * recent changes *since* that new token. So as tokens advance into the\n+ * future, older batch items will never be requested/needed. So we can\n+ * truncate them without loss of functionality.\n+ *\n+ * However, multiple commands may be talking to the daemon concurrently\n+ * or perform a slow command, so a little \"token skew\" is possible.\n+ * Therefore, we want this to be a little bit lazy and have a generous\n+ * delay.\n+ *\n+ * The current reader thread walked backwards in time from `token->batch_head`\n+ * back to `batch_marker` somewhere in the middle of the batch list.\n+ *\n+ * Let's walk backwards in time from that marker an arbitrary delay\n+ * and truncate the list there. Note that these timestamps are completely\n+ * artificial (based on when we pinned the batch item) and not on any\n+ * filesystem activity.\n+ *\n+ * Return the obsolete portion of the list after we have removed it from\n+ * the official list so that the caller can free it after leaving the lock.\n+ */\n+#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */\n+\n+static struct fsmonitor_batch *with_lock__truncate_old_batches(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst struct fsmonitor_batch *batch_marker)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder;\n+\n+\tif (!batch_marker)\n+\t\treturn NULL;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Truncate: mark (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t batch_marker->batch_seq_nr,\n+\t\t\t (uint64_t)batch_marker->pinned_time);\n+\n+\tfor (batch = batch_marker; batch; batch = batch->next) {\n+\t\ttime_t t;\n+\n+\t\tif (!batch->pinned_time) /* an overflow batch */\n+\t\t\tcontinue;\n+\n+\t\tt = batch->pinned_time + MY_TIME_DELAY_SECONDS;\n+\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n+\t\t\tcontinue;\n+\n+\t\tgoto truncate_past_here;\n+\t}\n+\n+\treturn NULL;\n+\n+truncate_past_here:\n+\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n+\n+\tremainder = ((struct fsmonitor_batch *)batch)->next;\n+\t((struct fsmonitor_batch *)batch)->next = NULL;\n+\n+\treturn remainder;\n+}\n+\n static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n {\n \tif (!token)\n@@ -425,6 +494,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tconst char *p;\n \tconst struct fsmonitor_batch *batch_head;\n \tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder = NULL;\n \tintmax_t count = 0, duplicates = 0;\n \tkh_str_t *shown;\n \tint hash_ret;\n@@ -654,11 +724,29 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * that work.\n \t\t\t */\n \t\t\tfsmonitor_free_token_data(token_data);\n+\t\t} else if (batch) {\n+\t\t\t/*\n+\t\t\t * We are holding the lock and are the only\n+\t\t\t * reader of the ref-counted portion of the\n+\t\t\t * list, so we get the honor of seeing if the\n+\t\t\t * list can be truncated to save memory.\n+\t\t\t *\n+\t\t\t * The main loop did not walk to the end of the\n+\t\t\t * list, so this batch is the first item in the\n+\t\t\t * batch-list that is older than the requested\n+\t\t\t * end-point sequence number. See if the tail\n+\t\t\t * end of the list is obsolete.\n+\t\t\t */\n+\t\t\tremainder = with_lock__truncate_old_batches(state,\n+\t\t\t\t\t\t\t\t batch);\n \t\t}\n \t}\n \n \tpthread_mutex_unlock(&state->main_lock);\n \n+\tif (remainder)\n+\t\tfsmonitor_batch__free_list(remainder);\n+\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n-- \ngitgitgadget\n\n"},{"id":"451887","messageId":"f256c3cbe8b862d8403cc3ac16d17b5bb986b302.1647972011.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 27/29] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:08Z","receivedAt":"2022-03-22T18:02:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon client threads to create a cookie file\ninside the .git directory and then wait until FS events for the\ncookie are observed by the FS listener thread.\n\nThis helps address the racy nature of file system events by\nblocking the client response until the kernel has drained any\nevent backlog.\n\nThis is especially important on MacOS where kernel events are\nonly issued with a limited frequency. See the `latency` argument\nof `FSeventStreamCreate()`. The kernel only signals every `latency`\nseconds, but does not guarantee that the kernel queue is completely\ndrained, so we may have to wait more than one interval. If we\nincrease the latency, the system is more likely to drop events.\nWe avoid these issues by having each client thread create a unique\ncookie file and then wait until it is seen in the event stream.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 237 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 5 +\n 2 files changed, 241 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex edd00379cca..a30ecf6cfac 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -107,6 +107,162 @@ static int do_as_client__status(void)\n \t}\n }\n \n+enum fsmonitor_cookie_item_result {\n+\tFCIR_ERROR = -1, /* could not create cookie file ? */\n+\tFCIR_INIT,\n+\tFCIR_SEEN,\n+\tFCIR_ABORT,\n+};\n+\n+struct fsmonitor_cookie_item {\n+\tstruct hashmap_entry entry;\n+\tchar *name;\n+\tenum fsmonitor_cookie_item_result result;\n+};\n+\n+static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n+\t\t const struct hashmap_entry *he2, const void *keydata)\n+{\n+\tconst struct fsmonitor_cookie_item *a =\n+\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n+\tconst struct fsmonitor_cookie_item *b =\n+\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n+\n+\treturn strcmp(a->name, keydata ? keydata : b->name);\n+}\n+\n+static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(\n+\tstruct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint fd;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tstruct strbuf cookie_pathname = STRBUF_INIT;\n+\tstruct strbuf cookie_filename = STRBUF_INIT;\n+\tenum fsmonitor_cookie_item_result result;\n+\tint my_cookie_seq;\n+\n+\tCALLOC_ARRAY(cookie, 1);\n+\n+\tmy_cookie_seq = state->cookie_seq++;\n+\n+\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n+\n+\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n+\tstrbuf_addbuf(&cookie_pathname, &cookie_filename);\n+\n+\tcookie->name = strbuf_detach(&cookie_filename, NULL);\n+\tcookie->result = FCIR_INIT;\n+\thashmap_entry_init(&cookie->entry, strhash(cookie->name));\n+\n+\thashmap_add(&state->cookies, &cookie->entry);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n+\t\t\t cookie->name, cookie_pathname.buf);\n+\n+\t/*\n+\t * Create the cookie file on disk and then wait for a notification\n+\t * that the listener thread has seen it.\n+\t */\n+\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n+\tif (fd < 0) {\n+\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n+\t\t\t cookie->name);\n+\n+\t\tcookie->result = FCIR_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * Technically, close() and unlink() can fail, but we don't\n+\t * care here. We only created the file to trigger a watch\n+\t * event from the FS to know that when we're up to date.\n+\t */\n+\tclose(fd);\n+\tunlink(cookie_pathname.buf);\n+\n+\t/*\n+\t * Technically, this is an infinite wait (well, unless another\n+\t * thread sends us an abort). I'd like to change this to\n+\t * use `pthread_cond_timedwait()` and return an error/timeout\n+\t * and let the caller do the trivial response thing, but we\n+\t * don't have that routine in our thread-utils.\n+\t *\n+\t * After extensive beta testing I'm not really worried about\n+\t * this. Also note that the above open() and unlink() calls\n+\t * will cause at least two FS events on that path, so the odds\n+\t * of getting stuck are pretty slim.\n+\t */\n+\twhile (cookie->result == FCIR_INIT)\n+\t\tpthread_cond_wait(&state->cookies_cond,\n+\t\t\t\t &state->main_lock);\n+\n+done:\n+\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n+\n+\tresult = cookie->result;\n+\n+\tfree(cookie->name);\n+\tfree(cookie);\n+\tstrbuf_release(&cookie_pathname);\n+\n+\treturn result;\n+}\n+\n+/*\n+ * Mark these cookies as _SEEN and wake up the corresponding client threads.\n+ */\n+static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,\n+\t\t\t\t\t const struct string_list *cookie_names)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint k;\n+\tint nr_seen = 0;\n+\n+\tfor (k = 0; k < cookie_names->nr; k++) {\n+\t\tstruct fsmonitor_cookie_item key;\n+\t\tstruct fsmonitor_cookie_item *cookie;\n+\n+\t\tkey.name = cookie_names->items[k].string;\n+\t\thashmap_entry_init(&key.entry, strhash(key.name));\n+\n+\t\tcookie = hashmap_get_entry(&state->cookies, &key, entry, NULL);\n+\t\tif (cookie) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-seen: '%s'\",\n+\t\t\t\t\t cookie->name);\n+\t\t\tcookie->result = FCIR_SEEN;\n+\t\t\tnr_seen++;\n+\t\t}\n+\t}\n+\n+\tif (nr_seen)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n+/*\n+ * Set _ABORT on all pending cookies and wake up all client threads.\n+ */\n+static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct hashmap_iter iter;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tint nr_aborted = 0;\n+\n+\thashmap_for_each_entry(&state->cookies, &iter, cookie, entry) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-abort: '%s'\",\n+\t\t\t\t cookie->name);\n+\t\tcookie->result = FCIR_ABORT;\n+\t\tnr_aborted++;\n+\t}\n+\n+\tif (nr_aborted)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n /*\n * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n * \"token\" as a virtual timestamp. Clients can request a summary of all\n@@ -404,6 +560,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n *\n+ * [2] Some of those lost events may have been for cookie files. We\n+ * should assume the worst and abort them rather letting them starve.\n+ *\n * If there are no concurrent threads reading the current token data\n * series, we can free it now. Otherwise, let the last reader free\n * it.\n@@ -425,6 +584,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n \tstate->current_token_data = new_one;\n \n \tfsmonitor_free_token_data(free_me);\n+\n+\twith_lock__abort_all_cookies(state);\n }\n \n void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n@@ -500,6 +661,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tint hash_ret;\n \tint do_trivial = 0;\n \tint do_flush = 0;\n+\tint do_cookie = 0;\n+\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n@@ -560,6 +723,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * We have a V2 valid token:\n \t\t\t * \"builtin::\"\n \t\t\t */\n+\t\t\tdo_cookie = 1;\n \t\t}\n \t}\n \n@@ -568,6 +732,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tif (!state->current_token_data)\n \t\tBUG(\"fsmonitor state does not have a current token\");\n \n+\t/*\n+\t * Write a cookie file inside the directory being watched in\n+\t * an effort to flush out existing filesystem events that we\n+\t * actually care about. Suspend this client thread until we\n+\t * see the filesystem events for this cookie file.\n+\t *\n+\t * Creating the cookie lets us guarantee that our FS listener\n+\t * thread has drained the kernel queue and we are caught up\n+\t * with the kernel.\n+\t *\n+\t * If we cannot create the cookie (or otherwise guarantee that\n+\t * we are caught up), we send a trivial response. We have to\n+\t * assume that there might be some very, very recent activity\n+\t * on the FS still in flight.\n+\t */\n+\tif (do_cookie) {\n+\t\tcookie_result = with_lock__wait_for_cookie(state);\n+\t\tif (cookie_result != FCIR_SEEN) {\n+\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n+\t\t\t cookie_result);\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n \tif (do_flush)\n \t\twith_lock__do_force_resync(state);\n \n@@ -788,7 +976,9 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n-#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+#define FSMONITOR_DIR \"fsmonitor--daemon\"\n+#define FSMONITOR_COOKIE_DIR \"cookies\"\n+#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n \n enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n \tconst char *rel)\n@@ -941,6 +1131,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n+\tif (cookie_names->nr)\n+\t\twith_lock__mark_cookies_seen(state, cookie_names);\n+\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n@@ -1032,7 +1225,9 @@ static int fsmonitor_run_daemon(void)\n \n \tmemset(&state, 0, sizeof(state));\n \n+\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n+\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n@@ -1057,6 +1252,44 @@ static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n+\t/*\n+\t * We will write filesystem syncing cookie files into\n+\t * ///-.\n+\t *\n+\t * The extra layers of subdirectories here keep us from\n+\t * changing the mtime on \".git/\" or \".git/foo/\" when we create\n+\t * or delete cookie files.\n+\t *\n+\t * There have been problems with some IDEs that do a\n+\t * non-recursive watch of the \".git/\" directory and run a\n+\t * series of commands any time something happens.\n+\t *\n+\t * For example, if we place our cookie files directly in\n+\t * \".git/\" or \".git/foo/\" then a `git status` (or similar\n+\t * command) from the IDE will cause a cookie file to be\n+\t * created in one of those dirs. This causes the mtime of\n+\t * those dirs to change. This triggers the IDE's watch\n+\t * notification. This triggers the IDE to run those commands\n+\t * again. And the process repeats and the machine never goes\n+\t * idle.\n+\t *\n+\t * Adding the extra layers of subdirectories prevents the\n+\t * mtime of \".git/\" and \".git/foo\" from changing when a\n+\t * cookie file is created.\n+\t */\n+\tstrbuf_init(&state.path_cookie_prefix, 0);\n+\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1069,6 +1302,7 @@ static int fsmonitor_run_daemon(void)\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n+\tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n \n@@ -1076,6 +1310,7 @@ done:\n \n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n+\tstrbuf_release(&state.path_cookie_prefix);\n \n \treturn err;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 010fbfe60e9..bd09fffc176 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n \n \tstruct fsmonitor_token_data *current_token_data;\n \n+\tstruct strbuf path_cookie_prefix;\n+\tpthread_cond_t cookies_cond;\n+\tint cookie_seq;\n+\tstruct hashmap cookies;\n+\n \tint error_code;\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n-- \ngitgitgadget\n\n"},{"id":"451888","messageId":"e6cf84dc8eb4933220187849b84e5cb96cda185f.1647972011.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v7 29/29] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:00:10Z","receivedAt":"2022-03-22T18:02:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate 2x2 test matrix with the untracked-cache and fsmonitor--daemon\nfeatures and a series of edits and verify that status output is\nidentical.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 93 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex d79635e7596..389ebf431c6 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -212,6 +212,8 @@ test_expect_success 'setup' '\n \t.gitignore\n \texpect*\n \tactual*\n+\tflush*\n+\ttrace*\n \tEOF\n \n \tgit -c core.fsmonitor=false add . &&\n@@ -498,4 +500,95 @@ test_expect_success 'cleanup worktrees' '\n \tstop_daemon_delete_repo wt-base\n '\n \n+# The next few tests perform arbitrary/contrived file operations and\n+# confirm that status is correct. That is, that the data (or lack of\n+# data) from fsmonitor doesn't cause incorrect results. And doesn't\n+# cause incorrect results when the untracked-cache is enabled.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\tgit update-index --test-untracked-cache\n+'\n+\n+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n+\ttest_unconfig core.fsmonitor &&\n+\tgit update-index --no-fsmonitor &&\n+\ttest_might_fail git fsmonitor--daemon stop\n+'\n+\n+matrix_clean_up_repo () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd\n+}\n+\n+matrix_try () {\n+\tuc=$1 &&\n+\tfsm=$2 &&\n+\tfn=$3 &&\n+\n+\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n+\t\tmatrix_clean_up_repo &&\n+\t\t$fn &&\n+\t\tif test $uc = false && test $fsm = false\n+\t\tthen\n+\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n+\t\telse\n+\t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n+\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n+\t\tfi\n+\t'\n+}\n+\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+for uc_val in $uc_values\n+do\n+\tif test $uc_val = false\n+\tthen\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n+\t\t\tgit config core.untrackedcache false &&\n+\t\t\tgit update-index --no-untracked-cache\n+\t\t'\n+\telse\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n+\t\t\tgit config core.untrackedcache true &&\n+\t\t\tgit update-index --untracked-cache\n+\t\t'\n+\tfi\n+\n+\tfsm_values=\"false true\"\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tif test $fsm_val = false\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\telse\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n+\t\t\t\tgit config core.fsmonitor true &&\n+\t\t\t\tgit fsmonitor--daemon start &&\n+\t\t\t\tgit update-index --fsmonitor\n+\t\t\t'\n+\t\tfi\n+\n+\t\tmatrix_try $uc_val $fsm_val edit_files\n+\t\tmatrix_try $uc_val $fsm_val delete_files\n+\t\tmatrix_try $uc_val $fsm_val create_files\n+\t\tmatrix_try $uc_val $fsm_val rename_files\n+\t\tmatrix_try $uc_val $fsm_val file_to_directory\n+\t\tmatrix_try $uc_val $fsm_val directory_to_file\n+\n+\t\tif test $fsm_val = true\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\tfi\n+\tdone\n+done\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"451918","messageId":"220322.86ils5q1um.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"14b775e9d8b1a4672f8175a546eb70e2790c1b23.1647972010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 16/29] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-22T18:19:53Z","receivedAt":"2022-03-22T18:27:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 22 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n> \n> compat/fsmonitor/fsm-darwin-gcc.h | 92 ++++++++++++++++++++++++++++\n\nIt's much nicer to have this compat-for-the-compat in its own\nheader. Thanks.\n\n> compat/fsmonitor/fsm-listen-darwin.c | 24 ++++++++\n> 2 files changed, 116 insertions(+)\n> create mode 100644 compat/fsmonitor/fsm-darwin-gcc.h\n>\n> diff --git a/compat/fsmonitor/fsm-darwin-gcc.h b/compat/fsmonitor/fsm-darwin-gcc.h\n> new file mode 100644\n> index 00000000000..1c75c3d48e7\n> --- /dev/null\n> +++ b/compat/fsmonitor/fsm-darwin-gcc.h\n> @@ -0,0 +1,92 @@\n> +#ifndef FSM_DARWIN_GCC_H\n> +#define FSM_DARWIN_GCC_H\n> +\n> +#ifndef __clang__\n\nThis was surprising, until I remembered that clang tries really hard to\npretend to be other compilers. I wonder if we should steal the macro\ncheck from compat/compiler.h into something more generic & use it here,\nprobably best as a follow-up...\n\n> [...]\n> +#endif /* !clang */\n> +#endif /* FSM_DARWIN_GCC_H */\n> diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\n> index c84e3344ab9..d2ce942cade 100644\n> --- a/compat/fsmonitor/fsm-listen-darwin.c\n> +++ b/compat/fsmonitor/fsm-listen-darwin.c\n> @@ -1,3 +1,27 @@\n> +#ifndef __clang__\n> +#include \"fsm-darwin-gcc.h\"\n> +#else\n> +#include \n> +#include \n> +\n> [...]\n"},{"id":"451919","messageId":"220322.86ee2tq1jr.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"1f4b5209bf6092dc389db4046e0bdedc61cc4581.1647972010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 18/29] fsmonitor--daemon: implement handle_client callback","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-22T18:30:35Z","receivedAt":"2022-03-22T18:34:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 22 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n> [...]\n> +static int do_handle_client(struct fsmonitor_daemon_state *state,\n> +\t\t\t const char *command,\n> +\t\t\t ipc_server_reply_cb *reply,\n> +\t\t\t struct ipc_server_reply_data *reply_data)\n> +{\n> +\tstruct fsmonitor_token_data *token_data = NULL;\n> +\tstruct strbuf response_token = STRBUF_INIT;\n> +\tstruct strbuf requested_token_id = STRBUF_INIT;\n> +\tstruct strbuf payload = STRBUF_INIT;\n> +\tuint64_t requested_oldest_seq_nr = 0;\n> +\tuint64_t total_response_len = 0;\n> +\tconst char *p;\n> +\tconst struct fsmonitor_batch *batch_head;\n> +\tconst struct fsmonitor_batch *batch;\n> +\tintmax_t count = 0, duplicates = 0;\n> +\tkh_str_t *shown;\n> +\tint hash_ret;\n> +\tint do_trivial = 0;\n> +\tint do_flush = 0;\n\nJust noticed while skimming, isn't initializing do_trivial here (didn't\nexhaustively check the others) getting in the way of uninitialized\nanalysis the compiler will do...\n\n> +\n> +\t/*\n> +\t * We expect `command` to be of the form:\n> +\t *\n> +\t * := quit NUL\n> +\t * | flush NUL\n> +\t * | NUL\n> +\t * | NUL\n> +\t */\n> +\n> +\tif (!strcmp(command, \"quit\")) {\n> +\t\t/*\n> +\t\t * A client has requested over the socket/pipe that the\n> +\t\t * daemon shutdown.\n> +\t\t *\n> +\t\t * Tell the IPC thread pool to shutdown (which completes\n> +\t\t * the await in the main thread (which can stop the\n> +\t\t * fsmonitor listener thread)).\n> +\t\t *\n> +\t\t * There is no reply to the client.\n> +\t\t */\n> +\t\treturn SIMPLE_IPC_QUIT;\n> +\n> +\t} else if (!strcmp(command, \"flush\")) {\n> +\t\t/*\n> +\t\t * Flush all of our cached data and generate a new token\n> +\t\t * just like if we lost sync with the filesystem.\n> +\t\t *\n> +\t\t * Then send a trivial response using the new token.\n> +\t\t */\n> +\t\tdo_flush = 1;\n> +\t\tdo_trivial = 1;\n> +\n> +\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n> +\t\t/* assume V1 timestamp or garbage */\n> +\n> +\t\tchar *p_end;\n> +\n> +\t\tstrtoumax(command, &p_end, 10);\n> +\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t ((*p_end) ?\n> +\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n> +\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n> +\t\t\t\t command);\n> +\t\tdo_trivial = 1;\n> +\n> +\t} else {\n> +\t\t/* We have \"builtin:*\" */\n> +\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n> +\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n> +\t\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n> +\t\t\t\t\t command);\n> +\t\t\tdo_trivial = 1;\n> +\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * We have a V2 valid token:\n> +\t\t\t * \"builtin::\"\n> +\t\t\t */\n> +\t\t}\n> +\t}\n\nSince we'll set it here in all branches except the \"else\" branch,\ni.e. if you move this to \"we have a v2 valid\" we'll catch future bugs if\nwe ever have it uninitialized still, \n\n> +\n> +\tpthread_mutex_lock(&state->main_lock);\n> +\n> +\tif (!state->current_token_data)\n> +\t\tBUG(\"fsmonitor state does not have a current token\");\n> +\n> +\tif (do_flush)\n> +\t\twith_lock__do_force_resync(state);\n> +\n> +\t/*\n> +\t * We mark the current head of the batch list as \"pinned\" so\n> +\t * that the listener thread will treat this item as read-only\n> +\t * (and prevent any more paths from being added to it) from\n> +\t * now on.\n> +\t */\n> +\ttoken_data = state->current_token_data;\n> +\tbatch_head = token_data->batch_head;\n> +\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n> +\n> +\t/*\n> +\t * FSMonitor Protocol V2 requires that we send a response header\n> +\t * with a \"new current token\" and then all of the paths that changed\n> +\t * since the \"requested token\". We send the seq_nr of the just-pinned\n> +\t * head batch so that future requests from a client will be relative\n> +\t * to it.\n> +\t */\n> +\twith_lock__format_response_token(&response_token,\n> +\t\t\t\t\t &token_data->token_id, batch_head);\n> +\n> +\treply(reply_data, response_token.buf, response_token.len + 1);\n> +\ttotal_response_len += response_token.len + 1;\n> +\n> +\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n> +\t\t\t response_token.buf);\n> +\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n> +\t\t\t response_token.buf);\n> +\n> +\tif (!do_trivial) {\n\nI.e. this would start warning.\n\n> +\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n> +\t\t\t/*\n> +\t\t\t * The client last spoke to a different daemon\n> +\t\t\t * instance -OR- the daemon had to resync with\n> +\t\t\t * the filesystem (and lost events), so reject.\n> +\t\t\t */\n> +\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n> +\t\t\t\t\t \"response/token\", \"different\");\n> +\t\t\tdo_trivial = 1;\n> +\n> +\t\t} else if (requested_oldest_seq_nr <\n> +\t\t\t token_data->batch_tail->batch_seq_nr) {\n> +\t\t\t/*\n> +\t\t\t * The client wants older events than we have for\n> +\t\t\t * this token_id. This means that the end of our\n> +\t\t\t * batch list was truncated and we cannot give the\n> +\t\t\t * client a complete snapshot relative to their\n> +\t\t\t * request.\n> +\t\t\t */\n> +\t\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t\t \"client requested truncated data\");\n> +\t\t\tdo_trivial = 1;\n> +\t\t}\n> +\t}\n> +\n> +\tif (do_trivial) {\n> +\t\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\t\treply(reply_data, \"/\", 2);\n> +\n> +\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n> +\t\t\t\t \"response/trivial\", 1);\n> +\n> +\t\tstrbuf_release(&response_token);\n> +\t\tstrbuf_release(&requested_token_id);\n> +\t\treturn 0;\n\nNit: instead of strbuf_release() here, just a 'goto cleanup' and ...\n\n> +\t}\n> +\n> +\t/*\n> +\t * We're going to hold onto a pointer to the current\n> +\t * token-data while we walk the list of batches of files.\n> +\t * During this time, we will NOT be under the lock.\n> +\t * So we ref-count it.\n> +\t *\n> +\t * This allows the listener thread to continue prepending\n> +\t * new batches of items to the token-data (which we'll ignore).\n> +\t *\n> +\t * AND it allows the listener thread to do a token-reset\n> +\t * (and install a new `current_token_data`).\n> +\t */\n> +\ttoken_data->client_ref_count++;\n> +\n> +\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\t/*\n> +\t * The client request is relative to the token that they sent,\n> +\t * so walk the batch list backwards from the current head back\n> +\t * to the batch (sequence number) they named.\n> +\t *\n> +\t * We use khash to de-dup the list of pathnames.\n> +\t *\n> +\t * NEEDSWORK: each batch contains a list of interned strings,\n> +\t * so we only need to do pointer comparisons here to build the\n> +\t * hash table. Currently, we're still comparing the string\n> +\t * values.\n> +\t */\n> +\tshown = kh_init_str();\n> +\tfor (batch = batch_head;\n> +\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n> +\t batch = batch->next) {\n> +\t\tsize_t k;\n> +\n> +\t\tfor (k = 0; k < batch->nr; k++) {\n> +\t\t\tconst char *s = batch->interned_paths[k];\n> +\t\t\tsize_t s_len;\n> +\n> +\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n> +\t\t\t\tduplicates++;\n> +\t\t\telse {\n> +\t\t\t\tkh_put_str(shown, s, &hash_ret);\n> +\n> +\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n> +\t\t\t\t\t\t count, s);\n> +\n> +\t\t\t\t/* Each path gets written with a trailing NUL */\n> +\t\t\t\ts_len = strlen(s) + 1;\n> +\n> +\t\t\t\tif (payload.len + s_len >=\n> +\t\t\t\t LARGE_PACKET_DATA_MAX) {\n> +\t\t\t\t\treply(reply_data, payload.buf,\n> +\t\t\t\t\t payload.len);\n> +\t\t\t\t\ttotal_response_len += payload.len;\n> +\t\t\t\t\tstrbuf_reset(&payload);\n> +\t\t\t\t}\n> +\n> +\t\t\t\tstrbuf_add(&payload, s, s_len);\n> +\t\t\t\tcount++;\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> +\n> +\tif (payload.len) {\n> +\t\treply(reply_data, payload.buf, payload.len);\n> +\t\ttotal_response_len += payload.len;\n> +\t}\n> +\n> +\tkh_release_str(shown);\n> +\n> +\tpthread_mutex_lock(&state->main_lock);\n> +\n> +\tif (token_data->client_ref_count > 0)\n> +\t\ttoken_data->client_ref_count--;\n> +\n> +\tif (token_data->client_ref_count == 0) {\n> +\t\tif (token_data != state->current_token_data) {\n> +\t\t\t/*\n> +\t\t\t * The listener thread did a token-reset while we were\n> +\t\t\t * walking the batch list. Therefore, this token is\n> +\t\t\t * stale and can be discarded completely. If we are\n> +\t\t\t * the last reader thread using this token, we own\n> +\t\t\t * that work.\n> +\t\t\t */\n> +\t\t\tfsmonitor_free_token_data(token_data);\n> +\t\t}\n> +\t}\n> +\n> +\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n> +\n\nadd a \"cleanup\" label here?\n\n> +\tstrbuf_release(&response_token);\n> +\tstrbuf_release(&requested_token_id);\n> +\tstrbuf_release(&payload);\n> +\n> +\treturn 0;\n> +}\n> +\n> static ipc_server_application_cb handle_client;\n"},{"id":"451920","messageId":"220322.86a6dhq1a6.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"4a920d0b54a25a442bf52efc171139c698d59dc7.1647972010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 21/29] t7527: create test for fsmonitor--daemon","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-22T18:35:27Z","receivedAt":"2022-03-22T18:40:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 22 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Junio C Hamano \n> ---\n> t/t7527-builtin-fsmonitor.sh | 501 +++++++++++++++++++++++++++++++++++\n> 1 file changed, 501 insertions(+)\n> create mode 100755 t/t7527-builtin-fsmonitor.sh\n>\n> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n> new file mode 100755\n> index 00000000000..d79635e7596\n> --- /dev/null\n> +++ b/t/t7527-builtin-fsmonitor.sh\n> @@ -0,0 +1,501 @@\n> +#!/bin/sh\n> +\n> +test_description='built-in file system watcher'\n> +\n> +. ./test-lib.sh\n> +\n> +if ! test_have_prereq FSMONITOR_DAEMON\n> +then\n> +\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n> +\ttest_done\n> +fi\n> +\n> +stop_daemon_delete_repo () {\n> +\tr=$1 &&\n> +\ttest_might_fail git -C $r fsmonitor--daemon stop &&\n> +\trm -rf $1\n> +}\n> +\n> +start_daemon () {\n> +\tr= &&\n> +\ttf= &&\n> +\tt2= &&\n> +\ttk= &&\n\nI think these all have &&-chains added since previous versions, good...\n\n> +\twhile test \"$#\" -ne 0\n> +\tdo\n> +\t\tcase \"$1\" in\n> +\t\t-C)\n> +\t\t\tshift;\n> +\t\t\ttest \"$#\" -ne 0 || BUG \"error: -C requires arg\"\n> +\t\t\tr=\"-C $1\"\n> +\t\t\tshift\n> +\t\t\t;;\n> +\t\t-tf)\n> +\t\t\tshift;\n> +\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tf requires arg\"\n> +\t\t\ttf=\"$1\"\n> +\t\t\tshift\n> +\t\t\t;;\n> +\t\t-t2)\n> +\t\t\tshift;\n> +\t\t\ttest \"$#\" -ne 0 || BUG \"error: -t2 requires arg\"\n> +\t\t\tt2=\"$1\"\n> +\t\t\tshift\n> +\t\t\t;;\n> +\t\t-tk)\n> +\t\t\tshift;\n> +\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tk requires arg\"\n> +\t\t\ttk=\"$1\"\n> +\t\t\tshift\n> +\t\t\t;;\n\nBut (and IIRC I noted this in a previous iteration) if you &&-chain the\n\"shift\" here you can lose the more verbose BUG\n\n> +\tstart_daemon -tf \"$PWD/.git/trace\" &&\n\nFWIW having an option parser take -tf to mean --tf is quite unlike our\ncommon conventions, usually it means both -t and -f.\n\nIn this case every single caller added here does provide -tf\nargument. Perhaps better as as unconditional $1 then?\n\n\n"},{"id":"451921","messageId":"220322.865yo5q13f.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"803a540cc0022e893a75eae8815b3275a7fac3af.1647972011.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 24/29] t/perf/p7519: speed up test on Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-22T18:43:04Z","receivedAt":"2022-03-22T18:44:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 22 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Change p7519 to use `test_seq` and `xargs` rather than a `for` loop\n> to touch thousands of files. This takes minutes off of test runs\n> on Windows because of process creation overhead.\n>\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Junio C Hamano \n> ---\n> t/perf/p7519-fsmonitor.sh | 32 ++++++++++++++++++++------------\n> 1 file changed, 20 insertions(+), 12 deletions(-)\n>\n> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n> index c8be58f3c76..0611e533951 100755\n> --- a/t/perf/p7519-fsmonitor.sh\n> +++ b/t/perf/p7519-fsmonitor.sh\n> @@ -72,7 +72,7 @@ then\n> \tfi\n> fi\n> \n> -trace_start() {\n> +trace_start () {\n> \tif test -n \"$GIT_PERF_7519_TRACE\"\n> \tthen\n> \t\tname=\"$1\"\n> @@ -91,13 +91,20 @@ trace_start() {\n> \tfi\n> }\n> \n> -trace_stop() {\n> +trace_stop () {\n> \tif test -n \"$GIT_PERF_7519_TRACE\"\n> \tthen\n> \t\tunset GIT_TRACE2_PERF\n> \tfi\n> }\n\n(I think I noted in a previous version): Would be nice to have an\noptimization change not do unrelated refactoring....\n\n> +touch_files () {\n> +\tn=$1\n> +\td=\"$n\"_files\n> +\n> +\t(cd $d ; test_seq 1 $n | xargs touch )\n\n\n...and here we don't &&-chain.\n\n> +}\n> +\n> test_expect_success \"one time repo setup\" '\n> \t# set untrackedCache depending on the environment\n> \tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n> @@ -119,10 +126,11 @@ test_expect_success \"one time repo setup\" '\n> \tfi &&\n> \n> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n> -\tfor i in $(test_seq 1 10); do touch 10_files/$i || return 1; done &&\n> -\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n> -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n> -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n> +\t: 1_file directory should be left empty &&\n> +\ttouch_files 10 &&\n> +\ttouch_files 100 &&\n> +\ttouch_files 1000 &&\n> +\ttouch_files 10000 &&\n> \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n> \tgit commit -qm \"Add files\" &&\n> \n> @@ -133,7 +141,7 @@ test_expect_success \"one time repo setup\" '\n> \tfi\n> '\n> \n> -setup_for_fsmonitor() {\n> +setup_for_fsmonitor () {\n> \t# set INTEGRATION_SCRIPT depending on the environment\n> \tif test -n \"$INTEGRATION_PATH\"\n> \tthen\n> @@ -173,7 +181,7 @@ test_perf_w_drop_caches () {\n> \ttest_perf \"$@\"\n> }\n> \n> -test_fsmonitor_suite() {\n> +test_fsmonitor_suite () {\n\nditto refactoring..\n"},{"id":"451990","messageId":"cc9ff6b9-9574-8170-f7de-b3f0e3815edd@jeffhostetler.com","threadId":"56518","inReplyTo":"220322.86ils5q1um.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v7 16/29] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-23T14:32:18Z","receivedAt":"2022-03-23T14:32:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/22/22 2:19 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 22 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n[...]\n>> +++ b/compat/fsmonitor/fsm-darwin-gcc.h\n>> @@ -0,0 +1,92 @@\n>> +#ifndef FSM_DARWIN_GCC_H\n>> +#define FSM_DARWIN_GCC_H\n>> +\n>> +#ifndef __clang__\n> \n> This was surprising, until I remembered that clang tries really hard to\n> pretend to be other compilers. I wonder if we should steal the macro\n> check from compat/compiler.h into something more generic & use it here,\n> probably best as a follow-up...\n> \n>> [...]\n\nyeah, V5 had this as an ifdef __GNUC__, I changed it in V6 because\nwe were always using the hack local declarations rather than the\nofficial header files in clang builds. (Technically, it doesn't\nreally matter, since the net result is the same, but it felt\ncleaner (er, less mysterious).)\n\nAs for extracting a better macro from compat/compiler.h for more\ngeneral use, maybe, but that's not something I want to think about\nright now. Doing a quick \"git grep __GNUC__\" turns up a lot of dark\nmagic. There are only a couple of references to __clang__, so it\nmight be that a good comment somewhere (say in compat/compiler.h)\nwould be sufficient.\n\nJeff\n"},{"id":"451991","messageId":"1bdfffe9-98ad-ce34-56cf-8b75bcc11227@jeffhostetler.com","threadId":"56518","inReplyTo":"220322.86ee2tq1jr.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v7 18/29] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-23T14:45:42Z","receivedAt":"2022-03-23T14:45:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/22/22 2:30 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 22 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>> [...]\n>> +static int do_handle_client(struct fsmonitor_daemon_state *state,\n>> +\t\t\t const char *command,\n>> +\t\t\t ipc_server_reply_cb *reply,\n>> +\t\t\t struct ipc_server_reply_data *reply_data)\n>> +{\n>> +\tstruct fsmonitor_token_data *token_data = NULL;\n>> +\tstruct strbuf response_token = STRBUF_INIT;\n>> +\tstruct strbuf requested_token_id = STRBUF_INIT;\n>> +\tstruct strbuf payload = STRBUF_INIT;\n>> +\tuint64_t requested_oldest_seq_nr = 0;\n>> +\tuint64_t total_response_len = 0;\n>> +\tconst char *p;\n>> +\tconst struct fsmonitor_batch *batch_head;\n>> +\tconst struct fsmonitor_batch *batch;\n>> +\tintmax_t count = 0, duplicates = 0;\n>> +\tkh_str_t *shown;\n>> +\tint hash_ret;\n>> +\tint do_trivial = 0;\n>> +\tint do_flush = 0;\n> \n> Just noticed while skimming, isn't initializing do_trivial here (didn't\n> exhaustively check the others) getting in the way of uninitialized\n> analysis the compiler will do...\n> \n>> +\n>> +\t/*\n>> +\t * We expect `command` to be of the form:\n>> +\t *\n>> +\t * := quit NUL\n>> +\t * | flush NUL\n>> +\t * | NUL\n>> +\t * | NUL\n>> +\t */\n>> +\n>> +\tif (!strcmp(command, \"quit\")) {\n>> +\t\t/*\n>> +\t\t * A client has requested over the socket/pipe that the\n>> +\t\t * daemon shutdown.\n>> +\t\t *\n>> +\t\t * Tell the IPC thread pool to shutdown (which completes\n>> +\t\t * the await in the main thread (which can stop the\n>> +\t\t * fsmonitor listener thread)).\n>> +\t\t *\n>> +\t\t * There is no reply to the client.\n>> +\t\t */\n>> +\t\treturn SIMPLE_IPC_QUIT;\n>> +\n>> +\t} else if (!strcmp(command, \"flush\")) {\n>> +\t\t/*\n>> +\t\t * Flush all of our cached data and generate a new token\n>> +\t\t * just like if we lost sync with the filesystem.\n>> +\t\t *\n>> +\t\t * Then send a trivial response using the new token.\n>> +\t\t */\n>> +\t\tdo_flush = 1;\n>> +\t\tdo_trivial = 1;\n>> +\n>> +\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n>> +\t\t/* assume V1 timestamp or garbage */\n>> +\n>> +\t\tchar *p_end;\n>> +\n>> +\t\tstrtoumax(command, &p_end, 10);\n>> +\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t ((*p_end) ?\n>> +\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n>> +\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n>> +\t\t\t\t command);\n>> +\t\tdo_trivial = 1;\n>> +\n>> +\t} else {\n>> +\t\t/* We have \"builtin:*\" */\n>> +\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n>> +\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n>> +\t\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n>> +\t\t\t\t\t command);\n>> +\t\t\tdo_trivial = 1;\n>> +\n>> +\t\t} else {\n>> +\t\t\t/*\n>> +\t\t\t * We have a V2 valid token:\n>> +\t\t\t * \"builtin::\"\n>> +\t\t\t */\n>> +\t\t}\n>> +\t}\n> \n> Since we'll set it here in all branches except the \"else\" branch,\n> i.e. if you move this to \"we have a v2 valid\" we'll catch future bugs if\n> we ever have it uninitialized still,\n\nI suppose we could do it that way. We'd want to do the same\nfor \"do_flush\" in that case. I could go either way on this\nchange.\n\n\n> \n>> +\n>> +\tpthread_mutex_lock(&state->main_lock);\n>> +\n>> +\tif (!state->current_token_data)\n>> +\t\tBUG(\"fsmonitor state does not have a current token\");\n>> +\n>> +\tif (do_flush)\n>> +\t\twith_lock__do_force_resync(state);\n>> +\n>> +\t/*\n>> +\t * We mark the current head of the batch list as \"pinned\" so\n>> +\t * that the listener thread will treat this item as read-only\n>> +\t * (and prevent any more paths from being added to it) from\n>> +\t * now on.\n>> +\t */\n>> +\ttoken_data = state->current_token_data;\n>> +\tbatch_head = token_data->batch_head;\n>> +\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n>> +\n>> +\t/*\n>> +\t * FSMonitor Protocol V2 requires that we send a response header\n>> +\t * with a \"new current token\" and then all of the paths that changed\n>> +\t * since the \"requested token\". We send the seq_nr of the just-pinned\n>> +\t * head batch so that future requests from a client will be relative\n>> +\t * to it.\n>> +\t */\n>> +\twith_lock__format_response_token(&response_token,\n>> +\t\t\t\t\t &token_data->token_id, batch_head);\n>> +\n>> +\treply(reply_data, response_token.buf, response_token.len + 1);\n>> +\ttotal_response_len += response_token.len + 1;\n>> +\n>> +\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n>> +\t\t\t response_token.buf);\n>> +\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n>> +\t\t\t response_token.buf);\n>> +\n>> +\tif (!do_trivial) {\n> \n> I.e. this would start warning.\n> \n>> +\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n>> +\t\t\t/*\n>> +\t\t\t * The client last spoke to a different daemon\n>> +\t\t\t * instance -OR- the daemon had to resync with\n>> +\t\t\t * the filesystem (and lost events), so reject.\n>> +\t\t\t */\n>> +\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n>> +\t\t\t\t\t \"response/token\", \"different\");\n>> +\t\t\tdo_trivial = 1;\n>> +\n>> +\t\t} else if (requested_oldest_seq_nr <\n>> +\t\t\t token_data->batch_tail->batch_seq_nr) {\n>> +\t\t\t/*\n>> +\t\t\t * The client wants older events than we have for\n>> +\t\t\t * this token_id. This means that the end of our\n>> +\t\t\t * batch list was truncated and we cannot give the\n>> +\t\t\t * client a complete snapshot relative to their\n>> +\t\t\t * request.\n>> +\t\t\t */\n>> +\t\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t\t \"client requested truncated data\");\n>> +\t\t\tdo_trivial = 1;\n>> +\t\t}\n>> +\t}\n>> +\n>> +\tif (do_trivial) {\n>> +\t\tpthread_mutex_unlock(&state->main_lock);\n>> +\n>> +\t\treply(reply_data, \"/\", 2);\n>> +\n>> +\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n>> +\t\t\t\t \"response/trivial\", 1);\n>> +\n>> +\t\tstrbuf_release(&response_token);\n>> +\t\tstrbuf_release(&requested_token_id);\n>> +\t\treturn 0;\n> \n> Nit: instead of strbuf_release() here, just a 'goto cleanup' and ...\n\nYeah, we could do that. That would also do an unnecesssary release\nof \"payload\", but that's not that bad (or I could order them at the\nbottom).\n\nI'm not sure either of these changes are enough for a re-roll,\nbut I'll include them if I do.\n\nThanks\nJeff\n\n> \n>> +\t}\n>> +\n>> +\t/*\n>> +\t * We're going to hold onto a pointer to the current\n>> +\t * token-data while we walk the list of batches of files.\n>> +\t * During this time, we will NOT be under the lock.\n>> +\t * So we ref-count it.\n>> +\t *\n>> +\t * This allows the listener thread to continue prepending\n>> +\t * new batches of items to the token-data (which we'll ignore).\n>> +\t *\n>> +\t * AND it allows the listener thread to do a token-reset\n>> +\t * (and install a new `current_token_data`).\n>> +\t */\n>> +\ttoken_data->client_ref_count++;\n>> +\n>> +\tpthread_mutex_unlock(&state->main_lock);\n>> +\n>> +\t/*\n>> +\t * The client request is relative to the token that they sent,\n>> +\t * so walk the batch list backwards from the current head back\n>> +\t * to the batch (sequence number) they named.\n>> +\t *\n>> +\t * We use khash to de-dup the list of pathnames.\n>> +\t *\n>> +\t * NEEDSWORK: each batch contains a list of interned strings,\n>> +\t * so we only need to do pointer comparisons here to build the\n>> +\t * hash table. Currently, we're still comparing the string\n>> +\t * values.\n>> +\t */\n>> +\tshown = kh_init_str();\n>> +\tfor (batch = batch_head;\n>> +\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n>> +\t batch = batch->next) {\n>> +\t\tsize_t k;\n>> +\n>> +\t\tfor (k = 0; k < batch->nr; k++) {\n>> +\t\t\tconst char *s = batch->interned_paths[k];\n>> +\t\t\tsize_t s_len;\n>> +\n>> +\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n>> +\t\t\t\tduplicates++;\n>> +\t\t\telse {\n>> +\t\t\t\tkh_put_str(shown, s, &hash_ret);\n>> +\n>> +\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n>> +\t\t\t\t\t\t count, s);\n>> +\n>> +\t\t\t\t/* Each path gets written with a trailing NUL */\n>> +\t\t\t\ts_len = strlen(s) + 1;\n>> +\n>> +\t\t\t\tif (payload.len + s_len >=\n>> +\t\t\t\t LARGE_PACKET_DATA_MAX) {\n>> +\t\t\t\t\treply(reply_data, payload.buf,\n>> +\t\t\t\t\t payload.len);\n>> +\t\t\t\t\ttotal_response_len += payload.len;\n>> +\t\t\t\t\tstrbuf_reset(&payload);\n>> +\t\t\t\t}\n>> +\n>> +\t\t\t\tstrbuf_add(&payload, s, s_len);\n>> +\t\t\t\tcount++;\n>> +\t\t\t}\n>> +\t\t}\n>> +\t}\n>> +\n>> +\tif (payload.len) {\n>> +\t\treply(reply_data, payload.buf, payload.len);\n>> +\t\ttotal_response_len += payload.len;\n>> +\t}\n>> +\n>> +\tkh_release_str(shown);\n>> +\n>> +\tpthread_mutex_lock(&state->main_lock);\n>> +\n>> +\tif (token_data->client_ref_count > 0)\n>> +\t\ttoken_data->client_ref_count--;\n>> +\n>> +\tif (token_data->client_ref_count == 0) {\n>> +\t\tif (token_data != state->current_token_data) {\n>> +\t\t\t/*\n>> +\t\t\t * The listener thread did a token-reset while we were\n>> +\t\t\t * walking the batch list. Therefore, this token is\n>> +\t\t\t * stale and can be discarded completely. If we are\n>> +\t\t\t * the last reader thread using this token, we own\n>> +\t\t\t * that work.\n>> +\t\t\t */\n>> +\t\t\tfsmonitor_free_token_data(token_data);\n>> +\t\t}\n>> +\t}\n>> +\n>> +\tpthread_mutex_unlock(&state->main_lock);\n>> +\n>> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n>> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n>> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n>> +\n> \n> add a \"cleanup\" label here?\n> \n>> +\tstrbuf_release(&response_token);\n>> +\tstrbuf_release(&requested_token_id);\n>> +\tstrbuf_release(&payload);\n>> +\n>> +\treturn 0;\n>> +}\n>> +\n>> static ipc_server_application_cb handle_client;\n"},{"id":"452001","messageId":"56a4a545-37b0-0cee-a8c8-1a019b0201c2@jeffhostetler.com","threadId":"56518","inReplyTo":"220322.86a6dhq1a6.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v7 21/29] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-23T16:22:31Z","receivedAt":"2022-03-23T16:22:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/22/22 2:35 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 22 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Signed-off-by: Jeff Hostetler \n>> Signed-off-by: Junio C Hamano \n>> ---\n>> t/t7527-builtin-fsmonitor.sh | 501 +++++++++++++++++++++++++++++++++++\n>> 1 file changed, 501 insertions(+)\n>> create mode 100755 t/t7527-builtin-fsmonitor.sh\n>>\n>> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n>> new file mode 100755\n>> index 00000000000..d79635e7596\n[...]\n> \n>> +\twhile test \"$#\" -ne 0\n>> +\tdo\n>> +\t\tcase \"$1\" in\n>> +\t\t-C)\n>> +\t\t\tshift;\n>> +\t\t\ttest \"$#\" -ne 0 || BUG \"error: -C requires arg\"\n>> +\t\t\tr=\"-C $1\"\n>> +\t\t\tshift\n>> +\t\t\t;;\n>> +\t\t-tf)\n>> +\t\t\tshift;\n>> +\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tf requires arg\"\n>> +\t\t\ttf=\"$1\"\n>> +\t\t\tshift\n>> +\t\t\t;;\n>> +\t\t-t2)\n>> +\t\t\tshift;\n>> +\t\t\ttest \"$#\" -ne 0 || BUG \"error: -t2 requires arg\"\n>> +\t\t\tt2=\"$1\"\n>> +\t\t\tshift\n>> +\t\t\t;;\n>> +\t\t-tk)\n>> +\t\t\tshift;\n>> +\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tk requires arg\"\n>> +\t\t\ttk=\"$1\"\n>> +\t\t\tshift\n>> +\t\t\t;;\n> \n> But (and IIRC I noted this in a previous iteration) if you &&-chain the\n> \"shift\" here you can lose the more verbose BUG\n\nYeah, I looked at the test_commit() example that you referenced.\nI thought it was too subtle and misleading.\n\nI mean, the \"shift\" after the \"case...esac\" will either clobber\nthe \"--key\" or the \"value\" depending on whether the particular\ncase-arm shifted. The shift error would only be thrown on a\nmissing value, since the while loop already tested $# for non-zero,\nbut at the point of the error, we'll just have a generic error message\nand not know which key should have had a value -- without reading\nthe script in detail.\n\nAlso, in the k/v case-arms, it references $2 without confirming\nthat it exists. In test_commit(), it just loads up local variables\n(in advance of the soon-to-be-thrown shift error, so no big deal)\nbut if other people copy this as a model, they may do more in their\ncase-arms that may be more serious.\n\n\nMy version on the other hand, shifts away the key immediately,\ntests whether the required value is present and errors with a\ndetailed message, and then references the value and shifts away\nthe value.\n\nMy way (IMHO) feels more straight-forward and easier for casual\nreaders to follow. Yes, it is a bit more wordy, but I think it\nis worth it.\n\n\nFWIW, as I was writing this note I noticed that both test_commit()\nand my start_daemon() examples have a bug where they won't detect\na missing value. For example, if someone changes\n \"test_commit -C repo\"\nto \"test_commit -C --no-tag repo\"\nthen $indir will be \"--no-tag\" and \"repo\" will be unclaimed and\nan error will follow at some point later (when $indir is used\nin a Git command).\n\n\nI think I'll fix my function to handle that error case, but keep\nthe basic design that I have.\n\n\n> \n>> +\tstart_daemon -tf \"$PWD/.git/trace\" &&\n> \n> FWIW having an option parser take -tf to mean --tf is quite unlike our\n> common conventions, usually it means both -t and -f.\n> \n> In this case every single caller added here does provide -tf\n> argument. Perhaps better as as unconditional $1 then?\n> \n\nyes, very old-school of me to use single dashes here. I'll change\nit/them to use double-dashes (or relabel the keys to be single chars).\nThere are callers that do not pass the -tf key, so I'd rather keep it\nas a key/value than assume a fixed $1.\n\nThanks\nJeff\n\n"},{"id":"452002","messageId":"3c3ec71c-d2f6-338d-721a-a76bcb588e0a@jeffhostetler.com","threadId":"56518","inReplyTo":"220322.865yo5q13f.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v7 24/29] t/perf/p7519: speed up test on Windows","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-23T16:33:48Z","receivedAt":"2022-03-23T16:33:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/22/22 2:43 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 22 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Change p7519 to use `test_seq` and `xargs` rather than a `for` loop\n>> to touch thousands of files. This takes minutes off of test runs\n>> on Windows because of process creation overhead.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> Signed-off-by: Junio C Hamano \n>> ---\n>> t/perf/p7519-fsmonitor.sh | 32 ++++++++++++++++++++------------\n>> 1 file changed, 20 insertions(+), 12 deletions(-)\n>>\n>> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n>> index c8be58f3c76..0611e533951 100755\n>> --- a/t/perf/p7519-fsmonitor.sh\n>> +++ b/t/perf/p7519-fsmonitor.sh\n>> @@ -72,7 +72,7 @@ then\n[...]\n>> \n>> -trace_stop() {\n>> +trace_stop () {\n>> \tif test -n \"$GIT_PERF_7519_TRACE\"\n>> \tthen\n>> \t\tunset GIT_TRACE2_PERF\n>> \tfi\n>> }\n> \n> (I think I noted in a previous version): Would be nice to have an\n> optimization change not do unrelated refactoring....\n\nIn V4 24/29, Junio pushed a fixup! commit onto my branch to\nadd the space between the function name and the parens of the\nfunction that I added. And fixed the other existing instances\nwhile he was at it. I squashed that down into my branch during\ncleanups between V4 and V5. It seemed less trouble/noise than\nsplitting it into two commits. (I was also at the 30 commit\nlimit of GGG in V5 and V6, so that was another reason not to\nsplit it.)\n\nNow that I'm at 29 in V7, I suppose that I could split it.\n\n\n\n> \n>> +touch_files () {\n>> +\tn=$1\n>> +\td=\"$n\"_files\n>> +\n>> +\t(cd $d ; test_seq 1 $n | xargs touch )\n> \n> \n> ...and here we don't &&-chain.\n> \n\nRight, thanks.\n\nJeff\n\n"},{"id":"452130","messageId":"e98373f997f5994a08423b3f42b6ddadb3a2a511.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 01/30] fsmonitor: enhance existing comments, clarify trivial response handling","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:17Z","receivedAt":"2022-03-24T16:49:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 64 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 41 insertions(+), 23 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..448d0ee33f5 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -168,29 +168,15 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf *\n \n \tif (result)\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/failed\", result);\n-\telse {\n+\telse\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/response-length\",\n \t\t\t\t query_result->len);\n \n-\t\tif (fsmonitor_is_trivial_response(query_result))\n-\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n-\t\t\t\t\t \"query/trivial-response\", 1);\n-\t}\n-\n \ttrace2_region_leave(\"fsm_hook\", \"query\", NULL);\n \n \treturn result;\n }\n \n-int fsmonitor_is_trivial_response(const struct strbuf *query_result)\n-{\n-\tstatic char trivial_response[3] = { '\\0', '/', '\\0' };\n-\n-\treturn query_result->len >= 3 &&\n-\t\t!memcmp(trivial_response,\n-\t\t\t&query_result->buf[query_result->len - 3], 3);\n-}\n-\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n@@ -238,6 +224,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tint is_trivial = 0;\n \n \tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n \t\treturn;\n@@ -283,6 +270,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t\tquery_success = 0;\n \t\t\t\t} else {\n \t\t\t\t\tbol = last_update_token.len + 1;\n+\t\t\t\t\tis_trivial = query_result.buf[bol] == '/';\n \t\t\t\t}\n \t\t\t} else if (hook_version < 0) {\n \t\t\t\thook_version = HOOK_INTERFACE_VERSION1;\n@@ -294,16 +282,38 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\t\tif (query_success)\n+\t\t\t\tis_trivial = query_result.buf[0] == '/';\n \t\t}\n \n+\t\tif (is_trivial)\n+\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n \t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n \t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n-\tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n+\tif (query_success && !is_trivial) {\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -318,11 +328,16 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We failed to get a response or received a trivial response,\n+\t\t * so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -330,7 +345,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"452131","messageId":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v7.git.1647972010.gitgitgadget@gmail.com","subject":"[PATCH v8 00/30] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:16Z","receivedAt":"2022-03-24T16:49:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V8 of Part 2 of my builtin FSMonitor series. This contains last\nminute comments on V7. I'll also send an updated version of Part 3 that\nbuilds upon this version.\n\nHere is a range-diff from V7 to V8 relative to 715d08a9e5 (The eighth batch,\n2022-02-25). Changes since V7 can be summarized as:\n\n * [] Improved arg checking in start_daemon() in t7527.\n * [] Switch start_daemon() args to use double-dash keywords.\n * [] Split out coding style changes in p7519 into separate commit from\n speed up changes.\n * [] Disabled some of the matrix tests in t7527 concerning the untracked\n cache.\n * [] Other minor style cleanups.\n\n 1: e98373f997 = 1: e98373f997 fsmonitor: enhance existing comments, clarify trivial response handling\n 2: ab68b94417 = 2: ab68b94417 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n 3: e04c7301f2 = 3: e04c7301f2 fsmonitor: config settings are repository-specific\n 4: ea02ba25d8 = 4: ea02ba25d8 fsmonitor: use IPC to query the builtin FSMonitor daemon\n 5: 6ab7db9cb7 = 5: 6ab7db9cb7 fsmonitor: document builtin fsmonitor\n 6: 0ce8ae3f2c = 6: 0ce8ae3f2c fsmonitor--daemon: add a built-in fsmonitor daemon\n 7: 4624ce2fa4 = 7: 4624ce2fa4 fsmonitor--daemon: implement 'stop' and 'status' commands\n 8: a29fe7266a = 8: a29fe7266a compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n 9: 2f8a42fdb9 = 9: 2f8a42fdb9 compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n10: f07800690e = 10: f07800690e fsmonitor--daemon: implement 'run' command\n11: a6a39a3306 = 11: a6a39a3306 fsmonitor--daemon: implement 'start' command\n12: d62e338d00 = 12: d62e338d00 fsmonitor--daemon: add pathname classification\n13: 53e06b4ae5 = 13: 53e06b4ae5 fsmonitor--daemon: define token-ids\n14: 39f43fabe0 = 14: 39f43fabe0 fsmonitor--daemon: create token-based changed path cache\n15: 239558e34f = 15: 239558e34f compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows\n16: 14b775e9d8 = 16: 14b775e9d8 compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n17: 55bd7aee06 = 17: 55bd7aee06 compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS\n18: 1f4b5209bf ! 18: c43009124f fsmonitor--daemon: implement handle_client callback\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n + trace2_data_intmax(\"fsmonitor\", the_repository,\n + \"response/trivial\", 1);\n +\n -+ strbuf_release(&response_token);\n -+ strbuf_release(&requested_token_id);\n -+ return 0;\n ++ goto cleanup;\n + }\n +\n + /*\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n + trace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n + trace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n +\n ++cleanup:\n + strbuf_release(&response_token);\n + strbuf_release(&requested_token_id);\n + strbuf_release(&payload);\n19: 8cf62c9fc6 = 19: ed338777b5 help: include fsmonitor--daemon feature flag in version info\n20: 1bd74a8159 = 20: c99bac29d4 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n21: 4a920d0b54 ! 21: c8709da945 t7527: create test for fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n + rm -rf $1\n +}\n +\n ++is_value () {\n ++ test -n \"$1\" && test \"${1::1}\" != \"-\"\n ++}\n ++\n +start_daemon () {\n + r= &&\n + tf= &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n + case \"$1\" in\n + -C)\n + shift;\n -+ test \"$#\" -ne 0 || BUG \"error: -C requires arg\"\n ++ is_value $1 || BUG \"error: -C requires value\"\n + r=\"-C $1\"\n + shift\n + ;;\n -+ -tf)\n ++ --tf)\n + shift;\n -+ test \"$#\" -ne 0 || BUG \"error: -tf requires arg\"\n ++ is_value $1 || BUG \"error: --tf requires value\"\n + tf=\"$1\"\n + shift\n + ;;\n -+ -t2)\n ++ --t2)\n + shift;\n -+ test \"$#\" -ne 0 || BUG \"error: -t2 requires arg\"\n ++ is_value $1 || BUG \"error: --t2 requires value\"\n + t2=\"$1\"\n + shift\n + ;;\n -+ -tk)\n ++ --tk)\n + shift;\n -+ test \"$#\" -ne 0 || BUG \"error: -tk requires arg\"\n ++ is_value $1 || BUG \"error: --tk requires value\"\n + tk=\"$1\"\n + shift\n + ;;\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'edit some files' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ start_daemon -tf \"$PWD/.git/trace\" &&\n ++ start_daemon --tf \"$PWD/.git/trace\" &&\n +\n + edit_files &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'create some files' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ start_daemon -tf \"$PWD/.git/trace\" &&\n ++ start_daemon --tf \"$PWD/.git/trace\" &&\n +\n + create_files &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'delete some files' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ start_daemon -tf \"$PWD/.git/trace\" &&\n ++ start_daemon --tf \"$PWD/.git/trace\" &&\n +\n + delete_files &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'rename some files' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ start_daemon -tf \"$PWD/.git/trace\" &&\n ++ start_daemon --tf \"$PWD/.git/trace\" &&\n +\n + rename_files &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'rename directory' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ start_daemon -tf \"$PWD/.git/trace\" &&\n ++ start_daemon --tf \"$PWD/.git/trace\" &&\n +\n + mv dirtorename dirrenamed &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'file changes to directory' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ start_daemon -tf \"$PWD/.git/trace\" &&\n ++ start_daemon --tf \"$PWD/.git/trace\" &&\n +\n + file_to_directory &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'directory changes to a file' '\n + test_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+ start_daemon -tf \"$PWD/.git/trace\" &&\n ++ start_daemon --tf \"$PWD/.git/trace\" &&\n +\n + directory_to_file &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n + git init test_flush &&\n +\n -+ start_daemon -C test_flush -tf \"$PWD/.git/trace_daemon\" -tk true &&\n ++ start_daemon -C test_flush --tf \"$PWD/.git/trace_daemon\" --tk true &&\n +\n + # The daemon should have an initial token with no events in _0 and\n + # then a few (probably platform-specific number of) events in _1.\n @@ t/t7527-builtin-fsmonitor.sh (new)\n + git -C wt-base worktree add ../wt-secondary &&\n +\n + start_daemon -C wt-secondary \\\n -+ -tf \"$PWD/trace_wt_secondary\" \\\n -+ -t2 \"$PWD/trace2_wt_secondary\" &&\n ++ --tf \"$PWD/trace_wt_secondary\" \\\n ++ --t2 \"$PWD/trace2_wt_secondary\" &&\n +\n + git -C wt-secondary fsmonitor--daemon stop &&\n + test_must_fail git -C wt-secondary fsmonitor--daemon status\n22: c925a9a745 = 22: cc39ecf10a t/perf: avoid copying builtin fsmonitor files into test repo\n23: 5b3381c223 = 23: 2bb3eb8476 t/helper/test-chmtime: skip directories on Windows\n -: ---------- > 24: bab9a9b080 t/perf/p7519: fix coding style\n24: 803a540cc0 ! 25: 2dd06ad2f7 t/perf/p7519: speed up test on Windows\n @@ Commit message\n Signed-off-by: Junio C Hamano \n \n ## t/perf/p7519-fsmonitor.sh ##\n -@@ t/perf/p7519-fsmonitor.sh: then\n - fi\n - fi\n - \n --trace_start() {\n -+trace_start () {\n - if test -n \"$GIT_PERF_7519_TRACE\"\n - then\n - name=\"$1\"\n -@@ t/perf/p7519-fsmonitor.sh: trace_start() {\n - fi\n - }\n - \n --trace_stop() {\n -+trace_stop () {\n - if test -n \"$GIT_PERF_7519_TRACE\"\n - then\n - unset GIT_TRACE2_PERF\n +@@ t/perf/p7519-fsmonitor.sh: trace_stop () {\n fi\n }\n \n +touch_files () {\n -+ n=$1\n -+ d=\"$n\"_files\n ++ n=$1 &&\n ++ d=\"$n\"_files &&\n +\n -+ (cd $d ; test_seq 1 $n | xargs touch )\n ++ (cd $d && test_seq 1 $n | xargs touch )\n +}\n +\n test_expect_success \"one time repo setup\" '\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n git add 1_file 10_files 100_files 1000_files 10000_files &&\n git commit -qm \"Add files\" &&\n \n -@@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n - fi\n - '\n - \n --setup_for_fsmonitor() {\n -+setup_for_fsmonitor () {\n - # set INTEGRATION_SCRIPT depending on the environment\n - if test -n \"$INTEGRATION_PATH\"\n - then\n -@@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n - test_perf \"$@\"\n - }\n - \n --test_fsmonitor_suite() {\n -+test_fsmonitor_suite () {\n - if test -n \"$INTEGRATION_SCRIPT\"; then\n - DESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n - else\n -@@ t/perf/p7519-fsmonitor.sh: test_fsmonitor_suite() {\n +@@ t/perf/p7519-fsmonitor.sh: test_fsmonitor_suite () {\n \n # Update the mtimes on upto 100k files to make status think\n # that they are dirty. For simplicity, omit any files with\n25: d5ca2df31c = 26: 6eaa5765ae t/perf/p7519: add fsmonitor--daemon test cases\n26: 42631259e8 = 27: 30957f3930 fsmonitor--daemon: periodically truncate list of modified files\n27: f256c3cbe8 = 28: c8ca2a1727 fsmonitor--daemon: use a cookie file to sync with file system\n28: 08af8296f9 = 29: 4caf1d89b8 fsmonitor: force update index after large responses\n29: e6cf84dc8e ! 30: f87a1eba69 t7527: test status with untracked-cache and fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n + fsm=$2 &&\n + fn=$3 &&\n +\n ++ if test $uc = true && test $fsm = false\n ++ then\n ++ # The untracked-cache is buggy when FSMonitor is\n ++ # DISABLED, so skip the tests for this matrix\n ++ # combination.\n ++ #\n ++ # We've observed random, occasional test failures on\n ++ # Windows and MacOS when the UC is turned on and FSM\n ++ # is turned off. These are rare, but they do happen\n ++ # indicating that it is probably a race condition within\n ++ # the untracked cache itself.\n ++ #\n ++ # It usually happens when a test does F/D trickery and\n ++ # then the NEXT test fails because of extra status\n ++ # output from stale UC data from the previous test.\n ++ #\n ++ # Since FSMonitor is not involved in the error, skip\n ++ # the tests for this matrix combination.\n ++ #\n ++ return 0\n ++ fi &&\n ++\n + test_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n + matrix_clean_up_repo &&\n + $fn &&\n\n\nJeff Hostetler (30):\n fsmonitor: enhance existing comments, clarify trivial response\n handling\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor: document builtin fsmonitor\n fsmonitor--daemon: add a built-in fsmonitor daemon\n fsmonitor--daemon: implement 'stop' and 'status' commands\n compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n fsmonitor--daemon: implement 'run' command\n fsmonitor--daemon: implement 'start' command\n fsmonitor--daemon: add pathname classification\n fsmonitor--daemon: define token-ids\n fsmonitor--daemon: create token-based changed path cache\n compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on\n Windows\n compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on\n MacOS\n fsmonitor--daemon: implement handle_client callback\n help: include fsmonitor--daemon feature flag in version info\n t/helper/fsmonitor-client: create IPC client to talk to FSMonitor\n Daemon\n t7527: create test for fsmonitor--daemon\n t/perf: avoid copying builtin fsmonitor files into test repo\n t/helper/test-chmtime: skip directories on Windows\n t/perf/p7519: fix coding style\n t/perf/p7519: speed up test on Windows\n t/perf/p7519: add fsmonitor--daemon test cases\n fsmonitor--daemon: periodically truncate list of modified files\n fsmonitor--daemon: use a cookie file to sync with file system\n fsmonitor: force update index after large responses\n t7527: test status with untracked-cache and fsmonitor--daemon\n\n .gitignore | 1 +\n Documentation/config/core.txt | 60 +-\n Documentation/git-fsmonitor--daemon.txt | 75 ++\n Documentation/git-update-index.txt | 8 +-\n Makefile | 17 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 1479 +++++++++++++++++++++++\n builtin/update-index.c | 7 +-\n cache.h | 1 -\n compat/fsmonitor/fsm-darwin-gcc.h | 92 ++\n compat/fsmonitor/fsm-listen-darwin.c | 427 +++++++\n compat/fsmonitor/fsm-listen-win32.c | 586 +++++++++\n compat/fsmonitor/fsm-listen.h | 49 +\n config.c | 14 -\n config.h | 1 -\n config.mak.uname | 20 +\n contrib/buildsystems/CMakeLists.txt | 10 +\n environment.c | 1 -\n fsmonitor--daemon.h | 166 +++\n fsmonitor-ipc.c | 171 +++\n fsmonitor-ipc.h | 48 +\n fsmonitor-settings.c | 114 ++\n fsmonitor-settings.h | 21 +\n fsmonitor.c | 216 +++-\n fsmonitor.h | 15 +-\n git.c | 1 +\n help.c | 4 +\n repo-settings.c | 1 +\n repository.h | 3 +\n t/README | 4 +-\n t/helper/test-chmtime.c | 15 +\n t/helper/test-fsmonitor-client.c | 116 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/perf/p7519-fsmonitor.sh | 68 +-\n t/perf/perf-lib.sh | 2 +-\n t/t7527-builtin-fsmonitor.sh | 620 ++++++++++\n t/test-lib.sh | 7 +\n 38 files changed, 4337 insertions(+), 106 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n create mode 100644 builtin/fsmonitor--daemon.c\n create mode 100644 compat/fsmonitor/fsm-darwin-gcc.h\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n create mode 100644 fsmonitor--daemon.h\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n create mode 100644 t/helper/test-fsmonitor-client.c\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\n\nbase-commit: 715d08a9e51251ad8290b181b6ac3b9e1f9719d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v8\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v8\nPull-Request: https://github.com/gitgitgadget/git/pull/1041\n\nRange-diff vs v7:\n\n 1: e98373f997f = 1: e98373f997f fsmonitor: enhance existing comments, clarify trivial response handling\n 2: ab68b944173 = 2: ab68b944173 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n 3: e04c7301f24 = 3: e04c7301f24 fsmonitor: config settings are repository-specific\n 4: ea02ba25d8f = 4: ea02ba25d8f fsmonitor: use IPC to query the builtin FSMonitor daemon\n 5: 6ab7db9cb76 = 5: 6ab7db9cb76 fsmonitor: document builtin fsmonitor\n 6: 0ce8ae3f2cf = 6: 0ce8ae3f2cf fsmonitor--daemon: add a built-in fsmonitor daemon\n 7: 4624ce2fa47 = 7: 4624ce2fa47 fsmonitor--daemon: implement 'stop' and 'status' commands\n 8: a29fe7266a4 = 8: a29fe7266a4 compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n 9: 2f8a42fdb93 = 9: 2f8a42fdb93 compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n 10: f07800690ee = 10: f07800690ee fsmonitor--daemon: implement 'run' command\n 11: a6a39a3306d = 11: a6a39a3306d fsmonitor--daemon: implement 'start' command\n 12: d62e338d008 = 12: d62e338d008 fsmonitor--daemon: add pathname classification\n 13: 53e06b4ae5d = 13: 53e06b4ae5d fsmonitor--daemon: define token-ids\n 14: 39f43fabe02 = 14: 39f43fabe02 fsmonitor--daemon: create token-based changed path cache\n 15: 239558e34ff = 15: 239558e34ff compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows\n 16: 14b775e9d8b = 16: 14b775e9d8b compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n 17: 55bd7aee06c = 17: 55bd7aee06c compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS\n 18: 1f4b5209bf6 ! 18: c43009124fb fsmonitor--daemon: implement handle_client callback\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n +\t\t\t\t \"response/trivial\", 1);\n +\n -+\t\tstrbuf_release(&response_token);\n -+\t\tstrbuf_release(&requested_token_id);\n -+\t\treturn 0;\n ++\t\tgoto cleanup;\n +\t}\n +\n +\t/*\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n +\n ++cleanup:\n +\tstrbuf_release(&response_token);\n +\tstrbuf_release(&requested_token_id);\n +\tstrbuf_release(&payload);\n 19: 8cf62c9fc6f = 19: ed338777b56 help: include fsmonitor--daemon feature flag in version info\n 20: 1bd74a81593 = 20: c99bac29d42 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n 21: 4a920d0b54a ! 21: c8709da9457 t7527: create test for fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\trm -rf $1\n +}\n +\n ++is_value () {\n ++\ttest -n \"$1\" && test \"${1::1}\" != \"-\"\n ++}\n ++\n +start_daemon () {\n +\tr= &&\n +\ttf= &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t\tcase \"$1\" in\n +\t\t-C)\n +\t\t\tshift;\n -+\t\t\ttest \"$#\" -ne 0 || BUG \"error: -C requires arg\"\n ++\t\t\tis_value $1 || BUG \"error: -C requires value\"\n +\t\t\tr=\"-C $1\"\n +\t\t\tshift\n +\t\t\t;;\n -+\t\t-tf)\n ++\t\t--tf)\n +\t\t\tshift;\n -+\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tf requires arg\"\n ++\t\t\tis_value $1 || BUG \"error: --tf requires value\"\n +\t\t\ttf=\"$1\"\n +\t\t\tshift\n +\t\t\t;;\n -+\t\t-t2)\n ++\t\t--t2)\n +\t\t\tshift;\n -+\t\t\ttest \"$#\" -ne 0 || BUG \"error: -t2 requires arg\"\n ++\t\t\tis_value $1 || BUG \"error: --t2 requires value\"\n +\t\t\tt2=\"$1\"\n +\t\t\tshift\n +\t\t\t;;\n -+\t\t-tk)\n ++\t\t--tk)\n +\t\t\tshift;\n -+\t\t\ttest \"$#\" -ne 0 || BUG \"error: -tk requires arg\"\n ++\t\t\tis_value $1 || BUG \"error: --tk requires value\"\n +\t\t\ttk=\"$1\"\n +\t\t\tshift\n +\t\t\t;;\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'edit some files' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n ++\tstart_daemon --tf \"$PWD/.git/trace\" &&\n +\n +\tedit_files &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'create some files' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n ++\tstart_daemon --tf \"$PWD/.git/trace\" &&\n +\n +\tcreate_files &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'delete some files' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n ++\tstart_daemon --tf \"$PWD/.git/trace\" &&\n +\n +\tdelete_files &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'rename some files' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n ++\tstart_daemon --tf \"$PWD/.git/trace\" &&\n +\n +\trename_files &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'rename directory' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n ++\tstart_daemon --tf \"$PWD/.git/trace\" &&\n +\n +\tmv dirtorename dirrenamed &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'file changes to directory' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n ++\tstart_daemon --tf \"$PWD/.git/trace\" &&\n +\n +\tfile_to_directory &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +test_expect_success 'directory changes to a file' '\n +\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n -+\tstart_daemon -tf \"$PWD/.git/trace\" &&\n ++\tstart_daemon --tf \"$PWD/.git/trace\" &&\n +\n +\tdirectory_to_file &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tgit init test_flush &&\n +\n -+\tstart_daemon -C test_flush -tf \"$PWD/.git/trace_daemon\" -tk true &&\n ++\tstart_daemon -C test_flush --tf \"$PWD/.git/trace_daemon\" --tk true &&\n +\n +\t# The daemon should have an initial token with no events in _0 and\n +\t# then a few (probably platform-specific number of) events in _1.\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\tgit -C wt-base worktree add ../wt-secondary &&\n +\n +\tstart_daemon -C wt-secondary \\\n -+\t\t-tf \"$PWD/trace_wt_secondary\" \\\n -+\t\t-t2 \"$PWD/trace2_wt_secondary\" &&\n ++\t\t--tf \"$PWD/trace_wt_secondary\" \\\n ++\t\t--t2 \"$PWD/trace2_wt_secondary\" &&\n +\n +\tgit -C wt-secondary fsmonitor--daemon stop &&\n +\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n 22: c925a9a7459 = 22: cc39ecf10ae t/perf: avoid copying builtin fsmonitor files into test repo\n 23: 5b3381c223e = 23: 2bb3eb84767 t/helper/test-chmtime: skip directories on Windows\n -: ----------- > 24: bab9a9b0802 t/perf/p7519: fix coding style\n 24: 803a540cc00 ! 25: 2dd06ad2f71 t/perf/p7519: speed up test on Windows\n @@ Commit message\n Signed-off-by: Junio C Hamano \n \n ## t/perf/p7519-fsmonitor.sh ##\n -@@ t/perf/p7519-fsmonitor.sh: then\n - \tfi\n - fi\n - \n --trace_start() {\n -+trace_start () {\n - \tif test -n \"$GIT_PERF_7519_TRACE\"\n - \tthen\n - \t\tname=\"$1\"\n -@@ t/perf/p7519-fsmonitor.sh: trace_start() {\n - \tfi\n - }\n - \n --trace_stop() {\n -+trace_stop () {\n - \tif test -n \"$GIT_PERF_7519_TRACE\"\n - \tthen\n - \t\tunset GIT_TRACE2_PERF\n +@@ t/perf/p7519-fsmonitor.sh: trace_stop () {\n \tfi\n }\n \n +touch_files () {\n -+\tn=$1\n -+\td=\"$n\"_files\n ++\tn=$1 &&\n ++\td=\"$n\"_files &&\n +\n -+\t(cd $d ; test_seq 1 $n | xargs touch )\n ++\t(cd $d && test_seq 1 $n | xargs touch )\n +}\n +\n test_expect_success \"one time repo setup\" '\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n \tgit commit -qm \"Add files\" &&\n \n -@@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n - \tfi\n - '\n - \n --setup_for_fsmonitor() {\n -+setup_for_fsmonitor () {\n - \t# set INTEGRATION_SCRIPT depending on the environment\n - \tif test -n \"$INTEGRATION_PATH\"\n - \tthen\n -@@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n - \ttest_perf \"$@\"\n - }\n - \n --test_fsmonitor_suite() {\n -+test_fsmonitor_suite () {\n - \tif test -n \"$INTEGRATION_SCRIPT\"; then\n - \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n - \telse\n -@@ t/perf/p7519-fsmonitor.sh: test_fsmonitor_suite() {\n +@@ t/perf/p7519-fsmonitor.sh: test_fsmonitor_suite () {\n \n \t# Update the mtimes on upto 100k files to make status think\n \t# that they are dirty. For simplicity, omit any files with\n 25: d5ca2df31c8 = 26: 6eaa5765ae1 t/perf/p7519: add fsmonitor--daemon test cases\n 26: 42631259e89 = 27: 30957f3930e fsmonitor--daemon: periodically truncate list of modified files\n 27: f256c3cbe8b = 28: c8ca2a17277 fsmonitor--daemon: use a cookie file to sync with file system\n 28: 08af8296f96 = 29: 4caf1d89b84 fsmonitor: force update index after large responses\n 29: e6cf84dc8eb ! 30: f87a1eba693 t7527: test status with untracked-cache and fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n +\tfsm=$2 &&\n +\tfn=$3 &&\n +\n ++\tif test $uc = true && test $fsm = false\n ++\tthen\n ++\t\t# The untracked-cache is buggy when FSMonitor is\n ++\t\t# DISABLED, so skip the tests for this matrix\n ++\t\t# combination.\n ++\t\t#\n ++\t\t# We've observed random, occasional test failures on\n ++\t\t# Windows and MacOS when the UC is turned on and FSM\n ++\t\t# is turned off. These are rare, but they do happen\n ++\t\t# indicating that it is probably a race condition within\n ++\t\t# the untracked cache itself.\n ++\t\t#\n ++\t\t# It usually happens when a test does F/D trickery and\n ++\t\t# then the NEXT test fails because of extra status\n ++\t\t# output from stale UC data from the previous test.\n ++\t\t#\n ++\t\t# Since FSMonitor is not involved in the error, skip\n ++\t\t# the tests for this matrix combination.\n ++\t\t#\n ++\t\treturn 0\n ++\tfi &&\n ++\n +\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n +\t\tmatrix_clean_up_repo &&\n +\t\t$fn &&\n\n-- \ngitgitgadget\n"},{"id":"452132","messageId":"ab68b944173cb8d4fdc0d7583a3d1dd11e5aef78.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 02/30] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:18Z","receivedAt":"2022-03-24T16:49:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 ++++++++++++++\n 3 files changed, 220 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 6f0b4b775fe..a19d850e716 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -907,6 +907,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..789e7397baa\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,171 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#else\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(_(\"fsmonitor--daemon is not running\"));\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(_(\"could not send '%s' command to fsmonitor--daemon\"), c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..b6a7067c3af\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"452133","messageId":"e04c7301f249a75a33e5498258a0dc628f3ebe66.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 03/30] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:19Z","receivedAt":"2022-03-24T16:49:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove fsmonitor config settings to a new and opaque\n`struct fsmonitor_settings` structure. Add a lazily-loaded pointer\nto this into `struct repo_settings`\n\nCreate an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\nrepresent the state of fsmonitor. This lets us represent which, if\nany, fsmonitor provider (hook or IPC) is enabled.\n\nCreate `fsm_settings__get_*()` getters to lazily look up fsmonitor-\nrelated config settings.\n\nGet rid of the `core_fsmonitor` global variable. Move the code to\nlookup the existing `core.fsmonitor` config value into the fsmonitor\nsettings.\n\nCreate a hook pathname variable in `struct fsmonitor-settings` and\nonly set it when in hook mode.\n\nExtend the definition of `core.fsmonitor` to be either a boolean\nor a hook pathname. When true, the builtin FSMonitor is used.\nWhen false or unset, no FSMonitor (neither builtin nor hook) is\nused.\n\nThe existing `core_fsmonitor` global variable was used to store the\npathname to the fsmonitor hook *and* it was used as a boolean to see\nif fsmonitor was enabled. This dual usage and global visibility leads\nto confusion when we add the IPC-based provider. So lets hide the\ndetails in fsmonitor-settings.c and let it decide which provider to\nuse in the case of multiple settings. This avoids cluttering up\nrepo-settings.c with these private details.\n\nA future commit in builtin-fsmonitor series will add the ability to\ndisqualify worktrees for various reasons, such as being mounted from a\nremote volume, where fsmonitor should not be started. Having the\nconfig settings hidden in fsmonitor-settings.c allows such worktree\nrestrictions to override the config values used.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n builtin/update-index.c | 7 ++-\n cache.h | 1 -\n config.c | 14 -----\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-settings.c | 114 +++++++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 21 ++++++++\n fsmonitor.c | 63 ++++++++++++++---------\n fsmonitor.h | 15 ++++--\n repository.h | 3 ++\n t/README | 4 +-\n 12 files changed, 196 insertions(+), 49 deletions(-)\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\ndiff --git a/Makefile b/Makefile\nindex a19d850e716..707a56d4c11 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -908,6 +908,7 @@ LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += fsmonitor-ipc.o\n+LIB_OBJS += fsmonitor-settings.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex aafe7eeac2a..876112abb21 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1236,14 +1236,17 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\n \t\t\t\t\"enable fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode > FSMONITOR_MODE_DISABLED)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex 04d4d2db25c..aaf334e2aa4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -999,7 +999,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex 383b1a4885b..3f9b0739a78 100644\n--- a/config.c\n+++ b/config.c\n@@ -2626,20 +2626,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex bb49baf1ee0..7654f61c634 100644\n--- a/config.h\n+++ b/config.h\n@@ -597,7 +597,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a5..00682e638d7 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nnew file mode 100644\nindex 00000000000..757d230d538\n--- /dev/null\n+++ b/fsmonitor-settings.c\n@@ -0,0 +1,114 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+/*\n+ * We keep this structure defintion private and have getters\n+ * for all fields so that we can lazy load it as needed.\n+ */\n+struct fsmonitor_settings {\n+\tenum fsmonitor_mode mode;\n+\tchar *hook_path;\n+};\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s;\n+\tconst char *const_str;\n+\tint bool_value;\n+\n+\tif (r->settings.fsmonitor)\n+\t\treturn;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\n+\tr->settings.fsmonitor = s;\n+\n+\t/*\n+\t * Overload the existing \"core.fsmonitor\" config setting (which\n+\t * has historically been either unset or a hook pathname) to\n+\t * now allow a boolean value to enable the builtin FSMonitor\n+\t * or to turn everything off. (This does imply that you can't\n+\t * use a hook script named \"true\" or \"false\", but that's OK.)\n+\t */\n+\tswitch (repo_config_get_maybe_bool(r, \"core.fsmonitor\", &bool_value)) {\n+\n+\tcase 0: /* config value was set to */\n+\t\tif (bool_value)\n+\t\t\tfsm_settings__set_ipc(r);\n+\t\treturn;\n+\n+\tcase 1: /* config value was unset */\n+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n+\t\tbreak;\n+\n+\tcase -1: /* config value set to an arbitrary string */\n+\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n+\t\t\treturn; /* should not happen */\n+\t\tbreak;\n+\n+\tdefault: /* should not happen */\n+\t\treturn;\n+\t}\n+\n+\tif (!const_str || !*const_str)\n+\t\treturn;\n+\n+\tfsm_settings__set_hook(r, const_str);\n+}\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->mode;\n+}\n+\n+const char *fsm_settings__get_hook_path(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->hook_path;\n+}\n+\n+void fsm_settings__set_ipc(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\n+\n+void fsm_settings__set_hook(struct repository *r, const char *path)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+\tr->settings.fsmonitor->hook_path = strdup(path);\n+}\n+\n+void fsm_settings__set_disabled(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nnew file mode 100644\nindex 00000000000..a4c5d7b4889\n--- /dev/null\n+++ b/fsmonitor-settings.h\n@@ -0,0 +1,21 @@\n+#ifndef FSMONITOR_SETTINGS_H\n+#define FSMONITOR_SETTINGS_H\n+\n+struct repository;\n+\n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n+\tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r);\n+void fsm_settings__set_hook(struct repository *r, const char *path);\n+void fsm_settings__set_disabled(struct repository *r);\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+const char *fsm_settings__get_hook_path(struct repository *r);\n+\n+struct fsmonitor_settings;\n+\n+#endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 448d0ee33f5..0e961b74d82 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tstrvec_push(&cp.args, fsm_settings__get_hook_path(r));\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -225,17 +229,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \tchar *buf;\n \tunsigned int i;\n \tint is_trivial = 0;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (fsm_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -243,13 +258,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -280,7 +296,8 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t\tif (query_success)\n \t\t\t\tis_trivial = query_result.buf[0] == '/';\n@@ -290,9 +307,12 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n \t\t\t\t\t \"query/trivial-response\", 1);\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tfsm_settings__get_hook_path(r));\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t fsm_settings__get_hook_path(r),\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/*\n@@ -429,7 +449,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tint fsmonitor_enabled = (fsm_settings__get_mode(istate->repo)\n+\t\t\t\t > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -449,16 +470,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..3f41f653691 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -3,6 +3,7 @@\n \n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"fsmonitor-settings.h\"\n \n extern struct trace_key trace_fsmonitor;\n \n@@ -57,7 +58,10 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +71,10 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +90,9 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repository.h b/repository.h\nindex ca837cb9e91..9bbb4659cc8 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n #include \"path.h\"\n \n struct config_set;\n+struct fsmonitor_settings;\n struct git_hash_algo;\n struct index_state;\n struct lock_file;\n@@ -35,6 +36,8 @@ struct repo_settings {\n \tint command_requires_full_index;\n \tint sparse_index;\n \n+\tstruct fsmonitor_settings *fsmonitor; /* lazily loaded */\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex f48e0542cdc..9ffea1d3147 100644\n--- a/t/README\n+++ b/t/README\n@@ -405,8 +405,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code paths for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\n-- \ngitgitgadget\n\n"},{"id":"452134","messageId":"ea02ba25d8f0e5e76a646d23306eb6a2b312ebe2.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 04/30] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:20Z","receivedAt":"2022-03-24T16:50:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.fsmonitor` is set to true.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 38 ++++++++++++++++++++++++++++++++++++--\n 1 file changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 0e961b74d82..a38b5710eb3 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -241,8 +241,41 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (fsm_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t\tis_trivial = query_result.buf[bol] == '/';\n+\t\t\tif (is_trivial)\n+\t\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t\t \"query/trivial-response\", 1);\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\tgoto apply_results;\n \t}\n \n \tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n@@ -315,6 +348,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/*\n \t * The response from FSMonitor (excluding the header token) is\n \t * either:\n-- \ngitgitgadget\n\n"},{"id":"452135","messageId":"6ab7db9cb769fe44db2c76e82da0b5d898538ec0.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 05/30] fsmonitor: document builtin fsmonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:21Z","receivedAt":"2022-03-24T16:50:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDocument how `core.fsmonitor` can be set to a boolean to enable\nor disable the builtin FSMonitor.\n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to refer to it.\n\nCreate `git-fsmonitor--daemon` manual page and describe its features.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Documentation/config/core.txt | 60 +++++++++++++++-----\n Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n Documentation/git-update-index.txt | 8 ++-\n 3 files changed, 126 insertions(+), 17 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..6303c36c7ed 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,54 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+\tOtherwise, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note that if you concurrently use multiple versions of Git, such\n+as one version on the command line and another version in an IDE\n+tool, that the definition of `core.fsmonitor` was extended to\n+allow boolean values in addition to hook pathnames. Git versions\n+2.35.1 and prior will not understand the boolean values and will\n+consider the \"true\" or \"false\" values as hook pathnames to be\n+invoked. Git versions 2.26 thru 2.35.1 default to hook protocol\n+V2 and will fall back to no fsmonitor (full scan). Git versions\n+prior to 2.26 default to hook protocol V1 and will silently\n+assume there were no changes to report (no scan), so status\n+commands may report incomplete results. For this reason, it is\n+best to upgrade all of your Git versions before using the built-in\n+file system monitor.\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\nnew file mode 100644\nindex 00000000000..0fedf5a4565\n--- /dev/null\n+++ b/Documentation/git-fsmonitor--daemon.txt\n@@ -0,0 +1,75 @@\n+git-fsmonitor--daemon(1)\n+========================\n+\n+NAME\n+----\n+git-fsmonitor--daemon - A Built-in File System Monitor\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git fsmonitor--daemon' start\n+'git fsmonitor--daemon' run\n+'git fsmonitor--daemon' stop\n+'git fsmonitor--daemon' status\n+\n+DESCRIPTION\n+-----------\n+\n+A daemon to watch the working directory for file and directory\n+changes using platform-specific file system notification facilities.\n+\n+This daemon communicates directly with commands like `git status`\n+using the link:technical/api-simple-ipc.html[simple IPC] interface\n+instead of the slower linkgit:githooks[5] interface.\n+\n+This daemon is built into Git so that no third-party tools are\n+required.\n+\n+OPTIONS\n+-------\n+\n+start::\n+\tStarts a daemon in the background.\n+\n+run::\n+\tRuns a daemon in the foreground.\n+\n+stop::\n+\tStops the daemon running in the current working\n+\tdirectory, if present.\n+\n+status::\n+\tExits with zero status if a daemon is watching the\n+\tcurrent working directory.\n+\n+REMARKS\n+-------\n+\n+This daemon is a long running process used to watch a single working\n+directory and maintain a list of the recently changed files and\n+directories. Performance of commands such as `git status` can be\n+increased if they just ask for a summary of changes to the working\n+directory and can avoid scanning the disk.\n+\n+When `core.fsmonitor` is set to `true` (see linkgit:git-config[1])\n+commands, such as `git status`, will ask the daemon for changes and\n+automatically start it (if necessary).\n+\n+For more information see the \"File System Monitor\" section in\n+linkgit:git-update-index[1].\n+\n+CAVEATS\n+-------\n+\n+The fsmonitor daemon does not currently know about submodules and does\n+not know to filter out file system events that happen within a\n+submodule. If fsmonitor daemon is watching a super repo and a file is\n+modified within the working directory of a submodule, it will report\n+the change (as happening against the super repo). However, the client\n+will properly ignore these extra events, so performance may be affected\n+but it will not cause an incorrect result.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..53ea48a04e2 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -509,8 +511,8 @@ looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n+linkgit:git-config[1]) than using the `--fsmonitor` option to `git\n+update-index` in each repository, especially if you want to do so\n across all repositories you use, because you can set the configuration\n variable in your `$HOME/.gitconfig` just once and have it affect all\n repositories you touch.\n-- \ngitgitgadget\n\n"},{"id":"452136","messageId":"0ce8ae3f2cf07bfffdb1cb14454768d2b6d84bf2.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 06/30] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:22Z","receivedAt":"2022-03-24T16:50:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a built-in file system monitoring daemon that can be used by\nthe existing `fsmonitor` feature (protocol API and index extension)\nto improve the performance of various Git commands, such as `status`.\n\nThe `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\nprovides an alternative to hook access to existing fsmonitors such\nas `watchman`.\n\nThis commit merely adds the new command without any functionality.\n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 46 +++++++++++++++++++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 50 insertions(+)\n create mode 100644 builtin/fsmonitor--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex f817c509ec0..e81de1063a4 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -72,6 +72,7 @@\n /git-format-patch\n /git-fsck\n /git-fsck-objects\n+/git-fsmonitor--daemon\n /git-gc\n /git-get-tar-commit-id\n /git-grep\ndiff --git a/Makefile b/Makefile\nindex 707a56d4c11..5af1d5b112e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1114,6 +1114,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n+BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex 83379f3832c..40e9ecc8485 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\n int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nnew file mode 100644\nindex 00000000000..f0498793379\n--- /dev/null\n+++ b/builtin/fsmonitor--daemon.c\n@@ -0,0 +1,46 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"simple-ipc.h\"\n+#include \"khash.h\"\n+\n+static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tNULL\n+};\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *subcmd;\n+\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\tif (argc != 1)\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\tsubcmd = argv[0];\n+\n+\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n+}\n+\n+#else\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n+}\n+#endif\ndiff --git a/git.c b/git.c\nindex a25940d72e8..3d8e48cf555 100644\n--- a/git.c\n+++ b/git.c\n@@ -537,6 +537,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"452137","messageId":"4624ce2fa471b3ebb3a08baf0de2609447eb9fe3.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 07/30] fsmonitor--daemon: implement 'stop' and 'status' commands","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:23Z","receivedAt":"2022-03-24T16:50:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `stop` and `status` client commands to control and query the\nstatus of a `fsmonitor--daemon` server process (and implicitly start a\nserver process if necessary).\n\nLater commits will implement the actual server and monitor the file\nsystem.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 51 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex f0498793379..5e3178b8bdd 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,10 +7,55 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon stop\"),\n+\tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n+ * and wait for it to shutdown.\n+ */\n+static int do_as_client__send_stop(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"quit\", &answer);\n+\n+\t/* The quit command does not return any response data. */\n+\tstrbuf_release(&answer);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\twhile (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tsleep_millisec(50);\n+\ttrace2_region_leave(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\n+\treturn 0;\n+}\n+\n+static int do_as_client__status(void)\n+{\n+\tenum ipc_active_state state = fsmonitor_ipc__get_state();\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tprintf(_(\"fsmonitor-daemon is watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tprintf(_(\"fsmonitor-daemon is not watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 1;\n+\t}\n+}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n@@ -28,6 +73,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (!strcmp(subcmd, \"stop\"))\n+\t\treturn !!do_as_client__send_stop();\n+\n+\tif (!strcmp(subcmd, \"status\"))\n+\t\treturn !!do_as_client__status();\n+\n \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"452138","messageId":"a29fe7266a456d88125ca1f87d4b16370caf486f.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 08/30] compat/fsmonitor/fsm-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:24Z","receivedAt":"2022-03-24T16:50:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty filesystem listener backend for fsmonitor--daemon on Windows.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 13 ++++++++\n compat/fsmonitor/fsm-listen-win32.c | 21 +++++++++++++\n compat/fsmonitor/fsm-listen.h | 49 +++++++++++++++++++++++++++++\n config.mak.uname | 10 ++++++\n contrib/buildsystems/CMakeLists.txt | 7 +++++\n repo-settings.c | 1 +\n 6 files changed, 101 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n\ndiff --git a/Makefile b/Makefile\nindex 5af1d5b112e..26567d4f772 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -470,6 +470,11 @@ all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n+# If your platform supports a built-in fsmonitor backend, set\n+# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n+# `compat/fsmonitor/fsm-listen-.c` that implements the\n+# `fsm_listen__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1968,6 +1973,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n \tCOMPAT_OBJS += compat/access.o\n endif\n \n+ifdef FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2887,6 +2897,9 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n \t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n+ifdef FSMONITOR_DAEMON_BACKEND\n+\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nnew file mode 100644\nindex 00000000000..916cbea254f\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nnew file mode 100644\nindex 00000000000..f0539349baf\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -0,0 +1,49 @@\n+#ifndef FSM_LISTEN_H\n+#define FSM_LISTEN_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * fsmonitor_fs_listener thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread AFTER joining the listener.\n+ */\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to watch for\n+ * filesystem events. This will run in the fsmonitor_fs_listen thread.\n+ *\n+ * It should call `ipc_server_stop_async()` if the listener thread\n+ * prematurely terminates (because of a filesystem error or if it\n+ * detects that the .git directory has been deleted). (It should NOT\n+ * do so if the listener thread receives a normal shutdown signal from\n+ * the IPC layer.)\n+ *\n+ * It should set `state->error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the fsmonitor listener thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_LISTEN_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 4352ea39e9b..26074f56bed 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -435,6 +435,11 @@ ifeq ($(uname_S),Windows)\n \t# so we don't need this:\n \t#\n \t# SNPRINTF_RETURNS_BOGUS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -619,6 +624,11 @@ ifeq ($(uname_S),MINGW)\n \tNO_STRTOUMAX = YesPlease\n \tNO_MKDTEMP = YesPlease\n \tNO_SVN_TESTS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex e44232f85d3..0963629db7f 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -285,6 +285,13 @@ else()\n \tendif()\n endif()\n \n+if(SUPPORTS_SIMPLE_IPC)\n+\tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\tendif()\n+endif()\n+\n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n \n #header checks\ndiff --git a/repo-settings.c b/repo-settings.c\nindex b4fbd16cdcc..2dfcb2b6542 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"midx.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n \n static void repo_cfg_bool(struct repository *r, const char *key, int *dest,\n \t\t\t int def)\n-- \ngitgitgadget\n\n"},{"id":"452139","messageId":"a6a39a3306df2602e06cc3d9a755fa247f8d41b0.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 11/30] fsmonitor--daemon: implement 'start' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:27Z","receivedAt":"2022-03-24T16:50:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement 'git fsmonitor--daemon start' command. This command starts\nan instance of 'git fsmonitor--daemon run' in the background using\nthe new 'start_bg_command()' function.\n\nWe avoid the fork-and-call technique on Unix systems in favor of a\nfork-and-exec technique. This gives us more uniform Trace2 child-*\nevents. It also makes our usage more consistent with Windows usage.\n\nOn Windows, teach 'git fsmonitor--daemon run' to optionally call\n'FreeConsole()' to release handles to the inherited Win32 console\n(despite being passed invalid handles for stdin/out/err). Without\nthis, command prompts and powershell terminal windows could hang\nin \"exit\" until the last background child process exited or released\ntheir Win32 console handle. (This was not seen with git-bash shells\nbecause they don't have a Win32 console attached to them.)\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 109 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 107 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 5591339399a..69dd39121a3 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -9,6 +9,7 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon start []\"),\n \tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n@@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n #define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n static int fsmonitor__ipc_threads = 8;\n \n+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n+static int fsmonitor__start_timeout_sec = 60;\n+\n #define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n static int fsmonitor__announce_startup = 0;\n \n@@ -36,6 +40,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n+\t\tfsmonitor__start_timeout_sec = i;\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n \t\tint is_bool;\n \t\tint i = git_config_bool_or_int(var, value, &is_bool);\n@@ -250,7 +263,7 @@ done:\n \treturn err;\n }\n \n-static int try_to_run_foreground_daemon(void)\n+static int try_to_run_foreground_daemon(int detach_console)\n {\n \t/*\n \t * Technically, we don't need to probe for an existing daemon\n@@ -270,17 +283,106 @@ static int try_to_run_foreground_daemon(void)\n \t\tfflush(stderr);\n \t}\n \n+#ifdef GIT_WINDOWS_NATIVE\n+\tif (detach_console)\n+\t\tFreeConsole();\n+#endif\n+\n \treturn !!fsmonitor_run_daemon();\n }\n \n+static start_bg_wait_cb bg_wait_cb;\n+\n+static int bg_wait_cb(const struct child_process *cp, void *cb_data)\n+{\n+\tenum ipc_active_state s = fsmonitor_ipc__get_state();\n+\n+\tswitch (s) {\n+\tcase IPC_STATE__LISTENING:\n+\t\t/* child is \"ready\" */\n+\t\treturn 0;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\t/* give child more time */\n+\t\treturn 1;\n+\n+\tdefault:\n+\tcase IPC_STATE__INVALID_PATH:\n+\tcase IPC_STATE__OTHER_ERROR:\n+\t\t/* all the time in world won't help */\n+\t\treturn -1;\n+\t}\n+}\n+\n+static int try_to_start_background_daemon(void)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tenum start_bg_result sbgr;\n+\n+\t/*\n+\t * Before we try to create a background daemon process, see\n+\t * if a daemon process is already listening. This makes it\n+\t * easier for us to report an already-listening error to the\n+\t * console, since our spawn/daemon can only report the success\n+\t * of creating the background process (and not whether it\n+\t * immediately exited).\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(_(\"fsmonitor--daemon is already running '%s'\"),\n+\t\t the_repository->worktree);\n+\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"starting fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n+\n+\tcp.git_cmd = 1;\n+\n+\tstrvec_push(&cp.args, \"fsmonitor--daemon\");\n+\tstrvec_push(&cp.args, \"run\");\n+\tstrvec_push(&cp.args, \"--detach\");\n+\tstrvec_pushf(&cp.args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n+\n+\tcp.no_stdin = 1;\n+\tcp.no_stdout = 1;\n+\tcp.no_stderr = 1;\n+\n+\tsbgr = start_bg_command(&cp, bg_wait_cb, NULL,\n+\t\t\t\tfsmonitor__start_timeout_sec);\n+\n+\tswitch (sbgr) {\n+\tcase SBGR_READY:\n+\t\treturn 0;\n+\n+\tdefault:\n+\tcase SBGR_ERROR:\n+\tcase SBGR_CB_ERROR:\n+\t\treturn error(_(\"daemon failed to start\"));\n+\n+\tcase SBGR_TIMEOUT:\n+\t\treturn error(_(\"daemon not online yet\"));\n+\n+\tcase SBGR_DIED:\n+\t\treturn error(_(\"daemon terminated\"));\n+\t}\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tint detach_console = 0;\n \n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"detach\", &detach_console, N_(\"detach from console\")),\n \t\tOPT_INTEGER(0, \"ipc-threads\",\n \t\t\t &fsmonitor__ipc_threads,\n \t\t\t N_(\"use ipc worker threads\")),\n+\t\tOPT_INTEGER(0, \"start-timeout\",\n+\t\t\t &fsmonitor__start_timeout_sec,\n+\t\t\t N_(\"max seconds to wait for background daemon startup\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -296,8 +398,11 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tif (!strcmp(subcmd, \"start\"))\n+\t\treturn !!try_to_start_background_daemon();\n+\n \tif (!strcmp(subcmd, \"run\"))\n-\t\treturn !!try_to_run_foreground_daemon();\n+\t\treturn !!try_to_run_foreground_daemon(detach_console);\n \n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n-- \ngitgitgadget\n\n"},{"id":"452140","messageId":"f07800690eeda000271b05435aa032ba17b35086.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 10/30] fsmonitor--daemon: implement 'run' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:26Z","receivedAt":"2022-03-24T16:50:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `run` command to try to begin listening for file system events.\n\nThis version defines the thread structure with a single fsmonitor_fs_listen\nthread to watch for file system events and a simple IPC thread pool to\nwatch for connection from Git clients over a well-known named pipe or\nUnix domain socket.\n\nThis commit does not actually do anything yet because the platform\nbackends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 34 ++++++\n 2 files changed, 261 insertions(+), 1 deletion(-)\n create mode 100644 fsmonitor--daemon.h\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 5e3178b8bdd..5591339399a 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,16 +3,52 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Global state loaded from config.\n+ */\n+#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n+static int fsmonitor__ipc_threads = 8;\n+\n+#define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n+static int fsmonitor__announce_startup = 0;\n+\n+static int fsmonitor_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 1)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__IPC_THREADS, i);\n+\t\tfsmonitor__ipc_threads = i;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n+\t\tint is_bool;\n+\t\tint i = git_config_bool_or_int(var, value, &is_bool);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' not bool or int: %d\"),\n+\t\t\t\t var, i);\n+\t\tfsmonitor__announce_startup = i;\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n /*\n * Acting as a CLIENT.\n *\n@@ -57,15 +93,198 @@ static int do_as_client__status(void)\n \t}\n }\n \n+static ipc_server_application_cb handle_client;\n+\n+static int handle_client(void *data,\n+\t\t\t const char *command, size_t command_len,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\t/* struct fsmonitor_daemon_state *state = data; */\n+\tint result;\n+\n+\t/*\n+\t * The Simple IPC API now supports {char*, len} arguments, but\n+\t * FSMonitor always uses proper null-terminated strings, so\n+\t * we can ignore the command_len argument. (Trust, but verify.)\n+\t */\n+\tif (command_len != strlen(command))\n+\t\tBUG(\"FSMonitor assumes text messages\");\n+\n+\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n+\n+\tresult = 0; /* TODO Do something here. */\n+\n+\ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n+\n+\treturn result;\n+}\n+\n+static void *fsm_listen__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-listen\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Watching: worktree '%s'\",\n+\t\t\t state->path_worktree_watch.buf);\n+\tif (state->nr_paths_watching > 1)\n+\t\ttrace_printf_key(&trace_fsmonitor, \"Watching: gitdir '%s'\",\n+\t\t\t\t state->path_gitdir_watch.buf);\n+\n+\tfsm_listen__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct ipc_server_opts ipc_opts = {\n+\t\t.nr_threads = fsmonitor__ipc_threads,\n+\n+\t\t/*\n+\t\t * We know that there are no other active threads yet,\n+\t\t * so we can let the IPC layer temporarily chdir() if\n+\t\t * it needs to when creating the server side of the\n+\t\t * Unix domain socket.\n+\t\t */\n+\t\t.uds_disallow_chdir = 0\n+\t};\n+\n+\t/*\n+\t * Start the IPC thread pool before the we've started the file\n+\t * system event listener thread so that we have the IPC handle\n+\t * before we need it.\n+\t */\n+\tif (ipc_server_run_async(&state->ipc_server_data,\n+\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t handle_client, state))\n+\t\treturn error_errno(\n+\t\t\t_(\"could not start IPC thread pool on '%s'\"),\n+\t\t\tfsmonitor_ipc__get_path());\n+\n+\t/*\n+\t * Start the fsmonitor listener thread to collect filesystem\n+\t * events.\n+\t */\n+\tif (pthread_create(&state->listener_thread, NULL,\n+\t\t\t fsm_listen__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\tipc_server_await(state->ipc_server_data);\n+\n+\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t}\n+\n+\t/*\n+\t * The daemon is now fully functional in background threads.\n+\t * Wait for the IPC thread pool to shutdown (whether by client\n+\t * request or from filesystem activity).\n+\t */\n+\tipc_server_await(state->ipc_server_data);\n+\n+\t/*\n+\t * The fsmonitor listener thread may have received a shutdown\n+\t * event from the IPC thread pool, but it doesn't hurt to tell\n+\t * it again. And wait for it to shutdown.\n+\t */\n+\tfsm_listen__stop_async(state);\n+\tpthread_join(state->listener_thread, NULL);\n+\n+\treturn state->error_code;\n+}\n+\n+static int fsmonitor_run_daemon(void)\n+{\n+\tstruct fsmonitor_daemon_state state;\n+\tint err;\n+\n+\tmemset(&state, 0, sizeof(state));\n+\n+\tpthread_mutex_init(&state.main_lock, NULL);\n+\tstate.error_code = 0;\n+\tstate.current_token_data = NULL;\n+\n+\t/* Prepare to (recursively) watch the directory. */\n+\tstrbuf_init(&state.path_worktree_watch, 0);\n+\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n+\tstate.nr_paths_watching = 1;\n+\n+\t/*\n+\t * We create and delete cookie files somewhere inside the .git\n+\t * directory to help us keep sync with the file system. If\n+\t * \".git\" is not a directory, then is not inside the\n+\t * cone of , so set up a second watch to watch\n+\t * the so that we get events for the cookie files.\n+\t */\n+\tstrbuf_init(&state.path_gitdir_watch, 0);\n+\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n+\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n+\tif (!is_directory(state.path_gitdir_watch.buf)) {\n+\t\tstrbuf_reset(&state.path_gitdir_watch);\n+\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n+\t\tstate.nr_paths_watching = 2;\n+\t}\n+\n+\t/*\n+\t * Confirm that we can create platform-specific resources for the\n+\t * filesystem listener before we bother starting all the threads.\n+\t */\n+\tif (fsm_listen__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize listener thread\"));\n+\t\tgoto done;\n+\t}\n+\n+\terr = fsmonitor_run_daemon_1(&state);\n+\n+done:\n+\tpthread_mutex_destroy(&state.main_lock);\n+\tfsm_listen__dtor(&state);\n+\n+\tipc_server_free(state.ipc_server_data);\n+\n+\tstrbuf_release(&state.path_worktree_watch);\n+\tstrbuf_release(&state.path_gitdir_watch);\n+\n+\treturn err;\n+}\n+\n+static int try_to_run_foreground_daemon(void)\n+{\n+\t/*\n+\t * Technically, we don't need to probe for an existing daemon\n+\t * process, since we could just call `fsmonitor_run_daemon()`\n+\t * and let it fail if the pipe/socket is busy.\n+\t *\n+\t * However, this method gives us a nicer error message for a\n+\t * common error case.\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(_(\"fsmonitor--daemon is already running '%s'\"),\n+\t\t the_repository->worktree);\n+\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"running fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n+\n+\treturn !!fsmonitor_run_daemon();\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n \n \tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"ipc-threads\",\n+\t\t\t &fsmonitor__ipc_threads,\n+\t\t\t N_(\"use ipc worker threads\")),\n \t\tOPT_END()\n \t};\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(fsmonitor_config, NULL);\n \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n@@ -73,6 +292,13 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (fsmonitor__ipc_threads < 1)\n+\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n+\t\t fsmonitor__ipc_threads);\n+\n+\tif (!strcmp(subcmd, \"run\"))\n+\t\treturn !!try_to_run_foreground_daemon();\n+\n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n \ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nnew file mode 100644\nindex 00000000000..3009c1a83de\n--- /dev/null\n+++ b/fsmonitor--daemon.h\n@@ -0,0 +1,34 @@\n+#ifndef FSMONITOR_DAEMON_H\n+#define FSMONITOR_DAEMON_H\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+#include \"thread-utils.h\"\n+\n+struct fsmonitor_batch;\n+struct fsmonitor_token_data;\n+\n+struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+\n+struct fsmonitor_daemon_state {\n+\tpthread_t listener_thread;\n+\tpthread_mutex_t main_lock;\n+\n+\tstruct strbuf path_worktree_watch;\n+\tstruct strbuf path_gitdir_watch;\n+\tint nr_paths_watching;\n+\n+\tstruct fsmonitor_token_data *current_token_data;\n+\n+\tint error_code;\n+\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\n+\tstruct ipc_server_data *ipc_server_data;\n+};\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"452141","messageId":"d62e338d00829bf3fe08e649ffa298c087a354a9.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 12/30] fsmonitor--daemon: add pathname classification","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:28Z","receivedAt":"2022-03-24T16:50:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to classify relative and absolute\npathnames and decide how they should be handled. This will\nbe used by the platform-specific backend to respond to each\nfilesystem event.\n\nWhen we register for filesystem notifications on a directory,\nwe get events for everything (recursively) in the directory.\nWe want to report to clients changes to tracked and untracked\npaths within the working directory proper. We do not want to\nreport changes within the .git directory, for example.\n\nThis classification will be used in a later commit by the\ndifferent backends to classify paths as events are received.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 81 ++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 87 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 168 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 69dd39121a3..1ce00b7c150 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -134,6 +134,87 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n+#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *rel)\n+{\n+\tif (fspathncmp(rel, \".git\", 4))\n+\t\treturn IS_WORKDIR_PATH;\n+\trel += 4;\n+\n+\tif (!*rel)\n+\t\treturn IS_DOT_GIT;\n+\tif (*rel != '/')\n+\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n+\trel++;\n+\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_DOT_GIT;\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *rel)\n+{\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_GITDIR;\n+}\n+\n+static enum fsmonitor_path_type try_classify_workdir_abs_path(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\n+\tif (fspathncmp(path, state->path_worktree_watch.buf,\n+\t\t state->path_worktree_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_worktree_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_workdir_relative(rel);\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\tenum fsmonitor_path_type t;\n+\n+\tt = try_classify_workdir_abs_path(state, path);\n+\tif (state->nr_paths_watching == 1)\n+\t\treturn t;\n+\tif (t != IS_OUTSIDE_CONE)\n+\t\treturn t;\n+\n+\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n+\t\t state->path_gitdir_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_gitdir_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_GITDIR; /* it is the exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_gitdir_relative(rel);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 3009c1a83de..8c3a71a48bd 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -30,5 +30,92 @@ struct fsmonitor_daemon_state {\n \tstruct ipc_server_data *ipc_server_data;\n };\n \n+/*\n+ * Pathname classifications.\n+ *\n+ * The daemon classifies the pathnames that it receives from file\n+ * system notification events into the following categories and uses\n+ * that to decide whether clients are told about them. (And to watch\n+ * for file system synchronization events.)\n+ *\n+ * The daemon only collects and reports on the set of modified paths\n+ * within the working directory (proper).\n+ *\n+ * The client should only care about paths within the working\n+ * directory proper (inside the working directory and not \".git\" nor\n+ * inside of \".git/\"). That is, the client has read the index and is\n+ * asking for a list of any paths in the working directory that have\n+ * been modified since the last token. The client does not care about\n+ * file system changes within the \".git/\" directory (such as new loose\n+ * objects or packfiles). So the client will only receive paths that\n+ * are classified as IS_WORKDIR_PATH.\n+ *\n+ * Note that \".git\" is usually a directory and is therefore inside\n+ * the cone of the FS watch that we have on the working directory root,\n+ * so we will also get FS events for disk activity on and within \".git/\"\n+ * that we need to respond to or filter from the client.\n+ *\n+ * But Git also allows \".git\" to be a *file* that points to a GITDIR\n+ * outside of the working directory. When this happens, we need to\n+ * create FS watches on both the working directory root *and* on the\n+ * (external) GITDIR root. (The latter is required because we put\n+ * cookie files inside it and use them to sync with the FS event\n+ * stream.)\n+ *\n+ * Note that in the context of this discussion, I'm using \"GITDIR\"\n+ * to only mean an external GITDIR referenced by a \".git\" file.\n+ *\n+ * The platform FS event backends will receive watch-specific\n+ * relative paths (except for those OS's that always emit absolute\n+ * paths). We use the following enum and routines to classify each\n+ * path so that we know how to handle it. There is a slight asymmetry\n+ * here because \".git/\" is inside the working directory and the\n+ * (external) GITDIR is not, and therefore how we handle events may\n+ * vary slightly, so I have different enums for \"IS...DOT_GIT...\" and\n+ * \"IS...GITDIR...\".\n+ *\n+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n+ * exact \".git\" file/directory or GITDIR directory. If the daemon\n+ * receives a delete event for either of these paths, it will\n+ * automatically shutdown, for example.\n+ *\n+ * Note that the daemon DOES NOT explicitly watch nor special case the\n+ * index. The daemon does not read the index nor have any internal\n+ * index-relative state, so there are no \"IS...INDEX...\" enum values.\n+ */\n+enum fsmonitor_path_type {\n+\tIS_WORKDIR_PATH = 0,\n+\n+\tIS_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n+\n+\tIS_GITDIR,\n+\tIS_INSIDE_GITDIR,\n+\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n+\n+\tIS_OUTSIDE_CONE,\n+};\n+\n+/*\n+ * Classify a pathname relative to the root of the working directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify a pathname relative to a that is external to the\n+ * worktree directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify an absolute pathname received from a filesystem event.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"452142","messageId":"53e06b4ae5d0eb1d3c2cb6a61ffd14abc36f6534.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 13/30] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:29Z","receivedAt":"2022-03-24T16:50:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to create token-ids and define the\noverall token naming scheme.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 115 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1ce00b7c150..1c7c156288d 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -106,6 +106,120 @@ static int do_as_client__status(void)\n \t}\n }\n \n+/*\n+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n+ * \"token\" as a virtual timestamp. Clients can request a summary of all\n+ * created/deleted/modified files relative to a token. In the response,\n+ * clients receive a new token for the next (relative) request.\n+ *\n+ *\n+ * Token Format\n+ * ============\n+ *\n+ * The contents of the token are private and provider-specific.\n+ *\n+ * For the built-in fsmonitor--daemon, we define a token as follows:\n+ *\n+ * \"builtin\" \":\" \":\" \n+ *\n+ * The \"builtin\" prefix is used as a namespace to avoid conflicts\n+ * with other providers (such as Watchman).\n+ *\n+ * The is an arbitrary OPAQUE string, such as a GUID,\n+ * UUID, or {timestamp,pid}. It is used to group all filesystem\n+ * events that happened while the daemon was monitoring (and in-sync\n+ * with the filesystem).\n+ *\n+ * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n+ * and does not define less-than/greater-than relationships.\n+ * (There are too many race conditions to rely on file system\n+ * event timestamps.)\n+ *\n+ * The is a simple integer incremented whenever the\n+ * daemon needs to make its state public. For example, if 1000 file\n+ * system events come in, but no clients have requested the data,\n+ * the daemon can continue to accumulate file changes in the same\n+ * bin and does not need to advance the sequence number. However,\n+ * as soon as a client does arrive, the daemon needs to start a new\n+ * bin and increment the sequence number.\n+ *\n+ * The sequence number serves as the boundary between 2 sets\n+ * of bins -- the older ones that the client has already seen\n+ * and the newer ones that it hasn't.\n+ *\n+ * When a new is created, the is reset to\n+ * zero.\n+ *\n+ *\n+ * About Token Ids\n+ * ===============\n+ *\n+ * A new token_id is created:\n+ *\n+ * [1] each time the daemon is started.\n+ *\n+ * [2] any time that the daemon must re-sync with the filesystem\n+ * (such as when the kernel drops or we miss events on a very\n+ * active volume).\n+ *\n+ * [3] in response to a client \"flush\" command (for dropped event\n+ * testing).\n+ *\n+ * When a new token_id is created, the daemon is free to discard all\n+ * cached filesystem events associated with any previous token_ids.\n+ * Events associated with a non-current token_id will never be sent\n+ * to a client. A token_id change implicitly means that the daemon\n+ * has gap in its event history.\n+ *\n+ * Therefore, clients that present a token with a stale (non-current)\n+ * token_id will always be given a trivial response.\n+ */\n+struct fsmonitor_token_data {\n+\tstruct strbuf token_id;\n+\tstruct fsmonitor_batch *batch_head;\n+\tstruct fsmonitor_batch *batch_tail;\n+\tuint64_t client_ref_count;\n+};\n+\n+static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n+{\n+\tstatic int test_env_value = -1;\n+\tstatic uint64_t flush_count = 0;\n+\tstruct fsmonitor_token_data *token;\n+\n+\tCALLOC_ARRAY(token, 1);\n+\n+\tstrbuf_init(&token->token_id, 0);\n+\ttoken->batch_head = NULL;\n+\ttoken->batch_tail = NULL;\n+\ttoken->client_ref_count = 0;\n+\n+\tif (test_env_value < 0)\n+\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n+\n+\tif (!test_env_value) {\n+\t\tstruct timeval tv;\n+\t\tstruct tm tm;\n+\t\ttime_t secs;\n+\n+\t\tgettimeofday(&tv, NULL);\n+\t\tsecs = tv.tv_sec;\n+\t\tgmtime_r(&secs, &tm);\n+\n+\t\tstrbuf_addf(&token->token_id,\n+\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n+\t\t\t flush_count++,\n+\t\t\t getpid(),\n+\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n+\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n+\t\t\t (long)tv.tv_usec);\n+\t} else {\n+\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n+\t}\n+\n+\treturn token;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -298,7 +412,7 @@ static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n-\tstate.current_token_data = NULL;\n+\tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n \tstrbuf_init(&state.path_worktree_watch, 0);\n-- \ngitgitgadget\n\n"},{"id":"452143","messageId":"39f43fabe024917e6c84b8916197b0dc0add97fc.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 14/30] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:30Z","receivedAt":"2022-03-24T16:50:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to build a list of changed paths and associate\nthem with a token-id. This will be used by the platform-specific\nbackends to accumulate changed paths in response to filesystem events.\n\nThe platform-specific file system listener thread receives file system\nevents containing one or more changed pathnames (with whatever\nbucketing or grouping that is convenient for the file system). These\npaths are accumulated (without locking) by the file system layer into\na `fsmonitor_batch`.\n\nWhen the file system layer has drained the kernel event queue, it will\n\"publish\" them to our token queue and make them visible to concurrent\nclient worker threads. The token layer is free to combine and/or de-dup\npaths within these batches for efficient presentation to clients.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 230 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 40 +++++++\n 2 files changed, 268 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1c7c156288d..69312119b07 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -181,17 +181,27 @@ struct fsmonitor_token_data {\n \tuint64_t client_ref_count;\n };\n \n+struct fsmonitor_batch {\n+\tstruct fsmonitor_batch *next;\n+\tuint64_t batch_seq_nr;\n+\tconst char **interned_paths;\n+\tsize_t nr, alloc;\n+\ttime_t pinned_time;\n+};\n+\n static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n {\n \tstatic int test_env_value = -1;\n \tstatic uint64_t flush_count = 0;\n \tstruct fsmonitor_token_data *token;\n+\tstruct fsmonitor_batch *batch;\n \n \tCALLOC_ARRAY(token, 1);\n+\tbatch = fsmonitor_batch__new();\n \n \tstrbuf_init(&token->token_id, 0);\n-\ttoken->batch_head = NULL;\n-\ttoken->batch_tail = NULL;\n+\ttoken->batch_head = batch;\n+\ttoken->batch_tail = batch;\n \ttoken->client_ref_count = 0;\n \n \tif (test_env_value < 0)\n@@ -217,9 +227,143 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n \t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n \t}\n \n+\t/*\n+\t * We created a new and are starting a new series\n+\t * of tokens with a zero .\n+\t *\n+\t * Since clients cannot guess our new (non test) \n+\t * they will always receive a trivial response (because of the\n+\t * mismatch on the ). The trivial response will\n+\t * tell them our new so that subsequent requests\n+\t * will be relative to our new series. (And when sending that\n+\t * response, we pin the current head of the batch list.)\n+\t *\n+\t * Even if the client correctly guesses the , their\n+\t * request of \"builtin::0\" asks for all changes MORE\n+\t * RECENT than batch/bin 0.\n+\t *\n+\t * This implies that it is a waste to accumulate paths in the\n+\t * initial batch/bin (because they will never be transmitted).\n+\t *\n+\t * So the daemon could be running for days and watching the\n+\t * file system, but doesn't need to actually accumulate any\n+\t * paths UNTIL we need to set a reference point for a later\n+\t * relative request.\n+\t *\n+\t * However, it is very useful for testing to always have a\n+\t * reference point set. Pin batch 0 to force early file system\n+\t * events to accumulate.\n+\t */\n+\tif (test_env_value)\n+\t\tbatch->pinned_time = time(NULL);\n+\n \treturn token;\n }\n \n+struct fsmonitor_batch *fsmonitor_batch__new(void)\n+{\n+\tstruct fsmonitor_batch *batch;\n+\n+\tCALLOC_ARRAY(batch, 1);\n+\n+\treturn batch;\n+}\n+\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch)\n+{\n+\twhile (batch) {\n+\t\tstruct fsmonitor_batch *next = batch->next;\n+\n+\t\t/*\n+\t\t * The actual strings within the array of this batch\n+\t\t * are interned, so we don't own them. We only own\n+\t\t * the array.\n+\t\t */\n+\t\tfree(batch->interned_paths);\n+\t\tfree(batch);\n+\n+\t\tbatch = next;\n+\t}\n+}\n+\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n+\t\t\t const char *path)\n+{\n+\tconst char *interned_path = strintern(path);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n+\n+\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n+\tbatch->interned_paths[batch->nr++] = interned_path;\n+}\n+\n+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n+\t\t\t\t const struct fsmonitor_batch *batch_src)\n+{\n+\tsize_t k;\n+\n+\tALLOC_GROW(batch_dest->interned_paths,\n+\t\t batch_dest->nr + batch_src->nr + 1,\n+\t\t batch_dest->alloc);\n+\n+\tfor (k = 0; k < batch_src->nr; k++)\n+\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n+\t\t\tbatch_src->interned_paths[k];\n+}\n+\n+static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n+{\n+\tif (!token)\n+\t\treturn;\n+\n+\tassert(token->client_ref_count == 0);\n+\n+\tstrbuf_release(&token->token_id);\n+\n+\tfsmonitor_batch__free_list(token->batch_head);\n+\n+\tfree(token);\n+}\n+\n+/*\n+ * Flush all of our cached data about the filesystem. Call this if we\n+ * lose sync with the filesystem and miss some notification events.\n+ *\n+ * [1] If we are missing events, then we no longer have a complete\n+ * history of the directory (relative to our current start token).\n+ * We should create a new token and start fresh (as if we just\n+ * booted up).\n+ *\n+ * If there are no concurrent threads reading the current token data\n+ * series, we can free it now. Otherwise, let the last reader free\n+ * it.\n+ *\n+ * Either way, the old token data series is no longer associated with\n+ * our state data.\n+ */\n+static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct fsmonitor_token_data *free_me = NULL;\n+\tstruct fsmonitor_token_data *new_one = NULL;\n+\n+\tnew_one = fsmonitor_new_token_data();\n+\n+\tif (state->current_token_data->client_ref_count == 0)\n+\t\tfree_me = state->current_token_data;\n+\tstate->current_token_data = new_one;\n+\n+\tfsmonitor_free_token_data(free_me);\n+}\n+\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\tpthread_mutex_lock(&state->main_lock);\n+\twith_lock__do_force_resync(state);\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -329,6 +473,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n \n+/*\n+ * We try to combine small batches at the front of the batch-list to avoid\n+ * having a long list. This hopefully makes it a little easier when we want\n+ * to truncate and maintain the list. However, we don't want the paths array\n+ * to just keep growing and growing with realloc, so we insert an arbitrary\n+ * limit.\n+ */\n+#define MY_COMBINE_LIMIT (1024)\n+\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names)\n+{\n+\tif (!batch && !cookie_names->nr)\n+\t\treturn;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (batch) {\n+\t\tstruct fsmonitor_batch *head;\n+\n+\t\thead = state->current_token_data->batch_head;\n+\t\tif (!head) {\n+\t\t\tBUG(\"token does not have batch\");\n+\t\t} else if (head->pinned_time) {\n+\t\t\t/*\n+\t\t\t * We cannot alter the current batch list\n+\t\t\t * because:\n+\t\t\t *\n+\t\t\t * [a] it is being transmitted to at least one\n+\t\t\t * client and the handle_client() thread has a\n+\t\t\t * ref-count, but not a lock on the batch list\n+\t\t\t * starting with this item.\n+\t\t\t *\n+\t\t\t * [b] it has been transmitted in the past to\n+\t\t\t * at least one client such that future\n+\t\t\t * requests are relative to this head batch.\n+\t\t\t *\n+\t\t\t * So, we can only prepend a new batch onto\n+\t\t\t * the front of the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else if (!head->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * Batch 0 is unpinned. See the note in\n+\t\t\t * `fsmonitor_new_token_data()` about why we\n+\t\t\t * don't need to accumulate these paths.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n+\t\t\t/*\n+\t\t\t * The head batch in the list has never been\n+\t\t\t * transmitted to a client, but folding the\n+\t\t\t * contents of the new batch onto it would\n+\t\t\t * exceed our arbitrary limit, so just prepend\n+\t\t\t * the new batch onto the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We are free to add the paths in the given\n+\t\t\t * batch onto the end of the current head batch.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__combine(head, batch);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -343,6 +562,13 @@ static void *fsm_listen__thread_proc(void *_state)\n \n \tfsm_listen__loop(state);\n \n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (state->current_token_data &&\n+\t state->current_token_data->client_ref_count == 0)\n+\t\tfsmonitor_free_token_data(state->current_token_data);\n+\tstate->current_token_data = NULL;\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n \ttrace2_thread_exit();\n \treturn NULL;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 8c3a71a48bd..010fbfe60e9 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -12,6 +12,27 @@\n struct fsmonitor_batch;\n struct fsmonitor_token_data;\n \n+/*\n+ * Create a new batch of path(s). The returned batch is considered\n+ * private and not linked into the fsmonitor daemon state. The caller\n+ * should fill this batch with one or more paths and then publish it.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__new(void);\n+\n+/*\n+ * Free the list of batches starting with this one.\n+ */\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n+\n+/*\n+ * Add this path to this batch of modified files.\n+ *\n+ * The batch should be private and NOT (yet) linked into the fsmonitor\n+ * daemon state and therefore not yet visible to worker threads and so\n+ * no locking is required.\n+ */\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n+\n struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n \n struct fsmonitor_daemon_state {\n@@ -117,5 +138,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \tstruct fsmonitor_daemon_state *state,\n \tconst char *path);\n \n+/*\n+ * Prepend the this batch of path(s) onto the list of batches associated\n+ * with the current token. This makes the batch visible to worker threads.\n+ *\n+ * The caller no longer owns the batch and must not free it.\n+ *\n+ * Wake up the client threads waiting on these cookies.\n+ */\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names);\n+\n+/*\n+ * If the platform-specific layer loses sync with the filesystem,\n+ * it should call this to invalidate cached data and abort waiting\n+ * threads.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"452144","messageId":"2f8a42fdb93623053256939fdbb45cf02fb1c5c9.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 09/30] compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:25Z","receivedAt":"2022-03-24T16:50:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty implementation of fsmonitor--daemon\nbackend for Darwin (aka MacOS).\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-darwin.c | 20 ++++++++++++++++++++\n config.mak.uname | 10 ++++++++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 33 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nnew file mode 100644\nindex 00000000000..c84e3344ab9\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -0,0 +1,20 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 26074f56bed..501970902da 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -157,6 +157,16 @@ ifeq ($(uname_S),Darwin)\n \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n \t\tendif\n \tendif\n+\n+\t# The builtin FSMonitor on MacOS builds upon Simple-IPC. Both require\n+\t# Unix domain sockets and PThreads.\n+\tifndef NO_PTHREADS\n+\tifndef NO_UNIX_SOCKETS\n+\tFSMONITOR_DAEMON_BACKEND = darwin\n+\tendif\n+\tendif\n+\n+\tBASIC_LDFLAGS += -framework CoreServices\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 0963629db7f..ee0d7257b77 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"452145","messageId":"14b775e9d8b1a4672f8175a546eb70e2790c1b23.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:32Z","receivedAt":"2022-03-24T16:50:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInclude MacOS system declarations to allow us to use FSEvent and\nCoreFoundation APIs. We need different versions of the declarations\nfor GCC vs. clang because of compiler and header file conflicts.\n\nWhile it is quite possible to #include Apple's CoreServices.h when\ncompiling C source code with clang, trying to build it with GCC\ncurrently fails with this error:\n\nIn file included\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Security.h:42,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/CSIdentity.h:43,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/OSServices.h:29,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/IconsCore.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/LaunchServices.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n\n /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Authorization.h:193:7:\n error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n\nThe underlying reason is that GCC (rightfully) objects that an `enum`\nvalue such as `kAuthorizationExternalFormLength` is not a constant\n(because it is not, the preprocessor has no knowledge of it, only the\nactual C compiler does) and can therefore not be used to define the size\nof a C array.\n\nThis is a known problem and tracked in GCC's bug tracker:\nhttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nIn the meantime, let's not block things and go the slightly ugly route\nof declaring/defining the FSEvents constants, data structures and\nfunctions that we need, so that we can avoid above-mentioned issue.\n\nLet's do this _only_ for GCC, though, so that the CI/PR builds (which\nbuild both with clang and with GCC) can guarantee that we _are_ using\nthe correct data types.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-darwin-gcc.h | 92 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-listen-darwin.c | 24 ++++++++\n 2 files changed, 116 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-darwin-gcc.h\n\ndiff --git a/compat/fsmonitor/fsm-darwin-gcc.h b/compat/fsmonitor/fsm-darwin-gcc.h\nnew file mode 100644\nindex 00000000000..1c75c3d48e7\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-darwin-gcc.h\n@@ -0,0 +1,92 @@\n+#ifndef FSM_DARWIN_GCC_H\n+#define FSM_DARWIN_GCC_H\n+\n+#ifndef __clang__\n+/*\n+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n+ * with clang, but not with GCC as of time of writing.\n+ *\n+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details.\n+ */\n+typedef unsigned int FSEventStreamCreateFlags;\n+#define kFSEventStreamEventFlagNone 0x00000000\n+#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n+#define kFSEventStreamEventFlagUserDropped 0x00000002\n+#define kFSEventStreamEventFlagKernelDropped 0x00000004\n+#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n+#define kFSEventStreamEventFlagHistoryDone 0x00000010\n+#define kFSEventStreamEventFlagRootChanged 0x00000020\n+#define kFSEventStreamEventFlagMount 0x00000040\n+#define kFSEventStreamEventFlagUnmount 0x00000080\n+#define kFSEventStreamEventFlagItemCreated 0x00000100\n+#define kFSEventStreamEventFlagItemRemoved 0x00000200\n+#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n+#define kFSEventStreamEventFlagItemRenamed 0x00000800\n+#define kFSEventStreamEventFlagItemModified 0x00001000\n+#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n+#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n+#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n+#define kFSEventStreamEventFlagItemIsFile 0x00010000\n+#define kFSEventStreamEventFlagItemIsDir 0x00020000\n+#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n+#define kFSEventStreamEventFlagOwnEvent 0x00080000\n+#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n+#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+\n+typedef struct __FSEventStream *FSEventStreamRef;\n+typedef const FSEventStreamRef ConstFSEventStreamRef;\n+\n+typedef unsigned int CFStringEncoding;\n+#define kCFStringEncodingUTF8 0x08000100\n+\n+typedef const struct __CFString *CFStringRef;\n+typedef const struct __CFArray *CFArrayRef;\n+typedef const struct __CFRunLoop *CFRunLoopRef;\n+\n+struct FSEventStreamContext {\n+ long long version;\n+ void *cb_data, *retain, *release, *copy_description;\n+};\n+\n+typedef struct FSEventStreamContext FSEventStreamContext;\n+typedef unsigned int FSEventStreamEventFlags;\n+#define kFSEventStreamCreateFlagNoDefer 0x02\n+#define kFSEventStreamCreateFlagWatchRoot 0x04\n+#define kFSEventStreamCreateFlagFileEvents 0x10\n+\n+typedef unsigned long long FSEventStreamEventId;\n+#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL\n+\n+typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef,\n+\t\t\t\t void *context,\n+\t\t\t\t __SIZE_TYPE__ num_of_events,\n+\t\t\t\t void *event_paths,\n+\t\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t\t const FSEventStreamEventId event_ids[]);\n+typedef double CFTimeInterval;\n+FSEventStreamRef FSEventStreamCreate(void *allocator,\n+\t\t\t\t FSEventStreamCallback callback,\n+\t\t\t\t FSEventStreamContext *context,\n+\t\t\t\t CFArrayRef paths_to_watch,\n+\t\t\t\t FSEventStreamEventId since_when,\n+\t\t\t\t CFTimeInterval latency,\n+\t\t\t\t FSEventStreamCreateFlags flags);\n+CFStringRef CFStringCreateWithCString(void *allocator, const char *string,\n+\t\t\t\t CFStringEncoding encoding);\n+CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count,\n+\t\t\t void *callbacks);\n+void CFRunLoopRun(void);\n+void CFRunLoopStop(CFRunLoopRef run_loop);\n+CFRunLoopRef CFRunLoopGetCurrent(void);\n+extern CFStringRef kCFRunLoopDefaultMode;\n+void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream,\n+\t\t\t\t CFRunLoopRef run_loop,\n+\t\t\t\t CFStringRef run_loop_mode);\n+unsigned char FSEventStreamStart(FSEventStreamRef stream);\n+void FSEventStreamStop(FSEventStreamRef stream);\n+void FSEventStreamInvalidate(FSEventStreamRef stream);\n+void FSEventStreamRelease(FSEventStreamRef stream);\n+\n+#endif /* !clang */\n+#endif /* FSM_DARWIN_GCC_H */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex c84e3344ab9..d2ce942cade 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -1,3 +1,27 @@\n+#ifndef __clang__\n+#include \"fsm-darwin-gcc.h\"\n+#else\n+#include \n+#include \n+\n+#ifndef AVAILABLE_MAC_OS_X_VERSION_10_13_AND_LATER\n+/*\n+ * This enum value was added in 10.13 to:\n+ *\n+ * /Applications/Xcode.app/Contents/Developer/Platforms/ \\\n+ * MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/ \\\n+ * Library/Frameworks/CoreServices.framework/Frameworks/ \\\n+ * FSEvents.framework/Versions/Current/Headers/FSEvents.h\n+ *\n+ * If we're compiling against an older SDK, this symbol won't be\n+ * present. Silently define it here so that we don't have to ifdef\n+ * the logging or masking below. This should be harmless since older\n+ * versions of macOS won't ever emit this FS event anyway.\n+ */\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+#endif\n+#endif\n+\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n-- \ngitgitgadget\n\n"},{"id":"452146","messageId":"239558e34ffea4240de8699707cd3b404b682488.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 15/30] compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:31Z","receivedAt":"2022-03-24T16:50:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the win32 backend to register a watch on the working tree\nroot directory (recursively). Also watch the if it is\nnot inside the working tree. And to collect path change notifications\ninto batches and publish.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-win32.c | 565 ++++++++++++++++++++++++++++\n 1 file changed, 565 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 916cbea254f..5b928ab66e5 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -2,20 +2,585 @@\n #include \"config.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+/*\n+ * The documentation of ReadDirectoryChangesW() states that the maximum\n+ * buffer size is 64K when the monitored directory is remote.\n+ *\n+ * Larger buffers may be used when the monitored directory is local and\n+ * will help us receive events faster from the kernel and avoid dropped\n+ * events.\n+ *\n+ * So we try to use a very large buffer and silently fallback to 64K if\n+ * we get an error.\n+ */\n+#define MAX_RDCW_BUF_FALLBACK (65536)\n+#define MAX_RDCW_BUF (65536 * 8)\n+\n+struct one_watch\n+{\n+\tchar buffer[MAX_RDCW_BUF];\n+\tDWORD buf_len;\n+\tDWORD count;\n+\n+\tstruct strbuf path;\n+\tHANDLE hDir;\n+\tHANDLE hEvent;\n+\tOVERLAPPED overlapped;\n+\n+\t/*\n+\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n+\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n+\t */\n+\tBOOL is_active;\n+};\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tstruct one_watch *watch_worktree;\n+\tstruct one_watch *watch_gitdir;\n+\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hListener[3]; /* we don't own these handles */\n+#define LISTENER_SHUTDOWN 0\n+#define LISTENER_HAVE_DATA_WORKTREE 1\n+#define LISTENER_HAVE_DATA_GITDIR 2\n+\tint nr_listener_handles;\n+};\n+\n+/*\n+ * Convert the WCHAR path from the notification into UTF8 and\n+ * then normalize it.\n+ */\n+static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+\t\t\t\t struct strbuf *normalized_path)\n+{\n+\tint reserve;\n+\tint len = 0;\n+\n+\tstrbuf_reset(normalized_path);\n+\tif (!info->FileNameLength)\n+\t\tgoto normalize;\n+\n+\t/*\n+\t * Pre-reserve enough space in the UTF8 buffer for\n+\t * each Unicode WCHAR character to be mapped into a\n+\t * sequence of 2 UTF8 characters. That should let us\n+\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n+\t */\n+\treserve = info->FileNameLength + 1;\n+\tstrbuf_grow(normalized_path, reserve);\n+\n+\tfor (;;) {\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n+\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t\t\t normalized_path->buf,\n+\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n+\t\t\t\t\t NULL, NULL);\n+\t\tif (len > 0)\n+\t\t\tgoto normalize;\n+\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n+\t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n+\t\t\t GetLastError(),\n+\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n+\t\t\t info->FileName);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tstrbuf_grow(normalized_path,\n+\t\t\t strbuf_avail(normalized_path) + reserve);\n+\t}\n+\n+normalize:\n+\tstrbuf_setlen(normalized_path, len);\n+\treturn strbuf_normalize_path(normalized_path);\n+}\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+}\n+\n+static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const char *path)\n+{\n+\tstruct one_watch *watch = NULL;\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\twchar_t wpath[MAX_PATH];\n+\n+\tif (xutftowcs_path(wpath, path) < 0) {\n+\t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n+\t\treturn NULL;\n+\t}\n+\n+\thDir = CreateFileW(wpath,\n+\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n+\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n+\t\t\t NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n+\t\t GetLastError(), path);\n+\t\treturn NULL;\n+\t}\n+\n+\tCALLOC_ARRAY(watch, 1);\n+\n+\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n+\n+\tstrbuf_init(&watch->path, 0);\n+\tstrbuf_addstr(&watch->path, path);\n+\n+\twatch->hDir = hDir;\n+\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\treturn watch;\n+}\n+\n+static void destroy_watch(struct one_watch *watch)\n+{\n+\tif (!watch)\n+\t\treturn;\n+\n+\tstrbuf_release(&watch->path);\n+\tif (watch->hDir != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hDir);\n+\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hEvent);\n+\n+\tfree(watch);\n+}\n+\n+static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+\t\t\t struct one_watch *watch)\n+{\n+\tDWORD dwNotifyFilter =\n+\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n+\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n+\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n+\t\tFILE_NOTIFY_CHANGE_SIZE |\n+\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n+\t\tFILE_NOTIFY_CHANGE_CREATION;\n+\n+\tResetEvent(watch->hEvent);\n+\n+\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n+\twatch->overlapped.hEvent = watch->hEvent;\n+\n+\t/*\n+\t * Queue an async call using Overlapped IO. This returns immediately.\n+\t * Our event handle will be signalled when the real result is available.\n+\t *\n+\t * The return value here just means that we successfully queued it.\n+\t * We won't know if the Read...() actually produces data until later.\n+\t */\n+\twatch->is_active = ReadDirectoryChangesW(\n+\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n+\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n+\n+\tif (watch->is_active)\n+\t\treturn 0;\n+\n+\terror(_(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\"),\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static int recv_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD gle;\n+\n+\twatch->is_active = FALSE;\n+\n+\t/*\n+\t * The overlapped result is ready. If the Read...() was successful\n+\t * we finally receive the actual result into our buffer.\n+\t */\n+\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n+\t\t\t\tTRUE))\n+\t\treturn 0;\n+\n+\tgle = GetLastError();\n+\tif (gle == ERROR_INVALID_PARAMETER &&\n+\t /*\n+\t * The kernel throws an invalid parameter error when our\n+\t * buffer is too big and we are pointed at a remote\n+\t * directory (and possibly for other reasons). Quietly\n+\t * set it down and try again.\n+\t *\n+\t * See note about MAX_RDCW_BUF at the top.\n+\t */\n+\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n+\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n+\t\treturn -2;\n+\t}\n+\n+\t/*\n+\t * NEEDSWORK: If an external is deleted, the above\n+\t * returns an error. I'm not sure that there's anything that\n+\t * we can do here other than failing -- the /.git\n+\t * link file would be broken anyway. We might try to check\n+\t * for that and return a better error message, but I'm not\n+\t * sure it is worth it.\n+\t */\n+\n+\terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\n+\t watch->path.buf, gle);\n+\treturn -1;\n+}\n+\n+static void cancel_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD count;\n+\n+\tif (!watch || !watch->is_active)\n+\t\treturn;\n+\n+\t/*\n+\t * The calls to ReadDirectoryChangesW() and GetOverlappedResult()\n+\t * form a \"pair\" (my term) where we queue an IO and promise to\n+\t * hang around and wait for the kernel to give us the result.\n+\t *\n+\t * If for some reason after we queue the IO, we have to quit\n+\t * or otherwise not stick around for the second half, we must\n+\t * tell the kernel to abort the IO. This prevents the kernel\n+\t * from writing to our buffer and/or signalling our event\n+\t * after we free them.\n+\t *\n+\t * (Ask me how much fun it was to track that one down).\n+\t */\n+\tCancelIoEx(watch->hDir, &watch->overlapped);\n+\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n+\twatch->is_active = FALSE;\n+}\n+\n+/*\n+ * Process filesystem events that happen anywhere (recursively) under the\n+ * root directory. For a normal working directory, this includes\n+ * both version controlled files and the contents of the .git/ directory.\n+ *\n+ * If /.git is a file, then we only see events for the file\n+ * itself.\n+ */\n+static int process_worktree_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_worktree;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tconst char *p = watch->buffer;\n+\n+\t/*\n+\t * If the kernel gets more events than will fit in the kernel\n+\t * buffer associated with our RDCW handle, it drops them and\n+\t * returns a count of zero.\n+\t *\n+\t * Yes, the call returns WITHOUT error and with length zero.\n+\t * This is the documented behavior. (My testing has confirmed\n+\t * that it also sets the last error to ERROR_NOTIFY_ENUM_DIR,\n+\t * but we do not rely on that since the function did not\n+\t * return an error and it is not documented.)\n+\t *\n+\t * (The \"overflow\" case is not ambiguous with the \"no data\" case\n+\t * because we did an INFINITE wait.)\n+\t *\n+\t * This means we have a gap in coverage. Tell the daemon layer\n+\t * to resync.\n+\t */\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\t}\n+\n+\t/*\n+\t * On Windows, `info` contains an \"array\" of paths that are\n+\t * relative to the root of whichever directory handle received\n+\t * the event.\n+\t */\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\t\t/* ignore everything inside of \"/.git/\" */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* queue normal pathname */\n+\t\t\tif (!batch)\n+\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tbatch = NULL;\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_SHUTDOWN;\n+}\n+\n+/*\n+ * Process filesystem events that happened anywhere (recursively) under the\n+ * external (such as non-primary worktrees or submodules).\n+ * We only care about cookie files that our client threads created here.\n+ *\n+ * Note that we DO NOT get filesystem events on the external \n+ * itself (it is not inside something that we are watching). In particular,\n+ * we do not get an event if the external is deleted.\n+ */\n+static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_gitdir;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *p = watch->buffer;\n+\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_GITDIR;\n+\t}\n+\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\tgoto skip_this_path;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, NULL, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_GITDIR;\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tDWORD dwWait;\n+\tint result;\n+\n+\tstate->error_code = 0;\n+\n+\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tif (data->watch_gitdir &&\n+\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tfor (;;) {\n+\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n+\t\t\t\t\t\tdata->hListener,\n+\t\t\t\t\t\tFALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_worktree);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_gitdir);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->error_code = -1;\n+\n+force_shutdown:\n+\t/*\n+\t * Tell the IPC thead pool to stop (which completes the await\n+\t * in the main thread (which will also signal this thread (if\n+\t * we are still alive))).\n+\t */\n+\tipc_server_stop_async(state->ipc_server_data);\n+\n+clean_shutdown:\n+\tcancel_rdcw_watch(data->watch_worktree);\n+\tcancel_rdcw_watch(data->watch_gitdir);\n }\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->watch_worktree = create_watch(state,\n+\t\t\t\t\t state->path_worktree_watch.buf);\n+\tif (!data->watch_worktree)\n+\t\tgoto failed;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->watch_gitdir = create_watch(state,\n+\t\t\t\t\t\t state->path_gitdir_watch.buf);\n+\t\tif (!data->watch_gitdir)\n+\t\t\tgoto failed;\n+\t}\n+\n+\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_listener_handles++;\n+\n+\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n+\t\tdata->watch_worktree->hEvent;\n+\tdata->nr_listener_handles++;\n+\n+\tif (data->watch_gitdir) {\n+\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n+\t\t\tdata->watch_gitdir->hEvent;\n+\t\tdata->nr_listener_handles++;\n+\t}\n+\n+\tstate->backend_data = data;\n+\treturn 0;\n+\n+failed:\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n-- \ngitgitgadget\n\n"},{"id":"452147","messageId":"55bd7aee06ca04e6ae9010700103a261d1255113.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 17/30] compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:33Z","receivedAt":"2022-03-24T16:50:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement file system event listener on MacOS using FSEvent,\nCoreFoundation, and CoreServices.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-darwin.c | 383 +++++++++++++++++++++++++++\n 1 file changed, 383 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex d2ce942cade..0741fe834c3 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -25,20 +25,403 @@\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tCFStringRef cfsr_worktree_path;\n+\tCFStringRef cfsr_gitdir_path;\n+\n+\tCFArrayRef cfar_paths_to_watch;\n+\tint nr_paths_watching;\n+\n+\tFSEventStreamRef stream;\n+\n+\tCFRunLoopRef rl;\n+\n+\tenum shutdown_style {\n+\t\tSHUTDOWN_EVENT = 0,\n+\t\tFORCE_SHUTDOWN,\n+\t\tFORCE_ERROR_STOP,\n+\t} shutdown_style;\n+\n+\tunsigned int stream_scheduled:1;\n+\tunsigned int stream_started:1;\n+};\n+\n+static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n+\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n+\tif (flag & kFSEventStreamEventFlagUserDropped)\n+\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n+\tif (flag & kFSEventStreamEventFlagKernelDropped)\n+\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n+\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n+\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n+\tif (flag & kFSEventStreamEventFlagHistoryDone)\n+\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n+\tif (flag & kFSEventStreamEventFlagRootChanged)\n+\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n+\tif (flag & kFSEventStreamEventFlagMount)\n+\t\tstrbuf_addstr(&msg, \"Mount|\");\n+\tif (flag & kFSEventStreamEventFlagUnmount)\n+\t\tstrbuf_addstr(&msg, \"Unmount|\");\n+\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n+\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n+\tif (flag & kFSEventStreamEventFlagItemCreated)\n+\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n+\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n+\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n+\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsDir)\n+\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsFile)\n+\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemModified)\n+\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n+\tif (flag & kFSEventStreamEventFlagItemRemoved)\n+\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n+\tif (flag & kFSEventStreamEventFlagItemRenamed)\n+\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n+\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n+\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n+\tif (flag & kFSEventStreamEventFlagOwnEvent)\n+\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n+\tif (flag & kFSEventStreamEventFlagItemCloned)\n+\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\t\t\t path, flag, msg.buf);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRemoved);\n+}\n+\n+static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRenamed);\n+}\n+\n+static int ef_is_dropped(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagMustScanSubDirs ||\n+\t\tef & kFSEventStreamEventFlagKernelDropped ||\n+\t\tef & kFSEventStreamEventFlagUserDropped);\n+}\n+\n+static void fsevent_callback(ConstFSEventStreamRef streamRef,\n+\t\t\t void *ctx,\n+\t\t\t size_t num_of_events,\n+\t\t\t void *event_paths,\n+\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t const FSEventStreamEventId event_ids[])\n+{\n+\tstruct fsmonitor_daemon_state *state = ctx;\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tchar **paths = (char **)event_paths;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *path_k;\n+\tconst char *slash;\n+\tint k;\n+\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t/*\n+\t * Build a list of all filesystem changes into a private/local\n+\t * list and without holding any locks.\n+\t */\n+\tfor (k = 0; k < num_of_events; k++) {\n+\t\t/*\n+\t\t * On Mac, we receive an array of absolute paths.\n+\t\t */\n+\t\tpath_k = paths[k];\n+\n+\t\t/*\n+\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n+\t\t * Please don't log them to Trace2.\n+\t\t *\n+\t\t * trace_printf_key(&trace_fsmonitor, \"Path: '%s'\", path_k);\n+\t\t */\n+\n+\t\t/*\n+\t\t * If event[k] is marked as dropped, we assume that we have\n+\t\t * lost sync with the filesystem and should flush our cached\n+\t\t * data. We need to:\n+\t\t *\n+\t\t * [1] Abort/wake any client threads waiting for a cookie and\n+\t\t * flush the cached state data (the current token), and\n+\t\t * create a new token.\n+\t\t *\n+\t\t * [2] Discard the batch that we were locally building (since\n+\t\t * they are conceptually relative to the just flushed\n+\t\t * token).\n+\t\t */\n+\t\tif (ef_is_dropped(event_flags[k])) {\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\tfsmonitor_force_resync(state);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t\tstring_list_clear(&cookie_list, 0);\n+\n+\t\t\t/*\n+\t\t\t * We assume that any events that we received\n+\t\t\t * in this callback after this dropped event\n+\t\t\t * may still be valid, so we continue rather\n+\t\t\t * than break. (And just in case there is a\n+\t\t\t * delete of \".git\" hiding in there.)\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n+\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git or gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path_k);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path_k);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\t/* ignore all other paths inside of .git or gitdir */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\tcase IS_GITDIR:\n+\t\t\t/*\n+\t\t\t * If .git directory is deleted or renamed away,\n+\t\t\t * we have to quit.\n+\t\t\t */\n+\t\t\tif (ef_is_root_delete(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir renamed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* try to queue normal pathnames */\n+\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\t/*\n+\t\t\t * Because of the implicit \"binning\" (the\n+\t\t\t * kernel calls us at a given frequency) and\n+\t\t\t * de-duping (the kernel is free to combine\n+\t\t\t * multiple events for a given pathname), an\n+\t\t\t * individual fsevent could be marked as both\n+\t\t\t * a file and directory. Add it to the queue\n+\t\t\t * with both spellings so that the client will\n+\t\t\t * know how much to invalidate/refresh.\n+\t\t\t */\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t}\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tstrbuf_reset(&tmp);\n+\t\t\t\tstrbuf_addstr(&tmp, rel);\n+\t\t\t\tstrbuf_addch(&tmp, '/');\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\n+\t\tcase IS_OUTSIDE_CONE:\n+\t\tdefault:\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignoring '%s'\", path_k);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\n+\tdata->shutdown_style = FORCE_SHUTDOWN;\n+\tCFRunLoopStop(data->rl);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+}\n+\n+/*\n+ * In the call to `FSEventStreamCreate()` to setup our watch, the\n+ * `latency` argument determines the frequency of calls to our callback\n+ * with new FS events. Too slow and events get dropped; too fast and\n+ * we burn CPU unnecessarily. Since it is rather obscure, I don't\n+ * think this needs to be a config setting. I've done extensive\n+ * testing on my systems and chosen the value below. It gives good\n+ * results and I've not seen any dropped events.\n+ *\n+ * With a latency of 0.1, I was seeing lots of dropped events during\n+ * the \"touch 100000\" files test within t/perf/p7519, but with a\n+ * latency of 0.001 I did not see any dropped events. So I'm going\n+ * to assume that this is the \"correct\" value.\n+ *\n+ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n+ */\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n+\t\tkFSEventStreamCreateFlagWatchRoot |\n+\t\tkFSEventStreamCreateFlagFileEvents;\n+\tFSEventStreamContext ctx = {\n+\t\t0,\n+\t\tstate,\n+\t\tNULL,\n+\t\tNULL,\n+\t\tNULL\n+\t};\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\tconst void *dir_array[2];\n+\n+\tCALLOC_ARRAY(data, 1);\n+\tstate->backend_data = data;\n+\n+\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n+\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n+\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n+\t\t\tNULL, state->path_gitdir_watch.buf,\n+\t\t\tkCFStringEncodingUTF8);\n+\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n+\t}\n+\n+\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n+\t\t\t\t\t\t data->nr_paths_watching,\n+\t\t\t\t\t\t NULL);\n+\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n+\t\t\t\t\t data->cfar_paths_to_watch,\n+\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n+\t\t\t\t\t 0.001, flags);\n+\tif (data->stream == NULL)\n+\t\tgoto failed;\n+\n+\t/*\n+\t * `data->rl` needs to be set inside the listener thread.\n+\t */\n+\n+\treturn 0;\n+\n+failed:\n+\terror(_(\"Unable to create FSEventStream.\"));\n+\n+\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tif (data->stream) {\n+\t\tif (data->stream_started)\n+\t\t\tFSEventStreamStop(data->stream);\n+\t\tif (data->stream_scheduled)\n+\t\t\tFSEventStreamInvalidate(data->stream);\n+\t\tFSEventStreamRelease(data->stream);\n+\t}\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\tdata->shutdown_style = SHUTDOWN_EVENT;\n+\n+\tCFRunLoopStop(data->rl);\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\n+\tdata->rl = CFRunLoopGetCurrent();\n+\n+\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n+\tdata->stream_scheduled = 1;\n+\n+\tif (!FSEventStreamStart(data->stream)) {\n+\t\terror(_(\"Failed to start the FSEventStream\"));\n+\t\tgoto force_error_stop_without_loop;\n+\t}\n+\tdata->stream_started = 1;\n+\n+\tCFRunLoopRun();\n+\n+\tswitch (data->shutdown_style) {\n+\tcase FORCE_ERROR_STOP:\n+\t\tstate->error_code = -1;\n+\t\t/* fall thru */\n+\tcase FORCE_SHUTDOWN:\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\t/* fall thru */\n+\tcase SHUTDOWN_EVENT:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn;\n+\n+force_error_stop_without_loop:\n+\tstate->error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+\treturn;\n }\n-- \ngitgitgadget\n\n"},{"id":"452148","messageId":"c43009124fbb8f10d4919dbaaf38a728789dd395.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 18/30] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:34Z","receivedAt":"2022-03-24T16:50:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to respond to IPC requests from client\nGit processes and respond with a list of modified pathnames\nrelative to the provided token.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 311 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 309 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 69312119b07..eafaafb45b1 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,6 +7,7 @@\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n+#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon start []\"),\n@@ -364,6 +365,310 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+/*\n+ * Format an opaque token string to send to the client.\n+ */\n+static void with_lock__format_response_token(\n+\tstruct strbuf *response_token,\n+\tconst struct strbuf *response_token_id,\n+\tconst struct fsmonitor_batch *batch)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstrbuf_reset(response_token);\n+\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n+\t\t response_token_id->buf, batch->batch_seq_nr);\n+}\n+\n+/*\n+ * Parse an opaque token from the client.\n+ * Returns -1 on error.\n+ */\n+static int fsmonitor_parse_client_token(const char *buf_token,\n+\t\t\t\t\tstruct strbuf *requested_token_id,\n+\t\t\t\t\tuint64_t *seq_nr)\n+{\n+\tconst char *p;\n+\tchar *p_end;\n+\n+\tstrbuf_reset(requested_token_id);\n+\t*seq_nr = 0;\n+\n+\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n+\t\treturn -1;\n+\n+\twhile (*p && *p != ':')\n+\t\tstrbuf_addch(requested_token_id, *p++);\n+\tif (!*p++)\n+\t\treturn -1;\n+\n+\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n+\tif (*p_end)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal)\n+\n+static int do_handle_client(struct fsmonitor_daemon_state *state,\n+\t\t\t const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\tstruct fsmonitor_token_data *token_data = NULL;\n+\tstruct strbuf response_token = STRBUF_INIT;\n+\tstruct strbuf requested_token_id = STRBUF_INIT;\n+\tstruct strbuf payload = STRBUF_INIT;\n+\tuint64_t requested_oldest_seq_nr = 0;\n+\tuint64_t total_response_len = 0;\n+\tconst char *p;\n+\tconst struct fsmonitor_batch *batch_head;\n+\tconst struct fsmonitor_batch *batch;\n+\tintmax_t count = 0, duplicates = 0;\n+\tkh_str_t *shown;\n+\tint hash_ret;\n+\tint do_trivial = 0;\n+\tint do_flush = 0;\n+\n+\t/*\n+\t * We expect `command` to be of the form:\n+\t *\n+\t * := quit NUL\n+\t * | flush NUL\n+\t * | NUL\n+\t * | NUL\n+\t */\n+\n+\tif (!strcmp(command, \"quit\")) {\n+\t\t/*\n+\t\t * A client has requested over the socket/pipe that the\n+\t\t * daemon shutdown.\n+\t\t *\n+\t\t * Tell the IPC thread pool to shutdown (which completes\n+\t\t * the await in the main thread (which can stop the\n+\t\t * fsmonitor listener thread)).\n+\t\t *\n+\t\t * There is no reply to the client.\n+\t\t */\n+\t\treturn SIMPLE_IPC_QUIT;\n+\n+\t} else if (!strcmp(command, \"flush\")) {\n+\t\t/*\n+\t\t * Flush all of our cached data and generate a new token\n+\t\t * just like if we lost sync with the filesystem.\n+\t\t *\n+\t\t * Then send a trivial response using the new token.\n+\t\t */\n+\t\tdo_flush = 1;\n+\t\tdo_trivial = 1;\n+\n+\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n+\t\t/* assume V1 timestamp or garbage */\n+\n+\t\tchar *p_end;\n+\n+\t\tstrtoumax(command, &p_end, 10);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t ((*p_end) ?\n+\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n+\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n+\t\t\t\t command);\n+\t\tdo_trivial = 1;\n+\n+\t} else {\n+\t\t/* We have \"builtin:*\" */\n+\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n+\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n+\t\t\t\t\t command);\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We have a V2 valid token:\n+\t\t\t * \"builtin::\"\n+\t\t\t */\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (!state->current_token_data)\n+\t\tBUG(\"fsmonitor state does not have a current token\");\n+\n+\tif (do_flush)\n+\t\twith_lock__do_force_resync(state);\n+\n+\t/*\n+\t * We mark the current head of the batch list as \"pinned\" so\n+\t * that the listener thread will treat this item as read-only\n+\t * (and prevent any more paths from being added to it) from\n+\t * now on.\n+\t */\n+\ttoken_data = state->current_token_data;\n+\tbatch_head = token_data->batch_head;\n+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n+\n+\t/*\n+\t * FSMonitor Protocol V2 requires that we send a response header\n+\t * with a \"new current token\" and then all of the paths that changed\n+\t * since the \"requested token\". We send the seq_nr of the just-pinned\n+\t * head batch so that future requests from a client will be relative\n+\t * to it.\n+\t */\n+\twith_lock__format_response_token(&response_token,\n+\t\t\t\t\t &token_data->token_id, batch_head);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttotal_response_len += response_token.len + 1;\n+\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n+\t\t\t response_token.buf);\n+\n+\tif (!do_trivial) {\n+\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n+\t\t\t/*\n+\t\t\t * The client last spoke to a different daemon\n+\t\t\t * instance -OR- the daemon had to resync with\n+\t\t\t * the filesystem (and lost events), so reject.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n+\t\t\t\t\t \"response/token\", \"different\");\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else if (requested_oldest_seq_nr <\n+\t\t\t token_data->batch_tail->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * The client wants older events than we have for\n+\t\t\t * this token_id. This means that the end of our\n+\t\t\t * batch list was truncated and we cannot give the\n+\t\t\t * client a complete snapshot relative to their\n+\t\t\t * request.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"client requested truncated data\");\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n+\tif (do_trivial) {\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\treply(reply_data, \"/\", 2);\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n+\t\t\t\t \"response/trivial\", 1);\n+\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/*\n+\t * We're going to hold onto a pointer to the current\n+\t * token-data while we walk the list of batches of files.\n+\t * During this time, we will NOT be under the lock.\n+\t * So we ref-count it.\n+\t *\n+\t * This allows the listener thread to continue prepending\n+\t * new batches of items to the token-data (which we'll ignore).\n+\t *\n+\t * AND it allows the listener thread to do a token-reset\n+\t * (and install a new `current_token_data`).\n+\t */\n+\ttoken_data->client_ref_count++;\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * The client request is relative to the token that they sent,\n+\t * so walk the batch list backwards from the current head back\n+\t * to the batch (sequence number) they named.\n+\t *\n+\t * We use khash to de-dup the list of pathnames.\n+\t *\n+\t * NEEDSWORK: each batch contains a list of interned strings,\n+\t * so we only need to do pointer comparisons here to build the\n+\t * hash table. Currently, we're still comparing the string\n+\t * values.\n+\t */\n+\tshown = kh_init_str();\n+\tfor (batch = batch_head;\n+\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n+\t batch = batch->next) {\n+\t\tsize_t k;\n+\n+\t\tfor (k = 0; k < batch->nr; k++) {\n+\t\t\tconst char *s = batch->interned_paths[k];\n+\t\t\tsize_t s_len;\n+\n+\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n+\t\t\t\tduplicates++;\n+\t\t\telse {\n+\t\t\t\tkh_put_str(shown, s, &hash_ret);\n+\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n+\t\t\t\t\t\t count, s);\n+\n+\t\t\t\t/* Each path gets written with a trailing NUL */\n+\t\t\t\ts_len = strlen(s) + 1;\n+\n+\t\t\t\tif (payload.len + s_len >=\n+\t\t\t\t LARGE_PACKET_DATA_MAX) {\n+\t\t\t\t\treply(reply_data, payload.buf,\n+\t\t\t\t\t payload.len);\n+\t\t\t\t\ttotal_response_len += payload.len;\n+\t\t\t\t\tstrbuf_reset(&payload);\n+\t\t\t\t}\n+\n+\t\t\t\tstrbuf_add(&payload, s, s_len);\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (payload.len) {\n+\t\treply(reply_data, payload.buf, payload.len);\n+\t\ttotal_response_len += payload.len;\n+\t}\n+\n+\tkh_release_str(shown);\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (token_data->client_ref_count > 0)\n+\t\ttoken_data->client_ref_count--;\n+\n+\tif (token_data->client_ref_count == 0) {\n+\t\tif (token_data != state->current_token_data) {\n+\t\t\t/*\n+\t\t\t * The listener thread did a token-reset while we were\n+\t\t\t * walking the batch list. Therefore, this token is\n+\t\t\t * stale and can be discarded completely. If we are\n+\t\t\t * the last reader thread using this token, we own\n+\t\t\t * that work.\n+\t\t\t */\n+\t\t\tfsmonitor_free_token_data(token_data);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n+\n+cleanup:\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\tstrbuf_release(&payload);\n+\n+\treturn 0;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -371,7 +676,7 @@ static int handle_client(void *data,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n-\t/* struct fsmonitor_daemon_state *state = data; */\n+\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n \t/*\n@@ -382,10 +687,12 @@ static int handle_client(void *data,\n \tif (command_len != strlen(command))\n \t\tBUG(\"FSMonitor assumes text messages\");\n \n+\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n+\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n \n-\tresult = 0; /* TODO Do something here. */\n+\tresult = do_handle_client(state, command, reply, reply_data);\n \n \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n \n-- \ngitgitgadget\n\n"},{"id":"452149","messageId":"ed338777b561ca7c2bb6753b0408da7ea3a64435.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 19/30] help: include fsmonitor--daemon feature flag in version info","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:35Z","receivedAt":"2022-03-24T16:50:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the \"feature: fsmonitor--daemon\" message to the output of\n`git version --build-options`.\n\nThe builtin FSMonitor is only available on certain platforms and\neven then only when certain Makefile flags are enabled, so print\na message in the verbose version output when it is available.\n\nThis can be used by test scripts for prereq testing. Granted, tests\ncould just try `git fsmonitor--daemon status` and look for a 128 exit\ncode or grep for a \"not supported\" message on stderr, but these\nmethods are rather obscure.\n\nThe main advantage is that the feature message will automatically\nappear in bug reports and other support requests.\n\nThis concept was also used during the development of Scalar for\nsimilar reasons.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n help.c | 4 ++++\n t/test-lib.sh | 7 +++++++\n 2 files changed, 11 insertions(+)\n\ndiff --git a/help.c b/help.c\nindex 71444906ddf..9112a51e84b 100644\n--- a/help.c\n+++ b/help.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"parse-options.h\"\n #include \"prompt.h\"\n+#include \"fsmonitor-ipc.h\"\n \n struct category_description {\n \tuint32_t category;\n@@ -695,6 +696,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n+\n+\t\tif (fsmonitor_ipc__is_supported())\n+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n \t}\n }\n \ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e4716b0b867..5d819c1bc11 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1799,3 +1799,10 @@ test_lazy_prereq SHA1 '\n # Tests that verify the scheduler integration must set this locally\n # to avoid errors.\n GIT_TEST_MAINT_SCHEDULER=\"none:exit 1\"\n+\n+# Does this platform support `git fsmonitor--daemon`\n+#\n+test_lazy_prereq FSMONITOR_DAEMON '\n+\tgit version --build-options >output &&\n+\tgrep \"feature: fsmonitor--daemon\" output\n+'\n-- \ngitgitgadget\n\n"},{"id":"452150","messageId":"c99bac29d4233a2957bf0cfa7f2470798ae495ce.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 20/30] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:36Z","receivedAt":"2022-03-24T16:50:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate an IPC client to send query and flush commands to the daemon.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n t/helper/test-fsmonitor-client.c | 116 +++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 119 insertions(+)\n create mode 100644 t/helper/test-fsmonitor-client.c\n\ndiff --git a/Makefile b/Makefile\nindex 26567d4f772..daa21bed6c3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -716,6 +716,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\n TEST_BUILTINS_OBJS += test-getcwd.o\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nnew file mode 100644\nindex 00000000000..3062c8a3c2b\n--- /dev/null\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -0,0 +1,116 @@\n+/*\n+ * test-fsmonitor-client.c: client code to send commands/requests to\n+ * a `git fsmonitor--daemon` daemon.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tdie(\"fsmonitor--daemon not available on this platform\");\n+}\n+#else\n+\n+/*\n+ * Read the `.git/index` to get the last token written to the\n+ * FSMonitor Index Extension.\n+ */\n+static const char *get_token_from_index(void)\n+{\n+\tstruct index_state *istate = the_repository->index;\n+\n+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update)\n+\t\tdie(\"index file does not have fsmonitor extension\");\n+\n+\treturn istate->fsmonitor_last_update;\n+}\n+\n+/*\n+ * Send an IPC query to a `git-fsmonitor--daemon` daemon and\n+ * ask for the changes since the given token or from the last\n+ * token in the index extension.\n+ *\n+ * This will implicitly start a daemon process if necessary. The\n+ * daemon process will persist after we exit.\n+ */\n+static int do_send_query(const char *token)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\n+\tret = fsmonitor_ipc__send_query(token, &answer);\n+\tif (ret < 0)\n+\t\tdie(\"could not query fsmonitor--daemon\");\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n+ * and tell it to flush its cache.\n+ *\n+ * This feature is primarily used by the test suite to simulate a loss of\n+ * sync with the filesystem where we miss kernel events.\n+ */\n+static int do_send_flush(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tconst char *subcmd;\n+\tconst char *token = NULL;\n+\n+\tconst char * const fsmonitor_client_usage[] = {\n+\t\t\"test-tool fsmonitor-client query []\",\n+\t\t\"test-tool fsmonitor-client flush\",\n+\t\tNULL,\n+\t};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING(0, \"token\", &token, \"token\",\n+\t\t\t \"command token to send to the server\"),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n+\n+\tif (argc != 1)\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tsubcmd = argv[0];\n+\n+\tsetup_git_directory();\n+\n+\tif (!strcmp(subcmd, \"query\"))\n+\t\treturn !!do_send_query(token);\n+\n+\tif (!strcmp(subcmd, \"flush\"))\n+\t\treturn !!do_send_flush();\n+\n+\tdie(\"Unhandled subcommand: '%s'\", subcmd);\n+}\n+#endif\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex e6ec69cf326..0424f7adf5d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -32,6 +32,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\n \t{ \"getcwd\", cmd__getcwd },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 20756eefdda..c876e8246fb 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"452151","messageId":"cc39ecf10ae4026fed608682f6c0cdc99cff705c.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 22/30] t/perf: avoid copying builtin fsmonitor files into test repo","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:38Z","receivedAt":"2022-03-24T16:51:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDo not copy any of the various fsmonitor--daemon files from the .git\ndirectory of the (GIT_PREF_REPO or GIT_PERF_LARGE_REPO) source repo\ninto the test's trash directory.\n\nWhen perf tests start, they copy the contents of the source repo into\nthe test's trash directory. If fsmonitor is running in the source repo,\nthere may be control files, such as the IPC socket and/or fsmonitor\ncookie files. These should not be copied into the test repo.\n\nUnix domain sockets cannot be copied in the manner used by the test\nsetup, so if present, the test setup fails.\n\nCookie files are harmless, but we should avoid them.\n\nThe builtin fsmonitor keeps all such control files/sockets in\n.git/fsmonitor--daemon*, so it is simple to exclude them.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/perf-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 407252bac70..932105cd12c 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -78,7 +78,7 @@ test_perf_copy_repo_contents () {\n \tfor stuff in \"$1\"/*\n \tdo\n \t\tcase \"$stuff\" in\n-\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees)\n+\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)\n \t\t\t;;\n \t\t*)\n \t\t\tcp -R \"$stuff\" \"$repo/.git/\" || exit 1\n-- \ngitgitgadget\n\n"},{"id":"452152","messageId":"c8709da9457eb303132b5cad6a204a1de27aabc0.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 21/30] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:37Z","receivedAt":"2022-03-24T16:51:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/t7527-builtin-fsmonitor.sh | 505 +++++++++++++++++++++++++++++++++++\n 1 file changed, 505 insertions(+)\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..3570bf764aa\n--- /dev/null\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -0,0 +1,505 @@\n+#!/bin/sh\n+\n+test_description='built-in file system watcher'\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+stop_daemon_delete_repo () {\n+\tr=$1 &&\n+\ttest_might_fail git -C $r fsmonitor--daemon stop &&\n+\trm -rf $1\n+}\n+\n+is_value () {\n+\ttest -n \"$1\" && test \"${1::1}\" != \"-\"\n+}\n+\n+start_daemon () {\n+\tr= &&\n+\ttf= &&\n+\tt2= &&\n+\ttk= &&\n+\n+\twhile test \"$#\" -ne 0\n+\tdo\n+\t\tcase \"$1\" in\n+\t\t-C)\n+\t\t\tshift;\n+\t\t\tis_value $1 || BUG \"error: -C requires value\"\n+\t\t\tr=\"-C $1\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--tf)\n+\t\t\tshift;\n+\t\t\tis_value $1 || BUG \"error: --tf requires value\"\n+\t\t\ttf=\"$1\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--t2)\n+\t\t\tshift;\n+\t\t\tis_value $1 || BUG \"error: --t2 requires value\"\n+\t\t\tt2=\"$1\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--tk)\n+\t\t\tshift;\n+\t\t\tis_value $1 || BUG \"error: --tk requires value\"\n+\t\t\ttk=\"$1\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t*)\n+\t\t\tBUG \"error: unknown option: '$1'\"\n+\t\t\t;;\n+\t\tesac\n+\tdone &&\n+\n+\t(\n+\t\tif test -n \"$tf\"\n+\t\tthen\n+\t\t\tGIT_TRACE_FSMONITOR=\"$tf\"\n+\t\t\texport GIT_TRACE_FSMONITOR\n+\t\tfi &&\n+\n+\t\tif test -n \"$t2\"\n+\t\tthen\n+\t\t\tGIT_TRACE2_PERF=\"$t2\"\n+\t\t\texport GIT_TRACE2_PERF\n+\t\tfi &&\n+\n+\t\tif test -n \"$tk\"\n+\t\tthen\n+\t\t\tGIT_TEST_FSMONITOR_TOKEN=\"$tk\"\n+\t\t\texport GIT_TEST_FSMONITOR_TOKEN\n+\t\tfi &&\n+\n+\t\tgit $r fsmonitor--daemon start &&\n+\t\tgit $r fsmonitor--daemon status\n+\t)\n+}\n+\n+# Is a Trace2 data event present with the given catetory and key?\n+# We do not care what the value is.\n+#\n+have_t2_data_event () {\n+\tc=$1 &&\n+\tk=$2 &&\n+\n+\tgrep -e '\"event\":\"data\".*\"category\":\"'\"$c\"'\".*\"key\":\"'\"$k\"'\"'\n+}\n+\n+test_expect_success 'explicit daemon start and stop' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n+\n+\tgit init test_explicit &&\n+\tstart_daemon -C test_explicit &&\n+\n+\tgit -C test_explicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon start' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit\" &&\n+\n+\tgit init test_implicit &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n+\n+\t# query will implicitly start the daemon.\n+\t#\n+\t# for test-script simplicity, we send a V1 timestamp rather than\n+\t# a V2 token. either way, the daemon response to any query contains\n+\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n+\t# but this test case is only concerned with whether the daemon was\n+\t# implicitly started.)\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n+\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n+\tnul_to_q actual.filtered &&\n+\tgrep \"builtin:\" actual.filtered &&\n+\n+\t# confirm that a daemon was started in the background.\n+\t#\n+\t# since the mechanism for starting the background daemon is platform\n+\t# dependent, just confirm that the foreground command received a\n+\t# response from the daemon.\n+\n+\thave_t2_data_event fsm_client query/response-length <.git/trace &&\n+\n+\tgit -C test_implicit fsmonitor--daemon status &&\n+\tgit -C test_implicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (delete .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n+\n+\tgit init test_implicit_1 &&\n+\n+\tstart_daemon -C test_implicit_1 &&\n+\n+\t# deleting the .git directory will implicitly stop the daemon.\n+\trm -rf test_implicit_1/.git &&\n+\n+\t# [1] Create an empty .git directory so that the following Git\n+\t# command will stay relative to the `-C` directory.\n+\t#\n+\t# Without this, the Git command will override the requested\n+\t# -C argument and crawl out to the containing Git source tree.\n+\t# This would make the test result dependent upon whether we\n+\t# were using fsmonitor on our development worktree.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_1/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (rename .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_2\" &&\n+\n+\tgit init test_implicit_2 &&\n+\n+\tstart_daemon -C test_implicit_2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n+\n+\t# See [1] above.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'cannot start multiple daemons' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n+\n+\tgit init test_multiple &&\n+\n+\tstart_daemon -C test_multiple &&\n+\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n+\tgrep \"fsmonitor--daemon is already running\" actual &&\n+\n+\tgit -C test_multiple fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n+'\n+\n+# These tests use the main repo in the trash directory\n+\n+test_expect_success 'setup' '\n+\t>tracked &&\n+\t>modified &&\n+\t>delete &&\n+\t>rename &&\n+\tmkdir dir1 &&\n+\t>dir1/tracked &&\n+\t>dir1/modified &&\n+\t>dir1/delete &&\n+\t>dir1/rename &&\n+\tmkdir dir2 &&\n+\t>dir2/tracked &&\n+\t>dir2/modified &&\n+\t>dir2/delete &&\n+\t>dir2/rename &&\n+\tmkdir dirtorename &&\n+\t>dirtorename/a &&\n+\t>dirtorename/b &&\n+\n+\tcat >.gitignore <<-\\EOF &&\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tEOF\n+\n+\tgit -c core.fsmonitor=false add . &&\n+\ttest_tick &&\n+\tgit -c core.fsmonitor=false commit -m initial &&\n+\n+\tgit config core.fsmonitor true\n+'\n+\n+# The test already explicitly stopped (or tried to stop) the daemon.\n+# This is here in case something else fails first.\n+#\n+redundant_stop_daemon () {\n+\ttest_might_fail git fsmonitor--daemon stop\n+}\n+\n+test_expect_success 'update-index implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n+\t\tgit update-index --fsmonitor &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_1\n+'\n+\n+test_expect_success 'status implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n+\t\tgit status >actual &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_2\n+'\n+\n+edit_files () {\n+\techo 1 >modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\t>dir1/untracked\n+}\n+\n+delete_files () {\n+\trm -f delete &&\n+\trm -f dir1/delete &&\n+\trm -f dir2/delete\n+}\n+\n+create_files () {\n+\techo 1 >new &&\n+\techo 2 >dir1/new &&\n+\techo 3 >dir2/new\n+}\n+\n+rename_files () {\n+\tmv rename renamed &&\n+\tmv dir1/rename dir1/renamed &&\n+\tmv dir2/rename dir2/renamed\n+}\n+\n+file_to_directory () {\n+\trm -f delete &&\n+\tmkdir delete &&\n+\techo 1 >delete/new\n+}\n+\n+directory_to_file () {\n+\trm -rf dir1 &&\n+\techo 1 >dir1\n+}\n+\n+# The next few test cases confirm that our fsmonitor daemon sees each type\n+# of OS filesystem notification that we care about. At this layer we just\n+# ensure we are getting the OS notifications and do not try to confirm what\n+# is reported by `git status`.\n+#\n+# We run a simple query after modifying the filesystem just to introduce\n+# a bit of a delay so that the trace logging from the daemon has time to\n+# get flushed to disk.\n+#\n+# We `reset` and `clean` at the bottom of each test (and before stopping the\n+# daemon) because these commands might implicitly restart the daemon.\n+\n+clean_up_repo_and_stop_daemon () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\trm -f .git/trace\n+}\n+\n+test_expect_success 'edit some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tedit_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/modified$\" .git/trace &&\n+\tgrep \"^event: dir2/modified$\" .git/trace &&\n+\tgrep \"^event: modified$\" .git/trace &&\n+\tgrep \"^event: dir1/untracked$\" .git/trace\n+'\n+\n+test_expect_success 'create some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tcreate_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/new$\" .git/trace &&\n+\tgrep \"^event: dir2/new$\" .git/trace &&\n+\tgrep \"^event: new$\" .git/trace\n+'\n+\n+test_expect_success 'delete some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tdelete_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/delete$\" .git/trace &&\n+\tgrep \"^event: dir2/delete$\" .git/trace &&\n+\tgrep \"^event: delete$\" .git/trace\n+'\n+\n+test_expect_success 'rename some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\trename_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/rename$\" .git/trace &&\n+\tgrep \"^event: dir2/rename$\" .git/trace &&\n+\tgrep \"^event: rename$\" .git/trace &&\n+\tgrep \"^event: dir1/renamed$\" .git/trace &&\n+\tgrep \"^event: dir2/renamed$\" .git/trace &&\n+\tgrep \"^event: renamed$\" .git/trace\n+'\n+\n+test_expect_success 'rename directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tmv dirtorename dirrenamed &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dirtorename/*$\" .git/trace &&\n+\tgrep \"^event: dirrenamed/*$\" .git/trace\n+'\n+\n+test_expect_success 'file changes to directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tfile_to_directory &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: delete$\" .git/trace &&\n+\tgrep \"^event: delete/new$\" .git/trace\n+'\n+\n+test_expect_success 'directory changes to a file' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tdirectory_to_file &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1$\" .git/trace\n+'\n+\n+# The next few test cases exercise the token-resync code. When filesystem\n+# drops events (because of filesystem velocity or because the daemon isn't\n+# polling fast enough), we need to discard the cached data (relative to the\n+# current token) and start collecting events under a new token.\n+#\n+# the 'test-tool fsmonitor-client flush' command can be used to send a\n+# \"flush\" message to a running daemon and ask it to do a flush/resync.\n+\n+test_expect_success 'flush cached data' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_flush\" &&\n+\n+\tgit init test_flush &&\n+\n+\tstart_daemon -C test_flush --tf \"$PWD/.git/trace_daemon\" --tk true &&\n+\n+\t# The daemon should have an initial token with no events in _0 and\n+\t# then a few (probably platform-specific number of) events in _1.\n+\t# These should both have the same .\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n+\tnul_to_q actual_q0 &&\n+\n+\t>test_flush/file_1 &&\n+\t>test_flush/file_2 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n+\tnul_to_q actual_q1 &&\n+\n+\tgrep \"file_1\" actual_q1 &&\n+\n+\t# Force a flush. This will change the , reset the , and\n+\t# flush the file data. Then create some events and ensure that the file\n+\t# again appears in the cache. It should have the new .\n+\n+\ttest-tool -C test_flush fsmonitor-client flush >flush_0 &&\n+\tnul_to_q flush_q0 &&\n+\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_2 &&\n+\tnul_to_q actual_q2 &&\n+\n+\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n+\n+\t>test_flush/file_3 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n+\tnul_to_q actual_q3 &&\n+\n+\tgrep \"file_3\" actual_q3\n+'\n+\n+# The next few test cases create repos where the .git directory is NOT\n+# inside the one of the working directory. That is, where .git is a file\n+# that points to a directory elsewhere. This happens for submodules and\n+# non-primary worktrees.\n+\n+test_expect_success 'setup worktree base' '\n+\tgit init wt-base &&\n+\techo 1 >wt-base/file1 &&\n+\tgit -C wt-base add file1 &&\n+\tgit -C wt-base commit -m \"c1\"\n+'\n+\n+test_expect_success 'worktree with .git file' '\n+\tgit -C wt-base worktree add ../wt-secondary &&\n+\n+\tstart_daemon -C wt-secondary \\\n+\t\t--tf \"$PWD/trace_wt_secondary\" \\\n+\t\t--t2 \"$PWD/trace2_wt_secondary\" &&\n+\n+\tgit -C wt-secondary fsmonitor--daemon stop &&\n+\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n+'\n+\n+# NEEDSWORK: Repeat one of the \"edit\" tests on wt-secondary and\n+# confirm that we get the same events and behavior -- that is, that\n+# fsmonitor--daemon correctly watches BOTH the working directory and\n+# the external GITDIR directory and behaves the same as when \".git\"\n+# is a directory inside the working directory.\n+\n+test_expect_success 'cleanup worktrees' '\n+\tstop_daemon_delete_repo wt-secondary &&\n+\tstop_daemon_delete_repo wt-base\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"452153","messageId":"2bb3eb84767b205a0bed2d00662aaa2b91de52e9.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 23/30] t/helper/test-chmtime: skip directories on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:39Z","receivedAt":"2022-03-24T16:51:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach `test-tool.exe chmtime` to ignore errors when setting the mtime\non a directory on Windows.\n\nNEEDSWORK: The Windows version of `utime()` (aka `mingw_utime()`) does\nnot properly handle directories because it uses `_wopen()`. It should\nbe converted to using `CreateFileW()` and backup semantics at a minimum.\nSince I'm already in the middle of a large patch series, I did not want\nto destabilize other callers of `utime()` right now. The problem has\nonly been observed in the t/perf/p7519 test when the test repo contains\nan empty directory on disk.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/helper/test-chmtime.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/t/helper/test-chmtime.c b/t/helper/test-chmtime.c\nindex 524b55ca496..dc28890a183 100644\n--- a/t/helper/test-chmtime.c\n+++ b/t/helper/test-chmtime.c\n@@ -134,6 +134,21 @@ int cmd__chmtime(int argc, const char **argv)\n \t\t}\n \n \t\tif (utb.modtime != sb.st_mtime && utime(argv[i], &utb) < 0) {\n+#ifdef GIT_WINDOWS_NATIVE\n+\t\t\tif (S_ISDIR(sb.st_mode)) {\n+\t\t\t\t/*\n+\t\t\t\t * NEEDSWORK: The Windows version of `utime()`\n+\t\t\t\t * (aka `mingw_utime()`) does not correctly\n+\t\t\t\t * handle directory arguments, since it uses\n+\t\t\t\t * `_wopen()`. Ignore it for now since this\n+\t\t\t\t * is just a test.\n+\t\t\t\t */\n+\t\t\t\tfprintf(stderr,\n+\t\t\t\t\t(\"Failed to modify time on directory %s. \"\n+\t\t\t\t\t \"Skipping\\n\"), argv[i]);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+#endif\n \t\t\tfprintf(stderr, \"Failed to modify time on %s: %s\\n\",\n \t\t\t argv[i], strerror(errno));\n \t\t\treturn 1;\n-- \ngitgitgadget\n\n"},{"id":"452158","messageId":"6eaa5765ae1cb24e1beff465a374a14d2ede16a0.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 26/30] t/perf/p7519: add fsmonitor--daemon test cases","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:42Z","receivedAt":"2022-03-24T16:51:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRepeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\nthe \"Simple IPC\" interface.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 38 ++++++++++++++++++++++++++++++++++----\n 1 file changed, 34 insertions(+), 4 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex a6c2a910e70..0b9129ca7bc 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -141,7 +141,7 @@ test_expect_success \"one time repo setup\" '\n \tfi\n '\n \n-setup_for_fsmonitor () {\n+setup_for_fsmonitor_hook () {\n \t# set INTEGRATION_SCRIPT depending on the environment\n \tif test -n \"$INTEGRATION_PATH\"\n \tthen\n@@ -182,7 +182,11 @@ test_perf_w_drop_caches () {\n }\n \n test_fsmonitor_suite () {\n-\tif test -n \"$INTEGRATION_SCRIPT\"; then\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n+\tthen\n+\t\tDESC=\"builtin fsmonitor--daemon\"\n+\telif test -n \"$INTEGRATION_SCRIPT\"\n+\tthen\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n \t\tDESC=\"fsmonitor=disabled\"\n@@ -261,11 +265,11 @@ test_fsmonitor_suite () {\n trace_start fsmonitor-watchman\n if test -n \"$GIT_PERF_7519_FSMONITOR\"; then\n \tfor INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do\n-\t\ttest_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor'\n+\t\ttest_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor_hook'\n \t\ttest_fsmonitor_suite\n \tdone\n else\n-\ttest_expect_success \"setup for fsmonitor\" 'setup_for_fsmonitor'\n+\ttest_expect_success \"setup for fsmonitor hook\" 'setup_for_fsmonitor_hook'\n \ttest_fsmonitor_suite\n fi\n \n@@ -293,4 +297,30 @@ test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n \n+#\n+# Run a full set of perf tests using the built-in fsmonitor--daemon.\n+# It does not use the Hook API, so it has a different setup.\n+# Explicitly start the daemon here and before we start client commands\n+# so that we can later add custom tracing.\n+#\n+if test_have_prereq FSMONITOR_DAEMON\n+then\n+\tUSE_FSMONITOR_DAEMON=t\n+\n+\ttest_expect_success \"setup for builtin fsmonitor\" '\n+\t\ttrace_start fsmonitor--daemon--server &&\n+\t\tgit fsmonitor--daemon start &&\n+\n+\t\ttrace_start fsmonitor--daemon--client &&\n+\n+\t\tgit config core.fsmonitor true &&\n+\t\tgit update-index --fsmonitor\n+\t'\n+\n+\ttest_fsmonitor_suite\n+\n+\tgit fsmonitor--daemon stop\n+\ttrace_stop\n+fi\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"452159","messageId":"bab9a9b0802dc6b0330b692b73d8850b3eb4e50e.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 24/30] t/perf/p7519: fix coding style","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:40Z","receivedAt":"2022-03-24T16:52:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex c8be58f3c76..5241eb6c4e5 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -72,7 +72,7 @@ then\n \tfi\n fi\n \n-trace_start() {\n+trace_start () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tname=\"$1\"\n@@ -91,7 +91,7 @@ trace_start() {\n \tfi\n }\n \n-trace_stop() {\n+trace_stop () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tunset GIT_TRACE2_PERF\n@@ -133,7 +133,7 @@ test_expect_success \"one time repo setup\" '\n \tfi\n '\n \n-setup_for_fsmonitor() {\n+setup_for_fsmonitor () {\n \t# set INTEGRATION_SCRIPT depending on the environment\n \tif test -n \"$INTEGRATION_PATH\"\n \tthen\n@@ -173,7 +173,7 @@ test_perf_w_drop_caches () {\n \ttest_perf \"$@\"\n }\n \n-test_fsmonitor_suite() {\n+test_fsmonitor_suite () {\n \tif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n-- \ngitgitgadget\n\n"},{"id":"452161","messageId":"2dd06ad2f712752e218de1d512edeedbed563ad4.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 25/30] t/perf/p7519: speed up test on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:41Z","receivedAt":"2022-03-24T16:52:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nChange p7519 to use `test_seq` and `xargs` rather than a `for` loop\nto touch thousands of files. This takes minutes off of test runs\non Windows because of process creation overhead.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 5241eb6c4e5..a6c2a910e70 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -98,6 +98,13 @@ trace_stop () {\n \tfi\n }\n \n+touch_files () {\n+\tn=$1 &&\n+\td=\"$n\"_files &&\n+\n+\t(cd $d && test_seq 1 $n | xargs touch )\n+}\n+\n test_expect_success \"one time repo setup\" '\n \t# set untrackedCache depending on the environment\n \tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n@@ -119,10 +126,11 @@ test_expect_success \"one time repo setup\" '\n \tfi &&\n \n \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n-\tfor i in $(test_seq 1 10); do touch 10_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n+\t: 1_file directory should be left empty &&\n+\ttouch_files 10 &&\n+\ttouch_files 100 &&\n+\ttouch_files 1000 &&\n+\ttouch_files 10000 &&\n \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n \tgit commit -qm \"Add files\" &&\n \n@@ -199,15 +207,15 @@ test_fsmonitor_suite () {\n \n \t# Update the mtimes on upto 100k files to make status think\n \t# that they are dirty. For simplicity, omit any files with\n-\t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n-\t# Then fully backslash-quote the paths to capture any\n-\t# whitespace so that they pass thru xargs properly.\n+\t# LFs (i.e. anything that ls-files thinks it needs to dquote)\n+\t# and any files with whitespace so that they pass thru xargs\n+\t# properly.\n \t#\n \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n \t\tgit ls-files | \\\n \t\t\thead -100000 | \\\n \t\t\tgrep -v \\\" | \\\n-\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n+\t\t\tgrep -v \" .\" | \\\n \t\t\txargs test-tool chmtime -300 &&\n \t\tgit status\n \t'\n-- \ngitgitgadget\n\n"},{"id":"452162","messageId":"30957f3930eb03d1dccbcae711b9e9a4383013e5.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 27/30] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:43Z","receivedAt":"2022-03-24T16:52:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to periodically truncate the list of\nmodified files to save some memory.\n\nClients will ask for the set of changes relative to a token that they\nfound in the FSMN index extension in the index. (This token is like a\npoint in time, but different). Clients will then update the index to\ncontain the response token (so that subsequent commands will be\nrelative to this new token).\n\nTherefore, the daemon can gradually truncate the in-memory list of\nchanged paths as they become obsolete (older than the previous token).\nSince we may have multiple clients making concurrent requests with a\nskew of tokens and clients may be racing to the talk to the daemon,\nwe lazily truncate the list.\n\nWe introduce a 5 minute delay and truncate batches 5 minutes after\nthey are considered obsolete.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 88 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 88 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex eafaafb45b1..ab9cc09f7ce 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -312,6 +312,75 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n \t\t\tbatch_src->interned_paths[k];\n }\n \n+/*\n+ * To keep the batch list from growing unbounded in response to filesystem\n+ * activity, we try to truncate old batches from the end of the list as\n+ * they become irrelevant.\n+ *\n+ * We assume that the .git/index will be updated with the most recent token\n+ * any time the index is updated. And future commands will only ask for\n+ * recent changes *since* that new token. So as tokens advance into the\n+ * future, older batch items will never be requested/needed. So we can\n+ * truncate them without loss of functionality.\n+ *\n+ * However, multiple commands may be talking to the daemon concurrently\n+ * or perform a slow command, so a little \"token skew\" is possible.\n+ * Therefore, we want this to be a little bit lazy and have a generous\n+ * delay.\n+ *\n+ * The current reader thread walked backwards in time from `token->batch_head`\n+ * back to `batch_marker` somewhere in the middle of the batch list.\n+ *\n+ * Let's walk backwards in time from that marker an arbitrary delay\n+ * and truncate the list there. Note that these timestamps are completely\n+ * artificial (based on when we pinned the batch item) and not on any\n+ * filesystem activity.\n+ *\n+ * Return the obsolete portion of the list after we have removed it from\n+ * the official list so that the caller can free it after leaving the lock.\n+ */\n+#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */\n+\n+static struct fsmonitor_batch *with_lock__truncate_old_batches(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst struct fsmonitor_batch *batch_marker)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder;\n+\n+\tif (!batch_marker)\n+\t\treturn NULL;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Truncate: mark (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t batch_marker->batch_seq_nr,\n+\t\t\t (uint64_t)batch_marker->pinned_time);\n+\n+\tfor (batch = batch_marker; batch; batch = batch->next) {\n+\t\ttime_t t;\n+\n+\t\tif (!batch->pinned_time) /* an overflow batch */\n+\t\t\tcontinue;\n+\n+\t\tt = batch->pinned_time + MY_TIME_DELAY_SECONDS;\n+\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n+\t\t\tcontinue;\n+\n+\t\tgoto truncate_past_here;\n+\t}\n+\n+\treturn NULL;\n+\n+truncate_past_here:\n+\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n+\n+\tremainder = ((struct fsmonitor_batch *)batch)->next;\n+\t((struct fsmonitor_batch *)batch)->next = NULL;\n+\n+\treturn remainder;\n+}\n+\n static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n {\n \tif (!token)\n@@ -425,6 +494,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tconst char *p;\n \tconst struct fsmonitor_batch *batch_head;\n \tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder = NULL;\n \tintmax_t count = 0, duplicates = 0;\n \tkh_str_t *shown;\n \tint hash_ret;\n@@ -652,11 +722,29 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * that work.\n \t\t\t */\n \t\t\tfsmonitor_free_token_data(token_data);\n+\t\t} else if (batch) {\n+\t\t\t/*\n+\t\t\t * We are holding the lock and are the only\n+\t\t\t * reader of the ref-counted portion of the\n+\t\t\t * list, so we get the honor of seeing if the\n+\t\t\t * list can be truncated to save memory.\n+\t\t\t *\n+\t\t\t * The main loop did not walk to the end of the\n+\t\t\t * list, so this batch is the first item in the\n+\t\t\t * batch-list that is older than the requested\n+\t\t\t * end-point sequence number. See if the tail\n+\t\t\t * end of the list is obsolete.\n+\t\t\t */\n+\t\t\tremainder = with_lock__truncate_old_batches(state,\n+\t\t\t\t\t\t\t\t batch);\n \t\t}\n \t}\n \n \tpthread_mutex_unlock(&state->main_lock);\n \n+\tif (remainder)\n+\t\tfsmonitor_batch__free_list(remainder);\n+\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n-- \ngitgitgadget\n\n"},{"id":"452165","messageId":"f87a1eba693b297d049281fa2d0c21b573027347.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 30/30] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:46Z","receivedAt":"2022-03-24T16:52:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate 2x2 test matrix with the untracked-cache and fsmonitor--daemon\nfeatures and a series of edits and verify that status output is\nidentical.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/t7527-builtin-fsmonitor.sh | 115 +++++++++++++++++++++++++++++++++++\n 1 file changed, 115 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 3570bf764aa..1c40d8d2035 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -216,6 +216,8 @@ test_expect_success 'setup' '\n \t.gitignore\n \texpect*\n \tactual*\n+\tflush*\n+\ttrace*\n \tEOF\n \n \tgit -c core.fsmonitor=false add . &&\n@@ -502,4 +504,117 @@ test_expect_success 'cleanup worktrees' '\n \tstop_daemon_delete_repo wt-base\n '\n \n+# The next few tests perform arbitrary/contrived file operations and\n+# confirm that status is correct. That is, that the data (or lack of\n+# data) from fsmonitor doesn't cause incorrect results. And doesn't\n+# cause incorrect results when the untracked-cache is enabled.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\tgit update-index --test-untracked-cache\n+'\n+\n+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n+\ttest_unconfig core.fsmonitor &&\n+\tgit update-index --no-fsmonitor &&\n+\ttest_might_fail git fsmonitor--daemon stop\n+'\n+\n+matrix_clean_up_repo () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd\n+}\n+\n+matrix_try () {\n+\tuc=$1 &&\n+\tfsm=$2 &&\n+\tfn=$3 &&\n+\n+\tif test $uc = true && test $fsm = false\n+\tthen\n+\t\t# The untracked-cache is buggy when FSMonitor is\n+\t\t# DISABLED, so skip the tests for this matrix\n+\t\t# combination.\n+\t\t#\n+\t\t# We've observed random, occasional test failures on\n+\t\t# Windows and MacOS when the UC is turned on and FSM\n+\t\t# is turned off. These are rare, but they do happen\n+\t\t# indicating that it is probably a race condition within\n+\t\t# the untracked cache itself.\n+\t\t#\n+\t\t# It usually happens when a test does F/D trickery and\n+\t\t# then the NEXT test fails because of extra status\n+\t\t# output from stale UC data from the previous test.\n+\t\t#\n+\t\t# Since FSMonitor is not involved in the error, skip\n+\t\t# the tests for this matrix combination.\n+\t\t#\n+\t\treturn 0\n+\tfi &&\n+\n+\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n+\t\tmatrix_clean_up_repo &&\n+\t\t$fn &&\n+\t\tif test $uc = false && test $fsm = false\n+\t\tthen\n+\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n+\t\telse\n+\t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n+\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n+\t\tfi\n+\t'\n+}\n+\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+for uc_val in $uc_values\n+do\n+\tif test $uc_val = false\n+\tthen\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n+\t\t\tgit config core.untrackedcache false &&\n+\t\t\tgit update-index --no-untracked-cache\n+\t\t'\n+\telse\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n+\t\t\tgit config core.untrackedcache true &&\n+\t\t\tgit update-index --untracked-cache\n+\t\t'\n+\tfi\n+\n+\tfsm_values=\"false true\"\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tif test $fsm_val = false\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\telse\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n+\t\t\t\tgit config core.fsmonitor true &&\n+\t\t\t\tgit fsmonitor--daemon start &&\n+\t\t\t\tgit update-index --fsmonitor\n+\t\t\t'\n+\t\tfi\n+\n+\t\tmatrix_try $uc_val $fsm_val edit_files\n+\t\tmatrix_try $uc_val $fsm_val delete_files\n+\t\tmatrix_try $uc_val $fsm_val create_files\n+\t\tmatrix_try $uc_val $fsm_val rename_files\n+\t\tmatrix_try $uc_val $fsm_val file_to_directory\n+\t\tmatrix_try $uc_val $fsm_val directory_to_file\n+\n+\t\tif test $fsm_val = true\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\tfi\n+\tdone\n+done\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"452174","messageId":"c8ca2a17277f8b8dab5e6cb832ba9f5cfa55466b.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 28/30] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:44Z","receivedAt":"2022-03-24T16:52:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon client threads to create a cookie file\ninside the .git directory and then wait until FS events for the\ncookie are observed by the FS listener thread.\n\nThis helps address the racy nature of file system events by\nblocking the client response until the kernel has drained any\nevent backlog.\n\nThis is especially important on MacOS where kernel events are\nonly issued with a limited frequency. See the `latency` argument\nof `FSeventStreamCreate()`. The kernel only signals every `latency`\nseconds, but does not guarantee that the kernel queue is completely\ndrained, so we may have to wait more than one interval. If we\nincrease the latency, the system is more likely to drop events.\nWe avoid these issues by having each client thread create a unique\ncookie file and then wait until it is seen in the event stream.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 237 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 5 +\n 2 files changed, 241 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex ab9cc09f7ce..46be55a4618 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -107,6 +107,162 @@ static int do_as_client__status(void)\n \t}\n }\n \n+enum fsmonitor_cookie_item_result {\n+\tFCIR_ERROR = -1, /* could not create cookie file ? */\n+\tFCIR_INIT,\n+\tFCIR_SEEN,\n+\tFCIR_ABORT,\n+};\n+\n+struct fsmonitor_cookie_item {\n+\tstruct hashmap_entry entry;\n+\tchar *name;\n+\tenum fsmonitor_cookie_item_result result;\n+};\n+\n+static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n+\t\t const struct hashmap_entry *he2, const void *keydata)\n+{\n+\tconst struct fsmonitor_cookie_item *a =\n+\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n+\tconst struct fsmonitor_cookie_item *b =\n+\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n+\n+\treturn strcmp(a->name, keydata ? keydata : b->name);\n+}\n+\n+static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(\n+\tstruct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint fd;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tstruct strbuf cookie_pathname = STRBUF_INIT;\n+\tstruct strbuf cookie_filename = STRBUF_INIT;\n+\tenum fsmonitor_cookie_item_result result;\n+\tint my_cookie_seq;\n+\n+\tCALLOC_ARRAY(cookie, 1);\n+\n+\tmy_cookie_seq = state->cookie_seq++;\n+\n+\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n+\n+\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n+\tstrbuf_addbuf(&cookie_pathname, &cookie_filename);\n+\n+\tcookie->name = strbuf_detach(&cookie_filename, NULL);\n+\tcookie->result = FCIR_INIT;\n+\thashmap_entry_init(&cookie->entry, strhash(cookie->name));\n+\n+\thashmap_add(&state->cookies, &cookie->entry);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n+\t\t\t cookie->name, cookie_pathname.buf);\n+\n+\t/*\n+\t * Create the cookie file on disk and then wait for a notification\n+\t * that the listener thread has seen it.\n+\t */\n+\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n+\tif (fd < 0) {\n+\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n+\t\t\t cookie->name);\n+\n+\t\tcookie->result = FCIR_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * Technically, close() and unlink() can fail, but we don't\n+\t * care here. We only created the file to trigger a watch\n+\t * event from the FS to know that when we're up to date.\n+\t */\n+\tclose(fd);\n+\tunlink(cookie_pathname.buf);\n+\n+\t/*\n+\t * Technically, this is an infinite wait (well, unless another\n+\t * thread sends us an abort). I'd like to change this to\n+\t * use `pthread_cond_timedwait()` and return an error/timeout\n+\t * and let the caller do the trivial response thing, but we\n+\t * don't have that routine in our thread-utils.\n+\t *\n+\t * After extensive beta testing I'm not really worried about\n+\t * this. Also note that the above open() and unlink() calls\n+\t * will cause at least two FS events on that path, so the odds\n+\t * of getting stuck are pretty slim.\n+\t */\n+\twhile (cookie->result == FCIR_INIT)\n+\t\tpthread_cond_wait(&state->cookies_cond,\n+\t\t\t\t &state->main_lock);\n+\n+done:\n+\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n+\n+\tresult = cookie->result;\n+\n+\tfree(cookie->name);\n+\tfree(cookie);\n+\tstrbuf_release(&cookie_pathname);\n+\n+\treturn result;\n+}\n+\n+/*\n+ * Mark these cookies as _SEEN and wake up the corresponding client threads.\n+ */\n+static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,\n+\t\t\t\t\t const struct string_list *cookie_names)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint k;\n+\tint nr_seen = 0;\n+\n+\tfor (k = 0; k < cookie_names->nr; k++) {\n+\t\tstruct fsmonitor_cookie_item key;\n+\t\tstruct fsmonitor_cookie_item *cookie;\n+\n+\t\tkey.name = cookie_names->items[k].string;\n+\t\thashmap_entry_init(&key.entry, strhash(key.name));\n+\n+\t\tcookie = hashmap_get_entry(&state->cookies, &key, entry, NULL);\n+\t\tif (cookie) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-seen: '%s'\",\n+\t\t\t\t\t cookie->name);\n+\t\t\tcookie->result = FCIR_SEEN;\n+\t\t\tnr_seen++;\n+\t\t}\n+\t}\n+\n+\tif (nr_seen)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n+/*\n+ * Set _ABORT on all pending cookies and wake up all client threads.\n+ */\n+static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct hashmap_iter iter;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tint nr_aborted = 0;\n+\n+\thashmap_for_each_entry(&state->cookies, &iter, cookie, entry) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-abort: '%s'\",\n+\t\t\t\t cookie->name);\n+\t\tcookie->result = FCIR_ABORT;\n+\t\tnr_aborted++;\n+\t}\n+\n+\tif (nr_aborted)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n /*\n * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n * \"token\" as a virtual timestamp. Clients can request a summary of all\n@@ -404,6 +560,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n *\n+ * [2] Some of those lost events may have been for cookie files. We\n+ * should assume the worst and abort them rather letting them starve.\n+ *\n * If there are no concurrent threads reading the current token data\n * series, we can free it now. Otherwise, let the last reader free\n * it.\n@@ -425,6 +584,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n \tstate->current_token_data = new_one;\n \n \tfsmonitor_free_token_data(free_me);\n+\n+\twith_lock__abort_all_cookies(state);\n }\n \n void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n@@ -500,6 +661,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tint hash_ret;\n \tint do_trivial = 0;\n \tint do_flush = 0;\n+\tint do_cookie = 0;\n+\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n@@ -560,6 +723,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * We have a V2 valid token:\n \t\t\t * \"builtin::\"\n \t\t\t */\n+\t\t\tdo_cookie = 1;\n \t\t}\n \t}\n \n@@ -568,6 +732,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tif (!state->current_token_data)\n \t\tBUG(\"fsmonitor state does not have a current token\");\n \n+\t/*\n+\t * Write a cookie file inside the directory being watched in\n+\t * an effort to flush out existing filesystem events that we\n+\t * actually care about. Suspend this client thread until we\n+\t * see the filesystem events for this cookie file.\n+\t *\n+\t * Creating the cookie lets us guarantee that our FS listener\n+\t * thread has drained the kernel queue and we are caught up\n+\t * with the kernel.\n+\t *\n+\t * If we cannot create the cookie (or otherwise guarantee that\n+\t * we are caught up), we send a trivial response. We have to\n+\t * assume that there might be some very, very recent activity\n+\t * on the FS still in flight.\n+\t */\n+\tif (do_cookie) {\n+\t\tcookie_result = with_lock__wait_for_cookie(state);\n+\t\tif (cookie_result != FCIR_SEEN) {\n+\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n+\t\t\t cookie_result);\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n \tif (do_flush)\n \t\twith_lock__do_force_resync(state);\n \n@@ -787,7 +975,9 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n-#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+#define FSMONITOR_DIR \"fsmonitor--daemon\"\n+#define FSMONITOR_COOKIE_DIR \"cookies\"\n+#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n \n enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n \tconst char *rel)\n@@ -940,6 +1130,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n+\tif (cookie_names->nr)\n+\t\twith_lock__mark_cookies_seen(state, cookie_names);\n+\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n@@ -1031,7 +1224,9 @@ static int fsmonitor_run_daemon(void)\n \n \tmemset(&state, 0, sizeof(state));\n \n+\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n+\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n@@ -1056,6 +1251,44 @@ static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n+\t/*\n+\t * We will write filesystem syncing cookie files into\n+\t * ///-.\n+\t *\n+\t * The extra layers of subdirectories here keep us from\n+\t * changing the mtime on \".git/\" or \".git/foo/\" when we create\n+\t * or delete cookie files.\n+\t *\n+\t * There have been problems with some IDEs that do a\n+\t * non-recursive watch of the \".git/\" directory and run a\n+\t * series of commands any time something happens.\n+\t *\n+\t * For example, if we place our cookie files directly in\n+\t * \".git/\" or \".git/foo/\" then a `git status` (or similar\n+\t * command) from the IDE will cause a cookie file to be\n+\t * created in one of those dirs. This causes the mtime of\n+\t * those dirs to change. This triggers the IDE's watch\n+\t * notification. This triggers the IDE to run those commands\n+\t * again. And the process repeats and the machine never goes\n+\t * idle.\n+\t *\n+\t * Adding the extra layers of subdirectories prevents the\n+\t * mtime of \".git/\" and \".git/foo\" from changing when a\n+\t * cookie file is created.\n+\t */\n+\tstrbuf_init(&state.path_cookie_prefix, 0);\n+\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1068,6 +1301,7 @@ static int fsmonitor_run_daemon(void)\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n+\tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n \n@@ -1075,6 +1309,7 @@ done:\n \n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n+\tstrbuf_release(&state.path_cookie_prefix);\n \n \treturn err;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 010fbfe60e9..bd09fffc176 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n \n \tstruct fsmonitor_token_data *current_token_data;\n \n+\tstruct strbuf path_cookie_prefix;\n+\tpthread_cond_t cookies_cond;\n+\tint cookie_seq;\n+\tstruct hashmap cookies;\n+\n \tint error_code;\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n-- \ngitgitgadget\n\n"},{"id":"452175","messageId":"4caf1d89b841726e9b76372bd6b238d5ebbffa5e.1648140586.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v8 29/30] fsmonitor: force update index after large responses","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:49:45Z","receivedAt":"2022-03-24T16:52:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMeasure the time taken to apply the FSMonitor query result\nto the index and the untracked-cache.\n\nSet the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when\nFSMonitor returns a very large repsonse to ensure that the index is\nwritten to disk.\n\nNormally, when the FSMonitor response includes a tracked file, the\nindex is always updated. Similarly, the index might be updated when\nthe response alters the untracked-cache (when enabled). However, in\ncases where neither of those cause the index to be considered changed,\nthe FSMonitor response is wasted. Subsequent Git commands will make\nrequests with the same token and receive the same response.\n\nIf that response is very large, performance may suffer. It would be\nmore efficient to force update the index now (and the token in the\nindex extension) in order to reduce the size of the response received\nby future commands.\n\nThis was observed on Windows after a large checkout. On Windows, the\nkernel emits events for the files that are changed as they are\nchanged. However, it might delay events for the containing\ndirectories until the system is more idle (or someone scans the\ndirectory (so it seems)). The first status following a checkout would\nget the list of files. The subsequent status commands would get the\nlist of directories as the events trickled out. But they would never\ncatch up because the token was not advanced because the index wasn't\nupdated.\n\nThis list of directories caused `wt_status_collect_untracked()` to\nunnecessarily spend time actually scanning them during each command.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 54 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex a38b5710eb3..292a6742b4f 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -219,6 +219,43 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n+/*\n+ * The number of pathnames that we need to receive from FSMonitor\n+ * before we force the index to be updated.\n+ *\n+ * Note that any pathname within the set of received paths MAY cause\n+ * cache-entry or istate flag bits to be updated and thus cause the\n+ * index to be updated on disk.\n+ *\n+ * However, the response may contain many paths (such as ignored\n+ * paths) that will not update any flag bits. And thus not force the\n+ * index to be updated. (This is fine and normal.) It also means\n+ * that the token will not be updated in the FSMonitor index\n+ * extension. So the next Git command will find the same token in the\n+ * index, make the same token-relative request, and receive the same\n+ * response (plus any newly changed paths). If this response is large\n+ * (and continues to grow), performance could be impacted.\n+ *\n+ * For example, if the user runs a build and it writes 100K object\n+ * files but doesn't modify any source files, the index would not need\n+ * to be updated. The FSMonitor response (after the build and\n+ * relative to a pre-build token) might be 5MB. Each subsequent Git\n+ * command will receive that same 100K/5MB response until something\n+ * causes the index to be updated. And `refresh_fsmonitor()` will\n+ * have to iterate over those 100K paths each time.\n+ *\n+ * Performance could be improved if we optionally force update the\n+ * index after a very large response and get an updated token into\n+ * the FSMonitor index extension. This should allow subsequent\n+ * commands to get smaller and more current responses.\n+ *\n+ * The value chosen here does not need to be precise. The index\n+ * will be updated automatically the first time the user touches\n+ * a tracked file and causes a command like `git status` to\n+ * update an mtime to be updated and/or set a flag bit.\n+ */\n+static int fsmonitor_force_update_threshold = 100;\n+\n void refresh_fsmonitor(struct index_state *istate)\n {\n \tstruct strbuf query_result = STRBUF_INIT;\n@@ -362,25 +399,39 @@ apply_results:\n \t * information and that we should consider everything\n \t * invalid. We call this a trivial response.\n \t */\n+\ttrace2_region_enter(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tif (query_success && !is_trivial) {\n \t\t/*\n \t\t * Mark all pathnames returned by the monitor as dirty.\n \t\t *\n \t\t * This updates both the cache-entries and the untracked-cache.\n \t\t */\n+\t\tint count = 0;\n+\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n \t\t\t\tcontinue;\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n \t\t\tbol = i + 1;\n+\t\t\tcount++;\n \t\t}\n-\t\tif (bol < query_result.len)\n+\t\tif (bol < query_result.len) {\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tcount++;\n+\t\t}\n \n \t\t/* Now mark the untracked cache for fsmonitor usage */\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n+\n+\t\tif (count > fsmonitor_force_update_threshold)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", istate->repo, \"apply_count\",\n+\t\t\t\t count);\n+\n \t} else {\n \t\t/*\n \t\t * We failed to get a response or received a trivial response,\n@@ -409,6 +460,8 @@ apply_results:\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 0;\n \t}\n+\ttrace2_region_leave(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tstrbuf_release(&query_result);\n \n \t/* Now that we've updated istate, save the last_update_token */\n-- \ngitgitgadget\n\n"},{"id":"452215","messageId":"xmqqils3gort.fsf@gitster.g","threadId":"56518","inReplyTo":"c8709da9457eb303132b5cad6a204a1de27aabc0.1648140586.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-24T18:59:34Z","receivedAt":"2022-03-24T18:59:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Junio C Hamano \n\nI hadn't signed off on this one yet ;-)\n\n> +is_value () {\n> +\ttest -n \"$1\" && test \"${1::1}\" != \"-\"\n> +}\n\n${var:ofs:len} is a bash-ism. If you run this test under /bin/dash\ninstead of /bin/dash, you'll likely see it fail.\n\nIf it were a good idea to see if $1 begins with a dash, a more\nnatural (to shell programmers) way to do so is\n\n\tcase \"$1\" in -*) false ;; ?*) true ;; *) false ;; esac\n\nbut given how this is used below, we do not want to special case\ndash. \n\nThere isn't anything wrong in \"mkdir ./-foo && start_daemon -C -foo\"\nin other words.\n\n> +start_daemon () {\n> +\tr= &&\n> +\ttf= &&\n> +\tt2= &&\n> +\ttk= &&\n\nFYI, you can write these on a single line, i.e.\n\n\tr= tf= t2= tk= &&\n\nSpending lines and spaces for the meat of the script would enhance\nreadability but for things like a boilerplate \"we clear variables\nbefore using them\", being concise may be less distracting.\n\n> +\twhile test \"$#\" -ne 0\n> +\tdo\n> +\t\tcase \"$1\" in\n> +\t\t-C)\n> +\t\t\tshift;\n> +\t\t\tis_value $1 || BUG \"error: -C requires value\"\n> +\t\t\tr=\"-C $1\"\n> +\t\t\tshift\n> +\t\t\t;;\n> +\t...\n> +\t\tesac\n> +\tdone &&\n\nA more natural way to write these loops is\n\n\twhile ...\n\tdo\n\t\tcase \"$1\" in\n\t\t-C)\n\t\t\tr=\"-C ${2?}\"\n\t\t\tshift\n\t\t\t;;\n\t\t... all other options you handle ...\n\t\t-*)\n\t\t\techo >&2 \"unknown option $1\"\n\t\t\texit 1\n\t\t\t;;\n\t\t*)\n\t\t\tbreak\n\t\t\t;;\n\t\tesac\n\t\tshift\n\tdone\n\ni.e. shifting out what we just saw is the default and happens\nimmediately after the case/esac, and extra shift after consuming\nan option parameter happens in each case arm.\n\nAn acceptable slight variation is\n\n\t\t-C)\n\t\t\tshift\n\t\t\tr=\"-C ${1?}\"\n\t\t\t;;\n\nbut the first form is more logical and clear, i.e. \"when we see '-C',\nwe want two on the command line, -C itself and the parameter it takes\"\nis conveyed more strongly with \"${2?}\" there.\n\nFor an additional bonus, we could also accept the stuck form, i.e.\n\n\t\tcase \"$1\" in\n\t\t-C)\n\t\t\tr=\"-C ${2?}\"\n\t\t\tshift\n\t\t\t;;\n\t\t-C*)\n\t\t\tr=\"-C {$1#-C}\"\n\t\t\t;;\n\t\t...\n"},{"id":"452217","messageId":"075d01d83fb2$1506ded0$3f149c70$@nexbridge.com","threadId":"56518","inReplyTo":"xmqqils3gort.fsf@gitster.g","subject":"RE: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2022-03-24T19:05:00Z","receivedAt":"2022-03-24T19:05:13Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On March 24, 2022 3:00 PM, Junio C Hamano wrote:\n>Subject: Re: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon\n>\n>\"Jeff Hostetler via GitGitGadget\" writes:\n>\n>> From: Jeff Hostetler \n>>\n>> Signed-off-by: Jeff Hostetler \n>> Signed-off-by: Junio C Hamano \n>\n>I hadn't signed off on this one yet ;-)\n>\n>> +is_value () {\n>> +\ttest -n \"$1\" && test \"${1::1}\" != \"-\"\n>> +}\n>\n>${var:ofs:len} is a bash-ism. If you run this test under /bin/dash instead of\n>/bin/dash, you'll likely see it fail.\n>\n>If it were a good idea to see if $1 begins with a dash, a more natural (to shell\n>programmers) way to do so is\n>\n>\tcase \"$1\" in -*) false ;; ?*) true ;; *) false ;; esac\n>\n>but given how this is used below, we do not want to special case dash.\n>\n>There isn't anything wrong in \"mkdir ./-foo && start_daemon -C -foo\"\n>in other words.\n>\n>> +start_daemon () {\n>> +\tr= &&\n>> +\ttf= &&\n>> +\tt2= &&\n>> +\ttk= &&\n>\n>FYI, you can write these on a single line, i.e.\n>\n>\tr= tf= t2= tk= &&\n>\n>Spending lines and spaces for the meat of the script would enhance readability but\n>for things like a boilerplate \"we clear variables before using them\", being concise\n>may be less distracting.\n>\n>> +\twhile test \"$#\" -ne 0\n>> +\tdo\n>> +\t\tcase \"$1\" in\n>> +\t\t-C)\n>> +\t\t\tshift;\n>> +\t\t\tis_value $1 || BUG \"error: -C requires value\"\n>> +\t\t\tr=\"-C $1\"\n>> +\t\t\tshift\n>> +\t\t\t;;\n>> +\t...\n>> +\t\tesac\n>> +\tdone &&\n>\n>A more natural way to write these loops is\n>\n>\twhile ...\n>\tdo\n>\t\tcase \"$1\" in\n>\t\t-C)\n>\t\t\tr=\"-C ${2?}\"\n>\t\t\tshift\n>\t\t\t;;\n>\t\t... all other options you handle ...\n>\t\t-*)\n>\t\t\techo >&2 \"unknown option $1\"\n>\t\t\texit 1\n>\t\t\t;;\n>\t\t*)\n>\t\t\tbreak\n>\t\t\t;;\n>\t\tesac\n>\t\tshift\n>\tdone\n>\n>i.e. shifting out what we just saw is the default and happens immediately after the\n>case/esac, and extra shift after consuming an option parameter happens in each\n>case arm.\n>\n>An acceptable slight variation is\n>\n>\t\t-C)\n>\t\t\tshift\n>\t\t\tr=\"-C ${1?}\"\n>\t\t\t;;\n>\n>but the first form is more logical and clear, i.e. \"when we see '-C', we want two on\n>the command line, -C itself and the parameter it takes\"\n>is conveyed more strongly with \"${2?}\" there.\n>\n>For an additional bonus, we could also accept the stuck form, i.e.\n>\n>\t\tcase \"$1\" in\n>\t\t-C)\n>\t\t\tr=\"-C ${2?}\"\n>\t\t\tshift\n>\t\t\t;;\n>\t\t-C*)\n>\t\t\tr=\"-C {$1#-C}\"\n>\t\t\t;;\n>\t\t...\n\nMay I request a bit of extra time on the -rc0 to -rc1 cycle for this? I have a feeling that while testing this is probably going to go well, I would like to have a bit of extra time for anything that might come up. There are a lot of moving parts to this series. Not being critical, but debugging scripts on my platforms can be a bit rough at times.\n\nThanks,\nRandall\n\n"},{"id":"452222","messageId":"0deea44e-b919-81cb-d8bf-ebea4c9ba426@jeffhostetler.com","threadId":"56518","inReplyTo":"075d01d83fb2$1506ded0$3f149c70$@nexbridge.com","subject":"Re: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-24T20:27:35Z","receivedAt":"2022-03-24T20:27:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/24/22 3:05 PM, rsbecker@nexbridge.com wrote:\n> On March 24, 2022 3:00 PM, Junio C Hamano wrote:\n>> Subject: Re: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon\n>>\n>> \"Jeff Hostetler via GitGitGadget\" writes:\n>>\n>>> From: Jeff Hostetler \n[...]\n> \n> May I request a bit of extra time on the -rc0 to -rc1 cycle for this? I have a feeling that while testing this is probably going to go well, I would like to have a bit of extra time for anything that might come up. There are a lot of moving parts to this series. Not being critical, but debugging scripts on my platforms can be a bit rough at times.\n> \n> Thanks,\n> Randall\n> \n\nI'll simplify the `start_daemon()` function as Junio suggests,\nso hopefully that'll reduce the amount of debugging that you need.\n\nBTW, which platforms are you worried about?\n\nJeff\n"},{"id":"452224","messageId":"076101d83fbe$e6ce5e50$b46b1af0$@nexbridge.com","threadId":"56518","inReplyTo":"0deea44e-b919-81cb-d8bf-ebea4c9ba426@jeffhostetler.com","subject":"RE: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2022-03-24T20:36:49Z","receivedAt":"2022-03-24T20:36:58Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On March 24, 2022 4:28 PM, Jeff Hostetler wrote:\n>On 3/24/22 3:05 PM, rsbecker@nexbridge.com wrote:\n>> On March 24, 2022 3:00 PM, Junio C Hamano wrote:\n>>> Subject: Re: [PATCH v8 21/30] t7527: create test for\n>>> fsmonitor--daemon\n>>>\n>>> \"Jeff Hostetler via GitGitGadget\" writes:\n>>>\n>>>> From: Jeff Hostetler \n>[...]\n>>\n>> May I request a bit of extra time on the -rc0 to -rc1 cycle for this? I have a feeling\n>that while testing this is probably going to go well, I would like to have a bit of\n>extra time for anything that might come up. There are a lot of moving parts to this\n>series. Not being critical, but debugging scripts on my platforms can be a bit rough\n>at times.\n>>\n>> Thanks,\n>> Randall\n>>\n>\n>I'll simplify the `start_daemon()` function as Junio suggests, so hopefully that'll\n>reduce the amount of debugging that you need.\nThanks.\n\n>BTW, which platforms are you worried about?\nI'm worried about NonStop ia64 and x86. It's not just this series but also the fsync series. I think it's going to be a bit of a rid this time, but hoping not.\n--Randall\n\n"},{"id":"452225","messageId":"cd5dd46d-fc47-1af8-6ba7-ddf1617a7b2b@jeffhostetler.com","threadId":"56518","inReplyTo":"076101d83fbe$e6ce5e50$b46b1af0$@nexbridge.com","subject":"Re: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-24T20:42:46Z","receivedAt":"2022-03-24T20:42:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/24/22 4:36 PM, rsbecker@nexbridge.com wrote:\n> On March 24, 2022 4:28 PM, Jeff Hostetler wrote:\n>> On 3/24/22 3:05 PM, rsbecker@nexbridge.com wrote:\n>>> On March 24, 2022 3:00 PM, Junio C Hamano wrote:\n>>>> Subject: Re: [PATCH v8 21/30] t7527: create test for\n>>>> fsmonitor--daemon\n>>>>\n>>>> \"Jeff Hostetler via GitGitGadget\" writes:\n>>>>\n>>>>> From: Jeff Hostetler \n>> [...]\n>>>\n>>> May I request a bit of extra time on the -rc0 to -rc1 cycle for this? I have a feeling\n>> that while testing this is probably going to go well, I would like to have a bit of\n>> extra time for anything that might come up. There are a lot of moving parts to this\n>> series. Not being critical, but debugging scripts on my platforms can be a bit rough\n>> at times.\n>>>\n>>> Thanks,\n>>> Randall\n>>>\n>>\n>> I'll simplify the `start_daemon()` function as Junio suggests, so hopefully that'll\n>> reduce the amount of debugging that you need.\n> Thanks.\n> \n>> BTW, which platforms are you worried about?\n> I'm worried about NonStop ia64 and x86. It's not just this series but also the fsync series. I think it's going to be a bit of a rid this time, but hoping not.\n> --Randall\n> \n\nI currently only have drivers/backends for Windows and MacOS for the\nbuiltin FSMonitor daemon feature. Since it relies on platform-specific\ncode to read whatever filesystem journal or event stream that the\nthe platform provides.\n\nSo t7527 should do a skip_all on NonStop IIUC.\n\nJeff\n"},{"id":"452226","messageId":"8c9e30be-5138-ecfe-5c37-4f91b4d3002f@jeffhostetler.com","threadId":"56518","inReplyTo":"xmqqils3gort.fsf@gitster.g","subject":"Re: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-24T20:45:53Z","receivedAt":"2022-03-24T20:45:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/24/22 2:59 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> From: Jeff Hostetler \n>>\n>> Signed-off-by: Jeff Hostetler \n>> Signed-off-by: Junio C Hamano \n> \n> I hadn't signed off on this one yet ;-)\n\nSorry, I started a recent rebase based upon your version\nof the commits and it had your signoff in it.\n\n> \n>> +is_value () {\n>> +\ttest -n \"$1\" && test \"${1::1}\" != \"-\"\n>> +}\n> \n> ${var:ofs:len} is a bash-ism. If you run this test under /bin/dash\n> instead of /bin/dash, you'll likely see it fail.\n[...]\n\nI'll simplify the function as you suggested and send a V9 in\nthe morning after GGG's CI finishes.\n\nThanks\nJeff\n\n"},{"id":"452227","messageId":"076201d83fc0$4a202140$de6063c0$@nexbridge.com","threadId":"56518","inReplyTo":"cd5dd46d-fc47-1af8-6ba7-ddf1617a7b2b@jeffhostetler.com","subject":"RE: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2022-03-24T20:46:45Z","receivedAt":"2022-03-24T20:48:23Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On: March 24, 2022 4:43 PM, Jeff Hostetler wrote:\n>On 3/24/22 4:36 PM, rsbecker@nexbridge.com wrote:\n>> On March 24, 2022 4:28 PM, Jeff Hostetler wrote:\n>>> On 3/24/22 3:05 PM, rsbecker@nexbridge.com wrote:\n>>>> On March 24, 2022 3:00 PM, Junio C Hamano wrote:\n>>>>> Subject: Re: [PATCH v8 21/30] t7527: create test for\n>>>>> fsmonitor--daemon\n>>>>>\n>>>>> \"Jeff Hostetler via GitGitGadget\" writes:\n>>>>>\n>>>>>> From: Jeff Hostetler \n>>> [...]\n>>>>\n>>>> May I request a bit of extra time on the -rc0 to -rc1 cycle for\n>>>> this? I have a feeling\n>>> that while testing this is probably going to go well, I would like to\n>>> have a bit of extra time for anything that might come up. There are a\n>>> lot of moving parts to this series. Not being critical, but debugging\n>>> scripts on my platforms can be a bit rough at times.\n>>>>\n>>>> Thanks,\n>>>> Randall\n>>>>\n>>>\n>>> I'll simplify the `start_daemon()` function as Junio suggests, so\n>>> hopefully that'll reduce the amount of debugging that you need.\n>> Thanks.\n>>\n>>> BTW, which platforms are you worried about?\n>> I'm worried about NonStop ia64 and x86. It's not just this series but also the\n>fsync series. I think it's going to be a bit of a rid this time, but hoping not.\n>> --Randall\n>>\n>\n>I currently only have drivers/backends for Windows and MacOS for the builtin\n>FSMonitor daemon feature. Since it relies on platform-specific code to read\n>whatever filesystem journal or event stream that the the platform provides.\n>\n>So t7527 should do a skip_all on NonStop IIUC.\n\nI would have thought this was applicable to Linux also. So use, it should skip. Be in touch when if you do down that path.\n--Randall\n\n"},{"id":"452230","messageId":"5f2bbcf3-fbaf-b1b4-7e53-b3f48cd971c2@jeffhostetler.com","threadId":"56518","inReplyTo":"076201d83fc0$4a202140$de6063c0$@nexbridge.com","subject":"Re: [PATCH v8 21/30] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-24T20:51:18Z","receivedAt":"2022-03-24T20:52:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/24/22 4:46 PM, rsbecker@nexbridge.com wrote:\n> On: March 24, 2022 4:43 PM, Jeff Hostetler wrote:\n>> On 3/24/22 4:36 PM, rsbecker@nexbridge.com wrote:\n>>> On March 24, 2022 4:28 PM, Jeff Hostetler wrote:\n>>>> On 3/24/22 3:05 PM, rsbecker@nexbridge.com wrote:\n>>>>> On March 24, 2022 3:00 PM, Junio C Hamano wrote:\n>>>>>> Subject: Re: [PATCH v8 21/30] t7527: create test for\n>>>>>> fsmonitor--daemon\n>>>>>>\n>>>>>> \"Jeff Hostetler via GitGitGadget\" writes:\n>>>>>>\n>>>>>>> From: Jeff Hostetler \n>>>> [...]\n>>>>>\n>>>>> May I request a bit of extra time on the -rc0 to -rc1 cycle for\n>>>>> this? I have a feeling\n>>>> that while testing this is probably going to go well, I would like to\n>>>> have a bit of extra time for anything that might come up. There are a\n>>>> lot of moving parts to this series. Not being critical, but debugging\n>>>> scripts on my platforms can be a bit rough at times.\n>>>>>\n>>>>> Thanks,\n>>>>> Randall\n>>>>>\n>>>>\n>>>> I'll simplify the `start_daemon()` function as Junio suggests, so\n>>>> hopefully that'll reduce the amount of debugging that you need.\n>>> Thanks.\n>>>\n>>>> BTW, which platforms are you worried about?\n>>> I'm worried about NonStop ia64 and x86. It's not just this series but also the\n>> fsync series. I think it's going to be a bit of a rid this time, but hoping not.\n>>> --Randall\n>>>\n>>\n>> I currently only have drivers/backends for Windows and MacOS for the builtin\n>> FSMonitor daemon feature. Since it relies on platform-specific code to read\n>> whatever filesystem journal or event stream that the the platform provides.\n>>\n>> So t7527 should do a skip_all on NonStop IIUC.\n> \n> I would have thought this was applicable to Linux also. So use, it should skip. Be in touch when if you do down that path.\n> --Randall\n> \n\nSorry, yeah, I wanted to do a Linux version too, but there was just too\nmuch platform-specific stuff to juggle three different backends at\nthe same time. I have the APIs in place between platform-specific and\n-independent code so that it shouldn't be hard to add a Linux version.\nI just didn't have the capacity to do all three (and the rest of\n$dayjob....)\n\nJeff\n\n"},{"id":"452303","messageId":"50c2afaa49e20f2b19824acb2ff6a64439e9207c.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 25/30] t/perf/p7519: speed up test on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:08Z","receivedAt":"2022-03-25T19:24:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nChange p7519 to use `test_seq` and `xargs` rather than a `for` loop\nto touch thousands of files. This takes minutes off of test runs\non Windows because of process creation overhead.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 5241eb6c4e5..a6c2a910e70 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -98,6 +98,13 @@ trace_stop () {\n \tfi\n }\n \n+touch_files () {\n+\tn=$1 &&\n+\td=\"$n\"_files &&\n+\n+\t(cd $d && test_seq 1 $n | xargs touch )\n+}\n+\n test_expect_success \"one time repo setup\" '\n \t# set untrackedCache depending on the environment\n \tif test -n \"$GIT_PERF_7519_UNTRACKED_CACHE\"\n@@ -119,10 +126,11 @@ test_expect_success \"one time repo setup\" '\n \tfi &&\n \n \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n-\tfor i in $(test_seq 1 10); do touch 10_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&\n-\tfor i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&\n+\t: 1_file directory should be left empty &&\n+\ttouch_files 10 &&\n+\ttouch_files 100 &&\n+\ttouch_files 1000 &&\n+\ttouch_files 10000 &&\n \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n \tgit commit -qm \"Add files\" &&\n \n@@ -199,15 +207,15 @@ test_fsmonitor_suite () {\n \n \t# Update the mtimes on upto 100k files to make status think\n \t# that they are dirty. For simplicity, omit any files with\n-\t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n-\t# Then fully backslash-quote the paths to capture any\n-\t# whitespace so that they pass thru xargs properly.\n+\t# LFs (i.e. anything that ls-files thinks it needs to dquote)\n+\t# and any files with whitespace so that they pass thru xargs\n+\t# properly.\n \t#\n \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n \t\tgit ls-files | \\\n \t\t\thead -100000 | \\\n \t\t\tgrep -v \\\" | \\\n-\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n+\t\t\tgrep -v \" .\" | \\\n \t\t\txargs test-tool chmtime -300 &&\n \t\tgit status\n \t'\n-- \ngitgitgadget\n\n"},{"id":"452304","messageId":"e04c7301f249a75a33e5498258a0dc628f3ebe66.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 03/30] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:46Z","receivedAt":"2022-03-25T19:25:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove fsmonitor config settings to a new and opaque\n`struct fsmonitor_settings` structure. Add a lazily-loaded pointer\nto this into `struct repo_settings`\n\nCreate an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to\nrepresent the state of fsmonitor. This lets us represent which, if\nany, fsmonitor provider (hook or IPC) is enabled.\n\nCreate `fsm_settings__get_*()` getters to lazily look up fsmonitor-\nrelated config settings.\n\nGet rid of the `core_fsmonitor` global variable. Move the code to\nlookup the existing `core.fsmonitor` config value into the fsmonitor\nsettings.\n\nCreate a hook pathname variable in `struct fsmonitor-settings` and\nonly set it when in hook mode.\n\nExtend the definition of `core.fsmonitor` to be either a boolean\nor a hook pathname. When true, the builtin FSMonitor is used.\nWhen false or unset, no FSMonitor (neither builtin nor hook) is\nused.\n\nThe existing `core_fsmonitor` global variable was used to store the\npathname to the fsmonitor hook *and* it was used as a boolean to see\nif fsmonitor was enabled. This dual usage and global visibility leads\nto confusion when we add the IPC-based provider. So lets hide the\ndetails in fsmonitor-settings.c and let it decide which provider to\nuse in the case of multiple settings. This avoids cluttering up\nrepo-settings.c with these private details.\n\nA future commit in builtin-fsmonitor series will add the ability to\ndisqualify worktrees for various reasons, such as being mounted from a\nremote volume, where fsmonitor should not be started. Having the\nconfig settings hidden in fsmonitor-settings.c allows such worktree\nrestrictions to override the config values used.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n builtin/update-index.c | 7 ++-\n cache.h | 1 -\n config.c | 14 -----\n config.h | 1 -\n environment.c | 1 -\n fsmonitor-settings.c | 114 +++++++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 21 ++++++++\n fsmonitor.c | 63 ++++++++++++++---------\n fsmonitor.h | 15 ++++--\n repository.h | 3 ++\n t/README | 4 +-\n 12 files changed, 196 insertions(+), 49 deletions(-)\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n\ndiff --git a/Makefile b/Makefile\nindex a19d850e716..707a56d4c11 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -908,6 +908,7 @@ LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n LIB_OBJS += fsmonitor-ipc.o\n+LIB_OBJS += fsmonitor-settings.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex aafe7eeac2a..876112abb21 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1236,14 +1236,17 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\n \t\t\t\t\"enable fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tif (fsm_mode > FSMONITOR_MODE_DISABLED)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex 04d4d2db25c..aaf334e2aa4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -999,7 +999,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex 383b1a4885b..3f9b0739a78 100644\n--- a/config.c\n+++ b/config.c\n@@ -2626,20 +2626,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex bb49baf1ee0..7654f61c634 100644\n--- a/config.h\n+++ b/config.h\n@@ -597,7 +597,6 @@ int git_config_get_pathname(const char *key, const char **dest);\n int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a5..00682e638d7 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nnew file mode 100644\nindex 00000000000..757d230d538\n--- /dev/null\n+++ b/fsmonitor-settings.c\n@@ -0,0 +1,114 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+/*\n+ * We keep this structure defintion private and have getters\n+ * for all fields so that we can lazy load it as needed.\n+ */\n+struct fsmonitor_settings {\n+\tenum fsmonitor_mode mode;\n+\tchar *hook_path;\n+};\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n+\tstruct fsmonitor_settings *s;\n+\tconst char *const_str;\n+\tint bool_value;\n+\n+\tif (r->settings.fsmonitor)\n+\t\treturn;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\n+\tr->settings.fsmonitor = s;\n+\n+\t/*\n+\t * Overload the existing \"core.fsmonitor\" config setting (which\n+\t * has historically been either unset or a hook pathname) to\n+\t * now allow a boolean value to enable the builtin FSMonitor\n+\t * or to turn everything off. (This does imply that you can't\n+\t * use a hook script named \"true\" or \"false\", but that's OK.)\n+\t */\n+\tswitch (repo_config_get_maybe_bool(r, \"core.fsmonitor\", &bool_value)) {\n+\n+\tcase 0: /* config value was set to */\n+\t\tif (bool_value)\n+\t\t\tfsm_settings__set_ipc(r);\n+\t\treturn;\n+\n+\tcase 1: /* config value was unset */\n+\t\tconst_str = getenv(\"GIT_TEST_FSMONITOR\");\n+\t\tbreak;\n+\n+\tcase -1: /* config value set to an arbitrary string */\n+\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_str))\n+\t\t\treturn; /* should not happen */\n+\t\tbreak;\n+\n+\tdefault: /* should not happen */\n+\t\treturn;\n+\t}\n+\n+\tif (!const_str || !*const_str)\n+\t\treturn;\n+\n+\tfsm_settings__set_hook(r, const_str);\n+}\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->mode;\n+}\n+\n+const char *fsm_settings__get_hook_path(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->hook_path;\n+}\n+\n+void fsm_settings__set_ipc(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\n+\n+void fsm_settings__set_hook(struct repository *r, const char *path)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+\tr->settings.fsmonitor->hook_path = strdup(path);\n+}\n+\n+void fsm_settings__set_disabled(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nnew file mode 100644\nindex 00000000000..a4c5d7b4889\n--- /dev/null\n+++ b/fsmonitor-settings.h\n@@ -0,0 +1,21 @@\n+#ifndef FSMONITOR_SETTINGS_H\n+#define FSMONITOR_SETTINGS_H\n+\n+struct repository;\n+\n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n+\tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n+};\n+\n+void fsm_settings__set_ipc(struct repository *r);\n+void fsm_settings__set_hook(struct repository *r, const char *path);\n+void fsm_settings__set_disabled(struct repository *r);\n+\n+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+const char *fsm_settings__get_hook_path(struct repository *r);\n+\n+struct fsmonitor_settings;\n+\n+#endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 448d0ee33f5..0e961b74d82 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tstrvec_push(&cp.args, fsm_settings__get_hook_path(r));\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -225,17 +229,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \tchar *buf;\n \tunsigned int i;\n \tint is_trivial = 0;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (fsm_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -243,13 +258,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -280,7 +296,8 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t\tif (query_success)\n \t\t\t\tis_trivial = query_result.buf[0] == '/';\n@@ -290,9 +307,12 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n \t\t\t\t\t \"query/trivial-response\", 1);\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tfsm_settings__get_hook_path(r));\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t fsm_settings__get_hook_path(r),\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/*\n@@ -429,7 +449,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tint fsmonitor_enabled = (fsm_settings__get_mode(istate->repo)\n+\t\t\t\t > FSMONITOR_MODE_DISABLED);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -449,16 +470,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..3f41f653691 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -3,6 +3,7 @@\n \n #include \"cache.h\"\n #include \"dir.h\"\n+#include \"fsmonitor-settings.h\"\n \n extern struct trace_key trace_fsmonitor;\n \n@@ -57,7 +58,10 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\treturn fsm_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +71,10 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +90,9 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);\n+\n+\tif (fsm_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repository.h b/repository.h\nindex ca837cb9e91..9bbb4659cc8 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n #include \"path.h\"\n \n struct config_set;\n+struct fsmonitor_settings;\n struct git_hash_algo;\n struct index_state;\n struct lock_file;\n@@ -35,6 +36,8 @@ struct repo_settings {\n \tint command_requires_full_index;\n \tint sparse_index;\n \n+\tstruct fsmonitor_settings *fsmonitor; /* lazily loaded */\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex f48e0542cdc..9ffea1d3147 100644\n--- a/t/README\n+++ b/t/README\n@@ -405,8 +405,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code paths for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\n-- \ngitgitgadget\n\n"},{"id":"452307","messageId":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v8.git.1648140586.gitgitgadget@gmail.com","subject":"[PATCH v9 00/30] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:43Z","receivedAt":"2022-03-25T19:26:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V9 of Part 2 of my builtin FSMonitor series. This version addresses\nbash style issues in t7527 raised on V8. These changes do not require a new\nversion of Part 3.\n\nHere is a range-diff from V8 to V9 relative to 715d08a9e5 (The eighth batch,\n2022-02-25).\n\n 1: e98373f997 = 1: e98373f997 fsmonitor: enhance existing comments, clarify trivial response handling\n 2: ab68b94417 = 2: ab68b94417 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n 3: e04c7301f2 = 3: e04c7301f2 fsmonitor: config settings are repository-specific\n 4: ea02ba25d8 = 4: ea02ba25d8 fsmonitor: use IPC to query the builtin FSMonitor daemon\n 5: 6ab7db9cb7 = 5: 6ab7db9cb7 fsmonitor: document builtin fsmonitor\n 6: 0ce8ae3f2c = 6: 0ce8ae3f2c fsmonitor--daemon: add a built-in fsmonitor daemon\n 7: 4624ce2fa4 = 7: 4624ce2fa4 fsmonitor--daemon: implement 'stop' and 'status' commands\n 8: a29fe7266a = 8: a29fe7266a compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n 9: 2f8a42fdb9 = 9: 2f8a42fdb9 compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n10: f07800690e = 10: f07800690e fsmonitor--daemon: implement 'run' command\n11: a6a39a3306 = 11: a6a39a3306 fsmonitor--daemon: implement 'start' command\n12: d62e338d00 = 12: d62e338d00 fsmonitor--daemon: add pathname classification\n13: 53e06b4ae5 = 13: 53e06b4ae5 fsmonitor--daemon: define token-ids\n14: 39f43fabe0 = 14: 39f43fabe0 fsmonitor--daemon: create token-based changed path cache\n15: 239558e34f = 15: 239558e34f compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows\n16: 14b775e9d8 = 16: 14b775e9d8 compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n17: 55bd7aee06 = 17: 55bd7aee06 compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS\n18: c43009124f = 18: c43009124f fsmonitor--daemon: implement handle_client callback\n19: ed338777b5 = 19: ed338777b5 help: include fsmonitor--daemon feature flag in version info\n20: c99bac29d4 = 20: c99bac29d4 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n21: c8709da945 ! 21: bc94e379b0 t7527: create test for fsmonitor--daemon\n @@ Commit message\n t7527: create test for fsmonitor--daemon\n \n Signed-off-by: Jeff Hostetler \n - Signed-off-by: Junio C Hamano \n \n ## t/t7527-builtin-fsmonitor.sh (new) ##\n @@\n @@ t/t7527-builtin-fsmonitor.sh (new)\n + rm -rf $1\n +}\n +\n -+is_value () {\n -+ test -n \"$1\" && test \"${1::1}\" != \"-\"\n -+}\n -+\n +start_daemon () {\n -+ r= &&\n -+ tf= &&\n -+ t2= &&\n -+ tk= &&\n ++ r= tf= t2= tk= &&\n +\n + while test \"$#\" -ne 0\n + do\n + case \"$1\" in\n + -C)\n -+ shift;\n -+ is_value $1 || BUG \"error: -C requires value\"\n -+ r=\"-C $1\"\n ++ r=\"-C ${2?}\"\n + shift\n + ;;\n + --tf)\n -+ shift;\n -+ is_value $1 || BUG \"error: --tf requires value\"\n -+ tf=\"$1\"\n ++ tf=\"${2?}\"\n + shift\n + ;;\n + --t2)\n -+ shift;\n -+ is_value $1 || BUG \"error: --t2 requires value\"\n -+ t2=\"$1\"\n ++ t2=\"${2?}\"\n + shift\n + ;;\n + --tk)\n -+ shift;\n -+ is_value $1 || BUG \"error: --tk requires value\"\n -+ tk=\"$1\"\n ++ tk=\"${2?}\"\n + shift\n + ;;\n -+ *)\n ++ -*)\n + BUG \"error: unknown option: '$1'\"\n + ;;\n ++ *)\n ++ BUG \"error: unbound argument: '$1'\"\n ++ ;;\n + esac\n ++ shift\n + done &&\n +\n + (\n22: cc39ecf10a = 22: 06d56d3a73 t/perf: avoid copying builtin fsmonitor files into test repo\n23: 2bb3eb8476 = 23: 2dd0215127 t/helper/test-chmtime: skip directories on Windows\n24: bab9a9b080 = 24: bb88cddc13 t/perf/p7519: fix coding style\n25: 2dd06ad2f7 = 25: 50c2afaa49 t/perf/p7519: speed up test on Windows\n26: 6eaa5765ae = 26: 5b18e3b692 t/perf/p7519: add fsmonitor--daemon test cases\n27: 30957f3930 = 27: 899c23f63c fsmonitor--daemon: periodically truncate list of modified files\n28: c8ca2a1727 = 28: 597a7192f9 fsmonitor--daemon: use a cookie file to sync with file system\n29: 4caf1d89b8 = 29: 68a05fd289 fsmonitor: force update index after large responses\n30: f87a1eba69 = 30: 5eb696daba t7527: test status with untracked-cache and fsmonitor--daemon\n\n\nJeff Hostetler (30):\n fsmonitor: enhance existing comments, clarify trivial response\n handling\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor: document builtin fsmonitor\n fsmonitor--daemon: add a built-in fsmonitor daemon\n fsmonitor--daemon: implement 'stop' and 'status' commands\n compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n fsmonitor--daemon: implement 'run' command\n fsmonitor--daemon: implement 'start' command\n fsmonitor--daemon: add pathname classification\n fsmonitor--daemon: define token-ids\n fsmonitor--daemon: create token-based changed path cache\n compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on\n Windows\n compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on\n MacOS\n fsmonitor--daemon: implement handle_client callback\n help: include fsmonitor--daemon feature flag in version info\n t/helper/fsmonitor-client: create IPC client to talk to FSMonitor\n Daemon\n t7527: create test for fsmonitor--daemon\n t/perf: avoid copying builtin fsmonitor files into test repo\n t/helper/test-chmtime: skip directories on Windows\n t/perf/p7519: fix coding style\n t/perf/p7519: speed up test on Windows\n t/perf/p7519: add fsmonitor--daemon test cases\n fsmonitor--daemon: periodically truncate list of modified files\n fsmonitor--daemon: use a cookie file to sync with file system\n fsmonitor: force update index after large responses\n t7527: test status with untracked-cache and fsmonitor--daemon\n\n .gitignore | 1 +\n Documentation/config/core.txt | 60 +-\n Documentation/git-fsmonitor--daemon.txt | 75 ++\n Documentation/git-update-index.txt | 8 +-\n Makefile | 17 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 1479 +++++++++++++++++++++++\n builtin/update-index.c | 7 +-\n cache.h | 1 -\n compat/fsmonitor/fsm-darwin-gcc.h | 92 ++\n compat/fsmonitor/fsm-listen-darwin.c | 427 +++++++\n compat/fsmonitor/fsm-listen-win32.c | 586 +++++++++\n compat/fsmonitor/fsm-listen.h | 49 +\n config.c | 14 -\n config.h | 1 -\n config.mak.uname | 20 +\n contrib/buildsystems/CMakeLists.txt | 10 +\n environment.c | 1 -\n fsmonitor--daemon.h | 166 +++\n fsmonitor-ipc.c | 171 +++\n fsmonitor-ipc.h | 48 +\n fsmonitor-settings.c | 114 ++\n fsmonitor-settings.h | 21 +\n fsmonitor.c | 216 +++-\n fsmonitor.h | 15 +-\n git.c | 1 +\n help.c | 4 +\n repo-settings.c | 1 +\n repository.h | 3 +\n t/README | 4 +-\n t/helper/test-chmtime.c | 15 +\n t/helper/test-fsmonitor-client.c | 116 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/perf/p7519-fsmonitor.sh | 68 +-\n t/perf/perf-lib.sh | 2 +-\n t/t7527-builtin-fsmonitor.sh | 609 ++++++++++\n t/test-lib.sh | 7 +\n 38 files changed, 4326 insertions(+), 106 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n create mode 100644 builtin/fsmonitor--daemon.c\n create mode 100644 compat/fsmonitor/fsm-darwin-gcc.h\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n create mode 100644 fsmonitor--daemon.h\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 fsmonitor-settings.c\n create mode 100644 fsmonitor-settings.h\n create mode 100644 t/helper/test-fsmonitor-client.c\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\n\nbase-commit: 715d08a9e51251ad8290b181b6ac3b9e1f9719d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v9\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v9\nPull-Request: https://github.com/gitgitgadget/git/pull/1041\n\nRange-diff vs v8:\n\n 1: e98373f997f = 1: e98373f997f fsmonitor: enhance existing comments, clarify trivial response handling\n 2: ab68b944173 = 2: ab68b944173 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n 3: e04c7301f24 = 3: e04c7301f24 fsmonitor: config settings are repository-specific\n 4: ea02ba25d8f = 4: ea02ba25d8f fsmonitor: use IPC to query the builtin FSMonitor daemon\n 5: 6ab7db9cb76 = 5: 6ab7db9cb76 fsmonitor: document builtin fsmonitor\n 6: 0ce8ae3f2cf = 6: 0ce8ae3f2cf fsmonitor--daemon: add a built-in fsmonitor daemon\n 7: 4624ce2fa47 = 7: 4624ce2fa47 fsmonitor--daemon: implement 'stop' and 'status' commands\n 8: a29fe7266a4 = 8: a29fe7266a4 compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n 9: 2f8a42fdb93 = 9: 2f8a42fdb93 compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n 10: f07800690ee = 10: f07800690ee fsmonitor--daemon: implement 'run' command\n 11: a6a39a3306d = 11: a6a39a3306d fsmonitor--daemon: implement 'start' command\n 12: d62e338d008 = 12: d62e338d008 fsmonitor--daemon: add pathname classification\n 13: 53e06b4ae5d = 13: 53e06b4ae5d fsmonitor--daemon: define token-ids\n 14: 39f43fabe02 = 14: 39f43fabe02 fsmonitor--daemon: create token-based changed path cache\n 15: 239558e34ff = 15: 239558e34ff compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows\n 16: 14b775e9d8b = 16: 14b775e9d8b compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n 17: 55bd7aee06c = 17: 55bd7aee06c compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS\n 18: c43009124fb = 18: c43009124fb fsmonitor--daemon: implement handle_client callback\n 19: ed338777b56 = 19: ed338777b56 help: include fsmonitor--daemon feature flag in version info\n 20: c99bac29d42 = 20: c99bac29d42 t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n 21: c8709da9457 ! 21: bc94e379b03 t7527: create test for fsmonitor--daemon\n @@ Commit message\n t7527: create test for fsmonitor--daemon\n \n Signed-off-by: Jeff Hostetler \n - Signed-off-by: Junio C Hamano \n \n ## t/t7527-builtin-fsmonitor.sh (new) ##\n @@\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\trm -rf $1\n +}\n +\n -+is_value () {\n -+\ttest -n \"$1\" && test \"${1::1}\" != \"-\"\n -+}\n -+\n +start_daemon () {\n -+\tr= &&\n -+\ttf= &&\n -+\tt2= &&\n -+\ttk= &&\n ++\tr= tf= t2= tk= &&\n +\n +\twhile test \"$#\" -ne 0\n +\tdo\n +\t\tcase \"$1\" in\n +\t\t-C)\n -+\t\t\tshift;\n -+\t\t\tis_value $1 || BUG \"error: -C requires value\"\n -+\t\t\tr=\"-C $1\"\n ++\t\t\tr=\"-C ${2?}\"\n +\t\t\tshift\n +\t\t\t;;\n +\t\t--tf)\n -+\t\t\tshift;\n -+\t\t\tis_value $1 || BUG \"error: --tf requires value\"\n -+\t\t\ttf=\"$1\"\n ++\t\t\ttf=\"${2?}\"\n +\t\t\tshift\n +\t\t\t;;\n +\t\t--t2)\n -+\t\t\tshift;\n -+\t\t\tis_value $1 || BUG \"error: --t2 requires value\"\n -+\t\t\tt2=\"$1\"\n ++\t\t\tt2=\"${2?}\"\n +\t\t\tshift\n +\t\t\t;;\n +\t\t--tk)\n -+\t\t\tshift;\n -+\t\t\tis_value $1 || BUG \"error: --tk requires value\"\n -+\t\t\ttk=\"$1\"\n ++\t\t\ttk=\"${2?}\"\n +\t\t\tshift\n +\t\t\t;;\n -+\t\t*)\n ++\t\t-*)\n +\t\t\tBUG \"error: unknown option: '$1'\"\n +\t\t\t;;\n ++\t\t*)\n ++\t\t\tBUG \"error: unbound argument: '$1'\"\n ++\t\t\t;;\n +\t\tesac\n ++\t\tshift\n +\tdone &&\n +\n +\t(\n 22: cc39ecf10ae = 22: 06d56d3a733 t/perf: avoid copying builtin fsmonitor files into test repo\n 23: 2bb3eb84767 = 23: 2dd02151278 t/helper/test-chmtime: skip directories on Windows\n 24: bab9a9b0802 = 24: bb88cddc137 t/perf/p7519: fix coding style\n 25: 2dd06ad2f71 = 25: 50c2afaa49e t/perf/p7519: speed up test on Windows\n 26: 6eaa5765ae1 = 26: 5b18e3b6926 t/perf/p7519: add fsmonitor--daemon test cases\n 27: 30957f3930e = 27: 899c23f63c3 fsmonitor--daemon: periodically truncate list of modified files\n 28: c8ca2a17277 = 28: 597a7192f94 fsmonitor--daemon: use a cookie file to sync with file system\n 29: 4caf1d89b84 = 29: 68a05fd2892 fsmonitor: force update index after large responses\n 30: f87a1eba693 = 30: 5eb696daba2 t7527: test status with untracked-cache and fsmonitor--daemon\n\n-- \ngitgitgadget\n"},{"id":"452308","messageId":"06d56d3a733f4fc5ae7fdfa90a667ae99bda1a0e.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 22/30] t/perf: avoid copying builtin fsmonitor files into test repo","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:05Z","receivedAt":"2022-03-25T19:26:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDo not copy any of the various fsmonitor--daemon files from the .git\ndirectory of the (GIT_PREF_REPO or GIT_PERF_LARGE_REPO) source repo\ninto the test's trash directory.\n\nWhen perf tests start, they copy the contents of the source repo into\nthe test's trash directory. If fsmonitor is running in the source repo,\nthere may be control files, such as the IPC socket and/or fsmonitor\ncookie files. These should not be copied into the test repo.\n\nUnix domain sockets cannot be copied in the manner used by the test\nsetup, so if present, the test setup fails.\n\nCookie files are harmless, but we should avoid them.\n\nThe builtin fsmonitor keeps all such control files/sockets in\n.git/fsmonitor--daemon*, so it is simple to exclude them.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/perf-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 407252bac70..932105cd12c 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -78,7 +78,7 @@ test_perf_copy_repo_contents () {\n \tfor stuff in \"$1\"/*\n \tdo\n \t\tcase \"$stuff\" in\n-\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees)\n+\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)\n \t\t\t;;\n \t\t*)\n \t\t\tcp -R \"$stuff\" \"$repo/.git/\" || exit 1\n-- \ngitgitgadget\n\n"},{"id":"452311","messageId":"c43009124fbb8f10d4919dbaaf38a728789dd395.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 18/30] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:01Z","receivedAt":"2022-03-25T19:26:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to respond to IPC requests from client\nGit processes and respond with a list of modified pathnames\nrelative to the provided token.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 311 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 309 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 69312119b07..eafaafb45b1 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,6 +7,7 @@\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n+#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon start []\"),\n@@ -364,6 +365,310 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+/*\n+ * Format an opaque token string to send to the client.\n+ */\n+static void with_lock__format_response_token(\n+\tstruct strbuf *response_token,\n+\tconst struct strbuf *response_token_id,\n+\tconst struct fsmonitor_batch *batch)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstrbuf_reset(response_token);\n+\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n+\t\t response_token_id->buf, batch->batch_seq_nr);\n+}\n+\n+/*\n+ * Parse an opaque token from the client.\n+ * Returns -1 on error.\n+ */\n+static int fsmonitor_parse_client_token(const char *buf_token,\n+\t\t\t\t\tstruct strbuf *requested_token_id,\n+\t\t\t\t\tuint64_t *seq_nr)\n+{\n+\tconst char *p;\n+\tchar *p_end;\n+\n+\tstrbuf_reset(requested_token_id);\n+\t*seq_nr = 0;\n+\n+\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n+\t\treturn -1;\n+\n+\twhile (*p && *p != ':')\n+\t\tstrbuf_addch(requested_token_id, *p++);\n+\tif (!*p++)\n+\t\treturn -1;\n+\n+\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n+\tif (*p_end)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal)\n+\n+static int do_handle_client(struct fsmonitor_daemon_state *state,\n+\t\t\t const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\tstruct fsmonitor_token_data *token_data = NULL;\n+\tstruct strbuf response_token = STRBUF_INIT;\n+\tstruct strbuf requested_token_id = STRBUF_INIT;\n+\tstruct strbuf payload = STRBUF_INIT;\n+\tuint64_t requested_oldest_seq_nr = 0;\n+\tuint64_t total_response_len = 0;\n+\tconst char *p;\n+\tconst struct fsmonitor_batch *batch_head;\n+\tconst struct fsmonitor_batch *batch;\n+\tintmax_t count = 0, duplicates = 0;\n+\tkh_str_t *shown;\n+\tint hash_ret;\n+\tint do_trivial = 0;\n+\tint do_flush = 0;\n+\n+\t/*\n+\t * We expect `command` to be of the form:\n+\t *\n+\t * := quit NUL\n+\t * | flush NUL\n+\t * | NUL\n+\t * | NUL\n+\t */\n+\n+\tif (!strcmp(command, \"quit\")) {\n+\t\t/*\n+\t\t * A client has requested over the socket/pipe that the\n+\t\t * daemon shutdown.\n+\t\t *\n+\t\t * Tell the IPC thread pool to shutdown (which completes\n+\t\t * the await in the main thread (which can stop the\n+\t\t * fsmonitor listener thread)).\n+\t\t *\n+\t\t * There is no reply to the client.\n+\t\t */\n+\t\treturn SIMPLE_IPC_QUIT;\n+\n+\t} else if (!strcmp(command, \"flush\")) {\n+\t\t/*\n+\t\t * Flush all of our cached data and generate a new token\n+\t\t * just like if we lost sync with the filesystem.\n+\t\t *\n+\t\t * Then send a trivial response using the new token.\n+\t\t */\n+\t\tdo_flush = 1;\n+\t\tdo_trivial = 1;\n+\n+\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n+\t\t/* assume V1 timestamp or garbage */\n+\n+\t\tchar *p_end;\n+\n+\t\tstrtoumax(command, &p_end, 10);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t ((*p_end) ?\n+\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n+\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n+\t\t\t\t command);\n+\t\tdo_trivial = 1;\n+\n+\t} else {\n+\t\t/* We have \"builtin:*\" */\n+\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n+\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n+\t\t\t\t\t command);\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We have a V2 valid token:\n+\t\t\t * \"builtin::\"\n+\t\t\t */\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (!state->current_token_data)\n+\t\tBUG(\"fsmonitor state does not have a current token\");\n+\n+\tif (do_flush)\n+\t\twith_lock__do_force_resync(state);\n+\n+\t/*\n+\t * We mark the current head of the batch list as \"pinned\" so\n+\t * that the listener thread will treat this item as read-only\n+\t * (and prevent any more paths from being added to it) from\n+\t * now on.\n+\t */\n+\ttoken_data = state->current_token_data;\n+\tbatch_head = token_data->batch_head;\n+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n+\n+\t/*\n+\t * FSMonitor Protocol V2 requires that we send a response header\n+\t * with a \"new current token\" and then all of the paths that changed\n+\t * since the \"requested token\". We send the seq_nr of the just-pinned\n+\t * head batch so that future requests from a client will be relative\n+\t * to it.\n+\t */\n+\twith_lock__format_response_token(&response_token,\n+\t\t\t\t\t &token_data->token_id, batch_head);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttotal_response_len += response_token.len + 1;\n+\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n+\t\t\t response_token.buf);\n+\n+\tif (!do_trivial) {\n+\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n+\t\t\t/*\n+\t\t\t * The client last spoke to a different daemon\n+\t\t\t * instance -OR- the daemon had to resync with\n+\t\t\t * the filesystem (and lost events), so reject.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n+\t\t\t\t\t \"response/token\", \"different\");\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else if (requested_oldest_seq_nr <\n+\t\t\t token_data->batch_tail->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * The client wants older events than we have for\n+\t\t\t * this token_id. This means that the end of our\n+\t\t\t * batch list was truncated and we cannot give the\n+\t\t\t * client a complete snapshot relative to their\n+\t\t\t * request.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"client requested truncated data\");\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n+\tif (do_trivial) {\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\treply(reply_data, \"/\", 2);\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n+\t\t\t\t \"response/trivial\", 1);\n+\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/*\n+\t * We're going to hold onto a pointer to the current\n+\t * token-data while we walk the list of batches of files.\n+\t * During this time, we will NOT be under the lock.\n+\t * So we ref-count it.\n+\t *\n+\t * This allows the listener thread to continue prepending\n+\t * new batches of items to the token-data (which we'll ignore).\n+\t *\n+\t * AND it allows the listener thread to do a token-reset\n+\t * (and install a new `current_token_data`).\n+\t */\n+\ttoken_data->client_ref_count++;\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * The client request is relative to the token that they sent,\n+\t * so walk the batch list backwards from the current head back\n+\t * to the batch (sequence number) they named.\n+\t *\n+\t * We use khash to de-dup the list of pathnames.\n+\t *\n+\t * NEEDSWORK: each batch contains a list of interned strings,\n+\t * so we only need to do pointer comparisons here to build the\n+\t * hash table. Currently, we're still comparing the string\n+\t * values.\n+\t */\n+\tshown = kh_init_str();\n+\tfor (batch = batch_head;\n+\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n+\t batch = batch->next) {\n+\t\tsize_t k;\n+\n+\t\tfor (k = 0; k < batch->nr; k++) {\n+\t\t\tconst char *s = batch->interned_paths[k];\n+\t\t\tsize_t s_len;\n+\n+\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n+\t\t\t\tduplicates++;\n+\t\t\telse {\n+\t\t\t\tkh_put_str(shown, s, &hash_ret);\n+\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n+\t\t\t\t\t\t count, s);\n+\n+\t\t\t\t/* Each path gets written with a trailing NUL */\n+\t\t\t\ts_len = strlen(s) + 1;\n+\n+\t\t\t\tif (payload.len + s_len >=\n+\t\t\t\t LARGE_PACKET_DATA_MAX) {\n+\t\t\t\t\treply(reply_data, payload.buf,\n+\t\t\t\t\t payload.len);\n+\t\t\t\t\ttotal_response_len += payload.len;\n+\t\t\t\t\tstrbuf_reset(&payload);\n+\t\t\t\t}\n+\n+\t\t\t\tstrbuf_add(&payload, s, s_len);\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (payload.len) {\n+\t\treply(reply_data, payload.buf, payload.len);\n+\t\ttotal_response_len += payload.len;\n+\t}\n+\n+\tkh_release_str(shown);\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (token_data->client_ref_count > 0)\n+\t\ttoken_data->client_ref_count--;\n+\n+\tif (token_data->client_ref_count == 0) {\n+\t\tif (token_data != state->current_token_data) {\n+\t\t\t/*\n+\t\t\t * The listener thread did a token-reset while we were\n+\t\t\t * walking the batch list. Therefore, this token is\n+\t\t\t * stale and can be discarded completely. If we are\n+\t\t\t * the last reader thread using this token, we own\n+\t\t\t * that work.\n+\t\t\t */\n+\t\t\tfsmonitor_free_token_data(token_data);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n+\n+cleanup:\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\tstrbuf_release(&payload);\n+\n+\treturn 0;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -371,7 +676,7 @@ static int handle_client(void *data,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n-\t/* struct fsmonitor_daemon_state *state = data; */\n+\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n \t/*\n@@ -382,10 +687,12 @@ static int handle_client(void *data,\n \tif (command_len != strlen(command))\n \t\tBUG(\"FSMonitor assumes text messages\");\n \n+\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n+\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n \n-\tresult = 0; /* TODO Do something here. */\n+\tresult = do_handle_client(state, command, reply, reply_data);\n \n \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n \n-- \ngitgitgadget\n\n"},{"id":"452312","messageId":"bc94e379b030da1295b524c8c47159368d7b63ba.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 21/30] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:04Z","receivedAt":"2022-03-25T19:27:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 494 +++++++++++++++++++++++++++++++++++\n 1 file changed, 494 insertions(+)\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..062e01c0dfc\n--- /dev/null\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -0,0 +1,494 @@\n+#!/bin/sh\n+\n+test_description='built-in file system watcher'\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+stop_daemon_delete_repo () {\n+\tr=$1 &&\n+\ttest_might_fail git -C $r fsmonitor--daemon stop &&\n+\trm -rf $1\n+}\n+\n+start_daemon () {\n+\tr= tf= t2= tk= &&\n+\n+\twhile test \"$#\" -ne 0\n+\tdo\n+\t\tcase \"$1\" in\n+\t\t-C)\n+\t\t\tr=\"-C ${2?}\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--tf)\n+\t\t\ttf=\"${2?}\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--t2)\n+\t\t\tt2=\"${2?}\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t--tk)\n+\t\t\ttk=\"${2?}\"\n+\t\t\tshift\n+\t\t\t;;\n+\t\t-*)\n+\t\t\tBUG \"error: unknown option: '$1'\"\n+\t\t\t;;\n+\t\t*)\n+\t\t\tBUG \"error: unbound argument: '$1'\"\n+\t\t\t;;\n+\t\tesac\n+\t\tshift\n+\tdone &&\n+\n+\t(\n+\t\tif test -n \"$tf\"\n+\t\tthen\n+\t\t\tGIT_TRACE_FSMONITOR=\"$tf\"\n+\t\t\texport GIT_TRACE_FSMONITOR\n+\t\tfi &&\n+\n+\t\tif test -n \"$t2\"\n+\t\tthen\n+\t\t\tGIT_TRACE2_PERF=\"$t2\"\n+\t\t\texport GIT_TRACE2_PERF\n+\t\tfi &&\n+\n+\t\tif test -n \"$tk\"\n+\t\tthen\n+\t\t\tGIT_TEST_FSMONITOR_TOKEN=\"$tk\"\n+\t\t\texport GIT_TEST_FSMONITOR_TOKEN\n+\t\tfi &&\n+\n+\t\tgit $r fsmonitor--daemon start &&\n+\t\tgit $r fsmonitor--daemon status\n+\t)\n+}\n+\n+# Is a Trace2 data event present with the given catetory and key?\n+# We do not care what the value is.\n+#\n+have_t2_data_event () {\n+\tc=$1 &&\n+\tk=$2 &&\n+\n+\tgrep -e '\"event\":\"data\".*\"category\":\"'\"$c\"'\".*\"key\":\"'\"$k\"'\"'\n+}\n+\n+test_expect_success 'explicit daemon start and stop' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n+\n+\tgit init test_explicit &&\n+\tstart_daemon -C test_explicit &&\n+\n+\tgit -C test_explicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon start' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit\" &&\n+\n+\tgit init test_implicit &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n+\n+\t# query will implicitly start the daemon.\n+\t#\n+\t# for test-script simplicity, we send a V1 timestamp rather than\n+\t# a V2 token. either way, the daemon response to any query contains\n+\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n+\t# but this test case is only concerned with whether the daemon was\n+\t# implicitly started.)\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n+\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n+\tnul_to_q actual.filtered &&\n+\tgrep \"builtin:\" actual.filtered &&\n+\n+\t# confirm that a daemon was started in the background.\n+\t#\n+\t# since the mechanism for starting the background daemon is platform\n+\t# dependent, just confirm that the foreground command received a\n+\t# response from the daemon.\n+\n+\thave_t2_data_event fsm_client query/response-length <.git/trace &&\n+\n+\tgit -C test_implicit fsmonitor--daemon status &&\n+\tgit -C test_implicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (delete .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n+\n+\tgit init test_implicit_1 &&\n+\n+\tstart_daemon -C test_implicit_1 &&\n+\n+\t# deleting the .git directory will implicitly stop the daemon.\n+\trm -rf test_implicit_1/.git &&\n+\n+\t# [1] Create an empty .git directory so that the following Git\n+\t# command will stay relative to the `-C` directory.\n+\t#\n+\t# Without this, the Git command will override the requested\n+\t# -C argument and crawl out to the containing Git source tree.\n+\t# This would make the test result dependent upon whether we\n+\t# were using fsmonitor on our development worktree.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_1/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (rename .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_2\" &&\n+\n+\tgit init test_implicit_2 &&\n+\n+\tstart_daemon -C test_implicit_2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n+\n+\t# See [1] above.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'cannot start multiple daemons' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n+\n+\tgit init test_multiple &&\n+\n+\tstart_daemon -C test_multiple &&\n+\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n+\tgrep \"fsmonitor--daemon is already running\" actual &&\n+\n+\tgit -C test_multiple fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n+'\n+\n+# These tests use the main repo in the trash directory\n+\n+test_expect_success 'setup' '\n+\t>tracked &&\n+\t>modified &&\n+\t>delete &&\n+\t>rename &&\n+\tmkdir dir1 &&\n+\t>dir1/tracked &&\n+\t>dir1/modified &&\n+\t>dir1/delete &&\n+\t>dir1/rename &&\n+\tmkdir dir2 &&\n+\t>dir2/tracked &&\n+\t>dir2/modified &&\n+\t>dir2/delete &&\n+\t>dir2/rename &&\n+\tmkdir dirtorename &&\n+\t>dirtorename/a &&\n+\t>dirtorename/b &&\n+\n+\tcat >.gitignore <<-\\EOF &&\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tEOF\n+\n+\tgit -c core.fsmonitor=false add . &&\n+\ttest_tick &&\n+\tgit -c core.fsmonitor=false commit -m initial &&\n+\n+\tgit config core.fsmonitor true\n+'\n+\n+# The test already explicitly stopped (or tried to stop) the daemon.\n+# This is here in case something else fails first.\n+#\n+redundant_stop_daemon () {\n+\ttest_might_fail git fsmonitor--daemon stop\n+}\n+\n+test_expect_success 'update-index implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n+\t\tgit update-index --fsmonitor &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_1\n+'\n+\n+test_expect_success 'status implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n+\t\tgit status >actual &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\t# Confirm that the trace2 log contains a record of the\n+\t# daemon starting.\n+\ttest_subcommand git fsmonitor--daemon start <.git/trace_implicit_2\n+'\n+\n+edit_files () {\n+\techo 1 >modified &&\n+\techo 2 >dir1/modified &&\n+\techo 3 >dir2/modified &&\n+\t>dir1/untracked\n+}\n+\n+delete_files () {\n+\trm -f delete &&\n+\trm -f dir1/delete &&\n+\trm -f dir2/delete\n+}\n+\n+create_files () {\n+\techo 1 >new &&\n+\techo 2 >dir1/new &&\n+\techo 3 >dir2/new\n+}\n+\n+rename_files () {\n+\tmv rename renamed &&\n+\tmv dir1/rename dir1/renamed &&\n+\tmv dir2/rename dir2/renamed\n+}\n+\n+file_to_directory () {\n+\trm -f delete &&\n+\tmkdir delete &&\n+\techo 1 >delete/new\n+}\n+\n+directory_to_file () {\n+\trm -rf dir1 &&\n+\techo 1 >dir1\n+}\n+\n+# The next few test cases confirm that our fsmonitor daemon sees each type\n+# of OS filesystem notification that we care about. At this layer we just\n+# ensure we are getting the OS notifications and do not try to confirm what\n+# is reported by `git status`.\n+#\n+# We run a simple query after modifying the filesystem just to introduce\n+# a bit of a delay so that the trace logging from the daemon has time to\n+# get flushed to disk.\n+#\n+# We `reset` and `clean` at the bottom of each test (and before stopping the\n+# daemon) because these commands might implicitly restart the daemon.\n+\n+clean_up_repo_and_stop_daemon () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\trm -f .git/trace\n+}\n+\n+test_expect_success 'edit some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tedit_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/modified$\" .git/trace &&\n+\tgrep \"^event: dir2/modified$\" .git/trace &&\n+\tgrep \"^event: modified$\" .git/trace &&\n+\tgrep \"^event: dir1/untracked$\" .git/trace\n+'\n+\n+test_expect_success 'create some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tcreate_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/new$\" .git/trace &&\n+\tgrep \"^event: dir2/new$\" .git/trace &&\n+\tgrep \"^event: new$\" .git/trace\n+'\n+\n+test_expect_success 'delete some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tdelete_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/delete$\" .git/trace &&\n+\tgrep \"^event: dir2/delete$\" .git/trace &&\n+\tgrep \"^event: delete$\" .git/trace\n+'\n+\n+test_expect_success 'rename some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\trename_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1/rename$\" .git/trace &&\n+\tgrep \"^event: dir2/rename$\" .git/trace &&\n+\tgrep \"^event: rename$\" .git/trace &&\n+\tgrep \"^event: dir1/renamed$\" .git/trace &&\n+\tgrep \"^event: dir2/renamed$\" .git/trace &&\n+\tgrep \"^event: renamed$\" .git/trace\n+'\n+\n+test_expect_success 'rename directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tmv dirtorename dirrenamed &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dirtorename/*$\" .git/trace &&\n+\tgrep \"^event: dirrenamed/*$\" .git/trace\n+'\n+\n+test_expect_success 'file changes to directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tfile_to_directory &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: delete$\" .git/trace &&\n+\tgrep \"^event: delete/new$\" .git/trace\n+'\n+\n+test_expect_success 'directory changes to a file' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\tstart_daemon --tf \"$PWD/.git/trace\" &&\n+\n+\tdirectory_to_file &&\n+\n+\ttest-tool fsmonitor-client query --token 0 &&\n+\n+\tgrep \"^event: dir1$\" .git/trace\n+'\n+\n+# The next few test cases exercise the token-resync code. When filesystem\n+# drops events (because of filesystem velocity or because the daemon isn't\n+# polling fast enough), we need to discard the cached data (relative to the\n+# current token) and start collecting events under a new token.\n+#\n+# the 'test-tool fsmonitor-client flush' command can be used to send a\n+# \"flush\" message to a running daemon and ask it to do a flush/resync.\n+\n+test_expect_success 'flush cached data' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_flush\" &&\n+\n+\tgit init test_flush &&\n+\n+\tstart_daemon -C test_flush --tf \"$PWD/.git/trace_daemon\" --tk true &&\n+\n+\t# The daemon should have an initial token with no events in _0 and\n+\t# then a few (probably platform-specific number of) events in _1.\n+\t# These should both have the same .\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n+\tnul_to_q actual_q0 &&\n+\n+\t>test_flush/file_1 &&\n+\t>test_flush/file_2 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n+\tnul_to_q actual_q1 &&\n+\n+\tgrep \"file_1\" actual_q1 &&\n+\n+\t# Force a flush. This will change the , reset the , and\n+\t# flush the file data. Then create some events and ensure that the file\n+\t# again appears in the cache. It should have the new .\n+\n+\ttest-tool -C test_flush fsmonitor-client flush >flush_0 &&\n+\tnul_to_q flush_q0 &&\n+\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_2 &&\n+\tnul_to_q actual_q2 &&\n+\n+\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n+\n+\t>test_flush/file_3 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n+\tnul_to_q actual_q3 &&\n+\n+\tgrep \"file_3\" actual_q3\n+'\n+\n+# The next few test cases create repos where the .git directory is NOT\n+# inside the one of the working directory. That is, where .git is a file\n+# that points to a directory elsewhere. This happens for submodules and\n+# non-primary worktrees.\n+\n+test_expect_success 'setup worktree base' '\n+\tgit init wt-base &&\n+\techo 1 >wt-base/file1 &&\n+\tgit -C wt-base add file1 &&\n+\tgit -C wt-base commit -m \"c1\"\n+'\n+\n+test_expect_success 'worktree with .git file' '\n+\tgit -C wt-base worktree add ../wt-secondary &&\n+\n+\tstart_daemon -C wt-secondary \\\n+\t\t--tf \"$PWD/trace_wt_secondary\" \\\n+\t\t--t2 \"$PWD/trace2_wt_secondary\" &&\n+\n+\tgit -C wt-secondary fsmonitor--daemon stop &&\n+\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n+'\n+\n+# NEEDSWORK: Repeat one of the \"edit\" tests on wt-secondary and\n+# confirm that we get the same events and behavior -- that is, that\n+# fsmonitor--daemon correctly watches BOTH the working directory and\n+# the external GITDIR directory and behaves the same as when \".git\"\n+# is a directory inside the working directory.\n+\n+test_expect_success 'cleanup worktrees' '\n+\tstop_daemon_delete_repo wt-secondary &&\n+\tstop_daemon_delete_repo wt-base\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"452313","messageId":"001701d8407a$d2a29dd0$77e7d970$@nexbridge.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"RE: [PATCH v9 00/30] Builtin FSMonitor Part 2","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2022-03-25T19:02:01Z","receivedAt":"2022-03-25T19:27:04Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On March 25, 2022 2:03 PM, Jeff Hostetler wrote:\n>Here is V9 of Part 2 of my builtin FSMonitor series. This version addresses bash\n>style issues in t7527 raised on V8. These changes do not require a new version of\n>Part 3.\n>\n>Here is a range-diff from V8 to V9 relative to 715d08a9e5 (The eighth batch, 2022-\n>02-25).\n>\n> 1: e98373f997 = 1: e98373f997 fsmonitor: enhance existing comments, clarify\n>trivial response handling\n> 2: ab68b94417 = 2: ab68b94417 fsmonitor-ipc: create client routines for git-\n>fsmonitor--daemon\n> 3: e04c7301f2 = 3: e04c7301f2 fsmonitor: config settings are repository-specific\n> 4: ea02ba25d8 = 4: ea02ba25d8 fsmonitor: use IPC to query the builtin FSMonitor\n>daemon\n> 5: 6ab7db9cb7 = 5: 6ab7db9cb7 fsmonitor: document builtin fsmonitor\n> 6: 0ce8ae3f2c = 6: 0ce8ae3f2c fsmonitor--daemon: add a built-in fsmonitor\n>daemon\n> 7: 4624ce2fa4 = 7: 4624ce2fa4 fsmonitor--daemon: implement 'stop' and 'status'\n>commands\n> 8: a29fe7266a = 8: a29fe7266a compat/fsmonitor/fsm-listen-win32: stub in\n>backend for Windows\n> 9: 2f8a42fdb9 = 9: 2f8a42fdb9 compat/fsmonitor/fsm-listen-darwin: stub in\n>backend for Darwin\n>10: f07800690e = 10: f07800690e fsmonitor--daemon: implement 'run' command\n>11: a6a39a3306 = 11: a6a39a3306 fsmonitor--daemon: implement 'start' command\n>12: d62e338d00 = 12: d62e338d00 fsmonitor--daemon: add pathname\n>classification\n>13: 53e06b4ae5 = 13: 53e06b4ae5 fsmonitor--daemon: define token-ids\n>14: 39f43fabe0 = 14: 39f43fabe0 fsmonitor--daemon: create token-based\n>changed path cache\n>15: 239558e34f = 15: 239558e34f compat/fsmonitor/fsm-listen-win32: implement\n>FSMonitor backend on Windows\n>16: 14b775e9d8 = 16: 14b775e9d8 compat/fsmonitor/fsm-listen-darwin: add\n>MacOS header files for FSEvent\n>17: 55bd7aee06 = 17: 55bd7aee06 compat/fsmonitor/fsm-listen-darwin:\n>implement FSEvent listener on MacOS\n>18: c43009124f = 18: c43009124f fsmonitor--daemon: implement handle_client\n>callback\n>19: ed338777b5 = 19: ed338777b5 help: include fsmonitor--daemon feature flag in\n>version info\n>20: c99bac29d4 = 20: c99bac29d4 t/helper/fsmonitor-client: create IPC client to\n>talk to FSMonitor Daemon\n>21: c8709da945 ! 21: bc94e379b0 t7527: create test for fsmonitor--daemon\n> @@ Commit message\n> t7527: create test for fsmonitor--daemon\n>\n> Signed-off-by: Jeff Hostetler \n> - Signed-off-by: Junio C Hamano \n>\n> ## t/t7527-builtin-fsmonitor.sh (new) ##\n> @@\n> @@ t/t7527-builtin-fsmonitor.sh (new)\n> + rm -rf $1\n> +}\n> +\n> -+is_value () {\n> -+ test -n \"$1\" && test \"${1::1}\" != \"-\"\n> -+}\n> -+\n> +start_daemon () {\n> -+ r= &&\n> -+ tf= &&\n> -+ t2= &&\n> -+ tk= &&\n> ++ r= tf= t2= tk= &&\n> +\n> + while test \"$#\" -ne 0\n> + do\n> + case \"$1\" in\n> + -C)\n> -+ shift;\n> -+ is_value $1 || BUG \"error: -C requires value\"\n> -+ r=\"-C $1\"\n> ++ r=\"-C ${2?}\"\n> + shift\n> + ;;\n> + --tf)\n> -+ shift;\n> -+ is_value $1 || BUG \"error: --tf requires value\"\n> -+ tf=\"$1\"\n> ++ tf=\"${2?}\"\n> + shift\n> + ;;\n> + --t2)\n> -+ shift;\n> -+ is_value $1 || BUG \"error: --t2 requires value\"\n> -+ t2=\"$1\"\n> ++ t2=\"${2?}\"\n> + shift\n> + ;;\n> + --tk)\n> -+ shift;\n> -+ is_value $1 || BUG \"error: --tk requires value\"\n> -+ tk=\"$1\"\n> ++ tk=\"${2?}\"\n> + shift\n> + ;;\n> -+ *)\n> ++ -*)\n> + BUG \"error: unknown option: '$1'\"\n> + ;;\n> ++ *)\n> ++ BUG \"error: unbound argument: '$1'\"\n> ++ ;;\n> + esac\n> ++ shift\n> + done &&\n> +\n> + (\n>22: cc39ecf10a = 22: 06d56d3a73 t/perf: avoid copying builtin fsmonitor files into\n>test repo\n>23: 2bb3eb8476 = 23: 2dd0215127 t/helper/test-chmtime: skip directories on\n>Windows\n>24: bab9a9b080 = 24: bb88cddc13 t/perf/p7519: fix coding style\n>25: 2dd06ad2f7 = 25: 50c2afaa49 t/perf/p7519: speed up test on Windows\n>26: 6eaa5765ae = 26: 5b18e3b692 t/perf/p7519: add fsmonitor--daemon test\n>cases\n>27: 30957f3930 = 27: 899c23f63c fsmonitor--daemon: periodically truncate list of\n>modified files\n>28: c8ca2a1727 = 28: 597a7192f9 fsmonitor--daemon: use a cookie file to sync with\n>file system\n>29: 4caf1d89b8 = 29: 68a05fd289 fsmonitor: force update index after large\n>responses\n>30: f87a1eba69 = 30: 5eb696daba t7527: test status with untracked-cache and\n>fsmonitor--daemon\n>\n>\n>Jeff Hostetler (30):\n> fsmonitor: enhance existing comments, clarify trivial response\n> handling\n> fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n> fsmonitor: config settings are repository-specific\n> fsmonitor: use IPC to query the builtin FSMonitor daemon\n> fsmonitor: document builtin fsmonitor\n> fsmonitor--daemon: add a built-in fsmonitor daemon\n> fsmonitor--daemon: implement 'stop' and 'status' commands\n> compat/fsmonitor/fsm-listen-win32: stub in backend for Windows\n> compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin\n> fsmonitor--daemon: implement 'run' command\n> fsmonitor--daemon: implement 'start' command\n> fsmonitor--daemon: add pathname classification\n> fsmonitor--daemon: define token-ids\n> fsmonitor--daemon: create token-based changed path cache\n> compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on\n> Windows\n> compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent\n> compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on\n> MacOS\n> fsmonitor--daemon: implement handle_client callback\n> help: include fsmonitor--daemon feature flag in version info\n> t/helper/fsmonitor-client: create IPC client to talk to FSMonitor\n> Daemon\n> t7527: create test for fsmonitor--daemon\n> t/perf: avoid copying builtin fsmonitor files into test repo\n> t/helper/test-chmtime: skip directories on Windows\n> t/perf/p7519: fix coding style\n> t/perf/p7519: speed up test on Windows\n> t/perf/p7519: add fsmonitor--daemon test cases\n> fsmonitor--daemon: periodically truncate list of modified files\n> fsmonitor--daemon: use a cookie file to sync with file system\n> fsmonitor: force update index after large responses\n> t7527: test status with untracked-cache and fsmonitor--daemon\n>\n> .gitignore | 1 +\n> Documentation/config/core.txt | 60 +-\n> Documentation/git-fsmonitor--daemon.txt | 75 ++\n> Documentation/git-update-index.txt | 8 +-\n> Makefile | 17 +\n> builtin.h | 1 +\n> builtin/fsmonitor--daemon.c | 1479 +++++++++++++++++++++++\n> builtin/update-index.c | 7 +-\n> cache.h | 1 -\n> compat/fsmonitor/fsm-darwin-gcc.h | 92 ++\n> compat/fsmonitor/fsm-listen-darwin.c | 427 +++++++\n> compat/fsmonitor/fsm-listen-win32.c | 586 +++++++++\n> compat/fsmonitor/fsm-listen.h | 49 +\n> config.c | 14 -\n> config.h | 1 -\n> config.mak.uname | 20 +\n> contrib/buildsystems/CMakeLists.txt | 10 +\n> environment.c | 1 -\n> fsmonitor--daemon.h | 166 +++\n> fsmonitor-ipc.c | 171 +++\n> fsmonitor-ipc.h | 48 +\n> fsmonitor-settings.c | 114 ++\n> fsmonitor-settings.h | 21 +\n> fsmonitor.c | 216 +++-\n> fsmonitor.h | 15 +-\n> git.c | 1 +\n> help.c | 4 +\n> repo-settings.c | 1 +\n> repository.h | 3 +\n> t/README | 4 +-\n> t/helper/test-chmtime.c | 15 +\n> t/helper/test-fsmonitor-client.c | 116 ++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> t/perf/p7519-fsmonitor.sh | 68 +-\n> t/perf/perf-lib.sh | 2 +-\n> t/t7527-builtin-fsmonitor.sh | 609 ++++++++++\n> t/test-lib.sh | 7 +\n> 38 files changed, 4326 insertions(+), 106 deletions(-) create mode 100644\n>Documentation/git-fsmonitor--daemon.txt\n> create mode 100644 builtin/fsmonitor--daemon.c create mode 100644\n>compat/fsmonitor/fsm-darwin-gcc.h create mode 100644 compat/fsmonitor/fsm-\n>listen-darwin.c\n> create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n> create mode 100644 compat/fsmonitor/fsm-listen.h create mode 100644\n>fsmonitor--daemon.h create mode 100644 fsmonitor-ipc.c create mode 100644\n>fsmonitor-ipc.h create mode 100644 fsmonitor-settings.c create mode 100644\n>fsmonitor-settings.h create mode 100644 t/helper/test-fsmonitor-client.c create\n>mode 100755 t/t7527-builtin-fsmonitor.sh\n>\n>\n>base-commit: 715d08a9e51251ad8290b181b6ac3b9e1f9719d7\n>Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-\n>1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v9\n>Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-\n>1041/jeffhostetler/builtin-fsmonitor-part2-v9\n>Pull-Request: https://github.com/gitgitgadget/git/pull/1041\n>\n>Range-diff vs v8:\n>\n> 1: e98373f997f = 1: e98373f997f fsmonitor: enhance existing comments, clarify\n>trivial response handling\n> 2: ab68b944173 = 2: ab68b944173 fsmonitor-ipc: create client routines for git-\n>fsmonitor--daemon\n> 3: e04c7301f24 = 3: e04c7301f24 fsmonitor: config settings are repository-\n>specific\n> 4: ea02ba25d8f = 4: ea02ba25d8f fsmonitor: use IPC to query the builtin\n>FSMonitor daemon\n> 5: 6ab7db9cb76 = 5: 6ab7db9cb76 fsmonitor: document builtin fsmonitor\n> 6: 0ce8ae3f2cf = 6: 0ce8ae3f2cf fsmonitor--daemon: add a built-in fsmonitor\n>daemon\n> 7: 4624ce2fa47 = 7: 4624ce2fa47 fsmonitor--daemon: implement 'stop' and\n>'status' commands\n> 8: a29fe7266a4 = 8: a29fe7266a4 compat/fsmonitor/fsm-listen-win32: stub in\n>backend for Windows\n> 9: 2f8a42fdb93 = 9: 2f8a42fdb93 compat/fsmonitor/fsm-listen-darwin: stub in\n>backend for Darwin\n> 10: f07800690ee = 10: f07800690ee fsmonitor--daemon: implement 'run'\n>command\n> 11: a6a39a3306d = 11: a6a39a3306d fsmonitor--daemon: implement 'start'\n>command\n> 12: d62e338d008 = 12: d62e338d008 fsmonitor--daemon: add pathname\n>classification\n> 13: 53e06b4ae5d = 13: 53e06b4ae5d fsmonitor--daemon: define token-ids\n> 14: 39f43fabe02 = 14: 39f43fabe02 fsmonitor--daemon: create token-based\n>changed path cache\n> 15: 239558e34ff = 15: 239558e34ff compat/fsmonitor/fsm-listen-win32:\n>implement FSMonitor backend on Windows\n> 16: 14b775e9d8b = 16: 14b775e9d8b compat/fsmonitor/fsm-listen-darwin: add\n>MacOS header files for FSEvent\n> 17: 55bd7aee06c = 17: 55bd7aee06c compat/fsmonitor/fsm-listen-darwin:\n>implement FSEvent listener on MacOS\n> 18: c43009124fb = 18: c43009124fb fsmonitor--daemon: implement handle_client\n>callback\n> 19: ed338777b56 = 19: ed338777b56 help: include fsmonitor--daemon feature\n>flag in version info\n> 20: c99bac29d42 = 20: c99bac29d42 t/helper/fsmonitor-client: create IPC client to\n>talk to FSMonitor Daemon\n> 21: c8709da9457 ! 21: bc94e379b03 t7527: create test for fsmonitor--daemon\n> @@ Commit message\n> t7527: create test for fsmonitor--daemon\n>\n> Signed-off-by: Jeff Hostetler \n> - Signed-off-by: Junio C Hamano \n>\n> ## t/t7527-builtin-fsmonitor.sh (new) ##\n> @@\n> @@ t/t7527-builtin-fsmonitor.sh (new)\n> +\trm -rf $1\n> +}\n> +\n> -+is_value () {\n> -+\ttest -n \"$1\" && test \"${1::1}\" != \"-\"\n> -+}\n> -+\n> +start_daemon () {\n> -+\tr= &&\n> -+\ttf= &&\n> -+\tt2= &&\n> -+\ttk= &&\n> ++\tr= tf= t2= tk= &&\n> +\n> +\twhile test \"$#\" -ne 0\n> +\tdo\n> +\t\tcase \"$1\" in\n> +\t\t-C)\n> -+\t\t\tshift;\n> -+\t\t\tis_value $1 || BUG \"error: -C requires value\"\n> -+\t\t\tr=\"-C $1\"\n> ++\t\t\tr=\"-C ${2?}\"\n> +\t\t\tshift\n> +\t\t\t;;\n> +\t\t--tf)\n> -+\t\t\tshift;\n> -+\t\t\tis_value $1 || BUG \"error: --tf requires value\"\n> -+\t\t\ttf=\"$1\"\n> ++\t\t\ttf=\"${2?}\"\n> +\t\t\tshift\n> +\t\t\t;;\n> +\t\t--t2)\n> -+\t\t\tshift;\n> -+\t\t\tis_value $1 || BUG \"error: --t2 requires value\"\n> -+\t\t\tt2=\"$1\"\n> ++\t\t\tt2=\"${2?}\"\n> +\t\t\tshift\n> +\t\t\t;;\n> +\t\t--tk)\n> -+\t\t\tshift;\n> -+\t\t\tis_value $1 || BUG \"error: --tk requires value\"\n> -+\t\t\ttk=\"$1\"\n> ++\t\t\ttk=\"${2?}\"\n> +\t\t\tshift\n> +\t\t\t;;\n> -+\t\t*)\n> ++\t\t-*)\n> +\t\t\tBUG \"error: unknown option: '$1'\"\n> +\t\t\t;;\n> ++\t\t*)\n> ++\t\t\tBUG \"error: unbound argument: '$1'\"\n> ++\t\t\t;;\n> +\t\tesac\n> ++\t\tshift\n> +\tdone &&\n> +\n> +\t(\n> 22: cc39ecf10ae = 22: 06d56d3a733 t/perf: avoid copying builtin fsmonitor files\n>into test repo\n> 23: 2bb3eb84767 = 23: 2dd02151278 t/helper/test-chmtime: skip directories on\n>Windows\n> 24: bab9a9b0802 = 24: bb88cddc137 t/perf/p7519: fix coding style\n> 25: 2dd06ad2f71 = 25: 50c2afaa49e t/perf/p7519: speed up test on Windows\n> 26: 6eaa5765ae1 = 26: 5b18e3b6926 t/perf/p7519: add fsmonitor--daemon test\n>cases\n> 27: 30957f3930e = 27: 899c23f63c3 fsmonitor--daemon: periodically truncate list\n>of modified files\n> 28: c8ca2a17277 = 28: 597a7192f94 fsmonitor--daemon: use a cookie file to sync\n>with file system\n> 29: 4caf1d89b84 = 29: 68a05fd2892 fsmonitor: force update index after large\n>responses\n> 30: f87a1eba693 = 30: 5eb696daba2 t7527: test status with untracked-cache and\n>fsmonitor--daemon\n\nSo that we can plan for next time, would you mind giving some thought to what would be required for Linux and whether it makes sense to extend this. No urgency, but I would not mind participating - assuming my $DAYJOB lets me. I have a backlog for git that I need to get done first anyway.\n\nKind Regards,\nRandall\n\n"},{"id":"452319","messageId":"5eb696daba2fe108d4d9ba2ccf4b357447ef9946.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 30/30] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:13Z","receivedAt":"2022-03-25T19:28:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate 2x2 test matrix with the untracked-cache and fsmonitor--daemon\nfeatures and a series of edits and verify that status output is\nidentical.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/t7527-builtin-fsmonitor.sh | 115 +++++++++++++++++++++++++++++++++++\n 1 file changed, 115 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 062e01c0dfc..bd0c952a116 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -205,6 +205,8 @@ test_expect_success 'setup' '\n \t.gitignore\n \texpect*\n \tactual*\n+\tflush*\n+\ttrace*\n \tEOF\n \n \tgit -c core.fsmonitor=false add . &&\n@@ -491,4 +493,117 @@ test_expect_success 'cleanup worktrees' '\n \tstop_daemon_delete_repo wt-base\n '\n \n+# The next few tests perform arbitrary/contrived file operations and\n+# confirm that status is correct. That is, that the data (or lack of\n+# data) from fsmonitor doesn't cause incorrect results. And doesn't\n+# cause incorrect results when the untracked-cache is enabled.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\tgit update-index --test-untracked-cache\n+'\n+\n+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n+\ttest_unconfig core.fsmonitor &&\n+\tgit update-index --no-fsmonitor &&\n+\ttest_might_fail git fsmonitor--daemon stop\n+'\n+\n+matrix_clean_up_repo () {\n+\tgit reset --hard HEAD &&\n+\tgit clean -fd\n+}\n+\n+matrix_try () {\n+\tuc=$1 &&\n+\tfsm=$2 &&\n+\tfn=$3 &&\n+\n+\tif test $uc = true && test $fsm = false\n+\tthen\n+\t\t# The untracked-cache is buggy when FSMonitor is\n+\t\t# DISABLED, so skip the tests for this matrix\n+\t\t# combination.\n+\t\t#\n+\t\t# We've observed random, occasional test failures on\n+\t\t# Windows and MacOS when the UC is turned on and FSM\n+\t\t# is turned off. These are rare, but they do happen\n+\t\t# indicating that it is probably a race condition within\n+\t\t# the untracked cache itself.\n+\t\t#\n+\t\t# It usually happens when a test does F/D trickery and\n+\t\t# then the NEXT test fails because of extra status\n+\t\t# output from stale UC data from the previous test.\n+\t\t#\n+\t\t# Since FSMonitor is not involved in the error, skip\n+\t\t# the tests for this matrix combination.\n+\t\t#\n+\t\treturn 0\n+\tfi &&\n+\n+\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n+\t\tmatrix_clean_up_repo &&\n+\t\t$fn &&\n+\t\tif test $uc = false && test $fsm = false\n+\t\tthen\n+\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n+\t\telse\n+\t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n+\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n+\t\tfi\n+\t'\n+}\n+\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+for uc_val in $uc_values\n+do\n+\tif test $uc_val = false\n+\tthen\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n+\t\t\tgit config core.untrackedcache false &&\n+\t\t\tgit update-index --no-untracked-cache\n+\t\t'\n+\telse\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n+\t\t\tgit config core.untrackedcache true &&\n+\t\t\tgit update-index --untracked-cache\n+\t\t'\n+\tfi\n+\n+\tfsm_values=\"false true\"\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tif test $fsm_val = false\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\telse\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n+\t\t\t\tgit config core.fsmonitor true &&\n+\t\t\t\tgit fsmonitor--daemon start &&\n+\t\t\t\tgit update-index --fsmonitor\n+\t\t\t'\n+\t\tfi\n+\n+\t\tmatrix_try $uc_val $fsm_val edit_files\n+\t\tmatrix_try $uc_val $fsm_val delete_files\n+\t\tmatrix_try $uc_val $fsm_val create_files\n+\t\tmatrix_try $uc_val $fsm_val rename_files\n+\t\tmatrix_try $uc_val $fsm_val file_to_directory\n+\t\tmatrix_try $uc_val $fsm_val directory_to_file\n+\n+\t\tif test $fsm_val = true\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n+\t\t\t\ttest_unconfig core.fsmonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop\n+\t\t\t'\n+\t\tfi\n+\tdone\n+done\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"452320","messageId":"d62e338d00829bf3fe08e649ffa298c087a354a9.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 12/30] fsmonitor--daemon: add pathname classification","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:55Z","receivedAt":"2022-03-25T19:28:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to classify relative and absolute\npathnames and decide how they should be handled. This will\nbe used by the platform-specific backend to respond to each\nfilesystem event.\n\nWhen we register for filesystem notifications on a directory,\nwe get events for everything (recursively) in the directory.\nWe want to report to clients changes to tracked and untracked\npaths within the working directory proper. We do not want to\nreport changes within the .git directory, for example.\n\nThis classification will be used in a later commit by the\ndifferent backends to classify paths as events are received.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 81 ++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 87 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 168 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 69dd39121a3..1ce00b7c150 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -134,6 +134,87 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n+#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *rel)\n+{\n+\tif (fspathncmp(rel, \".git\", 4))\n+\t\treturn IS_WORKDIR_PATH;\n+\trel += 4;\n+\n+\tif (!*rel)\n+\t\treturn IS_DOT_GIT;\n+\tif (*rel != '/')\n+\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n+\trel++;\n+\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_DOT_GIT;\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *rel)\n+{\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_GITDIR;\n+}\n+\n+static enum fsmonitor_path_type try_classify_workdir_abs_path(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\n+\tif (fspathncmp(path, state->path_worktree_watch.buf,\n+\t\t state->path_worktree_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_worktree_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_workdir_relative(rel);\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\tenum fsmonitor_path_type t;\n+\n+\tt = try_classify_workdir_abs_path(state, path);\n+\tif (state->nr_paths_watching == 1)\n+\t\treturn t;\n+\tif (t != IS_OUTSIDE_CONE)\n+\t\treturn t;\n+\n+\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n+\t\t state->path_gitdir_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_gitdir_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_GITDIR; /* it is the exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_gitdir_relative(rel);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 3009c1a83de..8c3a71a48bd 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -30,5 +30,92 @@ struct fsmonitor_daemon_state {\n \tstruct ipc_server_data *ipc_server_data;\n };\n \n+/*\n+ * Pathname classifications.\n+ *\n+ * The daemon classifies the pathnames that it receives from file\n+ * system notification events into the following categories and uses\n+ * that to decide whether clients are told about them. (And to watch\n+ * for file system synchronization events.)\n+ *\n+ * The daemon only collects and reports on the set of modified paths\n+ * within the working directory (proper).\n+ *\n+ * The client should only care about paths within the working\n+ * directory proper (inside the working directory and not \".git\" nor\n+ * inside of \".git/\"). That is, the client has read the index and is\n+ * asking for a list of any paths in the working directory that have\n+ * been modified since the last token. The client does not care about\n+ * file system changes within the \".git/\" directory (such as new loose\n+ * objects or packfiles). So the client will only receive paths that\n+ * are classified as IS_WORKDIR_PATH.\n+ *\n+ * Note that \".git\" is usually a directory and is therefore inside\n+ * the cone of the FS watch that we have on the working directory root,\n+ * so we will also get FS events for disk activity on and within \".git/\"\n+ * that we need to respond to or filter from the client.\n+ *\n+ * But Git also allows \".git\" to be a *file* that points to a GITDIR\n+ * outside of the working directory. When this happens, we need to\n+ * create FS watches on both the working directory root *and* on the\n+ * (external) GITDIR root. (The latter is required because we put\n+ * cookie files inside it and use them to sync with the FS event\n+ * stream.)\n+ *\n+ * Note that in the context of this discussion, I'm using \"GITDIR\"\n+ * to only mean an external GITDIR referenced by a \".git\" file.\n+ *\n+ * The platform FS event backends will receive watch-specific\n+ * relative paths (except for those OS's that always emit absolute\n+ * paths). We use the following enum and routines to classify each\n+ * path so that we know how to handle it. There is a slight asymmetry\n+ * here because \".git/\" is inside the working directory and the\n+ * (external) GITDIR is not, and therefore how we handle events may\n+ * vary slightly, so I have different enums for \"IS...DOT_GIT...\" and\n+ * \"IS...GITDIR...\".\n+ *\n+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n+ * exact \".git\" file/directory or GITDIR directory. If the daemon\n+ * receives a delete event for either of these paths, it will\n+ * automatically shutdown, for example.\n+ *\n+ * Note that the daemon DOES NOT explicitly watch nor special case the\n+ * index. The daemon does not read the index nor have any internal\n+ * index-relative state, so there are no \"IS...INDEX...\" enum values.\n+ */\n+enum fsmonitor_path_type {\n+\tIS_WORKDIR_PATH = 0,\n+\n+\tIS_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n+\n+\tIS_GITDIR,\n+\tIS_INSIDE_GITDIR,\n+\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n+\n+\tIS_OUTSIDE_CONE,\n+};\n+\n+/*\n+ * Classify a pathname relative to the root of the working directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify a pathname relative to a that is external to the\n+ * worktree directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify an absolute pathname received from a filesystem event.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"452323","messageId":"55bd7aee06ca04e6ae9010700103a261d1255113.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 17/30] compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:00Z","receivedAt":"2022-03-25T19:29:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement file system event listener on MacOS using FSEvent,\nCoreFoundation, and CoreServices.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-darwin.c | 383 +++++++++++++++++++++++++++\n 1 file changed, 383 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex d2ce942cade..0741fe834c3 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -25,20 +25,403 @@\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tCFStringRef cfsr_worktree_path;\n+\tCFStringRef cfsr_gitdir_path;\n+\n+\tCFArrayRef cfar_paths_to_watch;\n+\tint nr_paths_watching;\n+\n+\tFSEventStreamRef stream;\n+\n+\tCFRunLoopRef rl;\n+\n+\tenum shutdown_style {\n+\t\tSHUTDOWN_EVENT = 0,\n+\t\tFORCE_SHUTDOWN,\n+\t\tFORCE_ERROR_STOP,\n+\t} shutdown_style;\n+\n+\tunsigned int stream_scheduled:1;\n+\tunsigned int stream_started:1;\n+};\n+\n+static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n+\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n+\tif (flag & kFSEventStreamEventFlagUserDropped)\n+\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n+\tif (flag & kFSEventStreamEventFlagKernelDropped)\n+\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n+\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n+\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n+\tif (flag & kFSEventStreamEventFlagHistoryDone)\n+\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n+\tif (flag & kFSEventStreamEventFlagRootChanged)\n+\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n+\tif (flag & kFSEventStreamEventFlagMount)\n+\t\tstrbuf_addstr(&msg, \"Mount|\");\n+\tif (flag & kFSEventStreamEventFlagUnmount)\n+\t\tstrbuf_addstr(&msg, \"Unmount|\");\n+\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n+\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n+\tif (flag & kFSEventStreamEventFlagItemCreated)\n+\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n+\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n+\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n+\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsDir)\n+\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsFile)\n+\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemModified)\n+\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n+\tif (flag & kFSEventStreamEventFlagItemRemoved)\n+\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n+\tif (flag & kFSEventStreamEventFlagItemRenamed)\n+\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n+\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n+\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n+\tif (flag & kFSEventStreamEventFlagOwnEvent)\n+\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n+\tif (flag & kFSEventStreamEventFlagItemCloned)\n+\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\t\t\t path, flag, msg.buf);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRemoved);\n+}\n+\n+static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRenamed);\n+}\n+\n+static int ef_is_dropped(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagMustScanSubDirs ||\n+\t\tef & kFSEventStreamEventFlagKernelDropped ||\n+\t\tef & kFSEventStreamEventFlagUserDropped);\n+}\n+\n+static void fsevent_callback(ConstFSEventStreamRef streamRef,\n+\t\t\t void *ctx,\n+\t\t\t size_t num_of_events,\n+\t\t\t void *event_paths,\n+\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t const FSEventStreamEventId event_ids[])\n+{\n+\tstruct fsmonitor_daemon_state *state = ctx;\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tchar **paths = (char **)event_paths;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *path_k;\n+\tconst char *slash;\n+\tint k;\n+\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t/*\n+\t * Build a list of all filesystem changes into a private/local\n+\t * list and without holding any locks.\n+\t */\n+\tfor (k = 0; k < num_of_events; k++) {\n+\t\t/*\n+\t\t * On Mac, we receive an array of absolute paths.\n+\t\t */\n+\t\tpath_k = paths[k];\n+\n+\t\t/*\n+\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n+\t\t * Please don't log them to Trace2.\n+\t\t *\n+\t\t * trace_printf_key(&trace_fsmonitor, \"Path: '%s'\", path_k);\n+\t\t */\n+\n+\t\t/*\n+\t\t * If event[k] is marked as dropped, we assume that we have\n+\t\t * lost sync with the filesystem and should flush our cached\n+\t\t * data. We need to:\n+\t\t *\n+\t\t * [1] Abort/wake any client threads waiting for a cookie and\n+\t\t * flush the cached state data (the current token), and\n+\t\t * create a new token.\n+\t\t *\n+\t\t * [2] Discard the batch that we were locally building (since\n+\t\t * they are conceptually relative to the just flushed\n+\t\t * token).\n+\t\t */\n+\t\tif (ef_is_dropped(event_flags[k])) {\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\tfsmonitor_force_resync(state);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t\tstring_list_clear(&cookie_list, 0);\n+\n+\t\t\t/*\n+\t\t\t * We assume that any events that we received\n+\t\t\t * in this callback after this dropped event\n+\t\t\t * may still be valid, so we continue rather\n+\t\t\t * than break. (And just in case there is a\n+\t\t\t * delete of \".git\" hiding in there.)\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n+\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git or gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path_k);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path_k);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\t/* ignore all other paths inside of .git or gitdir */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\tcase IS_GITDIR:\n+\t\t\t/*\n+\t\t\t * If .git directory is deleted or renamed away,\n+\t\t\t * we have to quit.\n+\t\t\t */\n+\t\t\tif (ef_is_root_delete(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir renamed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* try to queue normal pathnames */\n+\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\t/*\n+\t\t\t * Because of the implicit \"binning\" (the\n+\t\t\t * kernel calls us at a given frequency) and\n+\t\t\t * de-duping (the kernel is free to combine\n+\t\t\t * multiple events for a given pathname), an\n+\t\t\t * individual fsevent could be marked as both\n+\t\t\t * a file and directory. Add it to the queue\n+\t\t\t * with both spellings so that the client will\n+\t\t\t * know how much to invalidate/refresh.\n+\t\t\t */\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t}\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tstrbuf_reset(&tmp);\n+\t\t\t\tstrbuf_addstr(&tmp, rel);\n+\t\t\t\tstrbuf_addch(&tmp, '/');\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\n+\t\tcase IS_OUTSIDE_CONE:\n+\t\tdefault:\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignoring '%s'\", path_k);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\n+\tdata->shutdown_style = FORCE_SHUTDOWN;\n+\tCFRunLoopStop(data->rl);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+}\n+\n+/*\n+ * In the call to `FSEventStreamCreate()` to setup our watch, the\n+ * `latency` argument determines the frequency of calls to our callback\n+ * with new FS events. Too slow and events get dropped; too fast and\n+ * we burn CPU unnecessarily. Since it is rather obscure, I don't\n+ * think this needs to be a config setting. I've done extensive\n+ * testing on my systems and chosen the value below. It gives good\n+ * results and I've not seen any dropped events.\n+ *\n+ * With a latency of 0.1, I was seeing lots of dropped events during\n+ * the \"touch 100000\" files test within t/perf/p7519, but with a\n+ * latency of 0.001 I did not see any dropped events. So I'm going\n+ * to assume that this is the \"correct\" value.\n+ *\n+ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n+ */\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n+\t\tkFSEventStreamCreateFlagWatchRoot |\n+\t\tkFSEventStreamCreateFlagFileEvents;\n+\tFSEventStreamContext ctx = {\n+\t\t0,\n+\t\tstate,\n+\t\tNULL,\n+\t\tNULL,\n+\t\tNULL\n+\t};\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\tconst void *dir_array[2];\n+\n+\tCALLOC_ARRAY(data, 1);\n+\tstate->backend_data = data;\n+\n+\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n+\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n+\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n+\t\t\tNULL, state->path_gitdir_watch.buf,\n+\t\t\tkCFStringEncodingUTF8);\n+\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n+\t}\n+\n+\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n+\t\t\t\t\t\t data->nr_paths_watching,\n+\t\t\t\t\t\t NULL);\n+\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n+\t\t\t\t\t data->cfar_paths_to_watch,\n+\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n+\t\t\t\t\t 0.001, flags);\n+\tif (data->stream == NULL)\n+\t\tgoto failed;\n+\n+\t/*\n+\t * `data->rl` needs to be set inside the listener thread.\n+\t */\n+\n+\treturn 0;\n+\n+failed:\n+\terror(_(\"Unable to create FSEventStream.\"));\n+\n+\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tif (data->stream) {\n+\t\tif (data->stream_started)\n+\t\t\tFSEventStreamStop(data->stream);\n+\t\tif (data->stream_scheduled)\n+\t\t\tFSEventStreamInvalidate(data->stream);\n+\t\tFSEventStreamRelease(data->stream);\n+\t}\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\tdata->shutdown_style = SHUTDOWN_EVENT;\n+\n+\tCFRunLoopStop(data->rl);\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\n+\tdata->rl = CFRunLoopGetCurrent();\n+\n+\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n+\tdata->stream_scheduled = 1;\n+\n+\tif (!FSEventStreamStart(data->stream)) {\n+\t\terror(_(\"Failed to start the FSEventStream\"));\n+\t\tgoto force_error_stop_without_loop;\n+\t}\n+\tdata->stream_started = 1;\n+\n+\tCFRunLoopRun();\n+\n+\tswitch (data->shutdown_style) {\n+\tcase FORCE_ERROR_STOP:\n+\t\tstate->error_code = -1;\n+\t\t/* fall thru */\n+\tcase FORCE_SHUTDOWN:\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\t/* fall thru */\n+\tcase SHUTDOWN_EVENT:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn;\n+\n+force_error_stop_without_loop:\n+\tstate->error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+\treturn;\n }\n-- \ngitgitgadget\n\n"},{"id":"452326","messageId":"c99bac29d4233a2957bf0cfa7f2470798ae495ce.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 20/30] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:03Z","receivedAt":"2022-03-25T19:29:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate an IPC client to send query and flush commands to the daemon.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n t/helper/test-fsmonitor-client.c | 116 +++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 119 insertions(+)\n create mode 100644 t/helper/test-fsmonitor-client.c\n\ndiff --git a/Makefile b/Makefile\nindex 26567d4f772..daa21bed6c3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -716,6 +716,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\n TEST_BUILTINS_OBJS += test-getcwd.o\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nnew file mode 100644\nindex 00000000000..3062c8a3c2b\n--- /dev/null\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -0,0 +1,116 @@\n+/*\n+ * test-fsmonitor-client.c: client code to send commands/requests to\n+ * a `git fsmonitor--daemon` daemon.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tdie(\"fsmonitor--daemon not available on this platform\");\n+}\n+#else\n+\n+/*\n+ * Read the `.git/index` to get the last token written to the\n+ * FSMonitor Index Extension.\n+ */\n+static const char *get_token_from_index(void)\n+{\n+\tstruct index_state *istate = the_repository->index;\n+\n+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update)\n+\t\tdie(\"index file does not have fsmonitor extension\");\n+\n+\treturn istate->fsmonitor_last_update;\n+}\n+\n+/*\n+ * Send an IPC query to a `git-fsmonitor--daemon` daemon and\n+ * ask for the changes since the given token or from the last\n+ * token in the index extension.\n+ *\n+ * This will implicitly start a daemon process if necessary. The\n+ * daemon process will persist after we exit.\n+ */\n+static int do_send_query(const char *token)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\n+\tret = fsmonitor_ipc__send_query(token, &answer);\n+\tif (ret < 0)\n+\t\tdie(\"could not query fsmonitor--daemon\");\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n+ * and tell it to flush its cache.\n+ *\n+ * This feature is primarily used by the test suite to simulate a loss of\n+ * sync with the filesystem where we miss kernel events.\n+ */\n+static int do_send_flush(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tconst char *subcmd;\n+\tconst char *token = NULL;\n+\n+\tconst char * const fsmonitor_client_usage[] = {\n+\t\t\"test-tool fsmonitor-client query []\",\n+\t\t\"test-tool fsmonitor-client flush\",\n+\t\tNULL,\n+\t};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING(0, \"token\", &token, \"token\",\n+\t\t\t \"command token to send to the server\"),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n+\n+\tif (argc != 1)\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tsubcmd = argv[0];\n+\n+\tsetup_git_directory();\n+\n+\tif (!strcmp(subcmd, \"query\"))\n+\t\treturn !!do_send_query(token);\n+\n+\tif (!strcmp(subcmd, \"flush\"))\n+\t\treturn !!do_send_flush();\n+\n+\tdie(\"Unhandled subcommand: '%s'\", subcmd);\n+}\n+#endif\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex e6ec69cf326..0424f7adf5d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -32,6 +32,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\n \t{ \"getcwd\", cmd__getcwd },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 20756eefdda..c876e8246fb 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"452328","messageId":"899c23f63c375438f060e9ee81440bee8c6613de.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 27/30] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:10Z","receivedAt":"2022-03-25T19:29:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to periodically truncate the list of\nmodified files to save some memory.\n\nClients will ask for the set of changes relative to a token that they\nfound in the FSMN index extension in the index. (This token is like a\npoint in time, but different). Clients will then update the index to\ncontain the response token (so that subsequent commands will be\nrelative to this new token).\n\nTherefore, the daemon can gradually truncate the in-memory list of\nchanged paths as they become obsolete (older than the previous token).\nSince we may have multiple clients making concurrent requests with a\nskew of tokens and clients may be racing to the talk to the daemon,\nwe lazily truncate the list.\n\nWe introduce a 5 minute delay and truncate batches 5 minutes after\nthey are considered obsolete.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 88 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 88 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex eafaafb45b1..ab9cc09f7ce 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -312,6 +312,75 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n \t\t\tbatch_src->interned_paths[k];\n }\n \n+/*\n+ * To keep the batch list from growing unbounded in response to filesystem\n+ * activity, we try to truncate old batches from the end of the list as\n+ * they become irrelevant.\n+ *\n+ * We assume that the .git/index will be updated with the most recent token\n+ * any time the index is updated. And future commands will only ask for\n+ * recent changes *since* that new token. So as tokens advance into the\n+ * future, older batch items will never be requested/needed. So we can\n+ * truncate them without loss of functionality.\n+ *\n+ * However, multiple commands may be talking to the daemon concurrently\n+ * or perform a slow command, so a little \"token skew\" is possible.\n+ * Therefore, we want this to be a little bit lazy and have a generous\n+ * delay.\n+ *\n+ * The current reader thread walked backwards in time from `token->batch_head`\n+ * back to `batch_marker` somewhere in the middle of the batch list.\n+ *\n+ * Let's walk backwards in time from that marker an arbitrary delay\n+ * and truncate the list there. Note that these timestamps are completely\n+ * artificial (based on when we pinned the batch item) and not on any\n+ * filesystem activity.\n+ *\n+ * Return the obsolete portion of the list after we have removed it from\n+ * the official list so that the caller can free it after leaving the lock.\n+ */\n+#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */\n+\n+static struct fsmonitor_batch *with_lock__truncate_old_batches(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst struct fsmonitor_batch *batch_marker)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder;\n+\n+\tif (!batch_marker)\n+\t\treturn NULL;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Truncate: mark (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t batch_marker->batch_seq_nr,\n+\t\t\t (uint64_t)batch_marker->pinned_time);\n+\n+\tfor (batch = batch_marker; batch; batch = batch->next) {\n+\t\ttime_t t;\n+\n+\t\tif (!batch->pinned_time) /* an overflow batch */\n+\t\t\tcontinue;\n+\n+\t\tt = batch->pinned_time + MY_TIME_DELAY_SECONDS;\n+\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n+\t\t\tcontinue;\n+\n+\t\tgoto truncate_past_here;\n+\t}\n+\n+\treturn NULL;\n+\n+truncate_past_here:\n+\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n+\n+\tremainder = ((struct fsmonitor_batch *)batch)->next;\n+\t((struct fsmonitor_batch *)batch)->next = NULL;\n+\n+\treturn remainder;\n+}\n+\n static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n {\n \tif (!token)\n@@ -425,6 +494,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tconst char *p;\n \tconst struct fsmonitor_batch *batch_head;\n \tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *remainder = NULL;\n \tintmax_t count = 0, duplicates = 0;\n \tkh_str_t *shown;\n \tint hash_ret;\n@@ -652,11 +722,29 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * that work.\n \t\t\t */\n \t\t\tfsmonitor_free_token_data(token_data);\n+\t\t} else if (batch) {\n+\t\t\t/*\n+\t\t\t * We are holding the lock and are the only\n+\t\t\t * reader of the ref-counted portion of the\n+\t\t\t * list, so we get the honor of seeing if the\n+\t\t\t * list can be truncated to save memory.\n+\t\t\t *\n+\t\t\t * The main loop did not walk to the end of the\n+\t\t\t * list, so this batch is the first item in the\n+\t\t\t * batch-list that is older than the requested\n+\t\t\t * end-point sequence number. See if the tail\n+\t\t\t * end of the list is obsolete.\n+\t\t\t */\n+\t\t\tremainder = with_lock__truncate_old_batches(state,\n+\t\t\t\t\t\t\t\t batch);\n \t\t}\n \t}\n \n \tpthread_mutex_unlock(&state->main_lock);\n \n+\tif (remainder)\n+\t\tfsmonitor_batch__free_list(remainder);\n+\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n \ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n-- \ngitgitgadget\n\n"},{"id":"452331","messageId":"53e06b4ae5d0eb1d3c2cb6a61ffd14abc36f6534.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 13/30] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:56Z","receivedAt":"2022-03-25T19:30:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to create token-ids and define the\noverall token naming scheme.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 115 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1ce00b7c150..1c7c156288d 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -106,6 +106,120 @@ static int do_as_client__status(void)\n \t}\n }\n \n+/*\n+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n+ * \"token\" as a virtual timestamp. Clients can request a summary of all\n+ * created/deleted/modified files relative to a token. In the response,\n+ * clients receive a new token for the next (relative) request.\n+ *\n+ *\n+ * Token Format\n+ * ============\n+ *\n+ * The contents of the token are private and provider-specific.\n+ *\n+ * For the built-in fsmonitor--daemon, we define a token as follows:\n+ *\n+ * \"builtin\" \":\" \":\" \n+ *\n+ * The \"builtin\" prefix is used as a namespace to avoid conflicts\n+ * with other providers (such as Watchman).\n+ *\n+ * The is an arbitrary OPAQUE string, such as a GUID,\n+ * UUID, or {timestamp,pid}. It is used to group all filesystem\n+ * events that happened while the daemon was monitoring (and in-sync\n+ * with the filesystem).\n+ *\n+ * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n+ * and does not define less-than/greater-than relationships.\n+ * (There are too many race conditions to rely on file system\n+ * event timestamps.)\n+ *\n+ * The is a simple integer incremented whenever the\n+ * daemon needs to make its state public. For example, if 1000 file\n+ * system events come in, but no clients have requested the data,\n+ * the daemon can continue to accumulate file changes in the same\n+ * bin and does not need to advance the sequence number. However,\n+ * as soon as a client does arrive, the daemon needs to start a new\n+ * bin and increment the sequence number.\n+ *\n+ * The sequence number serves as the boundary between 2 sets\n+ * of bins -- the older ones that the client has already seen\n+ * and the newer ones that it hasn't.\n+ *\n+ * When a new is created, the is reset to\n+ * zero.\n+ *\n+ *\n+ * About Token Ids\n+ * ===============\n+ *\n+ * A new token_id is created:\n+ *\n+ * [1] each time the daemon is started.\n+ *\n+ * [2] any time that the daemon must re-sync with the filesystem\n+ * (such as when the kernel drops or we miss events on a very\n+ * active volume).\n+ *\n+ * [3] in response to a client \"flush\" command (for dropped event\n+ * testing).\n+ *\n+ * When a new token_id is created, the daemon is free to discard all\n+ * cached filesystem events associated with any previous token_ids.\n+ * Events associated with a non-current token_id will never be sent\n+ * to a client. A token_id change implicitly means that the daemon\n+ * has gap in its event history.\n+ *\n+ * Therefore, clients that present a token with a stale (non-current)\n+ * token_id will always be given a trivial response.\n+ */\n+struct fsmonitor_token_data {\n+\tstruct strbuf token_id;\n+\tstruct fsmonitor_batch *batch_head;\n+\tstruct fsmonitor_batch *batch_tail;\n+\tuint64_t client_ref_count;\n+};\n+\n+static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n+{\n+\tstatic int test_env_value = -1;\n+\tstatic uint64_t flush_count = 0;\n+\tstruct fsmonitor_token_data *token;\n+\n+\tCALLOC_ARRAY(token, 1);\n+\n+\tstrbuf_init(&token->token_id, 0);\n+\ttoken->batch_head = NULL;\n+\ttoken->batch_tail = NULL;\n+\ttoken->client_ref_count = 0;\n+\n+\tif (test_env_value < 0)\n+\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n+\n+\tif (!test_env_value) {\n+\t\tstruct timeval tv;\n+\t\tstruct tm tm;\n+\t\ttime_t secs;\n+\n+\t\tgettimeofday(&tv, NULL);\n+\t\tsecs = tv.tv_sec;\n+\t\tgmtime_r(&secs, &tm);\n+\n+\t\tstrbuf_addf(&token->token_id,\n+\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n+\t\t\t flush_count++,\n+\t\t\t getpid(),\n+\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n+\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n+\t\t\t (long)tv.tv_usec);\n+\t} else {\n+\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n+\t}\n+\n+\treturn token;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -298,7 +412,7 @@ static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n-\tstate.current_token_data = NULL;\n+\tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n \tstrbuf_init(&state.path_worktree_watch, 0);\n-- \ngitgitgadget\n\n"},{"id":"452332","messageId":"39f43fabe024917e6c84b8916197b0dc0add97fc.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 14/30] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:57Z","receivedAt":"2022-03-25T19:31:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to build a list of changed paths and associate\nthem with a token-id. This will be used by the platform-specific\nbackends to accumulate changed paths in response to filesystem events.\n\nThe platform-specific file system listener thread receives file system\nevents containing one or more changed pathnames (with whatever\nbucketing or grouping that is convenient for the file system). These\npaths are accumulated (without locking) by the file system layer into\na `fsmonitor_batch`.\n\nWhen the file system layer has drained the kernel event queue, it will\n\"publish\" them to our token queue and make them visible to concurrent\nclient worker threads. The token layer is free to combine and/or de-dup\npaths within these batches for efficient presentation to clients.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 230 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 40 +++++++\n 2 files changed, 268 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1c7c156288d..69312119b07 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -181,17 +181,27 @@ struct fsmonitor_token_data {\n \tuint64_t client_ref_count;\n };\n \n+struct fsmonitor_batch {\n+\tstruct fsmonitor_batch *next;\n+\tuint64_t batch_seq_nr;\n+\tconst char **interned_paths;\n+\tsize_t nr, alloc;\n+\ttime_t pinned_time;\n+};\n+\n static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n {\n \tstatic int test_env_value = -1;\n \tstatic uint64_t flush_count = 0;\n \tstruct fsmonitor_token_data *token;\n+\tstruct fsmonitor_batch *batch;\n \n \tCALLOC_ARRAY(token, 1);\n+\tbatch = fsmonitor_batch__new();\n \n \tstrbuf_init(&token->token_id, 0);\n-\ttoken->batch_head = NULL;\n-\ttoken->batch_tail = NULL;\n+\ttoken->batch_head = batch;\n+\ttoken->batch_tail = batch;\n \ttoken->client_ref_count = 0;\n \n \tif (test_env_value < 0)\n@@ -217,9 +227,143 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n \t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n \t}\n \n+\t/*\n+\t * We created a new and are starting a new series\n+\t * of tokens with a zero .\n+\t *\n+\t * Since clients cannot guess our new (non test) \n+\t * they will always receive a trivial response (because of the\n+\t * mismatch on the ). The trivial response will\n+\t * tell them our new so that subsequent requests\n+\t * will be relative to our new series. (And when sending that\n+\t * response, we pin the current head of the batch list.)\n+\t *\n+\t * Even if the client correctly guesses the , their\n+\t * request of \"builtin::0\" asks for all changes MORE\n+\t * RECENT than batch/bin 0.\n+\t *\n+\t * This implies that it is a waste to accumulate paths in the\n+\t * initial batch/bin (because they will never be transmitted).\n+\t *\n+\t * So the daemon could be running for days and watching the\n+\t * file system, but doesn't need to actually accumulate any\n+\t * paths UNTIL we need to set a reference point for a later\n+\t * relative request.\n+\t *\n+\t * However, it is very useful for testing to always have a\n+\t * reference point set. Pin batch 0 to force early file system\n+\t * events to accumulate.\n+\t */\n+\tif (test_env_value)\n+\t\tbatch->pinned_time = time(NULL);\n+\n \treturn token;\n }\n \n+struct fsmonitor_batch *fsmonitor_batch__new(void)\n+{\n+\tstruct fsmonitor_batch *batch;\n+\n+\tCALLOC_ARRAY(batch, 1);\n+\n+\treturn batch;\n+}\n+\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch)\n+{\n+\twhile (batch) {\n+\t\tstruct fsmonitor_batch *next = batch->next;\n+\n+\t\t/*\n+\t\t * The actual strings within the array of this batch\n+\t\t * are interned, so we don't own them. We only own\n+\t\t * the array.\n+\t\t */\n+\t\tfree(batch->interned_paths);\n+\t\tfree(batch);\n+\n+\t\tbatch = next;\n+\t}\n+}\n+\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n+\t\t\t const char *path)\n+{\n+\tconst char *interned_path = strintern(path);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n+\n+\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n+\tbatch->interned_paths[batch->nr++] = interned_path;\n+}\n+\n+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n+\t\t\t\t const struct fsmonitor_batch *batch_src)\n+{\n+\tsize_t k;\n+\n+\tALLOC_GROW(batch_dest->interned_paths,\n+\t\t batch_dest->nr + batch_src->nr + 1,\n+\t\t batch_dest->alloc);\n+\n+\tfor (k = 0; k < batch_src->nr; k++)\n+\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n+\t\t\tbatch_src->interned_paths[k];\n+}\n+\n+static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n+{\n+\tif (!token)\n+\t\treturn;\n+\n+\tassert(token->client_ref_count == 0);\n+\n+\tstrbuf_release(&token->token_id);\n+\n+\tfsmonitor_batch__free_list(token->batch_head);\n+\n+\tfree(token);\n+}\n+\n+/*\n+ * Flush all of our cached data about the filesystem. Call this if we\n+ * lose sync with the filesystem and miss some notification events.\n+ *\n+ * [1] If we are missing events, then we no longer have a complete\n+ * history of the directory (relative to our current start token).\n+ * We should create a new token and start fresh (as if we just\n+ * booted up).\n+ *\n+ * If there are no concurrent threads reading the current token data\n+ * series, we can free it now. Otherwise, let the last reader free\n+ * it.\n+ *\n+ * Either way, the old token data series is no longer associated with\n+ * our state data.\n+ */\n+static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct fsmonitor_token_data *free_me = NULL;\n+\tstruct fsmonitor_token_data *new_one = NULL;\n+\n+\tnew_one = fsmonitor_new_token_data();\n+\n+\tif (state->current_token_data->client_ref_count == 0)\n+\t\tfree_me = state->current_token_data;\n+\tstate->current_token_data = new_one;\n+\n+\tfsmonitor_free_token_data(free_me);\n+}\n+\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\tpthread_mutex_lock(&state->main_lock);\n+\twith_lock__do_force_resync(state);\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -329,6 +473,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n \n+/*\n+ * We try to combine small batches at the front of the batch-list to avoid\n+ * having a long list. This hopefully makes it a little easier when we want\n+ * to truncate and maintain the list. However, we don't want the paths array\n+ * to just keep growing and growing with realloc, so we insert an arbitrary\n+ * limit.\n+ */\n+#define MY_COMBINE_LIMIT (1024)\n+\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names)\n+{\n+\tif (!batch && !cookie_names->nr)\n+\t\treturn;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (batch) {\n+\t\tstruct fsmonitor_batch *head;\n+\n+\t\thead = state->current_token_data->batch_head;\n+\t\tif (!head) {\n+\t\t\tBUG(\"token does not have batch\");\n+\t\t} else if (head->pinned_time) {\n+\t\t\t/*\n+\t\t\t * We cannot alter the current batch list\n+\t\t\t * because:\n+\t\t\t *\n+\t\t\t * [a] it is being transmitted to at least one\n+\t\t\t * client and the handle_client() thread has a\n+\t\t\t * ref-count, but not a lock on the batch list\n+\t\t\t * starting with this item.\n+\t\t\t *\n+\t\t\t * [b] it has been transmitted in the past to\n+\t\t\t * at least one client such that future\n+\t\t\t * requests are relative to this head batch.\n+\t\t\t *\n+\t\t\t * So, we can only prepend a new batch onto\n+\t\t\t * the front of the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else if (!head->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * Batch 0 is unpinned. See the note in\n+\t\t\t * `fsmonitor_new_token_data()` about why we\n+\t\t\t * don't need to accumulate these paths.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n+\t\t\t/*\n+\t\t\t * The head batch in the list has never been\n+\t\t\t * transmitted to a client, but folding the\n+\t\t\t * contents of the new batch onto it would\n+\t\t\t * exceed our arbitrary limit, so just prepend\n+\t\t\t * the new batch onto the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We are free to add the paths in the given\n+\t\t\t * batch onto the end of the current head batch.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__combine(head, batch);\n+\t\t\tfsmonitor_batch__free_list(batch);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -343,6 +562,13 @@ static void *fsm_listen__thread_proc(void *_state)\n \n \tfsm_listen__loop(state);\n \n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (state->current_token_data &&\n+\t state->current_token_data->client_ref_count == 0)\n+\t\tfsmonitor_free_token_data(state->current_token_data);\n+\tstate->current_token_data = NULL;\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n \ttrace2_thread_exit();\n \treturn NULL;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 8c3a71a48bd..010fbfe60e9 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -12,6 +12,27 @@\n struct fsmonitor_batch;\n struct fsmonitor_token_data;\n \n+/*\n+ * Create a new batch of path(s). The returned batch is considered\n+ * private and not linked into the fsmonitor daemon state. The caller\n+ * should fill this batch with one or more paths and then publish it.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__new(void);\n+\n+/*\n+ * Free the list of batches starting with this one.\n+ */\n+void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n+\n+/*\n+ * Add this path to this batch of modified files.\n+ *\n+ * The batch should be private and NOT (yet) linked into the fsmonitor\n+ * daemon state and therefore not yet visible to worker threads and so\n+ * no locking is required.\n+ */\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n+\n struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n \n struct fsmonitor_daemon_state {\n@@ -117,5 +138,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \tstruct fsmonitor_daemon_state *state,\n \tconst char *path);\n \n+/*\n+ * Prepend the this batch of path(s) onto the list of batches associated\n+ * with the current token. This makes the batch visible to worker threads.\n+ *\n+ * The caller no longer owns the batch and must not free it.\n+ *\n+ * Wake up the client threads waiting on these cookies.\n+ */\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names);\n+\n+/*\n+ * If the platform-specific layer loses sync with the filesystem,\n+ * it should call this to invalidate cached data and abort waiting\n+ * threads.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"452334","messageId":"ab68b944173cb8d4fdc0d7583a3d1dd11e5aef78.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 02/30] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:45Z","receivedAt":"2022-03-25T19:32:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 ++++++++++++++\n 3 files changed, 220 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 6f0b4b775fe..a19d850e716 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -907,6 +907,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..789e7397baa\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,171 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#else\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(_(\"fsmonitor--daemon is not running\"));\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(_(\"could not send '%s' command to fsmonitor--daemon\"), c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..b6a7067c3af\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"452336","messageId":"e98373f997f5994a08423b3f42b6ddadb3a2a511.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 01/30] fsmonitor: enhance existing comments, clarify trivial response handling","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:44Z","receivedAt":"2022-03-25T19:32:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 64 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 41 insertions(+), 23 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..448d0ee33f5 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -168,29 +168,15 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf *\n \n \tif (result)\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/failed\", result);\n-\telse {\n+\telse\n \t\ttrace2_data_intmax(\"fsm_hook\", NULL, \"query/response-length\",\n \t\t\t\t query_result->len);\n \n-\t\tif (fsmonitor_is_trivial_response(query_result))\n-\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n-\t\t\t\t\t \"query/trivial-response\", 1);\n-\t}\n-\n \ttrace2_region_leave(\"fsm_hook\", \"query\", NULL);\n \n \treturn result;\n }\n \n-int fsmonitor_is_trivial_response(const struct strbuf *query_result)\n-{\n-\tstatic char trivial_response[3] = { '\\0', '/', '\\0' };\n-\n-\treturn query_result->len >= 3 &&\n-\t\t!memcmp(trivial_response,\n-\t\t\t&query_result->buf[query_result->len - 3], 3);\n-}\n-\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n@@ -238,6 +224,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tint is_trivial = 0;\n \n \tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n \t\treturn;\n@@ -283,6 +270,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t\tquery_success = 0;\n \t\t\t\t} else {\n \t\t\t\t\tbol = last_update_token.len + 1;\n+\t\t\t\t\tis_trivial = query_result.buf[bol] == '/';\n \t\t\t\t}\n \t\t\t} else if (hook_version < 0) {\n \t\t\t\thook_version = HOOK_INTERFACE_VERSION1;\n@@ -294,16 +282,38 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\t\tif (query_success)\n+\t\t\t\tis_trivial = query_result.buf[0] == '/';\n \t\t}\n \n+\t\tif (is_trivial)\n+\t\t\ttrace2_data_intmax(\"fsm_hook\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n \t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n \t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n-\tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n+\tif (query_success && !is_trivial) {\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -318,11 +328,16 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We failed to get a response or received a trivial response,\n+\t\t * so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -330,7 +345,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"452338","messageId":"ea02ba25d8f0e5e76a646d23306eb6a2b312ebe2.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 04/30] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:47Z","receivedAt":"2022-03-25T19:32:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.fsmonitor` is set to true.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 38 ++++++++++++++++++++++++++++++++++++--\n 1 file changed, 36 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 0e961b74d82..a38b5710eb3 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -241,8 +241,41 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (fsm_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t\tis_trivial = query_result.buf[bol] == '/';\n+\t\t\tif (is_trivial)\n+\t\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t\t \"query/trivial-response\", 1);\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\tgoto apply_results;\n \t}\n \n \tassert(fsm_mode == FSMONITOR_MODE_HOOK);\n@@ -315,6 +348,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/*\n \t * The response from FSMonitor (excluding the header token) is\n \t * either:\n-- \ngitgitgadget\n\n"},{"id":"452339","messageId":"14b775e9d8b1a4672f8175a546eb70e2790c1b23.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:59Z","receivedAt":"2022-03-25T19:32:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInclude MacOS system declarations to allow us to use FSEvent and\nCoreFoundation APIs. We need different versions of the declarations\nfor GCC vs. clang because of compiler and header file conflicts.\n\nWhile it is quite possible to #include Apple's CoreServices.h when\ncompiling C source code with clang, trying to build it with GCC\ncurrently fails with this error:\n\nIn file included\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Security.h:42,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/CSIdentity.h:43,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...OSServices.framework/Headers/OSServices.h:29,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/IconsCore.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Frameworks/...\n ...LaunchServices.framework/Headers/LaunchServices.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n\n /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/...\n ...Library/Frameworks/Security.framework/Headers/Authorization.h:193:7:\n error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n\nThe underlying reason is that GCC (rightfully) objects that an `enum`\nvalue such as `kAuthorizationExternalFormLength` is not a constant\n(because it is not, the preprocessor has no knowledge of it, only the\nactual C compiler does) and can therefore not be used to define the size\nof a C array.\n\nThis is a known problem and tracked in GCC's bug tracker:\nhttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nIn the meantime, let's not block things and go the slightly ugly route\nof declaring/defining the FSEvents constants, data structures and\nfunctions that we need, so that we can avoid above-mentioned issue.\n\nLet's do this _only_ for GCC, though, so that the CI/PR builds (which\nbuild both with clang and with GCC) can guarantee that we _are_ using\nthe correct data types.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-darwin-gcc.h | 92 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-listen-darwin.c | 24 ++++++++\n 2 files changed, 116 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-darwin-gcc.h\n\ndiff --git a/compat/fsmonitor/fsm-darwin-gcc.h b/compat/fsmonitor/fsm-darwin-gcc.h\nnew file mode 100644\nindex 00000000000..1c75c3d48e7\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-darwin-gcc.h\n@@ -0,0 +1,92 @@\n+#ifndef FSM_DARWIN_GCC_H\n+#define FSM_DARWIN_GCC_H\n+\n+#ifndef __clang__\n+/*\n+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n+ * with clang, but not with GCC as of time of writing.\n+ *\n+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details.\n+ */\n+typedef unsigned int FSEventStreamCreateFlags;\n+#define kFSEventStreamEventFlagNone 0x00000000\n+#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n+#define kFSEventStreamEventFlagUserDropped 0x00000002\n+#define kFSEventStreamEventFlagKernelDropped 0x00000004\n+#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n+#define kFSEventStreamEventFlagHistoryDone 0x00000010\n+#define kFSEventStreamEventFlagRootChanged 0x00000020\n+#define kFSEventStreamEventFlagMount 0x00000040\n+#define kFSEventStreamEventFlagUnmount 0x00000080\n+#define kFSEventStreamEventFlagItemCreated 0x00000100\n+#define kFSEventStreamEventFlagItemRemoved 0x00000200\n+#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n+#define kFSEventStreamEventFlagItemRenamed 0x00000800\n+#define kFSEventStreamEventFlagItemModified 0x00001000\n+#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n+#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n+#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n+#define kFSEventStreamEventFlagItemIsFile 0x00010000\n+#define kFSEventStreamEventFlagItemIsDir 0x00020000\n+#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n+#define kFSEventStreamEventFlagOwnEvent 0x00080000\n+#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n+#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+\n+typedef struct __FSEventStream *FSEventStreamRef;\n+typedef const FSEventStreamRef ConstFSEventStreamRef;\n+\n+typedef unsigned int CFStringEncoding;\n+#define kCFStringEncodingUTF8 0x08000100\n+\n+typedef const struct __CFString *CFStringRef;\n+typedef const struct __CFArray *CFArrayRef;\n+typedef const struct __CFRunLoop *CFRunLoopRef;\n+\n+struct FSEventStreamContext {\n+ long long version;\n+ void *cb_data, *retain, *release, *copy_description;\n+};\n+\n+typedef struct FSEventStreamContext FSEventStreamContext;\n+typedef unsigned int FSEventStreamEventFlags;\n+#define kFSEventStreamCreateFlagNoDefer 0x02\n+#define kFSEventStreamCreateFlagWatchRoot 0x04\n+#define kFSEventStreamCreateFlagFileEvents 0x10\n+\n+typedef unsigned long long FSEventStreamEventId;\n+#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL\n+\n+typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef,\n+\t\t\t\t void *context,\n+\t\t\t\t __SIZE_TYPE__ num_of_events,\n+\t\t\t\t void *event_paths,\n+\t\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t\t const FSEventStreamEventId event_ids[]);\n+typedef double CFTimeInterval;\n+FSEventStreamRef FSEventStreamCreate(void *allocator,\n+\t\t\t\t FSEventStreamCallback callback,\n+\t\t\t\t FSEventStreamContext *context,\n+\t\t\t\t CFArrayRef paths_to_watch,\n+\t\t\t\t FSEventStreamEventId since_when,\n+\t\t\t\t CFTimeInterval latency,\n+\t\t\t\t FSEventStreamCreateFlags flags);\n+CFStringRef CFStringCreateWithCString(void *allocator, const char *string,\n+\t\t\t\t CFStringEncoding encoding);\n+CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count,\n+\t\t\t void *callbacks);\n+void CFRunLoopRun(void);\n+void CFRunLoopStop(CFRunLoopRef run_loop);\n+CFRunLoopRef CFRunLoopGetCurrent(void);\n+extern CFStringRef kCFRunLoopDefaultMode;\n+void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream,\n+\t\t\t\t CFRunLoopRef run_loop,\n+\t\t\t\t CFStringRef run_loop_mode);\n+unsigned char FSEventStreamStart(FSEventStreamRef stream);\n+void FSEventStreamStop(FSEventStreamRef stream);\n+void FSEventStreamInvalidate(FSEventStreamRef stream);\n+void FSEventStreamRelease(FSEventStreamRef stream);\n+\n+#endif /* !clang */\n+#endif /* FSM_DARWIN_GCC_H */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex c84e3344ab9..d2ce942cade 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -1,3 +1,27 @@\n+#ifndef __clang__\n+#include \"fsm-darwin-gcc.h\"\n+#else\n+#include \n+#include \n+\n+#ifndef AVAILABLE_MAC_OS_X_VERSION_10_13_AND_LATER\n+/*\n+ * This enum value was added in 10.13 to:\n+ *\n+ * /Applications/Xcode.app/Contents/Developer/Platforms/ \\\n+ * MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/ \\\n+ * Library/Frameworks/CoreServices.framework/Frameworks/ \\\n+ * FSEvents.framework/Versions/Current/Headers/FSEvents.h\n+ *\n+ * If we're compiling against an older SDK, this symbol won't be\n+ * present. Silently define it here so that we don't have to ifdef\n+ * the logging or masking below. This should be harmless since older\n+ * versions of macOS won't ever emit this FS event anyway.\n+ */\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+#endif\n+#endif\n+\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n-- \ngitgitgadget\n\n"},{"id":"452340","messageId":"2f8a42fdb93623053256939fdbb45cf02fb1c5c9.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 09/30] compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:52Z","receivedAt":"2022-03-25T19:32:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty implementation of fsmonitor--daemon\nbackend for Darwin (aka MacOS).\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-darwin.c | 20 ++++++++++++++++++++\n config.mak.uname | 10 ++++++++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 33 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nnew file mode 100644\nindex 00000000000..c84e3344ab9\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -0,0 +1,20 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 26074f56bed..501970902da 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -157,6 +157,16 @@ ifeq ($(uname_S),Darwin)\n \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n \t\tendif\n \tendif\n+\n+\t# The builtin FSMonitor on MacOS builds upon Simple-IPC. Both require\n+\t# Unix domain sockets and PThreads.\n+\tifndef NO_PTHREADS\n+\tifndef NO_UNIX_SOCKETS\n+\tFSMONITOR_DAEMON_BACKEND = darwin\n+\tendif\n+\tendif\n+\n+\tBASIC_LDFLAGS += -framework CoreServices\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 0963629db7f..ee0d7257b77 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"452342","messageId":"0ce8ae3f2cf07bfffdb1cb14454768d2b6d84bf2.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 06/30] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:49Z","receivedAt":"2022-03-25T19:32:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a built-in file system monitoring daemon that can be used by\nthe existing `fsmonitor` feature (protocol API and index extension)\nto improve the performance of various Git commands, such as `status`.\n\nThe `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\nprovides an alternative to hook access to existing fsmonitors such\nas `watchman`.\n\nThis commit merely adds the new command without any functionality.\n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 46 +++++++++++++++++++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 50 insertions(+)\n create mode 100644 builtin/fsmonitor--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex f817c509ec0..e81de1063a4 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -72,6 +72,7 @@\n /git-format-patch\n /git-fsck\n /git-fsck-objects\n+/git-fsmonitor--daemon\n /git-gc\n /git-get-tar-commit-id\n /git-grep\ndiff --git a/Makefile b/Makefile\nindex 707a56d4c11..5af1d5b112e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1114,6 +1114,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n+BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex 83379f3832c..40e9ecc8485 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\n int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nnew file mode 100644\nindex 00000000000..f0498793379\n--- /dev/null\n+++ b/builtin/fsmonitor--daemon.c\n@@ -0,0 +1,46 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"simple-ipc.h\"\n+#include \"khash.h\"\n+\n+static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tNULL\n+};\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *subcmd;\n+\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\tif (argc != 1)\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\tsubcmd = argv[0];\n+\n+\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n+}\n+\n+#else\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n+}\n+#endif\ndiff --git a/git.c b/git.c\nindex a25940d72e8..3d8e48cf555 100644\n--- a/git.c\n+++ b/git.c\n@@ -537,6 +537,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"452355","messageId":"4624ce2fa471b3ebb3a08baf0de2609447eb9fe3.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 07/30] fsmonitor--daemon: implement 'stop' and 'status' commands","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:50Z","receivedAt":"2022-03-25T19:35:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `stop` and `status` client commands to control and query the\nstatus of a `fsmonitor--daemon` server process (and implicitly start a\nserver process if necessary).\n\nLater commits will implement the actual server and monitor the file\nsystem.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 51 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex f0498793379..5e3178b8bdd 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,10 +7,55 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon stop\"),\n+\tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n+ * and wait for it to shutdown.\n+ */\n+static int do_as_client__send_stop(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"quit\", &answer);\n+\n+\t/* The quit command does not return any response data. */\n+\tstrbuf_release(&answer);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\twhile (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tsleep_millisec(50);\n+\ttrace2_region_leave(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\n+\treturn 0;\n+}\n+\n+static int do_as_client__status(void)\n+{\n+\tenum ipc_active_state state = fsmonitor_ipc__get_state();\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tprintf(_(\"fsmonitor-daemon is watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tprintf(_(\"fsmonitor-daemon is not watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 1;\n+\t}\n+}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n@@ -28,6 +73,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (!strcmp(subcmd, \"stop\"))\n+\t\treturn !!do_as_client__send_stop();\n+\n+\tif (!strcmp(subcmd, \"status\"))\n+\t\treturn !!do_as_client__status();\n+\n \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"452359","messageId":"a6a39a3306df2602e06cc3d9a755fa247f8d41b0.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 11/30] fsmonitor--daemon: implement 'start' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:54Z","receivedAt":"2022-03-25T19:38:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement 'git fsmonitor--daemon start' command. This command starts\nan instance of 'git fsmonitor--daemon run' in the background using\nthe new 'start_bg_command()' function.\n\nWe avoid the fork-and-call technique on Unix systems in favor of a\nfork-and-exec technique. This gives us more uniform Trace2 child-*\nevents. It also makes our usage more consistent with Windows usage.\n\nOn Windows, teach 'git fsmonitor--daemon run' to optionally call\n'FreeConsole()' to release handles to the inherited Win32 console\n(despite being passed invalid handles for stdin/out/err). Without\nthis, command prompts and powershell terminal windows could hang\nin \"exit\" until the last background child process exited or released\ntheir Win32 console handle. (This was not seen with git-bash shells\nbecause they don't have a Win32 console attached to them.)\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 109 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 107 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 5591339399a..69dd39121a3 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -9,6 +9,7 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon start []\"),\n \tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n@@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n #define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n static int fsmonitor__ipc_threads = 8;\n \n+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n+static int fsmonitor__start_timeout_sec = 60;\n+\n #define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n static int fsmonitor__announce_startup = 0;\n \n@@ -36,6 +40,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n+\t\tfsmonitor__start_timeout_sec = i;\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n \t\tint is_bool;\n \t\tint i = git_config_bool_or_int(var, value, &is_bool);\n@@ -250,7 +263,7 @@ done:\n \treturn err;\n }\n \n-static int try_to_run_foreground_daemon(void)\n+static int try_to_run_foreground_daemon(int detach_console)\n {\n \t/*\n \t * Technically, we don't need to probe for an existing daemon\n@@ -270,17 +283,106 @@ static int try_to_run_foreground_daemon(void)\n \t\tfflush(stderr);\n \t}\n \n+#ifdef GIT_WINDOWS_NATIVE\n+\tif (detach_console)\n+\t\tFreeConsole();\n+#endif\n+\n \treturn !!fsmonitor_run_daemon();\n }\n \n+static start_bg_wait_cb bg_wait_cb;\n+\n+static int bg_wait_cb(const struct child_process *cp, void *cb_data)\n+{\n+\tenum ipc_active_state s = fsmonitor_ipc__get_state();\n+\n+\tswitch (s) {\n+\tcase IPC_STATE__LISTENING:\n+\t\t/* child is \"ready\" */\n+\t\treturn 0;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\t/* give child more time */\n+\t\treturn 1;\n+\n+\tdefault:\n+\tcase IPC_STATE__INVALID_PATH:\n+\tcase IPC_STATE__OTHER_ERROR:\n+\t\t/* all the time in world won't help */\n+\t\treturn -1;\n+\t}\n+}\n+\n+static int try_to_start_background_daemon(void)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tenum start_bg_result sbgr;\n+\n+\t/*\n+\t * Before we try to create a background daemon process, see\n+\t * if a daemon process is already listening. This makes it\n+\t * easier for us to report an already-listening error to the\n+\t * console, since our spawn/daemon can only report the success\n+\t * of creating the background process (and not whether it\n+\t * immediately exited).\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(_(\"fsmonitor--daemon is already running '%s'\"),\n+\t\t the_repository->worktree);\n+\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"starting fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n+\n+\tcp.git_cmd = 1;\n+\n+\tstrvec_push(&cp.args, \"fsmonitor--daemon\");\n+\tstrvec_push(&cp.args, \"run\");\n+\tstrvec_push(&cp.args, \"--detach\");\n+\tstrvec_pushf(&cp.args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n+\n+\tcp.no_stdin = 1;\n+\tcp.no_stdout = 1;\n+\tcp.no_stderr = 1;\n+\n+\tsbgr = start_bg_command(&cp, bg_wait_cb, NULL,\n+\t\t\t\tfsmonitor__start_timeout_sec);\n+\n+\tswitch (sbgr) {\n+\tcase SBGR_READY:\n+\t\treturn 0;\n+\n+\tdefault:\n+\tcase SBGR_ERROR:\n+\tcase SBGR_CB_ERROR:\n+\t\treturn error(_(\"daemon failed to start\"));\n+\n+\tcase SBGR_TIMEOUT:\n+\t\treturn error(_(\"daemon not online yet\"));\n+\n+\tcase SBGR_DIED:\n+\t\treturn error(_(\"daemon terminated\"));\n+\t}\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tint detach_console = 0;\n \n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"detach\", &detach_console, N_(\"detach from console\")),\n \t\tOPT_INTEGER(0, \"ipc-threads\",\n \t\t\t &fsmonitor__ipc_threads,\n \t\t\t N_(\"use ipc worker threads\")),\n+\t\tOPT_INTEGER(0, \"start-timeout\",\n+\t\t\t &fsmonitor__start_timeout_sec,\n+\t\t\t N_(\"max seconds to wait for background daemon startup\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -296,8 +398,11 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tif (!strcmp(subcmd, \"start\"))\n+\t\treturn !!try_to_start_background_daemon();\n+\n \tif (!strcmp(subcmd, \"run\"))\n-\t\treturn !!try_to_run_foreground_daemon();\n+\t\treturn !!try_to_run_foreground_daemon(detach_console);\n \n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n-- \ngitgitgadget\n\n"},{"id":"452364","messageId":"bb88cddc1379dddd3a65bb2cf9978257fbd7d60d.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 24/30] t/perf/p7519: fix coding style","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:07Z","receivedAt":"2022-03-25T19:39:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex c8be58f3c76..5241eb6c4e5 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -72,7 +72,7 @@ then\n \tfi\n fi\n \n-trace_start() {\n+trace_start () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tname=\"$1\"\n@@ -91,7 +91,7 @@ trace_start() {\n \tfi\n }\n \n-trace_stop() {\n+trace_stop () {\n \tif test -n \"$GIT_PERF_7519_TRACE\"\n \tthen\n \t\tunset GIT_TRACE2_PERF\n@@ -133,7 +133,7 @@ test_expect_success \"one time repo setup\" '\n \tfi\n '\n \n-setup_for_fsmonitor() {\n+setup_for_fsmonitor () {\n \t# set INTEGRATION_SCRIPT depending on the environment\n \tif test -n \"$INTEGRATION_PATH\"\n \tthen\n@@ -173,7 +173,7 @@ test_perf_w_drop_caches () {\n \ttest_perf \"$@\"\n }\n \n-test_fsmonitor_suite() {\n+test_fsmonitor_suite () {\n \tif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n-- \ngitgitgadget\n\n"},{"id":"452368","messageId":"68a05fd2892b5d3aef6b3e390926e6d3c6b29c2d.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 29/30] fsmonitor: force update index after large responses","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:12Z","receivedAt":"2022-03-25T19:39:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMeasure the time taken to apply the FSMonitor query result\nto the index and the untracked-cache.\n\nSet the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when\nFSMonitor returns a very large repsonse to ensure that the index is\nwritten to disk.\n\nNormally, when the FSMonitor response includes a tracked file, the\nindex is always updated. Similarly, the index might be updated when\nthe response alters the untracked-cache (when enabled). However, in\ncases where neither of those cause the index to be considered changed,\nthe FSMonitor response is wasted. Subsequent Git commands will make\nrequests with the same token and receive the same response.\n\nIf that response is very large, performance may suffer. It would be\nmore efficient to force update the index now (and the token in the\nindex extension) in order to reduce the size of the response received\nby future commands.\n\nThis was observed on Windows after a large checkout. On Windows, the\nkernel emits events for the files that are changed as they are\nchanged. However, it might delay events for the containing\ndirectories until the system is more idle (or someone scans the\ndirectory (so it seems)). The first status following a checkout would\nget the list of files. The subsequent status commands would get the\nlist of directories as the events trickled out. But they would never\ncatch up because the token was not advanced because the index wasn't\nupdated.\n\nThis list of directories caused `wt_status_collect_untracked()` to\nunnecessarily spend time actually scanning them during each command.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n fsmonitor.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 54 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex a38b5710eb3..292a6742b4f 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -219,6 +219,43 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n+/*\n+ * The number of pathnames that we need to receive from FSMonitor\n+ * before we force the index to be updated.\n+ *\n+ * Note that any pathname within the set of received paths MAY cause\n+ * cache-entry or istate flag bits to be updated and thus cause the\n+ * index to be updated on disk.\n+ *\n+ * However, the response may contain many paths (such as ignored\n+ * paths) that will not update any flag bits. And thus not force the\n+ * index to be updated. (This is fine and normal.) It also means\n+ * that the token will not be updated in the FSMonitor index\n+ * extension. So the next Git command will find the same token in the\n+ * index, make the same token-relative request, and receive the same\n+ * response (plus any newly changed paths). If this response is large\n+ * (and continues to grow), performance could be impacted.\n+ *\n+ * For example, if the user runs a build and it writes 100K object\n+ * files but doesn't modify any source files, the index would not need\n+ * to be updated. The FSMonitor response (after the build and\n+ * relative to a pre-build token) might be 5MB. Each subsequent Git\n+ * command will receive that same 100K/5MB response until something\n+ * causes the index to be updated. And `refresh_fsmonitor()` will\n+ * have to iterate over those 100K paths each time.\n+ *\n+ * Performance could be improved if we optionally force update the\n+ * index after a very large response and get an updated token into\n+ * the FSMonitor index extension. This should allow subsequent\n+ * commands to get smaller and more current responses.\n+ *\n+ * The value chosen here does not need to be precise. The index\n+ * will be updated automatically the first time the user touches\n+ * a tracked file and causes a command like `git status` to\n+ * update an mtime to be updated and/or set a flag bit.\n+ */\n+static int fsmonitor_force_update_threshold = 100;\n+\n void refresh_fsmonitor(struct index_state *istate)\n {\n \tstruct strbuf query_result = STRBUF_INIT;\n@@ -362,25 +399,39 @@ apply_results:\n \t * information and that we should consider everything\n \t * invalid. We call this a trivial response.\n \t */\n+\ttrace2_region_enter(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tif (query_success && !is_trivial) {\n \t\t/*\n \t\t * Mark all pathnames returned by the monitor as dirty.\n \t\t *\n \t\t * This updates both the cache-entries and the untracked-cache.\n \t\t */\n+\t\tint count = 0;\n+\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n \t\t\t\tcontinue;\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n \t\t\tbol = i + 1;\n+\t\t\tcount++;\n \t\t}\n-\t\tif (bol < query_result.len)\n+\t\tif (bol < query_result.len) {\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tcount++;\n+\t\t}\n \n \t\t/* Now mark the untracked cache for fsmonitor usage */\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n+\n+\t\tif (count > fsmonitor_force_update_threshold)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", istate->repo, \"apply_count\",\n+\t\t\t\t count);\n+\n \t} else {\n \t\t/*\n \t\t * We failed to get a response or received a trivial response,\n@@ -409,6 +460,8 @@ apply_results:\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 0;\n \t}\n+\ttrace2_region_leave(\"fsmonitor\", \"apply_results\", istate->repo);\n+\n \tstrbuf_release(&query_result);\n \n \t/* Now that we've updated istate, save the last_update_token */\n-- \ngitgitgadget\n\n"},{"id":"452370","messageId":"5b18e3b692661010387b3438e86238dbf0fa2eb2.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 26/30] t/perf/p7519: add fsmonitor--daemon test cases","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:09Z","receivedAt":"2022-03-25T19:39:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRepeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\nthe \"Simple IPC\" interface.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/perf/p7519-fsmonitor.sh | 38 ++++++++++++++++++++++++++++++++++----\n 1 file changed, 34 insertions(+), 4 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex a6c2a910e70..0b9129ca7bc 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -141,7 +141,7 @@ test_expect_success \"one time repo setup\" '\n \tfi\n '\n \n-setup_for_fsmonitor () {\n+setup_for_fsmonitor_hook () {\n \t# set INTEGRATION_SCRIPT depending on the environment\n \tif test -n \"$INTEGRATION_PATH\"\n \tthen\n@@ -182,7 +182,11 @@ test_perf_w_drop_caches () {\n }\n \n test_fsmonitor_suite () {\n-\tif test -n \"$INTEGRATION_SCRIPT\"; then\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n+\tthen\n+\t\tDESC=\"builtin fsmonitor--daemon\"\n+\telif test -n \"$INTEGRATION_SCRIPT\"\n+\tthen\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n \t\tDESC=\"fsmonitor=disabled\"\n@@ -261,11 +265,11 @@ test_fsmonitor_suite () {\n trace_start fsmonitor-watchman\n if test -n \"$GIT_PERF_7519_FSMONITOR\"; then\n \tfor INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do\n-\t\ttest_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor'\n+\t\ttest_expect_success \"setup for fsmonitor $INTEGRATION_PATH\" 'setup_for_fsmonitor_hook'\n \t\ttest_fsmonitor_suite\n \tdone\n else\n-\ttest_expect_success \"setup for fsmonitor\" 'setup_for_fsmonitor'\n+\ttest_expect_success \"setup for fsmonitor hook\" 'setup_for_fsmonitor_hook'\n \ttest_fsmonitor_suite\n fi\n \n@@ -293,4 +297,30 @@ test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n \n+#\n+# Run a full set of perf tests using the built-in fsmonitor--daemon.\n+# It does not use the Hook API, so it has a different setup.\n+# Explicitly start the daemon here and before we start client commands\n+# so that we can later add custom tracing.\n+#\n+if test_have_prereq FSMONITOR_DAEMON\n+then\n+\tUSE_FSMONITOR_DAEMON=t\n+\n+\ttest_expect_success \"setup for builtin fsmonitor\" '\n+\t\ttrace_start fsmonitor--daemon--server &&\n+\t\tgit fsmonitor--daemon start &&\n+\n+\t\ttrace_start fsmonitor--daemon--client &&\n+\n+\t\tgit config core.fsmonitor true &&\n+\t\tgit update-index --fsmonitor\n+\t'\n+\n+\ttest_fsmonitor_suite\n+\n+\tgit fsmonitor--daemon stop\n+\ttrace_stop\n+fi\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"452371","messageId":"597a7192f94dd9c5c5101f5f92495565efdfe34f.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 28/30] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:11Z","receivedAt":"2022-03-25T19:39:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon client threads to create a cookie file\ninside the .git directory and then wait until FS events for the\ncookie are observed by the FS listener thread.\n\nThis helps address the racy nature of file system events by\nblocking the client response until the kernel has drained any\nevent backlog.\n\nThis is especially important on MacOS where kernel events are\nonly issued with a limited frequency. See the `latency` argument\nof `FSeventStreamCreate()`. The kernel only signals every `latency`\nseconds, but does not guarantee that the kernel queue is completely\ndrained, so we may have to wait more than one interval. If we\nincrease the latency, the system is more likely to drop events.\nWe avoid these issues by having each client thread create a unique\ncookie file and then wait until it is seen in the event stream.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 237 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 5 +\n 2 files changed, 241 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex ab9cc09f7ce..46be55a4618 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -107,6 +107,162 @@ static int do_as_client__status(void)\n \t}\n }\n \n+enum fsmonitor_cookie_item_result {\n+\tFCIR_ERROR = -1, /* could not create cookie file ? */\n+\tFCIR_INIT,\n+\tFCIR_SEEN,\n+\tFCIR_ABORT,\n+};\n+\n+struct fsmonitor_cookie_item {\n+\tstruct hashmap_entry entry;\n+\tchar *name;\n+\tenum fsmonitor_cookie_item_result result;\n+};\n+\n+static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n+\t\t const struct hashmap_entry *he2, const void *keydata)\n+{\n+\tconst struct fsmonitor_cookie_item *a =\n+\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n+\tconst struct fsmonitor_cookie_item *b =\n+\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n+\n+\treturn strcmp(a->name, keydata ? keydata : b->name);\n+}\n+\n+static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(\n+\tstruct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint fd;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tstruct strbuf cookie_pathname = STRBUF_INIT;\n+\tstruct strbuf cookie_filename = STRBUF_INIT;\n+\tenum fsmonitor_cookie_item_result result;\n+\tint my_cookie_seq;\n+\n+\tCALLOC_ARRAY(cookie, 1);\n+\n+\tmy_cookie_seq = state->cookie_seq++;\n+\n+\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n+\n+\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n+\tstrbuf_addbuf(&cookie_pathname, &cookie_filename);\n+\n+\tcookie->name = strbuf_detach(&cookie_filename, NULL);\n+\tcookie->result = FCIR_INIT;\n+\thashmap_entry_init(&cookie->entry, strhash(cookie->name));\n+\n+\thashmap_add(&state->cookies, &cookie->entry);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n+\t\t\t cookie->name, cookie_pathname.buf);\n+\n+\t/*\n+\t * Create the cookie file on disk and then wait for a notification\n+\t * that the listener thread has seen it.\n+\t */\n+\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n+\tif (fd < 0) {\n+\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n+\t\t\t cookie->name);\n+\n+\t\tcookie->result = FCIR_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * Technically, close() and unlink() can fail, but we don't\n+\t * care here. We only created the file to trigger a watch\n+\t * event from the FS to know that when we're up to date.\n+\t */\n+\tclose(fd);\n+\tunlink(cookie_pathname.buf);\n+\n+\t/*\n+\t * Technically, this is an infinite wait (well, unless another\n+\t * thread sends us an abort). I'd like to change this to\n+\t * use `pthread_cond_timedwait()` and return an error/timeout\n+\t * and let the caller do the trivial response thing, but we\n+\t * don't have that routine in our thread-utils.\n+\t *\n+\t * After extensive beta testing I'm not really worried about\n+\t * this. Also note that the above open() and unlink() calls\n+\t * will cause at least two FS events on that path, so the odds\n+\t * of getting stuck are pretty slim.\n+\t */\n+\twhile (cookie->result == FCIR_INIT)\n+\t\tpthread_cond_wait(&state->cookies_cond,\n+\t\t\t\t &state->main_lock);\n+\n+done:\n+\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n+\n+\tresult = cookie->result;\n+\n+\tfree(cookie->name);\n+\tfree(cookie);\n+\tstrbuf_release(&cookie_pathname);\n+\n+\treturn result;\n+}\n+\n+/*\n+ * Mark these cookies as _SEEN and wake up the corresponding client threads.\n+ */\n+static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,\n+\t\t\t\t\t const struct string_list *cookie_names)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint k;\n+\tint nr_seen = 0;\n+\n+\tfor (k = 0; k < cookie_names->nr; k++) {\n+\t\tstruct fsmonitor_cookie_item key;\n+\t\tstruct fsmonitor_cookie_item *cookie;\n+\n+\t\tkey.name = cookie_names->items[k].string;\n+\t\thashmap_entry_init(&key.entry, strhash(key.name));\n+\n+\t\tcookie = hashmap_get_entry(&state->cookies, &key, entry, NULL);\n+\t\tif (cookie) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-seen: '%s'\",\n+\t\t\t\t\t cookie->name);\n+\t\t\tcookie->result = FCIR_SEEN;\n+\t\t\tnr_seen++;\n+\t\t}\n+\t}\n+\n+\tif (nr_seen)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n+/*\n+ * Set _ABORT on all pending cookies and wake up all client threads.\n+ */\n+static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct hashmap_iter iter;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tint nr_aborted = 0;\n+\n+\thashmap_for_each_entry(&state->cookies, &iter, cookie, entry) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-abort: '%s'\",\n+\t\t\t\t cookie->name);\n+\t\tcookie->result = FCIR_ABORT;\n+\t\tnr_aborted++;\n+\t}\n+\n+\tif (nr_aborted)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n /*\n * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n * \"token\" as a virtual timestamp. Clients can request a summary of all\n@@ -404,6 +560,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n *\n+ * [2] Some of those lost events may have been for cookie files. We\n+ * should assume the worst and abort them rather letting them starve.\n+ *\n * If there are no concurrent threads reading the current token data\n * series, we can free it now. Otherwise, let the last reader free\n * it.\n@@ -425,6 +584,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n \tstate->current_token_data = new_one;\n \n \tfsmonitor_free_token_data(free_me);\n+\n+\twith_lock__abort_all_cookies(state);\n }\n \n void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n@@ -500,6 +661,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tint hash_ret;\n \tint do_trivial = 0;\n \tint do_flush = 0;\n+\tint do_cookie = 0;\n+\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n@@ -560,6 +723,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * We have a V2 valid token:\n \t\t\t * \"builtin::\"\n \t\t\t */\n+\t\t\tdo_cookie = 1;\n \t\t}\n \t}\n \n@@ -568,6 +732,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tif (!state->current_token_data)\n \t\tBUG(\"fsmonitor state does not have a current token\");\n \n+\t/*\n+\t * Write a cookie file inside the directory being watched in\n+\t * an effort to flush out existing filesystem events that we\n+\t * actually care about. Suspend this client thread until we\n+\t * see the filesystem events for this cookie file.\n+\t *\n+\t * Creating the cookie lets us guarantee that our FS listener\n+\t * thread has drained the kernel queue and we are caught up\n+\t * with the kernel.\n+\t *\n+\t * If we cannot create the cookie (or otherwise guarantee that\n+\t * we are caught up), we send a trivial response. We have to\n+\t * assume that there might be some very, very recent activity\n+\t * on the FS still in flight.\n+\t */\n+\tif (do_cookie) {\n+\t\tcookie_result = with_lock__wait_for_cookie(state);\n+\t\tif (cookie_result != FCIR_SEEN) {\n+\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n+\t\t\t cookie_result);\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n \tif (do_flush)\n \t\twith_lock__do_force_resync(state);\n \n@@ -787,7 +975,9 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n-#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+#define FSMONITOR_DIR \"fsmonitor--daemon\"\n+#define FSMONITOR_COOKIE_DIR \"cookies\"\n+#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n \n enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n \tconst char *rel)\n@@ -940,6 +1130,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n+\tif (cookie_names->nr)\n+\t\twith_lock__mark_cookies_seen(state, cookie_names);\n+\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n@@ -1031,7 +1224,9 @@ static int fsmonitor_run_daemon(void)\n \n \tmemset(&state, 0, sizeof(state));\n \n+\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n+\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n@@ -1056,6 +1251,44 @@ static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n+\t/*\n+\t * We will write filesystem syncing cookie files into\n+\t * ///-.\n+\t *\n+\t * The extra layers of subdirectories here keep us from\n+\t * changing the mtime on \".git/\" or \".git/foo/\" when we create\n+\t * or delete cookie files.\n+\t *\n+\t * There have been problems with some IDEs that do a\n+\t * non-recursive watch of the \".git/\" directory and run a\n+\t * series of commands any time something happens.\n+\t *\n+\t * For example, if we place our cookie files directly in\n+\t * \".git/\" or \".git/foo/\" then a `git status` (or similar\n+\t * command) from the IDE will cause a cookie file to be\n+\t * created in one of those dirs. This causes the mtime of\n+\t * those dirs to change. This triggers the IDE's watch\n+\t * notification. This triggers the IDE to run those commands\n+\t * again. And the process repeats and the machine never goes\n+\t * idle.\n+\t *\n+\t * Adding the extra layers of subdirectories prevents the\n+\t * mtime of \".git/\" and \".git/foo\" from changing when a\n+\t * cookie file is created.\n+\t */\n+\tstrbuf_init(&state.path_cookie_prefix, 0);\n+\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1068,6 +1301,7 @@ static int fsmonitor_run_daemon(void)\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n+\tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n \n@@ -1075,6 +1309,7 @@ done:\n \n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n+\tstrbuf_release(&state.path_cookie_prefix);\n \n \treturn err;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 010fbfe60e9..bd09fffc176 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n \n \tstruct fsmonitor_token_data *current_token_data;\n \n+\tstruct strbuf path_cookie_prefix;\n+\tpthread_cond_t cookies_cond;\n+\tint cookie_seq;\n+\tstruct hashmap cookies;\n+\n \tint error_code;\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n-- \ngitgitgadget\n\n"},{"id":"452372","messageId":"a29fe7266a456d88125ca1f87d4b16370caf486f.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 08/30] compat/fsmonitor/fsm-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:51Z","receivedAt":"2022-03-25T19:39:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty filesystem listener backend for fsmonitor--daemon on Windows.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Makefile | 13 ++++++++\n compat/fsmonitor/fsm-listen-win32.c | 21 +++++++++++++\n compat/fsmonitor/fsm-listen.h | 49 +++++++++++++++++++++++++++++\n config.mak.uname | 10 ++++++\n contrib/buildsystems/CMakeLists.txt | 7 +++++\n repo-settings.c | 1 +\n 6 files changed, 101 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-listen-win32.c\n create mode 100644 compat/fsmonitor/fsm-listen.h\n\ndiff --git a/Makefile b/Makefile\nindex 5af1d5b112e..26567d4f772 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -470,6 +470,11 @@ all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n+# If your platform supports a built-in fsmonitor backend, set\n+# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n+# `compat/fsmonitor/fsm-listen-.c` that implements the\n+# `fsm_listen__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1968,6 +1973,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n \tCOMPAT_OBJS += compat/access.o\n endif\n \n+ifdef FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2887,6 +2897,9 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n \t@echo SANITIZE_LEAK=\\''$(subst ','\\'',$(subst ','\\'',$(SANITIZE_LEAK)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n+ifdef FSMONITOR_DAEMON_BACKEND\n+\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nnew file mode 100644\nindex 00000000000..916cbea254f\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-listen.h\"\n+\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nnew file mode 100644\nindex 00000000000..f0539349baf\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -0,0 +1,49 @@\n+#ifndef FSM_LISTEN_H\n+#define FSM_LISTEN_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * fsmonitor_fs_listener thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_listen__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread AFTER joining the listener.\n+ */\n+void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to watch for\n+ * filesystem events. This will run in the fsmonitor_fs_listen thread.\n+ *\n+ * It should call `ipc_server_stop_async()` if the listener thread\n+ * prematurely terminates (because of a filesystem error or if it\n+ * detects that the .git directory has been deleted). (It should NOT\n+ * do so if the listener thread receives a normal shutdown signal from\n+ * the IPC layer.)\n+ *\n+ * It should set `state->error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_listen__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the fsmonitor listener thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_LISTEN_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 4352ea39e9b..26074f56bed 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -435,6 +435,11 @@ ifeq ($(uname_S),Windows)\n \t# so we don't need this:\n \t#\n \t# SNPRINTF_RETURNS_BOGUS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -619,6 +624,11 @@ ifeq ($(uname_S),MINGW)\n \tNO_STRTOUMAX = YesPlease\n \tNO_MKDTEMP = YesPlease\n \tNO_SVN_TESTS = YesPlease\n+\n+\t# The builtin FSMonitor requires Named Pipes and Threads on Windows.\n+\t# These are always available, so we do not have to conditionally\n+\t# support it.\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex e44232f85d3..0963629db7f 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -285,6 +285,13 @@ else()\n \tendif()\n endif()\n \n+if(SUPPORTS_SIMPLE_IPC)\n+\tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\tendif()\n+endif()\n+\n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n \n #header checks\ndiff --git a/repo-settings.c b/repo-settings.c\nindex b4fbd16cdcc..2dfcb2b6542 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"midx.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n \n static void repo_cfg_bool(struct repository *r, const char *key, int *dest,\n \t\t\t int def)\n-- \ngitgitgadget\n\n"},{"id":"452374","messageId":"f07800690eeda000271b05435aa032ba17b35086.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 10/30] fsmonitor--daemon: implement 'run' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:53Z","receivedAt":"2022-03-25T19:39:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `run` command to try to begin listening for file system events.\n\nThis version defines the thread structure with a single fsmonitor_fs_listen\nthread to watch for file system events and a simple IPC thread pool to\nwatch for connection from Git clients over a well-known named pipe or\nUnix domain socket.\n\nThis commit does not actually do anything yet because the platform\nbackends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 34 ++++++\n 2 files changed, 261 insertions(+), 1 deletion(-)\n create mode 100644 fsmonitor--daemon.h\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 5e3178b8bdd..5591339399a 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,16 +3,52 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Global state loaded from config.\n+ */\n+#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n+static int fsmonitor__ipc_threads = 8;\n+\n+#define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n+static int fsmonitor__announce_startup = 0;\n+\n+static int fsmonitor_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 1)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__IPC_THREADS, i);\n+\t\tfsmonitor__ipc_threads = i;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n+\t\tint is_bool;\n+\t\tint i = git_config_bool_or_int(var, value, &is_bool);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' not bool or int: %d\"),\n+\t\t\t\t var, i);\n+\t\tfsmonitor__announce_startup = i;\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n /*\n * Acting as a CLIENT.\n *\n@@ -57,15 +93,198 @@ static int do_as_client__status(void)\n \t}\n }\n \n+static ipc_server_application_cb handle_client;\n+\n+static int handle_client(void *data,\n+\t\t\t const char *command, size_t command_len,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\t/* struct fsmonitor_daemon_state *state = data; */\n+\tint result;\n+\n+\t/*\n+\t * The Simple IPC API now supports {char*, len} arguments, but\n+\t * FSMonitor always uses proper null-terminated strings, so\n+\t * we can ignore the command_len argument. (Trust, but verify.)\n+\t */\n+\tif (command_len != strlen(command))\n+\t\tBUG(\"FSMonitor assumes text messages\");\n+\n+\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n+\n+\tresult = 0; /* TODO Do something here. */\n+\n+\ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n+\n+\treturn result;\n+}\n+\n+static void *fsm_listen__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-listen\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Watching: worktree '%s'\",\n+\t\t\t state->path_worktree_watch.buf);\n+\tif (state->nr_paths_watching > 1)\n+\t\ttrace_printf_key(&trace_fsmonitor, \"Watching: gitdir '%s'\",\n+\t\t\t\t state->path_gitdir_watch.buf);\n+\n+\tfsm_listen__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct ipc_server_opts ipc_opts = {\n+\t\t.nr_threads = fsmonitor__ipc_threads,\n+\n+\t\t/*\n+\t\t * We know that there are no other active threads yet,\n+\t\t * so we can let the IPC layer temporarily chdir() if\n+\t\t * it needs to when creating the server side of the\n+\t\t * Unix domain socket.\n+\t\t */\n+\t\t.uds_disallow_chdir = 0\n+\t};\n+\n+\t/*\n+\t * Start the IPC thread pool before the we've started the file\n+\t * system event listener thread so that we have the IPC handle\n+\t * before we need it.\n+\t */\n+\tif (ipc_server_run_async(&state->ipc_server_data,\n+\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t handle_client, state))\n+\t\treturn error_errno(\n+\t\t\t_(\"could not start IPC thread pool on '%s'\"),\n+\t\t\tfsmonitor_ipc__get_path());\n+\n+\t/*\n+\t * Start the fsmonitor listener thread to collect filesystem\n+\t * events.\n+\t */\n+\tif (pthread_create(&state->listener_thread, NULL,\n+\t\t\t fsm_listen__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\tipc_server_await(state->ipc_server_data);\n+\n+\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t}\n+\n+\t/*\n+\t * The daemon is now fully functional in background threads.\n+\t * Wait for the IPC thread pool to shutdown (whether by client\n+\t * request or from filesystem activity).\n+\t */\n+\tipc_server_await(state->ipc_server_data);\n+\n+\t/*\n+\t * The fsmonitor listener thread may have received a shutdown\n+\t * event from the IPC thread pool, but it doesn't hurt to tell\n+\t * it again. And wait for it to shutdown.\n+\t */\n+\tfsm_listen__stop_async(state);\n+\tpthread_join(state->listener_thread, NULL);\n+\n+\treturn state->error_code;\n+}\n+\n+static int fsmonitor_run_daemon(void)\n+{\n+\tstruct fsmonitor_daemon_state state;\n+\tint err;\n+\n+\tmemset(&state, 0, sizeof(state));\n+\n+\tpthread_mutex_init(&state.main_lock, NULL);\n+\tstate.error_code = 0;\n+\tstate.current_token_data = NULL;\n+\n+\t/* Prepare to (recursively) watch the directory. */\n+\tstrbuf_init(&state.path_worktree_watch, 0);\n+\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n+\tstate.nr_paths_watching = 1;\n+\n+\t/*\n+\t * We create and delete cookie files somewhere inside the .git\n+\t * directory to help us keep sync with the file system. If\n+\t * \".git\" is not a directory, then is not inside the\n+\t * cone of , so set up a second watch to watch\n+\t * the so that we get events for the cookie files.\n+\t */\n+\tstrbuf_init(&state.path_gitdir_watch, 0);\n+\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n+\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n+\tif (!is_directory(state.path_gitdir_watch.buf)) {\n+\t\tstrbuf_reset(&state.path_gitdir_watch);\n+\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n+\t\tstate.nr_paths_watching = 2;\n+\t}\n+\n+\t/*\n+\t * Confirm that we can create platform-specific resources for the\n+\t * filesystem listener before we bother starting all the threads.\n+\t */\n+\tif (fsm_listen__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize listener thread\"));\n+\t\tgoto done;\n+\t}\n+\n+\terr = fsmonitor_run_daemon_1(&state);\n+\n+done:\n+\tpthread_mutex_destroy(&state.main_lock);\n+\tfsm_listen__dtor(&state);\n+\n+\tipc_server_free(state.ipc_server_data);\n+\n+\tstrbuf_release(&state.path_worktree_watch);\n+\tstrbuf_release(&state.path_gitdir_watch);\n+\n+\treturn err;\n+}\n+\n+static int try_to_run_foreground_daemon(void)\n+{\n+\t/*\n+\t * Technically, we don't need to probe for an existing daemon\n+\t * process, since we could just call `fsmonitor_run_daemon()`\n+\t * and let it fail if the pipe/socket is busy.\n+\t *\n+\t * However, this method gives us a nicer error message for a\n+\t * common error case.\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(_(\"fsmonitor--daemon is already running '%s'\"),\n+\t\t the_repository->worktree);\n+\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"running fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n+\n+\treturn !!fsmonitor_run_daemon();\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n \n \tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"ipc-threads\",\n+\t\t\t &fsmonitor__ipc_threads,\n+\t\t\t N_(\"use ipc worker threads\")),\n \t\tOPT_END()\n \t};\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(fsmonitor_config, NULL);\n \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n@@ -73,6 +292,13 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \tsubcmd = argv[0];\n \n+\tif (fsmonitor__ipc_threads < 1)\n+\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n+\t\t fsmonitor__ipc_threads);\n+\n+\tif (!strcmp(subcmd, \"run\"))\n+\t\treturn !!try_to_run_foreground_daemon();\n+\n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\n \ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nnew file mode 100644\nindex 00000000000..3009c1a83de\n--- /dev/null\n+++ b/fsmonitor--daemon.h\n@@ -0,0 +1,34 @@\n+#ifndef FSMONITOR_DAEMON_H\n+#define FSMONITOR_DAEMON_H\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+#include \"thread-utils.h\"\n+\n+struct fsmonitor_batch;\n+struct fsmonitor_token_data;\n+\n+struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+\n+struct fsmonitor_daemon_state {\n+\tpthread_t listener_thread;\n+\tpthread_mutex_t main_lock;\n+\n+\tstruct strbuf path_worktree_watch;\n+\tstruct strbuf path_gitdir_watch;\n+\tint nr_paths_watching;\n+\n+\tstruct fsmonitor_token_data *current_token_data;\n+\n+\tint error_code;\n+\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\n+\tstruct ipc_server_data *ipc_server_data;\n+};\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"452375","messageId":"ed338777b561ca7c2bb6753b0408da7ea3a64435.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 19/30] help: include fsmonitor--daemon feature flag in version info","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:02Z","receivedAt":"2022-03-25T19:39:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the \"feature: fsmonitor--daemon\" message to the output of\n`git version --build-options`.\n\nThe builtin FSMonitor is only available on certain platforms and\neven then only when certain Makefile flags are enabled, so print\na message in the verbose version output when it is available.\n\nThis can be used by test scripts for prereq testing. Granted, tests\ncould just try `git fsmonitor--daemon status` and look for a 128 exit\ncode or grep for a \"not supported\" message on stderr, but these\nmethods are rather obscure.\n\nThe main advantage is that the feature message will automatically\nappear in bug reports and other support requests.\n\nThis concept was also used during the development of Scalar for\nsimilar reasons.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n help.c | 4 ++++\n t/test-lib.sh | 7 +++++++\n 2 files changed, 11 insertions(+)\n\ndiff --git a/help.c b/help.c\nindex 71444906ddf..9112a51e84b 100644\n--- a/help.c\n+++ b/help.c\n@@ -12,6 +12,7 @@\n #include \"refs.h\"\n #include \"parse-options.h\"\n #include \"prompt.h\"\n+#include \"fsmonitor-ipc.h\"\n \n struct category_description {\n \tuint32_t category;\n@@ -695,6 +696,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n+\n+\t\tif (fsmonitor_ipc__is_supported())\n+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n \t}\n }\n \ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e4716b0b867..5d819c1bc11 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1799,3 +1799,10 @@ test_lazy_prereq SHA1 '\n # Tests that verify the scheduler integration must set this locally\n # to avoid errors.\n GIT_TEST_MAINT_SCHEDULER=\"none:exit 1\"\n+\n+# Does this platform support `git fsmonitor--daemon`\n+#\n+test_lazy_prereq FSMONITOR_DAEMON '\n+\tgit version --build-options >output &&\n+\tgrep \"feature: fsmonitor--daemon\" output\n+'\n-- \ngitgitgadget\n\n"},{"id":"452379","messageId":"2dd021512781a25ca3b6bc0c4d035b4fd4fc0456.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 23/30] t/helper/test-chmtime: skip directories on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:03:06Z","receivedAt":"2022-03-25T19:42:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach `test-tool.exe chmtime` to ignore errors when setting the mtime\non a directory on Windows.\n\nNEEDSWORK: The Windows version of `utime()` (aka `mingw_utime()`) does\nnot properly handle directories because it uses `_wopen()`. It should\nbe converted to using `CreateFileW()` and backup semantics at a minimum.\nSince I'm already in the middle of a large patch series, I did not want\nto destabilize other callers of `utime()` right now. The problem has\nonly been observed in the t/perf/p7519 test when the test repo contains\nan empty directory on disk.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n t/helper/test-chmtime.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/t/helper/test-chmtime.c b/t/helper/test-chmtime.c\nindex 524b55ca496..dc28890a183 100644\n--- a/t/helper/test-chmtime.c\n+++ b/t/helper/test-chmtime.c\n@@ -134,6 +134,21 @@ int cmd__chmtime(int argc, const char **argv)\n \t\t}\n \n \t\tif (utb.modtime != sb.st_mtime && utime(argv[i], &utb) < 0) {\n+#ifdef GIT_WINDOWS_NATIVE\n+\t\t\tif (S_ISDIR(sb.st_mode)) {\n+\t\t\t\t/*\n+\t\t\t\t * NEEDSWORK: The Windows version of `utime()`\n+\t\t\t\t * (aka `mingw_utime()`) does not correctly\n+\t\t\t\t * handle directory arguments, since it uses\n+\t\t\t\t * `_wopen()`. Ignore it for now since this\n+\t\t\t\t * is just a test.\n+\t\t\t\t */\n+\t\t\t\tfprintf(stderr,\n+\t\t\t\t\t(\"Failed to modify time on directory %s. \"\n+\t\t\t\t\t \"Skipping\\n\"), argv[i]);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+#endif\n \t\t\tfprintf(stderr, \"Failed to modify time on %s: %s\\n\",\n \t\t\t argv[i], strerror(errno));\n \t\t\treturn 1;\n-- \ngitgitgadget\n\n"},{"id":"452381","messageId":"239558e34ffea4240de8699707cd3b404b682488.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 15/30] compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:58Z","receivedAt":"2022-03-25T19:43:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the win32 backend to register a watch on the working tree\nroot directory (recursively). Also watch the if it is\nnot inside the working tree. And to collect path change notifications\ninto batches and publish.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n compat/fsmonitor/fsm-listen-win32.c | 565 ++++++++++++++++++++++++++++\n 1 file changed, 565 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 916cbea254f..5b928ab66e5 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -2,20 +2,585 @@\n #include \"config.h\"\n #include \"fsmonitor.h\"\n #include \"fsm-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+/*\n+ * The documentation of ReadDirectoryChangesW() states that the maximum\n+ * buffer size is 64K when the monitored directory is remote.\n+ *\n+ * Larger buffers may be used when the monitored directory is local and\n+ * will help us receive events faster from the kernel and avoid dropped\n+ * events.\n+ *\n+ * So we try to use a very large buffer and silently fallback to 64K if\n+ * we get an error.\n+ */\n+#define MAX_RDCW_BUF_FALLBACK (65536)\n+#define MAX_RDCW_BUF (65536 * 8)\n+\n+struct one_watch\n+{\n+\tchar buffer[MAX_RDCW_BUF];\n+\tDWORD buf_len;\n+\tDWORD count;\n+\n+\tstruct strbuf path;\n+\tHANDLE hDir;\n+\tHANDLE hEvent;\n+\tOVERLAPPED overlapped;\n+\n+\t/*\n+\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n+\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n+\t */\n+\tBOOL is_active;\n+};\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tstruct one_watch *watch_worktree;\n+\tstruct one_watch *watch_gitdir;\n+\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hListener[3]; /* we don't own these handles */\n+#define LISTENER_SHUTDOWN 0\n+#define LISTENER_HAVE_DATA_WORKTREE 1\n+#define LISTENER_HAVE_DATA_GITDIR 2\n+\tint nr_listener_handles;\n+};\n+\n+/*\n+ * Convert the WCHAR path from the notification into UTF8 and\n+ * then normalize it.\n+ */\n+static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+\t\t\t\t struct strbuf *normalized_path)\n+{\n+\tint reserve;\n+\tint len = 0;\n+\n+\tstrbuf_reset(normalized_path);\n+\tif (!info->FileNameLength)\n+\t\tgoto normalize;\n+\n+\t/*\n+\t * Pre-reserve enough space in the UTF8 buffer for\n+\t * each Unicode WCHAR character to be mapped into a\n+\t * sequence of 2 UTF8 characters. That should let us\n+\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n+\t */\n+\treserve = info->FileNameLength + 1;\n+\tstrbuf_grow(normalized_path, reserve);\n+\n+\tfor (;;) {\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n+\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t\t\t normalized_path->buf,\n+\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n+\t\t\t\t\t NULL, NULL);\n+\t\tif (len > 0)\n+\t\t\tgoto normalize;\n+\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n+\t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n+\t\t\t GetLastError(),\n+\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n+\t\t\t info->FileName);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tstrbuf_grow(normalized_path,\n+\t\t\t strbuf_avail(normalized_path) + reserve);\n+\t}\n+\n+normalize:\n+\tstrbuf_setlen(normalized_path, len);\n+\treturn strbuf_normalize_path(normalized_path);\n+}\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+}\n+\n+static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const char *path)\n+{\n+\tstruct one_watch *watch = NULL;\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\twchar_t wpath[MAX_PATH];\n+\n+\tif (xutftowcs_path(wpath, path) < 0) {\n+\t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n+\t\treturn NULL;\n+\t}\n+\n+\thDir = CreateFileW(wpath,\n+\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n+\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n+\t\t\t NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n+\t\t GetLastError(), path);\n+\t\treturn NULL;\n+\t}\n+\n+\tCALLOC_ARRAY(watch, 1);\n+\n+\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n+\n+\tstrbuf_init(&watch->path, 0);\n+\tstrbuf_addstr(&watch->path, path);\n+\n+\twatch->hDir = hDir;\n+\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\treturn watch;\n+}\n+\n+static void destroy_watch(struct one_watch *watch)\n+{\n+\tif (!watch)\n+\t\treturn;\n+\n+\tstrbuf_release(&watch->path);\n+\tif (watch->hDir != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hDir);\n+\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hEvent);\n+\n+\tfree(watch);\n+}\n+\n+static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+\t\t\t struct one_watch *watch)\n+{\n+\tDWORD dwNotifyFilter =\n+\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n+\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n+\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n+\t\tFILE_NOTIFY_CHANGE_SIZE |\n+\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n+\t\tFILE_NOTIFY_CHANGE_CREATION;\n+\n+\tResetEvent(watch->hEvent);\n+\n+\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n+\twatch->overlapped.hEvent = watch->hEvent;\n+\n+\t/*\n+\t * Queue an async call using Overlapped IO. This returns immediately.\n+\t * Our event handle will be signalled when the real result is available.\n+\t *\n+\t * The return value here just means that we successfully queued it.\n+\t * We won't know if the Read...() actually produces data until later.\n+\t */\n+\twatch->is_active = ReadDirectoryChangesW(\n+\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n+\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n+\n+\tif (watch->is_active)\n+\t\treturn 0;\n+\n+\terror(_(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\"),\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static int recv_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD gle;\n+\n+\twatch->is_active = FALSE;\n+\n+\t/*\n+\t * The overlapped result is ready. If the Read...() was successful\n+\t * we finally receive the actual result into our buffer.\n+\t */\n+\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n+\t\t\t\tTRUE))\n+\t\treturn 0;\n+\n+\tgle = GetLastError();\n+\tif (gle == ERROR_INVALID_PARAMETER &&\n+\t /*\n+\t * The kernel throws an invalid parameter error when our\n+\t * buffer is too big and we are pointed at a remote\n+\t * directory (and possibly for other reasons). Quietly\n+\t * set it down and try again.\n+\t *\n+\t * See note about MAX_RDCW_BUF at the top.\n+\t */\n+\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n+\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n+\t\treturn -2;\n+\t}\n+\n+\t/*\n+\t * NEEDSWORK: If an external is deleted, the above\n+\t * returns an error. I'm not sure that there's anything that\n+\t * we can do here other than failing -- the /.git\n+\t * link file would be broken anyway. We might try to check\n+\t * for that and return a better error message, but I'm not\n+\t * sure it is worth it.\n+\t */\n+\n+\terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\n+\t watch->path.buf, gle);\n+\treturn -1;\n+}\n+\n+static void cancel_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD count;\n+\n+\tif (!watch || !watch->is_active)\n+\t\treturn;\n+\n+\t/*\n+\t * The calls to ReadDirectoryChangesW() and GetOverlappedResult()\n+\t * form a \"pair\" (my term) where we queue an IO and promise to\n+\t * hang around and wait for the kernel to give us the result.\n+\t *\n+\t * If for some reason after we queue the IO, we have to quit\n+\t * or otherwise not stick around for the second half, we must\n+\t * tell the kernel to abort the IO. This prevents the kernel\n+\t * from writing to our buffer and/or signalling our event\n+\t * after we free them.\n+\t *\n+\t * (Ask me how much fun it was to track that one down).\n+\t */\n+\tCancelIoEx(watch->hDir, &watch->overlapped);\n+\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n+\twatch->is_active = FALSE;\n+}\n+\n+/*\n+ * Process filesystem events that happen anywhere (recursively) under the\n+ * root directory. For a normal working directory, this includes\n+ * both version controlled files and the contents of the .git/ directory.\n+ *\n+ * If /.git is a file, then we only see events for the file\n+ * itself.\n+ */\n+static int process_worktree_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_worktree;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tconst char *p = watch->buffer;\n+\n+\t/*\n+\t * If the kernel gets more events than will fit in the kernel\n+\t * buffer associated with our RDCW handle, it drops them and\n+\t * returns a count of zero.\n+\t *\n+\t * Yes, the call returns WITHOUT error and with length zero.\n+\t * This is the documented behavior. (My testing has confirmed\n+\t * that it also sets the last error to ERROR_NOTIFY_ENUM_DIR,\n+\t * but we do not rely on that since the function did not\n+\t * return an error and it is not documented.)\n+\t *\n+\t * (The \"overflow\" case is not ambiguous with the \"no data\" case\n+\t * because we did an INFINITE wait.)\n+\t *\n+\t * This means we have a gap in coverage. Tell the daemon layer\n+\t * to resync.\n+\t */\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\t}\n+\n+\t/*\n+\t * On Windows, `info` contains an \"array\" of paths that are\n+\t * relative to the root of whichever directory handle received\n+\t * the event.\n+\t */\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\t\t/* ignore everything inside of \"/.git/\" */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* queue normal pathname */\n+\t\t\tif (!batch)\n+\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tbatch = NULL;\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free_list(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_SHUTDOWN;\n+}\n+\n+/*\n+ * Process filesystem events that happened anywhere (recursively) under the\n+ * external (such as non-primary worktrees or submodules).\n+ * We only care about cookie files that our client threads created here.\n+ *\n+ * Note that we DO NOT get filesystem events on the external \n+ * itself (it is not inside something that we are watching). In particular,\n+ * we do not get an event if the external is deleted.\n+ */\n+static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_gitdir;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *p = watch->buffer;\n+\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_GITDIR;\n+\t}\n+\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\tgoto skip_this_path;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, NULL, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_GITDIR;\n }\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tDWORD dwWait;\n+\tint result;\n+\n+\tstate->error_code = 0;\n+\n+\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tif (data->watch_gitdir &&\n+\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tfor (;;) {\n+\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n+\t\t\t\t\t\tdata->hListener,\n+\t\t\t\t\t\tFALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_worktree);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n+\t\t\tresult = recv_rdcw_watch(data->watch_gitdir);\n+\t\t\tif (result == -1) {\n+\t\t\t\t/* hard error */\n+\t\t\t\tgoto force_error_stop;\n+\t\t\t}\n+\t\t\tif (result == -2) {\n+\t\t\t\t/* retryable error */\n+\t\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\t\tgoto force_error_stop;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\t/* have data */\n+\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->error_code = -1;\n+\n+force_shutdown:\n+\t/*\n+\t * Tell the IPC thead pool to stop (which completes the await\n+\t * in the main thread (which will also signal this thread (if\n+\t * we are still alive))).\n+\t */\n+\tipc_server_stop_async(state->ipc_server_data);\n+\n+clean_shutdown:\n+\tcancel_rdcw_watch(data->watch_worktree);\n+\tcancel_rdcw_watch(data->watch_gitdir);\n }\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->watch_worktree = create_watch(state,\n+\t\t\t\t\t state->path_worktree_watch.buf);\n+\tif (!data->watch_worktree)\n+\t\tgoto failed;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->watch_gitdir = create_watch(state,\n+\t\t\t\t\t\t state->path_gitdir_watch.buf);\n+\t\tif (!data->watch_gitdir)\n+\t\t\tgoto failed;\n+\t}\n+\n+\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_listener_handles++;\n+\n+\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n+\t\tdata->watch_worktree->hEvent;\n+\tdata->nr_listener_handles++;\n+\n+\tif (data->watch_gitdir) {\n+\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n+\t\t\tdata->watch_gitdir->hEvent;\n+\t\tdata->nr_listener_handles++;\n+\t}\n+\n+\tstate->backend_data = data;\n+\treturn 0;\n+\n+failed:\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n-- \ngitgitgadget\n\n"},{"id":"452382","messageId":"6ab7db9cb769fe44db2c76e82da0b5d898538ec0.1648231393.git.gitgitgadget@gmail.com","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"[PATCH v9 05/30] fsmonitor: document builtin fsmonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-25T18:02:48Z","receivedAt":"2022-03-25T19:43:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDocument how `core.fsmonitor` can be set to a boolean to enable\nor disable the builtin FSMonitor.\n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to refer to it.\n\nCreate `git-fsmonitor--daemon` manual page and describe its features.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Junio C Hamano \n---\n Documentation/config/core.txt | 60 +++++++++++++++-----\n Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n Documentation/git-update-index.txt | 8 ++-\n 3 files changed, 126 insertions(+), 17 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..6303c36c7ed 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,54 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+\tOtherwise, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note that if you concurrently use multiple versions of Git, such\n+as one version on the command line and another version in an IDE\n+tool, that the definition of `core.fsmonitor` was extended to\n+allow boolean values in addition to hook pathnames. Git versions\n+2.35.1 and prior will not understand the boolean values and will\n+consider the \"true\" or \"false\" values as hook pathnames to be\n+invoked. Git versions 2.26 thru 2.35.1 default to hook protocol\n+V2 and will fall back to no fsmonitor (full scan). Git versions\n+prior to 2.26 default to hook protocol V1 and will silently\n+assume there were no changes to report (no scan), so status\n+commands may report incomplete results. For this reason, it is\n+best to upgrade all of your Git versions before using the built-in\n+file system monitor.\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\nnew file mode 100644\nindex 00000000000..0fedf5a4565\n--- /dev/null\n+++ b/Documentation/git-fsmonitor--daemon.txt\n@@ -0,0 +1,75 @@\n+git-fsmonitor--daemon(1)\n+========================\n+\n+NAME\n+----\n+git-fsmonitor--daemon - A Built-in File System Monitor\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git fsmonitor--daemon' start\n+'git fsmonitor--daemon' run\n+'git fsmonitor--daemon' stop\n+'git fsmonitor--daemon' status\n+\n+DESCRIPTION\n+-----------\n+\n+A daemon to watch the working directory for file and directory\n+changes using platform-specific file system notification facilities.\n+\n+This daemon communicates directly with commands like `git status`\n+using the link:technical/api-simple-ipc.html[simple IPC] interface\n+instead of the slower linkgit:githooks[5] interface.\n+\n+This daemon is built into Git so that no third-party tools are\n+required.\n+\n+OPTIONS\n+-------\n+\n+start::\n+\tStarts a daemon in the background.\n+\n+run::\n+\tRuns a daemon in the foreground.\n+\n+stop::\n+\tStops the daemon running in the current working\n+\tdirectory, if present.\n+\n+status::\n+\tExits with zero status if a daemon is watching the\n+\tcurrent working directory.\n+\n+REMARKS\n+-------\n+\n+This daemon is a long running process used to watch a single working\n+directory and maintain a list of the recently changed files and\n+directories. Performance of commands such as `git status` can be\n+increased if they just ask for a summary of changes to the working\n+directory and can avoid scanning the disk.\n+\n+When `core.fsmonitor` is set to `true` (see linkgit:git-config[1])\n+commands, such as `git status`, will ask the daemon for changes and\n+automatically start it (if necessary).\n+\n+For more information see the \"File System Monitor\" section in\n+linkgit:git-update-index[1].\n+\n+CAVEATS\n+-------\n+\n+The fsmonitor daemon does not currently know about submodules and does\n+not know to filter out file system events that happen within a\n+submodule. If fsmonitor daemon is watching a super repo and a file is\n+modified within the working directory of a submodule, it will report\n+the change (as happening against the super repo). However, the client\n+will properly ignore these extra events, so performance may be affected\n+but it will not cause an incorrect result.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..53ea48a04e2 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -509,8 +511,8 @@ looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n+linkgit:git-config[1]) than using the `--fsmonitor` option to `git\n+update-index` in each repository, especially if you want to do so\n across all repositories you use, because you can set the configuration\n variable in your `$HOME/.gitconfig` just once and have it affect all\n repositories you touch.\n-- \ngitgitgadget\n\n"},{"id":"452387","messageId":"acfa4815-dd4a-3f35-2586-fa1ec55bca3c@jeffhostetler.com","threadId":"56518","inReplyTo":"001701d8407a$d2a29dd0$77e7d970$@nexbridge.com","subject":"Re: [PATCH v9 00/30] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-25T20:12:50Z","receivedAt":"2022-03-25T20:12:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/25/22 3:02 PM, rsbecker@nexbridge.com wrote:\n> On March 25, 2022 2:03 PM, Jeff Hostetler wrote:\n[...]\n> \n> So that we can plan for next time, would you mind giving some thought to what would be required for Linux and whether it makes sense to extend this. No urgency, but I would not mind participating - assuming my $DAYJOB lets me. I have a backlog for git that I need to get done first anyway.\n> \n> Kind Regards,\n> Randall\n> \n\nA Linux backend would need to:\n\n(1) stub in compat/fsmonitor/fsm-listen-linux.c (see commits\n 8/30 and 9/30 in this series). That gives you enough for\n the builtin/fsmonitor--daemon.c to link with your new\n backend.\n\n(2) populate those 4 routines.\n\n(2a) __ctor() and __dtor() will be called from the main thread\n before and after the listener thread is created. You can\n do anything you need there to register/deregister a watch\n on the FS.\n\n See inotify() and/or fanotify(). I haven't looked at those\n routines for a while, so all I can say is google it.\n\n(2b) __stop_async() will be called by \"some\" thread to request\n that the listener thread stop listening. This is an async\n request, so just notify the listener thread and return.\n (There is an example of this for Unix in\n compat/simple-ipc/ipc-unix-sockets.c)\n\n(2c) __loop() will be called (once) by the \"listener\" thread-proc\n to process/service events from the FS until a shutdown event\n is received. This runs in the body of the \"listener\" thread.\n It should probably use poll()/select()/whatever on fd's from\n inotify()/fsnotify() *and* whatever you set up in (2b) to\n wait for a shutdown event.\n\n (There are examples of this wait-loop in the Windows and Mac\n backends, but they have *very* different FS event and wait\n models, so they might not be very helpful here.)\n\n When you get a \"batch\" of one or more paths from the FS,\n use the fsmonitor_classify_*() routines to classify or\n discard them and then use fsmonitor_publish() to publish\n newly changed paths to the other threads.\n\n The core code will handle path de-dup and all locking so\n you don't have to.\n\nIt is important that you figure out how to get recursive data\nfrom the FS. We want to watch the complete worktree. Windows\nand Mac let you register the root directory of the watch and\nautomatically give me events for anything under it. IIRC,\ninotify() only gave you a single directory and you had to\nreaddir() and recurse to get fd's to the subdirs. I haven't\nlooked to see if fanotify() solves that or not. So there may\nbe some fd juggling and tree walking required. That could/should\nall be hidden inside the __ctor() and/or __loop() routines.\n\n(3) stub in compat/fsmonitor/fsm-settings-linux.c (see the\n peers. This is needed to link.\n\n(4) fill in any platform-specific reasons why you might want\n to reject a worktree. for example, when they are remote\n (NFS/SMB might support it, but do you trust it....)\n\n(5) stub in compat/fsmonitor/fsm-health-linux.c (see\n fsm-health-*.c in part 3). That will give you enough\n to link the health thread.\n\n(6) this part is probably optional (at least for now). we\n can use this if we want to add platform-specific things\n like auto-shutdown after idle. The mac version is currently\n empty, but the Windows version needs to watch the worktree\n root separately (because the FS watch is limited to what\n is *within* watched directory root).\n\nThat should help get you get started.\nLet me know if you have questions.\n\nJeff\n\n"},{"id":"452392","messageId":"xmqqpmm98w9u.fsf@gitster.g","threadId":"56518","inReplyTo":"pull.1041.v9.git.1648231393.gitgitgadget@gmail.com","subject":"Re: [PATCH v9 00/30] Builtin FSMonitor Part 2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-25T23:09:17Z","receivedAt":"2022-03-25T23:09:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> Here is V9 of Part 2 of my builtin FSMonitor series. This version addresses\n> bash style issues in t7527 raised on V8. These changes do not require a new\n> version of Part 3.\n\nThis addresses all the \"oops\" fix-ups from part 2.5, and good base\nto build part 3 on top, right?\n\nQueued.\n\nThanks.\n"},{"id":"452398","messageId":"220326.86a6ddleos.gmgdl@evledraar.gmail.com","threadId":"56518","inReplyTo":"acfa4815-dd4a-3f35-2586-fa1ec55bca3c@jeffhostetler.com","subject":"Re: [PATCH v9 00/30] Builtin FSMonitor Part 2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-26T00:48:10Z","receivedAt":"2022-03-26T00:50:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 25 2022, Jeff Hostetler wrote:\n\n> On 3/25/22 3:02 PM, rsbecker@nexbridge.com wrote:\n>> On March 25, 2022 2:03 PM, Jeff Hostetler wrote:\n> [...]\n>> So that we can plan for next time, would you mind giving some\n>> thought to what would be required for Linux and whether it makes\n>> sense to extend this. No urgency, but I would not mind participating\n>> - assuming my $DAYJOB lets me. I have a backlog for git that I need\n>> to get done first anyway.\n>> Kind Regards,\n>> Randall\n>> \n>\n> A Linux backend would need to:\n>\n> (1) stub in compat/fsmonitor/fsm-listen-linux.c (see commits\n> 8/30 and 9/30 in this series). That gives you enough for\n> the builtin/fsmonitor--daemon.c to link with your new\n> backend.\n>\n> (2) populate those 4 routines.\n>\n> (2a) __ctor() and __dtor() will be called from the main thread\n> before and after the listener thread is created. You can\n> do anything you need there to register/deregister a watch\n> on the FS.\n>\n> See inotify() and/or fanotify(). I haven't looked at those\n> routines for a while, so all I can say is google it.\n>\n> (2b) __stop_async() will be called by \"some\" thread to request\n> that the listener thread stop listening. This is an async\n> request, so just notify the listener thread and return.\n> (There is an example of this for Unix in\n> compat/simple-ipc/ipc-unix-sockets.c)\n>\n> (2c) __loop() will be called (once) by the \"listener\" thread-proc\n> to process/service events from the FS until a shutdown event\n> is received. This runs in the body of the \"listener\" thread.\n> It should probably use poll()/select()/whatever on fd's from\n> inotify()/fsnotify() *and* whatever you set up in (2b) to\n> wait for a shutdown event.\n>\n> (There are examples of this wait-loop in the Windows and Mac\n> backends, but they have *very* different FS event and wait\n> models, so they might not be very helpful here.)\n>\n> When you get a \"batch\" of one or more paths from the FS,\n> use the fsmonitor_classify_*() routines to classify or\n> discard them and then use fsmonitor_publish() to publish\n> newly changed paths to the other threads.\n>\n> The core code will handle path de-dup and all locking so\n> you don't have to.\n>\n> It is important that you figure out how to get recursive data\n> from the FS. We want to watch the complete worktree. Windows\n> and Mac let you register the root directory of the watch and\n> automatically give me events for anything under it. IIRC,\n> inotify() only gave you a single directory and you had to\n> readdir() and recurse to get fd's to the subdirs. I haven't\n> looked to see if fanotify() solves that or not. So there may\n> be some fd juggling and tree walking required. That could/should\n> all be hidden inside the __ctor() and/or __loop() routines.\n>\n> (3) stub in compat/fsmonitor/fsm-settings-linux.c (see the\n> peers. This is needed to link.\n>\n> (4) fill in any platform-specific reasons why you might want\n> to reject a worktree. for example, when they are remote\n> (NFS/SMB might support it, but do you trust it....)\n>\n> (5) stub in compat/fsmonitor/fsm-health-linux.c (see\n> fsm-health-*.c in part 3). That will give you enough\n> to link the health thread.\n>\n> (6) this part is probably optional (at least for now). we\n> can use this if we want to add platform-specific things\n> like auto-shutdown after idle. The mac version is currently\n> empty, but the Windows version needs to watch the worktree\n> root separately (because the FS watch is limited to what\n> is *within* watched directory root).\n>\n> That should help get you get started.\n> Let me know if you have questions.\n\nWouldn't it be much simpler POC in this case to write \"watchman\nbackend\"? Then we'd both get a Linux backend, and an alternate backend\nfor the other platforms to validate their implementation.\n\nSome past references to that:\nhttps://lore.kernel.org/git/871r8c73ej.fsf@evledraar.gmail.com/ &\nhttps://lore.kernel.org/git/87h7lgfchm.fsf@evledraar.gmail.com/\n"},{"id":"452478","messageId":"b89dcf8e-d16e-c604-0bf0-3a11d1528d59@jeffhostetler.com","threadId":"56518","inReplyTo":"xmqqpmm98w9u.fsf@gitster.g","subject":"Re: [PATCH v9 00/30] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-28T15:17:04Z","receivedAt":"2022-03-28T15:17:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/25/22 7:09 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> Here is V9 of Part 2 of my builtin FSMonitor series. This version addresses\n>> bash style issues in t7527 raised on V8. These changes do not require a new\n>> version of Part 3.\n> \n> This addresses all the \"oops\" fix-ups from part 2.5, and good base\n> to build part 3 on top, right?\n> \n> Queued.\n> \n> Thanks.\n> \n\nYes, this version of part 2 includes all of the fixups from 2.5.\nAnd 2.5 should be dropped.\nAnd yes, this part 2 should be the basis for part 3.\n\nThanks again!\nJeff\n"},{"id":"452483","messageId":"a5db97af-b678-4c32-257f-68117f7d08b3@jeffhostetler.com","threadId":"56518","inReplyTo":"220326.86a6ddleos.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v9 00/30] Builtin FSMonitor Part 2","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-28T16:27:54Z","receivedAt":"2022-03-28T16:28:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/25/22 8:48 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Mar 25 2022, Jeff Hostetler wrote:\n> \n>> On 3/25/22 3:02 PM, rsbecker@nexbridge.com wrote:\n>>> On March 25, 2022 2:03 PM, Jeff Hostetler wrote:\n>> [...]\n[...]\n> \n> Wouldn't it be much simpler POC in this case to write \"watchman\n> backend\"? Then we'd both get a Linux backend, and an alternate backend\n> for the other platforms to validate their implementation.\n> \n> Some past references to that:\n> https://lore.kernel.org/git/871r8c73ej.fsf@evledraar.gmail.com/ &\n> https://lore.kernel.org/git/87h7lgfchm.fsf@evledraar.gmail.com/\n> \n\nYes, there are several ways for a client command, such as anyone\nwho calls read_index/refresh_index, to get FS change data from a\nmonitoring service.\n\nLet's go thru the options here for the sake of conversation:\n\n(option 1): Use the hook-like mechanism that Ben built in 2017\n to talk to an interlude program, shell script, perl\n script, etc. That \"script\" itself then talks to a\n long-running service/daemon, such as Watchman, to get\n the list of changes and relays them back to the client.\n\n * This \"proxy\" has to handle protocol format conversions.\n * It may also have to start the service on new repos.\n * And depends upon a third-party service being installed.\n * We are limited to supporting platforms where the third-party\n tool is supported.\n\n(option 2): Replace the hook with builtin client code to talk\n directly to the service and bypass the need for\n the proxy script/executable.\n\n * Git client code would need client-side IPC to talk to\n an established and running service. (Similar to the client\n side of Simple-IPC but probably not pkt-line based.)\n * Git client code would now need to handle any protocol\n format conversions.\n * Git client code might also have to start the service.\n * And we'd still be dependent on a third-party service being\n installed.\n * And we are still limited to supporting platforms where\n the third-party tool is supported.\n * So far we've been assuming that that third-party tool is\n \"Watchman\", but technically, you could have other such\n services available.\n * So you may need multiple implementations of option 2,\n one for each third-party tool.\n * I'm not saying that this is hard, but just yet another\n detail that would have to be encoded in the Git source\n to get this \"free\" feature.\n\n(option 3): Git implements a daemon to monitor the file system\n directly.\n\n * Git owns the protocol between client and service.\n * Git owns the backend, so no third-party tools required.\n * Git owns service startup.\n * Unfortunately, we are also responsible for building the\n backends on each platform we want to support.\n\n * In the future, we could augment the service to be more\n \"Git-aware\", such as discarding data for ignored files,\n but that is just speculation at this point.\n\n\nNow, with that context in place:\n\n[1] Nothing prohibits us from having all three options be available\n on a platform. They should all be able to coexist.\n\n[2] One of my stated goals was to reduce the dependency on\n third-party tools -- especially on platforms that don't have\n a simple package management system. The point here was to\n make it easier for enterprises to deploy Git to 1k's or 10k's\n of users (and possibly unattended build machines) and make use\n of the feature without *also* having to deploy and track updates\n to yet-another third-party tool or otherwise complicate their ES\n deployment setups. Only option 3 gets rid of the third-party\n tool requirement.\n\n[3] Option 2 is a valuable suggestion, don't get me wrong. It can/\n will/should improve performance over option 1 by eliminating an\n extra process creation and the overhead of pumping all of that\n data thru another socket-pair/process and all of the context\n switches that that requires.\n\n[4] Option 2 and option 3 could/should perform relatively equally.\n And if we wanted to deprecate the hook-like interface, doing\n an option 2 implementation would allow us to transition the\n platforms for which I don't currently have a backend.\n\n[5] However, option 2 does not eliminate the need for a third-party\n tool, so it is of limited interest to me at this time. Yes, it\n would be nice to have it for testing and perf testing purposes\n and comparisons with option 3, but if I have to budget my time,\n I would rather spend my efforts on additional backends.\n\n I consider the question of doing option 2 and a Linux backend\n as two completely independent topics -- topics that we can\n discuss and/or pursue in parallel if there is interest.\n\n[6] Randall's question was about doing option 3 and I hope that I\n provided helpful information should he or anyone else want to\n pick up that effort before I can.\n\n[7] If you want to start a parallel conversation on option 2, let's\n do that in a new top-level email thread.\n\nCheers,\nJeff\n\n"}]}