{"thread":{"id":"55420","subject":"[PATCH 17/23] fsmonitor--daemon: periodically truncate list of modified files","startedAt":"2021-04-01T17:43:25Z","lastAt":"2021-07-27T17:45:31Z","messageCount":237,"participants":["Jeff Hostetler via GitGitGadget","Johannes Schindelin via GitGitGadget","Junio C Hamano","Johannes Schindelin","Jeff Hostetler","Derrick Stolee","Eric Sunshine","Ævar Arnfjörð Bjarmason","Jeff King","Felipe Contreras","Elijah Newren","Eric Wong"],"isPatch":true,"patchVersion":1,"patchTotal":23},"messages":[{"id":"420807","messageId":"9f263e70c7245a01210ac743ce3dfda4dcc08308.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 17/23] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:59Z","receivedAt":"2021-04-01T17:43:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to periodically truncate the list of\nmodified files to save some memory.\n\nClients will ask for the set of changes relative to a token that they\nfound in the FSMN index extension in the index. (This token is like a\npoint in time, but different). Clients will then update the index to\ncontain the response token (so that subsequent commands will be\nrelative to this new token).\n\nTherefore, the daemon can gradually truncate the in-memory list of\nchanged paths as they become obsolete (older that the previous token).\nSince we may have multiple clients making concurrent requests with a\nskew of tokens and clients may be racing to the talk to the daemon,\nwe lazily truncate the list.\n\nWe introduce a 5 minute delay and truncate batches 5 minutes after\nthey are considered obsolete.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 78 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 78 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 32df392b25d3..e9a9aea59ad6 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -316,6 +316,75 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n \t\t\tbatch_src->interned_paths[k];\n }\n \n+/*\n+ * To keep the batch list from growing unbounded in response to filesystem\n+ * activity, we try to truncate old batches from the end of the list as\n+ * they become irrelevant.\n+ *\n+ * We assume that the .git/index will be updated with the most recent token\n+ * any time the index is updated. And future commands will only ask for\n+ * recent changes *since* that new token. So as tokens advance into the\n+ * future, older batch items will never be requested/needed. So we can\n+ * truncate them without loss of functionality.\n+ *\n+ * However, multiple commands may be talking to the daemon concurrently\n+ * or perform a slow command, so a little \"token skew\" is possible.\n+ * Therefore, we want this to be a little bit lazy and have a generous\n+ * delay.\n+ *\n+ * The current reader thread walked backwards in time from `token->batch_head`\n+ * back to `batch_marker` somewhere in the middle of the batch list.\n+ *\n+ * Let's walk backwards in time from that marker an arbitrary delay\n+ * and truncate the list there. Note that these timestamps are completely\n+ * artificial (based on when we pinned the batch item) and not on any\n+ * filesystem activity.\n+ */\n+#define MY_TIME_DELAY (5 * 60) /* seconds */\n+\n+static void fsmonitor_batch__truncate(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const struct fsmonitor_batch *batch_marker)\n+{\n+\t/* assert state->main_lock */\n+\n+\tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *rest;\n+\tstruct fsmonitor_batch *p;\n+\ttime_t t;\n+\n+\tif (!batch_marker)\n+\t\treturn;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"TRNC mark (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t batch_marker->batch_seq_nr,\n+\t\t\t (uint64_t)batch_marker->pinned_time);\n+\n+\tfor (batch = batch_marker; batch; batch = batch->next) {\n+\t\tif (!batch->pinned_time) /* an overflow batch */\n+\t\t\tcontinue;\n+\n+\t\tt = batch->pinned_time + MY_TIME_DELAY;\n+\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n+\t\t\tcontinue;\n+\n+\t\tgoto truncate_past_here;\n+\t}\n+\n+\treturn;\n+\n+truncate_past_here:\n+\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n+\n+\trest = ((struct fsmonitor_batch *)batch)->next;\n+\t((struct fsmonitor_batch *)batch)->next = NULL;\n+\n+\tfor (p = rest; p; p = fsmonitor_batch__free(p)) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"TRNC kill (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t\t p->batch_seq_nr, (uint64_t)p->pinned_time);\n+\t}\n+}\n+\n static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n {\n \tstruct fsmonitor_batch *p;\n@@ -647,6 +716,15 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * that work.\n \t\t\t */\n \t\t\tfsmonitor_free_token_data(token_data);\n+\t\t} else if (batch) {\n+\t\t\t/*\n+\t\t\t * This batch is the first item in the list\n+\t\t\t * that is older than the requested sequence\n+\t\t\t * number and might be considered to be\n+\t\t\t * obsolete. See if we can truncate the list\n+\t\t\t * and save some memory.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__truncate(state, batch);\n \t\t}\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"420808","messageId":"67fa7c7b8ac79b05d0773dc0affbe39b77aa4893.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 14/23] fsmonitor-fs-listen-macos: add macos header files for FSEvent","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:56Z","receivedAt":"2021-04-01T17:43:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInclude MacOS system declarations to allow us to use FSEvent and\nCoreFoundation APIs. We need GCC and clang versions because of\ncompiler and header file conflicts.\n\nWhile it is quite possible to #include Apple's CoreServices.h when\ncompiling C source code with clang, trying to build it with GCC\ncurrently fails with this error:\n\nIn file included\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n\nThe underlying reason is that GCC (rightfully) objects that an `enum`\nvalue such as `kAuthorizationExternalFormLength` is not a constant\n(because it is not, the preprocessor has no knowledge of it, only the\nactual C compiler does) and can therefore not be used to define the size\nof a C array.\n\nThis is a known problem and tracked in GCC's bug tracker:\nhttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nIn the meantime, let's not block things and go the slightly ugly route\nof declaring/defining the FSEvents constants, data structures and\nfunctions that we need, so that we can avoid above-mentioned issue.\n\nLet's do this _only_ for GCC, though, so that the CI/PR builds (which\nbuild both with clang and with GCC) can guarantee that we _are_ using\nthe correct data types.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 96 ++++++++++++++++++++\n 1 file changed, 96 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\nindex b91058d1c4f8..bec5130d9e1d 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n@@ -1,3 +1,99 @@\n+#if defined(__GNUC__)\n+/*\n+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n+ * with clang, but not with GCC as of time of writing.\n+ *\n+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details.\n+ */\n+typedef unsigned int FSEventStreamCreateFlags;\n+#define kFSEventStreamEventFlagNone 0x00000000\n+#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n+#define kFSEventStreamEventFlagUserDropped 0x00000002\n+#define kFSEventStreamEventFlagKernelDropped 0x00000004\n+#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n+#define kFSEventStreamEventFlagHistoryDone 0x00000010\n+#define kFSEventStreamEventFlagRootChanged 0x00000020\n+#define kFSEventStreamEventFlagMount 0x00000040\n+#define kFSEventStreamEventFlagUnmount 0x00000080\n+#define kFSEventStreamEventFlagItemCreated 0x00000100\n+#define kFSEventStreamEventFlagItemRemoved 0x00000200\n+#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n+#define kFSEventStreamEventFlagItemRenamed 0x00000800\n+#define kFSEventStreamEventFlagItemModified 0x00001000\n+#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n+#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n+#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n+#define kFSEventStreamEventFlagItemIsFile 0x00010000\n+#define kFSEventStreamEventFlagItemIsDir 0x00020000\n+#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n+#define kFSEventStreamEventFlagOwnEvent 0x00080000\n+#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n+#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+\n+typedef struct __FSEventStream *FSEventStreamRef;\n+typedef const FSEventStreamRef ConstFSEventStreamRef;\n+\n+typedef unsigned int CFStringEncoding;\n+#define kCFStringEncodingUTF8 0x08000100\n+\n+typedef const struct __CFString *CFStringRef;\n+typedef const struct __CFArray *CFArrayRef;\n+typedef const struct __CFRunLoop *CFRunLoopRef;\n+\n+struct FSEventStreamContext {\n+ long long version;\n+ void *cb_data, *retain, *release, *copy_description;\n+};\n+\n+typedef struct FSEventStreamContext FSEventStreamContext;\n+typedef unsigned int FSEventStreamEventFlags;\n+#define kFSEventStreamCreateFlagNoDefer 0x02\n+#define kFSEventStreamCreateFlagWatchRoot 0x04\n+#define kFSEventStreamCreateFlagFileEvents 0x10\n+\n+typedef unsigned long long FSEventStreamEventId;\n+#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL\n+\n+typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef,\n+\t\t\t\t void *context,\n+\t\t\t\t __SIZE_TYPE__ num_of_events,\n+\t\t\t\t void *event_paths,\n+\t\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t\t const FSEventStreamEventId event_ids[]);\n+typedef double CFTimeInterval;\n+FSEventStreamRef FSEventStreamCreate(void *allocator,\n+\t\t\t\t FSEventStreamCallback callback,\n+\t\t\t\t FSEventStreamContext *context,\n+\t\t\t\t CFArrayRef paths_to_watch,\n+\t\t\t\t FSEventStreamEventId since_when,\n+\t\t\t\t CFTimeInterval latency,\n+\t\t\t\t FSEventStreamCreateFlags flags);\n+CFStringRef CFStringCreateWithCString(void *allocator, const char *string,\n+\t\t\t\t CFStringEncoding encoding);\n+CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count,\n+\t\t\t void *callbacks);\n+void CFRunLoopRun(void);\n+void CFRunLoopStop(CFRunLoopRef run_loop);\n+CFRunLoopRef CFRunLoopGetCurrent(void);\n+extern CFStringRef kCFRunLoopDefaultMode;\n+void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream,\n+\t\t\t\t CFRunLoopRef run_loop,\n+\t\t\t\t CFStringRef run_loop_mode);\n+unsigned char FSEventStreamStart(FSEventStreamRef stream);\n+void FSEventStreamStop(FSEventStreamRef stream);\n+void FSEventStreamInvalidate(FSEventStreamRef stream);\n+void FSEventStreamRelease(FSEventStreamRef stream);\n+#else\n+/*\n+ * Let Apple's headers declare `isalnum()` first, before\n+ * Git's headers override it via a constant\n+ */\n+#include \n+#include \n+#include \n+#endif\n+\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-fs-listen.h\"\n-- \ngitgitgadget\n\n"},{"id":"420809","messageId":"77170e521f67ce92587d833334f6951a9f375d4d.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 06/23] fsmonitor--daemon: implement client command options","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:48Z","receivedAt":"2021-04-01T17:43:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement command options `--stop`, `--is-running`, `--query`,\n`--query-index`, and `--flush` to control and query the status of a\n`fsmonitor--daemon` server process (and implicitly start a server\nprocess if necessary).\n\nLater commits will implement the actual server and monitor\nthe file system.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 144 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 144 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 6700bac92c7d..10434bce4b64 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,18 +7,144 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon --stop\"),\n+\tN_(\"git fsmonitor--daemon --is-running\"),\n+\tN_(\"git fsmonitor--daemon --query \"),\n+\tN_(\"git fsmonitor--daemon --query-index\"),\n+\tN_(\"git fsmonitor--daemon --flush\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send an IPC query to a `git-fsmonitor--daemon` SERVER process and\n+ * ask for the changes since the given token. This will implicitly\n+ * start a daemon process if necessary. The daemon process will\n+ * persist after we exit.\n+ *\n+ * This feature is primarily used by the test suite.\n+ */\n+static int do_as_client__query_token(const char *token)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_query(token, &answer);\n+\tif (ret < 0)\n+\t\tdie(_(\"could not query fsmonitor--daemon\"));\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Read the `.git/index` to get the last token written to the FSMonitor index\n+ * extension and use that to make a query.\n+ *\n+ * This feature is primarily used by the test suite.\n+ */\n+static int do_as_client__query_from_index(void)\n+{\n+\tstruct index_state *istate = the_repository->index;\n+\n+\tsetup_git_directory();\n+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update)\n+\t\tdie(\"index file does not have fsmonitor extension\");\n+\n+\treturn do_as_client__query_token(istate->fsmonitor_last_update);\n+}\n+\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n+ * and wait for it to shutdown.\n+ */\n+static int do_as_client__send_stop(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"quit\", &answer);\n+\n+\t/* The quit command does not return any response data. */\n+\tstrbuf_release(&answer);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\twhile (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tsleep_millisec(50);\n+\ttrace2_region_leave(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n+ * and tell it to flush its cache.\n+ *\n+ * This feature is primarily used by the test suite to simulate a loss of\n+ * sync with the filesystem where we miss kernel events.\n+ */\n+static int do_as_client__send_flush(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+static int is_ipc_daemon_listening(void)\n+{\n+\treturn fsmonitor_ipc__get_state() == IPC_STATE__LISTENING;\n+}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tenum daemon_mode {\n \t\tUNDEFINED_MODE,\n+\t\tSTOP,\n+\t\tIS_RUNNING,\n+\t\tQUERY,\n+\t\tQUERY_INDEX,\n+\t\tFLUSH,\n \t} mode = UNDEFINED_MODE;\n \n \tstruct option options[] = {\n+\t\tOPT_CMDMODE(0, \"stop\", &mode, N_(\"stop the running daemon\"),\n+\t\t\t STOP),\n+\n+\t\tOPT_CMDMODE(0, \"is-running\", &mode,\n+\t\t\t N_(\"test whether the daemon is running\"),\n+\t\t\t IS_RUNNING),\n+\n+\t\tOPT_CMDMODE(0, \"query\", &mode,\n+\t\t\t N_(\"query the daemon (starting if necessary)\"),\n+\t\t\t QUERY),\n+\t\tOPT_CMDMODE(0, \"query-index\", &mode,\n+\t\t\t N_(\"query the daemon (starting if necessary) using token from index\"),\n+\t\t\t QUERY_INDEX),\n+\t\tOPT_CMDMODE(0, \"flush\", &mode, N_(\"flush cached filesystem events\"),\n+\t\t\t FLUSH),\n \t\tOPT_END()\n \t};\n \n@@ -31,6 +157,24 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n \n \tswitch (mode) {\n+\tcase STOP:\n+\t\treturn !!do_as_client__send_stop();\n+\n+\tcase IS_RUNNING:\n+\t\treturn !is_ipc_daemon_listening();\n+\n+\tcase QUERY:\n+\t\tif (argc != 1)\n+\t\t\tusage_with_options(builtin_fsmonitor__daemon_usage,\n+\t\t\t\t\t options);\n+\t\treturn !!do_as_client__query_token(argv[0]);\n+\n+\tcase QUERY_INDEX:\n+\t\treturn !!do_as_client__query_from_index();\n+\n+\tcase FLUSH:\n+\t\treturn !!do_as_client__send_flush();\n+\n \tcase UNDEFINED_MODE:\n \tdefault:\n \t\tdie(_(\"Unhandled command mode %d\"), mode);\n-- \ngitgitgadget\n\n"},{"id":"420810","messageId":"d699ad597d2c40cd2c0fe8abbf75b5386d4cf0f2.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 20/23] fsmonitor: force update index when fsmonitor token advances","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:41:02Z","receivedAt":"2021-04-01T17:43:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSet the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when the\nfsmonitor response contains a different token to ensure that the index\nis written to disk.\n\nNormally, when the fsmonitor response includes a tracked file, the\nindex is always updated. Similarly, the index might be updated when\nthe response alters the untracked-cache (when enabled). However, in\ncases where neither of those cause the index to be considered changed,\nthe fsmonitor response is wasted. And subsequent commands will\ncontinue to make requests with the same token and if there have not\nbeen any changes in the working directory, they will receive the same\nresponse.\n\nThis was observed on Windows after a large checkout. On Windows, the\nkernel emits events for the files that are changed as they are\nchanged. However, it might delay events for the containing\ndirectories until the system is more idle (or someone scans the\ndirectory (so it seems)). The first status following a checkout would\nget the list of files. The subsequent status commands would get the\nlist of directories as the events trickled out. But they would never\ncatch up because the token was not advanced because the index wasn't\nupdated.\n\nThis list of directories caused `wt_status_collect_untracked()` to\nunnecessarily spend time actually scanning them during each command.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex d7e18fc8cd47..8b544e31f29f 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -353,6 +353,16 @@ void refresh_fsmonitor(struct index_state *istate)\n \t}\n \tstrbuf_release(&query_result);\n \n+\t/*\n+\t * If the fsmonitor response and the subsequent scan of the disk\n+\t * did not cause the in-memory index to be marked dirty, then force\n+\t * it so that we advance the fsmonitor token in our extension, so\n+\t * that future requests don't keep re-requesting the same range.\n+\t */\n+\tif (istate->fsmonitor_last_update &&\n+\t strcmp(istate->fsmonitor_last_update, last_update_token.buf))\n+\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\n \t/* Now that we've updated istate, save the last_update_token */\n \tFREE_AND_NULL(istate->fsmonitor_last_update);\n \tistate->fsmonitor_last_update = strbuf_detach(&last_update_token, NULL);\n-- \ngitgitgadget\n\n"},{"id":"420811","messageId":"038b62dc6744e8d8004f3f8423a40066821c0f1b.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 19/23] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:41:01Z","receivedAt":"2021-04-01T17:43:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon client threads to create a cookie file\ninside the .git directory and then wait until FS events for the\ncookie are observed by the FS listener thread.\n\nThis helps address the racy nature of file system events by\nblocking the client response until the kernel has drained any\nevent backlog.\n\nThis is especially important on MacOS where kernel events are\nonly issued with a limited frequency. See the `latency` argument\nof `FSeventStreamCreate()`. The kernel only signals every `latency`\nseconds, but does not guarantee that the kernel queue is completely\ndrained, so we may have to wait more than one interval. If we\nincrease the frequency, the system is more likely to drop events.\nWe avoid these issues by having each client thread create a unique\ncookie file and then wait until it is seen in the event stream.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 198 ++++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 5 +\n 2 files changed, 203 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 0cb09ef0b984..d6b59a98cedd 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -150,6 +150,149 @@ static int do_as_client__send_flush(void)\n \treturn 0;\n }\n \n+enum fsmonitor_cookie_item_result {\n+\tFCIR_ERROR = -1, /* could not create cookie file ? */\n+\tFCIR_INIT = 0,\n+\tFCIR_SEEN,\n+\tFCIR_ABORT,\n+};\n+\n+struct fsmonitor_cookie_item {\n+\tstruct hashmap_entry entry;\n+\tconst char *name;\n+\tenum fsmonitor_cookie_item_result result;\n+};\n+\n+static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n+\t\t const struct hashmap_entry *he2, const void *keydata)\n+{\n+\tconst struct fsmonitor_cookie_item *a =\n+\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n+\tconst struct fsmonitor_cookie_item *b =\n+\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n+\n+\treturn strcmp(a->name, keydata ? keydata : b->name);\n+}\n+\n+static enum fsmonitor_cookie_item_result fsmonitor_wait_for_cookie(\n+\tstruct fsmonitor_daemon_state *state)\n+{\n+\tint fd;\n+\tstruct fsmonitor_cookie_item cookie;\n+\tstruct strbuf cookie_pathname = STRBUF_INIT;\n+\tstruct strbuf cookie_filename = STRBUF_INIT;\n+\tconst char *slash;\n+\tint my_cookie_seq;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tmy_cookie_seq = state->cookie_seq++;\n+\n+\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n+\tstrbuf_addf(&cookie_pathname, \"%i-%i\", getpid(), my_cookie_seq);\n+\n+\tslash = find_last_dir_sep(cookie_pathname.buf);\n+\tif (slash)\n+\t\tstrbuf_addstr(&cookie_filename, slash + 1);\n+\telse\n+\t\tstrbuf_addbuf(&cookie_filename, &cookie_pathname);\n+\tcookie.name = strbuf_detach(&cookie_filename, NULL);\n+\tcookie.result = FCIR_INIT;\n+\t// TODO should we have case-insenstive hash (and in cookie_cmp()) ??\n+\thashmap_entry_init(&cookie.entry, strhash(cookie.name));\n+\n+\t/*\n+\t * Warning: we are putting the address of a stack variable into a\n+\t * global hashmap. This feels dodgy. We must ensure that we remove\n+\t * it before this thread and stack frame returns.\n+\t */\n+\thashmap_add(&state->cookies, &cookie.entry);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n+\t\t\t cookie.name, cookie_pathname.buf);\n+\n+\t/*\n+\t * Create the cookie file on disk and then wait for a notification\n+\t * that the listener thread has seen it.\n+\t */\n+\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n+\tif (fd >= 0) {\n+\t\tclose(fd);\n+\t\tunlink_or_warn(cookie_pathname.buf);\n+\n+\t\twhile (cookie.result == FCIR_INIT)\n+\t\t\tpthread_cond_wait(&state->cookies_cond,\n+\t\t\t\t\t &state->main_lock);\n+\n+\t\thashmap_remove(&state->cookies, &cookie.entry, NULL);\n+\t} else {\n+\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n+\t\t\t cookie.name);\n+\n+\t\tcookie.result = FCIR_ERROR;\n+\t\thashmap_remove(&state->cookies, &cookie.entry, NULL);\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\tfree((char*)cookie.name);\n+\tstrbuf_release(&cookie_pathname);\n+\treturn cookie.result;\n+}\n+\n+/*\n+ * Mark these cookies as _SEEN and wake up the corresponding client threads.\n+ */\n+static void fsmonitor_cookie_mark_seen(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const struct string_list *cookie_names)\n+{\n+\t/* assert state->main_lock */\n+\n+\tint k;\n+\tint nr_seen = 0;\n+\n+\tfor (k = 0; k < cookie_names->nr; k++) {\n+\t\tstruct fsmonitor_cookie_item key;\n+\t\tstruct fsmonitor_cookie_item *cookie;\n+\n+\t\tkey.name = cookie_names->items[k].string;\n+\t\thashmap_entry_init(&key.entry, strhash(key.name));\n+\n+\t\tcookie = hashmap_get_entry(&state->cookies, &key, entry, NULL);\n+\t\tif (cookie) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-seen: '%s'\",\n+\t\t\t\t\t cookie->name);\n+\t\t\tcookie->result = FCIR_SEEN;\n+\t\t\tnr_seen++;\n+\t\t}\n+\t}\n+\n+\tif (nr_seen)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n+/*\n+ * Set _ABORT on all pending cookies and wake up all client threads.\n+ */\n+static void fsmonitor_cookie_abort_all(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert state->main_lock */\n+\n+\tstruct hashmap_iter iter;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tint nr_aborted = 0;\n+\n+\thashmap_for_each_entry(&state->cookies, &iter, cookie, entry) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-abort: '%s'\",\n+\t\t\t\t cookie->name);\n+\t\tcookie->result = FCIR_ABORT;\n+\t\tnr_aborted++;\n+\t}\n+\n+\tif (nr_aborted)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n static int lookup_client_test_delay(void)\n {\n \tstatic int delay_ms = -1;\n@@ -435,6 +578,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n *\n+ * [2] Some of those lost events may have been for cookie files. We\n+ * should assume the worst and abort them rather letting them starve.\n+ *\n * If there are no readers of the the current token data series, we\n * can free it now. Otherwise, let the last reader free it. Either\n * way, the old token data series is no longer associated with our\n@@ -454,6 +600,8 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \t\t\t state->current_token_data->token_id.buf,\n \t\t\t new_one->token_id.buf);\n \n+\tfsmonitor_cookie_abort_all(state);\n+\n \tif (state->current_token_data->client_ref_count == 0)\n \t\tfree_me = state->current_token_data;\n \tstate->current_token_data = new_one;\n@@ -526,6 +674,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tkh_str_t *shown;\n \tint hash_ret;\n \tint result;\n+\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n@@ -654,6 +803,39 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\tgoto send_trivial_response;\n \t}\n \n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * Write a cookie file inside the directory being watched in an\n+\t * effort to flush out existing filesystem events that we actually\n+\t * care about. Suspend this client thread until we see the filesystem\n+\t * events for this cookie file.\n+\t */\n+\tcookie_result = fsmonitor_wait_for_cookie(state);\n+\tif (cookie_result != FCIR_SEEN) {\n+\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n+\t\t cookie_result);\n+\t\tresult = 0;\n+\t\tgoto send_trivial_response;\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (strcmp(requested_token_id.buf,\n+\t\t state->current_token_data->token_id.buf)) {\n+\t\t/*\n+\t\t * Ack! The listener thread lost sync with the filesystem\n+\t\t * and created a new token while we were waiting for the\n+\t\t * cookie file to be created! Just give up.\n+\t\t */\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"lost filesystem sync\");\n+\t\tresult = 0;\n+\t\tgoto send_trivial_response;\n+\t}\n+\n \t/*\n \t * We're going to hold onto a pointer to the current\n \t * token-data while we walk the list of batches of files.\n@@ -982,6 +1164,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n+\tif (cookie_names->nr)\n+\t\tfsmonitor_cookie_mark_seen(state, cookie_names);\n+\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n@@ -1071,7 +1256,9 @@ static int fsmonitor_run_daemon(void)\n \n \tmemset(&state, 0, sizeof(state));\n \n+\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n+\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \tstate.test_client_delay_ms = lookup_client_test_delay();\n@@ -1094,6 +1281,15 @@ static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n+\t/*\n+\t * We will write filesystem syncing cookie files into\n+\t * /-.\n+\t */\n+\tstrbuf_init(&state.path_cookie_prefix, 0);\n+\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_PREFIX);\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1106,6 +1302,7 @@ static int fsmonitor_run_daemon(void)\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n+\tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsmonitor_fs_listen__dtor(&state);\n \n@@ -1113,6 +1310,7 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n+\tstrbuf_release(&state.path_cookie_prefix);\n \n \treturn err;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 06563b6ed56c..4e580e285ed6 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n \n \tstruct fsmonitor_token_data *current_token_data;\n \n+\tstruct strbuf path_cookie_prefix;\n+\tpthread_cond_t cookies_cond;\n+\tint cookie_seq;\n+\tstruct hashmap cookies;\n+\n \tint error_code;\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n-- \ngitgitgadget\n\n"},{"id":"420812","messageId":"3dac63eae201e6d0b949680e682238625cad59bd.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 02/23] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:44Z","receivedAt":"2021-04-01T17:44:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate client routines to spawn a fsmonitor daemon and send it an IPC\nrequest using `simple-ipc`.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 153 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 +++++++++++++++\n help.c | 4 ++\n 4 files changed, 206 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex a6a73c574191..50977911d41a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -891,6 +891,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 000000000000..b0dc334ff02d\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,153 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+#define FSMONITOR_DAEMON_IS_SUPPORTED 1\n+#else\n+#define FSMONITOR_DAEMON_IS_SUPPORTED 0\n+#endif\n+\n+/*\n+ * A trivial function so that this source file always defines at least\n+ * one symbol even when the feature is not supported. This quiets an\n+ * annoying compiler error.\n+ */\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn FSMONITOR_DAEMON_IS_SUPPORTED;\n+}\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"--start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\",\n+\t\t\t since_token);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, since_token, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\n+\t\tif (fsmonitor_is_trivial_response(answer))\n+\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: daemon not available\"));\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(\"fsmonitor--daemon is not running\");\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, command, answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\",\n+\t\t command);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 000000000000..7d21c1260151\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+/*\n+ * Returns true if a filesystem notification backend is defined\n+ * for this platform. This symbol must always be visible and\n+ * outside of the HAVE_ ifdef.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * This DOES NOT use the hook interface.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_IPC_H */\ndiff --git a/help.c b/help.c\nindex 3c3bdec21356..e22ba1d246a5 100644\n--- a/help.c\n+++ b/help.c\n@@ -11,6 +11,7 @@\n #include \"version.h\"\n #include \"refs.h\"\n #include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n \n struct category_description {\n \tuint32_t category;\n@@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n+\n+\t\tif (fsmonitor_ipc__is_supported())\n+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"420814","messageId":"7082528d8f7c1afa33e1146e3d274e044735f6a1.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 04/23] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:46Z","receivedAt":"2021-04-01T17:47:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe `core.fsmonitor` setting is supposed to be set to a path pointing\nto a script or executable that (via the Hook API) queries an fsmonitor\nprocess such as watchman.\n\nWe are about to implement our own fsmonitor backend, and do not want\nto spawn hook processes just to query it. Let's use `Simple IPC` to\ndirectly communicate with the daemon (and start it if necessary),\nguarded by the brand-new `core.useBuiltinFSMonitor` toggle.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n config.c | 5 +++++\n fsmonitor.c | 20 +++++++++++++++++---\n repo-settings.c | 3 +++\n repository.h | 2 ++\n 4 files changed, 27 insertions(+), 3 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 955ff4f9461d..31f2cbaf6dfb 100644\n--- a/config.c\n+++ b/config.c\n@@ -2515,6 +2515,11 @@ int git_config_get_max_percent_split_change(void)\n \n int repo_config_get_fsmonitor(struct repository *r)\n {\n+\tif (r->settings.use_builtin_fsmonitor > 0) {\n+\t\tcore_fsmonitor = \"(built-in daemon)\";\n+\t\treturn 1;\n+\t}\n+\n \tif (repo_config_get_pathname(r, \"core.fsmonitor\", &core_fsmonitor))\n \t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n \ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 9c9b2abc9414..d7e18fc8cd47 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,14 +149,27 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor(int version, struct index_state *istate, struct strbuf *query_result)\n {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tconst char *last_update = istate->fsmonitor_last_update;\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n \tif (!core_fsmonitor)\n \t\treturn -1;\n \n+\tif (r->settings.use_builtin_fsmonitor > 0) {\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\t\treturn fsmonitor_ipc__send_query(last_update, query_result);\n+#else\n+\t\t/* Fake a trivial response. */\n+\t\twarning(_(\"fsmonitor--daemon unavailable; falling back\"));\n+\t\tstrbuf_add(query_result, \"/\", 2);\n+\t\treturn 0;\n+#endif\n+\t}\n+\n \tstrvec_push(&cp.args, core_fsmonitor);\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n@@ -263,7 +277,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n-\t\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\t\t\tistate, &query_result);\n \n \t\t\tif (query_success) {\n \t\t\t\tif (hook_version < 0)\n@@ -293,7 +307,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n-\t\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\t\t\tistate, &query_result);\n \t\t}\n \n \t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex f7fff0f5ab83..93aab92ff164 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r)\n \t\tr->settings.core_multi_pack_index = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n \n+\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n+\t\tr->settings.use_builtin_fsmonitor = 1;\n+\n \tif (!repo_config_get_bool(r, \"feature.manyfiles\", &value) && value) {\n \t\tUPDATE_DEFAULT_BOOL(r->settings.index_version, 4);\n \t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);\ndiff --git a/repository.h b/repository.h\nindex b385ca3c94b6..7eeab871ac3e 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -41,6 +41,8 @@ struct repo_settings {\n \tenum fetch_negotiation_setting fetch_negotiation_algorithm;\n \n \tint core_multi_pack_index;\n+\n+\tint use_builtin_fsmonitor;\n };\n \n struct repository {\n-- \ngitgitgadget\n\n"},{"id":"420816","messageId":"d469d3f02e335292f4debd55ff67184b2b91b04f.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 15/23] fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:57Z","receivedAt":"2021-04-01T17:47:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement file system event listener on MacOS using FSEvent,\nCoreFoundation, and CoreServices.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 368 +++++++++++++++++++\n 1 file changed, 368 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\nindex bec5130d9e1d..e055fb579cc4 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n@@ -97,20 +97,388 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-fs-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tCFStringRef cfsr_worktree_path;\n+\tCFStringRef cfsr_gitdir_path;\n+\n+\tCFArrayRef cfar_paths_to_watch;\n+\tint nr_paths_watching;\n+\n+\tFSEventStreamRef stream;\n+\n+\tCFRunLoopRef rl;\n+\n+\tenum shutdown_style {\n+\t\tSHUTDOWN_EVENT = 0,\n+\t\tFORCE_SHUTDOWN,\n+\t\tFORCE_ERROR_STOP,\n+\t} shutdown_style;\n+\n+\tunsigned int stream_scheduled:1;\n+\tunsigned int stream_started:1;\n+};\n+\n+static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n+\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n+\tif (flag & kFSEventStreamEventFlagUserDropped)\n+\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n+\tif (flag & kFSEventStreamEventFlagKernelDropped)\n+\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n+\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n+\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n+\tif (flag & kFSEventStreamEventFlagHistoryDone)\n+\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n+\tif (flag & kFSEventStreamEventFlagRootChanged)\n+\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n+\tif (flag & kFSEventStreamEventFlagMount)\n+\t\tstrbuf_addstr(&msg, \"Mount|\");\n+\tif (flag & kFSEventStreamEventFlagUnmount)\n+\t\tstrbuf_addstr(&msg, \"Unmount|\");\n+\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n+\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n+\tif (flag & kFSEventStreamEventFlagItemCreated)\n+\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n+\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n+\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n+\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsDir)\n+\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsFile)\n+\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemModified)\n+\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n+\tif (flag & kFSEventStreamEventFlagItemRemoved)\n+\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n+\tif (flag & kFSEventStreamEventFlagItemRenamed)\n+\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n+\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n+\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n+\tif (flag & kFSEventStreamEventFlagOwnEvent)\n+\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n+\tif (flag & kFSEventStreamEventFlagItemCloned)\n+\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\t\t\t path, flag, msg.buf);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRemoved);\n+}\n+\n+static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRenamed);\n+}\n+\n+static void fsevent_callback(ConstFSEventStreamRef streamRef,\n+\t\t\t void *ctx,\n+\t\t\t size_t num_of_events,\n+\t\t\t void *event_paths,\n+\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t const FSEventStreamEventId event_ids[])\n+{\n+\tstruct fsmonitor_daemon_state *state = ctx;\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tchar **paths = (char **)event_paths;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *path_k;\n+\tconst char *slash;\n+\tint k;\n+\n+\t/*\n+\t * Build a list of all filesystem changes into a private/local\n+\t * list and without holding any locks.\n+\t */\n+\tfor (k = 0; k < num_of_events; k++) {\n+\t\t/*\n+\t\t * On Mac, we receive an array of absolute paths.\n+\t\t */\n+\t\tpath_k = paths[k];\n+\n+\t\t/*\n+\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n+\t\t * Please don't log them to Trace2.\n+\t\t *\n+\t\t * trace_printf_key(&trace_fsmonitor, \"XXX '%s'\", path_k);\n+\t\t */\n+\n+\t\t/*\n+\t\t * If event[k] is marked as dropped, we assume that we have\n+\t\t * lost sync with the filesystem and should flush our cached\n+\t\t * data. We need to:\n+\t\t *\n+\t\t * [1] Abort/wake any client threads waiting for a cookie and\n+\t\t * flush the cached state data (the current token), and\n+\t\t * create a new token.\n+\t\t *\n+\t\t * [2] Discard the batch that we were locally building (since\n+\t\t * they are conceptually relative to the just flushed\n+\t\t * token).\n+\t\t */\n+\t\tif ((event_flags[k] & kFSEventStreamEventFlagKernelDropped) ||\n+\t\t (event_flags[k] & kFSEventStreamEventFlagUserDropped)) {\n+\t\t\t/*\n+\t\t\t * see also kFSEventStreamEventFlagMustScanSubDirs\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/kernel\", \"dropped\");\n+\n+\t\t\tfsmonitor_force_resync(state);\n+\n+\t\t\tif (fsmonitor_batch__free(batch))\n+\t\t\t\tBUG(\"batch should not have a next\");\n+\t\t\tstring_list_clear(&cookie_list, 0);\n+\n+\t\t\t/*\n+\t\t\t * We assume that any events that we received\n+\t\t\t * in this callback after this dropped event\n+\t\t\t * may still be valid, so we continue rather\n+\t\t\t * than break. (And just in case there is a\n+\t\t\t * delete of \".git\" hiding in there.)\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n+\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git or gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path_k);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path_k);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\t/* ignore all other paths inside of .git or gitdir */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\tcase IS_GITDIR:\n+\t\t\t/*\n+\t\t\t * If .git directory is deleted or renamed away,\n+\t\t\t * we have to quit.\n+\t\t\t */\n+\t\t\tif (ef_is_root_delete(event_flags[k])) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/gitdir\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/gitdir\",\n+\t\t\t\t\t\t \"renamed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* try to queue normal pathnames */\n+\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\t/* fsevent could be marked as both a file and directory */\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t}\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\t\t\t\tchar *p = xstrfmt(\"%s/\", rel);\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, p);\n+\n+\t\t\t\tfree(p);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\n+\t\tcase IS_OUTSIDE_CONE:\n+\t\tdefault:\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignoring '%s'\", path_k);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\treturn;\n+\n+force_shutdown:\n+\tif (fsmonitor_batch__free(batch))\n+\t\tBUG(\"batch should not have a next\");\n+\tstring_list_clear(&cookie_list, 0);\n+\n+\tdata->shutdown_style = FORCE_SHUTDOWN;\n+\tCFRunLoopStop(data->rl);\n+\treturn;\n+}\n+\n+/*\n+ * TODO Investigate the proper value for the `latency` argument in the call\n+ * TODO to `FSEventStreamCreate()`. I'm not sure that this needs to be a\n+ * TODO config setting or just something that we tune after some testing.\n+ * TODO\n+ * TODO With a latency of 0.1, I was seeing lots of dropped events during\n+ * TODO the \"touch 100000\" files test within t/perf/p7519, but with a\n+ * TODO latency of 0.001 I did not see any dropped events. So the \"correct\"\n+ * TODO value may be somewhere in between.\n+ * TODO\n+ * TODO https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n+ */\n \n int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n+\t\tkFSEventStreamCreateFlagWatchRoot |\n+\t\tkFSEventStreamCreateFlagFileEvents;\n+\tFSEventStreamContext ctx = {\n+\t\t0,\n+\t\tstate,\n+\t\tNULL,\n+\t\tNULL,\n+\t\tNULL\n+\t};\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\tconst void *dir_array[2];\n+\n+\tdata = xcalloc(1, sizeof(*data));\n+\tstate->backend_data = data;\n+\n+\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n+\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n+\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n+\t\t\tNULL, state->path_gitdir_watch.buf,\n+\t\t\tkCFStringEncodingUTF8);\n+\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n+\t}\n+\n+\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n+\t\t\t\t\t\t data->nr_paths_watching,\n+\t\t\t\t\t\t NULL);\n+\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n+\t\t\t\t\t data->cfar_paths_to_watch,\n+\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n+\t\t\t\t\t 0.001, flags);\n+\tif (data->stream == NULL)\n+\t\tgoto failed;\n+\n+\t/*\n+\t * `data->rl` needs to be set inside the listener thread.\n+\t */\n+\n+\treturn 0;\n+\n+failed:\n+\terror(\"Unable to create FSEventStream.\");\n+\n+\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n }\n \n void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tif (data->stream) {\n+\t\tif (data->stream_started)\n+\t\t\tFSEventStreamStop(data->stream);\n+\t\tif (data->stream_scheduled)\n+\t\t\tFSEventStreamInvalidate(data->stream);\n+\t\tFSEventStreamRelease(data->stream);\n+\t}\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n \n void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\tdata->shutdown_style = SHUTDOWN_EVENT;\n+\n+\tCFRunLoopStop(data->rl);\n }\n \n void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\n+\tdata->rl = CFRunLoopGetCurrent();\n+\n+\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n+\tdata->stream_scheduled = 1;\n+\n+\tif (!FSEventStreamStart(data->stream)) {\n+\t\terror(\"Failed to start the FSEventStream\");\n+\t\tgoto force_error_stop_without_loop;\n+\t}\n+\tdata->stream_started = 1;\n+\n+\tCFRunLoopRun();\n+\n+\tswitch (data->shutdown_style) {\n+\tcase FORCE_ERROR_STOP:\n+\t\tstate->error_code = -1;\n+\t\t/* fall thru */\n+\tcase FORCE_SHUTDOWN:\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\t/* fall thru */\n+\tcase SHUTDOWN_EVENT:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn;\n+\n+force_error_stop_without_loop:\n+\tstate->error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+\treturn;\n }\n-- \ngitgitgadget\n\n"},{"id":"420817","messageId":"f1fa803ebe9c9f78608c22e55ec590f8c6775c94.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 12/23] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:54Z","receivedAt":"2021-04-01T17:48:01Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to build lists of changed paths and associate\nthem with a token-id. This will be used by the platform-specific\nbackends to accumulate changed paths in response to filesystem events.\n\nThe platform-specific event loops receive batches containing one or\nmore changed paths. Their fs listener thread will accumulate them in\na `fsmonitor_batch` (and without locking) and then \"publish\" them to\nassociate them with the current token and to make them visible to the\nclient worker threads.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 192 ++++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 40 ++++++++\n 2 files changed, 232 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 2d25e36601fe..48071d445c49 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -255,6 +255,120 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n \treturn token;\n }\n \n+struct fsmonitor_batch {\n+\tstruct fsmonitor_batch *next;\n+\tuint64_t batch_seq_nr;\n+\tconst char **interned_paths;\n+\tsize_t nr, alloc;\n+\ttime_t pinned_time;\n+};\n+\n+struct fsmonitor_batch *fsmonitor_batch__new(void)\n+{\n+\tstruct fsmonitor_batch *batch = xcalloc(1, sizeof(*batch));\n+\n+\treturn batch;\n+}\n+\n+struct fsmonitor_batch *fsmonitor_batch__free(struct fsmonitor_batch *batch)\n+{\n+\tstruct fsmonitor_batch *next;\n+\n+\tif (!batch)\n+\t\treturn NULL;\n+\n+\tnext = batch->next;\n+\n+\t/*\n+\t * The actual strings within the array are interned, so we don't\n+\t * own them.\n+\t */\n+\tfree(batch->interned_paths);\n+\n+\treturn next;\n+}\n+\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n+\t\t\t const char *path)\n+{\n+\tconst char *interned_path = strintern(path);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n+\n+\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n+\tbatch->interned_paths[batch->nr++] = interned_path;\n+}\n+\n+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n+\t\t\t\t const struct fsmonitor_batch *batch_src)\n+{\n+\t/* assert state->main_lock */\n+\n+\tsize_t k;\n+\n+\tALLOC_GROW(batch_dest->interned_paths,\n+\t\t batch_dest->nr + batch_src->nr + 1,\n+\t\t batch_dest->alloc);\n+\n+\tfor (k = 0; k < batch_src->nr; k++)\n+\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n+\t\t\tbatch_src->interned_paths[k];\n+}\n+\n+static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n+{\n+\tstruct fsmonitor_batch *p;\n+\n+\tif (!token)\n+\t\treturn;\n+\n+\tassert(token->client_ref_count == 0);\n+\n+\tstrbuf_release(&token->token_id);\n+\n+\tfor (p = token->batch_head; p; p = fsmonitor_batch__free(p))\n+\t\t;\n+\n+\tfree(token);\n+}\n+\n+/*\n+ * Flush all of our cached data about the filesystem. Call this if we\n+ * lose sync with the filesystem and miss some notification events.\n+ *\n+ * [1] If we are missing events, then we no longer have a complete\n+ * history of the directory (relative to our current start token).\n+ * We should create a new token and start fresh (as if we just\n+ * booted up).\n+ *\n+ * If there are no readers of the the current token data series, we\n+ * can free it now. Otherwise, let the last reader free it. Either\n+ * way, the old token data series is no longer associated with our\n+ * state data.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_token_data *free_me = NULL;\n+\tstruct fsmonitor_token_data *new_one = NULL;\n+\n+\tnew_one = fsmonitor_new_token_data();\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"force resync [old '%s'][new '%s']\",\n+\t\t\t state->current_token_data->token_id.buf,\n+\t\t\t new_one->token_id.buf);\n+\n+\tif (state->current_token_data->client_ref_count == 0)\n+\t\tfree_me = state->current_token_data;\n+\tstate->current_token_data = new_one;\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\tfsmonitor_free_token_data(free_me);\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data, const char *command,\n@@ -355,6 +469,77 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n \n+/*\n+ * We try to combine small batches at the front of the batch-list to avoid\n+ * having a long list. This hopefully makes it a little easier when we want\n+ * to truncate and maintain the list. However, we don't want the paths array\n+ * to just keep growing and growing with realloc, so we insert an arbitrary\n+ * limit.\n+ */\n+#define MY_COMBINE_LIMIT (1024)\n+\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names)\n+{\n+\tif (!batch && !cookie_names->nr)\n+\t\treturn;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (batch) {\n+\t\tstruct fsmonitor_batch *head;\n+\n+\t\thead = state->current_token_data->batch_head;\n+\t\tif (!head) {\n+\t\t\tbatch->batch_seq_nr = 0;\n+\t\t\tbatch->next = NULL;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t\tstate->current_token_data->batch_tail = batch;\n+\t\t} else if (head->pinned_time) {\n+\t\t\t/*\n+\t\t\t * We cannot alter the current batch list\n+\t\t\t * because:\n+\t\t\t *\n+\t\t\t * [a] it is being transmitted to at least one\n+\t\t\t * client and the handle_client() thread has a\n+\t\t\t * ref-count, but not a lock on the batch list\n+\t\t\t * starting with this item.\n+\t\t\t *\n+\t\t\t * [b] it has been transmitted in the past to\n+\t\t\t * at least one client such that future\n+\t\t\t * requests are relative to this head batch.\n+\t\t\t *\n+\t\t\t * So, we can only prepend a new batch onto\n+\t\t\t * the front of the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n+\t\t\t/*\n+\t\t\t * The head batch in the list has never been\n+\t\t\t * transmitted to a client, but folding the\n+\t\t\t * contents of the new batch onto it would\n+\t\t\t * exceed our arbitrary limit, so just prepend\n+\t\t\t * the new batch onto the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We are free to append the paths in the given\n+\t\t\t * batch onto the end of the current head batch.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__combine(head, batch);\n+\t\t\tfsmonitor_batch__free(batch);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static void *fsmonitor_fs_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -369,6 +554,13 @@ static void *fsmonitor_fs_listen__thread_proc(void *_state)\n \n \tfsmonitor_fs_listen__loop(state);\n \n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (state->current_token_data &&\n+\t state->current_token_data->client_ref_count == 0)\n+\t\tfsmonitor_free_token_data(state->current_token_data);\n+\tstate->current_token_data = NULL;\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n \ttrace2_thread_exit();\n \treturn NULL;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 97ea3766e900..06563b6ed56c 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -12,6 +12,27 @@\n struct fsmonitor_batch;\n struct fsmonitor_token_data;\n \n+/*\n+ * Create a new batch of path(s). The returned batch is considered\n+ * private and not linked into the fsmonitor daemon state. The caller\n+ * should fill this batch with one or more paths and then publish it.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__new(void);\n+\n+/*\n+ * Free this batch and return the value of the batch->next field.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__free(struct fsmonitor_batch *batch);\n+\n+/*\n+ * Add this path to this batch of modified files.\n+ *\n+ * The batch should be private and NOT (yet) linked into the fsmonitor\n+ * daemon state and therefore not yet visible to worker threads and so\n+ * no locking is required.\n+ */\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n+\n struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n \n struct fsmonitor_daemon_state {\n@@ -93,5 +114,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \tstruct fsmonitor_daemon_state *state,\n \tconst char *path);\n \n+/*\n+ * Prepend the this batch of path(s) onto the list of batches associated\n+ * with the current token. This makes the batch visible to worker threads.\n+ *\n+ * The caller no longer owns the batch and must not free it.\n+ *\n+ * Wake up the client threads waiting on these cookies.\n+ */\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names);\n+\n+/*\n+ * If the platform-specific layer loses sync with the filesystem,\n+ * it should call this to invalidate cached data and abort waiting\n+ * threads.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"420818","messageId":"2b291d805d59b54203d939e048bb568782d5e17b.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 09/23] fsmonitor--daemon: implement daemon command options","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:51Z","receivedAt":"2021-04-01T17:48:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement command options `--run` and `--start` to try to\nbegin listening for file system events.\n\nThis version defines the thread structure with a single\nfsmonitor_fs_listen thread to watch for file system events\nand a simple IPC thread pool to wait for connections from\nGit clients over a well-known named pipe or Unix domain\nsocket.\n\nThis version does not actually do anything yet because the\nbackends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 395 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 36 ++++\n 2 files changed, 430 insertions(+), 1 deletion(-)\n create mode 100644 fsmonitor--daemon.h\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 10434bce4b64..23a063707972 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,10 +3,14 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsmonitor-fs-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon --start []\"),\n+\tN_(\"git fsmonitor--daemon --run []\"),\n \tN_(\"git fsmonitor--daemon --stop\"),\n \tN_(\"git fsmonitor--daemon --is-running\"),\n \tN_(\"git fsmonitor--daemon --query \"),\n@@ -16,6 +20,38 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Global state loaded from config.\n+ */\n+#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n+static int fsmonitor__ipc_threads = 8;\n+\n+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n+static int fsmonitor__start_timeout_sec = 60;\n+\n+static int fsmonitor_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 1)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__IPC_THREADS, i);\n+\t\tfsmonitor__ipc_threads = i;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n+\t\tfsmonitor__start_timeout_sec = i;\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n /*\n * Acting as a CLIENT.\n *\n@@ -113,15 +149,350 @@ static int do_as_client__send_flush(void)\n \treturn 0;\n }\n \n+static ipc_server_application_cb handle_client;\n+\n+static int handle_client(void *data, const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\t/* struct fsmonitor_daemon_state *state = data; */\n+\tint result;\n+\n+\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n+\n+\tresult = 0; /* TODO Do something here. */\n+\n+\ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n+\n+\treturn result;\n+}\n+\n+static void *fsmonitor_fs_listen__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-listen\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Watching: worktree '%s'\",\n+\t\t\t state->path_worktree_watch.buf);\n+\tif (state->nr_paths_watching > 1)\n+\t\ttrace_printf_key(&trace_fsmonitor, \"Watching: gitdir '%s'\",\n+\t\t\t\t state->path_gitdir_watch.buf);\n+\n+\tfsmonitor_fs_listen__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct ipc_server_opts ipc_opts = {\n+\t\t.nr_threads = fsmonitor__ipc_threads,\n+\n+\t\t/*\n+\t\t * We know that there are no other active threads yet,\n+\t\t * so we can let the IPC layer temporarily chdir() if\n+\t\t * it needs to when creating the server side of the\n+\t\t * Unix domain socket.\n+\t\t */\n+\t\t.uds_disallow_chdir = 0\n+\t};\n+\n+\t/*\n+\t * Start the IPC thread pool before the we've started the file\n+\t * system event listener thread so that we have the IPC handle\n+\t * before we need it.\n+\t */\n+\tif (ipc_server_run_async(&state->ipc_server_data,\n+\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t handle_client, state))\n+\t\treturn error(_(\"could not start IPC thread pool\"));\n+\n+\t/*\n+\t * Start the fsmonitor listener thread to collect filesystem\n+\t * events.\n+\t */\n+\tif (pthread_create(&state->listener_thread, NULL,\n+\t\t\t fsmonitor_fs_listen__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\tipc_server_await(state->ipc_server_data);\n+\n+\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t}\n+\n+\t/*\n+\t * The daemon is now fully functional in background threads.\n+\t * Wait for the IPC thread pool to shutdown (whether by client\n+\t * request or from filesystem activity).\n+\t */\n+\tipc_server_await(state->ipc_server_data);\n+\n+\t/*\n+\t * The fsmonitor listener thread may have received a shutdown\n+\t * event from the IPC thread pool, but it doesn't hurt to tell\n+\t * it again. And wait for it to shutdown.\n+\t */\n+\tfsmonitor_fs_listen__stop_async(state);\n+\tpthread_join(state->listener_thread, NULL);\n+\n+\treturn state->error_code;\n+}\n+\n+static int fsmonitor_run_daemon(void)\n+{\n+\tstruct fsmonitor_daemon_state state;\n+\tint err;\n+\n+\tmemset(&state, 0, sizeof(state));\n+\n+\tpthread_mutex_init(&state.main_lock, NULL);\n+\tstate.error_code = 0;\n+\tstate.current_token_data = NULL;\n+\tstate.test_client_delay_ms = 0;\n+\n+\t/* Prepare to (recursively) watch the directory. */\n+\tstrbuf_init(&state.path_worktree_watch, 0);\n+\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n+\tstate.nr_paths_watching = 1;\n+\n+\t/*\n+\t * If \".git\" is not a directory, then is not inside the\n+\t * cone of , so set up a second watch for it.\n+\t */\n+\tstrbuf_init(&state.path_gitdir_watch, 0);\n+\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n+\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n+\tif (!is_directory(state.path_gitdir_watch.buf)) {\n+\t\tstrbuf_reset(&state.path_gitdir_watch);\n+\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n+\t\tstate.nr_paths_watching = 2;\n+\t}\n+\n+\t/*\n+\t * Confirm that we can create platform-specific resources for the\n+\t * filesystem listener before we bother starting all the threads.\n+\t */\n+\tif (fsmonitor_fs_listen__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize listener thread\"));\n+\t\tgoto done;\n+\t}\n+\n+\terr = fsmonitor_run_daemon_1(&state);\n+\n+done:\n+\tpthread_mutex_destroy(&state.main_lock);\n+\tfsmonitor_fs_listen__dtor(&state);\n+\n+\tipc_server_free(state.ipc_server_data);\n+\n+\tstrbuf_release(&state.path_worktree_watch);\n+\tstrbuf_release(&state.path_gitdir_watch);\n+\n+\treturn err;\n+}\n+\n static int is_ipc_daemon_listening(void)\n {\n \treturn fsmonitor_ipc__get_state() == IPC_STATE__LISTENING;\n }\n \n+static int try_to_run_foreground_daemon(void)\n+{\n+\t/*\n+\t * Technically, we don't need to probe for an existing daemon\n+\t * process, since we could just call `fsmonitor_run_daemon()`\n+\t * and let it fail if the pipe/socket is busy.\n+\t *\n+\t * However, this method gives us a nicer error message for a\n+\t * common error case.\n+\t */\n+\tif (is_ipc_daemon_listening())\n+\t\tdie(\"fsmonitor--daemon is already running.\");\n+\n+\treturn !!fsmonitor_run_daemon();\n+}\n+\n+#ifndef GIT_WINDOWS_NATIVE\n+/*\n+ * This is adapted from `daemonize()`. Use `fork()` to directly create\n+ * and run the daemon in a child process. The fork-parent returns the\n+ * child PID so that we can wait for the child to startup before exiting.\n+ */\n+static int spawn_background_fsmonitor_daemon(pid_t *pid)\n+{\n+\t*pid = fork();\n+\n+\tswitch (*pid) {\n+\tcase 0:\n+\t\tif (setsid() == -1)\n+\t\t\terror_errno(_(\"setsid failed\"));\n+\t\tclose(0);\n+\t\tclose(1);\n+\t\tclose(2);\n+\t\tsanitize_stdfds();\n+\n+\t\treturn !!fsmonitor_run_daemon();\n+\n+\tcase -1:\n+\t\treturn error_errno(_(\"could not spawn fsmonitor--daemon in the background\"));\n+\n+\tdefault:\n+\t\treturn 0;\n+\t}\n+}\n+#else\n+/*\n+ * Conceptually like `daemonize()` but different because Windows does not\n+ * have `fork(2)`. Spawn a normal Windows child process but without the\n+ * limitations of `start_command()` and `finish_command()`.\n+ */\n+static int spawn_background_fsmonitor_daemon(pid_t *pid)\n+{\n+\tchar git_exe[MAX_PATH];\n+\tstruct strvec args = STRVEC_INIT;\n+\tint in, out;\n+\n+\tGetModuleFileNameA(NULL, git_exe, MAX_PATH);\n+\n+\tin = open(\"/dev/null\", O_RDONLY);\n+\tout = open(\"/dev/null\", O_WRONLY);\n+\n+\tstrvec_push(&args, git_exe);\n+\tstrvec_push(&args, \"fsmonitor--daemon\");\n+\tstrvec_push(&args, \"--run\");\n+\n+\t*pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);\n+\tclose(in);\n+\tclose(out);\n+\n+\tstrvec_clear(&args);\n+\n+\tif (*pid < 0)\n+\t\treturn error(_(\"could not spawn fsmonitor--daemon in the background\"));\n+\n+\treturn 0;\n+}\n+#endif\n+\n+/*\n+ * This is adapted from `wait_or_whine()`. Watch the child process and\n+ * let it get started and begin listening for requests on the socket\n+ * before reporting our success.\n+ */\n+static int wait_for_background_startup(pid_t pid_child)\n+{\n+\tint status;\n+\tpid_t pid_seen;\n+\tenum ipc_active_state s;\n+\ttime_t time_limit, now;\n+\n+\ttime(&time_limit);\n+\ttime_limit += fsmonitor__start_timeout_sec;\n+\n+\tfor (;;) {\n+\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n+\n+\t\tif (pid_seen == -1)\n+\t\t\treturn error_errno(_(\"waitpid failed\"));\n+\n+\t\telse if (pid_seen == 0) {\n+\t\t\t/*\n+\t\t\t * The child is still running (this should be\n+\t\t\t * the normal case). Try to connect to it on\n+\t\t\t * the socket and see if it is ready for\n+\t\t\t * business.\n+\t\t\t *\n+\t\t\t * If there is another daemon already running,\n+\t\t\t * our child will fail to start (possibly\n+\t\t\t * after a timeout on the lock), but we don't\n+\t\t\t * care (who responds) if the socket is live.\n+\t\t\t */\n+\t\t\ts = fsmonitor_ipc__get_state();\n+\t\t\tif (s == IPC_STATE__LISTENING)\n+\t\t\t\treturn 0;\n+\n+\t\t\ttime(&now);\n+\t\t\tif (now > time_limit)\n+\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n+\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\telse if (pid_seen == pid_child) {\n+\t\t\t/*\n+\t\t\t * The new child daemon process shutdown while\n+\t\t\t * it was starting up, so it is not listening\n+\t\t\t * on the socket.\n+\t\t\t *\n+\t\t\t * Try to ping the socket in the odd chance\n+\t\t\t * that another daemon started (or was already\n+\t\t\t * running) while our child was starting.\n+\t\t\t *\n+\t\t\t * Again, we don't care who services the socket.\n+\t\t\t */\n+\t\t\ts = fsmonitor_ipc__get_state();\n+\t\t\tif (s == IPC_STATE__LISTENING)\n+\t\t\t\treturn 0;\n+\n+\t\t\t/*\n+\t\t\t * We don't care about the WEXITSTATUS() nor\n+\t\t\t * any of the WIF*(status) values because\n+\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n+\t\t\t * trick on all function return values.\n+\t\t\t *\n+\t\t\t * So it is sufficient to just report the\n+\t\t\t * early shutdown as an error.\n+\t\t\t */\n+\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n+\t\t}\n+\n+\t\telse\n+\t\t\treturn error(_(\"waitpid is confused\"));\n+\t}\n+}\n+\n+static int try_to_start_background_daemon(void)\n+{\n+\tpid_t pid_child;\n+\tint ret;\n+\n+\t/*\n+\t * Before we try to create a background daemon process, see\n+\t * if a daemon process is already listening. This makes it\n+\t * easier for us to report an already-listening error to the\n+\t * console, since our spawn/daemon can only report the success\n+\t * of creating the background process (and not whether it\n+\t * immediately exited).\n+\t */\n+\tif (is_ipc_daemon_listening())\n+\t\tdie(\"fsmonitor--daemon is already running.\");\n+\n+\t/*\n+\t * Run the actual daemon in a background process.\n+\t */\n+\tret = spawn_background_fsmonitor_daemon(&pid_child);\n+\tif (pid_child <= 0)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Wait (with timeout) for the background child process get\n+\t * started and begin listening on the socket/pipe. This makes\n+\t * the \"start\" command more synchronous and more reliable in\n+\t * tests.\n+\t */\n+\tret = wait_for_background_startup(pid_child);\n+\n+\treturn ret;\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tenum daemon_mode {\n \t\tUNDEFINED_MODE,\n+\t\tSTART,\n+\t\tRUN,\n \t\tSTOP,\n \t\tIS_RUNNING,\n \t\tQUERY,\n@@ -130,6 +501,11 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t} mode = UNDEFINED_MODE;\n \n \tstruct option options[] = {\n+\t\tOPT_CMDMODE(0, \"start\", &mode,\n+\t\t\t N_(\"run the daemon in the background\"),\n+\t\t\t START),\n+\t\tOPT_CMDMODE(0, \"run\", &mode,\n+\t\t\t N_(\"run the daemon in the foreground\"), RUN),\n \t\tOPT_CMDMODE(0, \"stop\", &mode, N_(\"stop the running daemon\"),\n \t\t\t STOP),\n \n@@ -145,18 +521,35 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\t\t QUERY_INDEX),\n \t\tOPT_CMDMODE(0, \"flush\", &mode, N_(\"flush cached filesystem events\"),\n \t\t\t FLUSH),\n+\n+\t\tOPT_GROUP(N_(\"Daemon options\")),\n+\t\tOPT_INTEGER(0, \"ipc-threads\",\n+\t\t\t &fsmonitor__ipc_threads,\n+\t\t\t N_(\"use ipc worker threads\")),\n+\t\tOPT_INTEGER(0, \"start-timeout\",\n+\t\t\t &fsmonitor__start_timeout_sec,\n+\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n \t\tOPT_END()\n \t};\n \n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(fsmonitor_config, NULL);\n \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\tif (fsmonitor__ipc_threads < 1)\n+\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n+\t\t fsmonitor__ipc_threads);\n \n \tswitch (mode) {\n+\tcase START:\n+\t\treturn !!try_to_start_background_daemon();\n+\n+\tcase RUN:\n+\t\treturn !!try_to_run_foreground_daemon();\n+\n \tcase STOP:\n \t\treturn !!do_as_client__send_stop();\n \ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nnew file mode 100644\nindex 000000000000..09e4a6fb6675\n--- /dev/null\n+++ b/fsmonitor--daemon.h\n@@ -0,0 +1,36 @@\n+#ifndef FSMONITOR_DAEMON_H\n+#define FSMONITOR_DAEMON_H\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+#include \"thread-utils.h\"\n+\n+struct fsmonitor_batch;\n+struct fsmonitor_token_data;\n+\n+struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+\n+struct fsmonitor_daemon_state {\n+\tpthread_t listener_thread;\n+\tpthread_mutex_t main_lock;\n+\n+\tstruct strbuf path_worktree_watch;\n+\tstruct strbuf path_gitdir_watch;\n+\tint nr_paths_watching;\n+\n+\tstruct fsmonitor_token_data *current_token_data;\n+\n+\tint error_code;\n+\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\n+\tstruct ipc_server_data *ipc_server_data;\n+\n+\tint test_client_delay_ms;\n+};\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"420819","messageId":"304fe03034f8622aa8728d8872cc9bc539bab861.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 11/23] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:53Z","receivedAt":"2021-04-01T17:48:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to create token-ids and define the\noverall token naming scheme.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 108 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 107 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 16252487240a..2d25e36601fe 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -149,6 +149,112 @@ static int do_as_client__send_flush(void)\n \treturn 0;\n }\n \n+/*\n+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n+ * \"token\" as a virtual timestamp. Clients can request a summary of all\n+ * created/deleted/modified files relative to a token. In the response,\n+ * clients receive a new token for the next (relative) request.\n+ *\n+ *\n+ * Token Format\n+ * ============\n+ *\n+ * The contents of the token are private and provider-specific.\n+ *\n+ * For the built-in fsmonitor--daemon, we define a token as follows:\n+ *\n+ * \"builtin\" \":\" \":\" \n+ *\n+ * The is an arbitrary OPAQUE string, such as a GUID,\n+ * UUID, or {timestamp,pid}. It is used to group all filesystem\n+ * events that happened while the daemon was monitoring (and in-sync\n+ * with the filesystem).\n+ *\n+ * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n+ * and does not define less-than/greater-than relationships.\n+ * (There are too many race conditions to rely on file system\n+ * event timestamps.)\n+ *\n+ * The is a simple integer incremented for each event\n+ * received. When a new is created, the is\n+ * reset to zero.\n+ *\n+ *\n+ * About Token Ids\n+ * ===============\n+ *\n+ * A new token_id is created:\n+ *\n+ * [1] each time the daemon is started.\n+ *\n+ * [2] any time that the daemon must re-sync with the filesystem\n+ * (such as when the kernel drops or we miss events on a very\n+ * active volume).\n+ *\n+ * [3] in response to a client \"flush\" command (for dropped event\n+ * testing).\n+ *\n+ * [4] MAYBE We might want to change the token_id after very complex\n+ * filesystem operations are performed, such as a directory move\n+ * sequence that affects many files within. It might be simpler\n+ * to just give up and fake a re-sync (and let the client do a\n+ * full scan) than try to enumerate the effects of such a change.\n+ *\n+ * When a new token_id is created, the daemon is free to discard all\n+ * cached filesystem events associated with any previous token_ids.\n+ * Events associated with a non-current token_id will never be sent\n+ * to a client. A token_id change implicitly means that the daemon\n+ * has gap in its event history.\n+ *\n+ * Therefore, clients that present a token with a stale (non-current)\n+ * token_id will always be given a trivial response.\n+ */\n+struct fsmonitor_token_data {\n+\tstruct strbuf token_id;\n+\tstruct fsmonitor_batch *batch_head;\n+\tstruct fsmonitor_batch *batch_tail;\n+\tuint64_t client_ref_count;\n+};\n+\n+static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n+{\n+\tstatic int test_env_value = -1;\n+\tstatic uint64_t flush_count = 0;\n+\tstruct fsmonitor_token_data *token;\n+\n+\ttoken = (struct fsmonitor_token_data *)xcalloc(1, sizeof(*token));\n+\n+\tstrbuf_init(&token->token_id, 0);\n+\ttoken->batch_head = NULL;\n+\ttoken->batch_tail = NULL;\n+\ttoken->client_ref_count = 0;\n+\n+\tif (test_env_value < 0)\n+\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n+\n+\tif (!test_env_value) {\n+\t\tstruct timeval tv;\n+\t\tstruct tm tm;\n+\t\ttime_t secs;\n+\n+\t\tgettimeofday(&tv, NULL);\n+\t\tsecs = tv.tv_sec;\n+\t\tgmtime_r(&secs, &tm);\n+\n+\t\tstrbuf_addf(&token->token_id,\n+\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n+\t\t\t flush_count++,\n+\t\t\t getpid(),\n+\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n+\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n+\t\t\t (long)tv.tv_usec);\n+\t} else {\n+\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n+\t}\n+\n+\treturn token;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data, const char *command,\n@@ -330,7 +436,7 @@ static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n-\tstate.current_token_data = NULL;\n+\tstate.current_token_data = fsmonitor_new_token_data();\n \tstate.test_client_delay_ms = 0;\n \n \t/* Prepare to (recursively) watch the directory. */\n-- \ngitgitgadget\n\n"},{"id":"420822","messageId":"3eafd0b5cb09dbba8af2abf1a8d602ad27ad3bc4.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 23/23] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:41:05Z","receivedAt":"2021-04-01T17:53:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate 2x2 test matrix with the untracked-cache and fsmonitor--daemon\nfeatures and a series of edits and verify that status output is\nidentical.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 97 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 97 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1fd230f1d4c6..ad2188169db7 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -163,6 +163,8 @@ test_expect_success 'setup' '\n \t.gitignore\n \texpect*\n \tactual*\n+\tflush*\n+\ttrace*\n \tEOF\n \n \tgit -c core.useBuiltinFSMonitor= add . &&\n@@ -482,4 +484,99 @@ test_expect_success 'worktree with .git file' '\n \ttest_must_fail git -C wt-secondary fsmonitor--daemon --is-running\n '\n \n+# TODO Repeat one of the \"edit\" tests on wt-secondary and confirm that\n+# TODO we get the same events and behavior -- that is, that fsmonitor--daemon\n+# TODO correctly listens to events on both the working directory and to the\n+# TODO referenced GITDIR.\n+\n+test_expect_success 'cleanup worktrees' '\n+\tkill_repo wt-secondary &&\n+\tkill_repo wt-base\n+'\n+\n+# The next few tests perform arbitrary/contrived file operations and\n+# confirm that status is correct. That is, that the data (or lack of\n+# data) from fsmonitor doesn't cause incorrect results. And doesn't\n+# cause incorrect results when the untracked-cache is enabled.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n+\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n+\tgit update-index --no-fsmonitor &&\n+\ttest_might_fail git fsmonitor--daemon --stop\n+'\n+\n+matrix_clean_up_repo () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+}\n+\n+matrix_try () {\n+\tuc=$1\n+\tfsm=$2\n+\tfn=$3\n+\n+\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n+\t\tmatrix_clean_up_repo &&\n+\t\t$fn &&\n+\t\tif test $uc = false -a $fsm = false\n+\t\tthen\n+\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n+\t\telse\n+\t\t\tgit status --porcelain=v1 >.git/actual.$fn\n+\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n+\t\tfi\n+\t'\n+\n+\treturn $?\n+}\n+\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+for uc_val in $uc_values\n+do\n+\tif test $uc_val = false\n+\tthen\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n+\t\t\tgit config core.untrackedcache false &&\n+\t\t\tgit update-index --no-untracked-cache\n+\t\t'\n+\telse\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n+\t\t\tgit config core.untrackedcache true &&\n+\t\t\tgit update-index --untracked-cache\n+\t\t'\n+\tfi\n+\n+\tfsm_values=\"false true\"\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tif test $fsm_val = false\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n+\t\t\t\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon --stop 2>/dev/null\n+\t\t\t'\n+\t\telse\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n+\t\t\t\tgit config core.useBuiltinFSMonitor true &&\n+\t\t\t\tgit fsmonitor--daemon --start &&\n+\t\t\t\tgit update-index --fsmonitor\n+\t\t\t'\n+\t\tfi\n+\n+\t\tmatrix_try $uc_val $fsm_val edit_files\n+\t\tmatrix_try $uc_val $fsm_val delete_files\n+\t\tmatrix_try $uc_val $fsm_val create_files\n+\t\tmatrix_try $uc_val $fsm_val rename_files\n+\t\tmatrix_try $uc_val $fsm_val file_to_directory\n+\t\tmatrix_try $uc_val $fsm_val directory_to_file\n+\tdone\n+done\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"420823","messageId":"pull.923.git.1617291666.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":null,"subject":"[PATCH 00/23] [RFC] Builtin FSMonitor Feature","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:42Z","receivedAt":"2021-04-01T17:53:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"This patch series adds a builtin FSMonitor daemon to Git.\n\nThis daemon uses platform-specific filesystem notifications to keep track of\nchanges to a working directory. It also listens over the \"Simple IPC\"\nfacility for client requests and responds with a list of files/directories\nthat have been recently modified.\n\nClient commands, such as git status, already know how to request a list of\nmodified files via the FSMonitor Hook. This patch series teaches client\ncommands to talk directly to the daemon via IPC and avoid the overhead of\nthe hook API. (Hook process creation can be expensive on Windows.)\n\nSince the daemon is a feature of Git, rather than a generic third-party tool\nlike Watchman, the daemon can format its response to be exactly what the\nclient needs, so there is no need for a hook process to proxy and reformat\nthe data. For example, when Watchman is used, Watchman responds in JSON and\nthe hook process (typically a PERL script) must parse it and convert it into\na simple NUL-delimited list. FSMonitor daemon responses are already in this\nNUL-delimited format, so no processing is required.\n\nThe current daemon implementation is rather simple in that it just records\nthe set of files/directories that have changed. For example, it is not aware\nof specific Git features, such as .gitignore and doesn't attempt to filter\nout ignored files. Having a Git-specific daemon lets us explore such things\nin the future.\n\nFinally, having a builtin daemon eliminates the need for user to download\nand install a third-party tool. This makes enterprise deployments simpler\nsince there are fewer parts to install, maintain, and updates to track.\n\nThis RFC version includes support for Windows and MacOS file system events.\nA Linux version will be submitted in a later patch series.\n\nThis patch series is being previewed as an experimental feature in Git for\nWindows v2.31.0.windows.1.\n\nThis patch series requires the jh/simple-ipc and jh/fsmonitor-prework patch\nseries.\n\nJeff Hostetler (21):\n fsmonitor--daemon: man page and documentation\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n fsmonitor--daemon: add a built-in fsmonitor daemon\n fsmonitor--daemon: implement client command options\n fsmonitor-fs-listen-win32: stub in backend for Windows\n fsmonitor-fs-listen-macos: stub in backend for MacOS\n fsmonitor--daemon: implement daemon command options\n fsmonitor--daemon: add pathname classification\n fsmonitor--daemon: define token-ids\n fsmonitor--daemon: create token-based changed path cache\n fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows\n fsmonitor-fs-listen-macos: add macos header files for FSEvent\n fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS\n fsmonitor--daemon: implement handle_client callback\n fsmonitor--daemon: periodically truncate list of modified files\n fsmonitor--daemon:: introduce client delay for testing\n fsmonitor--daemon: use a cookie file to sync with file system\n fsmonitor: force update index when fsmonitor token advances\n t7527: create test for fsmonitor--daemon\n p7519: add fsmonitor--daemon\n t7527: test status with untracked-cache and fsmonitor--daemon\n\nJohannes Schindelin (2):\n config: FSMonitor is repository-specific\n fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via\n IPC\n\n .gitignore | 1 +\n Documentation/config/core.txt | 45 +-\n Documentation/git-fsmonitor--daemon.txt | 104 ++\n Documentation/git-update-index.txt | 4 +-\n Documentation/githooks.txt | 3 +-\n Makefile | 15 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 1611 ++++++++++++++++++\n builtin/update-index.c | 4 +-\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 484 ++++++\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 514 ++++++\n compat/fsmonitor/fsmonitor-fs-listen.h | 49 +\n config.c | 9 +-\n config.h | 2 +-\n config.mak.uname | 4 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 142 ++\n fsmonitor-ipc.c | 153 ++\n fsmonitor-ipc.h | 48 +\n fsmonitor.c | 32 +-\n git.c | 1 +\n help.c | 4 +\n repo-settings.c | 3 +\n repository.h | 2 +\n t/perf/p7519-fsmonitor.sh | 37 +-\n t/t7527-builtin-fsmonitor.sh | 582 +++++++\n 26 files changed, 3839 insertions(+), 23 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n create mode 100644 builtin/fsmonitor--daemon.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n create mode 100644 fsmonitor--daemon.h\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\n\nbase-commit: f1725819714fbcd96c47ae5f14e00cc01045272f\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-923%2Fjeffhostetler%2Fbuiltin-fsmonitor-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-923/jeffhostetler/builtin-fsmonitor-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/923\n-- \ngitgitgadget\n"},{"id":"420824","messageId":"8b2280e5c4d2cec1fe39e90bfc93f059a1d0eb05.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 21/23] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:41:03Z","receivedAt":"2021-04-01T17:53:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 485 +++++++++++++++++++++++++++++++++++\n 1 file changed, 485 insertions(+)\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 000000000000..1fd230f1d4c6\n--- /dev/null\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -0,0 +1,485 @@\n+#!/bin/sh\n+\n+test_description='built-in file system watcher'\n+\n+. ./test-lib.sh\n+\n+# Ask the fsmonitor daemon to insert a little delay before responding to\n+# client commands like `git status` and `git fsmonitor--daemon --query` to\n+# allow recent filesystem events to be received by the daemon. This helps\n+# the CI/PR builds be more stable.\n+#\n+# An arbitrary millisecond value.\n+#\n+GIT_TEST_FSMONITOR_CLIENT_DELAY=1000\n+export GIT_TEST_FSMONITOR_CLIENT_DELAY\n+\n+git version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\" || {\n+\tskip_all=\"The built-in FSMonitor is not supported on this platform\"\n+\ttest_done\n+}\n+\n+kill_repo () {\n+\tr=$1\n+\tgit -C $r fsmonitor--daemon --stop >/dev/null 2>/dev/null\n+\trm -rf $1\n+\treturn 0\n+}\n+\n+start_daemon () {\n+\tcase \"$#\" in\n+\t\t1) r=\"-C $1\";;\n+\t\t*) r=\"\";\n+\tesac\n+\n+\tgit $r fsmonitor--daemon --start || return $?\n+\tgit $r fsmonitor--daemon --is-running || return $?\n+\n+\treturn 0\n+}\n+\n+test_expect_success 'explicit daemon start and stop' '\n+\ttest_when_finished \"kill_repo test_explicit\" &&\n+\n+\tgit init test_explicit &&\n+\tstart_daemon test_explicit &&\n+\n+\tgit -C test_explicit fsmonitor--daemon --stop &&\n+\ttest_must_fail git -C test_explicit fsmonitor--daemon --is-running\n+'\n+\n+test_expect_success 'implicit daemon start' '\n+\ttest_when_finished \"kill_repo test_implicit\" &&\n+\n+\tgit init test_implicit &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon --is-running &&\n+\n+\t# query will implicitly start the daemon.\n+\t#\n+\t# for test-script simplicity, we send a V1 timestamp rather than\n+\t# a V2 token. either way, the daemon response to any query contains\n+\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n+\t# but this test case is only concerned with whether the daemon was\n+\t# implicitly started.)\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n+\t\tgit -C test_implicit fsmonitor--daemon --query 0 >actual &&\n+\tnul_to_q actual.filtered &&\n+\tgrep \"builtin:\" actual.filtered &&\n+\n+\t# confirm that a daemon was started in the background.\n+\t#\n+\t# since the mechanism for starting the background daemon is platform\n+\t# dependent, just confirm that the foreground command received a\n+\t# response from the daemon.\n+\n+\tgrep :\\\"query/response-length\\\" .git/trace &&\n+\n+\tgit -C test_implicit fsmonitor--daemon --is-running &&\n+\tgit -C test_implicit fsmonitor--daemon --stop &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon --is-running\n+'\n+\n+test_expect_success 'implicit daemon stop (delete .git)' '\n+\ttest_when_finished \"kill_repo test_implicit_1\" &&\n+\n+\tgit init test_implicit_1 &&\n+\n+\tstart_daemon test_implicit_1 &&\n+\n+\t# deleting the .git directory will implicitly stop the daemon.\n+\trm -rf test_implicit_1/.git &&\n+\n+\t# Create an empty .git directory so that the following Git command\n+\t# will stay relative to the `-C` directory. Without this, the Git\n+\t# command will (override the requested -C argument) and crawl out\n+\t# to the containing Git source tree. This would make the test\n+\t# result dependent upon whether we were using fsmonitor on our\n+\t# development worktree.\n+\n+\tsleep 1 &&\n+\tmkdir test_implicit_1/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon --is-running\n+'\n+\n+test_expect_success 'implicit daemon stop (rename .git)' '\n+\ttest_when_finished \"kill_repo test_implicit_2\" &&\n+\n+\tgit init test_implicit_2 &&\n+\n+\tstart_daemon test_implicit_2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n+\n+\t# Create an empty .git directory so that the following Git command\n+\t# will stay relative to the `-C` directory. Without this, the Git\n+\t# command will (override the requested -C argument) and crawl out\n+\t# to the containing Git source tree. This would make the test\n+\t# result dependent upon whether we were using fsmonitor on our\n+\t# development worktree.\n+\n+\tsleep 1 &&\n+\tmkdir test_implicit_2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon --is-running\n+'\n+\n+test_expect_success 'cannot start multiple daemons' '\n+\ttest_when_finished \"kill_repo test_multiple\" &&\n+\n+\tgit init test_multiple &&\n+\n+\tstart_daemon test_multiple &&\n+\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon --start 2>actual &&\n+\tgrep \"fsmonitor--daemon is already running\" actual &&\n+\n+\tgit -C test_multiple fsmonitor--daemon --stop &&\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon --is-running\n+'\n+\n+test_expect_success 'setup' '\n+\t>tracked &&\n+\t>modified &&\n+\t>delete &&\n+\t>rename &&\n+\tmkdir dir1 &&\n+\t>dir1/tracked &&\n+\t>dir1/modified &&\n+\t>dir1/delete &&\n+\t>dir1/rename &&\n+\tmkdir dir2 &&\n+\t>dir2/tracked &&\n+\t>dir2/modified &&\n+\t>dir2/delete &&\n+\t>dir2/rename &&\n+\tmkdir dirtorename &&\n+\t>dirtorename/a &&\n+\t>dirtorename/b &&\n+\n+\tcat >.gitignore <<-\\EOF &&\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tEOF\n+\n+\tgit -c core.useBuiltinFSMonitor= add . &&\n+\ttest_tick &&\n+\tgit -c core.useBuiltinFSMonitor= commit -m initial &&\n+\n+\tgit config core.useBuiltinFSMonitor true\n+'\n+\n+test_expect_success 'update-index implicitly starts daemon' '\n+\ttest_must_fail git fsmonitor--daemon --is-running &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n+\t\tgit update-index --fsmonitor &&\n+\n+\tgit fsmonitor--daemon --is-running &&\n+\ttest_might_fail git fsmonitor--daemon --stop &&\n+\n+\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_1\n+'\n+\n+test_expect_success 'status implicitly starts daemon' '\n+\ttest_must_fail git fsmonitor--daemon --is-running &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n+\t\tgit status >actual &&\n+\n+\tgit fsmonitor--daemon --is-running &&\n+\ttest_might_fail git fsmonitor--daemon --stop &&\n+\n+\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_2\n+'\n+\n+edit_files() {\n+\techo 1 >modified\n+\techo 2 >dir1/modified\n+\techo 3 >dir2/modified\n+\t>dir1/untracked\n+}\n+\n+delete_files() {\n+\trm -f delete\n+\trm -f dir1/delete\n+\trm -f dir2/delete\n+}\n+\n+create_files() {\n+\techo 1 >new\n+\techo 2 >dir1/new\n+\techo 3 >dir2/new\n+}\n+\n+rename_files() {\n+\tmv rename renamed\n+\tmv dir1/rename dir1/renamed\n+\tmv dir2/rename dir2/renamed\n+}\n+\n+file_to_directory() {\n+\trm -f delete\n+\tmkdir delete\n+\techo 1 >delete/new\n+}\n+\n+directory_to_file() {\n+\trm -rf dir1\n+\techo 1 >dir1\n+}\n+\n+verify_status() {\n+\tgit status >actual &&\n+\tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n+\tGIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect &&\n+\ttest_cmp expect actual &&\n+\techo HELLO AFTER &&\n+\tcat .git/trace &&\n+\techo HELLO AFTER\n+}\n+\n+# The next few test cases confirm that our fsmonitor daemon sees each type\n+# of OS filesystem notification that we care about. At this layer we just\n+# ensure we are getting the OS notifications and do not try to confirm what\n+# is reported by `git status`.\n+#\n+# We run a simple query after modifying the filesystem just to introduce\n+# a bit of a delay so that the trace logging from the daemon has time to\n+# get flushed to disk.\n+#\n+# We `reset` and `clean` at the bottom of each test (and before stopping the\n+# daemon) because these commands might implicitly restart the daemon.\n+\n+clean_up_repo_and_stop_daemon () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+\tgit fsmonitor--daemon --stop\n+\trm -f .git/trace\n+}\n+\n+test_expect_success 'edit some files' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tedit_files &&\n+\n+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/modified$\" .git/trace &&\n+\tgrep \"^event: dir2/modified$\" .git/trace &&\n+\tgrep \"^event: modified$\" .git/trace &&\n+\tgrep \"^event: dir1/untracked$\" .git/trace\n+'\n+\n+test_expect_success 'create some files' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tcreate_files &&\n+\n+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/new$\" .git/trace &&\n+\tgrep \"^event: dir2/new$\" .git/trace &&\n+\tgrep \"^event: new$\" .git/trace\n+'\n+\n+test_expect_success 'delete some files' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdelete_files &&\n+\n+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/delete$\" .git/trace &&\n+\tgrep \"^event: dir2/delete$\" .git/trace &&\n+\tgrep \"^event: delete$\" .git/trace\n+'\n+\n+test_expect_success 'rename some files' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\trename_files &&\n+\n+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/rename$\" .git/trace &&\n+\tgrep \"^event: dir2/rename$\" .git/trace &&\n+\tgrep \"^event: rename$\" .git/trace &&\n+\tgrep \"^event: dir1/renamed$\" .git/trace &&\n+\tgrep \"^event: dir2/renamed$\" .git/trace &&\n+\tgrep \"^event: renamed$\" .git/trace\n+'\n+\n+test_expect_success 'rename directory' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tmv dirtorename dirrenamed &&\n+\n+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dirtorename/*$\" .git/trace &&\n+\tgrep \"^event: dirrenamed/*$\" .git/trace\n+'\n+\n+test_expect_success 'file changes to directory' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tfile_to_directory &&\n+\n+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: delete$\" .git/trace &&\n+\tgrep \"^event: delete/new$\" .git/trace\n+'\n+\n+test_expect_success 'directory changes to a file' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdirectory_to_file &&\n+\n+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1$\" .git/trace\n+'\n+\n+# The next few test cases exercise the token-resync code. When filesystem\n+# drops events (because of filesystem velocity or because the daemon isn't\n+# polling fast enough), we need to discard the cached data (relative to the\n+# current token) and start collecting events under a new token.\n+#\n+# the 'git fsmonitor--daemon --flush' command can be used to send a \"flush\"\n+# message to a running daemon and ask it to do a flush/resync.\n+\n+test_expect_success 'flush cached data' '\n+\ttest_when_finished \"kill_repo test_flush\" &&\n+\n+\tgit init test_flush &&\n+\n+\t(\n+\t\tGIT_TEST_FSMONITOR_TOKEN=true &&\n+\t\texport GIT_TEST_FSMONITOR_TOKEN &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace_daemon\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon test_flush\n+\t) &&\n+\n+\t# The daemon should have an initial token with no events in _0 and\n+\t# then a few (probably platform-specific number of) events in _1.\n+\t# These should both have the same .\n+\n+\tgit -C test_flush fsmonitor--daemon --query \"builtin:test_00000001:0\" >actual_0 &&\n+\tnul_to_q actual_q0 &&\n+\n+\ttouch test_flush/file_1 &&\n+\ttouch test_flush/file_2 &&\n+\n+\tgit -C test_flush fsmonitor--daemon --query \"builtin:test_00000001:0\" >actual_1 &&\n+\tnul_to_q actual_q1 &&\n+\n+\tgrep \"file_1\" actual_q1 &&\n+\n+\t# Force a flush. This will change the , reset the , and\n+\t# flush the file data. Then create some events and ensure that the file\n+\t# again appears in the cache. It should have the new .\n+\n+\tgit -C test_flush fsmonitor--daemon --flush >flush_0 &&\n+\tnul_to_q flush_q0 &&\n+\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n+\n+\tgit -C test_flush fsmonitor--daemon --query \"builtin:test_00000002:0\" >actual_2 &&\n+\tnul_to_q actual_q2 &&\n+\n+\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n+\n+\ttouch test_flush/file_3 &&\n+\n+\tgit -C test_flush fsmonitor--daemon --query \"builtin:test_00000002:0\" >actual_3 &&\n+\tnul_to_q actual_q3 &&\n+\n+\tgrep \"file_3\" actual_q3\n+'\n+\n+# The next few test cases create repos where the .git directory is NOT\n+# inside the one of the working directory. That is, where .git is a file\n+# that points to a directory elsewhere. This happens for submodules and\n+# non-primary worktrees.\n+\n+test_expect_success 'setup worktree base' '\n+\tgit init wt-base &&\n+\techo 1 >wt-base/file1 &&\n+\tgit -C wt-base add file1 &&\n+\tgit -C wt-base commit -m \"c1\"\n+'\n+\n+test_expect_success 'worktree with .git file' '\n+\tgit -C wt-base worktree add ../wt-secondary &&\n+\n+\t(\n+\t\tGIT_TRACE2_PERF=\"$PWD/trace2_wt_secondary\" &&\n+\t\texport GIT_TRACE2_PERF &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/trace_wt_secondary\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon wt-secondary\n+\t) &&\n+\n+\tgit -C wt-secondary fsmonitor--daemon --stop &&\n+\ttest_must_fail git -C wt-secondary fsmonitor--daemon --is-running\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"420825","messageId":"2b4ae4fc3d622f0fee8eb5e527a51467e13acfc5.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 16/23] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:58Z","receivedAt":"2021-04-01T17:58:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to respond to IPC requests from client\nGit processes and respond with a list of modified pathnames\nrelative to the provided token.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 335 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 333 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 48071d445c49..32df392b25d3 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,6 +7,7 @@\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n+#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon --start []\"),\n@@ -369,19 +370,349 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \tfsmonitor_free_token_data(free_me);\n }\n \n+/*\n+ * Format an opaque token string to send to the client.\n+ */\n+static void fsmonitor_format_response_token(\n+\tstruct strbuf *response_token,\n+\tconst struct strbuf *response_token_id,\n+\tconst struct fsmonitor_batch *batch)\n+{\n+\tuint64_t seq_nr = (batch) ? batch->batch_seq_nr + 1 : 0;\n+\n+\tstrbuf_reset(response_token);\n+\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n+\t\t response_token_id->buf, seq_nr);\n+}\n+\n+/*\n+ * Parse an opaque token from the client.\n+ */\n+static int fsmonitor_parse_client_token(const char *buf_token,\n+\t\t\t\t\tstruct strbuf *requested_token_id,\n+\t\t\t\t\tuint64_t *seq_nr)\n+{\n+\tconst char *p;\n+\tchar *p_end;\n+\n+\tstrbuf_reset(requested_token_id);\n+\t*seq_nr = 0;\n+\n+\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n+\t\treturn 1;\n+\n+\twhile (*p && *p != ':')\n+\t\tstrbuf_addch(requested_token_id, *p++);\n+\tif (!*p++)\n+\t\treturn 1;\n+\n+\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n+\tif (*p_end)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal);\n+\n+static int do_handle_client(struct fsmonitor_daemon_state *state,\n+\t\t\t const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\tstruct fsmonitor_token_data *token_data = NULL;\n+\tstruct strbuf response_token = STRBUF_INIT;\n+\tstruct strbuf requested_token_id = STRBUF_INIT;\n+\tstruct strbuf payload = STRBUF_INIT;\n+\tuint64_t requested_oldest_seq_nr = 0;\n+\tuint64_t total_response_len = 0;\n+\tconst char *p;\n+\tconst struct fsmonitor_batch *batch_head;\n+\tconst struct fsmonitor_batch *batch;\n+\tintmax_t count = 0, duplicates = 0;\n+\tkh_str_t *shown;\n+\tint hash_ret;\n+\tint result;\n+\n+\t/*\n+\t * We expect `command` to be of the form:\n+\t *\n+\t * := quit NUL\n+\t * | flush NUL\n+\t * | NUL\n+\t * | NUL\n+\t */\n+\n+\tif (!strcmp(command, \"quit\")) {\n+\t\t/*\n+\t\t * A client has requested over the socket/pipe that the\n+\t\t * daemon shutdown.\n+\t\t *\n+\t\t * Tell the IPC thread pool to shutdown (which completes\n+\t\t * the await in the main thread (which can stop the\n+\t\t * fsmonitor listener thread)).\n+\t\t *\n+\t\t * There is no reply to the client.\n+\t\t */\n+\t\treturn SIMPLE_IPC_QUIT;\n+\t}\n+\n+\tif (!strcmp(command, \"flush\")) {\n+\t\t/*\n+\t\t * Flush all of our cached data and generate a new token\n+\t\t * just like if we lost sync with the filesystem.\n+\t\t *\n+\t\t * Then send a trivial response using the new token.\n+\t\t */\n+\t\tfsmonitor_force_resync(state);\n+\t\tresult = 0;\n+\t\tgoto send_trivial_response;\n+\t}\n+\n+\tif (!skip_prefix(command, \"builtin:\", &p)) {\n+\t\t/* assume V1 timestamp or garbage */\n+\n+\t\tchar *p_end;\n+\n+\t\tstrtoumax(command, &p_end, 10);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t ((*p_end) ?\n+\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n+\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n+\t\t\t\t command);\n+\t\tresult = -1;\n+\t\tgoto send_trivial_response;\n+\t}\n+\n+\t/* try V2 token */\n+\n+\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n+\t\t\t\t\t &requested_oldest_seq_nr)) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n+\t\t\t\t command);\n+\t\tresult = -1;\n+\t\tgoto send_trivial_response;\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (!state->current_token_data) {\n+\t\t/*\n+\t\t * We don't have a current token. This may mean that\n+\t\t * the listener thread has not yet started.\n+\t\t */\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\t\tresult = 0;\n+\t\tgoto send_trivial_response;\n+\t}\n+\tif (strcmp(requested_token_id.buf,\n+\t\t state->current_token_data->token_id.buf)) {\n+\t\t/*\n+\t\t * The client last spoke to a different daemon\n+\t\t * instance -OR- the daemon had to resync with\n+\t\t * the filesystem (and lost events), so reject.\n+\t\t */\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\t\tresult = 0;\n+\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n+\t\t\t\t \"response/token\", \"different\");\n+\t\tgoto send_trivial_response;\n+\t}\n+\tif (!state->current_token_data->batch_tail) {\n+\t\t/*\n+\t\t * The listener has not received any filesystem\n+\t\t * events yet since we created the current token.\n+\t\t * We can respond with an empty list, since the\n+\t\t * client has already seen the current token and\n+\t\t * we have nothing new to report. (This is\n+\t\t * instead of sending a trivial response.)\n+\t\t */\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\t\tresult = 0;\n+\t\tgoto send_empty_response;\n+\t}\n+\tif (requested_oldest_seq_nr <\n+\t state->current_token_data->batch_tail->batch_seq_nr) {\n+\t\t/*\n+\t\t * The client wants older events than we have for\n+\t\t * this token_id. This means that the end of our\n+\t\t * batch list was truncated and we cannot give the\n+\t\t * client a complete snapshot relative to their\n+\t\t * request.\n+\t\t */\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"client requested truncated data\");\n+\t\tresult = 0;\n+\t\tgoto send_trivial_response;\n+\t}\n+\n+\t/*\n+\t * We're going to hold onto a pointer to the current\n+\t * token-data while we walk the list of batches of files.\n+\t * During this time, we will NOT be under the lock.\n+\t * So we ref-count it.\n+\t *\n+\t * This allows the listener thread to continue prepending\n+\t * new batches of items to the token-data (which we'll ignore).\n+\t *\n+\t * AND it allows the listener thread to do a token-reset\n+\t * (and install a new `current_token_data`).\n+\t *\n+\t * We mark the current head of the batch list as \"pinned\" so\n+\t * that the listener thread will treat this item as read-only\n+\t * (and prevent any more paths from being added to it) from\n+\t * now on.\n+\t */\n+\ttoken_data = state->current_token_data;\n+\ttoken_data->client_ref_count++;\n+\n+\tbatch_head = token_data->batch_head;\n+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * FSMonitor Protocol V2 requires that we send a response header\n+\t * with a \"new current token\" and then all of the paths that changed\n+\t * since the \"requested token\".\n+\t */\n+\tfsmonitor_format_response_token(&response_token,\n+\t\t\t\t\t&token_data->token_id,\n+\t\t\t\t\tbatch_head);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttotal_response_len += response_token.len + 1;\n+\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\", response_token.buf);\n+\n+\tshown = kh_init_str();\n+\tfor (batch = batch_head;\n+\t batch && batch->batch_seq_nr >= requested_oldest_seq_nr;\n+\t batch = batch->next) {\n+\t\tsize_t k;\n+\n+\t\tfor (k = 0; k < batch->nr; k++) {\n+\t\t\tconst char *s = batch->interned_paths[k];\n+\t\t\tsize_t s_len;\n+\n+\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n+\t\t\t\tduplicates++;\n+\t\t\telse {\n+\t\t\t\tkh_put_str(shown, s, &hash_ret);\n+\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n+\t\t\t\t\t\t count, s);\n+\n+\t\t\t\t/* Each path gets written with a trailing NUL */\n+\t\t\t\ts_len = strlen(s) + 1;\n+\n+\t\t\t\tif (payload.len + s_len >=\n+\t\t\t\t LARGE_PACKET_DATA_MAX) {\n+\t\t\t\t\treply(reply_data, payload.buf,\n+\t\t\t\t\t payload.len);\n+\t\t\t\t\ttotal_response_len += payload.len;\n+\t\t\t\t\tstrbuf_reset(&payload);\n+\t\t\t\t}\n+\n+\t\t\t\tstrbuf_add(&payload, s, s_len);\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (payload.len) {\n+\t\treply(reply_data, payload.buf, payload.len);\n+\t\ttotal_response_len += payload.len;\n+\t}\n+\n+\tkh_release_str(shown);\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (token_data->client_ref_count > 0)\n+\t\ttoken_data->client_ref_count--;\n+\n+\tif (token_data->client_ref_count == 0) {\n+\t\tif (token_data != state->current_token_data) {\n+\t\t\t/*\n+\t\t\t * The listener thread did a token-reset while we were\n+\t\t\t * walking the batch list. Therefore, this token is\n+\t\t\t * stale and can be discarded completely. If we are\n+\t\t\t * the last reader thread using this token, we own\n+\t\t\t * that work.\n+\t\t\t */\n+\t\t\tfsmonitor_free_token_data(token_data);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n+\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\tstrbuf_release(&payload);\n+\n+\treturn 0;\n+\n+send_trivial_response:\n+\tpthread_mutex_lock(&state->main_lock);\n+\tfsmonitor_format_response_token(&response_token,\n+\t\t\t\t\t&state->current_token_data->token_id,\n+\t\t\t\t\tstate->current_token_data->batch_head);\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\treply(reply_data, \"/\", 2);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/trivial\", 1);\n+\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\n+\treturn result;\n+\n+send_empty_response:\n+\tpthread_mutex_lock(&state->main_lock);\n+\tfsmonitor_format_response_token(&response_token,\n+\t\t\t\t\t&state->current_token_data->token_id,\n+\t\t\t\t\tNULL);\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/empty\", 1);\n+\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\n+\treturn 0;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data, const char *command,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n-\t/* struct fsmonitor_daemon_state *state = data; */\n+\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n+\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n+\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n \n-\tresult = 0; /* TODO Do something here. */\n+\tresult = do_handle_client(state, command, reply, reply_data);\n \n \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n \n-- \ngitgitgadget\n\n"},{"id":"420827","messageId":"95d511d83b1211f24aeb17edbd4918750f406ece.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 05/23] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:47Z","receivedAt":"2021-04-01T18:03:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a built-in file system monitoring daemon that can be used by\nthe existing `fsmonitor` feature (protocol API and index extension)\nto improve the performance of various Git commands, such as `status`.\n\nThe `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\nprovides an alternative to hook access to existing fsmonitors such\nas `watchman`.\n\nThis commit merely adds the new command without any functionality.\n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 52 +++++++++++++++++++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 56 insertions(+)\n create mode 100644 builtin/fsmonitor--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 3dcdb6bb5ab8..beccf34abe9e 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -71,6 +71,7 @@\n /git-format-patch\n /git-fsck\n /git-fsck-objects\n+/git-fsmonitor--daemon\n /git-gc\n /git-get-tar-commit-id\n /git-grep\ndiff --git a/Makefile b/Makefile\nindex 50977911d41a..d792631d4250 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1091,6 +1091,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n+BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex b6ce981b7377..7554476f90a4 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -158,6 +158,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\n int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nnew file mode 100644\nindex 000000000000..6700bac92c7d\n--- /dev/null\n+++ b/builtin/fsmonitor--daemon.c\n@@ -0,0 +1,52 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"simple-ipc.h\"\n+#include \"khash.h\"\n+\n+static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tNULL\n+};\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tenum daemon_mode {\n+\t\tUNDEFINED_MODE,\n+\t} mode = UNDEFINED_MODE;\n+\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\n+\tswitch (mode) {\n+\tcase UNDEFINED_MODE:\n+\tdefault:\n+\t\tdie(_(\"Unhandled command mode %d\"), mode);\n+\t}\n+}\n+\n+#else\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n+}\n+#endif\ndiff --git a/git.c b/git.c\nindex 9bc077a025cb..239deb9823fc 100644\n--- a/git.c\n+++ b/git.c\n@@ -523,6 +523,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"420828","messageId":"a84dee24e13ebf93488a9258d7324387721f3833.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 08/23] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:50Z","receivedAt":"2021-04-01T18:04:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty implementation of fsmonitor--daemon\nbackend for MacOS.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++\n config.mak.uname | 2 ++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 25 insertions(+)\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\nnew file mode 100644\nindex 000000000000..b91058d1c4f8\n--- /dev/null\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n@@ -0,0 +1,20 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-fs-listen.h\"\n+\n+int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex fcd88b60b14a..394355463e1e 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin)\n \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n \t\tendif\n \tendif\n+\tFSMONITOR_DAEMON_BACKEND = macos\n+\tBASIC_LDFLAGS += -framework CoreServices\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 727cfd561169..341a85e7bfc9 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -255,6 +255,9 @@ endif()\n if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \tlist(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c)\n+elseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n+\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\tlist(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-macos.c)\n endif()\n \n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n-- \ngitgitgadget\n\n"},{"id":"420829","messageId":"c6d5f045fb5644306a3676e5fa4145ba4c6e9b93.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 18/23] fsmonitor--daemon:: introduce client delay for testing","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:41:00Z","receivedAt":"2021-04-01T18:04:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDefine GIT_TEST_FSMONITOR_CLIENT_DELAY as a millisecond delay.\n\nIntroduce an artificial delay when processing client requests.\nThis make the CI/PR test suite a little more stable and avoids\nthe need to load up test scripts with sleep statements to avoid\nracy failures. This was mostly seen on 1 or 2 core CI build\nmachines where the test script would create a file and quickly\ntry to confirm that the daemon had seen it *before* the daemon\nhad received the kernel event and causing a test failure.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 38 ++++++++++++++++++++++++++++++++++++-\n 1 file changed, 37 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex e9a9aea59ad6..0cb09ef0b984 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -150,6 +150,30 @@ static int do_as_client__send_flush(void)\n \treturn 0;\n }\n \n+static int lookup_client_test_delay(void)\n+{\n+\tstatic int delay_ms = -1;\n+\n+\tconst char *s;\n+\tint ms;\n+\n+\tif (delay_ms >= 0)\n+\t\treturn delay_ms;\n+\n+\tdelay_ms = 0;\n+\n+\ts = getenv(\"GIT_TEST_FSMONITOR_CLIENT_DELAY\");\n+\tif (!s)\n+\t\treturn delay_ms;\n+\n+\tms = atoi(s);\n+\tif (ms < 0)\n+\t\treturn delay_ms;\n+\n+\tdelay_ms = ms;\n+\treturn delay_ms;\n+}\n+\n /*\n * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n * \"token\" as a virtual timestamp. Clients can request a summary of all\n@@ -526,6 +550,18 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\treturn SIMPLE_IPC_QUIT;\n \t}\n \n+\t/*\n+\t * For testing purposes, introduce an artificial delay in this\n+\t * worker to allow the filesystem listener thread to receive\n+\t * any fs events that may have been generated by the client\n+\t * process on the other end of the pipe/socket. This helps\n+\t * make the CI/PR test suite runs a little more predictable\n+\t * and hopefully eliminates the need to introduce `sleep`\n+\t * commands in the test scripts.\n+\t */\n+\tif (state->test_client_delay_ms)\n+\t\tsleep_millisec(state->test_client_delay_ms);\n+\n \tif (!strcmp(command, \"flush\")) {\n \t\t/*\n \t\t * Flush all of our cached data and generate a new token\n@@ -1038,7 +1074,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n-\tstate.test_client_delay_ms = 0;\n+\tstate.test_client_delay_ms = lookup_client_test_delay();\n \n \t/* Prepare to (recursively) watch the directory. */\n \tstrbuf_init(&state.path_worktree_watch, 0);\n-- \ngitgitgadget\n\n"},{"id":"420830","messageId":"a57ddb3bc7cc2de9a48472c4faff32ef46132f73.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 13/23] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:55Z","receivedAt":"2021-04-01T18:04:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the win32 backend to register a watch on the working tree\nroot directory (recursively). Also watch the if it is\nnot inside the working tree. And to collect path change notifications\ninto batches and publish.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 493 +++++++++++++++++++\n 1 file changed, 493 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\nindex 880446b49e35..2f1fcf85a0a4 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n@@ -2,20 +2,513 @@\n #include \"config.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-fs-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+/*\n+ * The documentation of ReadDirectoryChangesW() states that the maximum\n+ * buffer size is 64K when the monitored directory is remote.\n+ *\n+ * Larger buffers may be used when the monitored directory is local and\n+ * will help us receive events faster from the kernel and avoid dropped\n+ * events.\n+ *\n+ * So we try to use a very large buffer and silently fallback to 64K if\n+ * we get an error.\n+ */\n+#define MAX_RDCW_BUF_FALLBACK (65536)\n+#define MAX_RDCW_BUF (65536 * 8)\n+\n+struct one_watch\n+{\n+\tchar buffer[MAX_RDCW_BUF];\n+\tDWORD buf_len;\n+\tDWORD count;\n+\n+\tstruct strbuf path;\n+\tHANDLE hDir;\n+\tHANDLE hEvent;\n+\tOVERLAPPED overlapped;\n+\n+\t/*\n+\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n+\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n+\t */\n+\tBOOL is_active;\n+};\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tstruct one_watch *watch_worktree;\n+\tstruct one_watch *watch_gitdir;\n+\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hListener[3]; /* we don't own these handles */\n+#define LISTENER_SHUTDOWN 0\n+#define LISTENER_HAVE_DATA_WORKTREE 1\n+#define LISTENER_HAVE_DATA_GITDIR 2\n+\tint nr_listener_handles;\n+};\n+\n+/*\n+ * Convert the WCHAR path from the notification into UTF8 and\n+ * then normalize it.\n+ */\n+static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+\t\t\t\t struct strbuf *normalized_path)\n+{\n+\tint reserve;\n+\tint len = 0;\n+\n+\tstrbuf_reset(normalized_path);\n+\tif (!info->FileNameLength)\n+\t\tgoto normalize;\n+\n+\t/*\n+\t * Pre-reserve enough space in the UTF8 buffer for\n+\t * each Unicode WCHAR character to be mapped into a\n+\t * sequence of 2 UTF8 characters. That should let us\n+\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n+\t */\n+\treserve = info->FileNameLength + 1;\n+\tstrbuf_grow(normalized_path, reserve);\n+\n+\tfor (;;) {\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n+\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t\t\t normalized_path->buf,\n+\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n+\t\t\t\t\t NULL, NULL);\n+\t\tif (len > 0)\n+\t\t\tgoto normalize;\n+\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n+\t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n+\t\t\t GetLastError(),\n+\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n+\t\t\t info->FileName);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tstrbuf_grow(normalized_path,\n+\t\t\t strbuf_avail(normalized_path) + reserve);\n+\t}\n+\n+normalize:\n+\tstrbuf_setlen(normalized_path, len);\n+\treturn strbuf_normalize_path(normalized_path);\n+}\n \n void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+}\n+\n+static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const char *path)\n+{\n+\tstruct one_watch *watch = NULL;\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileA(path,\n+\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n+\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n+\t\t\t NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n+\t\t GetLastError(), path);\n+\t\treturn NULL;\n+\t}\n+\n+\twatch = xcalloc(1, sizeof(*watch));\n+\n+\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n+\n+\tstrbuf_init(&watch->path, 0);\n+\tstrbuf_addstr(&watch->path, path);\n+\n+\twatch->hDir = hDir;\n+\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\treturn watch;\n+}\n+\n+static void destroy_watch(struct one_watch *watch)\n+{\n+\tif (!watch)\n+\t\treturn;\n+\n+\tstrbuf_release(&watch->path);\n+\tif (watch->hDir != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hDir);\n+\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hEvent);\n+\n+\tfree(watch);\n+}\n+\n+static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+\t\t\t struct one_watch *watch)\n+{\n+\tDWORD dwNotifyFilter =\n+\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n+\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n+\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n+\t\tFILE_NOTIFY_CHANGE_SIZE |\n+\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n+\t\tFILE_NOTIFY_CHANGE_CREATION;\n+\n+\tResetEvent(watch->hEvent);\n+\n+\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n+\twatch->overlapped.hEvent = watch->hEvent;\n+\n+start_watch:\n+\twatch->is_active = ReadDirectoryChangesW(\n+\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n+\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n+\n+\tif (!watch->is_active &&\n+\t GetLastError() == ERROR_INVALID_PARAMETER &&\n+\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n+\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n+\t\tgoto start_watch;\n+\t}\n+\n+\tif (watch->is_active)\n+\t\treturn 0;\n+\n+\terror(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static int recv_rdcw_watch(struct one_watch *watch)\n+{\n+\twatch->is_active = FALSE;\n+\n+\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n+\t\t\t\tTRUE))\n+\t\treturn 0;\n+\n+\t// TODO If an external is deleted, the above returns an error.\n+\t// TODO I'm not sure that there's anything that we can do here other\n+\t// TODO than failing -- the /.git link file would be broken\n+\t// TODO anyway. We might try to check for that and return a better\n+\t// TODO error message.\n+\n+\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static void cancel_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD count;\n+\n+\tif (!watch || !watch->is_active)\n+\t\treturn;\n+\n+\tCancelIoEx(watch->hDir, &watch->overlapped);\n+\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n+\twatch->is_active = FALSE;\n+}\n+\n+/*\n+ * Process filesystem events that happen anywhere (recursively) under the\n+ * root directory. For a normal working directory, this includes\n+ * both version controlled files and the contents of the .git/ directory.\n+ *\n+ * If /.git is a file, then we only see events for the file\n+ * itself.\n+ */\n+static int process_worktree_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_worktree;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tconst char *p = watch->buffer;\n+\n+\t/*\n+\t * If the kernel gets more events than will fit in the kernel\n+\t * buffer associated with our RDCW handle, it drops them and\n+\t * returns a count of zero. (A successful call, but with\n+\t * length zero.)\n+\t */\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\t}\n+\n+\t/*\n+\t * On Windows, `info` contains an \"array\" of paths that are\n+\t * relative to the root of whichever directory handle received\n+\t * the event.\n+\t */\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\t\t/* ignore everything inside of \"/.git/\" */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* queue normal pathname */\n+\t\t\tif (!batch)\n+\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t\tgoto skip_this_path;\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tbatch = NULL;\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\n+force_shutdown:\n+\tfsmonitor_batch__free(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_SHUTDOWN;\n+}\n+\n+/*\n+ * Process filesystem events that happend anywhere (recursively) under the\n+ * external (such as non-primary worktrees or submodules).\n+ * We only care about cookie files that our client threads created here.\n+ *\n+ * Note that we DO NOT get filesystem events on the external \n+ * itself (it is not inside something that we are watching). In particular,\n+ * we do not get an event if the external is deleted.\n+ */\n+static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_gitdir;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *p = watch->buffer;\n+\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_GITDIR;\n+\t}\n+\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n+\n+\t\ttrace_printf_key(&trace_fsmonitor, \"BBB: %s\", path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\tgoto skip_this_path;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t\tgoto skip_this_path;\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, NULL, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_GITDIR;\n }\n \n void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tDWORD dwWait;\n+\n+\tstate->error_code = 0;\n+\n+\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tif (data->watch_gitdir &&\n+\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tfor (;;) {\n+\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n+\t\t\t\t\t\tdata->hListener,\n+\t\t\t\t\t\tFALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n+\t\t\tif (recv_rdcw_watch(data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n+\t\t\tif (recv_rdcw_watch(data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->error_code = -1;\n+\n+force_shutdown:\n+\t/*\n+\t * Tell the IPC thead pool to stop (which completes the await\n+\t * in the main thread (which will also signal this thread (if\n+\t * we are still alive))).\n+\t */\n+\tipc_server_stop_async(state->ipc_server_data);\n+\n+clean_shutdown:\n+\tcancel_rdcw_watch(data->watch_worktree);\n+\tcancel_rdcw_watch(data->watch_gitdir);\n }\n \n int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = xcalloc(1, sizeof(*data));\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->watch_worktree = create_watch(state,\n+\t\t\t\t\t state->path_worktree_watch.buf);\n+\tif (!data->watch_worktree)\n+\t\tgoto failed;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->watch_gitdir = create_watch(state,\n+\t\t\t\t\t\t state->path_gitdir_watch.buf);\n+\t\tif (!data->watch_gitdir)\n+\t\t\tgoto failed;\n+\t}\n+\n+\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_listener_handles++;\n+\n+\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n+\t\tdata->watch_worktree->hEvent;\n+\tdata->nr_listener_handles++;\n+\n+\tif (data->watch_gitdir) {\n+\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n+\t\t\tdata->watch_gitdir->hEvent;\n+\t\tdata->nr_listener_handles++;\n+\t}\n+\n+\tstate->backend_data = data;\n+\treturn 0;\n+\n+failed:\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n \treturn -1;\n }\n \n void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n-- \ngitgitgadget\n\n"},{"id":"420832","messageId":"da5094e52032e28337cfcecd421dedb560952869.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 22/23] p7519: add fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:41:04Z","receivedAt":"2021-04-01T18:10:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRepeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\nthe \"Simple IPC\" interface.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 37 +++++++++++++++++++++++++++++++++++--\n 1 file changed, 35 insertions(+), 2 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 5eb5044a103c..2d018bc7d589 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -24,7 +24,8 @@ test_description=\"Test core.fsmonitor\"\n # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\n # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an\n # absolute path to an integration. May be a space delimited list of\n-# absolute paths to integrations.\n+# absolute paths to integrations. (This hook or list of hooks does not\n+# include the built-in fsmonitor--daemon.)\n #\n # The big win for using fsmonitor is the elimination of the need to scan the\n # working directory looking for changed and untracked files. If the file\n@@ -135,10 +136,16 @@ test_expect_success \"one time repo setup\" '\n \n setup_for_fsmonitor() {\n \t# set INTEGRATION_SCRIPT depending on the environment\n-\tif test -n \"$INTEGRATION_PATH\"\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n \tthen\n+\t\tgit config core.useBuiltinFSMonitor true &&\n+\t\tINTEGRATION_SCRIPT=false\n+\telif test -n \"$INTEGRATION_PATH\"\n+\tthen\n+\t\tgit config core.useBuiltinFSMonitor false &&\n \t\tINTEGRATION_SCRIPT=\"$INTEGRATION_PATH\"\n \telse\n+\t\tgit config core.useBuiltinFSMonitor false &&\n \t\t#\n \t\t# Choose integration script based on existence of Watchman.\n \t\t# Fall back to an empty integration script.\n@@ -285,4 +292,30 @@ test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n \n+#\n+# Run a full set of perf tests using the built-in fsmonitor--daemon.\n+# It does not use the Hook API, so it has a different setup.\n+# Explicitly start the daemon here and before we start client commands\n+# so that we can later add custom tracing.\n+#\n+\n+test_lazy_prereq HAVE_FSMONITOR_DAEMON '\n+\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n+'\n+\n+if test_have_prereq HAVE_FSMONITOR_DAEMON\n+then\n+\tUSE_FSMONITOR_DAEMON=t\n+\n+\ttrace_start fsmonitor--daemon--server\n+\tgit fsmonitor--daemon --start\n+\n+\ttrace_start fsmonitor--daemon--client\n+\ttest_expect_success \"setup for fsmonitor--daemon\" 'setup_for_fsmonitor'\n+\ttest_fsmonitor_suite\n+\n+\tgit fsmonitor--daemon --stop\n+\ttrace_stop\n+fi\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"420833","messageId":"451563314d84f9d6dee29a4899b5d18033aa227d.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 10/23] fsmonitor--daemon: add pathname classification","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:52Z","receivedAt":"2021-04-01T18:10:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to classify relative and absolute\npathnames and decide how they should be handled. This will\nbe used by the platform-specific backend to respond to each\nfilesystem event.\n\nWhen we register for filesystem notifications on a directory,\nwe get events for everything (recursively) in the directory.\nWe want to report to clients changes to tracked and untracked\npaths within the working directory. We do not want to report\nchanges within the .git directory, for example.\n\nThis classification will be used in a later commit by the\ndifferent backends to classify paths as events are received.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 81 +++++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 61 ++++++++++++++++++++++++++++\n 2 files changed, 142 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 23a063707972..16252487240a 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -168,6 +168,87 @@ static int handle_client(void *data, const char *command,\n \treturn result;\n }\n \n+#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *rel)\n+{\n+\tif (fspathncmp(rel, \".git\", 4))\n+\t\treturn IS_WORKDIR_PATH;\n+\trel += 4;\n+\n+\tif (!*rel)\n+\t\treturn IS_DOT_GIT;\n+\tif (*rel != '/')\n+\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n+\trel++;\n+\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_DOT_GIT;\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *rel)\n+{\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_GITDIR;\n+}\n+\n+static enum fsmonitor_path_type try_classify_workdir_abs_path(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\n+\tif (fspathncmp(path, state->path_worktree_watch.buf,\n+\t\t state->path_worktree_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_worktree_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_workdir_relative(rel);\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\tenum fsmonitor_path_type t;\n+\n+\tt = try_classify_workdir_abs_path(state, path);\n+\tif (state->nr_paths_watching == 1)\n+\t\treturn t;\n+\tif (t != IS_OUTSIDE_CONE)\n+\t\treturn t;\n+\n+\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n+\t\t state->path_gitdir_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_gitdir_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_GITDIR; /* it is the exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_gitdir_relative(rel);\n+}\n+\n static void *fsmonitor_fs_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 09e4a6fb6675..97ea3766e900 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -32,5 +32,66 @@ struct fsmonitor_daemon_state {\n \tint test_client_delay_ms;\n };\n \n+/*\n+ * Pathname classifications.\n+ *\n+ * The daemon classifies the pathnames that it receives from file\n+ * system notification events into the following categories and uses\n+ * that to decide whether clients are told about them. (And to watch\n+ * for file system synchronization events.)\n+ *\n+ * The client should only care about paths within the working\n+ * directory proper (inside the working directory and not \".git\" nor\n+ * inside of \".git/\"). That is, the client has read the index and is\n+ * asking for a list of any paths in the working directory that have\n+ * been modified since the last token. The client does not care about\n+ * file system changes within the .git directory (such as new loose\n+ * objects or packfiles). So the client will only receive paths that\n+ * are classified as IS_WORKDIR_PATH.\n+ *\n+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n+ * exact \".git\" directory or GITDIR. If the daemon receives a delete\n+ * event for either of these directories, it will automatically\n+ * shutdown, for example.\n+ *\n+ * Note that the daemon DOES NOT explicitly watch nor special case the\n+ * \".git/index\" file. The daemon does not read the index and does not\n+ * have any internal index-relative state. The daemon only collects\n+ * the set of modified paths within the working directory.\n+ */\n+enum fsmonitor_path_type {\n+\tIS_WORKDIR_PATH = 0,\n+\n+\tIS_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n+\n+\tIS_GITDIR,\n+\tIS_INSIDE_GITDIR,\n+\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n+\n+\tIS_OUTSIDE_CONE,\n+};\n+\n+/*\n+ * Classify a pathname relative to the root of the working directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify a pathname relative to a that is external to the\n+ * worktree directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify an absolute pathname received from a filesystem event.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"420838","messageId":"18c125ec73dc9c23725d7c4688fdc1e753e14780.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 03/23] config: FSMonitor is repository-specific","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:45Z","receivedAt":"2021-04-01T18:22:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis commit refactors `git_config_get_fsmonitor()` into the `repo_*()`\nform that takes a parameter `struct repository *r`.\n\nThat change prepares for the upcoming `core.useFSMonitorDaemon` flag which\nwill be stored in the `repo_settings` struct.\n\nSigned-off-by: Johannes Schindelin \n---\n builtin/update-index.c | 4 ++--\n config.c | 4 ++--\n config.h | 2 +-\n fsmonitor.c | 2 +-\n 4 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 79087bccea4b..84793df8b2b6 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1214,14 +1214,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tif (repo_config_get_fsmonitor(r) == 0)\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\n \t\t\t\t\"enable fsmonitor\"));\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tif (repo_config_get_fsmonitor(r) == 1)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/config.c b/config.c\nindex 6428393a4143..955ff4f9461d 100644\n--- a/config.c\n+++ b/config.c\n@@ -2513,9 +2513,9 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n+int repo_config_get_fsmonitor(struct repository *r)\n {\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n+\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &core_fsmonitor))\n \t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n \n \tif (core_fsmonitor && !*core_fsmonitor)\ndiff --git a/config.h b/config.h\nindex 19a9adbaa9a3..3139de81d986 100644\n--- a/config.h\n+++ b/config.h\n@@ -607,7 +607,7 @@ int git_config_get_index_threads(int *dest);\n int git_config_get_untracked_cache(void);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n+int repo_config_get_fsmonitor(struct repository *r);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34e..9c9b2abc9414 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -411,7 +411,7 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tint fsmonitor_enabled = repo_config_get_fsmonitor(istate->repo ? istate->repo : the_repository);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n-- \ngitgitgadget\n\n"},{"id":"420839","messageId":"27f47dfbd9cf1103fcc5ffff9da6239c1a441987.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 07/23] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:49Z","receivedAt":"2021-04-01T18:22:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty backend for fsmonitor--daemon on Windows.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 ++++++\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++\n compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++\n config.mak.uname | 2 +\n contrib/buildsystems/CMakeLists.txt | 5 ++\n 5 files changed, 90 insertions(+)\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n\ndiff --git a/Makefile b/Makefile\nindex d792631d4250..014bc1baa03a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -467,6 +467,11 @@ all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n+# If your platform supports an built-in fsmonitor backend, set\n+# FSMONITOR_DAEMON_BACKEND to the name of the corresponding\n+# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n+# `fsmonitor_fs_listen__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1904,6 +1909,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n \tCOMPAT_OBJS += compat/access.o\n endif\n \n+ifdef FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2761,6 +2771,9 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n+ifdef FSMONITOR_DAEMON_BACKEND\n+\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\nnew file mode 100644\nindex 000000000000..880446b49e35\n--- /dev/null\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-fs-listen.h\"\n+\n+void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen.h b/compat/fsmonitor/fsmonitor-fs-listen.h\nnew file mode 100644\nindex 000000000000..c7b5776b3b60\n--- /dev/null\n+++ b/compat/fsmonitor/fsmonitor-fs-listen.h\n@@ -0,0 +1,49 @@\n+#ifndef FSMONITOR_FS_LISTEN_H\n+#define FSMONITOR_FS_LISTEN_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * fsmonitor_fs_listener thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread AFTER joining the listener.\n+ */\n+void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to watch for\n+ * filesystem events. This will run in the fsmonitor_fs_listen thread.\n+ *\n+ * It should call `ipc_server_stop_async()` if the listener thread\n+ * prematurely terminates (because of a filesystem error or if it\n+ * detects that the .git directory has been deleted). (It should NOT\n+ * do so if the listener thread receives a normal shutdown signal from\n+ * the IPC layer.)\n+ *\n+ * It should set `state->error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the fsmonitor listener thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_FS_LISTEN_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cb443b4e023a..fcd88b60b14a 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -420,6 +420,7 @@ ifeq ($(uname_S),Windows)\n \t# so we don't need this:\n \t#\n \t# SNPRINTF_RETURNS_BOGUS = YesPlease\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -598,6 +599,7 @@ ifneq (,$(findstring MINGW,$(uname_S)))\n \tNO_STRTOUMAX = YesPlease\n \tNO_MKDTEMP = YesPlease\n \tNO_SVN_TESTS = YesPlease\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 9897fcc8ea2a..727cfd561169 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -252,6 +252,11 @@ else()\n \tlist(APPEND compat_SOURCES compat/simple-ipc/ipc-shared.c compat/simple-ipc/ipc-unix-socket.c)\n endif()\n \n+if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n+\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\tlist(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c)\n+endif()\n+\n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n \n #header checks\n-- \ngitgitgadget\n\n"},{"id":"420840","messageId":"074273330f8d6c656dfec7c8778fad20314c6ad1.1617291666.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH 01/23] fsmonitor--daemon: man page and documentation","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T15:40:43Z","receivedAt":"2021-04-01T18:33:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a manual page describing the `git fsmonitor--daemon` feature.\n\nUpdate references to `core.fsmonitor`, `core.fsmonitorHookVersion` and\npointers to `watchman` to mention the built-in FSMonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/config/core.txt | 45 +++++++---\n Documentation/git-fsmonitor--daemon.txt | 104 ++++++++++++++++++++++++\n Documentation/git-update-index.txt | 4 +-\n Documentation/githooks.txt | 3 +-\n 4 files changed, 144 insertions(+), 12 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a15..d6e2f01966cb 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -66,18 +66,43 @@ core.fsmonitor::\n \twill identify all files that may have changed since the\n \trequested date/time. This information is used to speed up git by\n \tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note: FSMonitor hooks (and this config setting) are ignored if the\n+built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`).\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the version of hook that is to be used when calling the\n+\tFSMonitor hook (as configured via `core.fsmonitor`).\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n++\n+Note: FSMonitor hooks (and this config setting) are ignored if the\n+built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`).\n+\n+core.useBuiltinFSMonitor::\n+\tIf set to true, enable the built-in filesystem event watcher (for\n+\ttechnical details, see linkgit:git-fsmonitor--daemon[1]).\n++\n+Like external (hook-based) FSMonitors, the built-in FSMonitor can speed up\n+Git commands that need to refresh the Git index (e.g. `git status`) in a\n+worktree with many files. The built-in FSMonitor facility eliminates the\n+need to install and maintain an external third-party monitoring tool.\n++\n+The built-in FSMonitor is currently available only on a limited set of\n+supported platforms.\n++\n+Note: if this config setting is set to `true`, any FSMonitor hook\n+configured via `core.fsmonitor` (and possibly `core.fsmonitorHookVersion`)\n+is ignored.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\nnew file mode 100644\nindex 000000000000..b94f57c97fe4\n--- /dev/null\n+++ b/Documentation/git-fsmonitor--daemon.txt\n@@ -0,0 +1,104 @@\n+git-fsmonitor--daemon(1)\n+========================\n+\n+NAME\n+----\n+git-fsmonitor--daemon - Builtin file system monitor daemon\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git fsmonitor--daemon' --start\n+'git fsmonitor--daemon' --run\n+'git fsmonitor--daemon' --stop\n+'git fsmonitor--daemon' --is-running\n+'git fsmonitor--daemon' --is-supported\n+'git fsmonitor--daemon' --query \n+'git fsmonitor--daemon' --query-index\n+'git fsmonitor--daemon' --flush\n+\n+DESCRIPTION\n+-----------\n+\n+Monitors files and directories in the working directory for changes using\n+platform-specific file system notification facilities.\n+\n+It communicates directly with commands like `git status` using the\n+link:technical/api-simple-ipc.html[simple IPC] interface instead of\n+the slower linkgit:githooks[5] interface.\n+\n+OPTIONS\n+-------\n+\n+--start::\n+\tStarts the fsmonitor daemon in the background.\n+\n+--run::\n+\tRuns the fsmonitor daemon in the foreground.\n+\n+--stop::\n+\tStops the fsmonitor daemon running for the current working\n+\tdirectory, if present.\n+\n+--is-running::\n+\tExits with zero status if the fsmonitor daemon is watching the\n+\tcurrent working directory.\n+\n+--is-supported::\n+\tExits with zero status if the fsmonitor daemon feature is supported\n+\ton this platform.\n+\n+--query ::\n+\tConnects to the fsmonitor daemon (starting it if necessary) and\n+\trequests the list of changed files and directories since the\n+\tgiven token.\n+\tThis is intended for testing purposes.\n+\n+--query-index::\n+\tRead the current `` from the File System Monitor index\n+\textension (if present) and use it to query the fsmonitor daemon.\n+\tThis is intended for testing purposes.\n+\n+--flush::\n+\tForce the fsmonitor daemon to flush its in-memory cache and\n+\tre-sync with the file system.\n+\tThis is intended for testing purposes.\n+\n+REMARKS\n+-------\n+The fsmonitor daemon is a long running process that will watch a single\n+working directory. Commands, such as `git status`, should automatically\n+start it (if necessary) when `core.useBuiltinFSMonitor` is set to `true`\n+(see linkgit:git-config[1]).\n+\n+Configure the built-in FSMonitor via `core.useBuiltinFSMonitor` in each\n+working directory separately, or globally via `git config --global\n+core.useBuiltinFSMonitor true`.\n+\n+Tokens are opaque strings. They are used by the fsmonitor daemon to\n+mark a point in time and the associated internal state. Callers should\n+make no assumptions about the content of the token. In particular,\n+the should not assume that it is a timestamp.\n+\n+Query commands send a request-token to the daemon and it responds with\n+a summary of the changes that have occurred since that token was\n+created. The daemon also returns a response-token that the client can\n+use in a future query.\n+\n+For more information see the \"File System Monitor\" section in\n+linkgit:git-update-index[1].\n+\n+CAVEATS\n+-------\n+\n+The fsmonitor daemon does not currently know about submodules and does\n+not know to filter out file system events that happen within a\n+submodule. If fsmonitor daemon is watching a super repo and a file is\n+modified within the working directory of a submodule, it will report\n+the change (as happening against the super repo). However, the client\n+should properly ignore these extra events, so performance may be affected\n+but it should not cause an incorrect result.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d976..8169aad7ee9f 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex b51959ff9418..b7d5e926f7b0 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -593,7 +593,8 @@ fsmonitor-watchman\n \n This hook is invoked when the configuration option `core.fsmonitor` is\n set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n-depending on the version of the hook to use.\n+depending on the version of the hook to use, unless overridden via\n+`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n \n Version 1 takes two arguments, a version (1) and the time in elapsed\n nanoseconds since midnight, January 1, 1970.\n-- \ngitgitgadget\n\n"},{"id":"422131","messageId":"xmqq8s5hkfxm.fsf@gitster.g","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/23] [RFC] Builtin FSMonitor Feature","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-16T22:44:37Z","receivedAt":"2021-04-16T22:44:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> This patch series adds a builtin FSMonitor daemon to Git.\n\nThis hasn't seen much (if any) activity for a few weeks.\n\nDoes that mean nobody (other than obviously the author and whoever\nwanted to have this feature) is interested?\n\nWhat does it need to get this topic unstuck?\n\n> Finally, having a builtin daemon eliminates the need for user to download\n> and install a third-party tool. This makes enterprise deployments simpler\n> since there are fewer parts to install, maintain, and updates to track.\n>\n> This RFC version includes support for Windows and MacOS file system events.\n> A Linux version will be submitted in a later patch series.\n\n"},{"id":"422442","messageId":"nycvar.QRO.7.76.6.2104201646210.54@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"xmqq8s5hkfxm.fsf@gitster.g","subject":"Re: [PATCH 00/23] [RFC] Builtin FSMonitor Feature","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-04-20T15:27:59Z","receivedAt":"2021-04-20T15:28:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Fri, 16 Apr 2021, Junio C Hamano wrote:\n\n> \"Jeff Hostetler via GitGitGadget\" writes:\n>\n> > This patch series adds a builtin FSMonitor daemon to Git.\n>\n> This hasn't seen much (if any) activity for a few weeks.\n\nIt actually is a good sign: I integrated this into Git for Windows (as an\nexperimental feature) and am running with it for a couple of weeks already\n(in _all_ worktrees, not just the massively large ones).\n\nAt first, I ran into a handful Blue Screens of Death, and I was worried\nthat they should be attributed to FSMonitor. But it turns out that this\nissue was most likely caused by a Windows update, and semi-resolved with\nanother Windows update (and only happens because I use WSL extensively).\nIn other words, those crashes are not related to FSMonitor.\n\nSo yeah, I find the lack of activity pretty good news.\n\nHowever, I would have hoped that this patch series would see a couple of\nreviews in the meantime. Since I was involved in the development of this\npatch series (I started it just before I got dragged into all that\nsecurity work that led to v2.24.1, and never quite got back to it after\nthat), I wondered whether it would be \"self review\" if I reviewed those\npatches, which is something I'd rather avoid.\n\nBut if nobody else reviews the patches, I will.\n\n> Does that mean nobody (other than obviously the author and whoever\n> wanted to have this feature) is interested?\n\nThe most likely reason why this does not see more reviews is that it\nmatters most for massive worktrees, and I don't think anybody here works\nwith those. The closest to a massive worktree I have is the `git-sdk-64`\nrepository (which has pretty much nothing to do with source code at all,\nit is just a matter of convenience that this is a Git repository; Think of\nit as if somebody mirrored their Ubuntu installation by tracking it in a\nGit repository and cloning it onto all of their machines). And that is not\nreally all that massive:\n\n\t$ git -C / ls-files | wc -l\n\t162975\n\nThat's tiny compared to some worktrees I saw.\n\nBut we should not mistake the needs of those on the Git mailing list (`git\nls-tree -r v2.31.1 | wc -l` says we have only 3901 files/symlinks) for the\nneeds of some of our biggest users.\n\nSo I would like to respectfully ask for this patch series to be kept under\nconsideration for `next`.\n\n> What does it need to get this topic unstuck?\n\nThe same resource that you keep complaining about, and that seems to be\ndrained more quickly than it can be replenished: reviewers.\n\nI am as guilty as the next person, of course, and it does not help that I\nget Cc:ed on several dozen patches seemingly every couple of days: this is\njust too much, and I cannot do it, so I admittedly neglect too many patch\nseries (even the ones that I _do_ want to review, such as the\n`bisect-in-c` one). My inbox is seriously no fun place to visit right now.\n\n> > Finally, having a builtin daemon eliminates the need for user to download\n> > and install a third-party tool. This makes enterprise deployments simpler\n> > since there are fewer parts to install, maintain, and updates to track.\n> >\n> > This RFC version includes support for Windows and MacOS file system events.\n> > A Linux version will be submitted in a later patch series.\n\nI guess this is another reason why this patch series did not see many\nreviews: the lack of a Linux backend. And I fear that the statement \"A\nLinux version will be submitted in a later patch series\" is a bit strong,\ngiven that my original implementation of that backend does not really do\nits job well: it uses `inotify` and therefore requires one handle _per\ndirectory_, which in turn drains the number of file handles rather quickly\nwhen your worktree has many directories. Meaning: It fails todoes not work in the\nmassive worktrees for which it was intended.\n\nNow, I heard rumors that there is a saner way to monitor directory trees\nin recent Linux kernel versions (Jeff, can you fill in where I am\nblanking?) and it might be a good idea to solicit volunteers to tackle\nthis backend, so that the Linux-leaning crowd on this here mailing list\nis interested a bit more?\n\nCiao,\nDscho\n"},{"id":"422462","messageId":"e5035fa3-4218-209a-28e7-7c1b57d7f388@jeffhostetler.com","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2104201646210.54@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 00/23] [RFC] Builtin FSMonitor Feature","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-20T19:13:35Z","receivedAt":"2021-04-20T19:13:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/20/21 11:27 AM, Johannes Schindelin wrote:\n> Hi Junio,\n...\n>>> This RFC version includes support for Windows and MacOS file system events.\n>>> A Linux version will be submitted in a later patch series.\n> \n> I guess this is another reason why this patch series did not see many\n> reviews: the lack of a Linux backend. And I fear that the statement \"A\n> Linux version will be submitted in a later patch series\" is a bit strong,\n> given that my original implementation of that backend does not really do\n> its job well: it uses `inotify` and therefore requires one handle _per\n> directory_, which in turn drains the number of file handles rather quickly\n> when your worktree has many directories. Meaning: It fails todoes not work in the\n> massive worktrees for which it was intended.\n> \n> Now, I heard rumors that there is a saner way to monitor directory trees\n> in recent Linux kernel versions (Jeff, can you fill in where I am\n> blanking?) and it might be a good idea to solicit volunteers to tackle\n> this backend, so that the Linux-leaning crowd on this here mailing list\n> is interested a bit more?\n\nYes, I removed the early inotify-based version because the kernel limits\nthe number of inotify handles to 8k (at least on my Mint box) and that\nis a global limit -- shared by any process wanting to use inotify.\nThe first monorepo that I tried it on had 120K directories in my\nsparse checkout...\n\nI'm told there is a newer \"fanotify\" facility available in newer\nLinux kernels that behaves more like Windows and MacOS and handles\nsubdirectories. I intend to jump into that shortly (unless someone\nis already familiar with fanotify and and wants to try it).\n\nJeff\n\n"},{"id":"422579","messageId":"f2f61f43-51ec-366f-9fac-f0d42d707de0@gmail.com","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2104201646210.54@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 00/23] [RFC] Builtin FSMonitor Feature","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-21T13:17:25Z","receivedAt":"2021-04-21T13:19:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/20/2021 11:27 AM, Johannes Schindelin wrote:\n> Hi Junio,\n> \n> On Fri, 16 Apr 2021, Junio C Hamano wrote:\n> \n>> \"Jeff Hostetler via GitGitGadget\" writes:\n>>\n>>> This patch series adds a builtin FSMonitor daemon to Git.\n>>\n>> This hasn't seen much (if any) activity for a few weeks.\n...\n>> What does it need to get this topic unstuck?\n> \n> The same resource that you keep complaining about, and that seems to be\n> drained more quickly than it can be replenished: reviewers.\n\nI purposefully stayed away from reviewing the series since we are on\nthe same team, but I have _not_ been involved in the development. At\nleast that lets me have fresh eyes.\n\nIf no external community members are willing to review it, then I will\ndedicate time for a careful review this week.\n\nThanks,\n-Stolee\n"},{"id":"422925","messageId":"b868522c-f07f-c431-aa2c-14af84015ced@gmail.com","threadId":"55420","inReplyTo":"074273330f8d6c656dfec7c8778fad20314c6ad1.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 01/23] fsmonitor--daemon: man page and documentation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T14:13:15Z","receivedAt":"2021-04-26T14:13:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Create a manual page describing the `git fsmonitor--daemon` feature.\n> \n> Update references to `core.fsmonitor`, `core.fsmonitorHookVersion` and\n> pointers to `watchman` to mention the built-in FSMonitor.\n\nMake sense to add clarity here, since there will be new ways\nto interact with a fileystem monitor.\n> core.fsmonitorHookVersion::\n> -\tSets the version of hook that is to be used when calling fsmonitor.\n> -\tThere are currently versions 1 and 2. When this is not set,\n> -\tversion 2 will be tried first and if it fails then version 1\n> -\twill be tried. Version 1 uses a timestamp as input to determine\n> -\twhich files have changes since that time but some monitors\n> -\tlike watchman have race conditions when used with a timestamp.\n> -\tVersion 2 uses an opaque string so that the monitor can return\n> -\tsomething that can be used to determine what files have changed\n> -\twithout race conditions.\n> +\tSets the version of hook that is to be used when calling the\n> +\tFSMonitor hook (as configured via `core.fsmonitor`).\n> ++\n> +There are currently versions 1 and 2. When this is not set,\n> +version 2 will be tried first and if it fails then version 1\n> +will be tried. Version 1 uses a timestamp as input to determine\n> +which files have changes since that time but some monitors\n> +like watchman have race conditions when used with a timestamp.\n> +Version 2 uses an opaque string so that the monitor can return\n> +something that can be used to determine what files have changed\n> +without race conditions.\n\nThis initially seemed like a big edit, but you just split the single\nparagraph into multiple, with a better leading sentence and a final\nstatement about the built-in FSMonitor. Good.\n> ++\n> +Note: FSMonitor hooks (and this config setting) are ignored if the\n> +built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`).\n> +\n> +core.useBuiltinFSMonitor::\n> +\tIf set to true, enable the built-in filesystem event watcher (for\n> +\ttechnical details, see linkgit:git-fsmonitor--daemon[1]).\n> ++\n> +Like external (hook-based) FSMonitors, the built-in FSMonitor can speed up\n> +Git commands that need to refresh the Git index (e.g. `git status`) in a\n> +worktree with many files. The built-in FSMonitor facility eliminates the\n> +need to install and maintain an external third-party monitoring tool.\n> ++\n> +The built-in FSMonitor is currently available only on a limited set of\n> +supported platforms.\n\nIs there a way for users to know this set of platforms? Can they run\na command to find out? Will 'git fsmonitor--daemon --start' send a\nhelpful message to assist here? Or, could there be a 'git\nfsmonitor--daemon --test' command?\n\n> +Note: if this config setting is set to `true`, any FSMonitor hook\n> +configured via `core.fsmonitor` (and possibly `core.fsmonitorHookVersion`)\n> +is ignored.\n...\n> +git-fsmonitor--daemon(1)\n> +========================\n> +\n> +NAME\n> +----\n> +git-fsmonitor--daemon - Builtin file system monitor daemon\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git fsmonitor--daemon' --start\n> +'git fsmonitor--daemon' --run\n> +'git fsmonitor--daemon' --stop\n> +'git fsmonitor--daemon' --is-running\n> +'git fsmonitor--daemon' --is-supported\n> +'git fsmonitor--daemon' --query \n> +'git fsmonitor--daemon' --query-index\n> +'git fsmonitor--daemon' --flush\n\nThese arguments with the \"--\" prefix make it seem like they are\noptions that could be grouped together, but you really want these\nto be verbs within the daemon. What do you think about removing\nthe \"--\" prefixes?\n\n> +\n> +DESCRIPTION\n> +-----------\n> +\n> +Monitors files and directories in the working directory for changes using\n> +platform-specific file system notification facilities.\n> +\n> +It communicates directly with commands like `git status` using the\n> +link:technical/api-simple-ipc.html[simple IPC] interface instead of\n> +the slower linkgit:githooks[5] interface.\n> +\n> +OPTIONS\n> +-------\n\nI typically view \"OPTIONS\" as arguments that can be grouped together,\nbut you are describing things more like verbs or subcommands. The\nmost recent example I know about is 'git maintenance ',\ndocumented at [1].\n\n[1] https://git-scm.com/docs/git-maintenance#_subcommands\n\n> +\n> +--start::\n> +\tStarts the fsmonitor daemon in the background.\n> +\n> +--run::\n> +\tRuns the fsmonitor daemon in the foreground.\n> +\n> +--stop::\n> +\tStops the fsmonitor daemon running for the current working\n> +\tdirectory, if present.\n\nI'm noticing \"fsmonitor\" in lowercase throughout this document. Is\nthat the intended case for user-facing documentation? I've been \nseeing \"FS Monitor\", \"filesystem monitor\", or even \"File System\nMonitor\" in other places.\n\n> +--is-running::\n> +\tExits with zero status if the fsmonitor daemon is watching the\n> +\tcurrent working directory.\n\nAnother potential name for this verb is \"status\".\n\n> +--is-supported::\n> +\tExits with zero status if the fsmonitor daemon feature is supported\n> +\ton this platform.\n\nAh, here is an indicator of whether the platform is supported. Please\ninclude details for this command in the earlier documentation. I'll\ncheck later to see if a message is also sent over 'stderr', which\nwould be helpful. Documenting the exit status is good for third-party\ntools that might use this.\n\n> +--query ::\n> +\tConnects to the fsmonitor daemon (starting it if necessary) and\n> +\trequests the list of changed files and directories since the\n> +\tgiven token.\n> +\tThis is intended for testing purposes.\n> +\n> +--query-index::\n> +\tRead the current `` from the File System Monitor index\n> +\textension (if present) and use it to query the fsmonitor daemon.\n> +\tThis is intended for testing purposes.\n\nThese two could be grouped as \"query [--token=X|--index]\", especially\nbecause they are for testing purposes.\n\n> +\n> +--flush::\n> +\tForce the fsmonitor daemon to flush its in-memory cache and\n> +\tre-sync with the file system.\n> +\tThis is intended for testing purposes.\n\nDo you see benefits to these being available in the CLI? Could these\nbe better served as a test helper?\n\n> +REMARKS\n> +-------\n> +The fsmonitor daemon is a long running process that will watch a single\n> +working directory. Commands, such as `git status`, should automatically\n> +start it (if necessary) when `core.useBuiltinFSMonitor` is set to `true`\n> +(see linkgit:git-config[1]).\n> +\n> +Configure the built-in FSMonitor via `core.useBuiltinFSMonitor` in each\n> +working directory separately, or globally via `git config --global\n> +core.useBuiltinFSMonitor true`.\n> +\n> +Tokens are opaque strings. They are used by the fsmonitor daemon to\n> +mark a point in time and the associated internal state. Callers should\n> +make no assumptions about the content of the token. In particular,\n> +the should not assume that it is a timestamp.\n> +\n> +Query commands send a request-token to the daemon and it responds with\n> +a summary of the changes that have occurred since that token was\n> +created. The daemon also returns a response-token that the client can\n> +use in a future query.\n> +\n> +For more information see the \"File System Monitor\" section in\n> +linkgit:git-update-index[1].\n> +\n> +CAVEATS\n> +-------\n> +\n> +The fsmonitor daemon does not currently know about submodules and does\n> +not know to filter out file system events that happen within a\n> +submodule. If fsmonitor daemon is watching a super repo and a file is\n> +modified within the working directory of a submodule, it will report\n> +the change (as happening against the super repo). However, the client\n> +should properly ignore these extra events, so performance may be affected\n> +but it should not cause an incorrect result.\n\nThere are several uses of the word \"should\" where I think \"will\" is a\nmore appropriate word. That is, unless we do not actually have confidence\nin this behavior.\n\n> --- a/Documentation/git-update-index.txt\n> +++ b/Documentation/git-update-index.txt\n> @@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n> This feature is intended to speed up git operations for repos that have\n> large working directories.\n> \n> -It enables git to work together with a file system monitor (see the\n> +It enables git to work together with a file system monitor (see\n> +linkgit:git-fsmonitor--daemon[1]\n> +and the\n> \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n> inform it as to what files have been modified. This enables git to avoid\n> having to lstat() every file to find modified files.\n> diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\n> index b51959ff9418..b7d5e926f7b0 100644\n> --- a/Documentation/githooks.txt\n> +++ b/Documentation/githooks.txt\n> @@ -593,7 +593,8 @@ fsmonitor-watchman\n> \n> This hook is invoked when the configuration option `core.fsmonitor` is\n> set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n> -depending on the version of the hook to use.\n> +depending on the version of the hook to use, unless overridden via\n> +`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n> \n> Version 1 takes two arguments, a version (1) and the time in elapsed\n> nanoseconds since midnight, January 1, 1970.\n\nThese are good connections to make.\n\nSince the documentation for the fsmonitor--daemon is so deep, this\npatch might be served well to split into two: one that just documents\nthe daemon, and another that updates existing documentation to point\nto the new file.\n\nThis does provide a good basis for me to investigate during the rest\nof the review.\n\nThanks,\n-Stolee\n"},{"id":"422929","messageId":"5b63eb3f-0b87-209b-3943-ced8412abb93@gmail.com","threadId":"55420","inReplyTo":"3dac63eae201e6d0b949680e682238625cad59bd.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 02/23] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T14:31:34Z","receivedAt":"2021-04-26T14:31:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> +#define FSMONITOR_DAEMON_IS_SUPPORTED 1\n> +#else\n> +#define FSMONITOR_DAEMON_IS_SUPPORTED 0\n> +#endif\n> +\n> +/*\n> + * A trivial function so that this source file always defines at least\n> + * one symbol even when the feature is not supported. This quiets an\n> + * annoying compiler error.\n> + */\n> +int fsmonitor_ipc__is_supported(void)\n> +{\n> +\treturn FSMONITOR_DAEMON_IS_SUPPORTED;\n> +}\n\nI don't see any other use of FSMONITOR_DAEMON_IS_SUPPORTED,\nso I was thinking you could use the #ifdef/#else/#endif\nconstruct within the implementation of this method instead\nof creating a macro outside. But my suggestion might be an\nanti-pattern, so feel free to ignore me.\n\n> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> +\n> +GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor\")\n> +\n> +enum ipc_active_state fsmonitor_ipc__get_state(void)\n> +{\n> +\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n> +}\n> +\n> +static int spawn_daemon(void)\n> +{\n> +\tconst char *args[] = { \"fsmonitor--daemon\", \"--start\", NULL };\n> +\n> +\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n> +\t\t\t\t \"fsmonitor\");\n> +}\n> +\n> +int fsmonitor_ipc__send_query(const char *since_token,\n> +\t\t\t struct strbuf *answer)\n> +{\n> +\tint ret = -1;\n> +\tint tried_to_spawn = 0;\n> +\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n> +\tstruct ipc_client_connection *connection = NULL;\n> +\tstruct ipc_client_connect_options options\n> +\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n> +\n> +\toptions.wait_if_busy = 1;\n> +\toptions.wait_if_not_found = 0;\n> +\n> +\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n> +\n> +\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\",\n> +\t\t\t since_token);\n> +\n> +try_again:\n> +\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n> +\t\t\t\t &connection);\n> +\n> +\tswitch (state) {\n> +\tcase IPC_STATE__LISTENING:\n> +\t\tret = ipc_client_send_command_to_connection(\n> +\t\t\tconnection, since_token, answer);\n> +\t\tipc_client_close_connection(connection);\n> +\n> +\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n> +\t\t\t\t \"query/response-length\", answer->len);\n> +\n> +\t\tif (fsmonitor_is_trivial_response(answer))\n> +\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n> +\t\t\t\t\t \"query/trivial-response\", 1);\n> +\n> +\t\tgoto done;\n> +\n> +\tcase IPC_STATE__NOT_LISTENING:\n> +\t\tret = error(_(\"fsmonitor_ipc__send_query: daemon not available\"));\n> +\t\tgoto done;\n\nI'll need to read up on the IPC layer a bit to find out the difference\nbetween IPC_STATE__NOT_LISTENING and IPC_STATE__PATH_NOT_FOUND. When\ntesting on my macOS machine, I got this error. I was expecting the\ndaemon to be spawned. After spawning it myself, it started working.\n\nI expect that there are some cases where the process can fail and the\nnamed pipe is not cleaned up. Let's investigate that soon. I should\nmake it clear that I had tested the builtin FS Monitor on this machine\na few weeks ago, but hadn't been using it much since. We should auto-\nrecover from this situation.\n\nBut also: what is the cost of treating these two cases the same? Could\nwe attempt to \"restart\" the daemon by spawning a new one? Will the new\none find a way to kill a stale one?\n\n(Reading on.)\n\n> +\tcase IPC_STATE__PATH_NOT_FOUND:\n> +\t\tif (tried_to_spawn)\n> +\t\t\tgoto done;\n> +\n> +\t\ttried_to_spawn++;\n> +\t\tif (spawn_daemon())\n> +\t\t\tgoto done;\n\nThis should return zero on success, OK.\n\n> +\t\t/*\n> +\t\t * Try again, but this time give the daemon a chance to\n> +\t\t * actually create the pipe/socket.\n> +\t\t *\n> +\t\t * Granted, the daemon just started so it can't possibly have\n> +\t\t * any FS cached yet, so we'll always get a trivial answer.\n> +\t\t * BUT the answer should include a new token that can serve\n> +\t\t * as the basis for subsequent requests.\n> +\t\t */\n> +\t\toptions.wait_if_not_found = 1;\n> +\t\tgoto try_again;\n\nBecause of the tried_to_spawn check, we will re-run the request over\nIPC but will not retry the spawn_daemon() request. I'm unsure how\nthis could be helpful: is it possible that spawn_daemon() returns a\nnon-zero error code after starting the daemon and somehow that\ndaemon starts working? Or, is this a race-condition thing with parallel\nprocesses also starting up the daemon? It could be good to use this\ncomment to describe why a retry might be helpful.\n\n> +\n> +\tcase IPC_STATE__INVALID_PATH:\n> +\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n> +\t\t\t fsmonitor_ipc__get_path());\n> +\t\tgoto done;\n> +\n> +\tcase IPC_STATE__OTHER_ERROR:\n> +\tdefault:\n> +\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n> +\t\t\t fsmonitor_ipc__get_path());\n> +\t\tgoto done;\n> +\t}\n> +\n> +done:\n> +\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n> +\n> +\treturn ret;\n> +}\n> +\n> +int fsmonitor_ipc__send_command(const char *command,\n> +\t\t\t\tstruct strbuf *answer)\n> +{\n> +\tstruct ipc_client_connection *connection = NULL;\n> +\tstruct ipc_client_connect_options options\n> +\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n> +\tint ret;\n> +\tenum ipc_active_state state;\n> +\n> +\tstrbuf_reset(answer);\n> +\n> +\toptions.wait_if_busy = 1;\n> +\toptions.wait_if_not_found = 0;\n> +\n> +\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n> +\t\t\t\t &connection);\n> +\tif (state != IPC_STATE__LISTENING) {\n> +\t\tdie(\"fsmonitor--daemon is not running\");\n> +\t\treturn -1;\n> +\t}\n> +\n> +\tret = ipc_client_send_command_to_connection(connection, command, answer);\n> +\tipc_client_close_connection(connection);\n> +\n> +\tif (ret == -1) {\n> +\t\tdie(\"could not send '%s' command to fsmonitor--daemon\",\n> +\t\t command);\n> +\t\treturn -1;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n\nI wondier if this ...send_command() method is too generic. It might\nbe nice to have more structure to its inputs and outputs to lessen\nthe cognitive load when plugging into other portions of the code.\nHowever, I'll wait to see what those consumers look like in case the\ngenerality is merited.\n> struct category_description {\n> \tuint32_t category;\n> @@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n> \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n> \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n> \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n> +\n> +\t\tif (fsmonitor_ipc__is_supported())\n> +\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n\nThis change might deserve its own patch, including some documentation\nabout how users can use 'git version --build-options' to determine if\nthe builtin FS Monitor feature is available on their platform.\n\nThanks,\n-Stolee\n"},{"id":"422931","messageId":"f77457d0-925d-2ea1-4ad4-b3315532b37f@gmail.com","threadId":"55420","inReplyTo":"7082528d8f7c1afa33e1146e3d274e044735f6a1.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 04/23] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T14:56:47Z","receivedAt":"2021-04-26T14:56:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/21 11:40 AM, Johannes Schindelin via GitGitGadget wrote:> @@ -2515,6 +2515,11 @@ int git_config_get_max_percent_split_change(void)\n> \n> int repo_config_get_fsmonitor(struct repository *r)\n> {\n> +\tif (r->settings.use_builtin_fsmonitor > 0) {\n\nDon't forget to run prepare_repo_settings(r) first.\n\n> +\t\tcore_fsmonitor = \"(built-in daemon)\";\n> +\t\treturn 1;\n> +\t}\n> +\n\nI found this odd, assigning a string to core_fsmonitor that\nwould definitely cause a problem trying to execute it as a\nhook. I wondered the need for it at all, but found that\nthere are several places in the FS Monitor subsystem that use\ncore_fsmonitor as if it was a boolean, indicating whether or\nnot the feature is enabled at all.\n\nA cleaner way to handle this would be to hide the data behind\na helper method, say \"fsmonitor_enabled()\" that could then\ncheck a value on the repository (or index) and store the hook\nvalue as a separate value that is only used by the hook-based\nimplementation.\n\nIt's probably a good idea to do that cleanup now, before we\nfind on accident that we missed a gap and start trying to run\nthis bogus string as a hook invocation.\n> -static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n> +static int query_fsmonitor(int version, struct index_state *istate, struct strbuf *query_result)\n> {\n> +\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n> +\tconst char *last_update = istate->fsmonitor_last_update;\n> \tstruct child_process cp = CHILD_PROCESS_INIT;\n> \tint result;\n> \n> \tif (!core_fsmonitor)\n> \t\treturn -1;\n\nHere is an example of it being used as a boolean.\n\n> +\tif (r->settings.use_builtin_fsmonitor > 0) {\n> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> +\t\treturn fsmonitor_ipc__send_query(last_update, query_result);\n> +#else\n> +\t\t/* Fake a trivial response. */\n> +\t\twarning(_(\"fsmonitor--daemon unavailable; falling back\"));\n> +\t\tstrbuf_add(query_result, \"/\", 2);\n> +\t\treturn 0;\n> +#endif\n\nThis seems like a case where the helper fsmonitor_ipc__is_supported()\ncould be used instead of compile-time macros.\n\n(I think this is especially true when we consider the future of the\nfeature on Linux and the possibility of the same compiled code needing\nto check run-time properties of the platform for compatibility.)\n\n> --- a/repo-settings.c\n> +++ b/repo-settings.c\n> @@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r)\n> \t\tr->settings.core_multi_pack_index = value;\n> \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n> \n> +\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n> +\t\tr->settings.use_builtin_fsmonitor = 1;\n> +\n\nFollows the patterns of repo settings. Good.\n\n"},{"id":"422933","messageId":"522e10e6-f18d-9c28-7102-f5a855d4be50@gmail.com","threadId":"55420","inReplyTo":"95d511d83b1211f24aeb17edbd4918750f406ece.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 05/23] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T15:08:36Z","receivedAt":"2021-04-26T15:08:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n\nI think these compile-time macros should be replaced with a\nmethod call, as I've said before. It should be simple to say\n\n\tif (!fsmonitor_ipc__is_supported())\n\t\tdie(_(\"fsmonitor--daemon is not supported on this platform\"));\n\nand call it a day. This can be done before parsing arguments.\n\n> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> +{\n> +\tenum daemon_mode {\n> +\t\tUNDEFINED_MODE,\n> +\t} mode = UNDEFINED_MODE;\n> +\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n\nI can see where you are going here, to use the parse-opts API\nto get your \"--\" arguments to populate an 'enum'. However,\nit seems like you will run into the problem where a user enters\nmultiple such arguments and you lose the information as the\nparser overwrites 'mode' here.\n\nBetter to use a positional argument and drop the \"--\" prefix,\nin my opinion.\n\nThanks,\n-Stolee\n"},{"id":"422934","messageId":"098e4123-676e-2ea0-4c16-dbeaa1198b46@gmail.com","threadId":"55420","inReplyTo":"77170e521f67ce92587d833334f6951a9f375d4d.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 06/23] fsmonitor--daemon: implement client command options","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T15:12:13Z","receivedAt":"2021-04-26T15:12:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Implement command options `--stop`, `--is-running`, `--query`,\n> `--query-index`, and `--flush` to control and query the status of a\n> `fsmonitor--daemon` server process (and implicitly start a server\n> process if necessary).\n> \n> Later commits will implement the actual server and monitor\n> the file system.\n\nAs mentioned before, I think the \"query\", \"query-index\", and\n\"flush\" commands are better served in a test helper. Luckily,\nthe implementation you give here seems rather straightforward\nand could fit into a test helper without a lot of duplicated\nboilerplate. That's a good sign for the API presented here.\n\nAs a bonus, you could delay the implementation of those test\nhelpers until they are going to be used in a test.\n\nThanks,\n-Stolee\n"},{"id":"422935","messageId":"36e7dfeb-e833-ec11-f1b9-a596d560b35e@gmail.com","threadId":"55420","inReplyTo":"27f47dfbd9cf1103fcc5ffff9da6239c1a441987.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 07/23] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T15:23:17Z","receivedAt":"2021-04-26T15:23:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n\n> +# If your platform supports an built-in fsmonitor backend, set\n> +# FSMONITOR_DAEMON_BACKEND to the name of the corresponding\n> +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n> +# `fsmonitor_fs_listen__*()` routines.\nI found this to be a little confusing, specifically that you\ncare about the \"\" part of the filename, not the full file\nname. Here is an option:\n\n# If your platform supports an built-in fsmonitor backend, set\n# FSMONITOR_DAEMON_BACKEND to \"\", corresponding to the file\n# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n# `fsmonitor_fs_listen__*()` routines.\n\nEverything else looks pretty standard. Good to create stubs this\nway so they can be consumed by a platform-agnostic caller and then\nimplemented with that context.\n\nThanks,\n-Stolee\n"},{"id":"422938","messageId":"1c45542a-5d80-6e51-26fe-94066987a566@gmail.com","threadId":"55420","inReplyTo":"522e10e6-f18d-9c28-7102-f5a855d4be50@gmail.com","subject":"Re: [PATCH 05/23] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T15:45:32Z","receivedAt":"2021-04-26T15:45:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/26/21 11:08 AM, Derrick Stolee wrote:\n> On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> \n> I think these compile-time macros should be replaced with a\n> method call, as I've said before. It should be simple to say\n> \n> \tif (!fsmonitor_ipc__is_supported())\n> \t\tdie(_(\"fsmonitor--daemon is not supported on this platform\"));\n> \n> and call it a day. This can be done before parsing arguments.\n> \n>> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n>> +{\n>> +\tenum daemon_mode {\n>> +\t\tUNDEFINED_MODE,\n>> +\t} mode = UNDEFINED_MODE;\n>> +\n>> +\tstruct option options[] = {\n>> +\t\tOPT_END()\n>> +\t};\n> \n> I can see where you are going here, to use the parse-opts API\n> to get your \"--\" arguments to populate an 'enum'. However,\n> it seems like you will run into the problem where a user enters\n> multiple such arguments and you lose the information as the\n> parser overwrites 'mode' here.\n\nI see that you use OPT_CMDMODE in your implementation, which\nmakes this concern invalid.\n\n> Better to use a positional argument and drop the \"--\" prefix,\n> in my opinion.\n\nThis is my personal taste, but the technical reason to do this\ndoesn't exist.\n\nThanks,\n-Stolee\n"},{"id":"422939","messageId":"91a0fc4b-505a-fa2b-c797-5e420e836642@gmail.com","threadId":"55420","inReplyTo":"2b291d805d59b54203d939e048bb568782d5e17b.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 09/23] fsmonitor--daemon: implement daemon command options","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T15:47:18Z","receivedAt":"2021-04-26T15:47:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n...\n> +\t/* Prepare to (recursively) watch the directory. */\n> +\tstrbuf_init(&state.path_worktree_watch, 0);\n> +\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n> +\tstate.nr_paths_watching = 1;\n\nYes, let's watch the working directory.\n\n> +\t/*\n> +\t * If \".git\" is not a directory, then is not inside the\n> +\t * cone of , so set up a second watch for it.\n> +\t */\n> +\tstrbuf_init(&state.path_gitdir_watch, 0);\n> +\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n> +\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n> +\tif (!is_directory(state.path_gitdir_watch.buf)) {\n> +\t\tstrbuf_reset(&state.path_gitdir_watch);\n> +\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n> +\t\tstate.nr_paths_watching = 2;\n> +\t}\n\nBut why watch the .git directory, especially for a worktree (or\nsubmodule I guess)? What benefit do we get from events within the\n.git directory? I'm expecting any event within the .git directory\nshould be silently ignored.\n\n> +\n> static int is_ipc_daemon_listening(void)\n> {\n> \treturn fsmonitor_ipc__get_state() == IPC_STATE__LISTENING;\n> }\n> \n> +static int try_to_run_foreground_daemon(void)\n> +{\n> +\t/*\n> +\t * Technically, we don't need to probe for an existing daemon\n> +\t * process, since we could just call `fsmonitor_run_daemon()`\n> +\t * and let it fail if the pipe/socket is busy.\n> +\t *\n> +\t * However, this method gives us a nicer error message for a\n> +\t * common error case.\n> +\t */\n> +\tif (is_ipc_daemon_listening())\n> +\t\tdie(\"fsmonitor--daemon is already running.\");\nHere, it seems like we only care about IPC_STATE_LISTENING, while\nearlier I mentioned that I ended up in IPC_STATE__NOT_LISTENING,\nand my manually running of the daemon helped.\n\n> +\treturn !!fsmonitor_run_daemon();\n> +}\n\nYou are ignoring the IPC_STATE__NOT_LISTENING and creating a new\nprocess, which is good. I'm just wondering why that state exists\nand what is the proper way to handle it?\n\n> +\n> +#ifndef GIT_WINDOWS_NATIVE\n\nYou are already creating a platform-specific mechanism for the\nfilesystem watcher. Shouldn't the implementation of this method\nbe part of that file in compat/fsmonitor/?\n\nI guess the biggest reason is that macOS and Linux share this\nimplementation, so maybe this is the cleanest approach.\n\n> +\n> +/*\n> + * This is adapted from `wait_or_whine()`. Watch the child process and\n> + * let it get started and begin listening for requests on the socket\n> + * before reporting our success.\n> + */\n> +static int wait_for_background_startup(pid_t pid_child)\n> +{\n> +\tint status;\n> +\tpid_t pid_seen;\n> +\tenum ipc_active_state s;\n> +\ttime_t time_limit, now;\n> +\n> +\ttime(&time_limit);\n> +\ttime_limit += fsmonitor__start_timeout_sec;\n> +\n> +\tfor (;;) {\n> +\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n> +\n> +\t\tif (pid_seen == -1)\n> +\t\t\treturn error_errno(_(\"waitpid failed\"));\n> +\n> +\t\telse if (pid_seen == 0) {\n\nThere is some non-standard whitespace throughout this\nif/else if/else:\n...> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\telse if (pid_seen == pid_child) {\n...\n> +\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n> +\t\t}\n> +\n> +\t\telse\n> +\t\t\treturn error(_(\"waitpid is confused\"));\n\nThe rest of the glue in this patch looks good.\n\nThanks,\n-Stolee\n"},{"id":"422940","messageId":"3aa3c802-0ad1-0a65-d646-0cd574cb8036@gmail.com","threadId":"55420","inReplyTo":"91a0fc4b-505a-fa2b-c797-5e420e836642@gmail.com","subject":"Re: [PATCH 09/23] fsmonitor--daemon: implement daemon command options","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T16:12:46Z","receivedAt":"2021-04-26T16:12:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/26/2021 11:47 AM, Derrick Stolee wrote:\n> On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n> ...\n>> +\t/* Prepare to (recursively) watch the directory. */\n>> +\tstrbuf_init(&state.path_worktree_watch, 0);\n>> +\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n>> +\tstate.nr_paths_watching = 1;\n> \n> Yes, let's watch the working directory.\n> \n>> +\t/*\n>> +\t * If \".git\" is not a directory, then is not inside the\n>> +\t * cone of , so set up a second watch for it.\n>> +\t */\n>> +\tstrbuf_init(&state.path_gitdir_watch, 0);\n>> +\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n>> +\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n>> +\tif (!is_directory(state.path_gitdir_watch.buf)) {\n>> +\t\tstrbuf_reset(&state.path_gitdir_watch);\n>> +\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n>> +\t\tstate.nr_paths_watching = 2;\n>> +\t}\n> \n> But why watch the .git directory, especially for a worktree (or\n> submodule I guess)? What benefit do we get from events within the\n> .git directory? I'm expecting any event within the .git directory\n> should be silently ignored.\n\nI see in a following patch that we place a cookie file within the\n.git directory. I'm reminded that this is done for a reason: other\nfilesystem watchers can get into a loop if we place the cookie\nfile outside of the .git directory. The classic example is VS Code\nrunning 'git status' in a loop because Watchman writes a cookie\ninto the root of the working directory.\n\nThanks,\n-Stolee\n"},{"id":"423028","messageId":"aca24c96-33ab-3b90-bf21-0da19283a3d7@gmail.com","threadId":"55420","inReplyTo":"451563314d84f9d6dee29a4899b5d18033aa227d.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 10/23] fsmonitor--daemon: add pathname classification","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T19:17:00Z","receivedAt":"2021-04-26T19:17:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n...\n> +#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n> +\n> +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n> +\tconst char *rel)\n> +{\n> +\tif (fspathncmp(rel, \".git\", 4))\n> +\t\treturn IS_WORKDIR_PATH;\n> +\trel += 4;\n> +\n> +\tif (!*rel)\n> +\t\treturn IS_DOT_GIT;\n> +\tif (*rel != '/')\n> +\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n> +\trel++;\n> +\n> +\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n> +\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n\nSeems like this strlen() could be abstracted out. Is it\nsomething the compiler can compute and set for us? Or,\nshould we create a macro for this constant?\n\n> +\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n> +\n> +\treturn IS_INSIDE_DOT_GIT;\n> +}\n\nHere is the reasoning I was missing for why we watch the .git\ndirectory.\n\n> +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n> +\tconst char *rel)\n> +{\n> +\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n> +\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n> +\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n> +\n> +\treturn IS_INSIDE_GITDIR;\n> +}\n\nAnd I was about to ask \"what happens if we are watching the .git\ndirectory of a worktree?\" but here we have a different classifier.\n\n> +static enum fsmonitor_path_type try_classify_workdir_abs_path(\n> +\tstruct fsmonitor_daemon_state *state,\n> +\tconst char *path)\n> +{\n> +\tconst char *rel;\n> +\n> +\tif (fspathncmp(path, state->path_worktree_watch.buf,\n> +\t\t state->path_worktree_watch.len))\n> +\t\treturn IS_OUTSIDE_CONE;\n> +\n> +\trel = path + state->path_worktree_watch.len;\n> +\n> +\tif (!*rel)\n> +\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n> +\tif (*rel != '/')\n> +\t\treturn IS_OUTSIDE_CONE;\n> +\trel++;\n> +\n> +\treturn fsmonitor_classify_path_workdir_relative(rel);\n> +}\n> +\n> +enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n> +\tstruct fsmonitor_daemon_state *state,\n> +\tconst char *path)\n> +{\n> +\tconst char *rel;\n> +\tenum fsmonitor_path_type t;\n> +\n> +\tt = try_classify_workdir_abs_path(state, path);\n> +\tif (state->nr_paths_watching == 1)\n> +\t\treturn t;\n> +\tif (t != IS_OUTSIDE_CONE)\n> +\t\treturn t;\n> +\n> +\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n> +\t\t state->path_gitdir_watch.len))\n> +\t\treturn IS_OUTSIDE_CONE;\n> +\n> +\trel = path + state->path_gitdir_watch.len;\n> +\n> +\tif (!*rel)\n> +\t\treturn IS_GITDIR; /* it is the exactly */\n> +\tif (*rel != '/')\n> +\t\treturn IS_OUTSIDE_CONE;\n> +\trel++;\n> +\n> +\treturn fsmonitor_classify_path_gitdir_relative(rel);\n> +}\n\nAnd here is where you differentiate the event across the two\ncases. OK.\n\n> +/*\n> + * Pathname classifications.\n> + *\n> + * The daemon classifies the pathnames that it receives from file\n> + * system notification events into the following categories and uses\n> + * that to decide whether clients are told about them. (And to watch\n> + * for file system synchronization events.)\n> + *\n> + * The client should only care about paths within the working\n> + * directory proper (inside the working directory and not \".git\" nor\n> + * inside of \".git/\"). That is, the client has read the index and is\n> + * asking for a list of any paths in the working directory that have\n> + * been modified since the last token. The client does not care about\n> + * file system changes within the .git directory (such as new loose\n> + * objects or packfiles). So the client will only receive paths that\n> + * are classified as IS_WORKDIR_PATH.\n> + *\n> + * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n> + * exact \".git\" directory or GITDIR. If the daemon receives a delete\n> + * event for either of these directories, it will automatically\n> + * shutdown, for example.\n> + *\n> + * Note that the daemon DOES NOT explicitly watch nor special case the\n> + * \".git/index\" file. The daemon does not read the index and does not\n> + * have any internal index-relative state. The daemon only collects\n> + * the set of modified paths within the working directory.\n> + */\n> +enum fsmonitor_path_type {\n> +\tIS_WORKDIR_PATH = 0,\n> +\n> +\tIS_DOT_GIT,\n> +\tIS_INSIDE_DOT_GIT,\n> +\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n> +\n> +\tIS_GITDIR,\n> +\tIS_INSIDE_GITDIR,\n> +\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n> +\n> +\tIS_OUTSIDE_CONE,\n> +};\n> +\n> +/*\n> + * Classify a pathname relative to the root of the working directory.\n> + */\n> +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n> +\tconst char *relative_path);\n> +\n> +/*\n> + * Classify a pathname relative to a that is external to the\n> + * worktree directory.\n> + */\n> +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n> +\tconst char *relative_path);\n> +\n> +/*\n> + * Classify an absolute pathname received from a filesystem event.\n> + */\n> +enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n> +\tstruct fsmonitor_daemon_state *state,\n> +\tconst char *path);\n> +\n> #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n> #endif /* FSMONITOR_DAEMON_H */\n\nHad I looked ahead and read these comments beforehand, then I would\nhave had an easier time determining the intended behavior from the\nimplementations. Oops.\n\nThanks,\n-Stolee\n"},{"id":"423029","messageId":"3f0c5c7d-8448-15ec-9e12-b2745d7bbf2b@gmail.com","threadId":"55420","inReplyTo":"304fe03034f8622aa8728d8872cc9bc539bab861.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 11/23] fsmonitor--daemon: define token-ids","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T19:49:17Z","receivedAt":"2021-04-26T19:49:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach fsmonitor--daemon to create token-ids and define the\n> overall token naming scheme.\n...\n> +/*\n> + * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n> + * \"token\" as a virtual timestamp. Clients can request a summary of all\n> + * created/deleted/modified files relative to a token. In the response,\n> + * clients receive a new token for the next (relative) request.\n> + *\n> + *\n> + * Token Format\n> + * ============\n> + *\n> + * The contents of the token are private and provider-specific.\n> + *\n> + * For the built-in fsmonitor--daemon, we define a token as follows:\n> + *\n> + * \"builtin\" \":\" \":\" \n> + *\n> + * The is an arbitrary OPAQUE string, such as a GUID,\n> + * UUID, or {timestamp,pid}. It is used to group all filesystem\n> + * events that happened while the daemon was monitoring (and in-sync\n> + * with the filesystem).\n> + *\n> + * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n> + * and does not define less-than/greater-than relationships.\n> + * (There are too many race conditions to rely on file system\n> + * event timestamps.)\n> + *\n> + * The is a simple integer incremented for each event\n> + * received. When a new is created, the is\n> + * reset to zero.\n> + *\n> + *\n> + * About Token Ids\n> + * ===============\n> + *\n> + * A new token_id is created:\n> + *\n> + * [1] each time the daemon is started.\n> + *\n> + * [2] any time that the daemon must re-sync with the filesystem\n> + * (such as when the kernel drops or we miss events on a very\n> + * active volume).\n> + *\n> + * [3] in response to a client \"flush\" command (for dropped event\n> + * testing).\n> + *\n> + * [4] MAYBE We might want to change the token_id after very complex\n> + * filesystem operations are performed, such as a directory move\n> + * sequence that affects many files within. It might be simpler\n> + * to just give up and fake a re-sync (and let the client do a\n> + * full scan) than try to enumerate the effects of such a change.\n> + *\n> + * When a new token_id is created, the daemon is free to discard all\n> + * cached filesystem events associated with any previous token_ids.\n> + * Events associated with a non-current token_id will never be sent\n> + * to a client. A token_id change implicitly means that the daemon\n> + * has gap in its event history.\n> + *\n> + * Therefore, clients that present a token with a stale (non-current)\n> + * token_id will always be given a trivial response.\n\nFrom this comment, it seems to be the case that concurrent Git\ncommands will race to advance the FS Monitor token and one of them\nwill lose, causing a full working directory scan. There is no list\nof \"recent\" tokens.\n\nI could see this changing in the future, but for now it is a\nreasonable simplification.\n\n> + */\n> +struct fsmonitor_token_data {\n> +\tstruct strbuf token_id;\n> +\tstruct fsmonitor_batch *batch_head;\n> +\tstruct fsmonitor_batch *batch_tail;\n> +\tuint64_t client_ref_count;\n> +};\n> +\n> +static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n> +{\n> +\tstatic int test_env_value = -1;\n> +\tstatic uint64_t flush_count = 0;\n> +\tstruct fsmonitor_token_data *token;\n> +\n> +\ttoken = (struct fsmonitor_token_data *)xcalloc(1, sizeof(*token));\n\nI think the best practice here is \"CALLOC_ARRAY(token, 1);\"\n\n> +\n> +\tstrbuf_init(&token->token_id, 0);\n\nThis is likely overkill since you used calloc() above.\n\n> +\ttoken->batch_head = NULL;\n> +\ttoken->batch_tail = NULL;\n> +\ttoken->client_ref_count = 0;\n> +\n> +\tif (test_env_value < 0)\n> +\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n> +\n> +\tif (!test_env_value) {\n> +\t\tstruct timeval tv;\n> +\t\tstruct tm tm;\n> +\t\ttime_t secs;\n> +\n> +\t\tgettimeofday(&tv, NULL);\n> +\t\tsecs = tv.tv_sec;\n> +\t\tgmtime_r(&secs, &tm);\n> +\n> +\t\tstrbuf_addf(&token->token_id,\n> +\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n> +\t\t\t flush_count++,\n> +\t\t\t getpid(),\n> +\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n> +\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n> +\t\t\t (long)tv.tv_usec);\n\nBetween the PID, the flush count, and how deep you go in the\ntimestamp, this seems to be specific enough.\n\n> +\t} else {\n> +\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n\nAnd this will be nice for testing.\n\n> +\t}\n> +\n> +\treturn token;\n> +}\n> +\n> static ipc_server_application_cb handle_client;\n> \n> static int handle_client(void *data, const char *command,\n> @@ -330,7 +436,7 @@ static int fsmonitor_run_daemon(void)\n> \n> \tpthread_mutex_init(&state.main_lock, NULL);\n> \tstate.error_code = 0;\n> -\tstate.current_token_data = NULL;\n> +\tstate.current_token_data = fsmonitor_new_token_data();\n> \tstate.test_client_delay_ms = 0;\n> \n> \t/* Prepare to (recursively) watch the directory. */\n> \n\nThanks,\n-Stolee\n"},{"id":"423033","messageId":"171678f4-75ee-f404-4dd3-215616d08373@gmail.com","threadId":"55420","inReplyTo":"CAPig+cSNJYh-cTxghPiaAxKhYENDKbYO8jrdvoBsB3Eq0GYWjA@mail.gmail.com","subject":"Re: [PATCH 11/23] fsmonitor--daemon: define token-ids","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T20:03:37Z","receivedAt":"2021-04-26T20:03:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/26/2021 4:01 PM, Eric Sunshine wrote:\n> On Mon, Apr 26, 2021 at 3:49 PM Derrick Stolee wrote:\n>> On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>>> + token = (struct fsmonitor_token_data *)xcalloc(1, sizeof(*token));\n>>\n>> I think the best practice here is \"CALLOC_ARRAY(token, 1);\"\n>>\n>>> +\n>>> + strbuf_init(&token->token_id, 0);\n>>\n>> This is likely overkill since you used calloc() above.\n> \n> Not quite. A strbuf must be initialized either with STRBUF_INIT or\n> strbuf_init() in order to make strbuf.buf point at strbuf_slopbuf.\n\nThanks! I didn't know that detail, but it makes a lot of sense.\n\n-Stolee\n"},{"id":"423034","messageId":"CAPig+cSNJYh-cTxghPiaAxKhYENDKbYO8jrdvoBsB3Eq0GYWjA@mail.gmail.com","threadId":"55420","inReplyTo":"3f0c5c7d-8448-15ec-9e12-b2745d7bbf2b@gmail.com","subject":"Re: [PATCH 11/23] fsmonitor--daemon: define token-ids","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-04-26T20:01:57Z","receivedAt":"2021-04-26T20:07:57Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 26, 2021 at 3:49 PM Derrick Stolee wrote:\n> On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> > + token = (struct fsmonitor_token_data *)xcalloc(1, sizeof(*token));\n>\n> I think the best practice here is \"CALLOC_ARRAY(token, 1);\"\n>\n> > +\n> > + strbuf_init(&token->token_id, 0);\n>\n> This is likely overkill since you used calloc() above.\n\nNot quite. A strbuf must be initialized either with STRBUF_INIT or\nstrbuf_init() in order to make strbuf.buf point at strbuf_slopbuf.\n"},{"id":"423035","messageId":"CAPig+cT1yz-M1MYTx1_HQRYe+Ryy=vzwtRgkhkX7bTYDB5rnkA@mail.gmail.com","threadId":"55420","inReplyTo":"aca24c96-33ab-3b90-bf21-0da19283a3d7@gmail.com","subject":"Re: [PATCH 10/23] fsmonitor--daemon: add pathname classification","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-04-26T20:11:58Z","receivedAt":"2021-04-26T20:12:11Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 26, 2021 at 3:17 PM Derrick Stolee wrote:\n> On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> > +#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n> > +\n> > + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n> > + strlen(FSMONITOR_COOKIE_PREFIX)))\n>\n> Seems like this strlen() could be abstracted out. Is it\n> something the compiler can compute and set for us? Or,\n> should we create a macro for this constant?\n\nIf you're asking whether the compiler will resolve strlen(\"literal\nstring\") to an integer constant at compile time rather than computing\nthe length at runtime, then the answer is that on this project we\npresume that the compiler is smart enough to do that.\n\nOr are you asking for a function something like this?\n\n fspathhasprefix(rel, FSMONITOR_COOKIE_PREFIX)\n"},{"id":"423036","messageId":"CAPig+cTjso8B=E0pGsr4TDU5zikVhjwbrrWD19uaiwJXvuXZ0w@mail.gmail.com","threadId":"55420","inReplyTo":"5b63eb3f-0b87-209b-3943-ced8412abb93@gmail.com","subject":"Re: [PATCH 02/23] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-04-26T20:20:29Z","receivedAt":"2021-04-26T20:20:56Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 26, 2021 at 10:31 AM Derrick Stolee wrote:\n> On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> > +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> > +#define FSMONITOR_DAEMON_IS_SUPPORTED 1\n> > +#else\n> > +#define FSMONITOR_DAEMON_IS_SUPPORTED 0\n> > +#endif\n> > +\n> > +int fsmonitor_ipc__is_supported(void)\n> > +{\n> > + return FSMONITOR_DAEMON_IS_SUPPORTED;\n> > +}\n>\n> I don't see any other use of FSMONITOR_DAEMON_IS_SUPPORTED,\n> so I was thinking you could use the #ifdef/#else/#endif\n> construct within the implementation of this method instead\n> of creating a macro outside. But my suggestion might be an\n> anti-pattern, so feel free to ignore me.\n\nOn this project, it is preferred to keep the #if / #else / #endif\noutside of functions since embedding them within functions often makes\nit difficult to follow how the code flows (and generally makes\nfunctions unnecessarily noisy). So, the way Jeff did this seems fine.\n\nAn alternative would have been:\n\n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n #define fsmonitor_ipc__is_supported() 1\n #else\n #define fsmonitor_ipc__is_supported() 0\n #endif\n\nwhich would still allow calling it as a function:\n\n if (fsmonitor_ipc__is_supported())\n ...\n\nbut it's subjective whether that's actually any cleaner or better.\n"},{"id":"423037","messageId":"df33c12b-ada3-05dc-3d17-6cc9d205b4cc@gmail.com","threadId":"55420","inReplyTo":"f1fa803ebe9c9f78608c22e55ec590f8c6775c94.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 12/23] fsmonitor--daemon: create token-based changed path cache","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T20:22:45Z","receivedAt":"2021-04-26T20:22:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach fsmonitor--daemon to build lists of changed paths and associate\n> them with a token-id. This will be used by the platform-specific\n> backends to accumulate changed paths in response to filesystem events.\n> \n> The platform-specific event loops receive batches containing one or\n> more changed paths. Their fs listener thread will accumulate them in\n\nI think the lowercase \"fs\" here is strange. \"Their listener thread\"\ncould be interpreted as the IPC listener, so it's probably best to\nspell it out: \"Their filesystem listener thread\".\n\n> a `fsmonitor_batch` (and without locking) and then \"publish\" them to\n> associate them with the current token and to make them visible to the\n> client worker threads.\n...\n> +struct fsmonitor_batch {\n> +\tstruct fsmonitor_batch *next;\n> +\tuint64_t batch_seq_nr;\n> +\tconst char **interned_paths;\n> +\tsize_t nr, alloc;\n> +\ttime_t pinned_time;\n> +};\n\nA linked list to help with adding while consuming it, but also\nbatching for efficiency. I can see how this will work out\nnicely.\n\n> +struct fsmonitor_batch *fsmonitor_batch__new(void)\n> +{\n> +\tstruct fsmonitor_batch *batch = xcalloc(1, sizeof(*batch));\n\nI mentioned earlier that I think `CALLOC_ARRAY(batch, 1)` is the\ntypical pattern here.\n\n> +\n> +\treturn batch;\n> +}\n> +\n> +struct fsmonitor_batch *fsmonitor_batch__free(struct fsmonitor_batch *batch)\n\nSince this method frees the tip of the list and returns the next\nitem (instead of freeing the entire list), perhaps this would be\nbetter named as _pop()?\n\n> +{\n> +\tstruct fsmonitor_batch *next;\n> +\n> +\tif (!batch)\n> +\t\treturn NULL;\n> +\n> +\tnext = batch->next;\n> +\n> +\t/*\n> +\t * The actual strings within the array are interned, so we don't\n> +\t * own them.\n> +\t */\n> +\tfree(batch->interned_paths);\n> +\n> +\treturn next;\n> +}\n> +\n> +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n> +\t\t\t const char *path)\n> +{\n> +\tconst char *interned_path = strintern(path);\n\nThis use of interned paths is interesting, although I become\nconcerned for the amount of memory we are consuming over the\nlifetime of the process. This could be considered as a target\nfor future improvements, perhaps with an LRU cache or something\nsimilar.\n\n> +\n> +\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n> +\n> +\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n> +\tbatch->interned_paths[batch->nr++] = interned_path;\n> +}\n> +\n> +static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n> +\t\t\t\t const struct fsmonitor_batch *batch_src)\n> +{\n> +\t/* assert state->main_lock */\n> +\n\nThis comment seems stale.\n\n> +\tsize_t k;\n> +\n> +\tALLOC_GROW(batch_dest->interned_paths,\n> +\t\t batch_dest->nr + batch_src->nr + 1,\n> +\t\t batch_dest->alloc);\n> +\n> +\tfor (k = 0; k < batch_src->nr; k++)\n> +\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n> +\t\t\tbatch_src->interned_paths[k];\n> +}\n> +\n> +static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n\nThis one _does_ free the whole list.\n\n> +{\n> +\tstruct fsmonitor_batch *p;\n> +\n> +\tif (!token)\n> +\t\treturn;\n> +\n> +\tassert(token->client_ref_count == 0);\n> +\n> +\tstrbuf_release(&token->token_id);\n> +\n> +\tfor (p = token->batch_head; p; p = fsmonitor_batch__free(p))\n> +\t\t;\n> +\n> +\tfree(token);\n> +}\n> +\n> +/*\n> + * Flush all of our cached data about the filesystem. Call this if we\n> + * lose sync with the filesystem and miss some notification events.\n> + *\n> + * [1] If we are missing events, then we no longer have a complete\n> + * history of the directory (relative to our current start token).\n> + * We should create a new token and start fresh (as if we just\n> + * booted up).\n> + *\n> + * If there are no readers of the the current token data series, we\n> + * can free it now. Otherwise, let the last reader free it. Either\n> + * way, the old token data series is no longer associated with our\n> + * state data.\n> + */\n> +void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n> +{\n> +\tstruct fsmonitor_token_data *free_me = NULL;\n> +\tstruct fsmonitor_token_data *new_one = NULL;\n> +\n> +\tnew_one = fsmonitor_new_token_data();\n> +\n> +\tpthread_mutex_lock(&state->main_lock);\n> +\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"force resync [old '%s'][new '%s']\",\n> +\t\t\t state->current_token_data->token_id.buf,\n> +\t\t\t new_one->token_id.buf);\n> +\n> +\tif (state->current_token_data->client_ref_count == 0)\n> +\t\tfree_me = state->current_token_data;\n> +\tstate->current_token_data = new_one;\n> +\n> +\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\tfsmonitor_free_token_data(free_me);\n> +}\n> +\n\nSwap the pointer under a lock, free outside of it. Good.\n\n> +/*\n> + * We try to combine small batches at the front of the batch-list to avoid\n> + * having a long list. This hopefully makes it a little easier when we want\n> + * to truncate and maintain the list. However, we don't want the paths array\n> + * to just keep growing and growing with realloc, so we insert an arbitrary\n> + * limit.\n> + */\n> +#define MY_COMBINE_LIMIT (1024)\n> +\n> +void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n> +\t\t struct fsmonitor_batch *batch,\n> +\t\t const struct string_list *cookie_names)\n> +{\n> +\tif (!batch && !cookie_names->nr)\n> +\t\treturn;\n> +\n> +\tpthread_mutex_lock(&state->main_lock);\n> +\n> +\tif (batch) {\n> +\t\tstruct fsmonitor_batch *head;\n> +\n> +\t\thead = state->current_token_data->batch_head;\n> +\t\tif (!head) {\n> +\t\t\tbatch->batch_seq_nr = 0;\n> +\t\t\tbatch->next = NULL;\n> +\t\t\tstate->current_token_data->batch_head = batch;\n> +\t\t\tstate->current_token_data->batch_tail = batch;\n> +\t\t} else if (head->pinned_time) {\n> +\t\t\t/*\n> +\t\t\t * We cannot alter the current batch list\n> +\t\t\t * because:\n> +\t\t\t *\n> +\t\t\t * [a] it is being transmitted to at least one\n> +\t\t\t * client and the handle_client() thread has a\n> +\t\t\t * ref-count, but not a lock on the batch list\n> +\t\t\t * starting with this item.\n> +\t\t\t *\n> +\t\t\t * [b] it has been transmitted in the past to\n> +\t\t\t * at least one client such that future\n> +\t\t\t * requests are relative to this head batch.\n> +\t\t\t *\n> +\t\t\t * So, we can only prepend a new batch onto\n> +\t\t\t * the front of the list.\n> +\t\t\t */\n> +\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n> +\t\t\tbatch->next = head;\n> +\t\t\tstate->current_token_data->batch_head = batch;\n> +\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n> +\t\t\t/*\n> +\t\t\t * The head batch in the list has never been\n> +\t\t\t * transmitted to a client, but folding the\n> +\t\t\t * contents of the new batch onto it would\n> +\t\t\t * exceed our arbitrary limit, so just prepend\n> +\t\t\t * the new batch onto the list.\n> +\t\t\t */\n> +\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n> +\t\t\tbatch->next = head;\n> +\t\t\tstate->current_token_data->batch_head = batch;\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * We are free to append the paths in the given\n> +\t\t\t * batch onto the end of the current head batch.\n> +\t\t\t */\n> +\t\t\tfsmonitor_batch__combine(head, batch);\n> +\t\t\tfsmonitor_batch__free(batch);\n> +\t\t}\n> +\t}\n> +\n> +\tpthread_mutex_unlock(&state->main_lock);\n> +}\n\nI appreciate the careful comments in this critical piece of the\ndata structure. Also, it is good that you already have a batch\nof results to merge into the list instead of updating a lock for\nevery filesystem event.\n\nThanks,\n-Stolee\n"},{"id":"423038","messageId":"9a9d6da0-1a31-c04a-f41f-de20d85996b2@gmail.com","threadId":"55420","inReplyTo":"CAPig+cT1yz-M1MYTx1_HQRYe+Ryy=vzwtRgkhkX7bTYDB5rnkA@mail.gmail.com","subject":"Re: [PATCH 10/23] fsmonitor--daemon: add pathname classification","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T20:24:23Z","receivedAt":"2021-04-26T20:24:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/26/2021 4:11 PM, Eric Sunshine wrote:\n> On Mon, Apr 26, 2021 at 3:17 PM Derrick Stolee wrote:\n>> On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>>> +#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n>>> +\n>>> + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n>>> + strlen(FSMONITOR_COOKIE_PREFIX)))\n>>\n>> Seems like this strlen() could be abstracted out. Is it\n>> something the compiler can compute and set for us? Or,\n>> should we create a macro for this constant?\n> \n> If you're asking whether the compiler will resolve strlen(\"literal\n> string\") to an integer constant at compile time rather than computing\n> the length at runtime, then the answer is that on this project we\n> presume that the compiler is smart enough to do that.\n\nThat is what I was asking.\n\n> Or are you asking for a function something like this?\n> \n> fspathhasprefix(rel, FSMONITOR_COOKIE_PREFIX)\n\nThe \"fix\" I would recommend otherwise would be\n\n\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n\t\t\tFSMONITOR_COOKIE_PREFIX_LEN))\n\nwhich is much uglier. I'm glad we can trust the compiler to\nbe smart enough.\n\nThanks,\n-Stolee\n"},{"id":"423042","messageId":"35876c33-4e1a-bc07-cd6c-8d43a25c7903@gmail.com","threadId":"55420","inReplyTo":"2b4ae4fc3d622f0fee8eb5e527a51467e13acfc5.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 16/23] fsmonitor--daemon: implement handle_client callback","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T21:01:02Z","receivedAt":"2021-04-26T21:01:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach fsmonitor--daemon to respond to IPC requests from client\n> Git processes and respond with a list of modified pathnames\n> relative to the provided token.\n\n(I'm skipping ahead to this part. I'll examine the platform\nspecific bits after I finish with \"the Git bits\".)\n\n> +static void fsmonitor_format_response_token(\n> +\tstruct strbuf *response_token,\n> +\tconst struct strbuf *response_token_id,\n> +\tconst struct fsmonitor_batch *batch)\n> +{\n> +\tuint64_t seq_nr = (batch) ? batch->batch_seq_nr + 1 : 0;\n> +\n> +\tstrbuf_reset(response_token);\n> +\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n> +\t\t response_token_id->buf, seq_nr);]\n\nAh, right. The token string gets _even more specific_ to allow\nfor multiple \"checkpoints\" within a batch.\n\n> +static int fsmonitor_parse_client_token(const char *buf_token,\n> +\t\t\t\t\tstruct strbuf *requested_token_id,\n> +\t\t\t\t\tuint64_t *seq_nr)\n> +{\n> +\tconst char *p;\n> +\tchar *p_end;\n> +\n> +\tstrbuf_reset(requested_token_id);\n> +\t*seq_nr = 0;\n> +\n> +\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n> +\t\treturn 1;\n> +\n> +\twhile (*p && *p != ':')\n> +\t\tstrbuf_addch(requested_token_id, *p++);\n\nMy mind is going towards microoptimizations, but I wonder if there\nis a difference using\n\n\tq = strchr(p, ':');\n\tif (!q)\n\t\treturn 1;\n\tstrbuf_add(requested_token_id, p, q - p);\n\nWe trade one scan with several method calls for instead two scans\nand two method calls, but also those two scans are very optimized.\n\nProbably not worth it, as this is something like 20 bytes of data\nper round-trip.\n\n> +\tif (!skip_prefix(command, \"builtin:\", &p)) {\n> +\t\t/* assume V1 timestamp or garbage */\n> +\n> +\t\tchar *p_end;\n> +\n> +\t\tstrtoumax(command, &p_end, 10);\n> +\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t ((*p_end) ?\n> +\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n> +\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n> +\t\t\t\t command);\n> +\t\tresult = -1;\n> +\t\tgoto send_trivial_response;\n> +\t}\n\nThis is an interesting protection for users currently using FS\nMonitor but upgrade to the builtin approach.\n\n> +\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n> +\t\t\t\t\t &requested_oldest_seq_nr)) {\n\nIt appears you will call skip_prefix() twice this way, once to\ndetermine we are actually the right kind of token, but a second\ntime as part of this call. Perhaps the helper method could start\nfrom 'p' which has already advanced beyond \"buildin:\"?\n\n> +\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n> +\t\t\t\t command);\n> +\t\tresult = -1;\n> +\t\tgoto send_trivial_response;\n> +\t}\n\nThis method is getting a bit long. Could the interesting data\nstructure code below be extracted as a method?\n\n> +\tpthread_mutex_lock(&state->main_lock);\n> +\n> +\tif (!state->current_token_data) {\n> +\t\t/*\n> +\t\t * We don't have a current token. This may mean that\n> +\t\t * the listener thread has not yet started.\n> +\t\t */\n> +\t\tpthread_mutex_unlock(&state->main_lock);\n> +\t\tresult = 0;\n> +\t\tgoto send_trivial_response;\n> +\t}\n> +\tif (strcmp(requested_token_id.buf,\n> +\t\t state->current_token_data->token_id.buf)) {\n> +\t\t/*\n> +\t\t * The client last spoke to a different daemon\n> +\t\t * instance -OR- the daemon had to resync with\n> +\t\t * the filesystem (and lost events), so reject.\n> +\t\t */\n> +\t\tpthread_mutex_unlock(&state->main_lock);\n> +\t\tresult = 0;\n> +\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n> +\t\t\t\t \"response/token\", \"different\");\n> +\t\tgoto send_trivial_response;\n> +\t}\n> +\tif (!state->current_token_data->batch_tail) {\n> +\t\t/*\n> +\t\t * The listener has not received any filesystem\n> +\t\t * events yet since we created the current token.\n> +\t\t * We can respond with an empty list, since the\n> +\t\t * client has already seen the current token and\n> +\t\t * we have nothing new to report. (This is\n> +\t\t * instead of sending a trivial response.)\n> +\t\t */\n> +\t\tpthread_mutex_unlock(&state->main_lock);\n> +\t\tresult = 0;\n> +\t\tgoto send_empty_response;\n> +\t}\n> +\tif (requested_oldest_seq_nr <\n> +\t state->current_token_data->batch_tail->batch_seq_nr) {\n> +\t\t/*\n> +\t\t * The client wants older events than we have for\n> +\t\t * this token_id. This means that the end of our\n> +\t\t * batch list was truncated and we cannot give the\n> +\t\t * client a complete snapshot relative to their\n> +\t\t * request.\n> +\t\t */\n> +\t\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t \"client requested truncated data\");\n> +\t\tresult = 0;\n> +\t\tgoto send_trivial_response;\n> +\t}\n\nIf these are part of a helper method, then they could be reorganized\nto \"goto\" the end of the method which returns an error code after\nunlocking the mutex. The multiple unlocks are making me nervous.\n\n> +\n> +\t/*\n> +\t * We're going to hold onto a pointer to the current\n> +\t * token-data while we walk the list of batches of files.\n> +\t * During this time, we will NOT be under the lock.\n> +\t * So we ref-count it.\n\nI was wondering if this would happen. I'm glad it is.\n\n> +\t * This allows the listener thread to continue prepending\n> +\t * new batches of items to the token-data (which we'll ignore).\n> +\t *\n> +\t * AND it allows the listener thread to do a token-reset\n> +\t * (and install a new `current_token_data`).\n> +\t *\n> +\t * We mark the current head of the batch list as \"pinned\" so\n> +\t * that the listener thread will treat this item as read-only\n> +\t * (and prevent any more paths from being added to it) from\n> +\t * now on.\n> +\t */\n> +\ttoken_data = state->current_token_data;\n> +\ttoken_data->client_ref_count++;\n> +\n> +\tbatch_head = token_data->batch_head;\n> +\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n> +\n> +\tpthread_mutex_unlock(&state->main_lock);\n\nWe are now pinned. Makes sense.\n\n> +\t/*\n> +\t * FSMonitor Protocol V2 requires that we send a response header\n> +\t * with a \"new current token\" and then all of the paths that changed\n> +\t * since the \"requested token\".\n> +\t */\n> +\tfsmonitor_format_response_token(&response_token,\n> +\t\t\t\t\t&token_data->token_id,\n> +\t\t\t\t\tbatch_head);\n> +\n> +\treply(reply_data, response_token.buf, response_token.len + 1);\n> +\ttotal_response_len += response_token.len + 1;\n\nI was going to say we should let \"reply\" return the number of bytes written,\nbut that is already an error code. But then we seem to be ignoring it here.\nShould we at least do something like \"err |= reply()\" to collect any errors?\n\n> +\n> +\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n> +\t\t\t response_token.buf);\n> +\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\", response_token.buf);\n> +\n> +\tshown = kh_init_str();\n> +\tfor (batch = batch_head;\n> +\t batch && batch->batch_seq_nr >= requested_oldest_seq_nr;\n> +\t batch = batch->next) {\n> +\t\tsize_t k;\n> +\n> +\t\tfor (k = 0; k < batch->nr; k++) {\n> +\t\t\tconst char *s = batch->interned_paths[k];\n> +\t\t\tsize_t s_len;\n> +\n> +\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n> +\t\t\t\tduplicates++;\n> +\t\t\telse {\n> +\t\t\t\tkh_put_str(shown, s, &hash_ret);\n\nIt appears that you could make use of 'struct strmap' instead of managing your\nown khash structure.\n\n> +\n> +\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n> +\t\t\t\t\t\t count, s);\n> +\n> +\t\t\t\t/* Each path gets written with a trailing NUL */\n> +\t\t\t\ts_len = strlen(s) + 1;\n> +\n> +\t\t\t\tif (payload.len + s_len >=\n> +\t\t\t\t LARGE_PACKET_DATA_MAX) {\n> +\t\t\t\t\treply(reply_data, payload.buf,\n> +\t\t\t\t\t payload.len);\n> +\t\t\t\t\ttotal_response_len += payload.len;\n> +\t\t\t\t\tstrbuf_reset(&payload);\n> +\t\t\t\t}\n> +\n> +\t\t\t\tstrbuf_add(&payload, s, s_len);\n> +\t\t\t\tcount++;\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> +\n> +\tif (payload.len) {\n> +\t\treply(reply_data, payload.buf, payload.len);\n> +\t\ttotal_response_len += payload.len;\n> +\t}\n> +\n> +\tkh_release_str(shown);\n> +\n> +\tpthread_mutex_lock(&state->main_lock);\n> +\tif (token_data->client_ref_count > 0)\n> +\t\ttoken_data->client_ref_count--;\n> +\n> +\tif (token_data->client_ref_count == 0) {\n> +\t\tif (token_data != state->current_token_data) {\n> +\t\t\t/*\n> +\t\t\t * The listener thread did a token-reset while we were\n> +\t\t\t * walking the batch list. Therefore, this token is\n> +\t\t\t * stale and can be discarded completely. If we are\n> +\t\t\t * the last reader thread using this token, we own\n> +\t\t\t * that work.\n> +\t\t\t */\n> +\t\t\tfsmonitor_free_token_data(token_data);\n> +\t\t}\n> +\t}\n\nPerhaps this could be extracted to a method, so that any (locked) caller\ncould run\n\n\tfree_token_if_unused(state, token_data);\n\nand the token will either keep around (because client_ref_count > 0 or\nstate->current_token_data is still on token_data). Otherwise I predict\nthis being implemented in two places, which is too many when dealing with\nmemory ownership.\n\n> +\n> +\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n> +\n> +\tstrbuf_release(&response_token);\n> +\tstrbuf_release(&requested_token_id);\n> +\tstrbuf_release(&payload);\n> +\n> +\treturn 0;\n> +\n> +send_trivial_response:\n> +\tpthread_mutex_lock(&state->main_lock);\n> +\tfsmonitor_format_response_token(&response_token,\n> +\t\t\t\t\t&state->current_token_data->token_id,\n> +\t\t\t\t\tstate->current_token_data->batch_head);\n> +\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\treply(reply_data, response_token.buf, response_token.len + 1);\n> +\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n> +\t\t\t response_token.buf);\n> +\treply(reply_data, \"/\", 2);\n> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/trivial\", 1);\n> +\n> +\tstrbuf_release(&response_token);\n> +\tstrbuf_release(&requested_token_id);\n> +\n> +\treturn result;\n> +\n> +send_empty_response:\n> +\tpthread_mutex_lock(&state->main_lock);\n> +\tfsmonitor_format_response_token(&response_token,\n> +\t\t\t\t\t&state->current_token_data->token_id,\n> +\t\t\t\t\tNULL);\n> +\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\treply(reply_data, response_token.buf, response_token.len + 1);\n> +\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n> +\t\t\t response_token.buf);\n> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/empty\", 1);\n> +\n> +\tstrbuf_release(&response_token);\n> +\tstrbuf_release(&requested_token_id);\n> +\n> +\treturn 0;\n> +}\n> +\n> static ipc_server_application_cb handle_client;\n> \n> static int handle_client(void *data, const char *command,\n> \t\t\t ipc_server_reply_cb *reply,\n> \t\t\t struct ipc_server_reply_data *reply_data)\n> {\n> -\t/* struct fsmonitor_daemon_state *state = data; */\n> +\tstruct fsmonitor_daemon_state *state = data;\n> \tint result;\n> \n> +\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n> +\n> \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n> \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n> \n> -\tresult = 0; /* TODO Do something here. */\n> +\tresult = do_handle_client(state, command, reply, reply_data);\n> \n> \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n> \n\nA simple integration with earlier work. Good.\n\nThanks,\n-Stolee\n"},{"id":"423043","messageId":"f18608a4-c4d3-9974-f8ab-ac416b82bab9@gmail.com","threadId":"55420","inReplyTo":"CAPig+cTjso8B=E0pGsr4TDU5zikVhjwbrrWD19uaiwJXvuXZ0w@mail.gmail.com","subject":"Re: [PATCH 02/23] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-26T21:02:30Z","receivedAt":"2021-04-26T21:02:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/26/2021 4:20 PM, Eric Sunshine wrote:\n> On Mon, Apr 26, 2021 at 10:31 AM Derrick Stolee wrote:\n>> On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>>> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n>>> +#define FSMONITOR_DAEMON_IS_SUPPORTED 1\n>>> +#else\n>>> +#define FSMONITOR_DAEMON_IS_SUPPORTED 0\n>>> +#endif\n>>> +\n>>> +int fsmonitor_ipc__is_supported(void)\n>>> +{\n>>> + return FSMONITOR_DAEMON_IS_SUPPORTED;\n>>> +}\n>>\n>> I don't see any other use of FSMONITOR_DAEMON_IS_SUPPORTED,\n>> so I was thinking you could use the #ifdef/#else/#endif\n>> construct within the implementation of this method instead\n>> of creating a macro outside. But my suggestion might be an\n>> anti-pattern, so feel free to ignore me.\n> \n> On this project, it is preferred to keep the #if / #else / #endif\n> outside of functions since embedding them within functions often makes\n> it difficult to follow how the code flows (and generally makes\n> functions unnecessarily noisy). So, the way Jeff did this seems fine.\n\nMakes sense.\n\n> An alternative would have been:\n> \n> #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> #define fsmonitor_ipc__is_supported() 1\n> #else\n> #define fsmonitor_ipc__is_supported() 0\n> #endif\n> \n> which would still allow calling it as a function:\n> \n> if (fsmonitor_ipc__is_supported())\n> ...\n> \n> but it's subjective whether that's actually any cleaner or better.\n \nTrue. I'm just thinking about a future where we need to do a runtime\ncheck for compatibility, but let's use the YAGNI principle and skip\nit for now.\n\nThanks,\n-Stolee\n"},{"id":"423076","messageId":"87mttkyrqq.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"f77457d0-925d-2ea1-4ad4-b3315532b37f@gmail.com","subject":"Re: [PATCH 04/23] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-27T09:20:56Z","receivedAt":"2021-04-27T09:57:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 26 2021, Derrick Stolee wrote:\n\n> On 4/1/21 11:40 AM, Johannes Schindelin via GitGitGadget wrote:> @@ -2515,6 +2515,11 @@ int git_config_get_max_percent_split_change(void)\n>> \n>> int repo_config_get_fsmonitor(struct repository *r)\n>> {\n>> +\tif (r->settings.use_builtin_fsmonitor > 0) {\n>\n> Don't forget to run prepare_repo_settings(r) first.\n>\n>> +\t\tcore_fsmonitor = \"(built-in daemon)\";\n>> +\t\treturn 1;\n>> +\t}\n>> +\n>\n> I found this odd, assigning a string to core_fsmonitor that\n> would definitely cause a problem trying to execute it as a\n> hook. I wondered the need for it at all, but found that\n> there are several places in the FS Monitor subsystem that use\n> core_fsmonitor as if it was a boolean, indicating whether or\n> not the feature is enabled at all.\n>\n> A cleaner way to handle this would be to hide the data behind\n> a helper method, say \"fsmonitor_enabled()\" that could then\n> check a value on the repository (or index) and store the hook\n> value as a separate value that is only used by the hook-based\n> implementation.\n>\n> It's probably a good idea to do that cleanup now, before we\n> find on accident that we missed a gap and start trying to run\n> this bogus string as a hook invocation.\n>> -static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n>> +static int query_fsmonitor(int version, struct index_state *istate, struct strbuf *query_result)\n>> {\n>> +\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n>> +\tconst char *last_update = istate->fsmonitor_last_update;\n>> \tstruct child_process cp = CHILD_PROCESS_INIT;\n>> \tint result;\n>> \n>> \tif (!core_fsmonitor)\n>> \t\treturn -1;\n>\n> Here is an example of it being used as a boolean.\n>\n>> +\tif (r->settings.use_builtin_fsmonitor > 0) {\n>> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n>> +\t\treturn fsmonitor_ipc__send_query(last_update, query_result);\n>> +#else\n>> +\t\t/* Fake a trivial response. */\n>> +\t\twarning(_(\"fsmonitor--daemon unavailable; falling back\"));\n>> +\t\tstrbuf_add(query_result, \"/\", 2);\n>> +\t\treturn 0;\n>> +#endif\n>\n> This seems like a case where the helper fsmonitor_ipc__is_supported()\n> could be used instead of compile-time macros.\n>\n> (I think this is especially true when we consider the future of the\n> feature on Linux and the possibility of the same compiled code needing\n> to check run-time properties of the platform for compatibility.)\n>\n>> --- a/repo-settings.c\n>> +++ b/repo-settings.c\n>> @@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r)\n>> \t\tr->settings.core_multi_pack_index = value;\n>> \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n>> \n>> +\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n>> +\t\tr->settings.use_builtin_fsmonitor = 1;\n>> +\n>\n> Follows the patterns of repo settings. Good.\n\nIt follows the pattern, but as an aside the pattern seems bit odd. I see\nit dates back to your 7211b9e7534 (repo-settings: consolidate some\nconfig settings, 2019-08-13).\n\nI.e. we memset() the whole thing to -1, then for most things do something like:\n\n if (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n r->settings.gc_write_commit_graph = value;\n UPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n\nBut could do:\n\n if (repo_config_get_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph))\n r->settings.gc_write_commit_graph = 1;\n\nNo? I.e. the repo_config_get_bool() function already returns non-zero if\nwe don't find it in the config.\n\nI see the UPDATE_DEFAULT_BOOL() macro has also drifted from \"set thing\ndefault boolean\" to \"set any default value\".\n"},{"id":"423103","messageId":"03b4def9-3435-bd44-69f9-1e037df83aaa@gmail.com","threadId":"55420","inReplyTo":"87mttkyrqq.fsf@evledraar.gmail.com","subject":"Re: [PATCH 04/23] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T12:42:49Z","receivedAt":"2021-04-27T12:42:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/27/2021 5:20 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Apr 26 2021, Derrick Stolee wrote:\n> \n>> On 4/1/21 11:40 AM, Johannes Schindelin via GitGitGadget wrote:> @@ -2515,6 +2515,11 @@ int git_config_get_max_percent_split_change(void)\n...\n>>> --- a/repo-settings.c\n>>> +++ b/repo-settings.c\n>>> @@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r)\n>>> \t\tr->settings.core_multi_pack_index = value;\n>>> \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n>>> \n>>> +\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n>>> +\t\tr->settings.use_builtin_fsmonitor = 1;\n>>> +\n>>\n>> Follows the patterns of repo settings. Good.\n> \n> It follows the pattern, but as an aside the pattern seems bit odd. I see\n> it dates back to your 7211b9e7534 (repo-settings: consolidate some\n> config settings, 2019-08-13).\n> \n> I.e. we memset() the whole thing to -1, then for most things do something like:\n> \n> if (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n> r->settings.gc_write_commit_graph = value;\n> UPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n> \n> But could do:\n> \n> if (repo_config_get_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph))\n> r->settings.gc_write_commit_graph = 1;\n> \n> No? I.e. the repo_config_get_bool() function already returns non-zero if\n> we don't find it in the config.\n\nI see how this is fewer lines of code, but it is harder to read the intent\nof the implementation. The current layout makes it clear that we set the\nvalue from the config, if it exists, but otherwise we choose a default.\n\nSometimes, this choice of a default _needs_ to be deferred, for example with\nthe fetch_negotiation_algorithm setting, which can be set both from the\nfetch.negotiationAlgorithm config, but also the feature.experimental config.\n\nHowever, perhaps it would be better still for these one-off requests to\ncreate a new macro, say USE_CONFIG_OR_DEFAULT_BOOL() that fills a value\nfrom config _or_ sets the given default:\n\n#define USE_CONFIG_OR_DEFAULT_BOOL(r, v, s, d) \\\n\tif (repo_config_get_bool(r, s, &v)) \\\n\t\tv = d\n\nAnd then for this example we would write\n\n\tUSE_CONFIG_OR_DEFAULT_BOOL(r, r->settings.core_commit_graph,\n\t\t\t\t \"core.commitgraph\", 1);\n\nThis would work for multiple config options in this file.\n\n> I see the UPDATE_DEFAULT_BOOL() macro has also drifted from \"set thing\n> default boolean\" to \"set any default value\".\n \nThis is correct. I suppose it would be a good change to make some time.\nSuch a rename could be combined with the refactor above.\n\nI would recommend waiting until such a change isn't conflicting with\nongoing topics, such as this one.\n\nThanks,\n-Stolee\n"},{"id":"423104","messageId":"1f4a4de9-4067-8608-0379-84098c9b404f@gmail.com","threadId":"55420","inReplyTo":"9f263e70c7245a01210ac743ce3dfda4dcc08308.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 17/23] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T13:24:38Z","receivedAt":"2021-04-27T13:24:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach fsmonitor--daemon to periodically truncate the list of\n> modified files to save some memory.\n> \n> Clients will ask for the set of changes relative to a token that they\n> found in the FSMN index extension in the index. (This token is like a\n> point in time, but different). Clients will then update the index to\n> contain the response token (so that subsequent commands will be\n> relative to this new token).\n> \n> Therefore, the daemon can gradually truncate the in-memory list of\n> changed paths as they become obsolete (older that the previous token).\n\ns/older that/older than/\n\n> Since we may have multiple clients making concurrent requests with a\n> skew of tokens and clients may be racing to the talk to the daemon,\n> we lazily truncate the list.\n> \n> We introduce a 5 minute delay and truncate batches 5 minutes after\n> they are considered obsolete.\n\n5 minutes seems like a good default timeframe. We can consider\nmaking this customizable in the future.\n\n> +/*\n> + * To keep the batch list from growing unbounded in response to filesystem\n> + * activity, we try to truncate old batches from the end of the list as\n> + * they become irrelevant.\n> + *\n> + * We assume that the .git/index will be updated with the most recent token\n> + * any time the index is updated. And future commands will only ask for\n> + * recent changes *since* that new token. So as tokens advance into the\n> + * future, older batch items will never be requested/needed. So we can\n> + * truncate them without loss of functionality.\n> + *\n> + * However, multiple commands may be talking to the daemon concurrently\n> + * or perform a slow command, so a little \"token skew\" is possible.\n> + * Therefore, we want this to be a little bit lazy and have a generous\n> + * delay.\n\nI appreciate this documentation of the \"expected\" behavior and how it\ncompares to the \"possible\" behavior.\n\n> + * The current reader thread walked backwards in time from `token->batch_head`\n> + * back to `batch_marker` somewhere in the middle of the batch list.\n> + *\n> + * Let's walk backwards in time from that marker an arbitrary delay\n> + * and truncate the list there. Note that these timestamps are completely\n> + * artificial (based on when we pinned the batch item) and not on any\n> + * filesystem activity.\n> + */\n> +#define MY_TIME_DELAY (5 * 60) /* seconds */\n\nPerhaps put the units into the macro? MY_TIME_DELAY_SECONDS?\n> +static void fsmonitor_batch__truncate(struct fsmonitor_daemon_state *state,\n> +\t\t\t\t const struct fsmonitor_batch *batch_marker)\n> +{\n> +\t/* assert state->main_lock */\n\nIf this comment is intended to be a warning for consumers that they should\nhave the lock around this method, then maybe that should be in a documentation\ncomment above the method declaration.\n\n> +\tconst struct fsmonitor_batch *batch;\n> +\tstruct fsmonitor_batch *rest;\n> +\tstruct fsmonitor_batch *p;\n> +\ttime_t t;\n\nThis is only used within the for loop, so it could be defined there.\n\n> +\n> +\tif (!batch_marker)\n> +\t\treturn;\n> +\n> +\ttrace_printf_key(&trace_fsmonitor, \"TRNC mark (%\"PRIu64\",%\"PRIu64\")\",\n\nWhat's the value of abbreviating \"truncate\" like this? Is there a special\nreason?\n\n> +\t\t\t batch_marker->batch_seq_nr,\n> +\t\t\t (uint64_t)batch_marker->pinned_time);\n> +\n> +\tfor (batch = batch_marker; batch; batch = batch->next) {\n> +\t\tif (!batch->pinned_time) /* an overflow batch */\n> +\t\t\tcontinue;\n> +\n> +\t\tt = batch->pinned_time + MY_TIME_DELAY;\n> +\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n> +\t\t\tcontinue;> +\n> +\t\tgoto truncate_past_here;\n> +\t}\n> +\n> +\treturn;\n> +\n> +truncate_past_here:\n> +\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n> +\n> +\trest = ((struct fsmonitor_batch *)batch)->next;\n> +\t((struct fsmonitor_batch *)batch)->next = NULL;\n> +\n> +\tfor (p = rest; p; p = fsmonitor_batch__free(p)) {\n> +\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t \"TRNC kill (%\"PRIu64\",%\"PRIu64\")\",\n> +\t\t\t\t p->batch_seq_nr, (uint64_t)p->pinned_time);\n> +\t}\n\nI see that you are not using the method that frees the entire list so\nyou can trace each entry as it is deleted. That works.\n\n> +}\n> +\n> static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n> {\n> \tstruct fsmonitor_batch *p;\n> @@ -647,6 +716,15 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n> \t\t\t * that work.\n> \t\t\t */\n> \t\t\tfsmonitor_free_token_data(token_data);\n> +\t\t} else if (batch) {\n> +\t\t\t/*\n> +\t\t\t * This batch is the first item in the list\n> +\t\t\t * that is older than the requested sequence\n> +\t\t\t * number and might be considered to be\n> +\t\t\t * obsolete. See if we can truncate the list\n> +\t\t\t * and save some memory.\n> +\t\t\t */\n> +\t\t\tfsmonitor_batch__truncate(state, batch);\n\nSeems to work as advertised.\n\nThanks,\n-Stolee\n"},{"id":"423105","messageId":"e09104e9-a75c-e7ec-09cf-18dae9b56a23@gmail.com","threadId":"55420","inReplyTo":"c6d5f045fb5644306a3676e5fa4145ba4c6e9b93.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 18/23] fsmonitor--daemon:: introduce client delay for testing","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T13:36:21Z","receivedAt":"2021-04-27T13:36:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:41 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Define GIT_TEST_FSMONITOR_CLIENT_DELAY as a millisecond delay.\n\nThis is a second delay introduced in this feature, but the units\nare different. Could we put a unit in the name? Perhaps a \"_MS\"\nsuffix.\n\n> Introduce an artificial delay when processing client requests.\n> This make the CI/PR test suite a little more stable and avoids\n> the need to load up test scripts with sleep statements to avoid\n> racy failures. This was mostly seen on 1 or 2 core CI build\n> machines where the test script would create a file and quickly\n> try to confirm that the daemon had seen it *before* the daemon\n> had received the kernel event and causing a test failure.\n\nIsn't the cookie file supposed to prevent this from happening?\n\nYes, our test suite interacts with the filesystem and Git commands\nmore quickly than a human user would, but Git is used all the time\nby scripts or build machines to quickly process data. The FS\nMonitor feature should be robust to such a situation.\n\nI feel that as currently described, this patch is only hiding a\nbug that shows up during heavy use.\n\nPerhaps the test failures are limited to a small number of\nspecific tests that are checking the FS Monitor daemon in a\nnon-standard way, especially in a way that circumvents the\ncookie file. In this case, I'd like to see _in this patch_ how\nthe environment variable is used in the test suite.\n\nI understand that it is difficult to simultaneously build a new\nfeature like this in small increments, but the biggest issue I\nhave with the series' organization so far is that we are 18\npatches deep and I still haven't seen a single test. This is\na case where I think this only serves the purpose of the test\nsuite, so it would be good to delay until only seeing its value\nin a test script.\n\nLooking ahead, I see that you insert it as a blanket statement\nin the t7527 test script, which seems like it has potential to\nhide bugs instead of being an isolated cover for a specific\ninteraction.\n\nAs for the code, it all looks correct. However, please update\nt/README with a description of the new GIT_TEST_* variable.\n\nThanks,\n-Stolee\n"},{"id":"423106","messageId":"5669d313-4084-dceb-1204-acd005e58657@gmail.com","threadId":"55420","inReplyTo":"038b62dc6744e8d8004f3f8423a40066821c0f1b.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 19/23] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T14:23:54Z","receivedAt":"2021-04-27T14:24:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:41 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach fsmonitor--daemon client threads to create a cookie file\n> inside the .git directory and then wait until FS events for the\n> cookie are observed by the FS listener thread.\n> \n> This helps address the racy nature of file system events by\n> blocking the client response until the kernel has drained any\n> event backlog.\n\nThis description matches my expectation of the cookie file,\nwhich furthers my confusion about GIT_TEST_FSMONITOR_CLIENT_DELAY.\n\n> +enum fsmonitor_cookie_item_result {\n> +\tFCIR_ERROR = -1, /* could not create cookie file ? */\n> +\tFCIR_INIT = 0,\n> +\tFCIR_SEEN,\n> +\tFCIR_ABORT,\n> +};\n> +\n> +struct fsmonitor_cookie_item {\n> +\tstruct hashmap_entry entry;\n> +\tconst char *name;\n> +\tenum fsmonitor_cookie_item_result result;\n> +};\n> +\n> +static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n> +\t\t const struct hashmap_entry *he2, const void *keydata)\n\nI'm interested to see why a hashset is necessary.\n\n> +static enum fsmonitor_cookie_item_result fsmonitor_wait_for_cookie(\n> +\tstruct fsmonitor_daemon_state *state)\n> +{\n> +\tint fd;\n> +\tstruct fsmonitor_cookie_item cookie;\n> +\tstruct strbuf cookie_pathname = STRBUF_INIT;\n> +\tstruct strbuf cookie_filename = STRBUF_INIT;\n> +\tconst char *slash;\n> +\tint my_cookie_seq;\n> +\n> +\tpthread_mutex_lock(&state->main_lock);\n\nHm. We are entering a locked region. I hope this is only for the\ncookie write and not the entire waiting period.\n\n> +\tmy_cookie_seq = state->cookie_seq++;\n> +\n> +\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n> +\tstrbuf_addf(&cookie_pathname, \"%i-%i\", getpid(), my_cookie_seq);\n> +\n> +\tslash = find_last_dir_sep(cookie_pathname.buf);\n> +\tif (slash)\n> +\t\tstrbuf_addstr(&cookie_filename, slash + 1);\n> +\telse\n> +\t\tstrbuf_addbuf(&cookie_filename, &cookie_pathname);\n\nThis business about the slash-or-not-slash is good defensive\nprogramming. I imagine the only possible way for there to not\nbe a slash is if the Git process is running with the .git\ndirectory as its working directory?\n\n> +\tcookie.name = strbuf_detach(&cookie_filename, NULL);\n> +\tcookie.result = FCIR_INIT;\n> +\t// TODO should we have case-insenstive hash (and in cookie_cmp()) ??\n\nThis TODO comment should be cleaned up. Doesn't match C-style, either.\n\nAs for the question, I believe that we can limit ourselves to names that\ndon't need case-insensitive hashes and trust that the filesystem will not\nchange the case. Using lowercase letters should help with this.\n\n> +\thashmap_entry_init(&cookie.entry, strhash(cookie.name));\n> +\n> +\t/*\n> +\t * Warning: we are putting the address of a stack variable into a\n> +\t * global hashmap. This feels dodgy. We must ensure that we remove\n> +\t * it before this thread and stack frame returns.\n> +\t */\n> +\thashmap_add(&state->cookies, &cookie.entry);\n\nI saw this warning and thought about avoiding it by using the heap, but\neven with a heap pointer we need to be careful to remove the result\nbefore returning and stopping the thread.\n\nHowever, there is likely a higher potential of a bug leading to a\nsecurity issue through an error causing stack corruption and unsafe\ncode execution. Perhaps it is worth converting to using heap data here.\n\n> +\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n> +\t\t\t cookie.name, cookie_pathname.buf);\n> +\n> +\t/*\n> +\t * Create the cookie file on disk and then wait for a notification\n> +\t * that the listener thread has seen it.\n> +\t */\n> +\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n> +\tif (fd >= 0) {\n> +\t\tclose(fd);\n> +\t\tunlink_or_warn(cookie_pathname.buf);\n\nInteresting that we are ignoring the warning here. Is it possible that\nthese cookie files will continue to grow if this unlink fails?\n\n> +\n> +\t\twhile (cookie.result == FCIR_INIT)\n> +\t\t\tpthread_cond_wait(&state->cookies_cond,\n> +\t\t\t\t\t &state->main_lock);\n\nOk, we are waiting here for another thread to signal that the cookie\nfile has been found in the events. What happens if the event gets lost?\nI'll look for a later signal that cookie.result can change based on a\ntimeout, too.\n\n> +\n> +\t\thashmap_remove(&state->cookies, &cookie.entry, NULL);\n> +\t} else {\n> +\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n> +\t\t\t cookie.name);\n> +\n> +\t\tcookie.result = FCIR_ERROR;\n> +\t\thashmap_remove(&state->cookies, &cookie.entry, NULL);\n> +\t}\n\nBoth blocks here remove the cookie entry, so move it to the end of the\nmethod with the other cleanups.\n\n> +\n> +\tpthread_mutex_unlock(&state->main_lock);\n\nHm. We are locking the main state throughout this process. I suppose that\nthe listener thread could be watching multiple repos and updating them\nwhile we wait here for one repo to update. This is a larger lock window\nthan I was hoping for, but I don't currently see how to reduce it safely.\n\n> +\n> +\tfree((char*)cookie.name);\n> +\tstrbuf_release(&cookie_pathname);\n> +\treturn cookie.result;\n\nRemove the cookie from the hashset along with these lines.\n\n> +}\n> +\n> +/*\n> + * Mark these cookies as _SEEN and wake up the corresponding client threads.\n> + */\n> +static void fsmonitor_cookie_mark_seen(struct fsmonitor_daemon_state *state,\n> +\t\t\t\t const struct string_list *cookie_names)\n> +{\n> +\t/* assert state->main_lock */\n\nI'm now confused what this is trying to document. The 'state' should be\nlocked by another thread while we are waiting for a cookie response, so\nthis method is updating the cookie as seen from a different thread that\ndoesn't have the lock.\n\n...\n> +/*\n> + * Set _ABORT on all pending cookies and wake up all client threads.\n> + */\n> +static void fsmonitor_cookie_abort_all(struct fsmonitor_daemon_state *state)\n...\n\n> + * [2] Some of those lost events may have been for cookie files. We\n> + * should assume the worst and abort them rather letting them starve.\n> + *\n> * If there are no readers of the the current token data series, we\n> * can free it now. Otherwise, let the last reader free it. Either\n> * way, the old token data series is no longer associated with our\n> @@ -454,6 +600,8 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n> \t\t\t state->current_token_data->token_id.buf,\n> \t\t\t new_one->token_id.buf);\n> \n> +\tfsmonitor_cookie_abort_all(state);\n> +\n\nI see we abort here if we force a resync. I lost the detail of whether\nthis is triggered by a timeout, too.\n\n> @@ -654,6 +803,39 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n> \t\tgoto send_trivial_response;\n> \t}\n> \n> +\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\t/*\n> +\t * Write a cookie file inside the directory being watched in an\n> +\t * effort to flush out existing filesystem events that we actually\n> +\t * care about. Suspend this client thread until we see the filesystem\n> +\t * events for this cookie file.\n> +\t */\n> +\tcookie_result = fsmonitor_wait_for_cookie(state);\n\nOdd that we unlock before calling this method, then just take the lock\nagain inside of it.\n\n> +\tif (cookie_result != FCIR_SEEN) {\n> +\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n> +\t\t cookie_result);\n> +\t\tresult = 0;\n> +\t\tgoto send_trivial_response;\n> +\t}\n> +\n> +\tpthread_mutex_lock(&state->main_lock);\n> +\n> +\tif (strcmp(requested_token_id.buf,\n> +\t\t state->current_token_data->token_id.buf)) {\n> +\t\t/*\n> +\t\t * Ack! The listener thread lost sync with the filesystem\n> +\t\t * and created a new token while we were waiting for the\n> +\t\t * cookie file to be created! Just give up.\n> +\t\t */\n> +\t\tpthread_mutex_unlock(&state->main_lock);\n> +\n> +\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t \"lost filesystem sync\");\n> +\t\tresult = 0;\n> +\t\tgoto send_trivial_response;\n> +\t}\n> +\n> \t/*\n> \t * We're going to hold onto a pointer to the current\n> \t * token-data while we walk the list of batches of files.\n> @@ -982,6 +1164,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n> \t\t}\n> \t}\n> \n> +\tif (cookie_names->nr)\n> +\t\tfsmonitor_cookie_mark_seen(state, cookie_names);\n> +\n\nI was confused as to what updates 'cookie_names', but it appears that\nthese are updated in the platform-specific code. That seems to happen\nin later patches.\n\nThanks,\n-Stolee\n"},{"id":"423108","messageId":"91ee1b83-7db6-1578-9725-b3ae263ea023@gmail.com","threadId":"55420","inReplyTo":"d699ad597d2c40cd2c0fe8abbf75b5386d4cf0f2.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 20/23] fsmonitor: force update index when fsmonitor token advances","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T14:52:09Z","receivedAt":"2021-04-27T14:52:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:41 AM, Jeff Hostetler via GitGitGadget wrote:...\n> +\t/*\n> +\t * If the fsmonitor response and the subsequent scan of the disk\n> +\t * did not cause the in-memory index to be marked dirty, then force\n> +\t * it so that we advance the fsmonitor token in our extension, so\n> +\t * that future requests don't keep re-requesting the same range.\n> +\t */\n> +\tif (istate->fsmonitor_last_update &&\n> +\t strcmp(istate->fsmonitor_last_update, last_update_token.buf))\n> +\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n> +\n\nThis could lead to extra index writes that don't normally happen in\nthe case without the FS Monitor feature. I'm particularly sensitive\nto this because of my sparse-index work is trying to solve for the\nI/O cost of large indexes, but perhaps this cost is worth the benefit.\n\nI'll keep an eye out as I do performance testing.\n\nThanks,\n-Stolee\n"},{"id":"423111","messageId":"b62694df-7dd5-71fa-304d-576c2550e5d2@gmail.com","threadId":"55420","inReplyTo":"8b2280e5c4d2cec1fe39e90bfc93f059a1d0eb05.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 21/23] t7527: create test for fsmonitor--daemon","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T15:41:10Z","receivedAt":"2021-04-27T15:41:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:41 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n\nIt might be nice to summarize the testing strategy here. Are these just\nthe basics? Is this a full list of every conceivable client/server\ninteraction? Do some platforms need special tests?\n\n> +# Ask the fsmonitor daemon to insert a little delay before responding to\n> +# client commands like `git status` and `git fsmonitor--daemon --query` to\n> +# allow recent filesystem events to be received by the daemon. This helps\n> +# the CI/PR builds be more stable.\n> +#\n> +# An arbitrary millisecond value.\n> +#\n> +GIT_TEST_FSMONITOR_CLIENT_DELAY=1000\n> +export GIT_TEST_FSMONITOR_CLIENT_DELAY\n\nAs I mentioned before, this seems like it is hiding a bug, especially\nbecause of a full second delay. But even a 1 millisecond delay seems\nlike it is incorrect to assume this feature works correctly if the\ntest requires this delay.\n\nIf there is a specific interaction that has issues, then it might be\nvalid to insert this delay in a specific test or two.\n\n> +git version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\" || {\n> +\tskip_all=\"The built-in FSMonitor is not supported on this platform\"\n> +\ttest_done\n> +}\n\nI see some precedent of this pattern, but it might be nice to instead\nregister a prereq and then test for the prereq here in the test script.\n\n> +kill_repo () {\n\nPerhaps \"kill_repo_daemon\" might be more specific?\n\n> +\tr=$1\n> +\tgit -C $r fsmonitor--daemon --stop >/dev/null 2>/dev/null\n> +\trm -rf $1\n> +\treturn 0\n> +}\n> +\n> +start_daemon () {\n> +\tcase \"$#\" in\n> +\t\t1) r=\"-C $1\";;\n> +\t\t*) r=\"\";\n> +\tesac\n> +\n> +\tgit $r fsmonitor--daemon --start || return $?\n> +\tgit $r fsmonitor--daemon --is-running || return $?\n\nPerhaps add 'test_when_finished kill_repo \"$r\"' as a line here so\nconsumers don't need to do it themselves.\n\n> +\treturn 0\n> +}\n> +\n> +test_expect_success 'explicit daemon start and stop' '\n> +\ttest_when_finished \"kill_repo test_explicit\" &&\n> +\n> +\tgit init test_explicit &&\n> +\tstart_daemon test_explicit &&\n> +\n> +\tgit -C test_explicit fsmonitor--daemon --stop &&\n> +\ttest_must_fail git -C test_explicit fsmonitor--daemon --is-running\n> +'\n\nThis is an example of a test that could have been created as early as\npatch 09/23.\n\n> +test_expect_success 'implicit daemon start' '\n> +\ttest_when_finished \"kill_repo test_implicit\" &&\n> +\n> +\tgit init test_implicit &&\n> +\ttest_must_fail git -C test_implicit fsmonitor--daemon --is-running &&\n> +\n> +\t# query will implicitly start the daemon.\n> +\t#\n> +\t# for test-script simplicity, we send a V1 timestamp rather than\n> +\t# a V2 token. either way, the daemon response to any query contains\n> +\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n> +\t# but this test case is only concerned with whether the daemon was\n> +\t# implicitly started.)\n> +\n> +\tGIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n> +\t\tgit -C test_implicit fsmonitor--daemon --query 0 >actual &&\n> +\tnul_to_q actual.filtered &&\n> +\tgrep \"builtin:\" actual.filtered &&\n> +\n> +\t# confirm that a daemon was started in the background.\n> +\t#\n> +\t# since the mechanism for starting the background daemon is platform\n> +\t# dependent, just confirm that the foreground command received a\n> +\t# response from the daemon.\n> +\n> +\tgrep :\\\"query/response-length\\\" .git/trace &&\n> +\n> +\tgit -C test_implicit fsmonitor--daemon --is-running &&\n> +\tgit -C test_implicit fsmonitor--daemon --stop &&\n> +\ttest_must_fail git -C test_implicit fsmonitor--daemon --is-running\n> +'\n> +\n> +test_expect_success 'implicit daemon stop (delete .git)' '\n> +\ttest_when_finished \"kill_repo test_implicit_1\" &&\n> +\n> +\tgit init test_implicit_1 &&\n> +\n> +\tstart_daemon test_implicit_1 &&\n> +\n> +\t# deleting the .git directory will implicitly stop the daemon.\n> +\trm -rf test_implicit_1/.git &&\n> +\n> +\t# Create an empty .git directory so that the following Git command\n> +\t# will stay relative to the `-C` directory. Without this, the Git\n> +\t# command will (override the requested -C argument) and crawl out\n\nWhy the parentheses here?\n\n> +\t# to the containing Git source tree. This would make the test\n> +\t# result dependent upon whether we were using fsmonitor on our\n> +\t# development worktree.\n> +\n> +\tsleep 1 &&\n\nI can understand this sleep, as we are waiting for a background process\nto end in response to a directory being deleted.\n\nI'm surprised this works on Windows! I recall having issues deleting\nrepos that are being watched by Watchman.\n\n> +\tmkdir test_implicit_1/.git &&\n> +\n> +\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon --is-running\n> +'\n> +\n> +test_expect_success 'implicit daemon stop (rename .git)' '\n> +\ttest_when_finished \"kill_repo test_implicit_2\" &&\n> +\n> +\tgit init test_implicit_2 &&\n> +\n> +\tstart_daemon test_implicit_2 &&\n> +\n> +\t# renaming the .git directory will implicitly stop the daemon.\n> +\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n> +\n> +\t# Create an empty .git directory so that the following Git command\n> +\t# will stay relative to the `-C` directory. Without this, the Git\n> +\t# command will (override the requested -C argument) and crawl out\n> +\t# to the containing Git source tree. This would make the test\n> +\t# result dependent upon whether we were using fsmonitor on our\n> +\t# development worktree.\n> +\n> +\tsleep 1 &&\n> +\tmkdir test_implicit_2/.git &&\n> +\n> +\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon --is-running\n> +'\n> +\n> +test_expect_success 'cannot start multiple daemons' '\n> +\ttest_when_finished \"kill_repo test_multiple\" &&\n> +\n> +\tgit init test_multiple &&\n> +\n> +\tstart_daemon test_multiple &&\n> +\n> +\ttest_must_fail git -C test_multiple fsmonitor--daemon --start 2>actual &&\n> +\tgrep \"fsmonitor--daemon is already running\" actual &&\n> +\n> +\tgit -C test_multiple fsmonitor--daemon --stop &&\n> +\ttest_must_fail git -C test_multiple fsmonitor--daemon --is-running\n> +'\n\nThe tests above seem like they could be inserted as soon as the\nplatform-specific listeners are created. None of this requires the\nlinked-list of batched updates or cookie file checks.\n\n> +test_expect_success 'setup' '\n> +\t>tracked &&\n> +\t>modified &&\n> +\t>delete &&\n> +\t>rename &&\n> +\tmkdir dir1 &&\n> +\t>dir1/tracked &&\n> +\t>dir1/modified &&\n> +\t>dir1/delete &&\n> +\t>dir1/rename &&\n> +\tmkdir dir2 &&\n> +\t>dir2/tracked &&\n> +\t>dir2/modified &&\n> +\t>dir2/delete &&\n> +\t>dir2/rename &&\n> +\tmkdir dirtorename &&\n> +\t>dirtorename/a &&\n> +\t>dirtorename/b &&\n> +\n> +\tcat >.gitignore <<-\\EOF &&\n> +\t.gitignore\n> +\texpect*\n> +\tactual*\n> +\tEOF\n> +\n> +\tgit -c core.useBuiltinFSMonitor= add . &&\n> +\ttest_tick &&\n> +\tgit -c core.useBuiltinFSMonitor= commit -m initial &&\n> +\n> +\tgit config core.useBuiltinFSMonitor true\n> +'\n\nNow we are getting into the meat of the interactions with Git\nfeatures. I can understand these not being ready until all of\nthe previous product patches are in place.\n\n> +test_expect_success 'update-index implicitly starts daemon' '\n> +\ttest_must_fail git fsmonitor--daemon --is-running &&\n> +\n> +\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n> +\t\tgit update-index --fsmonitor &&\n> +\n> +\tgit fsmonitor--daemon --is-running &&\n> +\ttest_might_fail git fsmonitor--daemon --stop &&\n\nShould this be a \"test_when_finished kill_repo .\" at the\nbeginning of the test?\n\n> +\n> +\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_1\n> +'\n> +\n> +test_expect_success 'status implicitly starts daemon' '\n> +\ttest_must_fail git fsmonitor--daemon --is-running &&\n> +\n> +\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n> +\t\tgit status >actual &&\n> +\n> +\tgit fsmonitor--daemon --is-running &&\n> +\ttest_might_fail git fsmonitor--daemon --stop &&\n> +\n> +\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_2\n> +'\n> +\n> +edit_files() {\n> +\techo 1 >modified\n> +\techo 2 >dir1/modified\n> +\techo 3 >dir2/modified\n> +\t>dir1/untracked\n> +}\n> +\n> +delete_files() {\n> +\trm -f delete\n> +\trm -f dir1/delete\n> +\trm -f dir2/delete\n> +}\n> +\n> +create_files() {\n> +\techo 1 >new\n> +\techo 2 >dir1/new\n> +\techo 3 >dir2/new\n> +}\n> +\n> +rename_files() {\n> +\tmv rename renamed\n> +\tmv dir1/rename dir1/renamed\n> +\tmv dir2/rename dir2/renamed\n> +}\n> +\n> +file_to_directory() {\n> +\trm -f delete\n> +\tmkdir delete\n> +\techo 1 >delete/new\n> +}\n> +\n> +directory_to_file() {\n> +\trm -rf dir1\n> +\techo 1 >dir1\n> +}\n> +\n> +verify_status() {\n> +\tgit status >actual &&\n> +\tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n> +\tGIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect &&\n> +\ttest_cmp expect actual &&\n> +\techo HELLO AFTER &&\n> +\tcat .git/trace &&\n> +\techo HELLO AFTER\n> +}\n> +\n> +# The next few test cases confirm that our fsmonitor daemon sees each type\n> +# of OS filesystem notification that we care about. At this layer we just\n> +# ensure we are getting the OS notifications and do not try to confirm what\n> +# is reported by `git status`.\n> +#\n> +# We run a simple query after modifying the filesystem just to introduce\n> +# a bit of a delay so that the trace logging from the daemon has time to\n> +# get flushed to disk.\n> +#\n> +# We `reset` and `clean` at the bottom of each test (and before stopping the\n> +# daemon) because these commands might implicitly restart the daemon.\n> +\n> +clean_up_repo_and_stop_daemon () {\n> +\tgit reset --hard HEAD\n> +\tgit clean -fd\n> +\tgit fsmonitor--daemon --stop\n> +\trm -f .git/trace\n> +}\n> +\n> +test_expect_success 'edit some files' '\n> +\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n\nDo you need the quotes here?\n\n> +\n> +\t(\n> +\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n\nUse \"$(pwd)/.git/trace\". There are some strange things with $PWD\nespecially on Windows.\n\n> +\t\texport GIT_TRACE_FSMONITOR &&\n> +\n> +\t\tstart_daemon\n> +\t) &&\n> +\n> +\tedit_files &&\n> +\n> +\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n> +\n> +\tgrep \"^event: dir1/modified$\" .git/trace &&\n> +\tgrep \"^event: dir2/modified$\" .git/trace &&\n> +\tgrep \"^event: modified$\" .git/trace &&\n> +\tgrep \"^event: dir1/untracked$\" .git/trace\n> +'\n> +\n> +test_expect_success 'create some files' '\n> +\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n> +\n> +\t(\n> +\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n> +\t\texport GIT_TRACE_FSMONITOR &&\n> +\n> +\t\tstart_daemon\n> +\t) &&\n> +\n> +\tcreate_files &&\n> +\n> +\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n> +\n> +\tgrep \"^event: dir1/new$\" .git/trace &&\n> +\tgrep \"^event: dir2/new$\" .git/trace &&\n> +\tgrep \"^event: new$\" .git/trace\n> +'\n\nI wonder if we can scan the trace for the number of events\nand ensure we have the right count, to ensure we aren't getting\n_extra_ events that we don't want?\n\nThe rest of the tests seem similarly structured and testing\nimportant cases. I'll delay thinking of new tests until I see\nthe rest of the tests you are adding.\n\nThanks,\n-Stolee\n"},{"id":"423113","messageId":"e44ea355-5e58-bbf4-12a3-7653278da2d6@gmail.com","threadId":"55420","inReplyTo":"da5094e52032e28337cfcecd421dedb560952869.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 22/23] p7519: add fsmonitor--daemon","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T15:45:52Z","receivedAt":"2021-04-27T15:45:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:41 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Repeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\n> the \"Simple IPC\" interface.\n\nIt would be nice to see some numbers for how this test performs\non some standard Git repositories across Windows and macOS.\n\n> Signed-off-by: Jeff Hostetler \n> ---\n> t/perf/p7519-fsmonitor.sh | 37 +++++++++++++++++++++++++++++++++++--\n> 1 file changed, 35 insertions(+), 2 deletions(-)\n> \n> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n> index 5eb5044a103c..2d018bc7d589 100755\n> --- a/t/perf/p7519-fsmonitor.sh\n> +++ b/t/perf/p7519-fsmonitor.sh\n> @@ -24,7 +24,8 @@ test_description=\"Test core.fsmonitor\"\n> # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\n> # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an\n> # absolute path to an integration. May be a space delimited list of\n> -# absolute paths to integrations.\n> +# absolute paths to integrations. (This hook or list of hooks does not\n> +# include the built-in fsmonitor--daemon.)\n> #\n> # The big win for using fsmonitor is the elimination of the need to scan the\n> # working directory looking for changed and untracked files. If the file\n> @@ -135,10 +136,16 @@ test_expect_success \"one time repo setup\" '\n> \n> setup_for_fsmonitor() {\n> \t# set INTEGRATION_SCRIPT depending on the environment\n> -\tif test -n \"$INTEGRATION_PATH\"\n> +\tif test -n \"$USE_FSMONITOR_DAEMON\"\n> \tthen\n> +\t\tgit config core.useBuiltinFSMonitor true &&\n> +\t\tINTEGRATION_SCRIPT=false\n> +\telif test -n \"$INTEGRATION_PATH\"\n> +\tthen\n> +\t\tgit config core.useBuiltinFSMonitor false &&\n> \t\tINTEGRATION_SCRIPT=\"$INTEGRATION_PATH\"\n> \telse\n> +\t\tgit config core.useBuiltinFSMonitor false &&\n> \t\t#\n> \t\t# Choose integration script based on existence of Watchman.\n> \t\t# Fall back to an empty integration script.\n> @@ -285,4 +292,30 @@ test_expect_success \"setup without fsmonitor\" '\n> test_fsmonitor_suite\n> trace_stop\n> \n> +#\n> +# Run a full set of perf tests using the built-in fsmonitor--daemon.\n> +# It does not use the Hook API, so it has a different setup.\n> +# Explicitly start the daemon here and before we start client commands\n> +# so that we can later add custom tracing.\n> +#\n> +\n> +test_lazy_prereq HAVE_FSMONITOR_DAEMON '\n> +\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n> +'\n\nHere you do create the prereq. Let's put this into t/test-lib.sh\nor t/test-lib-functions.sh, whichever is more appropriate.\n\n> +\n> +if test_have_prereq HAVE_FSMONITOR_DAEMON\n> +then\n> +\tUSE_FSMONITOR_DAEMON=t\n> +\n> +\ttrace_start fsmonitor--daemon--server\n> +\tgit fsmonitor--daemon --start\n> +\n> +\ttrace_start fsmonitor--daemon--client\n> +\ttest_expect_success \"setup for fsmonitor--daemon\" 'setup_for_fsmonitor'\n\nMaybe this is copied from the rest of the file, but we should probably\nuse the standard layout for tests here:\n\n\ttest_expect_success 'setup for fsmonitor--daemon' '\n\t\tsetup_for_fsmonitor\n\t'\n\n> +\ttest_fsmonitor_suite\n> +\n> +\tgit fsmonitor--daemon --stop\n> +\ttrace_stop\n> +fi\n> +\n\nThanks,\n-Stolee\n"},{"id":"423114","messageId":"14045741-64ea-5274-ae6f-aec832272024@gmail.com","threadId":"55420","inReplyTo":"3eafd0b5cb09dbba8af2abf1a8d602ad27ad3bc4.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 23/23] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T15:51:17Z","receivedAt":"2021-04-27T15:51:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:41 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Create 2x2 test matrix with the untracked-cache and fsmonitor--daemon\n> features and a series of edits and verify that status output is\n> identical.\n\nI value the detail here. It also signals that there is something\ninteresting going on with the untracked cache, which I have also\ndiscovered in my testing of this feature. I'll follow up in a\nresponse to the cover letter.\n\nThanks,\n-Stolee\n"},{"id":"423116","messageId":"4b0066f5-5962-e57e-5987-5358f3d394f7@gmail.com","threadId":"55420","inReplyTo":"a57ddb3bc7cc2de9a48472c4faff32ef46132f73.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 13/23] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T17:22:30Z","receivedAt":"2021-04-27T17:22:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach the win32 backend to register a watch on the working tree\n> root directory (recursively). Also watch the if it is\n> not inside the working tree. And to collect path change notifications\n> into batches and publish.\n\nIs it valuable to list the important API methods here for an interested\nreader to discover them? Perhaps using links to the docs [1] might be\ntoo ephemeral, in case those URLs stop being valid.\n\nIn any case, here are the URLs I found helpful:\n\n[1] https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-readdirectorychangesw\n[2] https://docs.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-getoverlappedresult\n[3] https://docs.microsoft.com/en-us/windows/win32/fileio/cancelioex-func\n[4] https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-resetevent\n[5] https://docs.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-file_notify_information\n[6] https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-waitformultipleobjects\n\n> Signed-off-by: Jeff Hostetler \n> ---\n> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 493 +++++++++++++++++++\n> 1 file changed, 493 insertions(+)\n> \n> diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> index 880446b49e35..2f1fcf85a0a4 100644\n> --- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> @@ -2,20 +2,513 @@\n> #include \"config.h\"\n> #include \"fsmonitor.h\"\n> #include \"fsmonitor-fs-listen.h\"\n> +#include \"fsmonitor--daemon.h\"\n> +\n> +/*\n> + * The documentation of ReadDirectoryChangesW() states that the maximum\n> + * buffer size is 64K when the monitored directory is remote.\n> + *\n> + * Larger buffers may be used when the monitored directory is local and\n> + * will help us receive events faster from the kernel and avoid dropped\n> + * events.\n> + *\n> + * So we try to use a very large buffer and silently fallback to 64K if\n> + * we get an error.\n> + */\n> +#define MAX_RDCW_BUF_FALLBACK (65536)\n> +#define MAX_RDCW_BUF (65536 * 8)\n> +\n> +struct one_watch\n> +{\n> +\tchar buffer[MAX_RDCW_BUF];\n> +\tDWORD buf_len;\n> +\tDWORD count;\n> +\n> +\tstruct strbuf path;\n> +\tHANDLE hDir;\n> +\tHANDLE hEvent;\n> +\tOVERLAPPED overlapped;\n> +\n> +\t/*\n> +\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n> +\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n> +\t */\n> +\tBOOL is_active;\n> +};\n> +\n> +struct fsmonitor_daemon_backend_data\n> +{\n> +\tstruct one_watch *watch_worktree;\n> +\tstruct one_watch *watch_gitdir;\n> +\n> +\tHANDLE hEventShutdown;\n> +\n> +\tHANDLE hListener[3]; /* we don't own these handles */\n> +#define LISTENER_SHUTDOWN 0\n> +#define LISTENER_HAVE_DATA_WORKTREE 1\n> +#define LISTENER_HAVE_DATA_GITDIR 2\n> +\tint nr_listener_handles;\n> +};\n> +\n> +/*\n> + * Convert the WCHAR path from the notification into UTF8 and\n> + * then normalize it.\n> + */\n> +static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n> +\t\t\t\t struct strbuf *normalized_path)\n> +{\n> +\tint reserve;\n> +\tint len = 0;\n> +\n> +\tstrbuf_reset(normalized_path);\n> +\tif (!info->FileNameLength)\n> +\t\tgoto normalize;\n> +\n> +\t/*\n> +\t * Pre-reserve enough space in the UTF8 buffer for\n> +\t * each Unicode WCHAR character to be mapped into a\n> +\t * sequence of 2 UTF8 characters. That should let us\n> +\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n> +\t */\n> +\treserve = info->FileNameLength + 1;\n> +\tstrbuf_grow(normalized_path, reserve);\n> +\n> +\tfor (;;) {\n> +\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n> +\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n> +\t\t\t\t\t normalized_path->buf,\n> +\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n> +\t\t\t\t\t NULL, NULL);\n> +\t\tif (len > 0)\n> +\t\t\tgoto normalize;\n> +\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n> +\t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n> +\t\t\t GetLastError(),\n> +\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n> +\t\t\t info->FileName);\n> +\t\t\treturn -1;\n> +\t\t}\n> +\n> +\t\tstrbuf_grow(normalized_path,\n> +\t\t\t strbuf_avail(normalized_path) + reserve);\n> +\t}\n> +\n> +normalize:\n> +\tstrbuf_setlen(normalized_path, len);\n> +\treturn strbuf_normalize_path(normalized_path);\n> +}\n> \n> void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n> {\n> +\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n> +}\n> +\n> +static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n> +\t\t\t\t const char *path)\n> +{\n> +\tstruct one_watch *watch = NULL;\n> +\tDWORD desired_access = FILE_LIST_DIRECTORY;\n> +\tDWORD share_mode =\n> +\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n\nAh, this is probably why we can delete a repo that is under a watch.\n\n> +\tHANDLE hDir;\n> +\n> +\thDir = CreateFileA(path,\n> +\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n> +\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n> +\t\t\t NULL);\n> +\tif (hDir == INVALID_HANDLE_VALUE) {\n> +\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n> +\t\t GetLastError(), path);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\twatch = xcalloc(1, sizeof(*watch));\n> +\n> +\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n> +\n> +\tstrbuf_init(&watch->path, 0);\n> +\tstrbuf_addstr(&watch->path, path);\n> +\n> +\twatch->hDir = hDir;\n> +\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n> +\n> +\treturn watch;\n> +}\n> +\n> +static void destroy_watch(struct one_watch *watch)\n> +{\n> +\tif (!watch)\n> +\t\treturn;\n> +\n> +\tstrbuf_release(&watch->path);\n> +\tif (watch->hDir != INVALID_HANDLE_VALUE)\n> +\t\tCloseHandle(watch->hDir);\n> +\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n> +\t\tCloseHandle(watch->hEvent);\n> +\n> +\tfree(watch);\n> +}\n> +\n> +static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n> +\t\t\t struct one_watch *watch)\n> +{\n> +\tDWORD dwNotifyFilter =\n> +\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n> +\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n> +\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n> +\t\tFILE_NOTIFY_CHANGE_SIZE |\n> +\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n> +\t\tFILE_NOTIFY_CHANGE_CREATION;\n> +\n> +\tResetEvent(watch->hEvent);\n> +\n> +\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n> +\twatch->overlapped.hEvent = watch->hEvent;\n> +\n> +start_watch:\n> +\twatch->is_active = ReadDirectoryChangesW(\n> +\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n> +\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n> +\n> +\tif (!watch->is_active &&\n> +\t GetLastError() == ERROR_INVALID_PARAMETER &&\n> +\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n> +\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n> +\t\tgoto start_watch;\n> +\t}\n> +\n> +\tif (watch->is_active)\n> +\t\treturn 0;\n> +\n> +\terror(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\",\n> +\t watch->path.buf, GetLastError());\n> +\treturn -1;\n> +}\n> +\n> +static int recv_rdcw_watch(struct one_watch *watch)\n> +{\n> +\twatch->is_active = FALSE;\n> +\n> +\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n> +\t\t\t\tTRUE))\n> +\t\treturn 0;\n> +\n> +\t// TODO If an external is deleted, the above returns an error.\n> +\t// TODO I'm not sure that there's anything that we can do here other\n> +\t// TODO than failing -- the /.git link file would be broken\n> +\t// TODO anyway. We might try to check for that and return a better\n> +\t// TODO error message.\n\nThese are not fit C-style comments. This situation can be handled\nby a later patch series, if valuable enough.\n\n> +\n> +\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n> +\t watch->path.buf, GetLastError());\n> +\treturn -1;\n> +}\n> +\n> +static void cancel_rdcw_watch(struct one_watch *watch)\n> +{\n> +\tDWORD count;\n> +\n> +\tif (!watch || !watch->is_active)\n> +\t\treturn;\n> +\n> +\tCancelIoEx(watch->hDir, &watch->overlapped);\n> +\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n> +\twatch->is_active = FALSE;\n> +}\n> +\n> +/*\n> + * Process filesystem events that happen anywhere (recursively) under the\n> + * root directory. For a normal working directory, this includes\n> + * both version controlled files and the contents of the .git/ directory.\n> + *\n> + * If /.git is a file, then we only see events for the file\n> + * itself.\n> + */\n> +static int process_worktree_events(struct fsmonitor_daemon_state *state)\n> +{\n> +\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n> +\tstruct one_watch *watch = data->watch_worktree;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n> +\tstruct fsmonitor_batch *batch = NULL;\n> +\tconst char *p = watch->buffer;\n> +\n> +\t/*\n> +\t * If the kernel gets more events than will fit in the kernel\n> +\t * buffer associated with our RDCW handle, it drops them and\n> +\t * returns a count of zero. (A successful call, but with\n> +\t * length zero.)\n> +\t */\n\nI suppose that since we create a cookie file, we don't expect a zero\nresult to ever be a meaningful value? Or, is there another way to\ndifferentiate between \"nothing happened\" and \"too much happened\"?\n\n> +\tif (!watch->count) {\n> +\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n> +\t\t\t\t \"overflow\");\n> +\t\tfsmonitor_force_resync(state);\n> +\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n> +\t}\n> +\n> +\t/*\n> +\t * On Windows, `info` contains an \"array\" of paths that are\n> +\t * relative to the root of whichever directory handle received\n> +\t * the event.\n> +\t */\n> +\tfor (;;) {\n> +\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n> +\t\tconst char *slash;\n> +\t\tenum fsmonitor_path_type t;\n> +\n> +\t\tstrbuf_reset(&path);\n> +\t\tif (normalize_path_in_utf8(info, &path) == -1)\n> +\t\t\tgoto skip_this_path;\n> +\n> +\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n> +\n> +\t\tswitch (t) {\n> +\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n> +\t\t\t/* special case cookie files within .git */\n> +\n> +\t\t\t/* Use just the filename of the cookie file. */\n> +\t\t\tslash = find_last_dir_sep(path.buf);\n> +\t\t\tstring_list_append(&cookie_list,\n> +\t\t\t\t\t slash ? slash + 1 : path.buf);\n\nOk, I see now how we special-case cookies in the list of events.\n\n> +\t\t\tbreak;\n> +\n> +\t\tcase IS_INSIDE_DOT_GIT:\n> +\t\t\t/* ignore everything inside of \"/.git/\" */\n> +\t\t\tbreak;\n> +\n> +\t\tcase IS_DOT_GIT:\n> +\t\t\t/* \"/.git\" was deleted (or renamed away) */\n> +\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n> +\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n> +\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n> +\t\t\t\t\t\t \"fsm-listen/dotgit\",\n> +\t\t\t\t\t\t \"removed\");\n> +\t\t\t\tgoto force_shutdown;\n> +\t\t\t}\n> +\t\t\tbreak;\n> +\n> +\t\tcase IS_WORKDIR_PATH:\n> +\t\t\t/* queue normal pathname */\n> +\t\t\tif (!batch)\n> +\t\t\t\tbatch = fsmonitor_batch__new();\n> +\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n> +\t\t\tbreak;\n> +\n> +\t\tcase IS_GITDIR:\n> +\t\tcase IS_INSIDE_GITDIR:\n> +\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n> +\t\tdefault:\n> +\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n> +\t\t\t t, path.buf);\nSo these events should be caught by the _other_ watcher. I suppose\nBUG() is somewhat appropriate, but also seems heavy-handed. For\nexample, the 'goto' in the next line will never be visited. A die()\nwould even be appropriate, but somewhat less harsh than a BUG(),\nespecially for a background process.\n\n> +\t\t\tgoto skip_this_path;\n> +\t\t}\n> +\n> +skip_this_path:\n> +\t\tif (!info->NextEntryOffset)\n> +\t\t\tbreak;\n> +\t\tp += info->NextEntryOffset;\n> +\t}\n> +\n> +\tfsmonitor_publish(state, batch, &cookie_list);\n> +\tbatch = NULL;\n> +\tstring_list_clear(&cookie_list, 0);\n> +\tstrbuf_release(&path);\n> +\treturn LISTENER_HAVE_DATA_WORKTREE;\n> +\n> +force_shutdown:\n> +\tfsmonitor_batch__free(batch);\n> +\tstring_list_clear(&cookie_list, 0);\n> +\tstrbuf_release(&path);\n> +\treturn LISTENER_SHUTDOWN;\n> +}\n> +\n> +/*\n> + * Process filesystem events that happend anywhere (recursively) under the\n\ns/happend/happened\n\n> + * external (such as non-primary worktrees or submodules).\n> + * We only care about cookie files that our client threads created here.\n> + *\n> + * Note that we DO NOT get filesystem events on the external \n> + * itself (it is not inside something that we are watching). In particular,\n> + * we do not get an event if the external is deleted.\n\nThis is an interesting change of behavior. I forget if it is listed in\nthe documentation file, but definitely could be. I imagine wanting a\n\"Troubleshooting\" section that describes special cases like this.\n\nAlso, because of this worktree-specific behavior, we might want to\nrecommend using 'git config --worktree' when choosing to use FS Monitor,\nso that each worktree is opted-in as requested. Without --worktree, all\nworktrees with a common base would stard using FS Monitor simultaneously.\n\n> + */\n> +static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n> +{\n> +\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n> +\tstruct one_watch *watch = data->watch_gitdir;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n> +\tconst char *p = watch->buffer;\n> +\n> +\tif (!watch->count) {\n> +\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n> +\t\t\t\t \"overflow\");\n> +\t\tfsmonitor_force_resync(state);\n> +\t\treturn LISTENER_HAVE_DATA_GITDIR;\n> +\t}\n> +\n> +\tfor (;;) {\n> +\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n> +\t\tconst char *slash;\n> +\t\tenum fsmonitor_path_type t;\n> +\n> +\t\tstrbuf_reset(&path);\n> +\t\tif (normalize_path_in_utf8(info, &path) == -1)\n> +\t\t\tgoto skip_this_path;\n> +\n> +\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n> +\n> +\t\ttrace_printf_key(&trace_fsmonitor, \"BBB: %s\", path.buf);\n> +\n> +\t\tswitch (t) {\n> +\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n> +\t\t\t/* special case cookie files within gitdir */\n> +\n> +\t\t\t/* Use just the filename of the cookie file. */\n> +\t\t\tslash = find_last_dir_sep(path.buf);\n> +\t\t\tstring_list_append(&cookie_list,\n> +\t\t\t\t\t slash ? slash + 1 : path.buf);\n> +\t\t\tbreak;\n> +\n> +\t\tcase IS_INSIDE_GITDIR:\n> +\t\t\tgoto skip_this_path;\n> +\n> +\t\tdefault:\n> +\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n> +\t\t\t t, path.buf);\n\nIf we decide against BUG() earlier, then also get this one.\n\n> +\t\t\tgoto skip_this_path;\n> +\t\t}\n> +\n> +skip_this_path:\n> +\t\tif (!info->NextEntryOffset)\n> +\t\t\tbreak;\n> +\t\tp += info->NextEntryOffset;\n> +\t}\n> +\n> +\tfsmonitor_publish(state, NULL, &cookie_list);\n> +\tstring_list_clear(&cookie_list, 0);\n> +\tstrbuf_release(&path);\n> +\treturn LISTENER_HAVE_DATA_GITDIR;\n> }\n> \n> void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n> {\n> +\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n> +\tDWORD dwWait;\n> +\n> +\tstate->error_code = 0;\n> +\n> +\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n> +\t\tgoto force_error_stop;\n> +\n> +\tif (data->watch_gitdir &&\n> +\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n> +\t\tgoto force_error_stop;\n> +\n> +\tfor (;;) {\n> +\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n> +\t\t\t\t\t\tdata->hListener,\n> +\t\t\t\t\t\tFALSE, INFINITE);\n\nSince you use INFINITE here, that says that we will wait for at least one\nsignal, solving the confusion about zero results: zero results unambiguously\nindicates a loss of events.\n\n> +\n> +\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n> +\t\t\tif (recv_rdcw_watch(data->watch_worktree) == -1)\n> +\t\t\t\tgoto force_error_stop;\n> +\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n> +\t\t\t\tgoto force_shutdown;\n> +\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n> +\t\t\t\tgoto force_error_stop;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n> +\t\t\tif (recv_rdcw_watch(data->watch_gitdir) == -1)\n> +\t\t\t\tgoto force_error_stop;\n> +\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n> +\t\t\t\tgoto force_shutdown;\n> +\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n> +\t\t\t\tgoto force_error_stop;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n> +\t\t\tgoto clean_shutdown;\n> +\n> +\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n> +\t\t GetLastError());\n> +\t\tgoto force_error_stop;\n> +\t}\n> +\n> +force_error_stop:\n> +\tstate->error_code = -1;\n> +\n> +force_shutdown:\n> +\t/*\n> +\t * Tell the IPC thead pool to stop (which completes the await\n> +\t * in the main thread (which will also signal this thread (if\n> +\t * we are still alive))).\n> +\t */\n> +\tipc_server_stop_async(state->ipc_server_data);\n> +\n> +clean_shutdown:\n> +\tcancel_rdcw_watch(data->watch_worktree);\n> +\tcancel_rdcw_watch(data->watch_gitdir);\n> }\n> \n> int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n> {\n> +\tstruct fsmonitor_daemon_backend_data *data;\n> +\n> +\tdata = xcalloc(1, sizeof(*data));\n\nCALLOC_ARRAY()\n\n> +\n> +\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n> +\n> +\tdata->watch_worktree = create_watch(state,\n> +\t\t\t\t\t state->path_worktree_watch.buf);\n> +\tif (!data->watch_worktree)\n> +\t\tgoto failed;\n> +\n> +\tif (state->nr_paths_watching > 1) {\n> +\t\tdata->watch_gitdir = create_watch(state,\n> +\t\t\t\t\t\t state->path_gitdir_watch.buf);\n> +\t\tif (!data->watch_gitdir)\n> +\t\t\tgoto failed;\n> +\t}\n> +\n> +\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n> +\tdata->nr_listener_handles++;\n> +\n> +\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n> +\t\tdata->watch_worktree->hEvent;\n> +\tdata->nr_listener_handles++;\n> +\n> +\tif (data->watch_gitdir) {\n> +\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n> +\t\t\tdata->watch_gitdir->hEvent;\n> +\t\tdata->nr_listener_handles++;\n> +\t}\n\nThis is a clever organization of the event handles. I imagine it\nwill requires some rework if we decide to include another optional\nhandle whose inclusion is orthogonal to the gitdir one, but that\nis unlikely enough to keep these well-defined array indices.\n\n> +\tstate->backend_data = data;\n> +\treturn 0;\n> +\n> +failed:\n> +\tCloseHandle(data->hEventShutdown);\n> +\tdestroy_watch(data->watch_worktree);\n> +\tdestroy_watch(data->watch_gitdir);\n> +\n> \treturn -1;\n> }\n> \n> void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n> {\n> +\tstruct fsmonitor_daemon_backend_data *data;\n> +\n> +\tif (!state || !state->backend_data)\n> +\t\treturn;\n> +\n> +\tdata = state->backend_data;\n> +\n> +\tCloseHandle(data->hEventShutdown);\n> +\tdestroy_watch(data->watch_worktree);\n> +\tdestroy_watch(data->watch_gitdir);\n> +\n> +\tFREE_AND_NULL(state->backend_data);\n> }\n\nI tried to follow all the API calls and check the documentation for\nany misuse, but did not find any. I can only contribute nitpicks\nhere, and rely on the tests to really see that this is working as\nexpected.\n\nI was hoping to find in here why we need to sleep in the test suite,\nbut have not pinpointed that issue yet.\n\nThanks,\n-Stolee\n"},{"id":"423118","messageId":"CAPig+cR8RUATo0cVZE5fQYOJjs+5v9-JfWdrz+A93Q5_Qr-ZpQ@mail.gmail.com","threadId":"55420","inReplyTo":"4b0066f5-5962-e57e-5987-5358f3d394f7@gmail.com","subject":"Re: [PATCH 13/23] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-04-27T17:41:36Z","receivedAt":"2021-04-27T17:41:50Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Apr 27, 2021 at 1:22 PM Derrick Stolee wrote:\n> On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> > + // TODO If an external is deleted, the above returns an error.\n> > + // TODO I'm not sure that there's anything that we can do here other\n> > + // TODO than failing -- the /.git link file would be broken\n> > + // TODO anyway. We might try to check for that and return a better\n> > + // TODO error message.\n>\n> These are not fit C-style comments. This situation can be handled\n> by a later patch series, if valuable enough.\n\nIn this project, a comment like this would normally be prefixed by\nNEEDSWORK rather than TODO, and only the first line would carry the\nprefix line, not all of them.\n\n /* NEEDSWORK: the foinkster blorps the wooz */\n"},{"id":"423119","messageId":"7da80a62-7820-3fb8-a0e9-4356d9ea597e@gmail.com","threadId":"55420","inReplyTo":"67fa7c7b8ac79b05d0773dc0affbe39b77aa4893.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 14/23] fsmonitor-fs-listen-macos: add macos header files for FSEvent","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T18:13:09Z","receivedAt":"2021-04-27T18:13:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Include MacOS system declarations to allow us to use FSEvent and\n> CoreFoundation APIs. We need GCC and clang versions because of\n> compiler and header file conflicts.\n...\n> This is a known problem and tracked in GCC's bug tracker:\n> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n> \n> In the meantime, let's not block things and go the slightly ugly route\n> of declaring/defining the FSEvents constants, data structures and\n> functions that we need, so that we can avoid above-mentioned issue.\n> \n> Let's do this _only_ for GCC, though, so that the CI/PR builds (which\n> build both with clang and with GCC) can guarantee that we _are_ using\n> the correct data types.\n\nI appreciate that this issue with header files is isolated to its own\npatch.\n\nThanks,\n-Stolee\n\n"},{"id":"423120","messageId":"26cd8593-58ad-bc97-5348-fdc6aaaba4ec@gmail.com","threadId":"55420","inReplyTo":"d469d3f02e335292f4debd55ff67184b2b91b04f.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 15/23] fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T18:35:53Z","receivedAt":"2021-04-27T18:35:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Implement file system event listener on MacOS using FSEvent,\n> CoreFoundation, and CoreServices.\n\nAgain, I'm not sure if we _should_ be including URLs to\ndocumentation in our messages, but here are some I found helpful:\n\n[1] https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html\n[2] https://developer.apple.com/documentation/corefoundation/1541796-cfrunloopstop\n[3] https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/Multithreading/RunLoopManagement/RunLoopManagement.html\n\n> Co-authored-by: Kevin Willford \n> Co-authored-by: Johannes Schindelin \n> Signed-off-by: Jeff Hostetler \n> ---\n> compat/fsmonitor/fsmonitor-fs-listen-macos.c | 368 +++++++++++++++++++\n> 1 file changed, 368 insertions(+)\n> \n> diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n> index bec5130d9e1d..e055fb579cc4 100644\n> --- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n> +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n> @@ -97,20 +97,388 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n> #include \"cache.h\"\n> #include \"fsmonitor.h\"\n> #include \"fsmonitor-fs-listen.h\"\n> +#include \"fsmonitor--daemon.h\"\n> +\n> +struct fsmonitor_daemon_backend_data\n> +{\n> +\tCFStringRef cfsr_worktree_path;\n> +\tCFStringRef cfsr_gitdir_path;\n> +\n> +\tCFArrayRef cfar_paths_to_watch;\n> +\tint nr_paths_watching;\n> +\n> +\tFSEventStreamRef stream;\n> +\n> +\tCFRunLoopRef rl;\n> +\n> +\tenum shutdown_style {\n> +\t\tSHUTDOWN_EVENT = 0,\n> +\t\tFORCE_SHUTDOWN,\n> +\t\tFORCE_ERROR_STOP,\n> +\t} shutdown_style;\n> +\n> +\tunsigned int stream_scheduled:1;\n> +\tunsigned int stream_started:1;\n> +};\n> +\n> +static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n> +{\n> +\tstruct strbuf msg = STRBUF_INIT;\n\nBefore going through these ifs and constructing a string, it\nmight be a good idea to check if the trace event will actually\nbe sent somewhere. If the logging method is switched to a\ntrace2 method, then up here we can do:\n\n\tif (!trace2_is_enabled())\n\t\treturn;\n\n> +\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n> +\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n> +\tif (flag & kFSEventStreamEventFlagUserDropped)\n> +\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n> +\tif (flag & kFSEventStreamEventFlagKernelDropped)\n> +\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n> +\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n> +\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n> +\tif (flag & kFSEventStreamEventFlagHistoryDone)\n> +\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n> +\tif (flag & kFSEventStreamEventFlagRootChanged)\n> +\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n> +\tif (flag & kFSEventStreamEventFlagMount)\n> +\t\tstrbuf_addstr(&msg, \"Mount|\");\n> +\tif (flag & kFSEventStreamEventFlagUnmount)\n> +\t\tstrbuf_addstr(&msg, \"Unmount|\");\n> +\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n> +\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n> +\tif (flag & kFSEventStreamEventFlagItemCreated)\n> +\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n> +\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n> +\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n> +\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n> +\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsDir)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsFile)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n> +\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n> +\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n> +\tif (flag & kFSEventStreamEventFlagItemModified)\n> +\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n> +\tif (flag & kFSEventStreamEventFlagItemRemoved)\n> +\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n> +\tif (flag & kFSEventStreamEventFlagItemRenamed)\n> +\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n> +\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n> +\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n> +\tif (flag & kFSEventStreamEventFlagOwnEvent)\n> +\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n> +\tif (flag & kFSEventStreamEventFlagItemCloned)\n> +\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n> +\n> +\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n> +\t\t\t path, flag, msg.buf);\n\nShould this be a trace2 call?\n\n> +\n> +\tstrbuf_release(&msg);\n> +}\n> +\n> +static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n> +{\n> +\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n> +\t\tef & kFSEventStreamEventFlagItemRemoved);\n> +}\n> +\n> +static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n> +{\n> +\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n> +\t\tef & kFSEventStreamEventFlagItemRenamed);\n> +}\n\nWill these be handled differently? Or is it enough to detect\nef_is_root_moved_or_deleted()?\n\n> +static void fsevent_callback(ConstFSEventStreamRef streamRef,\n> +\t\t\t void *ctx,\n> +\t\t\t size_t num_of_events,\n> +\t\t\t void *event_paths,\n> +\t\t\t const FSEventStreamEventFlags event_flags[],\n> +\t\t\t const FSEventStreamEventId event_ids[])\n> +{\n> +\tstruct fsmonitor_daemon_state *state = ctx;\n> +\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n> +\tchar **paths = (char **)event_paths;\n> +\tstruct fsmonitor_batch *batch = NULL;\n> +\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n> +\tconst char *path_k;\n> +\tconst char *slash;\n> +\tint k;\n> +\n> +\t/*\n> +\t * Build a list of all filesystem changes into a private/local\n> +\t * list and without holding any locks.\n> +\t */\n> +\tfor (k = 0; k < num_of_events; k++) {\n> +\t\t/*\n> +\t\t * On Mac, we receive an array of absolute paths.\n> +\t\t */\n> +\t\tpath_k = paths[k];\n> +\n> +\t\t/*\n> +\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n> +\t\t * Please don't log them to Trace2.\n> +\t\t *\n> +\t\t * trace_printf_key(&trace_fsmonitor, \"XXX '%s'\", path_k);\n> +\t\t */\n\nOh, I see. _Not_ trace2. What should we do to see if this is enabled\nto avoid over-working in the case we are not using GIT_TRACE_FSMONITOR?\n\n> +\t\t/*\n> +\t\t * If event[k] is marked as dropped, we assume that we have\n> +\t\t * lost sync with the filesystem and should flush our cached\n> +\t\t * data. We need to:\n> +\t\t *\n> +\t\t * [1] Abort/wake any client threads waiting for a cookie and\n> +\t\t * flush the cached state data (the current token), and\n> +\t\t * create a new token.\n> +\t\t *\n> +\t\t * [2] Discard the batch that we were locally building (since\n> +\t\t * they are conceptually relative to the just flushed\n> +\t\t * token).\n> +\t\t */\n> +\t\tif ((event_flags[k] & kFSEventStreamEventFlagKernelDropped) ||\n> +\t\t (event_flags[k] & kFSEventStreamEventFlagUserDropped)) {\n\nPerhaps create a macro EVENT_FLAG_DROPPED that is the union of these two? Then\na single \"event_flags[k] & EVENT_FLAG_DROPPED\" would suffice here. Helps cover\nup how complicated the macOS API names are, too.\n\n> +\t\t\t/*\n> +\t\t\t * see also kFSEventStreamEventFlagMustScanSubDirs\n> +\t\t\t */\n> +\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n> +\t\t\t\t\t \"fsm-listen/kernel\", \"dropped\");\n> +\n> +\t\t\tfsmonitor_force_resync(state);\n> +\n> +\t\t\tif (fsmonitor_batch__free(batch))\n> +\t\t\t\tBUG(\"batch should not have a next\");\n\nI mentioned before that BUG() seems overkill for these processes, but this\none fits. If this batch has a next, then we did something wrong, right? Do\nwe have an automated test that checks enough events to maybe cause a second\nbatch to be created?\n\n> +\t\t\tstring_list_clear(&cookie_list, 0);\n> +\n> +\t\t\t/*\n> +\t\t\t * We assume that any events that we received\n> +\t\t\t * in this callback after this dropped event\n> +\t\t\t * may still be valid, so we continue rather\n> +\t\t\t * than break. (And just in case there is a\n> +\t\t\t * delete of \".git\" hiding in there.)\n> +\t\t\t */\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n> +\n> +\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n> +\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n> +\t\t\t/* special case cookie files within .git or gitdir */\n> +\n> +\t\t\t/* Use just the filename of the cookie file. */\n> +\t\t\tslash = find_last_dir_sep(path_k);\n> +\t\t\tstring_list_append(&cookie_list,\n> +\t\t\t\t\t slash ? slash + 1 : path_k);\n> +\t\t\tbreak;\n> +\n> +\t\tcase IS_INSIDE_DOT_GIT:\n> +\t\tcase IS_INSIDE_GITDIR:\n> +\t\t\t/* ignore all other paths inside of .git or gitdir */\n> +\t\t\tbreak;\n> +\n> +\t\tcase IS_DOT_GIT:\n> +\t\tcase IS_GITDIR:\n> +\t\t\t/*\n> +\t\t\t * If .git directory is deleted or renamed away,\n> +\t\t\t * we have to quit.\n> +\t\t\t */\n> +\t\t\tif (ef_is_root_delete(event_flags[k])) {\n> +\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n> +\t\t\t\t\t\t \"fsm-listen/gitdir\",\n> +\t\t\t\t\t\t \"removed\");\n> +\t\t\t\tgoto force_shutdown;\n> +\t\t\t}\n> +\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n> +\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n> +\t\t\t\t\t\t \"fsm-listen/gitdir\",\n> +\t\t\t\t\t\t \"renamed\");\n> +\t\t\t\tgoto force_shutdown;\n> +\t\t\t}\n\nI see. The only difference is in how we trace the result. I'm not sure\nthis tracing message is worth the differentiation.\n\n> +\t\t\tbreak;\n> +\n> +\t\tcase IS_WORKDIR_PATH:\n> +\t\t\t/* try to queue normal pathnames */\n> +\n> +\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n> +\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n> +\n> +\t\t\t/* fsevent could be marked as both a file and directory */\n\nThe _same_ event? Interesting. And I see that you need to log the name\ndifferently in the case of a file or a directory.\n\n> +\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n> +\t\t\t\tconst char *rel = path_k +\n> +\t\t\t\t\tstate->path_worktree_watch.len + 1;\n> +\n> +\t\t\t\tif (!batch)\n> +\t\t\t\t\tbatch = fsmonitor_batch__new();\n> +\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n> +\t\t\t}\n> +\n> +\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n> +\t\t\t\tconst char *rel = path_k +\n> +\t\t\t\t\tstate->path_worktree_watch.len + 1;\n> +\t\t\t\tchar *p = xstrfmt(\"%s/\", rel);\n\nIn a critical path, xstrfmt() may be too slow for such a simple case.\nLikely we should instead use a strbuf with:\n\n\tstrbuf_addstr(&p, rel);\n\tstrbuf_addch(&p, '/');\n\nBonus points if we can use the data to predict the size of the strbuf's\nbuffer.\n\n> +\n> +\t\t\t\tif (!batch)\n> +\t\t\t\t\tbatch = fsmonitor_batch__new();\n> +\t\t\t\tfsmonitor_batch__add_path(batch, p);\n> +\n> +\t\t\t\tfree(p);\n> +\t\t\t}\n> +\n> +\t\t\tbreak;\n> +\n> +\t\tcase IS_OUTSIDE_CONE:\n> +\t\tdefault:\n> +\t\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t\t \"ignoring '%s'\", path_k);\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n> +\n> +\tfsmonitor_publish(state, batch, &cookie_list);\n> +\tstring_list_clear(&cookie_list, 0);\n> +\treturn;\n> +\n> +force_shutdown:\n> +\tif (fsmonitor_batch__free(batch))\n> +\t\tBUG(\"batch should not have a next\");\n> +\tstring_list_clear(&cookie_list, 0);\n> +\n> +\tdata->shutdown_style = FORCE_SHUTDOWN;\n> +\tCFRunLoopStop(data->rl);\n> +\treturn;\n> +}\n> +\n> +/*\n> + * TODO Investigate the proper value for the `latency` argument in the call\n> + * TODO to `FSEventStreamCreate()`. I'm not sure that this needs to be a\n> + * TODO config setting or just something that we tune after some testing.\n> + * TODO\n> + * TODO With a latency of 0.1, I was seeing lots of dropped events during\n> + * TODO the \"touch 100000\" files test within t/perf/p7519, but with a\n> + * TODO latency of 0.001 I did not see any dropped events. So the \"correct\"\n> + * TODO value may be somewhere in between.\n> + * TODO\n> + * TODO https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n> + */\n\nAs Eric mentioned in another thread, this should say \"NEEDSWORK\" at\nthe top. This is a good candidate for follow-up after the basics of\nthe series is stable.\n\n> int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n> {\n> +\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n> +\t\tkFSEventStreamCreateFlagWatchRoot |\n> +\t\tkFSEventStreamCreateFlagFileEvents;\n> +\tFSEventStreamContext ctx = {\n> +\t\t0,\n> +\t\tstate,\n> +\t\tNULL,\n> +\t\tNULL,\n> +\t\tNULL\n> +\t};\n> +\tstruct fsmonitor_daemon_backend_data *data;\n> +\tconst void *dir_array[2];\n> +\n> +\tdata = xcalloc(1, sizeof(*data));\n\nCALLOC_ARRAY()\n\n> +\tstate->backend_data = data;\n> +\n> +\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n> +\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n> +\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n> +\n> +\tif (state->nr_paths_watching > 1) {\n> +\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n> +\t\t\tNULL, state->path_gitdir_watch.buf,\n> +\t\t\tkCFStringEncodingUTF8);\n> +\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n> +\t}\n> +\n> +\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n> +\t\t\t\t\t\t data->nr_paths_watching,\n> +\t\t\t\t\t\t NULL);\n> +\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n> +\t\t\t\t\t data->cfar_paths_to_watch,\n> +\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n> +\t\t\t\t\t 0.001, flags);\n> +\tif (data->stream == NULL)\n> +\t\tgoto failed;\n> +\n> +\t/*\n> +\t * `data->rl` needs to be set inside the listener thread.\n> +\t */\n> +\n> +\treturn 0;\n> +\n> +failed:\n> +\terror(\"Unable to create FSEventStream.\");\n> +\n> +\tFREE_AND_NULL(state->backend_data);\n> \treturn -1;\n> }\n> \n> void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n> {\n> +\tstruct fsmonitor_daemon_backend_data *data;\n> +\n> +\tif (!state || !state->backend_data)\n> +\t\treturn;\n> +\n> +\tdata = state->backend_data;\n> +\n> +\tif (data->stream) {\n> +\t\tif (data->stream_started)\n> +\t\t\tFSEventStreamStop(data->stream);\n> +\t\tif (data->stream_scheduled)\n> +\t\t\tFSEventStreamInvalidate(data->stream);\n> +\t\tFSEventStreamRelease(data->stream);\n> +\t}\n> +\n> +\tFREE_AND_NULL(state->backend_data);\n> }\n> \n> void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n> {\n> +\tstruct fsmonitor_daemon_backend_data *data;\n> +\n> +\tdata = state->backend_data;\n> +\tdata->shutdown_style = SHUTDOWN_EVENT;\n> +\n> +\tCFRunLoopStop(data->rl);\n> }\n> \n> void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n> {\n> +\tstruct fsmonitor_daemon_backend_data *data;\n> +\n> +\tdata = state->backend_data;\n> +\n> +\tdata->rl = CFRunLoopGetCurrent();\n> +\n> +\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n> +\tdata->stream_scheduled = 1;\n> +\n> +\tif (!FSEventStreamStart(data->stream)) {\n> +\t\terror(\"Failed to start the FSEventStream\");\n> +\t\tgoto force_error_stop_without_loop;\n> +\t}\n> +\tdata->stream_started = 1;\n> +\n> +\tCFRunLoopRun();\n> +\n> +\tswitch (data->shutdown_style) {\n> +\tcase FORCE_ERROR_STOP:\n> +\t\tstate->error_code = -1;\n> +\t\t/* fall thru */\n> +\tcase FORCE_SHUTDOWN:\n> +\t\tipc_server_stop_async(state->ipc_server_data);\n> +\t\t/* fall thru */\n> +\tcase SHUTDOWN_EVENT:\n> +\tdefault:\n> +\t\tbreak;\n> +\t}\n> +\treturn;\n> +\n> +force_error_stop_without_loop:\n> +\tstate->error_code = -1;\n> +\tipc_server_stop_async(state->ipc_server_data);\n> +\treturn;\n> }\n\nThanks,\n-Stolee\n"},{"id":"423121","messageId":"af7a671c-fa32-6d9a-7d75-65582fdbcf24@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"FS Monitor Windows Performance (was [PATCH 00/23] [RFC] Builtin FSMonitor Feature)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T18:49:54Z","receivedAt":"2021-04-27T18:50:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> This patch series adds a builtin FSMonitor daemon to Git.\n> \n> This daemon uses platform-specific filesystem notifications to keep track of\n> changes to a working directory. It also listens over the \"Simple IPC\"\n> facility for client requests and responds with a list of files/directories\n> that have been recently modified.\n...\n> This RFC version includes support for Windows and MacOS file system events.\n> A Linux version will be submitted in a later patch series.\n\nI finished a full read-through of the series, and pointed out what I\ncould. I'm not an expert on filesystems or these platform-specific\nAPIs, so I could only do a surface-level check that those integrations\nare correct. They certainly appear to be, and the real proof is in the\ntests and its performance.\n\nI mentioned that I am concerned about the need for delays in the test\nsuite, since the feature itself should be robust to scripts and tools\ninteracting with Git shortly after modifying the filesystem. I hope we\ncan isolate the need for such delays.\n\nAs for performance, I wanted to check the timings for how this improves\nthe case for large repositories. I believe it should be clear that this\nmakes things easier when there is a large set of filesystem events,\ncausing Git to need to walk more of the workdir in a command like 'git\nstatus'. So, I wanted to focus on zero or one changes, and see how\nthat affects performance.\n\nThis message focuses only on the Windows case. I will provide my macOS\nperformance numbers in a separate message.\n\nI've been using two cases, one that tests 'git status' when there are no\nchanges to the filesystem, and another where a file is modified and then\ndeleted (with 'git status' run between each case).\n\nhyperfine \\\n\t-n \"none (clean)\" \"$GIT -c core.useBuiltinFSMonitor=false status\" \\\n\t-n \"builtin (clean)\" \"$GIT -c core.useBuiltinFSMonitor=true status\" \\\n\t--warmup=5\n\nhyperfine \\\n\t-n \"none (dirty)\" \"echo >>$FILE && $GIT -c core.useBuiltinFSMonitor=false status && rm $FILE && $GIT -c core.useBuiltinFSMonitor=false status\" \\\n\t-n \"builtin (dirty)\" \"echo >>$FILE && $GIT -c core.useBuiltinFSMonitor=true status && rm $FILE && $GIT -c core.useBuiltinFSMonitor=true status\" \\\n\t--warmup=5\n\nNote that we are running 'git status' twice in the dirty case, which\nwill make it appear like things are more than twice as slow as the\nclean case.\n\nI then got some disappointing results on my first run:\n\nsparse-index disabled, untracked cache disabled\n-----------------------------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 1.870 s ± 0.029 s [User: 6.1 ms, System: 13.9 ms]\n Range (min … max): 1.814 s … 1.903 s 10 runs\n\nBenchmark #2: builtin (clean)\n Time (mean ± σ): 1.961 s ± 0.102 s [User: 3.6 ms, System: 12.4 ms]\n Range (min … max): 1.832 s … 2.172 s 10 runs\n\nSummary\n 'none (clean)' ran\n 1.05 ± 0.06 times faster than 'builtin (clean)'\nBenchmark #1: none (dirty)\n Time (mean ± σ): 3.738 s ± 0.044 s [User: 5.3 ms, System: 17.0 ms]\n Range (min … max): 3.663 s … 3.832 s 10 runs\n\nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 5.987 s ± 0.062 s [User: 2.8 ms, System: 17.3 ms]\n Range (min … max): 5.895 s … 6.090 s 10 runs\n\nSummary\n 'none (dirty)' ran\n 1.60 ± 0.03 times faster than 'builtin (dirty)'\n\nThis all depends on the index being very large. I'm testing using a repo\nwith 2 million files at HEAD, but only 4% actually checked out on disk.\nThis exaggerates the cost of the index rewrite. The FS Monitor feature\nforces 'git status' to rewrite the index because it updates the token in\nthe extension. I wish I had a better understanding of why the index is\nnot updated in the default case.\n\nInterestingly, the untracked cache extension makes a big difference here.\nThe performance of the overall behavior is much faster if the untracked\ncache exists (when paired with the builtin FS Monitor; it doesn't make a\nsignificant difference when FS Monitor is disabled).\n\nsparse-index disabled, untracked cache enabled\n----------------------------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 1.803 s ± 0.037 s [User: 1.3 ms, System: 19.5 ms]\n Range (min … max): 1.748 s … 1.878 s 10 runs\n\nBenchmark #2: builtin (clean)\n Time (mean ± σ): 1.071 s ± 0.035 s [User: 1.3 ms, System: 14.0 ms]\n Range (min … max): 1.019 s … 1.138 s 10 runs\n\nSummary\n 'builtin (clean)' ran\n 1.68 ± 0.07 times faster than 'none (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 3.648 s ± 0.079 s [User: 3.9 ms, System: 20.5 ms]\n Range (min … max): 3.533 s … 3.761 s 10 runs\n\nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 4.268 s ± 0.095 s [User: 2.6 ms, System: 20.8 ms]\n Range (min … max): 4.115 s … 4.403 s 10 runs\n\nSummary\n 'none (dirty)' ran\n 1.17 ± 0.04 times faster than 'builtin (dirty)'\n\nHowever, when I enable the sparse-index and the code where 'git status'\nworks with it, then I get the results we hope for with the FS Monitor\nfeature:\n\nsparse-index enabled, untracked cache enabled\n---------------------------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 568.3 ms ± 21.7 ms [User: 5.0 ms, System: 10.4 ms]\n Range (min … max): 541.6 ms … 598.3 ms 10 runs\n\nBenchmark #2: builtin (clean)\n Time (mean ± σ): 214.8 ms ± 24.9 ms [User: 1.0 ms, System: 16.0 ms]\n Range (min … max): 175.9 ms … 249.4 ms 12 runs\n\nSummary\n 'builtin (clean)' ran\n 2.65 ± 0.32 times faster than 'none (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 979.1 ms ± 30.2 ms [User: 2.4 ms, System: 18.7 ms]\n Range (min … max): 951.8 ms … 1051.1 ms 10 runs\n\nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 529.2 ms ± 43.1 ms [User: 8.0 ms, System: 12.4 ms]\n Range (min … max): 461.2 ms … 590.6 ms 10 runs\n\nSummary\n 'builtin (dirty)' ran\n 1.85 ± 0.16 times faster than 'none (dirty)'\n\nHowever, if I disable the untracked cache, we are back to where we\nstarted:\n\nsparse-index enabled, untracked cache disabled\n----------------------------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 542.3 ms ± 28.5 ms [User: 4.0 ms, System: 17.2 ms]\n Range (min … max): 501.8 ms … 594.4 ms 10 runs\n\nBenchmark #2: builtin (clean)\n Time (mean ± σ): 1.126 s ± 0.034 s [User: 5.2 ms, System: 7.6 ms]\n Range (min … max): 1.074 s … 1.163 s 10 runs\n\nSummary\n 'none (clean)' ran\n 2.08 ± 0.13 times faster than 'builtin (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 1.128 s ± 0.032 s [User: 6.9 ms, System: 5.0 ms]\n Range (min … max): 1.078 s … 1.202 s 10 runs\n\nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 2.334 s ± 0.072 s [User: 2.9 ms, System: 21.7 ms]\n Range (min … max): 2.220 s … 2.444 s 10 runs\n\nSummary\n 'none (dirty)' ran\n 2.07 ± 0.09 times faster than 'builtin (dirty)'\n\nWhen I use a much smaller repository (git.git) without sparse-checkout,\nI see that this extra cost is not there, but the untracked cache still\nhelps FS Monitor more than the standard case:\n\nuntracked cache disabled\n-------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 118.0 ms ± 8.4 ms [User: 2.3 ms, System: 4.8 ms]\n Range (min … max): 102.6 ms … 133.7 ms 22 runs\n\nBenchmark #2: builtin (clean)\n Time (mean ± σ): 72.2 ms ± 9.5 ms [User: 3.0 ms, System: 7.8 ms]\n Range (min … max): 53.5 ms … 96.0 ms 33 runs\n\nSummary\n 'builtin (clean)' ran\n 1.63 ± 0.25 times faster than 'none (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 270.3 ms ± 17.6 ms [User: 1.3 ms, System: 13.6 ms]\n Range (min … max): 248.7 ms … 306.8 ms 10 runs\n\nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 165.5 ms ± 10.4 ms [User: 3.3 ms, System: 11.5 ms]\n Range (min … max): 146.0 ms … 183.7 ms 16 runs\n\nSummary\n 'builtin (dirty)' ran\n 1.63 ± 0.15 times faster than 'none (dirty)'\n\n\nuntracked cache enabled\n-------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 129.3 ms ± 10.9 ms [User: 2.2 ms, System: 6.9 ms]\n Range (min … max): 108.2 ms … 146.0 ms 19 runs\n\nBenchmark #2: builtin (clean)\n Time (mean ± σ): 51.6 ms ± 10.5 ms [User: 5.3 ms, System: 10.4 ms]\n Range (min … max): 34.9 ms … 99.1 ms 48 runs\n\nSummary\n 'builtin (clean)' ran\n 2.51 ± 0.55 times faster than 'none (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 214.5 ms ± 7.5 ms [User: 7.7 ms, System: 3.4 ms]\n Range (min … max): 207.1 ms … 234.5 ms 12 runs\n\nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 131.8 ms ± 13.1 ms [User: 2.9 ms, System: 9.8 ms]\n Range (min … max): 110.8 ms … 159.7 ms 22 runs\n\nSummary\n 'builtin (dirty)' ran\n 1.63 ± 0.17 times faster than 'none (dirty)'\n\nI think it would be valuable to discover why using the builtin FS Monitor\nwithout the untracked cache causes such performance problems (on Windows).\nThat might be reason enough to enable the untracked cache feature when the\nFS Monitor feature is enabled, as in the following diff:\n\n--- >8 ---\n\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 93aab92ff16..1f25609f019 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -58,9 +58,13 @@ void prepare_repo_settings(struct repository *r)\n \t\tr->settings.core_multi_pack_index = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n \n-\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n+\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value) {\n \t\tr->settings.use_builtin_fsmonitor = 1;\n \n+\t\t/* Use untracked cache if FS Monitor is enabled. */\n+\t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);\n+\t}\n+\n \tif (!repo_config_get_bool(r, \"feature.manyfiles\", &value) && value) {\n \t\tUPDATE_DEFAULT_BOOL(r->settings.index_version, 4);\n \t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);\n\n--- >8 ---\n\nThanks,\n-Stolee\n"},{"id":"423124","messageId":"aa7ab292-f574-f960-c270-79e298046fef@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"FS Monitor macOS Performance (was [PATCH 00/23] [RFC] Builtin FSMonitor Feature)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-27T19:31:54Z","receivedAt":"2021-04-27T19:31:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> This patch series adds a builtin FSMonitor daemon to Git.\n> \n> This daemon uses platform-specific filesystem notifications to keep track of\n> changes to a working directory. It also listens over the \"Simple IPC\"\n> facility for client requests and responds with a list of files/directories\n> that have been recently modified.\n...\n> This RFC version includes support for Windows and MacOS file system events.\n> A Linux version will be submitted in a later patch series.\n\nSimilarly to my message about testing the Windows performance, I\nrepeated those tests on macOS.\n\nThe same testing procedure was used, except now I'm on a MacBook\nPro laptop instead of a desktop, so the CPU power is likely to be\nsignificantly less.\n\nHowever, I am pleased to report that the FS Monitor feature is\na clear winner in all scenarios. Using the untracked cache is\nstill highly recommended, but not necessary in order to get a\nspeed boost from the builtin FS Montiro.\n\n\nSparse Index Disabled, Untracked Cache Enabled\n----------------------------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 3.980 s ± 0.026 s [User: 919.1 ms, System: 1891.8 ms]\n Range (min … max): 3.940 s … 4.028 s 10 runs\n \nBenchmark #2: builtin (clean)\n Time (mean ± σ): 477.9 ms ± 6.6 ms [User: 772.9 ms, System: 379.7 ms]\n Range (min … max): 468.1 ms … 489.5 ms 10 runs\n \nSummary\n 'builtin (clean)' ran\n 8.33 ± 0.13 times faster than 'none (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 5.411 s ± 0.199 s [User: 2.993 s, System: 4.120 s]\n Range (min … max): 5.026 s … 5.756 s 10 runs\n \nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 2.588 s ± 0.025 s [User: 3.752 s, System: 2.853 s]\n Range (min … max): 2.540 s … 2.628 s 10 runs\n \nSummary\n 'builtin (dirty)' ran\n 2.09 ± 0.08 times faster than 'none (dirty)'\n\nSparse Index Disabled, Untracked Cache Disabled\n-----------------------------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 2.993 s ± 0.115 s [User: 1.562 s, System: 2.289 s]\n Range (min … max): 2.741 s … 3.167 s 10 runs\n \nBenchmark #2: builtin (clean)\n Time (mean ± σ): 939.4 ms ± 10.1 ms [User: 1.452 s, System: 1.519 s]\n Range (min … max): 925.1 ms … 961.0 ms 10 runs\n \nSummary\n 'builtin (clean)' ran\n 3.19 ± 0.13 times faster than 'none (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 8.245 s ± 1.118 s [User: 3.204 s, System: 5.684 s]\n Range (min … max): 5.927 s … 8.985 s 10 runs\n \nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 2.969 s ± 0.034 s [User: 3.832 s, System: 3.160 s]\n Range (min … max): 2.927 s … 3.023 s 10 runs\n \nSummary\n 'builtin (dirty)' ran\n 2.78 ± 0.38 times faster than 'none (dirty)'\n\n\nSparse Index Enabled, Untracked Cache Enabled\n---------------------------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 1.250 s ± 0.050 s [User: 216.9 ms, System: 1836.9 ms]\n Range (min … max): 1.177 s … 1.300 s 10 runs\n \nBenchmark #2: builtin (clean)\n Time (mean ± σ): 89.3 ms ± 2.9 ms [User: 51.3 ms, System: 22.6 ms]\n Range (min … max): 81.9 ms … 93.5 ms 31 runs\n \nSummary\n 'builtin (clean)' ran\n 14.01 ± 0.72 times faster than 'none (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 2.087 s ± 0.095 s [User: 320.9 ms, System: 3327.5 ms]\n Range (min … max): 1.943 s … 2.242 s 10 runs\n \nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 233.5 ms ± 2.7 ms [User: 165.5 ms, System: 74.1 ms]\n Range (min … max): 227.8 ms … 237.1 ms 12 runs\n \nSummary\n 'builtin (dirty)' ran\n 8.94 ± 0.42 times faster than 'none (dirty)'\n\n\nSparse Index Enabled, Untracked Cache Disabled\n----------------------------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 1.277 s ± 0.101 s [User: 215.5 ms, System: 1877.9 ms]\n Range (min … max): 1.138 s … 1.458 s 10 runs\n \nBenchmark #2: builtin (clean)\n Time (mean ± σ): 300.0 ms ± 6.1 ms [User: 119.4 ms, System: 183.1 ms]\n Range (min … max): 293.0 ms … 313.2 ms 10 runs\n \nSummary\n 'builtin (clean)' ran\n 4.26 ± 0.35 times faster than 'none (clean)'\nBenchmark #1: none (dirty)\n Time (mean ± σ): 2.488 s ± 0.088 s [User: 432.6 ms, System: 3631.6 ms]\n Range (min … max): 2.328 s … 2.601 s 10 runs\n \nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 636.4 ms ± 12.8 ms [User: 266.2 ms, System: 374.0 ms]\n Range (min … max): 624.4 ms … 671.0 ms 10 runs\n\nSummary\n 'builtin (dirty)' ran\n 3.91 ± 0.16 times faster than 'none (dirty)'\n\n\nHere are my results for the Git repository:\n\nUntracked Cache Enabled\n-----------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 51.2 ms ± 4.0 ms [User: 12.9 ms, System: 61.2 ms]\n Range (min … max): 46.2 ms … 65.7 ms 54 runs\n \nBenchmark #2: builtin (clean)\n Time (mean ± σ): 38.6 ms ± 1.7 ms [User: 9.9 ms, System: 9.7 ms]\n Range (min … max): 28.6 ms … 42.4 ms 75 runs\n \nSummary\n 'builtin (clean)' ran\n 1.33 ± 0.12 times faster than 'none (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 108.1 ms ± 7.2 ms [User: 27.2 ms, System: 126.9 ms]\n Range (min … max): 97.6 ms … 130.4 ms 25 runs\n \nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 91.7 ms ± 3.8 ms [User: 25.4 ms, System: 27.0 ms]\n Range (min … max): 88.5 ms … 105.1 ms 32 runs\n \nSummary\n 'builtin (dirty)' ran\n 1.18 ± 0.09 times faster than 'none (dirty)'\n\n\nUntracked Cache Disabled\n------------------------\n\nBenchmark #1: none (clean)\n Time (mean ± σ): 59.5 ms ± 4.0 ms [User: 15.2 ms, System: 67.7 ms]\n Range (min … max): 55.5 ms … 71.6 ms 46 runs\n \nBenchmark #2: builtin (clean)\n Time (mean ± σ): 48.9 ms ± 1.0 ms [User: 12.5 ms, System: 17.3 ms]\n Range (min … max): 46.7 ms … 51.3 ms 58 runs\n \nSummary\n 'builtin (clean)' ran\n 1.22 ± 0.08 times faster than 'none (clean)'\n\nBenchmark #1: none (dirty)\n Time (mean ± σ): 124.4 ms ± 6.8 ms [User: 31.5 ms, System: 140.2 ms]\n Range (min … max): 116.8 ms … 140.0 ms 24 runs\n \nBenchmark #2: builtin (dirty)\n Time (mean ± σ): 104.1 ms ± 1.7 ms [User: 27.4 ms, System: 37.8 ms]\n Range (min … max): 99.7 ms … 106.6 ms 27 runs\n \nSummary\n 'builtin (dirty)' ran\n 1.19 ± 0.07 times faster than 'none (dirty)'\n\nI think it valuable to point out that in my initial tests I had forgotten\nto disable the Watchman-based FS Monitor hook, and the results looked even\nmore impressive (on the small Git repository). Dropping the hook overhead is\na huge benefit here.\n\nThanks,\n-Stolee\n\n"},{"id":"423186","messageId":"87k0omzv3h.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"03b4def9-3435-bd44-69f9-1e037df83aaa@gmail.com","subject":"Re: [PATCH 04/23] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-28T07:59:24Z","receivedAt":"2021-04-28T08:12:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Apr 27 2021, Derrick Stolee wrote:\n\n> On 4/27/2021 5:20 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Mon, Apr 26 2021, Derrick Stolee wrote:\n>> \n>>> On 4/1/21 11:40 AM, Johannes Schindelin via GitGitGadget wrote:> @@ -2515,6 +2515,11 @@ int git_config_get_max_percent_split_change(void)\n> ...\n>>>> --- a/repo-settings.c\n>>>> +++ b/repo-settings.c\n>>>> @@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r)\n>>>> \t\tr->settings.core_multi_pack_index = value;\n>>>> \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n>>>> \n>>>> +\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n>>>> +\t\tr->settings.use_builtin_fsmonitor = 1;\n>>>> +\n>>>\n>>> Follows the patterns of repo settings. Good.\n>> \n>> It follows the pattern, but as an aside the pattern seems bit odd. I see\n>> it dates back to your 7211b9e7534 (repo-settings: consolidate some\n>> config settings, 2019-08-13).\n>> \n>> I.e. we memset() the whole thing to -1, then for most things do something like:\n>> \n>> if (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n>> r->settings.gc_write_commit_graph = value;\n>> UPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n>> \n>> But could do:\n>> \n>> if (repo_config_get_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph))\n>> r->settings.gc_write_commit_graph = 1;\n>> \n>> No? I.e. the repo_config_get_bool() function already returns non-zero if\n>> we don't find it in the config.\n>\n> I see how this is fewer lines of code, but it is harder to read the intent\n> of the implementation. The current [...]\n\nThat's exactly the reason I find the existing version unreadable, i.e.:\n\n> layout makes it clear that we set the value from the config, if it\n> exists, but otherwise we choose a default.\n\nThe repo_config_get_*() functions only return non-zero if the value\ndoesn't exist, so the pattern of:\n\n if (repo_config_get(..., \"some.key\", &value))\n value = 123;\n\nIs idiomatic for \"use 123 if some.key doesn't exist in config\".\n\nMaybe I'm missing something and that isn't true, but it seems like a\ncase of going out of one's way to use what the return value is going to\ngive you.\n\n> Sometimes, this choice of a default _needs_ to be deferred, for example with\n> the fetch_negotiation_algorithm setting, which can be set both from the\n> fetch.negotiationAlgorithm config, but also the feature.experimental config.\n\nDon't FETCH_NEGOTIATION_UNSET and UNTRACKED_CACHE_UNSET only exist as\naction-at-a-distance interaction with the memset to -1 that this\nfunction does?\n\nI.e. it's somewhat complex state management, first we set it to\n\"uninit\", then later act on fetch.negotiationalgorithm, and then on\nfeature.experimental, and then set a default only if we didn't do any of\nthe previous things.;\n\nI.e. something like:\n\n x = -1;\n if (fetch.negotiationalgorithm is set)\n if (x != -1 && feature.experimental is set)\n if (x != -1) x = default\n settings->x = x;\n\nAs opposed to a more (to me at least) simpler:\n\n int x;\n if (fetch.negotiationalgorithm is set)\n else if (feature.experimental is set)\n else x = default\n settings->x = x;\n\n> However, perhaps it would be better still for these one-off requests to\n> create a new macro, say USE_CONFIG_OR_DEFAULT_BOOL() that fills a value\n> from config _or_ sets the given default:\n>\n> #define USE_CONFIG_OR_DEFAULT_BOOL(r, v, s, d) \\\n> \tif (repo_config_get_bool(r, s, &v)) \\\n> \t\tv = d\n>\n> And then for this example we would write\n>\n> \tUSE_CONFIG_OR_DEFAULT_BOOL(r, r->settings.core_commit_graph,\n> \t\t\t\t \"core.commitgraph\", 1);\n>\n> This would work for multiple config options in this file.\n\nI came up with this:\n\n+static void repo_env_config_bool_or_default(struct repository *r, const char *env,\n+\t\t\t\t\t const char *key, int *dest, int def)\n+{\n+\tif (env) {\n+\t\tint val = git_env_bool(env, -1);\n+\t\tif (val != -1) {\n+\t\t\t*dest = val;\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\tif (repo_config_get_bool(r, key, dest))\n+\t\t*dest = def;\n+}\n\nUsed as e.g.:\n\n+\trepo_env_config_bool_or_default(r, NULL, \"pack.usesparse\",\n+\t\t\t\t\t&r->settings.pack_use_sparse, 1);\n+\trepo_env_config_bool_or_default(r, GIT_TEST_MULTI_PACK_INDEX, \"core.multipackindex\",\n+\t\t\t\t\t&r->settings.core_multi_pack_index, 1);\n\nIt works for most things there.\n\nUsing that sort of pattern also fixes e.g. a bug in your 18e449f86b7\n(midx: enable core.multiPackIndex by default, 2020-09-25), where we'll\nignore a false-but-existing env config value over a true config key.\n\n>> I see the UPDATE_DEFAULT_BOOL() macro has also drifted from \"set thing\n>> default boolean\" to \"set any default value\".\n> \n> This is correct. I suppose it would be a good change to make some time.\n> Such a rename could be combined with the refactor above.\n>\n> I would recommend waiting until such a change isn't conflicting with\n> ongoing topics, such as this one.\n\nI'm not planning to work on it, but thought I'd ask/prod the original\nauthor if they were interested :)\n"},{"id":"423194","messageId":"e3ef8d87-355e-d3df-9f04-fb52456734dd@jeffhostetler.com","threadId":"55420","inReplyTo":"b868522c-f07f-c431-aa2c-14af84015ced@gmail.com","subject":"Re: [PATCH 01/23] fsmonitor--daemon: man page and documentation","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-28T13:54:23Z","receivedAt":"2021-04-28T13:54:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 10:13 AM, Derrick Stolee wrote:\n> On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Create a manual page describing the `git fsmonitor--daemon` feature.\n>>\n>> Update references to `core.fsmonitor`, `core.fsmonitorHookVersion` and\n>> pointers to `watchman` to mention the built-in FSMonitor.\n> \n> Make sense to add clarity here, since there will be new ways\n> to interact with a fileystem monitor.\n>> core.fsmonitorHookVersion::\n>> -\tSets the version of hook that is to be used when calling fsmonitor.\n>> -\tThere are currently versions 1 and 2. When this is not set,\n>> -\tversion 2 will be tried first and if it fails then version 1\n>> -\twill be tried. Version 1 uses a timestamp as input to determine\n>> -\twhich files have changes since that time but some monitors\n>> -\tlike watchman have race conditions when used with a timestamp.\n>> -\tVersion 2 uses an opaque string so that the monitor can return\n>> -\tsomething that can be used to determine what files have changed\n>> -\twithout race conditions.\n>> +\tSets the version of hook that is to be used when calling the\n>> +\tFSMonitor hook (as configured via `core.fsmonitor`).\n>> ++\n>> +There are currently versions 1 and 2. When this is not set,\n>> +version 2 will be tried first and if it fails then version 1\n>> +will be tried. Version 1 uses a timestamp as input to determine\n>> +which files have changes since that time but some monitors\n>> +like watchman have race conditions when used with a timestamp.\n>> +Version 2 uses an opaque string so that the monitor can return\n>> +something that can be used to determine what files have changed\n>> +without race conditions.\n> \n> This initially seemed like a big edit, but you just split the single\n> paragraph into multiple, with a better leading sentence and a final\n> statement about the built-in FSMonitor. Good.\n>> ++\n>> +Note: FSMonitor hooks (and this config setting) are ignored if the\n>> +built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`).\n>> +\n>> +core.useBuiltinFSMonitor::\n>> +\tIf set to true, enable the built-in filesystem event watcher (for\n>> +\ttechnical details, see linkgit:git-fsmonitor--daemon[1]).\n>> ++\n>> +Like external (hook-based) FSMonitors, the built-in FSMonitor can speed up\n>> +Git commands that need to refresh the Git index (e.g. `git status`) in a\n>> +worktree with many files. The built-in FSMonitor facility eliminates the\n>> +need to install and maintain an external third-party monitoring tool.\n>> ++\n>> +The built-in FSMonitor is currently available only on a limited set of\n>> +supported platforms.\n> \n> Is there a way for users to know this set of platforms? Can they run\n> a command to find out? Will 'git fsmonitor--daemon --start' send a\n> helpful message to assist here? Or, could there be a 'git\n> fsmonitor--daemon --test' command?\n\nI do have a `git fsmonitor--daemon --is-supported` option. It will\nexit with 0 if the current platform is supported.\n\nIt would probably be helpful to list the current platforms and/or\nadd a statement about the `--is-supported` command here.\n\n> \n>> +Note: if this config setting is set to `true`, any FSMonitor hook\n>> +configured via `core.fsmonitor` (and possibly `core.fsmonitorHookVersion`)\n>> +is ignored.\n> ...\n>> +git-fsmonitor--daemon(1)\n>> +========================\n>> +\n>> +NAME\n>> +----\n>> +git-fsmonitor--daemon - Builtin file system monitor daemon\n>> +\n>> +SYNOPSIS\n>> +--------\n>> +[verse]\n>> +'git fsmonitor--daemon' --start\n>> +'git fsmonitor--daemon' --run\n>> +'git fsmonitor--daemon' --stop\n>> +'git fsmonitor--daemon' --is-running\n>> +'git fsmonitor--daemon' --is-supported\n>> +'git fsmonitor--daemon' --query \n>> +'git fsmonitor--daemon' --query-index\n>> +'git fsmonitor--daemon' --flush\n> \n> These arguments with the \"--\" prefix make it seem like they are\n> options that could be grouped together, but you really want these\n> to be verbs within the daemon. What do you think about removing\n> the \"--\" prefixes?\n\nThat's easy enough. The OPT_CMDMODE() made it easy to do it this\nway.\n\n> \n>> +\n>> +DESCRIPTION\n>> +-----------\n>> +\n>> +Monitors files and directories in the working directory for changes using\n>> +platform-specific file system notification facilities.\n>> +\n>> +It communicates directly with commands like `git status` using the\n>> +link:technical/api-simple-ipc.html[simple IPC] interface instead of\n>> +the slower linkgit:githooks[5] interface.\n>> +\n>> +OPTIONS\n>> +-------\n> \n> I typically view \"OPTIONS\" as arguments that can be grouped together,\n> but you are describing things more like verbs or subcommands. The\n> most recent example I know about is 'git maintenance ',\n> documented at [1].\n> \n> [1] https://git-scm.com/docs/git-maintenance#_subcommands\n\nLet me take a look at doing the subcommand way.\n\n> \n>> +\n>> +--start::\n>> +\tStarts the fsmonitor daemon in the background.\n>> +\n>> +--run::\n>> +\tRuns the fsmonitor daemon in the foreground.\n>> +\n>> +--stop::\n>> +\tStops the fsmonitor daemon running for the current working\n>> +\tdirectory, if present.\n> \n> I'm noticing \"fsmonitor\" in lowercase throughout this document. Is\n> that the intended case for user-facing documentation? I've been\n> seeing \"FS Monitor\", \"filesystem monitor\", or even \"File System\n> Monitor\" in other places.\n\nI think I want to rewrite this whole man-page and address all\nof the different spellings and phrasing.\n\n\n> \n>> +--is-running::\n>> +\tExits with zero status if the fsmonitor daemon is watching the\n>> +\tcurrent working directory.\n> \n> Another potential name for this verb is \"status\".\n> \n>> +--is-supported::\n>> +\tExits with zero status if the fsmonitor daemon feature is supported\n>> +\ton this platform.\n> \n> Ah, here is an indicator of whether the platform is supported. Please\n> include details for this command in the earlier documentation. I'll\n> check later to see if a message is also sent over 'stderr', which\n> would be helpful. Documenting the exit status is good for third-party\n> tools that might use this.\n> \n>> +--query ::\n>> +\tConnects to the fsmonitor daemon (starting it if necessary) and\n>> +\trequests the list of changed files and directories since the\n>> +\tgiven token.\n>> +\tThis is intended for testing purposes.\n>> +\n>> +--query-index::\n>> +\tRead the current `` from the File System Monitor index\n>> +\textension (if present) and use it to query the fsmonitor daemon.\n>> +\tThis is intended for testing purposes.\n> \n> These two could be grouped as \"query [--token=X|--index]\", especially\n> because they are for testing purposes.\n> \n>> +\n>> +--flush::\n>> +\tForce the fsmonitor daemon to flush its in-memory cache and\n>> +\tre-sync with the file system.\n>> +\tThis is intended for testing purposes.\n> \n> Do you see benefits to these being available in the CLI? Could these\n> be better served as a test helper?\n\nI debated putting the 3 test options into a test helper.\nLet me take a look at that.\n\n> \n>> +REMARKS\n>> +-------\n>> +The fsmonitor daemon is a long running process that will watch a single\n>> +working directory. Commands, such as `git status`, should automatically\n>> +start it (if necessary) when `core.useBuiltinFSMonitor` is set to `true`\n>> +(see linkgit:git-config[1]).\n>> +\n>> +Configure the built-in FSMonitor via `core.useBuiltinFSMonitor` in each\n>> +working directory separately, or globally via `git config --global\n>> +core.useBuiltinFSMonitor true`.\n>> +\n>> +Tokens are opaque strings. They are used by the fsmonitor daemon to\n>> +mark a point in time and the associated internal state. Callers should\n>> +make no assumptions about the content of the token. In particular,\n>> +the should not assume that it is a timestamp.\n>> +\n>> +Query commands send a request-token to the daemon and it responds with\n>> +a summary of the changes that have occurred since that token was\n>> +created. The daemon also returns a response-token that the client can\n>> +use in a future query.\n>> +\n>> +For more information see the \"File System Monitor\" section in\n>> +linkgit:git-update-index[1].\n>> +\n>> +CAVEATS\n>> +-------\n>> +\n>> +The fsmonitor daemon does not currently know about submodules and does\n>> +not know to filter out file system events that happen within a\n>> +submodule. If fsmonitor daemon is watching a super repo and a file is\n>> +modified within the working directory of a submodule, it will report\n>> +the change (as happening against the super repo). However, the client\n>> +should properly ignore these extra events, so performance may be affected\n>> +but it should not cause an incorrect result.\n> \n> There are several uses of the word \"should\" where I think \"will\" is a\n> more appropriate word. That is, unless we do not actually have confidence\n> in this behavior.\n\nI think I was just being overly conservative in my language.\n\n> \n>> --- a/Documentation/git-update-index.txt\n>> +++ b/Documentation/git-update-index.txt\n>> @@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n>> This feature is intended to speed up git operations for repos that have\n>> large working directories.\n>> \n>> -It enables git to work together with a file system monitor (see the\n>> +It enables git to work together with a file system monitor (see\n>> +linkgit:git-fsmonitor--daemon[1]\n>> +and the\n>> \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n>> inform it as to what files have been modified. This enables git to avoid\n>> having to lstat() every file to find modified files.\n>> diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\n>> index b51959ff9418..b7d5e926f7b0 100644\n>> --- a/Documentation/githooks.txt\n>> +++ b/Documentation/githooks.txt\n>> @@ -593,7 +593,8 @@ fsmonitor-watchman\n>> \n>> This hook is invoked when the configuration option `core.fsmonitor` is\n>> set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n>> -depending on the version of the hook to use.\n>> +depending on the version of the hook to use, unless overridden via\n>> +`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n>> \n>> Version 1 takes two arguments, a version (1) and the time in elapsed\n>> nanoseconds since midnight, January 1, 1970.\n> \n> These are good connections to make.\n> \n> Since the documentation for the fsmonitor--daemon is so deep, this\n> patch might be served well to split into two: one that just documents\n> the daemon, and another that updates existing documentation to point\n> to the new file.\n\nGood point. Thanks!\n\n> \n> This does provide a good basis for me to investigate during the rest\n> of the review.\n> \n> Thanks,\n> -Stolee\n> \n\nThanks\nJeff\n"},{"id":"423201","messageId":"patch-1.1-e1d8c842c70-20210428T161817Z-avarab@gmail.com","threadId":"55420","inReplyTo":"87k0omzv3h.fsf@evledraar.gmail.com","subject":"[PATCH] repo-settings.c: simplify the setup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-28T16:26:43Z","receivedAt":"2021-04-28T16:26:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Simplify the setup code in repo-settings.c in various ways, making the\ncode shorter, easier to read, and requiring fewer hacks to do the same\nthing as it did before:\n\nSince 7211b9e7534 (repo-settings: consolidate some config settings,\n2019-08-13) we have memset() the whole \"settings\" structure to -1, and\nsubsequently relied on the -1 value. As it turns out most things did\nnot need to be initialized to -1, and e.g. UNTRACKED_CACHE_UNSET and\nFETCH_NEGOTIATION_UNSET existed purely to reflect the previous\ninternal state of the prepare_repo_settings() function.\n\nMuch of the \"are we -1, then read xyz\" can simply be removed by\nre-arranging what we read first. E.g. we should read\nfeature.experimental first, set some values, and then e.g. an explicit\nindex.version setting should override that. We don't need to read\nindex.version first, and then check when reading feature.experimental\nif it's still -1.\n\nInstead of the global ignore_untracked_cache_config variable added in\ndae6c322fa1 (test-dump-untracked-cache: don't modify the untracked\ncache, 2016-01-27) we can make use of the new facility to set config\nvia environment variables added in d8d77153eaf (config: allow\nspecifying config entries via envvar pairs, 2021-01-12).\n\nIt's arguably a bit hacky to use setenv() and getenv() to pass\nmessages between the same program, but since the test helpers are not\nthe main intended audience of repo-settings.c I think it's better than\nhardcoding the test-only special-case in prepare_repo_settings().\n\nIn ad0fb659993 (repo-settings: parse core.untrackedCache, 2019-08-13)\nthe \"unset\" and \"keep\" handling for core.untrackedCache was\nconsolidated. But it apparently wasn't noticed that while we\nunderstand the \"keep\" value, we actually don't handle it differently\nthan the case of any other unknown value.\n\nSo we can remove UNTRACKED_CACHE_KEEP from the codebase. It's not\nhandled any differently than UNTRACKED_CACHE_UNSET once we get past\nthe config parsing step.\n\nThe UPDATE_DEFAULT_BOOL() wrapper added in 31b1de6a09b (commit-graph:\nturn on commit-graph by default, 2019-08-13) is redundant to simply\nusing the return value from repo_config_get_bool(), which is non-zero\nif the provided key exists in the config.\n\nThis also fixes an (admittedly obscure) logic error in the previous\ncode where we'd conflate an explicit \"-1\" value in the config with our\nown earlier memset() -1.\n\nSince the two enum fields added in aaf633c2ad1 (repo-settings: create\nfeature.experimental setting, 2019-08-13) and\nad0fb659993 (repo-settings: parse core.untrackedCache, 2019-08-13)\ndon't rely on the memzero() setting them to \"-1\" anymore we don't have\nto provide them with explicit values. Let's also explicitly use the\nenum type in read-cache.c and fetch-negotiator.c for\nself-documentation. Since the FETCH_NEGOTIATION_UNSET is gone we can\nremove the \"default\" case in fetch-negotiator.c, and rely on the\ncompiler to complain about missing enum values instead.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n\nOn Wed, Apr 28 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Tue, Apr 27 2021, Derrick Stolee wrote:\n>\n>> This is correct. I suppose it would be a good change to make some time.\n>> Such a rename could be combined with the refactor above.\n>>\n>> I would recommend waiting until such a change isn't conflicting with\n>> ongoing topics, such as this one.\n>\n> I'm not planning to work on it, but thought I'd ask/prod the original\n> author if they were interested :)\n\nSeems I'm pretty bad at sticking to my plans. Here's that refactoring,\nsince I mostly had this hacked-up locally anyway.\n\nThe conflict with the fsmonitor work can be resolved by adding:\n\n\trepo_config_get_bool_or(r, \"core.usebuiltinfsmonitor\",\n\t\t\t\t&r->settings.use_builtin_fsmonitor, 0);\n\nTo the \"Boolean config or default, does not cascade (simple)\" section\nin my version. I.e. I assume nothing past 04/23 cares about the case\nwhere it was set to \"-1\", which as noted in the commit message above\nwas (like many other setting variables) leaking an internal\nimplementation detail.\n\n cache.h | 7 --\n environment.c | 7 --\n fetch-negotiator.c | 6 +-\n read-cache.c | 17 ++--\n repo-settings.c | 119 +++++++++++++++------------\n repository.h | 15 ++--\n t/helper/test-dump-untracked-cache.c | 6 +-\n 7 files changed, 92 insertions(+), 85 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 148d9ab5f18..7ea0feb3462 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1684,13 +1684,6 @@ int update_server_info(int);\n const char *get_log_output_encoding(void);\n const char *get_commit_output_encoding(void);\n \n-/*\n- * This is a hack for test programs like test-dump-untracked-cache to\n- * ensure that they do not modify the untracked cache when reading it.\n- * Do not use it otherwise!\n- */\n-extern int ignore_untracked_cache_config;\n-\n int committer_ident_sufficiently_given(void);\n int author_ident_sufficiently_given(void);\n \ndiff --git a/environment.c b/environment.c\nindex 2f27008424a..bc825cc7e05 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -96,13 +96,6 @@ int auto_comment_line_char;\n /* Parallel index stat data preload? */\n int core_preload_index = 1;\n \n-/*\n- * This is a hack for test programs like test-dump-untracked-cache to\n- * ensure that they do not modify the untracked cache when reading it.\n- * Do not use it otherwise!\n- */\n-int ignore_untracked_cache_config;\n-\n /* This is set by setup_git_dir_gently() and/or git_default_config() */\n char *git_work_tree_cfg;\n \ndiff --git a/fetch-negotiator.c b/fetch-negotiator.c\nindex 57ed5784e14..c7c0eda7e21 100644\n--- a/fetch-negotiator.c\n+++ b/fetch-negotiator.c\n@@ -8,8 +8,11 @@\n void fetch_negotiator_init(struct repository *r,\n \t\t\t struct fetch_negotiator *negotiator)\n {\n+\tenum fetch_negotiation_setting setting;\n \tprepare_repo_settings(r);\n-\tswitch(r->settings.fetch_negotiation_algorithm) {\n+\tsetting = r->settings.fetch_negotiation_algorithm;\n+\n+\tswitch (setting) {\n \tcase FETCH_NEGOTIATION_SKIPPING:\n \t\tskipping_negotiator_init(negotiator);\n \t\treturn;\n@@ -19,7 +22,6 @@ void fetch_negotiator_init(struct repository *r,\n \t\treturn;\n \n \tcase FETCH_NEGOTIATION_DEFAULT:\n-\tdefault:\n \t\tdefault_negotiator_init(negotiator);\n \t\treturn;\n \t}\ndiff --git a/read-cache.c b/read-cache.c\nindex 5a907af2fb5..1aefe4a5c23 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1889,16 +1889,23 @@ static void check_ce_order(struct index_state *istate)\n static void tweak_untracked_cache(struct index_state *istate)\n {\n \tstruct repository *r = the_repository;\n+\tenum untracked_cache_setting setting;\n \n \tprepare_repo_settings(r);\n+\tsetting = r->settings.core_untracked_cache;\n \n-\tif (r->settings.core_untracked_cache == UNTRACKED_CACHE_REMOVE) {\n+\tswitch (setting) {\n+\tcase UNTRACKED_CACHE_REMOVE:\n \t\tremove_untracked_cache(istate);\n-\t\treturn;\n-\t}\n-\n-\tif (r->settings.core_untracked_cache == UNTRACKED_CACHE_WRITE)\n+\t\tbreak;\n+\tcase UNTRACKED_CACHE_WRITE:\n \t\tadd_untracked_cache(istate);\n+\t\tbreak;\n+\tcase UNTRACKED_CACHE_UNSET:\n+\t\t/* This includes core.untrackedCache=keep */\n+\t\tbreak;\n+\t}\n+\treturn;\n }\n \n static void tweak_split_index(struct index_state *istate)\ndiff --git a/repo-settings.c b/repo-settings.c\nindex f7fff0f5ab8..2be242fde1d 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -3,40 +3,84 @@\n #include \"repository.h\"\n #include \"midx.h\"\n \n-#define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0)\n+static void repo_config_get_bool_or(struct repository *r, const char *key,\n+\t\t\t\t int *dest, int def)\n+{\n+\tif (repo_config_get_bool(r, key, dest))\n+\t\t*dest = def;\n+}\n \n void prepare_repo_settings(struct repository *r)\n {\n-\tint value;\n+\tint experimental;\n+\tint intval;\n \tchar *strval;\n+\tint manyfiles;\n \n \tif (r->settings.initialized)\n \t\treturn;\n \n \t/* Defaults */\n-\tmemset(&r->settings, -1, sizeof(r->settings));\n+\tr->settings.index_version = -1;\n+\tr->settings.core_untracked_cache = UNTRACKED_CACHE_UNSET;\n+\tr->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_DEFAULT;\n+\n+\t/* Booleans config or default, cascades to other settings */\n+\trepo_config_get_bool_or(r, \"feature.manyfiles\", &manyfiles, 0);\n+\trepo_config_get_bool_or(r, \"feature.experimental\", &experimental, 0);\n+\n+\t/* Defaults modified by feature.* */\n+\tif (experimental) {\n+\t\tr->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_SKIPPING;\n+\t}\n+\tif (manyfiles) {\n+\t\tr->settings.index_version = 4;\n+\t\tr->settings.core_untracked_cache = UNTRACKED_CACHE_WRITE;\n+\t}\n \n-\tif (!repo_config_get_bool(r, \"core.commitgraph\", &value))\n-\t\tr->settings.core_commit_graph = value;\n-\tif (!repo_config_get_bool(r, \"commitgraph.readchangedpaths\", &value))\n-\t\tr->settings.commit_graph_read_changed_paths = value;\n-\tif (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n-\t\tr->settings.gc_write_commit_graph = value;\n-\tUPDATE_DEFAULT_BOOL(r->settings.core_commit_graph, 1);\n-\tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n-\tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n+\t/* Boolean config or default, does not cascade (simple) */\n+\trepo_config_get_bool_or(r, \"core.commitgraph\",\n+\t\t\t\t&r->settings.core_commit_graph, 1);\n+\trepo_config_get_bool_or(r, \"commitgraph.readchangedpaths\",\n+\t\t\t\t&r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_config_get_bool_or(r, \"gc.writecommitgraph\",\n+\t\t\t\t&r->settings.gc_write_commit_graph, 1);\n+\trepo_config_get_bool_or(r, \"fetch.writecommitgraph\",\n+\t\t\t\t&r->settings.fetch_write_commit_graph, 0);\n+\trepo_config_get_bool_or(r, \"pack.usesparse\",\n+\t\t\t\t&r->settings.pack_use_sparse, 1);\n+\trepo_config_get_bool_or(r, \"core.multipackindex\",\n+\t\t\t\t&r->settings.core_multi_pack_index, 1);\n \n-\tif (!repo_config_get_int(r, \"index.version\", &value))\n-\t\tr->settings.index_version = value;\n-\tif (!repo_config_get_maybe_bool(r, \"core.untrackedcache\", &value)) {\n-\t\tif (value == 0)\n-\t\t\tr->settings.core_untracked_cache = UNTRACKED_CACHE_REMOVE;\n-\t\telse\n-\t\t\tr->settings.core_untracked_cache = UNTRACKED_CACHE_WRITE;\n-\t} else if (!repo_config_get_string(r, \"core.untrackedcache\", &strval)) {\n-\t\tif (!strcasecmp(strval, \"keep\"))\n-\t\t\tr->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;\n+\t/*\n+\t * The GIT_TEST_MULTI_PACK_INDEX variable is special in that\n+\t * either it *or* the config sets\n+\t * r->settings.core_multi_pack_index if true. We don't take\n+\t * the environment variable if it exists (even if false) over\n+\t * any config, as in other cases.\n+\t */\n+\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n+\t\tr->settings.core_multi_pack_index = 1;\n \n+\t/*\n+\t * Non-boolean config\n+\t */\n+\tif (!repo_config_get_int(r, \"index.version\", &intval))\n+\t\tr->settings.index_version = intval;\n+\n+\tif (!repo_config_get_string(r, \"core.untrackedcache\", &strval)) {\n+\t\tint maybe_bool = git_parse_maybe_bool(strval);\n+\t\tif (maybe_bool == -1) {\n+\t\t\t/*\n+\t\t\t * Set to \"keep\", or some other non-boolean\n+\t\t\t * value. In either case we do nothing but\n+\t\t\t * keep UNTRACKED_CACHE_UNSET.\n+\t\t\t */\n+\t\t} else {\n+\t\t\tr->settings.core_untracked_cache = maybe_bool\n+\t\t\t\t? UNTRACKED_CACHE_WRITE\n+\t\t\t\t: UNTRACKED_CACHE_REMOVE;\n+\t\t}\n \t\tfree(strval);\n \t}\n \n@@ -45,36 +89,5 @@ void prepare_repo_settings(struct repository *r)\n \t\t\tr->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_SKIPPING;\n \t\telse if (!strcasecmp(strval, \"noop\"))\n \t\t\tr->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_NOOP;\n-\t\telse\n-\t\t\tr->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_DEFAULT;\n \t}\n-\n-\tif (!repo_config_get_bool(r, \"pack.usesparse\", &value))\n-\t\tr->settings.pack_use_sparse = value;\n-\tUPDATE_DEFAULT_BOOL(r->settings.pack_use_sparse, 1);\n-\n-\tvalue = git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0);\n-\tif (value || !repo_config_get_bool(r, \"core.multipackindex\", &value))\n-\t\tr->settings.core_multi_pack_index = value;\n-\tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n-\n-\tif (!repo_config_get_bool(r, \"feature.manyfiles\", &value) && value) {\n-\t\tUPDATE_DEFAULT_BOOL(r->settings.index_version, 4);\n-\t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);\n-\t}\n-\n-\tif (!repo_config_get_bool(r, \"fetch.writecommitgraph\", &value))\n-\t\tr->settings.fetch_write_commit_graph = value;\n-\tUPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0);\n-\n-\tif (!repo_config_get_bool(r, \"feature.experimental\", &value) && value)\n-\t\tUPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING);\n-\n-\t/* Hack for test programs like test-dump-untracked-cache */\n-\tif (ignore_untracked_cache_config)\n-\t\tr->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;\n-\telse\n-\t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_KEEP);\n-\n-\tUPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_DEFAULT);\n }\ndiff --git a/repository.h b/repository.h\nindex b385ca3c94b..9345423c5ba 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -12,18 +12,15 @@ struct raw_object_store;\n struct submodule_cache;\n \n enum untracked_cache_setting {\n-\tUNTRACKED_CACHE_UNSET = -1,\n-\tUNTRACKED_CACHE_REMOVE = 0,\n-\tUNTRACKED_CACHE_KEEP = 1,\n-\tUNTRACKED_CACHE_WRITE = 2\n+\tUNTRACKED_CACHE_UNSET,\n+\tUNTRACKED_CACHE_REMOVE,\n+\tUNTRACKED_CACHE_WRITE,\n };\n \n enum fetch_negotiation_setting {\n-\tFETCH_NEGOTIATION_UNSET = -1,\n-\tFETCH_NEGOTIATION_NONE = 0,\n-\tFETCH_NEGOTIATION_DEFAULT = 1,\n-\tFETCH_NEGOTIATION_SKIPPING = 2,\n-\tFETCH_NEGOTIATION_NOOP = 3,\n+\tFETCH_NEGOTIATION_DEFAULT,\n+\tFETCH_NEGOTIATION_SKIPPING,\n+\tFETCH_NEGOTIATION_NOOP,\n };\n \n struct repo_settings {\ndiff --git a/t/helper/test-dump-untracked-cache.c b/t/helper/test-dump-untracked-cache.c\nindex cf0f2c7228e..8b73a2f8bc3 100644\n--- a/t/helper/test-dump-untracked-cache.c\n+++ b/t/helper/test-dump-untracked-cache.c\n@@ -45,8 +45,10 @@ int cmd__dump_untracked_cache(int ac, const char **av)\n \tstruct untracked_cache *uc;\n \tstruct strbuf base = STRBUF_INIT;\n \n-\t/* Hack to avoid modifying the untracked cache when we read it */\n-\tignore_untracked_cache_config = 1;\n+\t/* Set core.untrackedCache=keep before setup_git_directory() */\n+\tsetenv(\"GIT_CONFIG_COUNT\", \"1\", 1);\n+\tsetenv(\"GIT_CONFIG_KEY_0\", \"core.untrackedCache\", 1);\n+\tsetenv(\"GIT_CONFIG_VALUE_0\", \"keep\", 1);\n \n \tsetup_git_directory();\n \tif (read_cache() < 0)\n-- \n2.31.1.734.g8d26f61af32\n\n"},{"id":"423213","messageId":"1ecb3727-106f-3d04-976a-36aa03a61caf@gmail.com","threadId":"55420","inReplyTo":"patch-1.1-e1d8c842c70-20210428T161817Z-avarab@gmail.com","subject":"Nesting topics within other threads (was: [PATCH] repo-settings.c: simplify the setup)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-28T19:09:28Z","receivedAt":"2021-04-28T19:09:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/28/2021 12:26 PM, Ævar Arnfjörð Bjarmason wrote:\n> Simplify the setup code in repo-settings.c in various ways, making the\n> code shorter, easier to read, and requiring fewer hacks to do the same\n> thing as it did before:\n\nThis patch is interesting, and I'll review it when I have some more\ntime. Probably tomorrow.\n\nBut I thought that I would point out that this pattern of adding a\npatch within the thread of a larger series makes it very difficult\nto separate the two. I use an email client that groups messages by\nthread in order to help parse meaningful discussion from the list\nwhich otherwise looks like a fire hose of noise. Now, this patch is\nlinked to the FS Monitor thread and feedback to either will trigger\nthe thread as having unread messages.\n\nI find it very difficult to track multiple patch series that are\nbeing juggled in the same thread. It is mentally taxing enough that\nI have avoided reviewing code presented this way to save myself the\neffort of tracking which patches go with what topic in what order.\n\nSince I've committed to reviewing the FS Monitor code, I'd prefer if\nthis patch (or maybe its v2, since this is here already) be sent as\na top-level message so it can be discussed independently.\n\nThanks,\n-Stolee\n"},{"id":"423214","messageId":"98038a12-4750-456b-6535-8bc59a32693c@jeffhostetler.com","threadId":"55420","inReplyTo":"5b63eb3f-0b87-209b-3943-ced8412abb93@gmail.com","subject":"Re: [PATCH 02/23] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-28T19:26:41Z","receivedAt":"2021-04-28T19:26:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 10:31 AM, Derrick Stolee wrote:\n> On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n>> +#define FSMONITOR_DAEMON_IS_SUPPORTED 1\n>> +#else\n>> +#define FSMONITOR_DAEMON_IS_SUPPORTED 0\n>> +#endif\n>> +\n>> +/*\n>> + * A trivial function so that this source file always defines at least\n>> + * one symbol even when the feature is not supported. This quiets an\n>> + * annoying compiler error.\n>> + */\n>> +int fsmonitor_ipc__is_supported(void)\n>> +{\n>> +\treturn FSMONITOR_DAEMON_IS_SUPPORTED;\n>> +}\n> \n> I don't see any other use of FSMONITOR_DAEMON_IS_SUPPORTED,\n> so I was thinking you could use the #ifdef/#else/#endif\n> construct within the implementation of this method instead\n> of creating a macro outside. But my suggestion might be an\n> anti-pattern, so feel free to ignore me.\n\nI think an earlier draft did more with the macros\nand wasn't as distilled as it is here. So yes, I\ncould simplify it here.\n\n\n>> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n>> +\n>> +GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor\")\n>> +\n>> +enum ipc_active_state fsmonitor_ipc__get_state(void)\n>> +{\n>> +\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n>> +}\n>> +\n>> +static int spawn_daemon(void)\n>> +{\n>> +\tconst char *args[] = { \"fsmonitor--daemon\", \"--start\", NULL };\n>> +\n>> +\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n>> +\t\t\t\t \"fsmonitor\");\n>> +}\n>> +\n>> +int fsmonitor_ipc__send_query(const char *since_token,\n>> +\t\t\t struct strbuf *answer)\n>> +{\n>> +\tint ret = -1;\n>> +\tint tried_to_spawn = 0;\n>> +\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n>> +\tstruct ipc_client_connection *connection = NULL;\n>> +\tstruct ipc_client_connect_options options\n>> +\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n>> +\n>> +\toptions.wait_if_busy = 1;\n>> +\toptions.wait_if_not_found = 0;\n>> +\n>> +\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n>> +\n>> +\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\",\n>> +\t\t\t since_token);\n>> +\n>> +try_again:\n>> +\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n>> +\t\t\t\t &connection);\n>> +\n>> +\tswitch (state) {\n>> +\tcase IPC_STATE__LISTENING:\n>> +\t\tret = ipc_client_send_command_to_connection(\n>> +\t\t\tconnection, since_token, answer);\n>> +\t\tipc_client_close_connection(connection);\n>> +\n>> +\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n>> +\t\t\t\t \"query/response-length\", answer->len);\n>> +\n>> +\t\tif (fsmonitor_is_trivial_response(answer))\n>> +\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n>> +\t\t\t\t\t \"query/trivial-response\", 1);\n>> +\n>> +\t\tgoto done;\n>> +\n>> +\tcase IPC_STATE__NOT_LISTENING:\n>> +\t\tret = error(_(\"fsmonitor_ipc__send_query: daemon not available\"));\n>> +\t\tgoto done;\n> \n> I'll need to read up on the IPC layer a bit to find out the difference\n> between IPC_STATE__NOT_LISTENING and IPC_STATE__PATH_NOT_FOUND. When\n> testing on my macOS machine, I got this error. I was expecting the\n> daemon to be spawned. After spawning it myself, it started working.\n> \n> I expect that there are some cases where the process can fail and the\n> named pipe is not cleaned up. Let's investigate that soon. I should\n> make it clear that I had tested the builtin FS Monitor on this machine\n> a few weeks ago, but hadn't been using it much since. We should auto-\n> recover from this situation.\n\nThis probably means that you had a left over (dead) unix domain\nsocket in your .git directory from your prior testing. I delete\nit when the daemon shuts down normally, but if it was killed or\ncrashed, it may have been left behind.\n\nWhen the client tries to connect (to a socket with no listener)\nthe OS refuses the connection and the IPC layer maps that back\nto the __NOT_LISTENING error.\n\n(There are other ways to get __NOT_LISTENING error, such as when\nthe client times-out because the daemon is too busy to respond,\nbut I'll ignore that for now.)\n\nThe unix daemon startup code tries to gently create the socket\n(without stealing it from an existing server) and if that fails\n(because there is no server present), it force creates a new socket\nand starts listening on it. (There was a large conversation on the\nSimple IPC patch series about this.) So this is how it fixed itself\nafter you started the daemon.\n\n\n> But also: what is the cost of treating these two cases the same? Could\n> we attempt to \"restart\" the daemon by spawning a new one? Will the new\n> one find a way to kill a stale one?\n\nOn Windows, named pipes are magically deleted when the last handle\nis closed (they are hosted on a special Named Pipe FS rather than NTFS,\nso they have slightly different semantics). If a named pipe exists,\nbut the connect fails, then a server is present but busy (or wedged).\nThe __NOT_LISTENING error basically means that the connect timed out.\nSo we know that the server is technically present, but it did not\nrespond.\n\n\nOn both platforms, if the socket/pipe is not present then the connect\nreturns __PATH_NOT_FOUND. So we know that no daemon is present and are\nfree to implicitly start one.\n\n\nThe subtle difference in the __NOT_LISTENING case between the platforms\nis why I hesitated to implicitly start (or restart) the daemon in this\ncase.\n\nI would like to revisit auto-starting the daemon (at least on Unix)\nwhen we have a dead socket. I'll review this.\n\nThanks for the question.\n\n\n> \n> (Reading on.)\n> \n>> +\tcase IPC_STATE__PATH_NOT_FOUND:\n>> +\t\tif (tried_to_spawn)\n>> +\t\t\tgoto done;\n>> +\n>> +\t\ttried_to_spawn++;\n>> +\t\tif (spawn_daemon())\n>> +\t\t\tgoto done;\n> \n> This should return zero on success, OK.\n> \n>> +\t\t/*\n>> +\t\t * Try again, but this time give the daemon a chance to\n>> +\t\t * actually create the pipe/socket.\n>> +\t\t *\n>> +\t\t * Granted, the daemon just started so it can't possibly have\n>> +\t\t * any FS cached yet, so we'll always get a trivial answer.\n>> +\t\t * BUT the answer should include a new token that can serve\n>> +\t\t * as the basis for subsequent requests.\n>> +\t\t */\n>> +\t\toptions.wait_if_not_found = 1;\n>> +\t\tgoto try_again;\n> \n> Because of the tried_to_spawn check, we will re-run the request over\n> IPC but will not retry the spawn_daemon() request. I'm unsure how\n> this could be helpful: is it possible that spawn_daemon() returns a\n> non-zero error code after starting the daemon and somehow that\n> daemon starts working? Or, is this a race-condition thing with parallel\n> processes also starting up the daemon? It could be good to use this\n> comment to describe why a retry might be helpful.\n\nI'm trying to be fairly conservative here. If no daemon/socket/pipe\nis present, we try once to start it and then (with a small delay) try\nto connect to the new daemon. There is a little race with our process\nand the new daemon instance, but we have the client spin a little to\ngive the daemon a chance to get started up. Normally, that connect\nwill then succeed.\n\nIf that new daemon fails to start or we have some other error, we\nneed to just give up and tell the caller to do the work -- we've\nalready held up the caller long enough IMHO.\n\nThe thought here is that if that first daemon failed to start, then\nsubsequent attempts are likely to also fail. And we don't want to\ncause the client to get stuck trying to repeatedly start the daemon.\nBetter to just give up and go on.\n\n> \n>> +\n>> +\tcase IPC_STATE__INVALID_PATH:\n>> +\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n>> +\t\t\t fsmonitor_ipc__get_path());\n>> +\t\tgoto done;\n>> +\n>> +\tcase IPC_STATE__OTHER_ERROR:\n>> +\tdefault:\n>> +\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n>> +\t\t\t fsmonitor_ipc__get_path());\n>> +\t\tgoto done;\n>> +\t}\n>> +\n>> +done:\n>> +\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n>> +\n>> +\treturn ret;\n>> +}\n>> +\n>> +int fsmonitor_ipc__send_command(const char *command,\n>> +\t\t\t\tstruct strbuf *answer)\n>> +{\n>> +\tstruct ipc_client_connection *connection = NULL;\n>> +\tstruct ipc_client_connect_options options\n>> +\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n>> +\tint ret;\n>> +\tenum ipc_active_state state;\n>> +\n>> +\tstrbuf_reset(answer);\n>> +\n>> +\toptions.wait_if_busy = 1;\n>> +\toptions.wait_if_not_found = 0;\n>> +\n>> +\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n>> +\t\t\t\t &connection);\n>> +\tif (state != IPC_STATE__LISTENING) {\n>> +\t\tdie(\"fsmonitor--daemon is not running\");\n>> +\t\treturn -1;\n>> +\t}\n>> +\n>> +\tret = ipc_client_send_command_to_connection(connection, command, answer);\n>> +\tipc_client_close_connection(connection);\n>> +\n>> +\tif (ret == -1) {\n>> +\t\tdie(\"could not send '%s' command to fsmonitor--daemon\",\n>> +\t\t command);\n>> +\t\treturn -1;\n>> +\t}\n>> +\n>> +\treturn 0;\n>> +}\n> \n> I wondier if this ...send_command() method is too generic. It might\n> be nice to have more structure to its inputs and outputs to lessen\n> the cognitive load when plugging into other portions of the code.\n> However, I'll wait to see what those consumers look like in case the\n> generality is merited.\n>> struct category_description {\n>> \tuint32_t category;\n>> @@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n>> \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n>> \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n>> \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n>> +\n>> +\t\tif (fsmonitor_ipc__is_supported())\n>> +\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n> \n> This change might deserve its own patch, including some documentation\n> about how users can use 'git version --build-options' to determine if\n> the builtin FS Monitor feature is available on their platform.\n> \n\nGood point. Thanks.\n\n> Thanks,\n> -Stolee\n> \n"},{"id":"423219","messageId":"87eeeuymtl.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"1ecb3727-106f-3d04-976a-36aa03a61caf@gmail.com","subject":"Re: Nesting topics within other threads (was: [PATCH] repo-settings.c: simplify the setup)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-28T23:01:28Z","receivedAt":"2021-04-29T00:08:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Apr 28 2021, Derrick Stolee wrote:\n\n> On 4/28/2021 12:26 PM, Ævar Arnfjörð Bjarmason wrote:\n>> Simplify the setup code in repo-settings.c in various ways, making the\n>> code shorter, easier to read, and requiring fewer hacks to do the same\n>> thing as it did before:\n>\n> This patch is interesting, and I'll review it when I have some more\n> time. Probably tomorrow.\n>\n> But I thought that I would point out that this pattern of adding a\n> patch within the thread of a larger series makes it very difficult\n> to separate the two. I use an email client that groups messages by\n> thread in order to help parse meaningful discussion from the list\n> which otherwise looks like a fire hose of noise. Now, this patch is\n> linked to the FS Monitor thread and feedback to either will trigger\n> the thread as having unread messages.\n>\n> I find it very difficult to track multiple patch series that are\n> being juggled in the same thread. It is mentally taxing enough that\n> I have avoided reviewing code presented this way to save myself the\n> effort of tracking which patches go with what topic in what order.\n>\n> Since I've committed to reviewing the FS Monitor code, I'd prefer if\n> this patch (or maybe its v2, since this is here already) be sent as\n> a top-level message so it can be discussed independently.\n\nAs a practical matter I think any effort I make to accommodate your\nrequest will be dwarfed by your own starting of a sub-thread on\nE-Mail/MUA nuances :)\n\nWhen [1] was brought up the other day (showing that I'm probably not the\nbest person to ask about on-list In-Reply-To semantics) I was surprised\nto find that we don't have much (if any) explicit documentation about\nIn-Reply-To best practices. There's a passing mention in\nDocumentation/MyFirstContribution.txt, but as far as I can tell from a\ncursory glance that's it.\n\nPersonally I draw the line at \"this random unrelated thing occurred to\nme while reading X\" v.s. \"this is directly in reply to X\".\n\nReading the upthread I don't really see a good point at which to start\nbreaking the reply chain and not make things harder for others reading\nalong with clients that aren't yours (which, looking at your headers\nseems to be Thunderbird 78).\n\nI.e. the one feedback on the patch idea is your upthread \"waiting until\nsuch a change\". With threading you can see the context, but without\nyou'd need to get it via some not-MUA side-channel (presumably\nlore.kernel.org link). Sending a v2 (if any) without threading would\nbreak the chain again.\n\n1. https://lore.kernel.org/git/nycvar.QRO.7.76.6.2103191540330.57@tvgsbejvaqbjf.bet/\n"},{"id":"423243","messageId":"xmqqsg397jyj.fsf@gitster.g","threadId":"55420","inReplyTo":"1ecb3727-106f-3d04-976a-36aa03a61caf@gmail.com","subject":"Re: Nesting topics within other threads","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-29T05:12:20Z","receivedAt":"2021-04-29T05:13:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 4/28/2021 12:26 PM, Ævar Arnfjörð Bjarmason wrote:\n>> Simplify the setup code in repo-settings.c in various ways, making the\n>> code shorter, easier to read, and requiring fewer hacks to do the same\n>> thing as it did before:\n>\n> This patch is interesting, and I'll review it when I have some more\n> time. Probably tomorrow.\n>\n> But I thought that I would point out that this pattern of adding a\n> patch within the thread of a larger series makes it very difficult\n> to separate the two. I use an email client that groups messages by\n> thread in order to help parse meaningful discussion from the list\n> which otherwise looks like a fire hose of noise. Now, this patch is\n> linked to the FS Monitor thread and feedback to either will trigger\n> the thread as having unread messages.\n>\n> I find it very difficult to track multiple patch series that are\n> being juggled in the same thread. It is mentally taxing enough that\n> I have avoided reviewing code presented this way to save myself the\n> effort of tracking which patches go with what topic in what order.\n\nI do find it distracting to have a full \"ah, I just thought of\nsomething while discussing this unrelated series\" patch fairly\nirritating for the same reason. It however is unavoidable human\nnature that we come up with ideas while thinking about something not\nnecessarily related. So it largely is a presentation issue.\n\nI really appreciate the way some people (Peff is a stellar example,\nbut there are others who are as good at this) handle these tangents,\nwhere the message sent to an existing thread is limited to only give\nan outline of the idea (possibly with \"something like this?\" patch\nfor illustration) and then they quickly get out of the way of the\ndiscussion by starting a separate thread, while back-referencing \"So\nhere is a proper patch based on the idea I interjected in the\ndiscussion of that other topic.\" And the discussion on the tangent\nwill be done on its own thread.\n\n\n"},{"id":"423263","messageId":"8735v9z32l.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"xmqqsg397jyj.fsf@gitster.g","subject":"Re: Nesting topics within other threads","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-29T12:14:52Z","receivedAt":"2021-04-29T12:29:42Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Apr 29 2021, Junio C Hamano wrote:\n\n> Derrick Stolee writes:\n>\n>> On 4/28/2021 12:26 PM, Ævar Arnfjörð Bjarmason wrote:\n>>> Simplify the setup code in repo-settings.c in various ways, making the\n>>> code shorter, easier to read, and requiring fewer hacks to do the same\n>>> thing as it did before:\n>>\n>> This patch is interesting, and I'll review it when I have some more\n>> time. Probably tomorrow.\n>>\n>> But I thought that I would point out that this pattern of adding a\n>> patch within the thread of a larger series makes it very difficult\n>> to separate the two. I use an email client that groups messages by\n>> thread in order to help parse meaningful discussion from the list\n>> which otherwise looks like a fire hose of noise. Now, this patch is\n>> linked to the FS Monitor thread and feedback to either will trigger\n>> the thread as having unread messages.\n>>\n>> I find it very difficult to track multiple patch series that are\n>> being juggled in the same thread. It is mentally taxing enough that\n>> I have avoided reviewing code presented this way to save myself the\n>> effort of tracking which patches go with what topic in what order.\n>\n> I do find it distracting to have a full \"ah, I just thought of\n> something while discussing this unrelated series\" patch fairly\n> irritating for the same reason. It however is unavoidable human\n> nature that we come up with ideas while thinking about something not\n> necessarily related. So it largely is a presentation issue.\n>\n> I really appreciate the way some people (Peff is a stellar example,\n> but there are others who are as good at this) handle these tangents,\n> where the message sent to an existing thread is limited to only give\n> an outline of the idea (possibly with \"something like this?\" patch\n> for illustration) and then they quickly get out of the way of the\n> discussion by starting a separate thread, while back-referencing \"So\n> here is a proper patch based on the idea I interjected in the\n> discussion of that other topic.\" And the discussion on the tangent\n> will be done on its own thread.\n\nIn RFC 822 terms. Are you talking about the In-Reply-To[1] or\nReferences[2] headers, or both/neither?\n\nI'm happy to go along with whatever the convention is, but as noted\nthink it's valuable to come to some explicit decision to document the\nconvention.\n\nThreading isn't a concept that exists in E-Mail protocols per-se. Just\nIn-Reply-To and References. The References header can reference N\nmessages most would think about as a separate \"thread\", and \"thread\" is\nultimately some fuzzy MUA-specific concept on top of these (and others).\n\nE.g. in my client right now I'm looking at just 4 messages in this\n\"thread\", it doesn't descend down the whole In-Reply-To, others would\nact differently.\n\nSome (such as GMail) have their own ad-hoc concept of \"thread\" separate\nfrom anything in RFCs (which includes some fuzzy group-by-subject). In\nGMail's web UI everything as of my \"upthread\"\n is presented\nas its own thread.\n\nThe ML read as it happens, but it's also a collectively maintained\ndatastructure.\n\nIt seems to me to be better to veer on the side of using standard fields\nfor their intended purpose for archiving / future use. I.e. making \"a\nreference\" universally machine-readable, as opposed to a lore.kernel.org\nlink, or a free-form \"in a recent thread\" blurb.\n\nML Archive Formats Matter[3] :)\n\nBut yes, maybe MUAs in the wild these days mostly render things one way\nor another, so catering to them would be a good trade-off. I'm writing\nthis from within an Emacs MUA, so I don't have much of a feel for common\nMUA conventions these days.\n\nI'm prodding to see if we can define the problem exactly, because\ne.g. maybe \"References: [actual ]\" is\nsomething that would achieve both aims, i.e. make the references\nmachine-readable, but break up threading in common in-the-wild\nclients. We could then patch format-patch etc. to support such\n\"detached\" threading.\n\n1. https://tools.ietf.org/html/rfc822#section-4.6.2\n2. https://tools.ietf.org/html/rfc822#section-4.6.3\n3. https://keithp.com/blogs/Repository_Formats_Matter/\n"},{"id":"423298","messageId":"YIsTqmSoR5k2r6kD@coredump.intra.peff.net","threadId":"55420","inReplyTo":"8735v9z32l.fsf@evledraar.gmail.com","subject":"Re: Nesting topics within other threads","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-29T20:14:34Z","receivedAt":"2021-04-29T20:14:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 29, 2021 at 02:14:52PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > I really appreciate the way some people (Peff is a stellar example,\n> > but there are others who are as good at this) handle these tangents,\n> > where the message sent to an existing thread is limited to only give\n> > an outline of the idea (possibly with \"something like this?\" patch\n> > for illustration) and then they quickly get out of the way of the\n> > discussion by starting a separate thread, while back-referencing \"So\n> > here is a proper patch based on the idea I interjected in the\n> > discussion of that other topic.\" And the discussion on the tangent\n> > will be done on its own thread.\n> \n> In RFC 822 terms. Are you talking about the In-Reply-To[1] or\n> References[2] headers, or both/neither?\n\nSince I got listed as an example, I can tell you what I do: I start a\ntotally new thread with no in-reply-to or references to the old thread.\nAnd the subject is new (usually \"[PATCH 0/N] foo...\"), so no clever\ngroup-by-subject heuristics will link them.\n\nIt's usually a good idea to reference the message-id/lore link in at\nleast one direction, though (usually I'd do it in the new thread, saying\n\"this is a followup to ...\", but you could also follow-up in the\noriginal to say \"I've spun this off into its own series here...\").\n\nWhich is really _sort of_ like putting it into \"References\", except that\nit's not machine readable. Which is a good thing, because it's a weaker\nform and doesn't tell mail clients to group it all into one thread.\n\n> Threading isn't a concept that exists in E-Mail protocols per-se. Just\n> In-Reply-To and References. The References header can reference N\n> messages most would think about as a separate \"thread\", and \"thread\" is\n> ultimately some fuzzy MUA-specific concept on top of these (and others).\n> \n> E.g. in my client right now I'm looking at just 4 messages in this\n> \"thread\", it doesn't descend down the whole In-Reply-To, others would\n> act differently.\n\nInteresting. Mutt (and notmuch, and public-inbox) definitely view these\nas part of a larger thread. It looks like you're using mu4e; I'm\nsurprised it doesn't, too (of course some clients will give a partial\nview of a thread if you've already marked the older messages as read and\nmoved them into an archival folder).\n\n> It seems to me to be better to veer on the side of using standard fields\n> for their intended purpose for archiving / future use. I.e. making \"a\n> reference\" universally machine-readable, as opposed to a lore.kernel.org\n> link, or a free-form \"in a recent thread\" blurb.\n\nI'd disagree here. There's a long history of intentionally breaking the\nthread in mailing lists and newsgroups exactly because the topic is\nsufficiently different that you want to make it easy for people to treat\nit as a separate unit. I admit there's a bit of an art form to deciding\nwhen that is appropriate and when not.\n\n-Peff\n"},{"id":"423305","messageId":"xmqqa6pgfxef.fsf@gitster.g","threadId":"55420","inReplyTo":"8735v9z32l.fsf@evledraar.gmail.com","subject":"Re: Nesting topics within other threads","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-30T00:07:04Z","receivedAt":"2021-04-30T00:07:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> In RFC 822 terms. Are you talking about the In-Reply-To[1] or\n> References[2] headers, or both/neither?\n\nNeither (I think Peff explained why it is a good idea to defer to\nverbal communication not to confuse tools better than I could).\n"},{"id":"423339","messageId":"4552971c-0a23-c19a-6a23-cb5737e43b2a@jeffhostetler.com","threadId":"55420","inReplyTo":"f77457d0-925d-2ea1-4ad4-b3315532b37f@gmail.com","subject":"Re: [PATCH 04/23] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-30T14:23:36Z","receivedAt":"2021-04-30T14:23:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 10:56 AM, Derrick Stolee wrote:\n> On 4/1/21 11:40 AM, Johannes Schindelin via GitGitGadget wrote:> @@ -2515,6 +2515,11 @@ int git_config_get_max_percent_split_change(void)\n>> \n>> int repo_config_get_fsmonitor(struct repository *r)\n>> {\n>> +\tif (r->settings.use_builtin_fsmonitor > 0) {\n> \n> Don't forget to run prepare_repo_settings(r) first.\n> \n>> +\t\tcore_fsmonitor = \"(built-in daemon)\";\n>> +\t\treturn 1;\n>> +\t}\n>> +\n> \n> I found this odd, assigning a string to core_fsmonitor that\n> would definitely cause a problem trying to execute it as a\n> hook. I wondered the need for it at all, but found that\n> there are several places in the FS Monitor subsystem that use\n> core_fsmonitor as if it was a boolean, indicating whether or\n> not the feature is enabled at all.\n> \n> A cleaner way to handle this would be to hide the data behind\n> a helper method, say \"fsmonitor_enabled()\" that could then\n> check a value on the repository (or index) and store the hook\n> value as a separate value that is only used by the hook-based\n> implementation.\n> \n> It's probably a good idea to do that cleanup now, before we\n> find on accident that we missed a gap and start trying to run\n> this bogus string as a hook invocation.\n\nGood point. In an earlier draft we were using that known\nstring as a bogus hook path to indicate that we should\ncall the IPC routines rather than the hook API. But then\nwe added the `core.useBuiltinFSMonitor` boolean and had it\noverride all of the existing fsmonitor config settings.\nSo we don't technically need it to have a value now and can\nand should stop using the pointer as a boolean.\n\nThanks!\n\n>> -static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n>> +static int query_fsmonitor(int version, struct index_state *istate, struct strbuf *query_result)\n>> {\n>> +\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n>> +\tconst char *last_update = istate->fsmonitor_last_update;\n>> \tstruct child_process cp = CHILD_PROCESS_INIT;\n>> \tint result;\n>> \n>> \tif (!core_fsmonitor)\n>> \t\treturn -1;\n> \n> Here is an example of it being used as a boolean.\n> \n>> +\tif (r->settings.use_builtin_fsmonitor > 0) {\n>> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n>> +\t\treturn fsmonitor_ipc__send_query(last_update, query_result);\n>> +#else\n>> +\t\t/* Fake a trivial response. */\n>> +\t\twarning(_(\"fsmonitor--daemon unavailable; falling back\"));\n>> +\t\tstrbuf_add(query_result, \"/\", 2);\n>> +\t\treturn 0;\n>> +#endif\n> \n> This seems like a case where the helper fsmonitor_ipc__is_supported()\n> could be used instead of compile-time macros.\n> \n> (I think this is especially true when we consider the future of the\n> feature on Linux and the possibility of the same compiled code needing\n> to check run-time properties of the platform for compatibility.)\n\nYes.\n\n>> --- a/repo-settings.c\n>> +++ b/repo-settings.c\n>> @@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r)\n>> \t\tr->settings.core_multi_pack_index = value;\n>> \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n>> \n>> +\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n>> +\t\tr->settings.use_builtin_fsmonitor = 1;\n>> +\n> \n> Follows the patterns of repo settings. Good.\n> \n\nI'm going to ignore all of the thread responses to this patch\ndealing with how we acquire config settings and macros and etc.\nThose issues are completely independent of FSMonitor (which is\nalready way too big).\n\nJeff\n\n"},{"id":"423340","messageId":"9e4f67f0-9a94-b93c-2f21-76555199a38e@jeffhostetler.com","threadId":"55420","inReplyTo":"1c45542a-5d80-6e51-26fe-94066987a566@gmail.com","subject":"Re: [PATCH 05/23] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-30T14:31:34Z","receivedAt":"2021-04-30T14:31:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 11:45 AM, Derrick Stolee wrote:\n> On 4/26/21 11:08 AM, Derrick Stolee wrote:\n>> On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n>>\n>> I think these compile-time macros should be replaced with a\n>> method call, as I've said before. It should be simple to say\n>>\n>> \tif (!fsmonitor_ipc__is_supported())\n>> \t\tdie(_(\"fsmonitor--daemon is not supported on this platform\"));\n>>\n>> and call it a day. This can be done before parsing arguments.\n>>\n>>> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n>>> +{\n>>> +\tenum daemon_mode {\n>>> +\t\tUNDEFINED_MODE,\n>>> +\t} mode = UNDEFINED_MODE;\n>>> +\n>>> +\tstruct option options[] = {\n>>> +\t\tOPT_END()\n>>> +\t};\n>>\n>> I can see where you are going here, to use the parse-opts API\n>> to get your \"--\" arguments to populate an 'enum'. However,\n>> it seems like you will run into the problem where a user enters\n>> multiple such arguments and you lose the information as the\n>> parser overwrites 'mode' here.\n> \n> I see that you use OPT_CMDMODE in your implementation, which\n> makes this concern invalid.\n> \n>> Better to use a positional argument and drop the \"--\" prefix,\n>> in my opinion.\n> \n> This is my personal taste, but the technical reason to do this\n> doesn't exist.\n\nEither method is fine/equivalent and I'm open to doing it either\nway. (In fact, I did the t/helper/test-simple-ipc the other way\nand didn't even think about it.)\n\nDoes the mailing list have a preference for one form over the other?\nThat is:\n\n git fsmonitor--daemon --start []\nvs\n git fsmonitor--daemon start []\n\nJeff\n\n"},{"id":"423341","messageId":"b0fe4d10-1205-39e4-72c9-d33d1f023730@jeffhostetler.com","threadId":"55420","inReplyTo":"098e4123-676e-2ea0-4c16-dbeaa1198b46@gmail.com","subject":"Re: [PATCH 06/23] fsmonitor--daemon: implement client command options","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-30T14:33:49Z","receivedAt":"2021-04-30T14:33:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 11:12 AM, Derrick Stolee wrote:\n> On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Implement command options `--stop`, `--is-running`, `--query`,\n>> `--query-index`, and `--flush` to control and query the status of a\n>> `fsmonitor--daemon` server process (and implicitly start a server\n>> process if necessary).\n>>\n>> Later commits will implement the actual server and monitor\n>> the file system.\n> \n> As mentioned before, I think the \"query\", \"query-index\", and\n> \"flush\" commands are better served in a test helper. Luckily,\n> the implementation you give here seems rather straightforward\n> and could fit into a test helper without a lot of duplicated\n> boilerplate. That's a good sign for the API presented here.\n> \n> As a bonus, you could delay the implementation of those test\n> helpers until they are going to be used in a test.\n> \n> Thanks,\n> -Stolee\n> \n\nGood point. I'll take a look at this.\n\nThanks\nJeff\n"},{"id":"423347","messageId":"a7e8ef01-b707-5b4a-2652-cc3e08bb85f2@jeffhostetler.com","threadId":"55420","inReplyTo":"3aa3c802-0ad1-0a65-d646-0cd574cb8036@gmail.com","subject":"Re: [PATCH 09/23] fsmonitor--daemon: implement daemon command options","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-30T15:18:51Z","receivedAt":"2021-04-30T15:18:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 12:12 PM, Derrick Stolee wrote:\n> On 4/26/2021 11:47 AM, Derrick Stolee wrote:\n>> On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>>> From: Jeff Hostetler \n>> ...\n>>> +\t/* Prepare to (recursively) watch the directory. */\n>>> +\tstrbuf_init(&state.path_worktree_watch, 0);\n>>> +\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n>>> +\tstate.nr_paths_watching = 1;\n>>\n>> Yes, let's watch the working directory.\n>>\n>>> +\t/*\n>>> +\t * If \".git\" is not a directory, then is not inside the\n>>> +\t * cone of , so set up a second watch for it.\n>>> +\t */\n>>> +\tstrbuf_init(&state.path_gitdir_watch, 0);\n>>> +\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n>>> +\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n>>> +\tif (!is_directory(state.path_gitdir_watch.buf)) {\n>>> +\t\tstrbuf_reset(&state.path_gitdir_watch);\n>>> +\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n>>> +\t\tstate.nr_paths_watching = 2;\n>>> +\t}\n>>\n>> But why watch the .git directory, especially for a worktree (or\n>> submodule I guess)? What benefit do we get from events within the\n>> .git directory? I'm expecting any event within the .git directory\n>> should be silently ignored.\n> \n> I see in a following patch that we place a cookie file within the\n> .git directory. I'm reminded that this is done for a reason: other\n> filesystem watchers can get into a loop if we place the cookie\n> file outside of the .git directory. The classic example is VS Code\n> running 'git status' in a loop because Watchman writes a cookie\n> into the root of the working directory.\n\nYes. I'll add a comment explaining the need for the second watch.\n"},{"id":"423349","messageId":"673017fa-36b9-3a26-f1de-155cc2608cff@jeffhostetler.com","threadId":"55420","inReplyTo":"91a0fc4b-505a-fa2b-c797-5e420e836642@gmail.com","subject":"Re: [PATCH 09/23] fsmonitor--daemon: implement daemon command options","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-30T15:59:11Z","receivedAt":"2021-04-30T15:59:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 11:47 AM, Derrick Stolee wrote:\n...\n> \n>> +\n>> static int is_ipc_daemon_listening(void)\n>> {\n>> \treturn fsmonitor_ipc__get_state() == IPC_STATE__LISTENING;\n>> }\n>> \n>> +static int try_to_run_foreground_daemon(void)\n>> +{\n>> +\t/*\n>> +\t * Technically, we don't need to probe for an existing daemon\n>> +\t * process, since we could just call `fsmonitor_run_daemon()`\n>> +\t * and let it fail if the pipe/socket is busy.\n>> +\t *\n>> +\t * However, this method gives us a nicer error message for a\n>> +\t * common error case.\n>> +\t */\n>> +\tif (is_ipc_daemon_listening())\n>> +\t\tdie(\"fsmonitor--daemon is already running.\");\n> Here, it seems like we only care about IPC_STATE_LISTENING, while\n> earlier I mentioned that I ended up in IPC_STATE__NOT_LISTENING,\n> and my manually running of the daemon helped.\n> \n>> +\treturn !!fsmonitor_run_daemon();\n>> +}\n> \n> You are ignoring the IPC_STATE__NOT_LISTENING and creating a new\n> process, which is good. I'm just wondering why that state exists\n> and what is the proper way to handle it?\n\nI'll revisit this and clarify.\n\n> \n>> +\n>> +#ifndef GIT_WINDOWS_NATIVE\n> \n> You are already creating a platform-specific mechanism for the\n> filesystem watcher. Shouldn't the implementation of this method\n> be part of that file in compat/fsmonitor/?\n> \n> I guess the biggest reason is that macOS and Linux share this\n> implementation, so maybe this is the cleanest approach.\n\nThis has to do with how to spawn a background process and\ndisassociate from the console and all that.\n\nOn Windows, the \"git fsmonitor--daemon --start\" process[1] must\nstart a child process[2] with \"git fsmonitor--daemon --run\" and\nthen the [1] can exit (to let the caller/shell continue) while\n[2] is free to continue.\n\nOn Unix, the \"git fsmonitor-daemon --start\" process[1] can\nfork() a child process[2] and [2] can just call the _run_daemon()\ncode. We don't need to exec the child, so this is a bit faster.\n\nThis code is platform-specific, so maybe it should go elsewhere,\nbut it has knowledge of fsmonitor--daemon-specific command line\nargs and private functions (`fsmonitor_run_daemon()`) and it knows\nthat it is not a library function. So it made sense to keep it\nclose to the fsmonitor--daemon main entry point.\n\n\nIt didn't feel right to make these 2 versions of\n`spawn_background_fsmonitor_daemon()` more generic (such as putting\nthem near `daemonize()`), because they know too much about\nfsmonitor--daemon.\n\nI did the same thing in `t/helper/test-simple-ipc.c` where I\ncreated variants of this that started the test-tool in the background.\n36a7eb6876 (t0052: add simple-ipc tests and t/helper/test-simple-ipc \ntool, 2021-03-22)\n\n\nI thought about putting them inside `compat/fsmonitor/*.c`\nbut that has different problems. Those files are concerned strictly\nwith the FS layer and how to get FS notification events from the\nkernel and translating them into something cross-platform. They\nare, in a sense, a \"driver\" for that FS. Process spawning is outside\nof their scope.\n\nAnd as you say, MacOS and Linux can both use the same process\nspawning code, but will have vastly different FS layers.\n\nSo I left them here.\n\n\nThanks,\nJeff\n"},{"id":"423353","messageId":"6f531ece-f95f-5825-4c2e-0fd7373de6b7@jeffhostetler.com","threadId":"55420","inReplyTo":"3f0c5c7d-8448-15ec-9e12-b2745d7bbf2b@gmail.com","subject":"Re: [PATCH 11/23] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-30T16:17:16Z","receivedAt":"2021-04-30T16:17:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 3:49 PM, Derrick Stolee wrote:\n> On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Teach fsmonitor--daemon to create token-ids and define the\n>> overall token naming scheme.\n> ...\n>> +/*\n>> + * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n>> + * \"token\" as a virtual timestamp. Clients can request a summary of all\n>> + * created/deleted/modified files relative to a token. In the response,\n>> + * clients receive a new token for the next (relative) request.\n>> + *\n>> + *\n>> + * Token Format\n>> + * ============\n>> + *\n>> + * The contents of the token are private and provider-specific.\n>> + *\n>> + * For the built-in fsmonitor--daemon, we define a token as follows:\n>> + *\n>> + * \"builtin\" \":\" \":\" \n>> + *\n>> + * The is an arbitrary OPAQUE string, such as a GUID,\n>> + * UUID, or {timestamp,pid}. It is used to group all filesystem\n>> + * events that happened while the daemon was monitoring (and in-sync\n>> + * with the filesystem).\n>> + *\n>> + * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n>> + * and does not define less-than/greater-than relationships.\n>> + * (There are too many race conditions to rely on file system\n>> + * event timestamps.)\n>> + *\n>> + * The is a simple integer incremented for each event\n>> + * received. When a new is created, the is\n>> + * reset to zero.\n>> + *\n>> + *\n>> + * About Token Ids\n>> + * ===============\n>> + *\n>> + * A new token_id is created:\n>> + *\n>> + * [1] each time the daemon is started.\n>> + *\n>> + * [2] any time that the daemon must re-sync with the filesystem\n>> + * (such as when the kernel drops or we miss events on a very\n>> + * active volume).\n>> + *\n>> + * [3] in response to a client \"flush\" command (for dropped event\n>> + * testing).\n>> + *\n>> + * [4] MAYBE We might want to change the token_id after very complex\n>> + * filesystem operations are performed, such as a directory move\n>> + * sequence that affects many files within. It might be simpler\n>> + * to just give up and fake a re-sync (and let the client do a\n>> + * full scan) than try to enumerate the effects of such a change.\n>> + *\n>> + * When a new token_id is created, the daemon is free to discard all\n>> + * cached filesystem events associated with any previous token_ids.\n>> + * Events associated with a non-current token_id will never be sent\n>> + * to a client. A token_id change implicitly means that the daemon\n>> + * has gap in its event history.\n>> + *\n>> + * Therefore, clients that present a token with a stale (non-current)\n>> + * token_id will always be given a trivial response.\n> \n> From this comment, it seems to be the case that concurrent Git\n> commands will race to advance the FS Monitor token and one of them\n> will lose, causing a full working directory scan. There is no list\n> of \"recent\" tokens.\n> \n> I could see this changing in the future, but for now it is a\n> reasonable simplification.\n\nThe daemon only creates a new token-id when it needs to because of\na loss of sync with the FS. And the sequence-nr is advanced based\nupon the quantity of FS activity. Clients don't cause either to\nchange or advance (except for the flush, which is a testing hack).\n\nIdeally, the token-id is created when the daemon starts up and is\nnever changed.\n\nConcurrent clients all receive normalized event data from the\nin-memory cache/queue from threads reading the queue in parallel.\n\n\nI included [4] as a possible future enhancement, but so far haven't\nactually needed it. The event stream (at least on Windows and MacOS)\nfrom the OS is sufficient that I didn't need to implement that.\n\nI'll remove [4] from the comments.\n\nThanks,\nJeff\n"},{"id":"423363","messageId":"ab661886-076c-4f8b-a65b-db54ca28d4a0@jeffhostetler.com","threadId":"55420","inReplyTo":"df33c12b-ada3-05dc-3d17-6cc9d205b4cc@gmail.com","subject":"Re: [PATCH 12/23] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-30T17:36:58Z","receivedAt":"2021-04-30T17:37:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 4:22 PM, Derrick Stolee wrote:\n...\n>> +\n>> +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n>> +\t\t\t const char *path)\n>> +{\n>> +\tconst char *interned_path = strintern(path);\n> \n> This use of interned paths is interesting, although I become\n> concerned for the amount of memory we are consuming over the\n> lifetime of the process. This could be considered as a target\n> for future improvements, perhaps with an LRU cache or something\n> similar.\n\nInterning gives us a fixed pointer for any given path. This\ngives us a way to de-dup paths using just pointers rather than\nstring compares.\n\nYes, we will accumulate paths in that dictionary, but the set of\npaths present in the typical working directory are usually pretty\nfixed.\n\nWe only generate these for modified paths. Users don't typically\ncreate/modify/delete that many paths in their source trees during\nnormal development.\n\nCompilers may generate lots of trash files in their worktree, but\nthose names are usually repeated (with each \"make\"). So we might\nacculuate a lot of paths for a repo, it should become stable.\nHowever, if they use temp files in the tree, it might invalidate\nthis statement.\n\nWRT LRUs, that gets us into threading and lock contention problem\nand ref-counting. I have it designed such that parallel threads\nread and send the current queue to the client without a lock. They\nonly need a quick lock to get the current head pointer; the rest\nis done lock free. Also, purging from the end of the LRU would\nput is in contention with the FS listener thread that is adding\nnew paths to the LRU.\n\nSo, yeah, maybe this is something to keep an eye on -- especially\nin the monorepo case, but I don't think we need to address it now.\n\nThanks,\nJeff\n"},{"id":"423367","messageId":"1c6dec17-89be-7f5f-7125-263e6778d2ba@jeffhostetler.com","threadId":"55420","inReplyTo":"4b0066f5-5962-e57e-5987-5358f3d394f7@gmail.com","subject":"Re: [PATCH 13/23] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-30T19:32:44Z","receivedAt":"2021-04-30T19:32:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/27/21 1:22 PM, Derrick Stolee wrote:\n> On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Teach the win32 backend to register a watch on the working tree\n>> root directory (recursively). Also watch the if it is\n>> not inside the working tree. And to collect path change notifications\n>> into batches and publish.\n> \n> Is it valuable to list the important API methods here for an interested\n> reader to discover them? Perhaps using links to the docs [1] might be\n> too ephemeral, in case those URLs stop being valid.\n> \n> In any case, here are the URLs I found helpful:\n> \n> [1] https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-readdirectorychangesw\n> [2] https://docs.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-getoverlappedresult\n> [3] https://docs.microsoft.com/en-us/windows/win32/fileio/cancelioex-func\n> [4] https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-resetevent\n> [5] https://docs.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-file_notify_information\n> [6] https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-waitformultipleobjects\n\nI could see adding them. I think I had some in an earlier draft.\n(And I'm always glad to find them when I go back and revisit the\ncode later :-)\n\nAnd yes, the Win32 APIs and code are a bit dense and tricky to\nunderstand -- especially the async IO stuff, so this would be good.\n\n> \n>> Signed-off-by: Jeff Hostetler \n...\n>> +static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n>> +\t\t\t\t const char *path)\n>> +{\n>> +\tstruct one_watch *watch = NULL;\n>> +\tDWORD desired_access = FILE_LIST_DIRECTORY;\n>> +\tDWORD share_mode =\n>> +\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n> \n> Ah, this is probably why we can delete a repo that is under a watch.\n\nYes, we're holding a handle to the directory (and we're CD'd into it),\nbut I have code in there to detect when the .git directory is deleted\nand trigger a shutdown. A \"rm -rf\" on the worktree will delete the\n.git directory first and we \"usually\" win that race before \"rm\" gets\naround to deleting the root directory.\n\nI intend to harden this a bit more. The Git startup code does the CD\nto the root directory and I'd like to CD out after we get everything\nsetup.\n\n...\n>> +static int recv_rdcw_watch(struct one_watch *watch)\n>> +{\n>> +\twatch->is_active = FALSE;\n>> +\n>> +\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n>> +\t\t\t\tTRUE))\n>> +\t\treturn 0;\n>> +\n>> +\t// TODO If an external is deleted, the above returns an error.\n>> +\t// TODO I'm not sure that there's anything that we can do here other\n>> +\t// TODO than failing -- the /.git link file would be broken\n>> +\t// TODO anyway. We might try to check for that and return a better\n>> +\t// TODO error message.\n> \n> These are not fit C-style comments. This situation can be handled\n> by a later patch series, if valuable enough.\n\nOooops, I missed that one before I posted it.\n\n> \n>> +\n>> +\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n>> +\t watch->path.buf, GetLastError());\n>> +\treturn -1;\n>> +}\n>> +\n>> +static void cancel_rdcw_watch(struct one_watch *watch)\n>> +{\n>> +\tDWORD count;\n>> +\n>> +\tif (!watch || !watch->is_active)\n>> +\t\treturn;\n>> +\n>> +\tCancelIoEx(watch->hDir, &watch->overlapped);\n>> +\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n>> +\twatch->is_active = FALSE;\n>> +}\n>> +\n>> +/*\n>> + * Process filesystem events that happen anywhere (recursively) under the\n>> + * root directory. For a normal working directory, this includes\n>> + * both version controlled files and the contents of the .git/ directory.\n>> + *\n>> + * If /.git is a file, then we only see events for the file\n>> + * itself.\n>> + */\n>> +static int process_worktree_events(struct fsmonitor_daemon_state *state)\n>> +{\n>> +\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n>> +\tstruct one_watch *watch = data->watch_worktree;\n>> +\tstruct strbuf path = STRBUF_INIT;\n>> +\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n>> +\tstruct fsmonitor_batch *batch = NULL;\n>> +\tconst char *p = watch->buffer;\n>> +\n>> +\t/*\n>> +\t * If the kernel gets more events than will fit in the kernel\n>> +\t * buffer associated with our RDCW handle, it drops them and\n>> +\t * returns a count of zero. (A successful call, but with\n>> +\t * length zero.)\n>> +\t */\n> \n> I suppose that since we create a cookie file, we don't expect a zero\n> result to ever be a meaningful value? Or, is there another way to\n> differentiate between \"nothing happened\" and \"too much happened\"?\n\nThis is independent of cookie files. We have a thread watching\nwatching for FS events and building up batches of changes and\nassembling the list of batches. This is always running.\n\nCookies are created by the threads in the IPC thread pool and\nthose threads wait until we get a FS event for their cookie file.\n\nAll of these threads are running independently.\n\nThe code here is saying that Windows will give us a non-error but\nzero result when the kernel had to drop events. (The \"too much\nhappened\" case.)\n\nIf nothing happens on the FS, our Async IO doesn't trigger a\nwakeup and the FS listener thread stays in the WaitForMultipleObjects().\n\nAll of this is pretty dense I realize.\n\n> \n>> +\tif (!watch->count) {\n>> +\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n>> +\t\t\t\t \"overflow\");\n>> +\t\tfsmonitor_force_resync(state);\n>> +\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n>> +\t}\n>> +\n>> +\t/*\n>> +\t * On Windows, `info` contains an \"array\" of paths that are\n>> +\t * relative to the root of whichever directory handle received\n>> +\t * the event.\n>> +\t */\n>> +\tfor (;;) {\n>> +\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n>> +\t\tconst char *slash;\n>> +\t\tenum fsmonitor_path_type t;\n>> +\n>> +\t\tstrbuf_reset(&path);\n>> +\t\tif (normalize_path_in_utf8(info, &path) == -1)\n>> +\t\t\tgoto skip_this_path;\n>> +\n>> +\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n>> +\n>> +\t\tswitch (t) {\n>> +\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n>> +\t\t\t/* special case cookie files within .git */\n>> +\n>> +\t\t\t/* Use just the filename of the cookie file. */\n>> +\t\t\tslash = find_last_dir_sep(path.buf);\n>> +\t\t\tstring_list_append(&cookie_list,\n>> +\t\t\t\t\t slash ? slash + 1 : path.buf);\n> \n> Ok, I see now how we special-case cookies in the list of events.\n> \n>> +\t\t\tbreak;\n>> +\n>> +\t\tcase IS_INSIDE_DOT_GIT:\n>> +\t\t\t/* ignore everything inside of \"/.git/\" */\n>> +\t\t\tbreak;\n>> +\n>> +\t\tcase IS_DOT_GIT:\n>> +\t\t\t/* \"/.git\" was deleted (or renamed away) */\n>> +\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n>> +\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n>> +\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n>> +\t\t\t\t\t\t \"fsm-listen/dotgit\",\n>> +\t\t\t\t\t\t \"removed\");\n>> +\t\t\t\tgoto force_shutdown;\n>> +\t\t\t}\n>> +\t\t\tbreak;\n>> +\n>> +\t\tcase IS_WORKDIR_PATH:\n>> +\t\t\t/* queue normal pathname */\n>> +\t\t\tif (!batch)\n>> +\t\t\t\tbatch = fsmonitor_batch__new();\n>> +\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n>> +\t\t\tbreak;\n>> +\n>> +\t\tcase IS_GITDIR:\n>> +\t\tcase IS_INSIDE_GITDIR:\n>> +\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n>> +\t\tdefault:\n>> +\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n>> +\t\t\t t, path.buf);\n> So these events should be caught by the _other_ watcher. I suppose\n> BUG() is somewhat appropriate, but also seems heavy-handed. For\n> example, the 'goto' in the next line will never be visited. A die()\n> would even be appropriate, but somewhat less harsh than a BUG(),\n> especially for a background process.\n> \n>> +\t\t\tgoto skip_this_path;\n\nRight, the IS_*GITDIR events should only happen in the other\nwatcher. They are bugs in the sense that the path classifier\nfailed. And the \"default:\" is my usual backstop for such an enum\nwhere any other value would be a bug.\n\nI'm not sure whether BUG(), die(), or error() is better than the\nother. The daemon runs with without a console normally, so the\nmessages will not be seen -- unless you're debugging it or running\nit in the foreground.\n\nI could get rid of the goto.\n\n>> +\t\t}\n>> +\n>> +skip_this_path:\n>> +\t\tif (!info->NextEntryOffset)\n>> +\t\t\tbreak;\n>> +\t\tp += info->NextEntryOffset;\n>> +\t}\n>> +\n>> +\tfsmonitor_publish(state, batch, &cookie_list);\n>> +\tbatch = NULL;\n>> +\tstring_list_clear(&cookie_list, 0);\n>> +\tstrbuf_release(&path);\n>> +\treturn LISTENER_HAVE_DATA_WORKTREE;\n>> +\n>> +force_shutdown:\n>> +\tfsmonitor_batch__free(batch);\n>> +\tstring_list_clear(&cookie_list, 0);\n>> +\tstrbuf_release(&path);\n>> +\treturn LISTENER_SHUTDOWN;\n>> +}\n>> +\n>> +/*\n>> + * Process filesystem events that happend anywhere (recursively) under the\n> \n> s/happend/happened\n> \n>> + * external (such as non-primary worktrees or submodules).\n>> + * We only care about cookie files that our client threads created here.\n>> + *\n>> + * Note that we DO NOT get filesystem events on the external \n>> + * itself (it is not inside something that we are watching). In particular,\n>> + * we do not get an event if the external is deleted.\n> \n> This is an interesting change of behavior. I forget if it is listed in\n> the documentation file, but definitely could be. I imagine wanting a\n> \"Troubleshooting\" section that describes special cases like this.\n> \n> Also, because of this worktree-specific behavior, we might want to\n> recommend using 'git config --worktree' when choosing to use FS Monitor,\n> so that each worktree is opted-in as requested. Without --worktree, all\n> worktrees with a common base would stard using FS Monitor simultaneously.\n\nI'll take a look.\n\n> \n>> + */\n>> +static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n>> +{\n>> +\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n>> +\tstruct one_watch *watch = data->watch_gitdir;\n>> +\tstruct strbuf path = STRBUF_INIT;\n>> +\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n>> +\tconst char *p = watch->buffer;\n>> +\n>> +\tif (!watch->count) {\n>> +\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n>> +\t\t\t\t \"overflow\");\n>> +\t\tfsmonitor_force_resync(state);\n>> +\t\treturn LISTENER_HAVE_DATA_GITDIR;\n>> +\t}\n>> +\n>> +\tfor (;;) {\n>> +\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n>> +\t\tconst char *slash;\n>> +\t\tenum fsmonitor_path_type t;\n>> +\n>> +\t\tstrbuf_reset(&path);\n>> +\t\tif (normalize_path_in_utf8(info, &path) == -1)\n>> +\t\t\tgoto skip_this_path;\n>> +\n>> +\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n>> +\n>> +\t\ttrace_printf_key(&trace_fsmonitor, \"BBB: %s\", path.buf);\n>> +\n>> +\t\tswitch (t) {\n>> +\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n>> +\t\t\t/* special case cookie files within gitdir */\n>> +\n>> +\t\t\t/* Use just the filename of the cookie file. */\n>> +\t\t\tslash = find_last_dir_sep(path.buf);\n>> +\t\t\tstring_list_append(&cookie_list,\n>> +\t\t\t\t\t slash ? slash + 1 : path.buf);\n>> +\t\t\tbreak;\n>> +\n>> +\t\tcase IS_INSIDE_GITDIR:\n>> +\t\t\tgoto skip_this_path;\n>> +\n>> +\t\tdefault:\n>> +\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n>> +\t\t\t t, path.buf);\n> \n> If we decide against BUG() earlier, then also get this one.\n> \n>> +\t\t\tgoto skip_this_path;\n>> +\t\t}\n>> +\n>> +skip_this_path:\n>> +\t\tif (!info->NextEntryOffset)\n>> +\t\t\tbreak;\n>> +\t\tp += info->NextEntryOffset;\n>> +\t}\n>> +\n>> +\tfsmonitor_publish(state, NULL, &cookie_list);\n>> +\tstring_list_clear(&cookie_list, 0);\n>> +\tstrbuf_release(&path);\n>> +\treturn LISTENER_HAVE_DATA_GITDIR;\n>> }\n>> \n>> void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n>> {\n>> +\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n>> +\tDWORD dwWait;\n>> +\n>> +\tstate->error_code = 0;\n>> +\n>> +\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n>> +\t\tgoto force_error_stop;\n>> +\n>> +\tif (data->watch_gitdir &&\n>> +\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n>> +\t\tgoto force_error_stop;\n>> +\n>> +\tfor (;;) {\n>> +\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n>> +\t\t\t\t\t\tdata->hListener,\n>> +\t\t\t\t\t\tFALSE, INFINITE);\n> \n> Since you use INFINITE here, that says that we will wait for at least one\n> signal, solving the confusion about zero results: zero results unambiguously\n> indicates a loss of events.\n\nRight.\n\n> \n>> +\n>> +\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n>> +\t\t\tif (recv_rdcw_watch(data->watch_worktree) == -1)\n>> +\t\t\t\tgoto force_error_stop;\n>> +\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n>> +\t\t\t\tgoto force_shutdown;\n>> +\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n>> +\t\t\t\tgoto force_error_stop;\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n>> +\t\t\tif (recv_rdcw_watch(data->watch_gitdir) == -1)\n>> +\t\t\t\tgoto force_error_stop;\n>> +\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n>> +\t\t\t\tgoto force_shutdown;\n>> +\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n>> +\t\t\t\tgoto force_error_stop;\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n>> +\t\t\tgoto clean_shutdown;\n>> +\n>> +\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n>> +\t\t GetLastError());\n>> +\t\tgoto force_error_stop;\n>> +\t}\n>> +\n>> +force_error_stop:\n>> +\tstate->error_code = -1;\n>> +\n>> +force_shutdown:\n>> +\t/*\n>> +\t * Tell the IPC thead pool to stop (which completes the await\n>> +\t * in the main thread (which will also signal this thread (if\n>> +\t * we are still alive))).\n>> +\t */\n>> +\tipc_server_stop_async(state->ipc_server_data);\n>> +\n>> +clean_shutdown:\n>> +\tcancel_rdcw_watch(data->watch_worktree);\n>> +\tcancel_rdcw_watch(data->watch_gitdir);\n>> }\n>> \n>> int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n>> {\n>> +\tstruct fsmonitor_daemon_backend_data *data;\n>> +\n>> +\tdata = xcalloc(1, sizeof(*data));\n> \n> CALLOC_ARRAY()\n> \n>> +\n>> +\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n>> +\n>> +\tdata->watch_worktree = create_watch(state,\n>> +\t\t\t\t\t state->path_worktree_watch.buf);\n>> +\tif (!data->watch_worktree)\n>> +\t\tgoto failed;\n>> +\n>> +\tif (state->nr_paths_watching > 1) {\n>> +\t\tdata->watch_gitdir = create_watch(state,\n>> +\t\t\t\t\t\t state->path_gitdir_watch.buf);\n>> +\t\tif (!data->watch_gitdir)\n>> +\t\t\tgoto failed;\n>> +\t}\n>> +\n>> +\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n>> +\tdata->nr_listener_handles++;\n>> +\n>> +\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n>> +\t\tdata->watch_worktree->hEvent;\n>> +\tdata->nr_listener_handles++;\n>> +\n>> +\tif (data->watch_gitdir) {\n>> +\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n>> +\t\t\tdata->watch_gitdir->hEvent;\n>> +\t\tdata->nr_listener_handles++;\n>> +\t}\n> \n> This is a clever organization of the event handles. I imagine it\n> will requires some rework if we decide to include another optional\n> handle whose inclusion is orthogonal to the gitdir one, but that\n> is unlikely enough to keep these well-defined array indices.\n\nI think we're good here. I think an INVALID_HANDLE value can\nbe used to supply a hole in the vector of event handles.\nBut yes, we don't have to worry about that now.\n\n> \n>> +\tstate->backend_data = data;\n>> +\treturn 0;\n>> +\n>> +failed:\n>> +\tCloseHandle(data->hEventShutdown);\n>> +\tdestroy_watch(data->watch_worktree);\n>> +\tdestroy_watch(data->watch_gitdir);\n>> +\n>> \treturn -1;\n>> }\n>> \n>> void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n>> {\n>> +\tstruct fsmonitor_daemon_backend_data *data;\n>> +\n>> +\tif (!state || !state->backend_data)\n>> +\t\treturn;\n>> +\n>> +\tdata = state->backend_data;\n>> +\n>> +\tCloseHandle(data->hEventShutdown);\n>> +\tdestroy_watch(data->watch_worktree);\n>> +\tdestroy_watch(data->watch_gitdir);\n>> +\n>> +\tFREE_AND_NULL(state->backend_data);\n>> }\n> \n> I tried to follow all the API calls and check the documentation for\n> any misuse, but did not find any. I can only contribute nitpicks\n> here, and rely on the tests to really see that this is working as\n> expected.\n> \n> I was hoping to find in here why we need to sleep in the test suite,\n> but have not pinpointed that issue yet.\n\nThe sleep was only needed on severely under-powered CI machines\n(think 1 core). I revisit this. I think I might be able to get\nrid of it.\n\nThanks,\nJeff\n\n\n"},{"id":"423370","messageId":"6f1e3acc-c223-9e27-0aa4-328f4a0ed452@jeffhostetler.com","threadId":"55420","inReplyTo":"26cd8593-58ad-bc97-5348-fdc6aaaba4ec@gmail.com","subject":"Re: [PATCH 15/23] fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-30T20:05:11Z","receivedAt":"2021-04-30T20:05:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/27/21 2:35 PM, Derrick Stolee wrote:\n> On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Implement file system event listener on MacOS using FSEvent,\n>> CoreFoundation, and CoreServices.\n> \n> Again, I'm not sure if we _should_ be including URLs to\n> documentation in our messages, but here are some I found helpful:\n> \n> [1] https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html\n> [2] https://developer.apple.com/documentation/corefoundation/1541796-cfrunloopstop\n> [3] https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/Multithreading/RunLoopManagement/RunLoopManagement.html\n> \n\nSure.\n\n>> Co-authored-by: Kevin Willford \n>> Co-authored-by: Johannes Schindelin \n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> compat/fsmonitor/fsmonitor-fs-listen-macos.c | 368 +++++++++++++++++++\n>> 1 file changed, 368 insertions(+)\n>>\n>> diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n>> index bec5130d9e1d..e055fb579cc4 100644\n>> --- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n>> +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n>> @@ -97,20 +97,388 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n>> #include \"cache.h\"\n>> #include \"fsmonitor.h\"\n>> #include \"fsmonitor-fs-listen.h\"\n>> +#include \"fsmonitor--daemon.h\"\n>> +\n>> +struct fsmonitor_daemon_backend_data\n>> +{\n>> +\tCFStringRef cfsr_worktree_path;\n>> +\tCFStringRef cfsr_gitdir_path;\n>> +\n>> +\tCFArrayRef cfar_paths_to_watch;\n>> +\tint nr_paths_watching;\n>> +\n>> +\tFSEventStreamRef stream;\n>> +\n>> +\tCFRunLoopRef rl;\n>> +\n>> +\tenum shutdown_style {\n>> +\t\tSHUTDOWN_EVENT = 0,\n>> +\t\tFORCE_SHUTDOWN,\n>> +\t\tFORCE_ERROR_STOP,\n>> +\t} shutdown_style;\n>> +\n>> +\tunsigned int stream_scheduled:1;\n>> +\tunsigned int stream_started:1;\n>> +};\n>> +\n>> +static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n>> +{\n>> +\tstruct strbuf msg = STRBUF_INIT;\n> \n> Before going through these ifs and constructing a string, it\n> might be a good idea to check if the trace event will actually\n> be sent somewhere. If the logging method is switched to a\n> trace2 method, then up here we can do:\n> \n> \tif (!trace2_is_enabled())\n> \t\treturn;\n\nNo, we're routing these messages (very noisy and low value) to\nthe old-style GIT_TRACE_FSMONITOR key. There's a whole set of\nexisting FSMonitor tracing that Ben and Kevin added that is\nuseful for interactive debugging.\n\n> \n>> +\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n>> +\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n>> +\tif (flag & kFSEventStreamEventFlagUserDropped)\n>> +\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n>> +\tif (flag & kFSEventStreamEventFlagKernelDropped)\n>> +\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n>> +\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n>> +\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n>> +\tif (flag & kFSEventStreamEventFlagHistoryDone)\n>> +\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n>> +\tif (flag & kFSEventStreamEventFlagRootChanged)\n>> +\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n>> +\tif (flag & kFSEventStreamEventFlagMount)\n>> +\t\tstrbuf_addstr(&msg, \"Mount|\");\n>> +\tif (flag & kFSEventStreamEventFlagUnmount)\n>> +\t\tstrbuf_addstr(&msg, \"Unmount|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n>> +\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemCreated)\n>> +\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n>> +\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n>> +\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemIsDir)\n>> +\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemIsFile)\n>> +\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n>> +\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n>> +\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n>> +\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemModified)\n>> +\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemRemoved)\n>> +\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemRenamed)\n>> +\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n>> +\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n>> +\tif (flag & kFSEventStreamEventFlagOwnEvent)\n>> +\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n>> +\tif (flag & kFSEventStreamEventFlagItemCloned)\n>> +\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n>> +\n>> +\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n>> +\t\t\t path, flag, msg.buf);\n> \n> Should this be a trace2 call?\n\nNo. I wanted to keep these messages on the old-style key.\n\n> \n>> +\n>> +\tstrbuf_release(&msg);\n>> +}\n>> +\n>> +static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n>> +{\n>> +\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n>> +\t\tef & kFSEventStreamEventFlagItemRemoved);\n>> +}\n>> +\n>> +static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n>> +{\n>> +\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n>> +\t\tef & kFSEventStreamEventFlagItemRenamed);\n>> +}\n> \n> Will these be handled differently? Or is it enough to detect\n> ef_is_root_moved_or_deleted()?\n\nThe whole set of Apple kernel APIs are very foreign territory\nto me, so I kept it simple and distinct. Also, it will let us\nlog different trace events.\n\n> \n>> +static void fsevent_callback(ConstFSEventStreamRef streamRef,\n>> +\t\t\t void *ctx,\n>> +\t\t\t size_t num_of_events,\n>> +\t\t\t void *event_paths,\n>> +\t\t\t const FSEventStreamEventFlags event_flags[],\n>> +\t\t\t const FSEventStreamEventId event_ids[])\n>> +{\n>> +\tstruct fsmonitor_daemon_state *state = ctx;\n>> +\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n>> +\tchar **paths = (char **)event_paths;\n>> +\tstruct fsmonitor_batch *batch = NULL;\n>> +\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n>> +\tconst char *path_k;\n>> +\tconst char *slash;\n>> +\tint k;\n>> +\n>> +\t/*\n>> +\t * Build a list of all filesystem changes into a private/local\n>> +\t * list and without holding any locks.\n>> +\t */\n>> +\tfor (k = 0; k < num_of_events; k++) {\n>> +\t\t/*\n>> +\t\t * On Mac, we receive an array of absolute paths.\n>> +\t\t */\n>> +\t\tpath_k = paths[k];\n>> +\n>> +\t\t/*\n>> +\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n>> +\t\t * Please don't log them to Trace2.\n>> +\t\t *\n>> +\t\t * trace_printf_key(&trace_fsmonitor, \"XXX '%s'\", path_k);\n>> +\t\t */\n> \n> Oh, I see. _Not_ trace2. What should we do to see if this is enabled\n> to avoid over-working in the case we are not using GIT_TRACE_FSMONITOR?\n\nRight. I'll see if there's a nice way to avoid the log_flags_set() call.\nAnd I should rename the \"XXX\" message.\n\n> \n>> +\t\t/*\n>> +\t\t * If event[k] is marked as dropped, we assume that we have\n>> +\t\t * lost sync with the filesystem and should flush our cached\n>> +\t\t * data. We need to:\n>> +\t\t *\n>> +\t\t * [1] Abort/wake any client threads waiting for a cookie and\n>> +\t\t * flush the cached state data (the current token), and\n>> +\t\t * create a new token.\n>> +\t\t *\n>> +\t\t * [2] Discard the batch that we were locally building (since\n>> +\t\t * they are conceptually relative to the just flushed\n>> +\t\t * token).\n>> +\t\t */\n>> +\t\tif ((event_flags[k] & kFSEventStreamEventFlagKernelDropped) ||\n>> +\t\t (event_flags[k] & kFSEventStreamEventFlagUserDropped)) {\n> \n> Perhaps create a macro EVENT_FLAG_DROPPED that is the union of these two? Then\n> a single \"event_flags[k] & EVENT_FLAG_DROPPED\" would suffice here. Helps cover\n> up how complicated the macOS API names are, too.\n> \n>> +\t\t\t/*\n>> +\t\t\t * see also kFSEventStreamEventFlagMustScanSubDirs\n>> +\t\t\t */\n>> +\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n>> +\t\t\t\t\t \"fsm-listen/kernel\", \"dropped\");\n>> +\n>> +\t\t\tfsmonitor_force_resync(state);\n>> +\n>> +\t\t\tif (fsmonitor_batch__free(batch))\n>> +\t\t\t\tBUG(\"batch should not have a next\");\n> \n> I mentioned before that BUG() seems overkill for these processes, but this\n> one fits. If this batch has a next, then we did something wrong, right? Do\n> we have an automated test that checks enough events to maybe cause a second\n> batch to be created?\n\nThis is me being parnoid. I hate to see a returned pointer go to waste.\n\nThe local variable `batch` has a flexarray of interned paths, it should\nnever have a next pointer until it has been \"published\". Publishing the\nbatch links it into the list of batches (and we lose ownership of it).\n\nThe flexarray grows as needed, so we won't have a second batch locally.\n\n> \n>> +\t\t\tstring_list_clear(&cookie_list, 0);\n>> +\n>> +\t\t\t/*\n>> +\t\t\t * We assume that any events that we received\n>> +\t\t\t * in this callback after this dropped event\n>> +\t\t\t * may still be valid, so we continue rather\n>> +\t\t\t * than break. (And just in case there is a\n>> +\t\t\t * delete of \".git\" hiding in there.)\n>> +\t\t\t */\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n>> +\n>> +\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n>> +\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n>> +\t\t\t/* special case cookie files within .git or gitdir */\n>> +\n>> +\t\t\t/* Use just the filename of the cookie file. */\n>> +\t\t\tslash = find_last_dir_sep(path_k);\n>> +\t\t\tstring_list_append(&cookie_list,\n>> +\t\t\t\t\t slash ? slash + 1 : path_k);\n>> +\t\t\tbreak;\n>> +\n>> +\t\tcase IS_INSIDE_DOT_GIT:\n>> +\t\tcase IS_INSIDE_GITDIR:\n>> +\t\t\t/* ignore all other paths inside of .git or gitdir */\n>> +\t\t\tbreak;\n>> +\n>> +\t\tcase IS_DOT_GIT:\n>> +\t\tcase IS_GITDIR:\n>> +\t\t\t/*\n>> +\t\t\t * If .git directory is deleted or renamed away,\n>> +\t\t\t * we have to quit.\n>> +\t\t\t */\n>> +\t\t\tif (ef_is_root_delete(event_flags[k])) {\n>> +\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n>> +\t\t\t\t\t\t \"fsm-listen/gitdir\",\n>> +\t\t\t\t\t\t \"removed\");\n>> +\t\t\t\tgoto force_shutdown;\n>> +\t\t\t}\n>> +\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n>> +\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n>> +\t\t\t\t\t\t \"fsm-listen/gitdir\",\n>> +\t\t\t\t\t\t \"renamed\");\n>> +\t\t\t\tgoto force_shutdown;\n>> +\t\t\t}\n> \n> I see. The only difference is in how we trace the result. I'm not sure\n> this tracing message is worth the differentiation.\n\nI could get rid of it. It was helpful during development to ensure\nthat I had covered all the bases.\n\n> \n>> +\t\t\tbreak;\n>> +\n>> +\t\tcase IS_WORKDIR_PATH:\n>> +\t\t\t/* try to queue normal pathnames */\n>> +\n>> +\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n>> +\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n>> +\n>> +\t\t\t/* fsevent could be marked as both a file and directory */\n> \n> The _same_ event? Interesting. And I see that you need to log the name\n> differently in the case of a file or a directory.\n\nApple will consolidate messages, so if you delete a file and create a\ndirectory with the same name quick enough, they will send a single event\nwith both bits set. We receive a batch of events from the kernel with a\ncertain frequency, so we might get 2 events in separate bins, or 1 event\nwith both bits.\n\nYes, we queue up different paths. Adding the trailing slash tells the\nclient to invalidate a range of cache-entries, for example.\n\n> \n>> +\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n>> +\t\t\t\tconst char *rel = path_k +\n>> +\t\t\t\t\tstate->path_worktree_watch.len + 1;\n>> +\n>> +\t\t\t\tif (!batch)\n>> +\t\t\t\t\tbatch = fsmonitor_batch__new();\n>> +\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n>> +\t\t\t}\n>> +\n>> +\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n>> +\t\t\t\tconst char *rel = path_k +\n>> +\t\t\t\t\tstate->path_worktree_watch.len + 1;\n>> +\t\t\t\tchar *p = xstrfmt(\"%s/\", rel);\n> \n> In a critical path, xstrfmt() may be too slow for such a simple case.\n> Likely we should instead use a strbuf with:\n> \n> \tstrbuf_addstr(&p, rel);\n> \tstrbuf_addch(&p, '/');\n> \n> Bonus points if we can use the data to predict the size of the strbuf's\n> buffer.\n\nGood point!\n\n> \n>> +\n>> +\t\t\t\tif (!batch)\n>> +\t\t\t\t\tbatch = fsmonitor_batch__new();\n>> +\t\t\t\tfsmonitor_batch__add_path(batch, p);\n>> +\n>> +\t\t\t\tfree(p);\n>> +\t\t\t}\n>> +\n>> +\t\t\tbreak;\n>> +\n>> +\t\tcase IS_OUTSIDE_CONE:\n>> +\t\tdefault:\n>> +\t\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t\t \"ignoring '%s'\", path_k);\n>> +\t\t\tbreak;\n>> +\t\t}\n>> +\t}\n>> +\n>> +\tfsmonitor_publish(state, batch, &cookie_list);\n>> +\tstring_list_clear(&cookie_list, 0);\n>> +\treturn;\n>> +\n>> +force_shutdown:\n>> +\tif (fsmonitor_batch__free(batch))\n>> +\t\tBUG(\"batch should not have a next\");\n>> +\tstring_list_clear(&cookie_list, 0);\n>> +\n>> +\tdata->shutdown_style = FORCE_SHUTDOWN;\n>> +\tCFRunLoopStop(data->rl);\n>> +\treturn;\n>> +}\n>> +\n>> +/*\n>> + * TODO Investigate the proper value for the `latency` argument in the call\n>> + * TODO to `FSEventStreamCreate()`. I'm not sure that this needs to be a\n>> + * TODO config setting or just something that we tune after some testing.\n>> + * TODO\n>> + * TODO With a latency of 0.1, I was seeing lots of dropped events during\n>> + * TODO the \"touch 100000\" files test within t/perf/p7519, but with a\n>> + * TODO latency of 0.001 I did not see any dropped events. So the \"correct\"\n>> + * TODO value may be somewhere in between.\n>> + * TODO\n>> + * TODO https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n>> + */\n> \n> As Eric mentioned in another thread, this should say \"NEEDSWORK\" at\n> the top. This is a good candidate for follow-up after the basics of\n> the series is stable.\n\nRight.\n\n> \n>> int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n>> {\n>> +\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n>> +\t\tkFSEventStreamCreateFlagWatchRoot |\n>> +\t\tkFSEventStreamCreateFlagFileEvents;\n>> +\tFSEventStreamContext ctx = {\n>> +\t\t0,\n>> +\t\tstate,\n>> +\t\tNULL,\n>> +\t\tNULL,\n>> +\t\tNULL\n>> +\t};\n>> +\tstruct fsmonitor_daemon_backend_data *data;\n>> +\tconst void *dir_array[2];\n>> +\n>> +\tdata = xcalloc(1, sizeof(*data));\n> \n> CALLOC_ARRAY()\n> \n>> +\tstate->backend_data = data;\n>> +\n>> +\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n>> +\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n>> +\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n>> +\n>> +\tif (state->nr_paths_watching > 1) {\n>> +\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n>> +\t\t\tNULL, state->path_gitdir_watch.buf,\n>> +\t\t\tkCFStringEncodingUTF8);\n>> +\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n>> +\t}\n>> +\n>> +\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n>> +\t\t\t\t\t\t data->nr_paths_watching,\n>> +\t\t\t\t\t\t NULL);\n>> +\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n>> +\t\t\t\t\t data->cfar_paths_to_watch,\n>> +\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n>> +\t\t\t\t\t 0.001, flags);\n>> +\tif (data->stream == NULL)\n>> +\t\tgoto failed;\n>> +\n>> +\t/*\n>> +\t * `data->rl` needs to be set inside the listener thread.\n>> +\t */\n>> +\n>> +\treturn 0;\n>> +\n>> +failed:\n>> +\terror(\"Unable to create FSEventStream.\");\n>> +\n>> +\tFREE_AND_NULL(state->backend_data);\n>> \treturn -1;\n>> }\n>> \n>> void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n>> {\n>> +\tstruct fsmonitor_daemon_backend_data *data;\n>> +\n>> +\tif (!state || !state->backend_data)\n>> +\t\treturn;\n>> +\n>> +\tdata = state->backend_data;\n>> +\n>> +\tif (data->stream) {\n>> +\t\tif (data->stream_started)\n>> +\t\t\tFSEventStreamStop(data->stream);\n>> +\t\tif (data->stream_scheduled)\n>> +\t\t\tFSEventStreamInvalidate(data->stream);\n>> +\t\tFSEventStreamRelease(data->stream);\n>> +\t}\n>> +\n>> +\tFREE_AND_NULL(state->backend_data);\n>> }\n>> \n>> void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n>> {\n>> +\tstruct fsmonitor_daemon_backend_data *data;\n>> +\n>> +\tdata = state->backend_data;\n>> +\tdata->shutdown_style = SHUTDOWN_EVENT;\n>> +\n>> +\tCFRunLoopStop(data->rl);\n>> }\n>> \n>> void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n>> {\n>> +\tstruct fsmonitor_daemon_backend_data *data;\n>> +\n>> +\tdata = state->backend_data;\n>> +\n>> +\tdata->rl = CFRunLoopGetCurrent();\n>> +\n>> +\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n>> +\tdata->stream_scheduled = 1;\n>> +\n>> +\tif (!FSEventStreamStart(data->stream)) {\n>> +\t\terror(\"Failed to start the FSEventStream\");\n>> +\t\tgoto force_error_stop_without_loop;\n>> +\t}\n>> +\tdata->stream_started = 1;\n>> +\n>> +\tCFRunLoopRun();\n>> +\n>> +\tswitch (data->shutdown_style) {\n>> +\tcase FORCE_ERROR_STOP:\n>> +\t\tstate->error_code = -1;\n>> +\t\t/* fall thru */\n>> +\tcase FORCE_SHUTDOWN:\n>> +\t\tipc_server_stop_async(state->ipc_server_data);\n>> +\t\t/* fall thru */\n>> +\tcase SHUTDOWN_EVENT:\n>> +\tdefault:\n>> +\t\tbreak;\n>> +\t}\n>> +\treturn;\n>> +\n>> +force_error_stop_without_loop:\n>> +\tstate->error_code = -1;\n>> +\tipc_server_stop_async(state->ipc_server_data);\n>> +\treturn;\n>> }\n> \n> Thanks,\n> -Stolee\n> \n\nBig thanks for looking at all of this!\nJeff\n"},{"id":"423499","messageId":"cde39ed5-1175-9163-698a-eb5eca93ba59@jeffhostetler.com","threadId":"55420","inReplyTo":"35876c33-4e1a-bc07-cd6c-8d43a25c7903@gmail.com","subject":"Re: [PATCH 16/23] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-05-03T15:04:19Z","receivedAt":"2021-05-03T15:04:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/26/21 5:01 PM, Derrick Stolee wrote:\n> On 4/1/2021 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Teach fsmonitor--daemon to respond to IPC requests from client\n>> Git processes and respond with a list of modified pathnames\n>> relative to the provided token.\n> \n> (I'm skipping ahead to this part. I'll examine the platform\n> specific bits after I finish with \"the Git bits\".)\n> \n>> +static void fsmonitor_format_response_token(\n>> +\tstruct strbuf *response_token,\n>> +\tconst struct strbuf *response_token_id,\n>> +\tconst struct fsmonitor_batch *batch)\n>> +{\n>> +\tuint64_t seq_nr = (batch) ? batch->batch_seq_nr + 1 : 0;\n>> +\n>> +\tstrbuf_reset(response_token);\n>> +\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n>> +\t\t response_token_id->buf, seq_nr);]\n> \n> Ah, right. The token string gets _even more specific_ to allow\n> for multiple \"checkpoints\" within a batch.\n> \n>> +static int fsmonitor_parse_client_token(const char *buf_token,\n>> +\t\t\t\t\tstruct strbuf *requested_token_id,\n>> +\t\t\t\t\tuint64_t *seq_nr)\n>> +{\n>> +\tconst char *p;\n>> +\tchar *p_end;\n>> +\n>> +\tstrbuf_reset(requested_token_id);\n>> +\t*seq_nr = 0;\n>> +\n>> +\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n>> +\t\treturn 1;\n>> +\n>> +\twhile (*p && *p != ':')\n>> +\t\tstrbuf_addch(requested_token_id, *p++);\n> \n> My mind is going towards microoptimizations, but I wonder if there\n> is a difference using\n> \n> \tq = strchr(p, ':');\n> \tif (!q)\n> \t\treturn 1;\n> \tstrbuf_add(requested_token_id, p, q - p);\n> \n> We trade one scan with several method calls for instead two scans\n> and two method calls, but also those two scans are very optimized.\n> \n> Probably not worth it, as this is something like 20 bytes of data\n> per round-trip.\n\nI'll take a look at this and your later comment about double parsing\n\"builtin:\".\n\n> \n>> +\tif (!skip_prefix(command, \"builtin:\", &p)) {\n>> +\t\t/* assume V1 timestamp or garbage */\n>> +\n>> +\t\tchar *p_end;\n>> +\n>> +\t\tstrtoumax(command, &p_end, 10);\n>> +\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t ((*p_end) ?\n>> +\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n>> +\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n>> +\t\t\t\t command);\n>> +\t\tresult = -1;\n>> +\t\tgoto send_trivial_response;\n>> +\t}\n> \n> This is an interesting protection for users currently using FS\n> Monitor but upgrade to the builtin approach.\n\nYes, the token is stored in the .git/index extension, so we always\nhave to assume that we'll get an old-school token that they inherited\nfrom the last command executed before they switched FSMonitor providers.\n\nAlso, there is a chicken-n-egg problem in the FSMonitor protocol that\nwe inherited from the V1 effort -- the client has to initiate the\nconversation with a token/timestamp, but has yet to talk to the daemon\n(or hook) and doesn't know what a V2 token looks like (since they are\nopaque to the client). So I let the client blindly send a V1 timestamp\nwhich the daemon will silently reject and send a trivial response (which\ntells the client to do the work itself and gives it a V2 token for the\nnext call).\n\n> \n>> +\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n>> +\t\t\t\t\t &requested_oldest_seq_nr)) {\n> \n> It appears you will call skip_prefix() twice this way, once to\n> determine we are actually the right kind of token, but a second\n> time as part of this call. Perhaps the helper method could start\n> from 'p' which has already advanced beyond \"buildin:\"?\n> \n>> +\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n>> +\t\t\t\t command);\n>> +\t\tresult = -1;\n>> +\t\tgoto send_trivial_response;\n>> +\t}\n> \n> This method is getting a bit long. Could the interesting data\n> structure code below be extracted as a method?\n\nLet me try refactoring it.\n\n> \n>> +\tpthread_mutex_lock(&state->main_lock);\n>> +\n>> +\tif (!state->current_token_data) {\n>> +\t\t/*\n>> +\t\t * We don't have a current token. This may mean that\n>> +\t\t * the listener thread has not yet started.\n>> +\t\t */\n>> +\t\tpthread_mutex_unlock(&state->main_lock);\n>> +\t\tresult = 0;\n>> +\t\tgoto send_trivial_response;\n>> +\t}\n>> +\tif (strcmp(requested_token_id.buf,\n>> +\t\t state->current_token_data->token_id.buf)) {\n>> +\t\t/*\n>> +\t\t * The client last spoke to a different daemon\n>> +\t\t * instance -OR- the daemon had to resync with\n>> +\t\t * the filesystem (and lost events), so reject.\n>> +\t\t */\n>> +\t\tpthread_mutex_unlock(&state->main_lock);\n>> +\t\tresult = 0;\n>> +\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n>> +\t\t\t\t \"response/token\", \"different\");\n>> +\t\tgoto send_trivial_response;\n>> +\t}\n>> +\tif (!state->current_token_data->batch_tail) {\n>> +\t\t/*\n>> +\t\t * The listener has not received any filesystem\n>> +\t\t * events yet since we created the current token.\n>> +\t\t * We can respond with an empty list, since the\n>> +\t\t * client has already seen the current token and\n>> +\t\t * we have nothing new to report. (This is\n>> +\t\t * instead of sending a trivial response.)\n>> +\t\t */\n>> +\t\tpthread_mutex_unlock(&state->main_lock);\n>> +\t\tresult = 0;\n>> +\t\tgoto send_empty_response;\n>> +\t}\n>> +\tif (requested_oldest_seq_nr <\n>> +\t state->current_token_data->batch_tail->batch_seq_nr) {\n>> +\t\t/*\n>> +\t\t * The client wants older events than we have for\n>> +\t\t * this token_id. This means that the end of our\n>> +\t\t * batch list was truncated and we cannot give the\n>> +\t\t * client a complete snapshot relative to their\n>> +\t\t * request.\n>> +\t\t */\n>> +\t\tpthread_mutex_unlock(&state->main_lock);\n>> +\n>> +\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t \"client requested truncated data\");\n>> +\t\tresult = 0;\n>> +\t\tgoto send_trivial_response;\n>> +\t}\n> \n> If these are part of a helper method, then they could be reorganized\n> to \"goto\" the end of the method which returns an error code after\n> unlocking the mutex. The multiple unlocks are making me nervous.\n> \n>> +\n>> +\t/*\n>> +\t * We're going to hold onto a pointer to the current\n>> +\t * token-data while we walk the list of batches of files.\n>> +\t * During this time, we will NOT be under the lock.\n>> +\t * So we ref-count it.\n> \n> I was wondering if this would happen. I'm glad it is.\n> \n>> +\t * This allows the listener thread to continue prepending\n>> +\t * new batches of items to the token-data (which we'll ignore).\n>> +\t *\n>> +\t * AND it allows the listener thread to do a token-reset\n>> +\t * (and install a new `current_token_data`).\n>> +\t *\n>> +\t * We mark the current head of the batch list as \"pinned\" so\n>> +\t * that the listener thread will treat this item as read-only\n>> +\t * (and prevent any more paths from being added to it) from\n>> +\t * now on.\n>> +\t */\n>> +\ttoken_data = state->current_token_data;\n>> +\ttoken_data->client_ref_count++;\n>> +\n>> +\tbatch_head = token_data->batch_head;\n>> +\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n>> +\n>> +\tpthread_mutex_unlock(&state->main_lock);\n> \n> We are now pinned. Makes sense.\n> \n>> +\t/*\n>> +\t * FSMonitor Protocol V2 requires that we send a response header\n>> +\t * with a \"new current token\" and then all of the paths that changed\n>> +\t * since the \"requested token\".\n>> +\t */\n>> +\tfsmonitor_format_response_token(&response_token,\n>> +\t\t\t\t\t&token_data->token_id,\n>> +\t\t\t\t\tbatch_head);\n>> +\n>> +\treply(reply_data, response_token.buf, response_token.len + 1);\n>> +\ttotal_response_len += response_token.len + 1;\n> \n> I was going to say we should let \"reply\" return the number of bytes written,\n> but that is already an error code. But then we seem to be ignoring it here.\n> Should we at least do something like \"err |= reply()\" to collect any errors?\n\nMaybe. I'll have to look and see what the daemon thread can do\non such an error.\n\n> \n>> +\n>> +\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n>> +\t\t\t response_token.buf);\n>> +\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\", response_token.buf);\n>> +\n>> +\tshown = kh_init_str();\n>> +\tfor (batch = batch_head;\n>> +\t batch && batch->batch_seq_nr >= requested_oldest_seq_nr;\n>> +\t batch = batch->next) {\n>> +\t\tsize_t k;\n>> +\n>> +\t\tfor (k = 0; k < batch->nr; k++) {\n>> +\t\t\tconst char *s = batch->interned_paths[k];\n>> +\t\t\tsize_t s_len;\n>> +\n>> +\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n>> +\t\t\t\tduplicates++;\n>> +\t\t\telse {\n>> +\t\t\t\tkh_put_str(shown, s, &hash_ret);\n> \n> It appears that you could make use of 'struct strmap' instead of managing your\n> own khash structure.\n\nSince all of the strings in the batches are already interned\n(and we have constant fixed pointers for each string), I'd\neventually like to have this khash take advantage of that and\nmake this (essentially) a Set on the pointer values rather than\na Set on the strings. This version is a step in that direction.\n\n> \n>> +\n>> +\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n>> +\t\t\t\t\t\t count, s);\n>> +\n>> +\t\t\t\t/* Each path gets written with a trailing NUL */\n>> +\t\t\t\ts_len = strlen(s) + 1;\n>> +\n>> +\t\t\t\tif (payload.len + s_len >=\n>> +\t\t\t\t LARGE_PACKET_DATA_MAX) {\n>> +\t\t\t\t\treply(reply_data, payload.buf,\n>> +\t\t\t\t\t payload.len);\n>> +\t\t\t\t\ttotal_response_len += payload.len;\n>> +\t\t\t\t\tstrbuf_reset(&payload);\n>> +\t\t\t\t}\n>> +\n>> +\t\t\t\tstrbuf_add(&payload, s, s_len);\n>> +\t\t\t\tcount++;\n>> +\t\t\t}\n>> +\t\t}\n>> +\t}\n>> +\n>> +\tif (payload.len) {\n>> +\t\treply(reply_data, payload.buf, payload.len);\n>> +\t\ttotal_response_len += payload.len;\n>> +\t}\n>> +\n>> +\tkh_release_str(shown);\n>> +\n>> +\tpthread_mutex_lock(&state->main_lock);\n>> +\tif (token_data->client_ref_count > 0)\n>> +\t\ttoken_data->client_ref_count--;\n>> +\n>> +\tif (token_data->client_ref_count == 0) {\n>> +\t\tif (token_data != state->current_token_data) {\n>> +\t\t\t/*\n>> +\t\t\t * The listener thread did a token-reset while we were\n>> +\t\t\t * walking the batch list. Therefore, this token is\n>> +\t\t\t * stale and can be discarded completely. If we are\n>> +\t\t\t * the last reader thread using this token, we own\n>> +\t\t\t * that work.\n>> +\t\t\t */\n>> +\t\t\tfsmonitor_free_token_data(token_data);\n>> +\t\t}\n>> +\t}\n> \n> Perhaps this could be extracted to a method, so that any (locked) caller\n> could run\n> \n> \tfree_token_if_unused(state, token_data);\n> \n> and the token will either keep around (because client_ref_count > 0 or\n> state->current_token_data is still on token_data). Otherwise I predict\n> this being implemented in two places, which is too many when dealing with\n> memory ownership.\n\nI'll take a look at this.\n\n> \n>> +\n>> +\tpthread_mutex_unlock(&state->main_lock);\n>> +\n>> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n>> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n>> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n>> +\n>> +\tstrbuf_release(&response_token);\n>> +\tstrbuf_release(&requested_token_id);\n>> +\tstrbuf_release(&payload);\n>> +\n>> +\treturn 0;\n>> +\n>> +send_trivial_response:\n>> +\tpthread_mutex_lock(&state->main_lock);\n>> +\tfsmonitor_format_response_token(&response_token,\n>> +\t\t\t\t\t&state->current_token_data->token_id,\n>> +\t\t\t\t\tstate->current_token_data->batch_head);\n>> +\tpthread_mutex_unlock(&state->main_lock);\n>> +\n>> +\treply(reply_data, response_token.buf, response_token.len + 1);\n>> +\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n>> +\t\t\t response_token.buf);\n>> +\treply(reply_data, \"/\", 2);\n>> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/trivial\", 1);\n>> +\n>> +\tstrbuf_release(&response_token);\n>> +\tstrbuf_release(&requested_token_id);\n>> +\n>> +\treturn result;\n>> +\n>> +send_empty_response:\n>> +\tpthread_mutex_lock(&state->main_lock);\n>> +\tfsmonitor_format_response_token(&response_token,\n>> +\t\t\t\t\t&state->current_token_data->token_id,\n>> +\t\t\t\t\tNULL);\n>> +\tpthread_mutex_unlock(&state->main_lock);\n>> +\n>> +\treply(reply_data, response_token.buf, response_token.len + 1);\n>> +\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n>> +\t\t\t response_token.buf);\n>> +\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/empty\", 1);\n>> +\n>> +\tstrbuf_release(&response_token);\n>> +\tstrbuf_release(&requested_token_id);\n>> +\n>> +\treturn 0;\n>> +}\n>> +\n>> static ipc_server_application_cb handle_client;\n>> \n>> static int handle_client(void *data, const char *command,\n>> \t\t\t ipc_server_reply_cb *reply,\n>> \t\t\t struct ipc_server_reply_data *reply_data)\n>> {\n>> -\t/* struct fsmonitor_daemon_state *state = data; */\n>> +\tstruct fsmonitor_daemon_state *state = data;\n>> \tint result;\n>> \n>> +\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n>> +\n>> \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n>> \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n>> \n>> -\tresult = 0; /* TODO Do something here. */\n>> +\tresult = do_handle_client(state, command, reply, reply_data);\n>> \n>> \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n>> \n> \n> A simple integration with earlier work. Good.\n> \n> Thanks,\n> -Stolee\n> \n\nThanks,\nJeff\n"},{"id":"423541","messageId":"6bea878c-a9b6-ea88-0364-dc6ff60155f5@jeffhostetler.com","threadId":"55420","inReplyTo":"5669d313-4084-dceb-1204-acd005e58657@gmail.com","subject":"Re: [PATCH 19/23] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-05-03T21:59:27Z","receivedAt":"2021-05-03T21:59:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/27/21 10:23 AM, Derrick Stolee wrote:\n> On 4/1/2021 11:41 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Teach fsmonitor--daemon client threads to create a cookie file\n>> inside the .git directory and then wait until FS events for the\n>> cookie are observed by the FS listener thread.\n>>\n>> This helps address the racy nature of file system events by\n>> blocking the client response until the kernel has drained any\n>> event backlog.\n> \n> This description matches my expectation of the cookie file,\n> which furthers my confusion about GIT_TEST_FSMONITOR_CLIENT_DELAY.\n\nI'm going to try to create the cookie earlier in the thread\nand see if that lets me eliminate the delay. I don't remember\nif I added the delay first and then the cookie when I was testing\nor not. IIRC I was switching between the 2 techniques at one point.\n\n> \n>> +enum fsmonitor_cookie_item_result {\n>> +\tFCIR_ERROR = -1, /* could not create cookie file ? */\n>> +\tFCIR_INIT = 0,\n>> +\tFCIR_SEEN,\n>> +\tFCIR_ABORT,\n>> +};\n>> +\n>> +struct fsmonitor_cookie_item {\n>> +\tstruct hashmap_entry entry;\n>> +\tconst char *name;\n>> +\tenum fsmonitor_cookie_item_result result;\n>> +};\n>> +\n>> +static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n>> +\t\t const struct hashmap_entry *he2, const void *keydata)\n> \n> I'm interested to see why a hashset is necessary.\n\nI suppose I could search a linked list of active cookies, but\nthis seemed easier. Basically, we have an active cookie (and a\nsocket listener thread blocked) for each active client connection.\nWhen the FS event thread receives an FS notification for a cookie\nfile, it needs to do a quick lookup on the cookie file and release\nthe associate socket thread.\n\nGiven that we're only likely to have a few clients connected at\nany given time, a list might be faster.\n\n> \n>> +static enum fsmonitor_cookie_item_result fsmonitor_wait_for_cookie(\n>> +\tstruct fsmonitor_daemon_state *state)\n>> +{\n>> +\tint fd;\n>> +\tstruct fsmonitor_cookie_item cookie;\n>> +\tstruct strbuf cookie_pathname = STRBUF_INIT;\n>> +\tstruct strbuf cookie_filename = STRBUF_INIT;\n>> +\tconst char *slash;\n>> +\tint my_cookie_seq;\n>> +\n>> +\tpthread_mutex_lock(&state->main_lock);\n> \n> Hm. We are entering a locked region. I hope this is only for the\n> cookie write and not the entire waiting period.\n\nI'm taking the lock to increment the cookie_seq and to add\nthe hash-entry to the hashmap mainly. The cond_wait() after\nthe open() is an atomic unlock-and-wait-and-relock. So we\nwait there for the FS thread to tell us it has seen our cookie\nfile. Then we remove our hash-entry from the hashmap and unlock.\n\nYes, I am doing several things here, but it didn't seem\nworth it to lock-unlock-lock-unlock-lock-cond_wait...\n\n> \n>> +\tmy_cookie_seq = state->cookie_seq++;\n>> +\n>> +\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n>> +\tstrbuf_addf(&cookie_pathname, \"%i-%i\", getpid(), my_cookie_seq);\n>> +\n>> +\tslash = find_last_dir_sep(cookie_pathname.buf);\n>> +\tif (slash)\n>> +\t\tstrbuf_addstr(&cookie_filename, slash + 1);\n>> +\telse\n>> +\t\tstrbuf_addbuf(&cookie_filename, &cookie_pathname);\n> \n> This business about the slash-or-not-slash is good defensive\n> programming. I imagine the only possible way for there to not\n> be a slash is if the Git process is running with the .git\n> directory as its working directory?\n> \n>> +\tcookie.name = strbuf_detach(&cookie_filename, NULL);\n>> +\tcookie.result = FCIR_INIT;\n>> +\t// TODO should we have case-insenstive hash (and in cookie_cmp()) ??\n> \n> This TODO comment should be cleaned up. Doesn't match C-style, either.\n> \n> As for the question, I believe that we can limit ourselves to names that\n> don't need case-insensitive hashes and trust that the filesystem will not\n> change the case. Using lowercase letters should help with this.\n> \n\nI'm going to redo the pathname construction (to solve a conflict\nwith VSCode) and will clean up this.\n\n>> +\thashmap_entry_init(&cookie.entry, strhash(cookie.name));\n>> +\n>> +\t/*\n>> +\t * Warning: we are putting the address of a stack variable into a\n>> +\t * global hashmap. This feels dodgy. We must ensure that we remove\n>> +\t * it before this thread and stack frame returns.\n>> +\t */\n>> +\thashmap_add(&state->cookies, &cookie.entry);\n> \n> I saw this warning and thought about avoiding it by using the heap, but\n> even with a heap pointer we need to be careful to remove the result\n> before returning and stopping the thread.\n> \n> However, there is likely a higher potential of a bug leading to a\n> security issue through an error causing stack corruption and unsafe\n> code execution. Perhaps it is worth converting to using heap data here.\n\nI never liked the stack buffer. I'm going to move it to the heap.\n\n> \n>> +\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n>> +\t\t\t cookie.name, cookie_pathname.buf);\n>> +\n>> +\t/*\n>> +\t * Create the cookie file on disk and then wait for a notification\n>> +\t * that the listener thread has seen it.\n>> +\t */\n>> +\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n>> +\tif (fd >= 0) {\n>> +\t\tclose(fd);\n>> +\t\tunlink_or_warn(cookie_pathname.buf);\n> \n> Interesting that we are ignoring the warning here. Is it possible that\n> these cookie files will continue to grow if this unlink fails?\n\nIt is possible that the unlink() could fail, but I'm not\nsure what we can do about it. The FS event from the open()\n(and/or the close()) will be sufficient to wake up this thread.\n\n> \n>> +\n>> +\t\twhile (cookie.result == FCIR_INIT)\n>> +\t\t\tpthread_cond_wait(&state->cookies_cond,\n>> +\t\t\t\t\t &state->main_lock);\n> \n> Ok, we are waiting here for another thread to signal that the cookie\n> file has been found in the events. What happens if the event gets lost?\n> I'll look for a later signal that cookie.result can change based on a\n> timeout, too.\n\nI'd like to use `pthread_cond_timedwait()` here, but I'm not\nsure it is supported everywhere.\n\nI do have code in the FS layers to dump/alert all cookies\nat certain times, such as loss of sync.\n\n> \n>> +\n>> +\t\thashmap_remove(&state->cookies, &cookie.entry, NULL);\n>> +\t} else {\n>> +\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n>> +\t\t\t cookie.name);\n>> +\n>> +\t\tcookie.result = FCIR_ERROR;\n>> +\t\thashmap_remove(&state->cookies, &cookie.entry, NULL);\n>> +\t}\n> \n> Both blocks here remove the cookie entry, so move it to the end of the\n> method with the other cleanups.\n\nI can move it outside of the IF, but it has to be before we unlock.\n\n> \n>> +\n>> +\tpthread_mutex_unlock(&state->main_lock);\n> \n> Hm. We are locking the main state throughout this process. I suppose that\n> the listener thread could be watching multiple repos and updating them\n> while we wait here for one repo to update. This is a larger lock window\n> than I was hoping for, but I don't currently see how to reduce it safely.\n\nWe only watch a single repo/working directory. We're locking because\nwe could have multiple clients all hitting us at the same time.\n\n> \n>> +\n>> +\tfree((char*)cookie.name);\n>> +\tstrbuf_release(&cookie_pathname);\n>> +\treturn cookie.result;\n> \n> Remove the cookie from the hashset along with these lines.\n\nNo, it has to be within the lock above.\n\n> \n>> +}\n>> +\n>> +/*\n>> + * Mark these cookies as _SEEN and wake up the corresponding client threads.\n>> + */\n>> +static void fsmonitor_cookie_mark_seen(struct fsmonitor_daemon_state *state,\n>> +\t\t\t\t const struct string_list *cookie_names)\n>> +{\n>> +\t/* assert state->main_lock */\n> \n> I'm now confused what this is trying to document. The 'state' should be\n> locked by another thread while we are waiting for a cookie response, so\n> this method is updating the cookie as seen from a different thread that\n> doesn't have the lock.\n\nI'm trying to document that this function must be called while the\nthread is holding the main lock (without paying for a lock check or\ntrying to do a recursive lock or whatever).\n\nSince it is a little static function and I control the 2 or 3 callers,\nI can just visually check this without fuss.\n\n> \n> ...\n>> +/*\n>> + * Set _ABORT on all pending cookies and wake up all client threads.\n>> + */\n>> +static void fsmonitor_cookie_abort_all(struct fsmonitor_daemon_state *state)\n> ...\n> \n>> + * [2] Some of those lost events may have been for cookie files. We\n>> + * should assume the worst and abort them rather letting them starve.\n>> + *\n>> * If there are no readers of the the current token data series, we\n>> * can free it now. Otherwise, let the last reader free it. Either\n>> * way, the old token data series is no longer associated with our\n>> @@ -454,6 +600,8 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n>> \t\t\t state->current_token_data->token_id.buf,\n>> \t\t\t new_one->token_id.buf);\n>> \n>> +\tfsmonitor_cookie_abort_all(state);\n>> +\n> \n> I see we abort here if we force a resync. I lost the detail of whether\n> this is triggered by a timeout, too.\n\nI don't currently have a cookie timeout for each thread. I'd like\nto use pthread_cond_timedwait(), but I didn't see it in the\ncompat headers, so I'm not sure if it is portable. I'll make a note\nto look into this again.\n\n> \n>> @@ -654,6 +803,39 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n>> \t\tgoto send_trivial_response;\n>> \t}\n>> \n>> +\tpthread_mutex_unlock(&state->main_lock);\n>> +\n>> +\t/*\n>> +\t * Write a cookie file inside the directory being watched in an\n>> +\t * effort to flush out existing filesystem events that we actually\n>> +\t * care about. Suspend this client thread until we see the filesystem\n>> +\t * events for this cookie file.\n>> +\t */\n>> +\tcookie_result = fsmonitor_wait_for_cookie(state);\n> \n> Odd that we unlock before calling this method, then just take the lock\n> again inside of it.\n\nYeah, I didn't like doing that. I'll revisit.\n\n> \n>> +\tif (cookie_result != FCIR_SEEN) {\n>> +\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n>> +\t\t cookie_result);\n>> +\t\tresult = 0;\n>> +\t\tgoto send_trivial_response;\n>> +\t}\n>> +\n>> +\tpthread_mutex_lock(&state->main_lock);\n>> +\n>> +\tif (strcmp(requested_token_id.buf,\n>> +\t\t state->current_token_data->token_id.buf)) {\n>> +\t\t/*\n>> +\t\t * Ack! The listener thread lost sync with the filesystem\n>> +\t\t * and created a new token while we were waiting for the\n>> +\t\t * cookie file to be created! Just give up.\n>> +\t\t */\n>> +\t\tpthread_mutex_unlock(&state->main_lock);\n>> +\n>> +\t\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t\t \"lost filesystem sync\");\n>> +\t\tresult = 0;\n>> +\t\tgoto send_trivial_response;\n>> +\t}\n>> +\n>> \t/*\n>> \t * We're going to hold onto a pointer to the current\n>> \t * token-data while we walk the list of batches of files.\n>> @@ -982,6 +1164,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n>> \t\t}\n>> \t}\n>> \n>> +\tif (cookie_names->nr)\n>> +\t\tfsmonitor_cookie_mark_seen(state, cookie_names);\n>> +\n> \n> I was confused as to what updates 'cookie_names', but it appears that\n> these are updated in the platform-specific code. That seems to happen\n> in later patches.\n\nYes, this is a list of the cookies that the platform layer saw events\nfor. It was passed in along with the set of batched paths. So the\nplatform code can \"publish/prepend\" a new set of changed paths and\nwake any threads whose cookie was seen.\n\n> \n> Thanks,\n> -Stolee\n> \n\nThanks,\nJeff\n"},{"id":"423695","messageId":"nycvar.QRO.7.76.6.2105051808190.50@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"87eeeuymtl.fsf@evledraar.gmail.com","subject":"Re: Nesting topics within other threads (was: [PATCH] repo-settings.c: simplify the setup)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-05-05T16:12:18Z","receivedAt":"2021-05-05T16:12:30Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Thu, 29 Apr 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Wed, Apr 28 2021, Derrick Stolee wrote:\n>\n> > On 4/28/2021 12:26 PM, Ævar Arnfjörð Bjarmason wrote:\n> >> Simplify the setup code in repo-settings.c in various ways, making the\n> >> code shorter, easier to read, and requiring fewer hacks to do the same\n> >> thing as it did before:\n> >\n> > [...]\n> > Since I've committed to reviewing the FS Monitor code, I'd prefer if\n> > this patch (or maybe its v2, since this is here already) be sent as\n> > a top-level message so it can be discussed independently.\n>\n> As a practical matter I think any effort I make to accommodate your\n> request will be dwarfed by your own starting of a sub-thread on\n> E-Mail/MUA nuances :)\n>\n> When [1] was brought up the other day (showing that I'm probably not the\n> best person to ask about on-list In-Reply-To semantics) I was surprised\n> to find that we don't have much (if any) explicit documentation about\n> In-Reply-To best practices. [...]\n>\n> 1. https://lore.kernel.org/git/nycvar.QRO.7.76.6.2103191540330.57@tvgsbejvaqbjf.bet/\n\nI find it a bit disingenous to reference my complaint about your\ndisconnected cover letter (which _definitely_ belongs with the patches for\nwhich it covers) with the practice of hiding patches or patch\nseries deep in a thread discussion an (lengthy!) patch series,\n_especially_ if it threatens to totally conflict with that patch series\nand thereby disrupt the flow.\n\nCouldn't you hold off with your patch for a while, instead help FSMonitor\nget over the finish line, and _then_ submit that simplification of\nrepo-settings? That would be constructive, from my perspective.\n\nCiao,\nDscho\n"},{"id":"424494","messageId":"0b9e54ba-ac27-e537-7bef-1b4448f92352@gmail.com","threadId":"55420","inReplyTo":"2b4ae4fc3d622f0fee8eb5e527a51467e13acfc5.1617291666.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 16/23] fsmonitor--daemon: implement handle_client callback","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-13T18:52:20Z","receivedAt":"2021-05-13T18:52:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/21 11:40 AM, Jeff Hostetler via GitGitGadget wrote:\n\nHere is a rather important bug that I found on a whim while working\nwith sparse-index integrations. The sparse-index isn't important\nexcept that it caused a different pattern of batch creation and\nresponses from the daemon.\n\n> +/*\n> + * Format an opaque token string to send to the client.\n> + */\n> +static void fsmonitor_format_response_token(\n> +\tstruct strbuf *response_token,\n> +\tconst struct strbuf *response_token_id,\n> +\tconst struct fsmonitor_batch *batch)\n> +{\n> +\tuint64_t seq_nr = (batch) ? batch->batch_seq_nr + 1 : 0;\n\nHere, you add one to the batch value to indicate a difference\nbetween \"zero\" and \"positive\" values.\n\n> +\n> +\tstrbuf_reset(response_token);\n> +\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n> +\t\t response_token_id->buf, seq_nr);\n> +}\n> +\n> +/*\n> + * Parse an opaque token from the client.\n> + */\n> +static int fsmonitor_parse_client_token(const char *buf_token,\n> +\t\t\t\t\tstruct strbuf *requested_token_id,\n> +\t\t\t\t\tuint64_t *seq_nr)\n> +{\n> +\tconst char *p;\n> +\tchar *p_end;\n> +\n> +\tstrbuf_reset(requested_token_id);\n> +\t*seq_nr = 0;\n> +\n> +\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n> +\t\treturn 1;\n> +\n> +\twhile (*p && *p != ':')\n> +\t\tstrbuf_addch(requested_token_id, *p++);\n> +\tif (!*p++)\n> +\t\treturn 1;\n> +\n> +\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n\nWhich means here you should decrement one from the value, possibly,\n(except if it is zero).\n\n> +\tif (*p_end)\n> +\t\treturn 1;\n> +\n> +\treturn 0;\n> +}\n\n...\n\n> +\tshown = kh_init_str();\n> +\tfor (batch = batch_head;\n> +\t batch && batch->batch_seq_nr >= requested_oldest_seq_nr;\n> +\t batch = batch->next) {\n\nAnd without either decrementing one from requested_oldest_seq_nr or\nadding one to the batch_seq_nr here, this loop could terminate\nimmediately.\n\nIn my testing, I added one to the left-hand side of the inequality.\n\nThanks,\n-Stolee\n"},{"id":"425303","messageId":"763fa1ee7bb61d236cb0e7e2f01a3a03af87ac2d.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 01/28] simple-ipc: preparations for supporting binary messages.","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:40Z","receivedAt":"2021-05-22T13:57:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd `command_len` argument to the Simple IPC API.\n\nIn my original Simple IPC API, I assumed that the request\nwould always be a null-terminated string of text characters.\nThe command arg was just a `const char *`.\n\nI found a caller that would like to pass a binary command\nto the daemon, so I want to ammend the Simple IPC API to\ntake `const char *command, size_t command_len` and pass\nthat to the daemon. (Really, the first arg should just be\na `void *` or `const unsigned byte *` to make that clearer.)\n\nNote, the response side has always been a `struct strbuf`\nwhich includes the buffer and length, so we already support\nreturning a binary answer. (Yes, it feels a little weird\nreturning a binary buffer in a `strbuf`, but it works.)\n\nSigned-off-by: Jeff Hostetler \n---\n compat/simple-ipc/ipc-unix-socket.c | 14 +++++++-----\n compat/simple-ipc/ipc-win32.c | 14 +++++++-----\n simple-ipc.h | 7 ++++--\n t/helper/test-simple-ipc.c | 34 +++++++++++++++++++----------\n 4 files changed, 46 insertions(+), 23 deletions(-)\n\ndiff --git a/compat/simple-ipc/ipc-unix-socket.c b/compat/simple-ipc/ipc-unix-socket.c\nindex 38689b278df3..0a2d1c1162b9 100644\n--- a/compat/simple-ipc/ipc-unix-socket.c\n+++ b/compat/simple-ipc/ipc-unix-socket.c\n@@ -164,7 +164,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection)\n \n int ipc_client_send_command_to_connection(\n \tstruct ipc_client_connection *connection,\n-\tconst char *message, struct strbuf *answer)\n+\tconst char *message, size_t message_len,\n+\tstruct strbuf *answer)\n {\n \tint ret = 0;\n \n@@ -172,7 +173,7 @@ int ipc_client_send_command_to_connection(\n \n \ttrace2_region_enter(\"ipc-client\", \"send-command\", NULL);\n \n-\tif (write_packetized_from_buf_no_flush(message, strlen(message),\n+\tif (write_packetized_from_buf_no_flush(message, message_len,\n \t\t\t\t\t connection->fd) < 0 ||\n \t packet_flush_gently(connection->fd) < 0) {\n \t\tret = error(_(\"could not send IPC command\"));\n@@ -193,7 +194,8 @@ int ipc_client_send_command_to_connection(\n \n int ipc_client_send_command(const char *path,\n \t\t\t const struct ipc_client_connect_options *options,\n-\t\t\t const char *message, struct strbuf *answer)\n+\t\t\t const char *message, size_t message_len,\n+\t\t\t struct strbuf *answer)\n {\n \tint ret = -1;\n \tenum ipc_active_state state;\n@@ -204,7 +206,9 @@ int ipc_client_send_command(const char *path,\n \tif (state != IPC_STATE__LISTENING)\n \t\treturn ret;\n \n-\tret = ipc_client_send_command_to_connection(connection, message, answer);\n+\tret = ipc_client_send_command_to_connection(connection,\n+\t\t\t\t\t\t message, message_len,\n+\t\t\t\t\t\t answer);\n \n \tipc_client_close_connection(connection);\n \n@@ -499,7 +503,7 @@ static int worker_thread__do_io(\n \tif (ret >= 0) {\n \t\tret = worker_thread_data->server_data->application_cb(\n \t\t\tworker_thread_data->server_data->application_data,\n-\t\t\tbuf.buf, do_io_reply_callback, &reply_data);\n+\t\t\tbuf.buf, buf.len, do_io_reply_callback, &reply_data);\n \n \t\tpacket_flush_gently(reply_data.fd);\n \t}\ndiff --git a/compat/simple-ipc/ipc-win32.c b/compat/simple-ipc/ipc-win32.c\nindex 8f89c02037e3..632fb3c7ea24 100644\n--- a/compat/simple-ipc/ipc-win32.c\n+++ b/compat/simple-ipc/ipc-win32.c\n@@ -204,7 +204,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection)\n \n int ipc_client_send_command_to_connection(\n \tstruct ipc_client_connection *connection,\n-\tconst char *message, struct strbuf *answer)\n+\tconst char *message, size_t message_len,\n+\tstruct strbuf *answer)\n {\n \tint ret = 0;\n \n@@ -212,7 +213,7 @@ int ipc_client_send_command_to_connection(\n \n \ttrace2_region_enter(\"ipc-client\", \"send-command\", NULL);\n \n-\tif (write_packetized_from_buf_no_flush(message, strlen(message),\n+\tif (write_packetized_from_buf_no_flush(message, message_len,\n \t\t\t\t\t connection->fd) < 0 ||\n \t packet_flush_gently(connection->fd) < 0) {\n \t\tret = error(_(\"could not send IPC command\"));\n@@ -235,7 +236,8 @@ int ipc_client_send_command_to_connection(\n \n int ipc_client_send_command(const char *path,\n \t\t\t const struct ipc_client_connect_options *options,\n-\t\t\t const char *message, struct strbuf *response)\n+\t\t\t const char *message, size_t message_len,\n+\t\t\t struct strbuf *response)\n {\n \tint ret = -1;\n \tenum ipc_active_state state;\n@@ -246,7 +248,9 @@ int ipc_client_send_command(const char *path,\n \tif (state != IPC_STATE__LISTENING)\n \t\treturn ret;\n \n-\tret = ipc_client_send_command_to_connection(connection, message, response);\n+\tret = ipc_client_send_command_to_connection(connection,\n+\t\t\t\t\t\t message, message_len,\n+\t\t\t\t\t\t response);\n \n \tipc_client_close_connection(connection);\n \n@@ -454,7 +458,7 @@ static int do_io(struct ipc_server_thread_data *server_thread_data)\n \tif (ret >= 0) {\n \t\tret = server_thread_data->server_data->application_cb(\n \t\t\tserver_thread_data->server_data->application_data,\n-\t\t\tbuf.buf, do_io_reply_callback, &reply_data);\n+\t\t\tbuf.buf, buf.len, do_io_reply_callback, &reply_data);\n \n \t\tpacket_flush_gently(reply_data.fd);\n \ndiff --git a/simple-ipc.h b/simple-ipc.h\nindex dc3606e30bd6..c4d5225b41c2 100644\n--- a/simple-ipc.h\n+++ b/simple-ipc.h\n@@ -111,7 +111,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection);\n */\n int ipc_client_send_command_to_connection(\n \tstruct ipc_client_connection *connection,\n-\tconst char *message, struct strbuf *answer);\n+\tconst char *message, size_t message_len,\n+\tstruct strbuf *answer);\n \n /*\n * Used by the client to synchronously connect and send and receive a\n@@ -123,7 +124,8 @@ int ipc_client_send_command_to_connection(\n */\n int ipc_client_send_command(const char *path,\n \t\t\t const struct ipc_client_connect_options *options,\n-\t\t\t const char *message, struct strbuf *answer);\n+\t\t\t const char *message, size_t message_len,\n+\t\t\t struct strbuf *answer);\n \n /*\n * Simple IPC Server Side API.\n@@ -148,6 +150,7 @@ typedef int (ipc_server_reply_cb)(struct ipc_server_reply_data *,\n */\n typedef int (ipc_server_application_cb)(void *application_data,\n \t\t\t\t\tconst char *request,\n+\t\t\t\t\tsize_t request_len,\n \t\t\t\t\tipc_server_reply_cb *reply_cb,\n \t\t\t\t\tstruct ipc_server_reply_data *reply_data);\n \ndiff --git a/t/helper/test-simple-ipc.c b/t/helper/test-simple-ipc.c\nindex 42040ef81b1e..913451807509 100644\n--- a/t/helper/test-simple-ipc.c\n+++ b/t/helper/test-simple-ipc.c\n@@ -112,7 +112,7 @@ static int app__slow_command(ipc_server_reply_cb *reply_cb,\n /*\n * The client sent a command followed by a (possibly very) large buffer.\n */\n-static int app__sendbytes_command(const char *received,\n+static int app__sendbytes_command(const char *received, size_t received_len,\n \t\t\t\t ipc_server_reply_cb *reply_cb,\n \t\t\t\t struct ipc_server_reply_data *reply_data)\n {\n@@ -123,6 +123,13 @@ static int app__sendbytes_command(const char *received,\n \tint errs = 0;\n \tint ret;\n \n+\t/*\n+\t * The test is setup to send:\n+\t * \"sendbytes\" SP \n+\t */\n+\tif (received_len < strlen(\"sendbytes \"))\n+\t\tBUG(\"received_len is short in app__sendbytes_command\");\n+\n \tif (skip_prefix(received, \"sendbytes \", &p))\n \t\tlen_ballast = strlen(p);\n \n@@ -160,7 +167,7 @@ static ipc_server_application_cb test_app_cb;\n * by this application.\n */\n static int test_app_cb(void *application_data,\n-\t\t const char *command,\n+\t\t const char *command, size_t command_len,\n \t\t ipc_server_reply_cb *reply_cb,\n \t\t struct ipc_server_reply_data *reply_data)\n {\n@@ -173,7 +180,7 @@ static int test_app_cb(void *application_data,\n \tif (application_data != (void*)&my_app_data)\n \t\tBUG(\"application_cb: application_data pointer wrong\");\n \n-\tif (!strcmp(command, \"quit\")) {\n+\tif (command_len == 4 && !strncmp(command, \"quit\", 4)) {\n \t\t/*\n \t\t * The client sent a \"quit\" command. This is an async\n \t\t * request for the server to shutdown.\n@@ -193,22 +200,23 @@ static int test_app_cb(void *application_data,\n \t\treturn SIMPLE_IPC_QUIT;\n \t}\n \n-\tif (!strcmp(command, \"ping\")) {\n+\tif (command_len == 4 && !strncmp(command, \"ping\", 4)) {\n \t\tconst char *answer = \"pong\";\n \t\treturn reply_cb(reply_data, answer, strlen(answer));\n \t}\n \n-\tif (!strcmp(command, \"big\"))\n+\tif (command_len == 3 && !strncmp(command, \"big\", 3))\n \t\treturn app__big_command(reply_cb, reply_data);\n \n-\tif (!strcmp(command, \"chunk\"))\n+\tif (command_len == 5 && !strncmp(command, \"chunk\", 5))\n \t\treturn app__chunk_command(reply_cb, reply_data);\n \n-\tif (!strcmp(command, \"slow\"))\n+\tif (command_len == 4 && !strncmp(command, \"slow\", 4))\n \t\treturn app__slow_command(reply_cb, reply_data);\n \n-\tif (starts_with(command, \"sendbytes \"))\n-\t\treturn app__sendbytes_command(command, reply_cb, reply_data);\n+\tif (command_len >= 10 && starts_with(command, \"sendbytes \"))\n+\t\treturn app__sendbytes_command(command, command_len,\n+\t\t\t\t\t reply_cb, reply_data);\n \n \treturn app__unhandled_command(command, reply_cb, reply_data);\n }\n@@ -488,7 +496,9 @@ static int client__send_ipc(void)\n \toptions.wait_if_busy = 1;\n \toptions.wait_if_not_found = 0;\n \n-\tif (!ipc_client_send_command(cl_args.path, &options, command, &buf)) {\n+\tif (!ipc_client_send_command(cl_args.path, &options,\n+\t\t\t\t command, strlen(command),\n+\t\t\t\t &buf)) {\n \t\tif (buf.len) {\n \t\t\tprintf(\"%s\\n\", buf.buf);\n \t\t\tfflush(stdout);\n@@ -556,7 +566,9 @@ static int do_sendbytes(int bytecount, char byte, const char *path,\n \tstrbuf_addstr(&buf_send, \"sendbytes \");\n \tstrbuf_addchars(&buf_send, byte, bytecount);\n \n-\tif (!ipc_client_send_command(path, options, buf_send.buf, &buf_resp)) {\n+\tif (!ipc_client_send_command(path, options,\n+\t\t\t\t buf_send.buf, buf_send.len,\n+\t\t\t\t &buf_resp)) {\n \t\tstrbuf_rtrim(&buf_resp);\n \t\tprintf(\"sent:%c%08d %s\\n\", byte, bytecount, buf_resp.buf);\n \t\tfflush(stdout);\n-- \ngitgitgadget\n\n"},{"id":"425304","messageId":"d5d09eb1635bc6a0781df6218f0d2537605c4666.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 05/28] help: include fsmonitor--daemon feature flag in version info","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:44Z","receivedAt":"2021-05-22T13:57:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the \"feature: fsmonitor--daemon\" message to the output of\n`git version --build-options`.\n\nThis allows users to know if the built-in fsmonitor feature is\nsupported on their platform.\n\nSigned-off-by: Jeff Hostetler \n---\n help.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/help.c b/help.c\nindex 3c3bdec21356..e22ba1d246a5 100644\n--- a/help.c\n+++ b/help.c\n@@ -11,6 +11,7 @@\n #include \"version.h\"\n #include \"refs.h\"\n #include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n \n struct category_description {\n \tuint32_t category;\n@@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n+\n+\t\tif (fsmonitor_ipc__is_supported())\n+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"425305","messageId":"d56f3e91db9f02eee60ae904a16176d98f656534.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 03/28] fsmonitor--daemon: update fsmonitor documentation","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:42Z","receivedAt":"2021-05-22T13:57:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to mention the new built-in `fsmonitor--daemon`.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/config/core.txt | 56 ++++++++++++++++++++++--------\n Documentation/git-update-index.txt | 27 +++++++-------\n Documentation/githooks.txt | 3 +-\n 3 files changed, 59 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a15..4f6e519bc025 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,50 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n+\n+core.useBuiltinFSMonitor::\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+Note: if this config setting is set to `true`, the values of\n+`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d976..c7c31b3fcf9c 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory\n looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n-the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n-across all repositories you use, because you can set the configuration\n-variable in your `$HOME/.gitconfig` just once and have it affect all\n-repositories you touch.\n-\n-When the `core.fsmonitor` configuration variable is changed, the\n-file system monitor is added to or removed from the index the next time\n-a command reads the index. When `--[no-]fsmonitor` are used, the file\n-system monitor is immediately added to or removed from the index.\n+the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable (see linkgit:git-config[1]) than using the `--fsmonitor`\n+option to `git update-index` in each repository, especially if you\n+want to do so across all repositories you use, because you can set the\n+configuration variable in your `$HOME/.gitconfig` just once and have\n+it affect all repositories you touch.\n+\n+When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable is changed, the file system monitor is added to or removed\n+from the index the next time a command reads the index. When\n+`--[no-]fsmonitor` are used, the file system monitor is immediately\n+added to or removed from the index.\n \n CONFIGURATION\n -------------\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex b51959ff9418..b7d5e926f7b0 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -593,7 +593,8 @@ fsmonitor-watchman\n \n This hook is invoked when the configuration option `core.fsmonitor` is\n set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n-depending on the version of the hook to use.\n+depending on the version of the hook to use, unless overridden via\n+`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n \n Version 1 takes two arguments, a version (1) and the time in elapsed\n nanoseconds since midnight, January 1, 1970.\n-- \ngitgitgadget\n\n"},{"id":"425306","messageId":"fc180e8591bfaa41d31f8311c1fecb5f1c1be936.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 02/28] fsmonitor--daemon: man page","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:41Z","receivedAt":"2021-05-22T13:57:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a manual page describing the `git fsmonitor--daemon` feature.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n 1 file changed, 75 insertions(+)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n\ndiff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\nnew file mode 100644\nindex 000000000000..154e7684daae\n--- /dev/null\n+++ b/Documentation/git-fsmonitor--daemon.txt\n@@ -0,0 +1,75 @@\n+git-fsmonitor--daemon(1)\n+========================\n+\n+NAME\n+----\n+git-fsmonitor--daemon - A Built-in File System Monitor\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git fsmonitor--daemon' start\n+'git fsmonitor--daemon' run\n+'git fsmonitor--daemon' stop\n+'git fsmonitor--daemon' status\n+\n+DESCRIPTION\n+-----------\n+\n+A daemon to watch the working directory for file and directory\n+changes using platform-specific file system notification facilities.\n+\n+This daemon communicates directly with commands like `git status`\n+using the link:technical/api-simple-ipc.html[simple IPC] interface\n+instead of the slower linkgit:githooks[5] interface.\n+\n+This daemon is built into Git so that no third-party tools are\n+required.\n+\n+OPTIONS\n+-------\n+\n+start::\n+\tStarts a daemon in the background.\n+\n+run::\n+\tRuns a daemon in the foreground.\n+\n+stop::\n+\tStops the daemon running in the current working\n+\tdirectory, if present.\n+\n+status::\n+\tExits with zero status if a daemon is watching the\n+\tcurrent working directory.\n+\n+REMARKS\n+-------\n+\n+This daemon is a long running process used to watch a single working\n+directory and maintain a list of the recently changed files and\n+directories. Performance of commands such as `git status` can be\n+increased if they just ask for a summary of changes to the working\n+directory and can avoid scanning the disk.\n+\n+When `core.useBuiltinFSMonitor` is set to `true` (see\n+linkgit:git-config[1]) commands, such as `git status`, will ask the\n+daemon for changes and automatically start it (if necessary).\n+\n+For more information see the \"File System Monitor\" section in\n+linkgit:git-update-index[1].\n+\n+CAVEATS\n+-------\n+\n+The fsmonitor daemon does not currently know about submodules and does\n+not know to filter out file system events that happen within a\n+submodule. If fsmonitor daemon is watching a super repo and a file is\n+modified within the working directory of a submodule, it will report\n+the change (as happening against the super repo). However, the client\n+will properly ignore these extra events, so performance may be affected\n+but it will not cause an incorrect result.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\n-- \ngitgitgadget\n\n"},{"id":"425307","messageId":"e4a263728773381a64b8662c0577a3f12eff4ca0.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 04/28] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:43Z","receivedAt":"2021-05-22T13:57:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 179 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 +++++++++++++\n 3 files changed, 228 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex 21c0bf16672b..23f3b9890acd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -892,6 +892,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 000000000000..e62901a85b5d\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,179 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\",\n+\t\t\t since_token);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, since_token, strlen(since_token), answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\n+\t\tif (fsmonitor_is_trivial_response(answer))\n+\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: daemon not available\"));\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(\"fsmonitor--daemon is not running\");\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection,\n+\t\t\t\t\t\t command, strlen(command),\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\",\n+\t\t command);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 000000000000..837c5e5b64ad\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * This DOES NOT use the hook interface.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"425308","messageId":"67bcf57f5948038a86c7aa99a6f56655a33eb045.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 06/28] config: FSMonitor is repository-specific","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:45Z","receivedAt":"2021-05-22T13:57:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis commit refactors `git_config_get_fsmonitor()` into the `repo_*()`\nform that takes a parameter `struct repository *r`.\n\nThat change prepares for the upcoming `core.useBuiltinFSMonitor` flag which\nwill be stored in the `repo_settings` struct.\n\nSigned-off-by: Johannes Schindelin \n---\n builtin/update-index.c | 4 ++--\n config.c | 4 ++--\n config.h | 2 +-\n fsmonitor.c | 2 +-\n 4 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 79087bccea4b..84793df8b2b6 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1214,14 +1214,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tif (repo_config_get_fsmonitor(r) == 0)\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\n \t\t\t\t\"enable fsmonitor\"));\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tif (repo_config_get_fsmonitor(r) == 1)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/config.c b/config.c\nindex 870d9534defc..a896f44cba1f 100644\n--- a/config.c\n+++ b/config.c\n@@ -2499,9 +2499,9 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n+int repo_config_get_fsmonitor(struct repository *r)\n {\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n+\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &core_fsmonitor))\n \t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n \n \tif (core_fsmonitor && !*core_fsmonitor)\ndiff --git a/config.h b/config.h\nindex 19a9adbaa9a3..3139de81d986 100644\n--- a/config.h\n+++ b/config.h\n@@ -607,7 +607,7 @@ int git_config_get_index_threads(int *dest);\n int git_config_get_untracked_cache(void);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n+int repo_config_get_fsmonitor(struct repository *r);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34e..9c9b2abc9414 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -411,7 +411,7 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tint fsmonitor_enabled = repo_config_get_fsmonitor(istate->repo ? istate->repo : the_repository);\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n-- \ngitgitgadget\n\n"},{"id":"425309","messageId":"f362a88632e47267f67f5d805b899485eda46eb6.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 08/28] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:47Z","receivedAt":"2021-05-22T13:57:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a built-in file system monitoring daemon that can be used by\nthe existing `fsmonitor` feature (protocol API and index extension)\nto improve the performance of various Git commands, such as `status`.\n\nThe `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\nprovides an alternative to hook access to existing fsmonitors such\nas `watchman`.\n\nThis commit merely adds the new command without any functionality.\n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 53 +++++++++++++++++++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 57 insertions(+)\n create mode 100644 builtin/fsmonitor--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 3dcdb6bb5ab8..beccf34abe9e 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -71,6 +71,7 @@\n /git-format-patch\n /git-fsck\n /git-fsck-objects\n+/git-fsmonitor--daemon\n /git-gc\n /git-get-tar-commit-id\n /git-grep\ndiff --git a/Makefile b/Makefile\nindex 23f3b9890acd..74673acc9833 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1092,6 +1092,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n+BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex b6ce981b7377..7554476f90a4 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -158,6 +158,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\n int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nnew file mode 100644\nindex 000000000000..df2bad531118\n--- /dev/null\n+++ b/builtin/fsmonitor--daemon.c\n@@ -0,0 +1,53 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"simple-ipc.h\"\n+#include \"khash.h\"\n+\n+static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tNULL\n+};\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *subcmd;\n+\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc < 2)\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tsubcmd = argv[1];\n+\targv--;\n+\targc++;\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\n+\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n+}\n+\n+#else\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n+}\n+#endif\ndiff --git a/git.c b/git.c\nindex b53e66567138..41980c897964 100644\n--- a/git.c\n+++ b/git.c\n@@ -523,6 +523,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"425314","messageId":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.git.1617291666.gitgitgadget@gmail.com","subject":"[PATCH v2 00/28] Builtin FSMonitor Feature","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:39Z","receivedAt":"2021-05-22T13:57:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V2 of my patch series to add a builtin FSMonitor daemon to Git.\n\nV2 includes addresses most of the review comments from the RFC and many of\nthe issues seen during out beta-testing with selected GVFS users. However,\nthere are still a few items that I need to address:\n\n[ ] Revisit the how the client handles the IPC_STATE__NOT_LISTENING state\n(where a daemon appears to be running, but is non-responsive) [ ] Revisit\nuse of global core_fsmonitor as both a pathname and a boolean. The existing\nfsmonitor code uses it as the pathname to the fsmonitor hook and as a flag\nto indicate that a hook is configured. [ ] Consider having daemon chdir()\nout of the working directory to avoid directory handle issues on Windows. [\n] Some documentation recommendations. [ ] Split up the commit containing the\ntests and move some earlier in patch series. [ ] Move my FSMonitor PREREQ to\ntest-lib.sh instead of having it in my test scripts. [ ] Document\nperformance gains. [ ] On Windows, If the daemon is started as an elevated\nprocess, then client commands might not have access to communicate with it.\n[ ] Review if/how we decide to shutdown the FSMonitor daemon after and a\nsignificant idle period. [ ] Investigate ways to temporarily shutdown\nFSMonitor daemons processes so that the Git for Windows installer can\ninstall an upgrade.\n\nIn this version, the first commit updates the Simple IPC API to make it\neasier to pass binary data using {char *, size_t} rather than assuming that\nthe message is a null-terminated string. FSMonitor does not use binary\nmessages and doesn't really need this API change, but I thought it best to\nfix the API now before we have other callers of IPC.\n\nThis V2 version will hopefully be previewed as an experimental feature in\nGit for Windows v2.32.0.windows.*.\n\nJeff Hostetler (26):\n simple-ipc: preparations for supporting binary messages.\n fsmonitor--daemon: man page\n fsmonitor--daemon: update fsmonitor documentation\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n help: include fsmonitor--daemon feature flag in version info\n fsmonitor--daemon: add a built-in fsmonitor daemon\n fsmonitor--daemon: implement client command options\n t/helper/fsmonitor-client: create IPC client to talk to FSMonitor\n Daemon\n fsmonitor-fs-listen-win32: stub in backend for Windows\n fsmonitor-fs-listen-macos: stub in backend for MacOS\n fsmonitor--daemon: implement daemon command options\n fsmonitor--daemon: add pathname classification\n fsmonitor--daemon: define token-ids\n fsmonitor--daemon: create token-based changed path cache\n fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows\n fsmonitor-fs-listen-macos: add macos header files for FSEvent\n fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS\n fsmonitor--daemon: implement handle_client callback\n fsmonitor--daemon: periodically truncate list of modified files\n fsmonitor--daemon: use a cookie file to sync with file system\n fsmonitor: enhance existing comments\n fsmonitor: force update index after large responses\n t7527: create test for fsmonitor--daemon\n p7519: add fsmonitor--daemon\n t7527: test status with untracked-cache and fsmonitor--daemon\n t/perf: avoid copying builtin fsmonitor files into test repo\n\nJohannes Schindelin (2):\n config: FSMonitor is repository-specific\n fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via\n IPC\n\n .gitignore | 1 +\n Documentation/config/core.txt | 56 +-\n Documentation/git-fsmonitor--daemon.txt | 75 +\n Documentation/git-update-index.txt | 27 +-\n Documentation/githooks.txt | 3 +-\n Makefile | 16 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 1511 ++++++++++++++++++\n builtin/update-index.c | 4 +-\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 497 ++++++\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 553 +++++++\n compat/fsmonitor/fsmonitor-fs-listen.h | 49 +\n compat/simple-ipc/ipc-unix-socket.c | 14 +-\n compat/simple-ipc/ipc-win32.c | 14 +-\n config.c | 9 +-\n config.h | 2 +-\n config.mak.uname | 4 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 140 ++\n fsmonitor-ipc.c | 179 +++\n fsmonitor-ipc.h | 48 +\n fsmonitor.c | 132 +-\n git.c | 1 +\n help.c | 4 +\n repo-settings.c | 3 +\n repository.h | 2 +\n simple-ipc.h | 7 +-\n t/helper/test-fsmonitor-client.c | 125 ++\n t/helper/test-simple-ipc.c | 34 +-\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/perf/p7519-fsmonitor.sh | 42 +-\n t/perf/perf-lib.sh | 2 +-\n t/t7527-builtin-fsmonitor.sh | 572 +++++++\n 34 files changed, 4069 insertions(+), 68 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n create mode 100644 builtin/fsmonitor--daemon.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n create mode 100644 fsmonitor--daemon.h\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 t/helper/test-fsmonitor-client.c\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\n\nbase-commit: b0c09ab8796fb736efa432b8e817334f3e5ee75a\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-923%2Fjeffhostetler%2Fbuiltin-fsmonitor-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-923/jeffhostetler/builtin-fsmonitor-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/923\n\nRange-diff vs v1:\n\n -: ------------ > 1: 763fa1ee7bb6 simple-ipc: preparations for supporting binary messages.\n -: ------------ > 2: fc180e8591bf fsmonitor--daemon: man page\n 1: 074273330f8d ! 3: d56f3e91db9f fsmonitor--daemon: man page and documentation\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor--daemon: man page and documentation\n + fsmonitor--daemon: update fsmonitor documentation\n \n - Create a manual page describing the `git fsmonitor--daemon` feature.\n -\n - Update references to `core.fsmonitor`, `core.fsmonitorHookVersion` and\n - pointers to `watchman` to mention the built-in FSMonitor.\n + Update references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\n + pointers to `Watchman` to mention the new built-in `fsmonitor--daemon`.\n \n Signed-off-by: Jeff Hostetler \n \n ## Documentation/config/core.txt ##\n -@@ Documentation/config/core.txt: core.fsmonitor::\n - \twill identify all files that may have changed since the\n - \trequested date/time. This information is used to speed up git by\n - \tavoiding unnecessary processing of files that have not changed.\n +@@ Documentation/config/core.txt: core.protectNTFS::\n + \tDefaults to `true` on Windows, and `false` elsewhere.\n + \n + core.fsmonitor::\n +-\tIf set, the value of this variable is used as a command which\n +-\twill identify all files that may have changed since the\n +-\trequested date/time. This information is used to speed up git by\n +-\tavoiding unnecessary processing of files that have not changed.\n -\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n ++\tIf set, this variable contains the pathname of the \"fsmonitor\"\n ++\thook command.\n +++\n ++This hook command is used to identify all files that may have changed\n ++since the requested date/time. This information is used to speed up\n ++git by avoiding unnecessary scanning of files that have not changed.\n ++\n +See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n ++\n -+Note: FSMonitor hooks (and this config setting) are ignored if the\n -+built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`).\n ++Note: The value of this config setting is ignored if the\n ++built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n \n core.fsmonitorHookVersion::\n -\tSets the version of hook that is to be used when calling fsmonitor.\n @@ Documentation/config/core.txt: core.fsmonitor::\n -\tVersion 2 uses an opaque string so that the monitor can return\n -\tsomething that can be used to determine what files have changed\n -\twithout race conditions.\n -+\tSets the version of hook that is to be used when calling the\n -+\tFSMonitor hook (as configured via `core.fsmonitor`).\n ++\tSets the protocol version to be used when invoking the\n ++\t\"fsmonitor\" hook.\n ++\n +There are currently versions 1 and 2. When this is not set,\n +version 2 will be tried first and if it fails then version 1\n +will be tried. Version 1 uses a timestamp as input to determine\n +which files have changes since that time but some monitors\n -+like watchman have race conditions when used with a timestamp.\n ++like Watchman have race conditions when used with a timestamp.\n +Version 2 uses an opaque string so that the monitor can return\n +something that can be used to determine what files have changed\n +without race conditions.\n ++\n -+Note: FSMonitor hooks (and this config setting) are ignored if the\n -+built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`).\n ++Note: The value of this config setting is ignored if the\n ++built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n +\n +core.useBuiltinFSMonitor::\n -+\tIf set to true, enable the built-in filesystem event watcher (for\n -+\ttechnical details, see linkgit:git-fsmonitor--daemon[1]).\n ++\tIf set to true, enable the built-in file system monitor\n ++\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n ++\n -+Like external (hook-based) FSMonitors, the built-in FSMonitor can speed up\n -+Git commands that need to refresh the Git index (e.g. `git status`) in a\n -+worktree with many files. The built-in FSMonitor facility eliminates the\n -+need to install and maintain an external third-party monitoring tool.\n ++Like hook-based file system monitors, the built-in file system monitor\n ++can speed up Git commands that need to refresh the Git index\n ++(e.g. `git status`) in a working directory with many files. The\n ++built-in monitor eliminates the need to install and maintain an\n ++external third-party tool.\n ++\n -+The built-in FSMonitor is currently available only on a limited set of\n -+supported platforms.\n ++The built-in file system monitor is currently available only on a\n ++limited set of supported platforms. Currently, this includes Windows\n ++and MacOS.\n ++\n -+Note: if this config setting is set to `true`, any FSMonitor hook\n -+configured via `core.fsmonitor` (and possibly `core.fsmonitorHookVersion`)\n -+is ignored.\n ++Note: if this config setting is set to `true`, the values of\n ++`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \n - ## Documentation/git-fsmonitor--daemon.txt (new) ##\n -@@\n -+git-fsmonitor--daemon(1)\n -+========================\n -+\n -+NAME\n -+----\n -+git-fsmonitor--daemon - Builtin file system monitor daemon\n -+\n -+SYNOPSIS\n -+--------\n -+[verse]\n -+'git fsmonitor--daemon' --start\n -+'git fsmonitor--daemon' --run\n -+'git fsmonitor--daemon' --stop\n -+'git fsmonitor--daemon' --is-running\n -+'git fsmonitor--daemon' --is-supported\n -+'git fsmonitor--daemon' --query \n -+'git fsmonitor--daemon' --query-index\n -+'git fsmonitor--daemon' --flush\n -+\n -+DESCRIPTION\n -+-----------\n -+\n -+Monitors files and directories in the working directory for changes using\n -+platform-specific file system notification facilities.\n -+\n -+It communicates directly with commands like `git status` using the\n -+link:technical/api-simple-ipc.html[simple IPC] interface instead of\n -+the slower linkgit:githooks[5] interface.\n -+\n -+OPTIONS\n -+-------\n -+\n -+--start::\n -+\tStarts the fsmonitor daemon in the background.\n -+\n -+--run::\n -+\tRuns the fsmonitor daemon in the foreground.\n -+\n -+--stop::\n -+\tStops the fsmonitor daemon running for the current working\n -+\tdirectory, if present.\n -+\n -+--is-running::\n -+\tExits with zero status if the fsmonitor daemon is watching the\n -+\tcurrent working directory.\n -+\n -+--is-supported::\n -+\tExits with zero status if the fsmonitor daemon feature is supported\n -+\ton this platform.\n -+\n -+--query ::\n -+\tConnects to the fsmonitor daemon (starting it if necessary) and\n -+\trequests the list of changed files and directories since the\n -+\tgiven token.\n -+\tThis is intended for testing purposes.\n -+\n -+--query-index::\n -+\tRead the current `` from the File System Monitor index\n -+\textension (if present) and use it to query the fsmonitor daemon.\n -+\tThis is intended for testing purposes.\n -+\n -+--flush::\n -+\tForce the fsmonitor daemon to flush its in-memory cache and\n -+\tre-sync with the file system.\n -+\tThis is intended for testing purposes.\n -+\n -+REMARKS\n -+-------\n -+The fsmonitor daemon is a long running process that will watch a single\n -+working directory. Commands, such as `git status`, should automatically\n -+start it (if necessary) when `core.useBuiltinFSMonitor` is set to `true`\n -+(see linkgit:git-config[1]).\n -+\n -+Configure the built-in FSMonitor via `core.useBuiltinFSMonitor` in each\n -+working directory separately, or globally via `git config --global\n -+core.useBuiltinFSMonitor true`.\n -+\n -+Tokens are opaque strings. They are used by the fsmonitor daemon to\n -+mark a point in time and the associated internal state. Callers should\n -+make no assumptions about the content of the token. In particular,\n -+the should not assume that it is a timestamp.\n -+\n -+Query commands send a request-token to the daemon and it responds with\n -+a summary of the changes that have occurred since that token was\n -+created. The daemon also returns a response-token that the client can\n -+use in a future query.\n -+\n -+For more information see the \"File System Monitor\" section in\n -+linkgit:git-update-index[1].\n -+\n -+CAVEATS\n -+-------\n -+\n -+The fsmonitor daemon does not currently know about submodules and does\n -+not know to filter out file system events that happen within a\n -+submodule. If fsmonitor daemon is watching a super repo and a file is\n -+modified within the working directory of a submodule, it will report\n -+the change (as happening against the super repo). However, the client\n -+should properly ignore these extra events, so performance may be affected\n -+but it should not cause an incorrect result.\n -+\n -+GIT\n -+---\n -+Part of the linkgit:git[1] suite\n -\n ## Documentation/git-update-index.txt ##\n @@ Documentation/git-update-index.txt: FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n @@ Documentation/git-update-index.txt: FILE SYSTEM MONITOR\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n +@@ Documentation/git-update-index.txt: performance by avoiding the cost of scanning the entire working directory\n + looking for new files.\n + \n + If you want to enable (or disable) this feature, it is easier to use\n +-the `core.fsmonitor` configuration variable (see\n +-linkgit:git-config[1]) than using the `--fsmonitor` option to\n +-`git update-index` in each repository, especially if you want to do so\n +-across all repositories you use, because you can set the configuration\n +-variable in your `$HOME/.gitconfig` just once and have it affect all\n +-repositories you touch.\n +-\n +-When the `core.fsmonitor` configuration variable is changed, the\n +-file system monitor is added to or removed from the index the next time\n +-a command reads the index. When `--[no-]fsmonitor` are used, the file\n +-system monitor is immediately added to or removed from the index.\n ++the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n ++variable (see linkgit:git-config[1]) than using the `--fsmonitor`\n ++option to `git update-index` in each repository, especially if you\n ++want to do so across all repositories you use, because you can set the\n ++configuration variable in your `$HOME/.gitconfig` just once and have\n ++it affect all repositories you touch.\n ++\n ++When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n ++variable is changed, the file system monitor is added to or removed\n ++from the index the next time a command reads the index. When\n ++`--[no-]fsmonitor` are used, the file system monitor is immediately\n ++added to or removed from the index.\n + \n + CONFIGURATION\n + -------------\n \n ## Documentation/githooks.txt ##\n @@ Documentation/githooks.txt: fsmonitor-watchman\n 2: 3dac63eae201 ! 4: e4a263728773 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n @@ Metadata\n ## Commit message ##\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n \n - Create client routines to spawn a fsmonitor daemon and send it an IPC\n - request using `simple-ipc`.\n + Create fsmonitor_ipc__*() client routines to spawn the built-in file\n + system monitor daemon and send it an IPC request using the `Simple\n + IPC` API.\n +\n + Stub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n \n Signed-off-by: Jeff Hostetler \n \n @@ fsmonitor-ipc.c (new)\n @@\n +#include \"cache.h\"\n +#include \"fsmonitor.h\"\n ++#include \"simple-ipc.h\"\n +#include \"fsmonitor-ipc.h\"\n +#include \"run-command.h\"\n +#include \"strbuf.h\"\n +#include \"trace2.h\"\n +\n +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n -+#define FSMONITOR_DAEMON_IS_SUPPORTED 1\n -+#else\n -+#define FSMONITOR_DAEMON_IS_SUPPORTED 0\n -+#endif\n +\n -+/*\n -+ * A trivial function so that this source file always defines at least\n -+ * one symbol even when the feature is not supported. This quiets an\n -+ * annoying compiler error.\n -+ */\n +int fsmonitor_ipc__is_supported(void)\n +{\n -+\treturn FSMONITOR_DAEMON_IS_SUPPORTED;\n ++\treturn 1;\n +}\n +\n -+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n -+\n -+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor\")\n ++GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n +\n +enum ipc_active_state fsmonitor_ipc__get_state(void)\n +{\n @@ fsmonitor-ipc.c (new)\n +\n +static int spawn_daemon(void)\n +{\n -+\tconst char *args[] = { \"fsmonitor--daemon\", \"--start\", NULL };\n ++\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n +\n +\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n +\t\t\t\t \"fsmonitor\");\n @@ fsmonitor-ipc.c (new)\n +\tswitch (state) {\n +\tcase IPC_STATE__LISTENING:\n +\t\tret = ipc_client_send_command_to_connection(\n -+\t\t\tconnection, since_token, answer);\n ++\t\t\tconnection, since_token, strlen(since_token), answer);\n +\t\tipc_client_close_connection(connection);\n +\n +\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n @@ fsmonitor-ipc.c (new)\n +\t\treturn -1;\n +\t}\n +\n -+\tret = ipc_client_send_command_to_connection(connection, command, answer);\n ++\tret = ipc_client_send_command_to_connection(connection,\n ++\t\t\t\t\t\t command, strlen(command),\n ++\t\t\t\t\t\t answer);\n +\tipc_client_close_connection(connection);\n +\n +\tif (ret == -1) {\n @@ fsmonitor-ipc.c (new)\n +\treturn 0;\n +}\n +\n ++#else\n ++\n ++/*\n ++ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n ++ * platforms.\n ++ */\n ++\n ++int fsmonitor_ipc__is_supported(void)\n ++{\n ++\treturn 0;\n ++}\n ++\n ++const char *fsmonitor_ipc__get_path(void)\n ++{\n ++\treturn NULL;\n ++}\n ++\n ++enum ipc_active_state fsmonitor_ipc__get_state(void)\n ++{\n ++\treturn IPC_STATE__OTHER_ERROR;\n ++}\n ++\n ++int fsmonitor_ipc__send_query(const char *since_token,\n ++\t\t\t struct strbuf *answer)\n ++{\n ++\treturn -1;\n ++}\n ++\n ++int fsmonitor_ipc__send_command(const char *command,\n ++\t\t\t\tstruct strbuf *answer)\n ++{\n ++\treturn -1;\n ++}\n ++\n +#endif\n \n ## fsmonitor-ipc.h (new) ##\n @@ fsmonitor-ipc.h (new)\n +#define FSMONITOR_IPC_H\n +\n +/*\n -+ * Returns true if a filesystem notification backend is defined\n -+ * for this platform. This symbol must always be visible and\n -+ * outside of the HAVE_ ifdef.\n ++ * Returns true if built-in file system monitor daemon is defined\n ++ * for this platform.\n + */\n +int fsmonitor_ipc__is_supported(void);\n +\n -+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n -+#include \"run-command.h\"\n -+#include \"simple-ipc.h\"\n -+\n +/*\n + * Returns the pathname to the IPC named pipe or Unix domain socket\n + * where a `git-fsmonitor--daemon` process will listen. This is a\n + * per-worktree value.\n ++ *\n ++ * Returns NULL if the daemon is not supported on this platform.\n + */\n +const char *fsmonitor_ipc__get_path(void);\n +\n @@ fsmonitor-ipc.h (new)\n + * This DOES NOT use the hook interface.\n + *\n + * Spawn a daemon process in the background if necessary.\n ++ *\n ++ * Returns -1 on error; 0 on success.\n + */\n +int fsmonitor_ipc__send_query(const char *since_token,\n +\t\t\t struct strbuf *answer);\n @@ fsmonitor-ipc.h (new)\n + * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n + * send a command verb. If no daemon is available, we DO NOT try to\n + * start one.\n ++ *\n ++ * Returns -1 on error; 0 on success.\n + */\n +int fsmonitor_ipc__send_command(const char *command,\n +\t\t\t\tstruct strbuf *answer);\n +\n -+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n +#endif /* FSMONITOR_IPC_H */\n -\n - ## help.c ##\n -@@\n - #include \"version.h\"\n - #include \"refs.h\"\n - #include \"parse-options.h\"\n -+#include \"fsmonitor-ipc.h\"\n - \n - struct category_description {\n - \tuint32_t category;\n -@@ help.c: void get_version_info(struct strbuf *buf, int show_build_options)\n - \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n - \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n - \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n -+\n -+\t\tif (fsmonitor_ipc__is_supported())\n -+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n - \t}\n - }\n - \n -: ------------ > 5: d5d09eb1635b help: include fsmonitor--daemon feature flag in version info\n 3: 18c125ec73dc ! 6: 67bcf57f5948 config: FSMonitor is repository-specific\n @@ Commit message\n This commit refactors `git_config_get_fsmonitor()` into the `repo_*()`\n form that takes a parameter `struct repository *r`.\n \n - That change prepares for the upcoming `core.useFSMonitorDaemon` flag which\n + That change prepares for the upcoming `core.useBuiltinFSMonitor` flag which\n will be stored in the `repo_settings` struct.\n \n Signed-off-by: Johannes Schindelin \n 4: 7082528d8f7c ! 7: 7e097cebc143 fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC\n @@ Metadata\n ## Commit message ##\n fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC\n \n - The `core.fsmonitor` setting is supposed to be set to a path pointing\n - to a script or executable that (via the Hook API) queries an fsmonitor\n - process such as watchman.\n + Use simple IPC to directly communicate with the new builtin file\n + system monitor daemon.\n \n - We are about to implement our own fsmonitor backend, and do not want\n - to spawn hook processes just to query it. Let's use `Simple IPC` to\n - directly communicate with the daemon (and start it if necessary),\n - guarded by the brand-new `core.useBuiltinFSMonitor` toggle.\n + Define a new config setting `core.useBuiltinFSMonitor` to enable the\n + builtin file system monitor.\n +\n + The `core.fsmonitor` setting has already been defined as a HOOK\n + pathname. Historically, this has been set to a HOOK script that will\n + talk with Watchman. For compatibility reasons, we do not want to\n + overload that definition (and cause problems if users have multiple\n + versions of Git installed).\n \n Signed-off-by: Johannes Schindelin \n Signed-off-by: Jeff Hostetler \n @@ fsmonitor.c\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n -@@ fsmonitor.c: void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n - /*\n - * Call the query-fsmonitor hook passing the last update token of the saved results.\n - */\n --static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n -+static int query_fsmonitor(int version, struct index_state *istate, struct strbuf *query_result)\n +@@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n + \n + void refresh_fsmonitor(struct index_state *istate)\n {\n +\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n -+\tconst char *last_update = istate->fsmonitor_last_update;\n - \tstruct child_process cp = CHILD_PROCESS_INIT;\n - \tint result;\n - \n - \tif (!core_fsmonitor)\n - \t\treturn -1;\n + \tstruct strbuf query_result = STRBUF_INIT;\n + \tint query_success = 0, hook_version = -1;\n + \tsize_t bol = 0; /* beginning of line */\n +@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \tistate->fsmonitor_has_run_once = 1;\n \n + \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n ++\n +\tif (r->settings.use_builtin_fsmonitor > 0) {\n -+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n -+\t\treturn fsmonitor_ipc__send_query(last_update, query_result);\n -+#else\n -+\t\t/* Fake a trivial response. */\n -+\t\twarning(_(\"fsmonitor--daemon unavailable; falling back\"));\n -+\t\tstrbuf_add(query_result, \"/\", 2);\n -+\t\treturn 0;\n -+#endif\n ++\t\tquery_success = !fsmonitor_ipc__send_query(\n ++\t\t\tistate->fsmonitor_last_update, &query_result);\n ++\t\tif (query_success) {\n ++\t\t\t/*\n ++\t\t\t * The response contains a series of nul terminated\n ++\t\t\t * strings. The first is the new token.\n ++\t\t\t *\n ++\t\t\t * Use `char *buf` as an interlude to trick the CI\n ++\t\t\t * static analysis to let us use `strbuf_addstr()`\n ++\t\t\t * here (and only copy the token) rather than\n ++\t\t\t * `strbuf_addbuf()`.\n ++\t\t\t */\n ++\t\t\tbuf = query_result.buf;\n ++\t\t\tstrbuf_addstr(&last_update_token, buf);\n ++\t\t\tbol = last_update_token.len + 1;\n ++\t\t} else {\n ++\t\t\t/*\n ++\t\t\t * The builtin daemon is not available on this\n ++\t\t\t * platform -OR- we failed to get a response.\n ++\t\t\t *\n ++\t\t\t * Generate a fake token (rather than a V1\n ++\t\t\t * timestamp) for the index extension. (If\n ++\t\t\t * they switch back to the hook API, we don't\n ++\t\t\t * want ambiguous state.)\n ++\t\t\t */\n ++\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n ++\t\t}\n ++\n ++\t\t/*\n ++\t\t * Regardless of whether we successfully talked to a\n ++\t\t * fsmonitor daemon or not, we skip over and do not\n ++\t\t * try to use the hook. The \"core.useBuiltinFSMonitor\"\n ++\t\t * config setting ALWAYS overrides the \"core.fsmonitor\"\n ++\t\t * hook setting.\n ++\t\t */\n ++\t\tgoto apply_results;\n +\t}\n +\n - \tstrvec_push(&cp.args, core_fsmonitor);\n - \tstrvec_pushf(&cp.args, \"%d\", version);\n - \tstrvec_pushf(&cp.args, \"%s\", last_update);\n + \t/*\n + \t * This could be racy so save the date/time now and query_fsmonitor\n + \t * should be inclusive to ensure we don't miss potential changes.\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n - \tif (istate->fsmonitor_last_update) {\n - \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n - \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n --\t\t\t\tistate->fsmonitor_last_update, &query_result);\n -+\t\t\t\tistate, &query_result);\n + \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n + \t}\n \n - \t\t\tif (query_success) {\n - \t\t\t\tif (hook_version < 0)\n -@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n - \n - \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n - \t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n --\t\t\t\tistate->fsmonitor_last_update, &query_result);\n -+\t\t\t\tistate, &query_result);\n - \t\t}\n - \n - \t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n ++apply_results:\n + \t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n + \tif (query_success && query_result.buf[bol] != '/') {\n + \t\t/* Mark all entries returned by the monitor as dirty */\n \n ## repo-settings.c ##\n @@ repo-settings.c: void prepare_repo_settings(struct repository *r)\n @@ repo-settings.c: void prepare_repo_settings(struct repository *r)\n \t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);\n \n ## repository.h ##\n -@@ repository.h: struct repo_settings {\n - \tenum fetch_negotiation_setting fetch_negotiation_algorithm;\n +@@ repository.h: enum fetch_negotiation_setting {\n + struct repo_settings {\n + \tint initialized;\n \n - \tint core_multi_pack_index;\n -+\n +\tint use_builtin_fsmonitor;\n - };\n - \n - struct repository {\n ++\n + \tint core_commit_graph;\n + \tint commit_graph_read_changed_paths;\n + \tint gc_write_commit_graph;\n 5: 95d511d83b12 ! 8: f362a88632e4 fsmonitor--daemon: add a built-in fsmonitor daemon\n @@ builtin/fsmonitor--daemon.c (new)\n +\n +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n +{\n -+\tenum daemon_mode {\n -+\t\tUNDEFINED_MODE,\n -+\t} mode = UNDEFINED_MODE;\n ++\tconst char *subcmd;\n +\n +\tstruct option options[] = {\n +\t\tOPT_END()\n +\t};\n +\n ++\tif (argc < 2)\n ++\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n ++\n +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n +\n +\tgit_config(git_default_config, NULL);\n +\n ++\tsubcmd = argv[1];\n ++\targv--;\n ++\targc++;\n ++\n +\targc = parse_options(argc, argv, prefix, options,\n +\t\t\t builtin_fsmonitor__daemon_usage, 0);\n +\n -+\tswitch (mode) {\n -+\tcase UNDEFINED_MODE:\n -+\tdefault:\n -+\t\tdie(_(\"Unhandled command mode %d\"), mode);\n -+\t}\n ++\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n +}\n +\n +#else\n 6: 77170e521f67 ! 9: 4f401310539e fsmonitor--daemon: implement client command options\n @@ Metadata\n ## Commit message ##\n fsmonitor--daemon: implement client command options\n \n - Implement command options `--stop`, `--is-running`, `--query`,\n - `--query-index`, and `--flush` to control and query the status of a\n - `fsmonitor--daemon` server process (and implicitly start a server\n - process if necessary).\n + Implement `stop` and `status` client commands to control and query the\n + status of a `fsmonitor--daemon` server process (and implicitly start a\n + server process if necessary).\n \n - Later commits will implement the actual server and monitor\n - the file system.\n + Later commits will implement the actual server and monitor the file\n + system.\n \n Signed-off-by: Jeff Hostetler \n \n @@ builtin/fsmonitor--daemon.c\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n -+\tN_(\"git fsmonitor--daemon --stop\"),\n -+\tN_(\"git fsmonitor--daemon --is-running\"),\n -+\tN_(\"git fsmonitor--daemon --query \"),\n -+\tN_(\"git fsmonitor--daemon --query-index\"),\n -+\tN_(\"git fsmonitor--daemon --flush\"),\n ++\tN_(\"git fsmonitor--daemon stop\"),\n ++\tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n @@ builtin/fsmonitor--daemon.c\n +/*\n + * Acting as a CLIENT.\n + *\n -+ * Send an IPC query to a `git-fsmonitor--daemon` SERVER process and\n -+ * ask for the changes since the given token. This will implicitly\n -+ * start a daemon process if necessary. The daemon process will\n -+ * persist after we exit.\n -+ *\n -+ * This feature is primarily used by the test suite.\n -+ */\n -+static int do_as_client__query_token(const char *token)\n -+{\n -+\tstruct strbuf answer = STRBUF_INIT;\n -+\tint ret;\n -+\n -+\tret = fsmonitor_ipc__send_query(token, &answer);\n -+\tif (ret < 0)\n -+\t\tdie(_(\"could not query fsmonitor--daemon\"));\n -+\n -+\twrite_in_full(1, answer.buf, answer.len);\n -+\tstrbuf_release(&answer);\n -+\n -+\treturn 0;\n -+}\n -+\n -+/*\n -+ * Acting as a CLIENT.\n -+ *\n -+ * Read the `.git/index` to get the last token written to the FSMonitor index\n -+ * extension and use that to make a query.\n -+ *\n -+ * This feature is primarily used by the test suite.\n -+ */\n -+static int do_as_client__query_from_index(void)\n -+{\n -+\tstruct index_state *istate = the_repository->index;\n -+\n -+\tsetup_git_directory();\n -+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n -+\t\tdie(\"unable to read index file\");\n -+\tif (!istate->fsmonitor_last_update)\n -+\t\tdie(\"index file does not have fsmonitor extension\");\n -+\n -+\treturn do_as_client__query_token(istate->fsmonitor_last_update);\n -+}\n -+\n -+/*\n -+ * Acting as a CLIENT.\n -+ *\n + * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n + * and wait for it to shutdown.\n + */\n @@ builtin/fsmonitor--daemon.c\n +\treturn 0;\n +}\n +\n -+/*\n -+ * Acting as a CLIENT.\n -+ *\n -+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n -+ * and tell it to flush its cache.\n -+ *\n -+ * This feature is primarily used by the test suite to simulate a loss of\n -+ * sync with the filesystem where we miss kernel events.\n -+ */\n -+static int do_as_client__send_flush(void)\n ++static int do_as_client__status(void)\n +{\n -+\tstruct strbuf answer = STRBUF_INIT;\n -+\tint ret;\n -+\n -+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n -+\tif (ret)\n -+\t\treturn ret;\n -+\n -+\twrite_in_full(1, answer.buf, answer.len);\n -+\tstrbuf_release(&answer);\n ++\tenum ipc_active_state state = fsmonitor_ipc__get_state();\n +\n -+\treturn 0;\n -+}\n ++\tswitch (state) {\n ++\tcase IPC_STATE__LISTENING:\n ++\t\tprintf(_(\"The built-in file system monitor is active\\n\"));\n ++\t\treturn 0;\n +\n -+static int is_ipc_daemon_listening(void)\n -+{\n -+\treturn fsmonitor_ipc__get_state() == IPC_STATE__LISTENING;\n ++\tdefault:\n ++\t\tprintf(_(\"The built-in file system monitor is not active\\n\"));\n ++\t\treturn 1;\n ++\t}\n +}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n - \tenum daemon_mode {\n - \t\tUNDEFINED_MODE,\n -+\t\tSTOP,\n -+\t\tIS_RUNNING,\n -+\t\tQUERY,\n -+\t\tQUERY_INDEX,\n -+\t\tFLUSH,\n - \t} mode = UNDEFINED_MODE;\n - \n - \tstruct option options[] = {\n -+\t\tOPT_CMDMODE(0, \"stop\", &mode, N_(\"stop the running daemon\"),\n -+\t\t\t STOP),\n -+\n -+\t\tOPT_CMDMODE(0, \"is-running\", &mode,\n -+\t\t\t N_(\"test whether the daemon is running\"),\n -+\t\t\t IS_RUNNING),\n -+\n -+\t\tOPT_CMDMODE(0, \"query\", &mode,\n -+\t\t\t N_(\"query the daemon (starting if necessary)\"),\n -+\t\t\t QUERY),\n -+\t\tOPT_CMDMODE(0, \"query-index\", &mode,\n -+\t\t\t N_(\"query the daemon (starting if necessary) using token from index\"),\n -+\t\t\t QUERY_INDEX),\n -+\t\tOPT_CMDMODE(0, \"flush\", &mode, N_(\"flush cached filesystem events\"),\n -+\t\t\t FLUSH),\n - \t\tOPT_END()\n - \t};\n - \n @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n + \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n \n - \tswitch (mode) {\n -+\tcase STOP:\n ++\tif (!strcmp(subcmd, \"stop\"))\n +\t\treturn !!do_as_client__send_stop();\n +\n -+\tcase IS_RUNNING:\n -+\t\treturn !is_ipc_daemon_listening();\n -+\n -+\tcase QUERY:\n -+\t\tif (argc != 1)\n -+\t\t\tusage_with_options(builtin_fsmonitor__daemon_usage,\n -+\t\t\t\t\t options);\n -+\t\treturn !!do_as_client__query_token(argv[0]);\n ++\tif (!strcmp(subcmd, \"status\"))\n ++\t\treturn !!do_as_client__status();\n +\n -+\tcase QUERY_INDEX:\n -+\t\treturn !!do_as_client__query_from_index();\n -+\n -+\tcase FLUSH:\n -+\t\treturn !!do_as_client__send_flush();\n -+\n - \tcase UNDEFINED_MODE:\n - \tdefault:\n - \t\tdie(_(\"Unhandled command mode %d\"), mode);\n + \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n + }\n + \n -: ------------ > 10: d21af7ff842c t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n 7: 27f47dfbd9cf ! 11: 49f9e2e3d49c fsmonitor-fs-listen-win32: stub in backend for Windows\n @@ Makefile: all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n -+# If your platform supports an built-in fsmonitor backend, set\n -+# FSMONITOR_DAEMON_BACKEND to the name of the corresponding\n ++# If your platform supports a built-in fsmonitor backend, set\n ++# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n +# `fsmonitor_fs_listen__*()` routines.\n +#\n 8: a84dee24e13e = 12: 2aa85151f03c fsmonitor-fs-listen-macos: stub in backend for MacOS\n 9: 2b291d805d59 ! 13: 2aa05ad5c67f fsmonitor--daemon: implement daemon command options\n @@ Metadata\n ## Commit message ##\n fsmonitor--daemon: implement daemon command options\n \n - Implement command options `--run` and `--start` to try to\n + Implement `run` and `start` commands to try to\n begin listening for file system events.\n \n This version defines the thread structure with a single\n @@ builtin/fsmonitor--daemon.c\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n -+\tN_(\"git fsmonitor--daemon --start []\"),\n -+\tN_(\"git fsmonitor--daemon --run []\"),\n - \tN_(\"git fsmonitor--daemon --stop\"),\n - \tN_(\"git fsmonitor--daemon --is-running\"),\n - \tN_(\"git fsmonitor--daemon --query \"),\n -@@ builtin/fsmonitor--daemon.c: static const char * const builtin_fsmonitor__daemon_usage[] = {\n ++\tN_(\"git fsmonitor--daemon start []\"),\n ++\tN_(\"git fsmonitor--daemon run []\"),\n + \tN_(\"git fsmonitor--daemon stop\"),\n + \tN_(\"git fsmonitor--daemon status\"),\n + \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n @@ builtin/fsmonitor--daemon.c: static const char * const builtin_fsmonitor__daemon\n /*\n * Acting as a CLIENT.\n *\n -@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n - \treturn 0;\n +@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n + \t}\n }\n \n +static ipc_server_application_cb handle_client;\n +\n -+static int handle_client(void *data, const char *command,\n ++static int handle_client(void *data,\n ++\t\t\t const char *command, size_t command_len,\n +\t\t\t ipc_server_reply_cb *reply,\n +\t\t\t struct ipc_server_reply_data *reply_data)\n +{\n +\t/* struct fsmonitor_daemon_state *state = data; */\n +\tint result;\n +\n ++\t/*\n ++\t * The Simple IPC API now supports {char*, len} arguments, but\n ++\t * FSMonitor always uses proper null-terminated strings, so\n ++\t * we can ignore the command_len argument. (Trust, but verify.)\n ++\t */\n ++\tif (command_len != strlen(command))\n ++\t\tBUG(\"FSMonitor assumes text messages\");\n ++\n +\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n +\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n +\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\tpthread_mutex_init(&state.main_lock, NULL);\n +\tstate.error_code = 0;\n +\tstate.current_token_data = NULL;\n -+\tstate.test_client_delay_ms = 0;\n +\n +\t/* Prepare to (recursively) watch the directory. */\n +\tstrbuf_init(&state.path_worktree_watch, 0);\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\tstate.nr_paths_watching = 1;\n +\n +\t/*\n -+\t * If \".git\" is not a directory, then is not inside the\n -+\t * cone of , so set up a second watch for it.\n ++\t * We create/delete cookie files inside the .git directory to\n ++\t * help us keep sync with the file system. If \".git\" is not a\n ++\t * directory, then is not inside the cone of\n ++\t * , so set up a second watch for it.\n +\t */\n +\tstrbuf_init(&state.path_gitdir_watch, 0);\n +\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\treturn err;\n +}\n +\n - static int is_ipc_daemon_listening(void)\n - {\n - \treturn fsmonitor_ipc__get_state() == IPC_STATE__LISTENING;\n - }\n - \n +static int try_to_run_foreground_daemon(void)\n +{\n +\t/*\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\t * However, this method gives us a nicer error message for a\n +\t * common error case.\n +\t */\n -+\tif (is_ipc_daemon_listening())\n ++\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n +\t\tdie(\"fsmonitor--daemon is already running.\");\n +\n +\treturn !!fsmonitor_run_daemon();\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\n +\tstrvec_push(&args, git_exe);\n +\tstrvec_push(&args, \"fsmonitor--daemon\");\n -+\tstrvec_push(&args, \"--run\");\n ++\tstrvec_push(&args, \"run\");\n +\n +\t*pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);\n +\tclose(in);\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\n +\t\tif (pid_seen == -1)\n +\t\t\treturn error_errno(_(\"waitpid failed\"));\n -+\n +\t\telse if (pid_seen == 0) {\n +\t\t\t/*\n +\t\t\t * The child is still running (this should be\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\t\t\ttime(&now);\n +\t\t\tif (now > time_limit)\n +\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n -+\n -+\t\t\tcontinue;\n -+\t\t}\n -+\n -+\t\telse if (pid_seen == pid_child) {\n ++\t\t} else if (pid_seen == pid_child) {\n +\t\t\t/*\n +\t\t\t * The new child daemon process shutdown while\n +\t\t\t * it was starting up, so it is not listening\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\t\t\t * early shutdown as an error.\n +\t\t\t */\n +\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n -+\t\t}\n -+\n -+\t\telse\n ++\t\t} else\n +\t\t\treturn error(_(\"waitpid is confused\"));\n +\t}\n +}\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\t * of creating the background process (and not whether it\n +\t * immediately exited).\n +\t */\n -+\tif (is_ipc_daemon_listening())\n ++\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n +\t\tdie(\"fsmonitor--daemon is already running.\");\n +\n +\t/*\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n - \tenum daemon_mode {\n - \t\tUNDEFINED_MODE,\n -+\t\tSTART,\n -+\t\tRUN,\n - \t\tSTOP,\n - \t\tIS_RUNNING,\n - \t\tQUERY,\n -@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n - \t} mode = UNDEFINED_MODE;\n + \tconst char *subcmd;\n \n \tstruct option options[] = {\n -+\t\tOPT_CMDMODE(0, \"start\", &mode,\n -+\t\t\t N_(\"run the daemon in the background\"),\n -+\t\t\t START),\n -+\t\tOPT_CMDMODE(0, \"run\", &mode,\n -+\t\t\t N_(\"run the daemon in the foreground\"), RUN),\n - \t\tOPT_CMDMODE(0, \"stop\", &mode, N_(\"stop the running daemon\"),\n - \t\t\t STOP),\n - \n -@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n - \t\t\t QUERY_INDEX),\n - \t\tOPT_CMDMODE(0, \"flush\", &mode, N_(\"flush cached filesystem events\"),\n - \t\t\t FLUSH),\n -+\n -+\t\tOPT_GROUP(N_(\"Daemon options\")),\n +\t\tOPT_INTEGER(0, \"ipc-threads\",\n +\t\t\t &fsmonitor__ipc_threads,\n +\t\t\t N_(\"use ipc worker threads\")),\n +\t\tOPT_INTEGER(0, \"start-timeout\",\n +\t\t\t &fsmonitor__start_timeout_sec,\n +\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n ++\n \t\tOPT_END()\n \t};\n \n +@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \n -\tgit_config(git_default_config, NULL);\n +\tgit_config(fsmonitor_config, NULL);\n \n + \tsubcmd = argv[1];\n + \targv--;\n +@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n + \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n +\tif (fsmonitor__ipc_threads < 1)\n +\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n +\t\t fsmonitor__ipc_threads);\n - \n - \tswitch (mode) {\n -+\tcase START:\n ++\n ++\tif (!strcmp(subcmd, \"start\"))\n +\t\treturn !!try_to_start_background_daemon();\n +\n -+\tcase RUN:\n ++\tif (!strcmp(subcmd, \"run\"))\n +\t\treturn !!try_to_run_foreground_daemon();\n -+\n - \tcase STOP:\n - \t\treturn !!do_as_client__send_stop();\n \n + \tif (!strcmp(subcmd, \"stop\"))\n + \t\treturn !!do_as_client__send_stop();\n \n ## fsmonitor--daemon.h (new) ##\n @@\n @@ fsmonitor--daemon.h (new)\n +\tstruct fsmonitor_daemon_backend_data *backend_data;\n +\n +\tstruct ipc_server_data *ipc_server_data;\n -+\n -+\tint test_client_delay_ms;\n +};\n +\n +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n 10: 451563314d84 ! 14: d5ababfd03e9 fsmonitor--daemon: add pathname classification\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## builtin/fsmonitor--daemon.c ##\n -@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data, const char *command,\n +@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,\n \treturn result;\n }\n \n @@ builtin/fsmonitor--daemon.c: static int handle_client(void *data, const char *co\n \n ## fsmonitor--daemon.h ##\n @@ fsmonitor--daemon.h: struct fsmonitor_daemon_state {\n - \tint test_client_delay_ms;\n + \tstruct ipc_server_data *ipc_server_data;\n };\n \n +/*\n 11: 304fe03034f8 ! 15: c092cdf2c8b7 fsmonitor--daemon: define token-ids\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## builtin/fsmonitor--daemon.c ##\n -@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n - \treturn 0;\n +@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n + \t}\n }\n \n +/*\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n + *\n + * \"builtin\" \":\" \":\" \n + *\n ++ * The \"builtin\" prefix is used as a namespace to avoid conflicts\n ++ * with other providers (such as Watchman).\n ++ *\n + * The is an arbitrary OPAQUE string, such as a GUID,\n + * UUID, or {timestamp,pid}. It is used to group all filesystem\n + * events that happened while the daemon was monitoring (and in-sync\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n + * (There are too many race conditions to rely on file system\n + * event timestamps.)\n + *\n -+ * The is a simple integer incremented for each event\n -+ * received. When a new is created, the is\n -+ * reset to zero.\n ++ * The is a simple integer incremented whenever the\n ++ * daemon needs to make its state public. For example, if 1000 file\n ++ * system events come in, but no clients have requested the data,\n ++ * the daemon can continue to accumulate file changes in the same\n ++ * bin and does not need to advance the sequence number. However,\n ++ * as soon as a client does arrive, the daemon needs to start a new\n ++ * bin and increment the sequence number.\n ++ *\n ++ * The sequence number serves as the boundary between 2 sets\n ++ * of bins -- the older ones that the client has already seen\n ++ * and the newer ones that it hasn't.\n ++ *\n ++ * When a new is created, the is reset to\n ++ * zero.\n + *\n + *\n + * About Token Ids\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n + * [3] in response to a client \"flush\" command (for dropped event\n + * testing).\n + *\n -+ * [4] MAYBE We might want to change the token_id after very complex\n -+ * filesystem operations are performed, such as a directory move\n -+ * sequence that affects many files within. It might be simpler\n -+ * to just give up and fake a re-sync (and let the client do a\n -+ * full scan) than try to enumerate the effects of such a change.\n -+ *\n + * When a new token_id is created, the daemon is free to discard all\n + * cached filesystem events associated with any previous token_ids.\n + * Events associated with a non-current token_id will never be sent\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\tstatic uint64_t flush_count = 0;\n +\tstruct fsmonitor_token_data *token;\n +\n -+\ttoken = (struct fsmonitor_token_data *)xcalloc(1, sizeof(*token));\n ++\tCALLOC_ARRAY(token, 1);\n +\n +\tstrbuf_init(&token->token_id, 0);\n +\ttoken->batch_head = NULL;\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\n static ipc_server_application_cb handle_client;\n \n - static int handle_client(void *data, const char *command,\n + static int handle_client(void *data,\n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n -\tstate.current_token_data = NULL;\n +\tstate.current_token_data = fsmonitor_new_token_data();\n - \tstate.test_client_delay_ms = 0;\n \n \t/* Prepare to (recursively) watch the directory. */\n + \tstrbuf_init(&state.path_worktree_watch, 0);\n 12: f1fa803ebe9c ! 16: 2ed7bc3fae7a fsmonitor--daemon: create token-based changed path cache\n @@ Metadata\n ## Commit message ##\n fsmonitor--daemon: create token-based changed path cache\n \n - Teach fsmonitor--daemon to build lists of changed paths and associate\n + Teach fsmonitor--daemon to build a list of changed paths and associate\n them with a token-id. This will be used by the platform-specific\n backends to accumulate changed paths in response to filesystem events.\n \n - The platform-specific event loops receive batches containing one or\n - more changed paths. Their fs listener thread will accumulate them in\n - a `fsmonitor_batch` (and without locking) and then \"publish\" them to\n - associate them with the current token and to make them visible to the\n - client worker threads.\n + The platform-specific file system listener thread receives file system\n + events containing one or more changed pathnames (with whatever bucketing\n + or grouping that is convenient for the file system). These paths are\n + accumulated (without locking) by the file system layer into a `fsmonitor_batch`.\n +\n + When the file system layer has drained the kernel event queue, it will\n + \"publish\" them to our token queue and make them visible to concurrent\n + client worker threads. The token layer is free to combine and/or de-dup\n + paths within these batches for efficient presentation to clients.\n \n Signed-off-by: Jeff Hostetler \n \n ## builtin/fsmonitor--daemon.c ##\n -@@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n - \treturn token;\n - }\n +@@ builtin/fsmonitor--daemon.c: struct fsmonitor_token_data {\n + \tuint64_t client_ref_count;\n + };\n \n +struct fsmonitor_batch {\n +\tstruct fsmonitor_batch *next;\n @@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t\n +\ttime_t pinned_time;\n +};\n +\n + static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n + {\n + \tstatic int test_env_value = -1;\n + \tstatic uint64_t flush_count = 0;\n + \tstruct fsmonitor_token_data *token;\n ++\tstruct fsmonitor_batch *batch;\n + \n + \tCALLOC_ARRAY(token, 1);\n ++\tbatch = fsmonitor_batch__new();\n + \n + \tstrbuf_init(&token->token_id, 0);\n +-\ttoken->batch_head = NULL;\n +-\ttoken->batch_tail = NULL;\n ++\ttoken->batch_head = batch;\n ++\ttoken->batch_tail = batch;\n + \ttoken->client_ref_count = 0;\n + \n + \tif (test_env_value < 0)\n +@@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n + \t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n + \t}\n + \n ++\t/*\n ++\t * We created a new and are starting a new series\n ++\t * of tokens with a zero .\n ++\t *\n ++\t * Since clients cannot guess our new (non test) \n ++\t * they will always receive a trivial response (because of the\n ++\t * mismatch on the ). The trivial response will\n ++\t * tell them our new so that subsequent requests\n ++\t * will be relative to our new series. (And when sending that\n ++\t * response, we pin the current head of the batch list.)\n ++\t *\n ++\t * Even if the client correctly guesses the , their\n ++\t * request of \"builtin::0\" asks for all changes MORE\n ++\t * RECENT than batch/bin 0.\n ++\t *\n ++\t * This implies that it is a waste to accumulate paths in the\n ++\t * initial batch/bin (because they will never be transmitted).\n ++\t *\n ++\t * So the daemon could be running for days and watching the\n ++\t * file system, but doesn't need to actually accumulate any\n ++\t * paths UNTIL we need to set a reference point for a later\n ++\t * relative request.\n ++\t *\n ++\t * However, it is very useful for testing to always have a\n ++\t * reference point set. Pin batch 0 to force early file system\n ++\t * events to accumulate.\n ++\t */\n ++\tif (test_env_value)\n ++\t\tbatch->pinned_time = time(NULL);\n ++\n + \treturn token;\n + }\n + \n +struct fsmonitor_batch *fsmonitor_batch__new(void)\n +{\n -+\tstruct fsmonitor_batch *batch = xcalloc(1, sizeof(*batch));\n ++\tstruct fsmonitor_batch *batch;\n ++\n ++\tCALLOC_ARRAY(batch, 1);\n +\n +\treturn batch;\n +}\n +\n -+struct fsmonitor_batch *fsmonitor_batch__free(struct fsmonitor_batch *batch)\n ++struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch)\n +{\n +\tstruct fsmonitor_batch *next;\n +\n @@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t\n +static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n +\t\t\t\t const struct fsmonitor_batch *batch_src)\n +{\n -+\t/* assert state->main_lock */\n -+\n +\tsize_t k;\n +\n +\tALLOC_GROW(batch_dest->interned_paths,\n @@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t\n +\n +\tstrbuf_release(&token->token_id);\n +\n -+\tfor (p = token->batch_head; p; p = fsmonitor_batch__free(p))\n ++\tfor (p = token->batch_head; p; p = fsmonitor_batch__pop(p))\n +\t\t;\n +\n +\tfree(token);\n @@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t\n + * We should create a new token and start fresh (as if we just\n + * booted up).\n + *\n -+ * If there are no readers of the the current token data series, we\n -+ * can free it now. Otherwise, let the last reader free it. Either\n -+ * way, the old token data series is no longer associated with our\n -+ * state data.\n ++ * If there are no concurrent threads readering the current token data\n ++ * series, we can free it now. Otherwise, let the last reader free\n ++ * it.\n ++ *\n ++ * Either way, the old token data series is no longer associated with\n ++ * our state data.\n + */\n -+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n ++static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n +{\n ++\t/* assert current thread holding state->main_lock */\n ++\n +\tstruct fsmonitor_token_data *free_me = NULL;\n +\tstruct fsmonitor_token_data *new_one = NULL;\n +\n +\tnew_one = fsmonitor_new_token_data();\n +\n -+\tpthread_mutex_lock(&state->main_lock);\n -+\n -+\ttrace_printf_key(&trace_fsmonitor,\n -+\t\t\t \"force resync [old '%s'][new '%s']\",\n -+\t\t\t state->current_token_data->token_id.buf,\n -+\t\t\t new_one->token_id.buf);\n -+\n +\tif (state->current_token_data->client_ref_count == 0)\n +\t\tfree_me = state->current_token_data;\n +\tstate->current_token_data = new_one;\n +\n -+\tpthread_mutex_unlock(&state->main_lock);\n -+\n +\tfsmonitor_free_token_data(free_me);\n +}\n ++\n ++void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n ++{\n ++\tpthread_mutex_lock(&state->main_lock);\n ++\twith_lock__do_force_resync(state);\n ++\tpthread_mutex_unlock(&state->main_lock);\n ++}\n +\n static ipc_server_application_cb handle_client;\n \n - static int handle_client(void *data, const char *command,\n + static int handle_client(void *data,\n @@ builtin/fsmonitor--daemon.c: enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n @@ builtin/fsmonitor--daemon.c: enum fsmonitor_path_type fsmonitor_classify_path_ab\n +\n +\t\thead = state->current_token_data->batch_head;\n +\t\tif (!head) {\n -+\t\t\tbatch->batch_seq_nr = 0;\n -+\t\t\tbatch->next = NULL;\n -+\t\t\tstate->current_token_data->batch_head = batch;\n -+\t\t\tstate->current_token_data->batch_tail = batch;\n ++\t\t\tBUG(\"token does not have batch\");\n +\t\t} else if (head->pinned_time) {\n +\t\t\t/*\n +\t\t\t * We cannot alter the current batch list\n @@ builtin/fsmonitor--daemon.c: enum fsmonitor_path_type fsmonitor_classify_path_ab\n +\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n +\t\t\tbatch->next = head;\n +\t\t\tstate->current_token_data->batch_head = batch;\n ++\t\t} else if (!head->batch_seq_nr) {\n ++\t\t\t/*\n ++\t\t\t * Batch 0 is unpinned. See the note in\n ++\t\t\t * `fsmonitor_new_token_data()` about why we\n ++\t\t\t * don't need to accumulate these paths.\n ++\t\t\t */\n ++\t\t\tfsmonitor_batch__pop(batch);\n +\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n +\t\t\t/*\n +\t\t\t * The head batch in the list has never been\n @@ builtin/fsmonitor--daemon.c: enum fsmonitor_path_type fsmonitor_classify_path_ab\n +\t\t\t * batch onto the end of the current head batch.\n +\t\t\t */\n +\t\t\tfsmonitor_batch__combine(head, batch);\n -+\t\t\tfsmonitor_batch__free(batch);\n ++\t\t\tfsmonitor_batch__pop(batch);\n +\t\t}\n +\t}\n +\n @@ fsmonitor--daemon.h\n +/*\n + * Free this batch and return the value of the batch->next field.\n + */\n -+struct fsmonitor_batch *fsmonitor_batch__free(struct fsmonitor_batch *batch);\n ++struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch);\n +\n +/*\n + * Add this path to this batch of modified files.\n 13: a57ddb3bc7cc ! 17: 9ea4b04b8215 fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n +\t\treturn NULL;\n +\t}\n +\n -+\twatch = xcalloc(1, sizeof(*watch));\n ++\tCALLOC_ARRAY(watch, 1);\n +\n +\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n +\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n +\twatch->overlapped.hEvent = watch->hEvent;\n +\n +start_watch:\n ++\t/*\n ++\t * Queue an async call using Overlapped IO. This returns immediately.\n ++\t * Our event handle will be signalled when the real result is available.\n ++\t *\n ++\t * The return value here just means that we successfully queued it.\n ++\t * We won't know if the Read...() actually produces data until later.\n ++\t */\n +\twatch->is_active = ReadDirectoryChangesW(\n +\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n +\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n +\n ++\t/*\n ++\t * The kernel throws an invalid parameter error when our buffer\n ++\t * is too big and we are pointed at a remote directory (and possibly\n ++\t * for other reasons). Quietly set it down and try again.\n ++\t *\n ++\t * See note about MAX_RDCW_BUF at the top.\n ++\t */\n +\tif (!watch->is_active &&\n +\t GetLastError() == ERROR_INVALID_PARAMETER &&\n +\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n +{\n +\twatch->is_active = FALSE;\n +\n ++\t/*\n ++\t * The overlapped result is ready. If the Read...() was successful\n ++\t * we finally receive the actual result into our buffer.\n ++\t */\n +\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n +\t\t\t\tTRUE))\n +\t\treturn 0;\n +\n -+\t// TODO If an external is deleted, the above returns an error.\n -+\t// TODO I'm not sure that there's anything that we can do here other\n -+\t// TODO than failing -- the /.git link file would be broken\n -+\t// TODO anyway. We might try to check for that and return a better\n -+\t// TODO error message.\n ++\t/*\n ++\t * NEEDSWORK: If an external is deleted, the above\n ++\t * returns an error. I'm not sure that there's anything that\n ++\t * we can do here other than failing -- the /.git\n ++\t * link file would be broken anyway. We might try to check\n ++\t * for that and return a better error message, but I'm not\n ++\t * sure it is worth it.\n ++\t */\n +\n +\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n +\t watch->path.buf, GetLastError());\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n +\tif (!watch || !watch->is_active)\n +\t\treturn;\n +\n ++\t/*\n ++\t * The calls to ReadDirectoryChangesW() and GetOverlappedResult()\n ++\t * form a \"pair\" (my term) where we queue an IO and promise to\n ++\t * hang around and wait for the kernel to give us the result.\n ++\t *\n ++\t * If for some reason after we queue the IO, we have to quit\n ++\t * or otherwise not stick around for the second half, we must\n ++\t * tell the kernel to abort the IO. This prevents the kernel\n ++\t * from writing to our buffer and/or signalling our event\n ++\t * after we free them.\n ++\t *\n ++\t * (Ask me how much fun it was to track that one down).\n ++\t */\n +\tCancelIoEx(watch->hDir, &watch->overlapped);\n +\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n +\twatch->is_active = FALSE;\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n +\t/*\n +\t * If the kernel gets more events than will fit in the kernel\n +\t * buffer associated with our RDCW handle, it drops them and\n -+\t * returns a count of zero. (A successful call, but with\n -+\t * length zero.)\n ++\t * returns a count of zero.\n ++\t *\n ++\t * Yes, the call returns WITHOUT error and with length zero.\n ++\t *\n ++\t * (The \"overflow\" case is not ambiguous with the \"no data\" case\n ++\t * because we did an INFINITE wait.)\n ++\t *\n ++\t * This means we have a gap in coverage. Tell the daemon layer\n ++\t * to resync.\n +\t */\n +\tif (!watch->count) {\n +\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n +\t\tdefault:\n +\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n +\t\t\t t, path.buf);\n -+\t\t\tgoto skip_this_path;\n +\t\t}\n +\n +skip_this_path:\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n +\treturn LISTENER_HAVE_DATA_WORKTREE;\n +\n +force_shutdown:\n -+\tfsmonitor_batch__free(batch);\n ++\tfsmonitor_batch__pop(batch);\n +\tstring_list_clear(&cookie_list, 0);\n +\tstrbuf_release(&path);\n +\treturn LISTENER_SHUTDOWN;\n +}\n +\n +/*\n -+ * Process filesystem events that happend anywhere (recursively) under the\n ++ * Process filesystem events that happened anywhere (recursively) under the\n + * external (such as non-primary worktrees or submodules).\n + * We only care about cookie files that our client threads created here.\n + *\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n +\t\tdefault:\n +\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n +\t\t\t t, path.buf);\n -+\t\t\tgoto skip_this_path;\n +\t\t}\n +\n +skip_this_path:\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n {\n +\tstruct fsmonitor_daemon_backend_data *data;\n +\n -+\tdata = xcalloc(1, sizeof(*data));\n ++\tCALLOC_ARRAY(data, 1);\n +\n +\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n +\n 14: 67fa7c7b8ac7 = 18: 21b2b4f941b2 fsmonitor-fs-listen-macos: add macos header files for FSEvent\n 15: d469d3f02e33 ! 19: 08474bad8303 fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS\n @@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS\n +\t\tef & kFSEventStreamEventFlagItemRenamed);\n +}\n +\n ++static int ef_is_dropped(const FSEventStreamEventFlags ef)\n ++{\n ++\treturn (ef & kFSEventStreamEventFlagKernelDropped ||\n ++\t\tef & kFSEventStreamEventFlagUserDropped);\n ++}\n ++\n +static void fsevent_callback(ConstFSEventStreamRef streamRef,\n +\t\t\t void *ctx,\n +\t\t\t size_t num_of_events,\n @@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS\n +\tconst char *path_k;\n +\tconst char *slash;\n +\tint k;\n ++\tstruct strbuf tmp = STRBUF_INIT;\n +\n +\t/*\n +\t * Build a list of all filesystem changes into a private/local\n @@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS\n +\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n +\t\t * Please don't log them to Trace2.\n +\t\t *\n -+\t\t * trace_printf_key(&trace_fsmonitor, \"XXX '%s'\", path_k);\n ++\t\t * trace_printf_key(&trace_fsmonitor, \"Path: '%s'\", path_k);\n +\t\t */\n +\n +\t\t/*\n @@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS\n +\t\t * they are conceptually relative to the just flushed\n +\t\t * token).\n +\t\t */\n -+\t\tif ((event_flags[k] & kFSEventStreamEventFlagKernelDropped) ||\n -+\t\t (event_flags[k] & kFSEventStreamEventFlagUserDropped)) {\n ++\t\tif (ef_is_dropped(event_flags[k])) {\n +\t\t\t/*\n +\t\t\t * see also kFSEventStreamEventFlagMustScanSubDirs\n +\t\t\t */\n -+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n -+\t\t\t\t\t \"fsm-listen/kernel\", \"dropped\");\n ++\t\t\ttrace_printf_key(&trace_fsmonitor, \"event: dropped\");\n +\n +\t\t\tfsmonitor_force_resync(state);\n -+\n -+\t\t\tif (fsmonitor_batch__free(batch))\n -+\t\t\t\tBUG(\"batch should not have a next\");\n ++\t\t\tfsmonitor_batch__pop(batch);\n +\t\t\tstring_list_clear(&cookie_list, 0);\n +\n +\t\t\t/*\n @@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS\n +\t\t\t * we have to quit.\n +\t\t\t */\n +\t\t\tif (ef_is_root_delete(event_flags[k])) {\n -+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n -+\t\t\t\t\t\t \"fsm-listen/gitdir\",\n -+\t\t\t\t\t\t \"removed\");\n ++\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n ++\t\t\t\t\t\t \"event: gitdir removed\");\n +\t\t\t\tgoto force_shutdown;\n +\t\t\t}\n +\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n -+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n -+\t\t\t\t\t\t \"fsm-listen/gitdir\",\n -+\t\t\t\t\t\t \"renamed\");\n ++\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n ++\t\t\t\t\t\t \"event: gitdir renamed\");\n +\t\t\t\tgoto force_shutdown;\n +\t\t\t}\n +\t\t\tbreak;\n @@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS\n +\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n +\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n +\n -+\t\t\t/* fsevent could be marked as both a file and directory */\n ++\t\t\t/*\n ++\t\t\t * Because of the implicit \"binning\" (the\n ++\t\t\t * kernel calls us at a given frequency) and\n ++\t\t\t * de-duping (the kernel is free to combine\n ++\t\t\t * multiple events for a given pathname), an\n ++\t\t\t * individual fsevent could be marked as both\n ++\t\t\t * a file and directory. Add it to the queue\n ++\t\t\t * with both spellings so that the client will\n ++\t\t\t * know how much to invalidate/refresh.\n ++\t\t\t */\n +\n +\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n +\t\t\t\tconst char *rel = path_k +\n @@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS\n +\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n +\t\t\t\tconst char *rel = path_k +\n +\t\t\t\t\tstate->path_worktree_watch.len + 1;\n -+\t\t\t\tchar *p = xstrfmt(\"%s/\", rel);\n ++\n ++\t\t\t\tstrbuf_reset(&tmp);\n ++\t\t\t\tstrbuf_addstr(&tmp, rel);\n ++\t\t\t\tstrbuf_addch(&tmp, '/');\n +\n +\t\t\t\tif (!batch)\n +\t\t\t\t\tbatch = fsmonitor_batch__new();\n -+\t\t\t\tfsmonitor_batch__add_path(batch, p);\n -+\n -+\t\t\t\tfree(p);\n ++\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n +\t\t\t}\n +\n +\t\t\tbreak;\n @@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS\n +\n +\tfsmonitor_publish(state, batch, &cookie_list);\n +\tstring_list_clear(&cookie_list, 0);\n ++\tstrbuf_release(&tmp);\n +\treturn;\n +\n +force_shutdown:\n -+\tif (fsmonitor_batch__free(batch))\n -+\t\tBUG(\"batch should not have a next\");\n ++\tfsmonitor_batch__pop(batch);\n +\tstring_list_clear(&cookie_list, 0);\n +\n +\tdata->shutdown_style = FORCE_SHUTDOWN;\n +\tCFRunLoopStop(data->rl);\n ++\tstrbuf_release(&tmp);\n +\treturn;\n +}\n +\n +/*\n -+ * TODO Investigate the proper value for the `latency` argument in the call\n -+ * TODO to `FSEventStreamCreate()`. I'm not sure that this needs to be a\n -+ * TODO config setting or just something that we tune after some testing.\n -+ * TODO\n -+ * TODO With a latency of 0.1, I was seeing lots of dropped events during\n -+ * TODO the \"touch 100000\" files test within t/perf/p7519, but with a\n -+ * TODO latency of 0.001 I did not see any dropped events. So the \"correct\"\n -+ * TODO value may be somewhere in between.\n -+ * TODO\n -+ * TODO https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n ++ * NEEDSWORK: Investigate the proper value for the `latency` argument\n ++ * in the call to `FSEventStreamCreate()`. I'm not sure that this\n ++ * needs to be a config setting or just something that we tune after\n ++ * some testing.\n ++ *\n ++ * With a latency of 0.1, I was seeing lots of dropped events during\n ++ * the \"touch 100000\" files test within t/perf/p7519, but with a\n ++ * latency of 0.001 I did not see any dropped events. So the\n ++ * \"correct\" value may be somewhere in between.\n ++ *\n ++ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n + */\n \n int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n @@ compat/fsmonitor/fsmonitor-fs-listen-macos.c: void FSEventStreamRelease(FSEventS\n +\tstruct fsmonitor_daemon_backend_data *data;\n +\tconst void *dir_array[2];\n +\n -+\tdata = xcalloc(1, sizeof(*data));\n ++\tCALLOC_ARRAY(data, 1);\n +\tstate->backend_data = data;\n +\n +\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n 16: 2b4ae4fc3d62 ! 20: cc4a596d17c7 fsmonitor--daemon: implement handle_client callback\n @@ builtin/fsmonitor--daemon.c\n +#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n - \tN_(\"git fsmonitor--daemon --start []\"),\n + \tN_(\"git fsmonitor--daemon start []\"),\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n - \tfsmonitor_free_token_data(free_me);\n + \tpthread_mutex_unlock(&state->main_lock);\n }\n \n +/*\n + * Format an opaque token string to send to the client.\n + */\n -+static void fsmonitor_format_response_token(\n ++static void with_lock__format_response_token(\n +\tstruct strbuf *response_token,\n +\tconst struct strbuf *response_token_id,\n +\tconst struct fsmonitor_batch *batch)\n +{\n -+\tuint64_t seq_nr = (batch) ? batch->batch_seq_nr + 1 : 0;\n ++\t/* assert current thread holding state->main_lock */\n +\n +\tstrbuf_reset(response_token);\n +\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n -+\t\t response_token_id->buf, seq_nr);\n ++\t\t response_token_id->buf, batch->batch_seq_nr);\n +}\n +\n +/*\n + * Parse an opaque token from the client.\n ++ * Returns -1 on error.\n + */\n +static int fsmonitor_parse_client_token(const char *buf_token,\n +\t\t\t\t\tstruct strbuf *requested_token_id,\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\t*seq_nr = 0;\n +\n +\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n -+\t\treturn 1;\n ++\t\treturn -1;\n +\n +\twhile (*p && *p != ':')\n +\t\tstrbuf_addch(requested_token_id, *p++);\n +\tif (!*p++)\n -+\t\treturn 1;\n ++\t\treturn -1;\n +\n +\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n +\tif (*p_end)\n -+\t\treturn 1;\n ++\t\treturn -1;\n +\n +\treturn 0;\n +}\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\tintmax_t count = 0, duplicates = 0;\n +\tkh_str_t *shown;\n +\tint hash_ret;\n -+\tint result;\n ++\tint do_trivial = 0;\n ++\tint do_flush = 0;\n +\n +\t/*\n +\t * We expect `command` to be of the form:\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\t\t * There is no reply to the client.\n +\t\t */\n +\t\treturn SIMPLE_IPC_QUIT;\n -+\t}\n +\n -+\tif (!strcmp(command, \"flush\")) {\n ++\t} else if (!strcmp(command, \"flush\")) {\n +\t\t/*\n +\t\t * Flush all of our cached data and generate a new token\n +\t\t * just like if we lost sync with the filesystem.\n +\t\t *\n +\t\t * Then send a trivial response using the new token.\n +\t\t */\n -+\t\tfsmonitor_force_resync(state);\n -+\t\tresult = 0;\n -+\t\tgoto send_trivial_response;\n -+\t}\n ++\t\tdo_flush = 1;\n ++\t\tdo_trivial = 1;\n +\n -+\tif (!skip_prefix(command, \"builtin:\", &p)) {\n ++\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n +\t\t/* assume V1 timestamp or garbage */\n +\n +\t\tchar *p_end;\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n +\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n +\t\t\t\t command);\n -+\t\tresult = -1;\n -+\t\tgoto send_trivial_response;\n ++\t\tdo_trivial = 1;\n ++\n ++\t} else {\n ++\t\t/* We have \"builtin:*\" */\n ++\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n ++\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n ++\t\t\ttrace_printf_key(&trace_fsmonitor,\n ++\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n ++\t\t\t\t\t command);\n ++\t\t\tdo_trivial = 1;\n ++\n ++\t\t} else {\n ++\t\t\t/*\n ++\t\t\t * We have a V2 valid token:\n ++\t\t\t * \"builtin::\"\n ++\t\t\t */\n ++\t\t}\n +\t}\n +\n -+\t/* try V2 token */\n ++\tpthread_mutex_lock(&state->main_lock);\n +\n -+\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n -+\t\t\t\t\t &requested_oldest_seq_nr)) {\n -+\t\ttrace_printf_key(&trace_fsmonitor,\n -+\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n -+\t\t\t\t command);\n -+\t\tresult = -1;\n -+\t\tgoto send_trivial_response;\n -+\t}\n ++\tif (!state->current_token_data)\n ++\t\tBUG(\"fsmonitor state does not have a current token\");\n +\n -+\tpthread_mutex_lock(&state->main_lock);\n ++\tif (do_flush)\n ++\t\twith_lock__do_force_resync(state);\n +\n -+\tif (!state->current_token_data) {\n -+\t\t/*\n -+\t\t * We don't have a current token. This may mean that\n -+\t\t * the listener thread has not yet started.\n -+\t\t */\n -+\t\tpthread_mutex_unlock(&state->main_lock);\n -+\t\tresult = 0;\n -+\t\tgoto send_trivial_response;\n -+\t}\n -+\tif (strcmp(requested_token_id.buf,\n -+\t\t state->current_token_data->token_id.buf)) {\n -+\t\t/*\n -+\t\t * The client last spoke to a different daemon\n -+\t\t * instance -OR- the daemon had to resync with\n -+\t\t * the filesystem (and lost events), so reject.\n -+\t\t */\n -+\t\tpthread_mutex_unlock(&state->main_lock);\n -+\t\tresult = 0;\n -+\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n -+\t\t\t\t \"response/token\", \"different\");\n -+\t\tgoto send_trivial_response;\n -+\t}\n -+\tif (!state->current_token_data->batch_tail) {\n -+\t\t/*\n -+\t\t * The listener has not received any filesystem\n -+\t\t * events yet since we created the current token.\n -+\t\t * We can respond with an empty list, since the\n -+\t\t * client has already seen the current token and\n -+\t\t * we have nothing new to report. (This is\n -+\t\t * instead of sending a trivial response.)\n -+\t\t */\n -+\t\tpthread_mutex_unlock(&state->main_lock);\n -+\t\tresult = 0;\n -+\t\tgoto send_empty_response;\n ++\t/*\n ++\t * We mark the current head of the batch list as \"pinned\" so\n ++\t * that the listener thread will treat this item as read-only\n ++\t * (and prevent any more paths from being added to it) from\n ++\t * now on.\n ++\t */\n ++\ttoken_data = state->current_token_data;\n ++\tbatch_head = token_data->batch_head;\n ++\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n ++\n ++\t/*\n ++\t * FSMonitor Protocol V2 requires that we send a response header\n ++\t * with a \"new current token\" and then all of the paths that changed\n ++\t * since the \"requested token\". We send the seq_nr of the just-pinned\n ++\t * head batch so that future requests from a client will be relative\n ++\t * to it.\n ++\t */\n ++\twith_lock__format_response_token(&response_token,\n ++\t\t\t\t\t &token_data->token_id, batch_head);\n ++\n ++\treply(reply_data, response_token.buf, response_token.len + 1);\n ++\ttotal_response_len += response_token.len + 1;\n ++\n ++\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n ++\t\t\t response_token.buf);\n ++\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n ++\t\t\t response_token.buf);\n ++\n ++\tif (!do_trivial) {\n ++\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n ++\t\t\t/*\n ++\t\t\t * The client last spoke to a different daemon\n ++\t\t\t * instance -OR- the daemon had to resync with\n ++\t\t\t * the filesystem (and lost events), so reject.\n ++\t\t\t */\n ++\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n ++\t\t\t\t\t \"response/token\", \"different\");\n ++\t\t\tdo_trivial = 1;\n ++\n ++\t\t} else if (requested_oldest_seq_nr <\n ++\t\t\t token_data->batch_tail->batch_seq_nr) {\n ++\t\t\t/*\n ++\t\t\t * The client wants older events than we have for\n ++\t\t\t * this token_id. This means that the end of our\n ++\t\t\t * batch list was truncated and we cannot give the\n ++\t\t\t * client a complete snapshot relative to their\n ++\t\t\t * request.\n ++\t\t\t */\n ++\t\t\ttrace_printf_key(&trace_fsmonitor,\n ++\t\t\t\t\t \"client requested truncated data\");\n ++\t\t\tdo_trivial = 1;\n ++\t\t}\n +\t}\n -+\tif (requested_oldest_seq_nr <\n -+\t state->current_token_data->batch_tail->batch_seq_nr) {\n -+\t\t/*\n -+\t\t * The client wants older events than we have for\n -+\t\t * this token_id. This means that the end of our\n -+\t\t * batch list was truncated and we cannot give the\n -+\t\t * client a complete snapshot relative to their\n -+\t\t * request.\n -+\t\t */\n ++\n ++\tif (do_trivial) {\n +\t\tpthread_mutex_unlock(&state->main_lock);\n +\n -+\t\ttrace_printf_key(&trace_fsmonitor,\n -+\t\t\t\t \"client requested truncated data\");\n -+\t\tresult = 0;\n -+\t\tgoto send_trivial_response;\n ++\t\treply(reply_data, \"/\", 2);\n ++\n ++\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n ++\t\t\t\t \"response/trivial\", 1);\n ++\n ++\t\tstrbuf_release(&response_token);\n ++\t\tstrbuf_release(&requested_token_id);\n ++\t\treturn 0;\n +\t}\n +\n +\t/*\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\t *\n +\t * AND it allows the listener thread to do a token-reset\n +\t * (and install a new `current_token_data`).\n -+\t *\n -+\t * We mark the current head of the batch list as \"pinned\" so\n -+\t * that the listener thread will treat this item as read-only\n -+\t * (and prevent any more paths from being added to it) from\n -+\t * now on.\n +\t */\n -+\ttoken_data = state->current_token_data;\n +\ttoken_data->client_ref_count++;\n +\n -+\tbatch_head = token_data->batch_head;\n -+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n -+\n +\tpthread_mutex_unlock(&state->main_lock);\n +\n +\t/*\n -+\t * FSMonitor Protocol V2 requires that we send a response header\n -+\t * with a \"new current token\" and then all of the paths that changed\n -+\t * since the \"requested token\".\n ++\t * The client request is relative to the token that they sent,\n ++\t * so walk the batch list backwards from the current head back\n ++\t * to the batch (sequence number) they named.\n ++\t *\n ++\t * We use khash to de-dup the list of pathnames.\n ++\t *\n ++\t * NEEDSWORK: each batch contains a list of interned strings,\n ++\t * so we only need to do pointer comparisons here to build the\n ++\t * hash table. Currently, we're still comparing the string\n ++\t * values.\n +\t */\n -+\tfsmonitor_format_response_token(&response_token,\n -+\t\t\t\t\t&token_data->token_id,\n -+\t\t\t\t\tbatch_head);\n -+\n -+\treply(reply_data, response_token.buf, response_token.len + 1);\n -+\ttotal_response_len += response_token.len + 1;\n -+\n -+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n -+\t\t\t response_token.buf);\n -+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\", response_token.buf);\n -+\n +\tshown = kh_init_str();\n +\tfor (batch = batch_head;\n -+\t batch && batch->batch_seq_nr >= requested_oldest_seq_nr;\n ++\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n +\t batch = batch->next) {\n +\t\tsize_t k;\n +\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\tkh_release_str(shown);\n +\n +\tpthread_mutex_lock(&state->main_lock);\n ++\n +\tif (token_data->client_ref_count > 0)\n +\t\ttoken_data->client_ref_count--;\n +\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\tstrbuf_release(&payload);\n +\n +\treturn 0;\n -+\n -+send_trivial_response:\n -+\tpthread_mutex_lock(&state->main_lock);\n -+\tfsmonitor_format_response_token(&response_token,\n -+\t\t\t\t\t&state->current_token_data->token_id,\n -+\t\t\t\t\tstate->current_token_data->batch_head);\n -+\tpthread_mutex_unlock(&state->main_lock);\n -+\n -+\treply(reply_data, response_token.buf, response_token.len + 1);\n -+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n -+\t\t\t response_token.buf);\n -+\treply(reply_data, \"/\", 2);\n -+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/trivial\", 1);\n -+\n -+\tstrbuf_release(&response_token);\n -+\tstrbuf_release(&requested_token_id);\n -+\n -+\treturn result;\n -+\n -+send_empty_response:\n -+\tpthread_mutex_lock(&state->main_lock);\n -+\tfsmonitor_format_response_token(&response_token,\n -+\t\t\t\t\t&state->current_token_data->token_id,\n -+\t\t\t\t\tNULL);\n -+\tpthread_mutex_unlock(&state->main_lock);\n -+\n -+\treply(reply_data, response_token.buf, response_token.len + 1);\n -+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n -+\t\t\t response_token.buf);\n -+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/empty\", 1);\n -+\n -+\tstrbuf_release(&response_token);\n -+\tstrbuf_release(&requested_token_id);\n -+\n -+\treturn 0;\n +}\n +\n static ipc_server_application_cb handle_client;\n \n - static int handle_client(void *data, const char *command,\n + static int handle_client(void *data,\n +@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon\n +\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n + \t/*\n +@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,\n + \tif (command_len != strlen(command))\n + \t\tBUG(\"FSMonitor assumes text messages\");\n + \n +\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n +\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n 17: 9f263e70c724 ! 21: f0da90e9b050 fsmonitor--daemon: periodically truncate list of modified files\n @@ Commit message\n relative to this new token).\n \n Therefore, the daemon can gradually truncate the in-memory list of\n - changed paths as they become obsolete (older that the previous token).\n + changed paths as they become obsolete (older than the previous token).\n Since we may have multiple clients making concurrent requests with a\n skew of tokens and clients may be racing to the talk to the daemon,\n we lazily truncate the list.\n @@ builtin/fsmonitor--daemon.c: static void fsmonitor_batch__combine(struct fsmonit\n + * artificial (based on when we pinned the batch item) and not on any\n + * filesystem activity.\n + */\n -+#define MY_TIME_DELAY (5 * 60) /* seconds */\n ++#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */\n +\n -+static void fsmonitor_batch__truncate(struct fsmonitor_daemon_state *state,\n -+\t\t\t\t const struct fsmonitor_batch *batch_marker)\n ++static void with_lock__truncate_old_batches(\n ++\tstruct fsmonitor_daemon_state *state,\n ++\tconst struct fsmonitor_batch *batch_marker)\n +{\n -+\t/* assert state->main_lock */\n ++\t/* assert current thread holding state->main_lock */\n +\n +\tconst struct fsmonitor_batch *batch;\n +\tstruct fsmonitor_batch *rest;\n +\tstruct fsmonitor_batch *p;\n -+\ttime_t t;\n +\n +\tif (!batch_marker)\n +\t\treturn;\n +\n -+\ttrace_printf_key(&trace_fsmonitor, \"TRNC mark (%\"PRIu64\",%\"PRIu64\")\",\n ++\ttrace_printf_key(&trace_fsmonitor, \"Truncate: mark (%\"PRIu64\",%\"PRIu64\")\",\n +\t\t\t batch_marker->batch_seq_nr,\n +\t\t\t (uint64_t)batch_marker->pinned_time);\n +\n +\tfor (batch = batch_marker; batch; batch = batch->next) {\n ++\t\ttime_t t;\n ++\n +\t\tif (!batch->pinned_time) /* an overflow batch */\n +\t\t\tcontinue;\n +\n -+\t\tt = batch->pinned_time + MY_TIME_DELAY;\n ++\t\tt = batch->pinned_time + MY_TIME_DELAY_SECONDS;\n +\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n +\t\t\tcontinue;\n +\n @@ builtin/fsmonitor--daemon.c: static void fsmonitor_batch__combine(struct fsmonit\n +\trest = ((struct fsmonitor_batch *)batch)->next;\n +\t((struct fsmonitor_batch *)batch)->next = NULL;\n +\n -+\tfor (p = rest; p; p = fsmonitor_batch__free(p)) {\n ++\tfor (p = rest; p; p = fsmonitor_batch__pop(p)) {\n +\t\ttrace_printf_key(&trace_fsmonitor,\n -+\t\t\t\t \"TRNC kill (%\"PRIu64\",%\"PRIu64\")\",\n ++\t\t\t\t \"Truncate: kill (%\"PRIu64\",%\"PRIu64\")\",\n +\t\t\t\t p->batch_seq_nr, (uint64_t)p->pinned_time);\n +\t}\n +}\n @@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon\n +\t\t\t * obsolete. See if we can truncate the list\n +\t\t\t * and save some memory.\n +\t\t\t */\n -+\t\t\tfsmonitor_batch__truncate(state, batch);\n ++\t\t\twith_lock__truncate_old_batches(state, batch);\n \t\t}\n \t}\n \n 18: c6d5f045fb56 < -: ------------ fsmonitor--daemon:: introduce client delay for testing\n 19: 038b62dc6744 ! 22: bb7b1912bb47 fsmonitor--daemon: use a cookie file to sync with file system\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## builtin/fsmonitor--daemon.c ##\n -@@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n - \treturn 0;\n +@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n + \t}\n }\n \n +enum fsmonitor_cookie_item_result {\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\treturn strcmp(a->name, keydata ? keydata : b->name);\n +}\n +\n -+static enum fsmonitor_cookie_item_result fsmonitor_wait_for_cookie(\n ++static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(\n +\tstruct fsmonitor_daemon_state *state)\n +{\n ++\t/* assert current thread holding state->main_lock */\n ++\n +\tint fd;\n -+\tstruct fsmonitor_cookie_item cookie;\n ++\tstruct fsmonitor_cookie_item *cookie;\n +\tstruct strbuf cookie_pathname = STRBUF_INIT;\n +\tstruct strbuf cookie_filename = STRBUF_INIT;\n -+\tconst char *slash;\n ++\tenum fsmonitor_cookie_item_result result;\n +\tint my_cookie_seq;\n +\n -+\tpthread_mutex_lock(&state->main_lock);\n ++\tCALLOC_ARRAY(cookie, 1);\n +\n +\tmy_cookie_seq = state->cookie_seq++;\n +\n ++\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n ++\n +\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n -+\tstrbuf_addf(&cookie_pathname, \"%i-%i\", getpid(), my_cookie_seq);\n -+\n -+\tslash = find_last_dir_sep(cookie_pathname.buf);\n -+\tif (slash)\n -+\t\tstrbuf_addstr(&cookie_filename, slash + 1);\n -+\telse\n -+\t\tstrbuf_addbuf(&cookie_filename, &cookie_pathname);\n -+\tcookie.name = strbuf_detach(&cookie_filename, NULL);\n -+\tcookie.result = FCIR_INIT;\n -+\t// TODO should we have case-insenstive hash (and in cookie_cmp()) ??\n -+\thashmap_entry_init(&cookie.entry, strhash(cookie.name));\n ++\tstrbuf_addbuf(&cookie_pathname, &cookie_filename);\n +\n -+\t/*\n -+\t * Warning: we are putting the address of a stack variable into a\n -+\t * global hashmap. This feels dodgy. We must ensure that we remove\n -+\t * it before this thread and stack frame returns.\n -+\t */\n -+\thashmap_add(&state->cookies, &cookie.entry);\n ++\tcookie->name = strbuf_detach(&cookie_filename, NULL);\n ++\tcookie->result = FCIR_INIT;\n ++\thashmap_entry_init(&cookie->entry, strhash(cookie->name));\n ++\n ++\thashmap_add(&state->cookies, &cookie->entry);\n +\n +\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n -+\t\t\t cookie.name, cookie_pathname.buf);\n ++\t\t\t cookie->name, cookie_pathname.buf);\n +\n +\t/*\n +\t * Create the cookie file on disk and then wait for a notification\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n +\tif (fd >= 0) {\n +\t\tclose(fd);\n -+\t\tunlink_or_warn(cookie_pathname.buf);\n ++\t\tunlink(cookie_pathname.buf);\n +\n -+\t\twhile (cookie.result == FCIR_INIT)\n ++\t\t/*\n ++\t\t * NEEDSWORK: This is an infinite wait (well, unless another\n ++\t\t * thread sends us an abort). I'd like to change this to\n ++\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n ++\t\t * and let the caller do the trivial response thing.\n ++\t\t */\n ++\t\twhile (cookie->result == FCIR_INIT)\n +\t\t\tpthread_cond_wait(&state->cookies_cond,\n +\t\t\t\t\t &state->main_lock);\n -+\n -+\t\thashmap_remove(&state->cookies, &cookie.entry, NULL);\n +\t} else {\n +\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n -+\t\t\t cookie.name);\n ++\t\t\t cookie->name);\n +\n -+\t\tcookie.result = FCIR_ERROR;\n -+\t\thashmap_remove(&state->cookies, &cookie.entry, NULL);\n ++\t\tcookie->result = FCIR_ERROR;\n +\t}\n +\n -+\tpthread_mutex_unlock(&state->main_lock);\n ++\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n ++\n ++\tresult = cookie->result;\n +\n -+\tfree((char*)cookie.name);\n ++\tfree((char*)cookie->name);\n ++\tfree(cookie);\n +\tstrbuf_release(&cookie_pathname);\n -+\treturn cookie.result;\n ++\n ++\treturn result;\n +}\n +\n +/*\n + * Mark these cookies as _SEEN and wake up the corresponding client threads.\n + */\n -+static void fsmonitor_cookie_mark_seen(struct fsmonitor_daemon_state *state,\n -+\t\t\t\t const struct string_list *cookie_names)\n ++static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,\n ++\t\t\t\t\t const struct string_list *cookie_names)\n +{\n -+\t/* assert state->main_lock */\n ++\t/* assert current thread holding state->main_lock */\n +\n +\tint k;\n +\tint nr_seen = 0;\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +/*\n + * Set _ABORT on all pending cookies and wake up all client threads.\n + */\n -+static void fsmonitor_cookie_abort_all(struct fsmonitor_daemon_state *state)\n ++static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)\n +{\n -+\t/* assert state->main_lock */\n ++\t/* assert current thread holding state->main_lock */\n +\n +\tstruct hashmap_iter iter;\n +\tstruct fsmonitor_cookie_item *cookie;\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__send_flush(void)\n +\t\tpthread_cond_broadcast(&state->cookies_cond);\n +}\n +\n - static int lookup_client_test_delay(void)\n - {\n - \tstatic int delay_ms = -1;\n + /*\n + * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n + * \"token\" as a virtual timestamp. Clients can request a summary of all\n @@ builtin/fsmonitor--daemon.c: static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n @@ builtin/fsmonitor--daemon.c: static void fsmonitor_free_token_data(struct fsmoni\n + * [2] Some of those lost events may have been for cookie files. We\n + * should assume the worst and abort them rather letting them starve.\n + *\n - * If there are no readers of the the current token data series, we\n - * can free it now. Otherwise, let the last reader free it. Either\n - * way, the old token data series is no longer associated with our\n -@@ builtin/fsmonitor--daemon.c: void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n - \t\t\t state->current_token_data->token_id.buf,\n - \t\t\t new_one->token_id.buf);\n + * If there are no concurrent threads readering the current token data\n + * series, we can free it now. Otherwise, let the last reader free\n + * it.\n +@@ builtin/fsmonitor--daemon.c: static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n + \tstate->current_token_data = new_one;\n \n -+\tfsmonitor_cookie_abort_all(state);\n + \tfsmonitor_free_token_data(free_me);\n +\n - \tif (state->current_token_data->client_ref_count == 0)\n - \t\tfree_me = state->current_token_data;\n - \tstate->current_token_data = new_one;\n ++\twith_lock__abort_all_cookies(state);\n + }\n + \n + void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n @@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,\n - \tkh_str_t *shown;\n \tint hash_ret;\n - \tint result;\n + \tint do_trivial = 0;\n + \tint do_flush = 0;\n ++\tint do_cookie = 0;\n +\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n @@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,\n - \t\tgoto send_trivial_response;\n + \t\t */\n + \t\tdo_flush = 1;\n + \t\tdo_trivial = 1;\n ++\t\tdo_cookie = 1;\n + \n + \t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n + \t\t/* assume V1 timestamp or garbage */\n +@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,\n + \t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n + \t\t\t\t command);\n + \t\tdo_trivial = 1;\n ++\t\tdo_cookie = 1;\n + \n + \t} else {\n + \t\t/* We have \"builtin:*\" */\n +@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,\n + \t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n + \t\t\t\t\t command);\n + \t\t\tdo_trivial = 1;\n ++\t\t\tdo_cookie = 1;\n + \n + \t\t} else {\n + \t\t\t/*\n + \t\t\t * We have a V2 valid token:\n + \t\t\t * \"builtin::\"\n + \t\t\t */\n ++\t\t\tdo_cookie = 1;\n + \t\t}\n \t}\n \n -+\tpthread_mutex_unlock(&state->main_lock);\n -+\n +@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,\n + \tif (!state->current_token_data)\n + \t\tBUG(\"fsmonitor state does not have a current token\");\n + \n +\t/*\n -+\t * Write a cookie file inside the directory being watched in an\n -+\t * effort to flush out existing filesystem events that we actually\n -+\t * care about. Suspend this client thread until we see the filesystem\n -+\t * events for this cookie file.\n ++\t * Write a cookie file inside the directory being watched in\n ++\t * an effort to flush out existing filesystem events that we\n ++\t * actually care about. Suspend this client thread until we\n ++\t * see the filesystem events for this cookie file.\n ++\t *\n ++\t * Creating the cookie lets us guarantee that our FS listener\n ++\t * thread has drained the kernel queue and we are caught up\n ++\t * with the kernel.\n ++\t *\n ++\t * If we cannot create the cookie (or otherwise guarantee that\n ++\t * we are caught up), we send a trivial response. We have to\n ++\t * assume that there might be some very, very recent activity\n ++\t * on the FS still in flight.\n +\t */\n -+\tcookie_result = fsmonitor_wait_for_cookie(state);\n -+\tif (cookie_result != FCIR_SEEN) {\n -+\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n -+\t\t cookie_result);\n -+\t\tresult = 0;\n -+\t\tgoto send_trivial_response;\n -+\t}\n -+\n -+\tpthread_mutex_lock(&state->main_lock);\n -+\n -+\tif (strcmp(requested_token_id.buf,\n -+\t\t state->current_token_data->token_id.buf)) {\n -+\t\t/*\n -+\t\t * Ack! The listener thread lost sync with the filesystem\n -+\t\t * and created a new token while we were waiting for the\n -+\t\t * cookie file to be created! Just give up.\n -+\t\t */\n -+\t\tpthread_mutex_unlock(&state->main_lock);\n -+\n -+\t\ttrace_printf_key(&trace_fsmonitor,\n -+\t\t\t\t \"lost filesystem sync\");\n -+\t\tresult = 0;\n -+\t\tgoto send_trivial_response;\n ++\tif (do_cookie) {\n ++\t\tcookie_result = with_lock__wait_for_cookie(state);\n ++\t\tif (cookie_result != FCIR_SEEN) {\n ++\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n ++\t\t\t cookie_result);\n ++\t\t\tdo_trivial = 1;\n ++\t\t}\n +\t}\n +\n - \t/*\n - \t * We're going to hold onto a pointer to the current\n - \t * token-data while we walk the list of batches of files.\n + \tif (do_flush)\n + \t\twith_lock__do_force_resync(state);\n + \n +@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,\n + \treturn result;\n + }\n + \n +-#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n ++#define FSMONITOR_DIR \"fsmonitor--daemon\"\n ++#define FSMONITOR_COOKIE_DIR \"cookies\"\n ++#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n + \n + enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n + \tconst char *rel)\n @@ builtin/fsmonitor--daemon.c: void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n +\tif (cookie_names->nr)\n -+\t\tfsmonitor_cookie_mark_seen(state, cookie_names);\n ++\t\twith_lock__mark_cookies_seen(state, cookie_names);\n +\n \tpthread_mutex_unlock(&state->main_lock);\n }\n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)\n +\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n - \tstate.test_client_delay_ms = lookup_client_test_delay();\n + \n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n +\t/*\n +\t * We will write filesystem syncing cookie files into\n -+\t * /-.\n ++\t * ///-.\n +\t */\n +\tstrbuf_init(&state.path_cookie_prefix, 0);\n +\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n ++\n ++\tstrbuf_addch(&state.path_cookie_prefix, '/');\n ++\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n ++\tmkdir(state.path_cookie_prefix.buf, 0777);\n ++\n ++\tstrbuf_addch(&state.path_cookie_prefix, '/');\n ++\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n ++\tmkdir(state.path_cookie_prefix.buf, 0777);\n ++\n +\tstrbuf_addch(&state.path_cookie_prefix, '/');\n -+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_PREFIX);\n +\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n +\tstrbuf_release(&state.path_cookie_prefix);\n ++\n ++\t/*\n ++\t * NEEDSWORK: Consider \"rm -rf /\"\n ++\t */\n \n \treturn err;\n }\n 20: d699ad597d2c ! 23: 102e17cbc875 fsmonitor: force update index when fsmonitor token advances\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: force update index when fsmonitor token advances\n -\n - Set the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when the\n - fsmonitor response contains a different token to ensure that the index\n - is written to disk.\n -\n - Normally, when the fsmonitor response includes a tracked file, the\n - index is always updated. Similarly, the index might be updated when\n - the response alters the untracked-cache (when enabled). However, in\n - cases where neither of those cause the index to be considered changed,\n - the fsmonitor response is wasted. And subsequent commands will\n - continue to make requests with the same token and if there have not\n - been any changes in the working directory, they will receive the same\n - response.\n -\n - This was observed on Windows after a large checkout. On Windows, the\n - kernel emits events for the files that are changed as they are\n - changed. However, it might delay events for the containing\n - directories until the system is more idle (or someone scans the\n - directory (so it seems)). The first status following a checkout would\n - get the list of files. The subsequent status commands would get the\n - list of directories as the events trickled out. But they would never\n - catch up because the token was not advanced because the index wasn't\n - updated.\n -\n - This list of directories caused `wt_status_collect_untracked()` to\n - unnecessarily spend time actually scanning them during each command.\n + fsmonitor: enhance existing comments\n \n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n \t}\n - \tstrbuf_release(&query_result);\n \n + apply_results:\n +-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n +\t/*\n -+\t * If the fsmonitor response and the subsequent scan of the disk\n -+\t * did not cause the in-memory index to be marked dirty, then force\n -+\t * it so that we advance the fsmonitor token in our extension, so\n -+\t * that future requests don't keep re-requesting the same range.\n ++\t * The response from FSMonitor (excluding the header token) is\n ++\t * either:\n ++\t *\n ++\t * [a] a (possibly empty) list of NUL delimited relative\n ++\t * pathnames of changed paths. This list can contain\n ++\t * files and directories. Directories have a trailing\n ++\t * slash.\n ++\t *\n ++\t * [b] a single '/' to indicate the provider had no\n ++\t * information and that we should consider everything\n ++\t * invalid. We call this a trivial response.\n +\t */\n -+\tif (istate->fsmonitor_last_update &&\n -+\t strcmp(istate->fsmonitor_last_update, last_update_token.buf))\n -+\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n + \tif (query_success && query_result.buf[bol] != '/') {\n +-\t\t/* Mark all entries returned by the monitor as dirty */\n ++\t\t/*\n ++\t\t * Mark all pathnames returned by the monitor as dirty.\n ++\t\t *\n ++\t\t * This updates both the cache-entries and the untracked-cache.\n ++\t\t */\n + \t\tbuf = query_result.buf;\n + \t\tfor (i = bol; i < query_result.len; i++) {\n + \t\t\tif (buf[i] != '\\0')\n +@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \t\tif (istate->untracked)\n + \t\t\tistate->untracked->use_fsmonitor = 1;\n + \t} else {\n +-\n +-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n +-\t\t * if we actually changed entries or not */\n ++\t\t/*\n ++\t\t * We received a trivial response, so invalidate everything.\n ++\t\t *\n ++\t\t * We only want to run the post index changed hook if\n ++\t\t * we've actually changed entries, so keep track if we\n ++\t\t * actually changed entries or not.\n ++\t\t */\n + \t\tint is_cache_changed = 0;\n +-\t\t/* Mark all entries invalid */\n +\n - \t/* Now that we've updated istate, save the last_update_token */\n - \tFREE_AND_NULL(istate->fsmonitor_last_update);\n - \tistate->fsmonitor_last_update = strbuf_detach(&last_update_token, NULL);\n + \t\tfor (i = 0; i < istate->cache_nr; i++) {\n + \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n + \t\t\t\tis_cache_changed = 1;\n +@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \t\t\t}\n + \t\t}\n + \n +-\t\t/* If we're going to check every file, ensure we save the results */\n ++\t\t/*\n ++\t\t * If we're going to check every file, ensure we save\n ++\t\t * the results.\n ++\t\t */\n + \t\tif (is_cache_changed)\n + \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n + \n -: ------------ > 24: 11ea2f97def6 fsmonitor: force update index after large responses\n 21: 8b2280e5c4d2 ! 25: c9159db718a7 t7527: create test for fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +. ./test-lib.sh\n +\n -+# Ask the fsmonitor daemon to insert a little delay before responding to\n -+# client commands like `git status` and `git fsmonitor--daemon --query` to\n -+# allow recent filesystem events to be received by the daemon. This helps\n -+# the CI/PR builds be more stable.\n -+#\n -+# An arbitrary millisecond value.\n -+#\n -+GIT_TEST_FSMONITOR_CLIENT_DELAY=1000\n -+export GIT_TEST_FSMONITOR_CLIENT_DELAY\n -+\n +git version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\" || {\n +\tskip_all=\"The built-in FSMonitor is not supported on this platform\"\n +\ttest_done\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +kill_repo () {\n +\tr=$1\n -+\tgit -C $r fsmonitor--daemon --stop >/dev/null 2>/dev/null\n ++\tgit -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n +\trm -rf $1\n +\treturn 0\n +}\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t\t*) r=\"\";\n +\tesac\n +\n -+\tgit $r fsmonitor--daemon --start || return $?\n -+\tgit $r fsmonitor--daemon --is-running || return $?\n ++\tgit $r fsmonitor--daemon start || return $?\n ++\tgit $r fsmonitor--daemon status || return $?\n +\n +\treturn 0\n +}\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\tgit init test_explicit &&\n +\tstart_daemon test_explicit &&\n +\n -+\tgit -C test_explicit fsmonitor--daemon --stop &&\n -+\ttest_must_fail git -C test_explicit fsmonitor--daemon --is-running\n ++\tgit -C test_explicit fsmonitor--daemon stop &&\n ++\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n +'\n +\n +test_expect_success 'implicit daemon start' '\n +\ttest_when_finished \"kill_repo test_implicit\" &&\n +\n +\tgit init test_implicit &&\n -+\ttest_must_fail git -C test_implicit fsmonitor--daemon --is-running &&\n ++\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n +\n +\t# query will implicitly start the daemon.\n +\t#\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t# implicitly started.)\n +\n +\tGIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n -+\t\tgit -C test_implicit fsmonitor--daemon --query 0 >actual &&\n ++\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n +\tnul_to_q actual.filtered &&\n +\tgrep \"builtin:\" actual.filtered &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tgrep :\\\"query/response-length\\\" .git/trace &&\n +\n -+\tgit -C test_implicit fsmonitor--daemon --is-running &&\n -+\tgit -C test_implicit fsmonitor--daemon --stop &&\n -+\ttest_must_fail git -C test_implicit fsmonitor--daemon --is-running\n ++\tgit -C test_implicit fsmonitor--daemon status &&\n ++\tgit -C test_implicit fsmonitor--daemon stop &&\n ++\ttest_must_fail git -C test_implicit fsmonitor--daemon status\n +'\n +\n +test_expect_success 'implicit daemon stop (delete .git)' '\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\tsleep 1 &&\n +\tmkdir test_implicit_1/.git &&\n +\n -+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon --is-running\n ++\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n +'\n +\n +test_expect_success 'implicit daemon stop (rename .git)' '\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\tsleep 1 &&\n +\tmkdir test_implicit_2/.git &&\n +\n -+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon --is-running\n ++\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n +'\n +\n +test_expect_success 'cannot start multiple daemons' '\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tstart_daemon test_multiple &&\n +\n -+\ttest_must_fail git -C test_multiple fsmonitor--daemon --start 2>actual &&\n ++\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n +\tgrep \"fsmonitor--daemon is already running\" actual &&\n +\n -+\tgit -C test_multiple fsmonitor--daemon --stop &&\n -+\ttest_must_fail git -C test_multiple fsmonitor--daemon --is-running\n ++\tgit -C test_multiple fsmonitor--daemon stop &&\n ++\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n +'\n +\n +test_expect_success 'setup' '\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'update-index implicitly starts daemon' '\n -+\ttest_must_fail git fsmonitor--daemon --is-running &&\n ++\ttest_must_fail git fsmonitor--daemon status &&\n +\n +\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n +\t\tgit update-index --fsmonitor &&\n +\n -+\tgit fsmonitor--daemon --is-running &&\n -+\ttest_might_fail git fsmonitor--daemon --stop &&\n ++\tgit fsmonitor--daemon status &&\n ++\ttest_might_fail git fsmonitor--daemon stop &&\n +\n +\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_1\n +'\n +\n +test_expect_success 'status implicitly starts daemon' '\n -+\ttest_must_fail git fsmonitor--daemon --is-running &&\n ++\ttest_must_fail git fsmonitor--daemon status &&\n +\n +\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n +\t\tgit status >actual &&\n +\n -+\tgit fsmonitor--daemon --is-running &&\n -+\ttest_might_fail git fsmonitor--daemon --stop &&\n ++\tgit fsmonitor--daemon status &&\n ++\ttest_might_fail git fsmonitor--daemon stop &&\n +\n +\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_2\n +'\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +clean_up_repo_and_stop_daemon () {\n +\tgit reset --hard HEAD\n +\tgit clean -fd\n -+\tgit fsmonitor--daemon --stop\n ++\tgit fsmonitor--daemon stop\n +\trm -f .git/trace\n +}\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tedit_files &&\n +\n -+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n +\n +\tgrep \"^event: dir1/modified$\" .git/trace &&\n +\tgrep \"^event: dir2/modified$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tcreate_files &&\n +\n -+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n +\n +\tgrep \"^event: dir1/new$\" .git/trace &&\n +\tgrep \"^event: dir2/new$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tdelete_files &&\n +\n -+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n +\n +\tgrep \"^event: dir1/delete$\" .git/trace &&\n +\tgrep \"^event: dir2/delete$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\trename_files &&\n +\n -+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n +\n +\tgrep \"^event: dir1/rename$\" .git/trace &&\n +\tgrep \"^event: dir2/rename$\" .git/trace &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tmv dirtorename dirrenamed &&\n +\n -+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n +\n +\tgrep \"^event: dirtorename/*$\" .git/trace &&\n +\tgrep \"^event: dirrenamed/*$\" .git/trace\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tfile_to_directory &&\n +\n -+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n +\n +\tgrep \"^event: delete$\" .git/trace &&\n +\tgrep \"^event: delete/new$\" .git/trace\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +\tdirectory_to_file &&\n +\n -+\tgit fsmonitor--daemon --query 0 >/dev/null 2>&1 &&\n ++\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n +\n +\tgrep \"^event: dir1$\" .git/trace\n +'\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +# polling fast enough), we need to discard the cached data (relative to the\n +# current token) and start collecting events under a new token.\n +#\n -+# the 'git fsmonitor--daemon --flush' command can be used to send a \"flush\"\n -+# message to a running daemon and ask it to do a flush/resync.\n ++# the 'test-tool fsmonitor-client flush' command can be used to send a\n ++# \"flush\" message to a running daemon and ask it to do a flush/resync.\n +\n +test_expect_success 'flush cached data' '\n +\ttest_when_finished \"kill_repo test_flush\" &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t# then a few (probably platform-specific number of) events in _1.\n +\t# These should both have the same .\n +\n -+\tgit -C test_flush fsmonitor--daemon --query \"builtin:test_00000001:0\" >actual_0 &&\n ++\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n +\tnul_to_q actual_q0 &&\n +\n +\ttouch test_flush/file_1 &&\n +\ttouch test_flush/file_2 &&\n +\n -+\tgit -C test_flush fsmonitor--daemon --query \"builtin:test_00000001:0\" >actual_1 &&\n ++\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n +\tnul_to_q actual_q1 &&\n +\n +\tgrep \"file_1\" actual_q1 &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t# flush the file data. Then create some events and ensure that the file\n +\t# again appears in the cache. It should have the new .\n +\n -+\tgit -C test_flush fsmonitor--daemon --flush >flush_0 &&\n ++\ttest-tool -C test_flush fsmonitor-client flush >flush_0 &&\n +\tnul_to_q flush_q0 &&\n +\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n +\n -+\tgit -C test_flush fsmonitor--daemon --query \"builtin:test_00000002:0\" >actual_2 &&\n ++\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_2 &&\n +\tnul_to_q actual_q2 &&\n +\n +\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n +\n +\ttouch test_flush/file_3 &&\n +\n -+\tgit -C test_flush fsmonitor--daemon --query \"builtin:test_00000002:0\" >actual_3 &&\n ++\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n +\tnul_to_q actual_q3 &&\n +\n +\tgrep \"file_3\" actual_q3\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t\tstart_daemon wt-secondary\n +\t) &&\n +\n -+\tgit -C wt-secondary fsmonitor--daemon --stop &&\n -+\ttest_must_fail git -C wt-secondary fsmonitor--daemon --is-running\n ++\tgit -C wt-secondary fsmonitor--daemon stop &&\n ++\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n +'\n +\n +test_done\n 22: da5094e52032 ! 26: 5b035c6e0d60 p7519: add fsmonitor--daemon\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"one time repo setup\" '\n \t\t#\n \t\t# Choose integration script based on existence of Watchman.\n \t\t# Fall back to an empty integration script.\n +@@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {\n + }\n + \n + test_fsmonitor_suite() {\n +-\tif test -n \"$INTEGRATION_SCRIPT\"; then\n ++\tif test -n \"$USE_FSMONITOR_DAEMON\"\n ++\tthen\n ++\t\tDESC=\"builtin fsmonitor--daemon\"\n ++\telif test -n \"$INTEGRATION_SCRIPT\"; then\n + \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n + \telse\n + \t\tDESC=\"fsmonitor=disabled\"\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"setup without fsmonitor\" '\n +\tUSE_FSMONITOR_DAEMON=t\n +\n +\ttrace_start fsmonitor--daemon--server\n -+\tgit fsmonitor--daemon --start\n ++\tgit fsmonitor--daemon start\n +\n +\ttrace_start fsmonitor--daemon--client\n +\ttest_expect_success \"setup for fsmonitor--daemon\" 'setup_for_fsmonitor'\n +\ttest_fsmonitor_suite\n +\n -+\tgit fsmonitor--daemon --stop\n ++\tgit fsmonitor--daemon stop\n +\ttrace_stop\n +fi\n +\n 23: 3eafd0b5cb09 ! 27: 1483c68855cb t7527: test status with untracked-cache and fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '\n \n \tgit -c core.useBuiltinFSMonitor= add . &&\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'worktree with .git file' '\n - \ttest_must_fail git -C wt-secondary fsmonitor--daemon --is-running\n + \ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n '\n \n +# TODO Repeat one of the \"edit\" tests on wt-secondary and confirm that\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'worktree with .git file' '\n +test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n +\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n +\tgit update-index --no-fsmonitor &&\n -+\ttest_might_fail git fsmonitor--daemon --stop\n ++\ttest_might_fail git fsmonitor--daemon stop\n +'\n +\n +matrix_clean_up_repo () {\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'worktree with .git file' '\n +\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n +\t\t\t\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n +\t\t\t\tgit update-index --no-fsmonitor &&\n -+\t\t\t\ttest_might_fail git fsmonitor--daemon --stop 2>/dev/null\n ++\t\t\t\ttest_might_fail git fsmonitor--daemon stop 2>/dev/null\n +\t\t\t'\n +\t\telse\n +\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n +\t\t\t\tgit config core.useBuiltinFSMonitor true &&\n -+\t\t\t\tgit fsmonitor--daemon --start &&\n ++\t\t\t\tgit fsmonitor--daemon start &&\n +\t\t\t\tgit update-index --fsmonitor\n +\t\t\t'\n +\t\tfi\n -: ------------ > 28: 96a3eab819f4 t/perf: avoid copying builtin fsmonitor files into test repo\n\n-- \ngitgitgadget\n"},{"id":"425311","messageId":"7e097cebc14328bc1ec6e30fc164270e0889ff06.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 07/28] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:46Z","receivedAt":"2021-05-22T13:57:26Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon.\n\nDefine a new config setting `core.useBuiltinFSMonitor` to enable the\nbuiltin file system monitor.\n\nThe `core.fsmonitor` setting has already been defined as a HOOK\npathname. Historically, this has been set to a HOOK script that will\ntalk with Watchman. For compatibility reasons, we do not want to\noverload that definition (and cause problems if users have multiple\nversions of Git installed).\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n config.c | 5 +++++\n fsmonitor.c | 43 +++++++++++++++++++++++++++++++++++++++++++\n repo-settings.c | 3 +++\n repository.h | 2 ++\n 4 files changed, 53 insertions(+)\n\ndiff --git a/config.c b/config.c\nindex a896f44cba1f..c82f40c22b43 100644\n--- a/config.c\n+++ b/config.c\n@@ -2501,6 +2501,11 @@ int git_config_get_max_percent_split_change(void)\n \n int repo_config_get_fsmonitor(struct repository *r)\n {\n+\tif (r->settings.use_builtin_fsmonitor > 0) {\n+\t\tcore_fsmonitor = \"(built-in daemon)\";\n+\t\treturn 1;\n+\t}\n+\n \tif (repo_config_get_pathname(r, \"core.fsmonitor\", &core_fsmonitor))\n \t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n \ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 9c9b2abc9414..c6d3c34ad78e 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -231,6 +232,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \n void refresh_fsmonitor(struct index_state *istate)\n {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n \tstruct strbuf query_result = STRBUF_INIT;\n \tint query_success = 0, hook_version = -1;\n \tsize_t bol = 0; /* beginning of line */\n@@ -247,6 +249,46 @@ void refresh_fsmonitor(struct index_state *istate)\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (r->settings.use_builtin_fsmonitor > 0) {\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\t/*\n+\t\t * Regardless of whether we successfully talked to a\n+\t\t * fsmonitor daemon or not, we skip over and do not\n+\t\t * try to use the hook. The \"core.useBuiltinFSMonitor\"\n+\t\t * config setting ALWAYS overrides the \"core.fsmonitor\"\n+\t\t * hook setting.\n+\t\t */\n+\t\tgoto apply_results;\n+\t}\n+\n \t/*\n \t * This could be racy so save the date/time now and query_fsmonitor\n \t * should be inclusive to ensure we don't miss potential changes.\n@@ -301,6 +343,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n \tif (query_success && query_result.buf[bol] != '/') {\n \t\t/* Mark all entries returned by the monitor as dirty */\ndiff --git a/repo-settings.c b/repo-settings.c\nindex f7fff0f5ab83..93aab92ff164 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r)\n \t\tr->settings.core_multi_pack_index = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n \n+\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n+\t\tr->settings.use_builtin_fsmonitor = 1;\n+\n \tif (!repo_config_get_bool(r, \"feature.manyfiles\", &value) && value) {\n \t\tUPDATE_DEFAULT_BOOL(r->settings.index_version, 4);\n \t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);\ndiff --git a/repository.h b/repository.h\nindex b385ca3c94b6..d6e7f61f9cf7 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -29,6 +29,8 @@ enum fetch_negotiation_setting {\n struct repo_settings {\n \tint initialized;\n \n+\tint use_builtin_fsmonitor;\n+\n \tint core_commit_graph;\n \tint commit_graph_read_changed_paths;\n \tint gc_write_commit_graph;\n-- \ngitgitgadget\n\n"},{"id":"425310","messageId":"4f401310539ea4bebf662c7ddf82bb74e9f1d8c8.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 09/28] fsmonitor--daemon: implement client command options","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:48Z","receivedAt":"2021-05-22T13:57:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `stop` and `status` client commands to control and query the\nstatus of a `fsmonitor--daemon` server process (and implicitly start a\nserver process if necessary).\n\nLater commits will implement the actual server and monitor the file\nsystem.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 49 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 49 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex df2bad531118..16ff68b65407 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,10 +7,53 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon stop\"),\n+\tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n+ * and wait for it to shutdown.\n+ */\n+static int do_as_client__send_stop(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"quit\", &answer);\n+\n+\t/* The quit command does not return any response data. */\n+\tstrbuf_release(&answer);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\twhile (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tsleep_millisec(50);\n+\ttrace2_region_leave(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\n+\treturn 0;\n+}\n+\n+static int do_as_client__status(void)\n+{\n+\tenum ipc_active_state state = fsmonitor_ipc__get_state();\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tprintf(_(\"The built-in file system monitor is active\\n\"));\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tprintf(_(\"The built-in file system monitor is not active\\n\"));\n+\t\treturn 1;\n+\t}\n+}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n@@ -35,6 +78,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n \n+\tif (!strcmp(subcmd, \"stop\"))\n+\t\treturn !!do_as_client__send_stop();\n+\n+\tif (!strcmp(subcmd, \"status\"))\n+\t\treturn !!do_as_client__status();\n+\n \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"425312","messageId":"d21af7ff842c2a2642ae42df7b972a8817b8a3e8.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 10/28] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:49Z","receivedAt":"2021-05-22T13:57:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate an IPC client to send query and flush commands to the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n t/helper/test-fsmonitor-client.c | 125 +++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 128 insertions(+)\n create mode 100644 t/helper/test-fsmonitor-client.c\n\ndiff --git a/Makefile b/Makefile\nindex 74673acc9833..80059032c4e3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\n TEST_BUILTINS_OBJS += test-hash-speed.o\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nnew file mode 100644\nindex 000000000000..4961f28e3e02\n--- /dev/null\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -0,0 +1,125 @@\n+/*\n+ * test-fsmonitor-client.c: client code to send commands/requests to\n+ * a `git fsmonitor--daemon` daemon.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+//#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+//#include \"compat/fsmonitor/fsmonitor-fs-listen.h\"\n+//#include \"fsmonitor--daemon.h\"\n+//#include \"simple-ipc.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tdie(\"fsmonitor--daemon not available on this platform\");\n+}\n+#else\n+\n+/*\n+ * Read the `.git/index` to get the last token written to the\n+ * FSMonitor Index Extension.\n+ */\n+static const char *get_token_from_index(void)\n+{\n+\tstruct index_state *istate = the_repository->index;\n+\n+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update)\n+\t\tdie(\"index file does not have fsmonitor extension\");\n+\n+\treturn istate->fsmonitor_last_update;\n+}\n+\n+/*\n+ * Send an IPC query to a `git-fsmonitor--daemon` daemon and\n+ * ask for the changes since the given token or from the last\n+ * token in the index extension.\n+ *\n+ * This will implicitly start a daemon process if necessary. The\n+ * daemon process will persist after we exit.\n+ */\n+static int do_send_query(const char *token)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\n+\tret = fsmonitor_ipc__send_query(token, &answer);\n+\tif (ret < 0)\n+\t\tdie(_(\"could not query fsmonitor--daemon\"));\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n+ * and tell it to flush its cache.\n+ *\n+ * This feature is primarily used by the test suite to simulate a loss of\n+ * sync with the filesystem where we miss kernel events.\n+ */\n+static int do_send_flush(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tconst char *subcmd;\n+\tconst char *token = NULL;\n+\n+\tconst char * const fsmonitor_client_usage[] = {\n+\t\tN_(\"test-helper fsmonitor-client query []\"),\n+\t\tN_(\"test-helper fsmonitor-client flush\"),\n+\t\tNULL,\n+\t};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING(0, \"token\", &token, N_(\"token\"),\n+\t\t\t N_(\"command token to send to the server\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc < 2)\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tsubcmd = argv[1];\n+\targv--;\n+\targc++;\n+\n+\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n+\n+\tsetup_git_directory();\n+\n+\tif (!strcmp(subcmd, \"query\"))\n+\t\treturn !!do_send_query(token);\n+\n+\tif (!strcmp(subcmd, \"flush\"))\n+\t\treturn !!do_send_flush();\n+\n+\tdie(\"Unhandled subcommand: '%s'\", subcmd);\n+}\n+#endif\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 25c6a37e93e5..b15d328f9a41 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\n \t{ \"hashmap\", cmd__hashmap },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex f03c5988b20c..a8e96b97c419 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -21,6 +21,7 @@ int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"425313","messageId":"2aa85151f03c31989e3b11979711220d05f42b1a.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 12/28] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:51Z","receivedAt":"2021-05-22T13:57:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty implementation of fsmonitor--daemon\nbackend for MacOS.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++\n config.mak.uname | 2 ++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 25 insertions(+)\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\nnew file mode 100644\nindex 000000000000..b91058d1c4f8\n--- /dev/null\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n@@ -0,0 +1,20 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-fs-listen.h\"\n+\n+int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex fcd88b60b14a..394355463e1e 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin)\n \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n \t\tendif\n \tendif\n+\tFSMONITOR_DAEMON_BACKEND = macos\n+\tBASIC_LDFLAGS += -framework CoreServices\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 4e812462d955..22dec4600431 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -259,6 +259,9 @@ endif()\n if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \tlist(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c)\n+elseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n+\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\tlist(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-macos.c)\n endif()\n \n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n-- \ngitgitgadget\n\n"},{"id":"425315","messageId":"49f9e2e3d49ce6e7b56839bf44535f271216abeb.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 11/28] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:50Z","receivedAt":"2021-05-22T13:57:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty backend for fsmonitor--daemon on Windows.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 ++++++\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++\n compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++\n config.mak.uname | 2 +\n contrib/buildsystems/CMakeLists.txt | 5 ++\n 5 files changed, 90 insertions(+)\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n\ndiff --git a/Makefile b/Makefile\nindex 80059032c4e3..3f31adfd135c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -467,6 +467,11 @@ all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n+# If your platform supports a built-in fsmonitor backend, set\n+# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n+# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n+# `fsmonitor_fs_listen__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1906,6 +1911,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n \tCOMPAT_OBJS += compat/access.o\n endif\n \n+ifdef FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2765,6 +2775,9 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n+ifdef FSMONITOR_DAEMON_BACKEND\n+\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\nnew file mode 100644\nindex 000000000000..880446b49e35\n--- /dev/null\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-fs-listen.h\"\n+\n+void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen.h b/compat/fsmonitor/fsmonitor-fs-listen.h\nnew file mode 100644\nindex 000000000000..c7b5776b3b60\n--- /dev/null\n+++ b/compat/fsmonitor/fsmonitor-fs-listen.h\n@@ -0,0 +1,49 @@\n+#ifndef FSMONITOR_FS_LISTEN_H\n+#define FSMONITOR_FS_LISTEN_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * fsmonitor_fs_listener thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread AFTER joining the listener.\n+ */\n+void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to watch for\n+ * filesystem events. This will run in the fsmonitor_fs_listen thread.\n+ *\n+ * It should call `ipc_server_stop_async()` if the listener thread\n+ * prematurely terminates (because of a filesystem error or if it\n+ * detects that the .git directory has been deleted). (It should NOT\n+ * do so if the listener thread receives a normal shutdown signal from\n+ * the IPC layer.)\n+ *\n+ * It should set `state->error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the fsmonitor listener thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_FS_LISTEN_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cb443b4e023a..fcd88b60b14a 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -420,6 +420,7 @@ ifeq ($(uname_S),Windows)\n \t# so we don't need this:\n \t#\n \t# SNPRINTF_RETURNS_BOGUS = YesPlease\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -598,6 +599,7 @@ ifneq (,$(findstring MINGW,$(uname_S)))\n \tNO_STRTOUMAX = YesPlease\n \tNO_MKDTEMP = YesPlease\n \tNO_SVN_TESTS = YesPlease\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 75ed198a6a36..4e812462d955 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -256,6 +256,11 @@ else()\n \tlist(APPEND compat_SOURCES compat/simple-ipc/ipc-shared.c compat/simple-ipc/ipc-unix-socket.c)\n endif()\n \n+if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n+\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\tlist(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c)\n+endif()\n+\n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n \n #header checks\n-- \ngitgitgadget\n\n"},{"id":"425316","messageId":"2aa05ad5c67f2ce93b97c306a9723afeabd64c8c.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 13/28] fsmonitor--daemon: implement daemon command options","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:52Z","receivedAt":"2021-05-22T13:57:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `run` and `start` commands to try to\nbegin listening for file system events.\n\nThis version defines the thread structure with a single\nfsmonitor_fs_listen thread to watch for file system events\nand a simple IPC thread pool to wait for connections from\nGit clients over a well-known named pipe or Unix domain\nsocket.\n\nThis version does not actually do anything yet because the\nbackends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 390 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 34 ++++\n 2 files changed, 423 insertions(+), 1 deletion(-)\n create mode 100644 fsmonitor--daemon.h\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 16ff68b65407..85f99dba861f 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,16 +3,52 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsmonitor-fs-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon start []\"),\n+\tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Global state loaded from config.\n+ */\n+#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n+static int fsmonitor__ipc_threads = 8;\n+\n+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n+static int fsmonitor__start_timeout_sec = 60;\n+\n+static int fsmonitor_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 1)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__IPC_THREADS, i);\n+\t\tfsmonitor__ipc_threads = i;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n+\t\tfsmonitor__start_timeout_sec = i;\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n /*\n * Acting as a CLIENT.\n *\n@@ -55,11 +91,354 @@ static int do_as_client__status(void)\n \t}\n }\n \n+static ipc_server_application_cb handle_client;\n+\n+static int handle_client(void *data,\n+\t\t\t const char *command, size_t command_len,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\t/* struct fsmonitor_daemon_state *state = data; */\n+\tint result;\n+\n+\t/*\n+\t * The Simple IPC API now supports {char*, len} arguments, but\n+\t * FSMonitor always uses proper null-terminated strings, so\n+\t * we can ignore the command_len argument. (Trust, but verify.)\n+\t */\n+\tif (command_len != strlen(command))\n+\t\tBUG(\"FSMonitor assumes text messages\");\n+\n+\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n+\n+\tresult = 0; /* TODO Do something here. */\n+\n+\ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n+\n+\treturn result;\n+}\n+\n+static void *fsmonitor_fs_listen__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-listen\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Watching: worktree '%s'\",\n+\t\t\t state->path_worktree_watch.buf);\n+\tif (state->nr_paths_watching > 1)\n+\t\ttrace_printf_key(&trace_fsmonitor, \"Watching: gitdir '%s'\",\n+\t\t\t\t state->path_gitdir_watch.buf);\n+\n+\tfsmonitor_fs_listen__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct ipc_server_opts ipc_opts = {\n+\t\t.nr_threads = fsmonitor__ipc_threads,\n+\n+\t\t/*\n+\t\t * We know that there are no other active threads yet,\n+\t\t * so we can let the IPC layer temporarily chdir() if\n+\t\t * it needs to when creating the server side of the\n+\t\t * Unix domain socket.\n+\t\t */\n+\t\t.uds_disallow_chdir = 0\n+\t};\n+\n+\t/*\n+\t * Start the IPC thread pool before the we've started the file\n+\t * system event listener thread so that we have the IPC handle\n+\t * before we need it.\n+\t */\n+\tif (ipc_server_run_async(&state->ipc_server_data,\n+\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t handle_client, state))\n+\t\treturn error(_(\"could not start IPC thread pool\"));\n+\n+\t/*\n+\t * Start the fsmonitor listener thread to collect filesystem\n+\t * events.\n+\t */\n+\tif (pthread_create(&state->listener_thread, NULL,\n+\t\t\t fsmonitor_fs_listen__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\tipc_server_await(state->ipc_server_data);\n+\n+\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t}\n+\n+\t/*\n+\t * The daemon is now fully functional in background threads.\n+\t * Wait for the IPC thread pool to shutdown (whether by client\n+\t * request or from filesystem activity).\n+\t */\n+\tipc_server_await(state->ipc_server_data);\n+\n+\t/*\n+\t * The fsmonitor listener thread may have received a shutdown\n+\t * event from the IPC thread pool, but it doesn't hurt to tell\n+\t * it again. And wait for it to shutdown.\n+\t */\n+\tfsmonitor_fs_listen__stop_async(state);\n+\tpthread_join(state->listener_thread, NULL);\n+\n+\treturn state->error_code;\n+}\n+\n+static int fsmonitor_run_daemon(void)\n+{\n+\tstruct fsmonitor_daemon_state state;\n+\tint err;\n+\n+\tmemset(&state, 0, sizeof(state));\n+\n+\tpthread_mutex_init(&state.main_lock, NULL);\n+\tstate.error_code = 0;\n+\tstate.current_token_data = NULL;\n+\n+\t/* Prepare to (recursively) watch the directory. */\n+\tstrbuf_init(&state.path_worktree_watch, 0);\n+\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n+\tstate.nr_paths_watching = 1;\n+\n+\t/*\n+\t * We create/delete cookie files inside the .git directory to\n+\t * help us keep sync with the file system. If \".git\" is not a\n+\t * directory, then is not inside the cone of\n+\t * , so set up a second watch for it.\n+\t */\n+\tstrbuf_init(&state.path_gitdir_watch, 0);\n+\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n+\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n+\tif (!is_directory(state.path_gitdir_watch.buf)) {\n+\t\tstrbuf_reset(&state.path_gitdir_watch);\n+\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n+\t\tstate.nr_paths_watching = 2;\n+\t}\n+\n+\t/*\n+\t * Confirm that we can create platform-specific resources for the\n+\t * filesystem listener before we bother starting all the threads.\n+\t */\n+\tif (fsmonitor_fs_listen__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize listener thread\"));\n+\t\tgoto done;\n+\t}\n+\n+\terr = fsmonitor_run_daemon_1(&state);\n+\n+done:\n+\tpthread_mutex_destroy(&state.main_lock);\n+\tfsmonitor_fs_listen__dtor(&state);\n+\n+\tipc_server_free(state.ipc_server_data);\n+\n+\tstrbuf_release(&state.path_worktree_watch);\n+\tstrbuf_release(&state.path_gitdir_watch);\n+\n+\treturn err;\n+}\n+\n+static int try_to_run_foreground_daemon(void)\n+{\n+\t/*\n+\t * Technically, we don't need to probe for an existing daemon\n+\t * process, since we could just call `fsmonitor_run_daemon()`\n+\t * and let it fail if the pipe/socket is busy.\n+\t *\n+\t * However, this method gives us a nicer error message for a\n+\t * common error case.\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running.\");\n+\n+\treturn !!fsmonitor_run_daemon();\n+}\n+\n+#ifndef GIT_WINDOWS_NATIVE\n+/*\n+ * This is adapted from `daemonize()`. Use `fork()` to directly create\n+ * and run the daemon in a child process. The fork-parent returns the\n+ * child PID so that we can wait for the child to startup before exiting.\n+ */\n+static int spawn_background_fsmonitor_daemon(pid_t *pid)\n+{\n+\t*pid = fork();\n+\n+\tswitch (*pid) {\n+\tcase 0:\n+\t\tif (setsid() == -1)\n+\t\t\terror_errno(_(\"setsid failed\"));\n+\t\tclose(0);\n+\t\tclose(1);\n+\t\tclose(2);\n+\t\tsanitize_stdfds();\n+\n+\t\treturn !!fsmonitor_run_daemon();\n+\n+\tcase -1:\n+\t\treturn error_errno(_(\"could not spawn fsmonitor--daemon in the background\"));\n+\n+\tdefault:\n+\t\treturn 0;\n+\t}\n+}\n+#else\n+/*\n+ * Conceptually like `daemonize()` but different because Windows does not\n+ * have `fork(2)`. Spawn a normal Windows child process but without the\n+ * limitations of `start_command()` and `finish_command()`.\n+ */\n+static int spawn_background_fsmonitor_daemon(pid_t *pid)\n+{\n+\tchar git_exe[MAX_PATH];\n+\tstruct strvec args = STRVEC_INIT;\n+\tint in, out;\n+\n+\tGetModuleFileNameA(NULL, git_exe, MAX_PATH);\n+\n+\tin = open(\"/dev/null\", O_RDONLY);\n+\tout = open(\"/dev/null\", O_WRONLY);\n+\n+\tstrvec_push(&args, git_exe);\n+\tstrvec_push(&args, \"fsmonitor--daemon\");\n+\tstrvec_push(&args, \"run\");\n+\n+\t*pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);\n+\tclose(in);\n+\tclose(out);\n+\n+\tstrvec_clear(&args);\n+\n+\tif (*pid < 0)\n+\t\treturn error(_(\"could not spawn fsmonitor--daemon in the background\"));\n+\n+\treturn 0;\n+}\n+#endif\n+\n+/*\n+ * This is adapted from `wait_or_whine()`. Watch the child process and\n+ * let it get started and begin listening for requests on the socket\n+ * before reporting our success.\n+ */\n+static int wait_for_background_startup(pid_t pid_child)\n+{\n+\tint status;\n+\tpid_t pid_seen;\n+\tenum ipc_active_state s;\n+\ttime_t time_limit, now;\n+\n+\ttime(&time_limit);\n+\ttime_limit += fsmonitor__start_timeout_sec;\n+\n+\tfor (;;) {\n+\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n+\n+\t\tif (pid_seen == -1)\n+\t\t\treturn error_errno(_(\"waitpid failed\"));\n+\t\telse if (pid_seen == 0) {\n+\t\t\t/*\n+\t\t\t * The child is still running (this should be\n+\t\t\t * the normal case). Try to connect to it on\n+\t\t\t * the socket and see if it is ready for\n+\t\t\t * business.\n+\t\t\t *\n+\t\t\t * If there is another daemon already running,\n+\t\t\t * our child will fail to start (possibly\n+\t\t\t * after a timeout on the lock), but we don't\n+\t\t\t * care (who responds) if the socket is live.\n+\t\t\t */\n+\t\t\ts = fsmonitor_ipc__get_state();\n+\t\t\tif (s == IPC_STATE__LISTENING)\n+\t\t\t\treturn 0;\n+\n+\t\t\ttime(&now);\n+\t\t\tif (now > time_limit)\n+\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n+\t\t} else if (pid_seen == pid_child) {\n+\t\t\t/*\n+\t\t\t * The new child daemon process shutdown while\n+\t\t\t * it was starting up, so it is not listening\n+\t\t\t * on the socket.\n+\t\t\t *\n+\t\t\t * Try to ping the socket in the odd chance\n+\t\t\t * that another daemon started (or was already\n+\t\t\t * running) while our child was starting.\n+\t\t\t *\n+\t\t\t * Again, we don't care who services the socket.\n+\t\t\t */\n+\t\t\ts = fsmonitor_ipc__get_state();\n+\t\t\tif (s == IPC_STATE__LISTENING)\n+\t\t\t\treturn 0;\n+\n+\t\t\t/*\n+\t\t\t * We don't care about the WEXITSTATUS() nor\n+\t\t\t * any of the WIF*(status) values because\n+\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n+\t\t\t * trick on all function return values.\n+\t\t\t *\n+\t\t\t * So it is sufficient to just report the\n+\t\t\t * early shutdown as an error.\n+\t\t\t */\n+\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n+\t\t} else\n+\t\t\treturn error(_(\"waitpid is confused\"));\n+\t}\n+}\n+\n+static int try_to_start_background_daemon(void)\n+{\n+\tpid_t pid_child;\n+\tint ret;\n+\n+\t/*\n+\t * Before we try to create a background daemon process, see\n+\t * if a daemon process is already listening. This makes it\n+\t * easier for us to report an already-listening error to the\n+\t * console, since our spawn/daemon can only report the success\n+\t * of creating the background process (and not whether it\n+\t * immediately exited).\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running.\");\n+\n+\t/*\n+\t * Run the actual daemon in a background process.\n+\t */\n+\tret = spawn_background_fsmonitor_daemon(&pid_child);\n+\tif (pid_child <= 0)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Wait (with timeout) for the background child process get\n+\t * started and begin listening on the socket/pipe. This makes\n+\t * the \"start\" command more synchronous and more reliable in\n+\t * tests.\n+\t */\n+\tret = wait_for_background_startup(pid_child);\n+\n+\treturn ret;\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n \n \tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"ipc-threads\",\n+\t\t\t &fsmonitor__ipc_threads,\n+\t\t\t N_(\"use ipc worker threads\")),\n+\t\tOPT_INTEGER(0, \"start-timeout\",\n+\t\t\t &fsmonitor__start_timeout_sec,\n+\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -69,7 +448,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(fsmonitor_config, NULL);\n \n \tsubcmd = argv[1];\n \targv--;\n@@ -77,6 +456,15 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\tif (fsmonitor__ipc_threads < 1)\n+\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n+\t\t fsmonitor__ipc_threads);\n+\n+\tif (!strcmp(subcmd, \"start\"))\n+\t\treturn !!try_to_start_background_daemon();\n+\n+\tif (!strcmp(subcmd, \"run\"))\n+\t\treturn !!try_to_run_foreground_daemon();\n \n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nnew file mode 100644\nindex 000000000000..3009c1a83de7\n--- /dev/null\n+++ b/fsmonitor--daemon.h\n@@ -0,0 +1,34 @@\n+#ifndef FSMONITOR_DAEMON_H\n+#define FSMONITOR_DAEMON_H\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+#include \"thread-utils.h\"\n+\n+struct fsmonitor_batch;\n+struct fsmonitor_token_data;\n+\n+struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+\n+struct fsmonitor_daemon_state {\n+\tpthread_t listener_thread;\n+\tpthread_mutex_t main_lock;\n+\n+\tstruct strbuf path_worktree_watch;\n+\tstruct strbuf path_gitdir_watch;\n+\tint nr_paths_watching;\n+\n+\tstruct fsmonitor_token_data *current_token_data;\n+\n+\tint error_code;\n+\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\n+\tstruct ipc_server_data *ipc_server_data;\n+};\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"425317","messageId":"d5ababfd03e9bd799664cc33d4564e332811c0d4.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 14/28] fsmonitor--daemon: add pathname classification","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:53Z","receivedAt":"2021-05-22T13:57:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to classify relative and absolute\npathnames and decide how they should be handled. This will\nbe used by the platform-specific backend to respond to each\nfilesystem event.\n\nWhen we register for filesystem notifications on a directory,\nwe get events for everything (recursively) in the directory.\nWe want to report to clients changes to tracked and untracked\npaths within the working directory. We do not want to report\nchanges within the .git directory, for example.\n\nThis classification will be used in a later commit by the\ndifferent backends to classify paths as events are received.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 81 +++++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 61 ++++++++++++++++++++++++++++\n 2 files changed, 142 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 85f99dba861f..d6e35ded9f68 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -119,6 +119,87 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n+#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *rel)\n+{\n+\tif (fspathncmp(rel, \".git\", 4))\n+\t\treturn IS_WORKDIR_PATH;\n+\trel += 4;\n+\n+\tif (!*rel)\n+\t\treturn IS_DOT_GIT;\n+\tif (*rel != '/')\n+\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n+\trel++;\n+\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_DOT_GIT;\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *rel)\n+{\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_GITDIR;\n+}\n+\n+static enum fsmonitor_path_type try_classify_workdir_abs_path(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\n+\tif (fspathncmp(path, state->path_worktree_watch.buf,\n+\t\t state->path_worktree_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_worktree_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_workdir_relative(rel);\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\tenum fsmonitor_path_type t;\n+\n+\tt = try_classify_workdir_abs_path(state, path);\n+\tif (state->nr_paths_watching == 1)\n+\t\treturn t;\n+\tif (t != IS_OUTSIDE_CONE)\n+\t\treturn t;\n+\n+\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n+\t\t state->path_gitdir_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_gitdir_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_GITDIR; /* it is the exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_gitdir_relative(rel);\n+}\n+\n static void *fsmonitor_fs_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 3009c1a83de7..7bbb3a27a1ce 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -30,5 +30,66 @@ struct fsmonitor_daemon_state {\n \tstruct ipc_server_data *ipc_server_data;\n };\n \n+/*\n+ * Pathname classifications.\n+ *\n+ * The daemon classifies the pathnames that it receives from file\n+ * system notification events into the following categories and uses\n+ * that to decide whether clients are told about them. (And to watch\n+ * for file system synchronization events.)\n+ *\n+ * The client should only care about paths within the working\n+ * directory proper (inside the working directory and not \".git\" nor\n+ * inside of \".git/\"). That is, the client has read the index and is\n+ * asking for a list of any paths in the working directory that have\n+ * been modified since the last token. The client does not care about\n+ * file system changes within the .git directory (such as new loose\n+ * objects or packfiles). So the client will only receive paths that\n+ * are classified as IS_WORKDIR_PATH.\n+ *\n+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n+ * exact \".git\" directory or GITDIR. If the daemon receives a delete\n+ * event for either of these directories, it will automatically\n+ * shutdown, for example.\n+ *\n+ * Note that the daemon DOES NOT explicitly watch nor special case the\n+ * \".git/index\" file. The daemon does not read the index and does not\n+ * have any internal index-relative state. The daemon only collects\n+ * the set of modified paths within the working directory.\n+ */\n+enum fsmonitor_path_type {\n+\tIS_WORKDIR_PATH = 0,\n+\n+\tIS_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n+\n+\tIS_GITDIR,\n+\tIS_INSIDE_GITDIR,\n+\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n+\n+\tIS_OUTSIDE_CONE,\n+};\n+\n+/*\n+ * Classify a pathname relative to the root of the working directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify a pathname relative to a that is external to the\n+ * worktree directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify an absolute pathname received from a filesystem event.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"425318","messageId":"9ea4b04b8215b1e397623ea4c63e6af3655c74d3.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 17/28] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:56Z","receivedAt":"2021-05-22T13:57:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the win32 backend to register a watch on the working tree\nroot directory (recursively). Also watch the if it is\nnot inside the working tree. And to collect path change notifications\ninto batches and publish.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 532 +++++++++++++++++++\n 1 file changed, 532 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\nindex 880446b49e35..ba087b292df6 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n@@ -2,20 +2,552 @@\n #include \"config.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-fs-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+/*\n+ * The documentation of ReadDirectoryChangesW() states that the maximum\n+ * buffer size is 64K when the monitored directory is remote.\n+ *\n+ * Larger buffers may be used when the monitored directory is local and\n+ * will help us receive events faster from the kernel and avoid dropped\n+ * events.\n+ *\n+ * So we try to use a very large buffer and silently fallback to 64K if\n+ * we get an error.\n+ */\n+#define MAX_RDCW_BUF_FALLBACK (65536)\n+#define MAX_RDCW_BUF (65536 * 8)\n+\n+struct one_watch\n+{\n+\tchar buffer[MAX_RDCW_BUF];\n+\tDWORD buf_len;\n+\tDWORD count;\n+\n+\tstruct strbuf path;\n+\tHANDLE hDir;\n+\tHANDLE hEvent;\n+\tOVERLAPPED overlapped;\n+\n+\t/*\n+\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n+\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n+\t */\n+\tBOOL is_active;\n+};\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tstruct one_watch *watch_worktree;\n+\tstruct one_watch *watch_gitdir;\n+\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hListener[3]; /* we don't own these handles */\n+#define LISTENER_SHUTDOWN 0\n+#define LISTENER_HAVE_DATA_WORKTREE 1\n+#define LISTENER_HAVE_DATA_GITDIR 2\n+\tint nr_listener_handles;\n+};\n+\n+/*\n+ * Convert the WCHAR path from the notification into UTF8 and\n+ * then normalize it.\n+ */\n+static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+\t\t\t\t struct strbuf *normalized_path)\n+{\n+\tint reserve;\n+\tint len = 0;\n+\n+\tstrbuf_reset(normalized_path);\n+\tif (!info->FileNameLength)\n+\t\tgoto normalize;\n+\n+\t/*\n+\t * Pre-reserve enough space in the UTF8 buffer for\n+\t * each Unicode WCHAR character to be mapped into a\n+\t * sequence of 2 UTF8 characters. That should let us\n+\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n+\t */\n+\treserve = info->FileNameLength + 1;\n+\tstrbuf_grow(normalized_path, reserve);\n+\n+\tfor (;;) {\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n+\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t\t\t normalized_path->buf,\n+\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n+\t\t\t\t\t NULL, NULL);\n+\t\tif (len > 0)\n+\t\t\tgoto normalize;\n+\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n+\t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n+\t\t\t GetLastError(),\n+\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n+\t\t\t info->FileName);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tstrbuf_grow(normalized_path,\n+\t\t\t strbuf_avail(normalized_path) + reserve);\n+\t}\n+\n+normalize:\n+\tstrbuf_setlen(normalized_path, len);\n+\treturn strbuf_normalize_path(normalized_path);\n+}\n \n void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+}\n+\n+static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const char *path)\n+{\n+\tstruct one_watch *watch = NULL;\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileA(path,\n+\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n+\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n+\t\t\t NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n+\t\t GetLastError(), path);\n+\t\treturn NULL;\n+\t}\n+\n+\tCALLOC_ARRAY(watch, 1);\n+\n+\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n+\n+\tstrbuf_init(&watch->path, 0);\n+\tstrbuf_addstr(&watch->path, path);\n+\n+\twatch->hDir = hDir;\n+\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\treturn watch;\n+}\n+\n+static void destroy_watch(struct one_watch *watch)\n+{\n+\tif (!watch)\n+\t\treturn;\n+\n+\tstrbuf_release(&watch->path);\n+\tif (watch->hDir != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hDir);\n+\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hEvent);\n+\n+\tfree(watch);\n+}\n+\n+static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+\t\t\t struct one_watch *watch)\n+{\n+\tDWORD dwNotifyFilter =\n+\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n+\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n+\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n+\t\tFILE_NOTIFY_CHANGE_SIZE |\n+\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n+\t\tFILE_NOTIFY_CHANGE_CREATION;\n+\n+\tResetEvent(watch->hEvent);\n+\n+\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n+\twatch->overlapped.hEvent = watch->hEvent;\n+\n+start_watch:\n+\t/*\n+\t * Queue an async call using Overlapped IO. This returns immediately.\n+\t * Our event handle will be signalled when the real result is available.\n+\t *\n+\t * The return value here just means that we successfully queued it.\n+\t * We won't know if the Read...() actually produces data until later.\n+\t */\n+\twatch->is_active = ReadDirectoryChangesW(\n+\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n+\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n+\n+\t/*\n+\t * The kernel throws an invalid parameter error when our buffer\n+\t * is too big and we are pointed at a remote directory (and possibly\n+\t * for other reasons). Quietly set it down and try again.\n+\t *\n+\t * See note about MAX_RDCW_BUF at the top.\n+\t */\n+\tif (!watch->is_active &&\n+\t GetLastError() == ERROR_INVALID_PARAMETER &&\n+\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n+\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n+\t\tgoto start_watch;\n+\t}\n+\n+\tif (watch->is_active)\n+\t\treturn 0;\n+\n+\terror(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static int recv_rdcw_watch(struct one_watch *watch)\n+{\n+\twatch->is_active = FALSE;\n+\n+\t/*\n+\t * The overlapped result is ready. If the Read...() was successful\n+\t * we finally receive the actual result into our buffer.\n+\t */\n+\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n+\t\t\t\tTRUE))\n+\t\treturn 0;\n+\n+\t/*\n+\t * NEEDSWORK: If an external is deleted, the above\n+\t * returns an error. I'm not sure that there's anything that\n+\t * we can do here other than failing -- the /.git\n+\t * link file would be broken anyway. We might try to check\n+\t * for that and return a better error message, but I'm not\n+\t * sure it is worth it.\n+\t */\n+\n+\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static void cancel_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD count;\n+\n+\tif (!watch || !watch->is_active)\n+\t\treturn;\n+\n+\t/*\n+\t * The calls to ReadDirectoryChangesW() and GetOverlappedResult()\n+\t * form a \"pair\" (my term) where we queue an IO and promise to\n+\t * hang around and wait for the kernel to give us the result.\n+\t *\n+\t * If for some reason after we queue the IO, we have to quit\n+\t * or otherwise not stick around for the second half, we must\n+\t * tell the kernel to abort the IO. This prevents the kernel\n+\t * from writing to our buffer and/or signalling our event\n+\t * after we free them.\n+\t *\n+\t * (Ask me how much fun it was to track that one down).\n+\t */\n+\tCancelIoEx(watch->hDir, &watch->overlapped);\n+\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n+\twatch->is_active = FALSE;\n+}\n+\n+/*\n+ * Process filesystem events that happen anywhere (recursively) under the\n+ * root directory. For a normal working directory, this includes\n+ * both version controlled files and the contents of the .git/ directory.\n+ *\n+ * If /.git is a file, then we only see events for the file\n+ * itself.\n+ */\n+static int process_worktree_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_worktree;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tconst char *p = watch->buffer;\n+\n+\t/*\n+\t * If the kernel gets more events than will fit in the kernel\n+\t * buffer associated with our RDCW handle, it drops them and\n+\t * returns a count of zero.\n+\t *\n+\t * Yes, the call returns WITHOUT error and with length zero.\n+\t *\n+\t * (The \"overflow\" case is not ambiguous with the \"no data\" case\n+\t * because we did an INFINITE wait.)\n+\t *\n+\t * This means we have a gap in coverage. Tell the daemon layer\n+\t * to resync.\n+\t */\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\t}\n+\n+\t/*\n+\t * On Windows, `info` contains an \"array\" of paths that are\n+\t * relative to the root of whichever directory handle received\n+\t * the event.\n+\t */\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\t\t/* ignore everything inside of \"/.git/\" */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* queue normal pathname */\n+\t\t\tif (!batch)\n+\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tbatch = NULL;\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\n+force_shutdown:\n+\tfsmonitor_batch__pop(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_SHUTDOWN;\n+}\n+\n+/*\n+ * Process filesystem events that happened anywhere (recursively) under the\n+ * external (such as non-primary worktrees or submodules).\n+ * We only care about cookie files that our client threads created here.\n+ *\n+ * Note that we DO NOT get filesystem events on the external \n+ * itself (it is not inside something that we are watching). In particular,\n+ * we do not get an event if the external is deleted.\n+ */\n+static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_gitdir;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *p = watch->buffer;\n+\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_GITDIR;\n+\t}\n+\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n+\n+\t\ttrace_printf_key(&trace_fsmonitor, \"BBB: %s\", path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\tgoto skip_this_path;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, NULL, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_GITDIR;\n }\n \n void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tDWORD dwWait;\n+\n+\tstate->error_code = 0;\n+\n+\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tif (data->watch_gitdir &&\n+\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tfor (;;) {\n+\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n+\t\t\t\t\t\tdata->hListener,\n+\t\t\t\t\t\tFALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n+\t\t\tif (recv_rdcw_watch(data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n+\t\t\tif (recv_rdcw_watch(data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->error_code = -1;\n+\n+force_shutdown:\n+\t/*\n+\t * Tell the IPC thead pool to stop (which completes the await\n+\t * in the main thread (which will also signal this thread (if\n+\t * we are still alive))).\n+\t */\n+\tipc_server_stop_async(state->ipc_server_data);\n+\n+clean_shutdown:\n+\tcancel_rdcw_watch(data->watch_worktree);\n+\tcancel_rdcw_watch(data->watch_gitdir);\n }\n \n int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->watch_worktree = create_watch(state,\n+\t\t\t\t\t state->path_worktree_watch.buf);\n+\tif (!data->watch_worktree)\n+\t\tgoto failed;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->watch_gitdir = create_watch(state,\n+\t\t\t\t\t\t state->path_gitdir_watch.buf);\n+\t\tif (!data->watch_gitdir)\n+\t\t\tgoto failed;\n+\t}\n+\n+\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_listener_handles++;\n+\n+\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n+\t\tdata->watch_worktree->hEvent;\n+\tdata->nr_listener_handles++;\n+\n+\tif (data->watch_gitdir) {\n+\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n+\t\t\tdata->watch_gitdir->hEvent;\n+\t\tdata->nr_listener_handles++;\n+\t}\n+\n+\tstate->backend_data = data;\n+\treturn 0;\n+\n+failed:\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n \treturn -1;\n }\n \n void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n-- \ngitgitgadget\n\n"},{"id":"425319","messageId":"2ed7bc3fae7aece122f1c55f59f4c3e8201de341.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 16/28] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:55Z","receivedAt":"2021-05-22T13:57:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to build a list of changed paths and associate\nthem with a token-id. This will be used by the platform-specific\nbackends to accumulate changed paths in response to filesystem events.\n\nThe platform-specific file system listener thread receives file system\nevents containing one or more changed pathnames (with whatever bucketing\nor grouping that is convenient for the file system). These paths are\naccumulated (without locking) by the file system layer into a `fsmonitor_batch`.\n\nWhen the file system layer has drained the kernel event queue, it will\n\"publish\" them to our token queue and make them visible to concurrent\nclient worker threads. The token layer is free to combine and/or de-dup\npaths within these batches for efficient presentation to clients.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 234 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 40 ++++++\n 2 files changed, 272 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex ecd456dc9284..663fead0d66e 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -166,17 +166,27 @@ struct fsmonitor_token_data {\n \tuint64_t client_ref_count;\n };\n \n+struct fsmonitor_batch {\n+\tstruct fsmonitor_batch *next;\n+\tuint64_t batch_seq_nr;\n+\tconst char **interned_paths;\n+\tsize_t nr, alloc;\n+\ttime_t pinned_time;\n+};\n+\n static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n {\n \tstatic int test_env_value = -1;\n \tstatic uint64_t flush_count = 0;\n \tstruct fsmonitor_token_data *token;\n+\tstruct fsmonitor_batch *batch;\n \n \tCALLOC_ARRAY(token, 1);\n+\tbatch = fsmonitor_batch__new();\n \n \tstrbuf_init(&token->token_id, 0);\n-\ttoken->batch_head = NULL;\n-\ttoken->batch_tail = NULL;\n+\ttoken->batch_head = batch;\n+\ttoken->batch_tail = batch;\n \ttoken->client_ref_count = 0;\n \n \tif (test_env_value < 0)\n@@ -202,9 +212,147 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n \t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n \t}\n \n+\t/*\n+\t * We created a new and are starting a new series\n+\t * of tokens with a zero .\n+\t *\n+\t * Since clients cannot guess our new (non test) \n+\t * they will always receive a trivial response (because of the\n+\t * mismatch on the ). The trivial response will\n+\t * tell them our new so that subsequent requests\n+\t * will be relative to our new series. (And when sending that\n+\t * response, we pin the current head of the batch list.)\n+\t *\n+\t * Even if the client correctly guesses the , their\n+\t * request of \"builtin::0\" asks for all changes MORE\n+\t * RECENT than batch/bin 0.\n+\t *\n+\t * This implies that it is a waste to accumulate paths in the\n+\t * initial batch/bin (because they will never be transmitted).\n+\t *\n+\t * So the daemon could be running for days and watching the\n+\t * file system, but doesn't need to actually accumulate any\n+\t * paths UNTIL we need to set a reference point for a later\n+\t * relative request.\n+\t *\n+\t * However, it is very useful for testing to always have a\n+\t * reference point set. Pin batch 0 to force early file system\n+\t * events to accumulate.\n+\t */\n+\tif (test_env_value)\n+\t\tbatch->pinned_time = time(NULL);\n+\n \treturn token;\n }\n \n+struct fsmonitor_batch *fsmonitor_batch__new(void)\n+{\n+\tstruct fsmonitor_batch *batch;\n+\n+\tCALLOC_ARRAY(batch, 1);\n+\n+\treturn batch;\n+}\n+\n+struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch)\n+{\n+\tstruct fsmonitor_batch *next;\n+\n+\tif (!batch)\n+\t\treturn NULL;\n+\n+\tnext = batch->next;\n+\n+\t/*\n+\t * The actual strings within the array are interned, so we don't\n+\t * own them.\n+\t */\n+\tfree(batch->interned_paths);\n+\n+\treturn next;\n+}\n+\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n+\t\t\t const char *path)\n+{\n+\tconst char *interned_path = strintern(path);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n+\n+\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n+\tbatch->interned_paths[batch->nr++] = interned_path;\n+}\n+\n+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n+\t\t\t\t const struct fsmonitor_batch *batch_src)\n+{\n+\tsize_t k;\n+\n+\tALLOC_GROW(batch_dest->interned_paths,\n+\t\t batch_dest->nr + batch_src->nr + 1,\n+\t\t batch_dest->alloc);\n+\n+\tfor (k = 0; k < batch_src->nr; k++)\n+\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n+\t\t\tbatch_src->interned_paths[k];\n+}\n+\n+static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n+{\n+\tstruct fsmonitor_batch *p;\n+\n+\tif (!token)\n+\t\treturn;\n+\n+\tassert(token->client_ref_count == 0);\n+\n+\tstrbuf_release(&token->token_id);\n+\n+\tfor (p = token->batch_head; p; p = fsmonitor_batch__pop(p))\n+\t\t;\n+\n+\tfree(token);\n+}\n+\n+/*\n+ * Flush all of our cached data about the filesystem. Call this if we\n+ * lose sync with the filesystem and miss some notification events.\n+ *\n+ * [1] If we are missing events, then we no longer have a complete\n+ * history of the directory (relative to our current start token).\n+ * We should create a new token and start fresh (as if we just\n+ * booted up).\n+ *\n+ * If there are no concurrent threads readering the current token data\n+ * series, we can free it now. Otherwise, let the last reader free\n+ * it.\n+ *\n+ * Either way, the old token data series is no longer associated with\n+ * our state data.\n+ */\n+static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct fsmonitor_token_data *free_me = NULL;\n+\tstruct fsmonitor_token_data *new_one = NULL;\n+\n+\tnew_one = fsmonitor_new_token_data();\n+\n+\tif (state->current_token_data->client_ref_count == 0)\n+\t\tfree_me = state->current_token_data;\n+\tstate->current_token_data = new_one;\n+\n+\tfsmonitor_free_token_data(free_me);\n+}\n+\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\tpthread_mutex_lock(&state->main_lock);\n+\twith_lock__do_force_resync(state);\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -314,6 +462,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n \n+/*\n+ * We try to combine small batches at the front of the batch-list to avoid\n+ * having a long list. This hopefully makes it a little easier when we want\n+ * to truncate and maintain the list. However, we don't want the paths array\n+ * to just keep growing and growing with realloc, so we insert an arbitrary\n+ * limit.\n+ */\n+#define MY_COMBINE_LIMIT (1024)\n+\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names)\n+{\n+\tif (!batch && !cookie_names->nr)\n+\t\treturn;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (batch) {\n+\t\tstruct fsmonitor_batch *head;\n+\n+\t\thead = state->current_token_data->batch_head;\n+\t\tif (!head) {\n+\t\t\tBUG(\"token does not have batch\");\n+\t\t} else if (head->pinned_time) {\n+\t\t\t/*\n+\t\t\t * We cannot alter the current batch list\n+\t\t\t * because:\n+\t\t\t *\n+\t\t\t * [a] it is being transmitted to at least one\n+\t\t\t * client and the handle_client() thread has a\n+\t\t\t * ref-count, but not a lock on the batch list\n+\t\t\t * starting with this item.\n+\t\t\t *\n+\t\t\t * [b] it has been transmitted in the past to\n+\t\t\t * at least one client such that future\n+\t\t\t * requests are relative to this head batch.\n+\t\t\t *\n+\t\t\t * So, we can only prepend a new batch onto\n+\t\t\t * the front of the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else if (!head->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * Batch 0 is unpinned. See the note in\n+\t\t\t * `fsmonitor_new_token_data()` about why we\n+\t\t\t * don't need to accumulate these paths.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__pop(batch);\n+\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n+\t\t\t/*\n+\t\t\t * The head batch in the list has never been\n+\t\t\t * transmitted to a client, but folding the\n+\t\t\t * contents of the new batch onto it would\n+\t\t\t * exceed our arbitrary limit, so just prepend\n+\t\t\t * the new batch onto the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We are free to append the paths in the given\n+\t\t\t * batch onto the end of the current head batch.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__combine(head, batch);\n+\t\t\tfsmonitor_batch__pop(batch);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static void *fsmonitor_fs_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -328,6 +551,13 @@ static void *fsmonitor_fs_listen__thread_proc(void *_state)\n \n \tfsmonitor_fs_listen__loop(state);\n \n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (state->current_token_data &&\n+\t state->current_token_data->client_ref_count == 0)\n+\t\tfsmonitor_free_token_data(state->current_token_data);\n+\tstate->current_token_data = NULL;\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n \ttrace2_thread_exit();\n \treturn NULL;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 7bbb3a27a1ce..89a9ef20b24b 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -12,6 +12,27 @@\n struct fsmonitor_batch;\n struct fsmonitor_token_data;\n \n+/*\n+ * Create a new batch of path(s). The returned batch is considered\n+ * private and not linked into the fsmonitor daemon state. The caller\n+ * should fill this batch with one or more paths and then publish it.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__new(void);\n+\n+/*\n+ * Free this batch and return the value of the batch->next field.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch);\n+\n+/*\n+ * Add this path to this batch of modified files.\n+ *\n+ * The batch should be private and NOT (yet) linked into the fsmonitor\n+ * daemon state and therefore not yet visible to worker threads and so\n+ * no locking is required.\n+ */\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n+\n struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n \n struct fsmonitor_daemon_state {\n@@ -91,5 +112,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \tstruct fsmonitor_daemon_state *state,\n \tconst char *path);\n \n+/*\n+ * Prepend the this batch of path(s) onto the list of batches associated\n+ * with the current token. This makes the batch visible to worker threads.\n+ *\n+ * The caller no longer owns the batch and must not free it.\n+ *\n+ * Wake up the client threads waiting on these cookies.\n+ */\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names);\n+\n+/*\n+ * If the platform-specific layer loses sync with the filesystem,\n+ * it should call this to invalidate cached data and abort waiting\n+ * threads.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"425320","messageId":"c092cdf2c8b749768a3ec9c9998dd0d2842d748f.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 15/28] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:54Z","receivedAt":"2021-05-22T13:57:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to create token-ids and define the\noverall token naming scheme.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 115 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex d6e35ded9f68..ecd456dc9284 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -91,6 +91,120 @@ static int do_as_client__status(void)\n \t}\n }\n \n+/*\n+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n+ * \"token\" as a virtual timestamp. Clients can request a summary of all\n+ * created/deleted/modified files relative to a token. In the response,\n+ * clients receive a new token for the next (relative) request.\n+ *\n+ *\n+ * Token Format\n+ * ============\n+ *\n+ * The contents of the token are private and provider-specific.\n+ *\n+ * For the built-in fsmonitor--daemon, we define a token as follows:\n+ *\n+ * \"builtin\" \":\" \":\" \n+ *\n+ * The \"builtin\" prefix is used as a namespace to avoid conflicts\n+ * with other providers (such as Watchman).\n+ *\n+ * The is an arbitrary OPAQUE string, such as a GUID,\n+ * UUID, or {timestamp,pid}. It is used to group all filesystem\n+ * events that happened while the daemon was monitoring (and in-sync\n+ * with the filesystem).\n+ *\n+ * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n+ * and does not define less-than/greater-than relationships.\n+ * (There are too many race conditions to rely on file system\n+ * event timestamps.)\n+ *\n+ * The is a simple integer incremented whenever the\n+ * daemon needs to make its state public. For example, if 1000 file\n+ * system events come in, but no clients have requested the data,\n+ * the daemon can continue to accumulate file changes in the same\n+ * bin and does not need to advance the sequence number. However,\n+ * as soon as a client does arrive, the daemon needs to start a new\n+ * bin and increment the sequence number.\n+ *\n+ * The sequence number serves as the boundary between 2 sets\n+ * of bins -- the older ones that the client has already seen\n+ * and the newer ones that it hasn't.\n+ *\n+ * When a new is created, the is reset to\n+ * zero.\n+ *\n+ *\n+ * About Token Ids\n+ * ===============\n+ *\n+ * A new token_id is created:\n+ *\n+ * [1] each time the daemon is started.\n+ *\n+ * [2] any time that the daemon must re-sync with the filesystem\n+ * (such as when the kernel drops or we miss events on a very\n+ * active volume).\n+ *\n+ * [3] in response to a client \"flush\" command (for dropped event\n+ * testing).\n+ *\n+ * When a new token_id is created, the daemon is free to discard all\n+ * cached filesystem events associated with any previous token_ids.\n+ * Events associated with a non-current token_id will never be sent\n+ * to a client. A token_id change implicitly means that the daemon\n+ * has gap in its event history.\n+ *\n+ * Therefore, clients that present a token with a stale (non-current)\n+ * token_id will always be given a trivial response.\n+ */\n+struct fsmonitor_token_data {\n+\tstruct strbuf token_id;\n+\tstruct fsmonitor_batch *batch_head;\n+\tstruct fsmonitor_batch *batch_tail;\n+\tuint64_t client_ref_count;\n+};\n+\n+static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n+{\n+\tstatic int test_env_value = -1;\n+\tstatic uint64_t flush_count = 0;\n+\tstruct fsmonitor_token_data *token;\n+\n+\tCALLOC_ARRAY(token, 1);\n+\n+\tstrbuf_init(&token->token_id, 0);\n+\ttoken->batch_head = NULL;\n+\ttoken->batch_tail = NULL;\n+\ttoken->client_ref_count = 0;\n+\n+\tif (test_env_value < 0)\n+\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n+\n+\tif (!test_env_value) {\n+\t\tstruct timeval tv;\n+\t\tstruct tm tm;\n+\t\ttime_t secs;\n+\n+\t\tgettimeofday(&tv, NULL);\n+\t\tsecs = tv.tv_sec;\n+\t\tgmtime_r(&secs, &tm);\n+\n+\t\tstrbuf_addf(&token->token_id,\n+\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n+\t\t\t flush_count++,\n+\t\t\t getpid(),\n+\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n+\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n+\t\t\t (long)tv.tv_usec);\n+\t} else {\n+\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n+\t}\n+\n+\treturn token;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -281,7 +395,7 @@ static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n-\tstate.current_token_data = NULL;\n+\tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n \tstrbuf_init(&state.path_worktree_watch, 0);\n-- \ngitgitgadget\n\n"},{"id":"425321","messageId":"11ea2f97def686ca7f73e6c1a3eabba341bea145.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 24/28] fsmonitor: force update index after large responses","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:57:03Z","receivedAt":"2021-05-22T13:57:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSet the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when\nFSMonitor returns a very large repsonse to ensure that the index is\nwritten to disk.\n\nNormally, when the FSMonitor response includes a tracked file, the\nindex is always updated. Similarly, the index might be updated when\nthe response alters the untracked-cache (when enabled). However, in\ncases where neither of those cause the index to be considered changed,\nthe FSMonitor response is wasted. Subsequent Git commands will make\nrequests with the same token and receive the same response.\n\nIf that response is very large, performance may suffer. It would be\nmore efficient to force update the index now (and the token in the\nindex extension) in order to reduce the size of the response received\nby future commands.\n\nThis was observed on Windows after a large checkout. On Windows, the\nkernel emits events for the files that are changed as they are\nchanged. However, it might delay events for the containing\ndirectories until the system is more idle (or someone scans the\ndirectory (so it seems)). The first status following a checkout would\nget the list of files. The subsequent status commands would get the\nlist of directories as the events trickled out. But they would never\ncatch up because the token was not advanced because the index wasn't\nupdated.\n\nThis list of directories caused `wt_status_collect_untracked()` to\nunnecessarily spend time actually scanning them during each command.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 49 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 29adf3e53ef3..22623fd228fc 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -230,6 +230,45 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n+/*\n+ * The number of pathnames that we need to receive from FSMonitor\n+ * before we force the index to be updated.\n+ *\n+ * Note that any pathname within the set of received paths MAY cause\n+ * cache-entry or istate flag bits to be updated and thus cause the\n+ * index to be updated on disk.\n+ *\n+ * However, the response may contain many paths (such as ignored\n+ * paths) that will not update any flag bits. And thus not force the\n+ * index to be updated. (This is fine and normal.) It also means\n+ * that the token will not be updated in the FSMonitor index\n+ * extension. So the next Git command will find the same token in the\n+ * index, make the same token-relative request, and receive the same\n+ * response (plus any newly changed paths). If this response is large\n+ * (and continues to grow), performance could be impacted.\n+ *\n+ * For example, if the user runs a build and it writes 100K object\n+ * files but doesn't modify any source files, the index would not need\n+ * to be updated. The FSMonitor response (after the build and\n+ * relative to a pre-build token) might be 5MB. Each subsequent Git\n+ * command will receive that same 100K/5MB response until something\n+ * causes the index to be updated. And `refresh_fsmonitor()` will\n+ * have to iterate over those 100K paths each time.\n+ *\n+ * Performance could be improved if we optionally force update the\n+ * index after a very large response and get an updated token into\n+ * the FSMonitor index extension. This should allow subsequent\n+ * commands to get smaller and more current responses.\n+ *\n+ * The value chosen here does not need to be precise. The index\n+ * will be updated automatically the first time the user touches\n+ * a tracked file and causes a command like `git status` to\n+ * update an mtime to be updated and/or set a flag bit.\n+ *\n+ * NEEDSWORK: Does this need to be a config value?\n+ */\n+static int fsmonitor_force_update_threshold = 100;\n+\n void refresh_fsmonitor(struct index_state *istate)\n {\n \tstruct repository *r = istate->repo ? istate->repo : the_repository;\n@@ -363,19 +402,28 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t *\n \t\t * This updates both the cache-entries and the untracked-cache.\n \t\t */\n+\t\tint count = 0;\n+\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n \t\t\t\tcontinue;\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n \t\t\tbol = i + 1;\n+\t\t\tcount++;\n \t\t}\n-\t\tif (bol < query_result.len)\n+\t\tif (bol < query_result.len) {\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tcount++;\n+\t\t}\n \n \t\t/* Now mark the untracked cache for fsmonitor usage */\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n+\n+\t\tif (count > fsmonitor_force_update_threshold)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\n \t} else {\n \t\t/*\n \t\t * We received a trivial response, so invalidate everything.\n-- \ngitgitgadget\n\n"},{"id":"425322","messageId":"cc4a596d17c7036534bae9865170a8c9b7ad0818.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 20/28] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:59Z","receivedAt":"2021-05-22T13:57:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to respond to IPC requests from client\nGit processes and respond with a list of modified pathnames\nrelative to the provided token.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 312 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 310 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 663fead0d66e..33b4f09c72ca 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,6 +7,7 @@\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n+#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon start []\"),\n@@ -353,6 +354,311 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+/*\n+ * Format an opaque token string to send to the client.\n+ */\n+static void with_lock__format_response_token(\n+\tstruct strbuf *response_token,\n+\tconst struct strbuf *response_token_id,\n+\tconst struct fsmonitor_batch *batch)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstrbuf_reset(response_token);\n+\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n+\t\t response_token_id->buf, batch->batch_seq_nr);\n+}\n+\n+/*\n+ * Parse an opaque token from the client.\n+ * Returns -1 on error.\n+ */\n+static int fsmonitor_parse_client_token(const char *buf_token,\n+\t\t\t\t\tstruct strbuf *requested_token_id,\n+\t\t\t\t\tuint64_t *seq_nr)\n+{\n+\tconst char *p;\n+\tchar *p_end;\n+\n+\tstrbuf_reset(requested_token_id);\n+\t*seq_nr = 0;\n+\n+\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n+\t\treturn -1;\n+\n+\twhile (*p && *p != ':')\n+\t\tstrbuf_addch(requested_token_id, *p++);\n+\tif (!*p++)\n+\t\treturn -1;\n+\n+\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n+\tif (*p_end)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal);\n+\n+static int do_handle_client(struct fsmonitor_daemon_state *state,\n+\t\t\t const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\tstruct fsmonitor_token_data *token_data = NULL;\n+\tstruct strbuf response_token = STRBUF_INIT;\n+\tstruct strbuf requested_token_id = STRBUF_INIT;\n+\tstruct strbuf payload = STRBUF_INIT;\n+\tuint64_t requested_oldest_seq_nr = 0;\n+\tuint64_t total_response_len = 0;\n+\tconst char *p;\n+\tconst struct fsmonitor_batch *batch_head;\n+\tconst struct fsmonitor_batch *batch;\n+\tintmax_t count = 0, duplicates = 0;\n+\tkh_str_t *shown;\n+\tint hash_ret;\n+\tint do_trivial = 0;\n+\tint do_flush = 0;\n+\n+\t/*\n+\t * We expect `command` to be of the form:\n+\t *\n+\t * := quit NUL\n+\t * | flush NUL\n+\t * | NUL\n+\t * | NUL\n+\t */\n+\n+\tif (!strcmp(command, \"quit\")) {\n+\t\t/*\n+\t\t * A client has requested over the socket/pipe that the\n+\t\t * daemon shutdown.\n+\t\t *\n+\t\t * Tell the IPC thread pool to shutdown (which completes\n+\t\t * the await in the main thread (which can stop the\n+\t\t * fsmonitor listener thread)).\n+\t\t *\n+\t\t * There is no reply to the client.\n+\t\t */\n+\t\treturn SIMPLE_IPC_QUIT;\n+\n+\t} else if (!strcmp(command, \"flush\")) {\n+\t\t/*\n+\t\t * Flush all of our cached data and generate a new token\n+\t\t * just like if we lost sync with the filesystem.\n+\t\t *\n+\t\t * Then send a trivial response using the new token.\n+\t\t */\n+\t\tdo_flush = 1;\n+\t\tdo_trivial = 1;\n+\n+\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n+\t\t/* assume V1 timestamp or garbage */\n+\n+\t\tchar *p_end;\n+\n+\t\tstrtoumax(command, &p_end, 10);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t ((*p_end) ?\n+\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n+\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n+\t\t\t\t command);\n+\t\tdo_trivial = 1;\n+\n+\t} else {\n+\t\t/* We have \"builtin:*\" */\n+\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n+\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n+\t\t\t\t\t command);\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We have a V2 valid token:\n+\t\t\t * \"builtin::\"\n+\t\t\t */\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (!state->current_token_data)\n+\t\tBUG(\"fsmonitor state does not have a current token\");\n+\n+\tif (do_flush)\n+\t\twith_lock__do_force_resync(state);\n+\n+\t/*\n+\t * We mark the current head of the batch list as \"pinned\" so\n+\t * that the listener thread will treat this item as read-only\n+\t * (and prevent any more paths from being added to it) from\n+\t * now on.\n+\t */\n+\ttoken_data = state->current_token_data;\n+\tbatch_head = token_data->batch_head;\n+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n+\n+\t/*\n+\t * FSMonitor Protocol V2 requires that we send a response header\n+\t * with a \"new current token\" and then all of the paths that changed\n+\t * since the \"requested token\". We send the seq_nr of the just-pinned\n+\t * head batch so that future requests from a client will be relative\n+\t * to it.\n+\t */\n+\twith_lock__format_response_token(&response_token,\n+\t\t\t\t\t &token_data->token_id, batch_head);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttotal_response_len += response_token.len + 1;\n+\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n+\t\t\t response_token.buf);\n+\n+\tif (!do_trivial) {\n+\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n+\t\t\t/*\n+\t\t\t * The client last spoke to a different daemon\n+\t\t\t * instance -OR- the daemon had to resync with\n+\t\t\t * the filesystem (and lost events), so reject.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n+\t\t\t\t\t \"response/token\", \"different\");\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else if (requested_oldest_seq_nr <\n+\t\t\t token_data->batch_tail->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * The client wants older events than we have for\n+\t\t\t * this token_id. This means that the end of our\n+\t\t\t * batch list was truncated and we cannot give the\n+\t\t\t * client a complete snapshot relative to their\n+\t\t\t * request.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"client requested truncated data\");\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n+\tif (do_trivial) {\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\treply(reply_data, \"/\", 2);\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n+\t\t\t\t \"response/trivial\", 1);\n+\n+\t\tstrbuf_release(&response_token);\n+\t\tstrbuf_release(&requested_token_id);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * We're going to hold onto a pointer to the current\n+\t * token-data while we walk the list of batches of files.\n+\t * During this time, we will NOT be under the lock.\n+\t * So we ref-count it.\n+\t *\n+\t * This allows the listener thread to continue prepending\n+\t * new batches of items to the token-data (which we'll ignore).\n+\t *\n+\t * AND it allows the listener thread to do a token-reset\n+\t * (and install a new `current_token_data`).\n+\t */\n+\ttoken_data->client_ref_count++;\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * The client request is relative to the token that they sent,\n+\t * so walk the batch list backwards from the current head back\n+\t * to the batch (sequence number) they named.\n+\t *\n+\t * We use khash to de-dup the list of pathnames.\n+\t *\n+\t * NEEDSWORK: each batch contains a list of interned strings,\n+\t * so we only need to do pointer comparisons here to build the\n+\t * hash table. Currently, we're still comparing the string\n+\t * values.\n+\t */\n+\tshown = kh_init_str();\n+\tfor (batch = batch_head;\n+\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n+\t batch = batch->next) {\n+\t\tsize_t k;\n+\n+\t\tfor (k = 0; k < batch->nr; k++) {\n+\t\t\tconst char *s = batch->interned_paths[k];\n+\t\t\tsize_t s_len;\n+\n+\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n+\t\t\t\tduplicates++;\n+\t\t\telse {\n+\t\t\t\tkh_put_str(shown, s, &hash_ret);\n+\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n+\t\t\t\t\t\t count, s);\n+\n+\t\t\t\t/* Each path gets written with a trailing NUL */\n+\t\t\t\ts_len = strlen(s) + 1;\n+\n+\t\t\t\tif (payload.len + s_len >=\n+\t\t\t\t LARGE_PACKET_DATA_MAX) {\n+\t\t\t\t\treply(reply_data, payload.buf,\n+\t\t\t\t\t payload.len);\n+\t\t\t\t\ttotal_response_len += payload.len;\n+\t\t\t\t\tstrbuf_reset(&payload);\n+\t\t\t\t}\n+\n+\t\t\t\tstrbuf_add(&payload, s, s_len);\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (payload.len) {\n+\t\treply(reply_data, payload.buf, payload.len);\n+\t\ttotal_response_len += payload.len;\n+\t}\n+\n+\tkh_release_str(shown);\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (token_data->client_ref_count > 0)\n+\t\ttoken_data->client_ref_count--;\n+\n+\tif (token_data->client_ref_count == 0) {\n+\t\tif (token_data != state->current_token_data) {\n+\t\t\t/*\n+\t\t\t * The listener thread did a token-reset while we were\n+\t\t\t * walking the batch list. Therefore, this token is\n+\t\t\t * stale and can be discarded completely. If we are\n+\t\t\t * the last reader thread using this token, we own\n+\t\t\t * that work.\n+\t\t\t */\n+\t\t\tfsmonitor_free_token_data(token_data);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n+\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\tstrbuf_release(&payload);\n+\n+\treturn 0;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -360,7 +666,7 @@ static int handle_client(void *data,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n-\t/* struct fsmonitor_daemon_state *state = data; */\n+\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n \t/*\n@@ -371,10 +677,12 @@ static int handle_client(void *data,\n \tif (command_len != strlen(command))\n \t\tBUG(\"FSMonitor assumes text messages\");\n \n+\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n+\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n \n-\tresult = 0; /* TODO Do something here. */\n+\tresult = do_handle_client(state, command, reply, reply_data);\n \n \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n \n-- \ngitgitgadget\n\n"},{"id":"425323","messageId":"f0da90e9b050173e2fffe7da3b54b6bc5d0056a1.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 21/28] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:57:00Z","receivedAt":"2021-05-22T13:57:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to periodically truncate the list of\nmodified files to save some memory.\n\nClients will ask for the set of changes relative to a token that they\nfound in the FSMN index extension in the index. (This token is like a\npoint in time, but different). Clients will then update the index to\ncontain the response token (so that subsequent commands will be\nrelative to this new token).\n\nTherefore, the daemon can gradually truncate the in-memory list of\nchanged paths as they become obsolete (older than the previous token).\nSince we may have multiple clients making concurrent requests with a\nskew of tokens and clients may be racing to the talk to the daemon,\nwe lazily truncate the list.\n\nWe introduce a 5 minute delay and truncate batches 5 minutes after\nthey are considered obsolete.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 80 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 80 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 33b4f09c72ca..e807aa8f6741 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -298,6 +298,77 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n \t\t\tbatch_src->interned_paths[k];\n }\n \n+/*\n+ * To keep the batch list from growing unbounded in response to filesystem\n+ * activity, we try to truncate old batches from the end of the list as\n+ * they become irrelevant.\n+ *\n+ * We assume that the .git/index will be updated with the most recent token\n+ * any time the index is updated. And future commands will only ask for\n+ * recent changes *since* that new token. So as tokens advance into the\n+ * future, older batch items will never be requested/needed. So we can\n+ * truncate them without loss of functionality.\n+ *\n+ * However, multiple commands may be talking to the daemon concurrently\n+ * or perform a slow command, so a little \"token skew\" is possible.\n+ * Therefore, we want this to be a little bit lazy and have a generous\n+ * delay.\n+ *\n+ * The current reader thread walked backwards in time from `token->batch_head`\n+ * back to `batch_marker` somewhere in the middle of the batch list.\n+ *\n+ * Let's walk backwards in time from that marker an arbitrary delay\n+ * and truncate the list there. Note that these timestamps are completely\n+ * artificial (based on when we pinned the batch item) and not on any\n+ * filesystem activity.\n+ */\n+#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */\n+\n+static void with_lock__truncate_old_batches(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst struct fsmonitor_batch *batch_marker)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *rest;\n+\tstruct fsmonitor_batch *p;\n+\n+\tif (!batch_marker)\n+\t\treturn;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Truncate: mark (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t batch_marker->batch_seq_nr,\n+\t\t\t (uint64_t)batch_marker->pinned_time);\n+\n+\tfor (batch = batch_marker; batch; batch = batch->next) {\n+\t\ttime_t t;\n+\n+\t\tif (!batch->pinned_time) /* an overflow batch */\n+\t\t\tcontinue;\n+\n+\t\tt = batch->pinned_time + MY_TIME_DELAY_SECONDS;\n+\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n+\t\t\tcontinue;\n+\n+\t\tgoto truncate_past_here;\n+\t}\n+\n+\treturn;\n+\n+truncate_past_here:\n+\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n+\n+\trest = ((struct fsmonitor_batch *)batch)->next;\n+\t((struct fsmonitor_batch *)batch)->next = NULL;\n+\n+\tfor (p = rest; p; p = fsmonitor_batch__pop(p)) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"Truncate: kill (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t\t p->batch_seq_nr, (uint64_t)p->pinned_time);\n+\t}\n+}\n+\n static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n {\n \tstruct fsmonitor_batch *p;\n@@ -643,6 +714,15 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * that work.\n \t\t\t */\n \t\t\tfsmonitor_free_token_data(token_data);\n+\t\t} else if (batch) {\n+\t\t\t/*\n+\t\t\t * This batch is the first item in the list\n+\t\t\t * that is older than the requested sequence\n+\t\t\t * number and might be considered to be\n+\t\t\t * obsolete. See if we can truncate the list\n+\t\t\t * and save some memory.\n+\t\t\t */\n+\t\t\twith_lock__truncate_old_batches(state, batch);\n \t\t}\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"425324","messageId":"08474bad8303b1d9b1315af3fe97393190abb261.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 19/28] fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:58Z","receivedAt":"2021-05-22T13:57:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement file system event listener on MacOS using FSEvent,\nCoreFoundation, and CoreServices.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 381 +++++++++++++++++++\n 1 file changed, 381 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\nindex bec5130d9e1d..02f89de216e9 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n@@ -97,20 +97,401 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-fs-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tCFStringRef cfsr_worktree_path;\n+\tCFStringRef cfsr_gitdir_path;\n+\n+\tCFArrayRef cfar_paths_to_watch;\n+\tint nr_paths_watching;\n+\n+\tFSEventStreamRef stream;\n+\n+\tCFRunLoopRef rl;\n+\n+\tenum shutdown_style {\n+\t\tSHUTDOWN_EVENT = 0,\n+\t\tFORCE_SHUTDOWN,\n+\t\tFORCE_ERROR_STOP,\n+\t} shutdown_style;\n+\n+\tunsigned int stream_scheduled:1;\n+\tunsigned int stream_started:1;\n+};\n+\n+static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n+\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n+\tif (flag & kFSEventStreamEventFlagUserDropped)\n+\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n+\tif (flag & kFSEventStreamEventFlagKernelDropped)\n+\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n+\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n+\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n+\tif (flag & kFSEventStreamEventFlagHistoryDone)\n+\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n+\tif (flag & kFSEventStreamEventFlagRootChanged)\n+\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n+\tif (flag & kFSEventStreamEventFlagMount)\n+\t\tstrbuf_addstr(&msg, \"Mount|\");\n+\tif (flag & kFSEventStreamEventFlagUnmount)\n+\t\tstrbuf_addstr(&msg, \"Unmount|\");\n+\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n+\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n+\tif (flag & kFSEventStreamEventFlagItemCreated)\n+\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n+\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n+\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n+\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsDir)\n+\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsFile)\n+\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemModified)\n+\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n+\tif (flag & kFSEventStreamEventFlagItemRemoved)\n+\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n+\tif (flag & kFSEventStreamEventFlagItemRenamed)\n+\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n+\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n+\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n+\tif (flag & kFSEventStreamEventFlagOwnEvent)\n+\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n+\tif (flag & kFSEventStreamEventFlagItemCloned)\n+\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\t\t\t path, flag, msg.buf);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRemoved);\n+}\n+\n+static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRenamed);\n+}\n+\n+static int ef_is_dropped(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagKernelDropped ||\n+\t\tef & kFSEventStreamEventFlagUserDropped);\n+}\n+\n+static void fsevent_callback(ConstFSEventStreamRef streamRef,\n+\t\t\t void *ctx,\n+\t\t\t size_t num_of_events,\n+\t\t\t void *event_paths,\n+\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t const FSEventStreamEventId event_ids[])\n+{\n+\tstruct fsmonitor_daemon_state *state = ctx;\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tchar **paths = (char **)event_paths;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *path_k;\n+\tconst char *slash;\n+\tint k;\n+\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t/*\n+\t * Build a list of all filesystem changes into a private/local\n+\t * list and without holding any locks.\n+\t */\n+\tfor (k = 0; k < num_of_events; k++) {\n+\t\t/*\n+\t\t * On Mac, we receive an array of absolute paths.\n+\t\t */\n+\t\tpath_k = paths[k];\n+\n+\t\t/*\n+\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n+\t\t * Please don't log them to Trace2.\n+\t\t *\n+\t\t * trace_printf_key(&trace_fsmonitor, \"Path: '%s'\", path_k);\n+\t\t */\n+\n+\t\t/*\n+\t\t * If event[k] is marked as dropped, we assume that we have\n+\t\t * lost sync with the filesystem and should flush our cached\n+\t\t * data. We need to:\n+\t\t *\n+\t\t * [1] Abort/wake any client threads waiting for a cookie and\n+\t\t * flush the cached state data (the current token), and\n+\t\t * create a new token.\n+\t\t *\n+\t\t * [2] Discard the batch that we were locally building (since\n+\t\t * they are conceptually relative to the just flushed\n+\t\t * token).\n+\t\t */\n+\t\tif (ef_is_dropped(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * see also kFSEventStreamEventFlagMustScanSubDirs\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"event: dropped\");\n+\n+\t\t\tfsmonitor_force_resync(state);\n+\t\t\tfsmonitor_batch__pop(batch);\n+\t\t\tstring_list_clear(&cookie_list, 0);\n+\n+\t\t\t/*\n+\t\t\t * We assume that any events that we received\n+\t\t\t * in this callback after this dropped event\n+\t\t\t * may still be valid, so we continue rather\n+\t\t\t * than break. (And just in case there is a\n+\t\t\t * delete of \".git\" hiding in there.)\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n+\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git or gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path_k);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path_k);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\t/* ignore all other paths inside of .git or gitdir */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\tcase IS_GITDIR:\n+\t\t\t/*\n+\t\t\t * If .git directory is deleted or renamed away,\n+\t\t\t * we have to quit.\n+\t\t\t */\n+\t\t\tif (ef_is_root_delete(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir renamed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* try to queue normal pathnames */\n+\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\t/*\n+\t\t\t * Because of the implicit \"binning\" (the\n+\t\t\t * kernel calls us at a given frequency) and\n+\t\t\t * de-duping (the kernel is free to combine\n+\t\t\t * multiple events for a given pathname), an\n+\t\t\t * individual fsevent could be marked as both\n+\t\t\t * a file and directory. Add it to the queue\n+\t\t\t * with both spellings so that the client will\n+\t\t\t * know how much to invalidate/refresh.\n+\t\t\t */\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t}\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tstrbuf_reset(&tmp);\n+\t\t\t\tstrbuf_addstr(&tmp, rel);\n+\t\t\t\tstrbuf_addch(&tmp, '/');\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\n+\t\tcase IS_OUTSIDE_CONE:\n+\t\tdefault:\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignoring '%s'\", path_k);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+\n+force_shutdown:\n+\tfsmonitor_batch__pop(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\n+\tdata->shutdown_style = FORCE_SHUTDOWN;\n+\tCFRunLoopStop(data->rl);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+}\n+\n+/*\n+ * NEEDSWORK: Investigate the proper value for the `latency` argument\n+ * in the call to `FSEventStreamCreate()`. I'm not sure that this\n+ * needs to be a config setting or just something that we tune after\n+ * some testing.\n+ *\n+ * With a latency of 0.1, I was seeing lots of dropped events during\n+ * the \"touch 100000\" files test within t/perf/p7519, but with a\n+ * latency of 0.001 I did not see any dropped events. So the\n+ * \"correct\" value may be somewhere in between.\n+ *\n+ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n+ */\n \n int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n+\t\tkFSEventStreamCreateFlagWatchRoot |\n+\t\tkFSEventStreamCreateFlagFileEvents;\n+\tFSEventStreamContext ctx = {\n+\t\t0,\n+\t\tstate,\n+\t\tNULL,\n+\t\tNULL,\n+\t\tNULL\n+\t};\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\tconst void *dir_array[2];\n+\n+\tCALLOC_ARRAY(data, 1);\n+\tstate->backend_data = data;\n+\n+\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n+\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n+\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n+\t\t\tNULL, state->path_gitdir_watch.buf,\n+\t\t\tkCFStringEncodingUTF8);\n+\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n+\t}\n+\n+\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n+\t\t\t\t\t\t data->nr_paths_watching,\n+\t\t\t\t\t\t NULL);\n+\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n+\t\t\t\t\t data->cfar_paths_to_watch,\n+\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n+\t\t\t\t\t 0.001, flags);\n+\tif (data->stream == NULL)\n+\t\tgoto failed;\n+\n+\t/*\n+\t * `data->rl` needs to be set inside the listener thread.\n+\t */\n+\n+\treturn 0;\n+\n+failed:\n+\terror(\"Unable to create FSEventStream.\");\n+\n+\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n }\n \n void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tif (data->stream) {\n+\t\tif (data->stream_started)\n+\t\t\tFSEventStreamStop(data->stream);\n+\t\tif (data->stream_scheduled)\n+\t\t\tFSEventStreamInvalidate(data->stream);\n+\t\tFSEventStreamRelease(data->stream);\n+\t}\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n \n void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\tdata->shutdown_style = SHUTDOWN_EVENT;\n+\n+\tCFRunLoopStop(data->rl);\n }\n \n void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\n+\tdata->rl = CFRunLoopGetCurrent();\n+\n+\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n+\tdata->stream_scheduled = 1;\n+\n+\tif (!FSEventStreamStart(data->stream)) {\n+\t\terror(\"Failed to start the FSEventStream\");\n+\t\tgoto force_error_stop_without_loop;\n+\t}\n+\tdata->stream_started = 1;\n+\n+\tCFRunLoopRun();\n+\n+\tswitch (data->shutdown_style) {\n+\tcase FORCE_ERROR_STOP:\n+\t\tstate->error_code = -1;\n+\t\t/* fall thru */\n+\tcase FORCE_SHUTDOWN:\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\t/* fall thru */\n+\tcase SHUTDOWN_EVENT:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn;\n+\n+force_error_stop_without_loop:\n+\tstate->error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+\treturn;\n }\n-- \ngitgitgadget\n\n"},{"id":"425325","messageId":"21b2b4f941b20ff50fd7de0d777fa89d27729d6d.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 18/28] fsmonitor-fs-listen-macos: add macos header files for FSEvent","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:56:57Z","receivedAt":"2021-05-22T13:57:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInclude MacOS system declarations to allow us to use FSEvent and\nCoreFoundation APIs. We need GCC and clang versions because of\ncompiler and header file conflicts.\n\nWhile it is quite possible to #include Apple's CoreServices.h when\ncompiling C source code with clang, trying to build it with GCC\ncurrently fails with this error:\n\nIn file included\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n\nThe underlying reason is that GCC (rightfully) objects that an `enum`\nvalue such as `kAuthorizationExternalFormLength` is not a constant\n(because it is not, the preprocessor has no knowledge of it, only the\nactual C compiler does) and can therefore not be used to define the size\nof a C array.\n\nThis is a known problem and tracked in GCC's bug tracker:\nhttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nIn the meantime, let's not block things and go the slightly ugly route\nof declaring/defining the FSEvents constants, data structures and\nfunctions that we need, so that we can avoid above-mentioned issue.\n\nLet's do this _only_ for GCC, though, so that the CI/PR builds (which\nbuild both with clang and with GCC) can guarantee that we _are_ using\nthe correct data types.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 96 ++++++++++++++++++++\n 1 file changed, 96 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\nindex b91058d1c4f8..bec5130d9e1d 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n@@ -1,3 +1,99 @@\n+#if defined(__GNUC__)\n+/*\n+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n+ * with clang, but not with GCC as of time of writing.\n+ *\n+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details.\n+ */\n+typedef unsigned int FSEventStreamCreateFlags;\n+#define kFSEventStreamEventFlagNone 0x00000000\n+#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n+#define kFSEventStreamEventFlagUserDropped 0x00000002\n+#define kFSEventStreamEventFlagKernelDropped 0x00000004\n+#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n+#define kFSEventStreamEventFlagHistoryDone 0x00000010\n+#define kFSEventStreamEventFlagRootChanged 0x00000020\n+#define kFSEventStreamEventFlagMount 0x00000040\n+#define kFSEventStreamEventFlagUnmount 0x00000080\n+#define kFSEventStreamEventFlagItemCreated 0x00000100\n+#define kFSEventStreamEventFlagItemRemoved 0x00000200\n+#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n+#define kFSEventStreamEventFlagItemRenamed 0x00000800\n+#define kFSEventStreamEventFlagItemModified 0x00001000\n+#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n+#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n+#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n+#define kFSEventStreamEventFlagItemIsFile 0x00010000\n+#define kFSEventStreamEventFlagItemIsDir 0x00020000\n+#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n+#define kFSEventStreamEventFlagOwnEvent 0x00080000\n+#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n+#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+\n+typedef struct __FSEventStream *FSEventStreamRef;\n+typedef const FSEventStreamRef ConstFSEventStreamRef;\n+\n+typedef unsigned int CFStringEncoding;\n+#define kCFStringEncodingUTF8 0x08000100\n+\n+typedef const struct __CFString *CFStringRef;\n+typedef const struct __CFArray *CFArrayRef;\n+typedef const struct __CFRunLoop *CFRunLoopRef;\n+\n+struct FSEventStreamContext {\n+ long long version;\n+ void *cb_data, *retain, *release, *copy_description;\n+};\n+\n+typedef struct FSEventStreamContext FSEventStreamContext;\n+typedef unsigned int FSEventStreamEventFlags;\n+#define kFSEventStreamCreateFlagNoDefer 0x02\n+#define kFSEventStreamCreateFlagWatchRoot 0x04\n+#define kFSEventStreamCreateFlagFileEvents 0x10\n+\n+typedef unsigned long long FSEventStreamEventId;\n+#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL\n+\n+typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef,\n+\t\t\t\t void *context,\n+\t\t\t\t __SIZE_TYPE__ num_of_events,\n+\t\t\t\t void *event_paths,\n+\t\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t\t const FSEventStreamEventId event_ids[]);\n+typedef double CFTimeInterval;\n+FSEventStreamRef FSEventStreamCreate(void *allocator,\n+\t\t\t\t FSEventStreamCallback callback,\n+\t\t\t\t FSEventStreamContext *context,\n+\t\t\t\t CFArrayRef paths_to_watch,\n+\t\t\t\t FSEventStreamEventId since_when,\n+\t\t\t\t CFTimeInterval latency,\n+\t\t\t\t FSEventStreamCreateFlags flags);\n+CFStringRef CFStringCreateWithCString(void *allocator, const char *string,\n+\t\t\t\t CFStringEncoding encoding);\n+CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count,\n+\t\t\t void *callbacks);\n+void CFRunLoopRun(void);\n+void CFRunLoopStop(CFRunLoopRef run_loop);\n+CFRunLoopRef CFRunLoopGetCurrent(void);\n+extern CFStringRef kCFRunLoopDefaultMode;\n+void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream,\n+\t\t\t\t CFRunLoopRef run_loop,\n+\t\t\t\t CFStringRef run_loop_mode);\n+unsigned char FSEventStreamStart(FSEventStreamRef stream);\n+void FSEventStreamStop(FSEventStreamRef stream);\n+void FSEventStreamInvalidate(FSEventStreamRef stream);\n+void FSEventStreamRelease(FSEventStreamRef stream);\n+#else\n+/*\n+ * Let Apple's headers declare `isalnum()` first, before\n+ * Git's headers override it via a constant\n+ */\n+#include \n+#include \n+#include \n+#endif\n+\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-fs-listen.h\"\n-- \ngitgitgadget\n\n"},{"id":"425326","messageId":"102e17cbc87533ab7e95d32c1c258404b73252e5.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 23/28] fsmonitor: enhance existing comments","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:57:02Z","receivedAt":"2021-05-22T13:57:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 37 ++++++++++++++++++++++++++++++-------\n 1 file changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex c6d3c34ad78e..29adf3e53ef3 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -344,9 +344,25 @@ void refresh_fsmonitor(struct index_state *istate)\n \t}\n \n apply_results:\n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n \tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -361,11 +377,15 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We received a trivial response, so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -373,7 +393,10 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"425327","messageId":"bb7b1912bb47331ac5fea0b4fd8259bb8964e807.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 22/28] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:57:01Z","receivedAt":"2021-05-22T13:57:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon client threads to create a cookie file\ninside the .git directory and then wait until FS events for the\ncookie are observed by the FS listener thread.\n\nThis helps address the racy nature of file system events by\nblocking the client response until the kernel has drained any\nevent backlog.\n\nThis is especially important on MacOS where kernel events are\nonly issued with a limited frequency. See the `latency` argument\nof `FSeventStreamCreate()`. The kernel only signals every `latency`\nseconds, but does not guarantee that the kernel queue is completely\ndrained, so we may have to wait more than one interval. If we\nincrease the frequency, the system is more likely to drop events.\nWe avoid these issues by having each client thread create a unique\ncookie file and then wait until it is seen in the event stream.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 210 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 5 +\n 2 files changed, 214 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex e807aa8f6741..985a82cf39e0 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -92,6 +92,149 @@ static int do_as_client__status(void)\n \t}\n }\n \n+enum fsmonitor_cookie_item_result {\n+\tFCIR_ERROR = -1, /* could not create cookie file ? */\n+\tFCIR_INIT = 0,\n+\tFCIR_SEEN,\n+\tFCIR_ABORT,\n+};\n+\n+struct fsmonitor_cookie_item {\n+\tstruct hashmap_entry entry;\n+\tconst char *name;\n+\tenum fsmonitor_cookie_item_result result;\n+};\n+\n+static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n+\t\t const struct hashmap_entry *he2, const void *keydata)\n+{\n+\tconst struct fsmonitor_cookie_item *a =\n+\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n+\tconst struct fsmonitor_cookie_item *b =\n+\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n+\n+\treturn strcmp(a->name, keydata ? keydata : b->name);\n+}\n+\n+static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(\n+\tstruct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint fd;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tstruct strbuf cookie_pathname = STRBUF_INIT;\n+\tstruct strbuf cookie_filename = STRBUF_INIT;\n+\tenum fsmonitor_cookie_item_result result;\n+\tint my_cookie_seq;\n+\n+\tCALLOC_ARRAY(cookie, 1);\n+\n+\tmy_cookie_seq = state->cookie_seq++;\n+\n+\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n+\n+\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n+\tstrbuf_addbuf(&cookie_pathname, &cookie_filename);\n+\n+\tcookie->name = strbuf_detach(&cookie_filename, NULL);\n+\tcookie->result = FCIR_INIT;\n+\thashmap_entry_init(&cookie->entry, strhash(cookie->name));\n+\n+\thashmap_add(&state->cookies, &cookie->entry);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n+\t\t\t cookie->name, cookie_pathname.buf);\n+\n+\t/*\n+\t * Create the cookie file on disk and then wait for a notification\n+\t * that the listener thread has seen it.\n+\t */\n+\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n+\tif (fd >= 0) {\n+\t\tclose(fd);\n+\t\tunlink(cookie_pathname.buf);\n+\n+\t\t/*\n+\t\t * NEEDSWORK: This is an infinite wait (well, unless another\n+\t\t * thread sends us an abort). I'd like to change this to\n+\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n+\t\t * and let the caller do the trivial response thing.\n+\t\t */\n+\t\twhile (cookie->result == FCIR_INIT)\n+\t\t\tpthread_cond_wait(&state->cookies_cond,\n+\t\t\t\t\t &state->main_lock);\n+\t} else {\n+\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n+\t\t\t cookie->name);\n+\n+\t\tcookie->result = FCIR_ERROR;\n+\t}\n+\n+\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n+\n+\tresult = cookie->result;\n+\n+\tfree((char*)cookie->name);\n+\tfree(cookie);\n+\tstrbuf_release(&cookie_pathname);\n+\n+\treturn result;\n+}\n+\n+/*\n+ * Mark these cookies as _SEEN and wake up the corresponding client threads.\n+ */\n+static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,\n+\t\t\t\t\t const struct string_list *cookie_names)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint k;\n+\tint nr_seen = 0;\n+\n+\tfor (k = 0; k < cookie_names->nr; k++) {\n+\t\tstruct fsmonitor_cookie_item key;\n+\t\tstruct fsmonitor_cookie_item *cookie;\n+\n+\t\tkey.name = cookie_names->items[k].string;\n+\t\thashmap_entry_init(&key.entry, strhash(key.name));\n+\n+\t\tcookie = hashmap_get_entry(&state->cookies, &key, entry, NULL);\n+\t\tif (cookie) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-seen: '%s'\",\n+\t\t\t\t\t cookie->name);\n+\t\t\tcookie->result = FCIR_SEEN;\n+\t\t\tnr_seen++;\n+\t\t}\n+\t}\n+\n+\tif (nr_seen)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n+/*\n+ * Set _ABORT on all pending cookies and wake up all client threads.\n+ */\n+static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct hashmap_iter iter;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tint nr_aborted = 0;\n+\n+\thashmap_for_each_entry(&state->cookies, &iter, cookie, entry) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-abort: '%s'\",\n+\t\t\t\t cookie->name);\n+\t\tcookie->result = FCIR_ABORT;\n+\t\tnr_aborted++;\n+\t}\n+\n+\tif (nr_aborted)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n /*\n * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n * \"token\" as a virtual timestamp. Clients can request a summary of all\n@@ -395,6 +538,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n *\n+ * [2] Some of those lost events may have been for cookie files. We\n+ * should assume the worst and abort them rather letting them starve.\n+ *\n * If there are no concurrent threads readering the current token data\n * series, we can free it now. Otherwise, let the last reader free\n * it.\n@@ -416,6 +562,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n \tstate->current_token_data = new_one;\n \n \tfsmonitor_free_token_data(free_me);\n+\n+\twith_lock__abort_all_cookies(state);\n }\n \n void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n@@ -490,6 +638,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tint hash_ret;\n \tint do_trivial = 0;\n \tint do_flush = 0;\n+\tint do_cookie = 0;\n+\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n@@ -522,6 +672,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t */\n \t\tdo_flush = 1;\n \t\tdo_trivial = 1;\n+\t\tdo_cookie = 1;\n \n \t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n \t\t/* assume V1 timestamp or garbage */\n@@ -535,6 +686,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n \t\t\t\t command);\n \t\tdo_trivial = 1;\n+\t\tdo_cookie = 1;\n \n \t} else {\n \t\t/* We have \"builtin:*\" */\n@@ -544,12 +696,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n \t\t\t\t\t command);\n \t\t\tdo_trivial = 1;\n+\t\t\tdo_cookie = 1;\n \n \t\t} else {\n \t\t\t/*\n \t\t\t * We have a V2 valid token:\n \t\t\t * \"builtin::\"\n \t\t\t */\n+\t\t\tdo_cookie = 1;\n \t\t}\n \t}\n \n@@ -558,6 +712,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tif (!state->current_token_data)\n \t\tBUG(\"fsmonitor state does not have a current token\");\n \n+\t/*\n+\t * Write a cookie file inside the directory being watched in\n+\t * an effort to flush out existing filesystem events that we\n+\t * actually care about. Suspend this client thread until we\n+\t * see the filesystem events for this cookie file.\n+\t *\n+\t * Creating the cookie lets us guarantee that our FS listener\n+\t * thread has drained the kernel queue and we are caught up\n+\t * with the kernel.\n+\t *\n+\t * If we cannot create the cookie (or otherwise guarantee that\n+\t * we are caught up), we send a trivial response. We have to\n+\t * assume that there might be some very, very recent activity\n+\t * on the FS still in flight.\n+\t */\n+\tif (do_cookie) {\n+\t\tcookie_result = with_lock__wait_for_cookie(state);\n+\t\tif (cookie_result != FCIR_SEEN) {\n+\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n+\t\t\t cookie_result);\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n \tif (do_flush)\n \t\twith_lock__do_force_resync(state);\n \n@@ -769,7 +947,9 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n-#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+#define FSMONITOR_DIR \"fsmonitor--daemon\"\n+#define FSMONITOR_COOKIE_DIR \"cookies\"\n+#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n \n enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n \tconst char *rel)\n@@ -922,6 +1102,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n+\tif (cookie_names->nr)\n+\t\twith_lock__mark_cookies_seen(state, cookie_names);\n+\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n@@ -1011,7 +1194,9 @@ static int fsmonitor_run_daemon(void)\n \n \tmemset(&state, 0, sizeof(state));\n \n+\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n+\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n@@ -1035,6 +1220,23 @@ static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n+\t/*\n+\t * We will write filesystem syncing cookie files into\n+\t * ///-.\n+\t */\n+\tstrbuf_init(&state.path_cookie_prefix, 0);\n+\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1047,6 +1249,7 @@ static int fsmonitor_run_daemon(void)\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n+\tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsmonitor_fs_listen__dtor(&state);\n \n@@ -1054,6 +1257,11 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n+\tstrbuf_release(&state.path_cookie_prefix);\n+\n+\t/*\n+\t * NEEDSWORK: Consider \"rm -rf /\"\n+\t */\n \n \treturn err;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 89a9ef20b24b..e9fc099bae9c 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n \n \tstruct fsmonitor_token_data *current_token_data;\n \n+\tstruct strbuf path_cookie_prefix;\n+\tpthread_cond_t cookies_cond;\n+\tint cookie_seq;\n+\tstruct hashmap cookies;\n+\n \tint error_code;\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n-- \ngitgitgadget\n\n"},{"id":"425328","messageId":"c9159db718a7140afd150fe14d6d970c846d9b59.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 25/28] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:57:04Z","receivedAt":"2021-05-22T13:57:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 475 +++++++++++++++++++++++++++++++++++\n 1 file changed, 475 insertions(+)\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 000000000000..eaed44ebad63\n--- /dev/null\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -0,0 +1,475 @@\n+#!/bin/sh\n+\n+test_description='built-in file system watcher'\n+\n+. ./test-lib.sh\n+\n+git version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\" || {\n+\tskip_all=\"The built-in FSMonitor is not supported on this platform\"\n+\ttest_done\n+}\n+\n+kill_repo () {\n+\tr=$1\n+\tgit -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n+\trm -rf $1\n+\treturn 0\n+}\n+\n+start_daemon () {\n+\tcase \"$#\" in\n+\t\t1) r=\"-C $1\";;\n+\t\t*) r=\"\";\n+\tesac\n+\n+\tgit $r fsmonitor--daemon start || return $?\n+\tgit $r fsmonitor--daemon status || return $?\n+\n+\treturn 0\n+}\n+\n+test_expect_success 'explicit daemon start and stop' '\n+\ttest_when_finished \"kill_repo test_explicit\" &&\n+\n+\tgit init test_explicit &&\n+\tstart_daemon test_explicit &&\n+\n+\tgit -C test_explicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon start' '\n+\ttest_when_finished \"kill_repo test_implicit\" &&\n+\n+\tgit init test_implicit &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n+\n+\t# query will implicitly start the daemon.\n+\t#\n+\t# for test-script simplicity, we send a V1 timestamp rather than\n+\t# a V2 token. either way, the daemon response to any query contains\n+\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n+\t# but this test case is only concerned with whether the daemon was\n+\t# implicitly started.)\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n+\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n+\tnul_to_q actual.filtered &&\n+\tgrep \"builtin:\" actual.filtered &&\n+\n+\t# confirm that a daemon was started in the background.\n+\t#\n+\t# since the mechanism for starting the background daemon is platform\n+\t# dependent, just confirm that the foreground command received a\n+\t# response from the daemon.\n+\n+\tgrep :\\\"query/response-length\\\" .git/trace &&\n+\n+\tgit -C test_implicit fsmonitor--daemon status &&\n+\tgit -C test_implicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (delete .git)' '\n+\ttest_when_finished \"kill_repo test_implicit_1\" &&\n+\n+\tgit init test_implicit_1 &&\n+\n+\tstart_daemon test_implicit_1 &&\n+\n+\t# deleting the .git directory will implicitly stop the daemon.\n+\trm -rf test_implicit_1/.git &&\n+\n+\t# Create an empty .git directory so that the following Git command\n+\t# will stay relative to the `-C` directory. Without this, the Git\n+\t# command will (override the requested -C argument) and crawl out\n+\t# to the containing Git source tree. This would make the test\n+\t# result dependent upon whether we were using fsmonitor on our\n+\t# development worktree.\n+\n+\tsleep 1 &&\n+\tmkdir test_implicit_1/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (rename .git)' '\n+\ttest_when_finished \"kill_repo test_implicit_2\" &&\n+\n+\tgit init test_implicit_2 &&\n+\n+\tstart_daemon test_implicit_2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n+\n+\t# Create an empty .git directory so that the following Git command\n+\t# will stay relative to the `-C` directory. Without this, the Git\n+\t# command will (override the requested -C argument) and crawl out\n+\t# to the containing Git source tree. This would make the test\n+\t# result dependent upon whether we were using fsmonitor on our\n+\t# development worktree.\n+\n+\tsleep 1 &&\n+\tmkdir test_implicit_2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'cannot start multiple daemons' '\n+\ttest_when_finished \"kill_repo test_multiple\" &&\n+\n+\tgit init test_multiple &&\n+\n+\tstart_daemon test_multiple &&\n+\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n+\tgrep \"fsmonitor--daemon is already running\" actual &&\n+\n+\tgit -C test_multiple fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n+'\n+\n+test_expect_success 'setup' '\n+\t>tracked &&\n+\t>modified &&\n+\t>delete &&\n+\t>rename &&\n+\tmkdir dir1 &&\n+\t>dir1/tracked &&\n+\t>dir1/modified &&\n+\t>dir1/delete &&\n+\t>dir1/rename &&\n+\tmkdir dir2 &&\n+\t>dir2/tracked &&\n+\t>dir2/modified &&\n+\t>dir2/delete &&\n+\t>dir2/rename &&\n+\tmkdir dirtorename &&\n+\t>dirtorename/a &&\n+\t>dirtorename/b &&\n+\n+\tcat >.gitignore <<-\\EOF &&\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tEOF\n+\n+\tgit -c core.useBuiltinFSMonitor= add . &&\n+\ttest_tick &&\n+\tgit -c core.useBuiltinFSMonitor= commit -m initial &&\n+\n+\tgit config core.useBuiltinFSMonitor true\n+'\n+\n+test_expect_success 'update-index implicitly starts daemon' '\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n+\t\tgit update-index --fsmonitor &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_1\n+'\n+\n+test_expect_success 'status implicitly starts daemon' '\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n+\t\tgit status >actual &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_2\n+'\n+\n+edit_files() {\n+\techo 1 >modified\n+\techo 2 >dir1/modified\n+\techo 3 >dir2/modified\n+\t>dir1/untracked\n+}\n+\n+delete_files() {\n+\trm -f delete\n+\trm -f dir1/delete\n+\trm -f dir2/delete\n+}\n+\n+create_files() {\n+\techo 1 >new\n+\techo 2 >dir1/new\n+\techo 3 >dir2/new\n+}\n+\n+rename_files() {\n+\tmv rename renamed\n+\tmv dir1/rename dir1/renamed\n+\tmv dir2/rename dir2/renamed\n+}\n+\n+file_to_directory() {\n+\trm -f delete\n+\tmkdir delete\n+\techo 1 >delete/new\n+}\n+\n+directory_to_file() {\n+\trm -rf dir1\n+\techo 1 >dir1\n+}\n+\n+verify_status() {\n+\tgit status >actual &&\n+\tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n+\tGIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect &&\n+\ttest_cmp expect actual &&\n+\techo HELLO AFTER &&\n+\tcat .git/trace &&\n+\techo HELLO AFTER\n+}\n+\n+# The next few test cases confirm that our fsmonitor daemon sees each type\n+# of OS filesystem notification that we care about. At this layer we just\n+# ensure we are getting the OS notifications and do not try to confirm what\n+# is reported by `git status`.\n+#\n+# We run a simple query after modifying the filesystem just to introduce\n+# a bit of a delay so that the trace logging from the daemon has time to\n+# get flushed to disk.\n+#\n+# We `reset` and `clean` at the bottom of each test (and before stopping the\n+# daemon) because these commands might implicitly restart the daemon.\n+\n+clean_up_repo_and_stop_daemon () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+\tgit fsmonitor--daemon stop\n+\trm -f .git/trace\n+}\n+\n+test_expect_success 'edit some files' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tedit_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/modified$\" .git/trace &&\n+\tgrep \"^event: dir2/modified$\" .git/trace &&\n+\tgrep \"^event: modified$\" .git/trace &&\n+\tgrep \"^event: dir1/untracked$\" .git/trace\n+'\n+\n+test_expect_success 'create some files' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tcreate_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/new$\" .git/trace &&\n+\tgrep \"^event: dir2/new$\" .git/trace &&\n+\tgrep \"^event: new$\" .git/trace\n+'\n+\n+test_expect_success 'delete some files' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdelete_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/delete$\" .git/trace &&\n+\tgrep \"^event: dir2/delete$\" .git/trace &&\n+\tgrep \"^event: delete$\" .git/trace\n+'\n+\n+test_expect_success 'rename some files' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\trename_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/rename$\" .git/trace &&\n+\tgrep \"^event: dir2/rename$\" .git/trace &&\n+\tgrep \"^event: rename$\" .git/trace &&\n+\tgrep \"^event: dir1/renamed$\" .git/trace &&\n+\tgrep \"^event: dir2/renamed$\" .git/trace &&\n+\tgrep \"^event: renamed$\" .git/trace\n+'\n+\n+test_expect_success 'rename directory' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tmv dirtorename dirrenamed &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dirtorename/*$\" .git/trace &&\n+\tgrep \"^event: dirrenamed/*$\" .git/trace\n+'\n+\n+test_expect_success 'file changes to directory' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tfile_to_directory &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: delete$\" .git/trace &&\n+\tgrep \"^event: delete/new$\" .git/trace\n+'\n+\n+test_expect_success 'directory changes to a file' '\n+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdirectory_to_file &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1$\" .git/trace\n+'\n+\n+# The next few test cases exercise the token-resync code. When filesystem\n+# drops events (because of filesystem velocity or because the daemon isn't\n+# polling fast enough), we need to discard the cached data (relative to the\n+# current token) and start collecting events under a new token.\n+#\n+# the 'test-tool fsmonitor-client flush' command can be used to send a\n+# \"flush\" message to a running daemon and ask it to do a flush/resync.\n+\n+test_expect_success 'flush cached data' '\n+\ttest_when_finished \"kill_repo test_flush\" &&\n+\n+\tgit init test_flush &&\n+\n+\t(\n+\t\tGIT_TEST_FSMONITOR_TOKEN=true &&\n+\t\texport GIT_TEST_FSMONITOR_TOKEN &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace_daemon\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon test_flush\n+\t) &&\n+\n+\t# The daemon should have an initial token with no events in _0 and\n+\t# then a few (probably platform-specific number of) events in _1.\n+\t# These should both have the same .\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n+\tnul_to_q actual_q0 &&\n+\n+\ttouch test_flush/file_1 &&\n+\ttouch test_flush/file_2 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n+\tnul_to_q actual_q1 &&\n+\n+\tgrep \"file_1\" actual_q1 &&\n+\n+\t# Force a flush. This will change the , reset the , and\n+\t# flush the file data. Then create some events and ensure that the file\n+\t# again appears in the cache. It should have the new .\n+\n+\ttest-tool -C test_flush fsmonitor-client flush >flush_0 &&\n+\tnul_to_q flush_q0 &&\n+\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_2 &&\n+\tnul_to_q actual_q2 &&\n+\n+\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n+\n+\ttouch test_flush/file_3 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n+\tnul_to_q actual_q3 &&\n+\n+\tgrep \"file_3\" actual_q3\n+'\n+\n+# The next few test cases create repos where the .git directory is NOT\n+# inside the one of the working directory. That is, where .git is a file\n+# that points to a directory elsewhere. This happens for submodules and\n+# non-primary worktrees.\n+\n+test_expect_success 'setup worktree base' '\n+\tgit init wt-base &&\n+\techo 1 >wt-base/file1 &&\n+\tgit -C wt-base add file1 &&\n+\tgit -C wt-base commit -m \"c1\"\n+'\n+\n+test_expect_success 'worktree with .git file' '\n+\tgit -C wt-base worktree add ../wt-secondary &&\n+\n+\t(\n+\t\tGIT_TRACE2_PERF=\"$PWD/trace2_wt_secondary\" &&\n+\t\texport GIT_TRACE2_PERF &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$PWD/trace_wt_secondary\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon wt-secondary\n+\t) &&\n+\n+\tgit -C wt-secondary fsmonitor--daemon stop &&\n+\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"425329","messageId":"5b035c6e0d60157cae5da515a0b1ae366fc38e30.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 26/28] p7519: add fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:57:05Z","receivedAt":"2021-05-22T13:57:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRepeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\nthe \"Simple IPC\" interface.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 42 ++++++++++++++++++++++++++++++++++++---\n 1 file changed, 39 insertions(+), 3 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 5eb5044a103c..542ae61c99d2 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -24,7 +24,8 @@ test_description=\"Test core.fsmonitor\"\n # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\n # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an\n # absolute path to an integration. May be a space delimited list of\n-# absolute paths to integrations.\n+# absolute paths to integrations. (This hook or list of hooks does not\n+# include the built-in fsmonitor--daemon.)\n #\n # The big win for using fsmonitor is the elimination of the need to scan the\n # working directory looking for changed and untracked files. If the file\n@@ -135,10 +136,16 @@ test_expect_success \"one time repo setup\" '\n \n setup_for_fsmonitor() {\n \t# set INTEGRATION_SCRIPT depending on the environment\n-\tif test -n \"$INTEGRATION_PATH\"\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n \tthen\n+\t\tgit config core.useBuiltinFSMonitor true &&\n+\t\tINTEGRATION_SCRIPT=false\n+\telif test -n \"$INTEGRATION_PATH\"\n+\tthen\n+\t\tgit config core.useBuiltinFSMonitor false &&\n \t\tINTEGRATION_SCRIPT=\"$INTEGRATION_PATH\"\n \telse\n+\t\tgit config core.useBuiltinFSMonitor false &&\n \t\t#\n \t\t# Choose integration script based on existence of Watchman.\n \t\t# Fall back to an empty integration script.\n@@ -174,7 +181,10 @@ test_perf_w_drop_caches () {\n }\n \n test_fsmonitor_suite() {\n-\tif test -n \"$INTEGRATION_SCRIPT\"; then\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n+\tthen\n+\t\tDESC=\"builtin fsmonitor--daemon\"\n+\telif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n \t\tDESC=\"fsmonitor=disabled\"\n@@ -285,4 +295,30 @@ test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n \n+#\n+# Run a full set of perf tests using the built-in fsmonitor--daemon.\n+# It does not use the Hook API, so it has a different setup.\n+# Explicitly start the daemon here and before we start client commands\n+# so that we can later add custom tracing.\n+#\n+\n+test_lazy_prereq HAVE_FSMONITOR_DAEMON '\n+\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n+'\n+\n+if test_have_prereq HAVE_FSMONITOR_DAEMON\n+then\n+\tUSE_FSMONITOR_DAEMON=t\n+\n+\ttrace_start fsmonitor--daemon--server\n+\tgit fsmonitor--daemon start\n+\n+\ttrace_start fsmonitor--daemon--client\n+\ttest_expect_success \"setup for fsmonitor--daemon\" 'setup_for_fsmonitor'\n+\ttest_fsmonitor_suite\n+\n+\tgit fsmonitor--daemon stop\n+\ttrace_stop\n+fi\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"425330","messageId":"1483c68855cbdf2adfa27e1a429e1693ce3ba6ff.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 27/28] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:57:06Z","receivedAt":"2021-05-22T13:57:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate 2x2 test matrix with the untracked-cache and fsmonitor--daemon\nfeatures and a series of edits and verify that status output is\nidentical.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 97 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 97 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex eaed44ebad63..106adc2a7ee0 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -153,6 +153,8 @@ test_expect_success 'setup' '\n \t.gitignore\n \texpect*\n \tactual*\n+\tflush*\n+\ttrace*\n \tEOF\n \n \tgit -c core.useBuiltinFSMonitor= add . &&\n@@ -472,4 +474,99 @@ test_expect_success 'worktree with .git file' '\n \ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n '\n \n+# TODO Repeat one of the \"edit\" tests on wt-secondary and confirm that\n+# TODO we get the same events and behavior -- that is, that fsmonitor--daemon\n+# TODO correctly listens to events on both the working directory and to the\n+# TODO referenced GITDIR.\n+\n+test_expect_success 'cleanup worktrees' '\n+\tkill_repo wt-secondary &&\n+\tkill_repo wt-base\n+'\n+\n+# The next few tests perform arbitrary/contrived file operations and\n+# confirm that status is correct. That is, that the data (or lack of\n+# data) from fsmonitor doesn't cause incorrect results. And doesn't\n+# cause incorrect results when the untracked-cache is enabled.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n+\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n+\tgit update-index --no-fsmonitor &&\n+\ttest_might_fail git fsmonitor--daemon stop\n+'\n+\n+matrix_clean_up_repo () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+}\n+\n+matrix_try () {\n+\tuc=$1\n+\tfsm=$2\n+\tfn=$3\n+\n+\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n+\t\tmatrix_clean_up_repo &&\n+\t\t$fn &&\n+\t\tif test $uc = false -a $fsm = false\n+\t\tthen\n+\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n+\t\telse\n+\t\t\tgit status --porcelain=v1 >.git/actual.$fn\n+\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n+\t\tfi\n+\t'\n+\n+\treturn $?\n+}\n+\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+for uc_val in $uc_values\n+do\n+\tif test $uc_val = false\n+\tthen\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n+\t\t\tgit config core.untrackedcache false &&\n+\t\t\tgit update-index --no-untracked-cache\n+\t\t'\n+\telse\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n+\t\t\tgit config core.untrackedcache true &&\n+\t\t\tgit update-index --untracked-cache\n+\t\t'\n+\tfi\n+\n+\tfsm_values=\"false true\"\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tif test $fsm_val = false\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n+\t\t\t\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop 2>/dev/null\n+\t\t\t'\n+\t\telse\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n+\t\t\t\tgit config core.useBuiltinFSMonitor true &&\n+\t\t\t\tgit fsmonitor--daemon start &&\n+\t\t\t\tgit update-index --fsmonitor\n+\t\t\t'\n+\t\tfi\n+\n+\t\tmatrix_try $uc_val $fsm_val edit_files\n+\t\tmatrix_try $uc_val $fsm_val delete_files\n+\t\tmatrix_try $uc_val $fsm_val create_files\n+\t\tmatrix_try $uc_val $fsm_val rename_files\n+\t\tmatrix_try $uc_val $fsm_val file_to_directory\n+\t\tmatrix_try $uc_val $fsm_val directory_to_file\n+\tdone\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"425331","messageId":"96a3eab819f455e56643158cd135f03530b66be5.1621691828.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v2 28/28] t/perf: avoid copying builtin fsmonitor files into test repo","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-22T13:57:07Z","receivedAt":"2021-05-22T13:57:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDo not try to copy a fsmonitor--daemon socket from the current\ndevelopment directory into the test trash directory.\n\nWhen we run the perf suite without an explicit source repo set,\nwe copy of the current $GIT_DIR into the test trash directory.\nUnix domain sockets cannot be copied in that manner, so the test\nsetup fails.\n\nAdditionally, omit any other fsmonitor--daemon temp files inside\nthe $GIT_DIR directory.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/perf-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 601d9f67ddb0..3b97e3fc0f27 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -74,7 +74,7 @@ test_perf_copy_repo_contents () {\n \tfor stuff in \"$1\"/*\n \tdo\n \t\tcase \"$stuff\" in\n-\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees)\n+\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)\n \t\t\t;;\n \t\t*)\n \t\t\tcp -R \"$stuff\" \"$repo/.git/\" || exit 1\n-- \ngitgitgadget\n"},{"id":"425646","messageId":"xmqqmtsh9bhl.fsf@gitster.g","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/28] Builtin FSMonitor Feature","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-27T02:06:14Z","receivedAt":"2021-05-27T02:06:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"These new global symbols are introduced by the series, but never\nused outside the file they are added to:\n\nfsmonitor-ipc.o - fsmonitor_ipc__get_path\nfsmonitor-ipc.o - fsmonitor_ipc__get_state\nfsmonitor-ipc.o - fsmonitor_ipc__send_command\n\nPerhaps make them file-scope static?\n\nThanks.\n"},{"id":"426222","messageId":"nycvar.QRO.7.76.6.2106021320490.55@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"e4a263728773381a64b8662c0577a3f12eff4ca0.1621691828.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/28] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-06-02T11:24:38Z","receivedAt":"2021-06-02T11:25:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nI know you're on vacation, therefore I would like to apologize for adding\nto your post-vacation notification overload, but...\n\nOn Sat, 22 May 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> diff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\n> new file mode 100644\n> index 000000000000..e62901a85b5d\n> --- /dev/null\n> +++ b/fsmonitor-ipc.c\n> @@ -0,0 +1,179 @@\n> [...]\n> +\n> +int fsmonitor_ipc__send_query(const char *since_token,\n> +\t\t\t struct strbuf *answer)\n> +{\n> +\tint ret = -1;\n> +\tint tried_to_spawn = 0;\n> +\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n> +\tstruct ipc_client_connection *connection = NULL;\n> +\tstruct ipc_client_connect_options options\n> +\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n> +\n> +\toptions.wait_if_busy = 1;\n> +\toptions.wait_if_not_found = 0;\n> +\n> +\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n> +\n> +\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\",\n> +\t\t\t since_token);\n> +\n> +try_again:\n> +\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n> +\t\t\t\t &connection);\n> +\n> +\tswitch (state) {\n> +\tcase IPC_STATE__LISTENING:\n> +\t\tret = ipc_client_send_command_to_connection(\n> +\t\t\tconnection, since_token, strlen(since_token), answer);\n\nHere, `since_token` can be `NULL` (and hence the `strlen(since_token)` can\nlead to a segmentation fault). I ran into this situation while `git rebase\n-i --autostash` wanted to apply the stashed changes.\n\nSince I picked up your v2 and included it in Git for Windows v2.32.0-rc2,\nI needed this hotfix: https://github.com/git-for-windows/git/pull/3241\n\nThanks,\nDscho\n"},{"id":"426224","messageId":"nycvar.QRO.7.76.6.2106021325070.55@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"xmqqmtsh9bhl.fsf@gitster.g","subject":"Re: [PATCH v2 00/28] Builtin FSMonitor Feature","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-06-02T11:28:53Z","receivedAt":"2021-06-02T11:29:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 27 May 2021, Junio C Hamano wrote:\n\n> These new global symbols are introduced by the series, but never\n> used outside the file they are added to:\n>\n> fsmonitor-ipc.o - fsmonitor_ipc__get_path\n> fsmonitor-ipc.o - fsmonitor_ipc__get_state\n> fsmonitor-ipc.o - fsmonitor_ipc__send_command\n>\n> Perhaps make them file-scope static?\n\nGood idea!\n\nBy the way, GitGitGadget keeps getting confused by the fact that one of\nPeff's patches looks very similar to the tip commit of this here patch\nseries, and mislabels Jeff's PR as being closed.\n\nWould you terribly mind picking up v2 some time soon so that I do not have\nto click \"Reopen\" on Jeff's PR all the time (I had to reopen it a couple\ntimes already: https://github.com/gitgitgadget/git/pull/923, and I do lack\nthe time to teach GitGitGadget new tricks to avoid this mislabeling).\n\nThank you,\nDscho\n"},{"id":"427086","messageId":"xmqq35to52u6.fsf@gitster.g","threadId":"55420","inReplyTo":"d21af7ff842c2a2642ae42df7b972a8817b8a3e8.1621691828.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 10/28] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-11T06:32:01Z","receivedAt":"2021-06-11T06:32:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> +#include \"test-tool.h\"\n> +#include \"cache.h\"\n> +#include \"parse-options.h\"\n> +//#include \"fsmonitor.h\"\n> +#include \"fsmonitor-ipc.h\"\n> +//#include \"compat/fsmonitor/fsmonitor-fs-listen.h\"\n> +//#include \"fsmonitor--daemon.h\"\n> +//#include \"simple-ipc.h\"\n\nPlease never commit a commented-out code. Comments are for humans\n(and they shouldn't use the // style in this project)---use removal\nfor machines.\n\nThanks.\n\n\n"},{"id":"427384","messageId":"nycvar.QRO.7.76.6.2106142320070.57@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2106021320490.55@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 04/28] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-06-14T21:23:13Z","receivedAt":"2021-06-14T21:23:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Wed, 2 Jun 2021, Johannes Schindelin wrote:\n\n> I know you're on vacation, therefore I would like to apologize for adding\n> to your post-vacation notification overload, but...\n\nNow that you're back from vacation...\n\n> On Sat, 22 May 2021, Jeff Hostetler via GitGitGadget wrote:\n>\n> > From: Jeff Hostetler \n> >\n> > diff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\n> > new file mode 100644\n> > index 000000000000..e62901a85b5d\n> > --- /dev/null\n> > +++ b/fsmonitor-ipc.c\n> > @@ -0,0 +1,179 @@\n> > [...]\n> > +\n> > +int fsmonitor_ipc__send_query(const char *since_token,\n> > +\t\t\t struct strbuf *answer)\n> > +{\n> > +\tint ret = -1;\n> > +\tint tried_to_spawn = 0;\n> > +\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n> > +\tstruct ipc_client_connection *connection = NULL;\n> > +\tstruct ipc_client_connect_options options\n> > +\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n> > +\n> > +\toptions.wait_if_busy = 1;\n> > +\toptions.wait_if_not_found = 0;\n> > +\n> > +\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n> > +\n> > +\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\",\n> > +\t\t\t since_token);\n> > +\n> > +try_again:\n> > +\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n> > +\t\t\t\t &connection);\n> > +\n> > +\tswitch (state) {\n> > +\tcase IPC_STATE__LISTENING:\n> > +\t\tret = ipc_client_send_command_to_connection(\n> > +\t\t\tconnection, since_token, strlen(since_token), answer);\n>\n> Here, `since_token` can be `NULL` (and hence the `strlen(since_token)` can\n> lead to a segmentation fault). I ran into this situation while `git rebase\n> -i --autostash` wanted to apply the stashed changes.\n>\n> Since I picked up your v2 and included it in Git for Windows v2.32.0-rc2,\n> I needed this hotfix: https://github.com/git-for-windows/git/pull/3241\n\nI actually noticed another similar issue and fixed it in time for Git for\nWindows v2.32.0, but eventually figured out the actual culprit, with a\nmuch better fix:\n\n-- snip --\ncommit bc40a560d3c95040b55fd7be6fe5b7012d267f8f\nAuthor: Johannes Schindelin \nDate: Wed Jun 9 09:49:50 2021 +0200\n\n fixup! fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC\n\n In FSMonitor v1, we made sure to only use a valid `since_token` when\n querying the FSMonitor. This condition was accidentally lost in v2, and\n caused segmentation faults uncovered by Scalar's Functional Tests.\n\n I had tried to fix this in https://github.com/git-for-windows/pull/3241,\n but the fix was incomplete, and I had to follow up with\n https://github.com/git-for-windows/pull/3258. However, it turns out that\n both of them were actually only work-arounds; I should have dug deeper\n to figure out _why_ the `since_token` was no longer guaranteed not to be\n `NULL`, and I finally did.\n\n Signed-off-by: Johannes Schindelin \n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 22623fd228f..0b40643442e 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -290,8 +290,9 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n\n \tif (r->settings.use_builtin_fsmonitor > 0) {\n-\t\tquery_success = !fsmonitor_ipc__send_query(\n-\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\tquery_success = istate->fsmonitor_last_update &&\n+\t\t\t!fsmonitor_ipc__send_query(istate->fsmonitor_last_update,\n+\t\t\t\t\t\t &query_result);\n \t\tif (query_success) {\n \t\t\t/*\n \t\t\t * The response contains a series of nul terminated\n\n-- snap --\n\nWould you mind squashing this in when you re-roll?\n\nCiao,\nDscho\n"},{"id":"427385","messageId":"nycvar.QRO.7.76.6.2106142324250.57@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"7e097cebc14328bc1ec6e30fc164270e0889ff06.1621691828.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 07/28] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-06-14T21:28:18Z","receivedAt":"2021-06-14T21:28:26Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Sat, 22 May 2021, Johannes Schindelin via GitGitGadget wrote:\n\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 9c9b2abc9414..c6d3c34ad78e 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -3,6 +3,7 @@\n> #include \"dir.h\"\n> #include \"ewah/ewok.h\"\n> #include \"fsmonitor.h\"\n> +#include \"fsmonitor-ipc.h\"\n> #include \"run-command.h\"\n> #include \"strbuf.h\"\n>\n> @@ -231,6 +232,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n>\n> void refresh_fsmonitor(struct index_state *istate)\n> {\n> +\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n> \tstruct strbuf query_result = STRBUF_INIT;\n> \tint query_success = 0, hook_version = -1;\n> \tsize_t bol = 0; /* beginning of line */\n> @@ -247,6 +249,46 @@ void refresh_fsmonitor(struct index_state *istate)\n> \tistate->fsmonitor_has_run_once = 1;\n>\n> \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n> +\n> +\tif (r->settings.use_builtin_fsmonitor > 0) {\n> +\t\tquery_success = !fsmonitor_ipc__send_query(\n> +\t\t\tistate->fsmonitor_last_update, &query_result);\n\nAs I pointed out elsewhere in the thread, this is a slight change in\nbehavior: in the previous iteration, we had this call in\n`query_fsmonitor()`, which was only ever called if\n`istate->fsmonitor_last_update` is non-`NULL`.\n\nThe code in `fsmonitor_ipc__send_query()` does actually depend on this,\ntherefore we need this change to be squashed in:\n\n-- snip --\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 22623fd228f..0b40643442e 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -290,8 +290,9 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n\n \tif (r->settings.use_builtin_fsmonitor > 0) {\n-\t\tquery_success = !fsmonitor_ipc__send_query(\n-\t\t\tistate->fsmonitor_last_update, &query_result);\n+\t\tquery_success = istate->fsmonitor_last_update &&\n+\t\t\t!fsmonitor_ipc__send_query(istate->fsmonitor_last_update,\n+\t\t\t\t\t\t &query_result);\n \t\tif (query_success) {\n \t\t\t/*\n \t\t\t * The response contains a series of nul terminated\n-- snap --\n\nThanks,\nDscho\n\n> +\t\tif (query_success) {\n> +\t\t\t/*\n> +\t\t\t * The response contains a series of nul terminated\n> +\t\t\t * strings. The first is the new token.\n> +\t\t\t *\n> +\t\t\t * Use `char *buf` as an interlude to trick the CI\n> +\t\t\t * static analysis to let us use `strbuf_addstr()`\n> +\t\t\t * here (and only copy the token) rather than\n> +\t\t\t * `strbuf_addbuf()`.\n> +\t\t\t */\n> +\t\t\tbuf = query_result.buf;\n> +\t\t\tstrbuf_addstr(&last_update_token, buf);\n> +\t\t\tbol = last_update_token.len + 1;\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * The builtin daemon is not available on this\n> +\t\t\t * platform -OR- we failed to get a response.\n> +\t\t\t *\n> +\t\t\t * Generate a fake token (rather than a V1\n> +\t\t\t * timestamp) for the index extension. (If\n> +\t\t\t * they switch back to the hook API, we don't\n> +\t\t\t * want ambiguous state.)\n> +\t\t\t */\n> +\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n> +\t\t}\n> +\n> +\t\t/*\n> +\t\t * Regardless of whether we successfully talked to a\n> +\t\t * fsmonitor daemon or not, we skip over and do not\n> +\t\t * try to use the hook. The \"core.useBuiltinFSMonitor\"\n> +\t\t * config setting ALWAYS overrides the \"core.fsmonitor\"\n> +\t\t * hook setting.\n> +\t\t */\n> +\t\tgoto apply_results;\n> +\t}\n> +\n> \t/*\n> \t * This could be racy so save the date/time now and query_fsmonitor\n> \t * should be inclusive to ensure we don't miss potential changes.\n> @@ -301,6 +343,7 @@ void refresh_fsmonitor(struct index_state *istate)\n> \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n> \t}\n>\n> +apply_results:\n> \t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n> \tif (query_success && query_result.buf[bol] != '/') {\n> \t\t/* Mark all entries returned by the monitor as dirty */\n> diff --git a/repo-settings.c b/repo-settings.c\n> index f7fff0f5ab83..93aab92ff164 100644\n> --- a/repo-settings.c\n> +++ b/repo-settings.c\n> @@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r)\n> \t\tr->settings.core_multi_pack_index = value;\n> \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n>\n> +\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n> +\t\tr->settings.use_builtin_fsmonitor = 1;\n> +\n> \tif (!repo_config_get_bool(r, \"feature.manyfiles\", &value) && value) {\n> \t\tUPDATE_DEFAULT_BOOL(r->settings.index_version, 4);\n> \t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);\n> diff --git a/repository.h b/repository.h\n> index b385ca3c94b6..d6e7f61f9cf7 100644\n> --- a/repository.h\n> +++ b/repository.h\n> @@ -29,6 +29,8 @@ enum fetch_negotiation_setting {\n> struct repo_settings {\n> \tint initialized;\n>\n> +\tint use_builtin_fsmonitor;\n> +\n> \tint core_commit_graph;\n> \tint commit_graph_read_changed_paths;\n> \tint gc_write_commit_graph;\n> --\n> gitgitgadget\n>\n>\n"},{"id":"427387","messageId":"nycvar.QRO.7.76.6.2106142330000.57@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"bb7b1912bb47331ac5fea0b4fd8259bb8964e807.1621691828.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 22/28] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-06-14T21:42:50Z","receivedAt":"2021-06-14T21:43:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nI've found an issue with this patch that I unfortunately failed to fix in\ntime for Git for Windows v2.32.0. It is a subtle issue, and it only reared\nits head in the form of flakiness when running Scalar's Functional Tests\n(which is the most comprehensive test suite we have in the absence of\nproper integration tests for Git).\n\nThe issue is the change in behavior between the previous iteration and\nthis one when replying with the trivial response:\n\nOn Sat, 22 May 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n> index e807aa8f6741..985a82cf39e0 100644\n> --- a/builtin/fsmonitor--daemon.c\n> +++ b/builtin/fsmonitor--daemon.c\n> [...]\n> @@ -522,6 +672,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n> \t\t */\n> \t\tdo_flush = 1;\n> \t\tdo_trivial = 1;\n> +\t\tdo_cookie = 1;\n>\n> \t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n> \t\t/* assume V1 timestamp or garbage */\n> @@ -535,6 +686,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n> \t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n> \t\t\t\t command);\n> \t\tdo_trivial = 1;\n> +\t\tdo_cookie = 1;\n>\n> \t} else {\n> \t\t/* We have \"builtin:*\" */\n> @@ -544,12 +696,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n> \t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n> \t\t\t\t\t command);\n> \t\t\tdo_trivial = 1;\n> +\t\t\tdo_cookie = 1;\n\nIn the first iteration of this patch series, these three trivial responses\nwere sent without first writing a cookie file and then waiting for the\nevent to arrive.\n\nThe symptom of the issue here is that some of Scalar's Functional Test\nhung, ostensibly waiting for a cookie file that never arrived.\n\nI am not 100% clear on why this only happened in Scalar's Functional Tests\nand not in the regression test in Git's test suite, but here are my\nthoughts on this:\n\n- I _suspect_ that the .git/ directory gets deleted on the client side\n after receiving a trivial response and finishing the test case, and\n _before_ the daemon can receive the event. As the directory already got\n deleted, the event never arrives.\n\n- I _think_ that running the test cases in parallel (10 concurrent test\n cases, if memory serves) exacerbates this problem.\n\n- Simply _not_ writing that cookie file (i.e. removing those three\n `do_cookie = 1` assignments above is sufficient to let Scalar's\n Functional Tests pass.\n\n- Those trivial responses do not _actually_ need to be guarded behind that\n cookie file. The worst that could happen is for the FSMonitor daemon to\n over-report paths that need to be `lstat()`ed.\n\nTherefore, I would like to suggest this diff to be squashed into this\npatch in a re-roll:\n\n-- snip --\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 985a82cf39e..4afbb36fe61 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -672,7 +672,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t */\n \t\tdo_flush = 1;\n \t\tdo_trivial = 1;\n-\t\tdo_cookie = 1;\n\n \t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n \t\t/* assume V1 timestamp or garbage */\n@@ -686,7 +685,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n \t\t\t\t command);\n \t\tdo_trivial = 1;\n-\t\tdo_cookie = 1;\n\n \t} else {\n \t\t/* We have \"builtin:*\" */\n@@ -696,7 +694,6 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n \t\t\t\t\t command);\n \t\t\tdo_trivial = 1;\n-\t\t\tdo_cookie = 1;\n\n \t\t} else {\n \t\t\t/*\n-- snap --\n\nFor the record, the Windows installer and macOS/Linux packages available\nat https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.1 do come\nwith this fix.\n\nThanks,\nDscho\n\n>\n> \t\t} else {\n> \t\t\t/*\n> \t\t\t * We have a V2 valid token:\n> \t\t\t * \"builtin::\"\n> \t\t\t */\n> +\t\t\tdo_cookie = 1;\n> \t\t}\n> \t}\n>\n> @@ -558,6 +712,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n> \tif (!state->current_token_data)\n> \t\tBUG(\"fsmonitor state does not have a current token\");\n>\n> +\t/*\n> +\t * Write a cookie file inside the directory being watched in\n> +\t * an effort to flush out existing filesystem events that we\n> +\t * actually care about. Suspend this client thread until we\n> +\t * see the filesystem events for this cookie file.\n> +\t *\n> +\t * Creating the cookie lets us guarantee that our FS listener\n> +\t * thread has drained the kernel queue and we are caught up\n> +\t * with the kernel.\n> +\t *\n> +\t * If we cannot create the cookie (or otherwise guarantee that\n> +\t * we are caught up), we send a trivial response. We have to\n> +\t * assume that there might be some very, very recent activity\n> +\t * on the FS still in flight.\n> +\t */\n> +\tif (do_cookie) {\n> +\t\tcookie_result = with_lock__wait_for_cookie(state);\n> +\t\tif (cookie_result != FCIR_SEEN) {\n> +\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n> +\t\t\t cookie_result);\n> +\t\t\tdo_trivial = 1;\n> +\t\t}\n> +\t}\n> +\n> \tif (do_flush)\n> \t\twith_lock__do_force_resync(state);\n>\n> @@ -769,7 +947,9 @@ static int handle_client(void *data,\n> \treturn result;\n> }\n>\n> -#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n> +#define FSMONITOR_DIR \"fsmonitor--daemon\"\n> +#define FSMONITOR_COOKIE_DIR \"cookies\"\n> +#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n>\n> enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n> \tconst char *rel)\n> @@ -922,6 +1102,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n> \t\t}\n> \t}\n>\n> +\tif (cookie_names->nr)\n> +\t\twith_lock__mark_cookies_seen(state, cookie_names);\n> +\n> \tpthread_mutex_unlock(&state->main_lock);\n> }\n>\n> @@ -1011,7 +1194,9 @@ static int fsmonitor_run_daemon(void)\n>\n> \tmemset(&state, 0, sizeof(state));\n>\n> +\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n> \tpthread_mutex_init(&state.main_lock, NULL);\n> +\tpthread_cond_init(&state.cookies_cond, NULL);\n> \tstate.error_code = 0;\n> \tstate.current_token_data = fsmonitor_new_token_data();\n>\n> @@ -1035,6 +1220,23 @@ static int fsmonitor_run_daemon(void)\n> \t\tstate.nr_paths_watching = 2;\n> \t}\n>\n> +\t/*\n> +\t * We will write filesystem syncing cookie files into\n> +\t * ///-.\n> +\t */\n> +\tstrbuf_init(&state.path_cookie_prefix, 0);\n> +\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n> +\n> +\tstrbuf_addch(&state.path_cookie_prefix, '/');\n> +\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n> +\tmkdir(state.path_cookie_prefix.buf, 0777);\n> +\n> +\tstrbuf_addch(&state.path_cookie_prefix, '/');\n> +\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n> +\tmkdir(state.path_cookie_prefix.buf, 0777);\n> +\n> +\tstrbuf_addch(&state.path_cookie_prefix, '/');\n> +\n> \t/*\n> \t * Confirm that we can create platform-specific resources for the\n> \t * filesystem listener before we bother starting all the threads.\n> @@ -1047,6 +1249,7 @@ static int fsmonitor_run_daemon(void)\n> \terr = fsmonitor_run_daemon_1(&state);\n>\n> done:\n> +\tpthread_cond_destroy(&state.cookies_cond);\n> \tpthread_mutex_destroy(&state.main_lock);\n> \tfsmonitor_fs_listen__dtor(&state);\n>\n> @@ -1054,6 +1257,11 @@ static int fsmonitor_run_daemon(void)\n>\n> \tstrbuf_release(&state.path_worktree_watch);\n> \tstrbuf_release(&state.path_gitdir_watch);\n> +\tstrbuf_release(&state.path_cookie_prefix);\n> +\n> +\t/*\n> +\t * NEEDSWORK: Consider \"rm -rf /\"\n> +\t */\n>\n> \treturn err;\n> }\n> diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\n> index 89a9ef20b24b..e9fc099bae9c 100644\n> --- a/fsmonitor--daemon.h\n> +++ b/fsmonitor--daemon.h\n> @@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n>\n> \tstruct fsmonitor_token_data *current_token_data;\n>\n> +\tstruct strbuf path_cookie_prefix;\n> +\tpthread_cond_t cookies_cond;\n> +\tint cookie_seq;\n> +\tstruct hashmap cookies;\n> +\n> \tint error_code;\n> \tstruct fsmonitor_daemon_backend_data *backend_data;\n>\n> --\n> gitgitgadget\n>\n>\n"},{"id":"428203","messageId":"6d302977-5360-b257-c9c0-9744311c6fae@jeffhostetler.com","threadId":"55420","inReplyTo":"xmqqmtsh9bhl.fsf@gitster.g","subject":"Re: [PATCH v2 00/28] Builtin FSMonitor Feature","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-06-22T15:45:31Z","receivedAt":"2021-06-22T15:45:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/26/21 10:06 PM, Junio C Hamano wrote:\n> These new global symbols are introduced by the series, but never\n> used outside the file they are added to:\n> \n> fsmonitor-ipc.o - fsmonitor_ipc__get_path\n> fsmonitor-ipc.o - fsmonitor_ipc__get_state\n> fsmonitor-ipc.o - fsmonitor_ipc__send_command\n> \n> Perhaps make them file-scope static?\n\nI intended these to be part of the API for talking to a builtin\nFSMonitor. They are called from builtin/fsmonitor--daemon.c\nand t/helper/test-fsmonitor-client.c\n\nJeff\n"},{"id":"428903","messageId":"5db2c0390a657d5790cf24404201505c4ec3a829.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 02/34] fsmonitor--daemon: man page","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:12Z","receivedAt":"2021-07-01T14:47:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a manual page describing the `git fsmonitor--daemon` feature.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n 1 file changed, 75 insertions(+)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n\ndiff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\nnew file mode 100644\nindex 00000000000..154e7684daa\n--- /dev/null\n+++ b/Documentation/git-fsmonitor--daemon.txt\n@@ -0,0 +1,75 @@\n+git-fsmonitor--daemon(1)\n+========================\n+\n+NAME\n+----\n+git-fsmonitor--daemon - A Built-in File System Monitor\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git fsmonitor--daemon' start\n+'git fsmonitor--daemon' run\n+'git fsmonitor--daemon' stop\n+'git fsmonitor--daemon' status\n+\n+DESCRIPTION\n+-----------\n+\n+A daemon to watch the working directory for file and directory\n+changes using platform-specific file system notification facilities.\n+\n+This daemon communicates directly with commands like `git status`\n+using the link:technical/api-simple-ipc.html[simple IPC] interface\n+instead of the slower linkgit:githooks[5] interface.\n+\n+This daemon is built into Git so that no third-party tools are\n+required.\n+\n+OPTIONS\n+-------\n+\n+start::\n+\tStarts a daemon in the background.\n+\n+run::\n+\tRuns a daemon in the foreground.\n+\n+stop::\n+\tStops the daemon running in the current working\n+\tdirectory, if present.\n+\n+status::\n+\tExits with zero status if a daemon is watching the\n+\tcurrent working directory.\n+\n+REMARKS\n+-------\n+\n+This daemon is a long running process used to watch a single working\n+directory and maintain a list of the recently changed files and\n+directories. Performance of commands such as `git status` can be\n+increased if they just ask for a summary of changes to the working\n+directory and can avoid scanning the disk.\n+\n+When `core.useBuiltinFSMonitor` is set to `true` (see\n+linkgit:git-config[1]) commands, such as `git status`, will ask the\n+daemon for changes and automatically start it (if necessary).\n+\n+For more information see the \"File System Monitor\" section in\n+linkgit:git-update-index[1].\n+\n+CAVEATS\n+-------\n+\n+The fsmonitor daemon does not currently know about submodules and does\n+not know to filter out file system events that happen within a\n+submodule. If fsmonitor daemon is watching a super repo and a file is\n+modified within the working directory of a submodule, it will report\n+the change (as happening against the super repo). However, the client\n+will properly ignore these extra events, so performance may be affected\n+but it will not cause an incorrect result.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\n-- \ngitgitgadget\n\n"},{"id":"428904","messageId":"cafc71c8d7d03e69efc13e86f828031a76f6794f.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 01/34] simple-ipc: preparations for supporting binary messages.","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:11Z","receivedAt":"2021-07-01T14:47:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd `command_len` argument to the Simple IPC API.\n\nIn my original Simple IPC API, I assumed that the request\nwould always be a null-terminated string of text characters.\nThe command arg was just a `const char *`.\n\nI found a caller that would like to pass a binary command\nto the daemon, so I want to ammend the Simple IPC API to\ntake `const char *command, size_t command_len` and pass\nthat to the daemon. (Really, the first arg should just be\na `void *` or `const unsigned byte *` to make that clearer.)\n\nNote, the response side has always been a `struct strbuf`\nwhich includes the buffer and length, so we already support\nreturning a binary answer. (Yes, it feels a little weird\nreturning a binary buffer in a `strbuf`, but it works.)\n\nSigned-off-by: Jeff Hostetler \n---\n compat/simple-ipc/ipc-unix-socket.c | 14 +++++++-----\n compat/simple-ipc/ipc-win32.c | 14 +++++++-----\n simple-ipc.h | 7 ++++--\n t/helper/test-simple-ipc.c | 34 +++++++++++++++++++----------\n 4 files changed, 46 insertions(+), 23 deletions(-)\n\ndiff --git a/compat/simple-ipc/ipc-unix-socket.c b/compat/simple-ipc/ipc-unix-socket.c\nindex 1927e6ef4bc..4e28857a0a1 100644\n--- a/compat/simple-ipc/ipc-unix-socket.c\n+++ b/compat/simple-ipc/ipc-unix-socket.c\n@@ -168,7 +168,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection)\n \n int ipc_client_send_command_to_connection(\n \tstruct ipc_client_connection *connection,\n-\tconst char *message, struct strbuf *answer)\n+\tconst char *message, size_t message_len,\n+\tstruct strbuf *answer)\n {\n \tint ret = 0;\n \n@@ -176,7 +177,7 @@ int ipc_client_send_command_to_connection(\n \n \ttrace2_region_enter(\"ipc-client\", \"send-command\", NULL);\n \n-\tif (write_packetized_from_buf_no_flush(message, strlen(message),\n+\tif (write_packetized_from_buf_no_flush(message, message_len,\n \t\t\t\t\t connection->fd) < 0 ||\n \t packet_flush_gently(connection->fd) < 0) {\n \t\tret = error(_(\"could not send IPC command\"));\n@@ -197,7 +198,8 @@ done:\n \n int ipc_client_send_command(const char *path,\n \t\t\t const struct ipc_client_connect_options *options,\n-\t\t\t const char *message, struct strbuf *answer)\n+\t\t\t const char *message, size_t message_len,\n+\t\t\t struct strbuf *answer)\n {\n \tint ret = -1;\n \tenum ipc_active_state state;\n@@ -208,7 +210,9 @@ int ipc_client_send_command(const char *path,\n \tif (state != IPC_STATE__LISTENING)\n \t\treturn ret;\n \n-\tret = ipc_client_send_command_to_connection(connection, message, answer);\n+\tret = ipc_client_send_command_to_connection(connection,\n+\t\t\t\t\t\t message, message_len,\n+\t\t\t\t\t\t answer);\n \n \tipc_client_close_connection(connection);\n \n@@ -503,7 +507,7 @@ static int worker_thread__do_io(\n \tif (ret >= 0) {\n \t\tret = worker_thread_data->server_data->application_cb(\n \t\t\tworker_thread_data->server_data->application_data,\n-\t\t\tbuf.buf, do_io_reply_callback, &reply_data);\n+\t\t\tbuf.buf, buf.len, do_io_reply_callback, &reply_data);\n \n \t\tpacket_flush_gently(reply_data.fd);\n \t}\ndiff --git a/compat/simple-ipc/ipc-win32.c b/compat/simple-ipc/ipc-win32.c\nindex 8dc7bda087d..8e889d6a506 100644\n--- a/compat/simple-ipc/ipc-win32.c\n+++ b/compat/simple-ipc/ipc-win32.c\n@@ -208,7 +208,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection)\n \n int ipc_client_send_command_to_connection(\n \tstruct ipc_client_connection *connection,\n-\tconst char *message, struct strbuf *answer)\n+\tconst char *message, size_t message_len,\n+\tstruct strbuf *answer)\n {\n \tint ret = 0;\n \n@@ -216,7 +217,7 @@ int ipc_client_send_command_to_connection(\n \n \ttrace2_region_enter(\"ipc-client\", \"send-command\", NULL);\n \n-\tif (write_packetized_from_buf_no_flush(message, strlen(message),\n+\tif (write_packetized_from_buf_no_flush(message, message_len,\n \t\t\t\t\t connection->fd) < 0 ||\n \t packet_flush_gently(connection->fd) < 0) {\n \t\tret = error(_(\"could not send IPC command\"));\n@@ -239,7 +240,8 @@ done:\n \n int ipc_client_send_command(const char *path,\n \t\t\t const struct ipc_client_connect_options *options,\n-\t\t\t const char *message, struct strbuf *response)\n+\t\t\t const char *message, size_t message_len,\n+\t\t\t struct strbuf *response)\n {\n \tint ret = -1;\n \tenum ipc_active_state state;\n@@ -250,7 +252,9 @@ int ipc_client_send_command(const char *path,\n \tif (state != IPC_STATE__LISTENING)\n \t\treturn ret;\n \n-\tret = ipc_client_send_command_to_connection(connection, message, response);\n+\tret = ipc_client_send_command_to_connection(connection,\n+\t\t\t\t\t\t message, message_len,\n+\t\t\t\t\t\t response);\n \n \tipc_client_close_connection(connection);\n \n@@ -458,7 +462,7 @@ static int do_io(struct ipc_server_thread_data *server_thread_data)\n \tif (ret >= 0) {\n \t\tret = server_thread_data->server_data->application_cb(\n \t\t\tserver_thread_data->server_data->application_data,\n-\t\t\tbuf.buf, do_io_reply_callback, &reply_data);\n+\t\t\tbuf.buf, buf.len, do_io_reply_callback, &reply_data);\n \n \t\tpacket_flush_gently(reply_data.fd);\n \ndiff --git a/simple-ipc.h b/simple-ipc.h\nindex 2c48a5ee004..9c7330fcda0 100644\n--- a/simple-ipc.h\n+++ b/simple-ipc.h\n@@ -107,7 +107,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection);\n */\n int ipc_client_send_command_to_connection(\n \tstruct ipc_client_connection *connection,\n-\tconst char *message, struct strbuf *answer);\n+\tconst char *message, size_t message_len,\n+\tstruct strbuf *answer);\n \n /*\n * Used by the client to synchronously connect and send and receive a\n@@ -119,7 +120,8 @@ int ipc_client_send_command_to_connection(\n */\n int ipc_client_send_command(const char *path,\n \t\t\t const struct ipc_client_connect_options *options,\n-\t\t\t const char *message, struct strbuf *answer);\n+\t\t\t const char *message, size_t message_len,\n+\t\t\t struct strbuf *answer);\n \n /*\n * Simple IPC Server Side API.\n@@ -144,6 +146,7 @@ typedef int (ipc_server_reply_cb)(struct ipc_server_reply_data *,\n */\n typedef int (ipc_server_application_cb)(void *application_data,\n \t\t\t\t\tconst char *request,\n+\t\t\t\t\tsize_t request_len,\n \t\t\t\t\tipc_server_reply_cb *reply_cb,\n \t\t\t\t\tstruct ipc_server_reply_data *reply_data);\n \ndiff --git a/t/helper/test-simple-ipc.c b/t/helper/test-simple-ipc.c\nindex 42040ef81b1..91345180750 100644\n--- a/t/helper/test-simple-ipc.c\n+++ b/t/helper/test-simple-ipc.c\n@@ -112,7 +112,7 @@ static int app__slow_command(ipc_server_reply_cb *reply_cb,\n /*\n * The client sent a command followed by a (possibly very) large buffer.\n */\n-static int app__sendbytes_command(const char *received,\n+static int app__sendbytes_command(const char *received, size_t received_len,\n \t\t\t\t ipc_server_reply_cb *reply_cb,\n \t\t\t\t struct ipc_server_reply_data *reply_data)\n {\n@@ -123,6 +123,13 @@ static int app__sendbytes_command(const char *received,\n \tint errs = 0;\n \tint ret;\n \n+\t/*\n+\t * The test is setup to send:\n+\t * \"sendbytes\" SP \n+\t */\n+\tif (received_len < strlen(\"sendbytes \"))\n+\t\tBUG(\"received_len is short in app__sendbytes_command\");\n+\n \tif (skip_prefix(received, \"sendbytes \", &p))\n \t\tlen_ballast = strlen(p);\n \n@@ -160,7 +167,7 @@ static ipc_server_application_cb test_app_cb;\n * by this application.\n */\n static int test_app_cb(void *application_data,\n-\t\t const char *command,\n+\t\t const char *command, size_t command_len,\n \t\t ipc_server_reply_cb *reply_cb,\n \t\t struct ipc_server_reply_data *reply_data)\n {\n@@ -173,7 +180,7 @@ static int test_app_cb(void *application_data,\n \tif (application_data != (void*)&my_app_data)\n \t\tBUG(\"application_cb: application_data pointer wrong\");\n \n-\tif (!strcmp(command, \"quit\")) {\n+\tif (command_len == 4 && !strncmp(command, \"quit\", 4)) {\n \t\t/*\n \t\t * The client sent a \"quit\" command. This is an async\n \t\t * request for the server to shutdown.\n@@ -193,22 +200,23 @@ static int test_app_cb(void *application_data,\n \t\treturn SIMPLE_IPC_QUIT;\n \t}\n \n-\tif (!strcmp(command, \"ping\")) {\n+\tif (command_len == 4 && !strncmp(command, \"ping\", 4)) {\n \t\tconst char *answer = \"pong\";\n \t\treturn reply_cb(reply_data, answer, strlen(answer));\n \t}\n \n-\tif (!strcmp(command, \"big\"))\n+\tif (command_len == 3 && !strncmp(command, \"big\", 3))\n \t\treturn app__big_command(reply_cb, reply_data);\n \n-\tif (!strcmp(command, \"chunk\"))\n+\tif (command_len == 5 && !strncmp(command, \"chunk\", 5))\n \t\treturn app__chunk_command(reply_cb, reply_data);\n \n-\tif (!strcmp(command, \"slow\"))\n+\tif (command_len == 4 && !strncmp(command, \"slow\", 4))\n \t\treturn app__slow_command(reply_cb, reply_data);\n \n-\tif (starts_with(command, \"sendbytes \"))\n-\t\treturn app__sendbytes_command(command, reply_cb, reply_data);\n+\tif (command_len >= 10 && starts_with(command, \"sendbytes \"))\n+\t\treturn app__sendbytes_command(command, command_len,\n+\t\t\t\t\t reply_cb, reply_data);\n \n \treturn app__unhandled_command(command, reply_cb, reply_data);\n }\n@@ -488,7 +496,9 @@ static int client__send_ipc(void)\n \toptions.wait_if_busy = 1;\n \toptions.wait_if_not_found = 0;\n \n-\tif (!ipc_client_send_command(cl_args.path, &options, command, &buf)) {\n+\tif (!ipc_client_send_command(cl_args.path, &options,\n+\t\t\t\t command, strlen(command),\n+\t\t\t\t &buf)) {\n \t\tif (buf.len) {\n \t\t\tprintf(\"%s\\n\", buf.buf);\n \t\t\tfflush(stdout);\n@@ -556,7 +566,9 @@ static int do_sendbytes(int bytecount, char byte, const char *path,\n \tstrbuf_addstr(&buf_send, \"sendbytes \");\n \tstrbuf_addchars(&buf_send, byte, bytecount);\n \n-\tif (!ipc_client_send_command(path, options, buf_send.buf, &buf_resp)) {\n+\tif (!ipc_client_send_command(path, options,\n+\t\t\t\t buf_send.buf, buf_send.len,\n+\t\t\t\t &buf_resp)) {\n \t\tstrbuf_rtrim(&buf_resp);\n \t\tprintf(\"sent:%c%08d %s\\n\", byte, bytecount, buf_resp.buf);\n \t\tfflush(stdout);\n-- \ngitgitgadget\n\n"},{"id":"428907","messageId":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v2.git.1621691828.gitgitgadget@gmail.com","subject":"[PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:10Z","receivedAt":"2021-07-01T14:47:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V3 of my patch series to add a builtin FSMonitor daemon to Git. I\nrebased this series onto v2.32.0.\n\nV3 addresses most of the previous review comments and things we've learned\nfrom our experimental testing of V2. (A version of V2 was shipped as an\nexperimental feature in the v2.32.0-based releases of Git for Windows and\nVFS for Git.)\n\nThere are still a few items that I need to address, but that list is getting\nvery short.\n\nThe following items from my V2 cover have not yet been addressed:\n\n[ ] Revisit the how the client handles the IPC_STATE__NOT_LISTENING state\n(where a daemon appears to be running, but is non-responsive)\n\n[ ] Consider having daemon chdir() out of the working directory to avoid\ndirectory handle issues on Windows.\n\n[ ] On Windows, If the daemon is started as an elevated process, then client\ncommands might not have access to communicate with it.\n\n[ ] Review if/how we decide to shutdown the FSMonitor daemon after and a\nsignificant idle period.\n\nAlso, there are potential problems with the untracked-cache that we have\nbeen looking at. Concurrently, Tao Klerks independently noticed similar\nproblems with the untracked-cache and has reported/discussed them here on\nthe mailing list. I would like to get to the bottom of them before going\nfurther -- at this point I don't know they are related to FSMonitor or not.\n\nIn this version, the first commit updates the Simple IPC API to make it\neasier to pass binary data using {char *, size_t} rather than assuming that\nthe message is a null-terminated string. FSMonitor does not use binary\nmessages and doesn't really need this API change, but I thought it best to\nfix the API now before we have other callers of IPC.\n\nThis patch series contains 34 commits and is rather large. If it would help\nwith the review, I could try to divide it into a client-side part 1 and a\ndaemon-side part 2 -- if there is interest.\n\nJeff Hostetler (34):\n simple-ipc: preparations for supporting binary messages.\n fsmonitor--daemon: man page\n fsmonitor--daemon: update fsmonitor documentation\n fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n help: include fsmonitor--daemon feature flag in version info\n fsmonitor: config settings are repository-specific\n fsmonitor: use IPC to query the builtin FSMonitor daemon\n fsmonitor--daemon: add a built-in fsmonitor daemon\n fsmonitor--daemon: implement 'stop' and 'status' commands\n t/helper/fsmonitor-client: create IPC client to talk to FSMonitor\n Daemon\n fsmonitor-fs-listen-win32: stub in backend for Windows\n fsmonitor-fs-listen-macos: stub in backend for MacOS\n fsmonitor--daemon: implement 'run' command\n fsmonitor--daemon: implement 'start' command\n fsmonitor: do not try to operate on bare repos\n fsmonitor--daemon: add pathname classification\n fsmonitor--daemon: define token-ids\n fsmonitor--daemon: create token-based changed path cache\n fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows\n fsmonitor-fs-listen-macos: add macos header files for FSEvent\n fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS\n fsmonitor--daemon: implement handle_client callback\n t/helper/test-touch: add helper to touch a series of files\n t/perf/p7519: speed up test using \"test-tool touch\"\n t/perf: avoid copying builtin fsmonitor files into test repo\n t/perf/p7519: add fsmonitor--daemon test cases\n t7527: create test for fsmonitor--daemon\n fsmonitor--daemon: periodically truncate list of modified files\n fsmonitor--daemon: use a cookie file to sync with file system\n fsmonitor: enhance existing comments\n fsmonitor: force update index after large responses\n t7527: test status with untracked-cache and fsmonitor--daemon\n fsmonitor: handle shortname for .git\n t7527: test FS event reporing on MacOS WRT case and Unicode\n\n .gitignore | 1 +\n Documentation/config/core.txt | 56 +-\n Documentation/git-fsmonitor--daemon.txt | 75 +\n Documentation/git-update-index.txt | 27 +-\n Documentation/githooks.txt | 3 +-\n Makefile | 17 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 1567 ++++++++++++++++++\n builtin/update-index.c | 20 +-\n cache.h | 1 -\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 497 ++++++\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 663 ++++++++\n compat/fsmonitor/fsmonitor-fs-listen.h | 49 +\n compat/simple-ipc/ipc-unix-socket.c | 14 +-\n compat/simple-ipc/ipc-win32.c | 14 +-\n config.c | 14 -\n config.h | 1 -\n config.mak.uname | 4 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n environment.c | 1 -\n fsmonitor--daemon.h | 140 ++\n fsmonitor-ipc.c | 179 ++\n fsmonitor-ipc.h | 48 +\n fsmonitor.c | 189 ++-\n fsmonitor.h | 14 +-\n git.c | 1 +\n help.c | 4 +\n repo-settings.c | 48 +\n repository.h | 11 +\n simple-ipc.h | 7 +-\n t/README | 4 +-\n t/helper/test-fsmonitor-client.c | 121 ++\n t/helper/test-simple-ipc.c | 34 +-\n t/helper/test-tool.c | 2 +\n t/helper/test-tool.h | 2 +\n t/helper/test-touch.c | 126 ++\n t/perf/p7519-fsmonitor.sh | 51 +-\n t/perf/perf-lib.sh | 2 +-\n t/t7519-status-fsmonitor.sh | 38 +\n t/t7527-builtin-fsmonitor.sh | 679 ++++++++\n t/test-lib.sh | 6 +\n 41 files changed, 4618 insertions(+), 121 deletions(-)\n create mode 100644 Documentation/git-fsmonitor--daemon.txt\n create mode 100644 builtin/fsmonitor--daemon.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n create mode 100644 fsmonitor--daemon.h\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n create mode 100644 t/helper/test-fsmonitor-client.c\n create mode 100644 t/helper/test-touch.c\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\n\nbase-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-923%2Fjeffhostetler%2Fbuiltin-fsmonitor-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-923/jeffhostetler/builtin-fsmonitor-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/923\n\nRange-diff vs v2:\n\n 1: 763fa1ee7bb = 1: cafc71c8d7d simple-ipc: preparations for supporting binary messages.\n 2: fc180e8591b = 2: 5db2c0390a6 fsmonitor--daemon: man page\n 3: d56f3e91db9 = 3: 86413bfe347 fsmonitor--daemon: update fsmonitor documentation\n 4: e4a26372877 ! 4: dfcd3e5ac97 fsmonitor-ipc: create client routines for git-fsmonitor--daemon\n @@ fsmonitor-ipc.c (new)\n +\tstruct ipc_client_connection *connection = NULL;\n +\tstruct ipc_client_connect_options options\n +\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n ++\tconst char *tok = since_token ? since_token : \"\";\n ++\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n +\n +\toptions.wait_if_busy = 1;\n +\toptions.wait_if_not_found = 0;\n +\n +\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n -+\n -+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\",\n -+\t\t\t since_token);\n ++\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n +\n +try_again:\n +\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n @@ fsmonitor-ipc.c (new)\n +\tswitch (state) {\n +\tcase IPC_STATE__LISTENING:\n +\t\tret = ipc_client_send_command_to_connection(\n -+\t\t\tconnection, since_token, strlen(since_token), answer);\n ++\t\t\tconnection, tok, tok_len, answer);\n +\t\tipc_client_close_connection(connection);\n +\n +\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n @@ fsmonitor-ipc.c (new)\n +\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n +\tint ret;\n +\tenum ipc_active_state state;\n ++\tconst char *c = command ? command : \"\";\n ++\tsize_t c_len = command ? strlen(command) : 0;\n +\n +\tstrbuf_reset(answer);\n +\n @@ fsmonitor-ipc.c (new)\n +\t\treturn -1;\n +\t}\n +\n -+\tret = ipc_client_send_command_to_connection(connection,\n -+\t\t\t\t\t\t command, strlen(command),\n ++\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n +\t\t\t\t\t\t answer);\n +\tipc_client_close_connection(connection);\n +\n +\tif (ret == -1) {\n -+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\",\n -+\t\t command);\n ++\t\tdie(\"could not send '%s' command to fsmonitor--daemon\", c);\n +\t\treturn -1;\n +\t}\n +\n 5: d5d09eb1635 ! 5: 0aaca2f9390 help: include fsmonitor--daemon feature flag in version info\n @@ help.c: void get_version_info(struct strbuf *buf, int show_build_options)\n \t}\n }\n \n +\n + ## t/test-lib.sh ##\n +@@ t/test-lib.sh: test_lazy_prereq REBASE_P '\n + # Tests that verify the scheduler integration must set this locally\n + # to avoid errors.\n + GIT_TEST_MAINT_SCHEDULER=\"none:exit 1\"\n ++\n ++# Does this platform support `git fsmonitor--daemon`\n ++#\n ++test_lazy_prereq FSMONITOR_DAEMON '\n ++\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n ++'\n 6: 67bcf57f594 ! 6: 8b64b7cd367 config: FSMonitor is repository-specific\n @@\n ## Metadata ##\n -Author: Johannes Schindelin \n +Author: Jeff Hostetler \n \n ## Commit message ##\n - config: FSMonitor is repository-specific\n + fsmonitor: config settings are repository-specific\n \n - This commit refactors `git_config_get_fsmonitor()` into the `repo_*()`\n - form that takes a parameter `struct repository *r`.\n + Move FSMonitor config settings to `struct repo_settings`, get rid of\n + the `core_fsmonitor` global variable, and add support for the new\n + `core.useBuiltinFSMonitor` config setting. Move config code to lookup\n + `core.fsmonitor` into `prepare_repo_settings()` with the rest of the\n + setup code.\n \n - That change prepares for the upcoming `core.useBuiltinFSMonitor` flag which\n - will be stored in the `repo_settings` struct.\n + The `core_fsmonitor` global variable was used to store the pathname to\n + the FSMonitor hook and it was used as a boolean to see if FSMonitor\n + was enabled. This dual usage will lead to confusion when we add\n + support for a builtin FSMonitor based on IPC, since the builtin\n + FSMonitor doesn't need the hook pathname.\n \n - Signed-off-by: Johannes Schindelin \n + Replace the boolean usage with an `enum fsmonitor_mode` to represent\n + the state of FSMonitor. And only set the pathname when in HOOK mode.\n +\n + Also, disable FSMonitor when the repository working directory is\n + incompatible. For example, in bare repositories, since there aren't\n + any files to watch.\n +\n + Signed-off-by: Jeff Hostetler \n \n ## builtin/update-index.c ##\n @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const char *prefix)\n @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const\n \n \tif (fsmonitor > 0) {\n -\t\tif (git_config_get_fsmonitor() == 0)\n -+\t\tif (repo_config_get_fsmonitor(r) == 0)\n ++\t\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE)\n ++\t\t\treturn error(\n ++\t\t\t\t_(\"repository is incompatible with fsmonitor\"));\n ++\n ++\t\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_DISABLED) {\n ++\t\t\twarning(_(\"core.useBuiltinFSMonitor is unset; \"\n ++\t\t\t\t\"set it if you really want to enable the \"\n ++\t\t\t\t\"builtin fsmonitor\"));\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n - \t\t\t\t\"set it if you really want to \"\n - \t\t\t\t\"enable fsmonitor\"));\n +-\t\t\t\t\"set it if you really want to \"\n +-\t\t\t\t\"enable fsmonitor\"));\n ++\t\t\t\t\"set it if you really want to enable the \"\n ++\t\t\t\t\"hook-based fsmonitor\"));\n ++\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n -\t\tif (git_config_get_fsmonitor() == 1)\n -+\t\tif (repo_config_get_fsmonitor(r) == 1)\n ++\t\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC)\n ++\t\t\twarning(_(\"core.useBuiltinFSMonitor is set; \"\n ++\t\t\t\t\"remove it if you really want to \"\n ++\t\t\t\t\"disable fsmonitor\"));\n ++\t\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\n \n + ## cache.h ##\n +@@ cache.h: extern int core_preload_index;\n + extern int precomposed_unicode;\n + extern int protect_hfs;\n + extern int protect_ntfs;\n +-extern const char *core_fsmonitor;\n + \n + extern int core_apply_sparse_checkout;\n + extern int core_sparse_checkout_cone;\n +\n ## config.c ##\n @@ config.c: int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n -int git_config_get_fsmonitor(void)\n -+int repo_config_get_fsmonitor(struct repository *r)\n - {\n +-{\n -\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n -+\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &core_fsmonitor))\n - \t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n - \n - \tif (core_fsmonitor && !*core_fsmonitor)\n +-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n +-\n +-\tif (core_fsmonitor && !*core_fsmonitor)\n +-\t\tcore_fsmonitor = NULL;\n +-\n +-\tif (core_fsmonitor)\n +-\t\treturn 1;\n +-\n +-\treturn 0;\n +-}\n +-\n + int git_config_get_index_threads(int *dest)\n + {\n + \tint is_bool, val;\n \n ## config.h ##\n @@ config.h: int git_config_get_index_threads(int *dest);\n @@ config.h: int git_config_get_index_threads(int *dest);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n -int git_config_get_fsmonitor(void);\n -+int repo_config_get_fsmonitor(struct repository *r);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\n \n + ## environment.c ##\n +@@ environment.c: int protect_hfs = PROTECT_HFS_DEFAULT;\n + #define PROTECT_NTFS_DEFAULT 1\n + #endif\n + int protect_ntfs = PROTECT_NTFS_DEFAULT;\n +-const char *core_fsmonitor;\n + \n + /*\n + * The character that begins a commented line in user-editable file\n +\n ## fsmonitor.c ##\n +@@\n + #include \"dir.h\"\n + #include \"ewah/ewok.h\"\n + #include \"fsmonitor.h\"\n ++#include \"fsmonitor-ipc.h\"\n + #include \"run-command.h\"\n + #include \"strbuf.h\"\n + \n +@@ fsmonitor.c: void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n + /*\n + * Call the query-fsmonitor hook passing the last update token of the saved results.\n + */\n +-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n ++static int query_fsmonitor_hook(struct repository *r,\n ++\t\t\t\tint version,\n ++\t\t\t\tconst char *last_update,\n ++\t\t\t\tstruct strbuf *query_result)\n + {\n + \tstruct child_process cp = CHILD_PROCESS_INIT;\n + \tint result;\n + \n +-\tif (!core_fsmonitor)\n ++\tif (r->settings.fsmonitor_mode != FSMONITOR_MODE_HOOK)\n + \t\treturn -1;\n + \n +-\tstrvec_push(&cp.args, core_fsmonitor);\n ++\tassert(r->settings.fsmonitor_hook_path);\n ++\tassert(*r->settings.fsmonitor_hook_path);\n ++\n ++\tstrvec_push(&cp.args, r->settings.fsmonitor_hook_path);\n + \tstrvec_pushf(&cp.args, \"%d\", version);\n + \tstrvec_pushf(&cp.args, \"%s\", last_update);\n + \tcp.use_shell = 1;\n +@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \tstruct strbuf last_update_token = STRBUF_INIT;\n + \tchar *buf;\n + \tunsigned int i;\n ++\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n + \n +-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n ++\tif (r->settings.fsmonitor_mode <= FSMONITOR_MODE_DISABLED ||\n ++\t istate->fsmonitor_has_run_once)\n + \t\treturn;\n + \n +-\thook_version = fsmonitor_hook_version();\n +-\n + \tistate->fsmonitor_has_run_once = 1;\n + \n + \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n ++\n ++\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC) {\n ++\t\t/* TODO */\n ++\t\treturn;\n ++\t}\n ++\n ++\tassert(r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK);\n ++\n ++\thook_version = fsmonitor_hook_version();\n ++\n + \t/*\n +-\t * This could be racy so save the date/time now and query_fsmonitor\n ++\t * This could be racy so save the date/time now and query_fsmonitor_hook\n + \t * should be inclusive to ensure we don't miss potential changes.\n + \t */\n + \tlast_update = getnanotime();\n +@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n + \n + \t/*\n +-\t * If we have a last update token, call query_fsmonitor for the set of\n ++\t * If we have a last update token, call query_fsmonitor_hook for the set of\n + \t * changes since that token, else assume everything is possibly dirty\n + \t * and check it all.\n + \t */\n + \tif (istate->fsmonitor_last_update) {\n + \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n +-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n ++\t\t\tquery_success = !query_fsmonitor_hook(\n ++\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n + \t\t\t\tistate->fsmonitor_last_update, &query_result);\n + \n + \t\t\tif (query_success) {\n +@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n + \t\t}\n + \n + \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n +-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n ++\t\t\tquery_success = !query_fsmonitor_hook(\n ++\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n + \t\t\t\tistate->fsmonitor_last_update, &query_result);\n + \t\t}\n + \n +-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n +-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n +-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n ++\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n ++\t\t\t\t\tr->settings.fsmonitor_hook_path);\n ++\t\ttrace_printf_key(&trace_fsmonitor,\n ++\t\t\t\t \"fsmonitor process '%s' returned %s\",\n ++\t\t\t\t r->settings.fsmonitor_hook_path,\n ++\t\t\t\t query_success ? \"success\" : \"failure\");\n + \t}\n + \n + \t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n @@ fsmonitor.c: void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n -\tint fsmonitor_enabled = git_config_get_fsmonitor();\n -+\tint fsmonitor_enabled = repo_config_get_fsmonitor(istate->repo ? istate->repo : the_repository);\n ++\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n ++\tint fsmonitor_enabled = r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED;\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n +@@ fsmonitor.c: void tweak_fsmonitor(struct index_state *istate)\n + \t\tistate->fsmonitor_dirty = NULL;\n + \t}\n + \n +-\tswitch (fsmonitor_enabled) {\n +-\tcase -1: /* keep: do nothing */\n +-\t\tbreak;\n +-\tcase 0: /* false */\n +-\t\tremove_fsmonitor(istate);\n +-\t\tbreak;\n +-\tcase 1: /* true */\n ++\tif (fsmonitor_enabled)\n + \t\tadd_fsmonitor(istate);\n +-\t\tbreak;\n +-\tdefault: /* unknown value: do nothing */\n +-\t\tbreak;\n +-\t}\n ++\telse\n ++\t\tremove_fsmonitor(istate);\n + }\n +\n + ## fsmonitor.h ##\n +@@ fsmonitor.h: int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n + */\n + static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n + {\n +-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n ++\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n ++\n ++\treturn r->settings.fsmonitor_mode <= FSMONITOR_MODE_DISABLED ||\n ++\t\tistate->fsmonitor_has_run_once;\n + }\n + \n + /*\n +@@ fsmonitor.h: static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n + */\n + static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n + {\n +-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n ++\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n ++\n ++\tif (r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED &&\n ++\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n + \t\tistate->cache_changed = 1;\n + \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n + \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n +@@ fsmonitor.h: static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n + */\n + static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n + {\n +-\tif (core_fsmonitor) {\n ++\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n ++\n ++\tif (r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED) {\n + \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n + \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n + \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\n +\n + ## repo-settings.c ##\n +@@\n + \n + #define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0)\n + \n ++/*\n ++ * Return 1 if the repo/workdir is incompatible with FSMonitor.\n ++ */\n ++static int is_repo_incompatible_with_fsmonitor(struct repository *r)\n ++{\n ++\tconst char *const_strval;\n ++\n ++\t/*\n ++\t * Bare repositories don't have a working directory and\n ++\t * therefore, nothing to watch.\n ++\t */\n ++\tif (!r->worktree)\n ++\t\treturn 1;\n ++\n ++\t/*\n ++\t * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n ++\t *\n ++\t * Granted, core Git does not know anything about GVFS and\n ++\t * we shouldn't make assumptions about a downstream feature,\n ++\t * but users can install both versions. And this can lead\n ++\t * to incorrect results from core Git commands. So, without\n ++\t * bringing in any of the GVFS code, do a simple config test\n ++\t * for a published config setting. (We do not look at the\n ++\t * various *_TEST_* environment variables.)\n ++\t */\n ++\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_strval))\n ++\t\treturn 1;\n ++\n ++\treturn 0;\n ++}\n ++\n + void prepare_repo_settings(struct repository *r)\n + {\n + \tint value;\n + \tchar *strval;\n ++\tconst char *const_strval;\n + \n + \tif (r->settings.initialized)\n + \t\treturn;\n +@@ repo-settings.c: void prepare_repo_settings(struct repository *r)\n + \tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n + \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n + \n ++\tr->settings.fsmonitor_hook_path = NULL;\n ++\tr->settings.fsmonitor_mode = FSMONITOR_MODE_DISABLED;\n ++\tif (is_repo_incompatible_with_fsmonitor(r))\n ++\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_INCOMPATIBLE;\n ++\telse if (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value)\n ++\t\t && value)\n ++\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_IPC;\n ++\telse {\n ++\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_strval))\n ++\t\t\tconst_strval = getenv(\"GIT_TEST_FSMONITOR\");\n ++\t\tif (const_strval && *const_strval) {\n ++\t\t\tr->settings.fsmonitor_hook_path = strdup(const_strval);\n ++\t\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_HOOK;\n ++\t\t}\n ++\t}\n ++\n + \tif (!repo_config_get_int(r, \"index.version\", &value))\n + \t\tr->settings.index_version = value;\n + \tif (!repo_config_get_maybe_bool(r, \"core.untrackedcache\", &value)) {\n +\n + ## repository.h ##\n +@@ repository.h: enum fetch_negotiation_setting {\n + \tFETCH_NEGOTIATION_NOOP = 3,\n + };\n + \n ++enum fsmonitor_mode {\n ++\tFSMONITOR_MODE_INCOMPATIBLE = -2,\n ++\tFSMONITOR_MODE_UNSET = -1,\n ++\tFSMONITOR_MODE_DISABLED = 0,\n ++\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n ++\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n ++};\n ++\n + struct repo_settings {\n + \tint initialized;\n + \n +@@ repository.h: struct repo_settings {\n + \tint gc_write_commit_graph;\n + \tint fetch_write_commit_graph;\n + \n ++\tenum fsmonitor_mode fsmonitor_mode;\n ++\tchar *fsmonitor_hook_path;\n ++\n + \tint index_version;\n + \tenum untracked_cache_setting core_untracked_cache;\n + \n +\n + ## t/README ##\n +@@ t/README: every 'git commit-graph write', as if the `--changed-paths` option was\n + passed in.\n + \n + GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n +-code path for utilizing a file system monitor to speed up detecting\n +-new or changed files.\n ++code path for utilizing a (hook based) file system monitor to speed up\n ++detecting new or changed files.\n + \n + GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n + for the index version specified. Can be set to any valid version\n +\n + ## t/t7519-status-fsmonitor.sh ##\n +@@ t/t7519-status-fsmonitor.sh: test_expect_success 'status succeeds after staging/unstaging' '\n + \t)\n + '\n + \n ++# Test that we detect and disallow repos that are incompatible with FSMonitor.\n ++test_expect_success 'incompatible bare repo' '\n ++\ttest_when_finished \"rm -rf ./bare-clone\" &&\n ++\tgit clone --bare . ./bare-clone &&\n ++\tcat >expect <<-\\EOF &&\n ++\terror: repository is incompatible with fsmonitor\n ++\tEOF\n ++\ttest_must_fail git -C ./bare-clone update-index --fsmonitor 2>actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'incompatible core.virtualfilesystem' '\n ++\ttest_when_finished \"rm -rf ./fake-gvfs-clone\" &&\n ++\tgit clone . ./fake-gvfs-clone &&\n ++\tgit -C ./fake-gvfs-clone config core.virtualfilesystem true &&\n ++\tcat >expect <<-\\EOF &&\n ++\terror: repository is incompatible with fsmonitor\n ++\tEOF\n ++\ttest_must_fail git -C ./fake-gvfs-clone update-index --fsmonitor 2>actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n + test_done\n 7: 7e097cebc14 ! 7: c86b5651ecc fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC\n @@\n ## Metadata ##\n -Author: Johannes Schindelin \n +Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC\n + fsmonitor: use IPC to query the builtin FSMonitor daemon\n \n Use simple IPC to directly communicate with the new builtin file\n - system monitor daemon.\n -\n - Define a new config setting `core.useBuiltinFSMonitor` to enable the\n - builtin file system monitor.\n + system monitor daemon when `core.useBuiltinFSMonitor` is set.\n \n The `core.fsmonitor` setting has already been defined as a HOOK\n pathname. Historically, this has been set to a HOOK script that will\n @@ Commit message\n Signed-off-by: Johannes Schindelin \n Signed-off-by: Jeff Hostetler \n \n - ## config.c ##\n -@@ config.c: int git_config_get_max_percent_split_change(void)\n - \n - int repo_config_get_fsmonitor(struct repository *r)\n - {\n -+\tif (r->settings.use_builtin_fsmonitor > 0) {\n -+\t\tcore_fsmonitor = \"(built-in daemon)\";\n -+\t\treturn 1;\n -+\t}\n -+\n - \tif (repo_config_get_pathname(r, \"core.fsmonitor\", &core_fsmonitor))\n - \t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n - \n -\n ## fsmonitor.c ##\n -@@\n - #include \"dir.h\"\n - #include \"ewah/ewok.h\"\n - #include \"fsmonitor.h\"\n -+#include \"fsmonitor-ipc.h\"\n - #include \"run-command.h\"\n - #include \"strbuf.h\"\n - \n -@@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n - \n - void refresh_fsmonitor(struct index_state *istate)\n - {\n -+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n - \tstruct strbuf query_result = STRBUF_INIT;\n - \tint query_success = 0, hook_version = -1;\n - \tsize_t bol = 0; /* beginning of line */\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n - \tistate->fsmonitor_has_run_once = 1;\n - \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n -+\n -+\tif (r->settings.use_builtin_fsmonitor > 0) {\n + \n + \tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC) {\n +-\t\t/* TODO */\n +-\t\treturn;\n +\t\tquery_success = !fsmonitor_ipc__send_query(\n -+\t\t\tistate->fsmonitor_last_update, &query_result);\n ++\t\t\tistate->fsmonitor_last_update ?\n ++\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n ++\t\t\t&query_result);\n +\t\tif (query_success) {\n +\t\t\t/*\n +\t\t\t * The response contains a series of nul terminated\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n +\t\t * hook setting.\n +\t\t */\n +\t\tgoto apply_results;\n -+\t}\n -+\n - \t/*\n - \t * This could be racy so save the date/time now and query_fsmonitor\n - \t * should be inclusive to ensure we don't miss potential changes.\n + \t}\n + \n + \tassert(r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK);\n @@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)\n - \t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n + \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n +apply_results:\n \t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n \tif (query_success && query_result.buf[bol] != '/') {\n \t\t/* Mark all entries returned by the monitor as dirty */\n -\n - ## repo-settings.c ##\n -@@ repo-settings.c: void prepare_repo_settings(struct repository *r)\n - \t\tr->settings.core_multi_pack_index = value;\n - \tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n - \n -+\tif (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value) && value)\n -+\t\tr->settings.use_builtin_fsmonitor = 1;\n -+\n - \tif (!repo_config_get_bool(r, \"feature.manyfiles\", &value) && value) {\n - \t\tUPDATE_DEFAULT_BOOL(r->settings.index_version, 4);\n - \t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);\n -\n - ## repository.h ##\n -@@ repository.h: enum fetch_negotiation_setting {\n - struct repo_settings {\n - \tint initialized;\n - \n -+\tint use_builtin_fsmonitor;\n -+\n - \tint core_commit_graph;\n - \tint commit_graph_read_changed_paths;\n - \tint gc_write_commit_graph;\n 8: f362a88632e = 8: f88db92d425 fsmonitor--daemon: add a built-in fsmonitor daemon\n 9: 4f401310539 ! 9: 02e21384ef0 fsmonitor--daemon: implement client command options\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor--daemon: implement client command options\n + fsmonitor--daemon: implement 'stop' and 'status' commands\n \n Implement `stop` and `status` client commands to control and query the\n status of a `fsmonitor--daemon` server process (and implicitly start a\n @@ builtin/fsmonitor--daemon.c\n +\n +\tswitch (state) {\n +\tcase IPC_STATE__LISTENING:\n -+\t\tprintf(_(\"The built-in file system monitor is active\\n\"));\n ++\t\tprintf(_(\"fsmonitor-daemon is watching '%s'\\n\"),\n ++\t\t the_repository->worktree);\n +\t\treturn 0;\n +\n +\tdefault:\n -+\t\tprintf(_(\"The built-in file system monitor is not active\\n\"));\n ++\t\tprintf(_(\"fsmonitor-daemon is not watching '%s'\\n\"),\n ++\t\t the_repository->worktree);\n +\t\treturn 1;\n +\t}\n +}\n 10: d21af7ff842 ! 10: c2adac8ed4b t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon\n @@ t/helper/test-fsmonitor-client.c (new)\n +#include \"test-tool.h\"\n +#include \"cache.h\"\n +#include \"parse-options.h\"\n -+//#include \"fsmonitor.h\"\n +#include \"fsmonitor-ipc.h\"\n -+//#include \"compat/fsmonitor/fsmonitor-fs-listen.h\"\n -+//#include \"fsmonitor--daemon.h\"\n -+//#include \"simple-ipc.h\"\n +\n +#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n +int cmd__fsmonitor_client(int argc, const char **argv)\n 11: 49f9e2e3d49 ! 11: 5a9bda72203 fsmonitor-fs-listen-win32: stub in backend for Windows\n @@ config.mak.uname: ifneq (,$(findstring MINGW,$(uname_S)))\n \n ## contrib/buildsystems/CMakeLists.txt ##\n @@ contrib/buildsystems/CMakeLists.txt: else()\n - \tlist(APPEND compat_SOURCES compat/simple-ipc/ipc-shared.c compat/simple-ipc/ipc-unix-socket.c)\n + \tendif()\n endif()\n \n +if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n 12: 2aa85151f03 = 12: 58758048947 fsmonitor-fs-listen-macos: stub in backend for MacOS\n 13: 2aa05ad5c67 ! 13: 5d6646df93a fsmonitor--daemon: implement daemon command options\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor--daemon: implement daemon command options\n + fsmonitor--daemon: implement 'run' command\n \n - Implement `run` and `start` commands to try to\n - begin listening for file system events.\n + Implement `run` command to try to begin listening for file system events.\n \n - This version defines the thread structure with a single\n - fsmonitor_fs_listen thread to watch for file system events\n - and a simple IPC thread pool to wait for connections from\n - Git clients over a well-known named pipe or Unix domain\n - socket.\n + This version defines the thread structure with a single fsmonitor_fs_listen\n + thread to watch for file system events and a simple IPC thread pool to\n + watch for connection from Git clients over a well-known named pipe or\n + Unix domain socket.\n \n - This version does not actually do anything yet because the\n + This commit does not actually do anything yet because the platform\n backends are still just stubs.\n \n Signed-off-by: Jeff Hostetler \n @@ builtin/fsmonitor--daemon.c\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n -+\tN_(\"git fsmonitor--daemon start []\"),\n +\tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n @@ builtin/fsmonitor--daemon.c\n +#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n +static int fsmonitor__ipc_threads = 8;\n +\n -+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n -+static int fsmonitor__start_timeout_sec = 60;\n -+\n +static int fsmonitor_config(const char *var, const char *value, void *cb)\n +{\n +\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n @@ builtin/fsmonitor--daemon.c\n +\t\treturn 0;\n +\t}\n +\n -+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n -+\t\tint i = git_config_int(var, value);\n -+\t\tif (i < 0)\n -+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n -+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n -+\t\tfsmonitor__start_timeout_sec = i;\n -+\t\treturn 0;\n -+\t}\n -+\n +\treturn git_default_config(var, value, cb);\n +}\n +\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n +\tstate.nr_paths_watching = 1;\n +\n +\t/*\n -+\t * We create/delete cookie files inside the .git directory to\n -+\t * help us keep sync with the file system. If \".git\" is not a\n -+\t * directory, then is not inside the cone of\n -+\t * , so set up a second watch for it.\n ++\t * We create and delete cookie files somewhere inside the .git\n ++\t * directory to help us keep sync with the file system. If\n ++\t * \".git\" is not a directory, then is not inside the\n ++\t * cone of , so set up a second watch to watch\n ++\t * the so that we get events for the cookie files.\n +\t */\n +\tstrbuf_init(&state.path_gitdir_watch, 0);\n +\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n +\t * common error case.\n +\t */\n +\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n -+\t\tdie(\"fsmonitor--daemon is already running.\");\n -+\n -+\treturn !!fsmonitor_run_daemon();\n -+}\n -+\n -+#ifndef GIT_WINDOWS_NATIVE\n -+/*\n -+ * This is adapted from `daemonize()`. Use `fork()` to directly create\n -+ * and run the daemon in a child process. The fork-parent returns the\n -+ * child PID so that we can wait for the child to startup before exiting.\n -+ */\n -+static int spawn_background_fsmonitor_daemon(pid_t *pid)\n -+{\n -+\t*pid = fork();\n -+\n -+\tswitch (*pid) {\n -+\tcase 0:\n -+\t\tif (setsid() == -1)\n -+\t\t\terror_errno(_(\"setsid failed\"));\n -+\t\tclose(0);\n -+\t\tclose(1);\n -+\t\tclose(2);\n -+\t\tsanitize_stdfds();\n -+\n -+\t\treturn !!fsmonitor_run_daemon();\n -+\n -+\tcase -1:\n -+\t\treturn error_errno(_(\"could not spawn fsmonitor--daemon in the background\"));\n -+\n -+\tdefault:\n -+\t\treturn 0;\n -+\t}\n -+}\n -+#else\n -+/*\n -+ * Conceptually like `daemonize()` but different because Windows does not\n -+ * have `fork(2)`. Spawn a normal Windows child process but without the\n -+ * limitations of `start_command()` and `finish_command()`.\n -+ */\n -+static int spawn_background_fsmonitor_daemon(pid_t *pid)\n -+{\n -+\tchar git_exe[MAX_PATH];\n -+\tstruct strvec args = STRVEC_INIT;\n -+\tint in, out;\n -+\n -+\tGetModuleFileNameA(NULL, git_exe, MAX_PATH);\n -+\n -+\tin = open(\"/dev/null\", O_RDONLY);\n -+\tout = open(\"/dev/null\", O_WRONLY);\n -+\n -+\tstrvec_push(&args, git_exe);\n -+\tstrvec_push(&args, \"fsmonitor--daemon\");\n -+\tstrvec_push(&args, \"run\");\n ++\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n ++\t\t the_repository->worktree);\n +\n -+\t*pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);\n -+\tclose(in);\n -+\tclose(out);\n ++\tprintf(_(\"running fsmonitor-daemon in '%s'\\n\"),\n ++\t the_repository->worktree);\n ++\tfflush(stdout);\n +\n -+\tstrvec_clear(&args);\n -+\n -+\tif (*pid < 0)\n -+\t\treturn error(_(\"could not spawn fsmonitor--daemon in the background\"));\n -+\n -+\treturn 0;\n -+}\n -+#endif\n -+\n -+/*\n -+ * This is adapted from `wait_or_whine()`. Watch the child process and\n -+ * let it get started and begin listening for requests on the socket\n -+ * before reporting our success.\n -+ */\n -+static int wait_for_background_startup(pid_t pid_child)\n -+{\n -+\tint status;\n -+\tpid_t pid_seen;\n -+\tenum ipc_active_state s;\n -+\ttime_t time_limit, now;\n -+\n -+\ttime(&time_limit);\n -+\ttime_limit += fsmonitor__start_timeout_sec;\n -+\n -+\tfor (;;) {\n -+\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n -+\n -+\t\tif (pid_seen == -1)\n -+\t\t\treturn error_errno(_(\"waitpid failed\"));\n -+\t\telse if (pid_seen == 0) {\n -+\t\t\t/*\n -+\t\t\t * The child is still running (this should be\n -+\t\t\t * the normal case). Try to connect to it on\n -+\t\t\t * the socket and see if it is ready for\n -+\t\t\t * business.\n -+\t\t\t *\n -+\t\t\t * If there is another daemon already running,\n -+\t\t\t * our child will fail to start (possibly\n -+\t\t\t * after a timeout on the lock), but we don't\n -+\t\t\t * care (who responds) if the socket is live.\n -+\t\t\t */\n -+\t\t\ts = fsmonitor_ipc__get_state();\n -+\t\t\tif (s == IPC_STATE__LISTENING)\n -+\t\t\t\treturn 0;\n -+\n -+\t\t\ttime(&now);\n -+\t\t\tif (now > time_limit)\n -+\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n -+\t\t} else if (pid_seen == pid_child) {\n -+\t\t\t/*\n -+\t\t\t * The new child daemon process shutdown while\n -+\t\t\t * it was starting up, so it is not listening\n -+\t\t\t * on the socket.\n -+\t\t\t *\n -+\t\t\t * Try to ping the socket in the odd chance\n -+\t\t\t * that another daemon started (or was already\n -+\t\t\t * running) while our child was starting.\n -+\t\t\t *\n -+\t\t\t * Again, we don't care who services the socket.\n -+\t\t\t */\n -+\t\t\ts = fsmonitor_ipc__get_state();\n -+\t\t\tif (s == IPC_STATE__LISTENING)\n -+\t\t\t\treturn 0;\n -+\n -+\t\t\t/*\n -+\t\t\t * We don't care about the WEXITSTATUS() nor\n -+\t\t\t * any of the WIF*(status) values because\n -+\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n -+\t\t\t * trick on all function return values.\n -+\t\t\t *\n -+\t\t\t * So it is sufficient to just report the\n -+\t\t\t * early shutdown as an error.\n -+\t\t\t */\n -+\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n -+\t\t} else\n -+\t\t\treturn error(_(\"waitpid is confused\"));\n -+\t}\n -+}\n -+\n -+static int try_to_start_background_daemon(void)\n -+{\n -+\tpid_t pid_child;\n -+\tint ret;\n -+\n -+\t/*\n -+\t * Before we try to create a background daemon process, see\n -+\t * if a daemon process is already listening. This makes it\n -+\t * easier for us to report an already-listening error to the\n -+\t * console, since our spawn/daemon can only report the success\n -+\t * of creating the background process (and not whether it\n -+\t * immediately exited).\n -+\t */\n -+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n -+\t\tdie(\"fsmonitor--daemon is already running.\");\n -+\n -+\t/*\n -+\t * Run the actual daemon in a background process.\n -+\t */\n -+\tret = spawn_background_fsmonitor_daemon(&pid_child);\n -+\tif (pid_child <= 0)\n -+\t\treturn ret;\n -+\n -+\t/*\n -+\t * Wait (with timeout) for the background child process get\n -+\t * started and begin listening on the socket/pipe. This makes\n -+\t * the \"start\" command more synchronous and more reliable in\n -+\t * tests.\n -+\t */\n -+\tret = wait_for_background_startup(pid_child);\n -+\n -+\treturn ret;\n ++\treturn !!fsmonitor_run_daemon();\n +}\n +\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n @@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)\n +\t\tOPT_INTEGER(0, \"ipc-threads\",\n +\t\t\t &fsmonitor__ipc_threads,\n +\t\t\t N_(\"use ipc worker threads\")),\n -+\t\tOPT_INTEGER(0, \"start-timeout\",\n -+\t\t\t &fsmonitor__start_timeout_sec,\n -+\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n -+\n \t\tOPT_END()\n \t};\n \n @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar\n +\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n +\t\t fsmonitor__ipc_threads);\n +\n -+\tif (!strcmp(subcmd, \"start\"))\n -+\t\treturn !!try_to_start_background_daemon();\n -+\n +\tif (!strcmp(subcmd, \"run\"))\n +\t\treturn !!try_to_run_foreground_daemon();\n \n -: ----------- > 14: 9fe902aad87 fsmonitor--daemon: implement 'start' command\n -: ----------- > 15: eef39aa168f fsmonitor: do not try to operate on bare repos\n 14: d5ababfd03e = 16: 3b12f668060 fsmonitor--daemon: add pathname classification\n 15: c092cdf2c8b = 17: 37fdce5ec3a fsmonitor--daemon: define token-ids\n 16: 2ed7bc3fae7 = 18: 84444c44c32 fsmonitor--daemon: create token-based changed path cache\n 17: 9ea4b04b821 ! 19: 5bba5eb3d1b fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows\n @@ compat/fsmonitor/fsmonitor-fs-listen-win32.c\n +\n +\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n +\n -+\t\ttrace_printf_key(&trace_fsmonitor, \"BBB: %s\", path.buf);\n -+\n +\t\tswitch (t) {\n +\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n +\t\t\t/* special case cookie files within gitdir */\n 18: 21b2b4f941b = 20: 175ae9a757e fsmonitor-fs-listen-macos: add macos header files for FSEvent\n 19: 08474bad830 = 21: 5d12b5d808a fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS\n 20: cc4a596d17c = 22: 39df123143b fsmonitor--daemon: implement handle_client callback\n -: ----------- > 23: 3cf8f3cd771 t/helper/test-touch: add helper to touch a series of files\n -: ----------- > 24: f1ef9656fc3 t/perf/p7519: speed up test using \"test-tool touch\"\n 28: 96a3eab819f = 25: a83485fb10f t/perf: avoid copying builtin fsmonitor files into test repo\n 26: 5b035c6e0d6 ! 26: de517a8259a p7519: add fsmonitor--daemon\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - p7519: add fsmonitor--daemon\n + t/perf/p7519: add fsmonitor--daemon test cases\n \n Repeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\n the \"Simple IPC\" interface.\n @@ t/perf/p7519-fsmonitor.sh: test_expect_success \"setup without fsmonitor\" '\n +# Explicitly start the daemon here and before we start client commands\n +# so that we can later add custom tracing.\n +#\n -+\n -+test_lazy_prereq HAVE_FSMONITOR_DAEMON '\n -+\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n -+'\n -+\n -+if test_have_prereq HAVE_FSMONITOR_DAEMON\n ++if test_have_prereq FSMONITOR_DAEMON\n +then\n +\tUSE_FSMONITOR_DAEMON=t\n +\n 25: c9159db718a ! 27: 99279c0ebd2 t7527: create test for fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\n +. ./test-lib.sh\n +\n -+git version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\" || {\n -+\tskip_all=\"The built-in FSMonitor is not supported on this platform\"\n ++if ! test_have_prereq FSMONITOR_DAEMON\n ++then\n ++\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n +\ttest_done\n -+}\n ++fi\n +\n -+kill_repo () {\n ++stop_daemon_delete_repo () {\n +\tr=$1\n +\tgit -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n +\trm -rf $1\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +}\n +\n +test_expect_success 'explicit daemon start and stop' '\n -+\ttest_when_finished \"kill_repo test_explicit\" &&\n ++\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n +\n +\tgit init test_explicit &&\n +\tstart_daemon test_explicit &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'implicit daemon start' '\n -+\ttest_when_finished \"kill_repo test_implicit\" &&\n ++\ttest_when_finished \"stop_daemon_delete_repo test_implicit\" &&\n +\n +\tgit init test_implicit &&\n +\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t# but this test case is only concerned with whether the daemon was\n +\t# implicitly started.)\n +\n -+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace\" \\\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace\" \\\n +\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n +\tnul_to_q actual.filtered &&\n +\tgrep \"builtin:\" actual.filtered &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'implicit daemon stop (delete .git)' '\n -+\ttest_when_finished \"kill_repo test_implicit_1\" &&\n ++\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n +\n +\tgit init test_implicit_1 &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t# deleting the .git directory will implicitly stop the daemon.\n +\trm -rf test_implicit_1/.git &&\n +\n -+\t# Create an empty .git directory so that the following Git command\n -+\t# will stay relative to the `-C` directory. Without this, the Git\n -+\t# command will (override the requested -C argument) and crawl out\n -+\t# to the containing Git source tree. This would make the test\n -+\t# result dependent upon whether we were using fsmonitor on our\n -+\t# development worktree.\n -+\n ++\t# [1] Create an empty .git directory so that the following Git\n ++\t# command will stay relative to the `-C` directory.\n ++\t#\n ++\t# Without this, the Git command will override the requested\n ++\t# -C argument and crawl out to the containing Git source tree.\n ++\t# This would make the test result dependent upon whether we\n ++\t# were using fsmonitor on our development worktree.\n ++\t#\n +\tsleep 1 &&\n +\tmkdir test_implicit_1/.git &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'implicit daemon stop (rename .git)' '\n -+\ttest_when_finished \"kill_repo test_implicit_2\" &&\n ++\ttest_when_finished \"stop_daemon_delete_repo test_implicit_2\" &&\n +\n +\tgit init test_implicit_2 &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t# renaming the .git directory will implicitly stop the daemon.\n +\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n +\n -+\t# Create an empty .git directory so that the following Git command\n -+\t# will stay relative to the `-C` directory. Without this, the Git\n -+\t# command will (override the requested -C argument) and crawl out\n -+\t# to the containing Git source tree. This would make the test\n -+\t# result dependent upon whether we were using fsmonitor on our\n -+\t# development worktree.\n -+\n ++\t# See [1] above.\n ++\t#\n +\tsleep 1 &&\n +\tmkdir test_implicit_2/.git &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'cannot start multiple daemons' '\n -+\ttest_when_finished \"kill_repo test_multiple\" &&\n ++\ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n +\n +\tgit init test_multiple &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n +'\n +\n ++# These tests use the main repo in the trash directory\n ++\n +test_expect_success 'setup' '\n +\t>tracked &&\n +\t>modified &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\tgit config core.useBuiltinFSMonitor true\n +'\n +\n ++# The test already explicitly stopped (or tried to stop) the daemon.\n ++# This is here in case something else fails first.\n ++#\n ++redundant_stop_daemon () {\n ++\tgit fsmonitor--daemon stop\n ++\treturn 0\n ++}\n ++\n +test_expect_success 'update-index implicitly starts daemon' '\n ++\ttest_when_finished redundant_stop_daemon &&\n ++\n +\ttest_must_fail git fsmonitor--daemon status &&\n +\n -+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_1\" \\\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_1\" \\\n +\t\tgit update-index --fsmonitor &&\n +\n +\tgit fsmonitor--daemon status &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'status implicitly starts daemon' '\n ++\ttest_when_finished redundant_stop_daemon &&\n ++\n +\ttest_must_fail git fsmonitor--daemon status &&\n +\n -+\tGIT_TRACE2_EVENT=\"$PWD/.git/trace_implicit_2\" \\\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_2\" \\\n +\t\tgit status >actual &&\n +\n +\tgit fsmonitor--daemon status &&\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +}\n +\n +test_expect_success 'edit some files' '\n -+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n ++\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n +\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n +\t\texport GIT_TRACE_FSMONITOR &&\n +\n +\t\tstart_daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'create some files' '\n -+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n ++\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n +\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n +\t\texport GIT_TRACE_FSMONITOR &&\n +\n +\t\tstart_daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'delete some files' '\n -+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n ++\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n +\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n +\t\texport GIT_TRACE_FSMONITOR &&\n +\n +\t\tstart_daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'rename some files' '\n -+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n ++\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n +\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n +\t\texport GIT_TRACE_FSMONITOR &&\n +\n +\t\tstart_daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'rename directory' '\n -+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n ++\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n +\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n +\t\texport GIT_TRACE_FSMONITOR &&\n +\n +\t\tstart_daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'file changes to directory' '\n -+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n ++\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n +\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n +\t\texport GIT_TRACE_FSMONITOR &&\n +\n +\t\tstart_daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +'\n +\n +test_expect_success 'directory changes to a file' '\n -+\ttest_when_finished \"clean_up_repo_and_stop_daemon\" &&\n ++\ttest_when_finished clean_up_repo_and_stop_daemon &&\n +\n +\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace\" &&\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n +\t\texport GIT_TRACE_FSMONITOR &&\n +\n +\t\tstart_daemon\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +# \"flush\" message to a running daemon and ask it to do a flush/resync.\n +\n +test_expect_success 'flush cached data' '\n -+\ttest_when_finished \"kill_repo test_flush\" &&\n ++\ttest_when_finished \"stop_daemon_delete_repo test_flush\" &&\n +\n +\tgit init test_flush &&\n +\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\t\tGIT_TEST_FSMONITOR_TOKEN=true &&\n +\t\texport GIT_TEST_FSMONITOR_TOKEN &&\n +\n -+\t\tGIT_TRACE_FSMONITOR=\"$PWD/.git/trace_daemon\" &&\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace_daemon\" &&\n +\t\texport GIT_TRACE_FSMONITOR &&\n +\n +\t\tstart_daemon test_flush\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\tgit -C wt-base worktree add ../wt-secondary &&\n +\n +\t(\n -+\t\tGIT_TRACE2_PERF=\"$PWD/trace2_wt_secondary\" &&\n ++\t\tGIT_TRACE2_PERF=\"$(pwd)/trace2_wt_secondary\" &&\n +\t\texport GIT_TRACE2_PERF &&\n +\n -+\t\tGIT_TRACE_FSMONITOR=\"$PWD/trace_wt_secondary\" &&\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/trace_wt_secondary\" &&\n +\t\texport GIT_TRACE_FSMONITOR &&\n +\n +\t\tstart_daemon wt-secondary\n @@ t/t7527-builtin-fsmonitor.sh (new)\n +\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n +'\n +\n ++# NEEDSWORK: Repeat one of the \"edit\" tests on wt-secondary and\n ++# confirm that we get the same events and behavior -- that is, that\n ++# fsmonitor--daemon correctly watches BOTH the working directory and\n ++# the external GITDIR directory and behaves the same as when \".git\"\n ++# is a directory inside the working directory.\n ++\n ++test_expect_success 'cleanup worktrees' '\n ++\tstop_daemon_delete_repo wt-secondary &&\n ++\tstop_daemon_delete_repo wt-base\n ++'\n ++\n +test_done\n 21: f0da90e9b05 = 28: 3f36a31eb42 fsmonitor--daemon: periodically truncate list of modified files\n 22: bb7b1912bb4 ! 29: 555caca2216 fsmonitor--daemon: use a cookie file to sync with file system\n @@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon\n \t/*\n \t * We expect `command` to be of the form:\n @@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,\n - \t\t */\n - \t\tdo_flush = 1;\n - \t\tdo_trivial = 1;\n -+\t\tdo_cookie = 1;\n - \n - \t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n - \t\t/* assume V1 timestamp or garbage */\n -@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,\n - \t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n - \t\t\t\t command);\n - \t\tdo_trivial = 1;\n -+\t\tdo_cookie = 1;\n - \n - \t} else {\n - \t\t/* We have \"builtin:*\" */\n -@@ builtin/fsmonitor--daemon.c: static int do_handle_client(struct fsmonitor_daemon_state *state,\n - \t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n - \t\t\t\t\t command);\n - \t\t\tdo_trivial = 1;\n -+\t\t\tdo_cookie = 1;\n - \n - \t\t} else {\n - \t\t\t/*\n \t\t\t * We have a V2 valid token:\n \t\t\t * \"builtin::\"\n \t\t\t */\n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)\n +\t/*\n +\t * We will write filesystem syncing cookie files into\n +\t * ///-.\n ++\t *\n ++\t * The extra layers of subdirectories here keep us from\n ++\t * changing the mtime on \".git/\" or \".git/foo/\" when we create\n ++\t * or delete cookie files.\n ++\t *\n ++\t * There have been problems with some IDEs that do a\n ++\t * non-recursive watch of the \".git/\" directory and run a\n ++\t * series of commands any time something happens.\n ++\t *\n ++\t * For example, if we place our cookie files directly in\n ++\t * \".git/\" or \".git/foo/\" then a `git status` (or similar\n ++\t * command) from the IDE will cause a cookie file to be\n ++\t * created in one of those dirs. This causes the mtime of\n ++\t * those dirs to change. This triggers the IDE's watch\n ++\t * notification. This triggers the IDE to run those commands\n ++\t * again. And the process repeats and the machine never goes\n ++\t * idle.\n ++\t *\n ++\t * Adding the extra layers of subdirectories prevents the\n ++\t * mtime of \".git/\" and \".git/foo\" from changing when a\n ++\t * cookie file is created.\n +\t */\n +\tstrbuf_init(&state.path_cookie_prefix, 0);\n +\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n 23: 102e17cbc87 = 30: 75bb4bc8463 fsmonitor: enhance existing comments\n 24: 11ea2f97def ! 31: 8b3c4f4e6dd fsmonitor: force update index after large responses\n @@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate,\n +\n void refresh_fsmonitor(struct index_state *istate)\n {\n - \tstruct repository *r = istate->repo ? istate->repo : the_repository;\n + \tstruct strbuf query_result = STRBUF_INIT;\n @@ fsmonitor.c: apply_results:\n \t\t *\n \t\t * This updates both the cache-entries and the untracked-cache.\n 27: 1483c68855c ! 32: 97dce46d1d0 t7527: test status with untracked-cache and fsmonitor--daemon\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '\n \tEOF\n \n \tgit -c core.useBuiltinFSMonitor= add . &&\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'worktree with .git file' '\n - \ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'cleanup worktrees' '\n + \tstop_daemon_delete_repo wt-base\n '\n \n -+# TODO Repeat one of the \"edit\" tests on wt-secondary and confirm that\n -+# TODO we get the same events and behavior -- that is, that fsmonitor--daemon\n -+# TODO correctly listens to events on both the working directory and to the\n -+# TODO referenced GITDIR.\n -+\n -+test_expect_success 'cleanup worktrees' '\n -+\tkill_repo wt-secondary &&\n -+\tkill_repo wt-base\n -+'\n -+\n +# The next few tests perform arbitrary/contrived file operations and\n +# confirm that status is correct. That is, that the data (or lack of\n +# data) from fsmonitor doesn't cause incorrect results. And doesn't\n -: ----------- > 33: e32ba686f7e fsmonitor: handle shortname for .git\n -: ----------- > 34: 627e27fe60b t7527: test FS event reporing on MacOS WRT case and Unicode\n\n-- \ngitgitgadget\n"},{"id":"428905","messageId":"dfcd3e5ac97b9a0834d4d8f28893fbb80c79bf09.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 04/34] fsmonitor-ipc: create client routines for git-fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:14Z","receivedAt":"2021-07-01T14:47:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate fsmonitor_ipc__*() client routines to spawn the built-in file\nsystem monitor daemon and send it an IPC request using the `Simple\nIPC` API.\n\nStub in empty fsmonitor_ipc__*() functions for unsupported platforms.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n fsmonitor-ipc.c | 179 ++++++++++++++++++++++++++++++++++++++++++++++++\n fsmonitor-ipc.h | 48 +++++++++++++\n 3 files changed, 228 insertions(+)\n create mode 100644 fsmonitor-ipc.c\n create mode 100644 fsmonitor-ipc.h\n\ndiff --git a/Makefile b/Makefile\nindex c3565fc0f8f..209c97aa22d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -893,6 +893,7 @@ LIB_OBJS += fetch-pack.o\n LIB_OBJS += fmt-merge-msg.o\n LIB_OBJS += fsck.o\n LIB_OBJS += fsmonitor.o\n+LIB_OBJS += fsmonitor-ipc.o\n LIB_OBJS += gettext.o\n LIB_OBJS += gpg-interface.o\n LIB_OBJS += graph.o\ndiff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c\nnew file mode 100644\nindex 00000000000..41706972520\n--- /dev/null\n+++ b/fsmonitor-ipc.c\n@@ -0,0 +1,179 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"simple-ipc.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"trace2.h\"\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 1;\n+}\n+\n+GIT_PATH_FUNC(fsmonitor_ipc__get_path, \"fsmonitor--daemon.ipc\")\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn ipc_get_active_state(fsmonitor_ipc__get_path());\n+}\n+\n+static int spawn_daemon(void)\n+{\n+\tconst char *args[] = { \"fsmonitor--daemon\", \"start\", NULL };\n+\n+\treturn run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD,\n+\t\t\t\t \"fsmonitor\");\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\tint ret = -1;\n+\tint tried_to_spawn = 0;\n+\tenum ipc_active_state state = IPC_STATE__OTHER_ERROR;\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tconst char *tok = since_token ? since_token : \"\";\n+\tsize_t tok_len = since_token ? strlen(since_token) : 0;\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"query\", NULL);\n+\ttrace2_data_string(\"fsm_client\", NULL, \"query/command\", tok);\n+\n+try_again:\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tret = ipc_client_send_command_to_connection(\n+\t\t\tconnection, tok, tok_len, answer);\n+\t\tipc_client_close_connection(connection);\n+\n+\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t \"query/response-length\", answer->len);\n+\n+\t\tif (fsmonitor_is_trivial_response(answer))\n+\t\t\ttrace2_data_intmax(\"fsm_client\", NULL,\n+\t\t\t\t\t \"query/trivial-response\", 1);\n+\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__NOT_LISTENING:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: daemon not available\"));\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__PATH_NOT_FOUND:\n+\t\tif (tried_to_spawn)\n+\t\t\tgoto done;\n+\n+\t\ttried_to_spawn++;\n+\t\tif (spawn_daemon())\n+\t\t\tgoto done;\n+\n+\t\t/*\n+\t\t * Try again, but this time give the daemon a chance to\n+\t\t * actually create the pipe/socket.\n+\t\t *\n+\t\t * Granted, the daemon just started so it can't possibly have\n+\t\t * any FS cached yet, so we'll always get a trivial answer.\n+\t\t * BUT the answer should include a new token that can serve\n+\t\t * as the basis for subsequent requests.\n+\t\t */\n+\t\toptions.wait_if_not_found = 1;\n+\t\tgoto try_again;\n+\n+\tcase IPC_STATE__INVALID_PATH:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: invalid path '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\n+\tcase IPC_STATE__OTHER_ERROR:\n+\tdefault:\n+\t\tret = error(_(\"fsmonitor_ipc__send_query: unspecified error on '%s'\"),\n+\t\t\t fsmonitor_ipc__get_path());\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\ttrace2_region_leave(\"fsm_client\", \"query\", NULL);\n+\n+\treturn ret;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\tstruct ipc_client_connection *connection = NULL;\n+\tstruct ipc_client_connect_options options\n+\t\t= IPC_CLIENT_CONNECT_OPTIONS_INIT;\n+\tint ret;\n+\tenum ipc_active_state state;\n+\tconst char *c = command ? command : \"\";\n+\tsize_t c_len = command ? strlen(command) : 0;\n+\n+\tstrbuf_reset(answer);\n+\n+\toptions.wait_if_busy = 1;\n+\toptions.wait_if_not_found = 0;\n+\n+\tstate = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,\n+\t\t\t\t &connection);\n+\tif (state != IPC_STATE__LISTENING) {\n+\t\tdie(\"fsmonitor--daemon is not running\");\n+\t\treturn -1;\n+\t}\n+\n+\tret = ipc_client_send_command_to_connection(connection, c, c_len,\n+\t\t\t\t\t\t answer);\n+\tipc_client_close_connection(connection);\n+\n+\tif (ret == -1) {\n+\t\tdie(\"could not send '%s' command to fsmonitor--daemon\", c);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+/*\n+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported\n+ * platforms.\n+ */\n+\n+int fsmonitor_ipc__is_supported(void)\n+{\n+\treturn 0;\n+}\n+\n+const char *fsmonitor_ipc__get_path(void)\n+{\n+\treturn NULL;\n+}\n+\n+enum ipc_active_state fsmonitor_ipc__get_state(void)\n+{\n+\treturn IPC_STATE__OTHER_ERROR;\n+}\n+\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer)\n+{\n+\treturn -1;\n+}\n+\n+#endif\ndiff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h\nnew file mode 100644\nindex 00000000000..837c5e5b64a\n--- /dev/null\n+++ b/fsmonitor-ipc.h\n@@ -0,0 +1,48 @@\n+#ifndef FSMONITOR_IPC_H\n+#define FSMONITOR_IPC_H\n+\n+/*\n+ * Returns true if built-in file system monitor daemon is defined\n+ * for this platform.\n+ */\n+int fsmonitor_ipc__is_supported(void);\n+\n+/*\n+ * Returns the pathname to the IPC named pipe or Unix domain socket\n+ * where a `git-fsmonitor--daemon` process will listen. This is a\n+ * per-worktree value.\n+ *\n+ * Returns NULL if the daemon is not supported on this platform.\n+ */\n+const char *fsmonitor_ipc__get_path(void);\n+\n+/*\n+ * Try to determine whether there is a `git-fsmonitor--daemon` process\n+ * listening on the IPC pipe/socket.\n+ */\n+enum ipc_active_state fsmonitor_ipc__get_state(void);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc\n+ * and ask for the set of changed files since the given token.\n+ *\n+ * This DOES NOT use the hook interface.\n+ *\n+ * Spawn a daemon process in the background if necessary.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_query(const char *since_token,\n+\t\t\t struct strbuf *answer);\n+\n+/*\n+ * Connect to a `git-fsmonitor--daemon` process via simple-ipc and\n+ * send a command verb. If no daemon is available, we DO NOT try to\n+ * start one.\n+ *\n+ * Returns -1 on error; 0 on success.\n+ */\n+int fsmonitor_ipc__send_command(const char *command,\n+\t\t\t\tstruct strbuf *answer);\n+\n+#endif /* FSMONITOR_IPC_H */\n-- \ngitgitgadget\n\n"},{"id":"428906","messageId":"0aaca2f93908256b8d6385d321144dbd5cca813e.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 05/34] help: include fsmonitor--daemon feature flag in version info","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:15Z","receivedAt":"2021-07-01T14:47:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd the \"feature: fsmonitor--daemon\" message to the output of\n`git version --build-options`.\n\nThis allows users to know if the built-in fsmonitor feature is\nsupported on their platform.\n\nSigned-off-by: Jeff Hostetler \n---\n help.c | 4 ++++\n t/test-lib.sh | 6 ++++++\n 2 files changed, 10 insertions(+)\n\ndiff --git a/help.c b/help.c\nindex 3c3bdec2135..e22ba1d246a 100644\n--- a/help.c\n+++ b/help.c\n@@ -11,6 +11,7 @@\n #include \"version.h\"\n #include \"refs.h\"\n #include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n \n struct category_description {\n \tuint32_t category;\n@@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options)\n \t\tstrbuf_addf(buf, \"sizeof-size_t: %d\\n\", (int)sizeof(size_t));\n \t\tstrbuf_addf(buf, \"shell-path: %s\\n\", SHELL_PATH);\n \t\t/* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */\n+\n+\t\tif (fsmonitor_ipc__is_supported())\n+\t\t\tstrbuf_addstr(buf, \"feature: fsmonitor--daemon\\n\");\n \t}\n }\n \ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex adaf03543e8..bfb7ff6ed17 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1674,3 +1674,9 @@ test_lazy_prereq REBASE_P '\n # Tests that verify the scheduler integration must set this locally\n # to avoid errors.\n GIT_TEST_MAINT_SCHEDULER=\"none:exit 1\"\n+\n+# Does this platform support `git fsmonitor--daemon`\n+#\n+test_lazy_prereq FSMONITOR_DAEMON '\n+\tgit version --build-options | grep \"feature:\" | grep \"fsmonitor--daemon\"\n+'\n-- \ngitgitgadget\n\n"},{"id":"428908","messageId":"86413bfe34722dcd5c846cf7326340473471773c.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 03/34] fsmonitor--daemon: update fsmonitor documentation","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:13Z","receivedAt":"2021-07-01T14:47:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUpdate references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\npointers to `Watchman` to mention the new built-in `fsmonitor--daemon`.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/config/core.txt | 56 ++++++++++++++++++++++--------\n Documentation/git-update-index.txt | 27 +++++++-------\n Documentation/githooks.txt | 3 +-\n 3 files changed, 59 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..4f6e519bc02 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -62,22 +62,50 @@ core.protectNTFS::\n \tDefaults to `true` on Windows, and `false` elsewhere.\n \n core.fsmonitor::\n-\tIf set, the value of this variable is used as a command which\n-\twill identify all files that may have changed since the\n-\trequested date/time. This information is used to speed up git by\n-\tavoiding unnecessary processing of files that have not changed.\n-\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n+\tIf set, this variable contains the pathname of the \"fsmonitor\"\n+\thook command.\n++\n+This hook command is used to identify all files that may have changed\n+since the requested date/time. This information is used to speed up\n+git by avoiding unnecessary scanning of files that have not changed.\n++\n+See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n \n core.fsmonitorHookVersion::\n-\tSets the version of hook that is to be used when calling fsmonitor.\n-\tThere are currently versions 1 and 2. When this is not set,\n-\tversion 2 will be tried first and if it fails then version 1\n-\twill be tried. Version 1 uses a timestamp as input to determine\n-\twhich files have changes since that time but some monitors\n-\tlike watchman have race conditions when used with a timestamp.\n-\tVersion 2 uses an opaque string so that the monitor can return\n-\tsomething that can be used to determine what files have changed\n-\twithout race conditions.\n+\tSets the protocol version to be used when invoking the\n+\t\"fsmonitor\" hook.\n++\n+There are currently versions 1 and 2. When this is not set,\n+version 2 will be tried first and if it fails then version 1\n+will be tried. Version 1 uses a timestamp as input to determine\n+which files have changes since that time but some monitors\n+like Watchman have race conditions when used with a timestamp.\n+Version 2 uses an opaque string so that the monitor can return\n+something that can be used to determine what files have changed\n+without race conditions.\n++\n+Note: The value of this config setting is ignored if the\n+built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n+\n+core.useBuiltinFSMonitor::\n+\tIf set to true, enable the built-in file system monitor\n+\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n++\n+Like hook-based file system monitors, the built-in file system monitor\n+can speed up Git commands that need to refresh the Git index\n+(e.g. `git status`) in a working directory with many files. The\n+built-in monitor eliminates the need to install and maintain an\n+external third-party tool.\n++\n+The built-in file system monitor is currently available only on a\n+limited set of supported platforms. Currently, this includes Windows\n+and MacOS.\n++\n+Note: if this config setting is set to `true`, the values of\n+`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored.\n \n core.trustctime::\n \tIf false, the ctime differences between the index and the\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..c7c31b3fcf9 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n This feature is intended to speed up git operations for repos that have\n large working directories.\n \n-It enables git to work together with a file system monitor (see the\n+It enables git to work together with a file system monitor (see\n+linkgit:git-fsmonitor--daemon[1]\n+and the\n \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n inform it as to what files have been modified. This enables git to avoid\n having to lstat() every file to find modified files.\n@@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory\n looking for new files.\n \n If you want to enable (or disable) this feature, it is easier to use\n-the `core.fsmonitor` configuration variable (see\n-linkgit:git-config[1]) than using the `--fsmonitor` option to\n-`git update-index` in each repository, especially if you want to do so\n-across all repositories you use, because you can set the configuration\n-variable in your `$HOME/.gitconfig` just once and have it affect all\n-repositories you touch.\n-\n-When the `core.fsmonitor` configuration variable is changed, the\n-file system monitor is added to or removed from the index the next time\n-a command reads the index. When `--[no-]fsmonitor` are used, the file\n-system monitor is immediately added to or removed from the index.\n+the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable (see linkgit:git-config[1]) than using the `--fsmonitor`\n+option to `git update-index` in each repository, especially if you\n+want to do so across all repositories you use, because you can set the\n+configuration variable in your `$HOME/.gitconfig` just once and have\n+it affect all repositories you touch.\n+\n+When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n+variable is changed, the file system monitor is added to or removed\n+from the index the next time a command reads the index. When\n+`--[no-]fsmonitor` are used, the file system monitor is immediately\n+added to or removed from the index.\n \n CONFIGURATION\n -------------\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex b51959ff941..b7d5e926f7b 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -593,7 +593,8 @@ fsmonitor-watchman\n \n This hook is invoked when the configuration option `core.fsmonitor` is\n set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n-depending on the version of the hook to use.\n+depending on the version of the hook to use, unless overridden via\n+`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n \n Version 1 takes two arguments, a version (1) and the time in elapsed\n nanoseconds since midnight, January 1, 1970.\n-- \ngitgitgadget\n\n"},{"id":"428909","messageId":"c86b5651ecc63f885701e0850b351db62b6d5478.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 07/34] fsmonitor: use IPC to query the builtin FSMonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:17Z","receivedAt":"2021-07-01T14:47:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUse simple IPC to directly communicate with the new builtin file\nsystem monitor daemon when `core.useBuiltinFSMonitor` is set.\n\nThe `core.fsmonitor` setting has already been defined as a HOOK\npathname. Historically, this has been set to a HOOK script that will\ntalk with Watchman. For compatibility reasons, we do not want to\noverload that definition (and cause problems if users have multiple\nversions of Git installed).\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 41 +++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 39 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 374189be7d9..3719ddfeec9 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -256,8 +256,44 @@ void refresh_fsmonitor(struct index_state *istate)\n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n \n \tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC) {\n-\t\t/* TODO */\n-\t\treturn;\n+\t\tquery_success = !fsmonitor_ipc__send_query(\n+\t\t\tistate->fsmonitor_last_update ?\n+\t\t\tistate->fsmonitor_last_update : \"builtin:fake\",\n+\t\t\t&query_result);\n+\t\tif (query_success) {\n+\t\t\t/*\n+\t\t\t * The response contains a series of nul terminated\n+\t\t\t * strings. The first is the new token.\n+\t\t\t *\n+\t\t\t * Use `char *buf` as an interlude to trick the CI\n+\t\t\t * static analysis to let us use `strbuf_addstr()`\n+\t\t\t * here (and only copy the token) rather than\n+\t\t\t * `strbuf_addbuf()`.\n+\t\t\t */\n+\t\t\tbuf = query_result.buf;\n+\t\t\tstrbuf_addstr(&last_update_token, buf);\n+\t\t\tbol = last_update_token.len + 1;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The builtin daemon is not available on this\n+\t\t\t * platform -OR- we failed to get a response.\n+\t\t\t *\n+\t\t\t * Generate a fake token (rather than a V1\n+\t\t\t * timestamp) for the index extension. (If\n+\t\t\t * they switch back to the hook API, we don't\n+\t\t\t * want ambiguous state.)\n+\t\t\t */\n+\t\t\tstrbuf_addstr(&last_update_token, \"builtin:fake\");\n+\t\t}\n+\n+\t\t/*\n+\t\t * Regardless of whether we successfully talked to a\n+\t\t * fsmonitor daemon or not, we skip over and do not\n+\t\t * try to use the hook. The \"core.useBuiltinFSMonitor\"\n+\t\t * config setting ALWAYS overrides the \"core.fsmonitor\"\n+\t\t * hook setting.\n+\t\t */\n+\t\tgoto apply_results;\n \t}\n \n \tassert(r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK);\n@@ -323,6 +359,7 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n+apply_results:\n \t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n \tif (query_success && query_result.buf[bol] != '/') {\n \t\t/* Mark all entries returned by the monitor as dirty */\n-- \ngitgitgadget\n\n"},{"id":"428910","messageId":"f88db92d4259d1c29827e97e957daf6eda39c551.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 08/34] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:18Z","receivedAt":"2021-07-01T14:47:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a built-in file system monitoring daemon that can be used by\nthe existing `fsmonitor` feature (protocol API and index extension)\nto improve the performance of various Git commands, such as `status`.\n\nThe `fsmonitor--daemon` feature builds upon the `Simple IPC` API and\nprovides an alternative to hook access to existing fsmonitors such\nas `watchman`.\n\nThis commit merely adds the new command without any functionality.\n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/fsmonitor--daemon.c | 53 +++++++++++++++++++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 57 insertions(+)\n create mode 100644 builtin/fsmonitor--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 311841f9bed..4baba472aa8 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -72,6 +72,7 @@\n /git-format-patch\n /git-fsck\n /git-fsck-objects\n+/git-fsmonitor--daemon\n /git-gc\n /git-get-tar-commit-id\n /git-grep\ndiff --git a/Makefile b/Makefile\nindex 209c97aa22d..8fe1e42a435 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1097,6 +1097,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n+BUILTIN_OBJS += builtin/fsmonitor--daemon.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex 16ecd5586f0..2470d1cd3a2 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\n int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix);\n int cmd_grep(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nnew file mode 100644\nindex 00000000000..df2bad53111\n--- /dev/null\n+++ b/builtin/fsmonitor--daemon.c\n@@ -0,0 +1,53 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n+#include \"simple-ipc.h\"\n+#include \"khash.h\"\n+\n+static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tNULL\n+};\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *subcmd;\n+\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc < 2)\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tsubcmd = argv[1];\n+\targv--;\n+\targc++;\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\n+\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n+}\n+\n+#else\n+int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n+\n+\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n+}\n+#endif\ndiff --git a/git.c b/git.c\nindex 18bed9a9964..c6160f4a886 100644\n--- a/git.c\n+++ b/git.c\n@@ -533,6 +533,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"428911","messageId":"8b64b7cd3674163adb0588d42eccf4873b30974b.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 06/34] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:16Z","receivedAt":"2021-07-01T14:47:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove FSMonitor config settings to `struct repo_settings`, get rid of\nthe `core_fsmonitor` global variable, and add support for the new\n`core.useBuiltinFSMonitor` config setting. Move config code to lookup\n`core.fsmonitor` into `prepare_repo_settings()` with the rest of the\nsetup code.\n\nThe `core_fsmonitor` global variable was used to store the pathname to\nthe FSMonitor hook and it was used as a boolean to see if FSMonitor\nwas enabled. This dual usage will lead to confusion when we add\nsupport for a builtin FSMonitor based on IPC, since the builtin\nFSMonitor doesn't need the hook pathname.\n\nReplace the boolean usage with an `enum fsmonitor_mode` to represent\nthe state of FSMonitor. And only set the pathname when in HOOK mode.\n\nAlso, disable FSMonitor when the repository working directory is\nincompatible. For example, in bare repositories, since there aren't\nany files to watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/update-index.c | 20 +++++++++---\n cache.h | 1 -\n config.c | 14 --------\n config.h | 1 -\n environment.c | 1 -\n fsmonitor.c | 65 +++++++++++++++++++++++--------------\n fsmonitor.h | 14 ++++++--\n repo-settings.c | 48 +++++++++++++++++++++++++++\n repository.h | 11 +++++++\n t/README | 4 +--\n t/t7519-status-fsmonitor.sh | 22 +++++++++++++\n 11 files changed, 150 insertions(+), 51 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex f1f16f2de52..0141899caa3 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1216,14 +1216,26 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (fsmonitor > 0) {\n-\t\tif (git_config_get_fsmonitor() == 0)\n+\t\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE)\n+\t\t\treturn error(\n+\t\t\t\t_(\"repository is incompatible with fsmonitor\"));\n+\n+\t\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_DISABLED) {\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is unset; \"\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"builtin fsmonitor\"));\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n-\t\t\t\t\"set it if you really want to \"\n-\t\t\t\t\"enable fsmonitor\"));\n+\t\t\t\t\"set it if you really want to enable the \"\n+\t\t\t\t\"hook-based fsmonitor\"));\n+\t\t}\n \t\tadd_fsmonitor(&the_index);\n \t\treport(_(\"fsmonitor enabled\"));\n \t} else if (!fsmonitor) {\n-\t\tif (git_config_get_fsmonitor() == 1)\n+\t\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC)\n+\t\t\twarning(_(\"core.useBuiltinFSMonitor is set; \"\n+\t\t\t\t\"remove it if you really want to \"\n+\t\t\t\t\"disable fsmonitor\"));\n+\t\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK)\n \t\t\twarning(_(\"core.fsmonitor is set; \"\n \t\t\t\t\"remove it if you really want to \"\n \t\t\t\t\"disable fsmonitor\"));\ndiff --git a/cache.h b/cache.h\nindex ba04ff8bd36..50463876852 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -981,7 +981,6 @@ extern int core_preload_index;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n-extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\ndiff --git a/config.c b/config.c\nindex f9c400ad306..ec3b88b4639 100644\n--- a/config.c\n+++ b/config.c\n@@ -2516,20 +2516,6 @@ int git_config_get_max_percent_split_change(void)\n \treturn -1; /* default value */\n }\n \n-int git_config_get_fsmonitor(void)\n-{\n-\tif (git_config_get_pathname(\"core.fsmonitor\", &core_fsmonitor))\n-\t\tcore_fsmonitor = getenv(\"GIT_TEST_FSMONITOR\");\n-\n-\tif (core_fsmonitor && !*core_fsmonitor)\n-\t\tcore_fsmonitor = NULL;\n-\n-\tif (core_fsmonitor)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n int git_config_get_index_threads(int *dest)\n {\n \tint is_bool, val;\ndiff --git a/config.h b/config.h\nindex 9038538ffdc..b8d6b75d4fe 100644\n--- a/config.h\n+++ b/config.h\n@@ -609,7 +609,6 @@ int git_config_get_index_threads(int *dest);\n int git_config_get_untracked_cache(void);\n int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n-int git_config_get_fsmonitor(void);\n \n /* This dies if the configured or default date is in the future */\n int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex 2f27008424a..7b5e8ff78da 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT;\n #define PROTECT_NTFS_DEFAULT 1\n #endif\n int protect_ntfs = PROTECT_NTFS_DEFAULT;\n-const char *core_fsmonitor;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ab9bfc60b34..374189be7d9 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n+#include \"fsmonitor-ipc.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n \n@@ -148,15 +149,21 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)\n /*\n * Call the query-fsmonitor hook passing the last update token of the saved results.\n */\n-static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result)\n+static int query_fsmonitor_hook(struct repository *r,\n+\t\t\t\tint version,\n+\t\t\t\tconst char *last_update,\n+\t\t\t\tstruct strbuf *query_result)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tint result;\n \n-\tif (!core_fsmonitor)\n+\tif (r->settings.fsmonitor_mode != FSMONITOR_MODE_HOOK)\n \t\treturn -1;\n \n-\tstrvec_push(&cp.args, core_fsmonitor);\n+\tassert(r->settings.fsmonitor_hook_path);\n+\tassert(*r->settings.fsmonitor_hook_path);\n+\n+\tstrvec_push(&cp.args, r->settings.fsmonitor_hook_path);\n \tstrvec_pushf(&cp.args, \"%d\", version);\n \tstrvec_pushf(&cp.args, \"%s\", last_update);\n \tcp.use_shell = 1;\n@@ -238,17 +245,27 @@ void refresh_fsmonitor(struct index_state *istate)\n \tstruct strbuf last_update_token = STRBUF_INIT;\n \tchar *buf;\n \tunsigned int i;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n \n-\tif (!core_fsmonitor || istate->fsmonitor_has_run_once)\n+\tif (r->settings.fsmonitor_mode <= FSMONITOR_MODE_DISABLED ||\n+\t istate->fsmonitor_has_run_once)\n \t\treturn;\n \n-\thook_version = fsmonitor_hook_version();\n-\n \tistate->fsmonitor_has_run_once = 1;\n \n \ttrace_printf_key(&trace_fsmonitor, \"refresh fsmonitor\");\n+\n+\tif (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC) {\n+\t\t/* TODO */\n+\t\treturn;\n+\t}\n+\n+\tassert(r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK);\n+\n+\thook_version = fsmonitor_hook_version();\n+\n \t/*\n-\t * This could be racy so save the date/time now and query_fsmonitor\n+\t * This could be racy so save the date/time now and query_fsmonitor_hook\n \t * should be inclusive to ensure we don't miss potential changes.\n \t */\n \tlast_update = getnanotime();\n@@ -256,13 +273,14 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\tstrbuf_addf(&last_update_token, \"%\"PRIu64\"\", last_update);\n \n \t/*\n-\t * If we have a last update token, call query_fsmonitor for the set of\n+\t * If we have a last update token, call query_fsmonitor_hook for the set of\n \t * changes since that token, else assume everything is possibly dirty\n \t * and check it all.\n \t */\n \tif (istate->fsmonitor_last_update) {\n \t\tif (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION2,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \n \t\t\tif (query_success) {\n@@ -292,13 +310,17 @@ void refresh_fsmonitor(struct index_state *istate)\n \t\t}\n \n \t\tif (hook_version == HOOK_INTERFACE_VERSION1) {\n-\t\t\tquery_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1,\n+\t\t\tquery_success = !query_fsmonitor_hook(\n+\t\t\t\tr, HOOK_INTERFACE_VERSION1,\n \t\t\t\tistate->fsmonitor_last_update, &query_result);\n \t\t}\n \n-\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\", core_fsmonitor);\n-\t\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor process '%s' returned %s\",\n-\t\t\tcore_fsmonitor, query_success ? \"success\" : \"failure\");\n+\t\ttrace_performance_since(last_update, \"fsmonitor process '%s'\",\n+\t\t\t\t\tr->settings.fsmonitor_hook_path);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor process '%s' returned %s\",\n+\t\t\t\t r->settings.fsmonitor_hook_path,\n+\t\t\t\t query_success ? \"success\" : \"failure\");\n \t}\n \n \t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n@@ -411,7 +433,8 @@ void remove_fsmonitor(struct index_state *istate)\n void tweak_fsmonitor(struct index_state *istate)\n {\n \tunsigned int i;\n-\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\tint fsmonitor_enabled = r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED;\n \n \tif (istate->fsmonitor_dirty) {\n \t\tif (fsmonitor_enabled) {\n@@ -431,16 +454,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tistate->fsmonitor_dirty = NULL;\n \t}\n \n-\tswitch (fsmonitor_enabled) {\n-\tcase -1: /* keep: do nothing */\n-\t\tbreak;\n-\tcase 0: /* false */\n-\t\tremove_fsmonitor(istate);\n-\t\tbreak;\n-\tcase 1: /* true */\n+\tif (fsmonitor_enabled)\n \t\tadd_fsmonitor(istate);\n-\t\tbreak;\n-\tdefault: /* unknown value: do nothing */\n-\t\tbreak;\n-\t}\n+\telse\n+\t\tremove_fsmonitor(istate);\n }\ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex f20d72631d7..9cc14e05239 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -57,7 +57,10 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result);\n */\n static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n {\n-\treturn !core_fsmonitor || istate->fsmonitor_has_run_once;\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\n+\treturn r->settings.fsmonitor_mode <= FSMONITOR_MODE_DISABLED ||\n+\t\tistate->fsmonitor_has_run_once;\n }\n \n /*\n@@ -67,7 +70,10 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\n+\tif (r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED &&\n+\t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n@@ -83,7 +89,9 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n */\n static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)\n {\n-\tif (core_fsmonitor) {\n+\tstruct repository *r = istate->repo ? istate->repo : the_repository;\n+\n+\tif (r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED) {\n \t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\tuntracked_cache_invalidate_path(istate, ce->name, 1);\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_invalid '%s'\", ce->name);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 0cfe8b787db..faf197ff60a 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -5,10 +5,42 @@\n \n #define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0)\n \n+/*\n+ * Return 1 if the repo/workdir is incompatible with FSMonitor.\n+ */\n+static int is_repo_incompatible_with_fsmonitor(struct repository *r)\n+{\n+\tconst char *const_strval;\n+\n+\t/*\n+\t * Bare repositories don't have a working directory and\n+\t * therefore, nothing to watch.\n+\t */\n+\tif (!r->worktree)\n+\t\treturn 1;\n+\n+\t/*\n+\t * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n+\t *\n+\t * Granted, core Git does not know anything about GVFS and\n+\t * we shouldn't make assumptions about a downstream feature,\n+\t * but users can install both versions. And this can lead\n+\t * to incorrect results from core Git commands. So, without\n+\t * bringing in any of the GVFS code, do a simple config test\n+\t * for a published config setting. (We do not look at the\n+\t * various *_TEST_* environment variables.)\n+\t */\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_strval))\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n void prepare_repo_settings(struct repository *r)\n {\n \tint value;\n \tchar *strval;\n+\tconst char *const_strval;\n \n \tif (r->settings.initialized)\n \t\treturn;\n@@ -26,6 +58,22 @@ void prepare_repo_settings(struct repository *r)\n \tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n \n+\tr->settings.fsmonitor_hook_path = NULL;\n+\tr->settings.fsmonitor_mode = FSMONITOR_MODE_DISABLED;\n+\tif (is_repo_incompatible_with_fsmonitor(r))\n+\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\telse if (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value)\n+\t\t && value)\n+\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_IPC;\n+\telse {\n+\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_strval))\n+\t\t\tconst_strval = getenv(\"GIT_TEST_FSMONITOR\");\n+\t\tif (const_strval && *const_strval) {\n+\t\t\tr->settings.fsmonitor_hook_path = strdup(const_strval);\n+\t\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_HOOK;\n+\t\t}\n+\t}\n+\n \tif (!repo_config_get_int(r, \"index.version\", &value))\n \t\tr->settings.index_version = value;\n \tif (!repo_config_get_maybe_bool(r, \"core.untrackedcache\", &value)) {\ndiff --git a/repository.h b/repository.h\nindex a45f7520fd9..09154298ba1 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -26,6 +26,14 @@ enum fetch_negotiation_setting {\n \tFETCH_NEGOTIATION_NOOP = 3,\n };\n \n+enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -2,\n+\tFSMONITOR_MODE_UNSET = -1,\n+\tFSMONITOR_MODE_DISABLED = 0,\n+\tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */\n+\tFSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */\n+};\n+\n struct repo_settings {\n \tint initialized;\n \n@@ -34,6 +42,9 @@ struct repo_settings {\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \n+\tenum fsmonitor_mode fsmonitor_mode;\n+\tchar *fsmonitor_hook_path;\n+\n \tint index_version;\n \tenum untracked_cache_setting core_untracked_cache;\n \ndiff --git a/t/README b/t/README\nindex 1a2072b2c8a..852a4eae9da 100644\n--- a/t/README\n+++ b/t/README\n@@ -398,8 +398,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was\n passed in.\n \n GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor\n-code path for utilizing a file system monitor to speed up detecting\n-new or changed files.\n+code path for utilizing a (hook based) file system monitor to speed up\n+detecting new or changed files.\n \n GIT_TEST_INDEX_VERSION= exercises the index read/write code path\n for the index version specified. Can be set to any valid version\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 637391c6ce4..02919c68ddd 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -383,4 +383,26 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone\" &&\n+\tgit clone --bare . ./bare-clone &&\n+\tcat >expect <<-\\EOF &&\n+\terror: repository is incompatible with fsmonitor\n+\tEOF\n+\ttest_must_fail git -C ./bare-clone update-index --fsmonitor 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'incompatible core.virtualfilesystem' '\n+\ttest_when_finished \"rm -rf ./fake-gvfs-clone\" &&\n+\tgit clone . ./fake-gvfs-clone &&\n+\tgit -C ./fake-gvfs-clone config core.virtualfilesystem true &&\n+\tcat >expect <<-\\EOF &&\n+\terror: repository is incompatible with fsmonitor\n+\tEOF\n+\ttest_must_fail git -C ./fake-gvfs-clone update-index --fsmonitor 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"428912","messageId":"02e21384ef0ca4909e0bda2c78fa63c06be22a50.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 09/34] fsmonitor--daemon: implement 'stop' and 'status' commands","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:19Z","receivedAt":"2021-07-01T14:47:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `stop` and `status` client commands to control and query the\nstatus of a `fsmonitor--daemon` server process (and implicitly start a\nserver process if necessary).\n\nLater commits will implement the actual server and monitor the file\nsystem.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 51 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex df2bad53111..62efd5ea787 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,10 +7,55 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon stop\"),\n+\tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Acting as a CLIENT.\n+ *\n+ * Send a \"quit\" command to the `git-fsmonitor--daemon` (if running)\n+ * and wait for it to shutdown.\n+ */\n+static int do_as_client__send_stop(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"quit\", &answer);\n+\n+\t/* The quit command does not return any response data. */\n+\tstrbuf_release(&answer);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttrace2_region_enter(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\twhile (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tsleep_millisec(50);\n+\ttrace2_region_leave(\"fsm_client\", \"polling-for-daemon-exit\", NULL);\n+\n+\treturn 0;\n+}\n+\n+static int do_as_client__status(void)\n+{\n+\tenum ipc_active_state state = fsmonitor_ipc__get_state();\n+\n+\tswitch (state) {\n+\tcase IPC_STATE__LISTENING:\n+\t\tprintf(_(\"fsmonitor-daemon is watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tprintf(_(\"fsmonitor-daemon is not watching '%s'\\n\"),\n+\t\t the_repository->worktree);\n+\t\treturn 1;\n+\t}\n+}\n \n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n@@ -35,6 +80,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n \n+\tif (!strcmp(subcmd, \"stop\"))\n+\t\treturn !!do_as_client__send_stop();\n+\n+\tif (!strcmp(subcmd, \"status\"))\n+\t\treturn !!do_as_client__status();\n+\n \tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"428913","messageId":"5a9bda7220356ebf0689bb6aaa9068520dc6e33b.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:21Z","receivedAt":"2021-07-01T14:48:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty backend for fsmonitor--daemon on Windows.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 ++++++\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++\n compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++\n config.mak.uname | 2 +\n contrib/buildsystems/CMakeLists.txt | 5 ++\n 5 files changed, 90 insertions(+)\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n\ndiff --git a/Makefile b/Makefile\nindex c45caacf2c3..a2a6e1f20f6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -467,6 +467,11 @@ all::\n # directory, and the JSON compilation database 'compile_commands.json' will be\n # created at the root of the repository.\n #\n+# If your platform supports a built-in fsmonitor backend, set\n+# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n+# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n+# `fsmonitor_fs_listen__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n \tCOMPAT_OBJS += compat/access.o\n endif\n \n+ifdef FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n+\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n \t@echo X=\\'$(X)\\' >>$@+\n+ifdef FSMONITOR_DAEMON_BACKEND\n+\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\nnew file mode 100644\nindex 00000000000..880446b49e3\n--- /dev/null\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n@@ -0,0 +1,21 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-fs-listen.h\"\n+\n+void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen.h b/compat/fsmonitor/fsmonitor-fs-listen.h\nnew file mode 100644\nindex 00000000000..c7b5776b3b6\n--- /dev/null\n+++ b/compat/fsmonitor/fsmonitor-fs-listen.h\n@@ -0,0 +1,49 @@\n+#ifndef FSMONITOR_FS_LISTEN_H\n+#define FSMONITOR_FS_LISTEN_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * fsmonitor_fs_listener thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the fsmonitor listener thread.\n+ * This will be called from the main thread AFTER joining the listener.\n+ */\n+void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to watch for\n+ * filesystem events. This will run in the fsmonitor_fs_listen thread.\n+ *\n+ * It should call `ipc_server_stop_async()` if the listener thread\n+ * prematurely terminates (because of a filesystem error or if it\n+ * detects that the .git directory has been deleted). (It should NOT\n+ * do so if the listener thread receives a normal shutdown signal from\n+ * the IPC layer.)\n+ *\n+ * It should set `state->error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the fsmonitor listener thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_FS_LISTEN_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cb443b4e023..fcd88b60b14 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -420,6 +420,7 @@ ifeq ($(uname_S),Windows)\n \t# so we don't need this:\n \t#\n \t# SNPRINTF_RETURNS_BOGUS = YesPlease\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -598,6 +599,7 @@ ifneq (,$(findstring MINGW,$(uname_S)))\n \tNO_STRTOUMAX = YesPlease\n \tNO_MKDTEMP = YesPlease\n \tNO_SVN_TESTS = YesPlease\n+\tFSMONITOR_DAEMON_BACKEND = win32\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex a87841340e6..1ab94eb284f 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -263,6 +263,11 @@ else()\n \tendif()\n endif()\n \n+if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n+\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\tlist(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c)\n+endif()\n+\n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n \n #header checks\n-- \ngitgitgadget\n\n"},{"id":"428914","messageId":"5d6646df93a17659af66f136295444d1bd834090.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 13/34] fsmonitor--daemon: implement 'run' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:23Z","receivedAt":"2021-07-01T14:48:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement `run` command to try to begin listening for file system events.\n\nThis version defines the thread structure with a single fsmonitor_fs_listen\nthread to watch for file system events and a simple IPC thread pool to\nwatch for connection from Git clients over a well-known named pipe or\nUnix domain socket.\n\nThis commit does not actually do anything yet because the platform\nbackends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 210 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 34 ++++++\n 2 files changed, 243 insertions(+), 1 deletion(-)\n create mode 100644 fsmonitor--daemon.h\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 62efd5ea787..a265c962ccc 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,16 +3,39 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsmonitor-fs-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n \tNULL\n };\n \n #ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+/*\n+ * Global state loaded from config.\n+ */\n+#define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n+static int fsmonitor__ipc_threads = 8;\n+\n+static int fsmonitor_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 1)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__IPC_THREADS, i);\n+\t\tfsmonitor__ipc_threads = i;\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n /*\n * Acting as a CLIENT.\n *\n@@ -57,11 +80,190 @@ static int do_as_client__status(void)\n \t}\n }\n \n+static ipc_server_application_cb handle_client;\n+\n+static int handle_client(void *data,\n+\t\t\t const char *command, size_t command_len,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\t/* struct fsmonitor_daemon_state *state = data; */\n+\tint result;\n+\n+\t/*\n+\t * The Simple IPC API now supports {char*, len} arguments, but\n+\t * FSMonitor always uses proper null-terminated strings, so\n+\t * we can ignore the command_len argument. (Trust, but verify.)\n+\t */\n+\tif (command_len != strlen(command))\n+\t\tBUG(\"FSMonitor assumes text messages\");\n+\n+\ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n+\n+\tresult = 0; /* TODO Do something here. */\n+\n+\ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n+\n+\treturn result;\n+}\n+\n+static void *fsmonitor_fs_listen__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-listen\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Watching: worktree '%s'\",\n+\t\t\t state->path_worktree_watch.buf);\n+\tif (state->nr_paths_watching > 1)\n+\t\ttrace_printf_key(&trace_fsmonitor, \"Watching: gitdir '%s'\",\n+\t\t\t\t state->path_gitdir_watch.buf);\n+\n+\tfsmonitor_fs_listen__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct ipc_server_opts ipc_opts = {\n+\t\t.nr_threads = fsmonitor__ipc_threads,\n+\n+\t\t/*\n+\t\t * We know that there are no other active threads yet,\n+\t\t * so we can let the IPC layer temporarily chdir() if\n+\t\t * it needs to when creating the server side of the\n+\t\t * Unix domain socket.\n+\t\t */\n+\t\t.uds_disallow_chdir = 0\n+\t};\n+\n+\t/*\n+\t * Start the IPC thread pool before the we've started the file\n+\t * system event listener thread so that we have the IPC handle\n+\t * before we need it.\n+\t */\n+\tif (ipc_server_run_async(&state->ipc_server_data,\n+\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t handle_client, state))\n+\t\treturn error(_(\"could not start IPC thread pool\"));\n+\n+\t/*\n+\t * Start the fsmonitor listener thread to collect filesystem\n+\t * events.\n+\t */\n+\tif (pthread_create(&state->listener_thread, NULL,\n+\t\t\t fsmonitor_fs_listen__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\tipc_server_await(state->ipc_server_data);\n+\n+\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t}\n+\n+\t/*\n+\t * The daemon is now fully functional in background threads.\n+\t * Wait for the IPC thread pool to shutdown (whether by client\n+\t * request or from filesystem activity).\n+\t */\n+\tipc_server_await(state->ipc_server_data);\n+\n+\t/*\n+\t * The fsmonitor listener thread may have received a shutdown\n+\t * event from the IPC thread pool, but it doesn't hurt to tell\n+\t * it again. And wait for it to shutdown.\n+\t */\n+\tfsmonitor_fs_listen__stop_async(state);\n+\tpthread_join(state->listener_thread, NULL);\n+\n+\treturn state->error_code;\n+}\n+\n+static int fsmonitor_run_daemon(void)\n+{\n+\tstruct fsmonitor_daemon_state state;\n+\tint err;\n+\n+\tmemset(&state, 0, sizeof(state));\n+\n+\tpthread_mutex_init(&state.main_lock, NULL);\n+\tstate.error_code = 0;\n+\tstate.current_token_data = NULL;\n+\n+\t/* Prepare to (recursively) watch the directory. */\n+\tstrbuf_init(&state.path_worktree_watch, 0);\n+\tstrbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));\n+\tstate.nr_paths_watching = 1;\n+\n+\t/*\n+\t * We create and delete cookie files somewhere inside the .git\n+\t * directory to help us keep sync with the file system. If\n+\t * \".git\" is not a directory, then is not inside the\n+\t * cone of , so set up a second watch to watch\n+\t * the so that we get events for the cookie files.\n+\t */\n+\tstrbuf_init(&state.path_gitdir_watch, 0);\n+\tstrbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch);\n+\tstrbuf_addstr(&state.path_gitdir_watch, \"/.git\");\n+\tif (!is_directory(state.path_gitdir_watch.buf)) {\n+\t\tstrbuf_reset(&state.path_gitdir_watch);\n+\t\tstrbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir()));\n+\t\tstate.nr_paths_watching = 2;\n+\t}\n+\n+\t/*\n+\t * Confirm that we can create platform-specific resources for the\n+\t * filesystem listener before we bother starting all the threads.\n+\t */\n+\tif (fsmonitor_fs_listen__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize listener thread\"));\n+\t\tgoto done;\n+\t}\n+\n+\terr = fsmonitor_run_daemon_1(&state);\n+\n+done:\n+\tpthread_mutex_destroy(&state.main_lock);\n+\tfsmonitor_fs_listen__dtor(&state);\n+\n+\tipc_server_free(state.ipc_server_data);\n+\n+\tstrbuf_release(&state.path_worktree_watch);\n+\tstrbuf_release(&state.path_gitdir_watch);\n+\n+\treturn err;\n+}\n+\n+static int try_to_run_foreground_daemon(void)\n+{\n+\t/*\n+\t * Technically, we don't need to probe for an existing daemon\n+\t * process, since we could just call `fsmonitor_run_daemon()`\n+\t * and let it fail if the pipe/socket is busy.\n+\t *\n+\t * However, this method gives us a nicer error message for a\n+\t * common error case.\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n+\t\t the_repository->worktree);\n+\n+\tprintf(_(\"running fsmonitor-daemon in '%s'\\n\"),\n+\t the_repository->worktree);\n+\tfflush(stdout);\n+\n+\treturn !!fsmonitor_run_daemon();\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n \n \tstruct option options[] = {\n+\t\tOPT_INTEGER(0, \"ipc-threads\",\n+\t\t\t &fsmonitor__ipc_threads,\n+\t\t\t N_(\"use ipc worker threads\")),\n \t\tOPT_END()\n \t};\n \n@@ -71,7 +273,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(fsmonitor_config, NULL);\n \n \tsubcmd = argv[1];\n \targv--;\n@@ -79,6 +281,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \n \targc = parse_options(argc, argv, prefix, options,\n \t\t\t builtin_fsmonitor__daemon_usage, 0);\n+\tif (fsmonitor__ipc_threads < 1)\n+\t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n+\t\t fsmonitor__ipc_threads);\n+\n+\tif (!strcmp(subcmd, \"run\"))\n+\t\treturn !!try_to_run_foreground_daemon();\n \n \tif (!strcmp(subcmd, \"stop\"))\n \t\treturn !!do_as_client__send_stop();\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nnew file mode 100644\nindex 00000000000..3009c1a83de\n--- /dev/null\n+++ b/fsmonitor--daemon.h\n@@ -0,0 +1,34 @@\n+#ifndef FSMONITOR_DAEMON_H\n+#define FSMONITOR_DAEMON_H\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"run-command.h\"\n+#include \"simple-ipc.h\"\n+#include \"thread-utils.h\"\n+\n+struct fsmonitor_batch;\n+struct fsmonitor_token_data;\n+\n+struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+\n+struct fsmonitor_daemon_state {\n+\tpthread_t listener_thread;\n+\tpthread_mutex_t main_lock;\n+\n+\tstruct strbuf path_worktree_watch;\n+\tstruct strbuf path_gitdir_watch;\n+\tint nr_paths_watching;\n+\n+\tstruct fsmonitor_token_data *current_token_data;\n+\n+\tint error_code;\n+\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\n+\tstruct ipc_server_data *ipc_server_data;\n+};\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"428916","messageId":"c2adac8ed4b4f324b54b147e1fcb1aa4a74e80dd.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 10/34] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:20Z","receivedAt":"2021-07-01T14:48:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate an IPC client to send query and flush commands to the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n t/helper/test-fsmonitor-client.c | 121 +++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 124 insertions(+)\n create mode 100644 t/helper/test-fsmonitor-client.c\n\ndiff --git a/Makefile b/Makefile\nindex 8fe1e42a435..c45caacf2c3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-fast-rebase.o\n+TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-genzeros.o\n TEST_BUILTINS_OBJS += test-hash-speed.o\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nnew file mode 100644\nindex 00000000000..f7a5b3a32fa\n--- /dev/null\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -0,0 +1,121 @@\n+/*\n+ * test-fsmonitor-client.c: client code to send commands/requests to\n+ * a `git fsmonitor--daemon` daemon.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"fsmonitor-ipc.h\"\n+\n+#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tdie(\"fsmonitor--daemon not available on this platform\");\n+}\n+#else\n+\n+/*\n+ * Read the `.git/index` to get the last token written to the\n+ * FSMonitor Index Extension.\n+ */\n+static const char *get_token_from_index(void)\n+{\n+\tstruct index_state *istate = the_repository->index;\n+\n+\tif (do_read_index(istate, the_repository->index_file, 0) < 0)\n+\t\tdie(\"unable to read index file\");\n+\tif (!istate->fsmonitor_last_update)\n+\t\tdie(\"index file does not have fsmonitor extension\");\n+\n+\treturn istate->fsmonitor_last_update;\n+}\n+\n+/*\n+ * Send an IPC query to a `git-fsmonitor--daemon` daemon and\n+ * ask for the changes since the given token or from the last\n+ * token in the index extension.\n+ *\n+ * This will implicitly start a daemon process if necessary. The\n+ * daemon process will persist after we exit.\n+ */\n+static int do_send_query(const char *token)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\n+\tret = fsmonitor_ipc__send_query(token, &answer);\n+\tif (ret < 0)\n+\t\tdie(_(\"could not query fsmonitor--daemon\"));\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Send a \"flush\" command to the `git-fsmonitor--daemon` (if running)\n+ * and tell it to flush its cache.\n+ *\n+ * This feature is primarily used by the test suite to simulate a loss of\n+ * sync with the filesystem where we miss kernel events.\n+ */\n+static int do_send_flush(void)\n+{\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint ret;\n+\n+\tret = fsmonitor_ipc__send_command(\"flush\", &answer);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\twrite_in_full(1, answer.buf, answer.len);\n+\tstrbuf_release(&answer);\n+\n+\treturn 0;\n+}\n+\n+int cmd__fsmonitor_client(int argc, const char **argv)\n+{\n+\tconst char *subcmd;\n+\tconst char *token = NULL;\n+\n+\tconst char * const fsmonitor_client_usage[] = {\n+\t\tN_(\"test-helper fsmonitor-client query []\"),\n+\t\tN_(\"test-helper fsmonitor-client flush\"),\n+\t\tNULL,\n+\t};\n+\n+\tstruct option options[] = {\n+\t\tOPT_STRING(0, \"token\", &token, N_(\"token\"),\n+\t\t\t N_(\"command token to send to the server\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc < 2)\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(fsmonitor_client_usage, options);\n+\n+\tsubcmd = argv[1];\n+\targv--;\n+\targc++;\n+\n+\targc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0);\n+\n+\tsetup_git_directory();\n+\n+\tif (!strcmp(subcmd, \"query\"))\n+\t\treturn !!do_send_query(token);\n+\n+\tif (!strcmp(subcmd, \"flush\"))\n+\t\treturn !!do_send_flush();\n+\n+\tdie(\"Unhandled subcommand: '%s'\", subcmd);\n+}\n+#endif\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c5bd0c6d4c7..af879e4a5d7 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -31,6 +31,7 @@ static struct test_cmd cmds[] = {\n \t{ \"dump-untracked-cache\", cmd__dump_untracked_cache },\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"fast-rebase\", cmd__fast_rebase },\n+\t{ \"fsmonitor-client\", cmd__fsmonitor_client },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"genzeros\", cmd__genzeros },\n \t{ \"hashmap\", cmd__hashmap },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e8069a3b222..6c5134b46d9 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -21,6 +21,7 @@ int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n+int cmd__fsmonitor_client(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"428915","messageId":"587580489473a7a2ad665bdf3c482ea5d2c54f61.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:22Z","receivedAt":"2021-07-01T14:48:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nStub in empty implementation of fsmonitor--daemon\nbackend for MacOS.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++\n config.mak.uname | 2 ++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 25 insertions(+)\n create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\nnew file mode 100644\nindex 00000000000..b91058d1c4f\n--- /dev/null\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n@@ -0,0 +1,20 @@\n+#include \"cache.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsmonitor-fs-listen.h\"\n+\n+int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn -1;\n+}\n+\n+void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\n+\n+void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex fcd88b60b14..394355463e1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin)\n \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n \t\tendif\n \tendif\n+\tFSMONITOR_DAEMON_BACKEND = macos\n+\tBASIC_LDFLAGS += -framework CoreServices\n endif\n ifeq ($(uname_S),SunOS)\n \tNEEDS_SOCKET = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 1ab94eb284f..aa80671045a 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -266,6 +266,9 @@ endif()\n if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \tlist(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c)\n+elseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n+\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n+\tlist(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-macos.c)\n endif()\n \n set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX})\n-- \ngitgitgadget\n\n"},{"id":"428917","messageId":"9fe902aad87f1192705fb69ea212a2d066d0286d.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 14/34] fsmonitor--daemon: implement 'start' command","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:24Z","receivedAt":"2021-07-01T14:48:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement 'git fsmonitor--daemon start' command. This command\ntries to start a daemon in the background. It creates a background\nprocess to run the daemon.\n\nThe updated daemon does not actually do anything yet because the\nplatform backends are still just stubs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 208 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 208 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex a265c962ccc..7fcf960652f 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -9,6 +9,7 @@\n #include \"khash.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n+\tN_(\"git fsmonitor--daemon start []\"),\n \tN_(\"git fsmonitor--daemon run []\"),\n \tN_(\"git fsmonitor--daemon stop\"),\n \tN_(\"git fsmonitor--daemon status\"),\n@@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = {\n #define FSMONITOR__IPC_THREADS \"fsmonitor.ipcthreads\"\n static int fsmonitor__ipc_threads = 8;\n \n+#define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n+static int fsmonitor__start_timeout_sec = 60;\n+\n static int fsmonitor_config(const char *var, const char *value, void *cb)\n {\n \tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n@@ -33,6 +37,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, FSMONITOR__START_TIMEOUT)) {\n+\t\tint i = git_config_int(var, value);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' out of range: %d\"),\n+\t\t\t\t FSMONITOR__START_TIMEOUT, i);\n+\t\tfsmonitor__start_timeout_sec = i;\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -256,6 +269,194 @@ static int try_to_run_foreground_daemon(void)\n \treturn !!fsmonitor_run_daemon();\n }\n \n+#ifdef GIT_WINDOWS_NATIVE\n+/*\n+ * Create a background process to run the daemon. It should be completely\n+ * disassociated from the terminal.\n+ *\n+ * Conceptually like `daemonize()` but different because Windows does not\n+ * have `fork(2)`. Spawn a normal Windows child process but without the\n+ * limitations of `start_command()` and `finish_command()`.\n+ *\n+ * The child process execs the \"git fsmonitor--daemon run\" command.\n+ *\n+ * The current process returns so that the caller can wait for the child\n+ * to startup before exiting.\n+ */\n+static int spawn_background_fsmonitor_daemon(pid_t *pid)\n+{\n+\tchar git_exe[MAX_PATH];\n+\tstruct strvec args = STRVEC_INIT;\n+\tint in, out;\n+\n+\tGetModuleFileNameA(NULL, git_exe, MAX_PATH);\n+\n+\tin = open(\"/dev/null\", O_RDONLY);\n+\tout = open(\"/dev/null\", O_WRONLY);\n+\n+\tstrvec_push(&args, git_exe);\n+\tstrvec_push(&args, \"fsmonitor--daemon\");\n+\tstrvec_push(&args, \"run\");\n+\tstrvec_pushf(&args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n+\n+\t*pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);\n+\tclose(in);\n+\tclose(out);\n+\n+\tstrvec_clear(&args);\n+\n+\tif (*pid < 0)\n+\t\treturn error(_(\"could not spawn fsmonitor--daemon in the background\"));\n+\n+\treturn 0;\n+}\n+#else\n+/*\n+ * Create a background process to run the daemon. It should be completely\n+ * disassociated from the terminal.\n+ *\n+ * This is adapted from `daemonize()`. Use `fork()` to directly\n+ * create and run the daemon in the child process.\n+ *\n+ * The fork-child can just call the run code; it does not need to exec\n+ * it.\n+ *\n+ * The fork-parent returns the child PID so that we can wait for the\n+ * child to startup before exiting.\n+ */\n+static int spawn_background_fsmonitor_daemon(pid_t *pid)\n+{\n+\t*pid = fork();\n+\n+\tswitch (*pid) {\n+\tcase 0:\n+\t\tif (setsid() == -1)\n+\t\t\terror_errno(_(\"setsid failed\"));\n+\t\tclose(0);\n+\t\tclose(1);\n+\t\tclose(2);\n+\t\tsanitize_stdfds();\n+\n+\t\treturn !!fsmonitor_run_daemon();\n+\n+\tcase -1:\n+\t\treturn error_errno(_(\"could not spawn fsmonitor--daemon in the background\"));\n+\n+\tdefault:\n+\t\treturn 0;\n+\t}\n+}\n+#endif\n+\n+/*\n+ * This is adapted from `wait_or_whine()`. Watch the child process and\n+ * let it get started and begin listening for requests on the socket\n+ * before reporting our success.\n+ */\n+static int wait_for_background_startup(pid_t pid_child)\n+{\n+\tint status;\n+\tpid_t pid_seen;\n+\tenum ipc_active_state s;\n+\ttime_t time_limit, now;\n+\n+\ttime(&time_limit);\n+\ttime_limit += fsmonitor__start_timeout_sec;\n+\n+\tfor (;;) {\n+\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n+\n+\t\tif (pid_seen == -1)\n+\t\t\treturn error_errno(_(\"waitpid failed\"));\n+\t\telse if (pid_seen == 0) {\n+\t\t\t/*\n+\t\t\t * The child is still running (this should be\n+\t\t\t * the normal case). Try to connect to it on\n+\t\t\t * the socket and see if it is ready for\n+\t\t\t * business.\n+\t\t\t *\n+\t\t\t * If there is another daemon already running,\n+\t\t\t * our child will fail to start (possibly\n+\t\t\t * after a timeout on the lock), but we don't\n+\t\t\t * care (who responds) if the socket is live.\n+\t\t\t */\n+\t\t\ts = fsmonitor_ipc__get_state();\n+\t\t\tif (s == IPC_STATE__LISTENING)\n+\t\t\t\treturn 0;\n+\n+\t\t\ttime(&now);\n+\t\t\tif (now > time_limit)\n+\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n+\t\t} else if (pid_seen == pid_child) {\n+\t\t\t/*\n+\t\t\t * The new child daemon process shutdown while\n+\t\t\t * it was starting up, so it is not listening\n+\t\t\t * on the socket.\n+\t\t\t *\n+\t\t\t * Try to ping the socket in the odd chance\n+\t\t\t * that another daemon started (or was already\n+\t\t\t * running) while our child was starting.\n+\t\t\t *\n+\t\t\t * Again, we don't care who services the socket.\n+\t\t\t */\n+\t\t\ts = fsmonitor_ipc__get_state();\n+\t\t\tif (s == IPC_STATE__LISTENING)\n+\t\t\t\treturn 0;\n+\n+\t\t\t/*\n+\t\t\t * We don't care about the WEXITSTATUS() nor\n+\t\t\t * any of the WIF*(status) values because\n+\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n+\t\t\t * trick on all function return values.\n+\t\t\t *\n+\t\t\t * So it is sufficient to just report the\n+\t\t\t * early shutdown as an error.\n+\t\t\t */\n+\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n+\t\t} else\n+\t\t\treturn error(_(\"waitpid is confused\"));\n+\t}\n+}\n+\n+static int try_to_start_background_daemon(void)\n+{\n+\tpid_t pid_child;\n+\tint ret;\n+\n+\t/*\n+\t * Before we try to create a background daemon process, see\n+\t * if a daemon process is already listening. This makes it\n+\t * easier for us to report an already-listening error to the\n+\t * console, since our spawn/daemon can only report the success\n+\t * of creating the background process (and not whether it\n+\t * immediately exited).\n+\t */\n+\tif (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING)\n+\t\tdie(\"fsmonitor--daemon is already running '%s'\",\n+\t\t the_repository->worktree);\n+\n+\tprintf(_(\"starting fsmonitor-daemon in '%s'\\n\"),\n+\t the_repository->worktree);\n+\tfflush(stdout);\n+\n+\t/*\n+\t * Run the actual daemon in a background process.\n+\t */\n+\tret = spawn_background_fsmonitor_daemon(&pid_child);\n+\tif (pid_child <= 0)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Wait (with timeout) for the background child process get\n+\t * started and begin listening on the socket/pipe. This makes\n+\t * the \"start\" command more synchronous and more reliable in\n+\t * tests.\n+\t */\n+\tret = wait_for_background_startup(pid_child);\n+\n+\treturn ret;\n+}\n+\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n@@ -264,6 +465,10 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tOPT_INTEGER(0, \"ipc-threads\",\n \t\t\t &fsmonitor__ipc_threads,\n \t\t\t N_(\"use ipc worker threads\")),\n+\t\tOPT_INTEGER(0, \"start-timeout\",\n+\t\t\t &fsmonitor__start_timeout_sec,\n+\t\t\t N_(\"Max seconds to wait for background daemon startup\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -285,6 +490,9 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tif (!strcmp(subcmd, \"start\"))\n+\t\treturn !!try_to_start_background_daemon();\n+\n \tif (!strcmp(subcmd, \"run\"))\n \t\treturn !!try_to_run_foreground_daemon();\n \n-- \ngitgitgadget\n\n"},{"id":"428918","messageId":"eef39aa168fec58321ee233d907db696b56ee0b8.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 15/34] fsmonitor: do not try to operate on bare repos","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:25Z","receivedAt":"2021-07-01T14:48:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a working directory, so there is no\ndirectory for the daemon to register a watch upon. And therefore\nthere are no files within the directory for it to actually watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 8 ++++++++\n t/t7519-status-fsmonitor.sh | 16 ++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 7fcf960652f..d6161ad95a5 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -490,6 +490,14 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\tif (!the_repository->worktree)\n+\t\treturn error(_(\"fsmonitor-daemon does not support bare repos '%s'\"),\n+\t\t\t xgetcwd());\n+\tif (the_repository->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE)\n+\t\treturn error(_(\"fsmonitor-daemon is incompatible with this repo '%s'\"),\n+\t\t\t the_repository->worktree);\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 02919c68ddd..ed20a4f7fb9 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -394,6 +394,22 @@ test_expect_success 'incompatible bare repo' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success FSMONITOR_DAEMON 'try running fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone\" &&\n+\tgit clone --bare . ./bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"fsmonitor-daemon does not support bare repos\" actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'try running fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone\" &&\n+\tgit clone . ./fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"fsmonitor-daemon is incompatible with this repo\" actual\n+'\n+\n test_expect_success 'incompatible core.virtualfilesystem' '\n \ttest_when_finished \"rm -rf ./fake-gvfs-clone\" &&\n \tgit clone . ./fake-gvfs-clone &&\n-- \ngitgitgadget\n\n"},{"id":"428919","messageId":"3b12f668060fc757d7d36dcad2c86bbb917a8b8b.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 16/34] fsmonitor--daemon: add pathname classification","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:26Z","receivedAt":"2021-07-01T14:48:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to classify relative and absolute\npathnames and decide how they should be handled. This will\nbe used by the platform-specific backend to respond to each\nfilesystem event.\n\nWhen we register for filesystem notifications on a directory,\nwe get events for everything (recursively) in the directory.\nWe want to report to clients changes to tracked and untracked\npaths within the working directory. We do not want to report\nchanges within the .git directory, for example.\n\nThis classification will be used in a later commit by the\ndifferent backends to classify paths as events are received.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 81 +++++++++++++++++++++++++++++++++++++\n fsmonitor--daemon.h | 61 ++++++++++++++++++++++++++++\n 2 files changed, 142 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex d6161ad95a5..e942f7c5840 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -121,6 +121,87 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n+#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *rel)\n+{\n+\tif (fspathncmp(rel, \".git\", 4))\n+\t\treturn IS_WORKDIR_PATH;\n+\trel += 4;\n+\n+\tif (!*rel)\n+\t\treturn IS_DOT_GIT;\n+\tif (*rel != '/')\n+\t\treturn IS_WORKDIR_PATH; /* e.g. .gitignore */\n+\trel++;\n+\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_DOT_GIT;\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *rel)\n+{\n+\tif (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX,\n+\t\t\tstrlen(FSMONITOR_COOKIE_PREFIX)))\n+\t\treturn IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX;\n+\n+\treturn IS_INSIDE_GITDIR;\n+}\n+\n+static enum fsmonitor_path_type try_classify_workdir_abs_path(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\n+\tif (fspathncmp(path, state->path_worktree_watch.buf,\n+\t\t state->path_worktree_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_worktree_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_WORKDIR_PATH; /* it is the root dir exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_workdir_relative(rel);\n+}\n+\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path)\n+{\n+\tconst char *rel;\n+\tenum fsmonitor_path_type t;\n+\n+\tt = try_classify_workdir_abs_path(state, path);\n+\tif (state->nr_paths_watching == 1)\n+\t\treturn t;\n+\tif (t != IS_OUTSIDE_CONE)\n+\t\treturn t;\n+\n+\tif (fspathncmp(path, state->path_gitdir_watch.buf,\n+\t\t state->path_gitdir_watch.len))\n+\t\treturn IS_OUTSIDE_CONE;\n+\n+\trel = path + state->path_gitdir_watch.len;\n+\n+\tif (!*rel)\n+\t\treturn IS_GITDIR; /* it is the exactly */\n+\tif (*rel != '/')\n+\t\treturn IS_OUTSIDE_CONE;\n+\trel++;\n+\n+\treturn fsmonitor_classify_path_gitdir_relative(rel);\n+}\n+\n static void *fsmonitor_fs_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 3009c1a83de..7bbb3a27a1c 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -30,5 +30,66 @@ struct fsmonitor_daemon_state {\n \tstruct ipc_server_data *ipc_server_data;\n };\n \n+/*\n+ * Pathname classifications.\n+ *\n+ * The daemon classifies the pathnames that it receives from file\n+ * system notification events into the following categories and uses\n+ * that to decide whether clients are told about them. (And to watch\n+ * for file system synchronization events.)\n+ *\n+ * The client should only care about paths within the working\n+ * directory proper (inside the working directory and not \".git\" nor\n+ * inside of \".git/\"). That is, the client has read the index and is\n+ * asking for a list of any paths in the working directory that have\n+ * been modified since the last token. The client does not care about\n+ * file system changes within the .git directory (such as new loose\n+ * objects or packfiles). So the client will only receive paths that\n+ * are classified as IS_WORKDIR_PATH.\n+ *\n+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the\n+ * exact \".git\" directory or GITDIR. If the daemon receives a delete\n+ * event for either of these directories, it will automatically\n+ * shutdown, for example.\n+ *\n+ * Note that the daemon DOES NOT explicitly watch nor special case the\n+ * \".git/index\" file. The daemon does not read the index and does not\n+ * have any internal index-relative state. The daemon only collects\n+ * the set of modified paths within the working directory.\n+ */\n+enum fsmonitor_path_type {\n+\tIS_WORKDIR_PATH = 0,\n+\n+\tIS_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT,\n+\tIS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX,\n+\n+\tIS_GITDIR,\n+\tIS_INSIDE_GITDIR,\n+\tIS_INSIDE_GITDIR_WITH_COOKIE_PREFIX,\n+\n+\tIS_OUTSIDE_CONE,\n+};\n+\n+/*\n+ * Classify a pathname relative to the root of the working directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify a pathname relative to a that is external to the\n+ * worktree directory.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative(\n+\tconst char *relative_path);\n+\n+/*\n+ * Classify an absolute pathname received from a filesystem event.\n+ */\n+enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst char *path);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"428920","messageId":"84444c44c324751ecaed30a4fa2a868014da4397.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 18/34] fsmonitor--daemon: create token-based changed path cache","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:28Z","receivedAt":"2021-07-01T14:48:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to build a list of changed paths and associate\nthem with a token-id. This will be used by the platform-specific\nbackends to accumulate changed paths in response to filesystem events.\n\nThe platform-specific file system listener thread receives file system\nevents containing one or more changed pathnames (with whatever bucketing\nor grouping that is convenient for the file system). These paths are\naccumulated (without locking) by the file system layer into a `fsmonitor_batch`.\n\nWhen the file system layer has drained the kernel event queue, it will\n\"publish\" them to our token queue and make them visible to concurrent\nclient worker threads. The token layer is free to combine and/or de-dup\npaths within these batches for efficient presentation to clients.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 234 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 40 ++++++\n 2 files changed, 272 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex e991925fafc..ea3a52d34e3 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -168,17 +168,27 @@ struct fsmonitor_token_data {\n \tuint64_t client_ref_count;\n };\n \n+struct fsmonitor_batch {\n+\tstruct fsmonitor_batch *next;\n+\tuint64_t batch_seq_nr;\n+\tconst char **interned_paths;\n+\tsize_t nr, alloc;\n+\ttime_t pinned_time;\n+};\n+\n static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n {\n \tstatic int test_env_value = -1;\n \tstatic uint64_t flush_count = 0;\n \tstruct fsmonitor_token_data *token;\n+\tstruct fsmonitor_batch *batch;\n \n \tCALLOC_ARRAY(token, 1);\n+\tbatch = fsmonitor_batch__new();\n \n \tstrbuf_init(&token->token_id, 0);\n-\ttoken->batch_head = NULL;\n-\ttoken->batch_tail = NULL;\n+\ttoken->batch_head = batch;\n+\ttoken->batch_tail = batch;\n \ttoken->client_ref_count = 0;\n \n \tif (test_env_value < 0)\n@@ -204,9 +214,147 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n \t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n \t}\n \n+\t/*\n+\t * We created a new and are starting a new series\n+\t * of tokens with a zero .\n+\t *\n+\t * Since clients cannot guess our new (non test) \n+\t * they will always receive a trivial response (because of the\n+\t * mismatch on the ). The trivial response will\n+\t * tell them our new so that subsequent requests\n+\t * will be relative to our new series. (And when sending that\n+\t * response, we pin the current head of the batch list.)\n+\t *\n+\t * Even if the client correctly guesses the , their\n+\t * request of \"builtin::0\" asks for all changes MORE\n+\t * RECENT than batch/bin 0.\n+\t *\n+\t * This implies that it is a waste to accumulate paths in the\n+\t * initial batch/bin (because they will never be transmitted).\n+\t *\n+\t * So the daemon could be running for days and watching the\n+\t * file system, but doesn't need to actually accumulate any\n+\t * paths UNTIL we need to set a reference point for a later\n+\t * relative request.\n+\t *\n+\t * However, it is very useful for testing to always have a\n+\t * reference point set. Pin batch 0 to force early file system\n+\t * events to accumulate.\n+\t */\n+\tif (test_env_value)\n+\t\tbatch->pinned_time = time(NULL);\n+\n \treturn token;\n }\n \n+struct fsmonitor_batch *fsmonitor_batch__new(void)\n+{\n+\tstruct fsmonitor_batch *batch;\n+\n+\tCALLOC_ARRAY(batch, 1);\n+\n+\treturn batch;\n+}\n+\n+struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch)\n+{\n+\tstruct fsmonitor_batch *next;\n+\n+\tif (!batch)\n+\t\treturn NULL;\n+\n+\tnext = batch->next;\n+\n+\t/*\n+\t * The actual strings within the array are interned, so we don't\n+\t * own them.\n+\t */\n+\tfree(batch->interned_paths);\n+\n+\treturn next;\n+}\n+\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch,\n+\t\t\t const char *path)\n+{\n+\tconst char *interned_path = strintern(path);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"event: %s\", interned_path);\n+\n+\tALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc);\n+\tbatch->interned_paths[batch->nr++] = interned_path;\n+}\n+\n+static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n+\t\t\t\t const struct fsmonitor_batch *batch_src)\n+{\n+\tsize_t k;\n+\n+\tALLOC_GROW(batch_dest->interned_paths,\n+\t\t batch_dest->nr + batch_src->nr + 1,\n+\t\t batch_dest->alloc);\n+\n+\tfor (k = 0; k < batch_src->nr; k++)\n+\t\tbatch_dest->interned_paths[batch_dest->nr++] =\n+\t\t\tbatch_src->interned_paths[k];\n+}\n+\n+static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n+{\n+\tstruct fsmonitor_batch *p;\n+\n+\tif (!token)\n+\t\treturn;\n+\n+\tassert(token->client_ref_count == 0);\n+\n+\tstrbuf_release(&token->token_id);\n+\n+\tfor (p = token->batch_head; p; p = fsmonitor_batch__pop(p))\n+\t\t;\n+\n+\tfree(token);\n+}\n+\n+/*\n+ * Flush all of our cached data about the filesystem. Call this if we\n+ * lose sync with the filesystem and miss some notification events.\n+ *\n+ * [1] If we are missing events, then we no longer have a complete\n+ * history of the directory (relative to our current start token).\n+ * We should create a new token and start fresh (as if we just\n+ * booted up).\n+ *\n+ * If there are no concurrent threads readering the current token data\n+ * series, we can free it now. Otherwise, let the last reader free\n+ * it.\n+ *\n+ * Either way, the old token data series is no longer associated with\n+ * our state data.\n+ */\n+static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct fsmonitor_token_data *free_me = NULL;\n+\tstruct fsmonitor_token_data *new_one = NULL;\n+\n+\tnew_one = fsmonitor_new_token_data();\n+\n+\tif (state->current_token_data->client_ref_count == 0)\n+\t\tfree_me = state->current_token_data;\n+\tstate->current_token_data = new_one;\n+\n+\tfsmonitor_free_token_data(free_me);\n+}\n+\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n+{\n+\tpthread_mutex_lock(&state->main_lock);\n+\twith_lock__do_force_resync(state);\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -316,6 +464,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \treturn fsmonitor_classify_path_gitdir_relative(rel);\n }\n \n+/*\n+ * We try to combine small batches at the front of the batch-list to avoid\n+ * having a long list. This hopefully makes it a little easier when we want\n+ * to truncate and maintain the list. However, we don't want the paths array\n+ * to just keep growing and growing with realloc, so we insert an arbitrary\n+ * limit.\n+ */\n+#define MY_COMBINE_LIMIT (1024)\n+\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names)\n+{\n+\tif (!batch && !cookie_names->nr)\n+\t\treturn;\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (batch) {\n+\t\tstruct fsmonitor_batch *head;\n+\n+\t\thead = state->current_token_data->batch_head;\n+\t\tif (!head) {\n+\t\t\tBUG(\"token does not have batch\");\n+\t\t} else if (head->pinned_time) {\n+\t\t\t/*\n+\t\t\t * We cannot alter the current batch list\n+\t\t\t * because:\n+\t\t\t *\n+\t\t\t * [a] it is being transmitted to at least one\n+\t\t\t * client and the handle_client() thread has a\n+\t\t\t * ref-count, but not a lock on the batch list\n+\t\t\t * starting with this item.\n+\t\t\t *\n+\t\t\t * [b] it has been transmitted in the past to\n+\t\t\t * at least one client such that future\n+\t\t\t * requests are relative to this head batch.\n+\t\t\t *\n+\t\t\t * So, we can only prepend a new batch onto\n+\t\t\t * the front of the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else if (!head->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * Batch 0 is unpinned. See the note in\n+\t\t\t * `fsmonitor_new_token_data()` about why we\n+\t\t\t * don't need to accumulate these paths.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__pop(batch);\n+\t\t} else if (head->nr + batch->nr > MY_COMBINE_LIMIT) {\n+\t\t\t/*\n+\t\t\t * The head batch in the list has never been\n+\t\t\t * transmitted to a client, but folding the\n+\t\t\t * contents of the new batch onto it would\n+\t\t\t * exceed our arbitrary limit, so just prepend\n+\t\t\t * the new batch onto the list.\n+\t\t\t */\n+\t\t\tbatch->batch_seq_nr = head->batch_seq_nr + 1;\n+\t\t\tbatch->next = head;\n+\t\t\tstate->current_token_data->batch_head = batch;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We are free to append the paths in the given\n+\t\t\t * batch onto the end of the current head batch.\n+\t\t\t */\n+\t\t\tfsmonitor_batch__combine(head, batch);\n+\t\t\tfsmonitor_batch__pop(batch);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+}\n+\n static void *fsmonitor_fs_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -330,6 +553,13 @@ static void *fsmonitor_fs_listen__thread_proc(void *_state)\n \n \tfsmonitor_fs_listen__loop(state);\n \n+\tpthread_mutex_lock(&state->main_lock);\n+\tif (state->current_token_data &&\n+\t state->current_token_data->client_ref_count == 0)\n+\t\tfsmonitor_free_token_data(state->current_token_data);\n+\tstate->current_token_data = NULL;\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n \ttrace2_thread_exit();\n \treturn NULL;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 7bbb3a27a1c..89a9ef20b24 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -12,6 +12,27 @@\n struct fsmonitor_batch;\n struct fsmonitor_token_data;\n \n+/*\n+ * Create a new batch of path(s). The returned batch is considered\n+ * private and not linked into the fsmonitor daemon state. The caller\n+ * should fill this batch with one or more paths and then publish it.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__new(void);\n+\n+/*\n+ * Free this batch and return the value of the batch->next field.\n+ */\n+struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch);\n+\n+/*\n+ * Add this path to this batch of modified files.\n+ *\n+ * The batch should be private and NOT (yet) linked into the fsmonitor\n+ * daemon state and therefore not yet visible to worker threads and so\n+ * no locking is required.\n+ */\n+void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n+\n struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n \n struct fsmonitor_daemon_state {\n@@ -91,5 +112,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute(\n \tstruct fsmonitor_daemon_state *state,\n \tconst char *path);\n \n+/*\n+ * Prepend the this batch of path(s) onto the list of batches associated\n+ * with the current token. This makes the batch visible to worker threads.\n+ *\n+ * The caller no longer owns the batch and must not free it.\n+ *\n+ * Wake up the client threads waiting on these cookies.\n+ */\n+void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n+\t\t struct fsmonitor_batch *batch,\n+\t\t const struct string_list *cookie_names);\n+\n+/*\n+ * If the platform-specific layer loses sync with the filesystem,\n+ * it should call this to invalidate cached data and abort waiting\n+ * threads.\n+ */\n+void fsmonitor_force_resync(struct fsmonitor_daemon_state *state);\n+\n #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n #endif /* FSMONITOR_DAEMON_H */\n-- \ngitgitgadget\n\n"},{"id":"428921","messageId":"37fdce5ec3afaa9aae5001c648fced0675dae0c4.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 17/34] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:27Z","receivedAt":"2021-07-01T14:48:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to create token-ids and define the\noverall token naming scheme.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 115 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex e942f7c5840..e991925fafc 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -93,6 +93,120 @@ static int do_as_client__status(void)\n \t}\n }\n \n+/*\n+ * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n+ * \"token\" as a virtual timestamp. Clients can request a summary of all\n+ * created/deleted/modified files relative to a token. In the response,\n+ * clients receive a new token for the next (relative) request.\n+ *\n+ *\n+ * Token Format\n+ * ============\n+ *\n+ * The contents of the token are private and provider-specific.\n+ *\n+ * For the built-in fsmonitor--daemon, we define a token as follows:\n+ *\n+ * \"builtin\" \":\" \":\" \n+ *\n+ * The \"builtin\" prefix is used as a namespace to avoid conflicts\n+ * with other providers (such as Watchman).\n+ *\n+ * The is an arbitrary OPAQUE string, such as a GUID,\n+ * UUID, or {timestamp,pid}. It is used to group all filesystem\n+ * events that happened while the daemon was monitoring (and in-sync\n+ * with the filesystem).\n+ *\n+ * Unlike FSMonitor Protocol V1, it is not defined as a timestamp\n+ * and does not define less-than/greater-than relationships.\n+ * (There are too many race conditions to rely on file system\n+ * event timestamps.)\n+ *\n+ * The is a simple integer incremented whenever the\n+ * daemon needs to make its state public. For example, if 1000 file\n+ * system events come in, but no clients have requested the data,\n+ * the daemon can continue to accumulate file changes in the same\n+ * bin and does not need to advance the sequence number. However,\n+ * as soon as a client does arrive, the daemon needs to start a new\n+ * bin and increment the sequence number.\n+ *\n+ * The sequence number serves as the boundary between 2 sets\n+ * of bins -- the older ones that the client has already seen\n+ * and the newer ones that it hasn't.\n+ *\n+ * When a new is created, the is reset to\n+ * zero.\n+ *\n+ *\n+ * About Token Ids\n+ * ===============\n+ *\n+ * A new token_id is created:\n+ *\n+ * [1] each time the daemon is started.\n+ *\n+ * [2] any time that the daemon must re-sync with the filesystem\n+ * (such as when the kernel drops or we miss events on a very\n+ * active volume).\n+ *\n+ * [3] in response to a client \"flush\" command (for dropped event\n+ * testing).\n+ *\n+ * When a new token_id is created, the daemon is free to discard all\n+ * cached filesystem events associated with any previous token_ids.\n+ * Events associated with a non-current token_id will never be sent\n+ * to a client. A token_id change implicitly means that the daemon\n+ * has gap in its event history.\n+ *\n+ * Therefore, clients that present a token with a stale (non-current)\n+ * token_id will always be given a trivial response.\n+ */\n+struct fsmonitor_token_data {\n+\tstruct strbuf token_id;\n+\tstruct fsmonitor_batch *batch_head;\n+\tstruct fsmonitor_batch *batch_tail;\n+\tuint64_t client_ref_count;\n+};\n+\n+static struct fsmonitor_token_data *fsmonitor_new_token_data(void)\n+{\n+\tstatic int test_env_value = -1;\n+\tstatic uint64_t flush_count = 0;\n+\tstruct fsmonitor_token_data *token;\n+\n+\tCALLOC_ARRAY(token, 1);\n+\n+\tstrbuf_init(&token->token_id, 0);\n+\ttoken->batch_head = NULL;\n+\ttoken->batch_tail = NULL;\n+\ttoken->client_ref_count = 0;\n+\n+\tif (test_env_value < 0)\n+\t\ttest_env_value = git_env_bool(\"GIT_TEST_FSMONITOR_TOKEN\", 0);\n+\n+\tif (!test_env_value) {\n+\t\tstruct timeval tv;\n+\t\tstruct tm tm;\n+\t\ttime_t secs;\n+\n+\t\tgettimeofday(&tv, NULL);\n+\t\tsecs = tv.tv_sec;\n+\t\tgmtime_r(&secs, &tm);\n+\n+\t\tstrbuf_addf(&token->token_id,\n+\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n+\t\t\t flush_count++,\n+\t\t\t getpid(),\n+\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n+\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n+\t\t\t (long)tv.tv_usec);\n+\t} else {\n+\t\tstrbuf_addf(&token->token_id, \"test_%08x\", test_env_value++);\n+\t}\n+\n+\treturn token;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -283,7 +397,7 @@ static int fsmonitor_run_daemon(void)\n \n \tpthread_mutex_init(&state.main_lock, NULL);\n \tstate.error_code = 0;\n-\tstate.current_token_data = NULL;\n+\tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n \tstrbuf_init(&state.path_worktree_watch, 0);\n-- \ngitgitgadget\n\n"},{"id":"428922","messageId":"175ae9a757e33fb119b4e01c76f851fa71ff4c9b.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 20/34] fsmonitor-fs-listen-macos: add macos header files for FSEvent","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:30Z","receivedAt":"2021-07-01T14:48:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInclude MacOS system declarations to allow us to use FSEvent and\nCoreFoundation APIs. We need GCC and clang versions because of\ncompiler and header file conflicts.\n\nWhile it is quite possible to #include Apple's CoreServices.h when\ncompiling C source code with clang, trying to build it with GCC\ncurrently fails with this error:\n\nIn file included\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23,\n from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,\n /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope\n 193 | char bytes[kAuthorizationExternalFormLength];\n | ^~~~~\n\nThe underlying reason is that GCC (rightfully) objects that an `enum`\nvalue such as `kAuthorizationExternalFormLength` is not a constant\n(because it is not, the preprocessor has no knowledge of it, only the\nactual C compiler does) and can therefore not be used to define the size\nof a C array.\n\nThis is a known problem and tracked in GCC's bug tracker:\nhttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082\n\nIn the meantime, let's not block things and go the slightly ugly route\nof declaring/defining the FSEvents constants, data structures and\nfunctions that we need, so that we can avoid above-mentioned issue.\n\nLet's do this _only_ for GCC, though, so that the CI/PR builds (which\nbuild both with clang and with GCC) can guarantee that we _are_ using\nthe correct data types.\n\nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 96 ++++++++++++++++++++\n 1 file changed, 96 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\nindex b91058d1c4f..bec5130d9e1 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n@@ -1,3 +1,99 @@\n+#if defined(__GNUC__)\n+/*\n+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling\n+ * with clang, but not with GCC as of time of writing.\n+ *\n+ * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details.\n+ */\n+typedef unsigned int FSEventStreamCreateFlags;\n+#define kFSEventStreamEventFlagNone 0x00000000\n+#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001\n+#define kFSEventStreamEventFlagUserDropped 0x00000002\n+#define kFSEventStreamEventFlagKernelDropped 0x00000004\n+#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008\n+#define kFSEventStreamEventFlagHistoryDone 0x00000010\n+#define kFSEventStreamEventFlagRootChanged 0x00000020\n+#define kFSEventStreamEventFlagMount 0x00000040\n+#define kFSEventStreamEventFlagUnmount 0x00000080\n+#define kFSEventStreamEventFlagItemCreated 0x00000100\n+#define kFSEventStreamEventFlagItemRemoved 0x00000200\n+#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400\n+#define kFSEventStreamEventFlagItemRenamed 0x00000800\n+#define kFSEventStreamEventFlagItemModified 0x00001000\n+#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000\n+#define kFSEventStreamEventFlagItemChangeOwner 0x00004000\n+#define kFSEventStreamEventFlagItemXattrMod 0x00008000\n+#define kFSEventStreamEventFlagItemIsFile 0x00010000\n+#define kFSEventStreamEventFlagItemIsDir 0x00020000\n+#define kFSEventStreamEventFlagItemIsSymlink 0x00040000\n+#define kFSEventStreamEventFlagOwnEvent 0x00080000\n+#define kFSEventStreamEventFlagItemIsHardlink 0x00100000\n+#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000\n+#define kFSEventStreamEventFlagItemCloned 0x00400000\n+\n+typedef struct __FSEventStream *FSEventStreamRef;\n+typedef const FSEventStreamRef ConstFSEventStreamRef;\n+\n+typedef unsigned int CFStringEncoding;\n+#define kCFStringEncodingUTF8 0x08000100\n+\n+typedef const struct __CFString *CFStringRef;\n+typedef const struct __CFArray *CFArrayRef;\n+typedef const struct __CFRunLoop *CFRunLoopRef;\n+\n+struct FSEventStreamContext {\n+ long long version;\n+ void *cb_data, *retain, *release, *copy_description;\n+};\n+\n+typedef struct FSEventStreamContext FSEventStreamContext;\n+typedef unsigned int FSEventStreamEventFlags;\n+#define kFSEventStreamCreateFlagNoDefer 0x02\n+#define kFSEventStreamCreateFlagWatchRoot 0x04\n+#define kFSEventStreamCreateFlagFileEvents 0x10\n+\n+typedef unsigned long long FSEventStreamEventId;\n+#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL\n+\n+typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef,\n+\t\t\t\t void *context,\n+\t\t\t\t __SIZE_TYPE__ num_of_events,\n+\t\t\t\t void *event_paths,\n+\t\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t\t const FSEventStreamEventId event_ids[]);\n+typedef double CFTimeInterval;\n+FSEventStreamRef FSEventStreamCreate(void *allocator,\n+\t\t\t\t FSEventStreamCallback callback,\n+\t\t\t\t FSEventStreamContext *context,\n+\t\t\t\t CFArrayRef paths_to_watch,\n+\t\t\t\t FSEventStreamEventId since_when,\n+\t\t\t\t CFTimeInterval latency,\n+\t\t\t\t FSEventStreamCreateFlags flags);\n+CFStringRef CFStringCreateWithCString(void *allocator, const char *string,\n+\t\t\t\t CFStringEncoding encoding);\n+CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count,\n+\t\t\t void *callbacks);\n+void CFRunLoopRun(void);\n+void CFRunLoopStop(CFRunLoopRef run_loop);\n+CFRunLoopRef CFRunLoopGetCurrent(void);\n+extern CFStringRef kCFRunLoopDefaultMode;\n+void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream,\n+\t\t\t\t CFRunLoopRef run_loop,\n+\t\t\t\t CFStringRef run_loop_mode);\n+unsigned char FSEventStreamStart(FSEventStreamRef stream);\n+void FSEventStreamStop(FSEventStreamRef stream);\n+void FSEventStreamInvalidate(FSEventStreamRef stream);\n+void FSEventStreamRelease(FSEventStreamRef stream);\n+#else\n+/*\n+ * Let Apple's headers declare `isalnum()` first, before\n+ * Git's headers override it via a constant\n+ */\n+#include \n+#include \n+#include \n+#endif\n+\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-fs-listen.h\"\n-- \ngitgitgadget\n\n"},{"id":"428923","messageId":"5bba5eb3d1bd172f09fdf6eb2e9b8ac4dd7f940f.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:29Z","receivedAt":"2021-07-01T14:48:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the win32 backend to register a watch on the working tree\nroot directory (recursively). Also watch the if it is\nnot inside the working tree. And to collect path change notifications\ninto batches and publish.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++\n 1 file changed, 530 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\nindex 880446b49e3..d707d47a0d7 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n@@ -2,20 +2,550 @@\n #include \"config.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-fs-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+/*\n+ * The documentation of ReadDirectoryChangesW() states that the maximum\n+ * buffer size is 64K when the monitored directory is remote.\n+ *\n+ * Larger buffers may be used when the monitored directory is local and\n+ * will help us receive events faster from the kernel and avoid dropped\n+ * events.\n+ *\n+ * So we try to use a very large buffer and silently fallback to 64K if\n+ * we get an error.\n+ */\n+#define MAX_RDCW_BUF_FALLBACK (65536)\n+#define MAX_RDCW_BUF (65536 * 8)\n+\n+struct one_watch\n+{\n+\tchar buffer[MAX_RDCW_BUF];\n+\tDWORD buf_len;\n+\tDWORD count;\n+\n+\tstruct strbuf path;\n+\tHANDLE hDir;\n+\tHANDLE hEvent;\n+\tOVERLAPPED overlapped;\n+\n+\t/*\n+\t * Is there an active ReadDirectoryChangesW() call pending. If so, we\n+\t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n+\t */\n+\tBOOL is_active;\n+};\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tstruct one_watch *watch_worktree;\n+\tstruct one_watch *watch_gitdir;\n+\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hListener[3]; /* we don't own these handles */\n+#define LISTENER_SHUTDOWN 0\n+#define LISTENER_HAVE_DATA_WORKTREE 1\n+#define LISTENER_HAVE_DATA_GITDIR 2\n+\tint nr_listener_handles;\n+};\n+\n+/*\n+ * Convert the WCHAR path from the notification into UTF8 and\n+ * then normalize it.\n+ */\n+static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+\t\t\t\t struct strbuf *normalized_path)\n+{\n+\tint reserve;\n+\tint len = 0;\n+\n+\tstrbuf_reset(normalized_path);\n+\tif (!info->FileNameLength)\n+\t\tgoto normalize;\n+\n+\t/*\n+\t * Pre-reserve enough space in the UTF8 buffer for\n+\t * each Unicode WCHAR character to be mapped into a\n+\t * sequence of 2 UTF8 characters. That should let us\n+\t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n+\t */\n+\treserve = info->FileNameLength + 1;\n+\tstrbuf_grow(normalized_path, reserve);\n+\n+\tfor (;;) {\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n+\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t\t\t normalized_path->buf,\n+\t\t\t\t\t strbuf_avail(normalized_path) - 1,\n+\t\t\t\t\t NULL, NULL);\n+\t\tif (len > 0)\n+\t\t\tgoto normalize;\n+\t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n+\t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n+\t\t\t GetLastError(),\n+\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n+\t\t\t info->FileName);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tstrbuf_grow(normalized_path,\n+\t\t\t strbuf_avail(normalized_path) + reserve);\n+\t}\n+\n+normalize:\n+\tstrbuf_setlen(normalized_path, len);\n+\treturn strbuf_normalize_path(normalized_path);\n+}\n \n void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+}\n+\n+static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n+\t\t\t\t const char *path)\n+{\n+\tstruct one_watch *watch = NULL;\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileA(path,\n+\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n+\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n+\t\t\t NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] could not watch '%s'\"),\n+\t\t GetLastError(), path);\n+\t\treturn NULL;\n+\t}\n+\n+\tCALLOC_ARRAY(watch, 1);\n+\n+\twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n+\n+\tstrbuf_init(&watch->path, 0);\n+\tstrbuf_addstr(&watch->path, path);\n+\n+\twatch->hDir = hDir;\n+\twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\treturn watch;\n+}\n+\n+static void destroy_watch(struct one_watch *watch)\n+{\n+\tif (!watch)\n+\t\treturn;\n+\n+\tstrbuf_release(&watch->path);\n+\tif (watch->hDir != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hDir);\n+\tif (watch->hEvent != INVALID_HANDLE_VALUE)\n+\t\tCloseHandle(watch->hEvent);\n+\n+\tfree(watch);\n+}\n+\n+static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+\t\t\t struct one_watch *watch)\n+{\n+\tDWORD dwNotifyFilter =\n+\t\tFILE_NOTIFY_CHANGE_FILE_NAME |\n+\t\tFILE_NOTIFY_CHANGE_DIR_NAME |\n+\t\tFILE_NOTIFY_CHANGE_ATTRIBUTES |\n+\t\tFILE_NOTIFY_CHANGE_SIZE |\n+\t\tFILE_NOTIFY_CHANGE_LAST_WRITE |\n+\t\tFILE_NOTIFY_CHANGE_CREATION;\n+\n+\tResetEvent(watch->hEvent);\n+\n+\tmemset(&watch->overlapped, 0, sizeof(watch->overlapped));\n+\twatch->overlapped.hEvent = watch->hEvent;\n+\n+start_watch:\n+\t/*\n+\t * Queue an async call using Overlapped IO. This returns immediately.\n+\t * Our event handle will be signalled when the real result is available.\n+\t *\n+\t * The return value here just means that we successfully queued it.\n+\t * We won't know if the Read...() actually produces data until later.\n+\t */\n+\twatch->is_active = ReadDirectoryChangesW(\n+\t\twatch->hDir, watch->buffer, watch->buf_len, TRUE,\n+\t\tdwNotifyFilter, &watch->count, &watch->overlapped, NULL);\n+\n+\t/*\n+\t * The kernel throws an invalid parameter error when our buffer\n+\t * is too big and we are pointed at a remote directory (and possibly\n+\t * for other reasons). Quietly set it down and try again.\n+\t *\n+\t * See note about MAX_RDCW_BUF at the top.\n+\t */\n+\tif (!watch->is_active &&\n+\t GetLastError() == ERROR_INVALID_PARAMETER &&\n+\t watch->buf_len > MAX_RDCW_BUF_FALLBACK) {\n+\t\twatch->buf_len = MAX_RDCW_BUF_FALLBACK;\n+\t\tgoto start_watch;\n+\t}\n+\n+\tif (watch->is_active)\n+\t\treturn 0;\n+\n+\terror(\"ReadDirectoryChangedW failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static int recv_rdcw_watch(struct one_watch *watch)\n+{\n+\twatch->is_active = FALSE;\n+\n+\t/*\n+\t * The overlapped result is ready. If the Read...() was successful\n+\t * we finally receive the actual result into our buffer.\n+\t */\n+\tif (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count,\n+\t\t\t\tTRUE))\n+\t\treturn 0;\n+\n+\t/*\n+\t * NEEDSWORK: If an external is deleted, the above\n+\t * returns an error. I'm not sure that there's anything that\n+\t * we can do here other than failing -- the /.git\n+\t * link file would be broken anyway. We might try to check\n+\t * for that and return a better error message, but I'm not\n+\t * sure it is worth it.\n+\t */\n+\n+\terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n+\t watch->path.buf, GetLastError());\n+\treturn -1;\n+}\n+\n+static void cancel_rdcw_watch(struct one_watch *watch)\n+{\n+\tDWORD count;\n+\n+\tif (!watch || !watch->is_active)\n+\t\treturn;\n+\n+\t/*\n+\t * The calls to ReadDirectoryChangesW() and GetOverlappedResult()\n+\t * form a \"pair\" (my term) where we queue an IO and promise to\n+\t * hang around and wait for the kernel to give us the result.\n+\t *\n+\t * If for some reason after we queue the IO, we have to quit\n+\t * or otherwise not stick around for the second half, we must\n+\t * tell the kernel to abort the IO. This prevents the kernel\n+\t * from writing to our buffer and/or signalling our event\n+\t * after we free them.\n+\t *\n+\t * (Ask me how much fun it was to track that one down).\n+\t */\n+\tCancelIoEx(watch->hDir, &watch->overlapped);\n+\tGetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE);\n+\twatch->is_active = FALSE;\n+}\n+\n+/*\n+ * Process filesystem events that happen anywhere (recursively) under the\n+ * root directory. For a normal working directory, this includes\n+ * both version controlled files and the contents of the .git/ directory.\n+ *\n+ * If /.git is a file, then we only see events for the file\n+ * itself.\n+ */\n+static int process_worktree_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_worktree;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tconst char *p = watch->buffer;\n+\n+\t/*\n+\t * If the kernel gets more events than will fit in the kernel\n+\t * buffer associated with our RDCW handle, it drops them and\n+\t * returns a count of zero.\n+\t *\n+\t * Yes, the call returns WITHOUT error and with length zero.\n+\t *\n+\t * (The \"overflow\" case is not ambiguous with the \"no data\" case\n+\t * because we did an INFINITE wait.)\n+\t *\n+\t * This means we have a gap in coverage. Tell the daemon layer\n+\t * to resync.\n+\t */\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\t}\n+\n+\t/*\n+\t * On Windows, `info` contains an \"array\" of paths that are\n+\t * relative to the root of whichever directory handle received\n+\t * the event.\n+\t */\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\t\t/* ignore everything inside of \"/.git/\" */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t\t \"removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* queue normal pathname */\n+\t\t\tif (!batch)\n+\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tbatch = NULL;\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_WORKTREE;\n+\n+force_shutdown:\n+\tfsmonitor_batch__pop(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_SHUTDOWN;\n+}\n+\n+/*\n+ * Process filesystem events that happened anywhere (recursively) under the\n+ * external (such as non-primary worktrees or submodules).\n+ * We only care about cookie files that our client threads created here.\n+ *\n+ * Note that we DO NOT get filesystem events on the external \n+ * itself (it is not inside something that we are watching). In particular,\n+ * we do not get an event if the external is deleted.\n+ */\n+static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct one_watch *watch = data->watch_gitdir;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *p = watch->buffer;\n+\n+\tif (!watch->count) {\n+\t\ttrace2_data_string(\"fsmonitor\", NULL, \"fsm-listen/kernel\",\n+\t\t\t\t \"overflow\");\n+\t\tfsmonitor_force_resync(state);\n+\t\treturn LISTENER_HAVE_DATA_GITDIR;\n+\t}\n+\n+\tfor (;;) {\n+\t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n+\t\tconst char *slash;\n+\t\tenum fsmonitor_path_type t;\n+\n+\t\tstrbuf_reset(&path);\n+\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\t\tgoto skip_this_path;\n+\n+\t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n+\n+\t\tswitch (t) {\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path.buf);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path.buf);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\tgoto skip_this_path;\n+\n+\t\tdefault:\n+\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t\t t, path.buf);\n+\t\t}\n+\n+skip_this_path:\n+\t\tif (!info->NextEntryOffset)\n+\t\t\tbreak;\n+\t\tp += info->NextEntryOffset;\n+\t}\n+\n+\tfsmonitor_publish(state, NULL, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&path);\n+\treturn LISTENER_HAVE_DATA_GITDIR;\n }\n \n void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tDWORD dwWait;\n+\n+\tstate->error_code = 0;\n+\n+\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tif (data->watch_gitdir &&\n+\t start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\tgoto force_error_stop;\n+\n+\tfor (;;) {\n+\t\tdwWait = WaitForMultipleObjects(data->nr_listener_handles,\n+\t\t\t\t\t\tdata->hListener,\n+\t\t\t\t\t\tFALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) {\n+\t\t\tif (recv_rdcw_watch(data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (process_worktree_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) {\n+\t\t\tif (recv_rdcw_watch(data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (process_gitdir_events(state) == LISTENER_SHUTDOWN)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tif (start_rdcw_watch(data, data->watch_gitdir) == -1)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"could not read directory changes [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->error_code = -1;\n+\n+force_shutdown:\n+\t/*\n+\t * Tell the IPC thead pool to stop (which completes the await\n+\t * in the main thread (which will also signal this thread (if\n+\t * we are still alive))).\n+\t */\n+\tipc_server_stop_async(state->ipc_server_data);\n+\n+clean_shutdown:\n+\tcancel_rdcw_watch(data->watch_worktree);\n+\tcancel_rdcw_watch(data->watch_gitdir);\n }\n \n int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->watch_worktree = create_watch(state,\n+\t\t\t\t\t state->path_worktree_watch.buf);\n+\tif (!data->watch_worktree)\n+\t\tgoto failed;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->watch_gitdir = create_watch(state,\n+\t\t\t\t\t\t state->path_gitdir_watch.buf);\n+\t\tif (!data->watch_gitdir)\n+\t\t\tgoto failed;\n+\t}\n+\n+\tdata->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_listener_handles++;\n+\n+\tdata->hListener[LISTENER_HAVE_DATA_WORKTREE] =\n+\t\tdata->watch_worktree->hEvent;\n+\tdata->nr_listener_handles++;\n+\n+\tif (data->watch_gitdir) {\n+\t\tdata->hListener[LISTENER_HAVE_DATA_GITDIR] =\n+\t\t\tdata->watch_gitdir->hEvent;\n+\t\tdata->nr_listener_handles++;\n+\t}\n+\n+\tstate->backend_data = data;\n+\treturn 0;\n+\n+failed:\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n \treturn -1;\n }\n \n void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\tdestroy_watch(data->watch_worktree);\n+\tdestroy_watch(data->watch_gitdir);\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n-- \ngitgitgadget\n\n"},{"id":"428924","messageId":"3cf8f3cd7717dad69262ae884a89c1a6ca0ce5fc.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 23/34] t/helper/test-touch: add helper to touch a series of files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:33Z","receivedAt":"2021-07-01T14:48:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate `test-tool touch` that can update a series of files\nusing either a pattern given on the command line or a list\nof files read from stdin.\n\nThis will be used in a later commit to speed up p7519\nwhich needs to generate/update many thousands of files.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/helper/test-touch.c | 126 ++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 129 insertions(+)\n create mode 100644 t/helper/test-touch.c\n\ndiff --git a/Makefile b/Makefile\nindex a2a6e1f20f6..c07cfb75532 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -757,6 +757,7 @@ TEST_BUILTINS_OBJS += test-string-list.o\n TEST_BUILTINS_OBJS += test-submodule-config.o\n TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o\n TEST_BUILTINS_OBJS += test-subprocess.o\n+TEST_BUILTINS_OBJS += test-touch.o\n TEST_BUILTINS_OBJS += test-trace2.o\n TEST_BUILTINS_OBJS += test-urlmatch-normalization.o\n TEST_BUILTINS_OBJS += test-userdiff.o\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex af879e4a5d7..1ad8d5fbd82 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -73,6 +73,7 @@ static struct test_cmd cmds[] = {\n \t{ \"submodule-config\", cmd__submodule_config },\n \t{ \"submodule-nested-repo-config\", cmd__submodule_nested_repo_config },\n \t{ \"subprocess\", cmd__subprocess },\n+\t{ \"touch\", cmd__touch },\n \t{ \"trace2\", cmd__trace2 },\n \t{ \"userdiff\", cmd__userdiff },\n \t{ \"urlmatch-normalization\", cmd__urlmatch_normalization },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 6c5134b46d9..58fde0a62e5 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -63,6 +63,7 @@ int cmd__string_list(int argc, const char **argv);\n int cmd__submodule_config(int argc, const char **argv);\n int cmd__submodule_nested_repo_config(int argc, const char **argv);\n int cmd__subprocess(int argc, const char **argv);\n+int cmd__touch(int argc, const char **argv);\n int cmd__trace2(int argc, const char **argv);\n int cmd__userdiff(int argc, const char **argv);\n int cmd__urlmatch_normalization(int argc, const char **argv);\ndiff --git a/t/helper/test-touch.c b/t/helper/test-touch.c\nnew file mode 100644\nindex 00000000000..e9b3b754f1f\n--- /dev/null\n+++ b/t/helper/test-touch.c\n@@ -0,0 +1,126 @@\n+/*\n+ * test-touch.c: variation on /usr/bin/touch to speed up tests\n+ * with a large number of files (primarily on Windows where child\n+ * process are very, very expensive).\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+\n+char *seq_pattern;\n+int seq_start = 1;\n+int seq_count = 1;\n+\n+static int do_touch_one(const char *path)\n+{\n+\tint fd;\n+\n+\tif (!utime(path, NULL))\n+\t\treturn 0;\n+\n+\tif (errno != ENOENT) {\n+\t\twarning_errno(\"could not touch '%s'\", path);\n+\t\treturn 0;\n+\t}\n+\n+\tfd = open(path, O_RDWR | O_CREAT, 0644);\n+\tif (fd == -1) {\n+\t\twarning_errno(\"could not create '%s'\", path);\n+\t\treturn 0;\n+\t}\n+\tclose(fd);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Touch a series of files. We assume that any required subdirs\n+ * already exist. This function allows us to replace the following\n+ * test script fragment:\n+ *\n+ * for i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n+ *\n+ * with a single process:\n+ *\n+ * test-tool touch sequence --pattern=\"10000_files/%d\" --start=1 --count=10000\n+ *\n+ * which is much faster on Windows.\n+ */\n+static int do_sequence(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint k;\n+\n+\tfor (k = seq_start; k < seq_start + seq_count; k++) {\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, seq_pattern, k);\n+\n+\t\tif (do_touch_one(buf.buf))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Read a list of pathnames from stdin and touch them. We assume that\n+ * any required subdirs already exist.\n+ */\n+static int do_stdin(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\twhile (strbuf_getline(&buf, stdin) != EOF && buf.len)\n+\t\tif (do_touch_one(buf.buf))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+int cmd__touch(int argc, const char **argv)\n+{\n+\tconst char *touch_usage[] = {\n+\t\tN_(\"test-tool touch sequence \"),\n+\t\tN_(\"test-tool touch stdin\"),\n+\t\tNULL,\n+\t};\n+\n+\tstruct option touch_options[] = {\n+\t\tOPT_GROUP(N_(\"sequence\")),\n+\t\tOPT_STRING(0, \"pattern\", &seq_pattern, N_(\"format\"),\n+\t\t\t N_(\"sequence pathname pattern\")),\n+\t\tOPT_INTEGER(0, \"start\", &seq_start,\n+\t\t\t N_(\"sequence starting value\")),\n+\t\tOPT_INTEGER(0, \"count\", &seq_count,\n+\t\t\t N_(\"sequence count\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tconst char *subcmd;\n+\n+\tif (argc < 2)\n+\t\tusage_with_options(touch_usage, touch_options);\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(touch_usage, touch_options);\n+\n+\tsubcmd = argv[1];\n+\targv--;\n+\targc++;\n+\n+\targc = parse_options(argc, argv, NULL, touch_options, touch_usage, 0);\n+\n+\tif (!strcmp(subcmd, \"sequence\")) {\n+\t\tif (!seq_pattern || !strstr(seq_pattern, \"%d\"))\n+\t\t\tdie(\"invalid sequence pattern\");\n+\t\tif (seq_count < 1)\n+\t\t\tdie(\"invalid sequence count: %d\", seq_count);\n+\t\treturn !!do_sequence();\n+\t}\n+\n+\tif (!strcmp(subcmd, \"stdin\")) {\n+\t\treturn !!do_stdin();\n+\t}\n+\n+\tdie(\"Unhandled subcommand: '%s'\", subcmd);\n+}\n-- \ngitgitgadget\n\n"},{"id":"428925","messageId":"f1ef9656fc3adf079c8e40a74baeb5356bcf1586.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:34Z","receivedAt":"2021-07-01T14:48:14Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nChange p7519 to use a single \"test-tool touch\" command to update\nthe mtime on a series of (thousands) files instead of invoking\nthousands of commands to update a single file.\n\nThis is primarily for Windows where process creation is so\nvery slow and reduces the test run time by minutes.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 14 ++++++--------\n 1 file changed, 6 insertions(+), 8 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex 5eb5044a103..f74e6014a0a 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -119,10 +119,11 @@ test_expect_success \"one time repo setup\" '\n \tfi &&\n \n \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n-\tfor i in $(test_seq 1 10); do touch 10_files/$i; done &&\n-\tfor i in $(test_seq 1 100); do touch 100_files/$i; done &&\n-\tfor i in $(test_seq 1 1000); do touch 1000_files/$i; done &&\n-\tfor i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n+\ttest-tool touch sequence --pattern=\"10_files/%d\" --start=1 --count=10 &&\n+\ttest-tool touch sequence --pattern=\"100_files/%d\" --start=1 --count=100 &&\n+\ttest-tool touch sequence --pattern=\"1000_files/%d\" --start=1 --count=1000 &&\n+\ttest-tool touch sequence --pattern=\"10000_files/%d\" --start=1 --count=10000 &&\n+\n \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n \tgit commit -qm \"Add files\" &&\n \n@@ -200,15 +201,12 @@ test_fsmonitor_suite() {\n \t# Update the mtimes on upto 100k files to make status think\n \t# that they are dirty. For simplicity, omit any files with\n \t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n-\t# Then fully backslash-quote the paths to capture any\n-\t# whitespace so that they pass thru xargs properly.\n \t#\n \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n \t\tgit ls-files | \\\n \t\t\thead -100000 | \\\n \t\t\tgrep -v \\\" | \\\n-\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n-\t\t\txargs test-tool chmtime -300 &&\n+\t\t\ttest-tool touch stdin &&\n \t\tgit status\n \t'\n \n-- \ngitgitgadget\n\n"},{"id":"428926","messageId":"39df123143bd35fcce72d589fb8761eda34e212e.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 22/34] fsmonitor--daemon: implement handle_client callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:32Z","receivedAt":"2021-07-01T14:48:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to respond to IPC requests from client\nGit processes and respond with a list of modified pathnames\nrelative to the provided token.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 312 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 310 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex ea3a52d34e3..7a7fef681fe 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -7,6 +7,7 @@\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n+#include \"pkt-line.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon start []\"),\n@@ -355,6 +356,311 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+/*\n+ * Format an opaque token string to send to the client.\n+ */\n+static void with_lock__format_response_token(\n+\tstruct strbuf *response_token,\n+\tconst struct strbuf *response_token_id,\n+\tconst struct fsmonitor_batch *batch)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstrbuf_reset(response_token);\n+\tstrbuf_addf(response_token, \"builtin:%s:%\"PRIu64,\n+\t\t response_token_id->buf, batch->batch_seq_nr);\n+}\n+\n+/*\n+ * Parse an opaque token from the client.\n+ * Returns -1 on error.\n+ */\n+static int fsmonitor_parse_client_token(const char *buf_token,\n+\t\t\t\t\tstruct strbuf *requested_token_id,\n+\t\t\t\t\tuint64_t *seq_nr)\n+{\n+\tconst char *p;\n+\tchar *p_end;\n+\n+\tstrbuf_reset(requested_token_id);\n+\t*seq_nr = 0;\n+\n+\tif (!skip_prefix(buf_token, \"builtin:\", &p))\n+\t\treturn -1;\n+\n+\twhile (*p && *p != ':')\n+\t\tstrbuf_addch(requested_token_id, *p++);\n+\tif (!*p++)\n+\t\treturn -1;\n+\n+\t*seq_nr = (uint64_t)strtoumax(p, &p_end, 10);\n+\tif (*p_end)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal);\n+\n+static int do_handle_client(struct fsmonitor_daemon_state *state,\n+\t\t\t const char *command,\n+\t\t\t ipc_server_reply_cb *reply,\n+\t\t\t struct ipc_server_reply_data *reply_data)\n+{\n+\tstruct fsmonitor_token_data *token_data = NULL;\n+\tstruct strbuf response_token = STRBUF_INIT;\n+\tstruct strbuf requested_token_id = STRBUF_INIT;\n+\tstruct strbuf payload = STRBUF_INIT;\n+\tuint64_t requested_oldest_seq_nr = 0;\n+\tuint64_t total_response_len = 0;\n+\tconst char *p;\n+\tconst struct fsmonitor_batch *batch_head;\n+\tconst struct fsmonitor_batch *batch;\n+\tintmax_t count = 0, duplicates = 0;\n+\tkh_str_t *shown;\n+\tint hash_ret;\n+\tint do_trivial = 0;\n+\tint do_flush = 0;\n+\n+\t/*\n+\t * We expect `command` to be of the form:\n+\t *\n+\t * := quit NUL\n+\t * | flush NUL\n+\t * | NUL\n+\t * | NUL\n+\t */\n+\n+\tif (!strcmp(command, \"quit\")) {\n+\t\t/*\n+\t\t * A client has requested over the socket/pipe that the\n+\t\t * daemon shutdown.\n+\t\t *\n+\t\t * Tell the IPC thread pool to shutdown (which completes\n+\t\t * the await in the main thread (which can stop the\n+\t\t * fsmonitor listener thread)).\n+\t\t *\n+\t\t * There is no reply to the client.\n+\t\t */\n+\t\treturn SIMPLE_IPC_QUIT;\n+\n+\t} else if (!strcmp(command, \"flush\")) {\n+\t\t/*\n+\t\t * Flush all of our cached data and generate a new token\n+\t\t * just like if we lost sync with the filesystem.\n+\t\t *\n+\t\t * Then send a trivial response using the new token.\n+\t\t */\n+\t\tdo_flush = 1;\n+\t\tdo_trivial = 1;\n+\n+\t} else if (!skip_prefix(command, \"builtin:\", &p)) {\n+\t\t/* assume V1 timestamp or garbage */\n+\n+\t\tchar *p_end;\n+\n+\t\tstrtoumax(command, &p_end, 10);\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t ((*p_end) ?\n+\t\t\t\t \"fsmonitor: invalid command line '%s'\" :\n+\t\t\t\t \"fsmonitor: unsupported V1 protocol '%s'\"),\n+\t\t\t\t command);\n+\t\tdo_trivial = 1;\n+\n+\t} else {\n+\t\t/* We have \"builtin:*\" */\n+\t\tif (fsmonitor_parse_client_token(command, &requested_token_id,\n+\t\t\t\t\t\t &requested_oldest_seq_nr)) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"fsmonitor: invalid V2 protocol token '%s'\",\n+\t\t\t\t\t command);\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * We have a V2 valid token:\n+\t\t\t * \"builtin::\"\n+\t\t\t */\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (!state->current_token_data)\n+\t\tBUG(\"fsmonitor state does not have a current token\");\n+\n+\tif (do_flush)\n+\t\twith_lock__do_force_resync(state);\n+\n+\t/*\n+\t * We mark the current head of the batch list as \"pinned\" so\n+\t * that the listener thread will treat this item as read-only\n+\t * (and prevent any more paths from being added to it) from\n+\t * now on.\n+\t */\n+\ttoken_data = state->current_token_data;\n+\tbatch_head = token_data->batch_head;\n+\t((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL);\n+\n+\t/*\n+\t * FSMonitor Protocol V2 requires that we send a response header\n+\t * with a \"new current token\" and then all of the paths that changed\n+\t * since the \"requested token\". We send the seq_nr of the just-pinned\n+\t * head batch so that future requests from a client will be relative\n+\t * to it.\n+\t */\n+\twith_lock__format_response_token(&response_token,\n+\t\t\t\t\t &token_data->token_id, batch_head);\n+\n+\treply(reply_data, response_token.buf, response_token.len + 1);\n+\ttotal_response_len += response_token.len + 1;\n+\n+\ttrace2_data_string(\"fsmonitor\", the_repository, \"response/token\",\n+\t\t\t response_token.buf);\n+\ttrace_printf_key(&trace_fsmonitor, \"response token: %s\",\n+\t\t\t response_token.buf);\n+\n+\tif (!do_trivial) {\n+\t\tif (strcmp(requested_token_id.buf, token_data->token_id.buf)) {\n+\t\t\t/*\n+\t\t\t * The client last spoke to a different daemon\n+\t\t\t * instance -OR- the daemon had to resync with\n+\t\t\t * the filesystem (and lost events), so reject.\n+\t\t\t */\n+\t\t\ttrace2_data_string(\"fsmonitor\", the_repository,\n+\t\t\t\t\t \"response/token\", \"different\");\n+\t\t\tdo_trivial = 1;\n+\n+\t\t} else if (requested_oldest_seq_nr <\n+\t\t\t token_data->batch_tail->batch_seq_nr) {\n+\t\t\t/*\n+\t\t\t * The client wants older events than we have for\n+\t\t\t * this token_id. This means that the end of our\n+\t\t\t * batch list was truncated and we cannot give the\n+\t\t\t * client a complete snapshot relative to their\n+\t\t\t * request.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"client requested truncated data\");\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n+\tif (do_trivial) {\n+\t\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t\treply(reply_data, \"/\", 2);\n+\n+\t\ttrace2_data_intmax(\"fsmonitor\", the_repository,\n+\t\t\t\t \"response/trivial\", 1);\n+\n+\t\tstrbuf_release(&response_token);\n+\t\tstrbuf_release(&requested_token_id);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * We're going to hold onto a pointer to the current\n+\t * token-data while we walk the list of batches of files.\n+\t * During this time, we will NOT be under the lock.\n+\t * So we ref-count it.\n+\t *\n+\t * This allows the listener thread to continue prepending\n+\t * new batches of items to the token-data (which we'll ignore).\n+\t *\n+\t * AND it allows the listener thread to do a token-reset\n+\t * (and install a new `current_token_data`).\n+\t */\n+\ttoken_data->client_ref_count++;\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\t/*\n+\t * The client request is relative to the token that they sent,\n+\t * so walk the batch list backwards from the current head back\n+\t * to the batch (sequence number) they named.\n+\t *\n+\t * We use khash to de-dup the list of pathnames.\n+\t *\n+\t * NEEDSWORK: each batch contains a list of interned strings,\n+\t * so we only need to do pointer comparisons here to build the\n+\t * hash table. Currently, we're still comparing the string\n+\t * values.\n+\t */\n+\tshown = kh_init_str();\n+\tfor (batch = batch_head;\n+\t batch && batch->batch_seq_nr > requested_oldest_seq_nr;\n+\t batch = batch->next) {\n+\t\tsize_t k;\n+\n+\t\tfor (k = 0; k < batch->nr; k++) {\n+\t\t\tconst char *s = batch->interned_paths[k];\n+\t\t\tsize_t s_len;\n+\n+\t\t\tif (kh_get_str(shown, s) != kh_end(shown))\n+\t\t\t\tduplicates++;\n+\t\t\telse {\n+\t\t\t\tkh_put_str(shown, s, &hash_ret);\n+\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"send[%\"PRIuMAX\"]: %s\",\n+\t\t\t\t\t\t count, s);\n+\n+\t\t\t\t/* Each path gets written with a trailing NUL */\n+\t\t\t\ts_len = strlen(s) + 1;\n+\n+\t\t\t\tif (payload.len + s_len >=\n+\t\t\t\t LARGE_PACKET_DATA_MAX) {\n+\t\t\t\t\treply(reply_data, payload.buf,\n+\t\t\t\t\t payload.len);\n+\t\t\t\t\ttotal_response_len += payload.len;\n+\t\t\t\t\tstrbuf_reset(&payload);\n+\t\t\t\t}\n+\n+\t\t\t\tstrbuf_add(&payload, s, s_len);\n+\t\t\t\tcount++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (payload.len) {\n+\t\treply(reply_data, payload.buf, payload.len);\n+\t\ttotal_response_len += payload.len;\n+\t}\n+\n+\tkh_release_str(shown);\n+\n+\tpthread_mutex_lock(&state->main_lock);\n+\n+\tif (token_data->client_ref_count > 0)\n+\t\ttoken_data->client_ref_count--;\n+\n+\tif (token_data->client_ref_count == 0) {\n+\t\tif (token_data != state->current_token_data) {\n+\t\t\t/*\n+\t\t\t * The listener thread did a token-reset while we were\n+\t\t\t * walking the batch list. Therefore, this token is\n+\t\t\t * stale and can be discarded completely. If we are\n+\t\t\t * the last reader thread using this token, we own\n+\t\t\t * that work.\n+\t\t\t */\n+\t\t\tfsmonitor_free_token_data(token_data);\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&state->main_lock);\n+\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/length\", total_response_len);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/files\", count);\n+\ttrace2_data_intmax(\"fsmonitor\", the_repository, \"response/count/duplicates\", duplicates);\n+\n+\tstrbuf_release(&response_token);\n+\tstrbuf_release(&requested_token_id);\n+\tstrbuf_release(&payload);\n+\n+\treturn 0;\n+}\n+\n static ipc_server_application_cb handle_client;\n \n static int handle_client(void *data,\n@@ -362,7 +668,7 @@ static int handle_client(void *data,\n \t\t\t ipc_server_reply_cb *reply,\n \t\t\t struct ipc_server_reply_data *reply_data)\n {\n-\t/* struct fsmonitor_daemon_state *state = data; */\n+\tstruct fsmonitor_daemon_state *state = data;\n \tint result;\n \n \t/*\n@@ -373,10 +679,12 @@ static int handle_client(void *data,\n \tif (command_len != strlen(command))\n \t\tBUG(\"FSMonitor assumes text messages\");\n \n+\ttrace_printf_key(&trace_fsmonitor, \"requested token: %s\", command);\n+\n \ttrace2_region_enter(\"fsmonitor\", \"handle_client\", the_repository);\n \ttrace2_data_string(\"fsmonitor\", the_repository, \"request\", command);\n \n-\tresult = 0; /* TODO Do something here. */\n+\tresult = do_handle_client(state, command, reply, reply_data);\n \n \ttrace2_region_leave(\"fsmonitor\", \"handle_client\", the_repository);\n \n-- \ngitgitgadget\n\n"},{"id":"428927","messageId":"5d12b5d808a4bcdff87fe0c3a30f415081003459.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 21/34] fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:31Z","receivedAt":"2021-07-01T14:48:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImplement file system event listener on MacOS using FSEvent,\nCoreFoundation, and CoreServices.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-macos.c | 381 +++++++++++++++++++\n 1 file changed, 381 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\nindex bec5130d9e1..02f89de216e 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n@@ -97,20 +97,401 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n #include \"cache.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-fs-listen.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsmonitor_daemon_backend_data\n+{\n+\tCFStringRef cfsr_worktree_path;\n+\tCFStringRef cfsr_gitdir_path;\n+\n+\tCFArrayRef cfar_paths_to_watch;\n+\tint nr_paths_watching;\n+\n+\tFSEventStreamRef stream;\n+\n+\tCFRunLoopRef rl;\n+\n+\tenum shutdown_style {\n+\t\tSHUTDOWN_EVENT = 0,\n+\t\tFORCE_SHUTDOWN,\n+\t\tFORCE_ERROR_STOP,\n+\t} shutdown_style;\n+\n+\tunsigned int stream_scheduled:1;\n+\tunsigned int stream_started:1;\n+};\n+\n+static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tif (flag & kFSEventStreamEventFlagMustScanSubDirs)\n+\t\tstrbuf_addstr(&msg, \"MustScanSubDirs|\");\n+\tif (flag & kFSEventStreamEventFlagUserDropped)\n+\t\tstrbuf_addstr(&msg, \"UserDropped|\");\n+\tif (flag & kFSEventStreamEventFlagKernelDropped)\n+\t\tstrbuf_addstr(&msg, \"KernelDropped|\");\n+\tif (flag & kFSEventStreamEventFlagEventIdsWrapped)\n+\t\tstrbuf_addstr(&msg, \"EventIdsWrapped|\");\n+\tif (flag & kFSEventStreamEventFlagHistoryDone)\n+\t\tstrbuf_addstr(&msg, \"HistoryDone|\");\n+\tif (flag & kFSEventStreamEventFlagRootChanged)\n+\t\tstrbuf_addstr(&msg, \"RootChanged|\");\n+\tif (flag & kFSEventStreamEventFlagMount)\n+\t\tstrbuf_addstr(&msg, \"Mount|\");\n+\tif (flag & kFSEventStreamEventFlagUnmount)\n+\t\tstrbuf_addstr(&msg, \"Unmount|\");\n+\tif (flag & kFSEventStreamEventFlagItemChangeOwner)\n+\t\tstrbuf_addstr(&msg, \"ItemChangeOwner|\");\n+\tif (flag & kFSEventStreamEventFlagItemCreated)\n+\t\tstrbuf_addstr(&msg, \"ItemCreated|\");\n+\tif (flag & kFSEventStreamEventFlagItemFinderInfoMod)\n+\t\tstrbuf_addstr(&msg, \"ItemFinderInfoMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemInodeMetaMod)\n+\t\tstrbuf_addstr(&msg, \"ItemInodeMetaMod|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsDir)\n+\t\tstrbuf_addstr(&msg, \"ItemIsDir|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsFile)\n+\t\tstrbuf_addstr(&msg, \"ItemIsFile|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsLastHardlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsLastHardlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemIsSymlink)\n+\t\tstrbuf_addstr(&msg, \"ItemIsSymlink|\");\n+\tif (flag & kFSEventStreamEventFlagItemModified)\n+\t\tstrbuf_addstr(&msg, \"ItemModified|\");\n+\tif (flag & kFSEventStreamEventFlagItemRemoved)\n+\t\tstrbuf_addstr(&msg, \"ItemRemoved|\");\n+\tif (flag & kFSEventStreamEventFlagItemRenamed)\n+\t\tstrbuf_addstr(&msg, \"ItemRenamed|\");\n+\tif (flag & kFSEventStreamEventFlagItemXattrMod)\n+\t\tstrbuf_addstr(&msg, \"ItemXattrMod|\");\n+\tif (flag & kFSEventStreamEventFlagOwnEvent)\n+\t\tstrbuf_addstr(&msg, \"OwnEvent|\");\n+\tif (flag & kFSEventStreamEventFlagItemCloned)\n+\t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\t\t\t path, flag, msg.buf);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRemoved);\n+}\n+\n+static int ef_is_root_renamed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n+\t\tef & kFSEventStreamEventFlagItemRenamed);\n+}\n+\n+static int ef_is_dropped(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagKernelDropped ||\n+\t\tef & kFSEventStreamEventFlagUserDropped);\n+}\n+\n+static void fsevent_callback(ConstFSEventStreamRef streamRef,\n+\t\t\t void *ctx,\n+\t\t\t size_t num_of_events,\n+\t\t\t void *event_paths,\n+\t\t\t const FSEventStreamEventFlags event_flags[],\n+\t\t\t const FSEventStreamEventId event_ids[])\n+{\n+\tstruct fsmonitor_daemon_state *state = ctx;\n+\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tchar **paths = (char **)event_paths;\n+\tstruct fsmonitor_batch *batch = NULL;\n+\tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n+\tconst char *path_k;\n+\tconst char *slash;\n+\tint k;\n+\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t/*\n+\t * Build a list of all filesystem changes into a private/local\n+\t * list and without holding any locks.\n+\t */\n+\tfor (k = 0; k < num_of_events; k++) {\n+\t\t/*\n+\t\t * On Mac, we receive an array of absolute paths.\n+\t\t */\n+\t\tpath_k = paths[k];\n+\n+\t\t/*\n+\t\t * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.\n+\t\t * Please don't log them to Trace2.\n+\t\t *\n+\t\t * trace_printf_key(&trace_fsmonitor, \"Path: '%s'\", path_k);\n+\t\t */\n+\n+\t\t/*\n+\t\t * If event[k] is marked as dropped, we assume that we have\n+\t\t * lost sync with the filesystem and should flush our cached\n+\t\t * data. We need to:\n+\t\t *\n+\t\t * [1] Abort/wake any client threads waiting for a cookie and\n+\t\t * flush the cached state data (the current token), and\n+\t\t * create a new token.\n+\t\t *\n+\t\t * [2] Discard the batch that we were locally building (since\n+\t\t * they are conceptually relative to the just flushed\n+\t\t * token).\n+\t\t */\n+\t\tif (ef_is_dropped(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * see also kFSEventStreamEventFlagMustScanSubDirs\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"event: dropped\");\n+\n+\t\t\tfsmonitor_force_resync(state);\n+\t\t\tfsmonitor_batch__pop(batch);\n+\t\t\tstring_list_clear(&cookie_list, 0);\n+\n+\t\t\t/*\n+\t\t\t * We assume that any events that we received\n+\t\t\t * in this callback after this dropped event\n+\t\t\t * may still be valid, so we continue rather\n+\t\t\t * than break. (And just in case there is a\n+\t\t\t * delete of \".git\" hiding in there.)\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n+\n+\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\t\t\t/* special case cookie files within .git or gitdir */\n+\n+\t\t\t/* Use just the filename of the cookie file. */\n+\t\t\tslash = find_last_dir_sep(path_k);\n+\t\t\tstring_list_append(&cookie_list,\n+\t\t\t\t\t slash ? slash + 1 : path_k);\n+\t\t\tbreak;\n+\n+\t\tcase IS_INSIDE_DOT_GIT:\n+\t\tcase IS_INSIDE_GITDIR:\n+\t\t\t/* ignore all other paths inside of .git or gitdir */\n+\t\t\tbreak;\n+\n+\t\tcase IS_DOT_GIT:\n+\t\tcase IS_GITDIR:\n+\t\t\t/*\n+\t\t\t * If .git directory is deleted or renamed away,\n+\t\t\t * we have to quit.\n+\t\t\t */\n+\t\t\tif (ef_is_root_delete(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir removed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tif (ef_is_root_renamed(event_flags[k])) {\n+\t\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t\t \"event: gitdir renamed\");\n+\t\t\t\tgoto force_shutdown;\n+\t\t\t}\n+\t\t\tbreak;\n+\n+\t\tcase IS_WORKDIR_PATH:\n+\t\t\t/* try to queue normal pathnames */\n+\n+\t\t\tif (trace_pass_fl(&trace_fsmonitor))\n+\t\t\t\tlog_flags_set(path_k, event_flags[k]);\n+\n+\t\t\t/*\n+\t\t\t * Because of the implicit \"binning\" (the\n+\t\t\t * kernel calls us at a given frequency) and\n+\t\t\t * de-duping (the kernel is free to combine\n+\t\t\t * multiple events for a given pathname), an\n+\t\t\t * individual fsevent could be marked as both\n+\t\t\t * a file and directory. Add it to the queue\n+\t\t\t * with both spellings so that the client will\n+\t\t\t * know how much to invalidate/refresh.\n+\t\t\t */\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsFile) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t}\n+\n+\t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n+\t\t\t\tconst char *rel = path_k +\n+\t\t\t\t\tstate->path_worktree_watch.len + 1;\n+\n+\t\t\t\tstrbuf_reset(&tmp);\n+\t\t\t\tstrbuf_addstr(&tmp, rel);\n+\t\t\t\tstrbuf_addch(&tmp, '/');\n+\n+\t\t\t\tif (!batch)\n+\t\t\t\t\tbatch = fsmonitor_batch__new();\n+\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\n+\t\tcase IS_OUTSIDE_CONE:\n+\t\tdefault:\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignoring '%s'\", path_k);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfsmonitor_publish(state, batch, &cookie_list);\n+\tstring_list_clear(&cookie_list, 0);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+\n+force_shutdown:\n+\tfsmonitor_batch__pop(batch);\n+\tstring_list_clear(&cookie_list, 0);\n+\n+\tdata->shutdown_style = FORCE_SHUTDOWN;\n+\tCFRunLoopStop(data->rl);\n+\tstrbuf_release(&tmp);\n+\treturn;\n+}\n+\n+/*\n+ * NEEDSWORK: Investigate the proper value for the `latency` argument\n+ * in the call to `FSEventStreamCreate()`. I'm not sure that this\n+ * needs to be a config setting or just something that we tune after\n+ * some testing.\n+ *\n+ * With a latency of 0.1, I was seeing lots of dropped events during\n+ * the \"touch 100000\" files test within t/perf/p7519, but with a\n+ * latency of 0.001 I did not see any dropped events. So the\n+ * \"correct\" value may be somewhere in between.\n+ *\n+ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate\n+ */\n \n int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n+\tFSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer |\n+\t\tkFSEventStreamCreateFlagWatchRoot |\n+\t\tkFSEventStreamCreateFlagFileEvents;\n+\tFSEventStreamContext ctx = {\n+\t\t0,\n+\t\tstate,\n+\t\tNULL,\n+\t\tNULL,\n+\t\tNULL\n+\t};\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\tconst void *dir_array[2];\n+\n+\tCALLOC_ARRAY(data, 1);\n+\tstate->backend_data = data;\n+\n+\tdata->cfsr_worktree_path = CFStringCreateWithCString(\n+\t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n+\tdir_array[data->nr_paths_watching++] = data->cfsr_worktree_path;\n+\n+\tif (state->nr_paths_watching > 1) {\n+\t\tdata->cfsr_gitdir_path = CFStringCreateWithCString(\n+\t\t\tNULL, state->path_gitdir_watch.buf,\n+\t\t\tkCFStringEncodingUTF8);\n+\t\tdir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path;\n+\t}\n+\n+\tdata->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array,\n+\t\t\t\t\t\t data->nr_paths_watching,\n+\t\t\t\t\t\t NULL);\n+\tdata->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx,\n+\t\t\t\t\t data->cfar_paths_to_watch,\n+\t\t\t\t\t kFSEventStreamEventIdSinceNow,\n+\t\t\t\t\t 0.001, flags);\n+\tif (data->stream == NULL)\n+\t\tgoto failed;\n+\n+\t/*\n+\t * `data->rl` needs to be set inside the listener thread.\n+\t */\n+\n+\treturn 0;\n+\n+failed:\n+\terror(\"Unable to create FSEventStream.\");\n+\n+\tFREE_AND_NULL(state->backend_data);\n \treturn -1;\n }\n \n void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tif (!state || !state->backend_data)\n+\t\treturn;\n+\n+\tdata = state->backend_data;\n+\n+\tif (data->stream) {\n+\t\tif (data->stream_started)\n+\t\t\tFSEventStreamStop(data->stream);\n+\t\tif (data->stream_scheduled)\n+\t\t\tFSEventStreamInvalidate(data->stream);\n+\t\tFSEventStreamRelease(data->stream);\n+\t}\n+\n+\tFREE_AND_NULL(state->backend_data);\n }\n \n void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\tdata->shutdown_style = SHUTDOWN_EVENT;\n+\n+\tCFRunLoopStop(data->rl);\n }\n \n void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n {\n+\tstruct fsmonitor_daemon_backend_data *data;\n+\n+\tdata = state->backend_data;\n+\n+\tdata->rl = CFRunLoopGetCurrent();\n+\n+\tFSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode);\n+\tdata->stream_scheduled = 1;\n+\n+\tif (!FSEventStreamStart(data->stream)) {\n+\t\terror(\"Failed to start the FSEventStream\");\n+\t\tgoto force_error_stop_without_loop;\n+\t}\n+\tdata->stream_started = 1;\n+\n+\tCFRunLoopRun();\n+\n+\tswitch (data->shutdown_style) {\n+\tcase FORCE_ERROR_STOP:\n+\t\tstate->error_code = -1;\n+\t\t/* fall thru */\n+\tcase FORCE_SHUTDOWN:\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\t/* fall thru */\n+\tcase SHUTDOWN_EVENT:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn;\n+\n+force_error_stop_without_loop:\n+\tstate->error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+\treturn;\n }\n-- \ngitgitgadget\n\n"},{"id":"428928","messageId":"a83485fb10f57326a725579f329b73ebf9240ac6.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 25/34] t/perf: avoid copying builtin fsmonitor files into test repo","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:35Z","receivedAt":"2021-07-01T14:48:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nDo not try to copy a fsmonitor--daemon socket from the current\ndevelopment directory into the test trash directory.\n\nWhen we run the perf suite without an explicit source repo set,\nwe copy of the current $GIT_DIR into the test trash directory.\nUnix domain sockets cannot be copied in that manner, so the test\nsetup fails.\n\nAdditionally, omit any other fsmonitor--daemon temp files inside\nthe $GIT_DIR directory.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/perf-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 601d9f67ddb..3b97e3fc0f2 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -74,7 +74,7 @@ test_perf_copy_repo_contents () {\n \tfor stuff in \"$1\"/*\n \tdo\n \t\tcase \"$stuff\" in\n-\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees)\n+\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)\n \t\t\t;;\n \t\t*)\n \t\t\tcp -R \"$stuff\" \"$repo/.git/\" || exit 1\n-- \ngitgitgadget\n\n"},{"id":"428929","messageId":"555caca2216dd3e459c118d76b46eb983a58e051.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 29/34] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:39Z","receivedAt":"2021-07-01T14:48:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon client threads to create a cookie file\ninside the .git directory and then wait until FS events for the\ncookie are observed by the FS listener thread.\n\nThis helps address the racy nature of file system events by\nblocking the client response until the kernel has drained any\nevent backlog.\n\nThis is especially important on MacOS where kernel events are\nonly issued with a limited frequency. See the `latency` argument\nof `FSeventStreamCreate()`. The kernel only signals every `latency`\nseconds, but does not guarantee that the kernel queue is completely\ndrained, so we may have to wait more than one interval. If we\nincrease the frequency, the system is more likely to drop events.\nWe avoid these issues by having each client thread create a unique\ncookie file and then wait until it is seen in the event stream.\n\nCo-authored-by: Kevin Willford \nCo-authored-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++-\n fsmonitor--daemon.h | 5 +\n 2 files changed, 232 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 8249420ba18..25f18f2726b 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -94,6 +94,149 @@ static int do_as_client__status(void)\n \t}\n }\n \n+enum fsmonitor_cookie_item_result {\n+\tFCIR_ERROR = -1, /* could not create cookie file ? */\n+\tFCIR_INIT = 0,\n+\tFCIR_SEEN,\n+\tFCIR_ABORT,\n+};\n+\n+struct fsmonitor_cookie_item {\n+\tstruct hashmap_entry entry;\n+\tconst char *name;\n+\tenum fsmonitor_cookie_item_result result;\n+};\n+\n+static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n+\t\t const struct hashmap_entry *he2, const void *keydata)\n+{\n+\tconst struct fsmonitor_cookie_item *a =\n+\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n+\tconst struct fsmonitor_cookie_item *b =\n+\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n+\n+\treturn strcmp(a->name, keydata ? keydata : b->name);\n+}\n+\n+static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie(\n+\tstruct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint fd;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tstruct strbuf cookie_pathname = STRBUF_INIT;\n+\tstruct strbuf cookie_filename = STRBUF_INIT;\n+\tenum fsmonitor_cookie_item_result result;\n+\tint my_cookie_seq;\n+\n+\tCALLOC_ARRAY(cookie, 1);\n+\n+\tmy_cookie_seq = state->cookie_seq++;\n+\n+\tstrbuf_addf(&cookie_filename, \"%i-%i\", getpid(), my_cookie_seq);\n+\n+\tstrbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix);\n+\tstrbuf_addbuf(&cookie_pathname, &cookie_filename);\n+\n+\tcookie->name = strbuf_detach(&cookie_filename, NULL);\n+\tcookie->result = FCIR_INIT;\n+\thashmap_entry_init(&cookie->entry, strhash(cookie->name));\n+\n+\thashmap_add(&state->cookies, &cookie->entry);\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"cookie-wait: '%s' '%s'\",\n+\t\t\t cookie->name, cookie_pathname.buf);\n+\n+\t/*\n+\t * Create the cookie file on disk and then wait for a notification\n+\t * that the listener thread has seen it.\n+\t */\n+\tfd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600);\n+\tif (fd >= 0) {\n+\t\tclose(fd);\n+\t\tunlink(cookie_pathname.buf);\n+\n+\t\t/*\n+\t\t * NEEDSWORK: This is an infinite wait (well, unless another\n+\t\t * thread sends us an abort). I'd like to change this to\n+\t\t * use `pthread_cond_timedwait()` and return an error/timeout\n+\t\t * and let the caller do the trivial response thing.\n+\t\t */\n+\t\twhile (cookie->result == FCIR_INIT)\n+\t\t\tpthread_cond_wait(&state->cookies_cond,\n+\t\t\t\t\t &state->main_lock);\n+\t} else {\n+\t\terror_errno(_(\"could not create fsmonitor cookie '%s'\"),\n+\t\t\t cookie->name);\n+\n+\t\tcookie->result = FCIR_ERROR;\n+\t}\n+\n+\thashmap_remove(&state->cookies, &cookie->entry, NULL);\n+\n+\tresult = cookie->result;\n+\n+\tfree((char*)cookie->name);\n+\tfree(cookie);\n+\tstrbuf_release(&cookie_pathname);\n+\n+\treturn result;\n+}\n+\n+/*\n+ * Mark these cookies as _SEEN and wake up the corresponding client threads.\n+ */\n+static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state,\n+\t\t\t\t\t const struct string_list *cookie_names)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tint k;\n+\tint nr_seen = 0;\n+\n+\tfor (k = 0; k < cookie_names->nr; k++) {\n+\t\tstruct fsmonitor_cookie_item key;\n+\t\tstruct fsmonitor_cookie_item *cookie;\n+\n+\t\tkey.name = cookie_names->items[k].string;\n+\t\thashmap_entry_init(&key.entry, strhash(key.name));\n+\n+\t\tcookie = hashmap_get_entry(&state->cookies, &key, entry, NULL);\n+\t\tif (cookie) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-seen: '%s'\",\n+\t\t\t\t\t cookie->name);\n+\t\t\tcookie->result = FCIR_SEEN;\n+\t\t\tnr_seen++;\n+\t\t}\n+\t}\n+\n+\tif (nr_seen)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n+/*\n+ * Set _ABORT on all pending cookies and wake up all client threads.\n+ */\n+static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tstruct hashmap_iter iter;\n+\tstruct fsmonitor_cookie_item *cookie;\n+\tint nr_aborted = 0;\n+\n+\thashmap_for_each_entry(&state->cookies, &iter, cookie, entry) {\n+\t\ttrace_printf_key(&trace_fsmonitor, \"cookie-abort: '%s'\",\n+\t\t\t\t cookie->name);\n+\t\tcookie->result = FCIR_ABORT;\n+\t\tnr_aborted++;\n+\t}\n+\n+\tif (nr_aborted)\n+\t\tpthread_cond_broadcast(&state->cookies_cond);\n+}\n+\n /*\n * Requests to and from a FSMonitor Protocol V2 provider use an opaque\n * \"token\" as a virtual timestamp. Clients can request a summary of all\n@@ -397,6 +540,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n * We should create a new token and start fresh (as if we just\n * booted up).\n *\n+ * [2] Some of those lost events may have been for cookie files. We\n+ * should assume the worst and abort them rather letting them starve.\n+ *\n * If there are no concurrent threads readering the current token data\n * series, we can free it now. Otherwise, let the last reader free\n * it.\n@@ -418,6 +564,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)\n \tstate->current_token_data = new_one;\n \n \tfsmonitor_free_token_data(free_me);\n+\n+\twith_lock__abort_all_cookies(state);\n }\n \n void fsmonitor_force_resync(struct fsmonitor_daemon_state *state)\n@@ -492,6 +640,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tint hash_ret;\n \tint do_trivial = 0;\n \tint do_flush = 0;\n+\tint do_cookie = 0;\n+\tenum fsmonitor_cookie_item_result cookie_result;\n \n \t/*\n \t * We expect `command` to be of the form:\n@@ -552,6 +702,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * We have a V2 valid token:\n \t\t\t * \"builtin::\"\n \t\t\t */\n+\t\t\tdo_cookie = 1;\n \t\t}\n \t}\n \n@@ -560,6 +711,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \tif (!state->current_token_data)\n \t\tBUG(\"fsmonitor state does not have a current token\");\n \n+\t/*\n+\t * Write a cookie file inside the directory being watched in\n+\t * an effort to flush out existing filesystem events that we\n+\t * actually care about. Suspend this client thread until we\n+\t * see the filesystem events for this cookie file.\n+\t *\n+\t * Creating the cookie lets us guarantee that our FS listener\n+\t * thread has drained the kernel queue and we are caught up\n+\t * with the kernel.\n+\t *\n+\t * If we cannot create the cookie (or otherwise guarantee that\n+\t * we are caught up), we send a trivial response. We have to\n+\t * assume that there might be some very, very recent activity\n+\t * on the FS still in flight.\n+\t */\n+\tif (do_cookie) {\n+\t\tcookie_result = with_lock__wait_for_cookie(state);\n+\t\tif (cookie_result != FCIR_SEEN) {\n+\t\t\terror(_(\"fsmonitor: cookie_result '%d' != SEEN\"),\n+\t\t\t cookie_result);\n+\t\t\tdo_trivial = 1;\n+\t\t}\n+\t}\n+\n \tif (do_flush)\n \t\twith_lock__do_force_resync(state);\n \n@@ -771,7 +946,9 @@ static int handle_client(void *data,\n \treturn result;\n }\n \n-#define FSMONITOR_COOKIE_PREFIX \".fsmonitor-daemon-\"\n+#define FSMONITOR_DIR \"fsmonitor--daemon\"\n+#define FSMONITOR_COOKIE_DIR \"cookies\"\n+#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR \"/\" FSMONITOR_COOKIE_DIR \"/\")\n \n enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative(\n \tconst char *rel)\n@@ -924,6 +1101,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \t\t}\n \t}\n \n+\tif (cookie_names->nr)\n+\t\twith_lock__mark_cookies_seen(state, cookie_names);\n+\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n@@ -1013,7 +1193,9 @@ static int fsmonitor_run_daemon(void)\n \n \tmemset(&state, 0, sizeof(state));\n \n+\thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n+\tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n@@ -1038,6 +1220,44 @@ static int fsmonitor_run_daemon(void)\n \t\tstate.nr_paths_watching = 2;\n \t}\n \n+\t/*\n+\t * We will write filesystem syncing cookie files into\n+\t * ///-.\n+\t *\n+\t * The extra layers of subdirectories here keep us from\n+\t * changing the mtime on \".git/\" or \".git/foo/\" when we create\n+\t * or delete cookie files.\n+\t *\n+\t * There have been problems with some IDEs that do a\n+\t * non-recursive watch of the \".git/\" directory and run a\n+\t * series of commands any time something happens.\n+\t *\n+\t * For example, if we place our cookie files directly in\n+\t * \".git/\" or \".git/foo/\" then a `git status` (or similar\n+\t * command) from the IDE will cause a cookie file to be\n+\t * created in one of those dirs. This causes the mtime of\n+\t * those dirs to change. This triggers the IDE's watch\n+\t * notification. This triggers the IDE to run those commands\n+\t * again. And the process repeats and the machine never goes\n+\t * idle.\n+\t *\n+\t * Adding the extra layers of subdirectories prevents the\n+\t * mtime of \".git/\" and \".git/foo\" from changing when a\n+\t * cookie file is created.\n+\t */\n+\tstrbuf_init(&state.path_cookie_prefix, 0);\n+\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n+\tmkdir(state.path_cookie_prefix.buf, 0777);\n+\n+\tstrbuf_addch(&state.path_cookie_prefix, '/');\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1050,6 +1270,7 @@ static int fsmonitor_run_daemon(void)\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n+\tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsmonitor_fs_listen__dtor(&state);\n \n@@ -1057,6 +1278,11 @@ done:\n \n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n+\tstrbuf_release(&state.path_cookie_prefix);\n+\n+\t/*\n+\t * NEEDSWORK: Consider \"rm -rf /\"\n+\t */\n \n \treturn err;\n }\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 89a9ef20b24..e9fc099bae9 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -45,6 +45,11 @@ struct fsmonitor_daemon_state {\n \n \tstruct fsmonitor_token_data *current_token_data;\n \n+\tstruct strbuf path_cookie_prefix;\n+\tpthread_cond_t cookies_cond;\n+\tint cookie_seq;\n+\tstruct hashmap cookies;\n+\n \tint error_code;\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n-- \ngitgitgadget\n\n"},{"id":"428930","messageId":"99279c0ebd2b36fe276301f018ca4e24e4cdff36.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 27/34] t7527: create test for fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:37Z","receivedAt":"2021-07-01T14:48:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 497 +++++++++++++++++++++++++++++++++++\n 1 file changed, 497 insertions(+)\n create mode 100755 t/t7527-builtin-fsmonitor.sh\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..58b56dc9940\n--- /dev/null\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -0,0 +1,497 @@\n+#!/bin/sh\n+\n+test_description='built-in file system watcher'\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+stop_daemon_delete_repo () {\n+\tr=$1\n+\tgit -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null\n+\trm -rf $1\n+\treturn 0\n+}\n+\n+start_daemon () {\n+\tcase \"$#\" in\n+\t\t1) r=\"-C $1\";;\n+\t\t*) r=\"\";\n+\tesac\n+\n+\tgit $r fsmonitor--daemon start || return $?\n+\tgit $r fsmonitor--daemon status || return $?\n+\n+\treturn 0\n+}\n+\n+test_expect_success 'explicit daemon start and stop' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_explicit\" &&\n+\n+\tgit init test_explicit &&\n+\tstart_daemon test_explicit &&\n+\n+\tgit -C test_explicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_explicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon start' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit\" &&\n+\n+\tgit init test_implicit &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status &&\n+\n+\t# query will implicitly start the daemon.\n+\t#\n+\t# for test-script simplicity, we send a V1 timestamp rather than\n+\t# a V2 token. either way, the daemon response to any query contains\n+\t# a new V2 token. (the daemon may complain that we sent a V1 request,\n+\t# but this test case is only concerned with whether the daemon was\n+\t# implicitly started.)\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace\" \\\n+\t\ttest-tool -C test_implicit fsmonitor-client query --token 0 >actual &&\n+\tnul_to_q actual.filtered &&\n+\tgrep \"builtin:\" actual.filtered &&\n+\n+\t# confirm that a daemon was started in the background.\n+\t#\n+\t# since the mechanism for starting the background daemon is platform\n+\t# dependent, just confirm that the foreground command received a\n+\t# response from the daemon.\n+\n+\tgrep :\\\"query/response-length\\\" .git/trace &&\n+\n+\tgit -C test_implicit fsmonitor--daemon status &&\n+\tgit -C test_implicit fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_implicit fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (delete .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n+\n+\tgit init test_implicit_1 &&\n+\n+\tstart_daemon test_implicit_1 &&\n+\n+\t# deleting the .git directory will implicitly stop the daemon.\n+\trm -rf test_implicit_1/.git &&\n+\n+\t# [1] Create an empty .git directory so that the following Git\n+\t# command will stay relative to the `-C` directory.\n+\t#\n+\t# Without this, the Git command will override the requested\n+\t# -C argument and crawl out to the containing Git source tree.\n+\t# This would make the test result dependent upon whether we\n+\t# were using fsmonitor on our development worktree.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_1/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'implicit daemon stop (rename .git)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_2\" &&\n+\n+\tgit init test_implicit_2 &&\n+\n+\tstart_daemon test_implicit_2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\tmv test_implicit_2/.git test_implicit_2/.xxx &&\n+\n+\t# See [1] above.\n+\t#\n+\tsleep 1 &&\n+\tmkdir test_implicit_2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+'\n+\n+test_expect_success 'cannot start multiple daemons' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n+\n+\tgit init test_multiple &&\n+\n+\tstart_daemon test_multiple &&\n+\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon start 2>actual &&\n+\tgrep \"fsmonitor--daemon is already running\" actual &&\n+\n+\tgit -C test_multiple fsmonitor--daemon stop &&\n+\ttest_must_fail git -C test_multiple fsmonitor--daemon status\n+'\n+\n+# These tests use the main repo in the trash directory\n+\n+test_expect_success 'setup' '\n+\t>tracked &&\n+\t>modified &&\n+\t>delete &&\n+\t>rename &&\n+\tmkdir dir1 &&\n+\t>dir1/tracked &&\n+\t>dir1/modified &&\n+\t>dir1/delete &&\n+\t>dir1/rename &&\n+\tmkdir dir2 &&\n+\t>dir2/tracked &&\n+\t>dir2/modified &&\n+\t>dir2/delete &&\n+\t>dir2/rename &&\n+\tmkdir dirtorename &&\n+\t>dirtorename/a &&\n+\t>dirtorename/b &&\n+\n+\tcat >.gitignore <<-\\EOF &&\n+\t.gitignore\n+\texpect*\n+\tactual*\n+\tEOF\n+\n+\tgit -c core.useBuiltinFSMonitor= add . &&\n+\ttest_tick &&\n+\tgit -c core.useBuiltinFSMonitor= commit -m initial &&\n+\n+\tgit config core.useBuiltinFSMonitor true\n+'\n+\n+# The test already explicitly stopped (or tried to stop) the daemon.\n+# This is here in case something else fails first.\n+#\n+redundant_stop_daemon () {\n+\tgit fsmonitor--daemon stop\n+\treturn 0\n+}\n+\n+test_expect_success 'update-index implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_1\" \\\n+\t\tgit update-index --fsmonitor &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_1\n+'\n+\n+test_expect_success 'status implicitly starts daemon' '\n+\ttest_when_finished redundant_stop_daemon &&\n+\n+\ttest_must_fail git fsmonitor--daemon status &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_2\" \\\n+\t\tgit status >actual &&\n+\n+\tgit fsmonitor--daemon status &&\n+\ttest_might_fail git fsmonitor--daemon stop &&\n+\n+\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_2\n+'\n+\n+edit_files() {\n+\techo 1 >modified\n+\techo 2 >dir1/modified\n+\techo 3 >dir2/modified\n+\t>dir1/untracked\n+}\n+\n+delete_files() {\n+\trm -f delete\n+\trm -f dir1/delete\n+\trm -f dir2/delete\n+}\n+\n+create_files() {\n+\techo 1 >new\n+\techo 2 >dir1/new\n+\techo 3 >dir2/new\n+}\n+\n+rename_files() {\n+\tmv rename renamed\n+\tmv dir1/rename dir1/renamed\n+\tmv dir2/rename dir2/renamed\n+}\n+\n+file_to_directory() {\n+\trm -f delete\n+\tmkdir delete\n+\techo 1 >delete/new\n+}\n+\n+directory_to_file() {\n+\trm -rf dir1\n+\techo 1 >dir1\n+}\n+\n+verify_status() {\n+\tgit status >actual &&\n+\tGIT_INDEX_FILE=.git/fresh-index git read-tree master &&\n+\tGIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect &&\n+\ttest_cmp expect actual &&\n+\techo HELLO AFTER &&\n+\tcat .git/trace &&\n+\techo HELLO AFTER\n+}\n+\n+# The next few test cases confirm that our fsmonitor daemon sees each type\n+# of OS filesystem notification that we care about. At this layer we just\n+# ensure we are getting the OS notifications and do not try to confirm what\n+# is reported by `git status`.\n+#\n+# We run a simple query after modifying the filesystem just to introduce\n+# a bit of a delay so that the trace logging from the daemon has time to\n+# get flushed to disk.\n+#\n+# We `reset` and `clean` at the bottom of each test (and before stopping the\n+# daemon) because these commands might implicitly restart the daemon.\n+\n+clean_up_repo_and_stop_daemon () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+\tgit fsmonitor--daemon stop\n+\trm -f .git/trace\n+}\n+\n+test_expect_success 'edit some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tedit_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/modified$\" .git/trace &&\n+\tgrep \"^event: dir2/modified$\" .git/trace &&\n+\tgrep \"^event: modified$\" .git/trace &&\n+\tgrep \"^event: dir1/untracked$\" .git/trace\n+'\n+\n+test_expect_success 'create some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tcreate_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/new$\" .git/trace &&\n+\tgrep \"^event: dir2/new$\" .git/trace &&\n+\tgrep \"^event: new$\" .git/trace\n+'\n+\n+test_expect_success 'delete some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdelete_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/delete$\" .git/trace &&\n+\tgrep \"^event: dir2/delete$\" .git/trace &&\n+\tgrep \"^event: delete$\" .git/trace\n+'\n+\n+test_expect_success 'rename some files' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\trename_files &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1/rename$\" .git/trace &&\n+\tgrep \"^event: dir2/rename$\" .git/trace &&\n+\tgrep \"^event: rename$\" .git/trace &&\n+\tgrep \"^event: dir1/renamed$\" .git/trace &&\n+\tgrep \"^event: dir2/renamed$\" .git/trace &&\n+\tgrep \"^event: renamed$\" .git/trace\n+'\n+\n+test_expect_success 'rename directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tmv dirtorename dirrenamed &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dirtorename/*$\" .git/trace &&\n+\tgrep \"^event: dirrenamed/*$\" .git/trace\n+'\n+\n+test_expect_success 'file changes to directory' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tfile_to_directory &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: delete$\" .git/trace &&\n+\tgrep \"^event: delete/new$\" .git/trace\n+'\n+\n+test_expect_success 'directory changes to a file' '\n+\ttest_when_finished clean_up_repo_and_stop_daemon &&\n+\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon\n+\t) &&\n+\n+\tdirectory_to_file &&\n+\n+\ttest-tool fsmonitor-client query --token 0 >/dev/null 2>&1 &&\n+\n+\tgrep \"^event: dir1$\" .git/trace\n+'\n+\n+# The next few test cases exercise the token-resync code. When filesystem\n+# drops events (because of filesystem velocity or because the daemon isn't\n+# polling fast enough), we need to discard the cached data (relative to the\n+# current token) and start collecting events under a new token.\n+#\n+# the 'test-tool fsmonitor-client flush' command can be used to send a\n+# \"flush\" message to a running daemon and ask it to do a flush/resync.\n+\n+test_expect_success 'flush cached data' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_flush\" &&\n+\n+\tgit init test_flush &&\n+\n+\t(\n+\t\tGIT_TEST_FSMONITOR_TOKEN=true &&\n+\t\texport GIT_TEST_FSMONITOR_TOKEN &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/.git/trace_daemon\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon test_flush\n+\t) &&\n+\n+\t# The daemon should have an initial token with no events in _0 and\n+\t# then a few (probably platform-specific number of) events in _1.\n+\t# These should both have the same .\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_0 &&\n+\tnul_to_q actual_q0 &&\n+\n+\ttouch test_flush/file_1 &&\n+\ttouch test_flush/file_2 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000001:0\" >actual_1 &&\n+\tnul_to_q actual_q1 &&\n+\n+\tgrep \"file_1\" actual_q1 &&\n+\n+\t# Force a flush. This will change the , reset the , and\n+\t# flush the file data. Then create some events and ensure that the file\n+\t# again appears in the cache. It should have the new .\n+\n+\ttest-tool -C test_flush fsmonitor-client flush >flush_0 &&\n+\tnul_to_q flush_q0 &&\n+\tgrep \"^builtin:test_00000002:0Q/Q$\" flush_q0 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_2 &&\n+\tnul_to_q actual_q2 &&\n+\n+\tgrep \"^builtin:test_00000002:0Q$\" actual_q2 &&\n+\n+\ttouch test_flush/file_3 &&\n+\n+\ttest-tool -C test_flush fsmonitor-client query --token \"builtin:test_00000002:0\" >actual_3 &&\n+\tnul_to_q actual_q3 &&\n+\n+\tgrep \"file_3\" actual_q3\n+'\n+\n+# The next few test cases create repos where the .git directory is NOT\n+# inside the one of the working directory. That is, where .git is a file\n+# that points to a directory elsewhere. This happens for submodules and\n+# non-primary worktrees.\n+\n+test_expect_success 'setup worktree base' '\n+\tgit init wt-base &&\n+\techo 1 >wt-base/file1 &&\n+\tgit -C wt-base add file1 &&\n+\tgit -C wt-base commit -m \"c1\"\n+'\n+\n+test_expect_success 'worktree with .git file' '\n+\tgit -C wt-base worktree add ../wt-secondary &&\n+\n+\t(\n+\t\tGIT_TRACE2_PERF=\"$(pwd)/trace2_wt_secondary\" &&\n+\t\texport GIT_TRACE2_PERF &&\n+\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/trace_wt_secondary\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon wt-secondary\n+\t) &&\n+\n+\tgit -C wt-secondary fsmonitor--daemon stop &&\n+\ttest_must_fail git -C wt-secondary fsmonitor--daemon status\n+'\n+\n+# NEEDSWORK: Repeat one of the \"edit\" tests on wt-secondary and\n+# confirm that we get the same events and behavior -- that is, that\n+# fsmonitor--daemon correctly watches BOTH the working directory and\n+# the external GITDIR directory and behaves the same as when \".git\"\n+# is a directory inside the working directory.\n+\n+test_expect_success 'cleanup worktrees' '\n+\tstop_daemon_delete_repo wt-secondary &&\n+\tstop_daemon_delete_repo wt-base\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"428931","messageId":"de517a8259a6eaba353f7e1023cb97c95734d6b2.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 26/34] t/perf/p7519: add fsmonitor--daemon test cases","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:36Z","receivedAt":"2021-07-01T14:48:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRepeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and\nthe \"Simple IPC\" interface.\n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7519-fsmonitor.sh | 37 ++++++++++++++++++++++++++++++++++---\n 1 file changed, 34 insertions(+), 3 deletions(-)\n\ndiff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\nindex f74e6014a0a..3a3fc5748ae 100755\n--- a/t/perf/p7519-fsmonitor.sh\n+++ b/t/perf/p7519-fsmonitor.sh\n@@ -24,7 +24,8 @@ test_description=\"Test core.fsmonitor\"\n # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex\n # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an\n # absolute path to an integration. May be a space delimited list of\n-# absolute paths to integrations.\n+# absolute paths to integrations. (This hook or list of hooks does not\n+# include the built-in fsmonitor--daemon.)\n #\n # The big win for using fsmonitor is the elimination of the need to scan the\n # working directory looking for changed and untracked files. If the file\n@@ -136,10 +137,16 @@ test_expect_success \"one time repo setup\" '\n \n setup_for_fsmonitor() {\n \t# set INTEGRATION_SCRIPT depending on the environment\n-\tif test -n \"$INTEGRATION_PATH\"\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n \tthen\n+\t\tgit config core.useBuiltinFSMonitor true &&\n+\t\tINTEGRATION_SCRIPT=false\n+\telif test -n \"$INTEGRATION_PATH\"\n+\tthen\n+\t\tgit config core.useBuiltinFSMonitor false &&\n \t\tINTEGRATION_SCRIPT=\"$INTEGRATION_PATH\"\n \telse\n+\t\tgit config core.useBuiltinFSMonitor false &&\n \t\t#\n \t\t# Choose integration script based on existence of Watchman.\n \t\t# Fall back to an empty integration script.\n@@ -175,7 +182,10 @@ test_perf_w_drop_caches () {\n }\n \n test_fsmonitor_suite() {\n-\tif test -n \"$INTEGRATION_SCRIPT\"; then\n+\tif test -n \"$USE_FSMONITOR_DAEMON\"\n+\tthen\n+\t\tDESC=\"builtin fsmonitor--daemon\"\n+\telif test -n \"$INTEGRATION_SCRIPT\"; then\n \t\tDESC=\"fsmonitor=$(basename $INTEGRATION_SCRIPT)\"\n \telse\n \t\tDESC=\"fsmonitor=disabled\"\n@@ -283,4 +293,25 @@ test_expect_success \"setup without fsmonitor\" '\n test_fsmonitor_suite\n trace_stop\n \n+#\n+# Run a full set of perf tests using the built-in fsmonitor--daemon.\n+# It does not use the Hook API, so it has a different setup.\n+# Explicitly start the daemon here and before we start client commands\n+# so that we can later add custom tracing.\n+#\n+if test_have_prereq FSMONITOR_DAEMON\n+then\n+\tUSE_FSMONITOR_DAEMON=t\n+\n+\ttrace_start fsmonitor--daemon--server\n+\tgit fsmonitor--daemon start\n+\n+\ttrace_start fsmonitor--daemon--client\n+\ttest_expect_success \"setup for fsmonitor--daemon\" 'setup_for_fsmonitor'\n+\ttest_fsmonitor_suite\n+\n+\tgit fsmonitor--daemon stop\n+\ttrace_stop\n+fi\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"428932","messageId":"3f36a31eb42147867f2cfd33f95187bb16c44a8b.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 28/34] fsmonitor--daemon: periodically truncate list of modified files","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:38Z","receivedAt":"2021-07-01T14:48:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to periodically truncate the list of\nmodified files to save some memory.\n\nClients will ask for the set of changes relative to a token that they\nfound in the FSMN index extension in the index. (This token is like a\npoint in time, but different). Clients will then update the index to\ncontain the response token (so that subsequent commands will be\nrelative to this new token).\n\nTherefore, the daemon can gradually truncate the in-memory list of\nchanged paths as they become obsolete (older than the previous token).\nSince we may have multiple clients making concurrent requests with a\nskew of tokens and clients may be racing to the talk to the daemon,\nwe lazily truncate the list.\n\nWe introduce a 5 minute delay and truncate batches 5 minutes after\nthey are considered obsolete.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 80 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 80 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 7a7fef681fe..8249420ba18 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -300,6 +300,77 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,\n \t\t\tbatch_src->interned_paths[k];\n }\n \n+/*\n+ * To keep the batch list from growing unbounded in response to filesystem\n+ * activity, we try to truncate old batches from the end of the list as\n+ * they become irrelevant.\n+ *\n+ * We assume that the .git/index will be updated with the most recent token\n+ * any time the index is updated. And future commands will only ask for\n+ * recent changes *since* that new token. So as tokens advance into the\n+ * future, older batch items will never be requested/needed. So we can\n+ * truncate them without loss of functionality.\n+ *\n+ * However, multiple commands may be talking to the daemon concurrently\n+ * or perform a slow command, so a little \"token skew\" is possible.\n+ * Therefore, we want this to be a little bit lazy and have a generous\n+ * delay.\n+ *\n+ * The current reader thread walked backwards in time from `token->batch_head`\n+ * back to `batch_marker` somewhere in the middle of the batch list.\n+ *\n+ * Let's walk backwards in time from that marker an arbitrary delay\n+ * and truncate the list there. Note that these timestamps are completely\n+ * artificial (based on when we pinned the batch item) and not on any\n+ * filesystem activity.\n+ */\n+#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */\n+\n+static void with_lock__truncate_old_batches(\n+\tstruct fsmonitor_daemon_state *state,\n+\tconst struct fsmonitor_batch *batch_marker)\n+{\n+\t/* assert current thread holding state->main_lock */\n+\n+\tconst struct fsmonitor_batch *batch;\n+\tstruct fsmonitor_batch *rest;\n+\tstruct fsmonitor_batch *p;\n+\n+\tif (!batch_marker)\n+\t\treturn;\n+\n+\ttrace_printf_key(&trace_fsmonitor, \"Truncate: mark (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t batch_marker->batch_seq_nr,\n+\t\t\t (uint64_t)batch_marker->pinned_time);\n+\n+\tfor (batch = batch_marker; batch; batch = batch->next) {\n+\t\ttime_t t;\n+\n+\t\tif (!batch->pinned_time) /* an overflow batch */\n+\t\t\tcontinue;\n+\n+\t\tt = batch->pinned_time + MY_TIME_DELAY_SECONDS;\n+\t\tif (t > batch_marker->pinned_time) /* too close to marker */\n+\t\t\tcontinue;\n+\n+\t\tgoto truncate_past_here;\n+\t}\n+\n+\treturn;\n+\n+truncate_past_here:\n+\tstate->current_token_data->batch_tail = (struct fsmonitor_batch *)batch;\n+\n+\trest = ((struct fsmonitor_batch *)batch)->next;\n+\t((struct fsmonitor_batch *)batch)->next = NULL;\n+\n+\tfor (p = rest; p; p = fsmonitor_batch__pop(p)) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"Truncate: kill (%\"PRIu64\",%\"PRIu64\")\",\n+\t\t\t\t p->batch_seq_nr, (uint64_t)p->pinned_time);\n+\t}\n+}\n+\n static void fsmonitor_free_token_data(struct fsmonitor_token_data *token)\n {\n \tstruct fsmonitor_batch *p;\n@@ -645,6 +716,15 @@ static int do_handle_client(struct fsmonitor_daemon_state *state,\n \t\t\t * that work.\n \t\t\t */\n \t\t\tfsmonitor_free_token_data(token_data);\n+\t\t} else if (batch) {\n+\t\t\t/*\n+\t\t\t * This batch is the first item in the list\n+\t\t\t * that is older than the requested sequence\n+\t\t\t * number and might be considered to be\n+\t\t\t * obsolete. See if we can truncate the list\n+\t\t\t * and save some memory.\n+\t\t\t */\n+\t\t\twith_lock__truncate_old_batches(state, batch);\n \t\t}\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"428933","messageId":"75bb4bc846364ec80e087d15bb1cb84265257ce3.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 30/34] fsmonitor: enhance existing comments","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:40Z","receivedAt":"2021-07-01T14:48:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 37 ++++++++++++++++++++++++++++++-------\n 1 file changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 3719ddfeec9..f53791c8674 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -360,9 +360,25 @@ void refresh_fsmonitor(struct index_state *istate)\n \t}\n \n apply_results:\n-\t/* a fsmonitor process can return '/' to indicate all entries are invalid */\n+\t/*\n+\t * The response from FSMonitor (excluding the header token) is\n+\t * either:\n+\t *\n+\t * [a] a (possibly empty) list of NUL delimited relative\n+\t * pathnames of changed paths. This list can contain\n+\t * files and directories. Directories have a trailing\n+\t * slash.\n+\t *\n+\t * [b] a single '/' to indicate the provider had no\n+\t * information and that we should consider everything\n+\t * invalid. We call this a trivial response.\n+\t */\n \tif (query_success && query_result.buf[bol] != '/') {\n-\t\t/* Mark all entries returned by the monitor as dirty */\n+\t\t/*\n+\t\t * Mark all pathnames returned by the monitor as dirty.\n+\t\t *\n+\t\t * This updates both the cache-entries and the untracked-cache.\n+\t\t */\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n@@ -377,11 +393,15 @@ apply_results:\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n \t} else {\n-\n-\t\t/* We only want to run the post index changed hook if we've actually changed entries, so keep track\n-\t\t * if we actually changed entries or not */\n+\t\t/*\n+\t\t * We received a trivial response, so invalidate everything.\n+\t\t *\n+\t\t * We only want to run the post index changed hook if\n+\t\t * we've actually changed entries, so keep track if we\n+\t\t * actually changed entries or not.\n+\t\t */\n \t\tint is_cache_changed = 0;\n-\t\t/* Mark all entries invalid */\n+\n \t\tfor (i = 0; i < istate->cache_nr; i++) {\n \t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) {\n \t\t\t\tis_cache_changed = 1;\n@@ -389,7 +409,10 @@ apply_results:\n \t\t\t}\n \t\t}\n \n-\t\t/* If we're going to check every file, ensure we save the results */\n+\t\t/*\n+\t\t * If we're going to check every file, ensure we save\n+\t\t * the results.\n+\t\t */\n \t\tif (is_cache_changed)\n \t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n \n-- \ngitgitgadget\n\n"},{"id":"428934","messageId":"97dce46d1d04527a540de37d814ca094e487623a.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 32/34] t7527: test status with untracked-cache and fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:42Z","receivedAt":"2021-07-01T14:48:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate 2x2 test matrix with the untracked-cache and fsmonitor--daemon\nfeatures and a series of edits and verify that status output is\nidentical.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 87 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 87 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 58b56dc9940..d1832702397 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -152,6 +152,8 @@ test_expect_success 'setup' '\n \t.gitignore\n \texpect*\n \tactual*\n+\tflush*\n+\ttrace*\n \tEOF\n \n \tgit -c core.useBuiltinFSMonitor= add . &&\n@@ -494,4 +496,89 @@ test_expect_success 'cleanup worktrees' '\n \tstop_daemon_delete_repo wt-base\n '\n \n+# The next few tests perform arbitrary/contrived file operations and\n+# confirm that status is correct. That is, that the data (or lack of\n+# data) from fsmonitor doesn't cause incorrect results. And doesn't\n+# cause incorrect results when the untracked-cache is enabled.\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n+\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n+\tgit update-index --no-fsmonitor &&\n+\ttest_might_fail git fsmonitor--daemon stop\n+'\n+\n+matrix_clean_up_repo () {\n+\tgit reset --hard HEAD\n+\tgit clean -fd\n+}\n+\n+matrix_try () {\n+\tuc=$1\n+\tfsm=$2\n+\tfn=$3\n+\n+\ttest_expect_success \"Matrix[uc:$uc][fsm:$fsm] $fn\" '\n+\t\tmatrix_clean_up_repo &&\n+\t\t$fn &&\n+\t\tif test $uc = false -a $fsm = false\n+\t\tthen\n+\t\t\tgit status --porcelain=v1 >.git/expect.$fn\n+\t\telse\n+\t\t\tgit status --porcelain=v1 >.git/actual.$fn\n+\t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n+\t\tfi\n+\t'\n+\n+\treturn $?\n+}\n+\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+for uc_val in $uc_values\n+do\n+\tif test $uc_val = false\n+\tthen\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] disable untracked cache\" '\n+\t\t\tgit config core.untrackedcache false &&\n+\t\t\tgit update-index --no-untracked-cache\n+\t\t'\n+\telse\n+\t\ttest_expect_success \"Matrix[uc:$uc_val] enable untracked cache\" '\n+\t\t\tgit config core.untrackedcache true &&\n+\t\t\tgit update-index --untracked-cache\n+\t\t'\n+\tfi\n+\n+\tfsm_values=\"false true\"\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tif test $fsm_val = false\n+\t\tthen\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor\" '\n+\t\t\t\ttest_might_fail git config --unset core.useBuiltinFSMonitor &&\n+\t\t\t\tgit update-index --no-fsmonitor &&\n+\t\t\t\ttest_might_fail git fsmonitor--daemon stop 2>/dev/null\n+\t\t\t'\n+\t\telse\n+\t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor\" '\n+\t\t\t\tgit config core.useBuiltinFSMonitor true &&\n+\t\t\t\tgit fsmonitor--daemon start &&\n+\t\t\t\tgit update-index --fsmonitor\n+\t\t\t'\n+\t\tfi\n+\n+\t\tmatrix_try $uc_val $fsm_val edit_files\n+\t\tmatrix_try $uc_val $fsm_val delete_files\n+\t\tmatrix_try $uc_val $fsm_val create_files\n+\t\tmatrix_try $uc_val $fsm_val rename_files\n+\t\tmatrix_try $uc_val $fsm_val file_to_directory\n+\t\tmatrix_try $uc_val $fsm_val directory_to_file\n+\tdone\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"428935","messageId":"8b3c4f4e6dd989de88b0b7b543751e362fa0d649.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 31/34] fsmonitor: force update index after large responses","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:41Z","receivedAt":"2021-07-01T14:48:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSet the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when\nFSMonitor returns a very large repsonse to ensure that the index is\nwritten to disk.\n\nNormally, when the FSMonitor response includes a tracked file, the\nindex is always updated. Similarly, the index might be updated when\nthe response alters the untracked-cache (when enabled). However, in\ncases where neither of those cause the index to be considered changed,\nthe FSMonitor response is wasted. Subsequent Git commands will make\nrequests with the same token and receive the same response.\n\nIf that response is very large, performance may suffer. It would be\nmore efficient to force update the index now (and the token in the\nindex extension) in order to reduce the size of the response received\nby future commands.\n\nThis was observed on Windows after a large checkout. On Windows, the\nkernel emits events for the files that are changed as they are\nchanged. However, it might delay events for the containing\ndirectories until the system is more idle (or someone scans the\ndirectory (so it seems)). The first status following a checkout would\nget the list of files. The subsequent status commands would get the\nlist of directories as the events trickled out. But they would never\ncatch up because the token was not advanced because the index wasn't\nupdated.\n\nThis list of directories caused `wt_status_collect_untracked()` to\nunnecessarily spend time actually scanning them during each command.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 49 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex f53791c8674..eee653f9337 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -236,6 +236,45 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n+/*\n+ * The number of pathnames that we need to receive from FSMonitor\n+ * before we force the index to be updated.\n+ *\n+ * Note that any pathname within the set of received paths MAY cause\n+ * cache-entry or istate flag bits to be updated and thus cause the\n+ * index to be updated on disk.\n+ *\n+ * However, the response may contain many paths (such as ignored\n+ * paths) that will not update any flag bits. And thus not force the\n+ * index to be updated. (This is fine and normal.) It also means\n+ * that the token will not be updated in the FSMonitor index\n+ * extension. So the next Git command will find the same token in the\n+ * index, make the same token-relative request, and receive the same\n+ * response (plus any newly changed paths). If this response is large\n+ * (and continues to grow), performance could be impacted.\n+ *\n+ * For example, if the user runs a build and it writes 100K object\n+ * files but doesn't modify any source files, the index would not need\n+ * to be updated. The FSMonitor response (after the build and\n+ * relative to a pre-build token) might be 5MB. Each subsequent Git\n+ * command will receive that same 100K/5MB response until something\n+ * causes the index to be updated. And `refresh_fsmonitor()` will\n+ * have to iterate over those 100K paths each time.\n+ *\n+ * Performance could be improved if we optionally force update the\n+ * index after a very large response and get an updated token into\n+ * the FSMonitor index extension. This should allow subsequent\n+ * commands to get smaller and more current responses.\n+ *\n+ * The value chosen here does not need to be precise. The index\n+ * will be updated automatically the first time the user touches\n+ * a tracked file and causes a command like `git status` to\n+ * update an mtime to be updated and/or set a flag bit.\n+ *\n+ * NEEDSWORK: Does this need to be a config value?\n+ */\n+static int fsmonitor_force_update_threshold = 100;\n+\n void refresh_fsmonitor(struct index_state *istate)\n {\n \tstruct strbuf query_result = STRBUF_INIT;\n@@ -379,19 +418,28 @@ apply_results:\n \t\t *\n \t\t * This updates both the cache-entries and the untracked-cache.\n \t\t */\n+\t\tint count = 0;\n+\n \t\tbuf = query_result.buf;\n \t\tfor (i = bol; i < query_result.len; i++) {\n \t\t\tif (buf[i] != '\\0')\n \t\t\t\tcontinue;\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n \t\t\tbol = i + 1;\n+\t\t\tcount++;\n \t\t}\n-\t\tif (bol < query_result.len)\n+\t\tif (bol < query_result.len) {\n \t\t\tfsmonitor_refresh_callback(istate, buf + bol);\n+\t\t\tcount++;\n+\t\t}\n \n \t\t/* Now mark the untracked cache for fsmonitor usage */\n \t\tif (istate->untracked)\n \t\t\tistate->untracked->use_fsmonitor = 1;\n+\n+\t\tif (count > fsmonitor_force_update_threshold)\n+\t\t\tistate->cache_changed |= FSMONITOR_CHANGED;\n+\n \t} else {\n \t\t/*\n \t\t * We received a trivial response, so invalidate everything.\n-- \ngitgitgadget\n\n"},{"id":"428936","messageId":"e32ba686f7e54097fb97f615d357ed3779755f21.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 33/34] fsmonitor: handle shortname for .git","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:43Z","receivedAt":"2021-07-01T14:48:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nOn Windows, teach FSMonitor to recognize the shortname of \".git\"\nas an alias for \".git\".\n\nSometimes we receive FS events using the shortname, such as when\na CMD shell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS\nnotification arrives using whatever combination of long and\nshortnames used by the other process. (Shortnames do seem to\nbe case normalized, however.)\n\nNEEDSWORK: This only addresses the case of removing or renaming\nthe \".git\" directory using the shortname alias, so that the daemon\nproperly shuts down. I'm leaving it a task for later to handle\nthe general case of shortnames and report them to the fsmonitor\nclient process. This would include tracked and untracked paths\nthat just happen to have a shortname alias.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 192 +++++++++++++++----\n t/t7527-builtin-fsmonitor.sh | 65 +++++++\n 2 files changed, 217 insertions(+), 40 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\nindex d707d47a0d7..f2ea5940790 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n@@ -48,6 +48,8 @@ struct fsmonitor_daemon_backend_data\n #define LISTENER_HAVE_DATA_WORKTREE 1\n #define LISTENER_HAVE_DATA_GITDIR 2\n \tint nr_listener_handles;\n+\n+\tstruct strbuf dot_git_shortname;\n };\n \n /*\n@@ -250,6 +252,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tFILE_NOTIFY_INFORMATION *info,\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n+\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -294,7 +352,6 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n \t\tstrbuf_reset(&path);\n@@ -303,45 +360,45 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \n \t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n \n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n-\n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n-\t\t\t\tgoto force_shutdown;\n-\t\t\t}\n-\t\t\tbreak;\n-\n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tif (process_1_worktree_event(info, &cookie_list, &batch,\n+\t\t\t\t\t &path, t))\n+\t\t\tgoto force_shutdown;\n+\n+\t\t/*\n+\t\t * NEEDSWORK: If `path` contains a shortname (that is,\n+\t\t * if any component within it is a shortname), we\n+\t\t * should expand it to a longname (See\n+\t\t * `GetLongPathNameW()`) and re-normalize, classify,\n+\t\t * and process it because our client is probably\n+\t\t * expecting \"normal\" paths.\n+\t\t *\n+\t\t * HOWEVER, if our process has called `chdir()` to get\n+\t\t * us out of the root of the worktree (so that the\n+\t\t * root directory is not busy), then we have to be\n+\t\t * careful to convert the paths in the INFO array\n+\t\t * (which are relative to the directory of the RDCW\n+\t\t * watch and not the CWD) into absolute paths before\n+\t\t * calling GetLongPathNameW() and then convert the\n+\t\t * computed value back to a RDCW-relative pathname\n+\t\t * (which is what we and the client expect).\n+\t\t *\n+\t\t * FOR NOW, just handle case (1) exactly so that we\n+\t\t * shutdown properly when \".git\" is deleted via the\n+\t\t * shortname alias.\n+\t\t *\n+\t\t * We might see case (2) events for cookie files, but\n+\t\t * we can ignore them.\n+\t\t *\n+\t\t * FOR LATER, handle case (3) where the worktree\n+\t\t * events contain shortnames. We should convert\n+\t\t * them to longnames to avoid confusing the client.\n+\t\t */\n+\t\tif (data->dot_git_shortname.len &&\n+\t\t !strcmp(path.buf, data->dot_git_shortname.buf) &&\n+\t\t process_1_worktree_event(info, &cookie_list, &batch,\n+\t\t\t\t\t &data->dot_git_shortname,\n+\t\t\t\t\t IS_DOT_GIT))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -415,6 +472,14 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\t\t t, path.buf);\n \t\t}\n \n+\t\t/*\n+\t\t * WRT shortnames, this external gitdir will not see\n+\t\t * case (1) nor case (3) events.\n+\t\t *\n+\t\t * We might see case (2) events for cookie files, but\n+\t\t * we can ignore them.\n+\t\t */\n+\n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n \t\t\tbreak;\n@@ -493,6 +558,7 @@ clean_shutdown:\n int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsmonitor_daemon_backend_data *data;\n+\tchar shortname[16]; /* a padded 8.3 buffer */\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -523,6 +589,52 @@ int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n+\t/*\n+\t * NEEDSWORK: Properly handle 8.3 shortnames. RDCW events can\n+\t * contain a shortname (if another application uses a\n+\t * shortname in a system call). We care about aliasing and\n+\t * the use of shortnames for:\n+\t *\n+\t * (1) \".git\",\n+\t * -- if an external process deletes \".git\" using \"GIT~1\",\n+\t * we need to catch that and shutdown.\n+\t *\n+\t * (2) our cookie files,\n+\t * -- if an external process deletes one of our cookie\n+\t * files using a shortname, we will get a shortname\n+\t * event for it. However, we should have already\n+\t * gotten a longname event for it when we created the\n+\t * cookie, so we can safely discard the shortname\n+\t * events for cookie files.\n+\t *\n+\t * (3) the spelling of modified files that we report to clients.\n+\t * -- we need to report the longname to the client because\n+\t * that is what they are expecting. Presumably, the\n+\t * client is going to lookup the paths that we report\n+\t * in their index and untracked-cache, so we should\n+\t * normalize the data for them. (Technically, they\n+\t * could adapt, so we could relax this maybe.)\n+\t *\n+\t * FOR NOW, while our CWD is at the root of the worktree we\n+\t * can easily get the spelling of the shortname of \".git\" (if\n+\t * the volume has shortnames enabled). For most worktrees\n+\t * this value will be \"GIT~1\", but we don't want to assume\n+\t * that.\n+\t *\n+\t * Capture this so that we can handle (1).\n+\t *\n+\t * We leave (3) for a future effort.\n+\t */\n+\tstrbuf_init(&data->dot_git_shortname, 0);\n+\tGetShortPathNameA(\".git\", shortname, sizeof(shortname));\n+\tif (!strcmp(\".git\", shortname))\n+\t\ttrace_printf_key(&trace_fsmonitor, \"No shortname for '.git'\");\n+\telse {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"Shortname of '.git' is '%s'\", shortname);\n+\t\tstrbuf_addstr(&data->dot_git_shortname, shortname);\n+\t}\n+\n \tstate->backend_data = data;\n \treturn 0;\n \ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex d1832702397..b166b4a0a31 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -113,6 +113,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\t[ -f test_implicit_1s2/GIT~1 ] &&\n+\t[ -d test_implicit_1s2/GIT~2 ] &&\n+\n+\tstart_daemon test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"428937","messageId":"627e27fe60bb543902fdcc4b2179c620403d9c38.1625150864.git.gitgitgadget@gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"[PATCH v3 34/34] t7527: test FS event reporing on MacOS WRT case and Unicode","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-01T14:47:44Z","receivedAt":"2021-07-01T14:48:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that MacOS FS events are reported with a normalized spelling.\n\nAPFS (and/or HFS+) is case-insensitive. This means that case-independent\nlookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed. But that\ndoesn't tell us how FS events are reported if we try \"rm -rf .git\" versus\n\"rm -rf .GIT\". Are the events reported using the on-disk spelling of the\npathname or in the spelling used by the command.\n\nNEEDSWORK: I was only able to test case. It would be nice to add tests\nthat use different Unicode spellings/normalizations and understand the\ndifferences between APFS and HFS+ in this area. We should confirm that\nthe spelling of the workdir paths that the daemon sends to clients are\nalways properly normalized.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 30 ++++++++++++++++++++++++++++++\n 1 file changed, 30 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex b166b4a0a31..d2ff1bf6c49 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -178,6 +178,36 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n \ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n '\n \n+# Confirm that MacOS hides all of the Unicode normalization and/or\n+# case folding from the FS events. That is, are the pathnames in the\n+# FS events reported using the spelling on the disk or in the spelling\n+# used by the other process.\n+#\n+# Note that we assume that the filesystem is set to case insensitive.\n+#\n+# NEEDSWORK: APFS handles Unicode and Unicode normalization\n+# differently than HFS+. I only have an APFS partition, so\n+# more testing here would be helpful.\n+#\n+\n+# Rename .git using alternate spelling and confirm that the daemon\n+# sees the event using the correct spelling and shutdown.\n+test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_apfs\" &&\n+\n+\tgit init test_apfs &&\n+\tstart_daemon test_apfs &&\n+\n+\t[ -d test_apfs/.git ] &&\n+\t[ -d test_apfs/.GIT ] &&\n+\n+\tmv test_apfs/.GIT test_apfs/.FOO &&\n+\tsleep 1 &&\n+\tmv test_apfs/.FOO test_apfs/.git &&\n+\n+\ttest_must_fail git -C test_apfs fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n"},{"id":"428974","messageId":"87im1uc57c.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"8b64b7cd3674163adb0588d42eccf4873b30974b.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 06/34] fsmonitor: config settings are repository-specific","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T16:46:01Z","receivedAt":"2021-07-01T17:30:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\nIn a reference to a discussion[1] about an earlier version of this patch\nyou said:\n\n I'm going to ignore all of the thread responses to this patch\n dealing with how we acquire config settings and macros and etc.\n Those issues are completely independent of FSMonitor (which is\n already way too big).\n\nSince then the changes to repo-settings.c have become a lot larger, so\nlet's take a look...\n\n1. https://lore.kernel.org/git/87mttkyrqq.fsf@evledraar.gmail.com/\n2. https://lore.kernel.org/git/4552971c-0a23-c19a-6a23-cb5737e43b2a@jeffhostetler.com/\n\n\n> diff --git a/repo-settings.c b/repo-settings.c\n> index 0cfe8b787db..faf197ff60a 100644\n> --- a/repo-settings.c\n> +++ b/repo-settings.c\n> @@ -5,10 +5,42 @@\n> \n> #define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0)\n> \n> +/*\n> + * Return 1 if the repo/workdir is incompatible with FSMonitor.\n> + */\n> +static int is_repo_incompatible_with_fsmonitor(struct repository *r)\n> +{\n> +\tconst char *const_strval;\n> +\n> +\t/*\n> +\t * Bare repositories don't have a working directory and\n> +\t * therefore, nothing to watch.\n> +\t */\n> +\tif (!r->worktree)\n> +\t\treturn 1;\n\nLooking ahead in this series you end up using\nFSMONITOR_MODE_INCOMPATIBLE in two places in the codebase. In\nbuiltin/update-index.c to throw a \"repository is incompatible with\nfsmonitor\" error.\n\nCan't that case just be replaced with setup_work_tree()? Other sub-modes\nof update-index already die implicitly on that, e.g.:\n\n\t$ git update-index test\n\tfatal: this operation must be run in a work tree\n\nThe other case is:\n\t\n\t+ prepare_repo_settings(the_repository);\n\t+ if (!the_repository->worktree)\n\t+ return error(_(\"fsmonitor-daemon does not support bare repos '%s'\"),\n\t+ xgetcwd());\n\t+ if (the_repository->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE)\n\t+ return error(_(\"fsmonitor-daemon is incompatible with this repo '%s'\"),\n\t+ the_repository->worktree);\n\nI.e. we just checked the_repository->worktree, but it's not that, but....\n\n> +\n> +\t/*\n> +\t * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n> +\t *\n> +\t * Granted, core Git does not know anything about GVFS and\n> +\t * we shouldn't make assumptions about a downstream feature,\n> +\t * but users can install both versions. And this can lead\n> +\t * to incorrect results from core Git commands. So, without\n> +\t * bringing in any of the GVFS code, do a simple config test\n> +\t * for a published config setting. (We do not look at the\n> +\t * various *_TEST_* environment variables.)\n> +\t */\n> +\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_strval))\n> +\t\treturn 1;\n\nI'm skeptical of us hardcoding a third-party software config\nvariable. Can't GitVFS handle this somehow on its end?\n\nBut just in terms of implementation it seems the end result of that is\nto emit a very confusing error to the user. Sinc we already checked for\nbare repos we run into this and instead of sayingwhen we should really\nsay \"hey, maybe disable your core.virtualFileSystem setting\", we say\n\"your repo is incompatible\".\n\n> +\n> +\treturn 0;\n> +}\n> +\n> void prepare_repo_settings(struct repository *r)\n> {\n> \tint value;\n> \tchar *strval;\n> +\tconst char *const_strval;\n\nCan be declared in the \"else\" below.\n\n> \n> \tif (r->settings.initialized)\n> \t\treturn;\n> @@ -26,6 +58,22 @@ void prepare_repo_settings(struct repository *r)\n> \tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n> \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n> \n> +\tr->settings.fsmonitor_hook_path = NULL;\n> +\tr->settings.fsmonitor_mode = FSMONITOR_MODE_DISABLED;\n\nWith the memset earlier (b.t.w. I've got a patch to fix all this bizarre\nbehavior in repo-settings.c, but have been waiting on this series we\nimplicitly set it to FSMONITOR_MODE_UNSET (-1) with the memset, but then\nnever use that ever.\n\nYour code in update-index.c then for a check against\n\"FSMONITOR_MODE_DISABLED\" says \"core.useBuiltinFSMonitor is unset;\".\n\n> +\tif (is_repo_incompatible_with_fsmonitor(r))\n> +\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_INCOMPATIBLE;\n\nStyle: should have {} braces on all arms.\n\n> +\telse if (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value)\n> +\t\t && value)\n> +\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_IPC;\n\nHere you're conflating false with whether the variable is set at all. I\nguess that works out here since if it's false we want to fall through\nto...\n\n> +\telse {\n\n...ignoring it and looing at core.fsmonitor instead.\n\n> +\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_strval))\n> +\t\t\tconst_strval = getenv(\"GIT_TEST_FSMONITOR\");\n\nIf it's not set we pay attention to GIT_TEST_FSMONITOR, so the behavior\nfrom the old git_config_get_fsmonitor(). So even if the env variable is\nset we want to take the config variable over it, correct?\n\n> +\t\tif (const_strval && *const_strval) {\n> +\t\t\tr->settings.fsmonitor_hook_path = strdup(const_strval);\n\nWe had a strbuf_detach()'d string in the case of\nrepo_config_get_pathname(), but here we strdup() it again in case we\nwere in the getenv() codepath. This code probably leaks memory now\nanyway, but perhaps it's better to split up the two so we make it easier\nto deal with who owns/frees what in the future.\n"},{"id":"428977","messageId":"87fswyc4dz.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"pull.923.v3.git.1625150864.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T17:40:45Z","receivedAt":"2021-07-01T17:47:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> Here is V3 of my patch series to add a builtin FSMonitor daemon to Git. I\n> rebased this series onto v2.32.0.\n>\n> V3 addresses most of the previous review comments and things we've learned\n> from our experimental testing of V2. (A version of V2 was shipped as an\n> experimental feature in the v2.32.0-based releases of Git for Windows and\n> VFS for Git.)\n>\n> There are still a few items that I need to address, but that list is getting\n> very short.\n\n...\n> fsmonitor-fs-listen-win32: stub in backend for Windows\n> fsmonitor-fs-listen-macos: stub in backend for MacOS\n\nI left some light comments on the repo-settings.c part of this to follow\nup from a previous round.\n\nAny other testing of it is stalled by there being no linux backend for\nit as part of this series. I see from spelunking repos that Johannes had\na WIP compat/fsmonitor/linux.c which looks like it could/should mostly\nwork, but the API names all changed since then, and after a short try I\ngave up on trying to rebase it.\n\nI'd really prefer for git not to have features that place free platforms\nat a disadvantage against proprietary platforms if it can be avoided,\nand in this case the lack of a Linux backend also means much less\nwidespread testing of the feature among the development community / CI.\n"},{"id":"428978","messageId":"e1442a04-7c68-0a7a-6e95-304854adff39@jeffhostetler.com","threadId":"55420","inReplyTo":"87fswyc4dz.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-01T18:29:23Z","receivedAt":"2021-07-01T18:29:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 1:40 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n>> Here is V3 of my patch series to add a builtin FSMonitor daemon to Git. I\n>> rebased this series onto v2.32.0.\n>>\n>> V3 addresses most of the previous review comments and things we've learned\n>> from our experimental testing of V2. (A version of V2 was shipped as an\n>> experimental feature in the v2.32.0-based releases of Git for Windows and\n>> VFS for Git.)\n>>\n>> There are still a few items that I need to address, but that list is getting\n>> very short.\n> \n> ...\n>> fsmonitor-fs-listen-win32: stub in backend for Windows\n>> fsmonitor-fs-listen-macos: stub in backend for MacOS\n> \n> I left some light comments on the repo-settings.c part of this to follow\n> up from a previous round.\n\nThanks.\n\n> \n> Any other testing of it is stalled by there being no linux backend for\n> it as part of this series. I see from spelunking repos that Johannes had\n> a WIP compat/fsmonitor/linux.c which looks like it could/should mostly\n> work, but the API names all changed since then, and after a short try I\n> gave up on trying to rebase it.\n\nThe early Linux version was dropped because inotify does not give\nrecursive coverage -- only the requested directory. Using inotify\nrequires adding a watch to each subdirectory (recursively) in the\nworktree. There's a system limit on the maximum number of watched\ndirectories (defaults to 8K IIRC) and that limit is system-wide.\n\nSince the whole point was to support large very large repos, using\ninotify was a non-starter, so I removed the Linux version from our\npatch series. For example, the first repo I tried it on (outside\nof the test suite) had 25K subdirectories.\n\nI'm told there is a new fanotify API in recent Linux kernels that\nis a better fit for what we need, but we haven't had time to investigate\nit yet.\n\n> \n> I'd really prefer for git not to have features that place free platforms\n> at a disadvantage against proprietary platforms if it can be avoided,\n> and in this case the lack of a Linux backend also means much less\n> widespread testing of the feature among the development community / CI.\n> \n\nThis feature is always going to have platform-specific components,\nso the lack of one platform or another should not stop us from\ndiscussing it for the platforms that can be supported.\n\nAnd given the size and complexity of the platform-specific code,\nwe should not assume that \"just test it on Linux\" is sufficient.\nYes, there are some common/shared routines/data structures in the\ndaemon, but hard/tricky parts are in the platform layer.\n\nJeff\n"},{"id":"428981","messageId":"xmqqsg0xreht.fsf@gitster.g","threadId":"55420","inReplyTo":"3cf8f3cd7717dad69262ae884a89c1a6ca0ce5fc.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 23/34] t/helper/test-touch: add helper to touch a series of files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-01T20:00:30Z","receivedAt":"2021-07-01T20:00:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> diff --git a/t/helper/test-touch.c b/t/helper/test-touch.c\n> new file mode 100644\n> index 00000000000..e9b3b754f1f\n> --- /dev/null\n> +++ b/t/helper/test-touch.c\n> @@ -0,0 +1,126 @@\n> +/*\n> + * test-touch.c: variation on /usr/bin/touch to speed up tests\n> + * with a large number of files (primarily on Windows where child\n> + * process are very, very expensive).\n> + */\n> +\n> +#include \"test-tool.h\"\n> +#include \"cache.h\"\n> +#include \"parse-options.h\"\n> +\n> +char *seq_pattern;\n> +int seq_start = 1;\n> +int seq_count = 1;\n\nWith this in, \"make sparse\" dies like this:\n\n SP t/helper/test-touch.c\nt/helper/test-touch.c:11:6: error: symbol 'seq_pattern' was not declared. Should it be static?\nt/helper/test-touch.c:12:5: error: symbol 'seq_start' was not declared. Should it be static?\nt/helper/test-touch.c:13:5: error: symbol 'seq_count' was not declared. Should it be static?\n\n"},{"id":"428991","messageId":"87czs1d6uy.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"e1442a04-7c68-0a7a-6e95-304854adff39@jeffhostetler.com","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T21:26:06Z","receivedAt":"2021-07-01T22:09:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler wrote:\n\n> On 7/1/21 1:40 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> Here is V3 of my patch series to add a builtin FSMonitor daemon to Git. I\n>>> rebased this series onto v2.32.0.\n>>>\n>>> V3 addresses most of the previous review comments and things we've learned\n>>> from our experimental testing of V2. (A version of V2 was shipped as an\n>>> experimental feature in the v2.32.0-based releases of Git for Windows and\n>>> VFS for Git.)\n>>>\n>>> There are still a few items that I need to address, but that list is getting\n>>> very short.\n>> ...\n>>> fsmonitor-fs-listen-win32: stub in backend for Windows\n>>> fsmonitor-fs-listen-macos: stub in backend for MacOS\n>> I left some light comments on the repo-settings.c part of this to\n>> follow\n>> up from a previous round.\n>\n> Thanks.\n>\n>> Any other testing of it is stalled by there being no linux backend\n>> for\n>> it as part of this series. I see from spelunking repos that Johannes had\n>> a WIP compat/fsmonitor/linux.c which looks like it could/should mostly\n>> work, but the API names all changed since then, and after a short try I\n>> gave up on trying to rebase it.\n>\n> The early Linux version was dropped because inotify does not give\n> recursive coverage -- only the requested directory. Using inotify\n> requires adding a watch to each subdirectory (recursively) in the\n> worktree. There's a system limit on the maximum number of watched\n> directories (defaults to 8K IIRC) and that limit is system-wide.\n>\n> Since the whole point was to support large very large repos, using\n> inotify was a non-starter, so I removed the Linux version from our\n> patch series. For example, the first repo I tried it on (outside\n> of the test suite) had 25K subdirectories.\n>\n> I'm told there is a new fanotify API in recent Linux kernels that\n> is a better fit for what we need, but we haven't had time to investigate\n> it yet.\n\nThat default limit is a bit annoying, but I don't see how it's a blocker\nin any way.\n\nYou simply adjust the limit. E.g. I deployed and tested the hook version\nof inotify (using watchman) in a sizable development environment, and\nwritten my own code using the API. This was all before fanotify(7)\nexisted. IIRC I set most of the limits to 2^24 or 2^20. I've used it\nwith really large Git repos, including with David Turner's\n2015-04-03-1M-git for testing (`git ls-files | wc -l` on that is around\na quarter-million).\n\nIf you have a humongous repository and don't have root on your own\nmachine you're out of luck. But I think most people who'd use this are\neither using their own laptop, or e.g. in a corporate setting where\nadministrators would tweak the sysctl limits given the performance\nadvantages (as I did).\n\nOnce you adjust the limits Linux deals with large trees just fine, it\njust has overly conservative limits for most things in sysctl. The API\nis a bit annoying, your event loop needs to run around and add watches.\n\nAFAICT you need Linux 5.1 for fanotify(7) to be useful, e.g. Debian\nstable, RHEL etc. aren't using anything that new. So having an inotify\nbackend as well as possibly a fanotify one would be very useful.\n\nAnd linux.git's docs suggest that the default limit was bumped from 8192\nto 1M in v5.10, a really recent kernel, so if you've got that you've\nalso got fanotify.\n\nIn any case, even if Linux's inotify limit was something hardcoded and\nimpossible to change you could still use such an API to test & debug the\nbasics of this feature on that platform.\n\n>> I'd really prefer for git not to have features that place free\n>> platforms\n>> at a disadvantage against proprietary platforms if it can be avoided,\n>> and in this case the lack of a Linux backend also means much less\n>> widespread testing of the feature among the development community / CI.\n>> \n>\n> This feature is always going to have platform-specific components,\n> so the lack of one platform or another should not stop us from\n> discussing it for the platforms that can be supported.\n\n(I think per the above that's s/can be/are/)\n\n> And given the size and complexity of the platform-specific code,\n> we should not assume that \"just test it on Linux\" is sufficient.\n> Yes, there are some common/shared routines/data structures in the\n> daemon, but hard/tricky parts are in the platform layer.\n\nI think we're talking past each other a bit here. I'm not saying that\nyou can get full or meaningful testing for it on Windows if you test it\non Linux, or the other way around.\n\nOf course there's platform-specific stuff, although there's also a lot\nof non-platform-specific stuff, so even a very basic implementation of\ninotify would make reviwing this easier / give access to more reviewers.\n\nI'm saying that I prefer that Git as a free software project not be in\nthe situation of saying the best use-case for a given size/shape of repo\nis to use Git in combination with proprietary operating systems over\nfreely licensed ones.\n\nIOW what the FSF has a policy for GNU projects. Now, I think the FSF\nprobably goes too far in that (famously removing rather obscure font\nrendering features from Emacs on OSX), but \"manage lots of data\" is a\ncore feature of git.\n\n"},{"id":"428994","messageId":"87a6n5d600.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"9fe902aad87f1192705fb69ea212a2d066d0286d.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T22:18:01Z","receivedAt":"2021-07-01T22:27:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> +#ifdef GIT_WINDOWS_NATIVE\n> +/*\n> + * Create a background process to run the daemon. It should be completely\n> + * disassociated from the terminal.\n> + *\n> + * Conceptually like `daemonize()` but different because Windows does not\n> + * have `fork(2)`. Spawn a normal Windows child process but without the\n> + * limitations of `start_command()` and `finish_command()`.\n> + *\n> + * The child process execs the \"git fsmonitor--daemon run\" command.\n> + *\n> + * The current process returns so that the caller can wait for the child\n> + * to startup before exiting.\n> + */\n> +static int spawn_background_fsmonitor_daemon(pid_t *pid)\n> +{\n> +\tchar git_exe[MAX_PATH];\n> +\tstruct strvec args = STRVEC_INIT;\n> +\tint in, out;\n> +\n> +\tGetModuleFileNameA(NULL, git_exe, MAX_PATH);\n> +\n> +\tin = open(\"/dev/null\", O_RDONLY);\n> +\tout = open(\"/dev/null\", O_WRONLY);\n> +\n> +\tstrvec_push(&args, git_exe);\n> +\tstrvec_push(&args, \"fsmonitor--daemon\");\n> +\tstrvec_push(&args, \"run\");\n> +\tstrvec_pushf(&args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n> +\n> +\t*pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);\n> +\tclose(in);\n> +\tclose(out);\n> +\n> +\tstrvec_clear(&args);\n> +\n> +\tif (*pid < 0)\n> +\t\treturn error(_(\"could not spawn fsmonitor--daemon in the background\"));\n> +\n> +\treturn 0;\n> +}\n> +#else\n> +/*\n> + * Create a background process to run the daemon. It should be completely\n> + * disassociated from the terminal.\n> + *\n> + * This is adapted from `daemonize()`. Use `fork()` to directly\n> + * create and run the daemon in the child process.\n> + *\n> + * The fork-child can just call the run code; it does not need to exec\n> + * it.\n> + *\n> + * The fork-parent returns the child PID so that we can wait for the\n> + * child to startup before exiting.\n> + */\n> +static int spawn_background_fsmonitor_daemon(pid_t *pid)\n> +{\n> +\t*pid = fork();\n> +\n> +\tswitch (*pid) {\n> +\tcase 0:\n> +\t\tif (setsid() == -1)\n> +\t\t\terror_errno(_(\"setsid failed\"));\n> +\t\tclose(0);\n> +\t\tclose(1);\n> +\t\tclose(2);\n> +\t\tsanitize_stdfds();\n> +\n> +\t\treturn !!fsmonitor_run_daemon();\n> +\n> +\tcase -1:\n> +\t\treturn error_errno(_(\"could not spawn fsmonitor--daemon in the background\"));\n> +\n> +\tdefault:\n> +\t\treturn 0;\n> +\t}\n> +}\n> +#endif\n\nThe spawn_background_fsmonitor_daemon() function here is almost the same\nas daemonize(). I wonder if this & the Windows-specific one you have\nhere can't be refactored into an API from what's now in setup.c.\n\nThen we could make builtin/gc.c and daemon.c use that, so Windows could\nhave background GC, and we'd have a more battle-tested central codepath\nfor this tricky bit.\n\nIt seems to me like the only limitations on it are to have this return\nslightly more general things (e.g. not set its own errors, return\nstructured data), and maybe some callback for what to do in the\nchild/parent.\n\n> +/*\n> + * This is adapted from `wait_or_whine()`. Watch the child process and\n> + * let it get started and begin listening for requests on the socket\n> + * before reporting our success.\n> + */\n> +static int wait_for_background_startup(pid_t pid_child)\n> +{\n> +\tint status;\n> +\tpid_t pid_seen;\n> +\tenum ipc_active_state s;\n> +\ttime_t time_limit, now;\n> +\n> +\ttime(&time_limit);\n> +\ttime_limit += fsmonitor__start_timeout_sec;\n> +\n> +\tfor (;;) {\n> +\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n> +\n> +\t\tif (pid_seen == -1)\n> +\t\t\treturn error_errno(_(\"waitpid failed\"));\n> +\t\telse if (pid_seen == 0) {\n> +\t\t\t/*\n> +\t\t\t * The child is still running (this should be\n> +\t\t\t * the normal case). Try to connect to it on\n> +\t\t\t * the socket and see if it is ready for\n> +\t\t\t * business.\n> +\t\t\t *\n> +\t\t\t * If there is another daemon already running,\n> +\t\t\t * our child will fail to start (possibly\n> +\t\t\t * after a timeout on the lock), but we don't\n> +\t\t\t * care (who responds) if the socket is live.\n> +\t\t\t */\n> +\t\t\ts = fsmonitor_ipc__get_state();\n> +\t\t\tif (s == IPC_STATE__LISTENING)\n> +\t\t\t\treturn 0;\n> +\n> +\t\t\ttime(&now);\n> +\t\t\tif (now > time_limit)\n> +\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n> +\t\t} else if (pid_seen == pid_child) {\n> +\t\t\t/*\n> +\t\t\t * The new child daemon process shutdown while\n> +\t\t\t * it was starting up, so it is not listening\n> +\t\t\t * on the socket.\n> +\t\t\t *\n> +\t\t\t * Try to ping the socket in the odd chance\n> +\t\t\t * that another daemon started (or was already\n> +\t\t\t * running) while our child was starting.\n> +\t\t\t *\n> +\t\t\t * Again, we don't care who services the socket.\n> +\t\t\t */\n> +\t\t\ts = fsmonitor_ipc__get_state();\n> +\t\t\tif (s == IPC_STATE__LISTENING)\n> +\t\t\t\treturn 0;\n> +\n> +\t\t\t/*\n> +\t\t\t * We don't care about the WEXITSTATUS() nor\n> +\t\t\t * any of the WIF*(status) values because\n> +\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n> +\t\t\t * trick on all function return values.\n> +\t\t\t *\n> +\t\t\t * So it is sufficient to just report the\n> +\t\t\t * early shutdown as an error.\n> +\t\t\t */\n> +\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n> +\t\t} else\n> +\t\t\treturn error(_(\"waitpid is confused\"));\n> +\t}\n> +}\n\nDitto this. could we extend the wait_or_whine() function (or some\nextended version thereof) to do what you need with callbacks?\n\nIt seems the main difference is just being able to pass down a flag for\nwaitpid(), and the loop needing to check EINTR or not depending on\nwhether WNOHANG is passed.\n\nFor e.g. the \"We don't care about the WEXITSTATUS()\" you'd get that\nbehavior with an adjusted wait_or_whine(). Wouldn't it be better to\nreport what exit status it exits with e.g. if the top-level process is\nsignalled? We do so in trace2 for other things we spawn...\n"},{"id":"428995","messageId":"877di9d5uz.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"5db2c0390a657d5790cf24404201505c4ec3a829.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 02/34] fsmonitor--daemon: man page","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T22:29:02Z","receivedAt":"2021-07-01T22:30:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Create a manual page describing the `git fsmonitor--daemon` feature.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n> 1 file changed, 75 insertions(+)\n> create mode 100644 Documentation/git-fsmonitor--daemon.txt\n>\n> diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\n> new file mode 100644\n> index 00000000000..154e7684daa\n> --- /dev/null\n> +++ b/Documentation/git-fsmonitor--daemon.txt\n> @@ -0,0 +1,75 @@\n> +git-fsmonitor--daemon(1)\n> +========================\n> +\n> +NAME\n> +----\n> +git-fsmonitor--daemon - A Built-in File System Monitor\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git fsmonitor--daemon' start\n> +'git fsmonitor--daemon' run\n> +'git fsmonitor--daemon' stop\n> +'git fsmonitor--daemon' status\n> +\n> +DESCRIPTION\n> +-----------\n> +\n> +A daemon to watch the working directory for file and directory\n> +changes using platform-specific file system notification facilities.\n> +\n> +This daemon communicates directly with commands like `git status`\n> +using the link:technical/api-simple-ipc.html[simple IPC] interface\n> +instead of the slower linkgit:githooks[5] interface.\n> +\n> +This daemon is built into Git so that no third-party tools are\n> +required.\n> +\n> +OPTIONS\n> +-------\n> +\n> +start::\n> +\tStarts a daemon in the background.\n> +\n> +run::\n> +\tRuns a daemon in the foreground.\n> +\n> +stop::\n> +\tStops the daemon running in the current working\n> +\tdirectory, if present.\n> +\n> +status::\n> +\tExits with zero status if a daemon is watching the\n> +\tcurrent working directory.\n> +\n> +REMARKS\n> +-------\n> +\n> +This daemon is a long running process used to watch a single working\n> +directory and maintain a list of the recently changed files and\n> +directories. Performance of commands such as `git status` can be\n> +increased if they just ask for a summary of changes to the working\n> +directory and can avoid scanning the disk.\n> +\n> +When `core.useBuiltinFSMonitor` is set to `true` (see\n> +linkgit:git-config[1]) commands, such as `git status`, will ask the\n> +daemon for changes and automatically start it (if necessary).\n> +\n> +For more information see the \"File System Monitor\" section in\n> +linkgit:git-update-index[1].\n> +\n> +CAVEATS\n> +-------\n> +\n> +The fsmonitor daemon does not currently know about submodules and does\n> +not know to filter out file system events that happen within a\n> +submodule. If fsmonitor daemon is watching a super repo and a file is\n> +modified within the working directory of a submodule, it will report\n> +the change (as happening against the super repo). However, the client\n> +will properly ignore these extra events, so performance may be affected\n> +but it will not cause an incorrect result.\n> +\n> +GIT\n> +---\n> +Part of the linkgit:git[1] suite\n\nLater in the series we incrementally add features to the daemon, so this\nis describing a state that doesn't exist yet at this point.\n\nI think it would be better to start with a stup here and add\ndocumentation as we add features, e.g. the patch tha adds \"start\" should\nadd that to the synopsis + options etc.\n\nSee the outstanding ab/config-based-hooks-base for a small example of\nthat.\n"},{"id":"428996","messageId":"874kddd5sm.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"86413bfe34722dcd5c846cf7326340473471773c.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 03/34] fsmonitor--daemon: update fsmonitor documentation","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T22:31:57Z","receivedAt":"2021-07-01T22:32:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Update references to `core.fsmonitor` and `core.fsmonitorHookVersion` and\n> pointers to `Watchman` to mention the new built-in `fsmonitor--daemon`.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> Documentation/config/core.txt | 56 ++++++++++++++++++++++--------\n> Documentation/git-update-index.txt | 27 +++++++-------\n> Documentation/githooks.txt | 3 +-\n> 3 files changed, 59 insertions(+), 27 deletions(-)\n>\n> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\n> index c04f62a54a1..4f6e519bc02 100644\n> --- a/Documentation/config/core.txt\n> +++ b/Documentation/config/core.txt\n> @@ -62,22 +62,50 @@ core.protectNTFS::\n> \tDefaults to `true` on Windows, and `false` elsewhere.\n> \n> core.fsmonitor::\n> -\tIf set, the value of this variable is used as a command which\n> -\twill identify all files that may have changed since the\n> -\trequested date/time. This information is used to speed up git by\n> -\tavoiding unnecessary processing of files that have not changed.\n> -\tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n> +\tIf set, this variable contains the pathname of the \"fsmonitor\"\n> +\thook command.\n> ++\n> +This hook command is used to identify all files that may have changed\n> +since the requested date/time. This information is used to speed up\n> +git by avoiding unnecessary scanning of files that have not changed.\n> ++\n> +See the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n> ++\n> +Note: The value of this config setting is ignored if the\n> +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n> \n> core.fsmonitorHookVersion::\n> -\tSets the version of hook that is to be used when calling fsmonitor.\n> -\tThere are currently versions 1 and 2. When this is not set,\n> -\tversion 2 will be tried first and if it fails then version 1\n> -\twill be tried. Version 1 uses a timestamp as input to determine\n> -\twhich files have changes since that time but some monitors\n> -\tlike watchman have race conditions when used with a timestamp.\n> -\tVersion 2 uses an opaque string so that the monitor can return\n> -\tsomething that can be used to determine what files have changed\n> -\twithout race conditions.\n> +\tSets the protocol version to be used when invoking the\n> +\t\"fsmonitor\" hook.\n> ++\n> +There are currently versions 1 and 2. When this is not set,\n> +version 2 will be tried first and if it fails then version 1\n> +will be tried. Version 1 uses a timestamp as input to determine\n> +which files have changes since that time but some monitors\n> +like Watchman have race conditions when used with a timestamp.\n> +Version 2 uses an opaque string so that the monitor can return\n> +something that can be used to determine what files have changed\n> +without race conditions.\n> ++\n> +Note: The value of this config setting is ignored if the\n> +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`).\n> +\n> +core.useBuiltinFSMonitor::\n> +\tIf set to true, enable the built-in file system monitor\n> +\tdaemon for this working directory (linkgit:git-fsmonitor--daemon[1]).\n> ++\n> +Like hook-based file system monitors, the built-in file system monitor\n> +can speed up Git commands that need to refresh the Git index\n> +(e.g. `git status`) in a working directory with many files. The\n> +built-in monitor eliminates the need to install and maintain an\n> +external third-party tool.\n> ++\n> +The built-in file system monitor is currently available only on a\n> +limited set of supported platforms. Currently, this includes Windows\n> +and MacOS.\n> ++\n> +Note: if this config setting is set to `true`, the values of\n> +`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored.\n> \n> core.trustctime::\n> \tIf false, the ctime differences between the index and the\n> diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\n> index 2853f168d97..c7c31b3fcf9 100644\n> --- a/Documentation/git-update-index.txt\n> +++ b/Documentation/git-update-index.txt\n> @@ -498,7 +498,9 @@ FILE SYSTEM MONITOR\n> This feature is intended to speed up git operations for repos that have\n> large working directories.\n> \n> -It enables git to work together with a file system monitor (see the\n> +It enables git to work together with a file system monitor (see\n> +linkgit:git-fsmonitor--daemon[1]\n> +and the\n> \"fsmonitor-watchman\" section of linkgit:githooks[5]) that can\n> inform it as to what files have been modified. This enables git to avoid\n> having to lstat() every file to find modified files.\n> @@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory\n> looking for new files.\n> \n> If you want to enable (or disable) this feature, it is easier to use\n> -the `core.fsmonitor` configuration variable (see\n> -linkgit:git-config[1]) than using the `--fsmonitor` option to\n> -`git update-index` in each repository, especially if you want to do so\n> -across all repositories you use, because you can set the configuration\n> -variable in your `$HOME/.gitconfig` just once and have it affect all\n> -repositories you touch.\n> -\n> -When the `core.fsmonitor` configuration variable is changed, the\n> -file system monitor is added to or removed from the index the next time\n> -a command reads the index. When `--[no-]fsmonitor` are used, the file\n> -system monitor is immediately added to or removed from the index.\n> +the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n> +variable (see linkgit:git-config[1]) than using the `--fsmonitor`\n> +option to `git update-index` in each repository, especially if you\n> +want to do so across all repositories you use, because you can set the\n> +configuration variable in your `$HOME/.gitconfig` just once and have\n> +it affect all repositories you touch.\n> +\n> +When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration\n> +variable is changed, the file system monitor is added to or removed\n> +from the index the next time a command reads the index. When\n> +`--[no-]fsmonitor` are used, the file system monitor is immediately\n> +added to or removed from the index.\n> \n> CONFIGURATION\n> -------------\n> diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\n> index b51959ff941..b7d5e926f7b 100644\n> --- a/Documentation/githooks.txt\n> +++ b/Documentation/githooks.txt\n> @@ -593,7 +593,8 @@ fsmonitor-watchman\n> \n> This hook is invoked when the configuration option `core.fsmonitor` is\n> set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2`\n> -depending on the version of the hook to use.\n> +depending on the version of the hook to use, unless overridden via\n> +`core.useBuiltinFSMonitor` (see linkgit:git-config[1]).\n> \n> Version 1 takes two arguments, a version (1) and the time in elapsed\n> nanoseconds since midnight, January 1, 1970.\n\nDitto the comment on 02/34, mostly docs for things that don't exist\nuntil in later patches.\n"},{"id":"428997","messageId":"871r8hd5ew.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"f88db92d4259d1c29827e97e957daf6eda39c551.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 08/34] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T22:36:57Z","receivedAt":"2021-07-01T22:41:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\nA general comment on this series (including previous patches). We've\nusually tried to bend over backwards in git's codebase not to have big\nifdef blocks, so we compile most code the same everywhere. We waste a\nbit of object code, but that's fine.\n\nSee 9c897c5c2ad (pack-objects: remove #ifdef NO_PTHREADS, 2018-11-03)\nfor a good exmaple of bad code being turned to good.\n\nE.g. in this case:\n\n> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> +\n> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> +{\n> +\tconst char *subcmd;\n> +\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\tif (argc < 2)\n> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n> +\n> +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n> +\n> +\tgit_config(git_default_config, NULL);\n> +\n> +\tsubcmd = argv[1];\n> +\targv--;\n> +\targc++;\n> +\n> +\targc = parse_options(argc, argv, prefix, options,\n> +\t\t\t builtin_fsmonitor__daemon_usage, 0);\n> +\n> +\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n> +}\n> +\n> +#else\n> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstruct option options[] = {\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n> +\n> +\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n> +}\n> +#endif\n\nThis whole thing could really just be a\n-DHAVE_FSMONITOR_DAEMON_BACKEND=1 or -DHAVE_FSMONITOR_DAEMON_BACKEND=0\nsomewhere (depending), and then somewhere in the middle of the first\nfunction:\n\n\tif (!HAVE_FSMONITOR_DAEMON_BACKEND)\n\t \tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n"},{"id":"428998","messageId":"87y2apbqqk.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"c2adac8ed4b4f324b54b147e1fcb1aa4a74e80dd.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 10/34] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T22:41:22Z","receivedAt":"2021-07-01T22:42:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Create an IPC client to send query and flush commands to the daemon.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> Makefile | 1 +\n> t/helper/test-fsmonitor-client.c | 121 +++++++++++++++++++++++++++++++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> 4 files changed, 124 insertions(+)\n> create mode 100644 t/helper/test-fsmonitor-client.c\n>\n> diff --git a/Makefile b/Makefile\n> index 8fe1e42a435..c45caacf2c3 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n> TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n> TEST_BUILTINS_OBJS += test-example-decorate.o\n> TEST_BUILTINS_OBJS += test-fast-rebase.o\n> +TEST_BUILTINS_OBJS += test-fsmonitor-client.o\n> TEST_BUILTINS_OBJS += test-genrandom.o\n> TEST_BUILTINS_OBJS += test-genzeros.o\n> TEST_BUILTINS_OBJS += test-hash-speed.o\n> diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\n> new file mode 100644\n> index 00000000000..f7a5b3a32fa\n> --- /dev/null\n> +++ b/t/helper/test-fsmonitor-client.c\n> @@ -0,0 +1,121 @@\n> +/*\n> + * test-fsmonitor-client.c: client code to send commands/requests to\n> + * a `git fsmonitor--daemon` daemon.\n> + */\n> +\n> +#include \"test-tool.h\"\n> +#include \"cache.h\"\n> +#include \"parse-options.h\"\n> +#include \"fsmonitor-ipc.h\"\n> +\n> +#ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n> +int cmd__fsmonitor_client(int argc, const char **argv)\n> +{\n> +\tdie(\"fsmonitor--daemon not available on this platform\");\n> +}\n> +#else\n\nRe my earlier comments on excessive ifdefs: In this case don't we just\nwant to not compile test-fsmonitor-client at all unless\nHAVE_FSMONITOR_DAEMON_BACKEND is true.\n\nYou'll get the same error as though you ran \"helper/test-tool\ndoes-not-exist\", but the tests check for the prerequisite earlier\nanyway, so why get this far on an unsupported platform for a pure test\nhelper?\n"},{"id":"428999","messageId":"87v95tbqgh.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"5a9bda7220356ebf0689bb6aaa9068520dc6e33b.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T22:45:45Z","receivedAt":"2021-07-01T22:48:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Stub in empty backend for fsmonitor--daemon on Windows.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> Makefile | 13 ++++++\n> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++\n> compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++\n> config.mak.uname | 2 +\n> contrib/buildsystems/CMakeLists.txt | 5 ++\n> 5 files changed, 90 insertions(+)\n> create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n>\n> diff --git a/Makefile b/Makefile\n> index c45caacf2c3..a2a6e1f20f6 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -467,6 +467,11 @@ all::\n> # directory, and the JSON compilation database 'compile_commands.json' will be\n> # created at the root of the repository.\n> #\n> +# If your platform supports a built-in fsmonitor backend, set\n> +# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n> +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n> +# `fsmonitor_fs_listen__*()` routines.\n> +#\n> # Define DEVELOPER to enable more compiler warnings. Compiler version\n> # and family are auto detected, but could be overridden by defining\n> # COMPILER_FEATURES (see config.mak.dev). You can still set\n> @@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n> \tCOMPAT_OBJS += compat/access.o\n> endif\n> \n> +ifdef FSMONITOR_DAEMON_BACKEND\n> +\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n> +\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n> +endif\n> +\n> ifeq ($(TCLTK_PATH),)\n> NO_TCLTK = NoThanks\n> endif\n> @@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE\n> \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n> \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n> \t@echo X=\\'$(X)\\' >>$@+\n> +ifdef FSMONITOR_DAEMON_BACKEND\n> +\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n> +endif\n\nWhy put this in an ifdef?\n\nIn 342e9ef2d9e (Introduce a performance testing framework, 2012-02-17)\nwe started doing that for some perf/test options (which b.t.w., I don't\nreally see the reason for, maybe it's some subtlety in how test-lib.sh\npicks those up).\n\nBut for all the other compile-time stuff we don't ifdef it, we just\ndefine it, and then you get an empty value or not.\n\nThis would AFAICT be the first build-time-for-the-C-program option we\nifdef for writing a line to GIT-BUILD-OPTIONS.\n"},{"id":"429001","messageId":"87sg0xbq9v.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"587580489473a7a2ad665bdf3c482ea5d2c54f61.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T22:49:26Z","receivedAt":"2021-07-01T22:52:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Stub in empty implementation of fsmonitor--daemon\n> backend for MacOS.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++\n> config.mak.uname | 2 ++\n> contrib/buildsystems/CMakeLists.txt | 3 +++\n> 3 files changed, 25 insertions(+)\n> create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c\n>\n> diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n> new file mode 100644\n> index 00000000000..b91058d1c4f\n> --- /dev/null\n> +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n> @@ -0,0 +1,20 @@\n> +#include \"cache.h\"\n> +#include \"fsmonitor.h\"\n> +#include \"fsmonitor-fs-listen.h\"\n> +\n> +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n> +{\n> +\treturn -1;\n> +}\n> +\n> +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n> +{\n> +}\n> +\n> +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n> +{\n> +}\n> +\n> +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n> +{\n> +}\n> diff --git a/config.mak.uname b/config.mak.uname\n> index fcd88b60b14..394355463e1 100644\n> --- a/config.mak.uname\n> +++ b/config.mak.uname\n> @@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin)\n> \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n> \t\tendif\n> \tendif\n> +\tFSMONITOR_DAEMON_BACKEND = macos\n\nA rather trivial point, but can't we pick one of \"macos\" or \"darwin\"\n(I'd think going with the existing uname is better) and name the file\nafter the uname (or lower-case thereof)?\n\nMakes these make rules more consistent too, we could just set this to\n\"YesPlease\" here, and then lower case the uname for the file\ncompilation/include.\n"},{"id":"429002","messageId":"87pmw1bq64.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"eef39aa168fec58321ee233d907db696b56ee0b8.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 15/34] fsmonitor: do not try to operate on bare repos","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T22:53:31Z","receivedAt":"2021-07-01T22:55:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Bare repos do not have a working directory, so there is no\n> directory for the daemon to register a watch upon. And therefore\n> there are no files within the directory for it to actually watch.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> builtin/fsmonitor--daemon.c | 8 ++++++++\n> t/t7519-status-fsmonitor.sh | 16 ++++++++++++++++\n> 2 files changed, 24 insertions(+)\n>\n> diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n> index 7fcf960652f..d6161ad95a5 100644\n> --- a/builtin/fsmonitor--daemon.c\n> +++ b/builtin/fsmonitor--daemon.c\n> @@ -490,6 +490,14 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n> \t\t fsmonitor__ipc_threads);\n> \n> +\tprepare_repo_settings(the_repository);\n> +\tif (!the_repository->worktree)\n> +\t\treturn error(_(\"fsmonitor-daemon does not support bare repos '%s'\"),\n> +\t\t\t xgetcwd());\n> +\tif (the_repository->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE)\n> +\t\treturn error(_(\"fsmonitor-daemon is incompatible with this repo '%s'\"),\n> +\t\t\t the_repository->worktree);\n\nI commented on another patch that that second condition seems like it's\nonly hit under core.virtualfilesystem=true....\n\n\n> +test_expect_success FSMONITOR_DAEMON 'try running fsmonitor-daemon in bare repo' '\n> +\ttest_when_finished \"rm -rf ./bare-clone\" &&\n> +\tgit clone --bare . ./bare-clone &&\n> +\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n> +\tgrep \"fsmonitor-daemon does not support bare repos\" actual\n> +'\n\nIsn't just:\n\n git init --bare bare.git\n test_must_fail git -C bare.git [...]\n\nEnough, or does the repository need content to get to that error.\n\n> +test_expect_success FSMONITOR_DAEMON 'try running fsmonitor-daemon in virtual repo' '\n> +\ttest_when_finished \"rm -rf ./fake-virtual-clone\" &&\n> +\tgit clone . ./fake-virtual-clone &&\n> +\ttest_must_fail git -C ./fake-virtual-clone \\\n> +\t\t\t -c core.virtualfilesystem=true \\\n> +\t\t\t fsmonitor--daemon run 2>actual &&\n> +\tgrep \"fsmonitor-daemon is incompatible with this repo\" actual\n> +'\n> +\n\nDitto. \n"},{"id":"429003","messageId":"87mtr5bpuw.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"37fdce5ec3afaa9aae5001c648fced0675dae0c4.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 17/34] fsmonitor--daemon: define token-ids","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T22:58:28Z","receivedAt":"2021-07-01T23:01:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> +\tif (!test_env_value) {\n> +\t\tstruct timeval tv;\n> +\t\tstruct tm tm;\n> +\t\ttime_t secs;\n> +\n> +\t\tgettimeofday(&tv, NULL);\n> +\t\tsecs = tv.tv_sec;\n> +\t\tgmtime_r(&secs, &tm);\n> +\n> +\t\tstrbuf_addf(&token->token_id,\n> +\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n> +\t\t\t flush_count++,\n> +\t\t\t getpid(),\n> +\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n> +\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n> +\t\t\t (long)tv.tv_usec);\n\nJust bikeshedding, but can we have tokens that mostly sort numeric-wise\nby time order? So time at the start, not the flush_count/getpid.\n\nMaybe I'm missing something, but couldn't we just re-use the trace2 SID\n+ a more trivial trailer? It would have the nice property that you could\nfind the trace2 SID whenever you looked at such a token (could\ne.g. split them by \"/\" too), and add the tv_usec, flush_count+whatever\nelse is needed to make it unique after the \"/\", no?\n"},{"id":"429004","messageId":"87k0m9bpmv.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"5bba5eb3d1bd172f09fdf6eb2e9b8ac4dd7f940f.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T23:02:34Z","receivedAt":"2021-07-01T23:06:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Teach the win32 backend to register a watch on the working tree\n> root directory (recursively). Also watch the if it is\n> not inside the working tree. And to collect path change notifications\n> into batches and publish.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++\n\n Spying on the early history of this (looking for the\nLinux backend) I saw that at some point we had just\ncompat/fsmonitor/linux.c, and presumably some of\ncompat/fsmonitor/{windows,win32,macos,darwin}.c.\n\nAt some point those filenames became much much longer.\n\nI've noticed you tend to prefer really long file and function names,\ne.g. your borrowed daemonize() became\nspawn_background_fsmonitor_daemon(), I think aiming for shorter\nfilenames & function names helps, e.g. these long names widen diffstats,\nand many people who hack on the code stick religiously to 80 character\nwidth terminals.\n"},{"id":"429005","messageId":"87h7hdbpgb.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"f1ef9656fc3adf079c8e40a74baeb5356bcf1586.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T23:09:04Z","receivedAt":"2021-07-01T23:10:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Change p7519 to use a single \"test-tool touch\" command to update\n> the mtime on a series of (thousands) files instead of invoking\n> thousands of commands to update a single file.\n>\n> This is primarily for Windows where process creation is so\n> very slow and reduces the test run time by minutes.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> t/perf/p7519-fsmonitor.sh | 14 ++++++--------\n> 1 file changed, 6 insertions(+), 8 deletions(-)\n>\n> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n> index 5eb5044a103..f74e6014a0a 100755\n> --- a/t/perf/p7519-fsmonitor.sh\n> +++ b/t/perf/p7519-fsmonitor.sh\n> @@ -119,10 +119,11 @@ test_expect_success \"one time repo setup\" '\n> \tfi &&\n> \n> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n> -\tfor i in $(test_seq 1 10); do touch 10_files/$i; done &&\n> -\tfor i in $(test_seq 1 100); do touch 100_files/$i; done &&\n> -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i; done &&\n> -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n> +\ttest-tool touch sequence --pattern=\"10_files/%d\" --start=1 --count=10 &&\n> +\ttest-tool touch sequence --pattern=\"100_files/%d\" --start=1 --count=100 &&\n> +\ttest-tool touch sequence --pattern=\"1000_files/%d\" --start=1 --count=1000 &&\n> +\ttest-tool touch sequence --pattern=\"10000_files/%d\" --start=1 --count=10000 &&\n> +\n> \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n> \tgit commit -qm \"Add files\" &&\n> \n> @@ -200,15 +201,12 @@ test_fsmonitor_suite() {\n> \t# Update the mtimes on upto 100k files to make status think\n> \t# that they are dirty. For simplicity, omit any files with\n> \t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n> -\t# Then fully backslash-quote the paths to capture any\n> -\t# whitespace so that they pass thru xargs properly.\n> \t#\n> \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n> \t\tgit ls-files | \\\n> \t\t\thead -100000 | \\\n> \t\t\tgrep -v \\\" | \\\n> -\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n> -\t\t\txargs test-tool chmtime -300 &&\n> +\t\t\ttest-tool touch stdin &&\n> \t\tgit status\n> \t'\n\nDid you try to replace this with some variant of:\n\n test_seq 1 10000 | xargs touch\n\nWhich (depending on your xargs version) would invoke \"touch\" commands\nwith however many argv items it thinks you can handle.\n"},{"id":"429006","messageId":"87eechbpcb.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"a83485fb10f57326a725579f329b73ebf9240ac6.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 25/34] t/perf: avoid copying builtin fsmonitor files into test repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T23:11:26Z","receivedAt":"2021-07-01T23:12:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Do not try to copy a fsmonitor--daemon socket from the current\n> development directory into the test trash directory.\n\nOkey, the */fsmonitor--daemon* rule covers that...\n\n> When we run the perf suite without an explicit source repo set,\n> we copy of the current $GIT_DIR into the test trash directory.\n> Unix domain sockets cannot be copied in that manner, so the test\n> setup fails.\n>\n> Additionally, omit any other fsmonitor--daemon temp files inside\n> the $GIT_DIR directory.\n\nSo is the \"any other\" also matched by that rule? Not knowing the files\npart of this is just phrasing, would be less confusing (if that's true,\nand you didn't just forget to add a match for them) as:\n\n The */fsmonitor--daemon* glob will also match temporary files the\n daemon creates, but that's OK. We'd like to ignore these too.\n"},{"id":"429007","messageId":"87bl7lbp72.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"99279c0ebd2b36fe276301f018ca4e24e4cdff36.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 27/34] t7527: create test for fsmonitor--daemon","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T23:15:13Z","receivedAt":"2021-07-01T23:16:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> +\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_1\n> +'\n> +\n> +test_expect_success 'status implicitly starts daemon' '\n> +\ttest_when_finished redundant_stop_daemon &&\n> +\n> +\ttest_must_fail git fsmonitor--daemon status &&\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/.git/trace_implicit_2\" \\\n> +\t\tgit status >actual &&\n> +\n> +\tgit fsmonitor--daemon status &&\n> +\ttest_might_fail git fsmonitor--daemon stop &&\n> +\n> +\tgrep \\\"event\\\":\\\"start\\\".*\\\"fsmonitor--daemon\\\" .git/trace_implicit_2\n> +'\n\nSeems like this and test_region could eventually be factored into some\ncommon function that would be more flexible about grabbing trace2 data.\n"},{"id":"429008","messageId":"878s2pboha.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"555caca2216dd3e459c118d76b46eb983a58e051.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 29/34] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T23:17:30Z","receivedAt":"2021-07-01T23:31:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Teach fsmonitor--daemon client threads to create a cookie file\n> inside the .git directory and then wait until FS events for the\n> cookie are observed by the FS listener thread.\n>\n> This helps address the racy nature of file system events by\n> blocking the client response until the kernel has drained any\n> event backlog.\n>\n> This is especially important on MacOS where kernel events are\n> only issued with a limited frequency. See the `latency` argument\n> of `FSeventStreamCreate()`. The kernel only signals every `latency`\n> seconds, but does not guarantee that the kernel queue is completely\n> drained, so we may have to wait more than one interval. If we\n> increase the frequency, the system is more likely to drop events.\n> We avoid these issues by having each client thread create a unique\n> cookie file and then wait until it is seen in the event stream.\n\nIs this a guaranteed property of any API fsmonitor might need to work\nwith (linux, darwin, Windows) that if I perform a bunch of FS operations\non my working tree, that if I finish up by touching this cookie file\nthat that'll happen last?\n\nI'd think that wouldn't be the case, i.e. on POSIX filesystems unless\nyou run around fsyncing both files and directories you're not guaranteed\nthat they're on disk, and even then the kernel might decide to sync your\ncookie earlier, won't it?\n\nE.g. on Linux you can even have cross-FS watches, and mix & match\ndifferent FS types. I'd expect to get events in whatever\nimplementation-defined order the VFS layer + FS decided to sync them to\ndisk in & get to firing off an event for me.\n\nOr do these APIs all guarantee that a linear view of the world is\npresented to the API consumer?\n\n\n> Co-authored-by: Kevin Willford \n> Co-authored-by: Johannes Schindelin \n> Signed-off-by: Jeff Hostetler \n> ---\n> builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++-\n> fsmonitor--daemon.h | 5 +\n> 2 files changed, 232 insertions(+), 1 deletion(-)\n>\n> diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n> index 8249420ba18..25f18f2726b 100644\n> --- a/builtin/fsmonitor--daemon.c\n> +++ b/builtin/fsmonitor--daemon.c\n> @@ -94,6 +94,149 @@ static int do_as_client__status(void)\n> \t}\n> }\n> \n> +enum fsmonitor_cookie_item_result {\n> +\tFCIR_ERROR = -1, /* could not create cookie file ? */\n> +\tFCIR_INIT = 0,\n> +\tFCIR_SEEN,\n> +\tFCIR_ABORT,\n> +};\n> +\n> +struct fsmonitor_cookie_item {\n> +\tstruct hashmap_entry entry;\n> +\tconst char *name;\n> +\tenum fsmonitor_cookie_item_result result;\n> +};\n> +\n> +static int cookies_cmp(const void *data, const struct hashmap_entry *he1,\n> +\t\t const struct hashmap_entry *he2, const void *keydata)\n> +{\n> +\tconst struct fsmonitor_cookie_item *a =\n> +\t\tcontainer_of(he1, const struct fsmonitor_cookie_item, entry);\n> +\tconst struct fsmonitor_cookie_item *b =\n> +\t\tcontainer_of(he2, const struct fsmonitor_cookie_item, entry);\n\nRe earlier comments about verbose names, these are just enums in a\nbuiltin/*.c file, so a name like \"cookie_item\" is fine, and then the\nwhole thing might even fit on one line.. :)\n\n> [...]\n> +\t/*\n> +\t * We will write filesystem syncing cookie files into\n> +\t * ///-.\n> +\t *\n> +\t * The extra layers of subdirectories here keep us from\n> +\t * changing the mtime on \".git/\" or \".git/foo/\" when we create\n> +\t * or delete cookie files.\n> +\t *\n> +\t * There have been problems with some IDEs that do a\n> +\t * non-recursive watch of the \".git/\" directory and run a\n> +\t * series of commands any time something happens.\n> +\t *\n> +\t * For example, if we place our cookie files directly in\n> +\t * \".git/\" or \".git/foo/\" then a `git status` (or similar\n> +\t * command) from the IDE will cause a cookie file to be\n> +\t * created in one of those dirs. This causes the mtime of\n> +\t * those dirs to change. This triggers the IDE's watch\n> +\t * notification. This triggers the IDE to run those commands\n> +\t * again. And the process repeats and the machine never goes\n> +\t * idle.\n> +\t *\n> +\t * Adding the extra layers of subdirectories prevents the\n> +\t * mtime of \".git/\" and \".git/foo\" from changing when a\n> +\t * cookie file is created.\n> +\t */\n> +\tstrbuf_init(&state.path_cookie_prefix, 0);\n> +\tstrbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch);\n> +\n> +\tstrbuf_addch(&state.path_cookie_prefix, '/');\n> +\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR);\n> +\tmkdir(state.path_cookie_prefix.buf, 0777);\n> +\n> +\tstrbuf_addch(&state.path_cookie_prefix, '/');\n> +\tstrbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR);\n> +\tmkdir(state.path_cookie_prefix.buf, 0777);\n> +\n> +\tstrbuf_addch(&state.path_cookie_prefix, '/');\n\nSo, on some stupid IDEs (would be nice to have specifics in the commit\nmessage, which ones/versions?) this avoids causing infinite activity,\nbut on slightly more industrious stupid IDEs that would do their own\nrecursive watch we'll have the same problem?\n\nPerhaps we should just consider creating it at the top-level and those\nIDEs will eventually sort out their bugs, sooner than later if this\nfeature ships...\n\n> +\tstrbuf_release(&state.path_cookie_prefix);\n> +\n> +\t/*\n> +\t * NEEDSWORK: Consider \"rm -rf /\"\n> +\t */\n\nThat would also make this trivial, presumably it's a \"needswork\" since\nyou have this recursive structure, but if it's at the top-level we\nalready did the unlink() above, so NEEDSWORK solved then?\n"},{"id":"429009","messageId":"875yxtbo3z.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"627e27fe60bb543902fdcc4b2179c620403d9c38.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 34/34] t7527: test FS event reporing on MacOS WRT case and Unicode","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-01T23:39:09Z","receivedAt":"2021-07-01T23:39:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n\n> +\t[ -d test_apfs/.git ] &&\n> +\t[ -d test_apfs/.GIT ] &&\n\nBetter as \"test_path_is_dir\".\n"},{"id":"429066","messageId":"9ac45626-f995-a661-2746-6711f996e5a6@jeffhostetler.com","threadId":"55420","inReplyTo":"87czs1d6uy.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-02T19:06:32Z","receivedAt":"2021-07-02T19:06:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 5:26 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler wrote:\n> \n>> On 7/1/21 1:40 PM, Ævar Arnfjörð Bjarmason wrote:\n>>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>>>\n>>>> Here is V3 of my patch series to add a builtin FSMonitor daemon to Git. I\n>>>> rebased this series onto v2.32.0.\n>>>>\n>>>> V3 addresses most of the previous review comments and things we've learned\n>>>> from our experimental testing of V2. (A version of V2 was shipped as an\n>>>> experimental feature in the v2.32.0-based releases of Git for Windows and\n>>>> VFS for Git.)\n>>>>\n>>>> There are still a few items that I need to address, but that list is getting\n>>>> very short.\n>>> ...\n>>>> fsmonitor-fs-listen-win32: stub in backend for Windows\n>>>> fsmonitor-fs-listen-macos: stub in backend for MacOS\n>>> I left some light comments on the repo-settings.c part of this to\n>>> follow\n>>> up from a previous round.\n>>\n>> Thanks.\n>>\n>>> Any other testing of it is stalled by there being no linux backend\n>>> for\n>>> it as part of this series. I see from spelunking repos that Johannes had\n>>> a WIP compat/fsmonitor/linux.c which looks like it could/should mostly\n>>> work, but the API names all changed since then, and after a short try I\n>>> gave up on trying to rebase it.\n>>\n>> The early Linux version was dropped because inotify does not give\n>> recursive coverage -- only the requested directory. Using inotify\n>> requires adding a watch to each subdirectory (recursively) in the\n>> worktree. There's a system limit on the maximum number of watched\n>> directories (defaults to 8K IIRC) and that limit is system-wide.\n>>\n>> Since the whole point was to support large very large repos, using\n>> inotify was a non-starter, so I removed the Linux version from our\n>> patch series. For example, the first repo I tried it on (outside\n>> of the test suite) had 25K subdirectories.\n>>\n>> I'm told there is a new fanotify API in recent Linux kernels that\n>> is a better fit for what we need, but we haven't had time to investigate\n>> it yet.\n> \n> That default limit is a bit annoying, but I don't see how it's a blocker\n> in any way.\n> \n> You simply adjust the limit. E.g. I deployed and tested the hook version\n> of inotify (using watchman) in a sizable development environment, and\n> written my own code using the API. This was all before fanotify(7)\n> existed. IIRC I set most of the limits to 2^24 or 2^20. I've used it\n> with really large Git repos, including with David Turner's\n> 2015-04-03-1M-git for testing (`git ls-files | wc -l` on that is around\n> a quarter-million).\n> \n> If you have a humongous repository and don't have root on your own\n> machine you're out of luck. But I think most people who'd use this are\n> either using their own laptop, or e.g. in a corporate setting where\n> administrators would tweak the sysctl limits given the performance\n> advantages (as I did).\n> \n> Once you adjust the limits Linux deals with large trees just fine, it\n> just has overly conservative limits for most things in sysctl. The API\n> is a bit annoying, your event loop needs to run around and add watches.\n> \n> AFAICT you need Linux 5.1 for fanotify(7) to be useful, e.g. Debian\n> stable, RHEL etc. aren't using anything that new. So having an inotify\n> backend as well as possibly a fanotify one would be very useful.\n> \n> And linux.git's docs suggest that the default limit was bumped from 8192\n> to 1M in v5.10, a really recent kernel, so if you've got that you've\n> also got fanotify.\n> \n> In any case, even if Linux's inotify limit was something hardcoded and\n> impossible to change you could still use such an API to test & debug the\n> basics of this feature on that platform.\n\nGood points. If the inotify limits can be increased into the millions\nthen we can revisit supporting it. I do worry about possible race\nconditions as we have to scan the worktree and add/delete a watch for\neach directory, but we don't need to worry about that right now.\n\nI do want to have Linux support eventually, but I was saving it for\na later effort (and/or looking for volunteers). My IPC-based builtin\ndaemon complements the existing hook-based fsmonitor support that Ben\nPeart and Kevin Willford added a few years ago. That model (and PERL\nhook script and Watchman integration) work fine for Linux, so the\nadvantages of a builtin monitor aren't as compelling.\n\nFor example, on Linux, hook process creation is fast, PERL is fast,\nand it is easy to just apt-get a tool like Watchman. But on\nWindows, process creation is slow, NTFS is slow, PERL is available\nas part of the Git-for-Windows distribution, and installing third-party\ntools like Watchman onto a collection of enterprise users' machines is\na chore, so it made sense of us to pick platforms where the existing\nhook model has issues and add other platforms later.\n\nBesides, this patch series is already at 34 commits. And some of\nthem are quite large. Adding another platform would just make it\neven larger.\n\nRight now I'm more interested in the larger question of whether\nwe WANT to have a builtin fsmonitor and do we like the overall\ndesign of what I have here? Picking up a new platform, whether\nit is Linux, or AIX, or Solaris, or Nonstop, or whatever, should\nnicely fit into one platform-specific file in compat/fsmonitor\nand not take that long.\n\n> \n>>> I'd really prefer for git not to have features that place free\n>>> platforms\n>>> at a disadvantage against proprietary platforms if it can be avoided,\n>>> and in this case the lack of a Linux backend also means much less\n>>> widespread testing of the feature among the development community / CI.\n>>>\n>>\n>> This feature is always going to have platform-specific components,\n>> so the lack of one platform or another should not stop us from\n>> discussing it for the platforms that can be supported.\n> \n> (I think per the above that's s/can be/are/)\n> \n>> And given the size and complexity of the platform-specific code,\n>> we should not assume that \"just test it on Linux\" is sufficient.\n>> Yes, there are some common/shared routines/data structures in the\n>> daemon, but hard/tricky parts are in the platform layer.\n> \n> I think we're talking past each other a bit here. I'm not saying that\n> you can get full or meaningful testing for it on Windows if you test it\n> on Linux, or the other way around.\n> \n> Of course there's platform-specific stuff, although there's also a lot\n> of non-platform-specific stuff, so even a very basic implementation of\n> inotify would make reviwing this easier / give access to more reviewers.\n> \n> I'm saying that I prefer that Git as a free software project not be in\n> the situation of saying the best use-case for a given size/shape of repo\n> is to use Git in combination with proprietary operating systems over\n> freely licensed ones.\n\nI wouldn't worry about that. Even without Watchman integration,\nLinux runs things so much faster than anything else it's not funny.\n\nIf anything, we need things like fsmonitor on Windows to help keep\nup with Linux.\n\n> \n> IOW what the FSF has a policy for GNU projects. Now, I think the FSF\n> probably goes too far in that (famously removing rather obscure font\n> rendering features from Emacs on OSX), but \"manage lots of data\" is a\n> core feature of git.\n> \n\nJeff\n"},{"id":"429266","messageId":"nycvar.QRO.7.76.6.2107052259340.8230@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"87czs1d6uy.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-05T21:35:39Z","receivedAt":"2021-07-05T21:35:58Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Thu, 1 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Thu, Jul 01 2021, Jeff Hostetler wrote:\n>\n> > On 7/1/21 1:40 PM, Ævar Arnfjörð Bjarmason wrote:\n> >\n> >> Any other testing of it is stalled by there being no linux backend\n> >> for it as part of this series. I see from spelunking repos that\n> >> Johannes had a WIP compat/fsmonitor/linux.c which looks like it\n> >> could/should mostly work, but the API names all changed since then,\n> >> and after a short try I gave up on trying to rebase it.\n\nI am a bit surprised that you gave up so easily, it cannot be that hard if\nyou use `git rebase -i` smartly.\n\nBut I think there is an even bigger obstacle lurking than just the\nchallenge of rebasing that experimental backend.\n\n> > The early Linux version was dropped because inotify does not give\n> > recursive coverage -- only the requested directory. Using inotify\n> > requires adding a watch to each subdirectory (recursively) in the\n> > worktree. There's a system limit on the maximum number of watched\n> > directories (defaults to 8K IIRC) and that limit is system-wide.\n> >\n> > Since the whole point was to support large very large repos, using\n> > inotify was a non-starter, so I removed the Linux version from our\n> > patch series. For example, the first repo I tried it on (outside of\n> > the test suite) had 25K subdirectories.\n> >\n> > I'm told there is a new fanotify API in recent Linux kernels that is a\n> > better fit for what we need, but we haven't had time to investigate it\n> > yet.\n>\n> That default limit is a bit annoying, but I don't see how it's a blocker\n> in any way.\n\nLet me help you to see it.\n\nSo let's assume that you start FSMonitor-enabled Git, with the default\nvalues. What is going to happen if you have any decently-sized worktree?\nYou run out of handles. What then? Throw your hands in the air? Stop\nworking? Report incorrect results?\n\nThose are real design challenges, and together with the race problems Jeff\nmentioned, they pose a much bigger obstacle than the rebasing you\nmentioned above.\n\n> You simply adjust the limit. E.g. I deployed and tested the hook version\n> of inotify (using watchman) in a sizable development environment, and\n> written my own code using the API. This was all before fanotify(7)\n> existed. IIRC I set most of the limits to 2^24 or 2^20. I've used it\n> with really large Git repos, including with David Turner's\n> 2015-04-03-1M-git for testing (`git ls-files | wc -l` on that is around\n> a quarter-million).\n>\n> If you have a humongous repository and don't have root on your own\n> machine you're out of luck. But I think most people who'd use this are\n> either using their own laptop, or e.g. in a corporate setting where\n> administrators would tweak the sysctl limits given the performance\n> advantages (as I did).\n\nThis conjecture that most people who'd use this are using their own laptop\nor have a corporate setting where administrators would tweak the sysctl\nlimits according to engineers' wishes strikes me as totally made up from\nthin air, nothing else.\n\nIn other words, I find it an incredibly unconvincing argument.\n\nI prefer not to address the rest of your mail, as I found it not only a\nlengthy tangent (basically trying to talk Jeff into adding Linux support\nin what could have been a much shorter mail), but actively distracting\nfrom the already long patch series Jeff presented. It is so long, in fact,\nthat we had to put in an exemption in GitGitGadget because it is already\nlonger than a usually-unreasonable 30 patches. Also, at this point,\ninsisting on Linux support (in so many words) is unhelpful.\n\nLet me summarize why I think this is unhelpful: In Git, it is our\ntradition to develop incrementally, for better or worse. Jeff's effort\nbrought us to a point where we already have Windows and macOS support,\ni.e. support for the most prevalent development platforms (see e.g.\nhttps://insights.stackoverflow.com/survey/2020#technology-developers-primary-operating-systems).\nWe already established multiple obstacles for Linux support, therefore\ndemanding Linux support to be included Right Now would increase the patch\nseries even further, making it even less reviewable, being even less\nincremental, hold up the current known-to-work-well state, force Jeff to\nwork on something he probably cannot work on right now, and therefore\ndelaying the entire effort even further.\n\nCiao,\nJohannes\n"},{"id":"429267","messageId":"nycvar.QRO.7.76.6.2107052336480.8230@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"87a6n5d600.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-05T21:52:07Z","receivedAt":"2021-07-05T21:52:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Fri, 2 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>\n> > +#ifdef GIT_WINDOWS_NATIVE\n> > +/*\n> > + * Create a background process to run the daemon. It should be completely\n> > + * disassociated from the terminal.\n> > + *\n> > + * Conceptually like `daemonize()` but different because Windows does not\n> > + * have `fork(2)`. Spawn a normal Windows child process but without the\n> > + * limitations of `start_command()` and `finish_command()`.\n> > + *\n> > + * The child process execs the \"git fsmonitor--daemon run\" command.\n> > + *\n> > + * The current process returns so that the caller can wait for the child\n> > + * to startup before exiting.\n> > + */\n> > +static int spawn_background_fsmonitor_daemon(pid_t *pid)\n> > +{\n> > +\tchar git_exe[MAX_PATH];\n> > +\tstruct strvec args = STRVEC_INIT;\n> > +\tint in, out;\n> > +\n> > +\tGetModuleFileNameA(NULL, git_exe, MAX_PATH);\n> > +\n> > +\tin = open(\"/dev/null\", O_RDONLY);\n> > +\tout = open(\"/dev/null\", O_WRONLY);\n> > +\n> > +\tstrvec_push(&args, git_exe);\n> > +\tstrvec_push(&args, \"fsmonitor--daemon\");\n> > +\tstrvec_push(&args, \"run\");\n> > +\tstrvec_pushf(&args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n> > +\n> > +\t*pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);\n> > +\tclose(in);\n> > +\tclose(out);\n> > +\n> > +\tstrvec_clear(&args);\n> > +\n> > +\tif (*pid < 0)\n> > +\t\treturn error(_(\"could not spawn fsmonitor--daemon in the background\"));\n> > +\n> > +\treturn 0;\n> > +}\n> > +#else\n> > +/*\n> > + * Create a background process to run the daemon. It should be completely\n> > + * disassociated from the terminal.\n> > + *\n> > + * This is adapted from `daemonize()`. Use `fork()` to directly\n> > + * create and run the daemon in the child process.\n> > + *\n> > + * The fork-child can just call the run code; it does not need to exec\n> > + * it.\n> > + *\n> > + * The fork-parent returns the child PID so that we can wait for the\n> > + * child to startup before exiting.\n> > + */\n> > +static int spawn_background_fsmonitor_daemon(pid_t *pid)\n> > +{\n> > +\t*pid = fork();\n> > +\n> > +\tswitch (*pid) {\n> > +\tcase 0:\n> > +\t\tif (setsid() == -1)\n> > +\t\t\terror_errno(_(\"setsid failed\"));\n> > +\t\tclose(0);\n> > +\t\tclose(1);\n> > +\t\tclose(2);\n> > +\t\tsanitize_stdfds();\n> > +\n> > +\t\treturn !!fsmonitor_run_daemon();\n> > +\n> > +\tcase -1:\n> > +\t\treturn error_errno(_(\"could not spawn fsmonitor--daemon in the background\"));\n> > +\n> > +\tdefault:\n> > +\t\treturn 0;\n> > +\t}\n> > +}\n> > +#endif\n>\n> The spawn_background_fsmonitor_daemon() function here is almost the same\n> as daemonize().\n\nYes, the code comment above that function even says that it was adapted\nfrom `daemonize()`.\n\nAnd above that, of course, is a _completely different_ implementation that\nworks on Windows (you will notice that this is in stark contrast of\nWindows, where the `daemonize()` function will simply fail with `ENOSYS`).\n\n> I wonder if this & the Windows-specific one you have here can't be\n> refactored into an API from what's now in setup.c.\n\nGiven that there is no `fork()` on Windows (which has been the subject of\nmany a message to this mailing list), I think the answer to this question\nof yours is a resounding \"No\".\n\n> Then we could make builtin/gc.c and daemon.c use that, so Windows could\n> have background GC, and we'd have a more battle-tested central codepath\n> for this tricky bit.\n\nPlease. Not _more_ sidetracks.\n\nThe issue of getting `git gc --auto` to daemonize on Windows is a rather\ncomplicated one. I won't bore this list with the details, but link to\nhttps://github.com/git-for-windows/git/issues/2221#issuecomment-542589590\n(a ~950 word analysis of the problem).\n\n> It seems to me like the only limitations on it are to have this return\n> slightly more general things (e.g. not set its own errors, return\n> structured data), and maybe some callback for what to do in the\n> child/parent.\n\nAnd one version doesn't `die()`. Nor does it call `exit(0)` in the parent\nprocess. But it calls `fsmonitor_listen()` in the child process. And if\nyou wanted to refactor `daemonize()` to do all that, it would have to be\nrenamed (because it does no longer _necessarily_ daemonize), and it would\nhave to have a `gentle` flag, and it would somehow have to indicate in its\nreturn value whether `0` means that the parent process returned\nsuccessfully or the client process. And soon we'll end up with a function\nthat is both longer and more unreadable than what we have right now.\n\n>\n> > +/*\n> > + * This is adapted from `wait_or_whine()`. Watch the child process and\n> > + * let it get started and begin listening for requests on the socket\n> > + * before reporting our success.\n> > + */\n> > +static int wait_for_background_startup(pid_t pid_child)\n> > +{\n> > +\tint status;\n> > +\tpid_t pid_seen;\n> > +\tenum ipc_active_state s;\n> > +\ttime_t time_limit, now;\n> > +\n> > +\ttime(&time_limit);\n> > +\ttime_limit += fsmonitor__start_timeout_sec;\n> > +\n> > +\tfor (;;) {\n> > +\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n> > +\n> > +\t\tif (pid_seen == -1)\n> > +\t\t\treturn error_errno(_(\"waitpid failed\"));\n> > +\t\telse if (pid_seen == 0) {\n> > +\t\t\t/*\n> > +\t\t\t * The child is still running (this should be\n> > +\t\t\t * the normal case). Try to connect to it on\n> > +\t\t\t * the socket and see if it is ready for\n> > +\t\t\t * business.\n> > +\t\t\t *\n> > +\t\t\t * If there is another daemon already running,\n> > +\t\t\t * our child will fail to start (possibly\n> > +\t\t\t * after a timeout on the lock), but we don't\n> > +\t\t\t * care (who responds) if the socket is live.\n> > +\t\t\t */\n> > +\t\t\ts = fsmonitor_ipc__get_state();\n> > +\t\t\tif (s == IPC_STATE__LISTENING)\n> > +\t\t\t\treturn 0;\n> > +\n> > +\t\t\ttime(&now);\n> > +\t\t\tif (now > time_limit)\n> > +\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n> > +\t\t} else if (pid_seen == pid_child) {\n> > +\t\t\t/*\n> > +\t\t\t * The new child daemon process shutdown while\n> > +\t\t\t * it was starting up, so it is not listening\n> > +\t\t\t * on the socket.\n> > +\t\t\t *\n> > +\t\t\t * Try to ping the socket in the odd chance\n> > +\t\t\t * that another daemon started (or was already\n> > +\t\t\t * running) while our child was starting.\n> > +\t\t\t *\n> > +\t\t\t * Again, we don't care who services the socket.\n> > +\t\t\t */\n> > +\t\t\ts = fsmonitor_ipc__get_state();\n> > +\t\t\tif (s == IPC_STATE__LISTENING)\n> > +\t\t\t\treturn 0;\n> > +\n> > +\t\t\t/*\n> > +\t\t\t * We don't care about the WEXITSTATUS() nor\n> > +\t\t\t * any of the WIF*(status) values because\n> > +\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n> > +\t\t\t * trick on all function return values.\n> > +\t\t\t *\n> > +\t\t\t * So it is sufficient to just report the\n> > +\t\t\t * early shutdown as an error.\n> > +\t\t\t */\n> > +\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n> > +\t\t} else\n> > +\t\t\treturn error(_(\"waitpid is confused\"));\n> > +\t}\n> > +}\n>\n> Ditto this. could we extend the wait_or_whine() function (or some\n> extended version thereof) to do what you need with callbacks?\n>\n> It seems the main difference is just being able to pass down a flag for\n> waitpid(), and the loop needing to check EINTR or not depending on\n> whether WNOHANG is passed.\n\nGiven that over half of `wait_or_whine()` is concerned with signals, which\nthe `wait_for_background_startup()` function is not at all concerned with,\nI see another main difference.\n\n> For e.g. the \"We don't care about the WEXITSTATUS()\" you'd get that\n> behavior with an adjusted wait_or_whine(). Wouldn't it be better to\n> report what exit status it exits with e.g. if the top-level process is\n> signalled? We do so in trace2 for other things we spawn...\n\nThe `wait_or_whine()` function also adjusts `atexit()` behavior, which we\nwould not want here.\n\nTherefore, just like the suggestion about `daemonize()` above, it appears\nto me as if the suggested refactoring would make the code dramatically\nmore complex and less readable.\n\nIn other words, it would be a refactoring for refactoring's sake.\nDefinitely not something I would suggest.\n\nCiao,\nJohannes\n"},{"id":"429268","messageId":"nycvar.QRO.7.76.6.2107052352560.8230@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"877di9d5uz.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 02/34] fsmonitor--daemon: man page","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-05T22:00:42Z","receivedAt":"2021-07-05T22:00:55Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Fri, 2 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>\n> > From: Jeff Hostetler \n> >\n> > Create a manual page describing the `git fsmonitor--daemon` feature.\n> >\n> > Signed-off-by: Jeff Hostetler \n> > ---\n> > Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n> > 1 file changed, 75 insertions(+)\n> > create mode 100644 Documentation/git-fsmonitor--daemon.txt\n> >\n> > diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\n> > new file mode 100644\n> > index 00000000000..154e7684daa\n> > --- /dev/null\n> > +++ b/Documentation/git-fsmonitor--daemon.txt\n> > @@ -0,0 +1,75 @@\n> > +git-fsmonitor--daemon(1)\n> > +========================\n> > +\n> > +NAME\n> > +----\n> > +git-fsmonitor--daemon - A Built-in File System Monitor\n> > +\n> > +SYNOPSIS\n> > +--------\n> > +[verse]\n> > +'git fsmonitor--daemon' start\n> > +'git fsmonitor--daemon' run\n> > +'git fsmonitor--daemon' stop\n> > +'git fsmonitor--daemon' status\n> > +\n> > +DESCRIPTION\n> > +-----------\n> > +\n> > +A daemon to watch the working directory for file and directory\n> > +changes using platform-specific file system notification facilities.\n> > +\n> > +This daemon communicates directly with commands like `git status`\n> > +using the link:technical/api-simple-ipc.html[simple IPC] interface\n> > +instead of the slower linkgit:githooks[5] interface.\n> > +\n> > +This daemon is built into Git so that no third-party tools are\n> > +required.\n> > +\n> > +OPTIONS\n> > +-------\n> > +\n> > +start::\n> > +\tStarts a daemon in the background.\n> > +\n> > +run::\n> > +\tRuns a daemon in the foreground.\n> > +\n> > +stop::\n> > +\tStops the daemon running in the current working\n> > +\tdirectory, if present.\n> > +\n> > +status::\n> > +\tExits with zero status if a daemon is watching the\n> > +\tcurrent working directory.\n> > +\n> > +REMARKS\n> > +-------\n> > +\n> > +This daemon is a long running process used to watch a single working\n> > +directory and maintain a list of the recently changed files and\n> > +directories. Performance of commands such as `git status` can be\n> > +increased if they just ask for a summary of changes to the working\n> > +directory and can avoid scanning the disk.\n> > +\n> > +When `core.useBuiltinFSMonitor` is set to `true` (see\n> > +linkgit:git-config[1]) commands, such as `git status`, will ask the\n> > +daemon for changes and automatically start it (if necessary).\n> > +\n> > +For more information see the \"File System Monitor\" section in\n> > +linkgit:git-update-index[1].\n> > +\n> > +CAVEATS\n> > +-------\n> > +\n> > +The fsmonitor daemon does not currently know about submodules and does\n> > +not know to filter out file system events that happen within a\n> > +submodule. If fsmonitor daemon is watching a super repo and a file is\n> > +modified within the working directory of a submodule, it will report\n> > +the change (as happening against the super repo). However, the client\n> > +will properly ignore these extra events, so performance may be affected\n> > +but it will not cause an incorrect result.\n> > +\n> > +GIT\n> > +---\n> > +Part of the linkgit:git[1] suite\n>\n> Later in the series we incrementally add features to the daemon, so this\n> is describing a state that doesn't exist yet at this point.\n>\n> I think it would be better to start with a stup here and add\n> documentation as we add features, e.g. the patch tha adds \"start\" should\n> add that to the synopsis + options etc.\n\nIncidentally, I had structured the patch series that way until Jeff took\nover from me last year. And it was definitely not more reviewable because\nit was not clear from the get-go what I intended this command to do.\n\nTherefore, I object to your suggestion.\n\nCiao,\nJohannes\n\nP.S.: I will not address your other reviews in this patch series, mostly\nbecause I am technically off from work this week.\n"},{"id":"429269","messageId":"874kd874qv.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107052259340.8230@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-05T22:02:31Z","receivedAt":"2021-07-05T22:52:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 05 2021, Johannes Schindelin wrote:\n\n> On Thu, 1 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n>> On Thu, Jul 01 2021, Jeff Hostetler wrote:\n>>\n>> > On 7/1/21 1:40 PM, Ævar Arnfjörð Bjarmason wrote:\n> [...]\n>> > The early Linux version was dropped because inotify does not give\n>> > recursive coverage -- only the requested directory. Using inotify\n>> > requires adding a watch to each subdirectory (recursively) in the\n>> > worktree. There's a system limit on the maximum number of watched\n>> > directories (defaults to 8K IIRC) and that limit is system-wide.\n>> >\n>> > Since the whole point was to support large very large repos, using\n>> > inotify was a non-starter, so I removed the Linux version from our\n>> > patch series. For example, the first repo I tried it on (outside of\n>> > the test suite) had 25K subdirectories.\n>> >\n>> > I'm told there is a new fanotify API in recent Linux kernels that is a\n>> > better fit for what we need, but we haven't had time to investigate it\n>> > yet.\n>>\n>> That default limit is a bit annoying, but I don't see how it's a blocker\n>> in any way.\n>\n> Let me help you to see it.\n>\n> So let's assume that you start FSMonitor-enabled Git, with the default\n> values. What is going to happen if you have any decently-sized worktree?\n> You run out of handles. What then? Throw your hands in the air? Stop\n> working? Report incorrect results?\n>\n> Those are real design challenges, and together with the race problems Jeff\n> mentioned, they pose a much bigger obstacle than the rebasing you\n> mentioned above.\n\nYou report an error and tell the user to raise the limit, and cover this\nin your install docs. It's what watchman does:\n\nhttps://github.com/facebook/watchman/blob/master/watchman/error_category.cpp#L28-L45\nhttps://facebook.github.io/watchman/docs/install.html#system-specific-preparation\n\n>> You simply adjust the limit. E.g. I deployed and tested the hook version\n>> of inotify (using watchman) in a sizable development environment, and\n>> written my own code using the API. This was all before fanotify(7)\n>> existed. IIRC I set most of the limits to 2^24 or 2^20. I've used it\n>> with really large Git repos, including with David Turner's\n>> 2015-04-03-1M-git for testing (`git ls-files | wc -l` on that is around\n>> a quarter-million).\n>>\n>> If you have a humongous repository and don't have root on your own\n>> machine you're out of luck. But I think most people who'd use this are\n>> either using their own laptop, or e.g. in a corporate setting where\n>> administrators would tweak the sysctl limits given the performance\n>> advantages (as I did).\n>\n> This conjecture that most people who'd use this are using their own laptop\n> or have a corporate setting where administrators would tweak the sysctl\n> limits according to engineers' wishes strikes me as totally made up from\n> thin air, nothing else.\n>\n> In other words, I find it an incredibly unconvincing argument.\n\nIt's from a sample size of one experience of deploying this in a BigCorp\nsetting.\n\nBut sure, perhaps things are done differently where you work/have\nworked. My experience is that even if you're dealing with some BOFHs and\ne.g. are using shared racked development servers it's generally not an\ninsurmountable task to get some useful software installed, or some\nsystem configuration tweaked.\n\nIn this case we're talking about ~40MB of kernel memory for 1 million\ndirs IIRC, that coupled with the target audience that benefits most from\nthis probably being deployments that are *painfully* aware of their \"git\nstatus\" slowness...\n\n> I prefer not to address the rest of your mail, as I found it not only a\n> lengthy tangent (basically trying to talk Jeff into adding Linux support\n> in what could have been a much shorter mail), but actively distracting\n> from the already long patch series Jeff presented. It is so long, in fact,\n> that we had to put in an exemption in GitGitGadget because it is already\n> longer than a usually-unreasonable 30 patches. Also, at this point,\n> insisting on Linux support (in so many words) is unhelpful.\n\nThis part of the tread started because Jeff H. claimed upthread:\n\n [...]inotify was a non-starter, so I removed the Linux version from\n our patch series\n\nBut after I noted that it works just fine, you just need to change some\nsysctl limits.\n\nIt seems at this point we're debating whether some installations of\nLinux have BOFH-y enough administrators that they won't tweak sysctl\nlimits for you. OK, but given that I've run this thing in a production\nsetting it's clearly not a \"non-starter\". I think it could be useful for\na lot of users.\n\nI'll reply with more (and hopefully helpful) specifics to Jeff's mail.\n\n> Let me summarize why I think this is unhelpful: In Git, it is our\n> tradition to develop incrementally, for better or worse. Jeff's effort\n> brought us to a point where we already have Windows and macOS support,\n> i.e. support for the most prevalent development platforms (see e.g.\n> https://insights.stackoverflow.com/survey/2020#technology-developers-primary-operating-systems).\n> We already established multiple obstacles for Linux support, therefore\n> demanding Linux support to be included Right Now would increase the patch\n> series even further, making it even less reviewable, being even less\n> incremental, hold up the current known-to-work-well state, force Jeff to\n> work on something he probably cannot work on right now, and therefore\n> delaying the entire effort even further.\n\nI think we just disagree. I wouldn't call my opinion \"unhelpful\" any\nmore than yours.\n\nI don't think Git's ever had anything like a major feature (built in,\nconfig settings, etc. etc.) contributed by a propriterary OS vendor that\nworks on that vendor's OS, as well as another vendor's propriterary OS,\nbut not comparable free systems\n\nIs that less incremental and perhaps less practical? Sure. It's not an\nentirely practical viewpoint. I work on free software partly for\nidealistic reasons. I'd prefer if the project I'm working on doesn't\ngive users a carrot to pick proprietary systems over free ones.\n\nBut ultimately it's not my call, but Junio's.\n\n\n\n"},{"id":"429270","messageId":"871r8c73ej.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"9ac45626-f995-a661-2746-6711f996e5a6@jeffhostetler.com","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-05T22:52:44Z","receivedAt":"2021-07-05T23:21:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 02 2021, Jeff Hostetler wrote:\n\n> On 7/1/21 5:26 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Jul 01 2021, Jeff Hostetler wrote:\n>> \n>>> On 7/1/21 1:40 PM, Ævar Arnfjörð Bjarmason wrote:\n>>>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>>>>\n>>>>> Here is V3 of my patch series to add a builtin FSMonitor daemon to Git. I\n>>>>> rebased this series onto v2.32.0.\n>>>>>\n>>>>> V3 addresses most of the previous review comments and things we've learned\n>>>>> from our experimental testing of V2. (A version of V2 was shipped as an\n>>>>> experimental feature in the v2.32.0-based releases of Git for Windows and\n>>>>> VFS for Git.)\n>>>>>\n>>>>> There are still a few items that I need to address, but that list is getting\n>>>>> very short.\n>>>> ...\n>>>>> fsmonitor-fs-listen-win32: stub in backend for Windows\n>>>>> fsmonitor-fs-listen-macos: stub in backend for MacOS\n>>>> I left some light comments on the repo-settings.c part of this to\n>>>> follow\n>>>> up from a previous round.\n>>>\n>>> Thanks.\n>>>\n>>>> Any other testing of it is stalled by there being no linux backend\n>>>> for\n>>>> it as part of this series. I see from spelunking repos that Johannes had\n>>>> a WIP compat/fsmonitor/linux.c which looks like it could/should mostly\n>>>> work, but the API names all changed since then, and after a short try I\n>>>> gave up on trying to rebase it.\n>>>\n>>> The early Linux version was dropped because inotify does not give\n>>> recursive coverage -- only the requested directory. Using inotify\n>>> requires adding a watch to each subdirectory (recursively) in the\n>>> worktree. There's a system limit on the maximum number of watched\n>>> directories (defaults to 8K IIRC) and that limit is system-wide.\n>>>\n>>> Since the whole point was to support large very large repos, using\n>>> inotify was a non-starter, so I removed the Linux version from our\n>>> patch series. For example, the first repo I tried it on (outside\n>>> of the test suite) had 25K subdirectories.\n>>>\n>>> I'm told there is a new fanotify API in recent Linux kernels that\n>>> is a better fit for what we need, but we haven't had time to investigate\n>>> it yet.\n>> That default limit is a bit annoying, but I don't see how it's a\n>> blocker\n>> in any way.\n>> You simply adjust the limit. E.g. I deployed and tested the hook\n>> version\n>> of inotify (using watchman) in a sizable development environment, and\n>> written my own code using the API. This was all before fanotify(7)\n>> existed. IIRC I set most of the limits to 2^24 or 2^20. I've used it\n>> with really large Git repos, including with David Turner's\n>> 2015-04-03-1M-git for testing (`git ls-files | wc -l` on that is around\n>> a quarter-million).\n>> If you have a humongous repository and don't have root on your own\n>> machine you're out of luck. But I think most people who'd use this are\n>> either using their own laptop, or e.g. in a corporate setting where\n>> administrators would tweak the sysctl limits given the performance\n>> advantages (as I did).\n>> Once you adjust the limits Linux deals with large trees just fine,\n>> it\n>> just has overly conservative limits for most things in sysctl. The API\n>> is a bit annoying, your event loop needs to run around and add watches.\n>> AFAICT you need Linux 5.1 for fanotify(7) to be useful, e.g. Debian\n>> stable, RHEL etc. aren't using anything that new. So having an inotify\n>> backend as well as possibly a fanotify one would be very useful.\n>> And linux.git's docs suggest that the default limit was bumped from\n>> 8192\n>> to 1M in v5.10, a really recent kernel, so if you've got that you've\n>> also got fanotify.\n>> In any case, even if Linux's inotify limit was something hardcoded\n>> and\n>> impossible to change you could still use such an API to test & debug the\n>> basics of this feature on that platform.\n>\n> Good points. If the inotify limits can be increased into the millions\n> then we can revisit supporting it.\n\nI also replied in the side-thread on this topic:\nhttps://lore.kernel.org/git/874kd874qv.fsf@evledraar.gmail.com/\n\n> [...] I do worry about possible race\n> conditions as we have to scan the worktree and add/delete a watch for\n> each directory, but we don't need to worry about that right now.\n\nIn the watchman code there's a scheduleRecrawl() in such cases, but it's\nnot just something that happens on Linux.\n\nOn Windows it's if ReadDirectoryChangesW() returns\nERROR_NOTIFY_ENUM_DIR. This builtin version doesn't check that at\nall. Here's someone who seems to work at MSFT talking about it:\n\n https://devblogs.microsoft.com/oldnewthing/20110812-00/?p=9913\n\nSo isn't this an issue on Windows as well?\n\n> I do want to have Linux support eventually, but I was saving it for\n> a later effort (and/or looking for volunteers).\n\nOK, upthread you said it was a \"non-starter\" because of the adjustable\nlimits we've been discussing...\n\n> My IPC-based builtin daemon complements the existing hook-based\n> fsmonitor support that Ben Peart and Kevin Willford added a few years\n> ago. That model (and PERL hook script and Watchman integration) work\n> fine for Linux, so the advantages of a builtin monitor aren't as\n> compelling.\n>\n> For example, on Linux, hook process creation is fast, PERL is fast,[...]\n\nDid you or Ben ever try to reproduce what I noted at:\n\n https://lore.kernel.org/git/CACBZZX5e58bWuf3NdDYTxu2KyZj29hHONzN=rp-7vXd8nURyWQ@mail.gmail.com/\n\nAnd more recently with reference to that at:\n\n https://lore.kernel.org/git/87h7lgfchm.fsf@evledraar.gmail.com/\n\nI.e. when I tested it back then the issue wasn't Perl's performance, it\nwas something between watchman and the user, happening in\ngit. I.e. watchman would return in the single-digit milliseconds, git\nwould take hundreds or thousands of ms.\n\nMaybe I missed some intervening analysis, but this didn't have to do\nwith process overhead back then.\n\nAlso, watchman has a JSON network interface. So for a \"native\" solution\nI also wonder if you or Ben tried a solution that involved just talking\nto that over the local network. The hook/perl script was just a trivial\naid to make doing that easier. Wouldn't that be just as fast (or near\nenough) on Windows?\n\nBen mentioned in the linked thread that there was some talks between\nMicrosoft (or well, the \"Git team\") and the watchman people. So again, I\nmissed some in-between discussions. I'm just wondering about the design\nchoices...\n\n> and it is easy to just apt-get a tool like Watchman. But on\n> Windows, process creation is slow, NTFS is slow, PERL is available\n> as part of the Git-for-Windows distribution, and installing third-party\n> tools like Watchman onto a collection of enterprise users' machines is\n> a chore, so it made sense of us to pick platforms where the existing\n> hook model has issues and add other platforms later.\n\nIf we're trying to do an end-run around package systems being a pain\nwould including the relevant part of watchman in contrib + talking to it\nover its network interface be a replacement for some/most/all of this\nseries?\n\n> Besides, this patch series is already at 34 commits. And some of\n> them are quite large. Adding another platform would just make it\n> even larger.\n>\n> Right now I'm more interested in the larger question of whether\n> we WANT to have a builtin fsmonitor and do we like the overall\n> design of what I have here? Picking up a new platform, whether\n> it is Linux, or AIX, or Solaris, or Nonstop, or whatever, should\n> nicely fit into one platform-specific file in compat/fsmonitor\n> and not take that long.\n\nSure, FWIW I'm not opposed to this being a built-in per-se, or us\nre-implementing parts of watchman here, and not being able to test this\nseries meaningfully a lot of these questions are probably easy to answer\non a supported OS...\n\n>>>> I'd really prefer for git not to have features that place free\n>>>> platforms\n>>>> at a disadvantage against proprietary platforms if it can be avoided,\n>>>> and in this case the lack of a Linux backend also means much less\n>>>> widespread testing of the feature among the development community / CI.\n>>>>\n>>>\n>>> This feature is always going to have platform-specific components,\n>>> so the lack of one platform or another should not stop us from\n>>> discussing it for the platforms that can be supported.\n>> (I think per the above that's s/can be/are/)\n>> \n>>> And given the size and complexity of the platform-specific code,\n>>> we should not assume that \"just test it on Linux\" is sufficient.\n>>> Yes, there are some common/shared routines/data structures in the\n>>> daemon, but hard/tricky parts are in the platform layer.\n>> I think we're talking past each other a bit here. I'm not saying\n>> that\n>> you can get full or meaningful testing for it on Windows if you test it\n>> on Linux, or the other way around.\n>> Of course there's platform-specific stuff, although there's also a\n>> lot\n>> of non-platform-specific stuff, so even a very basic implementation of\n>> inotify would make reviwing this easier / give access to more reviewers.\n>> I'm saying that I prefer that Git as a free software project not be\n>> in\n>> the situation of saying the best use-case for a given size/shape of repo\n>> is to use Git in combination with proprietary operating systems over\n>> freely licensed ones.\n>\n> I wouldn't worry about that. Even without Watchman integration,\n> Linux runs things so much faster than anything else it's not funny.\n>\n> If anything, we need things like fsmonitor on Windows to help keep\n> up with Linux.\n\nEven on Linux if you do a \"git status\" on a humongous repository it can\ntake 1-2 seconds (see old but still valid linked numbers above), whereas\nif you've been logging fs events and ask a daemon \"what changed since\ntime xyz\" it can take <5ms.\n\nBut that's just been on repositories I've tested. I'd assumed that this\npart of GVFS wasn't something that could have been fairly easily\nreplaced by having the relevant developers run a Linux distro in vmware\nor whatever, adn that's been consistent with my own testing on\nstill-big-but-smaller repos than that.\n"},{"id":"429284","messageId":"nycvar.QRO.7.76.6.2107061508510.8230@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"874kd874qv.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-06T13:12:11Z","receivedAt":"2021-07-06T13:12:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Tue, 6 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> I think we just disagree. I wouldn't call my opinion \"unhelpful\" any\n> more than yours.\n\nYou still misunderstand. This is not about any \"opinion\" of yours, it is\nabout your delay tactics to make it deliberately difficult to finish this\npatch series, by raising the bar beyond what is reasonable for a single\npatch series.\n\nAnd you keep doing it. I would appreciate if you just stopped with all\nthose tangents and long and many replies that do not seem designed to help\nthe patch series stabilize, but do the opposite.\n\nCiao,\nJohannes\n"},{"id":"429323","messageId":"nycvar.QRO.7.76.6.2107062102500.8230@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"5bba5eb3d1bd172f09fdf6eb2e9b8ac4dd7f940f.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-06T19:09:43Z","receivedAt":"2021-07-06T19:23:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\n\nOn Thu, 1 Jul 2021, Jeff Hostetler via GitGitGadget wrote:\n\nJeff Hostetler \n\nthe win32 backend to register a watch on the working tree\nirectory (recursively). Also watch the if it is\nside the working tree. And to collect path change notifications\natches and publish.\n\n-off-by: Jeff Hostetler \n\nt/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++\ne changed, 530 insertions(+)\n\n> diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> index 880446b49e3..d707d47a0d7 100644\n> --- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> @@ -2,20 +2,550 @@\n> + [...]\n> +\n> +static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n> +\t\t\t\t const char *path)\n> +{\n> +\tstruct one_watch *watch = NULL;\n> +\tDWORD desired_access = FILE_LIST_DIRECTORY;\n> +\tDWORD share_mode =\n> +\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n> +\tHANDLE hDir;\n> +\n> +\thDir = CreateFileA(path,\n> +\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n> +\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n> +\t\t\t NULL);\n\nThe `*A()` family of Win32 API functions disagree with Git in one very\ninteresting aspect: Git always assumes UTF-8, while e.g. `CreateFileA()`\nwill use the current Win32 locale to internally transform to wide\ncharacters and then call `CreateFileW()`.\n\nThis poses no problem when your locale is US American and your paths\ncontain no non-ASCII characters.\n\nIn the Git for Windows bug tracker, it was reported that it _does_ cause\nproblems when venturing outside such a cozy scenario (for full details,\nsee https://github.com/git-for-windows/git/issues/3262)\n\nI need this (and merged it before starting the process to release Git for\nWindows v2.32.0(2)) to fix that (could I ask you to integrate this in case\na re-roll will become necessary?):\n\n-- snipsnap --\nFrom: Johannes Schindelin \nDate: Mon, 5 Jul 2021 13:51:05 +0200\nSubject: [PATCH] fixup! fsmonitor-fs-listen-win32: implement FSMonitor backend\n on Windows\n\nLet's keep avoiding the `*A()` family of Win32 API functions because\nthey are susceptible to incoherent encoding problems. In Git for\nWindows, we always assume paths to be UTF-8 encoded. Let's use the\ndedicated helper to convert such a path to the wide character version,\nand then use the `*W()` function instead.\n\nSigned-off-by: Johannes Schindelin \n---\n compat/fsmonitor/fsmonitor-fs-listen-win32.c | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\nindex ba087b292df..3b42ab311d9 100644\n--- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n+++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n@@ -111,8 +111,14 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n+\twchar_t wpath[MAX_PATH];\n\n-\thDir = CreateFileA(path,\n+\tif (xutftowcs_path(wpath, path) < 0) {\n+\t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n+\t\treturn NULL;\n+\t}\n+\n+\thDir = CreateFileW(wpath,\n \t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n \t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n \t\t\t NULL);\n--\n2.32.0.windows.1.15.gf1590a75e2d\n\n"},{"id":"429389","messageId":"60e509353500e_2fc0f5208ef@natae.notmuch","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107052259340.8230@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-07-07T01:53:57Z","receivedAt":"2021-07-07T01:54:01Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Johannes Schindelin wrote:\n\n> In Git, it is our tradition to develop incrementally, for better or\n> worse.\n\nWe develop incrementally, but the first version of a big feature has to\nhave as many eyes on it as possible in order to track potential issues\nin the future.\n\nIt is very common to look back at an API and say \"oh, if only wad\nthought of that back then\". Therefore we should do our best to think\nabout it now.\n\nWe want to avoid future aw-shucks.\n\n> Jeff's effort\n> brought us to a point where we already have Windows and macOS support,\n> i.e. support for the most prevalent development platforms (see e.g.\n> https://insights.stackoverflow.com/survey/2020#technology-developers-primary-operating-systems).\n> We already established multiple obstacles for Linux support, therefore\n> demanding Linux support to be included Right Now would increase the patch\n> series even further, making it even less reviewable, being even less\n> incremental, hold up the current known-to-work-well state, force Jeff to\n> work on something he probably cannot work on right now, and therefore\n> delaying the entire effort even further.\n\nThis is a red herring.\n\nYou don't need to send the Linux support as part of the patch series,\nyou can simply provide a branch for the people that want to give it a\ntry.\n\nEven if it's not ready for wide use, even if it needs a specific version\nof Linux, even if it's a proof of concept, you can provide it.\n\nThe real reason it's not provided is laziness (this is not an attack,\nlaziness is good trait in a programmer, although not always).\n\nCheers.\n\n-- \nFelipe Contreras\n"},{"id":"429391","messageId":"60e50e02dc5a6_2fc0f520811@natae.notmuch","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107061508510.8230@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 00/34] Builtin FSMonitor Feature","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-07-07T02:14:26Z","receivedAt":"2021-07-07T02:14:46Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Johannes Schindelin wrote:\n> On Tue, 6 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n> \n> > I think we just disagree. I wouldn't call my opinion \"unhelpful\" any\n> > more than yours.\n> \n> You still misunderstand. This is not about any \"opinion\" of yours, it is\n> about your delay tactics to make it deliberately difficult to finish this\n> patch series,\n\nThe whole job of a reviewer is to *delay* the inclusion of the patch\nseries being reviewed until it has passed his/her personal standards.\n\nÆvar is just doing his job (metaphorically).\n\n> by raising the bar beyond what is reasonable for a single patch\n> series.\n\nIt is not up to you to determine what is reasonable for a patch series,\nand given that you have a vested interest you are also not an objective\nobserver.\n\nI have hundreds of patches pending review, and I would love for anyone\nto try find issues with them, even if I ultimately disagreed with their\nassessment.\n\nThe Git project doesn't owe your patches any preferential treatment.\nThe review process will take as long as the review process takes. You\ncan't put deadlines on open source projects.\n\n-- \nFelipe Contreras"},{"id":"429815","messageId":"533d47fc-ea37-1e6a-b6f4-ac4fdd24555e@jeffhostetler.com","threadId":"55420","inReplyTo":"877di9d5uz.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 02/34] fsmonitor--daemon: man page","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-12T19:23:33Z","receivedAt":"2021-07-12T19:23:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 6:29 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Create a manual page describing the `git fsmonitor--daemon` feature.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n>> 1 file changed, 75 insertions(+)\n>> create mode 100644 Documentation/git-fsmonitor--daemon.txt\n>>\n>> diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\n>> new file mode 100644\n>> index 00000000000..154e7684daa\n>> --- /dev/null\n>> +++ b/Documentation/git-fsmonitor--daemon.txt\n>> @@ -0,0 +1,75 @@\n>> +git-fsmonitor--daemon(1)\n>> +========================\n>> +\n>> +NAME\n>> +----\n>> +git-fsmonitor--daemon - A Built-in File System Monitor\n>> +\n>> +SYNOPSIS\n>> +--------\n>> +[verse]\n>> +'git fsmonitor--daemon' start\n>> +'git fsmonitor--daemon' run\n>> +'git fsmonitor--daemon' stop\n>> +'git fsmonitor--daemon' status\n>> +\n>> +DESCRIPTION\n>> +-----------\n>> +\n>> +A daemon to watch the working directory for file and directory\n>> +changes using platform-specific file system notification facilities.\n>> +\n>> +This daemon communicates directly with commands like `git status`\n>> +using the link:technical/api-simple-ipc.html[simple IPC] interface\n>> +instead of the slower linkgit:githooks[5] interface.\n>> +\n>> +This daemon is built into Git so that no third-party tools are\n>> +required.\n>> +\n>> +OPTIONS\n>> +-------\n>> +\n>> +start::\n>> +\tStarts a daemon in the background.\n>> +\n>> +run::\n>> +\tRuns a daemon in the foreground.\n>> +\n>> +stop::\n>> +\tStops the daemon running in the current working\n>> +\tdirectory, if present.\n>> +\n>> +status::\n>> +\tExits with zero status if a daemon is watching the\n>> +\tcurrent working directory.\n>> +\n>> +REMARKS\n>> +-------\n>> +\n>> +This daemon is a long running process used to watch a single working\n>> +directory and maintain a list of the recently changed files and\n>> +directories. Performance of commands such as `git status` can be\n>> +increased if they just ask for a summary of changes to the working\n>> +directory and can avoid scanning the disk.\n>> +\n>> +When `core.useBuiltinFSMonitor` is set to `true` (see\n>> +linkgit:git-config[1]) commands, such as `git status`, will ask the\n>> +daemon for changes and automatically start it (if necessary).\n>> +\n>> +For more information see the \"File System Monitor\" section in\n>> +linkgit:git-update-index[1].\n>> +\n>> +CAVEATS\n>> +-------\n>> +\n>> +The fsmonitor daemon does not currently know about submodules and does\n>> +not know to filter out file system events that happen within a\n>> +submodule. If fsmonitor daemon is watching a super repo and a file is\n>> +modified within the working directory of a submodule, it will report\n>> +the change (as happening against the super repo). However, the client\n>> +will properly ignore these extra events, so performance may be affected\n>> +but it will not cause an incorrect result.\n>> +\n>> +GIT\n>> +---\n>> +Part of the linkgit:git[1] suite\n> \n> Later in the series we incrementally add features to the daemon, so this\n> is describing a state that doesn't exist yet at this point.\n> \n> I think it would be better to start with a stup here and add\n> documentation as we add features, e.g. the patch tha adds \"start\" should\n> add that to the synopsis + options etc.\n> \n> See the outstanding ab/config-based-hooks-base for a small example of\n> that.\n> \n\nI like to lead the series with the documentation that summarizes the\npurpose of the entire feature or patch series. This gives the reviewer\nthe context for the complete series that follows. In the past, we've\nhad discussions on the list about how hard it is to review a series when\nthe foo.c comes (alphabetically) before foo.h in the patch and all\nthe documentation is attached to the prototypes in the .h file so the\nreviewer needs to bounce around in the patch or series to read the\nintent and then go back to the beginning to read the code. In that\nspirit, I think that having the complete man page come first provides\nnecessary context and is helpful.\n\nThe argument that the man-page should grow as the feature grows\npresumes that there is a meaningful cut-point mid-series where you\nwould adopt the first portion and delay the second to a later release\nor something. That division would not be useful/usable.\n\nAnd it just clutters up later commits in the series with man-page\ndeltas.\n\nSo I'd like to keep it as it unless there are further objections.\n\nJeff\n\n"},{"id":"429893","messageId":"80fb703d-0f88-110e-5a5c-e8e9a90bbc17@jeffhostetler.com","threadId":"55420","inReplyTo":"87a6n5d600.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T14:39:56Z","receivedAt":"2021-07-13T14:39:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\nMy response here is in addition to Dscho's remarks on this topic.\nHe makes excellent points that I'll just #include here. I do want\nto add my own $0.02 here.\n\nOn 7/1/21 6:18 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n>> +#ifdef GIT_WINDOWS_NATIVE\n>> +/*\n>> + * Create a background process to run the daemon. It should be completely\n>> + * disassociated from the terminal.\n>> + *\n>> + * Conceptually like `daemonize()` but different because Windows does not\n>> + * have `fork(2)`. Spawn a normal Windows child process but without the\n>> + * limitations of `start_command()` and `finish_command()`.\n>> + *\n>> + * The child process execs the \"git fsmonitor--daemon run\" command.\n>> + *\n>> + * The current process returns so that the caller can wait for the child\n>> + * to startup before exiting.\n>> + */\n>> +static int spawn_background_fsmonitor_daemon(pid_t *pid)\n>> +{\n>> +\tchar git_exe[MAX_PATH];\n>> +\tstruct strvec args = STRVEC_INIT;\n>> +\tint in, out;\n>> +\n>> +\tGetModuleFileNameA(NULL, git_exe, MAX_PATH);\n>> +\n>> +\tin = open(\"/dev/null\", O_RDONLY);\n>> +\tout = open(\"/dev/null\", O_WRONLY);\n>> +\n>> +\tstrvec_push(&args, git_exe);\n>> +\tstrvec_push(&args, \"fsmonitor--daemon\");\n>> +\tstrvec_push(&args, \"run\");\n>> +\tstrvec_pushf(&args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n>> +\n>> +\t*pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);\n>> +\tclose(in);\n>> +\tclose(out);\n>> +\n>> +\tstrvec_clear(&args);\n>> +\n>> +\tif (*pid < 0)\n>> +\t\treturn error(_(\"could not spawn fsmonitor--daemon in the background\"));\n>> +\n>> +\treturn 0;\n>> +}\n>> +#else\n>> +/*\n>> + * Create a background process to run the daemon. It should be completely\n>> + * disassociated from the terminal.\n>> + *\n>> + * This is adapted from `daemonize()`. Use `fork()` to directly\n>> + * create and run the daemon in the child process.\n>> + *\n>> + * The fork-child can just call the run code; it does not need to exec\n>> + * it.\n>> + *\n>> + * The fork-parent returns the child PID so that we can wait for the\n>> + * child to startup before exiting.\n>> + */\n>> +static int spawn_background_fsmonitor_daemon(pid_t *pid)\n>> +{\n>> +\t*pid = fork();\n>> +\n>> +\tswitch (*pid) {\n>> +\tcase 0:\n>> +\t\tif (setsid() == -1)\n>> +\t\t\terror_errno(_(\"setsid failed\"));\n>> +\t\tclose(0);\n>> +\t\tclose(1);\n>> +\t\tclose(2);\n>> +\t\tsanitize_stdfds();\n>> +\n>> +\t\treturn !!fsmonitor_run_daemon();\n>> +\n>> +\tcase -1:\n>> +\t\treturn error_errno(_(\"could not spawn fsmonitor--daemon in the background\"));\n>> +\n>> +\tdefault:\n>> +\t\treturn 0;\n>> +\t}\n>> +}\n>> +#endif\n> \n> The spawn_background_fsmonitor_daemon() function here is almost the same\n> as daemonize(). I wonder if this & the Windows-specific one you have\n> here can't be refactored into an API from what's now in setup.c.\n> \n> Then we could make builtin/gc.c and daemon.c use that, so Windows could\n> have background GC, and we'd have a more battle-tested central codepath\n> for this tricky bit.\n> \n\nI'd rather not refactor all of this and add unnecessary generality\nand complexity just to save duplicating some of the code in daemonize().\n\nAnd I'd rather not destabilize existing commands like gc and daemon\nby changing the daemonize() layer on them. If those commands need help,\nlet's have a separate conversation _later_ about what help they need\nand if it makes sense to combine them.\n\n\n> It seems to me like the only limitations on it are to have this return\n> slightly more general things (e.g. not set its own errors, return\n> structured data), and maybe some callback for what to do in the\n> child/parent.\n\nThere are several issues here when trying to start a background process\nand we're already on the edge of the behavioral differences between\nWindows and Unix -- let's not make things more confusing with multiple\ncallbacks, returning structures, custom errors, and etc.\n\nAlso, since Windows doesn't do fork(), we don't have child/parent\nbranches in the call, so this whole \"just pretend it's all Unix\"\nmodel doesn't work.\n\nEven if we did pretend I'd still need ifdef'd callback routines to\neither call `fsmonitor_run_daemon()` or build a command line (or have\nblocks of functions that \"just happen to never be called on one\nplatform or the other\").\n\n\nWhat I have here is an API that the primary (read: parent) calls\nand gets back a 0 or -1 (with error message). And that's it.\nThe primary can then wait for the child (whether from fork or\nCreateProcess) to become responsive or fail to start. And then\nthe primary can exit (with or without error).\n\nSo I think we're good. Yes, there is an ifdef here, but I think\nit is worth it.\n\n\n> \n>> +/*\n>> + * This is adapted from `wait_or_whine()`. Watch the child process and\n>> + * let it get started and begin listening for requests on the socket\n>> + * before reporting our success.\n>> + */\n>> +static int wait_for_background_startup(pid_t pid_child)\n>> +{\n>> +\tint status;\n>> +\tpid_t pid_seen;\n>> +\tenum ipc_active_state s;\n>> +\ttime_t time_limit, now;\n>> +\n>> +\ttime(&time_limit);\n>> +\ttime_limit += fsmonitor__start_timeout_sec;\n>> +\n>> +\tfor (;;) {\n>> +\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n>> +\n>> +\t\tif (pid_seen == -1)\n>> +\t\t\treturn error_errno(_(\"waitpid failed\"));\n>> +\t\telse if (pid_seen == 0) {\n>> +\t\t\t/*\n>> +\t\t\t * The child is still running (this should be\n>> +\t\t\t * the normal case). Try to connect to it on\n>> +\t\t\t * the socket and see if it is ready for\n>> +\t\t\t * business.\n>> +\t\t\t *\n>> +\t\t\t * If there is another daemon already running,\n>> +\t\t\t * our child will fail to start (possibly\n>> +\t\t\t * after a timeout on the lock), but we don't\n>> +\t\t\t * care (who responds) if the socket is live.\n>> +\t\t\t */\n>> +\t\t\ts = fsmonitor_ipc__get_state();\n>> +\t\t\tif (s == IPC_STATE__LISTENING)\n>> +\t\t\t\treturn 0;\n>> +\n>> +\t\t\ttime(&now);\n>> +\t\t\tif (now > time_limit)\n>> +\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n>> +\t\t} else if (pid_seen == pid_child) {\n>> +\t\t\t/*\n>> +\t\t\t * The new child daemon process shutdown while\n>> +\t\t\t * it was starting up, so it is not listening\n>> +\t\t\t * on the socket.\n>> +\t\t\t *\n>> +\t\t\t * Try to ping the socket in the odd chance\n>> +\t\t\t * that another daemon started (or was already\n>> +\t\t\t * running) while our child was starting.\n>> +\t\t\t *\n>> +\t\t\t * Again, we don't care who services the socket.\n>> +\t\t\t */\n>> +\t\t\ts = fsmonitor_ipc__get_state();\n>> +\t\t\tif (s == IPC_STATE__LISTENING)\n>> +\t\t\t\treturn 0;\n>> +\n>> +\t\t\t/*\n>> +\t\t\t * We don't care about the WEXITSTATUS() nor\n>> +\t\t\t * any of the WIF*(status) values because\n>> +\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n>> +\t\t\t * trick on all function return values.\n>> +\t\t\t *\n>> +\t\t\t * So it is sufficient to just report the\n>> +\t\t\t * early shutdown as an error.\n>> +\t\t\t */\n>> +\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n>> +\t\t} else\n>> +\t\t\treturn error(_(\"waitpid is confused\"));\n>> +\t}\n>> +}\n> \n> Ditto this. could we extend the wait_or_whine() function (or some\n> extended version thereof) to do what you need with callbacks?\n> \n> It seems the main difference is just being able to pass down a flag for\n> waitpid(), and the loop needing to check EINTR or not depending on\n> whether WNOHANG is passed.\n> \n> For e.g. the \"We don't care about the WEXITSTATUS()\" you'd get that\n> behavior with an adjusted wait_or_whine(). Wouldn't it be better to\n> report what exit status it exits with e.g. if the top-level process is\n> signalled? We do so in trace2 for other things we spawn...\n> \n\nAgain, I don't want to mix my usage here with the existing code\nand destabilize all existing callers. Here we are spinning to give\nthe child a chance to *start* and confirm that it is in a listening\nstate and ready for connections. We do not wait for the child to\nexit (unless it dies quickly without becoming ready).\n\nWe want to end our wait as soon as we confirm that the child is\nready and return. All I really need from the system is `waitpid()`.\n\nAlso, since we started the child in my `spawn_background...()`, it\nis not in the `children_to_clean` list, so there is no need to mess\nwith that.\n\nSo I'd like to leave this as is.\n\nJeff\n"},{"id":"429895","messageId":"f38ed916-cfd8-06b2-43dc-689720a96a6e@jeffhostetler.com","threadId":"55420","inReplyTo":"87mtr5bpuw.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 17/34] fsmonitor--daemon: define token-ids","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T15:15:08Z","receivedAt":"2021-07-13T15:15:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 6:58 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n>> +\tif (!test_env_value) {\n>> +\t\tstruct timeval tv;\n>> +\t\tstruct tm tm;\n>> +\t\ttime_t secs;\n>> +\n>> +\t\tgettimeofday(&tv, NULL);\n>> +\t\tsecs = tv.tv_sec;\n>> +\t\tgmtime_r(&secs, &tm);\n>> +\n>> +\t\tstrbuf_addf(&token->token_id,\n>> +\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n>> +\t\t\t flush_count++,\n>> +\t\t\t getpid(),\n>> +\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n>> +\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n>> +\t\t\t (long)tv.tv_usec);\n> \n> Just bikeshedding, but can we have tokens that mostly sort numeric-wise\n> by time order? So time at the start, not the flush_count/getpid.\n\nAs I described in a rather large comment in the code, tokens are opaque\nstrings -- without a less-than / greater-than relationship -- just a\nrandom string that the daemon can use (along with a sequence number) to\nensure that a later request is well-defined.\n\nHere I'm using a counter, pid, and date-stamp. I'd prefer using a GUID\nor UUID just to drive that home, but I didn't want to add a new .lib or\n.a to the build if not necessary.\n\nPerhaps I should compute this portion as hex(hash(time())) to remove the\ntemptation to look inside my opaque token ??\n\n> \n> Maybe I'm missing something, but couldn't we just re-use the trace2 SID\n> + a more trivial trailer? It would have the nice property that you could\n> find the trace2 SID whenever you looked at such a token (could\n> e.g. split them by \"/\" too), and add the tv_usec, flush_count+whatever\n> else is needed to make it unique after the \"/\", no?\n> \n\nI would rather keep Trace2 out of this. The SID is another opaque\nstring and I don't want to reach inside it.\n\nJeff\n"},{"id":"429896","messageId":"806e2108-5391-9b92-368f-403e7b2d44b7@jeffhostetler.com","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107062102500.8230@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T15:18:31Z","receivedAt":"2021-07-13T15:18:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/6/21 3:09 PM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> \n> On Thu, 1 Jul 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n> Jeff Hostetler \n> \n> the win32 backend to register a watch on the working tree\n> irectory (recursively). Also watch the if it is\n> side the working tree. And to collect path change notifications\n> atches and publish.\n> \n> -off-by: Jeff Hostetler \n> \n> t/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++\n> e changed, 530 insertions(+)\n> \n>> diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n>> index 880446b49e3..d707d47a0d7 100644\n>> --- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n>> +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n>> @@ -2,20 +2,550 @@\n>> + [...]\n>> +\n>> +static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n>> +\t\t\t\t const char *path)\n>> +{\n>> +\tstruct one_watch *watch = NULL;\n>> +\tDWORD desired_access = FILE_LIST_DIRECTORY;\n>> +\tDWORD share_mode =\n>> +\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n>> +\tHANDLE hDir;\n>> +\n>> +\thDir = CreateFileA(path,\n>> +\t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n>> +\t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n>> +\t\t\t NULL);\n> \n> The `*A()` family of Win32 API functions disagree with Git in one very\n> interesting aspect: Git always assumes UTF-8, while e.g. `CreateFileA()`\n> will use the current Win32 locale to internally transform to wide\n> characters and then call `CreateFileW()`.\n> \n> This poses no problem when your locale is US American and your paths\n> contain no non-ASCII characters.\n> \n> In the Git for Windows bug tracker, it was reported that it _does_ cause\n> problems when venturing outside such a cozy scenario (for full details,\n> see https://github.com/git-for-windows/git/issues/3262)\n> \n> I need this (and merged it before starting the process to release Git for\n> Windows v2.32.0(2)) to fix that (could I ask you to integrate this in case\n> a re-roll will become necessary?):\n> \n> -- snipsnap --\n> From: Johannes Schindelin \n> Date: Mon, 5 Jul 2021 13:51:05 +0200\n> Subject: [PATCH] fixup! fsmonitor-fs-listen-win32: implement FSMonitor backend\n> on Windows\n> \n> Let's keep avoiding the `*A()` family of Win32 API functions because\n> they are susceptible to incoherent encoding problems. In Git for\n> Windows, we always assume paths to be UTF-8 encoded. Let's use the\n> dedicated helper to convert such a path to the wide character version,\n> and then use the `*W()` function instead.\n> \n> Signed-off-by: Johannes Schindelin \n> ---\n> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 8 +++++++-\n> 1 file changed, 7 insertions(+), 1 deletion(-)\n> \n> diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> index ba087b292df..3b42ab311d9 100644\n> --- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> @@ -111,8 +111,14 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n> \tDWORD share_mode =\n> \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n> \tHANDLE hDir;\n> +\twchar_t wpath[MAX_PATH];\n> \n> -\thDir = CreateFileA(path,\n> +\tif (xutftowcs_path(wpath, path) < 0) {\n> +\t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\thDir = CreateFileW(wpath,\n> \t\t\t desired_access, share_mode, NULL, OPEN_EXISTING,\n> \t\t\t FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,\n> \t\t\t NULL);\n> --\n> 2.32.0.windows.1.15.gf1590a75e2d\n> \n\nThanks for the heads up. I'll pull this into my next release.\nJeff\n"},{"id":"429897","messageId":"b19f3f2a-049f-acf2-f59e-de705dc54307@jeffhostetler.com","threadId":"55420","inReplyTo":"87k0m9bpmv.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T15:46:05Z","receivedAt":"2021-07-13T15:46:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 7:02 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Teach the win32 backend to register a watch on the working tree\n>> root directory (recursively). Also watch the if it is\n>> not inside the working tree. And to collect path change notifications\n>> into batches and publish.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++\n> \n> Spying on the early history of this (looking for the\n> Linux backend) I saw that at some point we had just\n> compat/fsmonitor/linux.c, and presumably some of\n> compat/fsmonitor/{windows,win32,macos,darwin}.c.\n> \n> At some point those filenames became much much longer.\n> \n\nOnce upon a time having \"foo/bar/win32.c\" and \"abc/def/win32.c\"\nwould cause confusion in the debugger (I've long since forgotten\nwhich). Breaking at win32.c:30 was no longer unique.\n\nAlso, if the Makefile sends all .o's to the root directory or a\nunified OBJS directory rather than to the subdir containing the .c,\nthen we have another issue during linking...\n\nSo, having been burned too many times, I prefer to make source\nfilenames unique when possible.\n\n\n> I've noticed you tend to prefer really long file and function names,\n> e.g. your borrowed daemonize() became\n> spawn_background_fsmonitor_daemon(), I think aiming for shorter\n> filenames & function names helps, e.g. these long names widen diffstats,\n> and many people who hack on the code stick religiously to 80 character\n> width terminals.\n> \n\nI prefer self-documenting code.\n\nJeff\n"},{"id":"429900","messageId":"58eee2a1-3fbd-e03e-2079-0d654f4b7e1f@jeffhostetler.com","threadId":"55420","inReplyTo":"xmqqsg0xreht.fsf@gitster.g","subject":"Re: [PATCH v3 23/34] t/helper/test-touch: add helper to touch a series of files","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T16:45:22Z","receivedAt":"2021-07-13T16:45:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 4:00 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> diff --git a/t/helper/test-touch.c b/t/helper/test-touch.c\n>> new file mode 100644\n>> index 00000000000..e9b3b754f1f\n>> --- /dev/null\n>> +++ b/t/helper/test-touch.c\n>> @@ -0,0 +1,126 @@\n>> +/*\n>> + * test-touch.c: variation on /usr/bin/touch to speed up tests\n>> + * with a large number of files (primarily on Windows where child\n>> + * process are very, very expensive).\n>> + */\n>> +\n>> +#include \"test-tool.h\"\n>> +#include \"cache.h\"\n>> +#include \"parse-options.h\"\n>> +\n>> +char *seq_pattern;\n>> +int seq_start = 1;\n>> +int seq_count = 1;\n> \n> With this in, \"make sparse\" dies like this:\n> \n> SP t/helper/test-touch.c\n> t/helper/test-touch.c:11:6: error: symbol 'seq_pattern' was not declared. Should it be static?\n> t/helper/test-touch.c:12:5: error: symbol 'seq_start' was not declared. Should it be static?\n> t/helper/test-touch.c:13:5: error: symbol 'seq_count' was not declared. Should it be static?\n> \n\nI'll fix. Thanks!\nJeff\n"},{"id":"429902","messageId":"c6793033-9bd4-e108-4a53-56c1dbd24a60@jeffhostetler.com","threadId":"55420","inReplyTo":"87h7hdbpgb.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T17:06:21Z","receivedAt":"2021-07-13T17:06:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 7:09 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Change p7519 to use a single \"test-tool touch\" command to update\n>> the mtime on a series of (thousands) files instead of invoking\n>> thousands of commands to update a single file.\n>>\n>> This is primarily for Windows where process creation is so\n>> very slow and reduces the test run time by minutes.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> t/perf/p7519-fsmonitor.sh | 14 ++++++--------\n>> 1 file changed, 6 insertions(+), 8 deletions(-)\n>>\n>> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n>> index 5eb5044a103..f74e6014a0a 100755\n>> --- a/t/perf/p7519-fsmonitor.sh\n>> +++ b/t/perf/p7519-fsmonitor.sh\n>> @@ -119,10 +119,11 @@ test_expect_success \"one time repo setup\" '\n>> \tfi &&\n>> \n>> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n>> -\tfor i in $(test_seq 1 10); do touch 10_files/$i; done &&\n>> -\tfor i in $(test_seq 1 100); do touch 100_files/$i; done &&\n>> -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i; done &&\n>> -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n>> +\ttest-tool touch sequence --pattern=\"10_files/%d\" --start=1 --count=10 &&\n>> +\ttest-tool touch sequence --pattern=\"100_files/%d\" --start=1 --count=100 &&\n>> +\ttest-tool touch sequence --pattern=\"1000_files/%d\" --start=1 --count=1000 &&\n>> +\ttest-tool touch sequence --pattern=\"10000_files/%d\" --start=1 --count=10000 &&\n>> +\n>> \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n>> \tgit commit -qm \"Add files\" &&\n>> \n>> @@ -200,15 +201,12 @@ test_fsmonitor_suite() {\n>> \t# Update the mtimes on upto 100k files to make status think\n>> \t# that they are dirty. For simplicity, omit any files with\n>> \t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n>> -\t# Then fully backslash-quote the paths to capture any\n>> -\t# whitespace so that they pass thru xargs properly.\n>> \t#\n>> \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n>> \t\tgit ls-files | \\\n>> \t\t\thead -100000 | \\\n>> \t\t\tgrep -v \\\" | \\\n>> -\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n>> -\t\t\txargs test-tool chmtime -300 &&\n>> +\t\t\ttest-tool touch stdin &&\n>> \t\tgit status\n>> \t'\n> \n> Did you try to replace this with some variant of:\n> \n> test_seq 1 10000 | xargs touch\n> \n> Which (depending on your xargs version) would invoke \"touch\" commands\n> with however many argv items it thinks you can handle.\n> \n\na quick test on my Windows machine shows that\n\n\ttest_seq 1 10000 | xargs touch\n\ntakes 3.1 seconds.\n\njust a simple\n\n\ttest_seq 1 10000 >/dev/null\n\ntake 0.2 seconds.\n\nusing my test-tool helper cuts that time in half.\n\nJeff\n"},{"id":"429905","messageId":"CABPp-BF+GiAZVxeC+MAr6pSssRpSi2pNwHLrxUp9HvrOhUBieg@mail.gmail.com","threadId":"55420","inReplyTo":"c6793033-9bd4-e108-4a53-56c1dbd24a60@jeffhostetler.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-13T17:36:23Z","receivedAt":"2021-07-13T17:36:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jul 13, 2021 at 10:10 AM Jeff Hostetler wrote:\n>\n> On 7/1/21 7:09 PM, Ævar Arnfjörð Bjarmason wrote:\n> >\n> > Did you try to replace this with some variant of:\n> >\n> > test_seq 1 10000 | xargs touch\n> >\n> > Which (depending on your xargs version) would invoke \"touch\" commands\n> > with however many argv items it thinks you can handle.\n>\n> a quick test on my Windows machine shows that\n>\n> test_seq 1 10000 | xargs touch\n>\n> takes 3.1 seconds.\n>\n> just a simple\n>\n> test_seq 1 10000 >/dev/null\n>\n> take 0.2 seconds.\n>\n> using my test-tool helper cuts that time in half.\n\nYeah, test_seq is pretty bad; it's just a loop in shell. Is there a\n'seq' on windows, and does using it instead of test_seq make things\nfaster with Ævar's suggested command?\n\nI'd really like to modify test_seq to use seq when it's available and\nfall back to the looping-in-shell when we need to for various\nplatforms.\n\nMaybe it'd even make sense to write a 'test-tool seq' and make\ntest_seq use that just so we can rip out that super lame shell\nlooping.\n"},{"id":"429907","messageId":"xmqq1r822jj8.fsf@gitster.g","threadId":"55420","inReplyTo":"CABPp-BF+GiAZVxeC+MAr6pSssRpSi2pNwHLrxUp9HvrOhUBieg@mail.gmail.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-13T17:47:07Z","receivedAt":"2021-07-13T17:47:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> On Tue, Jul 13, 2021 at 10:10 AM Jeff Hostetler wrote:\n>>\n>> a quick test on my Windows machine shows that\n>>\n>> test_seq 1 10000 | xargs touch\n>>\n>> takes 3.1 seconds.\n>>\n>> just a simple\n>>\n>> test_seq 1 10000 >/dev/null\n>>\n>> take 0.2 seconds.\n>>\n>> using my test-tool helper cuts that time in half.\n>\n> Yeah, test_seq is pretty bad; it's just a loop in shell. Is there a\n> 'seq' on windows, and does using it instead of test_seq make things\n> faster with Ævar's suggested command?\n\nUnless I am misreading Jeff's message, I do not think that makes\nsense. Counting to 10000 in shell loop is trivial (0.2 seconds),\nbut letting touch invoked 10000 times to create (or smudge mtime of,\nbut I suspect that is not what is going on here) 10000 files takes\n3.1 seconds, and of course a native binary that creates 10000 files\nwith a single invocation would be faster.\n\n> I'd really like to modify test_seq to use seq when it's available and\n> fall back to the looping-in-shell when we need to for various\n> platforms.\n\nSo, if I am reading Jeff correctly, that optimizes something that is\nnot a bottleneck.\n\n> Maybe it'd even make sense to write a 'test-tool seq' and make\n> test_seq use that just so we can rip out that super lame shell\n> looping.\n"},{"id":"429908","messageId":"CABPp-BEWL3U0gsmoHsCMQqy5a81yLSWi5U=AeyG023D2u+echw@mail.gmail.com","threadId":"55420","inReplyTo":"xmqq1r822jj8.fsf@gitster.g","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-13T17:50:54Z","receivedAt":"2021-07-13T17:51:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jul 13, 2021 at 10:47 AM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > On Tue, Jul 13, 2021 at 10:10 AM Jeff Hostetler wrote:\n> >>\n> >> a quick test on my Windows machine shows that\n> >>\n> >> test_seq 1 10000 | xargs touch\n> >>\n> >> takes 3.1 seconds.\n> >>\n> >> just a simple\n> >>\n> >> test_seq 1 10000 >/dev/null\n> >>\n> >> take 0.2 seconds.\n> >>\n> >> using my test-tool helper cuts that time in half.\n> >\n> > Yeah, test_seq is pretty bad; it's just a loop in shell. Is there a\n> > 'seq' on windows, and does using it instead of test_seq make things\n> > faster with Ævar's suggested command?\n>\n> Unless I am misreading Jeff's message, I do not think that makes\n> sense. Counting to 10000 in shell loop is trivial (0.2 seconds),\n> but letting touch invoked 10000 times to create (or smudge mtime of,\n> but I suspect that is not what is going on here) 10000 files takes\n> 3.1 seconds, and of course a native binary that creates 10000 files\n> with a single invocation would be faster.\n>\n> > I'd really like to modify test_seq to use seq when it's available and\n> > fall back to the looping-in-shell when we need to for various\n> > platforms.\n>\n> So, if I am reading Jeff correctly, that optimizes something that is\n> not a bottleneck.\n\nOh, indeed. Sorry, I misread.\n"},{"id":"429909","messageId":"87v95e2j71.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"533d47fc-ea37-1e6a-b6f4-ac4fdd24555e@jeffhostetler.com","subject":"Re: [PATCH v3 02/34] fsmonitor--daemon: man page","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-13T17:46:07Z","receivedAt":"2021-07-13T17:54:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 12 2021, Jeff Hostetler wrote:\n\n> On 7/1/21 6:29 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> From: Jeff Hostetler \n>>>\n>>> Create a manual page describing the `git fsmonitor--daemon` feature.\n>>>\n>>> Signed-off-by: Jeff Hostetler \n>>> ---\n>>> Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++\n>>> 1 file changed, 75 insertions(+)\n>>> create mode 100644 Documentation/git-fsmonitor--daemon.txt\n>>>\n>>> diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt\n>>> new file mode 100644\n>>> index 00000000000..154e7684daa\n>>> --- /dev/null\n>>> +++ b/Documentation/git-fsmonitor--daemon.txt\n>>> @@ -0,0 +1,75 @@\n>>> +git-fsmonitor--daemon(1)\n>>> +========================\n>>> +\n>>> +NAME\n>>> +----\n>>> +git-fsmonitor--daemon - A Built-in File System Monitor\n>>> +\n>>> +SYNOPSIS\n>>> +--------\n>>> +[verse]\n>>> +'git fsmonitor--daemon' start\n>>> +'git fsmonitor--daemon' run\n>>> +'git fsmonitor--daemon' stop\n>>> +'git fsmonitor--daemon' status\n>>> +\n>>> +DESCRIPTION\n>>> +-----------\n>>> +\n>>> +A daemon to watch the working directory for file and directory\n>>> +changes using platform-specific file system notification facilities.\n>>> +\n>>> +This daemon communicates directly with commands like `git status`\n>>> +using the link:technical/api-simple-ipc.html[simple IPC] interface\n>>> +instead of the slower linkgit:githooks[5] interface.\n>>> +\n>>> +This daemon is built into Git so that no third-party tools are\n>>> +required.\n>>> +\n>>> +OPTIONS\n>>> +-------\n>>> +\n>>> +start::\n>>> +\tStarts a daemon in the background.\n>>> +\n>>> +run::\n>>> +\tRuns a daemon in the foreground.\n>>> +\n>>> +stop::\n>>> +\tStops the daemon running in the current working\n>>> +\tdirectory, if present.\n>>> +\n>>> +status::\n>>> +\tExits with zero status if a daemon is watching the\n>>> +\tcurrent working directory.\n>>> +\n>>> +REMARKS\n>>> +-------\n>>> +\n>>> +This daemon is a long running process used to watch a single working\n>>> +directory and maintain a list of the recently changed files and\n>>> +directories. Performance of commands such as `git status` can be\n>>> +increased if they just ask for a summary of changes to the working\n>>> +directory and can avoid scanning the disk.\n>>> +\n>>> +When `core.useBuiltinFSMonitor` is set to `true` (see\n>>> +linkgit:git-config[1]) commands, such as `git status`, will ask the\n>>> +daemon for changes and automatically start it (if necessary).\n>>> +\n>>> +For more information see the \"File System Monitor\" section in\n>>> +linkgit:git-update-index[1].\n>>> +\n>>> +CAVEATS\n>>> +-------\n>>> +\n>>> +The fsmonitor daemon does not currently know about submodules and does\n>>> +not know to filter out file system events that happen within a\n>>> +submodule. If fsmonitor daemon is watching a super repo and a file is\n>>> +modified within the working directory of a submodule, it will report\n>>> +the change (as happening against the super repo). However, the client\n>>> +will properly ignore these extra events, so performance may be affected\n>>> +but it will not cause an incorrect result.\n>>> +\n>>> +GIT\n>>> +---\n>>> +Part of the linkgit:git[1] suite\n>> Later in the series we incrementally add features to the daemon, so\n>> this\n>> is describing a state that doesn't exist yet at this point.\n>> I think it would be better to start with a stup here and add\n>> documentation as we add features, e.g. the patch tha adds \"start\" should\n>> add that to the synopsis + options etc.\n>> See the outstanding ab/config-based-hooks-base for a small example\n>> of\n>> that.\n>> \n>\n> I like to lead the series with the documentation that summarizes the\n> purpose of the entire feature or patch series. This gives the reviewer\n> the context for the complete series that follows. In the past, we've\n> had discussions on the list about how hard it is to review a series when\n> the foo.c comes (alphabetically) before foo.h in the patch and all\n> the documentation is attached to the prototypes in the .h file so the\n> reviewer needs to bounce around in the patch or series to read the\n> intent and then go back to the beginning to read the code. In that\n> spirit, I think that having the complete man page come first provides\n> necessary context and is helpful.\n>\n> The argument that the man-page should grow as the feature grows\n> presumes that there is a meaningful cut-point mid-series where you\n> would adopt the first portion and delay the second to a later release\n> or something. That division would not be useful/usable.\n\nIsn't there such a meaningful cut-off point?\n\nIn 08/34[1] you add a skeleton of the daemon, so then the NAME/SYNOPSIS\n(empty)/DESCRIPTION/REMARKS/GIT sections could be added, in 09/34[3] you\nadd stop/status commands, so then the SYNOPSIS/OPTIONS could be\nupdated/created with those, same for the addition of run/start in\n13/34[5] and 14/35[6] respectively.\n\n> And it just clutters up later commits in the series with man-page\n> deltas.\n>\n> So I'd like to keep it as it unless there are further objections.\n\nI'll leave it to you to do with the feedback as you choose,\n\nI suggested this because I think it's much easier to read patches that\nare larger because they incrementally update docs or tests along with\ncode, than smaller ones that are e.g. \"add docs\" followed by\nincrementally modifying the code.\n\nThat's because you can consider those atomically. If earlier doc changes\nrefer to later code changes you're left jumping back & forth and\nwondering if the code you're reading that doesn't match the docs yet is\na bug, or if it's solved in some later change you're now needing to\nmentally keep track of.\n\nWhich is not some theoretical concern b.t.w., but exactly what I found\nmyself doing when reading this series, hence the suggestion.\n\n1. https://lore.kernel.org/git/f88db92d4259d1c29827e97e957daf6eda39c551.1625150864.git.gitgitgadget@gmail.com/\n2. https://lore.kernel.org/git/877di9d5uz.fsf@evledraar.gmail.com/\n3. https://lore.kernel.org/git/02e21384ef0ca4909e0bda2c78fa63c06be22a50.1625150864.git.gitgitgadget@gmail.com/\n4. https://lore.kernel.org/git/5d6646df93a17659af66f136295444d1bd834090.1625150864.git.gitgitgadget@gmail.com/\n5. https://lore.kernel.org/git/5d6646df93a17659af66f136295444d1bd834090.1625150864.git.gitgitgadget@gmail.com/\n6. https://lore.kernel.org/git/9fe902aad87f1192705fb69ea212a2d066d0286d.1625150864.git.gitgitgadget@gmail.com/\n"},{"id":"429910","messageId":"797f4e85-33cc-df7e-31d8-3a4f95a9109a@jeffhostetler.com","threadId":"55420","inReplyTo":"CABPp-BF+GiAZVxeC+MAr6pSssRpSi2pNwHLrxUp9HvrOhUBieg@mail.gmail.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T17:58:16Z","receivedAt":"2021-07-13T17:58:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/13/21 1:36 PM, Elijah Newren wrote:\n> On Tue, Jul 13, 2021 at 10:10 AM Jeff Hostetler wrote:\n>>\n>> On 7/1/21 7:09 PM, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> Did you try to replace this with some variant of:\n>>>\n>>> test_seq 1 10000 | xargs touch\n>>>\n>>> Which (depending on your xargs version) would invoke \"touch\" commands\n>>> with however many argv items it thinks you can handle.\n>>\n>> a quick test on my Windows machine shows that\n>>\n>> test_seq 1 10000 | xargs touch\n>>\n>> takes 3.1 seconds.\n>>\n>> just a simple\n>>\n>> test_seq 1 10000 >/dev/null\n>>\n>> take 0.2 seconds.\n>>\n>> using my test-tool helper cuts that time in half.\n> \n> Yeah, test_seq is pretty bad; it's just a loop in shell. Is there a\n> 'seq' on windows, and does using it instead of test_seq make things\n> faster with Ævar's suggested command?\n> \n\nThe Git for Windows SDK bash environment does have a /usr/bin/seq\nwhich appears to be from GNU coreutils 8.32. (This is different\nfrom the version that I have on my Mac (which doesn't have a version\nnumber).)\n\nUsing it:\n\n\tseq 1 10000 >/dev/null\n\ntakes 0.04 seconds instead of 0.2.\n\nHowever, it doesn't help the touch.\n\n\tseq 1 10000 | xargs touch\n\nstill takes ~3.1 seconds.\n\nFWIW, the xargs is clustering the 10,000 files into ~4 command lines,\nso there is a little bit of Windows process overhead, but not that\nmuch.\n\n\tseq 1 10000 | xargs wc -l | grep total\n\n> I'd really like to modify test_seq to use seq when it's available and\n> fall back to the looping-in-shell when we need to for various\n> platforms.\n> \n> Maybe it'd even make sense to write a 'test-tool seq' and make\n> test_seq use that just so we can rip out that super lame shell\n> looping.\n> \n"},{"id":"429911","messageId":"fae8cdcd-8222-b253-f656-d9f5b76de001@jeffhostetler.com","threadId":"55420","inReplyTo":"f1ef9656fc3adf079c8e40a74baeb5356bcf1586.1625150864.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T18:04:05Z","receivedAt":"2021-07-13T18:04:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 10:47 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Change p7519 to use a single \"test-tool touch\" command to update\n> the mtime on a series of (thousands) files instead of invoking\n> thousands of commands to update a single file.\n> \n> This is primarily for Windows where process creation is so\n> very slow and reduces the test run time by minutes.\n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> t/perf/p7519-fsmonitor.sh | 14 ++++++--------\n> 1 file changed, 6 insertions(+), 8 deletions(-)\n> \n> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n> index 5eb5044a103..f74e6014a0a 100755\n> --- a/t/perf/p7519-fsmonitor.sh\n> +++ b/t/perf/p7519-fsmonitor.sh\n> @@ -119,10 +119,11 @@ test_expect_success \"one time repo setup\" '\n> \tfi &&\n> \n> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n> -\tfor i in $(test_seq 1 10); do touch 10_files/$i; done &&\n> -\tfor i in $(test_seq 1 100); do touch 100_files/$i; done &&\n> -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i; done &&\n> -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n> +\ttest-tool touch sequence --pattern=\"10_files/%d\" --start=1 --count=10 &&\n> +\ttest-tool touch sequence --pattern=\"100_files/%d\" --start=1 --count=100 &&\n> +\ttest-tool touch sequence --pattern=\"1000_files/%d\" --start=1 --count=1000 &&\n> +\ttest-tool touch sequence --pattern=\"10000_files/%d\" --start=1 --count=10000 &&\n\nThe big win in taking *minutes* off of the run time of this\ntest was getting rid of the `for` loops and one `touch` invocation\nper file.\n\nSo whether we keep my `test-tool touch` command or switch to\n`test_seq` or `seq` is open for debate. Mine seems quicker, but\nit is more or less round off error in the larger picture considering\nwhat we started with.\n\nI'll play with this a bit.\nJeff\n\n"},{"id":"429912","messageId":"xmqqtuky141l.fsf@gitster.g","threadId":"55420","inReplyTo":"797f4e85-33cc-df7e-31d8-3a4f95a9109a@jeffhostetler.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-13T18:07:02Z","receivedAt":"2021-07-13T18:07:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> FWIW, the xargs is clustering the 10,000 files into ~4 command lines,\n> so there is a little bit of Windows process overhead, but not that\n> much.\n>\n> \tseq 1 10000 | xargs wc -l | grep total\n>\n>> I'd really like to modify test_seq to use seq when it's available and\n>> fall back to the looping-in-shell when we need to for various\n>> platforms.\n>> Maybe it'd even make sense to write a 'test-tool seq' and make\n>> test_seq use that just so we can rip out that super lame shell\n>> looping.\n>> \n\nSo what lame in this picture is not shell, or process overhead, but\nI/O performance.\n\nI've seen some noises about Windows file creation performance raised\nas an issue when doing initial checkout followed by \"git clone\", and\nan idea floated to create a bunch of open file handles for writing\nin threads when checkout (really the caller that repeatedly calls\nentry.c:write_entry() by iterating the in-core index) starts, and\nwrite out the contents in parallel, as a workaround. When I heard\nit, I somehow thought it was meant as a not-so-funny joke, but from\nthe sounds of it, the I/O performance may be so horrible to require\nsuch a hack to be usable there. Sigh...\n\n"},{"id":"429913","messageId":"87sg0i2iid.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"80fb703d-0f88-110e-5a5c-e8e9a90bbc17@jeffhostetler.com","subject":"Re: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-13T17:54:57Z","receivedAt":"2021-07-13T18:09:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 13 2021, Jeff Hostetler wrote:\n\n> My response here is in addition to Dscho's remarks on this topic.\n> He makes excellent points that I'll just #include here. I do want\n> to add my own $0.02 here.\n>\n> On 7/1/21 6:18 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> +#ifdef GIT_WINDOWS_NATIVE\n>>> +/*\n>>> + * Create a background process to run the daemon. It should be completely\n>>> + * disassociated from the terminal.\n>>> + *\n>>> + * Conceptually like `daemonize()` but different because Windows does not\n>>> + * have `fork(2)`. Spawn a normal Windows child process but without the\n>>> + * limitations of `start_command()` and `finish_command()`.\n>>> + *\n>>> + * The child process execs the \"git fsmonitor--daemon run\" command.\n>>> + *\n>>> + * The current process returns so that the caller can wait for the child\n>>> + * to startup before exiting.\n>>> + */\n>>> +static int spawn_background_fsmonitor_daemon(pid_t *pid)\n>>> +{\n>>> +\tchar git_exe[MAX_PATH];\n>>> +\tstruct strvec args = STRVEC_INIT;\n>>> +\tint in, out;\n>>> +\n>>> +\tGetModuleFileNameA(NULL, git_exe, MAX_PATH);\n>>> +\n>>> +\tin = open(\"/dev/null\", O_RDONLY);\n>>> +\tout = open(\"/dev/null\", O_WRONLY);\n>>> +\n>>> +\tstrvec_push(&args, git_exe);\n>>> +\tstrvec_push(&args, \"fsmonitor--daemon\");\n>>> +\tstrvec_push(&args, \"run\");\n>>> +\tstrvec_pushf(&args, \"--ipc-threads=%d\", fsmonitor__ipc_threads);\n>>> +\n>>> +\t*pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out);\n>>> +\tclose(in);\n>>> +\tclose(out);\n>>> +\n>>> +\tstrvec_clear(&args);\n>>> +\n>>> +\tif (*pid < 0)\n>>> +\t\treturn error(_(\"could not spawn fsmonitor--daemon in the background\"));\n>>> +\n>>> +\treturn 0;\n>>> +}\n>>> +#else\n>>> +/*\n>>> + * Create a background process to run the daemon. It should be completely\n>>> + * disassociated from the terminal.\n>>> + *\n>>> + * This is adapted from `daemonize()`. Use `fork()` to directly\n>>> + * create and run the daemon in the child process.\n>>> + *\n>>> + * The fork-child can just call the run code; it does not need to exec\n>>> + * it.\n>>> + *\n>>> + * The fork-parent returns the child PID so that we can wait for the\n>>> + * child to startup before exiting.\n>>> + */\n>>> +static int spawn_background_fsmonitor_daemon(pid_t *pid)\n>>> +{\n>>> +\t*pid = fork();\n>>> +\n>>> +\tswitch (*pid) {\n>>> +\tcase 0:\n>>> +\t\tif (setsid() == -1)\n>>> +\t\t\terror_errno(_(\"setsid failed\"));\n>>> +\t\tclose(0);\n>>> +\t\tclose(1);\n>>> +\t\tclose(2);\n>>> +\t\tsanitize_stdfds();\n>>> +\n>>> +\t\treturn !!fsmonitor_run_daemon();\n>>> +\n>>> +\tcase -1:\n>>> +\t\treturn error_errno(_(\"could not spawn fsmonitor--daemon in the background\"));\n>>> +\n>>> +\tdefault:\n>>> +\t\treturn 0;\n>>> +\t}\n>>> +}\n>>> +#endif\n>> The spawn_background_fsmonitor_daemon() function here is almost the\n>> same\n>> as daemonize(). I wonder if this & the Windows-specific one you have\n>> here can't be refactored into an API from what's now in setup.c.\n>> Then we could make builtin/gc.c and daemon.c use that, so Windows\n>> could\n>> have background GC, and we'd have a more battle-tested central codepath\n>> for this tricky bit.\n>> \n>\n> I'd rather not refactor all of this and add unnecessary generality\n> and complexity just to save duplicating some of the code in daemonize().\n>\n> And I'd rather not destabilize existing commands like gc and daemon\n> by changing the daemonize() layer on them. If those commands need help,\n> let's have a separate conversation _later_ about what help they need\n> and if it makes sense to combine them.\n\nJohannes suggested in\nhttps://lore.kernel.org/git/nycvar.QRO.7.76.6.2107052336480.8230@tvgsbejvaqbjf.bet/\nthat (if I understand that correctly, and I just skimmed the linked isse\nsome days ago), that even if such a refactoring was done these two\nfunctions are solving subtly different problems, or something. I.e. we\ncouldn't use it for daemonize().\n\nWhich I'd say is interesting for the code comments/commit message at\nleast, i.e. how they're solving subtly different problems (not being\nable to run this & not being able to test on Windows I haven't poked at\nit myself).\n\n>> It seems to me like the only limitations on it are to have this return\n>> slightly more general things (e.g. not set its own errors, return\n>> structured data), and maybe some callback for what to do in the\n>> child/parent.\n>\n> There are several issues here when trying to start a background process\n> and we're already on the edge of the behavioral differences between\n> Windows and Unix -- let's not make things more confusing with multiple\n> callbacks, returning structures, custom errors, and etc.\n>\n> Also, since Windows doesn't do fork(), we don't have child/parent\n> branches in the call, so this whole \"just pretend it's all Unix\"\n> model doesn't work.\n\nFair enough, And I think replied-to above.\n\n> Even if we did pretend I'd still need ifdef'd callback routines to\n> either call `fsmonitor_run_daemon()` or build a command line (or have\n> blocks of functions that \"just happen to never be called on one\n> platform or the other\").\n>\n>\n> What I have here is an API that the primary (read: parent) calls\n> and gets back a 0 or -1 (with error message). And that's it.\n> The primary can then wait for the child (whether from fork or\n> CreateProcess) to become responsive or fail to start. And then\n> the primary can exit (with or without error).\n>\n> So I think we're good. Yes, there is an ifdef here, but I think\n> it is worth it.\n\nFWIW what I was noting here & elsewhere is that yes, you need to ifdef\nsome of it, but the code you're proposing to add here is using a\ndifferent pattern than the one generally preferred in this codebase.\n\nI.e. check out how we do it for threading, we intentionally compile the\n\"if (thread) {code}\" clauses on platforms we know don't have threading,\nditto the code around PCRE in grep.c. \n\nSimilarly, here in e.g. spawn_background_fsmonitor_daemon just the\nGetModuleFileNameA() and mingw_spawnvpe() are Windows-specifics (and\ncould be calls to some helper that *is* ifdef'd).\n\nIn this case it's not a big deal, but as a general pattern it helps to\ne.g. avoid subtle syntax errors in nested ifdefs and the like, and\ngenerally encourages keeping the ifdef'd code as small as possible.\n\n>>> +/*\n>>> + * This is adapted from `wait_or_whine()`. Watch the child process and\n>>> + * let it get started and begin listening for requests on the socket\n>>> + * before reporting our success.\n>>> + */\n>>> +static int wait_for_background_startup(pid_t pid_child)\n>>> +{\n>>> +\tint status;\n>>> +\tpid_t pid_seen;\n>>> +\tenum ipc_active_state s;\n>>> +\ttime_t time_limit, now;\n>>> +\n>>> +\ttime(&time_limit);\n>>> +\ttime_limit += fsmonitor__start_timeout_sec;\n>>> +\n>>> +\tfor (;;) {\n>>> +\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n>>> +\n>>> +\t\tif (pid_seen == -1)\n>>> +\t\t\treturn error_errno(_(\"waitpid failed\"));\n>>> +\t\telse if (pid_seen == 0) {\n>>> +\t\t\t/*\n>>> +\t\t\t * The child is still running (this should be\n>>> +\t\t\t * the normal case). Try to connect to it on\n>>> +\t\t\t * the socket and see if it is ready for\n>>> +\t\t\t * business.\n>>> +\t\t\t *\n>>> +\t\t\t * If there is another daemon already running,\n>>> +\t\t\t * our child will fail to start (possibly\n>>> +\t\t\t * after a timeout on the lock), but we don't\n>>> +\t\t\t * care (who responds) if the socket is live.\n>>> +\t\t\t */\n>>> +\t\t\ts = fsmonitor_ipc__get_state();\n>>> +\t\t\tif (s == IPC_STATE__LISTENING)\n>>> +\t\t\t\treturn 0;\n>>> +\n>>> +\t\t\ttime(&now);\n>>> +\t\t\tif (now > time_limit)\n>>> +\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n>>> +\t\t} else if (pid_seen == pid_child) {\n>>> +\t\t\t/*\n>>> +\t\t\t * The new child daemon process shutdown while\n>>> +\t\t\t * it was starting up, so it is not listening\n>>> +\t\t\t * on the socket.\n>>> +\t\t\t *\n>>> +\t\t\t * Try to ping the socket in the odd chance\n>>> +\t\t\t * that another daemon started (or was already\n>>> +\t\t\t * running) while our child was starting.\n>>> +\t\t\t *\n>>> +\t\t\t * Again, we don't care who services the socket.\n>>> +\t\t\t */\n>>> +\t\t\ts = fsmonitor_ipc__get_state();\n>>> +\t\t\tif (s == IPC_STATE__LISTENING)\n>>> +\t\t\t\treturn 0;\n>>> +\n>>> +\t\t\t/*\n>>> +\t\t\t * We don't care about the WEXITSTATUS() nor\n>>> +\t\t\t * any of the WIF*(status) values because\n>>> +\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n>>> +\t\t\t * trick on all function return values.\n>>> +\t\t\t *\n>>> +\t\t\t * So it is sufficient to just report the\n>>> +\t\t\t * early shutdown as an error.\n>>> +\t\t\t */\n>>> +\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n>>> +\t\t} else\n>>> +\t\t\treturn error(_(\"waitpid is confused\"));\n>>> +\t}\n>>> +}\n>> Ditto this. could we extend the wait_or_whine() function (or some\n>> extended version thereof) to do what you need with callbacks?\n>> It seems the main difference is just being able to pass down a flag\n>> for\n>> waitpid(), and the loop needing to check EINTR or not depending on\n>> whether WNOHANG is passed.\n>> For e.g. the \"We don't care about the WEXITSTATUS()\" you'd get that\n>> behavior with an adjusted wait_or_whine(). Wouldn't it be better to\n>> report what exit status it exits with e.g. if the top-level process is\n>> signalled? We do so in trace2 for other things we spawn...\n>> \n>\n> Again, I don't want to mix my usage here with the existing code\n> and destabilize all existing callers. Here we are spinning to give\n> the child a chance to *start* and confirm that it is in a listening\n> state and ready for connections. We do not wait for the child to\n> exit (unless it dies quickly without becoming ready).\n>\n> We want to end our wait as soon as we confirm that the child is\n> ready and return. All I really need from the system is `waitpid()`.\n\nWill this code behave correctly if the daemon we start is signalled per\nthe WIFSIGNALED() cases the code this is derived handles, but this does\nnot?\n\nBut sure, I just meant to point out that the flip side to \"destabilize\nall existing callers\" is reviewing new code that may be subtly buggy,\nand those subtle bugs (if any) would be smoked out if we were forced to\nextend run-command.c, i.e. to use whatever feature(s) this needs for all\nexisting callers.\n"},{"id":"429914","messageId":"87pmvm2i8x.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"f38ed916-cfd8-06b2-43dc-689720a96a6e@jeffhostetler.com","subject":"Re: [PATCH v3 17/34] fsmonitor--daemon: define token-ids","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-13T18:11:38Z","receivedAt":"2021-07-13T18:14:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 13 2021, Jeff Hostetler wrote:\n\n> On 7/1/21 6:58 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> +\tif (!test_env_value) {\n>>> +\t\tstruct timeval tv;\n>>> +\t\tstruct tm tm;\n>>> +\t\ttime_t secs;\n>>> +\n>>> +\t\tgettimeofday(&tv, NULL);\n>>> +\t\tsecs = tv.tv_sec;\n>>> +\t\tgmtime_r(&secs, &tm);\n>>> +\n>>> +\t\tstrbuf_addf(&token->token_id,\n>>> +\t\t\t \"%\"PRIu64\".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ\",\n>>> +\t\t\t flush_count++,\n>>> +\t\t\t getpid(),\n>>> +\t\t\t tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,\n>>> +\t\t\t tm.tm_hour, tm.tm_min, tm.tm_sec,\n>>> +\t\t\t (long)tv.tv_usec);\n>> Just bikeshedding, but can we have tokens that mostly sort\n>> numeric-wise\n>> by time order? So time at the start, not the flush_count/getpid.\n>\n> As I described in a rather large comment in the code, tokens are opaque\n> strings -- without a less-than / greater-than relationship -- just a\n> random string that the daemon can use (along with a sequence number) to\n> ensure that a later request is well-defined.\n>\n> Here I'm using a counter, pid, and date-stamp. I'd prefer using a GUID\n> or UUID just to drive that home, but I didn't want to add a new .lib or\n> .a to the build if not necessary.\n>\n> Perhaps I should compute this portion as hex(hash(time())) to remove the\n> temptation to look inside my opaque token ??\n\nWhy does it matter if someone looks inside your opaque token if the code\nis treating it as opaque by just doing a strcmp(old,new) on it?\n\nI just suggested this as a debugging aid, i.e. when you the human (as\nopposed to the program) are looking at this behavior it's handy to look\nat the token and see that your cookies don't match, and that they look\nto be N seconds apart.\n\nAnd furthermore, if git crashes or whatever you can now easily look up\nwhat process crashed if you've got the leftover cookie, if you've also\ngot trace2 logs.\n\n>> Maybe I'm missing something, but couldn't we just re-use the trace2\n>> SID\n>> + a more trivial trailer? It would have the nice property that you could\n>> find the trace2 SID whenever you looked at such a token (could\n>> e.g. split them by \"/\" too), and add the tv_usec, flush_count+whatever\n>> else is needed to make it unique after the \"/\", no?\n>> \n>\n> I would rather keep Trace2 out of this. The SID is another opaque\n> string and I don't want to reach inside it.\n\nFor the purposes of the git.git codebase it's fine to reach inside of\nit, especially for a \"I'd like a near-enough-UUID, and I know the trace2\nSID already does that per-program\", so you just need e.g. a sequence\ncounter within the program to ensure global uniqueness with other git\nprocesses for such a cookie.\n"},{"id":"429915","messageId":"87mtqq2i3r.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"b19f3f2a-049f-acf2-f59e-de705dc54307@jeffhostetler.com","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-13T18:15:07Z","receivedAt":"2021-07-13T18:18:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 13 2021, Jeff Hostetler wrote:\n\n> On 7/1/21 7:02 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> From: Jeff Hostetler \n>>>\n>>> Teach the win32 backend to register a watch on the working tree\n>>> root directory (recursively). Also watch the if it is\n>>> not inside the working tree. And to collect path change notifications\n>>> into batches and publish.\n>>>\n>>> Signed-off-by: Jeff Hostetler \n>>> ---\n>>> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++\n>> Spying on the early history of this (looking for the\n>> Linux backend) I saw that at some point we had just\n>> compat/fsmonitor/linux.c, and presumably some of\n>> compat/fsmonitor/{windows,win32,macos,darwin}.c.\n>> At some point those filenames became much much longer.\n>> \n>\n> Once upon a time having \"foo/bar/win32.c\" and \"abc/def/win32.c\"\n> would cause confusion in the debugger (I've long since forgotten\n> which). Breaking at win32.c:30 was no longer unique.\n>\n> Also, if the Makefile sends all .o's to the root directory or a\n> unified OBJS directory rather than to the subdir containing the .c,\n> then we have another issue during linking...\n>\n> So, having been burned too many times, I prefer to make source\n> filenames unique when possible.\n\nA much shorter name like compat/fsmonitor/fsmon-win32.c would achieve\nthat goal.\n\n>> I've noticed you tend to prefer really long file and function names,\n>> e.g. your borrowed daemonize() became\n>> spawn_background_fsmonitor_daemon(), I think aiming for shorter\n>> filenames & function names helps, e.g. these long names widen diffstats,\n>> and many people who hack on the code stick religiously to 80 character\n>> width terminals.\n>> \n>\n> I prefer self-documenting code.\n\nSure, I'm not saying daemonize() is an ideal name, just suggesting that\nyou can both get uniqueness & self-documentation and not need to split\nto multiple lines in some common cases to stay within the \"We try to\nkeep to at most 80 characters per line\" in CodingGuidelines in this\nseries.\n"},{"id":"429916","messageId":"042e011c-1e13-0400-59d8-030481c5f329@jeffhostetler.com","threadId":"55420","inReplyTo":"xmqqtuky141l.fsf@gitster.g","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T18:19:16Z","receivedAt":"2021-07-13T18:19:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/13/21 2:07 PM, Junio C Hamano wrote:\n> Jeff Hostetler writes:\n> \n>> FWIW, the xargs is clustering the 10,000 files into ~4 command lines,\n>> so there is a little bit of Windows process overhead, but not that\n>> much.\n>>\n>> \tseq 1 10000 | xargs wc -l | grep total\n>>\n>>> I'd really like to modify test_seq to use seq when it's available and\n>>> fall back to the looping-in-shell when we need to for various\n>>> platforms.\n>>> Maybe it'd even make sense to write a 'test-tool seq' and make\n>>> test_seq use that just so we can rip out that super lame shell\n>>> looping.\n>>>\n> \n> So what lame in this picture is not shell, or process overhead, but\n> I/O performance.\n> \n> I've seen some noises about Windows file creation performance raised\n> as an issue when doing initial checkout followed by \"git clone\", and\n> an idea floated to create a bunch of open file handles for writing\n> in threads when checkout (really the caller that repeatedly calls\n> entry.c:write_entry() by iterating the in-core index) starts, and\n> write out the contents in parallel, as a workaround. When I heard\n> it, I somehow thought it was meant as a not-so-funny joke, but from\n> the sounds of it, the I/O performance may be so horrible to require\n> such a hack to be usable there. Sigh...\n> \n\nYes, there are some things here (that I believe to be I/O related)\non Windows that I want to look at when I wrap up FSMonitor. And\nyes, some of them sound pretty stupid.\n\nJeff\n\n"},{"id":"429917","messageId":"87k0lu2hw8.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"c6793033-9bd4-e108-4a53-56c1dbd24a60@jeffhostetler.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-13T18:18:09Z","receivedAt":"2021-07-13T18:22:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 13 2021, Jeff Hostetler wrote:\n\n> On 7/1/21 7:09 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> From: Jeff Hostetler \n>>>\n>>> Change p7519 to use a single \"test-tool touch\" command to update\n>>> the mtime on a series of (thousands) files instead of invoking\n>>> thousands of commands to update a single file.\n>>>\n>>> This is primarily for Windows where process creation is so\n>>> very slow and reduces the test run time by minutes.\n>>>\n>>> Signed-off-by: Jeff Hostetler \n>>> ---\n>>> t/perf/p7519-fsmonitor.sh | 14 ++++++--------\n>>> 1 file changed, 6 insertions(+), 8 deletions(-)\n>>>\n>>> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n>>> index 5eb5044a103..f74e6014a0a 100755\n>>> --- a/t/perf/p7519-fsmonitor.sh\n>>> +++ b/t/perf/p7519-fsmonitor.sh\n>>> @@ -119,10 +119,11 @@ test_expect_success \"one time repo setup\" '\n>>> \tfi &&\n>>> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n>>> -\tfor i in $(test_seq 1 10); do touch 10_files/$i; done &&\n>>> -\tfor i in $(test_seq 1 100); do touch 100_files/$i; done &&\n>>> -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i; done &&\n>>> -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n>>> +\ttest-tool touch sequence --pattern=\"10_files/%d\" --start=1 --count=10 &&\n>>> +\ttest-tool touch sequence --pattern=\"100_files/%d\" --start=1 --count=100 &&\n>>> +\ttest-tool touch sequence --pattern=\"1000_files/%d\" --start=1 --count=1000 &&\n>>> +\ttest-tool touch sequence --pattern=\"10000_files/%d\" --start=1 --count=10000 &&\n>>> +\n>>> \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n>>> \tgit commit -qm \"Add files\" &&\n>>> @@ -200,15 +201,12 @@ test_fsmonitor_suite() {\n>>> \t# Update the mtimes on upto 100k files to make status think\n>>> \t# that they are dirty. For simplicity, omit any files with\n>>> \t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n>>> -\t# Then fully backslash-quote the paths to capture any\n>>> -\t# whitespace so that they pass thru xargs properly.\n>>> \t#\n>>> \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n>>> \t\tgit ls-files | \\\n>>> \t\t\thead -100000 | \\\n>>> \t\t\tgrep -v \\\" | \\\n>>> -\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n>>> -\t\t\txargs test-tool chmtime -300 &&\n>>> +\t\t\ttest-tool touch stdin &&\n>>> \t\tgit status\n>>> \t'\n>> Did you try to replace this with some variant of:\n>> test_seq 1 10000 | xargs touch\n>> Which (depending on your xargs version) would invoke \"touch\"\n>> commands\n>> with however many argv items it thinks you can handle.\n>> \n>\n> a quick test on my Windows machine shows that\n>\n> \ttest_seq 1 10000 | xargs touch\n>\n> takes 3.1 seconds.\n>\n> just a simple\n>\n> \ttest_seq 1 10000 >/dev/null\n>\n> take 0.2 seconds.\n>\n> using my test-tool helper cuts that time in half.\n\nThere's what Elijah mentioned about test_seq, so maybe it's just that.\n\nBut what I was suggesting was using the xargs mode where it does N\narguments at a time.\n\nDoes this work for you, and does it cause xargs to invoke \"touch\" with\nthe relevant N number of arguments, and does it help with the\nperformance?\n\n test_seq 1 10000 | xargs touch\n test_seq 1 10000 | xargs -n 10 touch\n test_seq 1 10000 | xargs -n 100 touch\n test_seq 1 10000 | xargs -n 1000 touch\n\netc.\n\nAlso I didn't notice this before, but the -300 part of \"chmtime -300\"\nwas redundant before then? I.e. you're implicitly changing it to \"=+0\"\ninstead with your \"touch\" helper, are you not?\n"},{"id":"429919","messageId":"55e15a7a-b15e-78bd-b3c3-4adff3363edd@jeffhostetler.com","threadId":"55420","inReplyTo":"87sg0i2iid.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T18:44:41Z","receivedAt":"2021-07-13T18:44:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/13/21 1:54 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Jul 13 2021, Jeff Hostetler wrote:\n> \n>> My response here is in addition to Dscho's remarks on this topic.\n>> He makes excellent points that I'll just #include here. I do want\n>> to add my own $0.02 here.\n>>\n>> On 7/1/21 6:18 PM, Ævar Arnfjörð Bjarmason wrote:\n>>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>>>\n\n>>>> +/*\n>>>> + * This is adapted from `wait_or_whine()`. Watch the child process and\n>>>> + * let it get started and begin listening for requests on the socket\n>>>> + * before reporting our success.\n>>>> + */\n>>>> +static int wait_for_background_startup(pid_t pid_child)\n>>>> +{\n>>>> +\tint status;\n>>>> +\tpid_t pid_seen;\n>>>> +\tenum ipc_active_state s;\n>>>> +\ttime_t time_limit, now;\n>>>> +\n>>>> +\ttime(&time_limit);\n>>>> +\ttime_limit += fsmonitor__start_timeout_sec;\n>>>> +\n>>>> +\tfor (;;) {\n>>>> +\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n>>>> +\n>>>> +\t\tif (pid_seen == -1)\n>>>> +\t\t\treturn error_errno(_(\"waitpid failed\"));\n>>>> +\t\telse if (pid_seen == 0) {\n>>>> +\t\t\t/*\n>>>> +\t\t\t * The child is still running (this should be\n>>>> +\t\t\t * the normal case). Try to connect to it on\n>>>> +\t\t\t * the socket and see if it is ready for\n>>>> +\t\t\t * business.\n>>>> +\t\t\t *\n>>>> +\t\t\t * If there is another daemon already running,\n>>>> +\t\t\t * our child will fail to start (possibly\n>>>> +\t\t\t * after a timeout on the lock), but we don't\n>>>> +\t\t\t * care (who responds) if the socket is live.\n>>>> +\t\t\t */\n>>>> +\t\t\ts = fsmonitor_ipc__get_state();\n>>>> +\t\t\tif (s == IPC_STATE__LISTENING)\n>>>> +\t\t\t\treturn 0;\n>>>> +\n>>>> +\t\t\ttime(&now);\n>>>> +\t\t\tif (now > time_limit)\n>>>> +\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n>>>> +\t\t} else if (pid_seen == pid_child) {\n>>>> +\t\t\t/*\n>>>> +\t\t\t * The new child daemon process shutdown while\n>>>> +\t\t\t * it was starting up, so it is not listening\n>>>> +\t\t\t * on the socket.\n>>>> +\t\t\t *\n>>>> +\t\t\t * Try to ping the socket in the odd chance\n>>>> +\t\t\t * that another daemon started (or was already\n>>>> +\t\t\t * running) while our child was starting.\n>>>> +\t\t\t *\n>>>> +\t\t\t * Again, we don't care who services the socket.\n>>>> +\t\t\t */\n>>>> +\t\t\ts = fsmonitor_ipc__get_state();\n>>>> +\t\t\tif (s == IPC_STATE__LISTENING)\n>>>> +\t\t\t\treturn 0;\n>>>> +\n>>>> +\t\t\t/*\n>>>> +\t\t\t * We don't care about the WEXITSTATUS() nor\n>>>> +\t\t\t * any of the WIF*(status) values because\n>>>> +\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n>>>> +\t\t\t * trick on all function return values.\n>>>> +\t\t\t *\n>>>> +\t\t\t * So it is sufficient to just report the\n>>>> +\t\t\t * early shutdown as an error.\n>>>> +\t\t\t */\n>>>> +\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n>>>> +\t\t} else\n>>>> +\t\t\treturn error(_(\"waitpid is confused\"));\n>>>> +\t}\n>>>> +}\n>>> Ditto this. could we extend the wait_or_whine() function (or some\n>>> extended version thereof) to do what you need with callbacks?\n>>> It seems the main difference is just being able to pass down a flag\n>>> for\n>>> waitpid(), and the loop needing to check EINTR or not depending on\n>>> whether WNOHANG is passed.\n>>> For e.g. the \"We don't care about the WEXITSTATUS()\" you'd get that\n>>> behavior with an adjusted wait_or_whine(). Wouldn't it be better to\n>>> report what exit status it exits with e.g. if the top-level process is\n>>> signalled? We do so in trace2 for other things we spawn...\n>>>\n>>\n>> Again, I don't want to mix my usage here with the existing code\n>> and destabilize all existing callers. Here we are spinning to give\n>> the child a chance to *start* and confirm that it is in a listening\n>> state and ready for connections. We do not wait for the child to\n>> exit (unless it dies quickly without becoming ready).\n>>\n>> We want to end our wait as soon as we confirm that the child is\n>> ready and return. All I really need from the system is `waitpid()`.\n> \n> Will this code behave correctly if the daemon we start is signalled per\n> the WIFSIGNALED() cases the code this is derived handles, but this does\n> not?\n\nWe're only waiting until the child gets started and is able to receive\nrequests -- what happens to it after we have confirmed that it is ready\nis not our concern (after all, the parent is about to exit anyway and\nthe child is going to continue on).\n\nIf waitpid() gives us a WIFSIGNALED (or any other WIF*() state) before\nwe have spoken to it, we will return a \"failed to start\".\n\nBut again, that signal would have to arrive immediately after we spawned\nit and *before* we could talk to it. If the child is signaled after we\nconfirmed it was ready, we don't care because the parent process will be\ngone.\n\n(If the child is signaled or is killed (or crashes or whatever), the\nnext Git command (like \"status\") that tries to talk to it will re-start\nit implicitly -- the `git fsmonitor--daemon start` command here is an\nexplicit start.)\n\n\n> \n> But sure, I just meant to point out that the flip side to \"destabilize\n> all existing callers\" is reviewing new code that may be subtly buggy,\n> and those subtle bugs (if any) would be smoked out if we were forced to\n> extend run-command.c, i.e. to use whatever feature(s) this needs for all\n> existing callers.\n> \n\nThat would/could have a massive footprint. And I've already established\nthat my usage here is sufficiently different from existing uses that the\nresult would be a mess. IMHO.\n\nJeff\n"},{"id":"429920","messageId":"fc37043d-5a6c-ae8a-09a9-8498c3715d43@jeffhostetler.com","threadId":"55420","inReplyTo":"87k0lu2hw8.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-13T19:05:18Z","receivedAt":"2021-07-13T19:05:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/13/21 2:18 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Jul 13 2021, Jeff Hostetler wrote:\n> \n>> On 7/1/21 7:09 PM, Ævar Arnfjörð Bjarmason wrote:\n>>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>>>\n>>>> From: Jeff Hostetler \n>>>>\n>>>> Change p7519 to use a single \"test-tool touch\" command to update\n>>>> the mtime on a series of (thousands) files instead of invoking\n>>>> thousands of commands to update a single file.\n>>>>\n>>>> This is primarily for Windows where process creation is so\n>>>> very slow and reduces the test run time by minutes.\n>>>>\n>>>> Signed-off-by: Jeff Hostetler \n>>>> ---\n>>>> t/perf/p7519-fsmonitor.sh | 14 ++++++--------\n>>>> 1 file changed, 6 insertions(+), 8 deletions(-)\n>>>>\n>>>> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n>>>> index 5eb5044a103..f74e6014a0a 100755\n>>>> --- a/t/perf/p7519-fsmonitor.sh\n>>>> +++ b/t/perf/p7519-fsmonitor.sh\n>>>> @@ -119,10 +119,11 @@ test_expect_success \"one time repo setup\" '\n>>>> \tfi &&\n>>>> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n>>>> -\tfor i in $(test_seq 1 10); do touch 10_files/$i; done &&\n>>>> -\tfor i in $(test_seq 1 100); do touch 100_files/$i; done &&\n>>>> -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i; done &&\n>>>> -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n>>>> +\ttest-tool touch sequence --pattern=\"10_files/%d\" --start=1 --count=10 &&\n>>>> +\ttest-tool touch sequence --pattern=\"100_files/%d\" --start=1 --count=100 &&\n>>>> +\ttest-tool touch sequence --pattern=\"1000_files/%d\" --start=1 --count=1000 &&\n>>>> +\ttest-tool touch sequence --pattern=\"10000_files/%d\" --start=1 --count=10000 &&\n>>>> +\n>>>> \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n>>>> \tgit commit -qm \"Add files\" &&\n>>>> @@ -200,15 +201,12 @@ test_fsmonitor_suite() {\n>>>> \t# Update the mtimes on upto 100k files to make status think\n>>>> \t# that they are dirty. For simplicity, omit any files with\n>>>> \t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n>>>> -\t# Then fully backslash-quote the paths to capture any\n>>>> -\t# whitespace so that they pass thru xargs properly.\n>>>> \t#\n>>>> \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n>>>> \t\tgit ls-files | \\\n>>>> \t\t\thead -100000 | \\\n>>>> \t\t\tgrep -v \\\" | \\\n>>>> -\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n>>>> -\t\t\txargs test-tool chmtime -300 &&\n>>>> +\t\t\ttest-tool touch stdin &&\n>>>> \t\tgit status\n>>>> \t'\n>>> Did you try to replace this with some variant of:\n>>> test_seq 1 10000 | xargs touch\n>>> Which (depending on your xargs version) would invoke \"touch\"\n>>> commands\n>>> with however many argv items it thinks you can handle.\n>>>\n>>\n>> a quick test on my Windows machine shows that\n>>\n>> \ttest_seq 1 10000 | xargs touch\n>>\n>> takes 3.1 seconds.\n>>\n>> just a simple\n>>\n>> \ttest_seq 1 10000 >/dev/null\n>>\n>> take 0.2 seconds.\n>>\n>> using my test-tool helper cuts that time in half.\n> \n> There's what Elijah mentioned about test_seq, so maybe it's just that.\n> \n> But what I was suggesting was using the xargs mode where it does N\n> arguments at a time.\n> \n> Does this work for you, and does it cause xargs to invoke \"touch\" with\n> the relevant N number of arguments, and does it help with the\n> performance?\n> \n> test_seq 1 10000 | xargs touch\n> test_seq 1 10000 | xargs -n 10 touch\n> test_seq 1 10000 | xargs -n 100 touch\n> test_seq 1 10000 | xargs -n 1000 touch\n\nThe GFW SDK version of xargs does have `-n N` and it does work as\nadvertised. And it does slow down things considerably. Letting it\ndo ~2500 per command in 4 commands took the 3.1 seconds listed above.\n\nAdd a -n 100 to it takes 5.7 seconds, so process creation overhead\nis a factor here.\n\n\n> \n> etc.\n> \n> Also I didn't notice this before, but the -300 part of \"chmtime -300\"\n> was redundant before then? I.e. you're implicitly changing it to \"=+0\"\n> instead with your \"touch\" helper, are you not?\n> \n\nRight. I'm changing it to the current time.\n\nJeff\n"},{"id":"430346","messageId":"nycvar.QRO.7.76.6.2107161732150.59@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"87v95e2j71.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 02/34] fsmonitor--daemon: man page","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-16T15:45:37Z","receivedAt":"2021-07-16T15:45:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Tue, 13 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> [snip Ævar's suggestion to populate the manual page incrementally,\n> interspersed with the commits that finalize implementing the respective\n> functionality]\n>\n> I suggested this because I think it's much easier to read patches that\n> are larger because they incrementally update docs or tests along with\n> code, than smaller ones that are e.g. \"add docs\" followed by\n> incrementally modifying the code.\n\nMy experience is the exact opposite of yours: shorter patches are easier\nto read.\n\n> That's because you can consider those atomically.\n\nNo, in a patch series you cannot consider any patch completely atomically.\nJust like you don't consider any paragraph in any well-written book out of\ncontext.\n\n> If earlier doc changes refer to later code changes you're left jumping\n> back & forth and wondering if the code you're reading that doesn't match\n> the docs yet is a bug, or if it's solved in some later change you're now\n> needing to mentally keep track of.\n\nYou only keep jumping back and forth when reviewing _patches_. We try to\ndo code review on this mailing list, which means that you have the code\nlocally and review it in the correct context. When you do that, a design\ndocument is quite helpful. And the proposed manual page serves as such a\ndesign document.\n\n> Which is not some theoretical concern b.t.w., but exactly what I found\n> myself doing when reading this series, hence the suggestion.\n\nPart of the problem here seems to be that this patch series saw many\nreviewer suggestions that forced it to increase in length. That was\nprobably not very helpful, after all.\n\nThe first iteration had 23 patches. Reviews forced it to grow to 28\npatches in the second iteration. And that was still not enough, therefore\nthe third iteration consisted of 34 patches. And if you had had your way,\nrequiring Jeff to include a Linux backend in the same patch series, it\nwould have to increase in size again, and not just by a little.\n\nThis all sounds like we're truly falling into the trap of ignoring the\nrule that the perfect is the enemy of the good.\n\nI really would like to come back to a focused review that truly improves\nthe patches at hand, and avoids conflating the review of the actual patch\nseries with matters of personal taste (which is a recipe for\ndisagreement). After all, we are interested in getting this feature out to\nusers who need to work with very large worktrees, right? At least that's\nmy goal here.\n\nCiao,\nJohannes\n"},{"id":"430347","messageId":"nycvar.QRO.7.76.6.2107161747010.59@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"87v95tbqgh.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-16T15:47:26Z","receivedAt":"2021-07-16T15:47:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Fri, 2 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n\n>\n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>\n> > From: Jeff Hostetler \n> >\n> > Stub in empty backend for fsmonitor--daemon on Windows.\n> >\n> > Signed-off-by: Jeff Hostetler \n> > ---\n> > Makefile | 13 ++++++\n> > compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++\n> > compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++\n> > config.mak.uname | 2 +\n> > contrib/buildsystems/CMakeLists.txt | 5 ++\n> > 5 files changed, 90 insertions(+)\n> > create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n> > create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n> >\n> > diff --git a/Makefile b/Makefile\n> > index c45caacf2c3..a2a6e1f20f6 100644\n> > --- a/Makefile\n> > +++ b/Makefile\n> > @@ -467,6 +467,11 @@ all::\n> > # directory, and the JSON compilation database 'compile_commands.json' will be\n> > # created at the root of the repository.\n> > #\n> > +# If your platform supports a built-in fsmonitor backend, set\n> > +# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n> > +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n> > +# `fsmonitor_fs_listen__*()` routines.\n> > +#\n> > # Define DEVELOPER to enable more compiler warnings. Compiler version\n> > # and family are auto detected, but could be overridden by defining\n> > # COMPILER_FEATURES (see config.mak.dev). You can still set\n> > @@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n> > \tCOMPAT_OBJS += compat/access.o\n> > endif\n> >\n> > +ifdef FSMONITOR_DAEMON_BACKEND\n> > +\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n> > +\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n> > +endif\n> > +\n> > ifeq ($(TCLTK_PATH),)\n> > NO_TCLTK = NoThanks\n> > endif\n> > @@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE\n> > \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n> > \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n> > \t@echo X=\\'$(X)\\' >>$@+\n> > +ifdef FSMONITOR_DAEMON_BACKEND\n> > +\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n> > +endif\n>\n> Why put this in an ifdef?\n\nWhy not? What benefit does this question bring to improving this patch\nseries?\n\nCiao,\nDscho\n"},{"id":"430348","messageId":"nycvar.QRO.7.76.6.2107161748050.59@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"87sg0xbq9v.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-16T15:51:02Z","receivedAt":"2021-07-16T15:51:11Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Fri, 2 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>\n> > From: Jeff Hostetler \n> >\n> > Stub in empty implementation of fsmonitor--daemon\n> > backend for MacOS.\n> >\n> > Signed-off-by: Jeff Hostetler \n> > ---\n> > compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++\n> > config.mak.uname | 2 ++\n> > contrib/buildsystems/CMakeLists.txt | 3 +++\n> > 3 files changed, 25 insertions(+)\n> > create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c\n> >\n> > diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n> > new file mode 100644\n> > index 00000000000..b91058d1c4f\n> > --- /dev/null\n> > +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n> > @@ -0,0 +1,20 @@\n> > +#include \"cache.h\"\n> > +#include \"fsmonitor.h\"\n> > +#include \"fsmonitor-fs-listen.h\"\n> > +\n> > +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n> > +{\n> > +\treturn -1;\n> > +}\n> > +\n> > +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n> > +{\n> > +}\n> > +\n> > +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n> > +{\n> > +}\n> > +\n> > +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n> > +{\n> > +}\n> > diff --git a/config.mak.uname b/config.mak.uname\n> > index fcd88b60b14..394355463e1 100644\n> > --- a/config.mak.uname\n> > +++ b/config.mak.uname\n> > @@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin)\n> > \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n> > \t\tendif\n> > \tendif\n> > +\tFSMONITOR_DAEMON_BACKEND = macos\n>\n> A rather trivial point, but can't we pick one of \"macos\" or \"darwin\"\n> (I'd think going with the existing uname is better) and name the file\n> after the uname (or lower-case thereof)?\n>\n> Makes these make rules more consistent too, we could just set this to\n> \"YesPlease\" here, and then lower case the uname for the file\n> compilation/include.\n\nSo you suggest that we name the new stuff after an `uname` that reflects a\nname that is no longer relevant? I haven't seen a real Darwin system in\nquite a long time, have you?\n\nI don't find such a suggestion constructive, I have to admit.\n\nCiao,\nJohannes\n"},{"id":"430349","messageId":"nycvar.QRO.7.76.6.2107161754180.59@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"87mtqq2i3r.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-16T15:55:28Z","receivedAt":"2021-07-16T15:55:38Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Tue, 13 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n\n>\n> On Tue, Jul 13 2021, Jeff Hostetler wrote:\n>\n> > On 7/1/21 7:02 PM, Ævar Arnfjörð Bjarmason wrote:\n> >> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> >>\n> >>> From: Jeff Hostetler \n> >>>\n> >>> Teach the win32 backend to register a watch on the working tree\n> >>> root directory (recursively). Also watch the if it is\n> >>> not inside the working tree. And to collect path change notifications\n> >>> into batches and publish.\n> >>>\n> >>> Signed-off-by: Jeff Hostetler \n> >>> ---\n> >>> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++\n> >> Spying on the early history of this (looking for the\n> >> Linux backend) I saw that at some point we had just\n> >> compat/fsmonitor/linux.c, and presumably some of\n> >> compat/fsmonitor/{windows,win32,macos,darwin}.c.\n> >> At some point those filenames became much much longer.\n> >>\n> >\n> > Once upon a time having \"foo/bar/win32.c\" and \"abc/def/win32.c\"\n> > would cause confusion in the debugger (I've long since forgotten\n> > which). Breaking at win32.c:30 was no longer unique.\n> >\n> > Also, if the Makefile sends all .o's to the root directory or a\n> > unified OBJS directory rather than to the subdir containing the .c,\n> > then we have another issue during linking...\n> >\n> > So, having been burned too many times, I prefer to make source\n> > filenames unique when possible.\n>\n> A much shorter name like compat/fsmonitor/fsmon-win32.c would achieve\n> that goal.\n>\n> >> I've noticed you tend to prefer really long file and function names,\n> >> e.g. your borrowed daemonize() became\n> >> spawn_background_fsmonitor_daemon(), I think aiming for shorter\n> >> filenames & function names helps, e.g. these long names widen diffstats,\n> >> and many people who hack on the code stick religiously to 80 character\n> >> width terminals.\n> >>\n> >\n> > I prefer self-documenting code.\n>\n> Sure, I'm not saying daemonize() is an ideal name, just suggesting that\n> you can both get uniqueness & self-documentation and not need to split\n> to multiple lines in some common cases to stay within the \"We try to\n> keep to at most 80 characters per line\" in CodingGuidelines in this\n> series.\n\nWhile you are entitled to have your taste, I have to point out that Jeff\nis just as entitled to their taste, and I don't think that you can claim\nthat yours is better.\n\nSo I wonder what the intended outcome of this review is? To make the patch\nbetter? Or to pit taste against taste?\n\nCiao,\nJohannes\n"},{"id":"430353","messageId":"874kcuxkz8.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107161754180.59@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-16T16:27:44Z","receivedAt":"2021-07-16T16:49:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 16 2021, Johannes Schindelin wrote:\n\n> Hi Ævar,\n>\n> On Tue, 13 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n>>\n>> On Tue, Jul 13 2021, Jeff Hostetler wrote:\n>>\n>> > On 7/1/21 7:02 PM, Ævar Arnfjörð Bjarmason wrote:\n>> >> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>> >>\n>> >>> From: Jeff Hostetler \n>> >>>\n>> >>> Teach the win32 backend to register a watch on the working tree\n>> >>> root directory (recursively). Also watch the if it is\n>> >>> not inside the working tree. And to collect path change notifications\n>> >>> into batches and publish.\n>> >>>\n>> >>> Signed-off-by: Jeff Hostetler \n>> >>> ---\n>> >>> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++\n>> >> Spying on the early history of this (looking for the\n>> >> Linux backend) I saw that at some point we had just\n>> >> compat/fsmonitor/linux.c, and presumably some of\n>> >> compat/fsmonitor/{windows,win32,macos,darwin}.c.\n>> >> At some point those filenames became much much longer.\n>> >>\n>> >\n>> > Once upon a time having \"foo/bar/win32.c\" and \"abc/def/win32.c\"\n>> > would cause confusion in the debugger (I've long since forgotten\n>> > which). Breaking at win32.c:30 was no longer unique.\n>> >\n>> > Also, if the Makefile sends all .o's to the root directory or a\n>> > unified OBJS directory rather than to the subdir containing the .c,\n>> > then we have another issue during linking...\n>> >\n>> > So, having been burned too many times, I prefer to make source\n>> > filenames unique when possible.\n>>\n>> A much shorter name like compat/fsmonitor/fsmon-win32.c would achieve\n>> that goal.\n>>\n>> >> I've noticed you tend to prefer really long file and function names,\n>> >> e.g. your borrowed daemonize() became\n>> >> spawn_background_fsmonitor_daemon(), I think aiming for shorter\n>> >> filenames & function names helps, e.g. these long names widen diffstats,\n>> >> and many people who hack on the code stick religiously to 80 character\n>> >> width terminals.\n>> >>\n>> >\n>> > I prefer self-documenting code.\n>>\n>> Sure, I'm not saying daemonize() is an ideal name, just suggesting that\n>> you can both get uniqueness & self-documentation and not need to split\n>> to multiple lines in some common cases to stay within the \"We try to\n>> keep to at most 80 characters per line\" in CodingGuidelines in this\n>> series.\n>\n> While you are entitled to have your taste, I have to point out that Jeff\n> is just as entitled to their taste, and I don't think that you can claim\n> that yours is better.\n>\n> So I wonder what the intended outcome of this review is? To make the patch\n> better? Or to pit taste against taste?\n\nNeither, to address a misunderstanding.\n\nSure, if a reviewer points out \"maybe change X to Y\" and the reply is \"I\nlike X better than Y\", fair enough.\n\nMy reading of Jeff H.'s upthread was that he'd misunderstood my\nsuggesting of that Y for a Z.\n\nI.e. that shortening a name like fsmonitor-fs-listen-win32.c (X)\nnecessarily had to mean that we'd have a win32.c (Z), negatively\nimpacting some debugging workflows, as opposed to just a\nshorter-but-unique name like fsmon-win32.c (Y).\n\nDitto for daemonize() (X/Z) and spawn_background_fsmonitor_daemon() (X).\n\nI'm certain that with this reply we're thoroughly into the \"respectfully\ndisagree\" territory as opposed to having a misunderstanding.\n\nI also take and agree your implied point that there's no point in having\na yes/no/yes/no/yes argument on-list, and I did not mean to engage in\nsuch a thing, only to clear up the misunderstanding, if any.\n\nI'll only say that I don't think that something like long variable/file\netc. names is *just* a matter of taste, seeing as we have a fairly\nstrict \"keep to at most 80 characters per line\" as the 2nd item in the C\ncoding style (after \"use tabs, not spaces\").\n\nThat matter of taste for one developer objectively makes it harder to\nstay within the bounds of the coding style for furute maintenance.\n\nWe do have active contributors that I understand actually use terminals\nof that size to work on this project (CC'd, but maybe I misrecall that\nfor one/both). I'm not one of those people, but I do find that\nmaintaining code with needlessly long identifiers in this codebase is\npainful.\n\nE.g. in a patch I just submitted I've been working on similarly long\nidentifiers in the refs code[1], and with say a long variable/type name\nand using a long-named function you get to the point of needing to place\neach individual argument of the function on its own line, or near enough\nto that.\n\n1. https://lore.kernel.org/git/patch-7.7-cb32b5c0526-20210716T142032Z-avarab@gmail.com/\n\n"},{"id":"430356","messageId":"871r7yxkq8.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107161748050.59@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-16T16:52:07Z","receivedAt":"2021-07-16T16:54:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 16 2021, Johannes Schindelin wrote:\n\n> Hi Ævar,\n>\n> On Fri, 2 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>>\n>> > From: Jeff Hostetler \n>> >\n>> > Stub in empty implementation of fsmonitor--daemon\n>> > backend for MacOS.\n>> >\n>> > Signed-off-by: Jeff Hostetler \n>> > ---\n>> > compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++\n>> > config.mak.uname | 2 ++\n>> > contrib/buildsystems/CMakeLists.txt | 3 +++\n>> > 3 files changed, 25 insertions(+)\n>> > create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c\n>> >\n>> > diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n>> > new file mode 100644\n>> > index 00000000000..b91058d1c4f\n>> > --- /dev/null\n>> > +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c\n>> > @@ -0,0 +1,20 @@\n>> > +#include \"cache.h\"\n>> > +#include \"fsmonitor.h\"\n>> > +#include \"fsmonitor-fs-listen.h\"\n>> > +\n>> > +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state)\n>> > +{\n>> > +\treturn -1;\n>> > +}\n>> > +\n>> > +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state)\n>> > +{\n>> > +}\n>> > +\n>> > +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state)\n>> > +{\n>> > +}\n>> > +\n>> > +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state)\n>> > +{\n>> > +}\n>> > diff --git a/config.mak.uname b/config.mak.uname\n>> > index fcd88b60b14..394355463e1 100644\n>> > --- a/config.mak.uname\n>> > +++ b/config.mak.uname\n>> > @@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin)\n>> > \t\t\tMSGFMT = /usr/local/opt/gettext/bin/msgfmt\n>> > \t\tendif\n>> > \tendif\n>> > +\tFSMONITOR_DAEMON_BACKEND = macos\n>>\n>> A rather trivial point, but can't we pick one of \"macos\" or \"darwin\"\n>> (I'd think going with the existing uname is better) and name the file\n>> after the uname (or lower-case thereof)?\n>>\n>> Makes these make rules more consistent too, we could just set this to\n>> \"YesPlease\" here, and then lower case the uname for the file\n>> compilation/include.\n>\n> So you suggest that we name the new stuff after an `uname` that reflects a\n> name that is no longer relevant? I haven't seen a real Darwin system in\n> quite a long time, have you?\n\nIt's not current? On an Mac Mini M1 which got released this year:\n\n % uname -s\n Darwin\n\nWe then have the same in config.mak.uname, it seemed the most obvious\nand consistent to carry that through to file inclusion.\n"},{"id":"430357","messageId":"60f1ba1b2deb2_ade92085e@natae.notmuch","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107161754180.59@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-07-16T16:55:55Z","receivedAt":"2021-07-16T16:55:58Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Johannes Schindelin wrote:\n> On Tue, 13 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n> > On Tue, Jul 13 2021, Jeff Hostetler wrote:\n\n> > > I prefer self-documenting code.\n> >\n> > Sure, I'm not saying daemonize() is an ideal name, just suggesting that\n> > you can both get uniqueness & self-documentation and not need to split\n> > to multiple lines in some common cases to stay within the \"We try to\n> > keep to at most 80 characters per line\" in CodingGuidelines in this\n> > series.\n> \n> While you are entitled to have your taste, I have to point out that Jeff\n> is just as entitled to their taste, and I don't think that you can claim\n> that yours is better.\n> \n> So I wonder what the intended outcome of this review is? To make the patch\n> better? Or to pit taste against taste?\n\nUnless you read minds you can't possibly know what the taste of other\npeople will be.\n\nSo you put forward what *you* think is better, and then find out if\nothers agree with your taste or not. If it turns out you are the only\none that thinks it's better, so be it.\n\nFor what it's worth, I agree with Ævar that daemonize() is better and I\nfind your statement \"I prefer self-documenting code\" a) not an argument,\nb) not a valid argument if we fill in the dots, and c) passively\naggressive.\n\nEach one of us can only do one thing: express our opinion. What else can\nwe do?\n\nReviewers should not be chastised for expressing their opinion.\n\n-- \nFelipe Contreras"},{"id":"430358","messageId":"87y2a6w61l.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107161747010.59@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-16T16:55:12Z","receivedAt":"2021-07-16T16:57:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 16 2021, Johannes Schindelin wrote:\n\n> Hi Ævar,\n>\n> On Fri, 2 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n>>\n>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>>\n>> > From: Jeff Hostetler \n>> >\n>> > Stub in empty backend for fsmonitor--daemon on Windows.\n>> >\n>> > Signed-off-by: Jeff Hostetler \n>> > ---\n>> > Makefile | 13 ++++++\n>> > compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++\n>> > compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++\n>> > config.mak.uname | 2 +\n>> > contrib/buildsystems/CMakeLists.txt | 5 ++\n>> > 5 files changed, 90 insertions(+)\n>> > create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n>> > create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n>> >\n>> > diff --git a/Makefile b/Makefile\n>> > index c45caacf2c3..a2a6e1f20f6 100644\n>> > --- a/Makefile\n>> > +++ b/Makefile\n>> > @@ -467,6 +467,11 @@ all::\n>> > # directory, and the JSON compilation database 'compile_commands.json' will be\n>> > # created at the root of the repository.\n>> > #\n>> > +# If your platform supports a built-in fsmonitor backend, set\n>> > +# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n>> > +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n>> > +# `fsmonitor_fs_listen__*()` routines.\n>> > +#\n>> > # Define DEVELOPER to enable more compiler warnings. Compiler version\n>> > # and family are auto detected, but could be overridden by defining\n>> > # COMPILER_FEATURES (see config.mak.dev). You can still set\n>> > @@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n>> > \tCOMPAT_OBJS += compat/access.o\n>> > endif\n>> >\n>> > +ifdef FSMONITOR_DAEMON_BACKEND\n>> > +\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n>> > +\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n>> > +endif\n>> > +\n>> > ifeq ($(TCLTK_PATH),)\n>> > NO_TCLTK = NoThanks\n>> > endif\n>> > @@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE\n>> > \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n>> > \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n>> > \t@echo X=\\'$(X)\\' >>$@+\n>> > +ifdef FSMONITOR_DAEMON_BACKEND\n>> > +\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n>> > +endif\n>>\n>> Why put this in an ifdef?\n>\n> Why not? What benefit does this question bring to improving this patch\n> series?\n\nI think that when adding code to the Makefile it makes sense to follow\nthe prevailing pattern, unless there's a good reason to do otherwise,\ne.g. on my build:\n\t\n\t$ grep \"''\" GIT-BUILD-OPTIONS \n\tNO_CURL=''\n\tNO_EXPAT=''\n\tNO_PERL=''\n\tNO_PTHREADS=''\n\tNO_PYTHON=''\n\tNO_UNIX_SOCKETS=''\n\tX=''\n\nWhy does the FSMONITOR_DAEMON_BACKEND option require a nonexistent line\nas opposed to an empty one?\n"},{"id":"430359","messageId":"60f1bb0d67e97_ade9208a8@natae.notmuch","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107161747010.59@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-07-16T16:59:57Z","receivedAt":"2021-07-16T17:00:06Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Johannes Schindelin wrote:\n> On Fri, 2 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n> > On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n\n> > > @@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE\n> > > \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n> > > \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n> > > \t@echo X=\\'$(X)\\' >>$@+\n> > > +ifdef FSMONITOR_DAEMON_BACKEND\n> > > +\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n> > > +endif\n> >\n> > Why put this in an ifdef?\n> \n> Why not? What benefit does this question bring to improving this patch\n> series?\n\nThis is a common debate tactic known as \"shifting the burden of proof\".\n\nÆvar does not need to prove that your patch is undesirable, *you* have\nto prove that it is desirable.\n\nYou have the burden of proof, so you should answer the question.\n\nhttps://www.logicallyfallacious.com/logicalfallacies/Shifting-of-the-Burden-of-Proof\n\n-- \nFelipe Contreras"},{"id":"430360","messageId":"60f1bc162a797_ade92086e@natae.notmuch","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107161732150.59@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 02/34] fsmonitor--daemon: man page","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-07-16T17:04:22Z","receivedAt":"2021-07-16T17:04:31Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Johannes Schindelin wrote:\n> On Tue, 13 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n> \n> > [snip Ævar's suggestion to populate the manual page incrementally,\n> > interspersed with the commits that finalize implementing the respective\n> > functionality]\n> >\n> > I suggested this because I think it's much easier to read patches that\n> > are larger because they incrementally update docs or tests along with\n> > code, than smaller ones that are e.g. \"add docs\" followed by\n> > incrementally modifying the code.\n> \n> My experience is the exact opposite of yours: shorter patches are easier\n> to read.\n\nIt depends.\n\n> > That's because you can consider those atomically.\n> \n> No, in a patch series you cannot consider any patch completely atomically.\n> Just like you don't consider any paragraph in any well-written book out of\n> context.\n\nBut you do not put every sentence in a paragraph.\n\nSometimes a paragraph can contain a single sentence, or a single word\neven. But other times to properly read what is being tried to say you\nneed a pretty big paragraph.\n\n> This all sounds like we're truly falling into the trap of ignoring the\n> rule that the perfect is the enemy of the good.\n\nHave you established that this is good enough?\n\n-- \nFelipe Contreras"},{"id":"430412","messageId":"xmqqa6mlplo7.fsf@gitster.g","threadId":"55420","inReplyTo":"87y2a6w61l.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-17T05:13:44Z","receivedAt":"2021-07-17T05:13:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n>>> > +ifdef FSMONITOR_DAEMON_BACKEND\n>>> > +\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n>>> > +\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n>>> > +endif\n>>> > +\n>>> > ifeq ($(TCLTK_PATH),)\n>>> > NO_TCLTK = NoThanks\n>>> > endif\n>>> ...\n>>>\n>>> Why put this in an ifdef?\n>>\n>> Why not? What benefit does this question bring to improving this patch\n>> series?\n>\n> I think that when adding code to the Makefile it makes sense to follow\n> the prevailing pattern, unless there's a good reason to do otherwise,\n> e.g. on my build:\n> \t\n> \t$ grep \"''\" GIT-BUILD-OPTIONS \n> \tNO_CURL=''\n> \tNO_EXPAT=''\n> \tNO_PERL=''\n> \tNO_PTHREADS=''\n> \tNO_PYTHON=''\n> \tNO_UNIX_SOCKETS=''\n> \tX=''\n>\n> Why does the FSMONITOR_DAEMON_BACKEND option require a nonexistent line\n> as opposed to an empty one?\n\nI do not quite get the question.\n\n#!/bin/sh\ncat >make.file <<\\EOF\nall::\nifeq ($(FSMONITOR_DAEMON_BACKEND),)\n\techo it is empty\nendif\nifdef FSMONITOR_DAEMON_BACKEND\n\techo it is undefined\nendif\nEOF\n\necho \"unset???\"\nmake -f make.file\n\necho \"set to empty???\"\nmake -f make.file FSMONITOR_DAEMON_BACKEND=\n\nThese two make invocations will give us the same result, showing\nthat \"is it set to empty\" and \"is it unset\" are the same.\n\n"},{"id":"430413","messageId":"xmqq5yx9plax.fsf@gitster.g","threadId":"55420","inReplyTo":"xmqqa6mlplo7.fsf@gitster.g","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-17T05:21:42Z","receivedAt":"2021-07-17T05:21:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> #!/bin/sh\n> cat >make.file <<\\EOF\n> all::\n> ifeq ($(FSMONITOR_DAEMON_BACKEND),)\n> \techo it is empty\n> endif\n> ifdef FSMONITOR_DAEMON_BACKEND\n\nAn obvious typo. This must be \"ifndef\", of course.\n\n> \techo it is undefined\n> endif\n> EOF\n>\n> echo \"unset???\"\n> make -f make.file\n>\n> echo \"set to empty???\"\n> make -f make.file FSMONITOR_DAEMON_BACKEND=\n>\n> These two make invocations will give us the same result, showing\n> that \"is it set to empty\" and \"is it unset\" are the same.\n"},{"id":"430417","messageId":"20210717124535.GB16801@dcvr","threadId":"55420","inReplyTo":"874kcuxkz8.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-07-17T12:45:35Z","receivedAt":"2021-07-17T12:45:38Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Ævar Arnfjörð Bjarmason wrote:\n> On Fri, Jul 16 2021, Johannes Schindelin wrote:\n> > Hi Ævar,\n> >\n> > On Tue, 13 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n> >\n> >>\n> >> On Tue, Jul 13 2021, Jeff Hostetler wrote:\n> >>\n> >> > On 7/1/21 7:02 PM, Ævar Arnfjörð Bjarmason wrote:\n> >> >> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> >> >>\n> >> >>> From: Jeff Hostetler \n> >> >>>\n> >> >>> Teach the win32 backend to register a watch on the working tree\n> >> >>> root directory (recursively). Also watch the if it is\n> >> >>> not inside the working tree. And to collect path change notifications\n> >> >>> into batches and publish.\n> >> >>>\n> >> >>> Signed-off-by: Jeff Hostetler \n> >> >>> ---\n> >> >>> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++\n> >> >> Spying on the early history of this (looking for the\n> >> >> Linux backend) I saw that at some point we had just\n> >> >> compat/fsmonitor/linux.c, and presumably some of\n> >> >> compat/fsmonitor/{windows,win32,macos,darwin}.c.\n> >> >> At some point those filenames became much much longer.\n> >> >>\n> >> >\n> >> > Once upon a time having \"foo/bar/win32.c\" and \"abc/def/win32.c\"\n> >> > would cause confusion in the debugger (I've long since forgotten\n> >> > which). Breaking at win32.c:30 was no longer unique.\n> >> >\n> >> > Also, if the Makefile sends all .o's to the root directory or a\n> >> > unified OBJS directory rather than to the subdir containing the .c,\n> >> > then we have another issue during linking...\n> >> >\n> >> > So, having been burned too many times, I prefer to make source\n> >> > filenames unique when possible.\n> >>\n> >> A much shorter name like compat/fsmonitor/fsmon-win32.c would achieve\n> >> that goal.\n> >>\n> >> >> I've noticed you tend to prefer really long file and function names,\n> >> >> e.g. your borrowed daemonize() became\n> >> >> spawn_background_fsmonitor_daemon(), I think aiming for shorter\n> >> >> filenames & function names helps, e.g. these long names widen diffstats,\n> >> >> and many people who hack on the code stick religiously to 80 character\n> >> >> width terminals.\n\nAt least \"daemon\"/\"daemonize\" already implies \"background\"; so\neven if we have the extra function, \"spawn_fsmon_daemon()\" would\nbe enough info.\n\n> >> >>\n> >> >\n> >> > I prefer self-documenting code.\n> >>\n> >> Sure, I'm not saying daemonize() is an ideal name, just suggesting that\n> >> you can both get uniqueness & self-documentation and not need to split\n> >> to multiple lines in some common cases to stay within the \"We try to\n> >> keep to at most 80 characters per line\" in CodingGuidelines in this\n> >> series.\n> >\n> > While you are entitled to have your taste, I have to point out that Jeff\n> > is just as entitled to their taste, and I don't think that you can claim\n> > that yours is better.\n> >\n> > So I wonder what the intended outcome of this review is? To make the patch\n> > better? Or to pit taste against taste?\n> \n> Neither, to address a misunderstanding.\n> \n> Sure, if a reviewer points out \"maybe change X to Y\" and the reply is \"I\n> like X better than Y\", fair enough.\n> \n> My reading of Jeff H.'s upthread was that he'd misunderstood my\n> suggesting of that Y for a Z.\n> \n> I.e. that shortening a name like fsmonitor-fs-listen-win32.c (X)\n> necessarily had to mean that we'd have a win32.c (Z), negatively\n> impacting some debugging workflows, as opposed to just a\n> shorter-but-unique name like fsmon-win32.c (Y).\n\nShort-as-possible-while-being-meaningful is a pretty important\nusability thing git. There's a good reason git supports OID\nprefix abbreviations, after all.\n\nNot my area of expertise, but AFAIK git's rename detection is\naffected by basename; and I've encountered debugger confusion\nwith non-unique basenames while debugging other codebases.\n\nMy brain works like a naive \"strcmp\"/\"memcmp\": long common\nprefixes slows down my ability to differentiate filenames.\n\nHaving lots of common terms/prefixes on the screen works like\ncamoflage to me and slows down my ability to process things.\nI suppose my eyes and cognitive abilities are below average;\nand even worse due to the pandemic numbing my brain.\n\n> Ditto for daemonize() (X/Z) and spawn_background_fsmonitor_daemon() (X).\n\n(what I said above)\n\n> I'm certain that with this reply we're thoroughly into the \"respectfully\n> disagree\" territory as opposed to having a misunderstanding.\n> \n> I also take and agree your implied point that there's no point in having\n> a yes/no/yes/no/yes argument on-list, and I did not mean to engage in\n> such a thing, only to clear up the misunderstanding, if any.\n> \n> I'll only say that I don't think that something like long variable/file\n> etc. names is *just* a matter of taste, seeing as we have a fairly\n> strict \"keep to at most 80 characters per line\" as the 2nd item in the C\n> coding style (after \"use tabs, not spaces\").\n> \n> That matter of taste for one developer objectively makes it harder to\n> stay within the bounds of the coding style for furute maintenance.\n> \n> We do have active contributors that I understand actually use terminals\n> of that size to work on this project (CC'd, but maybe I misrecall that\n> for one/both). I'm not one of those people, but I do find that\n> maintaining code with needlessly long identifiers in this codebase is\n> painful.\n\nThanks for Cc-ing me.\n\nYes, I'm one of those developers. Accessibility matters to me:\nmy eyesight certainly isn't getting better with age (nor do I\nexpect anyone elses'). I need giant fonts to reduce eye and\nneck strain.\n\nFwiw, newspaper publishers figured out line width\ndecades/centuries ago and wrap lines despite having large sheets\nto work on.\n\n\nI mostly work over mosh or ssh to reduce noise and heat locally.\nThere's no bandwidth for VNC or similar, and graphical stuff\ntends to be unstable UI-wise anyways so I stick to the terminal.\n\nTaste does have much to do with it: I favor stable, reliable\ntools (e.g. POSIX, Perl5, git) that works well on both old and\nnew hardware. I avoid mainstream \"desktop\" software since they\ntend to have unstable UIs which break users' workflows while\nrequiring more powerful HW.\n\nComplex graphics drivers tend to get unreliable, too, especially\nwhen one is stuck with old HW that gets limited support from\nvendors. It's also difficult to fix complex drivers as a\nhobbyist given the one-off HW/vendor-specific knowledge\nrequired.\n\nSo we shouldn't expect a developer with old HW can have more\nthan a standard text terminal. This is an accessibility problem\nfor developers lacking in finances.\n\nThis is also a problem for developers wishing to backdoors+bugs\nfound in modern systems (IntelME, AMD-PSP, endless stream of CPU\nbugs).\n\n\nBack to health-related accessibility; I've also had joint\nproblems for many years so shorter identifiers helps reduce\ntyping I need to do. I mostly had that under control\npre-pandemic, but it's been a huge struggle to find adequate\nreplacements for activities I used to rely on to manage the\npain.\n"},{"id":"430449","messageId":"87k0lowr6u.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"xmqqa6mlplo7.fsf@gitster.g","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-17T21:43:00Z","receivedAt":"2021-07-17T21:44:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jul 16 2021, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>>>> > +ifdef FSMONITOR_DAEMON_BACKEND\n>>>> > +\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n>>>> > +\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n>>>> > +endif\n>>>> > +\n>>>> > ifeq ($(TCLTK_PATH),)\n>>>> > NO_TCLTK = NoThanks\n>>>> > endif\n>>>> ...\n>>>>\n>>>> Why put this in an ifdef?\n>>>\n>>> Why not? What benefit does this question bring to improving this patch\n>>> series?\n>>\n>> I think that when adding code to the Makefile it makes sense to follow\n>> the prevailing pattern, unless there's a good reason to do otherwise,\n>> e.g. on my build:\n>> \t\n>> \t$ grep \"''\" GIT-BUILD-OPTIONS \n>> \tNO_CURL=''\n>> \tNO_EXPAT=''\n>> \tNO_PERL=''\n>> \tNO_PTHREADS=''\n>> \tNO_PYTHON=''\n>> \tNO_UNIX_SOCKETS=''\n>> \tX=''\n>>\n>> Why does the FSMONITOR_DAEMON_BACKEND option require a nonexistent line\n>> as opposed to an empty one?\n>\n> I do not quite get the question.\n>\n> #!/bin/sh\n> cat >make.file <<\\EOF\n> all::\n> ifeq ($(FSMONITOR_DAEMON_BACKEND),)\n> \techo it is empty\n> endif\n> ifdef FSMONITOR_DAEMON_BACKEND\n> \techo it is undefined\n> endif\n> EOF\n>\n> echo \"unset???\"\n> make -f make.file\n>\n> echo \"set to empty???\"\n> make -f make.file FSMONITOR_DAEMON_BACKEND=\n>\n> These two make invocations will give us the same result, showing\n> that \"is it set to empty\" and \"is it unset\" are the same.\n\nIndeed, which is why I'm pointing out that wrapping it in an ifdef is\npointless, which is why we don't do it for the other ones.\n\nWe do have a bunch of ifdef'd things there for perf etc., I'm not sure\nif it matters or not for those.\n"},{"id":"430512","messageId":"c62cf635-1d24-3b63-9566-b98aa2ec2af5@jeffhostetler.com","threadId":"55420","inReplyTo":"87v95tbqgh.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-19T16:54:48Z","receivedAt":"2021-07-19T17:00:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 6:45 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Stub in empty backend for fsmonitor--daemon on Windows.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> Makefile | 13 ++++++\n>> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++\n>> compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++\n>> config.mak.uname | 2 +\n>> contrib/buildsystems/CMakeLists.txt | 5 ++\n>> 5 files changed, 90 insertions(+)\n>> create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n>> create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n>>\n>> diff --git a/Makefile b/Makefile\n>> index c45caacf2c3..a2a6e1f20f6 100644\n>> --- a/Makefile\n>> +++ b/Makefile\n>> @@ -467,6 +467,11 @@ all::\n>> # directory, and the JSON compilation database 'compile_commands.json' will be\n>> # created at the root of the repository.\n>> #\n>> +# If your platform supports a built-in fsmonitor backend, set\n>> +# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n>> +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n>> +# `fsmonitor_fs_listen__*()` routines.\n>> +#\n>> # Define DEVELOPER to enable more compiler warnings. Compiler version\n>> # and family are auto detected, but could be overridden by defining\n>> # COMPILER_FEATURES (see config.mak.dev). You can still set\n>> @@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n>> \tCOMPAT_OBJS += compat/access.o\n>> endif\n>> \n>> +ifdef FSMONITOR_DAEMON_BACKEND\n>> +\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n>> +\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n>> +endif\n>> +\n>> ifeq ($(TCLTK_PATH),)\n>> NO_TCLTK = NoThanks\n>> endif\n>> @@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE\n>> \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n>> \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n>> \t@echo X=\\'$(X)\\' >>$@+\n>> +ifdef FSMONITOR_DAEMON_BACKEND\n>> +\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n>> +endif\n> \n> Why put this in an ifdef?\n> \n> In 342e9ef2d9e (Introduce a performance testing framework, 2012-02-17)\n> we started doing that for some perf/test options (which b.t.w., I don't\n> really see the reason for, maybe it's some subtlety in how test-lib.sh\n> picks those up).\n> \n> But for all the other compile-time stuff we don't ifdef it, we just\n> define it, and then you get an empty value or not.\n> \n> This would AFAICT be the first build-time-for-the-C-program option we\n> ifdef for writing a line to GIT-BUILD-OPTIONS.\n> \n\n(I'm going to respond here on the original question rather than on any\nof the follow up responses in an attempt at diffusing things a bit.)\n\nI added the ifdef because I thought it to be the *most conservative*\nthing that I could do. The output of the generated file on unsupported\nplatforms should be *identical* to what it was before my changes. I\nonly alter the contents of the generated file on supported platforms.\n\nLater, when the generated file is consumed, we don't need to worry about\nthe effect (if any) on incremental compiles -- we will know that it\nwon't be set -- just like it was not set in the original compile.\n\nThat change appears right before a 12 other ifdef'd symbols also being\nwritten to that generated file. Most are test and perf, but some are\nnot. But my point is that the pattern is present already.\n\nThe original question also references a 9.5 year old commit which\nuses the same pattern as I've used here. It also muddies the water\non why it was/wasn't needed back then. And hints at possible\nside-effects in some of our test scripts. So it is clear that the\nconfusion/disagreements that we are having with the current patch\nand whether or not to ifdef are not new.\n\n\nSo, is there value in being explicit and having the ifdef ??\n\n\nThere are well defined Make rules (and Junio gave us a very elegant\nlittle script to demonstrate that), but the subtleties are there.\nEspecially with our use generated files like `GIT-BUILD-OPTIONS`.\nWe have a mailing list full of experts and yet this question received\na lot more discussion than I thought possible or necessary, but it\ntook a test script to demonstrate that the results are the same and it\ndoesn't matter. Perhaps the clarity is worth it for the price of a\nsimple ifdef.\n\n\nSo, how much time have we (collectively) wasted discussing this\nsubtlety ??\n\n\nTo summarize, I added the ifdef to make it explicitly clear that\nI'm not altering behavior on unsupported platforms. I can remove it\nfrom V4 if desired or I can keep it. (We all now know that it doesn't\nfunctionally matter -- it does however, provide clarity.)\n\n\nSorry if this sounded like a rant,\nJeff\n"},{"id":"430545","messageId":"xmqqczrekrd0.fsf@gitster.g","threadId":"55420","inReplyTo":"87k0lowr6u.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-19T19:58:51Z","receivedAt":"2021-07-19T23:11:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n>>>>> Why put this in an ifdef?\n>>>> ...\n>>> Why does the FSMONITOR_DAEMON_BACKEND option require a nonexistent line\n>>> as opposed to an empty one?\n>>\n>> I do not quite get the question.\n>>\n>> #!/bin/sh\n>> cat >make.file <<\\EOF\n>> all::\n>> ifeq ($(FSMONITOR_DAEMON_BACKEND),)\n>> \techo it is empty\n>> endif\n>> ifndef FSMONITOR_DAEMON_BACKEND\n>> \techo it is undefined\n>> endif\n>> EOF\n>>\n>> echo \"unset???\"\n>> make -f make.file\n>>\n>> echo \"set to empty???\"\n>> make -f make.file FSMONITOR_DAEMON_BACKEND=\n>>\n>> These two make invocations will give us the same result, showing\n>> that \"is it set to empty\" and \"is it unset\" are the same.\n>\n> Indeed, which is why I'm pointing out that wrapping it in an ifdef is\n> pointless, which is why we don't do it for the other ones.\n>\n> We do have a bunch of ifdef'd things there for perf etc., I'm not sure\n> if it matters or not for those.\n\nSorry, but I still do not get the question. There are bunch of\nifndef in Makefile in addition to ifeq/ifneq and your question\n\n FSMONITOR_DAEMON_BACKEND option require a nonexistent line as\n opposed to an empty one?\n\nis asking \"why is it X\" when X is not quite true. I presume that\nyour \"wrapping it in an ifdef\" refers to a construct like this:\n\n>>> > +ifdef FSMONITOR_DAEMON_BACKEND\n>>> > +\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n>>> > +\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n>>> > +endif\n\nbut is your suggestion that it should be written like this instead?\n\n>>> > +ifneq ($(FSMONITOR_DAEMON_BACKEND),)\n>>> > +\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n>>> > +\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n>>> > +endif\n\nI do not think the latter is any easier to follow (and we have many\nifdef and ifndef in our Makefile already). Perhaps I will see what\nyou mean when I see your \"better alternative\", but so far, I am not\nsuccessfully guessing what it is.\n\n"},{"id":"430547","messageId":"40ab8cbc-a0fd-92a8-dd97-883c8752127d@jeffhostetler.com","threadId":"55420","inReplyTo":"87im1uc57c.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 06/34] fsmonitor: config settings are repository-specific","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-19T20:36:54Z","receivedAt":"2021-07-19T23:11:21Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 12:46 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n> In a reference to a discussion[1] about an earlier version of this patch\n> you said:\n> \n> I'm going to ignore all of the thread responses to this patch\n> dealing with how we acquire config settings and macros and etc.\n> Those issues are completely independent of FSMonitor (which is\n> already way too big).\n> \n> Since then the changes to repo-settings.c have become a lot larger, so\n> let's take a look...\n> \n> 1. https://lore.kernel.org/git/87mttkyrqq.fsf@evledraar.gmail.com/\n> 2. https://lore.kernel.org/git/4552971c-0a23-c19a-6a23-cb5737e43b2a@jeffhostetler.com/\n\nYes, there was a large conversation about re-doing how config\nvalues are acquired or whatever and it was nested inside of\nthe context of a completely unrelated topic. It still is.\n\nI would like to focus on FSMonitor using the existing config API.\nI'm only looking up ~2 config values. And that code is fairly\nminor considering everything else in this patch series.\n\nYes, there's a bizarre initialization with memset(-1), but it\ncan wait.\n\nLater (in a clean context) we can address and focus on the config\nAPI and/or structure initialization that you're talking about here.\n\nFWIW, in V4 I'll refactor the block of code that I added into the\nbody of prepare_repo_settings() to address your later concerns.\nThis version will hopefully be more readable.\n\n> \n> \n>> diff --git a/repo-settings.c b/repo-settings.c\n>> index 0cfe8b787db..faf197ff60a 100644\n>> --- a/repo-settings.c\n>> +++ b/repo-settings.c\n>> @@ -5,10 +5,42 @@\n>> \n>> #define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0)\n>> \n>> +/*\n>> + * Return 1 if the repo/workdir is incompatible with FSMonitor.\n>> + */\n>> +static int is_repo_incompatible_with_fsmonitor(struct repository *r)\n>> +{\n>> +\tconst char *const_strval;\n>> +\n>> +\t/*\n>> +\t * Bare repositories don't have a working directory and\n>> +\t * therefore, nothing to watch.\n>> +\t */\n>> +\tif (!r->worktree)\n>> +\t\treturn 1;\n> \n> Looking ahead in this series you end up using\n> FSMONITOR_MODE_INCOMPATIBLE in two places in the codebase. In\n> builtin/update-index.c to throw a \"repository is incompatible with\n> fsmonitor\" error.\n> \n> Can't that case just be replaced with setup_work_tree()? Other sub-modes\n> of update-index already die implicitly on that, e.g.:\n> \n> \t$ git update-index test\n> \tfatal: this operation must be run in a work tree\n\nI will refactor that static function in V4, but I want it to return\nan indication of whether the repo is compatible only and let the\ncommand print the error/die as is appropriate for the daemon and/or\nupdate-index. The daemon should not start on an incompatible repo.\nLikewise, update-index should not enable the extension in the index.\n\nWe share some of that code with the client side code, like status,\nwhich wants to talk to the hook or daemon if supported/present/allowed.\nIf the repo is incompatible, then status should just behave in the\nclassic manner. So I don't want the detection code to print those\nerror messages or die.\n\n> \n> The other case is:\n> \t\n> \t+ prepare_repo_settings(the_repository);\n> \t+ if (!the_repository->worktree)\n> \t+ return error(_(\"fsmonitor-daemon does not support bare repos '%s'\"),\n> \t+ xgetcwd());\n> \t+ if (the_repository->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE)\n> \t+ return error(_(\"fsmonitor-daemon is incompatible with this repo '%s'\"),\n> \t+ the_repository->worktree);\n> \n> I.e. we just checked the_repository->worktree, but it's not that, but....\n\nyes, I currently have 2 types of repos that I want to disable\nboth the hook and daemon version of FSMonitor. I'll update the\nerror messages to specify the reason why we are incompatible.\n\n> \n>> +\n>> +\t/*\n>> +\t * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n>> +\t *\n>> +\t * Granted, core Git does not know anything about GVFS and\n>> +\t * we shouldn't make assumptions about a downstream feature,\n>> +\t * but users can install both versions. And this can lead\n>> +\t * to incorrect results from core Git commands. So, without\n>> +\t * bringing in any of the GVFS code, do a simple config test\n>> +\t * for a published config setting. (We do not look at the\n>> +\t * various *_TEST_* environment variables.)\n>> +\t */\n>> +\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_strval))\n>> +\t\treturn 1;\n> \n> I'm skeptical of us hardcoding a third-party software config\n> variable. Can't GitVFS handle this somehow on its end?\n\nAdding the test for a GVFS-specific config setting is questionable.\nAnd perhaps we should move it to our downstream fork.\n\nThe value in putting it here for now (at least) is that it makes\nclear the structure for supporting other types of incompatible\nrepos.\n\nFor example, perhaps we want to disallow repos that are on remote\nfile systems (since we might not be able to get FS events). It\nwould be nice to be able to prevent the daemon from starting and/or\nfrom status from trying to connect to a daemon that will be a\ndisappointment. The code I have here serves as a model for adding\nsuch additional restrictions. And keeps us from trying to prematurely\ncollapse the code into a simple expression.\n\n> \n> But just in terms of implementation it seems the end result of that is\n> to emit a very confusing error to the user. Sinc we already checked for\n> bare repos we run into this and instead of sayingwhen we should really\n> say \"hey, maybe disable your core.virtualFileSystem setting\", we say\n> \"your repo is incompatible\".\n> \n\nI'll update the error messages to make that clear.\n\n>> +\n>> +\treturn 0;\n>> +}\n>> +\n>> void prepare_repo_settings(struct repository *r)\n>> {\n>> \tint value;\n>> \tchar *strval;\n>> +\tconst char *const_strval;\n> \n> Can be declared in the \"else\" below.\n> \n>> \n>> \tif (r->settings.initialized)\n>> \t\treturn;\n>> @@ -26,6 +58,22 @@ void prepare_repo_settings(struct repository *r)\n>> \tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n>> \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n>> \n>> +\tr->settings.fsmonitor_hook_path = NULL;\n>> +\tr->settings.fsmonitor_mode = FSMONITOR_MODE_DISABLED;\n> \n> With the memset earlier (b.t.w. I've got a patch to fix all this bizarre\n> behavior in repo-settings.c, but have been waiting on this series we\n> implicitly set it to FSMONITOR_MODE_UNSET (-1) with the memset, but then\n> never use that ever.\n\nI'm working around the bogus -1 value that the structure has\nbeen initialize with and that I'm inheriting. I'll completely\nset my `fsmonitor_mode` variable so that I don't care what it\nis initialized to.\n\nI created the _UNSET value as a reminder that there is this memset(-1)\nthere and that must be attended to.\n\nIn my V4 I'll add comments to that effect.\n\n> \n> Your code in update-index.c then for a check against\n> \"FSMONITOR_MODE_DISABLED\" says \"core.useBuiltinFSMonitor is unset;\".\n> \n>> +\tif (is_repo_incompatible_with_fsmonitor(r))\n>> +\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_INCOMPATIBLE;\n> \n> Style: should have {} braces on all arms.\n> \n>> +\telse if (!repo_config_get_bool(r, \"core.usebuiltinfsmonitor\", &value)\n>> +\t\t && value)\n>> +\t\tr->settings.fsmonitor_mode = FSMONITOR_MODE_IPC;\n> \n> Here you're conflating false with whether the variable is set at all. I\n> guess that works out here since if it's false we want to fall through\n> to...\n> \n>> +\telse {\n> \n> ...ignoring it and looing at core.fsmonitor instead.\n\nyes, if core.useBuiltinFSMonitor is true, we do not need to look\nat the hook pathname.\n\n> \n>> +\t\tif (repo_config_get_pathname(r, \"core.fsmonitor\", &const_strval))\n>> +\t\t\tconst_strval = getenv(\"GIT_TEST_FSMONITOR\");\n> \n> If it's not set we pay attention to GIT_TEST_FSMONITOR, so the behavior\n> from the old git_config_get_fsmonitor(). So even if the env variable is\n> set we want to take the config variable over it, correct?\n\nGIT_TEST_FSMONITOR sets the hook path for testing and we're not using\nthe hook API at all. So it is kind of ill-defined what that test env\nvar should do if you have IPC turned on, so I'm ignoring it.\n\n> \n>> +\t\tif (const_strval && *const_strval) {\n>> +\t\t\tr->settings.fsmonitor_hook_path = strdup(const_strval);\n> \n> We had a strbuf_detach()'d string in the case of\n> repo_config_get_pathname(), but here we strdup() it again in case we\n> were in the getenv() codepath. This code probably leaks memory now\n> anyway, but perhaps it's better to split up the two so we make it easier\n> to deal with who owns/frees what in the future.\n> \n\nrepo_config_get_pathname() returns a const char** which implies that\nit is not a detached buffer. it is a deep trek thru the config code,\nbut it eventually gets to git_configset_get_*() which is documented as\nreturning a pointer into a configset cache and that the caller should\nnot free it. so dup'ing it is appropriate.\n\nSimilarly, getenv() is also returning a pointer to a buffer that the\ncaller does not own. so dup'ing it here is also appropriate.\n\nThanks,\nJeff\n"},{"id":"430553","messageId":"d2936252-50d7-7b3f-4707-7c07b17a09f3@jeffhostetler.com","threadId":"55420","inReplyTo":"20210717124535.GB16801@dcvr","subject":"Re: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-19T22:35:33Z","receivedAt":"2021-07-19T23:11:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/17/21 8:45 AM, Eric Wong wrote:\n> Ævar Arnfjörð Bjarmason wrote:\n>> On Fri, Jul 16 2021, Johannes Schindelin wrote:\n>>> Hi Ævar,\n>>>\n>>> On Tue, 13 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>>>\n>>>> On Tue, Jul 13 2021, Jeff Hostetler wrote:\n>>>>\n>>>>> On 7/1/21 7:02 PM, Ævar Arnfjörð Bjarmason wrote:\n>>>>>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>>>>>>\n>>>>>>> From: Jeff Hostetler \n>>>>>>>\n...\n\nEric, welcome to the conversation and thanks for sharing your concerns.\n\n\nFor my upcoming V4 I've shortened the filenames of the various backends,\nrenamed the -macos one to -darwin, and shortened the names of the\nfsm-listener API, and the names of those static functions associated\nwith starting the daemon in the background.\n\nI think this covers all of the issues raised across several\npatches in the series.\n\nJeff\n"},{"id":"430556","messageId":"91cd5dfa-d325-7eb3-e948-c075269781a5@jeffhostetler.com","threadId":"55420","inReplyTo":"871r8hd5ew.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 08/34] fsmonitor--daemon: add a built-in fsmonitor daemon","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-19T20:56:44Z","receivedAt":"2021-07-19T23:11:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 6:36 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n> A general comment on this series (including previous patches). We've\n> usually tried to bend over backwards in git's codebase not to have big\n> ifdef blocks, so we compile most code the same everywhere. We waste a\n> bit of object code, but that's fine.\n> \n> See 9c897c5c2ad (pack-objects: remove #ifdef NO_PTHREADS, 2018-11-03)\n> for a good exmaple of bad code being turned to good.\n> \n> E.g. in this case:\n> \n>> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n>> +\n>> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n>> +{\n>> +\tconst char *subcmd;\n>> +\n>> +\tstruct option options[] = {\n>> +\t\tOPT_END()\n>> +\t};\n>> +\n>> +\tif (argc < 2)\n>> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n>> +\n>> +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n>> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n>> +\n>> +\tgit_config(git_default_config, NULL);\n>> +\n>> +\tsubcmd = argv[1];\n>> +\targv--;\n>> +\targc++;\n>> +\n>> +\targc = parse_options(argc, argv, prefix, options,\n>> +\t\t\t builtin_fsmonitor__daemon_usage, 0);\n>> +\n>> +\tdie(_(\"Unhandled subcommand '%s'\"), subcmd);\n>> +}\n>> +\n>> +#else\n>> +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n>> +{\n>> +\tstruct option options[] = {\n>> +\t\tOPT_END()\n>> +\t};\n>> +\n>> +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n>> +\t\tusage_with_options(builtin_fsmonitor__daemon_usage, options);\n>> +\n>> +\tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n>> +}\n>> +#endif\n> \n> This whole thing could really just be a\n> -DHAVE_FSMONITOR_DAEMON_BACKEND=1 or -DHAVE_FSMONITOR_DAEMON_BACKEND=0\n> somewhere (depending), and then somewhere in the middle of the first\n> function:\n> \n> \tif (!HAVE_FSMONITOR_DAEMON_BACKEND)\n> \t \tdie(_(\"fsmonitor--daemon not supported on this platform\"));\n> \n\nThis whole file will be filled up with ~1500 lines of static functions\nthat only make sense when the daemon is supported and that make calls\nto platform-specific backends.\n\nI suppose we could stub in an empty backend (something like that in\n11/34 and 12/34) and hack in all stuff in the makefile to link to it\nin the unsupported case, but that seems like a lot of effort just to\navoid an ifdef here.\n\nI mean, the intent of the #else block is quite clear and we're not\nfooling the reader with a large source file of code that will never\nbe used on their platform.\n\nWe could consider splitting this source file into a supported and\nunsupported version and have the makefile select the right .c file.\nWe'd have to move the usage and stuff to a shared header and etc.\nThat would eliminate the ifdef, but it would break the convention of\nthe source filename matching the command name.\n\nI'm not sure it's worth the bother TBH.\n\nJeff\n"},{"id":"430678","messageId":"87wnpkvkxj.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"fc37043d-5a6c-ae8a-09a9-8498c3715d43@jeffhostetler.com","subject":"Re: [PATCH v3 24/34] t/perf/p7519: speed up test using \"test-tool touch\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-20T19:18:56Z","receivedAt":"2021-07-20T19:34:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 13 2021, Jeff Hostetler wrote:\n\n> On 7/13/21 2:18 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Tue, Jul 13 2021, Jeff Hostetler wrote:\n>> \n>>> On 7/1/21 7:09 PM, Ævar Arnfjörð Bjarmason wrote:\n>>>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>>>>\n>>>>> From: Jeff Hostetler \n>>>>>\n>>>>> Change p7519 to use a single \"test-tool touch\" command to update\n>>>>> the mtime on a series of (thousands) files instead of invoking\n>>>>> thousands of commands to update a single file.\n>>>>>\n>>>>> This is primarily for Windows where process creation is so\n>>>>> very slow and reduces the test run time by minutes.\n>>>>>\n>>>>> Signed-off-by: Jeff Hostetler \n>>>>> ---\n>>>>> t/perf/p7519-fsmonitor.sh | 14 ++++++--------\n>>>>> 1 file changed, 6 insertions(+), 8 deletions(-)\n>>>>>\n>>>>> diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh\n>>>>> index 5eb5044a103..f74e6014a0a 100755\n>>>>> --- a/t/perf/p7519-fsmonitor.sh\n>>>>> +++ b/t/perf/p7519-fsmonitor.sh\n>>>>> @@ -119,10 +119,11 @@ test_expect_success \"one time repo setup\" '\n>>>>> \tfi &&\n>>>>> \tmkdir 1_file 10_files 100_files 1000_files 10000_files &&\n>>>>> -\tfor i in $(test_seq 1 10); do touch 10_files/$i; done &&\n>>>>> -\tfor i in $(test_seq 1 100); do touch 100_files/$i; done &&\n>>>>> -\tfor i in $(test_seq 1 1000); do touch 1000_files/$i; done &&\n>>>>> -\tfor i in $(test_seq 1 10000); do touch 10000_files/$i; done &&\n>>>>> +\ttest-tool touch sequence --pattern=\"10_files/%d\" --start=1 --count=10 &&\n>>>>> +\ttest-tool touch sequence --pattern=\"100_files/%d\" --start=1 --count=100 &&\n>>>>> +\ttest-tool touch sequence --pattern=\"1000_files/%d\" --start=1 --count=1000 &&\n>>>>> +\ttest-tool touch sequence --pattern=\"10000_files/%d\" --start=1 --count=10000 &&\n>>>>> +\n>>>>> \tgit add 1_file 10_files 100_files 1000_files 10000_files &&\n>>>>> \tgit commit -qm \"Add files\" &&\n>>>>> @@ -200,15 +201,12 @@ test_fsmonitor_suite() {\n>>>>> \t# Update the mtimes on upto 100k files to make status think\n>>>>> \t# that they are dirty. For simplicity, omit any files with\n>>>>> \t# LFs (i.e. anything that ls-files thinks it needs to dquote).\n>>>>> -\t# Then fully backslash-quote the paths to capture any\n>>>>> -\t# whitespace so that they pass thru xargs properly.\n>>>>> \t#\n>>>>> \ttest_perf_w_drop_caches \"status (dirty) ($DESC)\" '\n>>>>> \t\tgit ls-files | \\\n>>>>> \t\t\thead -100000 | \\\n>>>>> \t\t\tgrep -v \\\" | \\\n>>>>> -\t\t\tsed '\\''s/\\(.\\)/\\\\\\1/g'\\'' | \\\n>>>>> -\t\t\txargs test-tool chmtime -300 &&\n>>>>> +\t\t\ttest-tool touch stdin &&\n>>>>> \t\tgit status\n>>>>> \t'\n>>>> Did you try to replace this with some variant of:\n>>>> test_seq 1 10000 | xargs touch\n>>>> Which (depending on your xargs version) would invoke \"touch\"\n>>>> commands\n>>>> with however many argv items it thinks you can handle.\n>>>>\n>>>\n>>> a quick test on my Windows machine shows that\n>>>\n>>> \ttest_seq 1 10000 | xargs touch\n>>>\n>>> takes 3.1 seconds.\n>>>\n>>> just a simple\n>>>\n>>> \ttest_seq 1 10000 >/dev/null\n>>>\n>>> take 0.2 seconds.\n>>>\n>>> using my test-tool helper cuts that time in half.\n>> There's what Elijah mentioned about test_seq, so maybe it's just\n>> that.\n>> But what I was suggesting was using the xargs mode where it does N\n>> arguments at a time.\n>> Does this work for you, and does it cause xargs to invoke \"touch\"\n>> with\n>> the relevant N number of arguments, and does it help with the\n>> performance?\n>> test_seq 1 10000 | xargs touch\n>> test_seq 1 10000 | xargs -n 10 touch\n>> test_seq 1 10000 | xargs -n 100 touch\n>> test_seq 1 10000 | xargs -n 1000 touch\n>\n> The GFW SDK version of xargs does have `-n N` and it does work as\n> advertised. And it does slow down things considerably. Letting it\n> do ~2500 per command in 4 commands took the 3.1 seconds listed above.\n>\n> Add a -n 100 to it takes 5.7 seconds, so process creation overhead\n> is a factor here.\n\nDoesn't -n 2500 being faster than -n 100 suggest the opposite of process\noverhead being the deciding factor? With -n 2500 you'll invoke 4 touch\nprocesses, so one takes 2500/3.1 =~ 0.8s to run, whereas with -n 100 you\ninvoke 100 of them, so if the overall time is then 5.7 seconds that's\n5.7/100 =~ ~0.06s.\n\nOr am I misunderstanding you, or does some implicit parallelism kick in\nwith that version of xargs depending on -n?\n\n>> etc.\n>> Also I didn't notice this before, but the -300 part of \"chmtime\n>> -300\"\n>> was redundant before then? I.e. you're implicitly changing it to \"=+0\"\n>> instead with your \"touch\" helper, are you not?\n>> \n>\n> Right. I'm changing it to the current time.\n\nIf that \"while we're at it change the behavior of the test\" is wanted I\nthink it should be called out in the commit message. Right now it looks\nlike it might be an unintentional regression in the test.\n"},{"id":"430695","messageId":"87tukovidd.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"55e15a7a-b15e-78bd-b3c3-4adff3363edd@jeffhostetler.com","subject":"Re: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-20T19:38:33Z","receivedAt":"2021-07-20T20:44:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 13 2021, Jeff Hostetler wrote:\n\n> On 7/13/21 1:54 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Tue, Jul 13 2021, Jeff Hostetler wrote:\n>> \n>>> My response here is in addition to Dscho's remarks on this topic.\n>>> He makes excellent points that I'll just #include here. I do want\n>>> to add my own $0.02 here.\n>>>\n>>> On 7/1/21 6:18 PM, Ævar Arnfjörð Bjarmason wrote:\n>>>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>>>>\n>\n>>>>> +/*\n>>>>> + * This is adapted from `wait_or_whine()`. Watch the child process and\n>>>>> + * let it get started and begin listening for requests on the socket\n>>>>> + * before reporting our success.\n>>>>> + */\n>>>>> +static int wait_for_background_startup(pid_t pid_child)\n>>>>> +{\n>>>>> +\tint status;\n>>>>> +\tpid_t pid_seen;\n>>>>> +\tenum ipc_active_state s;\n>>>>> +\ttime_t time_limit, now;\n>>>>> +\n>>>>> +\ttime(&time_limit);\n>>>>> +\ttime_limit += fsmonitor__start_timeout_sec;\n>>>>> +\n>>>>> +\tfor (;;) {\n>>>>> +\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n>>>>> +\n>>>>> +\t\tif (pid_seen == -1)\n>>>>> +\t\t\treturn error_errno(_(\"waitpid failed\"));\n>>>>> +\t\telse if (pid_seen == 0) {\n>>>>> +\t\t\t/*\n>>>>> +\t\t\t * The child is still running (this should be\n>>>>> +\t\t\t * the normal case). Try to connect to it on\n>>>>> +\t\t\t * the socket and see if it is ready for\n>>>>> +\t\t\t * business.\n>>>>> +\t\t\t *\n>>>>> +\t\t\t * If there is another daemon already running,\n>>>>> +\t\t\t * our child will fail to start (possibly\n>>>>> +\t\t\t * after a timeout on the lock), but we don't\n>>>>> +\t\t\t * care (who responds) if the socket is live.\n>>>>> +\t\t\t */\n>>>>> +\t\t\ts = fsmonitor_ipc__get_state();\n>>>>> +\t\t\tif (s == IPC_STATE__LISTENING)\n>>>>> +\t\t\t\treturn 0;\n>>>>> +\n>>>>> +\t\t\ttime(&now);\n>>>>> +\t\t\tif (now > time_limit)\n>>>>> +\t\t\t\treturn error(_(\"fsmonitor--daemon not online yet\"));\n>>>>> +\t\t} else if (pid_seen == pid_child) {\n>>>>> +\t\t\t/*\n>>>>> +\t\t\t * The new child daemon process shutdown while\n>>>>> +\t\t\t * it was starting up, so it is not listening\n>>>>> +\t\t\t * on the socket.\n>>>>> +\t\t\t *\n>>>>> +\t\t\t * Try to ping the socket in the odd chance\n>>>>> +\t\t\t * that another daemon started (or was already\n>>>>> +\t\t\t * running) while our child was starting.\n>>>>> +\t\t\t *\n>>>>> +\t\t\t * Again, we don't care who services the socket.\n>>>>> +\t\t\t */\n>>>>> +\t\t\ts = fsmonitor_ipc__get_state();\n>>>>> +\t\t\tif (s == IPC_STATE__LISTENING)\n>>>>> +\t\t\t\treturn 0;\n>>>>> +\n>>>>> +\t\t\t/*\n>>>>> +\t\t\t * We don't care about the WEXITSTATUS() nor\n>>>>> +\t\t\t * any of the WIF*(status) values because\n>>>>> +\t\t\t * `cmd_fsmonitor__daemon()` does the `!!result`\n>>>>> +\t\t\t * trick on all function return values.\n>>>>> +\t\t\t *\n>>>>> +\t\t\t * So it is sufficient to just report the\n>>>>> +\t\t\t * early shutdown as an error.\n>>>>> +\t\t\t */\n>>>>> +\t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n>>>>> +\t\t} else\n>>>>> +\t\t\treturn error(_(\"waitpid is confused\"));\n>>>>> +\t}\n>>>>> +}\n>>>> Ditto this. could we extend the wait_or_whine() function (or some\n>>>> extended version thereof) to do what you need with callbacks?\n>>>> It seems the main difference is just being able to pass down a flag\n>>>> for\n>>>> waitpid(), and the loop needing to check EINTR or not depending on\n>>>> whether WNOHANG is passed.\n>>>> For e.g. the \"We don't care about the WEXITSTATUS()\" you'd get that\n>>>> behavior with an adjusted wait_or_whine(). Wouldn't it be better to\n>>>> report what exit status it exits with e.g. if the top-level process is\n>>>> signalled? We do so in trace2 for other things we spawn...\n>>>>\n>>>\n>>> Again, I don't want to mix my usage here with the existing code\n>>> and destabilize all existing callers. Here we are spinning to give\n>>> the child a chance to *start* and confirm that it is in a listening\n>>> state and ready for connections. We do not wait for the child to\n>>> exit (unless it dies quickly without becoming ready).\n>>>\n>>> We want to end our wait as soon as we confirm that the child is\n>>> ready and return. All I really need from the system is `waitpid()`.\n>> Will this code behave correctly if the daemon we start is signalled\n>> per\n>> the WIFSIGNALED() cases the code this is derived handles, but this does\n>> not?\n>\n> We're only waiting until the child gets started and is able to receive\n> requests -- what happens to it after we have confirmed that it is ready\n> is not our concern (after all, the parent is about to exit anyway and\n> the child is going to continue on).\n>\n> If waitpid() gives us a WIFSIGNALED (or any other WIF*() state) before\n> we have spoken to it, we will return a \"failed to start\".\n\nSo in wait_or_whine() and finish_command() we capture all of that in\ntrace2 logs, we explicitly don't want that in this case? We do concern\nourselves with the exact exit status/signal status etc. of children we\nstart in most other scenarios for trace2 logging purposes.\n\n> But again, that signal would have to arrive immediately after we spawned\n> it and *before* we could talk to it. If the child is signaled after we\n> confirmed it was ready, we don't care because the parent process will be\n> gone.\n>\n> (If the child is signaled or is killed (or crashes or whatever), the\n> next Git command (like \"status\") that tries to talk to it will re-start\n> it implicitly -- the `git fsmonitor--daemon start` command here is an\n> explicit start.)\n>\n>\n>> But sure, I just meant to point out that the flip side to\n>> \"destabilize\n>> all existing callers\" is reviewing new code that may be subtly buggy,\n>> and those subtle bugs (if any) would be smoked out if we were forced to\n>> extend run-command.c, i.e. to use whatever feature(s) this needs for all\n>> existing callers.\n>> \n>\n> That would/could have a massive footprint. And I've already established\n> that my usage here is sufficiently different from existing uses that the\n> result would be a mess. IMHO.\n\nI hadn't see this before but it seems pretty much exactly the same code\nwas already added (by you) in 36a7eb68760 (t0052: add simple-ipc tests\nand t/helper/test-simple-ipc tool, 2021-03-22), perhaps splitting it\ninto a utility function for the two to use with a callback mechanism\nwould reduce the footprint?\n\nWhat I was suggesting was some continuation of the below.\n\n(I stopped once I noticed the changes I was making to\nbuiltin/fsmonitor--daemon.c didn't even compile (almost the entire file\nis hidden behind a macro), but I've commented on that aspect\nelsewhere. I.e. it's nice to be able to do tree-wide refactoring without\ntripping over code hidden by ifdefs)\n\nIt passes all current tests for whatever that's worth, obviously not a\npretty API, and I'm not claiming it's correct.\n\nBut I think it's clear how the trace2/error handling part of it could be\nfurther extracted into some utility, so this would just be a mode of\nrun-command.\n\nNot saying you need to do it, but the comments about us explicitly not\ncaring at all about the exit state make me wonder if there's some reason\nfor why someone else would be tripping over some landmine if they did\nthat refactoring.\n\nAnyway, just a thought. I see from other feedback that you seem to be\ngetting pretty exasperated with me.\n\nI'm just trying to help this along, usually being able to piggy-back on\nexisting in-tree code and proving the correctness by passing all in-tree\ntests with that piggy-backing helps more than hinders.\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 25f18f2726b..7365fff95f4 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -8,6 +8,7 @@\n #include \"simple-ipc.h\"\n #include \"khash.h\"\n #include \"pkt-line.h\"\n+#include \"run-command.h\"\n \n static const char * const builtin_fsmonitor__daemon_usage[] = {\n \tN_(\"git fsmonitor--daemon start []\"),\n@@ -1403,11 +1404,12 @@ static int wait_for_background_startup(pid_t pid_child)\n \ttime_limit += fsmonitor__start_timeout_sec;\n \n \tfor (;;) {\n+\t\tint saved_errno = 0;\n+\t\tcode = wait_or_whine_extended(pid_child, &pid_seen, \"TODO\",\n+\t\t\t\t\t 0, WNOHANG, &saved_errno);\n \t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n \n-\t\tif (pid_seen == -1)\n-\t\t\treturn error_errno(_(\"waitpid failed\"));\n-\t\telse if (pid_seen == 0) {\n+\t\tif (pid_seen == 0) {\n \t\t\t/*\n \t\t\t * The child is still running (this should be\n \t\t\t * the normal case). Try to connect to it on\n@@ -1452,8 +1454,7 @@ static int wait_for_background_startup(pid_t pid_child)\n \t\t\t * early shutdown as an error.\n \t\t\t */\n \t\t\treturn error(_(\"fsmonitor--daemon failed to start\"));\n-\t\t} else\n-\t\t\treturn error(_(\"waitpid is confused\"));\n+\t\t}\n \t}\n }\n \ndiff --git a/run-command.c b/run-command.c\nindex aacc336f951..856e7d87c40 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -543,24 +543,28 @@ static inline void set_cloexec(int fd)\n \t\tfcntl(fd, F_SETFD, flags | FD_CLOEXEC);\n }\n \n-static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n+int wait_or_whine_extended(pid_t pid, pid_t *waiting, const char *argv0,\n+\t\t\t int in_signal, int waitpid_options, int *failed_errno)\n {\n \tint status, code = -1;\n-\tpid_t waiting;\n-\tint failed_errno = 0;\n \n-\twhile ((waiting = waitpid(pid, &status, 0)) < 0 && errno == EINTR)\n+\tif (waitpid_options & WNOHANG)\n+\t\t*waiting = waitpid(pid, &status, waitpid_options);\n+\telse\n+\t\twhile ((*waiting = waitpid(pid, &status, waitpid_options)) < 0 &&\n+\t\t errno == EINTR)\n \t\t;\t/* nothing */\n+\n \tif (in_signal) {\n \t\tif (WIFEXITED(status))\n \t\t\tcode = WEXITSTATUS(status);\n \t\treturn code;\n \t}\n \n-\tif (waiting < 0) {\n-\t\tfailed_errno = errno;\n+\tif (*waiting < 0) {\n+\t\t*failed_errno = errno;\n \t\terror_errno(\"waitpid for %s failed\", argv0);\n-\t} else if (waiting != pid) {\n+\t} else if (*waiting != pid) {\n \t\terror(\"waitpid is confused (%s)\", argv0);\n \t} else if (WIFSIGNALED(status)) {\n \t\tcode = WTERMSIG(status);\n@@ -574,14 +578,23 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n \t\tcode += 128;\n \t} else if (WIFEXITED(status)) {\n \t\tcode = WEXITSTATUS(status);\n+\t} else if (waitpid_options & WNOHANG && *waiting == 0) {\n+\t\tcode = 0;\n \t} else {\n \t\terror(\"waitpid is confused (%s)\", argv0);\n \t}\n \n-\tclear_child_for_cleanup(pid);\n+\treturn code;\n+}\n \n+static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)\n+{\n+\tpid_t ignore;\n+\tint failed_errno = 0;\n+\tint ret = wait_or_whine_extended(pid, &ignore, argv0, in_signal, 0, &failed_errno);\n+\tclear_child_for_cleanup(pid);\n \terrno = failed_errno;\n-\treturn code;\n+\treturn ret;\n }\n \n static void trace_add_env(struct strbuf *dst, const char *const *deltaenv)\ndiff --git a/run-command.h b/run-command.h\nindex e321d23bbd2..fe39730f87a 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -182,6 +182,9 @@ void child_process_clear(struct child_process *);\n \n int is_executable(const char *name);\n \n+int wait_or_whine_extended(pid_t pid, pid_t *waiting, const char *argv0,\n+\t\t\t int in_signal, int waitpid_options, int *failed_errno);\n+\n /**\n * Start a sub-process. Takes a pointer to a `struct child_process`\n * that specifies the details and returns pipe FDs (if requested).\ndiff --git a/t/helper/test-simple-ipc.c b/t/helper/test-simple-ipc.c\nindex 91345180750..44658a46713 100644\n--- a/t/helper/test-simple-ipc.c\n+++ b/t/helper/test-simple-ipc.c\n@@ -9,6 +9,7 @@\n #include \"parse-options.h\"\n #include \"thread-utils.h\"\n #include \"strvec.h\"\n+#include \"run-command.h\"\n \n #ifndef SUPPORTS_SIMPLE_IPC\n int cmd__simple_ipc(int argc, const char **argv)\n@@ -349,7 +350,7 @@ static int spawn_server(pid_t *pid)\n */\n static int wait_for_server_startup(pid_t pid_child)\n {\n-\tint status;\n+\tint code;\n \tpid_t pid_seen;\n \tenum ipc_active_state s;\n \ttime_t time_limit, now;\n@@ -358,12 +359,10 @@ static int wait_for_server_startup(pid_t pid_child)\n \ttime_limit += cl_args.max_wait_sec;\n \n \tfor (;;) {\n-\t\tpid_seen = waitpid(pid_child, &status, WNOHANG);\n-\n-\t\tif (pid_seen == -1)\n-\t\t\treturn error_errno(_(\"waitpid failed\"));\n-\n-\t\telse if (pid_seen == 0) {\n+\t\tint saved_errno = 0;\n+\t\tcode = wait_or_whine_extended(pid_child, &pid_seen, \"TODO\",\n+\t\t\t\t\t 0, WNOHANG, &saved_errno);\n+\t\tif (pid_seen == 0) {\n \t\t\t/*\n \t\t\t * The child is still running (this should be\n \t\t\t * the normal case). Try to connect to it on\n@@ -384,9 +383,7 @@ static int wait_for_server_startup(pid_t pid_child)\n \t\t\t\treturn error(_(\"daemon not online yet\"));\n \n \t\t\tcontinue;\n-\t\t}\n-\n-\t\telse if (pid_seen == pid_child) {\n+\t\t} else if (pid_seen == pid_child) {\n \t\t\t/*\n \t\t\t * The new child daemon process shutdown while\n \t\t\t * it was starting up, so it is not listening\n@@ -412,10 +409,9 @@ static int wait_for_server_startup(pid_t pid_child)\n \t\t\t * early shutdown as an error.\n \t\t\t */\n \t\t\treturn error(_(\"daemon failed to start\"));\n+\t\t} else if (code) {\n+\t\t\tBUG(\"??\");\n \t\t}\n-\n-\t\telse\n-\t\t\treturn error(_(\"waitpid is confused\"));\n \t}\n }\n \n\n"},{"id":"430697","messageId":"87r1fsvhcr.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"c62cf635-1d24-3b63-9566-b98aa2ec2af5@jeffhostetler.com","subject":"Re: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-20T20:32:38Z","receivedAt":"2021-07-20T21:02:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 19 2021, Jeff Hostetler wrote:\n\n> On 7/1/21 6:45 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n>> \n>>> From: Jeff Hostetler \n>>>\n>>> Stub in empty backend for fsmonitor--daemon on Windows.\n>>>\n>>> Signed-off-by: Jeff Hostetler \n>>> ---\n>>> Makefile | 13 ++++++\n>>> compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++\n>>> compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++\n>>> config.mak.uname | 2 +\n>>> contrib/buildsystems/CMakeLists.txt | 5 ++\n>>> 5 files changed, 90 insertions(+)\n>>> create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c\n>>> create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h\n>>>\n>>> diff --git a/Makefile b/Makefile\n>>> index c45caacf2c3..a2a6e1f20f6 100644\n>>> --- a/Makefile\n>>> +++ b/Makefile\n>>> @@ -467,6 +467,11 @@ all::\n>>> # directory, and the JSON compilation database 'compile_commands.json' will be\n>>> # created at the root of the repository.\n>>> #\n>>> +# If your platform supports a built-in fsmonitor backend, set\n>>> +# FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n>>> +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the\n>>> +# `fsmonitor_fs_listen__*()` routines.\n>>> +#\n>>> # Define DEVELOPER to enable more compiler warnings. Compiler version\n>>> # and family are auto detected, but could be overridden by defining\n>>> # COMPILER_FEATURES (see config.mak.dev). You can still set\n>>> @@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER\n>>> \tCOMPAT_OBJS += compat/access.o\n>>> endif\n>>> +ifdef FSMONITOR_DAEMON_BACKEND\n>>> +\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n>>> +\tCOMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o\n>>> +endif\n>>> +\n>>> ifeq ($(TCLTK_PATH),)\n>>> NO_TCLTK = NoThanks\n>>> endif\n>>> @@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE\n>>> \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n>>> \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n>>> \t@echo X=\\'$(X)\\' >>$@+\n>>> +ifdef FSMONITOR_DAEMON_BACKEND\n>>> +\t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n>>> +endif\n>> Why put this in an ifdef?\n>> In 342e9ef2d9e (Introduce a performance testing framework,\n>> 2012-02-17)\n>> we started doing that for some perf/test options (which b.t.w., I don't\n>> really see the reason for, maybe it's some subtlety in how test-lib.sh\n>> picks those up).\n>> But for all the other compile-time stuff we don't ifdef it, we just\n>> define it, and then you get an empty value or not.\n>> This would AFAICT be the first build-time-for-the-C-program option\n>> we\n>> ifdef for writing a line to GIT-BUILD-OPTIONS.\n>> \n>\n> (I'm going to respond here on the original question rather than on any\n> of the follow up responses in an attempt at diffusing things a bit.)\n>\n> I added the ifdef because I thought it to be the *most conservative*\n> thing that I could do. The output of the generated file on unsupported\n> platforms should be *identical* to what it was before my changes. I\n> only alter the contents of the generated file on supported platforms.\n>\n> Later, when the generated file is consumed, we don't need to worry about\n> the effect (if any) on incremental compiles -- we will know that it\n> won't be set -- just like it was not set in the original compile.\n\nOkey, so e.g. when we added e.g. USE_LIBPCRE2 we added it TO\nGIT-BUILD-OPTIONS unconditionally, so if you pulled that commit you'd\ntrigger a rebuild on anything that cares about GIT-BUILD-OPTIONS (which\nis almost everything).\n\nBut you'd like to have the line not added to avoid that one-off\nrecompile....\n\n> That change appears right before a 12 other ifdef'd symbols also being\n> written to that generated file. Most are test and perf, but some are\n> not. But my point is that the pattern is present already.\n>\n> The original question also references a 9.5 year old commit which\n> uses the same pattern as I've used here. It also muddies the water\n> on why it was/wasn't needed back then. And hints at possible\n> side-effects in some of our test scripts. So it is clear that the\n> confusion/disagreements that we are having with the current patch\n> and whether or not to ifdef are not new.\n>\n>\n> So, is there value in being explicit and having the ifdef ??\n>\n>\n> There are well defined Make rules (and Junio gave us a very elegant\n> little script to demonstrate that), but the subtleties are there.\n> Especially with our use generated files like `GIT-BUILD-OPTIONS`.\n> We have a mailing list full of experts and yet this question received\n> a lot more discussion than I thought possible or necessary, but it\n> took a test script to demonstrate that the results are the same and it\n> doesn't matter. Perhaps the clarity is worth it for the price of a\n> simple ifdef.\n>\n>\n> So, how much time have we (collectively) wasted discussing this\n> subtlety ??\n>\n>\n> To summarize, I added the ifdef to make it explicitly clear that\n> I'm not altering behavior on unsupported platforms. I can remove it\n> from V4 if desired or I can keep it. (We all now know that it doesn't\n> functionally matter -- it does however, provide clarity.)\n>\n>\n> Sorry if this sounded like a rant,\n\n...I asked because I've looked at that ifdef soup around\nGIT-BUILD-OPTIONS and wondered if I could make it go away, and before a\npatch lands is a good time to ask \"what's this pattern for?\", as opposed\nto inferring this after the fact.\n\nFor me it was just a minor curiosity, I didn't expect to start this big\ndiscussion about it. I expected just a \"oh, I just copy/pasted that from\nthe lines at the end\" or something, which would be fair enough.\n\nI really don't care which one we go for here. If you want to change it\nfine, if not that's fine too.\n\nI have noticed a pattern where you seem to really carefully consider why\nyou'd like X over Y. I.e. it wasn't just copy/pasting in this case if I\nunderstand you correctly, but a carefully thought out decision to not do\nit like the other C-level-GIT-BUILD-OPTIONS.\n\nOkey, fair enough, but that decision then doesn't go into the commit\nmessage, and then when I innocently ask about it...\n\n..I guess I'll stop before this starts resembling a rant on my part :)\n\nAnyway, I have also had really non-trivial comments on this fsmonitor\nseries, not just a few bikeshed comments. I.e. the un-addressed question\nabout the wildly different performance numbers we seem to have seen in\nour respective testing:\nhttps://lore.kernel.org/git/871r8c73ej.fsf@evledraar.gmail.com\n\nI think that's much more interesting than this relatively light-reading\nbikeshedding I had while giving this a read-through.\n"},{"id":"430773","messageId":"8b7e1d69-b89a-239b-af4b-53d90dbec651@jeffhostetler.com","threadId":"55420","inReplyTo":"878s2pboha.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 29/34] fsmonitor--daemon: use a cookie file to sync with file system","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-21T14:40:54Z","receivedAt":"2021-07-21T14:40:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/1/21 7:17 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Jul 01 2021, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Teach fsmonitor--daemon client threads to create a cookie file\n>> inside the .git directory and then wait until FS events for the\n>> cookie are observed by the FS listener thread.\n>>\n>> This helps address the racy nature of file system events by\n>> blocking the client response until the kernel has drained any\n>> event backlog.\n>>\n>> This is especially important on MacOS where kernel events are\n>> only issued with a limited frequency. See the `latency` argument\n>> of `FSeventStreamCreate()`. The kernel only signals every `latency`\n>> seconds, but does not guarantee that the kernel queue is completely\n>> drained, so we may have to wait more than one interval. If we\n>> increase the frequency, the system is more likely to drop events.\n>> We avoid these issues by having each client thread create a unique\n>> cookie file and then wait until it is seen in the event stream.\n> \n> Is this a guaranteed property of any API fsmonitor might need to work\n> with (linux, darwin, Windows) that if I perform a bunch of FS operations\n> on my working tree, that if I finish up by touching this cookie file\n> that that'll happen last?\n> \n> I'd think that wouldn't be the case, i.e. on POSIX filesystems unless\n> you run around fsyncing both files and directories you're not guaranteed\n> that they're on disk, and even then the kernel might decide to sync your\n> cookie earlier, won't it?\n> \n> E.g. on Linux you can even have cross-FS watches, and mix & match\n> different FS types. I'd expect to get events in whatever\n> implementation-defined order the VFS layer + FS decided to sync them to\n> disk in & get to firing off an event for me.\n> \n> Or do these APIs all guarantee that a linear view of the world is\n> presented to the API consumer?\n> \n\n\nTheoretically, none of these APIs guarantee a complete linear ordering.\nWe receive events from the FS in the order that the FS decides to\nperform the actual IO. And the inner workings of the FS is private.\nEven if we directly read the journal rather than listening for\nnotifications, we probably still don't know whether the FS reordered\nthe queue of things heading to disk.\n\nHowever in practice, the events for the cookie files do tend to arrive\nin order. And the net effect is that the worker thread in the daemon\nis sync'd up with IO activity that was initiated before the request.\n\n\nBTW Watchman also uses cookie files for this same reason.\n\n\nIt should also be noted that some operations are just racy. If you're\ndoing a bunch of IO in one window and a 'git status' in another window,\nyour result will be racy -- status (without FSM) makes 2 passes on the\ndisk: the first to verify mtimes on items in the index and the second\nto look for untracked files. the status result may be \"blurry\" (for\nlack of a better word). So the same questions\n \"does the FS reorder my IO?\",\n \"did status see the fully sync'd FS?\",\nand etc can also be asked in the normal (non FSM) case, right?\n\nSo it may be the case that having an fsmonitor (mine, Watchman, etc)\nand the untracked-cache, we'll have less skew in status results\nbecause the status process shouldn't have to do any scanning.\nBut I'm not sure I want to make that assertion yet.\n\nThanks,\nJeff\n\n"},{"id":"431237","messageId":"nycvar.QRO.7.76.6.2107262338320.55@tvgsbejvaqbjf.bet","threadId":"55420","inReplyTo":"871r7yxkq8.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-07-26T21:40:18Z","receivedAt":"2021-07-26T21:40:33Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Fri, 16 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, Jul 16 2021, Johannes Schindelin wrote:\n>\n> > So you suggest that we name the new stuff after an `uname` that\n> > reflects a name that is no longer relevant? I haven't seen a real\n> > Darwin system in quite a long time, have you?\n>\n> It's not current? On an Mac Mini M1 which got released this year:\n>\n> % uname -s\n> Darwin\n>\n> We then have the same in config.mak.uname, it seemed the most obvious\n> and consistent to carry that through to file inclusion.\n\nSorry. I assumed that you knew that Darwin was the name for an open source\nOperating System. See\nhttps://en.wikipedia.org/wiki/Darwin_%28operating_system%29 for more\ndetails.\n\nCiao,\nJohannes\n"},{"id":"431248","messageId":"xmqqr1fkfyh3.fsf@gitster.g","threadId":"55420","inReplyTo":"nycvar.QRO.7.76.6.2107262338320.55@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-26T23:26:48Z","receivedAt":"2021-07-26T23:26:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Hi Ævar,\n>\n> On Fri, 16 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n>> On Fri, Jul 16 2021, Johannes Schindelin wrote:\n>>\n>> > So you suggest that we name the new stuff after an `uname` that\n>> > reflects a name that is no longer relevant? I haven't seen a real\n>> > Darwin system in quite a long time, have you?\n>>\n>> It's not current? On an Mac Mini M1 which got released this year:\n>>\n>> % uname -s\n>> Darwin\n>>\n>> We then have the same in config.mak.uname, it seemed the most obvious\n>> and consistent to carry that through to file inclusion.\n>\n> Sorry. I assumed that you knew that Darwin was the name for an open source\n> Operating System. See\n> https://en.wikipedia.org/wiki/Darwin_%28operating_system%29 for more\n> details.\n>\n> Ciao,\n> Johannes\n\nSorry, but I do not see that you are being more constructive than\nthe other party, whom you blame to be not constructive, in this\nexchange.\n\nThe part of the file that the patch applies to uses $(uname_S) to\nimplement platform specific special cases, and we are looking at\n\n\tifeq ($(uname_S),Darwin)\n\t\t...\n\t\tFSMONITOR_DAEMON_BACKEND = macos\n\t\t...\n\tendif\n\nI find it a fair question why the name used there has to be\ndifferent from the one we can automatically and mechanically\nget out of \"uname -s\".\n\nThen you respond that uname output is no longer relevant because\nDarwin is a name that is no longer relevant? And when asked why the\nname is no longer relevant, you make a sniding comment implying that\nthe other party does not know the name is an operating system?\n\nWhat is going on here?\n\nIt does not really matter how \"Darwin\" is described in an\nencyclopedia in the context of this discussion. What matters is\nthat it is what the system's \"uname -s\" currently uses to identify\nitself, and what we guard the section of makefile snippet with,\nisn't it?\n\nci/lib.sh seems to have an attempt to unify/translate among these\nnames, and\n\n * on azure-pipelines, it wants to translate darwin to osx\n * on github-actions, it wants to translate macos to osx\n\nPresumably that is because these two systems call the platform with\nthese two different names, and you want to pick a middle ground that\nnobody uses to be neutral, or something?\n\nAlso, in contrib/vscode/init.sh, I see Darwin obtained from \"uname -s\"\ngets translated to \"macOS\".\n\nIn any case, if your argument was \"we picked macos because we use\nthe same token elsewhere, while trying to translate away from Darwin\nas much as possible for such and such reasons\", I would have found\nit a productive exchange, but unfortunately that is not what I am\nseeing here.\n"},{"id":"431297","messageId":"38c22648-df46-2464-0c04-347dedbd04c6@jeffhostetler.com","threadId":"55420","inReplyTo":"xmqqr1fkfyh3.fsf@gitster.g","subject":"Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-07-27T12:46:16Z","receivedAt":"2021-07-27T12:46:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/26/21 7:26 PM, Junio C Hamano wrote:\n> Johannes Schindelin writes:\n> \n>> Hi Ævar,\n>>\n>> On Fri, 16 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n>>\n>>> On Fri, Jul 16 2021, Johannes Schindelin wrote:\n>>>\n...\n\n\nI'm not sure that there is a \"correct\" answer here, but for the sake\nof harmony, in V4 I'll set this to \"darwin\" and update the name of\nthe backend driver source file to match. So that we are consistently\nusing 1 term throughout \"Makefile\" and \"config.mak.uname\".\n\n\tifeq ($(uname_S),Darwin)\n\t...\n\t\tFSMONITOR_DAEMON_BACKEND = darwin\n\tendif\n\n\n\nFWIW, I suspect that it is not worth the effort to directly set the\nbackend name from $(uname_S). For example, on Windows we currently have\ntwo different uname values depending on which compiler is being used.\n\n\tifeq ($(uname_S),Windows)\n\t...\n\t\tFSMONITOR_DAEMON_BACKEND = win32\n\tendif\n\n\tifneq (,$(findstring MINGW,$(uname_S)))\n\t...\n\t\tFSMONITOR_DAEMON_BACKEND = win32\n\tendif\n\n\nAlso, since the backend layer is highly platform-specific, it may be\na while (if ever) before we have universal coverage for all platforms.\nUntil then, we can simply set $FSMONITOR_DAEMON_BACKEND to a literal\nvalue on a platform-by-platform basis as support is added.\n\n\nThanks,\nJeff\n"},{"id":"431314","messageId":"87pmv3r9ud.fsf@evledraar.gmail.com","threadId":"55420","inReplyTo":"38c22648-df46-2464-0c04-347dedbd04c6@jeffhostetler.com","subject":"Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-27T15:56:37Z","receivedAt":"2021-07-27T16:37:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 27 2021, Jeff Hostetler wrote:\n\n> On 7/26/21 7:26 PM, Junio C Hamano wrote:\n>> Johannes Schindelin writes:\n>> \n>>> Hi Ævar,\n>>>\n>>> On Fri, 16 Jul 2021, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>>> On Fri, Jul 16 2021, Johannes Schindelin wrote:\n>>>>\n> ...\n>\n>\n> I'm not sure that there is a \"correct\" answer here, but for the sake\n> of harmony, in V4 I'll set this to \"darwin\" and update the name of\n> the backend driver source file to match. So that we are consistently\n> using 1 term throughout \"Makefile\" and \"config.mak.uname\".\n>\n> \tifeq ($(uname_S),Darwin)\n> \t...\n> \t\tFSMONITOR_DAEMON_BACKEND = darwin\n> \tendif\n>\n>\n>\n> FWIW, I suspect that it is not worth the effort to directly set the\n> backend name from $(uname_S). For example, on Windows we currently have\n> two different uname values depending on which compiler is being used.\n>\n> \tifeq ($(uname_S),Windows)\n> \t...\n> \t\tFSMONITOR_DAEMON_BACKEND = win32\n> \tendif\n>\n> \tifneq (,$(findstring MINGW,$(uname_S)))\n> \t...\n> \t\tFSMONITOR_DAEMON_BACKEND = win32\n> \tendif\n>\n>\n> Also, since the backend layer is highly platform-specific, it may be\n> a while (if ever) before we have universal coverage for all platforms.\n> Until then, we can simply set $FSMONITOR_DAEMON_BACKEND to a literal\n> value on a platform-by-platform basis as support is added.\n\nRe \"harmony\": For what it's worth I don't think you should change it on\nmy accord.\n\nI should probably have more explicitly said (but I've also been trying\nto check the general verbosity of my E-Mails), that when I read a series\nlike this and have some general trivial comments like this, I mean them\nas something like:\n\n Just a thought while reading this through, i.e. a person familiar\n with the general codebase but not necessarily your specific\n are. Maybe this suggestion makes things easier/simpler, but if you\n think not and decide not to take the suggestion that's fine too.\n\nI.e. that along with the general implicit suggestion that I'd say\napplies in general on list that if someone is perplexed by a patch by\ndefault that's a comment on the commit message.\n\nThat person (i.e. me in this case) could also just be hopelessly\nconfused & nothing needs to change. When I get comments like that I\nsometimes change things, sometimes not. You should do the same.\n\nAs noted in another reply on this general thread & what's cooking I seem\nto have poked a bit of a hornet's nest here that I wasn't expecting to\npoke. I'd not been following earlier rounds of this topic, and didn't\nknow that it had (seemingly) reached some phase of critical updates only\nin the minds of its authors.\n\n"},{"id":"431321","messageId":"xmqqo8anekk2.fsf@gitster.g","threadId":"55420","inReplyTo":"38c22648-df46-2464-0c04-347dedbd04c6@jeffhostetler.com","subject":"Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-27T17:25:01Z","receivedAt":"2021-07-27T17:25:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> I'm not sure that there is a \"correct\" answer here, but for the sake\n> of harmony, in V4 I'll set this to \"darwin\" and update the name of\n> the backend driver source file to match. So that we are consistently\n> using 1 term throughout \"Makefile\" and \"config.mak.uname\".\n>\n> \tifeq ($(uname_S),Darwin)\n> \t...\n> \t\tFSMONITOR_DAEMON_BACKEND = darwin\n> \tendif\n> ...\n\nI do not think it would help \"harmony\" to change the name, but any\none of 'darwin', 'macos' and 'osx' would be fine.\n\nIt was irritating to see a simple \"why is this particular name\nchosen?\" answered with such a hostility, when even a \"we have no\ndeep reasoning behind the choice of the name. It is only seen by\ndevelopers in names of the source files, and it does not make much\ndifference\" would have been sufficient. I somehow view it more\nproblematic.\n\nI guess the blame goes both ways, though. We all have worked with\neach other long enough to know which of your recipients are prone to\ngo overly defensive when asked questions, and we should know that it\nwould help to prepend \"I am just being curious but...\" to your\nquestions whose answers do not make a huge difference at the end\neither way (or not asking such questions at all).\n\nThanks.\n"},{"id":"431327","messageId":"61004635d03c5_932820832@natae.notmuch","threadId":"55420","inReplyTo":"xmqqo8anekk2.fsf@gitster.g","subject":"Re: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-07-27T17:45:25Z","receivedAt":"2021-07-27T17:45:31Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Junio C Hamano wrote:\n> I guess the blame goes both ways, though. We all have worked with\n> each other long enough to know which of your recipients are prone to\n> go overly defensive when asked questions, and we should know that it\n> would help to prepend \"I am just being curious but...\" to your\n> questions whose answers do not make a huge difference at the end\n> either way (or not asking such questions at all).\n\nI disagree. It's not OK for contributors to react defensively when asked\nquestions, and in particular I don't think it's OK that some\ncontributors are punished for merely disagreeing, while others are given\na pass for snarling at feedback. This is double standards.\n\nThe Git project should not play favorites, and all contributors should\nbe asked to assume good faith.\n\nhttps://en.wikipedia.org/wiki/Wikipedia:Assume_good_faith\n\nIt should be implied that the feedback given is to try to improve the\nproject, it should not need to be stated.\n\n-- \nFelipe Contreras\n"}]}