[PATCH v6 00/30] Builtin FSMonitor Part 2
- From
- Jeff Hostetler via GitGitGadget <gitgitgadget@gmail.com>
- Date
- Mar 1, 2022, 18:43 UTC
- Message-ID
- <pull.1041.v6.git.1646160212.gitgitgadget@gmail.com>
- In-Reply-To
- <pull.1041.v5.git.1644612979.gitgitgadget@gmail.com>
Here is V6 of Part 2 of my builtin FSmonitor series.
This version contains mostly cleanup based on feedback from V5. Of note:
* I squashed in the 1_file fix for p7519. * I squashed in a commit from part 3 to optionally print the "running daemon on..." message on stderr. * I added a note to the documentation about incompatible changes around core.fsmonitor. * Removed/rephrased some obsolete NEEDSWORK items.
Tao has an ongoing parallel series to fix test-chmtime on Windows. https://lore.kernel.org/all/pull.1166.git.1646041236.gitgitgadget@gmail.com/
If that lands first, we should be able to drop my 't/helper/test-chmtime: skip directories on Windows' commit.
A followup Part 3 will contain additional refinements to the daemon and additional tests. I drew the line here between Part 2 and 3 to make it easier to review.
Jeff Hostetler (30):
fsmonitor: enhance existing comments, clarify trivial response
handling
fsmonitor-ipc: create client routines for git-fsmonitor--daemon
fsmonitor: config settings are repository-specific
fsmonitor: use IPC to query the builtin FSMonitor daemon
fsmonitor: document builtin fsmonitor
fsmonitor--daemon: add a built-in fsmonitor daemon
fsmonitor--daemon: implement 'stop' and 'status' commands
compat/fsmonitor/fsm-listen-win32: stub in backend for Windows
compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin
fsmonitor--daemon: implement 'run' command
fsmonitor--daemon: implement 'start' command
fsmonitor--daemon: add pathname classification
fsmonitor--daemon: define token-ids
fsmonitor--daemon: create token-based changed path cache
compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on
Windows
compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent
compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on
MacOS
fsmonitor--daemon: implement handle_client callback
help: include fsmonitor--daemon feature flag in version info
t/helper/fsmonitor-client: create IPC client to talk to FSMonitor
Daemon
t7527: create test for fsmonitor--daemon
t/perf: avoid copying builtin fsmonitor files into test repo
t/helper/test-chmtime: skip directories on Windows
t/perf/p7519: speed up test on Windows
t/perf/p7519: add fsmonitor--daemon test cases
fsmonitor--daemon: periodically truncate list of modified files
fsmonitor--daemon: use a cookie file to sync with file system
fsmonitor: force update index after large responses
t7527: test status with untracked-cache and fsmonitor--daemon
update-index: convert fsmonitor warnings to advise.gitignore | 1 + Documentation/config/core.txt | 60 +- Documentation/git-fsmonitor--daemon.txt | 75 ++ Documentation/git-update-index.txt | 8 +- Makefile | 17 + builtin.h | 1 + builtin/fsmonitor--daemon.c | 1473 +++++++++++++++++++++++ builtin/update-index.c | 19 +- cache.h | 1 - compat/fsmonitor/fsm-listen-darwin.c | 499 ++++++++ compat/fsmonitor/fsm-listen-win32.c | 586 +++++++++ compat/fsmonitor/fsm-listen.h | 49 + config.c | 14 - config.h | 1 - config.mak.uname | 20 + contrib/buildsystems/CMakeLists.txt | 10 + environment.c | 1 - fsmonitor--daemon.h | 166 +++ fsmonitor-ipc.c | 171 +++ fsmonitor-ipc.h | 48 + fsmonitor-settings.c | 115 ++ fsmonitor-settings.h | 21 + fsmonitor.c | 216 +++- fsmonitor.h | 15 +- git.c | 1 + help.c | 4 + repo-settings.c | 1 + repository.h | 3 + t/README | 4 +- t/helper/test-chmtime.c | 15 + t/helper/test-fsmonitor-client.c | 121 ++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/perf/p7519-fsmonitor.sh | 61 +- t/perf/perf-lib.sh | 2 +- t/t7527-builtin-fsmonitor.sh | 604 ++++++++++ t/test-lib.sh | 6 + 37 files changed, 4301 insertions(+), 110 deletions(-) create mode 100644 Documentation/git-fsmonitor--daemon.txt create mode 100644 builtin/fsmonitor--daemon.c create mode 100644 compat/fsmonitor/fsm-listen-darwin.c create mode 100644 compat/fsmonitor/fsm-listen-win32.c create mode 100644 compat/fsmonitor/fsm-listen.h create mode 100644 fsmonitor--daemon.h create mode 100644 fsmonitor-ipc.c create mode 100644 fsmonitor-ipc.h create mode 100644 fsmonitor-settings.c create mode 100644 fsmonitor-settings.h create mode 100644 t/helper/test-fsmonitor-client.c create mode 100755 t/t7527-builtin-fsmonitor.sh
base-commit: 715d08a9e51251ad8290b181b6ac3b9e1f9719d7 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1041%2Fjeffhostetler%2Fbuiltin-fsmonitor-part2-v6 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1041/jeffhostetler/builtin-fsmonitor-part2-v6 Pull-Request: https://github.com/gitgitgadget/git/pull/1041
Range-diff vs v5:
1: a5ecabb4d02 ! 1: 10a34082fcc fsmonitor: enhance existing comments, clarify trivial response handling
@@ Commit message
fsmonitor: enhance existing comments, clarify trivial response handling
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## fsmonitor.c ##
@@ fsmonitor.c: static int query_fsmonitor(int version, const char *last_update, struct strbuf *
2: 365964b7664 ! 2: e1c946af2c5 fsmonitor-ipc: create client routines for git-fsmonitor--daemon
@@ Commit message
Stub in empty fsmonitor_ipc__*() functions for unsupported platforms.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## Makefile ##
@@ Makefile: LIB_OBJS += fetch-pack.o
@@ fsmonitor-ipc.c (new)
+#include "strbuf.h"
+#include "trace2.h"
+
-+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND
++#ifndef HAVE_FSMONITOR_DAEMON_BACKEND
++
++/*
++ * A trivial implementation of the fsmonitor_ipc__ API for unsupported
++ * platforms.
++ */
++
++int fsmonitor_ipc__is_supported(void)
++{
++ return 0;
++}
++
++const char *fsmonitor_ipc__get_path(void)
++{
++ return NULL;
++}
++
++enum ipc_active_state fsmonitor_ipc__get_state(void)
++{
++ return IPC_STATE__OTHER_ERROR;
++}
++
++int fsmonitor_ipc__send_query(const char *since_token,
++ struct strbuf *answer)
++{
++ return -1;
++}
++
++int fsmonitor_ipc__send_command(const char *command,
++ struct strbuf *answer)
++{
++ return -1;
++}
++
++#else
+
+int fsmonitor_ipc__is_supported(void)
+{
@@ fsmonitor-ipc.c (new)
+ return 0;
+}
+
-+#else
-+
-+/*
-+ * A trivial implementation of the fsmonitor_ipc__ API for unsupported
-+ * platforms.
-+ */
-+
-+int fsmonitor_ipc__is_supported(void)
-+{
-+ return 0;
-+}
-+
-+const char *fsmonitor_ipc__get_path(void)
-+{
-+ return NULL;
-+}
-+
-+enum ipc_active_state fsmonitor_ipc__get_state(void)
-+{
-+ return IPC_STATE__OTHER_ERROR;
-+}
-+
-+int fsmonitor_ipc__send_query(const char *since_token,
-+ struct strbuf *answer)
-+{
-+ return -1;
-+}
-+
-+int fsmonitor_ipc__send_command(const char *command,
-+ struct strbuf *answer)
-+{
-+ return -1;
-+}
-+
+#endif
## fsmonitor-ipc.h (new) ##
3: 384516ce1a1 ! 3: ae622a517cf fsmonitor: config settings are repository-specific
@@ Commit message
restrictions to override the config values used.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## Makefile ##
@@ Makefile: LIB_OBJS += fmt-merge-msg.o
@@ fsmonitor-settings.c (new)
+
+enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)
+{
++ if (!r)
++ r = the_repository;
++
+ lookup_fsmonitor_settings(r);
+
+ return r->settings.fsmonitor->mode;
@@ fsmonitor-settings.c (new)
+
+const char *fsm_settings__get_hook_path(struct repository *r)
+{
++ if (!r)
++ r = the_repository;
++
+ lookup_fsmonitor_settings(r);
+
+ return r->settings.fsmonitor->hook_path;
@@ fsmonitor-settings.c (new)
+
+void fsm_settings__set_ipc(struct repository *r)
+{
++ if (!r)
++ r = the_repository;
++
+ lookup_fsmonitor_settings(r);
+
+ r->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;
@@ fsmonitor-settings.c (new)
+
+void fsm_settings__set_hook(struct repository *r, const char *path)
+{
++ if (!r)
++ r = the_repository;
++
+ lookup_fsmonitor_settings(r);
+
+ r->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;
@@ fsmonitor-settings.c (new)
+
+void fsm_settings__set_disabled(struct repository *r)
+{
++ if (!r)
++ r = the_repository;
++
+ lookup_fsmonitor_settings(r);
+
+ r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;
@@ fsmonitor.c: void remove_fsmonitor(struct index_state *istate)
{
unsigned int i;
- int fsmonitor_enabled = git_config_get_fsmonitor();
-+ struct repository *r = istate->repo ? istate->repo : the_repository;
-+ int fsmonitor_enabled = (fsm_settings__get_mode(r) > FSMONITOR_MODE_DISABLED);
++ int fsmonitor_enabled = (fsm_settings__get_mode(istate->repo)
++ > FSMONITOR_MODE_DISABLED);
if (istate->fsmonitor_dirty) {
if (fsmonitor_enabled) {
@@ fsmonitor.h: int fsmonitor_is_trivial_response(const struct strbuf *query_result
static inline int is_fsmonitor_refreshed(const struct index_state *istate)
{
- return !core_fsmonitor || istate->fsmonitor_has_run_once;
-+ struct repository *r = istate->repo ? istate->repo : the_repository;
-+ enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
++ enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);
+
+ return fsm_mode <= FSMONITOR_MODE_DISABLED ||
+ istate->fsmonitor_has_run_once;
@@ fsmonitor.h: static inline int is_fsmonitor_refreshed(const struct index_state *
static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)
{
- if (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {
-+ struct repository *r = istate->repo ? istate->repo : the_repository;
-+ enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
++ enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);
+
+ if (fsm_mode > FSMONITOR_MODE_DISABLED &&
+ !(ce->ce_flags & CE_FSMONITOR_VALID)) {
@@ fsmonitor.h: static inline void mark_fsmonitor_valid(struct index_state *istate,
static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)
{
- if (core_fsmonitor) {
-+ struct repository *r = istate->repo ? istate->repo : the_repository;
-+ enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
++ enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo);
+
+ if (fsm_mode > FSMONITOR_MODE_DISABLED) {
ce->ce_flags &= ~CE_FSMONITOR_VALID;
4: 8e738a83bc5 ! 4: 55974867da5 fsmonitor: use IPC to query the builtin FSMonitor daemon
@@ Commit message
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## fsmonitor.c ##
@@ fsmonitor.c: void refresh_fsmonitor(struct index_state *istate)
5: 49e4c146e02 ! 5: ce42d5bbaf6 fsmonitor: document builtin fsmonitor
@@ Commit message
Create `git-fsmonitor--daemon` manual page and describe its features.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## Documentation/config/core.txt ##
@@ Documentation/config/core.txt: core.protectNTFS::
@@ Documentation/config/core.txt: core.protectNTFS::
+git by avoiding unnecessary scanning of files that have not changed.
++
+See the "fsmonitor-watchman" section of linkgit:githooks[5].
+++
++Note that if you concurrently use multiple versions of Git, such
++as one version on the command line and another version in an IDE
++tool, that the definition of `core.fsmonitor` was extended to
++allow boolean values in addition to hook pathnames. Git versions
++2.35.1 and prior will not understand the boolean values and will
++consider the "true" or "false" values as hook pathnames to be
++invoked. Git versions 2.26 thru 2.35.1 default to hook protocol
++V2 and will fall back to no fsmonitor (full scan). Git versions
++prior to 2.26 default to hook protocol V1 and will silently
++assume there were no changes to report (no scan), so status
++commands may report incomplete results. For this reason, it is
++best to upgrade all of your Git versions before using the built-in
++file system monitor.
core.fsmonitorHookVersion::
- Sets the version of hook that is to be used when calling fsmonitor.
6: bdd7334da31 ! 6: 9ce938c69b5 fsmonitor--daemon: add a built-in fsmonitor daemon
@@ Commit message
Co-authored-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## .gitignore ##
@@
7: 9f00ada3dd3 ! 7: 3bd7e6a051e fsmonitor--daemon: implement 'stop' and 'status' commands
@@ Commit message
system.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/fsmonitor--daemon.c ##
@@
8: d6819bdad66 ! 8: 92d3e54da7f compat/fsmonitor/fsm-listen-win32: stub in backend for Windows
@@ Commit message
Stub in empty filesystem listener backend for fsmonitor--daemon on Windows.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## Makefile ##
@@ Makefile: all::
9: 7de3d01cccc ! 9: f1219221270 compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin
@@ Commit message
backend for Darwin (aka MacOS).
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## compat/fsmonitor/fsm-listen-darwin.c (new) ##
@@
10: 6fe5a2bc79e ! 10: a3f94627cf6 fsmonitor--daemon: implement 'run' command
@@ Commit message
backends are still just stubs.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/fsmonitor--daemon.c ##
@@
@@ builtin/fsmonitor--daemon.c
+#define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads"
+static int fsmonitor__ipc_threads = 8;
+
++#define FSMONITOR__ANNOUNCE_STARTUP "fsmonitor.announcestartup"
++static int fsmonitor__announce_startup = 0;
++
+static int fsmonitor_config(const char *var, const char *value, void *cb)
+{
+ if (!strcmp(var, FSMONITOR__IPC_THREADS)) {
@@ builtin/fsmonitor--daemon.c
+ return 0;
+ }
+
++ if (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {
++ int is_bool;
++ int i = git_config_bool_or_int(var, value, &is_bool);
++ if (i < 0)
++ return error(_("value of '%s' not bool or int: %d"),
++ var, i);
++ fsmonitor__announce_startup = i;
++ return 0;
++ }
++
+ return git_default_config(var, value, cb);
+}
+
@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
+ die("fsmonitor--daemon is already running '%s'",
+ the_repository->worktree);
+
-+ printf(_("running fsmonitor-daemon in '%s'\n"),
-+ the_repository->worktree);
-+ fflush(stdout);
++ if (fsmonitor__announce_startup) {
++ fprintf(stderr, _("running fsmonitor-daemon in '%s'\n"),
++ the_repository->worktree);
++ fflush(stderr);
++ }
+
+ return !!fsmonitor_run_daemon();
+}
11: 69fc0998286 ! 11: 8de40b0fe8b fsmonitor--daemon: implement 'start' command
@@ Commit message
because they don't have a Win32 console attached to them.)
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/fsmonitor--daemon.c ##
@@
@@ builtin/fsmonitor--daemon.c: static const char * const builtin_fsmonitor__daemon
+#define FSMONITOR__START_TIMEOUT "fsmonitor.starttimeout"
+static int fsmonitor__start_timeout_sec = 60;
+
- static int fsmonitor_config(const char *var, const char *value, void *cb)
- {
- if (!strcmp(var, FSMONITOR__IPC_THREADS)) {
+ #define FSMONITOR__ANNOUNCE_STARTUP "fsmonitor.announcestartup"
+ static int fsmonitor__announce_startup = 0;
+
@@ builtin/fsmonitor--daemon.c: static int fsmonitor_config(const char *var, const char *value, void *cb)
return 0;
}
@@ builtin/fsmonitor--daemon.c: static int fsmonitor_config(const char *var, const
+ return 0;
+ }
+
- return git_default_config(var, value, cb);
- }
-
+ if (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {
+ int is_bool;
+ int i = git_config_bool_or_int(var, value, &is_bool);
@@ builtin/fsmonitor--daemon.c: done:
return err;
}
-static int try_to_run_foreground_daemon(void)
-+static int try_to_run_foreground_daemon(int free_console)
++static int try_to_run_foreground_daemon(int detach_console)
{
/*
* Technically, we don't need to probe for an existing daemon
@@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)
- the_repository->worktree);
- fflush(stdout);
+ fflush(stderr);
+ }
+#ifdef GIT_WINDOWS_NATIVE
-+ if (free_console)
++ if (detach_console)
+ FreeConsole();
+#endif
+
@@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)
+ die("fsmonitor--daemon is already running '%s'",
+ the_repository->worktree);
+
-+ printf(_("starting fsmonitor-daemon in '%s'\n"),
-+ the_repository->worktree);
-+ fflush(stdout);
++ if (fsmonitor__announce_startup) {
++ fprintf(stderr, _("starting fsmonitor-daemon in '%s'\n"),
++ the_repository->worktree);
++ fflush(stderr);
++ }
+
+ cp.git_cmd = 1;
+
+ strvec_push(&cp.args, "fsmonitor--daemon");
+ strvec_push(&cp.args, "run");
-+ strvec_push(&cp.args, "--free-console");
++ strvec_push(&cp.args, "--detach");
+ strvec_pushf(&cp.args, "--ipc-threads=%d", fsmonitor__ipc_threads);
+
+ cp.no_stdin = 1;
@@ builtin/fsmonitor--daemon.c: static int try_to_run_foreground_daemon(void)
int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
{
const char *subcmd;
-+ int free_console = 0;
++ int detach_console = 0;
struct option options[] = {
-+ OPT_BOOL(0, "free-console", &free_console, N_("free console")),
++ OPT_BOOL(0, "detach", &detach_console, N_("detach from console")),
OPT_INTEGER(0, "ipc-threads",
&fsmonitor__ipc_threads,
N_("use <n> ipc worker threads")),
+ OPT_INTEGER(0, "start-timeout",
+ &fsmonitor__start_timeout_sec,
-+ N_("Max seconds to wait for background daemon startup")),
++ N_("max seconds to wait for background daemon startup")),
+
OPT_END()
};
@@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar
+
if (!strcmp(subcmd, "run"))
- return !!try_to_run_foreground_daemon();
-+ return !!try_to_run_foreground_daemon(free_console);
++ return !!try_to_run_foreground_daemon(detach_console);
if (!strcmp(subcmd, "stop"))
return !!do_as_client__send_stop();
12: 21c099c5197 ! 12: e5d419b1ea0 fsmonitor--daemon: add pathname classification
@@ Commit message
When we register for filesystem notifications on a directory,
we get events for everything (recursively) in the directory.
We want to report to clients changes to tracked and untracked
- paths within the working directory. We do not want to report
- changes within the .git directory, for example.
+ paths within the working directory proper. We do not want to
+ report changes within the .git directory, for example.
This classification will be used in a later commit by the
different backends to classify paths as events are received.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/fsmonitor--daemon.c ##
@@ builtin/fsmonitor--daemon.c: static int handle_client(void *data,
@@ fsmonitor--daemon.h: struct fsmonitor_daemon_state {
+ * that to decide whether clients are told about them. (And to watch
+ * for file system synchronization events.)
+ *
++ * The daemon only collects and reports on the set of modified paths
++ * within the working directory (proper).
++ *
+ * The client should only care about paths within the working
+ * directory proper (inside the working directory and not ".git" nor
+ * inside of ".git/"). That is, the client has read the index and is
+ * asking for a list of any paths in the working directory that have
+ * been modified since the last token. The client does not care about
-+ * file system changes within the .git directory (such as new loose
++ * file system changes within the ".git/" directory (such as new loose
+ * objects or packfiles). So the client will only receive paths that
+ * are classified as IS_WORKDIR_PATH.
+ *
++ * Note that ".git" is usually a directory and is therefore inside
++ * the cone of the FS watch that we have on the working directory root,
++ * so we will also get FS events for disk activity on and within ".git/"
++ * that we need to respond to or filter from the client.
++ *
++ * But Git also allows ".git" to be a *file* that points to a GITDIR
++ * outside of the working directory. When this happens, we need to
++ * create FS watches on both the working directory root *and* on the
++ * (external) GITDIR root. (The latter is required because we put
++ * cookie files inside it and use them to sync with the FS event
++ * stream.)
++ *
++ * Note that in the context of this discussion, I'm using "GITDIR"
++ * to only mean an external GITDIR referenced by a ".git" file.
++ *
++ * The platform FS event backends will receive watch-specific
++ * relative paths (except for those OS's that always emit absolute
++ * paths). We use the following enum and routines to classify each
++ * path so that we know how to handle it. There is a slight asymmetry
++ * here because ".git/" is inside the working directory and the
++ * (external) GITDIR is not, and therefore how we handle events may
++ * vary slightly, so I have different enums for "IS...DOT_GIT..." and
++ * "IS...GITDIR...".
++ *
+ * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the
-+ * exact ".git" directory or GITDIR. If the daemon receives a delete
-+ * event for either of these directories, it will automatically
-+ * shutdown, for example.
++ * exact ".git" file/directory or GITDIR directory. If the daemon
++ * receives a delete event for either of these paths, it will
++ * automatically shutdown, for example.
+ *
+ * Note that the daemon DOES NOT explicitly watch nor special case the
-+ * ".git/index" file. The daemon does not read the index and does not
-+ * have any internal index-relative state. The daemon only collects
-+ * the set of modified paths within the working directory.
++ * index. The daemon does not read the index nor have any internal
++ * index-relative state, so there are no "IS...INDEX..." enum values.
+ */
+enum fsmonitor_path_type {
+ IS_WORKDIR_PATH = 0,
13: 72979c35ceb ! 13: c2f3668a866 fsmonitor--daemon: define token-ids
@@ Commit message
overall token naming scheme.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/fsmonitor--daemon.c ##
@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
14: 8d1db644409 ! 14: bd492d13fb1 fsmonitor--daemon: create token-based changed path cache
@@ Commit message
paths within these batches for efficient presentation to clients.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/fsmonitor--daemon.c ##
@@ builtin/fsmonitor--daemon.c: struct fsmonitor_token_data {
@@ builtin/fsmonitor--daemon.c: static struct fsmonitor_token_data *fsmonitor_new_t
+ * We should create a new token and start fresh (as if we just
+ * booted up).
+ *
-+ * If there are no concurrent threads readering the current token data
++ * If there are no concurrent threads reading the current token data
+ * series, we can free it now. Otherwise, let the last reader free
+ * it.
+ *
15: 98c5adf8ca0 ! 15: 6b5b3d08227 compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows
@@ Commit message
into batches and publish.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## compat/fsmonitor/fsm-listen-win32.c ##
@@
16: a3b881315fa ! 16: cdef9730b3f compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent
@@ Metadata
Author: Jeff Hostetler <jeffhost@microsoft.com>
## Commit message ##
- compat/fsmonitor/fsm-listen-darwin: add macos header files for FSEvent
+ compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent
Include MacOS system declarations to allow us to use FSEvent and
- CoreFoundation APIs. We need GCC and clang versions because of
- compiler and header file conflicts.
+ CoreFoundation APIs. We need different versions of the declarations
+ for GCC vs. clang because of compiler and header file conflicts.
While it is quite possible to #include Apple's CoreServices.h when
compiling C source code with clang, trying to build it with GCC
@@ Commit message
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## compat/fsmonitor/fsm-listen-darwin.c ##
@@
-+#if defined(__GNUC__)
++#ifndef __clang__
+/*
+ * It is possible to #include CoreFoundation/CoreFoundation.h when compiling
+ * with clang, but not with GCC as of time of writing.
17: 162e357db72 ! 17: aabfebd3a41 compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS
@@ Commit message
Co-authored-by: Kevin Willford <Kevin.Willford@microsoft.com>
Co-authored-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## compat/fsmonitor/fsm-listen-darwin.c ##
-@@
--#if defined(__GNUC__)
-+#ifndef __clang__
- /*
- * It is possible to #include CoreFoundation/CoreFoundation.h when compiling
- * with clang, but not with GCC as of time of writing.
@@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef stream);
#include "cache.h"
#include "fsmonitor.h"
@@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef
+}
+
+/*
-+ * NEEDSWORK: Investigate the proper value for the `latency` argument
-+ * in the call to `FSEventStreamCreate()`. I'm not sure that this
-+ * needs to be a config setting or just something that we tune after
-+ * some testing.
++ * In the call to `FSEventStreamCreate()` to setup our watch, the
++ * `latency` argument determines the frequency of calls to our callback
++ * with new FS events. Too slow and events get dropped; too fast and
++ * we burn CPU unnecessarily. Since it is rather obscure, I don't
++ * think this needs to be a config setting. I've done extensive
++ * testing on my systems and chosen the value below. It gives good
++ * results and I've not seen any dropped events.
+ *
+ * With a latency of 0.1, I was seeing lots of dropped events during
+ * the "touch 100000" files test within t/perf/p7519, but with a
-+ * latency of 0.001 I did not see any dropped events. So the
-+ * "correct" value may be somewhere in between.
++ * latency of 0.001 I did not see any dropped events. So I'm going
++ * to assume that this is the "correct" value.
+ *
+ * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate
+ */
18: 3de1c43beaf ! 18: 0896e72e2b8 fsmonitor--daemon: implement handle_client callback
@@ Commit message
relative to the provided token.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/fsmonitor--daemon.c ##
@@
19: 3517c4a3c13 ! 19: bc240a9e665 help: include fsmonitor--daemon feature flag in version info
@@ Commit message
This can be used by test scripts for prereq testing. Granted, tests
could just try `git fsmonitor--daemon status` and look for a 128 exit
- code or grep for a "not supported" message on stderr, but this is
- rather obscure.
+ code or grep for a "not supported" message on stderr, but these
+ methods are rather obscure.
The main advantage is that the feature message will automatically
appear in bug reports and other support requests.
+ This concept was also used during the development of Scalar for
+ similar reasons.
+
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## help.c ##
@@
20: 4ffc2ddf516 ! 20: 2b563c240eb t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon
@@ Commit message
Create an IPC client to send query and flush commands to the daemon.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## Makefile ##
@@ Makefile: TEST_BUILTINS_OBJS += test-dump-split-index.o
21: d93310a7c64 ! 21: db8809da089 t7527: create test for fsmonitor--daemon
@@ Commit message
t7527: create test for fsmonitor--daemon
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## t/t7527-builtin-fsmonitor.sh (new) ##
@@
22: 27e47108908 ! 22: 15bd5aaff36 t/perf: avoid copying builtin fsmonitor files into test repo
@@ Commit message
.git/fsmonitor--daemon*, so it is simple to exclude them.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## t/perf/perf-lib.sh ##
@@ t/perf/perf-lib.sh: test_perf_copy_repo_contents () {
23: 6cba1d950b0 ! 23: 4cb97918d20 t/helper/test-chmtime: skip directories on Windows
@@ Commit message
an empty directory on disk.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## t/helper/test-chmtime.c ##
@@ t/helper/test-chmtime.c: int cmd__chmtime(int argc, const char **argv)
24: fcf843a0d42 ! 24: a70748b4640 t/perf/p7519: speed up test on Windows
@@ Commit message
on Windows because of process creation overhead.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## t/perf/p7519-fsmonitor.sh ##
@@ t/perf/p7519-fsmonitor.sh: then
@@ t/perf/p7519-fsmonitor.sh: test_expect_success "one time repo setup" '
- for i in $(test_seq 1 100); do touch 100_files/$i || return 1; done &&
- for i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done &&
- for i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done &&
-+ touch_files 1 &&
++ : 1_file directory should be left empty &&
+ touch_files 10 &&
+ touch_files 100 &&
+ touch_files 1000 &&
25: 198f47bda5a ! 25: b0c9c9c7a40 t/perf/p7519: add fsmonitor--daemon test cases
@@ Commit message
the "Simple IPC" interface.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## t/perf/p7519-fsmonitor.sh ##
@@ t/perf/p7519-fsmonitor.sh: test_perf_w_drop_caches () {
26: 19993c130d2 ! 26: 64a5b741670 fsmonitor--daemon: periodically truncate list of modified files
@@ Commit message
they are considered obsolete.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/fsmonitor--daemon.c ##
@@ builtin/fsmonitor--daemon.c: static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest,
27: f47a763dc26 ! 27: 5b06eb5d0e6 fsmonitor--daemon: use a cookie file to sync with file system
@@ Commit message
of `FSeventStreamCreate()`. The kernel only signals every `latency`
seconds, but does not guarantee that the kernel queue is completely
drained, so we may have to wait more than one interval. If we
- increase the frequency, the system is more likely to drop events.
+ increase the latency, the system is more likely to drop events.
We avoid these issues by having each client thread create a unique
cookie file and then wait until it is seen in the event stream.
Co-authored-by: Kevin Willford <Kevin.Willford@microsoft.com>
Co-authored-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/fsmonitor--daemon.c ##
@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
@@ builtin/fsmonitor--daemon.c: static int do_as_client__status(void)
+ unlink(cookie_pathname.buf);
+
+ /*
-+ * NEEDSWORK: This is an infinite wait (well, unless another
++ * Technically, this is an infinite wait (well, unless another
+ * thread sends us an abort). I'd like to change this to
+ * use `pthread_cond_timedwait()` and return an error/timeout
-+ * and let the caller do the trivial response thing.
++ * and let the caller do the trivial response thing, but we
++ * don't have that routine in our thread-utils.
++ *
++ * After extensive beta testing I'm not really worried about
++ * this. Also note that the above open() and unlink() calls
++ * will cause at least two FS events on that path, so the odds
++ * of getting stuck are pretty slim.
+ */
+ while (cookie->result == FCIR_INIT)
+ pthread_cond_wait(&state->cookies_cond,
@@ builtin/fsmonitor--daemon.c: static void fsmonitor_free_token_data(struct fsmoni
+ * [2] Some of those lost events may have been for cookie files. We
+ * should assume the worst and abort them rather letting them starve.
+ *
- * If there are no concurrent threads readering the current token data
+ * If there are no concurrent threads reading the current token data
* series, we can free it now. Otherwise, let the last reader free
* it.
@@ builtin/fsmonitor--daemon.c: static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state)
@@ builtin/fsmonitor--daemon.c: done:
strbuf_release(&state.path_worktree_watch);
strbuf_release(&state.path_gitdir_watch);
+ strbuf_release(&state.path_cookie_prefix);
-+
-+ /*
-+ * NEEDSWORK: Consider "rm -rf <gitdir>/<fsmonitor-dir>"
-+ */
return err;
}
28: aec44a21afd ! 28: 1fd5439de03 fsmonitor: force update index after large responses
@@ Commit message
unnecessarily spend time actually scanning them during each command.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## fsmonitor.c ##
@@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
@@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate,
+ * will be updated automatically the first time the user touches
+ * a tracked file and causes a command like `git status` to
+ * update an mtime to be updated and/or set a flag bit.
-+ *
-+ * NEEDSWORK: Does this need to be a config value?
+ */
+static int fsmonitor_force_update_threshold = 100;
+
29: d6039987df8 ! 29: 6fc1430285f t7527: test status with untracked-cache and fsmonitor--daemon
@@ Commit message
identical.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## t/t7527-builtin-fsmonitor.sh ##
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '
30: 5117fbdfc63 ! 30: b915b95cc2f update-index: convert fsmonitor warnings to advise
@@ Commit message
update-index: convert fsmonitor warnings to advise
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
+ Signed-off-by: Junio C Hamano <gitster@pobox.com>
## builtin/update-index.c ##
@@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const char *prefix)-- gitgitgadget