# [PATCH v3] 0/4 fsmonitor fixes 10 messages from 2017-10-27 to 2017-10-31. Participants: Alex Vandiver, Johannes Schindelin, Ben Peart, Junio C Hamano. Thread: https://gitlist.dev/t/47048 ## Alex Vandiver, 2017-10-27 23:26 Subject: [PATCH v3] 0/4 fsmonitor fixes Message-ID: <20171027232637.30395-1-alexmv@dropbox.com> URL: https://gitlist.dev/e/20171027232637.30395-1-alexmv%40dropbox.com ``` Updates since v2: - Fix tab which crept into 1/4 - Fixed the benchmarking code in the commit message in 2/4 to just always load JSON::XS -- the previous version was the version where I'd broken that to force loading of JSON::PP. - Remove the --no-pretty from the t/ version of query-watchman in 2/4; I don't know how I messed up diff'ing the file previously, but if there are already differences, it makes sense to let them slide. ``` ## Alex Vandiver, 2017-10-27 23:26 Subject: [PATCH v3 1/4] fsmonitor: Set the PWD to the top of the working tree Message-ID: <4b488da5e0710e9699f92d2dabe5e3352f3eb394.1509146542.git.alexmv@dropbox.com> URL: https://gitlist.dev/e/4b488da5e0710e9699f92d2dabe5e3352f3eb394.1509146542.git.alexmv%40dropbox.com In-Reply-To: <20171027232637.30395-1-alexmv@dropbox.com> ``` The fsmonitor command inherits the PWD of its caller, which may be anywhere in the working copy. This makes is difficult for the fsmonitor command to operate on the whole repository. Specifically, for the watchman integration, this causes each subdirectory to get its own watch entry. Set the CWD to the top of the working directory, for consistency. Signed-off-by: Alex Vandiver --- fsmonitor.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fsmonitor.c b/fsmonitor.c index 7c1540c05..4ea44dcc6 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -121,6 +121,7 @@ static int query_fsmonitor(int version, uint64_t last_update, struct strbuf *que argv[3] = NULL; cp.argv = argv; cp.use_shell = 1; + cp.dir = get_git_work_tree(); return capture_command(&cp, query_result, 1024); } -- 2.15.0.rc1.413.g76aedb451 ``` ## Alex Vandiver, 2017-10-27 23:26 Subject: [PATCH v3 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman Message-ID: URL: https://gitlist.dev/e/ff7745089999ff3bb2c014d2d4f1659a9de4e859.1509146542.git.alexmv%40dropbox.com In-Reply-To: <20171027232637.30395-1-alexmv@dropbox.com> ``` This provides modest performance savings. Benchmarking with the following program, with and without `--no-pretty`, we find savings of 23% (0.316s -> 0.242s) in the git repository, and savings of 8% (5.24s -> 4.86s) on a large repository with 580k files in the working copy. #!/usr/bin/perl use strict; use warnings; use IPC::Open2; use JSON::XS; my $pid = open2(\*CHLD_OUT, \*CHLD_IN, "watchman -j @ARGV") or die "open2() failed: $!\n" . "Falling back to scanning...\n"; my $query = qq|["query", "$ENV{PWD}", {}]|; print CHLD_IN $query; close CHLD_IN; my $response = do {local $/; }; JSON::XS->new->utf8->decode($response); Signed-off-by: Alex Vandiver --- templates/hooks--fsmonitor-watchman.sample | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample index 9eba8a740..9a082f278 100755 --- a/templates/hooks--fsmonitor-watchman.sample +++ b/templates/hooks--fsmonitor-watchman.sample @@ -49,7 +49,7 @@ launch_watchman(); sub launch_watchman { - my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j') + my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty') or die "open2() failed: $!\n" . "Falling back to scanning...\n"; -- 2.15.0.rc1.413.g76aedb451 ``` ## Alex Vandiver, 2017-10-27 23:26 Subject: [PATCH v3 3/4] fsmonitor: Document GIT_TRACE_FSMONITOR Message-ID: URL: https://gitlist.dev/e/cdf4e3f3310d49751cea5317323e11597a66c966.1509146542.git.alexmv%40dropbox.com In-Reply-To: <20171027232637.30395-1-alexmv@dropbox.com> ``` Signed-off-by: Alex Vandiver --- Documentation/git.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Documentation/git.txt b/Documentation/git.txt index 1fca63634..720db196e 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -594,6 +594,10 @@ into it. Unsetting the variable, or setting it to empty, "0" or "false" (case insensitive) disables trace messages. +`GIT_TRACE_FSMONITOR`:: + Enables trace messages for the filesystem monitor extension. + See `GIT_TRACE` for available trace output options. + `GIT_TRACE_PACK_ACCESS`:: Enables trace messages for all accesses to any packs. For each access, the pack file name and an offset in the pack is -- 2.15.0.rc1.413.g76aedb451 ``` ## Alex Vandiver, 2017-10-27 23:26 Subject: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged Message-ID: <5cb81a33c31ffa585861f0d3f5a7c7eef5bd8fe0.1509146542.git.alexmv@dropbox.com> URL: https://gitlist.dev/e/5cb81a33c31ffa585861f0d3f5a7c7eef5bd8fe0.1509146542.git.alexmv%40dropbox.com In-Reply-To: <20171027232637.30395-1-alexmv@dropbox.com> ``` If the fsmonitor extension is used in conjunction with the split index extension, the set of entries in the index when it is first loaded is only a subset of the real index. This leads to only the non-"base" index being marked as CE_FSMONITOR_VALID. Delay the expansion of the ewah bitmap until after tweak_split_index has been called to merge in the base index as well. The new fsmonitor_dirty is kept from being leaked by dint of being cleaned up in post_read_index_from, which is guaranteed to be called after do_read_index in read_index_from. Signed-off-by: Alex Vandiver --- cache.h | 1 + fsmonitor.c | 39 ++++++++++++++++++++++++--------------- 2 files changed, 25 insertions(+), 15 deletions(-) diff --git a/cache.h b/cache.h index 25adcf681..0a4f43ec2 100644 --- a/cache.h +++ b/cache.h @@ -348,6 +348,7 @@ struct index_state { unsigned char sha1[20]; struct untracked_cache *untracked; uint64_t fsmonitor_last_update; + struct ewah_bitmap *fsmonitor_dirty; }; extern struct index_state the_index; diff --git a/fsmonitor.c b/fsmonitor.c index 4ea44dcc6..417759224 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data, ewah_free(fsmonitor_dirty); return error("failed to parse ewah bitmap reading fsmonitor index extension"); } - - if (git_config_get_fsmonitor()) { - /* Mark all entries valid */ - for (i = 0; i < istate->cache_nr; i++) - istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID; - - /* Mark all previously saved entries as dirty */ - ewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate); - - /* Now mark the untracked cache for fsmonitor usage */ - if (istate->untracked) - istate->untracked->use_fsmonitor = 1; - } - ewah_free(fsmonitor_dirty); + istate->fsmonitor_dirty = fsmonitor_dirty; trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful"); return 0; @@ -239,7 +226,29 @@ void remove_fsmonitor(struct index_state *istate) void tweak_fsmonitor(struct index_state *istate) { - switch (git_config_get_fsmonitor()) { + int i; + int fsmonitor_enabled = git_config_get_fsmonitor(); + + if (istate->fsmonitor_dirty) { + if (fsmonitor_enabled) { + /* Mark all entries valid */ + for (i = 0; i < istate->cache_nr; i++) { + istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID; + } + + /* Mark all previously saved entries as dirty */ + ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate); + + /* Now mark the untracked cache for fsmonitor usage */ + if (istate->untracked) + istate->untracked->use_fsmonitor = 1; + } + + ewah_free(istate->fsmonitor_dirty); + istate->fsmonitor_dirty = NULL; + } + + switch (fsmonitor_enabled) { case -1: /* keep: do nothing */ break; case 0: /* false */ -- 2.15.0.rc1.413.g76aedb451 ``` ## Johannes Schindelin, 2017-10-29 12:34 Subject: Re: [PATCH v3] 0/4 fsmonitor fixes Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.21.1.1710291334290.6482%40virtualbox In-Reply-To: <20171027232637.30395-1-alexmv@dropbox.com> ``` Hi Alex, On Fri, 27 Oct 2017, Alex Vandiver wrote: > Updates since v2: > > - Fix tab which crept into 1/4 > > - Fixed the benchmarking code in the commit message in 2/4 to just > always load JSON::XS -- the previous version was the version where > I'd broken that to force loading of JSON::PP. > > - Remove the --no-pretty from the t/ version of query-watchman in > 2/4; I don't know how I messed up diff'ing the file previously, but > if there are already differences, it makes sense to let them slide. Sounds good! Dscho ``` ## Ben Peart, 2017-10-30 12:38 Subject: Re: [PATCH v3] 0/4 fsmonitor fixes Message-ID: <7fc39ba0-d1a3-7e96-8585-e82c54dc3e7c@gmail.com> URL: https://gitlist.dev/e/7fc39ba0-d1a3-7e96-8585-e82c54dc3e7c%40gmail.com In-Reply-To: <20171027232637.30395-1-alexmv@dropbox.com> ``` On 10/27/2017 7:26 PM, Alex Vandiver wrote: > Updates since v2: > > - Fix tab which crept into 1/4 > > - Fixed the benchmarking code in the commit message in 2/4 to just > always load JSON::XS -- the previous version was the version where > I'd broken that to force loading of JSON::PP. > > - Remove the --no-pretty from the t/ version of query-watchman in > 2/4; I don't know how I messed up diff'ing the file previously, but > if there are already differences, it makes sense to let them slide. > Thanks Alex, nice improvements. This looks good to go. ``` ## Junio C Hamano, 2017-10-31 05:24 Subject: Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged Message-ID: URL: https://gitlist.dev/e/xmqqa80728lo.fsf%40gitster.mtv.corp.google.com In-Reply-To: <5cb81a33c31ffa585861f0d3f5a7c7eef5bd8fe0.1509146542.git.alexmv@dropbox.com> ``` Alex Vandiver writes: > diff --git a/fsmonitor.c b/fsmonitor.c > index 4ea44dcc6..417759224 100644 > --- a/fsmonitor.c > +++ b/fsmonitor.c > @@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data, > ewah_free(fsmonitor_dirty); > return error("failed to parse ewah bitmap reading fsmonitor index extension"); > } > - > - if (git_config_get_fsmonitor()) { > - /* Mark all entries valid */ > - for (i = 0; i < istate->cache_nr; i++) > - istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID; > - > - /* Mark all previously saved entries as dirty */ > - ewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate); > - > - /* Now mark the untracked cache for fsmonitor usage */ > - if (istate->untracked) > - istate->untracked->use_fsmonitor = 1; > - } > - ewah_free(fsmonitor_dirty); > + istate->fsmonitor_dirty = fsmonitor_dirty; This makes local variable "int i;" in this function unused and gets compiler warning. ``` ## Johannes Schindelin, 2017-10-31 17:31 Subject: Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.21.1.1710311830330.6482%40virtualbox In-Reply-To: ``` Hi, On Tue, 31 Oct 2017, Junio C Hamano wrote: > Alex Vandiver writes: > > > diff --git a/fsmonitor.c b/fsmonitor.c > > index 4ea44dcc6..417759224 100644 > > --- a/fsmonitor.c > > +++ b/fsmonitor.c > > @@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data, > > ewah_free(fsmonitor_dirty); > > return error("failed to parse ewah bitmap reading fsmonitor index extension"); > > } > > - > > - if (git_config_get_fsmonitor()) { > > - /* Mark all entries valid */ > > - for (i = 0; i < istate->cache_nr; i++) > > - istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID; > > - > > - /* Mark all previously saved entries as dirty */ > > - ewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate); > > - > > - /* Now mark the untracked cache for fsmonitor usage */ > > - if (istate->untracked) > > - istate->untracked->use_fsmonitor = 1; > > - } > > - ewah_free(fsmonitor_dirty); > > + istate->fsmonitor_dirty = fsmonitor_dirty; > > This makes local variable "int i;" in this function unused and gets > compiler warning. ... to which end we introduced the DEVELOPER flag to catch these: if you call make DEVELOPER=1 and compile with GCC or Clang, it will elevate such warnings to errors, and we highly encourage contributors to build their patched source code with said flag. Thanks, Johannes ``` ## Alex Vandiver, 2017-10-31 18:43 Subject: Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.10.1710311139560.5248%40alexmv-linux In-Reply-To: ``` On Tue, 31 Oct 2017, Junio C Hamano wrote: > This makes local variable "int i;" in this function unused and gets > compiler warning. Apologies for leaving that detritus -- I saw you added a 'SQUASH??' commit to deal with it, which LGTM. On Tue, 31 Oct 2017, Johannes Schindelin wrote: > ... to which end we introduced the DEVELOPER flag to catch these: if you > call > > make DEVELOPER=1 Aha! Thanks for the tip; I'll be sure to use that from now on. - Alex ```