{"thread":{"id":"47048","subject":"[PATCH v3] 0/4 fsmonitor fixes","startedAt":"2017-10-27T23:27:06Z","lastAt":"2017-10-31T18:43:26Z","messageCount":10,"participants":["Alex Vandiver","Johannes Schindelin","Ben Peart","Junio C Hamano"],"isPatch":true,"patchVersion":3,"patchTotal":null},"messages":[{"id":"331194","messageId":"20171027232637.30395-1-alexmv@dropbox.com","threadId":"47048","inReplyTo":null,"subject":"[PATCH v3] 0/4 fsmonitor fixes","fromName":"Alex Vandiver","fromEmail":"alexmv@dropbox.com","sentAt":"2017-10-27T23:26:33Z","receivedAt":"2017-10-27T23:27:06Z","isPatch":true,"sender":{"key":"alexmv@dropbox.com","avatar":null},"body":"Updates since v2:\n\n - Fix tab which crept into 1/4\n\n - Fixed the benchmarking code in the commit message in 2/4 to just\n   always load JSON::XS -- the previous version was the version where\n   I'd broken that to force loading of JSON::PP.\n\n - Remove the --no-pretty from the t/ version of query-watchman in\n   2/4; I don't know how I messed up diff'ing the file previously, but\n   if there are already differences, it makes sense to let them slide.\n\n"},{"id":"331195","messageId":"4b488da5e0710e9699f92d2dabe5e3352f3eb394.1509146542.git.alexmv@dropbox.com","threadId":"47048","inReplyTo":"20171027232637.30395-1-alexmv@dropbox.com","subject":"[PATCH v3 1/4] fsmonitor: Set the PWD to the top of the working tree","fromName":"Alex Vandiver","fromEmail":"alexmv@dropbox.com","sentAt":"2017-10-27T23:26:34Z","receivedAt":"2017-10-27T23:27:09Z","isPatch":true,"sender":{"key":"alexmv@dropbox.com","avatar":null},"body":"The fsmonitor command inherits the PWD of its caller, which may be\nanywhere in the working copy.  This makes is difficult for the\nfsmonitor command to operate on the whole repository.  Specifically,\nfor the watchman integration, this causes each subdirectory to get its\nown watch entry.\n\nSet the CWD to the top of the working directory, for consistency.\n\nSigned-off-by: Alex Vandiver <alexmv@dropbox.com>\n---\n fsmonitor.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 7c1540c05..4ea44dcc6 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -121,6 +121,7 @@ static int query_fsmonitor(int version, uint64_t last_update, struct strbuf *que\n \targv[3] = NULL;\n \tcp.argv = argv;\n \tcp.use_shell = 1;\n+\tcp.dir = get_git_work_tree();\n \n \treturn capture_command(&cp, query_result, 1024);\n }\n-- \n2.15.0.rc1.413.g76aedb451\n\n"},{"id":"331196","messageId":"ff7745089999ff3bb2c014d2d4f1659a9de4e859.1509146542.git.alexmv@dropbox.com","threadId":"47048","inReplyTo":"20171027232637.30395-1-alexmv@dropbox.com","subject":"[PATCH v3 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman","fromName":"Alex Vandiver","fromEmail":"alexmv@dropbox.com","sentAt":"2017-10-27T23:26:35Z","receivedAt":"2017-10-27T23:27:12Z","isPatch":true,"sender":{"key":"alexmv@dropbox.com","avatar":null},"body":"This provides modest performance savings.  Benchmarking with the\nfollowing program, with and without `--no-pretty`, we find savings of\n23% (0.316s -> 0.242s) in the git repository, and savings of 8% (5.24s\n-> 4.86s) on a large repository with 580k files in the working copy.\n\n    #!/usr/bin/perl\n\n    use strict;\n    use warnings;\n    use IPC::Open2;\n    use JSON::XS;\n\n    my $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, \"watchman -j @ARGV\")\n        or die \"open2() failed: $!\\n\" .\n        \"Falling back to scanning...\\n\";\n\n    my $query = qq|[\"query\", \"$ENV{PWD}\", {}]|;\n\n    print CHLD_IN $query;\n    close CHLD_IN;\n    my $response = do {local $/; <CHLD_OUT>};\n\n    JSON::XS->new->utf8->decode($response);\n\nSigned-off-by: Alex Vandiver <alexmv@dropbox.com>\n---\n templates/hooks--fsmonitor-watchman.sample | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample\nindex 9eba8a740..9a082f278 100755\n--- a/templates/hooks--fsmonitor-watchman.sample\n+++ b/templates/hooks--fsmonitor-watchman.sample\n@@ -49,7 +49,7 @@ launch_watchman();\n \n sub launch_watchman {\n \n-\tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j')\n+\tmy $pid = open2(\\*CHLD_OUT, \\*CHLD_IN, 'watchman -j --no-pretty')\n \t    or die \"open2() failed: $!\\n\" .\n \t    \"Falling back to scanning...\\n\";\n \n-- \n2.15.0.rc1.413.g76aedb451\n\n"},{"id":"331197","messageId":"cdf4e3f3310d49751cea5317323e11597a66c966.1509146542.git.alexmv@dropbox.com","threadId":"47048","inReplyTo":"20171027232637.30395-1-alexmv@dropbox.com","subject":"[PATCH v3 3/4] fsmonitor: Document GIT_TRACE_FSMONITOR","fromName":"Alex Vandiver","fromEmail":"alexmv@dropbox.com","sentAt":"2017-10-27T23:26:36Z","receivedAt":"2017-10-27T23:27:16Z","isPatch":true,"sender":{"key":"alexmv@dropbox.com","avatar":null},"body":"Signed-off-by: Alex Vandiver <alexmv@dropbox.com>\n---\n Documentation/git.txt | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 1fca63634..720db196e 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -594,6 +594,10 @@ into it.\n Unsetting the variable, or setting it to empty, \"0\" or\n \"false\" (case insensitive) disables trace messages.\n \n+`GIT_TRACE_FSMONITOR`::\n+\tEnables trace messages for the filesystem monitor extension.\n+\tSee `GIT_TRACE` for available trace output options.\n+\n `GIT_TRACE_PACK_ACCESS`::\n \tEnables trace messages for all accesses to any packs. For each\n \taccess, the pack file name and an offset in the pack is\n-- \n2.15.0.rc1.413.g76aedb451\n\n"},{"id":"331198","messageId":"5cb81a33c31ffa585861f0d3f5a7c7eef5bd8fe0.1509146542.git.alexmv@dropbox.com","threadId":"47048","inReplyTo":"20171027232637.30395-1-alexmv@dropbox.com","subject":"[PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged","fromName":"Alex Vandiver","fromEmail":"alexmv@dropbox.com","sentAt":"2017-10-27T23:26:37Z","receivedAt":"2017-10-27T23:27:18Z","isPatch":true,"sender":{"key":"alexmv@dropbox.com","avatar":null},"body":"If the fsmonitor extension is used in conjunction with the split index\nextension, the set of entries in the index when it is first loaded is\nonly a subset of the real index.  This leads to only the non-\"base\"\nindex being marked as CE_FSMONITOR_VALID.\n\nDelay the expansion of the ewah bitmap until after tweak_split_index\nhas been called to merge in the base index as well.\n\nThe new fsmonitor_dirty is kept from being leaked by dint of being\ncleaned up in post_read_index_from, which is guaranteed to be called\nafter do_read_index in read_index_from.\n\nSigned-off-by: Alex Vandiver <alexmv@dropbox.com>\n---\n cache.h     |  1 +\n fsmonitor.c | 39 ++++++++++++++++++++++++---------------\n 2 files changed, 25 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 25adcf681..0a4f43ec2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -348,6 +348,7 @@ struct index_state {\n \tunsigned char sha1[20];\n \tstruct untracked_cache *untracked;\n \tuint64_t fsmonitor_last_update;\n+\tstruct ewah_bitmap *fsmonitor_dirty;\n };\n \n extern struct index_state the_index;\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 4ea44dcc6..417759224 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,\n \t\tewah_free(fsmonitor_dirty);\n \t\treturn error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n \t}\n-\n-\tif (git_config_get_fsmonitor()) {\n-\t\t/* Mark all entries valid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++)\n-\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n-\n-\t\t/* Mark all previously saved entries as dirty */\n-\t\tewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n-\n-\t\t/* Now mark the untracked cache for fsmonitor usage */\n-\t\tif (istate->untracked)\n-\t\t\tistate->untracked->use_fsmonitor = 1;\n-\t}\n-\tewah_free(fsmonitor_dirty);\n+\tistate->fsmonitor_dirty = fsmonitor_dirty;\n \n \ttrace_printf_key(&trace_fsmonitor, \"read fsmonitor extension successful\");\n \treturn 0;\n@@ -239,7 +226,29 @@ void remove_fsmonitor(struct index_state *istate)\n \n void tweak_fsmonitor(struct index_state *istate)\n {\n-\tswitch (git_config_get_fsmonitor()) {\n+\tint i;\n+\tint fsmonitor_enabled = git_config_get_fsmonitor();\n+\n+\tif (istate->fsmonitor_dirty) {\n+\t\tif (fsmonitor_enabled) {\n+\t\t\t/* Mark all entries valid */\n+\t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n+\t\t\t}\n+\n+\t\t\t/* Mark all previously saved entries as dirty */\n+\t\t\tewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n+\n+\t\t\t/* Now mark the untracked cache for fsmonitor usage */\n+\t\t\tif (istate->untracked)\n+\t\t\t\tistate->untracked->use_fsmonitor = 1;\n+\t\t}\n+\n+\t\tewah_free(istate->fsmonitor_dirty);\n+\t\tistate->fsmonitor_dirty = NULL;\n+\t}\n+\n+\tswitch (fsmonitor_enabled) {\n \tcase -1: /* keep: do nothing */\n \t\tbreak;\n \tcase 0: /* false */\n-- \n2.15.0.rc1.413.g76aedb451\n\n"},{"id":"331268","messageId":"alpine.DEB.2.21.1.1710291334290.6482@virtualbox","threadId":"47048","inReplyTo":"20171027232637.30395-1-alexmv@dropbox.com","subject":"Re: [PATCH v3] 0/4 fsmonitor fixes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-10-29T12:34:41Z","receivedAt":"2017-10-29T12:35:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Alex,\n\nOn Fri, 27 Oct 2017, Alex Vandiver wrote:\n\n> Updates since v2:\n> \n>  - Fix tab which crept into 1/4\n> \n>  - Fixed the benchmarking code in the commit message in 2/4 to just\n>    always load JSON::XS -- the previous version was the version where\n>    I'd broken that to force loading of JSON::PP.\n> \n>  - Remove the --no-pretty from the t/ version of query-watchman in\n>    2/4; I don't know how I messed up diff'ing the file previously, but\n>    if there are already differences, it makes sense to let them slide.\n\nSounds good!\nDscho\n"},{"id":"331349","messageId":"7fc39ba0-d1a3-7e96-8585-e82c54dc3e7c@gmail.com","threadId":"47048","inReplyTo":"20171027232637.30395-1-alexmv@dropbox.com","subject":"Re: [PATCH v3] 0/4 fsmonitor fixes","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-10-30T12:38:56Z","receivedAt":"2017-10-30T12:39:06Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 10/27/2017 7:26 PM, Alex Vandiver wrote:\n> Updates since v2:\n> \n>   - Fix tab which crept into 1/4\n> \n>   - Fixed the benchmarking code in the commit message in 2/4 to just\n>     always load JSON::XS -- the previous version was the version where\n>     I'd broken that to force loading of JSON::PP.\n> \n>   - Remove the --no-pretty from the t/ version of query-watchman in\n>     2/4; I don't know how I messed up diff'ing the file previously, but\n>     if there are already differences, it makes sense to let them slide.\n> \n\nThanks Alex, nice improvements.  This looks good to go.\n"},{"id":"331453","messageId":"xmqqa80728lo.fsf@gitster.mtv.corp.google.com","threadId":"47048","inReplyTo":"5cb81a33c31ffa585861f0d3f5a7c7eef5bd8fe0.1509146542.git.alexmv@dropbox.com","subject":"Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-31T05:24:03Z","receivedAt":"2017-10-31T05:24:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Alex Vandiver <alexmv@dropbox.com> writes:\n\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 4ea44dcc6..417759224 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,\n>  \t\tewah_free(fsmonitor_dirty);\n>  \t\treturn error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n>  \t}\n> -\n> -\tif (git_config_get_fsmonitor()) {\n> -\t\t/* Mark all entries valid */\n> -\t\tfor (i = 0; i < istate->cache_nr; i++)\n> -\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n> -\n> -\t\t/* Mark all previously saved entries as dirty */\n> -\t\tewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n> -\n> -\t\t/* Now mark the untracked cache for fsmonitor usage */\n> -\t\tif (istate->untracked)\n> -\t\t\tistate->untracked->use_fsmonitor = 1;\n> -\t}\n> -\tewah_free(fsmonitor_dirty);\n> +\tistate->fsmonitor_dirty = fsmonitor_dirty;\n\nThis makes local variable \"int i;\" in this function unused and gets\ncompiler warning.\n\n"},{"id":"331489","messageId":"alpine.DEB.2.21.1.1710311830330.6482@virtualbox","threadId":"47048","inReplyTo":"xmqqa80728lo.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-10-31T17:31:52Z","receivedAt":"2017-10-31T17:32:03Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 31 Oct 2017, Junio C Hamano wrote:\n\n> Alex Vandiver <alexmv@dropbox.com> writes:\n> \n> > diff --git a/fsmonitor.c b/fsmonitor.c\n> > index 4ea44dcc6..417759224 100644\n> > --- a/fsmonitor.c\n> > +++ b/fsmonitor.c\n> > @@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,\n> >  \t\tewah_free(fsmonitor_dirty);\n> >  \t\treturn error(\"failed to parse ewah bitmap reading fsmonitor index extension\");\n> >  \t}\n> > -\n> > -\tif (git_config_get_fsmonitor()) {\n> > -\t\t/* Mark all entries valid */\n> > -\t\tfor (i = 0; i < istate->cache_nr; i++)\n> > -\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n> > -\n> > -\t\t/* Mark all previously saved entries as dirty */\n> > -\t\tewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);\n> > -\n> > -\t\t/* Now mark the untracked cache for fsmonitor usage */\n> > -\t\tif (istate->untracked)\n> > -\t\t\tistate->untracked->use_fsmonitor = 1;\n> > -\t}\n> > -\tewah_free(fsmonitor_dirty);\n> > +\tistate->fsmonitor_dirty = fsmonitor_dirty;\n> \n> This makes local variable \"int i;\" in this function unused and gets\n> compiler warning.\n\n... to which end we introduced the DEVELOPER flag to catch these: if you\ncall\n\n\tmake DEVELOPER=1\n\nand compile with GCC or Clang, it will elevate such warnings to errors,\nand we highly encourage contributors to build their patched source code\nwith said flag.\n\nThanks,\nJohannes\n"},{"id":"331507","messageId":"alpine.DEB.2.10.1710311139560.5248@alexmv-linux","threadId":"47048","inReplyTo":"alpine.DEB.2.21.1.1710311830330.6482@virtualbox","subject":"Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged","fromName":"Alex Vandiver","fromEmail":"alexmv@dropbox.com","sentAt":"2017-10-31T18:43:10Z","receivedAt":"2017-10-31T18:43:26Z","isPatch":true,"sender":{"key":"alexmv@dropbox.com","avatar":null},"body":"On Tue, 31 Oct 2017, Junio C Hamano wrote:\n> This makes local variable \"int i;\" in this function unused and gets\n> compiler warning.\n\nApologies for leaving that detritus -- I saw you added a 'SQUASH??' commit\nto deal with it, which LGTM.\n\nOn Tue, 31 Oct 2017, Johannes Schindelin wrote:\n> ... to which end we introduced the DEVELOPER flag to catch these: if you\n> call\n> \n> \tmake DEVELOPER=1\n\nAha!  Thanks for the tip; I'll be sure to use that from now on.\n - Alex\n"}]}