{"thread":{"id":"57414","subject":"[PATCH 00/23] Builtin FSMonitor Part 3","startedAt":"2022-02-15T15:59:40Z","lastAt":"2022-06-02T09:57:03Z","messageCount":345,"participants":["Jeff Hostetler via GitGitGadget","Junio C Hamano","Jeff Hostetler","Derrick Stolee","Torsten B��gershausen","Ævar Arnfjörð Bjarmason","Torsten Bögershausen","rsbecker@nexbridge.com","Bagas Sanjaya","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":23},"messages":[{"id":"448434","messageId":"pull.1143.git.1644940773.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":null,"subject":"[PATCH 00/23] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:10Z","receivedAt":"2022-02-15T15:59:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is part 3 of my builtin FSMonitor series.\n\nPart 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in\n\"seen\", so this series should be considered a preview until part 2 moves to\n\"next\". (And part 2 should not graduate to \"master\" without this part.)\n\nPart 2 established the client code (used by commands like git status) and an\nMVP implementation of the FSMonitor daemon. This was sufficient to test the\nconcepts and basic functionality.\n\nPart 3 finishes the daemon and adds additional tests. This includes:\n\n * On Windows, handle short- and long-name aliasing.\n * On Mac, handle Unicode aliasing.\n * Mark bare, virtual, and remote working directories incompatible with\n FSMonitor.\n * On Mac, ignore xattr change FSEvents.\n * On Windows, cd out of the working directory root.\n * Create a thread to monitor the health and shutdown the daemon if\n necessary.\n * Speed up handling of directory notification events.\n * Test directory move and rename events.\n * Add performance test.\n\nHere is performance data from t/perf/p7527-builtin-fsmonitor.sh on a\nsynthetic repo containing 1M files on a Macbook Pro. It shows the effects of\nthe untracked cache (uc) and FSMonitor (fsm) on git status.\n\n$ ./p7527-builtin-fsmonitor.sh \n# passed all 67 test(s)\n1..67\nTest this tree \n---------------------------------------------------------------------------------------\n7527.4: [uc false][fsm false] status after checkout 29.99(3.14+80.12) \n7527.6: [uc false][fsm false] status after big change 73.32(5.11+97.24) \n7527.8: [uc false][fsm false] status after add all 47.80(5.12+90.47) \n7527.10: [uc false][fsm false] status after add dot 49.22(5.16+92.05) \n7527.12: [uc false][fsm false] status after commit 51.53(3.35+100.74)\n7527.14: [uc false][fsm false] status after reset hard 33.74(3.03+85.31) \n7527.16: [uc false][fsm false] status after create untracked files 41.71(3.24+89.75) \n7527.18: [uc false][fsm false] status after clean 34.33(3.07+89.36) \n\n7527.20: [uc false][fsm true] status after checkout 29.23(1.94+10.84) \n7527.22: [uc false][fsm true] status after big change 64.23(4.66+24.86) \n7527.24: [uc false][fsm true] status after add all 45.45(4.37+18.70) \n7527.26: [uc false][fsm true] status after add dot 44.42(4.02+17.10) \n7527.28: [uc false][fsm true] status after commit 30.52(1.95+10.91) \n7527.30: [uc false][fsm true] status after reset hard 28.70(2.70+13.89) \n7527.32: [uc false][fsm true] status after create untracked files 28.63(2.59+10.71) \n7527.34: [uc false][fsm true] status after clean 28.97(2.59+10.78) \n\n7527.36: [uc true][fsm false] status after checkout 35.06(3.17+86.11) \n7527.38: [uc true][fsm false] status after big change 74.65(5.14+101.50)\n7527.40: [uc true][fsm false] status after add all 49.96(5.22+90.96) \n7527.42: [uc true][fsm false] status after add dot 49.77(5.24+91.72) \n7527.44: [uc true][fsm false] status after commit 36.95(3.27+92.25) \n7527.46: [uc true][fsm false] status after reset hard 33.89(3.18+85.68) \n7527.48: [uc true][fsm false] status after create untracked files 41.44(3.40+92.99) \n7527.50: [uc true][fsm false] status after clean 34.60(3.26+90.19) \n\n7527.52: [uc true][fsm true] status after checkout 0.58(0.45+0.10) \n7527.54: [uc true][fsm true] status after big change 65.16(4.91+25.64) \n7527.56: [uc true][fsm true] status after add all 45.43(4.45+18.92) \n7527.58: [uc true][fsm true] status after add dot 15.56(2.57+6.32) \n7527.60: [uc true][fsm true] status after commit 0.98(0.46+0.11) \n7527.62: [uc true][fsm true] status after reset hard 30.30(2.96+14.49) \n7527.64: [uc true][fsm true] status after create untracked files 2.15(1.73+0.40) \n7527.66: [uc true][fsm true] status after clean 1.68(1.56+0.32) \n\n\nHere is performance data from t/perf/p7519-fsmonitor.sh on the same\nsynthetic repo containing 1M files on a Macbook Pro. This test performs the\nsame series of commands on all three FSMonitor options: (1) using the hook\ninterface to talk to Watchman, (2) no FSMonitor, and (3) the builtin\nFSMonitor daemon.\n\n$ ./p7519-fsmonitor.sh \n# passed all 42 test(s)\n1..42\nTest this tree \n-------------------------------------------------------------------------------\n7519.4: status (fsmonitor=fsmonitor-watchman) 0.89(0.63+0.17) \n7519.5: status -uno (fsmonitor=fsmonitor-watchman) 0.56(0.43+0.13) \n7519.6: status -uall (fsmonitor=fsmonitor-watchman) 24.52(1.82+9.90) \n7519.7: status (dirty) (fsmonitor=fsmonitor-watchman) 32.40(6.54+9.94) \n7519.8: diff (fsmonitor=fsmonitor-watchman) 0.66(0.38+0.42) \n7519.9: diff HEAD (fsmonitor=fsmonitor-watchman) 1.60(0.69+0.80) \n7519.10: diff -- 0_files (fsmonitor=fsmonitor-watchman) 0.37(0.29+0.08) \n7519.11: diff -- 10_files (fsmonitor=fsmonitor-watchman) 0.42(0.32+0.09) \n7519.12: diff -- 100_files (fsmonitor=fsmonitor-watchman) 0.42(0.32+0.10) \n7519.13: diff -- 1000_files (fsmonitor=fsmonitor-watchman) 0.41(0.31+0.09) \n7519.14: diff -- 10000_files (fsmonitor=fsmonitor-watchman) 0.44(0.32+0.12) \n7519.15: add (fsmonitor=fsmonitor-watchman) 24.81(2.44+10.06)\n\n7519.18: status (fsmonitor=disabled) 11.18(1.80+74.68)\n7519.19: status -uno (fsmonitor=disabled) 7.11(1.50+73.95) \n7519.20: status -uall (fsmonitor=disabled) 30.09(3.01+91.11)\n7519.21: status (dirty) (fsmonitor=disabled) 31.46(5.03+77.75)\n7519.22: diff (fsmonitor=disabled) 5.79(1.27+62.97) \n7519.23: diff HEAD (fsmonitor=disabled) 7.29(1.57+76.03) \n7519.24: diff -- 0_files (fsmonitor=disabled) 0.68(0.30+0.09) \n7519.25: diff -- 10_files (fsmonitor=disabled) 0.32(0.29+0.07) \n7519.26: diff -- 100_files (fsmonitor=disabled) 0.32(0.29+0.07) \n7519.27: diff -- 1000_files (fsmonitor=disabled) 0.35(0.29+0.09) \n7519.28: diff -- 10000_files (fsmonitor=disabled) 0.56(0.29+0.21) \n7519.29: add (fsmonitor=disabled) 29.49(3.52+85.88)\n\n7519.31: status (builtin fsmonitor--daemon) 0.67(0.54+0.09) \n7519.32: status -uno (builtin fsmonitor--daemon) 0.42(0.35+0.08) \n7519.33: status -uall (builtin fsmonitor--daemon) 22.77(1.71+10.35)\n7519.34: status (dirty) (builtin fsmonitor--daemon) 21.79(4.18+9.97) \n7519.35: diff (builtin fsmonitor--daemon) 0.52(0.34+0.36) \n7519.36: diff HEAD (builtin fsmonitor--daemon) 1.55(0.66+0.80) \n7519.37: diff -- 0_files (builtin fsmonitor--daemon) 0.37(0.31+0.08) \n7519.38: diff -- 10_files (builtin fsmonitor--daemon) 0.35(0.29+0.07) \n7519.39: diff -- 100_files (builtin fsmonitor--daemon) 0.35(0.29+0.07) \n7519.40: diff -- 1000_files (builtin fsmonitor--daemon) 0.35(0.29+0.07) \n7519.41: diff -- 10000_files (builtin fsmonitor--daemon) 0.37(0.29+0.09) \n7519.42: add (builtin fsmonitor--daemon) 24.64(2.40+10.04)\n\n\nJeff Hostetler (23):\n fsm-listen-win32: handle shortnames\n t7527: test FS event reporing on macOS WRT case and Unicode\n t7527: test builtin FSMonitor watching repos with unicode paths\n t/helper/fsmonitor-client: create stress test\n fsmonitor-settings: bare repos are incompatible with FSMonitor\n fsmonitor-settings: stub in platform-specific incompatibility checking\n fsmonitor-settings: virtual repos are incompatible with FSMonitor\n fsmonitor-settings: stub in macOS-specific incompatibility checking\n fsmonitor-settings: remote repos on macOS are incompatible with\n FSMonitor\n fsmonitor-settings: remote repos on Windows are incompatible with\n FSMonitor\n unpack-trees: initialize fsmonitor_has_run_once in o->result\n fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n fsmonitor--daemon: print start message only if\n fsmonitor.announceStartup\n fsmonitor--daemon: cd out of worktree root\n fsmonitor--daemon: prepare for adding health thread\n fsmonitor--daemon: rename listener thread related variables\n fsmonitor--daemon: stub in health thread\n fsm-health-win32: add framework to monitor daemon health\n fsm-health-win32: force shutdown daemon if worktree root moves\n fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n fsmonitor: optimize processing of directory events\n t7527: FSMonitor tests for directory moves\n t/perf/p7527: add perf test for builtin FSMonitor\n\n Makefile | 19 +-\n builtin/fsmonitor--daemon.c | 138 ++++++++-\n builtin/update-index.c | 9 +\n compat/fsmonitor/fsm-health-darwin.c | 24 ++\n compat/fsmonitor/fsm-health-win32.c | 257 ++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 +++\n compat/fsmonitor/fsm-listen-darwin.c | 89 +++++-\n compat/fsmonitor/fsm-listen-win32.c | 409 ++++++++++++++++++++-----\n compat/fsmonitor/fsm-listen.h | 2 +-\n compat/fsmonitor/fsm-settings-darwin.c | 75 +++++\n compat/fsmonitor/fsm-settings-win32.c | 137 +++++++++\n config.mak.uname | 5 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 11 +-\n fsmonitor-settings.c | 84 +++++\n fsmonitor-settings.h | 26 ++\n fsmonitor.c | 71 ++++-\n t/helper/test-fsmonitor-client.c | 105 +++++++\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++\n t/t7519-status-fsmonitor.sh | 35 +++\n t/t7527-builtin-fsmonitor.sh | 157 ++++++++++\n unpack-trees.c | 1 +\n 22 files changed, 1838 insertions(+), 128 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\n\nbase-commit: ce8aa175364c7a02115c4b4f91d5674f4d6d8ae8\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1143\n-- \ngitgitgadget\n"},{"id":"448435","messageId":"ad8cf6d9a47b61d9fe41a961466122be16e4f041.1644940773.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:12Z","receivedAt":"2022-02-15T15:59:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that macOS FS events are reported with a normalized spelling.\n\nAPFS (and/or HFS+) is case-insensitive. This means that case-independent\nlookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed. But that\ndoesn't tell us how FS events are reported if we try \"rm -rf .git\" versus\n\"rm -rf .GIT\". Are the events reported using the on-disk spelling of the\npathname or in the spelling used by the command.\n\nNEEDSWORK: I was only able to test case. It would be nice to add tests\nthat use different Unicode spellings/normalizations and understand the\ndifferences between APFS and HFS+ in this area. We should confirm that\nthe spelling of the workdir paths that the daemon sends to clients are\nalways properly normalized.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 30 ++++++++++++++++++++++++++++++\n 1 file changed, 30 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex dbca7f835eb..1fdabfc4f1e 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -188,6 +188,36 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n \ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n '\n \n+# Confirm that MacOS hides all of the Unicode normalization and/or\n+# case folding from the FS events. That is, are the pathnames in the\n+# FS events reported using the spelling on the disk or in the spelling\n+# used by the other process.\n+#\n+# Note that we assume that the filesystem is set to case insensitive.\n+#\n+# NEEDSWORK: APFS handles Unicode and Unicode normalization\n+# differently than HFS+. I only have an APFS partition, so\n+# more testing here would be helpful.\n+#\n+\n+# Rename .git using alternate spelling and confirm that the daemon\n+# sees the event using the correct spelling and shutdown.\n+test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_apfs\" &&\n+\n+\tgit init test_apfs &&\n+\tstart_daemon test_apfs &&\n+\n+\ttest_path_is_dir test_apfs/.git &&\n+\ttest_path_is_dir test_apfs/.GIT &&\n+\n+\tmv test_apfs/.GIT test_apfs/.FOO &&\n+\tsleep 1 &&\n+\tmv test_apfs/.FOO test_apfs/.git &&\n+\n+\ttest_must_fail git -C test_apfs fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"448436","messageId":"23f38338cecd270c15fd1bba42de79c53e14ca2a.1644940773.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 01/23] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:11Z","receivedAt":"2022-02-15T15:59:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach FSMonitor daemon on Windows to recognize shortname paths as\naliases of normal longname paths. FSMonitor clients, such as `git\nstatus`, should receive the longname spelling of changed files (when\npossible).\n\nSometimes we receive FS events using the shortname, such as when a CMD\nshell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\narrives using whatever combination of long and shortnames were used by\nthe other process. (Shortnames do seem to be case normalized,\nhowever.)\n\nUse Windows GetLongPathNameW() to try to map the pathname spelling in\nthe notification event into the normalized longname spelling. (This\ncan fail if the file/directory is deleted, moved, or renamed, because\nwe are asking the FS for the mapping in response to the event and\nafter it has already happened, but we try.)\n\nSpecial case the shortname spelling of \".git\" to avoid under-reporting\nthese events.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 359 +++++++++++++++++++++++-----\n t/t7527-builtin-fsmonitor.sh | 65 +++++\n 2 files changed, 370 insertions(+), 54 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex c2d11acbc1e..eb407b0748c 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -25,6 +25,9 @@ struct one_watch\n \tDWORD count;\n \n \tstruct strbuf path;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n+\tDWORD wpath_longname_len;\n+\n \tHANDLE hDir;\n \tHANDLE hEvent;\n \tOVERLAPPED overlapped;\n@@ -34,6 +37,21 @@ struct one_watch\n \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n \t */\n \tBOOL is_active;\n+\n+\t/*\n+\t * Are shortnames enabled on the containing drive? This is\n+\t * always true for \"C:/\" drives and usually never true for\n+\t * other drives.\n+\t *\n+\t * We only set this for the worktree because we only need to\n+\t * convert shortname paths to longname paths for items we send\n+\t * to clients. (We don't care about shortname expansion for\n+\t * paths inside a GITDIR because we never send them to\n+\t * clients.)\n+\t */\n+\tBOOL has_shortnames;\n+\tBOOL has_tilda;\n+\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n struct fsmonitor_daemon_backend_data\n@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n };\n \n /*\n- * Convert the WCHAR path from the notification into UTF8 and\n- * then normalize it.\n+ * Convert the WCHAR path from the event into UTF8 and normalize it.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n */\n-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n \t\t\t\t struct strbuf *normalized_path)\n {\n \tint reserve;\n \tint len = 0;\n \n \tstrbuf_reset(normalized_path);\n-\tif (!info->FileNameLength)\n+\tif (!wpath_len)\n \t\tgoto normalize;\n \n \t/*\n@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t * sequence of 2 UTF8 characters. That should let us\n \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n \t */\n-\treserve = info->FileNameLength + 1;\n+\treserve = 2 * wpath_len + 1;\n \tstrbuf_grow(normalized_path, reserve);\n \n \tfor (;;) {\n-\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n-\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n+\t\t\t\t\t wpath, wpath_len,\n \t\t\t\t\t normalized_path->buf,\n \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n \t\t\t\t\t NULL, NULL);\n@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n \t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n-\t\t\t GetLastError(),\n-\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n-\t\t\t info->FileName);\n+\t\t\t GetLastError(), (int)wpath_len, wpath);\n \t\t\treturn -1;\n \t\t}\n \n@@ -98,6 +115,148 @@ normalize:\n \treturn strbuf_normalize_path(normalized_path);\n }\n \n+/*\n+ * See if the worktree root directory has shortnames enabled.\n+ * This will help us decide if we need to do an expensive shortname\n+ * to longname conversion on every notification event.\n+ *\n+ * We do not want to create a file to test this, so we assume that the\n+ * root directory contains a \".git\" file or directory. (Out caller\n+ * only calls us for the worktree root, so this should be fine.)\n+ *\n+ * Remember the spelling of the shortname for \".git\" if it exists.\n+ */\n+static void check_for_shortnames(struct one_watch *watch)\n+{\n+\twchar_t buf_in[MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\twchar_t *last_slash = NULL;\n+\twchar_t *last_bslash = NULL;\n+\twchar_t *last;\n+\n+\t/* build L\"/.git\" */\n+\twcscpy(buf_in, watch->wpath_longname);\n+\twcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n+\n+\tif (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n+\t\treturn;\n+\n+\tlast_slash = wcsrchr(buf_out, L'/');\n+\tlast_bslash = wcsrchr(buf_out, L'\\\\');\n+\tif (last_slash > last_bslash)\n+\t\tlast = last_slash + 1;\n+\telse if (last_bslash)\n+\t\tlast = last_bslash + 1;\n+\telse\n+\t\tlast = buf_out;\n+\n+\tif (!wcscmp(last, L\".git\"))\n+\t\treturn;\n+\n+\twatch->has_shortnames = 1;\n+\twcsncpy(watch->dotgit_shortname, last,\n+\t\tARRAY_SIZE(watch->dotgit_shortname));\n+\n+\t/*\n+\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n+\t * we should be able to avoid shortname to longname mapping on\n+\t * every notification event if the source string does not\n+\t * contain a \"~\".\n+\t *\n+\t * However, the documentation for GetLongPathNameW() says\n+\t * that there are filesystems that don't follow that pattern\n+\t * and warns against this optimization.\n+\t *\n+\t * Lets test this.\n+\t */\n+\tif (wcschr(watch->dotgit_shortname, L'~'))\n+\t\twatch->has_tilda = 1;\n+}\n+\n+enum get_relative_result {\n+\tGRR_NO_CONVERSION_NEEDED,\n+\tGRR_HAVE_CONVERSION,\n+\tGRR_SHUTDOWN,\n+};\n+\n+/*\n+ * Info notification paths are relative to the root of the watch.\n+ * If our CWD is still at the root, then we can use relative paths\n+ * to convert from shortnames to longnames. If our process has a\n+ * different CWD, then we need to construct an absolute path, do\n+ * the conversion, and then return the root-relative portion.\n+ *\n+ * We use the longname form of the root as our basis and assume that\n+ * it already has a trailing slash.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n+ */\n+static enum get_relative_result get_relative_longname(\n+\tstruct one_watch *watch,\n+\tconst wchar_t *wpath, DWORD wpath_len,\n+\twchar_t *wpath_longname)\n+{\n+\twchar_t buf_in[2 * MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\tDWORD root_len;\n+\n+\t/* Build L\"/\" */\n+\troot_len = watch->wpath_longname_len;\n+\twcsncpy(buf_in, watch->wpath_longname, root_len);\n+\twcsncpy(buf_in + root_len, wpath, wpath_len);\n+\tbuf_in[root_len + wpath_len] = 0;\n+\n+\t/*\n+\t * We don't actually know if the source pathname is a\n+\t * shortname or a longname. This routine allows either to be\n+\t * given as input.\n+\t */\n+\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n+\t\t/*\n+\t\t * The shortname to longname conversion can fail for\n+\t\t * various reasons, for example if the file has been\n+\t\t * deleted. (That is, if we just received a\n+\t\t * delete-file notification event and the file is\n+\t\t * already gone, we can't ask the file system to\n+\t\t * lookup the longname for it. Likewise, for moves\n+\t\t * and renames where we are given the old name.)\n+\t\t *\n+\t\t * NEEDSWORK: Since deleting or moving a file or\n+\t\t * directory by shortname is rather obscure, I'm going\n+\t\t * ignore the failure and ask the caller to report the\n+\t\t * original relative path. This seemds kinder than\n+\t\t * failing here and forcing a resync.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (!wcscmp(buf_in, buf_out)) {\n+\t\t/*\n+\t\t * The path does not have a shortname alias.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (wcsncmp(buf_in, buf_out, root_len)) {\n+\t\t/*\n+\t\t * The spelling of the root directory portion of the computed\n+\t\t * longname has changed. This should not happen. Basically,\n+\t\t * it means that we don't know where (without recomputing the\n+\t\t * longname of just the root directory) to split out the\n+\t\t * relative path. Since this should not happen, I'm just\n+\t\t * going to let this fail and force a shutdown (because all\n+\t\t * subsequent events are probably going to see the same\n+\t\t * mismatch).\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\t/* Return the worktree root-relative portion of the longname. */\n+\n+\twcscpy(wpath_longname, buf_out + root_len);\n+\treturn GRR_HAVE_CONVERSION;\n+}\n+\n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n@@ -111,7 +270,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n-\twchar_t wpath[MAX_PATH];\n+\tDWORD len_longname;\n+\twchar_t wpath[MAX_PATH + 1];\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \tif (xutftowcs_path(wpath, path) < 0) {\n \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n@@ -128,6 +289,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \t\treturn NULL;\n \t}\n \n+\tif (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n+\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n+\t\t GetLastError(), path);\n+\t\tCloseHandle(hDir);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen_longname = wcslen(wpath_longname);\n+\tif (wpath_longname[len_longname - 1] != L'/' &&\n+\t wpath_longname[len_longname - 1] != L'\\\\') {\n+\t\twpath_longname[len_longname++] = L'/';\n+\t\twpath_longname[len_longname] = 0;\n+\t}\n+\n \tCALLOC_ARRAY(watch, 1);\n \n \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n@@ -135,6 +310,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tstrbuf_init(&watch->path, 0);\n \tstrbuf_addstr(&watch->path, path);\n \n+\twcscpy(watch->wpath_longname, wpath_longname);\n+\twatch->wpath_longname_len = len_longname;\n+\n \twatch->hDir = hDir;\n \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n \n@@ -258,6 +436,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t,\n+\tDWORD info_action)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n+\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -274,6 +508,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n \tstruct fsmonitor_batch *batch = NULL;\n \tconst char *p = watch->buffer;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \t/*\n \t * If the kernel gets more events than will fit in the kernel\n@@ -306,54 +541,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n+\t\twchar_t *wpath = info->FileName;\n+\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n \t\tenum fsmonitor_path_type t;\n+\t\tenum get_relative_result grr;\n+\n+\t\tif (watch->has_shortnames) {\n+\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n+\t\t\t\t/*\n+\t\t\t\t * This event exactly matches the\n+\t\t\t\t * spelling of the shortname of\n+\t\t\t\t * \".git\", so we can skip some steps.\n+\t\t\t\t *\n+\t\t\t\t * (This case is odd because the user\n+\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n+\t\t\t\t * use the filesystem to map it back\n+\t\t\t\t * to \".git\".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_reset(&path);\n+\t\t\t\tstrbuf_addstr(&path, \".git\");\n+\t\t\t\tt = IS_DOT_GIT;\n+\t\t\t\tgoto process_it;\n+\t\t\t}\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n-\t\t\tgoto skip_this_path;\n-\n-\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n-\n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n+\t\t\tif (watch->has_tilda && !wcschr(wpath, L'~')) {\n+\t\t\t\t/*\n+\t\t\t\t * Shortnames on this filesystem have tildas\n+\t\t\t\t * and the notification path does not have\n+\t\t\t\t * one, so we assume that it is a longname.\n+\t\t\t\t */\n+\t\t\t\tgoto normalize_it;\n+\t\t\t}\n \n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n+\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n+\t\t\t\t\t\t wpath_longname);\n+\t\t\tswitch (grr) {\n+\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n+\t\t\t\tbreak;\n+\t\t\tcase GRR_HAVE_CONVERSION:\n+\t\t\t\twpath = wpath_longname;\n+\t\t\t\twpath_len = wcslen(wpath);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\tcase GRR_SHUTDOWN:\n \t\t\t\tgoto force_shutdown;\n \t\t\t}\n-\t\t\tbreak;\n+\t\t}\n \n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n+normalize_it:\n+\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n+\t\t\tgoto skip_this_path;\n \n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+process_it:\n+\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n+\t\t\t\t\t info->Action))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -382,6 +626,9 @@ force_shutdown:\n * Note that we DO NOT get filesystem events on the external \n * itself (it is not inside something that we are watching). In particular,\n * we do not get an event if the external is deleted.\n+ *\n+ * Also, we do not care about shortnames within the external , since\n+ * we never send these paths to clients.\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n@@ -403,8 +650,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\tif (normalize_path_in_utf8(\n+\t\t\t info->FileName,\n+\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t &path) == -1)\n \t\t\tgoto skip_this_path;\n \n \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n@@ -538,6 +787,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \tif (!data->watch_worktree)\n \t\tgoto failed;\n \n+\tcheck_for_shortnames(data->watch_worktree);\n+\n \tif (state->nr_paths_watching > 1) {\n \t\tdata->watch_gitdir = create_watch(state,\n \t\t\t\t\t\t state->path_gitdir_watch.buf);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 0ccbfb9616f..dbca7f835eb 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -123,6 +123,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n+\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n+\n+\tstart_daemon test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"448437","messageId":"9af952e4d170d71d66db721368af42ae5ce0fca8.1644940773.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 03/23] t7527: test builtin FSMonitor watching repos with unicode paths","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:13Z","receivedAt":"2022-02-15T15:59:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate some test repos with UTF8 pathnames and verify that\nthe builtin FSMonitor can watch them. This test is mainly\nfor Windows where we need to avoid `*A()` routines.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1fdabfc4f1e..c0145544ccb 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -696,4 +696,27 @@ do\n \tdone\n done\n \n+# Test Unicode UTF-8 characters in the pathname of the working\n+# directory. Use of \"*A()\" routines rather than \"*W()\" routines\n+# on Windows can sometimes lead to odd failures.\n+#\n+u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n+u2=$(printf \"u_e2_99_ab__\\xE2\\x99\\xAB\")\n+u_values=\"$u1 $u2\"\n+for u in $u_values\n+do\n+\ttest_expect_success \"Unicode path: $u\" '\n+\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n+\n+\t\tgit init \"$u\" &&\n+\t\techo 1 >\"$u\"/file1 &&\n+\t\tgit -C \"$u\" add file1 &&\n+\t\tgit -C \"$u\" config core.fsmonitor true &&\n+\n+\t\tstart_daemon \"$u\" &&\n+\t\tgit -C \"$u\" status >actual &&\n+\t\tgrep \"new file: file1\" actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"448438","messageId":"9efdbe28223e371dc22b4ba32fc791729af5577a.1644940773.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 04/23] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:14Z","receivedAt":"2022-02-15T15:59:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a stress test to hammer on the fsmonitor daemon.\nCreate a client-side thread pool of n threads and have\neach of them make m requests as fast as they can.\n\nNEEDSWORK: This is just the client-side thread pool and\nis useful for interactive testing and experimentation.\nWe need to add a script test to drive this.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-fsmonitor-client.c | 105 +++++++++++++++++++++++++++++++\n 1 file changed, 105 insertions(+)\n\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nindex f7a5b3a32fa..9dd2f9af553 100644\n--- a/t/helper/test-fsmonitor-client.c\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -7,6 +7,8 @@\n #include \"cache.h\"\n #include \"parse-options.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2.h\"\n \n #ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n int cmd__fsmonitor_client(int argc, const char **argv)\n@@ -79,20 +81,120 @@ static int do_send_flush(void)\n \treturn 0;\n }\n \n+struct hammer_thread_data\n+{\n+\tpthread_t pthread_id;\n+\tint thread_nr;\n+\n+\tint nr_requests;\n+\tconst char *token;\n+\n+\tint sum_successful;\n+\tint sum_errors;\n+};\n+\n+static void *hammer_thread_proc(void *_hammer_thread_data)\n+{\n+\tstruct hammer_thread_data *data = _hammer_thread_data;\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint k;\n+\tint ret;\n+\n+\ttrace2_thread_start(\"hammer\");\n+\n+\tfor (k = 0; k < data->nr_requests; k++) {\n+\t\tstrbuf_reset(&answer);\n+\n+\t\tret = fsmonitor_ipc__send_query(data->token, &answer);\n+\t\tif (ret < 0)\n+\t\t\tdata->sum_errors++;\n+\t\telse\n+\t\t\tdata->sum_successful++;\n+\t}\n+\n+\tstrbuf_release(&answer);\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Start a pool of client threads that will each send a series of\n+ * commands to the daemon.\n+ *\n+ * The goal is to overload the daemon with a sustained series of\n+ * concurrent requests.\n+ */\n+static int do_hammer(const char *token, int nr_threads, int nr_requests)\n+{\n+\tstruct hammer_thread_data *data = NULL;\n+\tint k;\n+\tint sum_join_errors = 0;\n+\tint sum_commands = 0;\n+\tint sum_errors = 0;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\tif (nr_threads < 1)\n+\t\tnr_threads = 1;\n+\tif (nr_requests < 1)\n+\t\tnr_requests = 1;\n+\n+\tCALLOC_ARRAY(data, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\t\tp->thread_nr = k;\n+\t\tp->nr_requests = nr_requests;\n+\t\tp->token = token;\n+\n+\t\tif (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {\n+\t\t\twarning(\"failed to create thread[%d] skipping remainder\", k);\n+\t\t\tnr_threads = k;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\n+\t\tif (pthread_join(p->pthread_id, NULL))\n+\t\t\tsum_join_errors++;\n+\t\tsum_commands += p->sum_successful;\n+\t\tsum_errors += p->sum_errors;\n+\t}\n+\n+\tfprintf(stderr, \"HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\\n\",\n+\t\tnr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);\n+\n+\tfree(data);\n+\n+\t/*\n+\t * TODO Decide if/when to return an error or call die().\n+\t */\n+\treturn 0;\n+}\n+\n int cmd__fsmonitor_client(int argc, const char **argv)\n {\n \tconst char *subcmd;\n \tconst char *token = NULL;\n+\tint nr_threads = 1;\n+\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n \t\tN_(\"test-helper fsmonitor-client query []\"),\n \t\tN_(\"test-helper fsmonitor-client flush\"),\n+\t\tN_(\"test-helper fsmonitor-client hammer [] [] []\"),\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n \t\tOPT_STRING(0, \"token\", &token, N_(\"token\"),\n \t\t\t N_(\"command token to send to the server\")),\n+\n+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, N_(\"number of client threads\")),\n+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, N_(\"number of requests per thread\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -116,6 +218,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)\n \tif (!strcmp(subcmd, \"flush\"))\n \t\treturn !!do_send_flush();\n \n+\tif (!strcmp(subcmd, \"hammer\"))\n+\t\treturn !!do_hammer(token, nr_threads, nr_requests);\n+\n \tdie(\"Unhandled subcommand: '%s'\", subcmd);\n }\n #endif\n-- \ngitgitgadget\n\n"},{"id":"448439","messageId":"4e856d60e385d64158f17ec1226f97eb323bc55e.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 07/23] fsmonitor-settings: virtual repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:17Z","receivedAt":"2022-02-15T15:59:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nVirtual repos, such as GVFS (aka VFS for Git), are incompatible\nwith FSMonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 5 +++++\n fsmonitor-settings.h | 1 +\n t/t7519-status-fsmonitor.sh | 9 +++++++++\n 4 files changed, 41 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 176a6f5726c..7caa79570af 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -3,7 +3,33 @@\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n \n+/*\n+ * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n+ *\n+ * Granted, core Git does not know anything about GVFS and we\n+ * shouldn't make assumptions about a downstream feature, but users\n+ * can install both versions. And this can lead to incorrect results\n+ * from core Git commands. So, without bringing in any of the GVFS\n+ * code, do a simple config test for a published config setting. (We\n+ * do not look at the various *_TEST_* environment variables.)\n+ */\n+static enum fsmonitor_reason is_virtual(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n+\t\treturn FSMONITOR_REASON_VIRTUAL;\n+\n+\treturn FSMONITOR_REASON_ZERO;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = is_virtual(r);\n+\tif (reason)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_ZERO;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex e445572354e..bb2ddd2457f 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -156,6 +156,11 @@ static void create_reason_message(struct repository *r,\n \t\t\t _(\"bare repos are incompatible with fsmonitor\"));\n \t\treturn;\n \n+\tcase FSMONITOR_REASON_VIRTUAL:\n+\t\tstrbuf_addstr(buf_reason,\n+\t\t\t _(\"virtual repos are incompatible with fsmonitor\"));\n+\t\treturn;\n+\n \tdefault:\n \t\tBUG(\"Unhandled case in create_reason_message '%d'\", s->reason);\n \t}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex b52bf8edaf1..c169683bf2d 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -16,6 +16,7 @@ enum fsmonitor_mode {\n enum fsmonitor_reason {\n \tFSMONITOR_REASON_ZERO = 0,\n \tFSMONITOR_REASON_BARE = 1,\n+\tFSMONITOR_REASON_VIRTUAL = 2,\n };\n \n void fsm_settings__set_ipc(struct repository *r);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 3c4e6f5f89c..d7540931a16 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -81,6 +81,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n \tgrep \"bare repos are incompatible with fsmonitor\" actual\n '\n \n+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone actual\" &&\n+\tgit init fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"virtual repos are incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"448440","messageId":"4715677f85ff3f47b437f82e2b82a8b5dd371c99.1644940773.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 06/23] fsmonitor-settings: stub in platform-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:16Z","receivedAt":"2022-02-15T15:59:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend generic incompatibility checkout with platform-specific\nmechanism. Stub in Win32 version.\n\nIn the existing fsmonitor-settings code we have a way to mark\ntypes of repos as incompatible with fsmonitor (whether via the\nhook and ipc APIs). For example, we do this for bare repos,\nsince there are no files to watch.\n\nExtend this exclusion mechanism for platfor-specific reasons.\nThis commit just creates the framework and adds a stub for Win32.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 +++++++++++++\n compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++\n config.mak.uname | 4 ++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n fsmonitor-settings.c | 12 ++++++++++++\n fsmonitor-settings.h | 13 +++++++++++++\n 6 files changed, 54 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n\ndiff --git a/Makefile b/Makefile\nindex 5ad0cef69f4..ed3d5231ea1 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -470,6 +470,11 @@ all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n+# If your platform has os-specific ways to tell if a repo is incompatible with\n+# fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS\n+# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n+# that implements the `fsm_os_settings__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1941,6 +1946,11 @@ ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n+ifdef FSMONITOR_OS_SETTINGS\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2819,6 +2829,9 @@ GIT-BUILD-OPTIONS: FORCE\n ifdef FSMONITOR_DAEMON_BACKEND\n \t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n endif\n+ifdef FSMONITOR_OS_SETTINGS\n+\t@echo FSMONITOR_OS_SETTINGS=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_OS_SETTINGS)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nnew file mode 100644\nindex 00000000000..176a6f5726c\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_ZERO;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 81f21d9e201..219a04b4c66 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -441,6 +441,8 @@ ifeq ($(uname_S),Windows)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -629,6 +631,8 @@ ifeq ($(uname_S),MINGW)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 41395705ac6..7b8ea41967b 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 0fc5566eb8a..e445572354e 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r)\n \t\treturn 1;\n \t}\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+\t{\n+\t\tenum fsmonitor_reason reason;\n+\n+\t\treason = fsm_os__incompatible(r);\n+\t\tif (reason != FSMONITOR_REASON_ZERO) {\n+\t\t\tset_incompatible(r, reason);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+#endif\n+\n \treturn 0;\n }\n \ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex e5f145a2f79..b52bf8edaf1 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -29,4 +29,17 @@ enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n \n struct fsmonitor_settings;\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+/*\n+ * Ask platform-specific code whether the repository is incompatible\n+ * with fsmonitor (both hook and ipc modes). For example, if the working\n+ * directory is on a remote volume and mounted via a technology that does\n+ * not support notification events.\n+ *\n+ * fsm_os__* routines should considered private to fsm_settings__\n+ * routines.\n+ */\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);\n+#endif /* HAVE_FSMONITOR_OS_SETTINGS */\n+\n #endif /* FSMONITOR_SETTINGS_H */\n-- \ngitgitgadget\n\n"},{"id":"448441","messageId":"44cc61e186cb65fa6b2c1d5a0f080fc0b2265e57.1644940773.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 05/23] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:15Z","receivedAt":"2022-02-15T15:59:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a worktree, so there is nothing for the\ndaemon watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 11 +++++++\n builtin/update-index.c | 9 ++++++\n fsmonitor-settings.c | 62 +++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 11 +++++++\n t/t7519-status-fsmonitor.sh | 26 ++++++++++++++++\n 5 files changed, 119 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 6011fe42ee0..899355c55aa 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1424,6 +1424,17 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\tfsm_settings__set_ipc(the_repository);\n+\n+\tif (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {\n+\t\tstruct strbuf buf_reason = STRBUF_INIT;\n+\t\tfsm_settings__get_reason(the_repository, &buf_reason);\n+\t\terror(\"%s '%s'\", buf_reason.buf, xgetcwd());\n+\t\tstrbuf_release(&buf_reason);\n+\t\treturn -1;\n+\t}\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 1232f0d214e..9a2c45f2e56 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1215,6 +1215,15 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\t\tif (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {\n+\t\t\tstruct strbuf buf_reason = STRBUF_INIT;\n+\t\t\tfsm_settings__get_reason(r, &buf_reason);\n+\t\t\terror(\"%s\", buf_reason.buf);\n+\t\t\tstrbuf_release(&buf_reason);\n+\t\t\treturn -1;\n+\t\t}\n+\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\tadvise(_(\"core.fsmonitor is unset; \"\n \t\t\t\t \"set it if you really want to \"\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex eb4d661c81e..0fc5566eb8a 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -9,9 +9,33 @@\n */\n struct fsmonitor_settings {\n \tenum fsmonitor_mode mode;\n+\tenum fsmonitor_reason reason;\n \tchar *hook_path;\n };\n \n+static void set_incompatible(struct repository *r,\n+\t\t\t enum fsmonitor_reason reason)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\ts->reason = reason;\n+}\n+\n+static int check_for_incompatible(struct repository *r)\n+{\n+\tif (!r->worktree) {\n+\t\t/*\n+\t\t * Bare repositories don't have a working directory and\n+\t\t * therefore have nothing to watch.\n+\t\t */\n+\t\tset_incompatible(r, FSMONITOR_REASON_BARE);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static void lookup_fsmonitor_settings(struct repository *r)\n {\n \tstruct fsmonitor_settings *s;\n@@ -78,6 +102,9 @@ void fsm_settings__set_ipc(struct repository *r)\n {\n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n@@ -86,6 +113,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path)\n {\n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n@@ -96,5 +126,37 @@ void fsm_settings__set_disabled(struct repository *r)\n \tlookup_fsmonitor_settings(r);\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_ZERO;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n+\n+static void create_reason_message(struct repository *r,\n+\t\t\t\t struct strbuf *buf_reason)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\tswitch (s->reason) {\n+\tcase FSMONITOR_REASON_ZERO:\n+\t\treturn;\n+\n+\tcase FSMONITOR_REASON_BARE:\n+\t\tstrbuf_addstr(buf_reason,\n+\t\t\t _(\"bare repos are incompatible with fsmonitor\"));\n+\t\treturn;\n+\n+\tdefault:\n+\t\tBUG(\"Unhandled case in create_reason_message '%d'\", s->reason);\n+\t}\n+}\n+\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n+\t\t\t\t\t struct strbuf *buf_reason)\n+{\n+\tlookup_fsmonitor_settings(r);\n+\n+\tstrbuf_reset(buf_reason);\n+\tif (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)\n+\t\tcreate_reason_message(r, buf_reason);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a4c5d7b4889..e5f145a2f79 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -4,17 +4,28 @@\n struct repository;\n \n enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n \tFSMONITOR_MODE_DISABLED = 0,\n \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n };\n \n+/*\n+ * Incompatibility reasons.\n+ */\n+enum fsmonitor_reason {\n+\tFSMONITOR_REASON_ZERO = 0,\n+\tFSMONITOR_REASON_BARE = 1,\n+};\n+\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n+\t\t\t\t\t struct strbuf *buf_reason);\n \n struct fsmonitor_settings;\n \ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex f488d930dfd..3c4e6f5f89c 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -55,6 +55,32 @@ test_lazy_prereq UNTRACKED_CACHE '\n \ttest $ret -ne 1\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n+\tgit init --bare bare-clone &&\n+\tcat >expect <<-\\EOF &&\n+\terror: bare repos are incompatible with fsmonitor\n+\tEOF\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\ttest_cmp expect actual &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n+\tgit init --bare bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"bare repos are incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"448442","messageId":"e5511ef0f8b1d20d2c3b8cce3e59161b14d9c37a.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 08/23] fsmonitor-settings: stub in macOS-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:18Z","receivedAt":"2022-02-15T15:59:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++\n config.mak.uname | 1 +\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 13 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nnew file mode 100644\nindex 00000000000..176a6f5726c\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_ZERO;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 219a04b4c66..529d58aae5d 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -159,6 +159,7 @@ ifeq ($(uname_S),Darwin)\n \tifndef NO_PTHREADS\n \tifndef NO_UNIX_SOCKETS\n \tFSMONITOR_DAEMON_BACKEND = darwin\n+\tFSMONITOR_OS_SETTINGS = darwin\n \tendif\n \tendif\n \ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 7b8ea41967b..fc70dd2fc1d 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"448443","messageId":"412fbc45868e7f2a05a03e424584b53ded4842e9.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 09/23] fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:19Z","receivedAt":"2022-02-15T16:00:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on macOS and mark them as\nincompatible with FSMonitor.\n\nWith this, `git fsmonitor--daemon run` will error out with a message\nlike it does for bare repos.\n\nClient commands, like `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 5 ++\n fsmonitor-settings.h | 1 +\n 3 files changed, 72 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex 176a6f5726c..c3b719d6fb0 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -2,8 +2,74 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n+#include \n+#include \n+\n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type (NFS, SAMBA, etc.) dictates whether notification events\n+ * are available at all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * For the builtin FSMonitor, we create the Unix domain socket for the\n+ * IPC in the .git directory. If the working directory is remote,\n+ * then the socket will be created on the remote file system. This\n+ * can fail if the remote file system does not support UDS file types\n+ * (e.g. smbfs to a Windows server) or if the remote kernel does not\n+ * allow a non-local process to bind() the socket. (These problems\n+ * could be fixed by moving the UDS out of the .git directory and to a\n+ * well-known local directory on the client machine, but care should\n+ * be taken to ensure that $HOME is actually local and not a managed\n+ * file share.)\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ */\n+static enum fsmonitor_reason is_remote(struct repository *r)\n+{\n+\tstruct statfs fs;\n+\n+\tif (statfs(r->worktree, &fs) == -1) {\n+\t\tint saved_errno = errno;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n+\t\t\t\t r->worktree, strerror(saved_errno));\n+\t\terrno = saved_errno;\n+\t\treturn FSMONITOR_REASON_ZERO;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n+\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n+\n+\tif (!(fs.f_flags & MNT_LOCAL))\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\n+\treturn FSMONITOR_REASON_ZERO;\n+}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = is_remote(r);\n+\tif (reason)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_ZERO;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex bb2ddd2457f..de69ace246a 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -161,6 +161,11 @@ static void create_reason_message(struct repository *r,\n \t\t\t _(\"virtual repos are incompatible with fsmonitor\"));\n \t\treturn;\n \n+\tcase FSMONITOR_REASON_REMOTE:\n+\t\tstrbuf_addstr(buf_reason,\n+\t\t\t _(\"remote repos are incompatible with fsmonitor\"));\n+\t\treturn;\n+\n \tdefault:\n \t\tBUG(\"Unhandled case in create_reason_message '%d'\", s->reason);\n \t}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex c169683bf2d..fca25887c0f 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -17,6 +17,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_ZERO = 0,\n \tFSMONITOR_REASON_BARE = 1,\n \tFSMONITOR_REASON_VIRTUAL = 2,\n+\tFSMONITOR_REASON_REMOTE = 3,\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n-- \ngitgitgadget\n\n"},{"id":"448444","messageId":"ae09fb10c3ad5ea129c88addd07ebe674a22793d.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 10/23] fsmonitor-settings: remote repos on Windows are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:20Z","receivedAt":"2022-02-15T16:00:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on Windows and mark them as\nincompatible with FSMonitor.\n\nWith this `git fsmonitor--daemon run` will error out with a message like it\ndoes for bare repos.\n\nClient commands, such as `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++\n 1 file changed, 102 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 7caa79570af..77156c0ca20 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n@@ -23,6 +24,103 @@ static enum fsmonitor_reason is_virtual(struct repository *r)\n \treturn FSMONITOR_REASON_ZERO;\n }\n \n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type dictates whether notification events are available at\n+ * all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ *\n+ * Notes for testing:\n+ *\n+ * (a) Windows allows a network share to be mapped to a drive letter.\n+ * (This is the normal method to access it.)\n+ *\n+ * $ NET USE Z: \\\\server\\share\n+ * $ git -C Z:/repo status\n+ *\n+ * (b) Windows allows a network share to be referenced WITHOUT mapping\n+ * it to drive letter.\n+ *\n+ * $ NET USE \\\\server\\share\\dir\n+ * $ git -C //server/share/repo status\n+ *\n+ * (c) Windows allows \"SUBST\" to create a fake drive mapping to an\n+ * arbitrary path (which may be remote)\n+ *\n+ * $ SUBST Q: Z:\\repo\n+ * $ git -C Q:/ status\n+ *\n+ * (d) Windows allows a directory symlink to be created on a local\n+ * file system that points to a remote repo.\n+ *\n+ * $ mklink /d ./link //server/share/repo\n+ * $ git -C ./link status\n+ */\n+static enum fsmonitor_reason is_remote(struct repository *r)\n+{\n+\twchar_t wpath[MAX_PATH];\n+\twchar_t wfullpath[MAX_PATH];\n+\tsize_t wlen;\n+\tUINT driveType;\n+\n+\t/*\n+\t * Do everything in wide chars because the drive letter might be\n+\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n+\t */\n+\tif (xutftowcs_path(wpath, r->worktree) < 0)\n+\t\treturn FSMONITOR_REASON_ZERO;\n+\n+\t/*\n+\t * GetDriveTypeW() requires a final slash. We assume that the\n+\t * worktree pathname points to an actual directory.\n+\t */\n+\twlen = wcslen(wpath);\n+\tif (wpath[wlen - 1] != L'\\\\' && wpath[wlen - 1] != L'/') {\n+\t\twpath[wlen++] = L'\\\\';\n+\t\twpath[wlen] = 0;\n+\t}\n+\n+\t/*\n+\t * Normalize the path. If nothing else, this converts forward\n+\t * slashes to backslashes. This is essential to get GetDriveTypeW()\n+\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n+\t */\n+\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n+\t\treturn FSMONITOR_REASON_ZERO;\n+\n+\tdriveType = GetDriveTypeW(wfullpath);\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"DriveType '%s' L'%ls' (%u)\",\n+\t\t\t r->worktree, wfullpath, driveType);\n+\n+\tif (driveType == DRIVE_REMOTE) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"is_remote('%s') true\",\n+\t\t\t\t r->worktree);\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_ZERO;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n \tif (reason)\n \t\treturn reason;\n \n+\treason = is_remote(r);\n+\tif (reason)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_ZERO;\n }\n-- \ngitgitgadget\n\n"},{"id":"448445","messageId":"71babe7243a312c024e6bc6f7c54357cac12c9b1.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 12/23] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:22Z","receivedAt":"2022-02-15T16:00:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIgnore FSEvents resulting from `xattr` changes. Git does not care about\nxattr's or changes to xattr's, so don't waste time collecting these\nevents in the daemon nor transmitting them to clients.\n\nVarious security tools add xattrs to files and/or directories, such as\nto mark them as having been downloaded. We should ignore these events\nsince it doesn't affect the content of the file/directory or the normal\nmeta-data that Git cares about.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 2aefdc14d89..79d08517d7b 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -172,7 +172,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tif (flag & kFSEventStreamEventFlagItemCloned)\n \t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n \n-\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n \t\t\t path, flag, msg.buf);\n \n \tstrbuf_release(&msg);\n@@ -197,6 +197,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)\n \t\tef & kFSEventStreamEventFlagUserDropped);\n }\n \n+/*\n+ * If an `xattr` change is the only reason we received this event,\n+ * then silently ignore it. Git doesn't care about xattr's. We\n+ * have to be careful here because the kernel can combine multiple\n+ * events for a single path. And because events always have certain\n+ * bits set, such as `ItemIsFile` or `ItemIsDir`.\n+ *\n+ * Return 1 if we should ignore it.\n+ */\n+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n+{\n+\tstatic const FSEventStreamEventFlags mask =\n+\t\tkFSEventStreamEventFlagItemChangeOwner |\n+\t\tkFSEventStreamEventFlagItemCreated |\n+\t\tkFSEventStreamEventFlagItemFinderInfoMod |\n+\t\tkFSEventStreamEventFlagItemInodeMetaMod |\n+\t\tkFSEventStreamEventFlagItemModified |\n+\t\tkFSEventStreamEventFlagItemRemoved |\n+\t\tkFSEventStreamEventFlagItemRenamed |\n+\t\tkFSEventStreamEventFlagItemXattrMod |\n+\t\tkFSEventStreamEventFlagItemCloned;\n+\n+\treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n+}\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -262,6 +287,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_ignore_xattr(event_flags[k])) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n+\t\t\t\t\t path_k, event_flags[k]);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n \n \t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-- \ngitgitgadget\n\n"},{"id":"448446","messageId":"be1672e32b24e45dd0916c92d10f077e3673ec2c.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 11/23] unpack-trees: initialize fsmonitor_has_run_once in o->result","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:21Z","receivedAt":"2022-02-15T16:00:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInitialize `o->result.fsmonitor_has_run_once` based upon value\nin `o->src_index->fsmonitor_has_run_once` to prevent a second\nfsmonitor query during the tree traversal and possibly getting\na skewed view of the working directory.\n\nThe checkout code has already talked to the fsmonitor and the\ntraversal is updating the index as it traverses, so there is\nno need to query the fsmonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n unpack-trees.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a7e1712d236..13e2e5de82c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1756,6 +1756,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \to->result.fsmonitor_last_update =\n \t\txstrdup_or_null(o->src_index->fsmonitor_last_update);\n+\to->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;\n \n \t/*\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n-- \ngitgitgadget\n\n"},{"id":"448447","messageId":"f19671f7defef336bdc567ae06da9d2e46c805b8.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 13/23] fsmonitor--daemon: print start message only if fsmonitor.announceStartup","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:23Z","receivedAt":"2022-02-15T16:00:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor--daemon to print a startup message only when\n`fsmonitor.announceStartup` is true. This setting is false by default\nso that the output of client commands, like `git status`, is not\nchanged if the daemon is implicitly started.\n\nThe message is conditionally printed by \"run\" and \"start\" subcommands\nand is sent to stderr. It contains the path to the work tree root.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 29 +++++++++++++++++++++++------\n 1 file changed, 23 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 899355c55aa..dd0561cfc51 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -27,6 +27,9 @@ static int fsmonitor__ipc_threads = 8;\n #define FSMONITOR__START_TIMEOUT \"fsmonitor.starttimeout\"\n static int fsmonitor__start_timeout_sec = 60;\n \n+#define FSMONITOR__ANNOUNCE_STARTUP \"fsmonitor.announcestartup\"\n+static int fsmonitor__announce_startup = 0;\n+\n static int fsmonitor_config(const char *var, const char *value, void *cb)\n {\n \tif (!strcmp(var, FSMONITOR__IPC_THREADS)) {\n@@ -47,6 +50,16 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {\n+\t\tint is_bool;\n+\t\tint i = git_config_bool_or_int(var, value, &is_bool);\n+\t\tif (i < 0)\n+\t\t\treturn error(_(\"value of '%s' not bool or int: %d\"),\n+\t\t\t\t var, i);\n+\t\tfsmonitor__announce_startup = i;\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -1307,9 +1320,11 @@ static int try_to_run_foreground_daemon(int free_console)\n \t\tdie(\"fsmonitor--daemon is already running '%s'\",\n \t\t the_repository->worktree);\n \n-\tprintf(_(\"running fsmonitor-daemon in '%s'\\n\"),\n-\t the_repository->worktree);\n-\tfflush(stdout);\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"running fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n \n #ifdef GIT_WINDOWS_NATIVE\n \tif (free_console)\n@@ -1360,9 +1375,11 @@ static int try_to_start_background_daemon(void)\n \t\tdie(\"fsmonitor--daemon is already running '%s'\",\n \t\t the_repository->worktree);\n \n-\tprintf(_(\"starting fsmonitor-daemon in '%s'\\n\"),\n-\t the_repository->worktree);\n-\tfflush(stdout);\n+\tif (fsmonitor__announce_startup) {\n+\t\tfprintf(stderr, _(\"starting fsmonitor-daemon in '%s'\\n\"),\n+\t\t\tthe_repository->worktree);\n+\t\tfflush(stderr);\n+\t}\n \n \tcp.git_cmd = 1;\n \n-- \ngitgitgadget\n\n"},{"id":"448448","messageId":"95cdff22be0b0ee6f364dfed031836ed2327cadb.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 14/23] fsmonitor--daemon: cd out of worktree root","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:24Z","receivedAt":"2022-02-15T16:00:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the fsmonitor--daemon to CD outside of the worktree\nbefore starting up.\n\nThe common Git startup mechanism causes the CWD of the daemon process\nto be in the root of the worktree. On Windows, this causes the daemon\nprocess to hold a locked handle on the CWD and prevents other\nprocesses from moving or deleting the worktree while the daemon is\nrunning.\n\nCD to HOME before entering main event loops.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++--\n compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------\n fsmonitor--daemon.h | 1 +\n 3 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex dd0561cfc51..e9b3f44d791 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1169,11 +1169,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * before we need it.\n \t */\n \tif (ipc_server_run_async(&state->ipc_server_data,\n-\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t state->path_ipc.buf, &ipc_opts,\n \t\t\t\t handle_client, state))\n \t\treturn error_errno(\n \t\t\t_(\"could not start IPC thread pool on '%s'\"),\n-\t\t\tfsmonitor_ipc__get_path());\n+\t\t\tstate->path_ipc.buf);\n \n \t/*\n \t * Start the fsmonitor listener thread to collect filesystem\n@@ -1208,6 +1208,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n static int fsmonitor_run_daemon(void)\n {\n \tstruct fsmonitor_daemon_state state;\n+\tconst char *home;\n \tint err;\n \n \tmemset(&state, 0, sizeof(state));\n@@ -1277,6 +1278,15 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_addch(&state.path_cookie_prefix, '/');\n \n+\t/*\n+\t * We create a named-pipe or unix domain socket inside of the\n+\t * \".git\" directory. (Well, on Windows, we base our named\n+\t * pipe in the NPFS on the absolute path of the git\n+\t * directory.)\n+\t */\n+\tstrbuf_init(&state.path_ipc, 0);\n+\tstrbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1286,6 +1296,23 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\t/*\n+\t * CD out of the worktree root directory.\n+\t *\n+\t * The common Git startup mechanism causes our CWD to be the\n+\t * root of the worktree. On Windows, this causes our process\n+\t * to hold a locked handle on the CWD. This prevents the\n+\t * worktree from being moved or deleted while the daemon is\n+\t * running.\n+\t *\n+\t * We assume that our FS and IPC listener threads have either\n+\t * opened all of the handles that they need or will do\n+\t * everything using absolute paths.\n+\t */\n+\thome = getenv(\"HOME\");\n+\tif (home && *home && chdir(home))\n+\t\tdie_errno(\"could not cd home '%s'\", home);\n+\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n@@ -1298,6 +1325,7 @@ done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n \tstrbuf_release(&state.path_cookie_prefix);\n+\tstrbuf_release(&state.path_ipc);\n \n \t/*\n \t * NEEDSWORK: Consider \"rm -rf /\"\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex eb407b0748c..6985903c95b 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -398,12 +398,22 @@ static int recv_rdcw_watch(struct one_watch *watch)\n \t}\n \n \t/*\n-\t * NEEDSWORK: If an external is deleted, the above\n-\t * returns an error. I'm not sure that there's anything that\n-\t * we can do here other than failing -- the /.git\n-\t * link file would be broken anyway. We might try to check\n-\t * for that and return a better error message, but I'm not\n-\t * sure it is worth it.\n+\t * GetOverlappedResult() fails if the watched directory is\n+\t * deleted while we were waiting for an overlapped IO to\n+\t * complete. The documentation did not list specific errors,\n+\t * but I observed ERROR_ACCESS_DENIED (0x05) errors during\n+\t * testing.\n+\t *\n+\t * Note that we only get notificaiton events for events\n+\t * *within* the directory, not *on* the directory itself.\n+\t * (These might be properies of the parent directory, for\n+\t * example).\n+\t *\n+\t * NEEDSWORK: We might try to check for the deleted directory\n+\t * case and return a better error message, but I'm not sure it\n+\t * is worth it.\n+\t *\n+\t * Shutdown if we get any error.\n \t */\n \n \terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex c16ef095688..a777c3a0590 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n+\tstruct strbuf path_ipc;\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"448449","messageId":"6b5642f7770d95f318eb04066e0291072bd8814e.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 15/23] fsmonitor--daemon: prepare for adding health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:25Z","receivedAt":"2022-02-15T16:00:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor daemon thread startup to make it easier to start\na third thread class to monitor the health of the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex e9b3f44d791..f42fb2ab626 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1162,6 +1162,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint listener_started = 0;\n+\tint err = 0;\n \n \t/*\n \t * Start the IPC thread pool before the we've started the file\n@@ -1182,15 +1184,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \tif (pthread_create(&state->listener_thread, NULL,\n \t\t\t fsm_listen__thread_proc, state) < 0) {\n \t\tipc_server_stop_async(state->ipc_server_data);\n-\t\tipc_server_await(state->ipc_server_data);\n-\n-\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t\terr = error(_(\"could not start fsmonitor listener thread\"));\n+\t\tgoto cleanup;\n \t}\n+\tlistener_started = 1;\n \n \t/*\n \t * The daemon is now fully functional in background threads.\n+\t * Our primary thread should now just wait while the threads\n+\t * do all the work.\n+\t */\n+cleanup:\n+\t/*\n \t * Wait for the IPC thread pool to shutdown (whether by client\n-\t * request or from filesystem activity).\n+\t * request, from filesystem activity, or an error).\n \t */\n \tipc_server_await(state->ipc_server_data);\n \n@@ -1199,10 +1206,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * event from the IPC thread pool, but it doesn't hurt to tell\n \t * it again. And wait for it to shutdown.\n \t */\n-\tfsm_listen__stop_async(state);\n-\tpthread_join(state->listener_thread, NULL);\n+\tif (listener_started) {\n+\t\tfsm_listen__stop_async(state);\n+\t\tpthread_join(state->listener_thread, NULL);\n+\t}\n \n-\treturn state->error_code;\n+\tif (err)\n+\t\treturn err;\n+\tif (state->error_code)\n+\t\treturn state->error_code;\n+\treturn 0;\n }\n \n static int fsmonitor_run_daemon(void)\n-- \ngitgitgadget\n\n"},{"id":"448450","messageId":"63c502c20bd995636d551d7d24d5885c40bfda19.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 16/23] fsmonitor--daemon: rename listener thread related variables","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:26Z","receivedAt":"2022-02-15T16:00:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRename platform-specific listener thread related variables\nand data types as we prepare to add another backend thread\ntype.\n\n[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`\n[] `state->backend_data` becomes `state->listen_data`\n[] `state->error_code` becomes `state->listen_error_code`\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 +++---\n compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------\n compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++-------------\n compat/fsmonitor/fsm-listen.h | 2 +-\n fsmonitor--daemon.h | 6 +++---\n 5 files changed, 36 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex f42fb2ab626..f3fb865a9d8 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1213,8 +1213,8 @@ cleanup:\n \n \tif (err)\n \t\treturn err;\n-\tif (state->error_code)\n-\t\treturn state->error_code;\n+\tif (state->listen_error_code)\n+\t\treturn state->listen_error_code;\n \treturn 0;\n }\n \n@@ -1229,7 +1229,7 @@ static int fsmonitor_run_daemon(void)\n \thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n-\tstate.error_code = 0;\n+\tstate.listen_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 79d08517d7b..87a8476b09f 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -99,7 +99,7 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tCFStringRef cfsr_worktree_path;\n \tCFStringRef cfsr_gitdir_path;\n@@ -230,7 +230,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t const FSEventStreamEventId event_ids[])\n {\n \tstruct fsmonitor_daemon_state *state = ctx;\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tchar **paths = (char **)event_paths;\n \tstruct fsmonitor_batch *batch = NULL;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -419,11 +419,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tNULL,\n \t\tNULL\n \t};\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \tconst void *dir_array[2];\n \n \tCALLOC_ARRAY(data, 1);\n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \n \tdata->cfsr_worktree_path = CFStringCreateWithCString(\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n@@ -455,18 +455,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n \terror(\"Unable to create FSEventStream.\");\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tif (data->stream) {\n \t\tif (data->stream_started)\n@@ -476,14 +476,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n \t\tFSEventStreamRelease(data->stream);\n \t}\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \tdata->shutdown_style = SHUTDOWN_EVENT;\n \n \tCFRunLoopStop(data->rl);\n@@ -491,9 +491,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tdata->rl = CFRunLoopGetCurrent();\n \n@@ -510,7 +510,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \n \tswitch (data->shutdown_style) {\n \tcase FORCE_ERROR_STOP:\n-\t\tstate->error_code = -1;\n+\t\tstate->listen_error_code = -1;\n \t\t/* fall thru */\n \tcase FORCE_SHUTDOWN:\n \t\tipc_server_stop_async(state->ipc_server_data);\n@@ -522,7 +522,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \treturn;\n \n force_error_stop_without_loop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \tipc_server_stop_async(state->ipc_server_data);\n \treturn;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 6985903c95b..8406f8277dc 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -54,7 +54,7 @@ struct one_watch\n \twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tstruct one_watch *watch_worktree;\n \tstruct one_watch *watch_gitdir;\n@@ -259,7 +259,7 @@ static enum get_relative_result get_relative_longname(\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+\tSetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);\n }\n \n static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n@@ -333,7 +333,7 @@ static void destroy_watch(struct one_watch *watch)\n \tfree(watch);\n }\n \n-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+static int start_rdcw_watch(struct fsm_listen_data *data,\n \t\t\t struct one_watch *watch)\n {\n \tDWORD dwNotifyFilter =\n@@ -512,7 +512,7 @@ static int process_1_worktree_event(\n */\n static int process_worktree_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_worktree;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -642,7 +642,7 @@ force_shutdown:\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_gitdir;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -700,11 +700,11 @@ skip_this_path:\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tDWORD dwWait;\n \tint result;\n \n-\tstate->error_code = 0;\n+\tstate->listen_error_code = 0;\n \n \tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n \t\tgoto force_error_stop;\n@@ -769,7 +769,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \t}\n \n force_error_stop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \n force_shutdown:\n \t/*\n@@ -786,7 +786,7 @@ clean_shutdown:\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -819,7 +819,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \treturn 0;\n \n failed:\n@@ -832,16 +832,16 @@ failed:\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tCloseHandle(data->hEventShutdown);\n \tdestroy_watch(data->watch_worktree);\n \tdestroy_watch(data->watch_gitdir);\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nindex f0539349baf..41650bf8972 100644\n--- a/compat/fsmonitor/fsm-listen.h\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n * do so if the listener thread receives a normal shutdown signal from\n * the IPC layer.)\n *\n- * It should set `state->error_code` to -1 if the daemon should exit\n+ * It should set `state->listen_error_code` to -1 if the daemon should exit\n * with an error.\n */\n void fsm_listen__loop(struct fsmonitor_daemon_state *state);\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex a777c3a0590..f7de7882517 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n */\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {\n \tint cookie_seq;\n \tstruct hashmap cookies;\n \n-\tint error_code;\n-\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\tint listen_error_code;\n+\tstruct fsm_listen_data *listen_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"448451","messageId":"023fcd6e2b1163ab3d23b0d5933c14586d814ce0.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 19/23] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:29Z","receivedAt":"2022-02-15T16:00:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nForce shutdown fsmonitor daemon if the worktree root directory\nis moved, renamed, or deleted.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 133 ++++++++++++++++++++++++++++\n 1 file changed, 133 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 3c3453369cd..2526ad9194f 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -14,7 +14,10 @@ enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };\n typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n \t\t\t enum interval_fn_ctx ctx);\n \n+static interval_fn has_worktree_moved;\n+\n static interval_fn *table[] = {\n+\thas_worktree_moved,\n \tNULL, /* must be last */\n };\n \n@@ -45,6 +48,12 @@ struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n \tint nr_handles; /* number of active event handles */\n+\n+\tstruct wt_moved\n+\t{\n+\t\twchar_t wpath[MAX_PATH + 1];\n+\t\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\t} wt_moved;\n };\n \n int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n@@ -76,6 +85,130 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n \tFREE_AND_NULL(state->health_data);\n }\n \n+static int lookup_bhfi(wchar_t *wpath,\n+\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n+{\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileW(wpath, desired_access, share_mode, NULL,\n+\t\t\t OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] health thread could not open '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\treturn -1;\n+\t}\n+\n+\tif (!GetFileInformationByHandle(hDir, bhfi)) {\n+\t\terror(_(\"[GLE %ld] health thread getting BHFI for '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\tCloseHandle(hDir);\n+\t\treturn -1;\n+\t}\n+\n+\tCloseHandle(hDir);\n+\treturn 0;\n+}\n+\n+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n+\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n+{\n+\treturn (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&\n+\t\tbhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&\n+\t\tbhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);\n+}\n+\n+/*\n+ * Shutdown if the original worktree root directory been deleted,\n+ * moved, or renamed?\n+ *\n+ * Since the main thread did a \"chdir(getenv($HOME))\" and our CWD\n+ * is not in the worktree root directory and because the listener\n+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible\n+ * for the root directory to be moved or deleted while we are still\n+ * watching it. We want to detect that here and force a shutdown.\n+ *\n+ * Granted, a delete MAY cause some operations to fail, such as\n+ * GetOverlappedResult(), but it is not guaranteed. And because\n+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the\n+ * directory, not changes *ON* the directory, our watch will not\n+ * receive a delete event for it.\n+ *\n+ * A move/rename of the worktree root will also not generate an event.\n+ * And since the listener thread already has an open handle, it may\n+ * continue to receive events for events within the directory.\n+ * However, the pathname of the named-pipe was constructed using the\n+ * original location of the worktree root. (Remember named-pipes are\n+ * stored in the NPFS and not in the actual file system.) Clients\n+ * trying to talk to the worktree after the move/rename will not\n+ * reach our daemon process, since we're still listening on the\n+ * pipe with original path.\n+ *\n+ * Furthermore, if the user does something like:\n+ *\n+ * $ mv repo repo.old\n+ * $ git init repo\n+ *\n+ * A new daemon cannot be started in the new instance of \"repo\"\n+ * because the named-pipe is still being used by the daemon on\n+ * the original instance.\n+ *\n+ * So, detect move/rename/delete and shutdown. This should also\n+ * handle unsafe drive removal.\n+ *\n+ * We use the file system unique ID to distinguish the original\n+ * directory instance from a new instance and force a shutdown\n+ * if the unique ID changes.\n+ *\n+ * Since a worktree move/rename/delete/unmount doesn't happen\n+ * that often (and we can't get an immediate event anyway), we\n+ * use a timeout and periodically poll it.\n+ */\n+static int has_worktree_moved(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\tint r;\n+\n+\tswitch (ctx) {\n+\tcase CTX_TERM:\n+\t\treturn 0;\n+\n+\tcase CTX_INIT:\n+\t\tif (xutftowcs_path(data->wt_moved.wpath,\n+\t\t\t\t state->path_worktree_watch.buf) < 0) {\n+\t\t\terror(_(\"could not convert to wide characters: '%s'\"),\n+\t\t\t state->path_worktree_watch.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/*\n+\t\t * On the first call we lookup the unique sequence ID for\n+\t\t * the worktree root directory.\n+\t\t */\n+\t\treturn lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);\n+\n+\tcase CTX_TIMER:\n+\t\tr = lookup_bhfi(data->wt_moved.wpath, &bhfi);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t\tif (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {\n+\t\t\terror(_(\"BHFI changed '%ls'\"), data->wt_moved.wpath);\n+\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tdie(\"unhandled case in 'has_worktree_moved': %d\",\n+\t\t (int)ctx);\n+\t}\n+\n+\treturn 0;\n+}\n+\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n-- \ngitgitgadget\n\n"},{"id":"448452","messageId":"f065e8c9a90d611888d8aa5ec8753c394f525e8e.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 22/23] t7527: FSMonitor tests for directory moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:32Z","receivedAt":"2022-02-15T16:00:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate unit tests to move a directory. Verify that `git status`\ngives the same result with and without FSMonitor enabled.\n\nNEEDSWORK: This test exposes a bug in the untracked-cache on\nWindows when FSMonitor is disabled. These are commented out\nfor the moment.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 39 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex c0145544ccb..408d614b28b 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -261,6 +261,16 @@ test_expect_success 'setup' '\n \ttrace*\n \tEOF\n \n+\tmkdir -p T1/T2/T3/T4 &&\n+\techo 1 >T1/F1 &&\n+\techo 1 >T1/T2/F1 &&\n+\techo 1 >T1/T2/T3/F1 &&\n+\techo 1 >T1/T2/T3/T4/F1 &&\n+\techo 2 >T1/F2 &&\n+\techo 2 >T1/T2/F2 &&\n+\techo 2 >T1/T2/T3/F2 &&\n+\techo 2 >T1/T2/T3/T4/F2 &&\n+\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n@@ -354,6 +364,19 @@ verify_status () {\n \techo HELLO AFTER\n }\n \n+move_directory_contents_deeper() {\n+\tmkdir T1/_new_\n+\tmv T1/[A-Z]* T1/_new_\n+}\n+\n+move_directory_up() {\n+\tmv T1/T2/T3 T1\n+}\n+\n+move_directory() {\n+\tmv T1/T2/T3 T1/T2/NewT3\n+}\n+\n # The next few test cases confirm that our fsmonitor daemon sees each type\n # of OS filesystem notification that we care about. At this layer we just\n # ensure we are getting the OS notifications and do not try to confirm what\n@@ -685,6 +708,22 @@ do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n+\t\t# NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor\n+\t\t# is DISABLED. Turn off a few test that cause it problems until\n+\t\t# we can debug it.\n+\t\t#\n+\t\ttry_moves=\"true\"\n+\t\ttest_have_prereq UNTRACKED_CACHE,WINDOWS && \\\n+\t\t\ttest $uc_val = true && \\\n+\t\t\ttest $fsm_val = false && \\\n+\t\t\ttry_moves=\"false\"\n+\t\tif test $try_moves = true\n+\t\tthen\n+\t\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n+\t\t\tmatrix_try $uc_val $fsm_val move_directory_up\n+\t\t\tmatrix_try $uc_val $fsm_val move_directory\n+\t\tfi\n+\n \t\tif test $fsm_val = true\n \t\tthen\n \t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n-- \ngitgitgadget\n\n"},{"id":"448453","messageId":"496b3da35d0f9b8c68c6bc6c2bf2ca93ffe224b9.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 20/23] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:30Z","receivedAt":"2022-02-15T16:00:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the listener thread to shutdown the daemon if the spelling of the\nworktree root directory changes.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 87a8476b09f..d3afbbc53d6 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -178,6 +178,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tstrbuf_release(&msg);\n }\n \n+static int ef_is_root_changed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagRootChanged);\n+}\n+\n static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n {\n \treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n@@ -287,6 +292,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_is_root_changed(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * The spelling of the pathname of the root directory\n+\t\t\t * has changed. This includes the name of the root\n+\t\t\t * directory itself of of any parent directory in the\n+\t\t\t * path.\n+\t\t\t *\n+\t\t\t * (There may be other conditions that throw this,\n+\t\t\t * but I couldn't find any information on it.)\n+\t\t\t *\n+\t\t\t * Force a shutdown now and avoid things getting\n+\t\t\t * out of sync. The Unix domain socket is inside\n+\t\t\t * the .git directory and a spelling change will make\n+\t\t\t * it hard for clients to rendezvous with us.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"event: root changed\");\n+\t\t\tgoto force_shutdown;\n+\t\t}\n+\n \t\tif (ef_ignore_xattr(event_flags[k])) {\n \t\t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n-- \ngitgitgadget\n\n"},{"id":"448454","messageId":"e5a12832afaa69485a68fc7920b55e60d7872a22.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 23/23] t/perf/p7527: add perf test for builtin FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:33Z","receivedAt":"2022-02-15T16:00:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++\n 1 file changed, 257 insertions(+)\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\ndiff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..9338b9ea008\n--- /dev/null\n+++ b/t/perf/p7527-builtin-fsmonitor.sh\n@@ -0,0 +1,257 @@\n+#!/bin/sh\n+\n+test_description=\"Perf test for the builtin FSMonitor\"\n+\n+. ./perf-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+# Lie to perf-lib and ask for a new empty repo and avoid\n+# the complaints about GIT_PERF_REPO not being big enough\n+# the perf hit when GIT_PERF_LARGE_REPO is copied into\n+# the trash directory.\n+#\n+# NEEDSWORK: It would be nice if perf-lib had an option to\n+# \"borrow\" an existing large repo (especially for gigantic\n+# monorepos) and use it in-place. For now, fake it here.\n+#\n+test_perf_fresh_repo\n+\n+\n+# Use a generated synthetic monorepo. If it doesn't exist, we will\n+# generate it. If it does exist, we will put it in a known state\n+# before we start our timings.\n+#\n+PARAM_D=5\n+PARAM_W=10\n+PARAM_F=9\n+\n+PARAMS=\"$PARAM_D\".\"$PARAM_W\".\"$PARAM_F\"\n+\n+BALLAST_BR=p0006-ballast\n+export BALLAST_BR\n+\n+TMP_BR=tmp_br\n+export TMP_BR\n+\n+REPO=../repos/gen-many-files-\"$PARAMS\".git\n+export REPO\n+\n+if ! test -d $REPO\n+then\n+\t(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)\n+fi\n+\n+\n+enable_uc () {\n+\tgit -C $REPO config core.untrackedcache true\n+\tgit -C $REPO update-index --untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+disable_uc () {\n+\tgit -C $REPO config core.untrackedcache false\n+\tgit -C $REPO update-index --no-untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+start_fsm () {\n+\tgit -C $REPO fsmonitor--daemon start\n+\tgit -C $REPO fsmonitor--daemon status\n+\tgit -C $REPO config core.fsmonitor true\n+\tgit -C $REPO update-index --fsmonitor\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+stop_fsm () {\n+\tgit -C $REPO config --unset core.fsmonitor\n+\tgit -C $REPO update-index --no-fsmonitor\n+\ttest_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+\n+# Ensure that FSMonitor is turned off on the borrowed repo.\n+#\n+test_expect_success \"Setup borrowed repo (fsm+uc)\" \"\n+\tstop_fsm &&\n+\tdisable_uc\n+\"\n+\n+# Also ensure that it starts in a known state.\n+#\n+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time\n+# the ballast checkout, since only the first invocation does any work and the\n+# subsequent ones just print \"already on branch\" and quit, so the reported\n+# time is not useful.\n+#\n+# Create a temp branch and do all work relative to it so that we don't\n+# accidentially alter the real ballast branch.\n+#\n+test_expect_success \"Setup borrowed repo (temp ballast branch)\" \"\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO reset --hard &&\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tgit -C $REPO branch $TMP_BR $BALLAST_BR &&\n+\tgit -C $REPO checkout $TMP_BR\n+\"\n+\n+\n+echo Data >data.txt\n+\n+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With\n+# FSMonitor enabled, we can get a skewed view of status times, since\n+# the index MAY (or may not) be updated after the first invocation\n+# which will update the FSMonitor Token, so the subsequent invocations\n+# may get a smaller response from the daemon.\n+#\n+do_status () {\n+\tmsg=$1\n+\n+\ttest_perf \"$msg\" \"\n+\t\tgit -C $REPO status >/dev/null 2>&1\n+\t\"\n+}\n+\n+do_matrix () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tt=\"[uc $uc][fsm $fsm]\"\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_expect_success \"$t Setup matrix branch\" \"\n+\t\tgit -C $REPO clean -d -f &&\n+\t\tgit -C $REPO checkout $TMP_BR &&\n+\t\ttest_might_fail git -C $REPO branch -D $MATRIX_BR &&\n+\t\tgit -C $REPO branch $MATRIX_BR $TMP_BR &&\n+\t\tgit -C $REPO checkout $MATRIX_BR\n+\t\"\n+\n+\tif test $uc = true\n+\tthen\n+\t\tenable_uc\n+\telse\n+\t\tdisable_uc\n+\tfi\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstart_fsm\n+\telse\n+\t\tstop_fsm\n+\tfi\n+\n+\tdo_status \"$t status after checkout\"\n+\n+\t# Modify many files in the matrix branch.\n+\t# Stage them.\n+\t# Commit them.\n+\t# Rollback.\n+\t#\n+\ttest_expect_success \"$t modify tracked files\" \"\n+\t\tfind $REPO -name file1 -exec cp data.txt {} \\\\;\n+\t\"\n+\n+\tdo_status \"$t status after big change\"\n+\n+\t# Don't bother timing the \"add\" because _REPEAT_COUNT\n+\t# issue described above.\n+\t#\n+\ttest_expect_success \"$t add all\" \"\n+\t\tgit -C $REPO add -A\n+\t\"\n+\n+\tdo_status \"$t status after add all\"\n+\n+\ttest_expect_success \"$t add dot\" \"\n+\t\tgit -C $REPO add .\n+\t\"\n+\n+\tdo_status \"$t status after add dot\"\n+\n+\ttest_expect_success \"$t commit staged\" \"\n+\t\tgit -C $REPO commit -a -m data\n+\t\"\n+\n+\tdo_status \"$t status after commit\"\n+\n+\ttest_expect_success \"$t reset HEAD~1 hard\" \"\n+\t\tgit -C $REPO reset --hard HEAD~1 >/dev/null 2>&1\n+\t\"\n+\n+\tdo_status \"$t status after reset hard\"\n+\n+\t# Create some untracked files.\n+\t#\n+\ttest_expect_success \"$t create untracked files\" \"\n+\t\tcp -R $REPO/ballast/dir1 $REPO/ballast/xxx1\n+\t\"\n+\n+\tdo_status \"$t status after create untracked files\"\n+\n+\t# Remove the new untracked files.\n+\t#\n+\ttest_expect_success \"$t clean -df\" \"\n+\t\tgit -C $REPO clean -d -f\n+\t\"\n+\n+\tdo_status \"$t status after clean\"\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstop_fsm\n+\tfi\n+}\n+\n+# Begin testing each case in the matrix that we care about.\n+#\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\n+fsm_values=\"false true\"\n+\n+for uc_val in $uc_values\n+do\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tdo_matrix $uc_val $fsm_val\n+\tdone\n+done\n+\n+cleanup () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_might_fail git -C $REPO branch -D $MATRIX_BR\n+}\n+\n+\n+# We're borrowing this repo. We should leave it in a clean state.\n+#\n+test_expect_success \"Cleanup temp and matrix branches\" \"\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\tfor fsm_val in $fsm_values\n+\t\tdo\n+\t\t\tcleanup $uc_val $fsm_val\n+\t\tdone\n+\tdone\n+\"\n+\n+test_done\n-- \ngitgitgadget\n"},{"id":"448455","messageId":"07a9c7542be79fb60861253848eca258bec87694.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 21/23] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:31Z","receivedAt":"2022-02-15T16:00:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to perform binary search over the cache-entries for a directory\nnotification and then linearly scan forward to find the immediate children.\n\nPreviously, when the FSMonitor reported a modified directory Git would\nperform a linear search on the entire cache-entry array for all\nentries matching that directory prefix and invalidate them. Since the\ncache-entry array is already sorted, we can use a binary search to\nfind the first matching entry and then only linearly walk forward and\ninvalidate entries until the prefix changes.\n\nAlso, the original code would invalidate anything having the same\ndirectory prefix. Since a directory event should only be received for\nitems that are immediately within the directory (and not within\nsub-directories of it), only invalidate those entries and not the\nwhole subtree.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 54 insertions(+), 17 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 8e3499d0667..be08861a18e 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n-\tif (name[len - 1] == '/') {\n+\tint pos = index_name_pos(istate, name, len);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n+\t\t\t name, pos);\n \n+\tif (name[len - 1] == '/') {\n \t\t/*\n-\t\t * TODO We should binary search to find the first path with\n-\t\t * TODO this directory prefix. Then linearly update entries\n-\t\t * TODO while the prefix matches. Taking care to search without\n-\t\t * TODO the trailing slash -- because '/' sorts after a few\n-\t\t * TODO interesting special chars, like '.' and ' '.\n+\t\t * The daemon can decorate directory events, such as\n+\t\t * moves or renames, with a trailing slash if the OS\n+\t\t * FS Event contains sufficient information, such as\n+\t\t * MacOS.\n+\t\t *\n+\t\t * Use this to invalidate the entire cone under that\n+\t\t * directory.\n+\t\t *\n+\t\t * We do not expect an exact match because the index\n+\t\t * does not normally contain directory entries, so we\n+\t\t * start at the insertion point and scan.\n \t\t */\n+\t\tif (pos < 0)\n+\t\t\tpos = -pos - 1;\n \n \t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n-\t\t\t starts_with(istate->cache[i]->name, name))\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n-\t\t/* Need to remove the / from the path for the untracked cache */\n+\n+\t\t/*\n+\t\t * We need to remove the traling \"/\" from the path\n+\t\t * for the untracked cache.\n+\t\t */\n \t\tname[len - 1] = '\\0';\n+\t} else if (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\tint pos = index_name_pos(istate, name, strlen(name));\n-\n-\t\tif (pos >= 0) {\n-\t\t\tstruct cache_entry *ce = istate->cache[pos];\n-\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n \t}\n \n@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n \t * as it could be a new untracked file.\n \t */\n-\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"448456","messageId":"4a77f5b1fdef3af20fbfefccccc5fee194eb3422.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 17/23] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:27Z","receivedAt":"2022-02-15T16:00:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate another thread to watch over the daemon process and\nautomatically shut it down if necessary.\n\nThis commit creates the basic framework for a \"health\" thread\nto monitor the daemon and/or the file system. Later commits\nwill add platform-specific code to do the actual work.\n\nThe \"health\" thread is intended to monitor conditions that\nwould be difficult to track inside the IPC thread pool and/or\nthe file system listener threads. For example, when there are\nfile system events outside of the watched worktree root or if\nwe want to have an idle-timeout auto-shutdown feature.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 6 ++-\n builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n contrib/buildsystems/CMakeLists.txt | 2 +\n fsmonitor--daemon.h | 4 ++\n 7 files changed, 192 insertions(+), 2 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n\ndiff --git a/Makefile b/Makefile\nindex ed3d5231ea1..85a61934f21 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -467,8 +467,9 @@ all::\n #\n # If your platform supports a built-in fsmonitor backend, set\n # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n-# `compat/fsmonitor/fsm-listen-.c` that implements the\n-# `fsm_listen__*()` routines.\n+# `compat/fsmonitor/fsm-listen-.c` and\n+# `compat/fsmonitor/fsm-health-.c` files\n+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n # If your platform has os-specific ways to tell if a repo is incompatible with\n # fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS\n@@ -1944,6 +1945,7 @@ endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n ifdef FSMONITOR_OS_SETTINGS\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex f3fb865a9d8..591433e897d 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n@@ -1124,6 +1125,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+static void *fsm_health__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-health\");\n+\n+\tfsm_health__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -1162,6 +1175,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint health_started = 0;\n \tint listener_started = 0;\n \tint err = 0;\n \n@@ -1189,6 +1203,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t}\n \tlistener_started = 1;\n \n+\t/*\n+\t * Start the health thread to watch over our process.\n+\t */\n+\tif (pthread_create(&state->health_thread, NULL,\n+\t\t\t fsm_health__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\terr = error(_(\"could not start fsmonitor health thread\"));\n+\t\tgoto cleanup;\n+\t}\n+\thealth_started = 1;\n+\n \t/*\n \t * The daemon is now fully functional in background threads.\n \t * Our primary thread should now just wait while the threads\n@@ -1211,10 +1236,17 @@ cleanup:\n \t\tpthread_join(state->listener_thread, NULL);\n \t}\n \n+\tif (health_started) {\n+\t\tfsm_health__stop_async(state);\n+\t\tpthread_join(state->health_thread, NULL);\n+\t}\n+\n \tif (err)\n \t\treturn err;\n \tif (state->listen_error_code)\n \t\treturn state->listen_error_code;\n+\tif (state->health_error_code)\n+\t\treturn state->health_error_code;\n \treturn 0;\n }\n \n@@ -1230,6 +1262,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.listen_error_code = 0;\n+\tstate.health_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n@@ -1309,6 +1342,11 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\tif (fsm_health__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize health thread\"));\n+\t\tgoto done;\n+\t}\n+\n \t/*\n \t * CD out of the worktree root directory.\n \t *\n@@ -1332,6 +1370,7 @@ done:\n \tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n+\tfsm_health__dtor(&state);\n \n \tipc_server_free(state.ipc_server_data);\n \ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nnew file mode 100644\nindex 00000000000..b9f709e8548\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -0,0 +1,24 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nnew file mode 100644\nindex 00000000000..94b1d020f25\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsm_health_data\n+{\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hHandles[1]; /* the array does not own these handles */\n+#define HEALTH_SHUTDOWN 0\n+\tint nr_handles; /* number of active event handles */\n+};\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_handles++;\n+\n+\tstate->health_data = data;\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tif (!state || !state->health_data)\n+\t\treturn;\n+\n+\tdata = state->health_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\n+\tFREE_AND_NULL(state->health_data);\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\n+\tfor (;;) {\n+\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n+\t\t\t\t\t\t data->hHandles,\n+\t\t\t\t\t\t FALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->health_error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+clean_shutdown:\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n+}\ndiff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\nnew file mode 100644\nindex 00000000000..45547ba9380\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health.h\n@@ -0,0 +1,47 @@\n+#ifndef FSM_HEALTH_H\n+#define FSM_HEALTH_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor health thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the health thread.\n+ * This will be called from the main thread AFTER joining the thread.\n+ */\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to monitor the\n+ * health of the daemon process. This will run in the health thread.\n+ *\n+ * The health thread should call `ipc_server_stop_async()` if it needs\n+ * to cause a shutdown. (It should NOT do so if it receives a shutdown\n+ * shutdown signal.)\n+ *\n+ * It should set `state->health_error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_health__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the health thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_HEALTH_H */\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex fc70dd2fc1d..80704406895 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex f7de7882517..716e0e4d28d 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n+struct fsm_health_data; /* opaque platform-specific data for health thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n+\tpthread_t health_thread;\n \tpthread_mutex_t main_lock;\n \n \tstruct strbuf path_worktree_watch;\n@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n \tstruct hashmap cookies;\n \n \tint listen_error_code;\n+\tint health_error_code;\n \tstruct fsm_listen_data *listen_data;\n+\tstruct fsm_health_data *health_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"448457","messageId":"a398cdb8a04228ff94e2f5467c010f4917b27042.1644940774.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH 18/23] fsm-health-win32: add framework to monitor daemon health","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-15T15:59:28Z","receivedAt":"2022-02-15T16:00:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate framework in Win32 version of the \"health\" thread to\nperiodically inspect the system and shutdown if warranted.\n\nThis version just includes the setup for the timeout in\nWaitForMultipleObjects() and calls the (currently empty) table\nof functions.\n\nA later commit will add functions to the table to actually\ninspect the system.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 54 ++++++++++++++++++++++++++++-\n 1 file changed, 53 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 94b1d020f25..3c3453369cd 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -4,6 +4,40 @@\n #include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n+/*\n+ * Every minute wake up and test our health.\n+ */\n+#define WAIT_FREQ_MS (60 * 1000)\n+\n+enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };\n+\n+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx);\n+\n+static interval_fn *table[] = {\n+\tNULL, /* must be last */\n+};\n+\n+/*\n+ * Call all of the functions in the table.\n+ * Shortcut and return first error.\n+ *\n+ * Return 0 if all succeeded.\n+ */\n+static int call_all(struct fsmonitor_daemon_state *state,\n+\t\t enum interval_fn_ctx ctx)\n+{\n+\tint k;\n+\n+\tfor (k = 0; table[k]; k++) {\n+\t\tint r = table[k](state, ctx);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n+\treturn 0;\n+}\n+\n struct fsm_health_data\n {\n \tHANDLE hEventShutdown;\n@@ -45,15 +79,31 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n+\tint r;\n+\n+\tr = call_all(state, CTX_INIT);\n+\tif (r < 0)\n+\t\tgoto force_error_stop;\n+\tif (r > 0)\n+\t\tgoto force_shutdown;\n \n \tfor (;;) {\n \t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n \t\t\t\t\t\t data->hHandles,\n-\t\t\t\t\t\t FALSE, INFINITE);\n+\t\t\t\t\t\t FALSE, WAIT_FREQ_MS);\n \n \t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n \t\t\tgoto clean_shutdown;\n \n+\t\tif (dwWait == WAIT_TIMEOUT) {\n+\t\t\tr = call_all(state, CTX_TIMER);\n+\t\t\tif (r < 0)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (r > 0)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\terror(_(\"health thread wait failed [GLE %ld]\"),\n \t\t GetLastError());\n \t\tgoto force_error_stop;\n@@ -61,8 +111,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)\n \n force_error_stop:\n \tstate->health_error_code = -1;\n+force_shutdown:\n \tipc_server_stop_async(state->ipc_server_data);\n clean_shutdown:\n+\tcall_all(state, CTX_TERM);\n \treturn;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"448520","messageId":"xmqqv8xfy6ab.fsf@gitster.g","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/23] Builtin FSMonitor Part 3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-16T01:00:28Z","receivedAt":"2022-02-16T01:00:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> Here is part 3 of my builtin FSMonitor series.\n>\n> Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in\n> \"seen\", so this series should be considered a preview until part 2 moves to\n> \"next\". (And part 2 should not graduate to \"master\" without this part.)\n>\n> Part 2 established the client code (used by commands like git status) and an\n> MVP implementation of the FSMonitor daemon. This was sufficient to test the\n> concepts and basic functionality.\n\nSounds like part 2 is sufficiently done to be eligible for being in\n'master', waiting for an improved daemon, no? \n\nHave people been reviewing the patches in part 2? I haven't had a\nchance to take a deeper look myself.\n\nThanks.\n\n"},{"id":"448594","messageId":"0e9f0152-4421-976b-51ff-8687c7f5d329@jeffhostetler.com","threadId":"57414","inReplyTo":"xmqqv8xfy6ab.fsf@gitster.g","subject":"Re: [PATCH 00/23] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-02-16T14:04:49Z","receivedAt":"2022-02-16T14:04:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/15/22 8:00 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> Here is part 3 of my builtin FSMonitor series.\n>>\n>> Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in\n>> \"seen\", so this series should be considered a preview until part 2 moves to\n>> \"next\". (And part 2 should not graduate to \"master\" without this part.)\n>>\n>> Part 2 established the client code (used by commands like git status) and an\n>> MVP implementation of the FSMonitor daemon. This was sufficient to test the\n>> concepts and basic functionality.\n> \n> Sounds like part 2 is sufficiently done to be eligible for being in\n> 'master', waiting for an improved daemon, no?\n> \n> Have people been reviewing the patches in part 2? I haven't had a\n> chance to take a deeper look myself.\n> \n> Thanks.\n> \n\nYes, I think Part 2 could advance if we wanted to. I just didn't\nwant to presume that, so I was being conservative. I've always\ntreated it and Part 3 as a unit.\n\nOn the other hand, we've been distributing Part 2 V4 and an\nun-submitted version of Part 3 experimentally in git-for-windows\nand microsoft/git.git (gvfs) since last summer, so I think we're\ngood.\n\nIt would be good to get some eyes on the Part 2 V5 changes having\nto do with removing `core.useBuiltinFSMonitor` and overloading\nthe existing `core.fsmonitor` to take either a hook path or bool.\nThat was the only major change between V4 and V5. And that V5\nchange has not yet been shipped in GFW nor GVFS.\n\nThanks\nJeff\n"},{"id":"449441","messageId":"e3b854c5-540a-980a-ceac-492d1ad3af90@github.com","threadId":"57414","inReplyTo":"23f38338cecd270c15fd1bba42de79c53e14ca2a.1644940773.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 01/23] fsm-listen-win32: handle shortnames","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T14:48:31Z","receivedAt":"2022-02-24T14:48:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach FSMonitor daemon on Windows to recognize shortname paths as\n> aliases of normal longname paths. FSMonitor clients, such as `git\n> status`, should receive the longname spelling of changed files (when\n> possible).\n> \n> Sometimes we receive FS events using the shortname, such as when a CMD\n> shell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\n> arrives using whatever combination of long and shortnames were used by\n> the other process. (Shortnames do seem to be case normalized,\n> however.)\n> \n> Use Windows GetLongPathNameW() to try to map the pathname spelling in\n> the notification event into the normalized longname spelling. (This\n> can fail if the file/directory is deleted, moved, or renamed, because\n> we are asking the FS for the mapping in response to the event and\n> after it has already happened, but we try.)\n\nI recall when we first discovered that this was important for users.\nI'm glad we haven't needed to worry about it since this fix.\n\n> Special case the shortname spelling of \".git\" to avoid under-reporting\n> these events.\n\nThis is particularly critical due to the case of deleting the .git\ndirectory being really important to catch, but GetLongPathNameW()\nwon't work in that case.\n\n> +\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n> +\t\t/*\n> +\t\t * The shortname to longname conversion can fail for\n> +\t\t * various reasons, for example if the file has been\n> +\t\t * deleted. (That is, if we just received a\n> +\t\t * delete-file notification event and the file is\n> +\t\t * already gone, we can't ask the file system to\n> +\t\t * lookup the longname for it. Likewise, for moves\n> +\t\t * and renames where we are given the old name.)\n> +\t\t *\n> +\t\t * NEEDSWORK: Since deleting or moving a file or\n> +\t\t * directory by shortname is rather obscure, I'm going\n> +\t\t * ignore the failure and ask the caller to report the\n> +\t\t * original relative path. This seemds kinder than\n\ns/seemds/seems/\n\n> +\t\t * failing here and forcing a resync.\n> +\t\t */\n> +\t\treturn GRR_NO_CONVERSION_NEEDED;\n> +\t}\n\nThis NEEDSWORK is something I saw while reading that makes it seem\nlike we _need_ to do something here. I think this is an appropriate\nshort-term solution that we can revisit if necessary. I don't\nanticipate that actually being the case, though.\n\nThanks,\n-Stolee\n"},{"id":"449442","messageId":"c7ee2394-cda0-a997-3b9d-fb8c3d65b312@github.com","threadId":"57414","inReplyTo":"ad8cf6d9a47b61d9fe41a961466122be16e4f041.1644940773.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T14:52:28Z","receivedAt":"2022-02-24T14:52:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Confirm that macOS FS events are reported with a normalized spelling.\n> \n> APFS (and/or HFS+) is case-insensitive. This means that case-independent\n> lookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed. But that\n> doesn't tell us how FS events are reported if we try \"rm -rf .git\" versus\n> \"rm -rf .GIT\". Are the events reported using the on-disk spelling of the\n> pathname or in the spelling used by the command.\n\nWas this last sentence supposed to be a question?\n \n> NEEDSWORK: I was only able to test case. It would be nice to add tests\n\n\"I was only able test the APFS case.\"?\n\n> that use different Unicode spellings/normalizations and understand the\n> differences between APFS and HFS+ in this area. We should confirm that\n> the spelling of the workdir paths that the daemon sends to clients are\n> always properly normalized.\n\nAre there any macOS experts out there who can help us find the answers\nto these questions?\n\n> +# Confirm that MacOS hides all of the Unicode normalization and/or\n> +# case folding from the FS events. That is, are the pathnames in the\n> +# FS events reported using the spelling on the disk or in the spelling\n> +# used by the other process.\n> +#\n> +# Note that we assume that the filesystem is set to case insensitive.\n> +#\n> +# NEEDSWORK: APFS handles Unicode and Unicode normalization\n> +# differently than HFS+. I only have an APFS partition, so\n> +# more testing here would be helpful.\n> +#\n> +\n> +# Rename .git using alternate spelling and confirm that the daemon\n> +# sees the event using the correct spelling and shutdown.\n> +test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' '\n> +\ttest_when_finished \"stop_daemon_delete_repo test_apfs\" &&\n> +\n> +\tgit init test_apfs &&\n> +\tstart_daemon test_apfs &&\n> +\n> +\ttest_path_is_dir test_apfs/.git &&\n> +\ttest_path_is_dir test_apfs/.GIT &&\n> +\n> +\tmv test_apfs/.GIT test_apfs/.FOO &&\n> +\tsleep 1 &&\n\nThis sleep is unfortunate. Do we really need it? Or does this test\nbecome flaky without it?\n\n> +\tmv test_apfs/.FOO test_apfs/.git &&\n> +\n> +\ttest_must_fail git -C test_apfs fsmonitor--daemon status\n> +'\n> +\n\nThis test is helpful in that it will help us discover if HFS+ or\nany future filesystem would break these assumptions.\n\nThanks,\n-Stolee\n"},{"id":"449444","messageId":"ab3d1fc7-e63a-dff4-f373-a6ac32b8294a@github.com","threadId":"57414","inReplyTo":"9efdbe28223e371dc22b4ba32fc791729af5577a.1644940773.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 04/23] t/helper/fsmonitor-client: create stress test","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T14:58:52Z","receivedAt":"2022-02-24T14:59:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Create a stress test to hammer on the fsmonitor daemon.\n> Create a client-side thread pool of n threads and have\n> each of them make m requests as fast as they can.\n> \n> NEEDSWORK: This is just the client-side thread pool and\n> is useful for interactive testing and experimentation.\n> We need to add a script test to drive this.\n\nI haven't gotten far enough in the series to know if you\n_do_ use this in a test eventually. If so, this NEEDSWORK\ncould be replaced with a mention of a future change.\n\n> +\t/*\n> +\t * TODO Decide if/when to return an error or call die().\n> +\t */\n> +\treturn 0;\n\nThis TODO could be cleaned up.\n\nThanks,\n-Stolee\n"},{"id":"449445","messageId":"b459111d-a536-b4d7-44d5-7f5aeb9ffb08@github.com","threadId":"57414","inReplyTo":"4715677f85ff3f47b437f82e2b82a8b5dd371c99.1644940773.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 06/23] fsmonitor-settings: stub in platform-specific incompatibility checking","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T15:05:39Z","receivedAt":"2022-02-24T15:05:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Extend generic incompatibility checkout with platform-specific\n> mechanism. Stub in Win32 version.\n> \n> In the existing fsmonitor-settings code we have a way to mark\n> types of repos as incompatible with fsmonitor (whether via the\n> hook and ipc APIs). For example, we do this for bare repos,\n> since there are no files to watch.\n> \n> Extend this exclusion mechanism for platfor-specific reasons.\n\ns/platfor-specific/platform-specific/\n\n> +# If your platform has os-specific ways to tell if a repo is incompatible with\n> +# fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS\n> +# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n> +# that implements the `fsm_os_settings__*()` routines.\n\nnit: It's just a comment, but I think \"OS\" and \"IPC\" should probably\nbe capitalized.\n\n> +#ifdef HAVE_FSMONITOR_OS_SETTINGS\n> +\t{\n> +\t\tenum fsmonitor_reason reason;\n> +\n> +\t\treason = fsm_os__incompatible(r);\n> +\t\tif (reason != FSMONITOR_REASON_ZERO) {\n\nA naming nit about FSMONITOR_REASON_ZERO. It seems named ZERO\non purpose so a non-zero reason signals incompatibility. That\nwould mean you could use\n\n\t\tif (reason)\n\nhere. Alternatively, FSMONITOR_REASON_COMPATIBLE would signal\nthe meaning better here:\n\n\t\tif (reason != FSMONITOR_REASON_COMPATIBLE)\n\n> +\t\t\tset_incompatible(r, reason);\n> +\t\t\treturn 1;\n> +\t\t}\n\nThanks,\n-Stolee\n"},{"id":"449447","messageId":"05747ff2-f839-5408-e25b-698b147ef158@github.com","threadId":"57414","inReplyTo":"4e856d60e385d64158f17ec1226f97eb323bc55e.1644940774.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 07/23] fsmonitor-settings: virtual repos are incompatible with FSMonitor","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T15:11:07Z","receivedAt":"2022-02-24T15:11:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Virtual repos, such as GVFS (aka VFS for Git), are incompatible\n> with FSMonitor.\n\nI would swap all of your \"GVFS (aka VFS for Git)\" for just\n\"VFS for Git\".\n\n> +/*\n> + * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n> + *\n> + * Granted, core Git does not know anything about GVFS and we\n> + * shouldn't make assumptions about a downstream feature, but users\n> + * can install both versions. And this can lead to incorrect results\n> + * from core Git commands. So, without bringing in any of the GVFS\n> + * code, do a simple config test for a published config setting. (We\n> + * do not look at the various *_TEST_* environment variables.)\n> + */\n> +static enum fsmonitor_reason is_virtual(struct repository *r)\n> +{\n> +\tconst char *const_str;\n> +\n> +\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n> +\t\treturn FSMONITOR_REASON_VIRTUAL;\n> +\n> +\treturn FSMONITOR_REASON_ZERO;\n> +}\n\nThis reason seems to be specific to a config setting that only\nexists in the microsoft/git fork. Perhaps this patch should remain\nthere.\n\nHowever, there is also the discussion of vfsd going on, so something\nsimilar might be necessary for that system. Junio also mentioned\nwanting to collaborate on a common indicator that virtualization was\nbeing used, so maybe we _should_ make core.virtualFilesystem a config\nsetting in the core Git project.\n\nThe reason for the incompatibility here is that VFS for Git has its\nown filesystem watcher and Git gets updates from it via custom code\nthat is a precursor to this FS Monitor feature. I don't know if vfsd\nhas plans for a similar setup. (It would probably be best to fit\ninto the FS Monitor client/server model and use a different daemon\nfor those virtualized repos, but I don't know enough details to be\nsure.)\n\nCC'ing Jonathan Nieder for thoughts on this.\n\nThanks,\n-Stolee\n"},{"id":"449450","messageId":"c35cb692-4539-d43e-6df0-6ae5ee911f5a@github.com","threadId":"57414","inReplyTo":"412fbc45868e7f2a05a03e424584b53ded4842e9.1644940774.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 09/23] fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T15:26:35Z","receivedAt":"2022-02-24T15:27:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach Git to detect remote working directories on macOS and mark them as\n> incompatible with FSMonitor.\n> \n> With this, `git fsmonitor--daemon run` will error out with a message\n> like it does for bare repos.\n> \n> Client commands, like `git status`, will not attempt to start the daemon.\n\n...\n\n> + * A client machine (such as a laptop) may choose to suspend/resume\n> + * and it is unclear (without lots of testing) whether the watcher can\n> + * resync after a resume. We might be able to treat this as a normal\n> + * \"events were dropped by the kernel\" event and do our normal \"flush\n> + * and resync\" --or-- we might need to close the existing (zombie?)\n> + * notification fd and create a new one.\n> + *\n> + * In theory, the above issues need to be addressed whether we are\n> + * using the Hook or IPC API.\n\nThe only thing I can think about is a case where the filesystem\nmonitor is actually running on the remote machine and Git\ncommunicates with it over the network. This is currently possible\nwith the hook, but I am not aware of a hook implementation that\ndoes this.\n\nWe can find a way to update the hook interface to communicate to\nGit that a remote disk is an appropriate case, but only if there\nis a real need for that.\n\n> + * For the builtin FSMonitor, we create the Unix domain socket for the\n> + * IPC in the .git directory. If the working directory is remote,\n> + * then the socket will be created on the remote file system. This\n\nThe socket is on the remote file system, but the daemon process is still\nlocal, so I still see this a problem.\n\n> + * can fail if the remote file system does not support UDS file types\n> + * (e.g. smbfs to a Windows server) or if the remote kernel does not\n> + * allow a non-local process to bind() the socket. (These problems\n> + * could be fixed by moving the UDS out of the .git directory and to a\n> + * well-known local directory on the client machine, but care should\n> + * be taken to ensure that $HOME is actually local and not a managed\n> + * file share.)\n> + *\n> + * So (for now at least), mark remote working directories as\n> + * incompatible.\n> + */\n> +static enum fsmonitor_reason is_remote(struct repository *r)\n> +{\n> +\tstruct statfs fs;\n> +\n> +\tif (statfs(r->worktree, &fs) == -1) {\n> +\t\tint saved_errno = errno;\n> +\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n> +\t\t\t\t r->worktree, strerror(saved_errno));\n> +\t\terrno = saved_errno;\n> +\t\treturn FSMONITOR_REASON_ZERO;\n\nSo if we fail to inspect the filesystem, we report it as compatible?\nI suppose other things are likely to fail if checks like this are\nfialing, but I wonder if we should preempt that by marking this as\nincompatible due to filesystem errors.\n\n> +\t}\n> +\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n> +\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n> +\n> +\tif (!(fs.f_flags & MNT_LOCAL))\n> +\t\treturn FSMONITOR_REASON_REMOTE;\n\nI do see that we need a successful response to give this specific\nreason for incompatibility.\n\n> +\treturn FSMONITOR_REASON_ZERO;\n> +}\n> \n> enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n> {\n> +\tenum fsmonitor_reason reason;\n> +\n> +\treason = is_remote(r);\n> +\tif (reason)\n> +\t\treturn reason;\n\nThis organization is looking like you want to short-circuit the\nchecks if you find an incompatibility, with the intent of having\nmultiple checks in the future.\n\nBut this can be done with simple || operators:\n\n\treturn is_remote() ||\n\t reason_check_2() ||\n\t reason_check_3();\n\nThanks,\n-Stolee\n"},{"id":"449457","messageId":"ac59cfaa-1323-df83-de03-f7886699bc24@github.com","threadId":"57414","inReplyTo":"ae09fb10c3ad5ea129c88addd07ebe674a22793d.1644940774.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 10/23] fsmonitor-settings: remote repos on Windows are incompatible with FSMonitor","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T15:56:44Z","receivedAt":"2022-02-24T15:57:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n...\n> @@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n> \tif (reason)\n> \t\treturn reason;\n> \n> +\treason = is_remote(r);\n> +\tif (reason)\n> +\t\treturn reason;\n> +\n> \treturn FSMONITOR_REASON_ZERO;\n\nJust popping in to say that\n\n\treturn is_virtual(r) ||\n\t is_remote(r);\n\nwould work here, but we might remove is_virtual().\n\nThanks,\n-Stolee\n"},{"id":"449458","messageId":"4c784936-7b70-3910-6413-0438e86b23c4@github.com","threadId":"57414","inReplyTo":"4a77f5b1fdef3af20fbfefccccc5fee194eb3422.1644940774.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T16:04:07Z","receivedAt":"2022-02-24T16:07:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Create another thread to watch over the daemon process and\n> automatically shut it down if necessary.\n> \n> This commit creates the basic framework for a \"health\" thread\n> to monitor the daemon and/or the file system. Later commits\n> will add platform-specific code to do the actual work.\n\n...\n\n> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\n> new file mode 100644\n> index 00000000000..b9f709e8548\n> --- /dev/null\n> +++ b/compat/fsmonitor/fsm-health-darwin.c\n> @@ -0,0 +1,24 @@\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"fsmonitor.h\"\n> +#include \"fsm-health.h\"\n> +#include \"fsmonitor--daemon.h\"\n> +\n> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n> +{\n> +\treturn 0;\n> +}\n> +\n> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n> +{\n> +\treturn;\n> +}\n> +\n> +void fsm_health__loop(struct fsmonitor_daemon_state *state)\n> +{\n> +\treturn;\n> +}\n> +\n> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n> +{\n> +}\n\nThe macOS implementation is stubbed, as advertised.\n\n> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\n> new file mode 100644\n> index 00000000000..94b1d020f25\n> --- /dev/null\n> +++ b/compat/fsmonitor/fsm-health-win32.c\n> @@ -0,0 +1,72 @@\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"fsmonitor.h\"\n> +#include \"fsm-health.h\"\n> +#include \"fsmonitor--daemon.h\"\n> +\n> +struct fsm_health_data\n> +{\n> +\tHANDLE hEventShutdown;\n> +\n> +\tHANDLE hHandles[1]; /* the array does not own these handles */\n> +#define HEALTH_SHUTDOWN 0\n> +\tint nr_handles; /* number of active event handles */\n> +};\n> +\n> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n> +{\n> +\tstruct fsm_health_data *data;\n> +\n> +\tCALLOC_ARRAY(data, 1);\n> +\n> +\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n> +\n> +\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n> +\tdata->nr_handles++;\n> +\n> +\tstate->health_data = data;\n> +\treturn 0;\n> +}\n> +\n> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n> +{\n> +\tstruct fsm_health_data *data;\n> +\n> +\tif (!state || !state->health_data)\n> +\t\treturn;\n> +\n> +\tdata = state->health_data;\n> +\n> +\tCloseHandle(data->hEventShutdown);\n> +\n> +\tFREE_AND_NULL(state->health_data);\n> +}\n> +\n> +void fsm_health__loop(struct fsmonitor_daemon_state *state)\n> +{\n> +\tstruct fsm_health_data *data = state->health_data;\n> +\n> +\tfor (;;) {\n> +\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n> +\t\t\t\t\t\t data->hHandles,\n> +\t\t\t\t\t\t FALSE, INFINITE);\n> +\n> +\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n> +\t\t\tgoto clean_shutdown;\n> +\n> +\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n> +\t\t GetLastError());\n> +\t\tgoto force_error_stop;\n> +\t}\n> +\n> +force_error_stop:\n> +\tstate->health_error_code = -1;\n> +\tipc_server_stop_async(state->ipc_server_data);\n> +clean_shutdown:\n> +\treturn;\n> +}\n> +\n> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n> +{\n> +\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n> +}\n\nBut it apppears the Windows code is actually implemented. Did you\nmean to do that as separate step, or should the commit message\nmention that the Windows implementation is included?\n\nThanks,\n-Stolee\n\n"},{"id":"449459","messageId":"0367bce8-3744-5835-aa26-6c1f9f359fdb@github.com","threadId":"57414","inReplyTo":"a398cdb8a04228ff94e2f5467c010f4917b27042.1644940774.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 18/23] fsm-health-win32: add framework to monitor daemon health","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T16:05:59Z","receivedAt":"2022-02-24T16:08:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Create framework in Win32 version of the \"health\" thread to\n> periodically inspect the system and shutdown if warranted.\n\nThis specific step makes sense to not be included in the\nprevious step.\n\n> +/*\n> + * Every minute wake up and test our health.\n> + */\n> +#define WAIT_FREQ_MS (60 * 1000)\n> +\n> +enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };\n\nPlease split out the values into their own lines.\n\nThanks,\n-Stolee\n"},{"id":"449460","messageId":"c0d69427-aee1-7341-5b5f-1fe44ee0542d@github.com","threadId":"57414","inReplyTo":"023fcd6e2b1163ab3d23b0d5933c14586d814ce0.1644940774.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 19/23] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T16:09:12Z","receivedAt":"2022-02-24T16:09:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\n> index 3c3453369cd..2526ad9194f 100644\n> --- a/compat/fsmonitor/fsm-health-win32.c\n> +++ b/compat/fsmonitor/fsm-health-win32.c\n> @@ -14,7 +14,10 @@ enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };\n> typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n> \t\t\t enum interval_fn_ctx ctx);\n> \n> +static interval_fn has_worktree_moved;\n> +\n> static interval_fn *table[] = {\n> +\thas_worktree_moved,\n> \tNULL, /* must be last */\n> };\n\nLooking at this now, I think table[] should be defined immediately\nbefore fsm_health__loop() so it is easier to see how they interact.\nIt also avoids this static declaration of the function before its\nimplementation.\n\nOr, is there a reason it is so high up in the file?\n\nThanks,\n-Stolee\n\n"},{"id":"449461","messageId":"87f9a63c-97d5-9aa7-c952-8d7498a5825c@github.com","threadId":"57414","inReplyTo":"496b3da35d0f9b8c68c6bc6c2bf2ca93ffe224b9.1644940774.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 20/23] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T16:10:18Z","receivedAt":"2022-02-24T16:11:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n\n> +\t\tif (ef_is_root_changed(event_flags[k])) {\n> +\t\t\t/*\n> +\t\t\t * The spelling of the pathname of the root directory\n> +\t\t\t * has changed. This includes the name of the root\n> +\t\t\t * directory itself of of any parent directory in the\n\ns/of of/or of/\n\n> +\t\t\t * path.\n> +\t\t\t *\n> +\t\t\t * (There may be other conditions that throw this,\n> +\t\t\t * but I couldn't find any information on it.)\n> +\t\t\t *\n> +\t\t\t * Force a shutdown now and avoid things getting\n> +\t\t\t * out of sync. The Unix domain socket is inside\n> +\t\t\t * the .git directory and a spelling change will make\n> +\t\t\t * it hard for clients to rendezvous with us.\n> +\t\t\t */\n> +\t\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t\t \"event: root changed\");\n> +\t\t\tgoto force_shutdown;\n> +\t\t}\n> +\n\nThanks,\n-Stolee\n\n"},{"id":"449462","messageId":"932d7a6a-010d-f659-137e-bf59f0e0b6a8@github.com","threadId":"57414","inReplyTo":"07a9c7542be79fb60861253848eca258bec87694.1644940774.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 21/23] fsmonitor: optimize processing of directory events","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T16:13:02Z","receivedAt":"2022-02-24T16:18:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach Git to perform binary search over the cache-entries for a directory\n> notification and then linearly scan forward to find the immediate children.\n> \n> Previously, when the FSMonitor reported a modified directory Git would\n> perform a linear search on the entire cache-entry array for all\n> entries matching that directory prefix and invalidate them. Since the\n> cache-entry array is already sorted, we can use a binary search to\n> find the first matching entry and then only linearly walk forward and\n> invalidate entries until the prefix changes.\n> \n> Also, the original code would invalidate anything having the same\n> directory prefix. Since a directory event should only be received for\n> items that are immediately within the directory (and not within\n> sub-directories of it), only invalidate those entries and not the\n> whole subtree.\n\nThis makes sense as an optimization. I've also taken a close look at\nthe code to be sure this makes sense if the file system events are\nfor a directory within a sparse directory entry, and the end result\nwill be that the sparse directory is marked as invalid, which is fine.\n\nThanks,\n-Stolee\n"},{"id":"449463","messageId":"f9249541-bb92-1c61-825f-44a034a17bb8@github.com","threadId":"57414","inReplyTo":"4c784936-7b70-3910-6413-0438e86b23c4@github.com","subject":"Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T16:15:18Z","receivedAt":"2022-02-24T16:19:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/24/2022 11:04 AM, Derrick Stolee wrote:\n> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Create another thread to watch over the daemon process and\n>> automatically shut it down if necessary.\n>>\n>> This commit creates the basic framework for a \"health\" thread\n>> to monitor the daemon and/or the file system. Later commits\n>> will add platform-specific code to do the actual work.\n> \n> ...\n> \n>> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\n>> new file mode 100644\n>> index 00000000000..b9f709e8548\n>> --- /dev/null\n>> +++ b/compat/fsmonitor/fsm-health-darwin.c\n>> @@ -0,0 +1,24 @@\n>> +#include \"cache.h\"\n>> +#include \"config.h\"\n>> +#include \"fsmonitor.h\"\n>> +#include \"fsm-health.h\"\n>> +#include \"fsmonitor--daemon.h\"\n>> +\n>> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n>> +{\n>> +\treturn 0;\n>> +}\n>> +\n>> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n>> +{\n>> +\treturn;\n>> +}\n>> +\n>> +void fsm_health__loop(struct fsmonitor_daemon_state *state)\n>> +{\n>> +\treturn;\n>> +}\n>> +\n>> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n>> +{\n>> +}\n> \n> The macOS implementation is stubbed, as advertised.\n\nAfter looking at the rest of the patch series, it seems that these\nare never filled in. Are some of the win32 health monitors also\nappropriate for macOS? (They would need platform-specific checks,\nprobably.)\n\nThanks,\n-Stolee\n"},{"id":"449472","messageId":"e3c0a046-8041-421d-6640-91d1a560f79b@github.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/23] Builtin FSMonitor Part 3","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T16:21:50Z","receivedAt":"2022-02-24T16:32:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> Here is part 3 of my builtin FSMonitor series.\n> \n> Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in\n> \"seen\", so this series should be considered a preview until part 2 moves to\n> \"next\". (And part 2 should not graduate to \"master\" without this part.)\n> \n> Part 2 established the client code (used by commands like git status) and an\n> MVP implementation of the FSMonitor daemon. This was sufficient to test the\n> concepts and basic functionality.\n> \n> Part 3 finishes the daemon and adds additional tests. This includes:\n> \n> * On Windows, handle short- and long-name aliasing.\n> * On Mac, handle Unicode aliasing.\n> * Mark bare, virtual, and remote working directories incompatible with\n> FSMonitor.\n> * On Mac, ignore xattr change FSEvents.\n> * On Windows, cd out of the working directory root.\n> * Create a thread to monitor the health and shutdown the daemon if\n> necessary.\n> * Speed up handling of directory notification events.\n> * Test directory move and rename events.\n> * Add performance test.\n\nI had been more involved in early versions of parts 1 and 2, but was\nnot paying attention much as these updates for part 3 were coming into\ngit-for-windows/git and microsoft/git. (I do recall some of the support\nissues that motivated them.)\n\nSince these have been battle-tested in the wild for a while, I do not\ndoubt that they are solving the user needs they intend to solve.\n\nInstead, I focused on looking at the patches for anything that might\nbe abnormal. The layout of the code is clear enough that I can easily\nsee how information from the platform-specific code is being handled,\nso I do not worry about corner cases being an issue from the Git side\nof the code. As for the platform-specific stuff, maybe there are cases\nthat are missed that require deep expertise in those platform calls. I\ncannot speak to that until we get a user report of something even\nstranger happening than the special cases targeted here.\n\nWith that said, I mostly found superficial things to improve the\nseries. Some typos, some rename possibilities, things like that.\n \n> Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a\n> synthetic repo containing 1M files on a Macbook Pro. It shows the effects of\n> the untracked cache (uc) and FSMonitor (fsm) on git status.\n> \n> $ ./p7527-builtin-fsmonitor.sh \n> # passed all 67 test(s)\n> 1..67\n> Test this tree \n> ---------------------------------------------------------------------------------------\n> 7527.4: [uc false][fsm false] status after checkout 29.99(3.14+80.12) \n> 7527.6: [uc false][fsm false] status after big change 73.32(5.11+97.24) \n> 7527.8: [uc false][fsm false] status after add all 47.80(5.12+90.47) \n> 7527.10: [uc false][fsm false] status after add dot 49.22(5.16+92.05) \n> 7527.12: [uc false][fsm false] status after commit 51.53(3.35+100.74)\n> 7527.14: [uc false][fsm false] status after reset hard 33.74(3.03+85.31) \n> 7527.16: [uc false][fsm false] status after create untracked files 41.71(3.24+89.75) \n> 7527.18: [uc false][fsm false] status after clean 34.33(3.07+89.36) \n> \n> 7527.20: [uc false][fsm true] status after checkout 29.23(1.94+10.84) \n> 7527.22: [uc false][fsm true] status after big change 64.23(4.66+24.86) \n> 7527.24: [uc false][fsm true] status after add all 45.45(4.37+18.70) \n> 7527.26: [uc false][fsm true] status after add dot 44.42(4.02+17.10) \n> 7527.28: [uc false][fsm true] status after commit 30.52(1.95+10.91) \n> 7527.30: [uc false][fsm true] status after reset hard 28.70(2.70+13.89) \n> 7527.32: [uc false][fsm true] status after create untracked files 28.63(2.59+10.71) \n> 7527.34: [uc false][fsm true] status after clean 28.97(2.59+10.78) \n> \n> 7527.36: [uc true][fsm false] status after checkout 35.06(3.17+86.11) \n> 7527.38: [uc true][fsm false] status after big change 74.65(5.14+101.50)\n> 7527.40: [uc true][fsm false] status after add all 49.96(5.22+90.96) \n> 7527.42: [uc true][fsm false] status after add dot 49.77(5.24+91.72) \n> 7527.44: [uc true][fsm false] status after commit 36.95(3.27+92.25) \n> 7527.46: [uc true][fsm false] status after reset hard 33.89(3.18+85.68) \n> 7527.48: [uc true][fsm false] status after create untracked files 41.44(3.40+92.99) \n> 7527.50: [uc true][fsm false] status after clean 34.60(3.26+90.19) \n> \n> 7527.52: [uc true][fsm true] status after checkout 0.58(0.45+0.10) \n> 7527.54: [uc true][fsm true] status after big change 65.16(4.91+25.64) \n> 7527.56: [uc true][fsm true] status after add all 45.43(4.45+18.92) \n> 7527.58: [uc true][fsm true] status after add dot 15.56(2.57+6.32) \n> 7527.60: [uc true][fsm true] status after commit 0.98(0.46+0.11) \n> 7527.62: [uc true][fsm true] status after reset hard 30.30(2.96+14.49) \n> 7527.64: [uc true][fsm true] status after create untracked files 2.15(1.73+0.40) \n> 7527.66: [uc true][fsm true] status after clean 1.68(1.56+0.32) \n\nThe other stylistic thing is this performance test. It would be nice if\nthese tests were grouped by the operation (like \"status after checkout\")\nso it is easier to compare the same operation across the matrix definitions.\n\nThis would require reordering the test definition as well as allowing the\ndifferent cases to simultaneously live in different repositories. The\np2000-sparse-operations.sh has this kind of organization, but you'll need\nmore involved test cases than \"run this command\".\n\nThanks,\n-Stolee\n"},{"id":"449483","messageId":"20220224173305.gbr2waw77xpuieub@tb-raspi4","threadId":"57414","inReplyTo":"c7ee2394-cda0-a997-3b9d-fb8c3d65b312@github.com","subject":"Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode","fromName":"Torsten B��gershausen","fromEmail":"tboegi@web.de","sentAt":"2022-02-24T17:33:06Z","receivedAt":"2022-02-24T17:38:22Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:\n> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> > From: Jeff Hostetler \n> >\n> > Confirm that macOS FS events are reported with a normalized spelling.\n> >\n> > APFS (and/or HFS+) is case-insensitive. This means that case-independent\n\nThis is not true, strictly speaking.\nYou can format a disk with \"case sensitive\" or \"case-insenstive, case preserving\".\nBoth APFS and HFS+ can be formated that way.\nThe default, which is what you get when you get a new machine,\nis \"case-insenstive, case preserving\".\nAnd I assume, that more 99% of all disks are formated that way.\nThe \"core.ignorecase\" is used in the same way as it is used under NTFS,\nFAT or all other case-insenstive file systems.\n(and even ext4 can be formated case-insensitive these days.)\n\nAn interesting article can be found here:\nhttps://lwn.net/Articles/784041/\n\nAnd to be technically correct, I think that even NTFS can be\n\"configured to be case insensitive in an empty directory\".\n\nIn that sense, I would like to avoid this statement, which\nfile system is case insensitive, and which is not.\nGit assumes that after probing the FS in \"git init\" we have\na valid configuration in core.ignorecase.\n\n> > lookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed. But that\n> > doesn't tell us how FS events are reported if we try \"rm -rf .git\" versus\n> > \"rm -rf .GIT\". Are the events reported using the on-disk spelling of the\n> > pathname or in the spelling used by the command.\n>\n> Was this last sentence supposed to be a question?\n>\n> > NEEDSWORK: I was only able to test case. It would be nice to add tests\n>\n> \"I was only able test the APFS case.\"?\n>\n> > that use different Unicode spellings/normalizations and understand the\n> > differences between APFS and HFS+ in this area. We should confirm that\n> > the spelling of the workdir paths that the daemon sends to clients are\n> > always properly normalized.\n>\n> Are there any macOS experts out there who can help us find the answers\n> to these questions?\n\nThere is a difference between HFS+ and APFS.\nHFS+ is \"unicode decomposing\" when you call readdir() - file names\nare stored decomposed on disk once created.\nHowever, opening file in precompsed form succeds.\nIn that sense I would strongly suspect, that any monitors are \"sending\"\nthe decomposed form (on HFS+).\n\nAPFS does not manipulate file names in that way, it is\n\"unicode normalization preserving and ignoring\".\n\n\n>\n> > +# Confirm that MacOS hides all of the Unicode normalization and/or\n> > +# case folding from the FS events. That is, are the pathnames in the\n> > +# FS events reported using the spelling on the disk or in the spelling\n> > +# used by the other process.\n> > +#\n> > +# Note that we assume that the filesystem is set to case insensitive.\n> > +#\n> > +# NEEDSWORK: APFS handles Unicode and Unicode normalization\n> > +# differently than HFS+. I only have an APFS partition, so\n> > +# more testing here would be helpful.\n\nI have an older system and may be able to help, (but am busy this week)\n> > +#\n> > +\n> > +# Rename .git using alternate spelling and confirm that the daemon\n> > +# sees the event using the correct spelling and shutdown.\n> > +test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' '\n\nIsn't this precondition is the wrong one and CASE_INSENSITIVE_FS is a better one ?\n\n[snip]\n\nI diddn't follow you series, but if there is a need to test under MacOS with HFS+,\nfeel free to cc me in the next round\n"},{"id":"449635","messageId":"220225.86ee3q7k2s.gmgdl@evledraar.gmail.com","threadId":"57414","inReplyTo":"44cc61e186cb65fa6b2c1d5a0f080fc0b2265e57.1644940773.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 05/23] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-25T20:42:09Z","receivedAt":"2022-02-25T20:44:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 15 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> +static void create_reason_message(struct repository *r,\n> +\t\t\t\t struct strbuf *buf_reason)\n> +{\n> +\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n> +\n> +\tswitch (s->reason) {\n> +\tcase FSMONITOR_REASON_ZERO:\n> +\t\treturn;\n> +\n> +\tcase FSMONITOR_REASON_BARE:\n> +\t\tstrbuf_addstr(buf_reason,\n> +\t\t\t _(\"bare repos are incompatible with fsmonitor\"));\n> +\t\treturn;\n> +\n> +\tdefault:\n> +\t\tBUG(\"Unhandled case in create_reason_message '%d'\", s->reason);\n> +\t}\n> +}\n> +\n> +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n> +\t\t\t\t\t struct strbuf *buf_reason)\n> +{\n> +\tlookup_fsmonitor_settings(r);\n> +\n> +\tstrbuf_reset(buf_reason);\n> +\tif (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)\n> +\t\tcreate_reason_message(r, buf_reason);\n> +\n> +\treturn r->settings.fsmonitor->reason;\n> +}\n\nThis API (just looking at one small bit discussed because related bits\nconflict with another series) seems to require a lot of ceremony just to\nget a const char * error.\n\nI tried this on top of \"seen\", and the parts I compile on Linux (so not\nthe fsmonitor--daemon.c) were happy with it.\n\n builtin/fsmonitor--daemon.c | 14 +++++++-------\n builtin/update-index.c | 9 ++++-----\n fsmonitor-settings.c | 47 +++++++++++++++------------------------------\n fsmonitor-settings.h | 5 +++--\n 4 files changed, 29 insertions(+), 46 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 591433e897d..7ad7bc718b3 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1496,7 +1496,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n \tint free_console = 0;\n-\n+\tenum fsmonitor_mode fsm_mode;\n \tstruct option options[] = {\n \t\tOPT_BOOL(0, \"free-console\", &free_console, N_(\"free console\")),\n \t\tOPT_INTEGER(0, \"ipc-threads\",\n@@ -1524,12 +1524,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \tprepare_repo_settings(the_repository);\n \tfsm_settings__set_ipc(the_repository);\n \n-\tif (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {\n-\t\tstruct strbuf buf_reason = STRBUF_INIT;\n-\t\tfsm_settings__get_reason(the_repository, &buf_reason);\n-\t\terror(\"%s '%s'\", buf_reason.buf, xgetcwd());\n-\t\tstrbuf_release(&buf_reason);\n-\t\treturn -1;\n+\tfsm_mode = fsm_settings__get_mode(the_repository);\n+\tif (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {\n+\t\tenum fsmonitor_reason fsm_reason = fsm_settings__get_reason(the_repository);\n+\t\tconst char *reason = fsm_settings_incompatible_reason_msg(fsm_mode, fsm_reason);\n+\n+\t\treturn error(\"%s\", reason ? reason : \"???\");\n \t}\n \n \tif (!strcmp(subcmd, \"start\"))\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 61b0b98ccaf..f8f638d33d9 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1237,13 +1237,12 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tenum fsmonitor_reason fsm_reason = fsm_settings__get_reason(r);\n \n \t\tif (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {\n-\t\t\tstruct strbuf buf_reason = STRBUF_INIT;\n-\t\t\tfsm_settings__get_reason(r, &buf_reason);\n-\t\t\terror(\"%s\", buf_reason.buf);\n-\t\t\tstrbuf_release(&buf_reason);\n-\t\t\treturn -1;\n+\t\t\tconst char *reason = fsm_settings_incompatible_reason_msg(fsm_mode, fsm_reason);\n+\n+\t\t\treturn error(\"%s\", reason ? reason : \"???\");\n \t\t}\n \n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex de69ace246a..e37b342aa2b 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -103,6 +103,13 @@ enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n \treturn r->settings.fsmonitor->mode;\n }\n \n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n+{\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\n+\n const char *fsm_settings__get_hook_path(struct repository *r)\n {\n \tlookup_fsmonitor_settings(r);\n@@ -142,43 +149,19 @@ void fsm_settings__set_disabled(struct repository *r)\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n \n-static void create_reason_message(struct repository *r,\n-\t\t\t\t struct strbuf *buf_reason)\n+const char *fsm_settings_incompatible_reason_msg(enum fsmonitor_mode mode,\n+\t\t\t\t\t\t enum fsmonitor_reason reason)\n {\n-\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\tassert(mode == FSMONITOR_MODE_INCOMPATIBLE);\n \n-\tswitch (s->reason) {\n+\tswitch (reason) {\n \tcase FSMONITOR_REASON_ZERO:\n-\t\treturn;\n-\n+\t\treturn NULL;\n \tcase FSMONITOR_REASON_BARE:\n-\t\tstrbuf_addstr(buf_reason,\n-\t\t\t _(\"bare repos are incompatible with fsmonitor\"));\n-\t\treturn;\n-\n+\t\treturn _(\"bare repos are incompatible with fsmonitor\");\n \tcase FSMONITOR_REASON_VIRTUAL:\n-\t\tstrbuf_addstr(buf_reason,\n-\t\t\t _(\"virtual repos are incompatible with fsmonitor\"));\n-\t\treturn;\n-\n+\t\treturn _(\"virtual repos are incompatible with fsmonitor\");\n \tcase FSMONITOR_REASON_REMOTE:\n-\t\tstrbuf_addstr(buf_reason,\n-\t\t\t _(\"remote repos are incompatible with fsmonitor\"));\n-\t\treturn;\n-\n-\tdefault:\n-\t\tBUG(\"Unhandled case in create_reason_message '%d'\", s->reason);\n+\t\treturn _(\"remote repos are incompatible with fsmonitor\");\n \t}\n }\n-\n-enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n-\t\t\t\t\t struct strbuf *buf_reason)\n-{\n-\tlookup_fsmonitor_settings(r);\n-\n-\tstrbuf_reset(buf_reason);\n-\tif (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)\n-\t\tcreate_reason_message(r, buf_reason);\n-\n-\treturn r->settings.fsmonitor->reason;\n-}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex fca25887c0f..81be1ef1801 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -25,9 +25,10 @@ void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n-enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n-\t\t\t\t\t struct strbuf *buf_reason);\n+const char *fsm_settings_incompatible_reason_msg(enum fsmonitor_mode mode,\n+\t\t\t\t\t\t enum fsmonitor_reason reason);\n \n struct fsmonitor_settings;\n \n\n"},{"id":"449990","messageId":"0baa2939-bf3c-b08b-2cfc-5a96be760b1c@jeffhostetler.com","threadId":"57414","inReplyTo":"ab3d1fc7-e63a-dff4-f373-a6ac32b8294a@github.com","subject":"Re: [PATCH 04/23] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-01T19:37:16Z","receivedAt":"2022-03-01T19:37:21Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 9:58 AM, Derrick Stolee wrote:\n> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Create a stress test to hammer on the fsmonitor daemon.\n>> Create a client-side thread pool of n threads and have\n>> each of them make m requests as fast as they can.\n>>\n>> NEEDSWORK: This is just the client-side thread pool and\n>> is useful for interactive testing and experimentation.\n>> We need to add a script test to drive this.\n> \n> I haven't gotten far enough in the series to know if you\n> _do_ use this in a test eventually. If so, this NEEDSWORK\n> could be replaced with a mention of a future change.\n\nRight. I mainly use this test helper to hammer on the\ndaemon during interactive tests. I don't have a script\nto actually use. I'm currently not sure what that would\nlook like given our test script framework.\n\n> \n>> +\t/*\n>> +\t * TODO Decide if/when to return an error or call die().\n>> +\t */\n>> +\treturn 0;\n> \n> This TODO could be cleaned up.\n\ngood catch. thanks!\n\nJeff\n\n"},{"id":"450018","messageId":"49d981a6-be6e-c995-160f-593d3ef3cf27@jeffhostetler.com","threadId":"57414","inReplyTo":"05747ff2-f839-5408-e25b-698b147ef158@github.com","subject":"Re: [PATCH 07/23] fsmonitor-settings: virtual repos are incompatible with FSMonitor","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-01T21:01:48Z","receivedAt":"2022-03-01T21:01:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 10:11 AM, Derrick Stolee wrote:\n> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Virtual repos, such as GVFS (aka VFS for Git), are incompatible\n>> with FSMonitor.\n> \n> I would swap all of your \"GVFS (aka VFS for Git)\" for just\n> \"VFS for Git\".\n> \n>> +/*\n>> + * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n>> + *\n>> + * Granted, core Git does not know anything about GVFS and we\n>> + * shouldn't make assumptions about a downstream feature, but users\n>> + * can install both versions. And this can lead to incorrect results\n>> + * from core Git commands. So, without bringing in any of the GVFS\n>> + * code, do a simple config test for a published config setting. (We\n>> + * do not look at the various *_TEST_* environment variables.)\n>> + */\n>> +static enum fsmonitor_reason is_virtual(struct repository *r)\n>> +{\n>> +\tconst char *const_str;\n>> +\n>> +\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n>> +\t\treturn FSMONITOR_REASON_VIRTUAL;\n>> +\n>> +\treturn FSMONITOR_REASON_ZERO;\n>> +}\n> \n> This reason seems to be specific to a config setting that only\n> exists in the microsoft/git fork. Perhaps this patch should remain\n> there.\n> \n> However, there is also the discussion of vfsd going on, so something\n> similar might be necessary for that system. Junio also mentioned\n> wanting to collaborate on a common indicator that virtualization was\n> being used, so maybe we _should_ make core.virtualFilesystem a config\n> setting in the core Git project.\n> \n> The reason for the incompatibility here is that VFS for Git has its\n> own filesystem watcher and Git gets updates from it via custom code\n> that is a precursor to this FS Monitor feature. I don't know if vfsd\n> has plans for a similar setup. (It would probably be best to fit\n> into the FS Monitor client/server model and use a different daemon\n> for those virtualized repos, but I don't know enough details to be\n> sure.)\n> \n> CC'ing Jonathan Nieder for thoughts on this.\n\nI was wondering whether this should be upstream or just in our\ndownstream forks, but I thought it better to include it so that\nwe don't try to monitor a virtualized repo and not mislead the\nuser. It may be that we can correctly watch the repo and\ngenerate correct results, but without knowing any details of\nwhat the virtualization is doing behind the scenes, we would\nbe making some unsafe assumptions. And since Windows users often\nhave multiple versions of Git installed (their CL tools and\nwhatever their IDE installed), promoting this check to upstream\nfelt important.\n\nWRT the ongoing \"vfsd\" effort, I'm not sure what that looks like\nyet and/or whether repos managed by \"vfsd\" have similar concerns.\n\nI'll rename the variables in my patch here to be \"vfs4git\" rather\nthe generic \"virtual\". This will avoid confusion later if another\ncase needs to be added for \"vfsd\". My code is Win32-specific and\nit is unclear it theirs will be Linux-only or cross-platform, so\nhopefully with the rename we can coexist sanely.\n\nJeff\n\n\n"},{"id":"450021","messageId":"11c3c3b7-82f2-f34c-e79e-5773593c19c2@jeffhostetler.com","threadId":"57414","inReplyTo":"c35cb692-4539-d43e-6df0-6ae5ee911f5a@github.com","subject":"Re: [PATCH 09/23] fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-01T21:30:34Z","receivedAt":"2022-03-01T21:30:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 10:26 AM, Derrick Stolee wrote:\n> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Teach Git to detect remote working directories on macOS and mark them as\n>> incompatible with FSMonitor.\n>>\n>> With this, `git fsmonitor--daemon run` will error out with a message\n>> like it does for bare repos.\n>>\n>> Client commands, like `git status`, will not attempt to start the daemon.\n> \n> ...\n> \n>> + * A client machine (such as a laptop) may choose to suspend/resume\n>> + * and it is unclear (without lots of testing) whether the watcher can\n>> + * resync after a resume. We might be able to treat this as a normal\n>> + * \"events were dropped by the kernel\" event and do our normal \"flush\n>> + * and resync\" --or-- we might need to close the existing (zombie?)\n>> + * notification fd and create a new one.\n>> + *\n>> + * In theory, the above issues need to be addressed whether we are\n>> + * using the Hook or IPC API.\n> \n> The only thing I can think about is a case where the filesystem\n> monitor is actually running on the remote machine and Git\n> communicates with it over the network. This is currently possible\n> with the hook, but I am not aware of a hook implementation that\n> does this.\n> \n> We can find a way to update the hook interface to communicate to\n> Git that a remote disk is an appropriate case, but only if there\n> is a real need for that.\n\nI'm not saying we can't support remote working directories.\nWe do get events from SMB for example, but there network\nbuffer limits and all the usual connectivity issues with a\nlocal daemon reading an event stream from a remote file system.\nSo out of caution, I want to disable it for now. We can always\nrevisit it later.\n\nWRT the builtin IPC-based daemon, I have the socket/named pipe\nrestricted to local access only. This prevents a client like\n\"git status\" from talking to a remote daemon, but it also prevents\na remote bad guy from talking our daemon. But those are secondary\nconcerns right now -- I'm mainly concerned with correctly getting\nall of the (possibly high volume/speed) events to the client.\n\nThere's also the possible incompatibilities of vendor X client-side\nOS and vendor Y server-side OS and how well notifications are\nsupported between them....\n\nSo I'd just like to shut it off for now.\n\n> \n>> + * For the builtin FSMonitor, we create the Unix domain socket for the\n>> + * IPC in the .git directory. If the working directory is remote,\n>> + * then the socket will be created on the remote file system. This\n> \n> The socket is on the remote file system, but the daemon process is still\n> local, so I still see this a problem.\n> \n>> + * can fail if the remote file system does not support UDS file types\n>> + * (e.g. smbfs to a Windows server) or if the remote kernel does not\n>> + * allow a non-local process to bind() the socket. (These problems\n>> + * could be fixed by moving the UDS out of the .git directory and to a\n>> + * well-known local directory on the client machine, but care should\n>> + * be taken to ensure that $HOME is actually local and not a managed\n>> + * file share.)\n>> + *\n>> + * So (for now at least), mark remote working directories as\n>> + * incompatible.\n>> + */\n>> +static enum fsmonitor_reason is_remote(struct repository *r)\n>> +{\n>> +\tstruct statfs fs;\n>> +\n>> +\tif (statfs(r->worktree, &fs) == -1) {\n>> +\t\tint saved_errno = errno;\n>> +\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n>> +\t\t\t\t r->worktree, strerror(saved_errno));\n>> +\t\terrno = saved_errno;\n>> +\t\treturn FSMONITOR_REASON_ZERO;\n> \n> So if we fail to inspect the filesystem, we report it as compatible?\n> I suppose other things are likely to fail if checks like this are\n> fialing, but I wonder if we should preempt that by marking this as\n> incompatible due to filesystem errors.\n\nGood point.\n\n> \n>> +\t}\n>> +\n>> +\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n>> +\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n>> +\n>> +\tif (!(fs.f_flags & MNT_LOCAL))\n>> +\t\treturn FSMONITOR_REASON_REMOTE;\n> \n> I do see that we need a successful response to give this specific\n> reason for incompatibility.\n> \n>> +\treturn FSMONITOR_REASON_ZERO;\n>> +}\n>> \n>> enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n>> {\n>> +\tenum fsmonitor_reason reason;\n>> +\n>> +\treason = is_remote(r);\n>> +\tif (reason)\n>> +\t\treturn reason;\n> \n> This organization is looking like you want to short-circuit the\n> checks if you find an incompatibility, with the intent of having\n> multiple checks in the future.\n> \n> But this can be done with simple || operators:\n> \n> \treturn is_remote() ||\n> \t reason_check_2() ||\n> \t reason_check_3();\n\nTrue. That is a little shorter.\n\nThanks\nJeff\n"},{"id":"450173","messageId":"1fe07a1b-df7f-5230-5f49-4cb454c3df9f@jeffhostetler.com","threadId":"57414","inReplyTo":"220225.86ee3q7k2s.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 05/23] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-02T21:09:28Z","receivedAt":"2022-03-02T21:09:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/25/22 3:42 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Feb 15 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> +static void create_reason_message(struct repository *r,\n>> +\t\t\t\t struct strbuf *buf_reason)\n>> +{\n>> +\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n>> +\n>> +\tswitch (s->reason) {\n>> +\tcase FSMONITOR_REASON_ZERO:\n>> +\t\treturn;\n>> +\n>> +\tcase FSMONITOR_REASON_BARE:\n>> +\t\tstrbuf_addstr(buf_reason,\n>> +\t\t\t _(\"bare repos are incompatible with fsmonitor\"));\n>> +\t\treturn;\n>> +\n>> +\tdefault:\n>> +\t\tBUG(\"Unhandled case in create_reason_message '%d'\", s->reason);\n>> +\t}\n>> +}\n>> +\n>> +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n>> +\t\t\t\t\t struct strbuf *buf_reason)\n>> +{\n>> +\tlookup_fsmonitor_settings(r);\n>> +\n>> +\tstrbuf_reset(buf_reason);\n>> +\tif (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)\n>> +\t\tcreate_reason_message(r, buf_reason);\n>> +\n>> +\treturn r->settings.fsmonitor->reason;\n>> +}\n> \n> This API (just looking at one small bit discussed because related bits\n> conflict with another series) seems to require a lot of ceremony just to\n> get a const char * error.\n\nMy thought at the time was that the __get_reason() code might want to\nformat a message and include details from the repo (such as the working\ndirectory root) or the kind of objection (such as a remote SMB mount).\nSo I had it take a strbuf rather than a \"const char *\" return value.\n\nBut so far (in the successive commits) all of the reason messages have\nbeen constant and I think I'm OK with having the non-specific messages.\n\nSo yes, I could simplify it.\n\nThanks\nJeff\n\n"},{"id":"450283","messageId":"5bb246d3-82d0-835a-cfc4-be2824479a82@jeffhostetler.com","threadId":"57414","inReplyTo":"4c784936-7b70-3910-6413-0438e86b23c4@github.com","subject":"Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-03T16:16:01Z","receivedAt":"2022-03-03T16:16:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 11:04 AM, Derrick Stolee wrote:\n> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Create another thread to watch over the daemon process and\n>> automatically shut it down if necessary.\n>>\n>> This commit creates the basic framework for a \"health\" thread\n>> to monitor the daemon and/or the file system. Later commits\n>> will add platform-specific code to do the actual work.\n> \n> ...\n...\n> \n>> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\n>> new file mode 100644\n>> index 00000000000..94b1d020f25\n>> --- /dev/null\n>> +++ b/compat/fsmonitor/fsm-health-win32.c\n>> @@ -0,0 +1,72 @@\n>> +#include \"cache.h\"\n>> +#include \"config.h\"\n>> +#include \"fsmonitor.h\"\n>> +#include \"fsm-health.h\"\n>> +#include \"fsmonitor--daemon.h\"\n>> +\n>> +struct fsm_health_data\n>> +{\n>> +\tHANDLE hEventShutdown;\n>> +\n>> +\tHANDLE hHandles[1]; /* the array does not own these handles */\n>> +#define HEALTH_SHUTDOWN 0\n>> +\tint nr_handles; /* number of active event handles */\n>> +};\n>> +\n>> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n>> +{\n>> +\tstruct fsm_health_data *data;\n>> +\n>> +\tCALLOC_ARRAY(data, 1);\n>> +\n>> +\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n>> +\n>> +\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n>> +\tdata->nr_handles++;\n>> +\n>> +\tstate->health_data = data;\n>> +\treturn 0;\n>> +}\n>> +\n>> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n>> +{\n>> +\tstruct fsm_health_data *data;\n>> +\n>> +\tif (!state || !state->health_data)\n>> +\t\treturn;\n>> +\n>> +\tdata = state->health_data;\n>> +\n>> +\tCloseHandle(data->hEventShutdown);\n>> +\n>> +\tFREE_AND_NULL(state->health_data);\n>> +}\n>> +\n>> +void fsm_health__loop(struct fsmonitor_daemon_state *state)\n>> +{\n>> +\tstruct fsm_health_data *data = state->health_data;\n>> +\n>> +\tfor (;;) {\n>> +\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n>> +\t\t\t\t\t\t data->hHandles,\n>> +\t\t\t\t\t\t FALSE, INFINITE);\n>> +\n>> +\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n>> +\t\t\tgoto clean_shutdown;\n>> +\n>> +\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n>> +\t\t GetLastError());\n>> +\t\tgoto force_error_stop;\n>> +\t}\n>> +\n>> +force_error_stop:\n>> +\tstate->health_error_code = -1;\n>> +\tipc_server_stop_async(state->ipc_server_data);\n>> +clean_shutdown:\n>> +\treturn;\n>> +}\n>> +\n>> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n>> +{\n>> +\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n>> +}\n> \n> But it apppears the Windows code is actually implemented. Did you\n> mean to do that as separate step, or should the commit message\n> mention that the Windows implementation is included?\n\nThe Windows version stubs in just enough of the thread-proc to keep\nthe health thread alive and waiting for a shutdown event. It doesn'ta\nactually have any health monitor in it yet.\n\nI'll update the commit message to clarify.\n\nThanks\nJeff\n\n"},{"id":"450287","messageId":"47e413dd-e748-a698-8c69-d24c8137fc1a@jeffhostetler.com","threadId":"57414","inReplyTo":"f9249541-bb92-1c61-825f-44a034a17bb8@github.com","subject":"Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-03T16:40:34Z","receivedAt":"2022-03-03T16:40:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 11:15 AM, Derrick Stolee wrote:\n> On 2/24/2022 11:04 AM, Derrick Stolee wrote:\n>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>>> From: Jeff Hostetler \n>>>\n>>> Create another thread to watch over the daemon process and\n>>> automatically shut it down if necessary.\n>>>\n>>> This commit creates the basic framework for a \"health\" thread\n>>> to monitor the daemon and/or the file system. Later commits\n>>> will add platform-specific code to do the actual work.\n>>\n>> ...\n>>\n>>> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\n>>> new file mode 100644\n>>> index 00000000000..b9f709e8548\n>>> --- /dev/null\n>>> +++ b/compat/fsmonitor/fsm-health-darwin.c\n>>> @@ -0,0 +1,24 @@\n>>> +#include \"cache.h\"\n>>> +#include \"config.h\"\n>>> +#include \"fsmonitor.h\"\n>>> +#include \"fsm-health.h\"\n>>> +#include \"fsmonitor--daemon.h\"\n>>> +\n>>> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n>>> +{\n>>> +\treturn 0;\n>>> +}\n>>> +\n>>> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n>>> +{\n>>> +\treturn;\n>>> +}\n>>> +\n>>> +void fsm_health__loop(struct fsmonitor_daemon_state *state)\n>>> +{\n>>> +\treturn;\n>>> +}\n>>> +\n>>> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n>>> +{\n>>> +}\n>>\n>> The macOS implementation is stubbed, as advertised.\n> \n> After looking at the rest of the patch series, it seems that these\n> are never filled in. Are some of the win32 health monitors also\n> appropriate for macOS? (They would need platform-specific checks,\n> probably.)\n> \n> Thanks,\n> -Stolee\n> \n"},{"id":"450290","messageId":"3ac0116c-4d82-55d0-6e17-75c3de392bbf@jeffhostetler.com","threadId":"57414","inReplyTo":"f9249541-bb92-1c61-825f-44a034a17bb8@github.com","subject":"Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-03T16:50:24Z","receivedAt":"2022-03-03T16:50:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 11:15 AM, Derrick Stolee wrote:\n> On 2/24/2022 11:04 AM, Derrick Stolee wrote:\n>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>>> From: Jeff Hostetler \n>>>\n>>> Create another thread to watch over the daemon process and\n>>> automatically shut it down if necessary.\n>>>\n>>> This commit creates the basic framework for a \"health\" thread\n>>> to monitor the daemon and/or the file system. Later commits\n>>> will add platform-specific code to do the actual work.\n>>\n>> ...\n>>\n>>> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\n>>> new file mode 100644\n>>> index 00000000000..b9f709e8548\n>>> --- /dev/null\n>>> +++ b/compat/fsmonitor/fsm-health-darwin.c\n>>> @@ -0,0 +1,24 @@\n>>> +#include \"cache.h\"\n>>> +#include \"config.h\"\n>>> +#include \"fsmonitor.h\"\n>>> +#include \"fsm-health.h\"\n>>> +#include \"fsmonitor--daemon.h\"\n>>> +\n>>> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n>>> +{\n>>> +\treturn 0;\n>>> +}\n>>> +\n>>> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n>>> +{\n>>> +\treturn;\n>>> +}\n>>> +\n>>> +void fsm_health__loop(struct fsmonitor_daemon_state *state)\n>>> +{\n>>> +\treturn;\n>>> +}\n>>> +\n>>> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n>>> +{\n>>> +}\n>>\n>> The macOS implementation is stubbed, as advertised.\n> \n> After looking at the rest of the patch series, it seems that these\n> are never filled in. Are some of the win32 health monitors also\n> appropriate for macOS? (They would need platform-specific checks,\n> probably.)\n\nYes, there are some asymmetries here. For example: On MacOS we\nget a notification at part of the existing watch if the repo root\ndirectory is moved or renamed, so (in a later commit) we can add\ncode to the listener thread to check for that; however, on Windows\nwe don't, so we have to poll for it in the health thread using\ntimers and the BY_HANDLE_FILE_INFORMATION data.\n\nHere I'm just stubbing in a trivial health thread so that the framework\nis complete for the supported platforms.\n\nThanks,\nJeff\n\n"},{"id":"450296","messageId":"2812e2af-4a63-09ae-89c9-bab8fb487100@jeffhostetler.com","threadId":"57414","inReplyTo":"c0d69427-aee1-7341-5b5f-1fe44ee0542d@github.com","subject":"Re: [PATCH 19/23] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-03T18:00:08Z","receivedAt":"2022-03-03T18:00:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 11:09 AM, Derrick Stolee wrote:\n> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\n>> index 3c3453369cd..2526ad9194f 100644\n>> --- a/compat/fsmonitor/fsm-health-win32.c\n>> +++ b/compat/fsmonitor/fsm-health-win32.c\n>> @@ -14,7 +14,10 @@ enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };\n>> typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n>> \t\t\t enum interval_fn_ctx ctx);\n>> \n>> +static interval_fn has_worktree_moved;\n>> +\n>> static interval_fn *table[] = {\n>> +\thas_worktree_moved,\n>> \tNULL, /* must be last */\n>> };\n> \n> Looking at this now, I think table[] should be defined immediately\n> before fsm_health__loop() so it is easier to see how they interact.\n> It also avoids this static declaration of the function before its\n> implementation.\n> \n> Or, is there a reason it is so high up in the file?\n\nI don't think so. I was trying to keep all of the public API\nroutines at the bottom, but all of the static stuff is pretty\nmuch free to move around.\n\nI'll revisit.\n\nThanks\nJeff\n"},{"id":"450458","messageId":"8a32ed13-ed53-3605-ca78-83dac2be1e28@jeffhostetler.com","threadId":"57414","inReplyTo":"20220224173305.gbr2waw77xpuieub@tb-raspi4","subject":"Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-04T23:40:27Z","receivedAt":"2022-03-04T23:40:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 12:33 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:\n> On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:\n>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>>> From: Jeff Hostetler \n>>>\n>>> Confirm that macOS FS events are reported with a normalized spelling.\n>>>\n>>> APFS (and/or HFS+) is case-insensitive. This means that case-independent\n> \n> This is not true, strictly speaking.\n> You can format a disk with \"case sensitive\" or \"case-insenstive, case preserving\".\n> Both APFS and HFS+ can be formated that way.\n> The default, which is what you get when you get a new machine,\n> is \"case-insenstive, case preserving\".\n> And I assume, that more 99% of all disks are formated that way.\n> The \"core.ignorecase\" is used in the same way as it is used under NTFS,\n> FAT or all other case-insenstive file systems.\n> (and even ext4 can be formated case-insensitive these days.)\n> \n> An interesting article can be found here:\n> https://lwn.net/Articles/784041/\n> \n> And to be technically correct, I think that even NTFS can be\n> \"configured to be case insensitive in an empty directory\".\n> \n> In that sense, I would like to avoid this statement, which\n> file system is case insensitive, and which is not.\n> Git assumes that after probing the FS in \"git init\" we have\n> a valid configuration in core.ignorecase.\n\nYou're right. I was incorrectly glossing over the differences\nbetween APFS and HFS+ -- and conflating case and nfc/nfd\nissues.\n\n[...]\n>>\n>>> NEEDSWORK: I was only able to test case. It would be nice to add tests\n>>\n>> \"I was only able test the APFS case.\"?\n\nI'm going to completely redo this commit in the next version.\nI now have both APFS and HFS+ partitions on my machine and\ncan compare the differences in behaviors and will have a new\nset of tests to cover this.\n\n>>\n>>> that use different Unicode spellings/normalizations and understand the\n>>> differences between APFS and HFS+ in this area. We should confirm that\n>>> the spelling of the workdir paths that the daemon sends to clients are\n>>> always properly normalized.\n>>\n>> Are there any macOS experts out there who can help us find the answers\n>> to these questions?\n> \n> There is a difference between HFS+ and APFS.\n> HFS+ is \"unicode decomposing\" when you call readdir() - file names\n> are stored decomposed on disk once created.\n> However, opening file in precompsed form succeds.\n> In that sense I would strongly suspect, that any monitors are \"sending\"\n> the decomposed form (on HFS+).\n> \n> APFS does not manipulate file names in that way, it is\n> \"unicode normalization preserving and ignoring\".\n\nIt took a few hours of poking to figure out what Apple is doing,\nbut yes on HFS+ they convert to NFD and use that as the on-disk\nformat. And they do collision detection as they always have in\nNFD-space.\n\nWhereas on APFS, they preserve the NFC/NFD as given when the file\nis created, but always do the same collision detection in NFD-space.\nThe net result is similar, but subtlety different.\n\nFS Events from MacOS are sent using the on-disk format (NFD on HFS+\nand whichever on APFS) and my FSMonitor daemon is sending them to\nthe client as received.\n\nI'm not sure whether or not the daemon should respect the\n`core.precompseUnicode` setting and when watching an HFS+\nvolume do the NFD-->NFC conversion for the client. I'm not\nsure whether that would be any more or less correct than just\nreporting the paths as received. I'm going to leave this as a\nquestion for the future.\n\n\nThanks for all of your background information on this topic.\nJeff\n\n\n"},{"id":"450462","messageId":"2bbc2802-8ab8-74ec-3fca-a933100752f3@jeffhostetler.com","threadId":"57414","inReplyTo":"20220224173305.gbr2waw77xpuieub@tb-raspi4","subject":"Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-04T23:47:46Z","receivedAt":"2022-03-04T23:47:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 12:33 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:\n> On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:\n>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>>> From: Jeff Hostetler \n>>>\n>>> Confirm that macOS FS events are reported with a normalized spelling.\n>>>\n>>> APFS (and/or HFS+) is case-insensitive. This means that case-independent\n> \n[...]\n> \n> An interesting article can be found here:\n> https://lwn.net/Articles/784041/\n> \n> And to be technically correct, I think that even NTFS can be\n> \"configured to be case insensitive in an empty directory\".\n\nYes, it is now possible to have NTFS be case sensitive (on a\ndirectory by directory basis). I haven't had a chance to\nexperiment with this yet, but I'm hoping that if we can always\nhave the daemon report using the on-disk spelling, we can\navoid most of this insanity.\n\nThanks,\nJeff\n"},{"id":"450487","messageId":"20220305085916.7rn6lhmbjkebrm5n@tb-raspi4","threadId":"57414","inReplyTo":"8a32ed13-ed53-3605-ca78-83dac2be1e28@jeffhostetler.com","subject":"Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode","fromName":"Torsten B��gershausen","fromEmail":"tboegi@web.de","sentAt":"2022-03-05T08:59:16Z","receivedAt":"2022-03-05T08:59:28Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, Mar 04, 2022 at 06:40:27PM -0500, Jeff Hostetler wrote:\n>\n>\n> On 2/24/22 12:33 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:\n> > On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:\n> > > On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n> > > > From: Jeff Hostetler \n> > > >\n> > > > Confirm that macOS FS events are reported with a normalized spelling.\n> > > >\n> > > > APFS (and/or HFS+) is case-insensitive. This means that case-independent\n> >\n> > This is not true, strictly speaking.\n> > You can format a disk with \"case sensitive\" or \"case-insenstive, case preserving\".\n> > Both APFS and HFS+ can be formated that way.\n> > The default, which is what you get when you get a new machine,\n> > is \"case-insenstive, case preserving\".\n> > And I assume, that more 99% of all disks are formated that way.\n> > The \"core.ignorecase\" is used in the same way as it is used under NTFS,\n> > FAT or all other case-insenstive file systems.\n> > (and even ext4 can be formated case-insensitive these days.)\n> >\n> > An interesting article can be found here:\n> > https://lwn.net/Articles/784041/\n> >\n> > And to be technically correct, I think that even NTFS can be\n> > \"configured to be case insensitive in an empty directory\".\n> >\n> > In that sense, I would like to avoid this statement, which\n> > file system is case insensitive, and which is not.\n> > Git assumes that after probing the FS in \"git init\" we have\n> > a valid configuration in core.ignorecase.\n>\n> You're right. I was incorrectly glossing over the differences\n> between APFS and HFS+ -- and conflating case and nfc/nfd\n> issues.\n>\n> [...]\n> > >\n> > > > NEEDSWORK: I was only able to test case. It would be nice to add tests\n> > >\n> > > \"I was only able test the APFS case.\"?\n>\n> I'm going to completely redo this commit in the next version.\n> I now have both APFS and HFS+ partitions on my machine and\n> can compare the differences in behaviors and will have a new\n> set of tests to cover this.\n>\n> > >\n> > > > that use different Unicode spellings/normalizations and understand the\n> > > > differences between APFS and HFS+ in this area. We should confirm that\n> > > > the spelling of the workdir paths that the daemon sends to clients are\n> > > > always properly normalized.\n> > >\n> > > Are there any macOS experts out there who can help us find the answers\n> > > to these questions?\n> >\n> > There is a difference between HFS+ and APFS.\n> > HFS+ is \"unicode decomposing\" when you call readdir() - file names\n> > are stored decomposed on disk once created.\n> > However, opening file in precompsed form succeds.\n> > In that sense I would strongly suspect, that any monitors are \"sending\"\n> > the decomposed form (on HFS+).\n> >\n> > APFS does not manipulate file names in that way, it is\n> > \"unicode normalization preserving and ignoring\".\n>\n> It took a few hours of poking to figure out what Apple is doing,\n> but yes on HFS+ they convert to NFD and use that as the on-disk\n> format. And they do collision detection as they always have in\n> NFD-space.\n>\n> Whereas on APFS, they preserve the NFC/NFD as given when the file\n> is created, but always do the same collision detection in NFD-space.\n> The net result is similar, but subtlety different.\n\nThat depends what you mean with \"net result\".\nWhat Git sees after calling precompose_utf8_readdir() with\ncore.precomposeunicode=true ?\n\n\n>\n> FS Events from MacOS are sent using the on-disk format (NFD on HFS+\n> and whichever on APFS) and my FSMonitor daemon is sending them to\n> the client as received.\n>\n> I'm not sure whether or not the daemon should respect the\n> `core.precompseUnicode` setting and when watching an HFS+\n> volume do the NFD-->NFC conversion for the client. I'm not\n> sure whether that would be any more or less correct than just\n> reporting the paths as received. I'm going to leave this as a\n> question for the future.\n\nI think that I have a suggestion for an answer:\nWe still have HFS+ systems around, and we still have an NFD feature\nin MacOs for USB sticks with FAT or SAMBA mounted network volumes.\nBoth return NFD in readdir().\nEven if NFC is on disk for FAT or going over the wire for SAMBA.\nHaving a precompose() function in the FSMonitor would help to make\nthings consistent.\nAnd the answer is yes.\n\n>\n>\n> Thanks for all of your background information on this topic.\n> Jeff\n>\n\nThe pleasure is on my side.\nPlease feel free to cc me for the next round, so that I don't miss\nto review them.\n"},{"id":"450646","messageId":"15d59c97-bcb3-549d-0a22-f08ea0505d45@jeffhostetler.com","threadId":"57414","inReplyTo":"20220305085916.7rn6lhmbjkebrm5n@tb-raspi4","subject":"Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-07T20:45:43Z","receivedAt":"2022-03-07T20:45:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/5/22 3:59 AM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:\n> On Fri, Mar 04, 2022 at 06:40:27PM -0500, Jeff Hostetler wrote:\n>>\n>>\n>> On 2/24/22 12:33 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:\n>>> On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:\n>>>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>>>>> From: Jeff Hostetler \n>>>>>\n>>>>> Confirm that macOS FS events are reported with a normalized spelling.\n[...]\n>>\n>>>>\n>>>>> that use different Unicode spellings/normalizations and understand the\n>>>>> differences between APFS and HFS+ in this area. We should confirm that\n>>>>> the spelling of the workdir paths that the daemon sends to clients are\n>>>>> always properly normalized.\n>>>>\n>>>> Are there any macOS experts out there who can help us find the answers\n>>>> to these questions?\n>>>\n>>> There is a difference between HFS+ and APFS.\n>>> HFS+ is \"unicode decomposing\" when you call readdir() - file names\n>>> are stored decomposed on disk once created.\n>>> However, opening file in precompsed form succeds.\n>>> In that sense I would strongly suspect, that any monitors are \"sending\"\n>>> the decomposed form (on HFS+).\n>>>\n>>> APFS does not manipulate file names in that way, it is\n>>> \"unicode normalization preserving and ignoring\".\n>>\n>> It took a few hours of poking to figure out what Apple is doing,\n>> but yes on HFS+ they convert to NFD and use that as the on-disk\n>> format. And they do collision detection as they always have in\n>> NFD-space.\n>>\n>> Whereas on APFS, they preserve the NFC/NFD as given when the file\n>> is created, but always do the same collision detection in NFD-space.\n>> The net result is similar, but subtlety different.\n> \n> That depends what you mean with \"net result\".\n> What Git sees after calling precompose_utf8_readdir() with\n> core.precomposeunicode=true ?\n\nI just meant that the OS always does the collision detection\nregardless of whether the filesystem is APFS or HFS. (They even\ndo it on a FAT32 thumb drive). So the OS layer is always\ncomposition insensitive while the underlying filesystem may\nor may not be composition preserving.\n\n\n>>\n>> FS Events from MacOS are sent using the on-disk format (NFD on HFS+\n>> and whichever on APFS) and my FSMonitor daemon is sending them to\n>> the client as received.\n>>\n>> I'm not sure whether or not the daemon should respect the\n>> `core.precompseUnicode` setting and when watching an HFS+\n>> volume do the NFD-->NFC conversion for the client. I'm not\n>> sure whether that would be any more or less correct than just\n>> reporting the paths as received. I'm going to leave this as a\n>> question for the future.\n> \n> I think that I have a suggestion for an answer:\n> We still have HFS+ systems around, and we still have an NFD feature\n> in MacOs for USB sticks with FAT or SAMBA mounted network volumes.\n> Both return NFD in readdir().\n> Even if NFC is on disk for FAT or going over the wire for SAMBA.\n> Having a precompose() function in the FSMonitor would help to make\n> things consistent.\n> And the answer is yes.\n\nI agree. I'm going to have the mac version always send the NFC\nversion (like core.precomposeUnicode suggests), but also send NFD\nfor *if* we get an NFD spelling from the FS event. So on an HFS\nvolume we'll get two events for each path that have different\nUTF8 spellings. In the case of APFS we should only get the NFC\nspelling -- unless the user really wants to use NFD spellings --\nin which case we will respect that and send both.\n\nThis maybe be overkill, but the daemon (at the time that it\nreceives the FS event) doesn't know how the client process will\nbe configured when it makes a query at some point in the future.\nSo by recording both and reporting them to the client, the client\ncan decide how to process them.\n\nWhichever spelling the client sees, it is only using it to mark\nthe cache-entry or untracked-cache data as possibly dirty so that\nthe original scanning code can actually check it. This is a little\ndifferent from the diff-tree code that needs the official spelling\nfrom the tree entry to match up items with the working directory.\n\nI've overhauled all of the MacOS Unicode handling in V2.\n\nJeff\n\n"},{"id":"450648","messageId":"eccdf8b7-93b5-f984-7754-622fa7c7771c@jeffhostetler.com","threadId":"57414","inReplyTo":"e3c0a046-8041-421d-6640-91d1a560f79b@github.com","subject":"Re: [PATCH 00/23] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-07T21:23:45Z","receivedAt":"2022-03-07T21:23:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/24/22 11:21 AM, Derrick Stolee wrote:\n> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>> Here is part 3 of my builtin FSMonitor series.\n[...]\n>> Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a\n>> synthetic repo containing 1M files on a Macbook Pro. It shows the effects of\n>> the untracked cache (uc) and FSMonitor (fsm) on git status.\n>>\n>> $ ./p7527-builtin-fsmonitor.sh\n>> # passed all 67 test(s)\n>> 1..67\n>> Test this tree\n>> ---------------------------------------------------------------------------------------\n>> 7527.4: [uc false][fsm false] status after checkout 29.99(3.14+80.12)\n>> 7527.6: [uc false][fsm false] status after big change 73.32(5.11+97.24)\n>> 7527.8: [uc false][fsm false] status after add all 47.80(5.12+90.47)\n>> 7527.10: [uc false][fsm false] status after add dot 49.22(5.16+92.05)\n>> 7527.12: [uc false][fsm false] status after commit 51.53(3.35+100.74)\n>> 7527.14: [uc false][fsm false] status after reset hard 33.74(3.03+85.31)\n>> 7527.16: [uc false][fsm false] status after create untracked files 41.71(3.24+89.75)\n>> 7527.18: [uc false][fsm false] status after clean 34.33(3.07+89.36)\n>>\n>> 7527.20: [uc false][fsm true] status after checkout 29.23(1.94+10.84)\n>> 7527.22: [uc false][fsm true] status after big change 64.23(4.66+24.86)\n>> 7527.24: [uc false][fsm true] status after add all 45.45(4.37+18.70)\n>> 7527.26: [uc false][fsm true] status after add dot 44.42(4.02+17.10)\n>> 7527.28: [uc false][fsm true] status after commit 30.52(1.95+10.91)\n>> 7527.30: [uc false][fsm true] status after reset hard 28.70(2.70+13.89)\n>> 7527.32: [uc false][fsm true] status after create untracked files 28.63(2.59+10.71)\n>> 7527.34: [uc false][fsm true] status after clean 28.97(2.59+10.78)\n>>\n>> 7527.36: [uc true][fsm false] status after checkout 35.06(3.17+86.11)\n>> 7527.38: [uc true][fsm false] status after big change 74.65(5.14+101.50)\n>> 7527.40: [uc true][fsm false] status after add all 49.96(5.22+90.96)\n>> 7527.42: [uc true][fsm false] status after add dot 49.77(5.24+91.72)\n>> 7527.44: [uc true][fsm false] status after commit 36.95(3.27+92.25)\n>> 7527.46: [uc true][fsm false] status after reset hard 33.89(3.18+85.68)\n>> 7527.48: [uc true][fsm false] status after create untracked files 41.44(3.40+92.99)\n>> 7527.50: [uc true][fsm false] status after clean 34.60(3.26+90.19)\n>>\n>> 7527.52: [uc true][fsm true] status after checkout 0.58(0.45+0.10)\n>> 7527.54: [uc true][fsm true] status after big change 65.16(4.91+25.64)\n>> 7527.56: [uc true][fsm true] status after add all 45.43(4.45+18.92)\n>> 7527.58: [uc true][fsm true] status after add dot 15.56(2.57+6.32)\n>> 7527.60: [uc true][fsm true] status after commit 0.98(0.46+0.11)\n>> 7527.62: [uc true][fsm true] status after reset hard 30.30(2.96+14.49)\n>> 7527.64: [uc true][fsm true] status after create untracked files 2.15(1.73+0.40)\n>> 7527.66: [uc true][fsm true] status after clean 1.68(1.56+0.32)\n> \n> The other stylistic thing is this performance test. It would be nice if\n> these tests were grouped by the operation (like \"status after checkout\")\n> so it is easier to compare the same operation across the matrix definitions.\n> \n> This would require reordering the test definition as well as allowing the\n> different cases to simultaneously live in different repositories. The\n> p2000-sparse-operations.sh has this kind of organization, but you'll need\n> more involved test cases than \"run this command\".\n\nYeah, it would be nice to turn this test inside-out so that\ncould group the outputs by test case rather than by (uc,fsm)\ncombination. That would certainly make it easier to see how\nthe two terms affect things.\n\nThe problem is I'd either need 4 parallel repos that I could\nsetup with each (uc,fsm) pair or I'd need to start/stop the\ndaemon and swap out the {.git/index, .git/config} between\neach step. The former is a problem for monorepos. The latter\nis doable, but I'm not sure it is worth the effort right now.\n\nJeff\n"},{"id":"450780","messageId":"34619e0652b4723de40d9b8972ec45aaac9009e6.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 01/27] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:01Z","receivedAt":"2022-03-08T22:15:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach FSMonitor daemon on Windows to recognize shortname paths as\naliases of normal longname paths. FSMonitor clients, such as `git\nstatus`, should receive the longname spelling of changed files (when\npossible).\n\nSometimes we receive FS events using the shortname, such as when a CMD\nshell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\narrives using whatever combination of long and shortnames were used by\nthe other process. (Shortnames do seem to be case normalized,\nhowever.)\n\nUse Windows GetLongPathNameW() to try to map the pathname spelling in\nthe notification event into the normalized longname spelling. (This\ncan fail if the file/directory is deleted, moved, or renamed, because\nwe are asking the FS for the mapping in response to the event and\nafter it has already happened, but we try.)\n\nSpecial case the shortname spelling of \".git\" to avoid under-reporting\nthese events.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++-----\n t/t7527-builtin-fsmonitor.sh | 65 +++++\n 2 files changed, 374 insertions(+), 54 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex c2d11acbc1e..f4673d7d8b1 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -25,6 +25,9 @@ struct one_watch\n \tDWORD count;\n \n \tstruct strbuf path;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n+\tDWORD wpath_longname_len;\n+\n \tHANDLE hDir;\n \tHANDLE hEvent;\n \tOVERLAPPED overlapped;\n@@ -34,6 +37,21 @@ struct one_watch\n \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n \t */\n \tBOOL is_active;\n+\n+\t/*\n+\t * Are shortnames enabled on the containing drive? This is\n+\t * always true for \"C:/\" drives and usually never true for\n+\t * other drives.\n+\t *\n+\t * We only set this for the worktree because we only need to\n+\t * convert shortname paths to longname paths for items we send\n+\t * to clients. (We don't care about shortname expansion for\n+\t * paths inside a GITDIR because we never send them to\n+\t * clients.)\n+\t */\n+\tBOOL has_shortnames;\n+\tBOOL has_tilda;\n+\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n struct fsmonitor_daemon_backend_data\n@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n };\n \n /*\n- * Convert the WCHAR path from the notification into UTF8 and\n- * then normalize it.\n+ * Convert the WCHAR path from the event into UTF8 and normalize it.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n */\n-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n \t\t\t\t struct strbuf *normalized_path)\n {\n \tint reserve;\n \tint len = 0;\n \n \tstrbuf_reset(normalized_path);\n-\tif (!info->FileNameLength)\n+\tif (!wpath_len)\n \t\tgoto normalize;\n \n \t/*\n@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t * sequence of 2 UTF8 characters. That should let us\n \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n \t */\n-\treserve = info->FileNameLength + 1;\n+\treserve = 2 * wpath_len + 1;\n \tstrbuf_grow(normalized_path, reserve);\n \n \tfor (;;) {\n-\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n-\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n+\t\t\t\t\t wpath, wpath_len,\n \t\t\t\t\t normalized_path->buf,\n \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n \t\t\t\t\t NULL, NULL);\n@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n \t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n-\t\t\t GetLastError(),\n-\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n-\t\t\t info->FileName);\n+\t\t\t GetLastError(), (int)wpath_len, wpath);\n \t\t\treturn -1;\n \t\t}\n \n@@ -98,6 +115,152 @@ normalize:\n \treturn strbuf_normalize_path(normalized_path);\n }\n \n+/*\n+ * See if the worktree root directory has shortnames enabled.\n+ * This will help us decide if we need to do an expensive shortname\n+ * to longname conversion on every notification event.\n+ *\n+ * We do not want to create a file to test this, so we assume that the\n+ * root directory contains a \".git\" file or directory. (Our caller\n+ * only calls us for the worktree root, so this should be fine.)\n+ *\n+ * Remember the spelling of the shortname for \".git\" if it exists.\n+ */\n+static void check_for_shortnames(struct one_watch *watch)\n+{\n+\twchar_t buf_in[MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\twchar_t *last_slash = NULL;\n+\twchar_t *last_bslash = NULL;\n+\twchar_t *last;\n+\n+\t/* build L\"/.git\" */\n+\twcscpy(buf_in, watch->wpath_longname);\n+\twcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n+\n+\tif (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n+\t\treturn;\n+\n+\tlast_slash = wcsrchr(buf_out, L'/');\n+\tlast_bslash = wcsrchr(buf_out, L'\\\\');\n+\tif (last_slash > last_bslash)\n+\t\tlast = last_slash + 1;\n+\telse if (last_bslash)\n+\t\tlast = last_bslash + 1;\n+\telse\n+\t\tlast = buf_out;\n+\n+\tif (!wcscmp(last, L\".git\"))\n+\t\treturn;\n+\n+\twatch->has_shortnames = 1;\n+\twcsncpy(watch->dotgit_shortname, last,\n+\t\tARRAY_SIZE(watch->dotgit_shortname));\n+\n+\t/*\n+\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n+\t * we should be able to avoid shortname to longname mapping on\n+\t * every notification event if the source string does not\n+\t * contain a \"~\".\n+\t *\n+\t * However, the documentation for GetLongPathNameW() says\n+\t * that there are filesystems that don't follow that pattern\n+\t * and warns against this optimization.\n+\t *\n+\t * Lets test this.\n+\t */\n+\tif (wcschr(watch->dotgit_shortname, L'~'))\n+\t\twatch->has_tilda = 1;\n+}\n+\n+enum get_relative_result {\n+\tGRR_NO_CONVERSION_NEEDED,\n+\tGRR_HAVE_CONVERSION,\n+\tGRR_SHUTDOWN,\n+};\n+\n+/*\n+ * Info notification paths are relative to the root of the watch.\n+ * If our CWD is still at the root, then we can use relative paths\n+ * to convert from shortnames to longnames. If our process has a\n+ * different CWD, then we need to construct an absolute path, do\n+ * the conversion, and then return the root-relative portion.\n+ *\n+ * We use the longname form of the root as our basis and assume that\n+ * it already has a trailing slash.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n+ */\n+static enum get_relative_result get_relative_longname(\n+\tstruct one_watch *watch,\n+\tconst wchar_t *wpath, DWORD wpath_len,\n+\twchar_t *wpath_longname)\n+{\n+\twchar_t buf_in[2 * MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\tDWORD root_len;\n+\n+\t/* Build L\"/\" */\n+\troot_len = watch->wpath_longname_len;\n+\twcsncpy(buf_in, watch->wpath_longname, root_len);\n+\twcsncpy(buf_in + root_len, wpath, wpath_len);\n+\tbuf_in[root_len + wpath_len] = 0;\n+\n+\t/*\n+\t * We don't actually know if the source pathname is a\n+\t * shortname or a longname. This routine allows either to be\n+\t * given as input.\n+\t */\n+\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n+\t\t/*\n+\t\t * The shortname to longname conversion can fail for\n+\t\t * various reasons, for example if the file has been\n+\t\t * deleted. (That is, if we just received a\n+\t\t * delete-file notification event and the file is\n+\t\t * already gone, we can't ask the file system to\n+\t\t * lookup the longname for it. Likewise, for moves\n+\t\t * and renames where we are given the old name.)\n+\t\t *\n+\t\t * Since deleting or moving a file or directory by its\n+\t\t * shortname is rather obscure, I'm going ignore the\n+\t\t * failure and ask the caller to report the original\n+\t\t * relative path. This seems kinder than failing here\n+\t\t * and forcing a resync. Besides, forcing a resync on\n+\t\t * every file/directory delete would effectively\n+\t\t * cripple monitoring.\n+\t\t *\n+\t\t * We might revisit this in the future.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (!wcscmp(buf_in, buf_out)) {\n+\t\t/*\n+\t\t * The path does not have a shortname alias.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (wcsncmp(buf_in, buf_out, root_len)) {\n+\t\t/*\n+\t\t * The spelling of the root directory portion of the computed\n+\t\t * longname has changed. This should not happen. Basically,\n+\t\t * it means that we don't know where (without recomputing the\n+\t\t * longname of just the root directory) to split out the\n+\t\t * relative path. Since this should not happen, I'm just\n+\t\t * going to let this fail and force a shutdown (because all\n+\t\t * subsequent events are probably going to see the same\n+\t\t * mismatch).\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\t/* Return the worktree root-relative portion of the longname. */\n+\n+\twcscpy(wpath_longname, buf_out + root_len);\n+\treturn GRR_HAVE_CONVERSION;\n+}\n+\n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n@@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n-\twchar_t wpath[MAX_PATH];\n+\tDWORD len_longname;\n+\twchar_t wpath[MAX_PATH + 1];\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \tif (xutftowcs_path(wpath, path) < 0) {\n \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n@@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \t\treturn NULL;\n \t}\n \n+\tif (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n+\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n+\t\t GetLastError(), path);\n+\t\tCloseHandle(hDir);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen_longname = wcslen(wpath_longname);\n+\tif (wpath_longname[len_longname - 1] != L'/' &&\n+\t wpath_longname[len_longname - 1] != L'\\\\') {\n+\t\twpath_longname[len_longname++] = L'/';\n+\t\twpath_longname[len_longname] = 0;\n+\t}\n+\n \tCALLOC_ARRAY(watch, 1);\n \n \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n@@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tstrbuf_init(&watch->path, 0);\n \tstrbuf_addstr(&watch->path, path);\n \n+\twcscpy(watch->wpath_longname, wpath_longname);\n+\twatch->wpath_longname_len = len_longname;\n+\n \twatch->hDir = hDir;\n \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n \n@@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t,\n+\tDWORD info_action)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n+\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n \tstruct fsmonitor_batch *batch = NULL;\n \tconst char *p = watch->buffer;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \t/*\n \t * If the kernel gets more events than will fit in the kernel\n@@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n+\t\twchar_t *wpath = info->FileName;\n+\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n \t\tenum fsmonitor_path_type t;\n+\t\tenum get_relative_result grr;\n+\n+\t\tif (watch->has_shortnames) {\n+\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n+\t\t\t\t/*\n+\t\t\t\t * This event exactly matches the\n+\t\t\t\t * spelling of the shortname of\n+\t\t\t\t * \".git\", so we can skip some steps.\n+\t\t\t\t *\n+\t\t\t\t * (This case is odd because the user\n+\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n+\t\t\t\t * use the filesystem to map it back\n+\t\t\t\t * to \".git\".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_reset(&path);\n+\t\t\t\tstrbuf_addstr(&path, \".git\");\n+\t\t\t\tt = IS_DOT_GIT;\n+\t\t\t\tgoto process_it;\n+\t\t\t}\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n-\t\t\tgoto skip_this_path;\n-\n-\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n-\n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n+\t\t\tif (watch->has_tilda && !wcschr(wpath, L'~')) {\n+\t\t\t\t/*\n+\t\t\t\t * Shortnames on this filesystem have tildas\n+\t\t\t\t * and the notification path does not have\n+\t\t\t\t * one, so we assume that it is a longname.\n+\t\t\t\t */\n+\t\t\t\tgoto normalize_it;\n+\t\t\t}\n \n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n+\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n+\t\t\t\t\t\t wpath_longname);\n+\t\t\tswitch (grr) {\n+\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n+\t\t\t\tbreak;\n+\t\t\tcase GRR_HAVE_CONVERSION:\n+\t\t\t\twpath = wpath_longname;\n+\t\t\t\twpath_len = wcslen(wpath);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\tcase GRR_SHUTDOWN:\n \t\t\t\tgoto force_shutdown;\n \t\t\t}\n-\t\t\tbreak;\n+\t\t}\n \n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n+normalize_it:\n+\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n+\t\t\tgoto skip_this_path;\n \n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+process_it:\n+\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n+\t\t\t\t\t info->Action))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -382,6 +630,9 @@ force_shutdown:\n * Note that we DO NOT get filesystem events on the external \n * itself (it is not inside something that we are watching). In particular,\n * we do not get an event if the external is deleted.\n+ *\n+ * Also, we do not care about shortnames within the external , since\n+ * we never send these paths to clients.\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n@@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\tif (normalize_path_in_utf8(\n+\t\t\t info->FileName,\n+\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t &path) == -1)\n \t\t\tgoto skip_this_path;\n \n \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n@@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \tif (!data->watch_worktree)\n \t\tgoto failed;\n \n+\tcheck_for_shortnames(data->watch_worktree);\n+\n \tif (state->nr_paths_watching > 1) {\n \t\tdata->watch_gitdir = create_watch(state,\n \t\t\t\t\t\t state->path_gitdir_watch.buf);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 0ccbfb9616f..dbca7f835eb 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -123,6 +123,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n+\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n+\n+\tstart_daemon test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"450781","messageId":"3a0f30b849a85ee0f2737a08fcf6a746020efdcf.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 02/27] t7527: test FSMonitor on repos with Unicode root paths","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:02Z","receivedAt":"2022-03-08T22:15:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate some test repos with UTF8 characters in the pathname of the\nroot directory and verify that the builtin FSMonitor can watch them.\n\nThis test is mainly for Windows where we need to avoid `*A()`\nroutines.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex dbca7f835eb..c2627f28865 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -666,4 +666,27 @@ do\n \tdone\n done\n \n+# Test Unicode UTF-8 characters in the pathname of the working\n+# directory root. Use of \"*A()\" routines rather than \"*W()\" routines\n+# on Windows can sometimes lead to odd failures.\n+#\n+u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n+u2=$(printf \"u_e2_99_ab__\\xE2\\x99\\xAB\")\n+u_values=\"$u1 $u2\"\n+for u in $u_values\n+do\n+\ttest_expect_success \"Unicode in repo root path: $u\" '\n+\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n+\n+\t\tgit init \"$u\" &&\n+\t\techo 1 >\"$u\"/file1 &&\n+\t\tgit -C \"$u\" add file1 &&\n+\t\tgit -C \"$u\" config core.fsmonitor true &&\n+\n+\t\tstart_daemon \"$u\" &&\n+\t\tgit -C \"$u\" status >actual &&\n+\t\tgrep \"new file: file1\" actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450782","messageId":"87d1c0b6f2afda53bc8a8fc5949af6caca5a0329.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 03/27] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:03Z","receivedAt":"2022-03-08T22:15:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a stress test to hammer on the fsmonitor daemon.\nCreate a client-side thread pool of n threads and have\neach of them make m requests as fast as they can.\n\nWe do not currently inspect the contents of the response.\nWe're only interested in placing a heavy request load on\nthe daemon.\n\nThis test is useful for interactive testing and various\nexperimentation. For example, to place additional load\non the daemon while another test is running. We currently\ndo not have a test script that actually uses this helper.\nWe might add such a test in the future.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++\n 1 file changed, 106 insertions(+)\n\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nindex f7a5b3a32fa..985340ba719 100644\n--- a/t/helper/test-fsmonitor-client.c\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -7,6 +7,8 @@\n #include \"cache.h\"\n #include \"parse-options.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2.h\"\n \n #ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n int cmd__fsmonitor_client(int argc, const char **argv)\n@@ -79,20 +81,121 @@ static int do_send_flush(void)\n \treturn 0;\n }\n \n+struct hammer_thread_data\n+{\n+\tpthread_t pthread_id;\n+\tint thread_nr;\n+\n+\tint nr_requests;\n+\tconst char *token;\n+\n+\tint sum_successful;\n+\tint sum_errors;\n+};\n+\n+static void *hammer_thread_proc(void *_hammer_thread_data)\n+{\n+\tstruct hammer_thread_data *data = _hammer_thread_data;\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint k;\n+\tint ret;\n+\n+\ttrace2_thread_start(\"hammer\");\n+\n+\tfor (k = 0; k < data->nr_requests; k++) {\n+\t\tstrbuf_reset(&answer);\n+\n+\t\tret = fsmonitor_ipc__send_query(data->token, &answer);\n+\t\tif (ret < 0)\n+\t\t\tdata->sum_errors++;\n+\t\telse\n+\t\t\tdata->sum_successful++;\n+\t}\n+\n+\tstrbuf_release(&answer);\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Start a pool of client threads that will each send a series of\n+ * commands to the daemon.\n+ *\n+ * The goal is to overload the daemon with a sustained series of\n+ * concurrent requests.\n+ */\n+static int do_hammer(const char *token, int nr_threads, int nr_requests)\n+{\n+\tstruct hammer_thread_data *data = NULL;\n+\tint k;\n+\tint sum_join_errors = 0;\n+\tint sum_commands = 0;\n+\tint sum_errors = 0;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\tif (nr_threads < 1)\n+\t\tnr_threads = 1;\n+\tif (nr_requests < 1)\n+\t\tnr_requests = 1;\n+\n+\tCALLOC_ARRAY(data, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\t\tp->thread_nr = k;\n+\t\tp->nr_requests = nr_requests;\n+\t\tp->token = token;\n+\n+\t\tif (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {\n+\t\t\twarning(\"failed to create thread[%d] skipping remainder\", k);\n+\t\t\tnr_threads = k;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\n+\t\tif (pthread_join(p->pthread_id, NULL))\n+\t\t\tsum_join_errors++;\n+\t\tsum_commands += p->sum_successful;\n+\t\tsum_errors += p->sum_errors;\n+\t}\n+\n+\tfprintf(stderr, \"HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\\n\",\n+\t\tnr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);\n+\n+\tfree(data);\n+\n+\t/*\n+\t * Return an error if any of the _send_query requests failed.\n+\t * We don't care about thread create/join errors.\n+\t */\n+\treturn sum_errors > 0;\n+}\n+\n int cmd__fsmonitor_client(int argc, const char **argv)\n {\n \tconst char *subcmd;\n \tconst char *token = NULL;\n+\tint nr_threads = 1;\n+\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n \t\tN_(\"test-helper fsmonitor-client query []\"),\n \t\tN_(\"test-helper fsmonitor-client flush\"),\n+\t\tN_(\"test-helper fsmonitor-client hammer [] [] []\"),\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n \t\tOPT_STRING(0, \"token\", &token, N_(\"token\"),\n \t\t\t N_(\"command token to send to the server\")),\n+\n+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, N_(\"number of client threads\")),\n+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, N_(\"number of requests per thread\")),\n+\n \t\tOPT_END()\n \t};\n \n@@ -116,6 +219,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)\n \tif (!strcmp(subcmd, \"flush\"))\n \t\treturn !!do_send_flush();\n \n+\tif (!strcmp(subcmd, \"hammer\"))\n+\t\treturn !!do_hammer(token, nr_threads, nr_requests);\n+\n \tdie(\"Unhandled subcommand: '%s'\", subcmd);\n }\n #endif\n-- \ngitgitgadget\n\n"},{"id":"450783","messageId":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.git.1644940773.gitgitgadget@gmail.com","subject":"[PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:00Z","receivedAt":"2022-03-08T22:15:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V2 of part 3 of my builtin FSMonitor series.\n\nI think I have addressed all of the feedback from V1. This includes:\n\n[] AEvar's suggestion to simplify the API to get the repo incompatibility\nmessage.\n\n[] I updated the daemon on MacOS to report both the NFC and NFD spellings of\na pathname when appropriate. This is a little more general than the\n\"core.precomposeUnicode\" setting, since the daemon does not know how the\nclient has (or will have) it set when they make a query.\n\n[] I replaced my Unicode NFC/NFD test for MacOS to focus exclusively on\nUnicode composition/decomposition sensitivity and to not confuse that with\ncase sensitivity.\n\n[] Added code on MacOS to mark repos on FAT32 and NTFS volumes as\nincompatible with FSMonitor, since they don't support Unix domain sockets.\n\n[] I reordered some of the static functions in the Windows version of the\nhealth monitoring code to reduce the need for forward declarations.\n\n[] Lots of typos and cleanup.\n\nThis version has been rebased upon V6 of Part 2 which is currently in\n\"next\".\n\nI'm not going to repeat the perf test results in the V2 version of the cover\nletter; interested readers should look at the cover letter for V1.\n\nJeff Hostetler (27):\n fsm-listen-win32: handle shortnames\n t7527: test FSMonitor on repos with Unicode root paths\n t/helper/fsmonitor-client: create stress test\n fsmonitor-settings: bare repos are incompatible with FSMonitor\n fsmonitor-settings: stub in platform-specific incompatibility checking\n fsmonitor-settings: VFS for Git virtual repos are incompatible\n fsmonitor-settings: stub in macOS-specific incompatibility checking\n fsmonitor-settings: remote repos on macOS are incompatible\n fsmonitor-settings: remote repos on Windows are incompatible\n unpack-trees: initialize fsmonitor_has_run_once in o->result\n fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n fsmonitor--daemon: cd out of worktree root\n fsmonitor--daemon: prepare for adding health thread\n fsmonitor--daemon: rename listener thread related variables\n fsmonitor--daemon: stub in health thread\n fsm-health-win32: add polling framework to monitor daemon health\n fsm-health-win32: force shutdown daemon if worktree root moves\n fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n fsmonitor: optimize processing of directory events\n t7527: FSMonitor tests for directory moves\n t/perf/p7527: add perf test for builtin FSMonitor\n fsmonitor: never set CE_FSMONITOR_VALID on submodules\n t7527: test FSMonitor on case insensitive+preserving file system\n fsmonitor: on macOS also emit NFC spelling for NFD pathname\n t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n t7527: test Unicode NFC/NFD handling on MacOS\n fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n\n Makefile | 19 +-\n builtin/fsmonitor--daemon.c | 107 ++++++-\n builtin/update-index.c | 7 +\n compat/fsmonitor/fsm-health-darwin.c | 24 ++\n compat/fsmonitor/fsm-health-win32.c | 278 +++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 +++\n compat/fsmonitor/fsm-listen-darwin.c | 122 ++++++--\n compat/fsmonitor/fsm-listen-win32.c | 413 ++++++++++++++++++++-----\n compat/fsmonitor/fsm-listen.h | 2 +-\n compat/fsmonitor/fsm-settings-darwin.c | 89 ++++++\n compat/fsmonitor/fsm-settings-win32.c | 137 ++++++++\n config.mak.uname | 5 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 11 +-\n fsmonitor-settings.c | 81 +++++\n fsmonitor-settings.h | 29 ++\n fsmonitor.c | 73 ++++-\n fsmonitor.h | 11 +\n t/helper/test-fsmonitor-client.c | 106 +++++++\n t/lib-unicode-nfc-nfd.sh | 159 ++++++++++\n t/perf/p7527-builtin-fsmonitor.sh | 257 +++++++++++++++\n t/t7519-status-fsmonitor.sh | 32 ++\n t/t7527-builtin-fsmonitor.sh | 314 +++++++++++++++++++\n unpack-trees.c | 1 +\n 24 files changed, 2208 insertions(+), 124 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\n\nbase-commit: 1a9241e1fee9d418e436849853f031329e792192\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1143\n\nRange-diff vs v1:\n\n 1: 23f38338cec ! 1: 34619e0652b fsm-listen-win32: handle shortnames\n @@ compat/fsmonitor/fsm-listen-win32.c: normalize:\n + * to longname conversion on every notification event.\n + *\n + * We do not want to create a file to test this, so we assume that the\n -+ * root directory contains a \".git\" file or directory. (Out caller\n ++ * root directory contains a \".git\" file or directory. (Our caller\n + * only calls us for the worktree root, so this should be fine.)\n + *\n + * Remember the spelling of the shortname for \".git\" if it exists.\n @@ compat/fsmonitor/fsm-listen-win32.c: normalize:\n +\t\t * lookup the longname for it. Likewise, for moves\n +\t\t * and renames where we are given the old name.)\n +\t\t *\n -+\t\t * NEEDSWORK: Since deleting or moving a file or\n -+\t\t * directory by shortname is rather obscure, I'm going\n -+\t\t * ignore the failure and ask the caller to report the\n -+\t\t * original relative path. This seemds kinder than\n -+\t\t * failing here and forcing a resync.\n ++\t\t * Since deleting or moving a file or directory by its\n ++\t\t * shortname is rather obscure, I'm going ignore the\n ++\t\t * failure and ask the caller to report the original\n ++\t\t * relative path. This seems kinder than failing here\n ++\t\t * and forcing a resync. Besides, forcing a resync on\n ++\t\t * every file/directory delete would effectively\n ++\t\t * cripple monitoring.\n ++\t\t *\n ++\t\t * We might revisit this in the future.\n +\t\t */\n +\t\treturn GRR_NO_CONVERSION_NEEDED;\n +\t}\n 3: 9af952e4d17 ! 2: 3a0f30b849a t7527: test builtin FSMonitor watching repos with unicode paths\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - t7527: test builtin FSMonitor watching repos with unicode paths\n + t7527: test FSMonitor on repos with Unicode root paths\n \n - Create some test repos with UTF8 pathnames and verify that\n - the builtin FSMonitor can watch them. This test is mainly\n - for Windows where we need to avoid `*A()` routines.\n + Create some test repos with UTF8 characters in the pathname of the\n + root directory and verify that the builtin FSMonitor can watch them.\n +\n + This test is mainly for Windows where we need to avoid `*A()`\n + routines.\n \n Signed-off-by: Jeff Hostetler \n \n @@ t/t7527-builtin-fsmonitor.sh: do\n done\n \n +# Test Unicode UTF-8 characters in the pathname of the working\n -+# directory. Use of \"*A()\" routines rather than \"*W()\" routines\n ++# directory root. Use of \"*A()\" routines rather than \"*W()\" routines\n +# on Windows can sometimes lead to odd failures.\n +#\n +u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n @@ t/t7527-builtin-fsmonitor.sh: do\n +u_values=\"$u1 $u2\"\n +for u in $u_values\n +do\n -+\ttest_expect_success \"Unicode path: $u\" '\n ++\ttest_expect_success \"Unicode in repo root path: $u\" '\n +\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n +\n +\t\tgit init \"$u\" &&\n 4: 9efdbe28223 ! 3: 87d1c0b6f2a t/helper/fsmonitor-client: create stress test\n @@ Commit message\n Create a client-side thread pool of n threads and have\n each of them make m requests as fast as they can.\n \n - NEEDSWORK: This is just the client-side thread pool and\n - is useful for interactive testing and experimentation.\n - We need to add a script test to drive this.\n + We do not currently inspect the contents of the response.\n + We're only interested in placing a heavy request load on\n + the daemon.\n +\n + This test is useful for interactive testing and various\n + experimentation. For example, to place additional load\n + on the daemon while another test is running. We currently\n + do not have a test script that actually uses this helper.\n + We might add such a test in the future.\n \n Signed-off-by: Jeff Hostetler \n \n @@ t/helper/test-fsmonitor-client.c: static int do_send_flush(void)\n +\tfree(data);\n +\n +\t/*\n -+\t * TODO Decide if/when to return an error or call die().\n ++\t * Return an error if any of the _send_query requests failed.\n ++\t * We don't care about thread create/join errors.\n +\t */\n -+\treturn 0;\n ++\treturn sum_errors > 0;\n +}\n +\n int cmd__fsmonitor_client(int argc, const char **argv)\n 5: 44cc61e186c ! 4: 8c4f90ae4fd fsmonitor-settings: bare repos are incompatible with FSMonitor\n @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar\n +\tfsm_settings__set_ipc(the_repository);\n +\n +\tif (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {\n -+\t\tstruct strbuf buf_reason = STRBUF_INIT;\n -+\t\tfsm_settings__get_reason(the_repository, &buf_reason);\n -+\t\terror(\"%s '%s'\", buf_reason.buf, xgetcwd());\n -+\t\tstrbuf_release(&buf_reason);\n -+\t\treturn -1;\n ++\t\tconst char *msg = fsm_settings__get_reason_msg(the_repository);\n ++\n ++\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n +\t}\n +\n \tif (!strcmp(subcmd, \"start\"))\n @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n +\n +\t\tif (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {\n -+\t\t\tstruct strbuf buf_reason = STRBUF_INIT;\n -+\t\t\tfsm_settings__get_reason(r, &buf_reason);\n -+\t\t\terror(\"%s\", buf_reason.buf);\n -+\t\t\tstrbuf_release(&buf_reason);\n -+\t\t\treturn -1;\n ++\t\t\tconst char *msg = fsm_settings__get_reason_msg(r);\n ++\n ++\t\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n +\t\t}\n +\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n @@ fsmonitor-settings.c\n {\n \tstruct fsmonitor_settings *s;\n @@ fsmonitor-settings.c: void fsm_settings__set_ipc(struct repository *r)\n - {\n + \n \tlookup_fsmonitor_settings(r);\n \n +\tif (check_for_incompatible(r))\n @@ fsmonitor-settings.c: void fsm_settings__set_ipc(struct repository *r)\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n @@ fsmonitor-settings.c: void fsm_settings__set_hook(struct repository *r, const char *path)\n - {\n + \n \tlookup_fsmonitor_settings(r);\n \n +\tif (check_for_incompatible(r))\n @@ fsmonitor-settings.c: void fsm_settings__set_disabled(struct repository *r)\n \tlookup_fsmonitor_settings(r);\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n -+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_ZERO;\n ++\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n +\n -+static void create_reason_message(struct repository *r,\n -+\t\t\t\t struct strbuf *buf_reason)\n ++enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n +{\n -+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n -+\n -+\tswitch (s->reason) {\n -+\tcase FSMONITOR_REASON_ZERO:\n -+\t\treturn;\n ++\tif (!r)\n ++\t\tr = the_repository;\n +\n -+\tcase FSMONITOR_REASON_BARE:\n -+\t\tstrbuf_addstr(buf_reason,\n -+\t\t\t _(\"bare repos are incompatible with fsmonitor\"));\n -+\t\treturn;\n ++\tlookup_fsmonitor_settings(r);\n +\n -+\tdefault:\n -+\t\tBUG(\"Unhandled case in create_reason_message '%d'\", s->reason);\n -+\t}\n ++\treturn r->settings.fsmonitor->reason;\n +}\n +\n -+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n -+\t\t\t\t\t struct strbuf *buf_reason)\n ++const char *fsm_settings__get_reason_msg(struct repository *r)\n +{\n -+\tlookup_fsmonitor_settings(r);\n ++\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n +\n -+\tstrbuf_reset(buf_reason);\n -+\tif (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)\n -+\t\tcreate_reason_message(r, buf_reason);\n ++\tswitch (reason) {\n ++\tcase FSMONITOR_REASON_OK:\n ++\t\treturn NULL;\n +\n -+\treturn r->settings.fsmonitor->reason;\n ++\tcase FSMONITOR_REASON_BARE:\n ++\t\treturn _(\"bare repos are incompatible with fsmonitor\");\n ++\t}\n ++\n ++\tBUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\n ++\t reason);\n +}\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h\n + * Incompatibility reasons.\n + */\n +enum fsmonitor_reason {\n -+\tFSMONITOR_REASON_ZERO = 0,\n -+\tFSMONITOR_REASON_BARE = 1,\n ++\tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n ++\tFSMONITOR_REASON_BARE,\n +};\n +\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n - \n +@@ fsmonitor-settings.h: void fsm_settings__set_disabled(struct repository *r);\n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n -+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n -+\t\t\t\t\t struct strbuf *buf_reason);\n \n ++enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n ++const char *fsm_settings__get_reason_msg(struct repository *r);\n ++\n struct fsmonitor_settings;\n \n + #endif /* FSMONITOR_SETTINGS_H */\n \n ## t/t7519-status-fsmonitor.sh ##\n @@ t/t7519-status-fsmonitor.sh: test_lazy_prereq UNTRACKED_CACHE '\n @@ t/t7519-status-fsmonitor.sh: test_lazy_prereq UNTRACKED_CACHE '\n +test_expect_success 'incompatible bare repo' '\n +\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n +\tgit init --bare bare-clone &&\n -+\tcat >expect <<-\\EOF &&\n -+\terror: bare repos are incompatible with fsmonitor\n -+\tEOF\n +\n +\ttest_must_fail \\\n +\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n +\t\t\tupdate-index --fsmonitor 2>actual &&\n -+\ttest_cmp expect actual &&\n ++\tgrep \"bare repos are incompatible with fsmonitor\" actual &&\n +\n +\ttest_must_fail \\\n +\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n +\t\t\tupdate-index --fsmonitor 2>actual &&\n -+\ttest_cmp expect actual\n ++\tgrep \"bare repos are incompatible with fsmonitor\" actual\n +'\n +\n +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n 6: 4715677f85f ! 5: 6329328d185 fsmonitor-settings: stub in platform-specific incompatibility checking\n @@ Commit message\n \n In the existing fsmonitor-settings code we have a way to mark\n types of repos as incompatible with fsmonitor (whether via the\n - hook and ipc APIs). For example, we do this for bare repos,\n + hook and IPC APIs). For example, we do this for bare repos,\n since there are no files to watch.\n \n - Extend this exclusion mechanism for platfor-specific reasons.\n + Extend this exclusion mechanism for platform-specific reasons.\n This commit just creates the framework and adds a stub for Win32.\n \n Signed-off-by: Jeff Hostetler \n @@ Makefile: all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n -+# If your platform has os-specific ways to tell if a repo is incompatible with\n -+# fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS\n ++# If your platform has OS-specific ways to tell if a repo is incompatible with\n ++# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n +# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n +# that implements the `fsm_os_settings__*()` routines.\n +#\n @@ compat/fsmonitor/fsm-settings-win32.c (new)\n +\n +enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n +{\n -+\treturn FSMONITOR_REASON_ZERO;\n ++\treturn FSMONITOR_REASON_OK;\n +}\n \n ## config.mak.uname ##\n @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)\n +\t\tenum fsmonitor_reason reason;\n +\n +\t\treason = fsm_os__incompatible(r);\n -+\t\tif (reason != FSMONITOR_REASON_ZERO) {\n ++\t\tif (reason != FSMONITOR_REASON_OK) {\n +\t\t\tset_incompatible(r, reason);\n +\t\t\treturn 1;\n +\t\t}\n @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)\n \n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,\n +@@ fsmonitor-settings.h: const char *fsm_settings__get_reason_msg(struct repository *r);\n \n struct fsmonitor_settings;\n \n @@ fsmonitor-settings.h: enum fsmonitor_reason fsm_settings__get_reason(struct repo\n + * Ask platform-specific code whether the repository is incompatible\n + * with fsmonitor (both hook and ipc modes). For example, if the working\n + * directory is on a remote volume and mounted via a technology that does\n -+ * not support notification events.\n ++ * not support notification events, then we should not pretend to watch it.\n + *\n + * fsm_os__* routines should considered private to fsm_settings__\n + * routines.\n 7: 4e856d60e38 ! 6: fa9e86e7de7 fsmonitor-settings: virtual repos are incompatible with FSMonitor\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor-settings: virtual repos are incompatible with FSMonitor\n + fsmonitor-settings: VFS for Git virtual repos are incompatible\n \n - Virtual repos, such as GVFS (aka VFS for Git), are incompatible\n - with FSMonitor.\n + VFS for Git virtual repositories are incompatible with FSMonitor.\n +\n + VFS for Git is a downstream fork of Git. It contains its own custom\n + file system watcher that is aware of the virtualization. If a working\n + directory is being managed by VFS for Git, we should not try to watch\n + it because we may get incomplete results.\n +\n + We do not know anything about how VFS for Git works, but we do\n + know that VFS for Git working directories contain a well-defined\n + config setting. If it is set, mark the working directory as\n + incompatible.\n \n Signed-off-by: Jeff Hostetler \n \n @@ compat/fsmonitor/fsm-settings-win32.c\n #include \"fsmonitor-settings.h\"\n \n +/*\n -+ * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n ++ * VFS for Git is incompatible with FSMonitor.\n + *\n -+ * Granted, core Git does not know anything about GVFS and we\n ++ * Granted, core Git does not know anything about VFS for Git and we\n + * shouldn't make assumptions about a downstream feature, but users\n + * can install both versions. And this can lead to incorrect results\n -+ * from core Git commands. So, without bringing in any of the GVFS\n -+ * code, do a simple config test for a published config setting. (We\n -+ * do not look at the various *_TEST_* environment variables.)\n ++ * from core Git commands. So, without bringing in any of the VFS for\n ++ * Git code, do a simple config test for a published config setting.\n ++ * (We do not look at the various *_TEST_* environment variables.)\n + */\n -+static enum fsmonitor_reason is_virtual(struct repository *r)\n ++static enum fsmonitor_reason check_vfs4git(struct repository *r)\n +{\n +\tconst char *const_str;\n +\n +\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n -+\t\treturn FSMONITOR_REASON_VIRTUAL;\n ++\t\treturn FSMONITOR_REASON_VFS4GIT;\n +\n -+\treturn FSMONITOR_REASON_ZERO;\n ++\treturn FSMONITOR_REASON_OK;\n +}\n +\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n +\tenum fsmonitor_reason reason;\n +\n -+\treason = is_virtual(r);\n -+\tif (reason)\n ++\treason = check_vfs4git(r);\n ++\tif (reason != FSMONITOR_REASON_OK)\n +\t\treturn reason;\n +\n - \treturn FSMONITOR_REASON_ZERO;\n + \treturn FSMONITOR_REASON_OK;\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: static void create_reason_message(struct repository *r,\n - \t\t\t _(\"bare repos are incompatible with fsmonitor\"));\n - \t\treturn;\n +@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)\n \n -+\tcase FSMONITOR_REASON_VIRTUAL:\n -+\t\tstrbuf_addstr(buf_reason,\n -+\t\t\t _(\"virtual repos are incompatible with fsmonitor\"));\n -+\t\treturn;\n + \tcase FSMONITOR_REASON_BARE:\n + \t\treturn _(\"bare repos are incompatible with fsmonitor\");\n +\n - \tdefault:\n - \t\tBUG(\"Unhandled case in create_reason_message '%d'\", s->reason);\n ++\tcase FSMONITOR_REASON_VFS4GIT:\n ++\t\treturn _(\"virtual repos are incompatible with fsmonitor\");\n \t}\n + \n + \tBUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_mode {\n enum fsmonitor_reason {\n - \tFSMONITOR_REASON_ZERO = 0,\n - \tFSMONITOR_REASON_BARE = 1,\n -+\tFSMONITOR_REASON_VIRTUAL = 2,\n + \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n + \tFSMONITOR_REASON_BARE,\n ++\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n 8: e5511ef0f8b ! 7: c1802410410 fsmonitor-settings: stub in macOS-specific incompatibility checking\n @@ compat/fsmonitor/fsm-settings-darwin.c (new)\n +\n +enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n +{\n -+\treturn FSMONITOR_REASON_ZERO;\n ++\treturn FSMONITOR_REASON_OK;\n +}\n \n ## config.mak.uname ##\n 9: 412fbc45868 ! 8: e3bfa0bd69d fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor\n + fsmonitor-settings: remote repos on macOS are incompatible\n \n Teach Git to detect remote working directories on macOS and mark them as\n incompatible with FSMonitor.\n @@ compat/fsmonitor/fsm-settings-darwin.c\n + * So (for now at least), mark remote working directories as\n + * incompatible.\n + */\n -+static enum fsmonitor_reason is_remote(struct repository *r)\n ++static enum fsmonitor_reason check_remote(struct repository *r)\n +{\n +\tstruct statfs fs;\n +\n @@ compat/fsmonitor/fsm-settings-darwin.c\n +\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n +\t\t\t\t r->worktree, strerror(saved_errno));\n +\t\terrno = saved_errno;\n -+\t\treturn FSMONITOR_REASON_ZERO;\n ++\t\treturn FSMONITOR_REASON_ERROR;\n +\t}\n +\n +\ttrace_printf_key(&trace_fsmonitor,\n @@ compat/fsmonitor/fsm-settings-darwin.c\n +\tif (!(fs.f_flags & MNT_LOCAL))\n +\t\treturn FSMONITOR_REASON_REMOTE;\n +\n -+\treturn FSMONITOR_REASON_ZERO;\n ++\treturn FSMONITOR_REASON_OK;\n +}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n +\tenum fsmonitor_reason reason;\n +\n -+\treason = is_remote(r);\n -+\tif (reason)\n ++\treason = check_remote(r);\n ++\tif (reason != FSMONITOR_REASON_OK)\n +\t\treturn reason;\n +\n - \treturn FSMONITOR_REASON_ZERO;\n + \treturn FSMONITOR_REASON_OK;\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: static void create_reason_message(struct repository *r,\n - \t\t\t _(\"virtual repos are incompatible with fsmonitor\"));\n - \t\treturn;\n +@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)\n + \tcase FSMONITOR_REASON_BARE:\n + \t\treturn _(\"bare repos are incompatible with fsmonitor\");\n \n ++\tcase FSMONITOR_REASON_ERROR:\n ++\t\treturn _(\"repo incompatible with fsmonitor due to errors\");\n ++\n +\tcase FSMONITOR_REASON_REMOTE:\n -+\t\tstrbuf_addstr(buf_reason,\n -+\t\t\t _(\"remote repos are incompatible with fsmonitor\"));\n -+\t\treturn;\n ++\t\treturn _(\"remote repos are incompatible with fsmonitor\");\n +\n - \tdefault:\n - \t\tBUG(\"Unhandled case in create_reason_message '%d'\", s->reason);\n + \tcase FSMONITOR_REASON_VFS4GIT:\n + \t\treturn _(\"virtual repos are incompatible with fsmonitor\");\n \t}\n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: enum fsmonitor_reason {\n - \tFSMONITOR_REASON_ZERO = 0,\n - \tFSMONITOR_REASON_BARE = 1,\n - \tFSMONITOR_REASON_VIRTUAL = 2,\n -+\tFSMONITOR_REASON_REMOTE = 3,\n +@@ fsmonitor-settings.h: enum fsmonitor_mode {\n + enum fsmonitor_reason {\n + \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n + \tFSMONITOR_REASON_BARE,\n ++\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n ++\tFSMONITOR_REASON_REMOTE,\n + \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n - void fsm_settings__set_ipc(struct repository *r);\n 10: ae09fb10c3a ! 9: e32da3118fb fsmonitor-settings: remote repos on Windows are incompatible with FSMonitor\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor-settings: remote repos on Windows are incompatible with FSMonitor\n + fsmonitor-settings: remote repos on Windows are incompatible\n \n Teach Git to detect remote working directories on Windows and mark them as\n incompatible with FSMonitor.\n @@ compat/fsmonitor/fsm-settings-win32.c\n +#include \"fsmonitor.h\"\n \n /*\n - * GVFS (aka VFS for Git) is incompatible with FSMonitor.\n -@@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(struct repository *r)\n - \treturn FSMONITOR_REASON_ZERO;\n + * VFS for Git is incompatible with FSMonitor.\n +@@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason check_vfs4git(struct repository *r)\n + \treturn FSMONITOR_REASON_OK;\n }\n \n +/*\n @@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(s\n + * $ mklink /d ./link //server/share/repo\n + * $ git -C ./link status\n + */\n -+static enum fsmonitor_reason is_remote(struct repository *r)\n ++static enum fsmonitor_reason check_remote(struct repository *r)\n +{\n +\twchar_t wpath[MAX_PATH];\n +\twchar_t wfullpath[MAX_PATH];\n @@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(s\n +\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n +\t */\n +\tif (xutftowcs_path(wpath, r->worktree) < 0)\n -+\t\treturn FSMONITOR_REASON_ZERO;\n ++\t\treturn FSMONITOR_REASON_ERROR;\n +\n +\t/*\n +\t * GetDriveTypeW() requires a final slash. We assume that the\n @@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(s\n +\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n +\t */\n +\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n -+\t\treturn FSMONITOR_REASON_ZERO;\n ++\t\treturn FSMONITOR_REASON_ERROR;\n +\n +\tdriveType = GetDriveTypeW(wfullpath);\n +\ttrace_printf_key(&trace_fsmonitor,\n @@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(s\n +\n +\tif (driveType == DRIVE_REMOTE) {\n +\t\ttrace_printf_key(&trace_fsmonitor,\n -+\t\t\t\t \"is_remote('%s') true\",\n ++\t\t\t\t \"check_remote('%s') true\",\n +\t\t\t\t r->worktree);\n +\t\treturn FSMONITOR_REASON_REMOTE;\n +\t}\n +\n -+\treturn FSMONITOR_REASON_ZERO;\n ++\treturn FSMONITOR_REASON_OK;\n +}\n +\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n @@ compat/fsmonitor/fsm-settings-win32.c: enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n - \tif (reason)\n + \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \n -+\treason = is_remote(r);\n -+\tif (reason)\n ++\treason = check_remote(r);\n ++\tif (reason != FSMONITOR_REASON_OK)\n +\t\treturn reason;\n +\n - \treturn FSMONITOR_REASON_ZERO;\n + \treturn FSMONITOR_REASON_OK;\n }\n 11: be1672e32b2 = 10: f63de4eda31 unpack-trees: initialize fsmonitor_has_run_once in o->result\n 12: 71babe7243a = 11: fe305f5f287 fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n 14: 95cdff22be0 ! 12: c8f3e251b1f fsmonitor--daemon: cd out of worktree root\n @@ builtin/fsmonitor--daemon.c: done:\n \tstrbuf_release(&state.path_cookie_prefix);\n +\tstrbuf_release(&state.path_ipc);\n \n - \t/*\n - \t * NEEDSWORK: Consider \"rm -rf /\"\n + \treturn err;\n + }\n \n ## compat/fsmonitor/fsm-listen-win32.c ##\n @@ compat/fsmonitor/fsm-listen-win32.c: static int recv_rdcw_watch(struct one_watch *watch)\n 15: 6b5642f7770 = 13: 71673be2da5 fsmonitor--daemon: prepare for adding health thread\n 16: 63c502c20bd = 14: 5387baaf5d7 fsmonitor--daemon: rename listener thread related variables\n 17: 4a77f5b1fde ! 15: f78e4ad87c0 fsmonitor--daemon: stub in health thread\n @@ Commit message\n file system events outside of the watched worktree root or if\n we want to have an idle-timeout auto-shutdown feature.\n \n + This commit creates the health thread itself, defines the thread-proc\n + and sets up the thread's event loop. It integrates this new thread\n + into the existing IPC and Listener thread models.\n +\n + This commit defines the API to the platform-specific code where all of\n + the monitoring will actually happen.\n +\n + The platform-specific code for MacOS is just stubs. Meaning that the\n + health thread will immediately exit on MacOS, but that is OK and\n + expected. Future work can define MacOS-specific monitoring.\n +\n + The platform-specific code for Windows sets up enough of the\n + WaitForMultipleObjects() machinery to watch for system and/or custom\n + events. Currently, the set of wait handles only includes our custom\n + shutdown event (sent from our other theads). Later commits in this\n + series will extend the set of wait handles to monitor other\n + conditions.\n +\n Signed-off-by: Jeff Hostetler \n \n ## Makefile ##\n @@ Makefile: all::\n +# `compat/fsmonitor/fsm-health-.c` files\n +# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n - # If your platform has os-specific ways to tell if a repo is incompatible with\n - # fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS\n + # If your platform has OS-specific ways to tell if a repo is incompatible with\n + # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n @@ Makefile: endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n 18: a398cdb8a04 ! 16: bb72f911a05 fsm-health-win32: add framework to monitor daemon health\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsm-health-win32: add framework to monitor daemon health\n + fsm-health-win32: add polling framework to monitor daemon health\n \n - Create framework in Win32 version of the \"health\" thread to\n - periodically inspect the system and shutdown if warranted.\n + Extend the Windows version of the \"health\" thread to periodically\n + inspect the system and shutdown if warranted.\n \n - This version just includes the setup for the timeout in\n - WaitForMultipleObjects() and calls the (currently empty) table\n - of functions.\n + This commit updates the thread's wait loop to use a timeout and\n + defines a (currently empty) table of functions to poll the system.\n \n A later commit will add functions to the table to actually\n inspect the system.\n @@ compat/fsmonitor/fsm-health-win32.c\n + */\n +#define WAIT_FREQ_MS (60 * 1000)\n +\n -+enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };\n ++/*\n ++ * State machine states for each of the interval functions\n ++ * used for polling our health.\n ++ */\n ++enum interval_fn_ctx {\n ++\tCTX_INIT = 0,\n ++\tCTX_TERM,\n ++\tCTX_TIMER\n ++};\n +\n +typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n +\t\t\t enum interval_fn_ctx ctx);\n +\n + struct fsm_health_data\n + {\n + \tHANDLE hEventShutdown;\n +@@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n + \tFREE_AND_NULL(state->health_data);\n + }\n + \n ++/*\n ++ * A table of the polling functions.\n ++ */\n +static interval_fn *table[] = {\n +\tNULL, /* must be last */\n +};\n +\n +/*\n -+ * Call all of the functions in the table.\n ++ * Call all of the polling functions in the table.\n + * Shortcut and return first error.\n + *\n + * Return 0 if all succeeded.\n @@ compat/fsmonitor/fsm-health-win32.c\n +\treturn 0;\n +}\n +\n - struct fsm_health_data\n - {\n - \tHANDLE hEventShutdown;\n -@@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n 19: 023fcd6e2b1 ! 17: baf8c031a97 fsm-health-win32: force shutdown daemon if worktree root moves\n @@ Commit message\n Force shutdown fsmonitor daemon if the worktree root directory\n is moved, renamed, or deleted.\n \n + Use Windows low-level GetFileInformationByHandle() to get and\n + compare the Windows system unique ID for the directory with a\n + cached version when we started up. This lets us detect the\n + case where someone renames the directory that we are watching\n + and then creates a new directory with the original pathname.\n +\n + This is important because we are listening to a named pipe for\n + requests and they are stored in the Named Pipe File System (NPFS)\n + which a kernel-resident pseudo filesystem not associated with\n + the actual NTFS directory.\n +\n + For example, if the daemon was watching \"~/foo/\", it would have\n + a directory-watch handle on that directory and a named-pipe\n + handle for \"//./pipe/...foo\". Moving the directory to \"~/bar/\"\n + does not invalidate the directory handle. (So the daemon would\n + actually be watching \"~/bar\" but listening on \"//./pipe/...foo\".\n + If the user then does \"git init ~/foo\" and causes another daemon\n + to start, the first daemon will still have ownership of the pipe\n + and the second daemon instance will fail to start. \"git status\"\n + clients in \"~/foo\" will ask \"//./pipe/...foo\" about changes and\n + the first daemon instance will tell them about \"~/bar\".\n +\n + This commit causes the first daemon to shutdown if the system unique\n + ID for \"~/foo\" changes (changes from what it was when the daemon\n + started). Shutdown occurs after a periodic poll. After the\n + first daemon exits and releases the lock on the named pipe,\n + subsequent Git commands may cause another daemon to be started\n + on \"~/foo\". Similarly, a subsequent Git command may cause another\n + daemon to be started on \"~/bar\".\n +\n Signed-off-by: Jeff Hostetler \n \n ## compat/fsmonitor/fsm-health-win32.c ##\n -@@ compat/fsmonitor/fsm-health-win32.c: enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };\n - typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n - \t\t\t enum interval_fn_ctx ctx);\n - \n -+static interval_fn has_worktree_moved;\n -+\n - static interval_fn *table[] = {\n -+\thas_worktree_moved,\n - \tNULL, /* must be last */\n - };\n - \n @@ compat/fsmonitor/fsm-health-win32.c: struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n @@ compat/fsmonitor/fsm-health-win32.c: struct fsm_health_data\n +\t} wt_moved;\n };\n \n - int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n -@@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n - \tFREE_AND_NULL(state->health_data);\n - }\n - \n ++/*\n ++ * Lookup the system unique ID for the path. This is as close as\n ++ * we get to an inode number, but this also contains volume info,\n ++ * so it is a little stronger.\n ++ */\n +static int lookup_bhfi(wchar_t *wpath,\n +\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n +{\n @@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daem\n +\treturn 0;\n +}\n +\n ++/*\n ++ * Compare the relevant fields from two system unique IDs.\n ++ * We use this to see if two different handles to the same\n ++ * path actually refer to the same *instance* of the file\n ++ * or directory.\n ++ */\n +static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n +\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n +{\n @@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daem\n +\treturn 0;\n +}\n +\n - void fsm_health__loop(struct fsmonitor_daemon_state *state)\n ++\n + int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n {\n - \tstruct fsm_health_data *data = state->health_data;\n + \tstruct fsm_health_data *data;\n +@@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n + * A table of the polling functions.\n + */\n + static interval_fn *table[] = {\n ++\thas_worktree_moved,\n + \tNULL, /* must be last */\n + };\n + \n 20: 496b3da35d0 ! 18: 796b6591393 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n @@ compat/fsmonitor/fsm-listen-darwin.c: static void fsevent_callback(ConstFSEventS\n +\t\t\t/*\n +\t\t\t * The spelling of the pathname of the root directory\n +\t\t\t * has changed. This includes the name of the root\n -+\t\t\t * directory itself of of any parent directory in the\n ++\t\t\t * directory itself or of any parent directory in the\n +\t\t\t * path.\n +\t\t\t *\n +\t\t\t * (There may be other conditions that throw this,\n 21: 07a9c7542be = 19: 24591920878 fsmonitor: optimize processing of directory events\n 22: f065e8c9a90 = 20: 06a32413854 t7527: FSMonitor tests for directory moves\n 23: e5a12832afa = 21: 4b59013cadd t/perf/p7527: add perf test for builtin FSMonitor\n -: ----------- > 22: 524d449ed64 fsmonitor: never set CE_FSMONITOR_VALID on submodules\n 2: ad8cf6d9a47 ! 23: c7264decaf6 t7527: test FS event reporing on macOS WRT case and Unicode\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - t7527: test FS event reporing on macOS WRT case and Unicode\n + t7527: test FSMonitor on case insensitive+preserving file system\n \n - Confirm that macOS FS events are reported with a normalized spelling.\n -\n - APFS (and/or HFS+) is case-insensitive. This means that case-independent\n - lookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed. But that\n - doesn't tell us how FS events are reported if we try \"rm -rf .git\" versus\n - \"rm -rf .GIT\". Are the events reported using the on-disk spelling of the\n - pathname or in the spelling used by the command.\n -\n - NEEDSWORK: I was only able to test case. It would be nice to add tests\n - that use different Unicode spellings/normalizations and understand the\n - differences between APFS and HFS+ in this area. We should confirm that\n - the spelling of the workdir paths that the daemon sends to clients are\n - always properly normalized.\n + Test that FS events from the OS are received using the preserved,\n + on-disk spelling of files/directories rather than spelling used\n + to make the change.\n \n Signed-off-by: Jeff Hostetler \n \n ## t/t7527-builtin-fsmonitor.sh ##\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n - \ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule\" '\n + \tmy_match_and_clean\n '\n \n -+# Confirm that MacOS hides all of the Unicode normalization and/or\n -+# case folding from the FS events. That is, are the pathnames in the\n -+# FS events reported using the spelling on the disk or in the spelling\n -+# used by the other process.\n -+#\n -+# Note that we assume that the filesystem is set to case insensitive.\n -+#\n -+# NEEDSWORK: APFS handles Unicode and Unicode normalization\n -+# differently than HFS+. I only have an APFS partition, so\n -+# more testing here would be helpful.\n ++# On a case-insensitive file system, confirm that the daemon\n ++# notices when the .git directory is moved/renamed/deleted\n ++# regardless of how it is spelled in the the FS event.\n ++# That is, does the FS event receive the spelling of the\n ++# operation or does it receive the spelling preserved with\n ++# the file/directory.\n +#\n ++test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n ++#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n ++\n ++\tgit init test_insensitive &&\n ++\t(\n ++\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/insensitive.trace\" &&\n ++\t\texport GIT_TRACE_FSMONITOR &&\n +\n -+# Rename .git using alternate spelling and confirm that the daemon\n -+# sees the event using the correct spelling and shutdown.\n -+test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' '\n -+\ttest_when_finished \"stop_daemon_delete_repo test_apfs\" &&\n ++\t\tstart_daemon test_insensitive\n ++\t) &&\n +\n -+\tgit init test_apfs &&\n -+\tstart_daemon test_apfs &&\n ++\tmkdir -p test_insensitive/abc/def &&\n ++\techo xyz >test_insensitive/ABC/DEF/xyz &&\n +\n -+\ttest_path_is_dir test_apfs/.git &&\n -+\ttest_path_is_dir test_apfs/.GIT &&\n ++\ttest_path_is_dir test_insensitive/.git &&\n ++\ttest_path_is_dir test_insensitive/.GIT &&\n +\n -+\tmv test_apfs/.GIT test_apfs/.FOO &&\n ++\t# Rename .git using an alternate spelling to verify that that\n ++\t# daemon detects it and automatically shuts down.\n ++\tmv test_insensitive/.GIT test_insensitive/.FOO &&\n +\tsleep 1 &&\n -+\tmv test_apfs/.FOO test_apfs/.git &&\n ++\tmv test_insensitive/.FOO test_insensitive/.git &&\n ++\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n +\n -+\ttest_must_fail git -C test_apfs fsmonitor--daemon status\n ++\t# Verify that events were reported using on-disk spellings of the\n ++\t# directories and files that we touched. We may or may not get a\n ++\t# trailing slash on modified directories.\n ++\t#\n ++\tegrep \"^event: abc/?$\" ./insensitive.trace &&\n ++\tegrep \"^event: abc/def/?$\" ./insensitive.trace &&\n ++\tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n +'\n +\n - test_expect_success 'cannot start multiple daemons' '\n - \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n - \n + test_done\n 13: f19671f7def < -: ----------- fsmonitor--daemon: print start message only if fsmonitor.announceStartup\n -: ----------- > 24: 95b9d4210d2 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n -: ----------- > 25: 5a0c1b7a287 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n -: ----------- > 26: a45c1fd3000 t7527: test Unicode NFC/NFD handling on MacOS\n -: ----------- > 27: e3e01677d93 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n\n-- \ngitgitgadget\n"},{"id":"450784","messageId":"8c4f90ae4fd5d9fbac9acb9307ee82ceffc7df08.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:04Z","receivedAt":"2022-03-08T22:15:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a worktree, so there is nothing for the\ndaemon watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 9 ++++++\n builtin/update-index.c | 7 +++++\n fsmonitor-settings.c | 57 +++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 12 ++++++++\n t/t7519-status-fsmonitor.sh | 23 +++++++++++++++\n 5 files changed, 108 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 97ca2a356e5..00eaffbb945 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1443,6 +1443,15 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\tfsm_settings__set_ipc(the_repository);\n+\n+\tif (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {\n+\t\tconst char *msg = fsm_settings__get_reason_msg(the_repository);\n+\n+\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n+\t}\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex d335f1ac72a..8f460e7195f 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1237,6 +1237,13 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\t\tif (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {\n+\t\t\tconst char *msg = fsm_settings__get_reason_msg(r);\n+\n+\t\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n+\t\t}\n+\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\tadvise(_(\"core.fsmonitor is unset; \"\n \t\t\t\t \"set it if you really want to \"\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 3b54e7a51f6..8410cc73404 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -9,9 +9,33 @@\n */\n struct fsmonitor_settings {\n \tenum fsmonitor_mode mode;\n+\tenum fsmonitor_reason reason;\n \tchar *hook_path;\n };\n \n+static void set_incompatible(struct repository *r,\n+\t\t\t enum fsmonitor_reason reason)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\ts->reason = reason;\n+}\n+\n+static int check_for_incompatible(struct repository *r)\n+{\n+\tif (!r->worktree) {\n+\t\t/*\n+\t\t * Bare repositories don't have a working directory and\n+\t\t * therefore have nothing to watch.\n+\t\t */\n+\t\tset_incompatible(r, FSMONITOR_REASON_BARE);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static void lookup_fsmonitor_settings(struct repository *r)\n {\n \tstruct fsmonitor_settings *s;\n@@ -87,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r)\n \n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n@@ -98,6 +125,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path)\n \n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n@@ -111,5 +141,32 @@ void fsm_settings__set_disabled(struct repository *r)\n \tlookup_fsmonitor_settings(r);\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n+\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\n+\n+const char *fsm_settings__get_reason_msg(struct repository *r)\n+{\n+\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n+\n+\tswitch (reason) {\n+\tcase FSMONITOR_REASON_OK:\n+\t\treturn NULL;\n+\n+\tcase FSMONITOR_REASON_BARE:\n+\t\treturn _(\"bare repos are incompatible with fsmonitor\");\n+\t}\n+\n+\tBUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\n+\t reason);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a4c5d7b4889..a1af058a287 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -4,11 +4,20 @@\n struct repository;\n \n enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n \tFSMONITOR_MODE_DISABLED = 0,\n \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n };\n \n+/*\n+ * Incompatibility reasons.\n+ */\n+enum fsmonitor_reason {\n+\tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n+\tFSMONITOR_REASON_BARE,\n+};\n+\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n@@ -16,6 +25,9 @@ void fsm_settings__set_disabled(struct repository *r);\n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n+const char *fsm_settings__get_reason_msg(struct repository *r);\n+\n struct fsmonitor_settings;\n \n #endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex a6308acf006..cf258d88f04 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '\n \ttest $ret -ne 1\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n+\tgit init --bare bare-clone &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repos are incompatible with fsmonitor\" actual &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repos are incompatible with fsmonitor\" actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n+\tgit init --bare bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"bare repos are incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"450785","messageId":"6329328d1859bc360dc6305a1d7be6734c9c84ea.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 05/27] fsmonitor-settings: stub in platform-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:05Z","receivedAt":"2022-03-08T22:15:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend generic incompatibility checkout with platform-specific\nmechanism. Stub in Win32 version.\n\nIn the existing fsmonitor-settings code we have a way to mark\ntypes of repos as incompatible with fsmonitor (whether via the\nhook and IPC APIs). For example, we do this for bare repos,\nsince there are no files to watch.\n\nExtend this exclusion mechanism for platform-specific reasons.\nThis commit just creates the framework and adds a stub for Win32.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 +++++++++++++\n compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++\n config.mak.uname | 4 ++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n fsmonitor-settings.c | 12 ++++++++++++\n fsmonitor-settings.h | 13 +++++++++++++\n 6 files changed, 54 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n\ndiff --git a/Makefile b/Makefile\nindex daa21bed6c3..93604fe8ef7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -475,6 +475,11 @@ all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n+# If your platform has OS-specific ways to tell if a repo is incompatible with\n+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n+# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n+# that implements the `fsm_os_settings__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n+ifdef FSMONITOR_OS_SETTINGS\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE\n ifdef FSMONITOR_DAEMON_BACKEND\n \t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n endif\n+ifdef FSMONITOR_OS_SETTINGS\n+\t@echo FSMONITOR_OS_SETTINGS=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_OS_SETTINGS)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 501970902da..cf224768ad6 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex ee0d7257b77..16705da2000 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 8410cc73404..1b16b551d87 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r)\n \t\treturn 1;\n \t}\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+\t{\n+\t\tenum fsmonitor_reason reason;\n+\n+\t\treason = fsm_os__incompatible(r);\n+\t\tif (reason != FSMONITOR_REASON_OK) {\n+\t\t\tset_incompatible(r, reason);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+#endif\n+\n \treturn 0;\n }\n \ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a1af058a287..be25272c012 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -30,4 +30,17 @@ const char *fsm_settings__get_reason_msg(struct repository *r);\n \n struct fsmonitor_settings;\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+/*\n+ * Ask platform-specific code whether the repository is incompatible\n+ * with fsmonitor (both hook and ipc modes). For example, if the working\n+ * directory is on a remote volume and mounted via a technology that does\n+ * not support notification events, then we should not pretend to watch it.\n+ *\n+ * fsm_os__* routines should considered private to fsm_settings__\n+ * routines.\n+ */\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);\n+#endif /* HAVE_FSMONITOR_OS_SETTINGS */\n+\n #endif /* FSMONITOR_SETTINGS_H */\n-- \ngitgitgadget\n\n"},{"id":"450786","messageId":"c1802410410b5de3fb9037cfe6e2bb073d53feca.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:07Z","receivedAt":"2022-03-08T22:15:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++\n config.mak.uname | 1 +\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 13 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cf224768ad6..cf911d141f2 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)\n \tifndef NO_PTHREADS\n \tifndef NO_UNIX_SOCKETS\n \tFSMONITOR_DAEMON_BACKEND = darwin\n+\tFSMONITOR_OS_SETTINGS = darwin\n \tendif\n \tendif\n \ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 16705da2000..b8f9f7a0388 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"450787","messageId":"fa9e86e7de7352a2cf9c1f465a38f834c6ec4ffe.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:06Z","receivedAt":"2022-03-08T22:15:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nVFS for Git virtual repositories are incompatible with FSMonitor.\n\nVFS for Git is a downstream fork of Git. It contains its own custom\nfile system watcher that is aware of the virtualization. If a working\ndirectory is being managed by VFS for Git, we should not try to watch\nit because we may get incomplete results.\n\nWe do not know anything about how VFS for Git works, but we do\nknow that VFS for Git working directories contain a well-defined\nconfig setting. If it is set, mark the working directory as\nincompatible.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 3 +++\n fsmonitor-settings.h | 1 +\n t/t7519-status-fsmonitor.sh | 9 +++++++++\n 4 files changed, 39 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 7fce32a3c5b..ee78bba38e3 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -3,7 +3,33 @@\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n \n+/*\n+ * VFS for Git is incompatible with FSMonitor.\n+ *\n+ * Granted, core Git does not know anything about VFS for Git and we\n+ * shouldn't make assumptions about a downstream feature, but users\n+ * can install both versions. And this can lead to incorrect results\n+ * from core Git commands. So, without bringing in any of the VFS for\n+ * Git code, do a simple config test for a published config setting.\n+ * (We do not look at the various *_TEST_* environment variables.)\n+ */\n+static enum fsmonitor_reason check_vfs4git(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n+\t\treturn FSMONITOR_REASON_VFS4GIT;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_vfs4git(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 1b16b551d87..ea3e365dfc4 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -177,6 +177,9 @@ const char *fsm_settings__get_reason_msg(struct repository *r)\n \n \tcase FSMONITOR_REASON_BARE:\n \t\treturn _(\"bare repos are incompatible with fsmonitor\");\n+\n+\tcase FSMONITOR_REASON_VFS4GIT:\n+\t\treturn _(\"virtual repos are incompatible with fsmonitor\");\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex be25272c012..7950529611b 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -16,6 +16,7 @@ enum fsmonitor_mode {\n enum fsmonitor_reason {\n \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex cf258d88f04..285508fb67e 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n \tgrep \"bare repos are incompatible with fsmonitor\" actual\n '\n \n+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone actual\" &&\n+\tgit init fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"virtual repos are incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"450788","messageId":"e3bfa0bd69d937efcf7f7bd879d3d372927ff06a.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 08/27] fsmonitor-settings: remote repos on macOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:08Z","receivedAt":"2022-03-08T22:16:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on macOS and mark them as\nincompatible with FSMonitor.\n\nWith this, `git fsmonitor--daemon run` will error out with a message\nlike it does for bare repos.\n\nClient commands, like `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 6 +++\n fsmonitor-settings.h | 2 +\n 3 files changed, 74 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex 7fce32a3c5b..fdd762bf79d 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -2,8 +2,74 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n+#include \n+#include \n+\n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type (NFS, SAMBA, etc.) dictates whether notification events\n+ * are available at all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * For the builtin FSMonitor, we create the Unix domain socket for the\n+ * IPC in the .git directory. If the working directory is remote,\n+ * then the socket will be created on the remote file system. This\n+ * can fail if the remote file system does not support UDS file types\n+ * (e.g. smbfs to a Windows server) or if the remote kernel does not\n+ * allow a non-local process to bind() the socket. (These problems\n+ * could be fixed by moving the UDS out of the .git directory and to a\n+ * well-known local directory on the client machine, but care should\n+ * be taken to ensure that $HOME is actually local and not a managed\n+ * file share.)\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\tstruct statfs fs;\n+\n+\tif (statfs(r->worktree, &fs) == -1) {\n+\t\tint saved_errno = errno;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n+\t\t\t\t r->worktree, strerror(saved_errno));\n+\t\terrno = saved_errno;\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n+\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n+\n+\tif (!(fs.f_flags & MNT_LOCAL))\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex ea3e365dfc4..7ff3f98964d 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -178,6 +178,12 @@ const char *fsm_settings__get_reason_msg(struct repository *r)\n \tcase FSMONITOR_REASON_BARE:\n \t\treturn _(\"bare repos are incompatible with fsmonitor\");\n \n+\tcase FSMONITOR_REASON_ERROR:\n+\t\treturn _(\"repo incompatible with fsmonitor due to errors\");\n+\n+\tcase FSMONITOR_REASON_REMOTE:\n+\t\treturn _(\"remote repos are incompatible with fsmonitor\");\n+\n \tcase FSMONITOR_REASON_VFS4GIT:\n \t\treturn _(\"virtual repos are incompatible with fsmonitor\");\n \t}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 7950529611b..6aa9a00379b 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -16,6 +16,8 @@ enum fsmonitor_mode {\n enum fsmonitor_reason {\n \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n+\tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"450789","messageId":"fe305f5f287de09f01faa1c9e738d231cc7a62c9.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 11/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:11Z","receivedAt":"2022-03-08T22:16:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIgnore FSEvents resulting from `xattr` changes. Git does not care about\nxattr's or changes to xattr's, so don't waste time collecting these\nevents in the daemon nor transmitting them to clients.\n\nVarious security tools add xattrs to files and/or directories, such as\nto mark them as having been downloaded. We should ignore these events\nsince it doesn't affect the content of the file/directory or the normal\nmeta-data that Git cares about.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 5c5de1ae702..f9b61b7b1c0 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -172,7 +172,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tif (flag & kFSEventStreamEventFlagItemCloned)\n \t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n \n-\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n \t\t\t path, flag, msg.buf);\n \n \tstrbuf_release(&msg);\n@@ -197,6 +197,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)\n \t\tef & kFSEventStreamEventFlagUserDropped);\n }\n \n+/*\n+ * If an `xattr` change is the only reason we received this event,\n+ * then silently ignore it. Git doesn't care about xattr's. We\n+ * have to be careful here because the kernel can combine multiple\n+ * events for a single path. And because events always have certain\n+ * bits set, such as `ItemIsFile` or `ItemIsDir`.\n+ *\n+ * Return 1 if we should ignore it.\n+ */\n+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n+{\n+\tstatic const FSEventStreamEventFlags mask =\n+\t\tkFSEventStreamEventFlagItemChangeOwner |\n+\t\tkFSEventStreamEventFlagItemCreated |\n+\t\tkFSEventStreamEventFlagItemFinderInfoMod |\n+\t\tkFSEventStreamEventFlagItemInodeMetaMod |\n+\t\tkFSEventStreamEventFlagItemModified |\n+\t\tkFSEventStreamEventFlagItemRemoved |\n+\t\tkFSEventStreamEventFlagItemRenamed |\n+\t\tkFSEventStreamEventFlagItemXattrMod |\n+\t\tkFSEventStreamEventFlagItemCloned;\n+\n+\treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n+}\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -262,6 +287,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_ignore_xattr(event_flags[k])) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n+\t\t\t\t\t path_k, event_flags[k]);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n \n \t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-- \ngitgitgadget\n\n"},{"id":"450790","messageId":"e32da3118fb9a4e55989ab24a58183c81d720523.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 09/27] fsmonitor-settings: remote repos on Windows are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:09Z","receivedAt":"2022-03-08T22:16:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on Windows and mark them as\nincompatible with FSMonitor.\n\nWith this `git fsmonitor--daemon run` will error out with a message like it\ndoes for bare repos.\n\nClient commands, such as `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++\n 1 file changed, 102 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex ee78bba38e3..907655720bb 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * VFS for Git is incompatible with FSMonitor.\n@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)\n \treturn FSMONITOR_REASON_OK;\n }\n \n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type dictates whether notification events are available at\n+ * all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ *\n+ * Notes for testing:\n+ *\n+ * (a) Windows allows a network share to be mapped to a drive letter.\n+ * (This is the normal method to access it.)\n+ *\n+ * $ NET USE Z: \\\\server\\share\n+ * $ git -C Z:/repo status\n+ *\n+ * (b) Windows allows a network share to be referenced WITHOUT mapping\n+ * it to drive letter.\n+ *\n+ * $ NET USE \\\\server\\share\\dir\n+ * $ git -C //server/share/repo status\n+ *\n+ * (c) Windows allows \"SUBST\" to create a fake drive mapping to an\n+ * arbitrary path (which may be remote)\n+ *\n+ * $ SUBST Q: Z:\\repo\n+ * $ git -C Q:/ status\n+ *\n+ * (d) Windows allows a directory symlink to be created on a local\n+ * file system that points to a remote repo.\n+ *\n+ * $ mklink /d ./link //server/share/repo\n+ * $ git -C ./link status\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\twchar_t wpath[MAX_PATH];\n+\twchar_t wfullpath[MAX_PATH];\n+\tsize_t wlen;\n+\tUINT driveType;\n+\n+\t/*\n+\t * Do everything in wide chars because the drive letter might be\n+\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n+\t */\n+\tif (xutftowcs_path(wpath, r->worktree) < 0)\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\t/*\n+\t * GetDriveTypeW() requires a final slash. We assume that the\n+\t * worktree pathname points to an actual directory.\n+\t */\n+\twlen = wcslen(wpath);\n+\tif (wpath[wlen - 1] != L'\\\\' && wpath[wlen - 1] != L'/') {\n+\t\twpath[wlen++] = L'\\\\';\n+\t\twpath[wlen] = 0;\n+\t}\n+\n+\t/*\n+\t * Normalize the path. If nothing else, this converts forward\n+\t * slashes to backslashes. This is essential to get GetDriveTypeW()\n+\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n+\t */\n+\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\tdriveType = GetDriveTypeW(wfullpath);\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"DriveType '%s' L'%ls' (%u)\",\n+\t\t\t r->worktree, wfullpath, driveType);\n+\n+\tif (driveType == DRIVE_REMOTE) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"check_remote('%s') true\",\n+\t\t\t\t r->worktree);\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n-- \ngitgitgadget\n\n"},{"id":"450791","messageId":"c8f3e251b1f1ffe0e058da01ddc918273ae9d9f2.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 12/27] fsmonitor--daemon: cd out of worktree root","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:12Z","receivedAt":"2022-03-08T22:16:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the fsmonitor--daemon to CD outside of the worktree\nbefore starting up.\n\nThe common Git startup mechanism causes the CWD of the daemon process\nto be in the root of the worktree. On Windows, this causes the daemon\nprocess to hold a locked handle on the CWD and prevents other\nprocesses from moving or deleting the worktree while the daemon is\nrunning.\n\nCD to HOME before entering main event loops.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++--\n compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------\n fsmonitor--daemon.h | 1 +\n 3 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 00eaffbb945..dbd37a2b3b8 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1175,11 +1175,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * before we need it.\n \t */\n \tif (ipc_server_run_async(&state->ipc_server_data,\n-\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t state->path_ipc.buf, &ipc_opts,\n \t\t\t\t handle_client, state))\n \t\treturn error_errno(\n \t\t\t_(\"could not start IPC thread pool on '%s'\"),\n-\t\t\tfsmonitor_ipc__get_path());\n+\t\t\tstate->path_ipc.buf);\n \n \t/*\n \t * Start the fsmonitor listener thread to collect filesystem\n@@ -1214,6 +1214,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n static int fsmonitor_run_daemon(void)\n {\n \tstruct fsmonitor_daemon_state state;\n+\tconst char *home;\n \tint err;\n \n \tmemset(&state, 0, sizeof(state));\n@@ -1283,6 +1284,15 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_addch(&state.path_cookie_prefix, '/');\n \n+\t/*\n+\t * We create a named-pipe or unix domain socket inside of the\n+\t * \".git\" directory. (Well, on Windows, we base our named\n+\t * pipe in the NPFS on the absolute path of the git\n+\t * directory.)\n+\t */\n+\tstrbuf_init(&state.path_ipc, 0);\n+\tstrbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1292,6 +1302,23 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\t/*\n+\t * CD out of the worktree root directory.\n+\t *\n+\t * The common Git startup mechanism causes our CWD to be the\n+\t * root of the worktree. On Windows, this causes our process\n+\t * to hold a locked handle on the CWD. This prevents the\n+\t * worktree from being moved or deleted while the daemon is\n+\t * running.\n+\t *\n+\t * We assume that our FS and IPC listener threads have either\n+\t * opened all of the handles that they need or will do\n+\t * everything using absolute paths.\n+\t */\n+\thome = getenv(\"HOME\");\n+\tif (home && *home && chdir(home))\n+\t\tdie_errno(\"could not cd home '%s'\", home);\n+\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n@@ -1304,6 +1331,7 @@ done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n \tstrbuf_release(&state.path_cookie_prefix);\n+\tstrbuf_release(&state.path_ipc);\n \n \treturn err;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex f4673d7d8b1..948f0e63915 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch)\n \t}\n \n \t/*\n-\t * NEEDSWORK: If an external is deleted, the above\n-\t * returns an error. I'm not sure that there's anything that\n-\t * we can do here other than failing -- the /.git\n-\t * link file would be broken anyway. We might try to check\n-\t * for that and return a better error message, but I'm not\n-\t * sure it is worth it.\n+\t * GetOverlappedResult() fails if the watched directory is\n+\t * deleted while we were waiting for an overlapped IO to\n+\t * complete. The documentation did not list specific errors,\n+\t * but I observed ERROR_ACCESS_DENIED (0x05) errors during\n+\t * testing.\n+\t *\n+\t * Note that we only get notificaiton events for events\n+\t * *within* the directory, not *on* the directory itself.\n+\t * (These might be properies of the parent directory, for\n+\t * example).\n+\t *\n+\t * NEEDSWORK: We might try to check for the deleted directory\n+\t * case and return a better error message, but I'm not sure it\n+\t * is worth it.\n+\t *\n+\t * Shutdown if we get any error.\n \t */\n \n \terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex bd09fffc176..223c2131b58 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n+\tstruct strbuf path_ipc;\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"450792","messageId":"f63de4eda3152e01c1a682d7bd002efd1e45878e.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 10/27] unpack-trees: initialize fsmonitor_has_run_once in o->result","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:10Z","receivedAt":"2022-03-08T22:16:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInitialize `o->result.fsmonitor_has_run_once` based upon value\nin `o->src_index->fsmonitor_has_run_once` to prevent a second\nfsmonitor query during the tree traversal and possibly getting\na skewed view of the working directory.\n\nThe checkout code has already talked to the fsmonitor and the\ntraversal is updating the index as it traverses, so there is\nno need to query the fsmonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n unpack-trees.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..888cff81f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \to->result.fsmonitor_last_update =\n \t\txstrdup_or_null(o->src_index->fsmonitor_last_update);\n+\to->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;\n \n \t/*\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n-- \ngitgitgadget\n\n"},{"id":"450793","messageId":"5387baaf5d78244ff079bdd92b99604d33e957bc.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 14/27] fsmonitor--daemon: rename listener thread related variables","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:14Z","receivedAt":"2022-03-08T22:16:14Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRename platform-specific listener thread related variables\nand data types as we prepare to add another backend thread\ntype.\n\n[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`\n[] `state->backend_data` becomes `state->listen_data`\n[] `state->error_code` becomes `state->listen_error_code`\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 +++---\n compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------\n compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++-------------\n compat/fsmonitor/fsm-listen.h | 2 +-\n fsmonitor--daemon.h | 6 +++---\n 5 files changed, 36 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex d5a8fc2ddc2..92c61d5b94d 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1219,8 +1219,8 @@ cleanup:\n \n \tif (err)\n \t\treturn err;\n-\tif (state->error_code)\n-\t\treturn state->error_code;\n+\tif (state->listen_error_code)\n+\t\treturn state->listen_error_code;\n \treturn 0;\n }\n \n@@ -1235,7 +1235,7 @@ static int fsmonitor_run_daemon(void)\n \thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n-\tstate.error_code = 0;\n+\tstate.listen_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex f9b61b7b1c0..7c81bc7e5bf 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -99,7 +99,7 @@ void FSEventStreamRelease(FSEventStreamRef stream);\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tCFStringRef cfsr_worktree_path;\n \tCFStringRef cfsr_gitdir_path;\n@@ -230,7 +230,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t const FSEventStreamEventId event_ids[])\n {\n \tstruct fsmonitor_daemon_state *state = ctx;\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tchar **paths = (char **)event_paths;\n \tstruct fsmonitor_batch *batch = NULL;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -422,11 +422,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tNULL,\n \t\tNULL\n \t};\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \tconst void *dir_array[2];\n \n \tCALLOC_ARRAY(data, 1);\n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \n \tdata->cfsr_worktree_path = CFStringCreateWithCString(\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n@@ -458,18 +458,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n \terror(\"Unable to create FSEventStream.\");\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tif (data->stream) {\n \t\tif (data->stream_started)\n@@ -479,14 +479,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n \t\tFSEventStreamRelease(data->stream);\n \t}\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \tdata->shutdown_style = SHUTDOWN_EVENT;\n \n \tCFRunLoopStop(data->rl);\n@@ -494,9 +494,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tdata->rl = CFRunLoopGetCurrent();\n \n@@ -513,7 +513,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \n \tswitch (data->shutdown_style) {\n \tcase FORCE_ERROR_STOP:\n-\t\tstate->error_code = -1;\n+\t\tstate->listen_error_code = -1;\n \t\t/* fall thru */\n \tcase FORCE_SHUTDOWN:\n \t\tipc_server_stop_async(state->ipc_server_data);\n@@ -525,7 +525,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \treturn;\n \n force_error_stop_without_loop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \tipc_server_stop_async(state->ipc_server_data);\n \treturn;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 948f0e63915..a7c5d263940 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -54,7 +54,7 @@ struct one_watch\n \twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tstruct one_watch *watch_worktree;\n \tstruct one_watch *watch_gitdir;\n@@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname(\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+\tSetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);\n }\n \n static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n@@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch)\n \tfree(watch);\n }\n \n-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+static int start_rdcw_watch(struct fsm_listen_data *data,\n \t\t\t struct one_watch *watch)\n {\n \tDWORD dwNotifyFilter =\n@@ -516,7 +516,7 @@ static int process_1_worktree_event(\n */\n static int process_worktree_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_worktree;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -646,7 +646,7 @@ force_shutdown:\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_gitdir;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -704,11 +704,11 @@ skip_this_path:\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tDWORD dwWait;\n \tint result;\n \n-\tstate->error_code = 0;\n+\tstate->listen_error_code = 0;\n \n \tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n \t\tgoto force_error_stop;\n@@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \t}\n \n force_error_stop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \n force_shutdown:\n \t/*\n@@ -790,7 +790,7 @@ clean_shutdown:\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \treturn 0;\n \n failed:\n@@ -836,16 +836,16 @@ failed:\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tCloseHandle(data->hEventShutdown);\n \tdestroy_watch(data->watch_worktree);\n \tdestroy_watch(data->watch_gitdir);\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nindex f0539349baf..41650bf8972 100644\n--- a/compat/fsmonitor/fsm-listen.h\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n * do so if the listener thread receives a normal shutdown signal from\n * the IPC layer.)\n *\n- * It should set `state->error_code` to -1 if the daemon should exit\n+ * It should set `state->listen_error_code` to -1 if the daemon should exit\n * with an error.\n */\n void fsm_listen__loop(struct fsmonitor_daemon_state *state);\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 223c2131b58..2c6fa1a5d91 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n */\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {\n \tint cookie_seq;\n \tstruct hashmap cookies;\n \n-\tint error_code;\n-\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\tint listen_error_code;\n+\tstruct fsm_listen_data *listen_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"450794","messageId":"f78e4ad87c0bd6b5f4c216a8bd5b61d1bcb6947f.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 15/27] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:15Z","receivedAt":"2022-03-08T22:16:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate another thread to watch over the daemon process and\nautomatically shut it down if necessary.\n\nThis commit creates the basic framework for a \"health\" thread\nto monitor the daemon and/or the file system. Later commits\nwill add platform-specific code to do the actual work.\n\nThe \"health\" thread is intended to monitor conditions that\nwould be difficult to track inside the IPC thread pool and/or\nthe file system listener threads. For example, when there are\nfile system events outside of the watched worktree root or if\nwe want to have an idle-timeout auto-shutdown feature.\n\nThis commit creates the health thread itself, defines the thread-proc\nand sets up the thread's event loop. It integrates this new thread\ninto the existing IPC and Listener thread models.\n\nThis commit defines the API to the platform-specific code where all of\nthe monitoring will actually happen.\n\nThe platform-specific code for MacOS is just stubs. Meaning that the\nhealth thread will immediately exit on MacOS, but that is OK and\nexpected. Future work can define MacOS-specific monitoring.\n\nThe platform-specific code for Windows sets up enough of the\nWaitForMultipleObjects() machinery to watch for system and/or custom\nevents. Currently, the set of wait handles only includes our custom\nshutdown event (sent from our other theads). Later commits in this\nseries will extend the set of wait handles to monitor other\nconditions.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 6 ++-\n builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n contrib/buildsystems/CMakeLists.txt | 2 +\n fsmonitor--daemon.h | 4 ++\n 7 files changed, 192 insertions(+), 2 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n\ndiff --git a/Makefile b/Makefile\nindex 93604fe8ef7..5f1623baadd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -472,8 +472,9 @@ all::\n #\n # If your platform supports a built-in fsmonitor backend, set\n # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n-# `compat/fsmonitor/fsm-listen-.c` that implements the\n-# `fsm_listen__*()` routines.\n+# `compat/fsmonitor/fsm-listen-.c` and\n+# `compat/fsmonitor/fsm-health-.c` files\n+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n # If your platform has OS-specific ways to tell if a repo is incompatible with\n # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n@@ -1982,6 +1983,7 @@ endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n ifdef FSMONITOR_OS_SETTINGS\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 92c61d5b94d..bebb3a292e2 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n@@ -1130,6 +1131,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+static void *fsm_health__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-health\");\n+\n+\tfsm_health__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -1168,6 +1181,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint health_started = 0;\n \tint listener_started = 0;\n \tint err = 0;\n \n@@ -1195,6 +1209,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t}\n \tlistener_started = 1;\n \n+\t/*\n+\t * Start the health thread to watch over our process.\n+\t */\n+\tif (pthread_create(&state->health_thread, NULL,\n+\t\t\t fsm_health__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\terr = error(_(\"could not start fsmonitor health thread\"));\n+\t\tgoto cleanup;\n+\t}\n+\thealth_started = 1;\n+\n \t/*\n \t * The daemon is now fully functional in background threads.\n \t * Our primary thread should now just wait while the threads\n@@ -1217,10 +1242,17 @@ cleanup:\n \t\tpthread_join(state->listener_thread, NULL);\n \t}\n \n+\tif (health_started) {\n+\t\tfsm_health__stop_async(state);\n+\t\tpthread_join(state->health_thread, NULL);\n+\t}\n+\n \tif (err)\n \t\treturn err;\n \tif (state->listen_error_code)\n \t\treturn state->listen_error_code;\n+\tif (state->health_error_code)\n+\t\treturn state->health_error_code;\n \treturn 0;\n }\n \n@@ -1236,6 +1268,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.listen_error_code = 0;\n+\tstate.health_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n@@ -1315,6 +1348,11 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\tif (fsm_health__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize health thread\"));\n+\t\tgoto done;\n+\t}\n+\n \t/*\n \t * CD out of the worktree root directory.\n \t *\n@@ -1338,6 +1376,7 @@ done:\n \tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n+\tfsm_health__dtor(&state);\n \n \tipc_server_free(state.ipc_server_data);\n \ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nnew file mode 100644\nindex 00000000000..b9f709e8548\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -0,0 +1,24 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nnew file mode 100644\nindex 00000000000..94b1d020f25\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsm_health_data\n+{\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hHandles[1]; /* the array does not own these handles */\n+#define HEALTH_SHUTDOWN 0\n+\tint nr_handles; /* number of active event handles */\n+};\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_handles++;\n+\n+\tstate->health_data = data;\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tif (!state || !state->health_data)\n+\t\treturn;\n+\n+\tdata = state->health_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\n+\tFREE_AND_NULL(state->health_data);\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\n+\tfor (;;) {\n+\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n+\t\t\t\t\t\t data->hHandles,\n+\t\t\t\t\t\t FALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->health_error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+clean_shutdown:\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n+}\ndiff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\nnew file mode 100644\nindex 00000000000..45547ba9380\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health.h\n@@ -0,0 +1,47 @@\n+#ifndef FSM_HEALTH_H\n+#define FSM_HEALTH_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor health thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the health thread.\n+ * This will be called from the main thread AFTER joining the thread.\n+ */\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to monitor the\n+ * health of the daemon process. This will run in the health thread.\n+ *\n+ * The health thread should call `ipc_server_stop_async()` if it needs\n+ * to cause a shutdown. (It should NOT do so if it receives a shutdown\n+ * shutdown signal.)\n+ *\n+ * It should set `state->health_error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_health__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the health thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_HEALTH_H */\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex b8f9f7a0388..16ace43d1c7 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 2c6fa1a5d91..2102a5c9ff5 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n+struct fsm_health_data; /* opaque platform-specific data for health thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n+\tpthread_t health_thread;\n \tpthread_mutex_t main_lock;\n \n \tstruct strbuf path_worktree_watch;\n@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n \tstruct hashmap cookies;\n \n \tint listen_error_code;\n+\tint health_error_code;\n \tstruct fsm_listen_data *listen_data;\n+\tstruct fsm_health_data *health_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"450795","messageId":"baf8c031a976fc18189df772e1a2bdff1c213e61.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 17/27] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:17Z","receivedAt":"2022-03-08T22:16:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nForce shutdown fsmonitor daemon if the worktree root directory\nis moved, renamed, or deleted.\n\nUse Windows low-level GetFileInformationByHandle() to get and\ncompare the Windows system unique ID for the directory with a\ncached version when we started up. This lets us detect the\ncase where someone renames the directory that we are watching\nand then creates a new directory with the original pathname.\n\nThis is important because we are listening to a named pipe for\nrequests and they are stored in the Named Pipe File System (NPFS)\nwhich a kernel-resident pseudo filesystem not associated with\nthe actual NTFS directory.\n\nFor example, if the daemon was watching \"~/foo/\", it would have\na directory-watch handle on that directory and a named-pipe\nhandle for \"//./pipe/...foo\". Moving the directory to \"~/bar/\"\ndoes not invalidate the directory handle. (So the daemon would\nactually be watching \"~/bar\" but listening on \"//./pipe/...foo\".\nIf the user then does \"git init ~/foo\" and causes another daemon\nto start, the first daemon will still have ownership of the pipe\nand the second daemon instance will fail to start. \"git status\"\nclients in \"~/foo\" will ask \"//./pipe/...foo\" about changes and\nthe first daemon instance will tell them about \"~/bar\".\n\nThis commit causes the first daemon to shutdown if the system unique\nID for \"~/foo\" changes (changes from what it was when the daemon\nstarted). Shutdown occurs after a periodic poll. After the\nfirst daemon exits and releases the lock on the named pipe,\nsubsequent Git commands may cause another daemon to be started\non \"~/foo\". Similarly, a subsequent Git command may cause another\ndaemon to be started on \"~/bar\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++\n 1 file changed, 143 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 24fc612bf02..0132ca79305 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -29,8 +29,150 @@ struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n \tint nr_handles; /* number of active event handles */\n+\n+\tstruct wt_moved\n+\t{\n+\t\twchar_t wpath[MAX_PATH + 1];\n+\t\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\t} wt_moved;\n };\n \n+/*\n+ * Lookup the system unique ID for the path. This is as close as\n+ * we get to an inode number, but this also contains volume info,\n+ * so it is a little stronger.\n+ */\n+static int lookup_bhfi(wchar_t *wpath,\n+\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n+{\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileW(wpath, desired_access, share_mode, NULL,\n+\t\t\t OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] health thread could not open '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\treturn -1;\n+\t}\n+\n+\tif (!GetFileInformationByHandle(hDir, bhfi)) {\n+\t\terror(_(\"[GLE %ld] health thread getting BHFI for '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\tCloseHandle(hDir);\n+\t\treturn -1;\n+\t}\n+\n+\tCloseHandle(hDir);\n+\treturn 0;\n+}\n+\n+/*\n+ * Compare the relevant fields from two system unique IDs.\n+ * We use this to see if two different handles to the same\n+ * path actually refer to the same *instance* of the file\n+ * or directory.\n+ */\n+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n+\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n+{\n+\treturn (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&\n+\t\tbhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&\n+\t\tbhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);\n+}\n+\n+/*\n+ * Shutdown if the original worktree root directory been deleted,\n+ * moved, or renamed?\n+ *\n+ * Since the main thread did a \"chdir(getenv($HOME))\" and our CWD\n+ * is not in the worktree root directory and because the listener\n+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible\n+ * for the root directory to be moved or deleted while we are still\n+ * watching it. We want to detect that here and force a shutdown.\n+ *\n+ * Granted, a delete MAY cause some operations to fail, such as\n+ * GetOverlappedResult(), but it is not guaranteed. And because\n+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the\n+ * directory, not changes *ON* the directory, our watch will not\n+ * receive a delete event for it.\n+ *\n+ * A move/rename of the worktree root will also not generate an event.\n+ * And since the listener thread already has an open handle, it may\n+ * continue to receive events for events within the directory.\n+ * However, the pathname of the named-pipe was constructed using the\n+ * original location of the worktree root. (Remember named-pipes are\n+ * stored in the NPFS and not in the actual file system.) Clients\n+ * trying to talk to the worktree after the move/rename will not\n+ * reach our daemon process, since we're still listening on the\n+ * pipe with original path.\n+ *\n+ * Furthermore, if the user does something like:\n+ *\n+ * $ mv repo repo.old\n+ * $ git init repo\n+ *\n+ * A new daemon cannot be started in the new instance of \"repo\"\n+ * because the named-pipe is still being used by the daemon on\n+ * the original instance.\n+ *\n+ * So, detect move/rename/delete and shutdown. This should also\n+ * handle unsafe drive removal.\n+ *\n+ * We use the file system unique ID to distinguish the original\n+ * directory instance from a new instance and force a shutdown\n+ * if the unique ID changes.\n+ *\n+ * Since a worktree move/rename/delete/unmount doesn't happen\n+ * that often (and we can't get an immediate event anyway), we\n+ * use a timeout and periodically poll it.\n+ */\n+static int has_worktree_moved(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\tint r;\n+\n+\tswitch (ctx) {\n+\tcase CTX_TERM:\n+\t\treturn 0;\n+\n+\tcase CTX_INIT:\n+\t\tif (xutftowcs_path(data->wt_moved.wpath,\n+\t\t\t\t state->path_worktree_watch.buf) < 0) {\n+\t\t\terror(_(\"could not convert to wide characters: '%s'\"),\n+\t\t\t state->path_worktree_watch.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/*\n+\t\t * On the first call we lookup the unique sequence ID for\n+\t\t * the worktree root directory.\n+\t\t */\n+\t\treturn lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);\n+\n+\tcase CTX_TIMER:\n+\t\tr = lookup_bhfi(data->wt_moved.wpath, &bhfi);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t\tif (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {\n+\t\t\terror(_(\"BHFI changed '%ls'\"), data->wt_moved.wpath);\n+\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tdie(\"unhandled case in 'has_worktree_moved': %d\",\n+\t\t (int)ctx);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+\n int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data;\n@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n * A table of the polling functions.\n */\n static interval_fn *table[] = {\n+\thas_worktree_moved,\n \tNULL, /* must be last */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"450796","messageId":"24591920878c37b0d0b9d91e8b2d36b2eb81b24e.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 19/27] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:19Z","receivedAt":"2022-03-08T22:16:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to perform binary search over the cache-entries for a directory\nnotification and then linearly scan forward to find the immediate children.\n\nPreviously, when the FSMonitor reported a modified directory Git would\nperform a linear search on the entire cache-entry array for all\nentries matching that directory prefix and invalidate them. Since the\ncache-entry array is already sorted, we can use a binary search to\nfind the first matching entry and then only linearly walk forward and\ninvalidate entries until the prefix changes.\n\nAlso, the original code would invalidate anything having the same\ndirectory prefix. Since a directory event should only be received for\nitems that are immediately within the directory (and not within\nsub-directories of it), only invalidate those entries and not the\nwhole subtree.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 54 insertions(+), 17 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 292a6742b4f..e1229c289cf 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n-\tif (name[len - 1] == '/') {\n+\tint pos = index_name_pos(istate, name, len);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n+\t\t\t name, pos);\n \n+\tif (name[len - 1] == '/') {\n \t\t/*\n-\t\t * TODO We should binary search to find the first path with\n-\t\t * TODO this directory prefix. Then linearly update entries\n-\t\t * TODO while the prefix matches. Taking care to search without\n-\t\t * TODO the trailing slash -- because '/' sorts after a few\n-\t\t * TODO interesting special chars, like '.' and ' '.\n+\t\t * The daemon can decorate directory events, such as\n+\t\t * moves or renames, with a trailing slash if the OS\n+\t\t * FS Event contains sufficient information, such as\n+\t\t * MacOS.\n+\t\t *\n+\t\t * Use this to invalidate the entire cone under that\n+\t\t * directory.\n+\t\t *\n+\t\t * We do not expect an exact match because the index\n+\t\t * does not normally contain directory entries, so we\n+\t\t * start at the insertion point and scan.\n \t\t */\n+\t\tif (pos < 0)\n+\t\t\tpos = -pos - 1;\n \n \t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n-\t\t\t starts_with(istate->cache[i]->name, name))\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n-\t\t/* Need to remove the / from the path for the untracked cache */\n+\n+\t\t/*\n+\t\t * We need to remove the traling \"/\" from the path\n+\t\t * for the untracked cache.\n+\t\t */\n \t\tname[len - 1] = '\\0';\n+\t} else if (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\tint pos = index_name_pos(istate, name, strlen(name));\n-\n-\t\tif (pos >= 0) {\n-\t\t\tstruct cache_entry *ce = istate->cache[pos];\n-\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n \t}\n \n@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n \t * as it could be a new untracked file.\n \t */\n-\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"450797","messageId":"796b65913939e4de5ff745981e3d36fe4926ae08.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 18/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:18Z","receivedAt":"2022-03-08T22:16:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the listener thread to shutdown the daemon if the spelling of the\nworktree root directory changes.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 7c81bc7e5bf..b6c33f2cf3b 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -178,6 +178,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tstrbuf_release(&msg);\n }\n \n+static int ef_is_root_changed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagRootChanged);\n+}\n+\n static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n {\n \treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n@@ -287,6 +292,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_is_root_changed(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * The spelling of the pathname of the root directory\n+\t\t\t * has changed. This includes the name of the root\n+\t\t\t * directory itself or of any parent directory in the\n+\t\t\t * path.\n+\t\t\t *\n+\t\t\t * (There may be other conditions that throw this,\n+\t\t\t * but I couldn't find any information on it.)\n+\t\t\t *\n+\t\t\t * Force a shutdown now and avoid things getting\n+\t\t\t * out of sync. The Unix domain socket is inside\n+\t\t\t * the .git directory and a spelling change will make\n+\t\t\t * it hard for clients to rendezvous with us.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"event: root changed\");\n+\t\t\tgoto force_shutdown;\n+\t\t}\n+\n \t\tif (ef_ignore_xattr(event_flags[k])) {\n \t\t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n-- \ngitgitgadget\n\n"},{"id":"450798","messageId":"524d449ed64fab482cbe8eb1f325490ca3860440.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 22/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:22Z","receivedAt":"2022-03-08T22:16:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNever set CE_FSMONITOR_VALID on the cache-entry of submodule\ndirectories.\n\nDuring a client command like 'git status', we may need to recurse\ninto each submodule to compute a status summary for the submodule.\nSince the purpose of the ce_flag is to let Git avoid scanning a\ncache-entry, setting the flag causes the recursive call to be\navoided and we report incorrect (no status) for the submodule.\n\nWe created an OS watch on the root directory of our working\ndirectory and we receive events for everything in the cone\nunder it. When submodules are present inside our working\ndirectory, we receive events for both our repo (the super) and\nany subs within it. Since our index doesn't have any information\nfor items within the submodules, we can't use those events.\n\nWe could try to truncate the paths of those events back to the\nsubmodule boundary and mark the GITLINK as dirty, but that\nfeels expensive since we would have to prefix compare every FS\nevent that we receive against a list of submodule roots. And\nit still wouldn't be sufficient to correctly report status on\nthe submodule, since we don't have any space in the cache-entry\nto cache the submodule's status (the 'SCMU' bits in porcelain\nV2 speak). That is, the CE_FSMONITOR_VALID bit just says that\nwe don't need to scan/inspect it because we already know the\nanswer -- it doesn't say that the item is clean -- and we\ndon't have space in the cache-entry to store those answers.\nSo we should always do the recursive scan.\n\nTherefore, we should never set the flag on GITLINK cache-entries.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 2 +\n fsmonitor.h | 11 +++++\n t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 106 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex e1229c289cf..57d6a483bee 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tif (fsmonitor_enabled) {\n \t\t\t/* Mark all entries valid */\n \t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\t\tif (S_ISGITLINK(istate->cache[i]->ce_mode))\n+\t\t\t\t\tcontinue;\n \t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n \t\t\t}\n \ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 3f41f653691..edf7ce5203b 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n * called any time the cache entry has been updated to reflect the\n * current state of the file on disk.\n+ *\n+ * However, never mark submodules as valid. When commands like \"git\n+ * status\" run they might need to recurse into the submodule (using a\n+ * child process) to get a summary of the submodule state. We don't\n+ * have (and don't want to create) the facility to translate every\n+ * FS event that we receive and that happens to be deep inside of a\n+ * submodule back to the submodule root, so we cannot correctly keep\n+ * track of this bit on the gitlink directory. Therefore, we never\n+ * set it on submodules.\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n \n \tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n \t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tif (S_ISGITLINK(ce->ce_mode))\n+\t\t\treturn;\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 8b63067cbf9..a18e077d375 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -728,4 +728,97 @@ do\n \t'\n done\n \n+# Test fsmonitor interaction with submodules.\n+#\n+# If we start the daemon in the super, it will see FS events for\n+# everything in the working directory cone and this includes any\n+# files/directories contained *within* the submodules.\n+#\n+# A `git status` at top level will get events for items within the\n+# submodule and ignore them, since they aren't named in the index\n+# of the super repo. This makes the fsmonitor response a little\n+# noisy, but it doesn't alter the correctness of the state of the\n+# super-proper.\n+#\n+# When we have submodules, `git status` normally does a recursive\n+# status on each of the submodules and adds a summary row for any\n+# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n+#\n+# It is therefore important that the top level status not be tricked\n+# by the FSMonitor response to skip those recursive calls.\n+\n+my_match_and_clean () {\n+\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n+\tgit -C super --no-optional-locks -c core.fsmonitor=false \\\n+\t\tstatus --porcelain=v2 >actual.without &&\n+\ttest_cmp actual.with actual.without &&\n+\n+\tgit -C super/dir_1/dir_2/sub reset --hard &&\n+\tgit -C super/dir_1/dir_2/sub clean -d -f\n+}\n+\n+test_expect_success \"Submodule\" '\n+\ttest_when_finished \"git -C super fsmonitor--daemon stop\" &&\n+\n+\tgit init \"super\" &&\n+\techo x >super/file_1 &&\n+\techo y >super/file_2 &&\n+\techo z >super/file_3 &&\n+\tmkdir super/dir_1 &&\n+\techo a >super/dir_1/file_11 &&\n+\techo b >super/dir_1/file_12 &&\n+\tmkdir super/dir_1/dir_2 &&\n+\techo a >super/dir_1/dir_2/file_21 &&\n+\techo b >super/dir_1/dir_2/file_22 &&\n+\tgit -C super add . &&\n+\tgit -C super commit -m \"initial super commit\" &&\n+\n+\tgit init \"sub\" &&\n+\techo x >sub/file_x &&\n+\techo y >sub/file_y &&\n+\techo z >sub/file_z &&\n+\tmkdir sub/dir_x &&\n+\techo a >sub/dir_x/file_a &&\n+\techo b >sub/dir_x/file_b &&\n+\tmkdir sub/dir_x/dir_y &&\n+\techo a >sub/dir_x/dir_y/file_a &&\n+\techo b >sub/dir_x/dir_y/file_b &&\n+\tgit -C sub add . &&\n+\tgit -C sub commit -m \"initial sub commit\" &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\tstart_daemon super &&\n+\tgit -C super config core.fsmonitor true &&\n+\tgit -C super update-index --fsmonitor &&\n+\tgit -C super status &&\n+\n+\t# Now run pairs of commands w/ and w/o FSMonitor while we make\n+\t# some dirt in the submodule and confirm matching output.\n+\n+\t# Completely clean status.\n+\tmy_match_and_clean &&\n+\n+\t# .M S..U\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M SC..\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tgit -C super/dir_1/dir_2/sub commit -m \"SC..\" &&\n+\tmy_match_and_clean\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450799","messageId":"4b59013caddc0d8ea6bdc2e4b78166c31b6e0027.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 21/27] t/perf/p7527: add perf test for builtin FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:21Z","receivedAt":"2022-03-08T22:16:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++\n 1 file changed, 257 insertions(+)\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\ndiff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..9338b9ea008\n--- /dev/null\n+++ b/t/perf/p7527-builtin-fsmonitor.sh\n@@ -0,0 +1,257 @@\n+#!/bin/sh\n+\n+test_description=\"Perf test for the builtin FSMonitor\"\n+\n+. ./perf-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+# Lie to perf-lib and ask for a new empty repo and avoid\n+# the complaints about GIT_PERF_REPO not being big enough\n+# the perf hit when GIT_PERF_LARGE_REPO is copied into\n+# the trash directory.\n+#\n+# NEEDSWORK: It would be nice if perf-lib had an option to\n+# \"borrow\" an existing large repo (especially for gigantic\n+# monorepos) and use it in-place. For now, fake it here.\n+#\n+test_perf_fresh_repo\n+\n+\n+# Use a generated synthetic monorepo. If it doesn't exist, we will\n+# generate it. If it does exist, we will put it in a known state\n+# before we start our timings.\n+#\n+PARAM_D=5\n+PARAM_W=10\n+PARAM_F=9\n+\n+PARAMS=\"$PARAM_D\".\"$PARAM_W\".\"$PARAM_F\"\n+\n+BALLAST_BR=p0006-ballast\n+export BALLAST_BR\n+\n+TMP_BR=tmp_br\n+export TMP_BR\n+\n+REPO=../repos/gen-many-files-\"$PARAMS\".git\n+export REPO\n+\n+if ! test -d $REPO\n+then\n+\t(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)\n+fi\n+\n+\n+enable_uc () {\n+\tgit -C $REPO config core.untrackedcache true\n+\tgit -C $REPO update-index --untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+disable_uc () {\n+\tgit -C $REPO config core.untrackedcache false\n+\tgit -C $REPO update-index --no-untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+start_fsm () {\n+\tgit -C $REPO fsmonitor--daemon start\n+\tgit -C $REPO fsmonitor--daemon status\n+\tgit -C $REPO config core.fsmonitor true\n+\tgit -C $REPO update-index --fsmonitor\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+stop_fsm () {\n+\tgit -C $REPO config --unset core.fsmonitor\n+\tgit -C $REPO update-index --no-fsmonitor\n+\ttest_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+\n+# Ensure that FSMonitor is turned off on the borrowed repo.\n+#\n+test_expect_success \"Setup borrowed repo (fsm+uc)\" \"\n+\tstop_fsm &&\n+\tdisable_uc\n+\"\n+\n+# Also ensure that it starts in a known state.\n+#\n+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time\n+# the ballast checkout, since only the first invocation does any work and the\n+# subsequent ones just print \"already on branch\" and quit, so the reported\n+# time is not useful.\n+#\n+# Create a temp branch and do all work relative to it so that we don't\n+# accidentially alter the real ballast branch.\n+#\n+test_expect_success \"Setup borrowed repo (temp ballast branch)\" \"\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO reset --hard &&\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tgit -C $REPO branch $TMP_BR $BALLAST_BR &&\n+\tgit -C $REPO checkout $TMP_BR\n+\"\n+\n+\n+echo Data >data.txt\n+\n+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With\n+# FSMonitor enabled, we can get a skewed view of status times, since\n+# the index MAY (or may not) be updated after the first invocation\n+# which will update the FSMonitor Token, so the subsequent invocations\n+# may get a smaller response from the daemon.\n+#\n+do_status () {\n+\tmsg=$1\n+\n+\ttest_perf \"$msg\" \"\n+\t\tgit -C $REPO status >/dev/null 2>&1\n+\t\"\n+}\n+\n+do_matrix () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tt=\"[uc $uc][fsm $fsm]\"\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_expect_success \"$t Setup matrix branch\" \"\n+\t\tgit -C $REPO clean -d -f &&\n+\t\tgit -C $REPO checkout $TMP_BR &&\n+\t\ttest_might_fail git -C $REPO branch -D $MATRIX_BR &&\n+\t\tgit -C $REPO branch $MATRIX_BR $TMP_BR &&\n+\t\tgit -C $REPO checkout $MATRIX_BR\n+\t\"\n+\n+\tif test $uc = true\n+\tthen\n+\t\tenable_uc\n+\telse\n+\t\tdisable_uc\n+\tfi\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstart_fsm\n+\telse\n+\t\tstop_fsm\n+\tfi\n+\n+\tdo_status \"$t status after checkout\"\n+\n+\t# Modify many files in the matrix branch.\n+\t# Stage them.\n+\t# Commit them.\n+\t# Rollback.\n+\t#\n+\ttest_expect_success \"$t modify tracked files\" \"\n+\t\tfind $REPO -name file1 -exec cp data.txt {} \\\\;\n+\t\"\n+\n+\tdo_status \"$t status after big change\"\n+\n+\t# Don't bother timing the \"add\" because _REPEAT_COUNT\n+\t# issue described above.\n+\t#\n+\ttest_expect_success \"$t add all\" \"\n+\t\tgit -C $REPO add -A\n+\t\"\n+\n+\tdo_status \"$t status after add all\"\n+\n+\ttest_expect_success \"$t add dot\" \"\n+\t\tgit -C $REPO add .\n+\t\"\n+\n+\tdo_status \"$t status after add dot\"\n+\n+\ttest_expect_success \"$t commit staged\" \"\n+\t\tgit -C $REPO commit -a -m data\n+\t\"\n+\n+\tdo_status \"$t status after commit\"\n+\n+\ttest_expect_success \"$t reset HEAD~1 hard\" \"\n+\t\tgit -C $REPO reset --hard HEAD~1 >/dev/null 2>&1\n+\t\"\n+\n+\tdo_status \"$t status after reset hard\"\n+\n+\t# Create some untracked files.\n+\t#\n+\ttest_expect_success \"$t create untracked files\" \"\n+\t\tcp -R $REPO/ballast/dir1 $REPO/ballast/xxx1\n+\t\"\n+\n+\tdo_status \"$t status after create untracked files\"\n+\n+\t# Remove the new untracked files.\n+\t#\n+\ttest_expect_success \"$t clean -df\" \"\n+\t\tgit -C $REPO clean -d -f\n+\t\"\n+\n+\tdo_status \"$t status after clean\"\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstop_fsm\n+\tfi\n+}\n+\n+# Begin testing each case in the matrix that we care about.\n+#\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\n+fsm_values=\"false true\"\n+\n+for uc_val in $uc_values\n+do\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tdo_matrix $uc_val $fsm_val\n+\tdone\n+done\n+\n+cleanup () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_might_fail git -C $REPO branch -D $MATRIX_BR\n+}\n+\n+\n+# We're borrowing this repo. We should leave it in a clean state.\n+#\n+test_expect_success \"Cleanup temp and matrix branches\" \"\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\tfor fsm_val in $fsm_values\n+\t\tdo\n+\t\t\tcleanup $uc_val $fsm_val\n+\t\tdone\n+\tdone\n+\"\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"450800","messageId":"95b9d4210d2c888689d2d6ffd60adb4c43d03bcc.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 24/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:24Z","receivedAt":"2022-03-08T22:16:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nEmit NFC or NFC and NFD spellings of pathnames on macOS.\n\nMacOS is Unicode composition insensitive, so NFC and NFD spellings are\ntreated as aliases and collide. While the spelling of pathnames in\nfilesystem events depends upon the underlying filesystem, such as\nAPFS, HFS+ or FAT32, the OS enforces such collisions regardless of\nfilesystem.\n\nTeach the daemon to always report the NFC spelling and to report\nthe NFD spelling when stored in that format on the disk.\n\nThis is slightly more general than \"core.precomposeUnicode\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex b6c33f2cf3b..3332d3b7792 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -227,6 +227,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n \treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n }\n \n+/*\n+ * On MacOS we have to adjust for Unicode composition insensitivity\n+ * (where NFC and NFD spellings are not respected). The different\n+ * spellings are essentially aliases regardless of how the path is\n+ * actually stored on the disk.\n+ *\n+ * This is related to \"core.precomposeUnicode\" (which wants to try\n+ * to hide NFD completely and treat everything as NFC). Here, we\n+ * don't know what the value the client has (or will have) for this\n+ * config setting when they make a query, so assume the worst and\n+ * emit both when the OS gives us an NFD path.\n+ */\n+static void my_add_path(struct fsmonitor_batch *batch, const char *path)\n+{\n+\tchar *composed;\n+\n+\t/* add the NFC or NFD path as received from the OS */\n+\tfsmonitor_batch__add_path(batch, path);\n+\n+\t/* if NFD, also add the corresponding NFC spelling */\n+\tcomposed = (char *)precompose_string_if_needed(path);\n+\tif (!composed || composed == path)\n+\t\treturn;\n+\n+\tfsmonitor_batch__add_path(batch, composed);\n+\tfree(composed);\n+}\n+\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -377,7 +406,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t\tmy_add_path(batch, rel);\n \t\t\t}\n \n \t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n@@ -390,7 +419,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t\tmy_add_path(batch, tmp.buf);\n \t\t\t}\n \n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"450801","messageId":"06a32413854c2b35f289da3b42861fc051e1d4cc.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 20/27] t7527: FSMonitor tests for directory moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:20Z","receivedAt":"2022-03-08T22:16:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate unit tests to move a directory. Verify that `git status`\ngives the same result with and without FSMonitor enabled.\n\nNEEDSWORK: This test exposes a bug in the untracked-cache on\nWindows when FSMonitor is disabled. These are commented out\nfor the moment.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 39 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex c2627f28865..8b63067cbf9 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -231,6 +231,16 @@ test_expect_success 'setup' '\n \ttrace*\n \tEOF\n \n+\tmkdir -p T1/T2/T3/T4 &&\n+\techo 1 >T1/F1 &&\n+\techo 1 >T1/T2/F1 &&\n+\techo 1 >T1/T2/T3/F1 &&\n+\techo 1 >T1/T2/T3/T4/F1 &&\n+\techo 2 >T1/F2 &&\n+\techo 2 >T1/T2/F2 &&\n+\techo 2 >T1/T2/T3/F2 &&\n+\techo 2 >T1/T2/T3/T4/F2 &&\n+\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n@@ -324,6 +334,19 @@ verify_status () {\n \techo HELLO AFTER\n }\n \n+move_directory_contents_deeper() {\n+\tmkdir T1/_new_\n+\tmv T1/[A-Z]* T1/_new_\n+}\n+\n+move_directory_up() {\n+\tmv T1/T2/T3 T1\n+}\n+\n+move_directory() {\n+\tmv T1/T2/T3 T1/T2/NewT3\n+}\n+\n # The next few test cases confirm that our fsmonitor daemon sees each type\n # of OS filesystem notification that we care about. At this layer we just\n # ensure we are getting the OS notifications and do not try to confirm what\n@@ -655,6 +678,22 @@ do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n+\t\t# NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor\n+\t\t# is DISABLED. Turn off a few test that cause it problems until\n+\t\t# we can debug it.\n+\t\t#\n+\t\ttry_moves=\"true\"\n+\t\ttest_have_prereq UNTRACKED_CACHE,WINDOWS && \\\n+\t\t\ttest $uc_val = true && \\\n+\t\t\ttest $fsm_val = false && \\\n+\t\t\ttry_moves=\"false\"\n+\t\tif test $try_moves = true\n+\t\tthen\n+\t\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n+\t\t\tmatrix_try $uc_val $fsm_val move_directory_up\n+\t\t\tmatrix_try $uc_val $fsm_val move_directory\n+\t\tfi\n+\n \t\tif test $fsm_val = true\n \t\tthen\n \t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n-- \ngitgitgadget\n\n"},{"id":"450802","messageId":"bb72f911a05727e75f44e7a4517b34d474724bea.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 16/27] fsm-health-win32: add polling framework to monitor daemon health","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:16Z","receivedAt":"2022-03-08T22:16:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend the Windows version of the \"health\" thread to periodically\ninspect the system and shutdown if warranted.\n\nThis commit updates the thread's wait loop to use a timeout and\ndefines a (currently empty) table of functions to poll the system.\n\nA later commit will add functions to the table to actually\ninspect the system.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-\n 1 file changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 94b1d020f25..24fc612bf02 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -4,6 +4,24 @@\n #include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n+/*\n+ * Every minute wake up and test our health.\n+ */\n+#define WAIT_FREQ_MS (60 * 1000)\n+\n+/*\n+ * State machine states for each of the interval functions\n+ * used for polling our health.\n+ */\n+enum interval_fn_ctx {\n+\tCTX_INIT = 0,\n+\tCTX_TERM,\n+\tCTX_TIMER\n+};\n+\n+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx);\n+\n struct fsm_health_data\n {\n \tHANDLE hEventShutdown;\n@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n \tFREE_AND_NULL(state->health_data);\n }\n \n+/*\n+ * A table of the polling functions.\n+ */\n+static interval_fn *table[] = {\n+\tNULL, /* must be last */\n+};\n+\n+/*\n+ * Call all of the polling functions in the table.\n+ * Shortcut and return first error.\n+ *\n+ * Return 0 if all succeeded.\n+ */\n+static int call_all(struct fsmonitor_daemon_state *state,\n+\t\t enum interval_fn_ctx ctx)\n+{\n+\tint k;\n+\n+\tfor (k = 0; table[k]; k++) {\n+\t\tint r = table[k](state, ctx);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n+\tint r;\n+\n+\tr = call_all(state, CTX_INIT);\n+\tif (r < 0)\n+\t\tgoto force_error_stop;\n+\tif (r > 0)\n+\t\tgoto force_shutdown;\n \n \tfor (;;) {\n \t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n \t\t\t\t\t\t data->hHandles,\n-\t\t\t\t\t\t FALSE, INFINITE);\n+\t\t\t\t\t\t FALSE, WAIT_FREQ_MS);\n \n \t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n \t\t\tgoto clean_shutdown;\n \n+\t\tif (dwWait == WAIT_TIMEOUT) {\n+\t\t\tr = call_all(state, CTX_TIMER);\n+\t\t\tif (r < 0)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (r > 0)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\terror(_(\"health thread wait failed [GLE %ld]\"),\n \t\t GetLastError());\n \t\tgoto force_error_stop;\n@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)\n \n force_error_stop:\n \tstate->health_error_code = -1;\n+force_shutdown:\n \tipc_server_stop_async(state->ipc_server_data);\n clean_shutdown:\n+\tcall_all(state, CTX_TERM);\n \treturn;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"450803","messageId":"71673be2da5e87163ec95f5d99f0a22b698276e6.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 13/27] fsmonitor--daemon: prepare for adding health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:13Z","receivedAt":"2022-03-08T22:16:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor daemon thread startup to make it easier to start\na third thread class to monitor the health of the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex dbd37a2b3b8..d5a8fc2ddc2 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1168,6 +1168,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint listener_started = 0;\n+\tint err = 0;\n \n \t/*\n \t * Start the IPC thread pool before the we've started the file\n@@ -1188,15 +1190,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \tif (pthread_create(&state->listener_thread, NULL,\n \t\t\t fsm_listen__thread_proc, state) < 0) {\n \t\tipc_server_stop_async(state->ipc_server_data);\n-\t\tipc_server_await(state->ipc_server_data);\n-\n-\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t\terr = error(_(\"could not start fsmonitor listener thread\"));\n+\t\tgoto cleanup;\n \t}\n+\tlistener_started = 1;\n \n \t/*\n \t * The daemon is now fully functional in background threads.\n+\t * Our primary thread should now just wait while the threads\n+\t * do all the work.\n+\t */\n+cleanup:\n+\t/*\n \t * Wait for the IPC thread pool to shutdown (whether by client\n-\t * request or from filesystem activity).\n+\t * request, from filesystem activity, or an error).\n \t */\n \tipc_server_await(state->ipc_server_data);\n \n@@ -1205,10 +1212,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * event from the IPC thread pool, but it doesn't hurt to tell\n \t * it again. And wait for it to shutdown.\n \t */\n-\tfsm_listen__stop_async(state);\n-\tpthread_join(state->listener_thread, NULL);\n+\tif (listener_started) {\n+\t\tfsm_listen__stop_async(state);\n+\t\tpthread_join(state->listener_thread, NULL);\n+\t}\n \n-\treturn state->error_code;\n+\tif (err)\n+\t\treturn err;\n+\tif (state->error_code)\n+\t\treturn state->error_code;\n+\treturn 0;\n }\n \n static int fsmonitor_run_daemon(void)\n-- \ngitgitgadget\n\n"},{"id":"450804","messageId":"c7264decaf6de8e3d39c3ea5421f45c93a7a84bd.1646777727.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 23/27] t7527: test FSMonitor on case insensitive+preserving file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:23Z","receivedAt":"2022-03-08T22:16:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTest that FS events from the OS are received using the preserved,\non-disk spelling of files/directories rather than spelling used\nto make the change.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 40 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 40 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex a18e077d375..a41e37236b5 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -821,4 +821,44 @@ test_expect_success \"Submodule\" '\n \tmy_match_and_clean\n '\n \n+# On a case-insensitive file system, confirm that the daemon\n+# notices when the .git directory is moved/renamed/deleted\n+# regardless of how it is spelled in the the FS event.\n+# That is, does the FS event receive the spelling of the\n+# operation or does it receive the spelling preserved with\n+# the file/directory.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n+#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n+\n+\tgit init test_insensitive &&\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/insensitive.trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon test_insensitive\n+\t) &&\n+\n+\tmkdir -p test_insensitive/abc/def &&\n+\techo xyz >test_insensitive/ABC/DEF/xyz &&\n+\n+\ttest_path_is_dir test_insensitive/.git &&\n+\ttest_path_is_dir test_insensitive/.GIT &&\n+\n+\t# Rename .git using an alternate spelling to verify that that\n+\t# daemon detects it and automatically shuts down.\n+\tmv test_insensitive/.GIT test_insensitive/.FOO &&\n+\tsleep 1 &&\n+\tmv test_insensitive/.FOO test_insensitive/.git &&\n+\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\t# Verify that events were reported using on-disk spellings of the\n+\t# directories and files that we touched. We may or may not get a\n+\t# trailing slash on modified directories.\n+\t#\n+\tegrep \"^event: abc/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450805","messageId":"5a0c1b7a2873accc6db4b34493962378819eacd4.1646777728.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:25Z","receivedAt":"2022-03-08T22:16:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a set of prereqs to help understand how file names\nare handled by the filesystem when they contain NFC and NFD\nUnicode characters.\n\nSigned-off-by: Jeff Hostetler \n---\n t/lib-unicode-nfc-nfd.sh | 159 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 159 insertions(+)\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n\ndiff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\nnew file mode 100755\nindex 00000000000..a09e910c302\n--- /dev/null\n+++ b/t/lib-unicode-nfc-nfd.sh\n@@ -0,0 +1,159 @@\n+# Help detect how Unicode NFC and NFD are handled on the filesystem.\n+\n+# A simple character that has a NFD form.\n+#\n+# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n+# UTF8(NFC): \\xc3 \\xa9\n+#\n+# NFD: U+0065 LATIN SMALL LETTER E\n+# U+0301 COMBINING ACUTE ACCENT\n+# UTF8(NFD): \\x65 + \\xcc \\x81\n+#\n+utf8_nfc=$(printf \"\\xc3\\xa9\")\n+utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n+\n+# Is the OS or the filesystem \"Unicode composition sensitive\"?\n+#\n+# That is, does the OS or the filesystem allow files to exist with\n+# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n+# tell us that the NFC and NFD forms are equivalent.\n+#\n+# This is or may be independent of what type of filesystem we have,\n+# since it might be handled by the OS at a layer above the FS.\n+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n+# collision, for example.\n+#\n+# This does not tell us how the Unicode pathname will be spelled\n+# on disk, but rather only that the two spelling \"collide\". We\n+# will examine the actual on disk spelling in a later prereq.\n+#\n+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${utf8_nfc} &&\n+\tmkdir trial_${utf8_nfd}\n+'\n+\n+# Is the spelling of an NFC pathname preserved on disk?\n+#\n+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n+# and on APFS, NFC paths are preserved. As we have established\n+# above, this is independent of \"composition sensitivity\".\n+#\n+# 0000000 63 5f c3 a9\n+#\n+# (/usr/bin/od output contains different amount of whitespace\n+# on different platforms, so we need the wildcards here.)\n+#\n+test_lazy_prereq UNICODE_NFC_PRESERVED '\n+\tmkdir c_${utf8_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *c3 *a9\"\n+'\n+\n+# Is the spelling of an NFD pathname preserved on disk?\n+#\n+# 0000000 64 5f 65 cc 81\n+#\n+test_lazy_prereq UNICODE_NFD_PRESERVED '\n+\tmkdir d_${utf8_nfd} &&\n+\tls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n+'\n+\tmkdir c_${utf8_nfc} &&\n+\tmkdir d_${utf8_nfd} &&\n+\n+# The following _DOUBLE_ forms are more for my curiosity,\n+# but there may be quirks lurking when there are multiple\n+# combining characters in non-canonical order.\n+\n+# Unicode also allows multiple combining characters\n+# that can be decomposed in pieces.\n+#\n+# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n+# UTF8(NFC): \\xe1 \\xbd \\xa7\n+#\n+# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n+# U+0342 COMBINING GREEK PERISPOMENI\n+# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n+#\n+# But U+1f61 decomposes into\n+# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n+# U+0314 COMBINING REVERSED COMMA ABOVE\n+# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n+#\n+# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n+#\n+# Note that I've used the canonical ordering of the\n+# combinining characters. It is also possible to\n+# swap them. My testing shows that that non-standard\n+# ordering also causes a collision in mkdir. However,\n+# the resulting names don't draw correctly on the\n+# terminal (implying that the on-disk format also has\n+# them out of order).\n+#\n+greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n+greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n+greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n+\n+# See if a double decomposition also collides.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${greek_nfc} &&\n+\tmkdir trial_${greek_nfd2}\n+'\n+\n+# See if the NFC spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n+\tmkdir c_${greek_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n+'\n+\n+# See if the NFD spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n+\tmkdir d_${greek_nfd2} &&\n+\tls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n+'\n+\n+if test $unicode_debug = 1\n+then\n+\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\techo NFC maintains original spelling.\n+\telse\n+\t\techo NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFD_PRESERVED\n+\tthen\n+\t\techo NFD maintains original spelling.\n+\telse\n+\t\techo NFD is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n+\tthen\n+\t\techo Double NFC maintains original spelling.\n+\telse\n+\t\techo Double NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n+\tthen\n+\t\techo Double NFD maintains original spelling.\n+\telse\n+\t\techo Double NFD is modified.\n+\tfi\n+fi\n-- \ngitgitgadget\n\n"},{"id":"450806","messageId":"e3e01677d9338310c3731e9fc02a9e7fd6af0032.1646777728.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 27/27] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:27Z","receivedAt":"2022-03-08T22:16:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nOn MacOS mark repos on NTFS or FAT32 volumes as incompatible.\n\nThe builtin FSMonitor used Unix domain sockets on MacOS for IPC\nwith clients. These sockets are kept in the .git directory.\nUnix sockets are not supported by NTFS and FAT32, so the daemon\ncannot start up.\n\nTest for this during our compatibility checking so that client\ncommands do not keep trying to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---\n fsmonitor-settings.c | 3 +++\n fsmonitor-settings.h | 1 +\n 3 files changed, 21 insertions(+), 3 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex fdd762bf79d..efc732c0f31 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -7,7 +7,7 @@\n #include \n \n /*\n- * Remote working directories are problematic for FSMonitor.\n+ * [1] Remote working directories are problematic for FSMonitor.\n *\n * The underlying file system on the server machine and/or the remote\n * mount type (NFS, SAMBA, etc.) dictates whether notification events\n@@ -40,8 +40,16 @@\n *\n * So (for now at least), mark remote working directories as\n * incompatible.\n+ *\n+ *\n+ * [2] FAT32 and NTFS working directories are problematic too.\n+ *\n+ * The builtin FSMonitor uses a Unix domain socket in the .git\n+ * directory for IPC. These Windows drive formats do not support\n+ * Unix domain sockets, so mark them as incompatible for the daemon.\n+ *\n */\n-static enum fsmonitor_reason check_remote(struct repository *r)\n+static enum fsmonitor_reason check_volume(struct repository *r)\n {\n \tstruct statfs fs;\n \n@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)\n \tif (!(fs.f_flags & MNT_LOCAL))\n \t\treturn FSMONITOR_REASON_REMOTE;\n \n+\tif (!strcmp(fs.f_fstypename, \"msdos\")) /* aka FAT32 */\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n+\tif (!strcmp(fs.f_fstypename, \"ntfs\"))\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \n@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n \n-\treason = check_remote(r);\n+\treason = check_volume(r);\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 7ff3f98964d..5734c93baf9 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -186,6 +186,9 @@ const char *fsm_settings__get_reason_msg(struct repository *r)\n \n \tcase FSMONITOR_REASON_VFS4GIT:\n \t\treturn _(\"virtual repos are incompatible with fsmonitor\");\n+\n+\tcase FSMONITOR_REASON_NOSOCKETS:\n+\t\treturn _(\"repo filesystem does not support Unix sockets\");\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 6aa9a00379b..af792313413 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -19,6 +19,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n \tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n+\tFSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n-- \ngitgitgadget\n"},{"id":"450807","messageId":"a45c1fd3000802ef3c8855bd3e9d7c3f2b11f6fd.1646777728.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v2 26/27] t7527: test Unicode NFC/NFD handling on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T22:15:26Z","receivedAt":"2022-03-08T22:16:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that the daemon reports events using the on-disk\nspelling for Unicode NFC/NFD characters. On APFS we still\nhave Unicode aliasing, so we cannot create two files that\nonly differ by NFC/NFD, but the on-disk format preserves\nthe spelling used to create the file. On HFS+ we also\nhave aliasing, but the path is always stored on disk in\nNFD.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 54 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 54 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex a41e37236b5..48c9125d8da 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -861,4 +861,58 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n+unicode_debug=0\n+. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n+\n+# See if the OS or filesystem does NFC/NFD aliasing/munging.\n+#\n+# The daemon should err on the side of caution and send BOTH the\n+# NFC and NFD forms. It does not know the original spelling of\n+# the pathname (how the user thinks it should be spelled), so\n+# emit both and let the client decide (when necessary). This is\n+# similar to \"core.precomposeUnicode\".\n+#\n+test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n+\n+\tgit init test_unicode &&\n+\t(\n+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/unicode.trace\" &&\n+\t\texport GIT_TRACE_FSMONITOR &&\n+\n+\t\tstart_daemon test_unicode\n+\t) &&\n+\n+\t# Create a directory using an NFC spelling.\n+\t#\n+\tmkdir test_unicode/nfc &&\n+\tmkdir test_unicode/nfc/c_${utf8_nfc} &&\n+\n+\t# Create a directory using an NFD spelling.\n+\t#\n+\tmkdir test_unicode/nfd &&\n+\tmkdir test_unicode/nfd/d_${utf8_nfd} &&\n+\n+\tgit -C test_unicode fsmonitor--daemon stop &&\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\t# We should have seen NFC event from OS.\n+\t\t# We should not have synthesized an NFD event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace &&\n+\t\tegrep -v \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace\n+\telse\n+\t\t# We should have seen NFD event from OS.\n+\t\t# We should have synthesized an NFC event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace &&\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace\n+\tfi &&\n+\n+\t# We assume UNICODE_NFD_PRESERVED.\n+\t# We should have seen explicit NFD from OS.\n+\t# We should have synthesized an NFC event.\n+\tegrep \"^event: nfd/d_${utf8_nfd}/?$\" ./unicode.trace &&\n+\tegrep \"^event: nfd/d_${utf8_nfc}/?$\" ./unicode.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450876","messageId":"2caf716e-116f-c10f-1af9-5612e2763a2a@github.com","threadId":"57414","inReplyTo":"eccdf8b7-93b5-f984-7754-622fa7c7771c@jeffhostetler.com","subject":"Re: [PATCH 00/23] Builtin FSMonitor Part 3","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-09T15:34:51Z","receivedAt":"2022-03-09T15:34:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/7/2022 4:23 PM, Jeff Hostetler wrote:\n> \n> \n> On 2/24/22 11:21 AM, Derrick Stolee wrote:\n>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:\n>>> Here is part 3 of my builtin FSMonitor series.\n> [...]\n>>> Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a\n>>> synthetic repo containing 1M files on a Macbook Pro. It shows the effects of\n>>> the untracked cache (uc) and FSMonitor (fsm) on git status.\n>>>\n>>> $ ./p7527-builtin-fsmonitor.sh\n>>> # passed all 67 test(s)\n>>> 1..67\n>>> Test                                                                 this tree\n>>> ---------------------------------------------------------------------------------------\n>>> 7527.4: [uc false][fsm false] status after checkout                  29.99(3.14+80.12)\n>>> 7527.6: [uc false][fsm false] status after big change                73.32(5.11+97.24)\n>>> 7527.8: [uc false][fsm false] status after add all                   47.80(5.12+90.47)\n>>> 7527.10: [uc false][fsm false] status after add dot                  49.22(5.16+92.05)\n>>> 7527.12: [uc false][fsm false] status after commit                   51.53(3.35+100.74)\n>>> 7527.14: [uc false][fsm false] status after reset hard               33.74(3.03+85.31)\n>>> 7527.16: [uc false][fsm false] status after create untracked files   41.71(3.24+89.75)\n>>> 7527.18: [uc false][fsm false] status after clean                    34.33(3.07+89.36)\n>>>\n>>> 7527.20: [uc false][fsm true] status after checkout                  29.23(1.94+10.84)\n>>> 7527.22: [uc false][fsm true] status after big change                64.23(4.66+24.86)\n>>> 7527.24: [uc false][fsm true] status after add all                   45.45(4.37+18.70)\n>>> 7527.26: [uc false][fsm true] status after add dot                   44.42(4.02+17.10)\n>>> 7527.28: [uc false][fsm true] status after commit                    30.52(1.95+10.91)\n>>> 7527.30: [uc false][fsm true] status after reset hard                28.70(2.70+13.89)\n>>> 7527.32: [uc false][fsm true] status after create untracked files    28.63(2.59+10.71)\n>>> 7527.34: [uc false][fsm true] status after clean                     28.97(2.59+10.78)\n>>>\n>>> 7527.36: [uc true][fsm false] status after checkout                  35.06(3.17+86.11)\n>>> 7527.38: [uc true][fsm false] status after big change                74.65(5.14+101.50)\n>>> 7527.40: [uc true][fsm false] status after add all                   49.96(5.22+90.96)\n>>> 7527.42: [uc true][fsm false] status after add dot                   49.77(5.24+91.72)\n>>> 7527.44: [uc true][fsm false] status after commit                    36.95(3.27+92.25)\n>>> 7527.46: [uc true][fsm false] status after reset hard                33.89(3.18+85.68)\n>>> 7527.48: [uc true][fsm false] status after create untracked files    41.44(3.40+92.99)\n>>> 7527.50: [uc true][fsm false] status after clean                     34.60(3.26+90.19)\n>>>\n>>> 7527.52: [uc true][fsm true] status after checkout                    0.58(0.45+0.10)\n>>> 7527.54: [uc true][fsm true] status after big change                 65.16(4.91+25.64)\n>>> 7527.56: [uc true][fsm true] status after add all                    45.43(4.45+18.92)\n>>> 7527.58: [uc true][fsm true] status after add dot                    15.56(2.57+6.32)\n>>> 7527.60: [uc true][fsm true] status after commit                      0.98(0.46+0.11)\n>>> 7527.62: [uc true][fsm true] status after reset hard                 30.30(2.96+14.49)\n>>> 7527.64: [uc true][fsm true] status after create untracked files      2.15(1.73+0.40)\n>>> 7527.66: [uc true][fsm true] status after clean                       1.68(1.56+0.32)\n>>\n>> The other stylistic thing is this performance test. It would be nice if\n>> these tests were grouped by the operation (like \"status after checkout\")\n>> so it is easier to compare the same operation across the matrix definitions.\n>>\n>> This would require reordering the test definition as well as allowing the\n>> different cases to simultaneously live in different repositories. The\n>> p2000-sparse-operations.sh has this kind of organization, but you'll need\n>> more involved test cases than \"run this command\".\n> \n> Yeah, it would be nice to turn this test inside-out so that\n> could group the outputs by test case rather than by (uc,fsm)\n> combination.  That would certainly make it easier to see how\n> the two terms affect things.\n> \n> The problem is I'd either need 4 parallel repos that I could\n> setup with each (uc,fsm) pair or I'd need to start/stop the\n> daemon and swap out the {.git/index, .git/config} between\n> each step.  The former is a problem for monorepos.  The latter\n> is doable, but I'm not sure it is worth the effort right now.\n\nYeah, that extra data is not something to take lightly.\n\nReworking this test is not critical right now, so I'll leave it\nfor another time, if we need it. But, perhaps we could set up\nthe test to create four _worktrees_ so at least the only\nduplication is the working tree and not the entire repository.\n\n(The working tree is still a huge amount of data in some cases,\nso this doesn't actually solve the problem, just reduces it.)\n\nThanks,\n-Stolee\n"},{"id":"450898","messageId":"af6c456a-d49f-e0f2-50ae-f44fdb6351e3@github.com","threadId":"57414","inReplyTo":"5a0c1b7a2873accc6db4b34493962378819eacd4.1646777728.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-09T18:40:32Z","receivedAt":"2022-03-09T18:40:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Create a set of prereqs to help understand how file names\n> are handled by the filesystem when they contain NFC and NFD\n> Unicode characters.\n\nPrereqs look good and are well documented.\n\n> +if test $unicode_debug = 1\n\nIs this $unicode_debug something I should know from a previous\npatch? or is it a leftover from local debugging?\n\nThanks,\n-Stolee\n"},{"id":"450900","messageId":"6555f3dc-e72c-2b0f-fe66-d81d48dff6f7@github.com","threadId":"57414","inReplyTo":"af6c456a-d49f-e0f2-50ae-f44fdb6351e3@github.com","subject":"Re: [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-09T18:42:32Z","receivedAt":"2022-03-09T18:42:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/9/2022 1:40 PM, Derrick Stolee wrote:\n> On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Create a set of prereqs to help understand how file names\n>> are handled by the filesystem when they contain NFC and NFD\n>> Unicode characters.\n> \n> Prereqs look good and are well documented.\n> \n>> +if test $unicode_debug = 1\n> \n> Is this $unicode_debug something I should know from a previous\n> patch? or is it a leftover from local debugging?\n\nI see that you set unicode_debug = 0 in a later patch, but I\nsuppose that we might want this output no matter what. Or, do\nwe think it will interrupt the output parsing of 'prove' and\nother tools?\n\nThanks,\n-Stolee\n"},{"id":"450902","messageId":"45b381da-af41-9e6b-f3b6-a83678a20319@github.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-09T18:48:07Z","receivedAt":"2022-03-09T18:48:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:\n> Here is V2 of part 3 of my builtin FSMonitor series.\n> \n> I think I have addressed all of the feedback from V1. This includes:\n\n> Range-diff vs v1:\n\n> -: ----------- > 22: 524d449ed64 fsmonitor: never set CE_FSMONITOR_VALID on submodules\n> -: ----------- > 24: 95b9d4210d2 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n> -: ----------- > 25: 5a0c1b7a287 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n> -: ----------- > 26: a45c1fd3000 t7527: test Unicode NFC/NFD handling on MacOS\n> -: ----------- > 27: e3e01677d93 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n\nI looked closely through the range-diff for the edits, then looked at\nthese new patches closely. Outside of one thought about some debug\noutput, I'm happy with this version.\n\nThanks,\n-Stolee\n"},{"id":"450984","messageId":"20220310053100.bkz5z243b3kwndos@tb-raspi4","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"Torsten B��gershausen","fromEmail":"tboegi@web.de","sentAt":"2022-03-10T05:31:00Z","receivedAt":"2022-03-10T05:31:15Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Tue, Mar 08, 2022 at 10:15:00PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> Here is V2 of part 3 of my builtin FSMonitor series.\n\n\nHej Jeff,\n\nFirst of all, the new test case passes here on one machine.\nI will do another round on another machine the next days.\n\nHaving said that, there are some small questions here and there,\nI'll send them as soon as I find the time for a more proper review.\n"},{"id":"451017","messageId":"0ba86794-e932-b443-2483-b93b171766f0@jeffhostetler.com","threadId":"57414","inReplyTo":"af6c456a-d49f-e0f2-50ae-f44fdb6351e3@github.com","subject":"Re: [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-10T14:23:51Z","receivedAt":"2022-03-10T14:31:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/9/22 1:40 PM, Derrick Stolee wrote:\n> On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Create a set of prereqs to help understand how file names\n>> are handled by the filesystem when they contain NFC and NFD\n>> Unicode characters.\n> \n> Prereqs look good and are well documented.\n> \n>> +if test $unicode_debug = 1\n> \n> Is this $unicode_debug something I should know from a previous\n> patch? or is it a leftover from local debugging?\n\n\nI added that and all of the print statements to help\ndescribe the characteristics of the (OS, FS) pair,\nfor example what happens on (MacOS, FAT32) and is that\nany different from (MacOS, APFS). I found this very\nuseful in trying to decipher the docs.\n\nHowever, it is kinda noisy and appears directly on the\nconsole. Since most people don't need to see it (unless\nthey are working on Unicode/UTF8 issues), I decided to\nturn it off for now.\n\nI'm not sure if we have a way to handle such output or\nnot. I thought about maybe hooking it into the -d or -x\noptions, but I'm not sure if that helps or not. So I\njust turned it off.\n\nAlso, by not always testing the prereqs just to print\nthe result here, we avoid actually doing the lazy evals\nuntil a real test wants to use one of them.\n\n\nI'll add a comment in the script documenting it.\n\nThanks\nJeff\n"},{"id":"451018","messageId":"1b35ccf2-fe23-145d-53b9-4a0cbb5bba8b@jeffhostetler.com","threadId":"57414","inReplyTo":"6555f3dc-e72c-2b0f-fe66-d81d48dff6f7@github.com","subject":"Re: [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-10T14:28:45Z","receivedAt":"2022-03-10T14:31:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/9/22 1:42 PM, Derrick Stolee wrote:\n> On 3/9/2022 1:40 PM, Derrick Stolee wrote:\n>> On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:\n>>> From: Jeff Hostetler \n>>>\n>>> Create a set of prereqs to help understand how file names\n>>> are handled by the filesystem when they contain NFC and NFD\n>>> Unicode characters.\n>>\n>> Prereqs look good and are well documented.\n>>\n>>> +if test $unicode_debug = 1\n>>\n>> Is this $unicode_debug something I should know from a previous\n>> patch? or is it a leftover from local debugging?\n> \n> I see that you set unicode_debug = 0 in a later patch, but I\n> suppose that we might want this output no matter what. Or, do\n> we think it will interrupt the output parsing of 'prove' and\n> other tools?\n\nI was afraid that it might interrupt tools like prove, but\nI just tried it and it didn't. But yeah it would be safer\nto turn it off until someone actually wants to do some debugging\nin this area.\n\nJeff\n\n\n"},{"id":"451105","messageId":"220311.86k0d1i7xr.gmgdl@evledraar.gmail.com","threadId":"57414","inReplyTo":"8c4f90ae4fd5d9fbac9acb9307ee82ceffc7df08.1646777727.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-11T01:31:00Z","receivedAt":"2022-03-11T01:39:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 08 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n> [...]\n> +\tprepare_repo_settings(the_repository);\n> +\tfsm_settings__set_ipc(the_repository);\n> +\n> +\tif (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {\n> +\t\tconst char *msg = fsm_settings__get_reason_msg(the_repository);\n> +\n> +\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n> +\t}\n> +\n> \tif (!strcmp(subcmd, \"start\"))\n> \t\treturn !!try_to_start_background_daemon();\n> \n> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> index d335f1ac72a..8f460e7195f 100644\n> --- a/builtin/update-index.c\n> +++ b/builtin/update-index.c\n> @@ -1237,6 +1237,13 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n> \n> \tif (fsmonitor > 0) {\n> \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n> +\n> +\t\tif (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {\n> +\t\t\tconst char *msg = fsm_settings__get_reason_msg(r);\n> +\n> +\t\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n> +\t\t}\n> +\n> \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n> \t\t\tadvise(_(\"core.fsmonitor is unset; \"\n> \t\t\t\t \"set it if you really want to \"\n\nCan w assert somewhere earlier that ->mode can't be\nFSMONITOR_MODE_INCOMPATIBLE at the same time that ->reason ==\nFSMONITOR_REASON_OK, should that ever happen?\n\nThen we can get rid of the \"???\" case here.\n\nThe \"%s '%s'\" here should really be marked for translation, but just\n\"some reason '$path'\" is a pretty confusing message. This will emit\ne.g.:\n\n \"bare repos are incompatible with fsmonitor '/some/path/to/repo'\"\n\nSince we always hand these to error maybe have the helper do e.g.:\n\n error(_(\"bare repository '%s' is incompatible with fsmonitor\"), path);\n\nI find the second-guessing in fsmonitor-settings.c really hard to\nfollow, i.e. how seemingly every function has some \"not loaded yet? load\nit\" instead of a more typical \"init it\", \"use it\", \"free it\"\npattern. Including stuff like this:\n\t\n\tenum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n\t{\n\t if (!r)\n\t r = the_repository;\n\nBut anyway, seeing as we do try really hard to load the_repository (or a\nrepository) can't we use the_repository->gitdir etc. here instead of\nxgetcwd(), or the_repository->worktree when non-bare?\n"},{"id":"451174","messageId":"73f6702b-95cd-794e-6ca7-5c995294935a@jeffhostetler.com","threadId":"57414","inReplyTo":"220311.86k0d1i7xr.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-11T22:25:49Z","receivedAt":"2022-03-11T22:51:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/10/22 8:31 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 08 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>> [...]\n>> +\tprepare_repo_settings(the_repository);\n>> +\tfsm_settings__set_ipc(the_repository);\n>> +\n>> +\tif (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {\n>> +\t\tconst char *msg = fsm_settings__get_reason_msg(the_repository);\n>> +\n>> +\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n>> +\t}\n>> +\n>> \tif (!strcmp(subcmd, \"start\"))\n>> \t\treturn !!try_to_start_background_daemon();\n>> \n>> diff --git a/builtin/update-index.c b/builtin/update-index.c\n>> index d335f1ac72a..8f460e7195f 100644\n>> --- a/builtin/update-index.c\n>> +++ b/builtin/update-index.c\n>> @@ -1237,6 +1237,13 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n>> \n>> \tif (fsmonitor > 0) {\n>> \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n>> +\n>> +\t\tif (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {\n>> +\t\t\tconst char *msg = fsm_settings__get_reason_msg(r);\n>> +\n>> +\t\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n>> +\t\t}\n>> +\n>> \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n>> \t\t\tadvise(_(\"core.fsmonitor is unset; \"\n>> \t\t\t\t \"set it if you really want to \"\n> \n> Can w assert somewhere earlier that ->mode can't be\n> FSMONITOR_MODE_INCOMPATIBLE at the same time that ->reason ==\n> FSMONITOR_REASON_OK, should that ever happen?\n> \n> Then we can get rid of the \"???\" case here.\n> \n\nYeah, it would be nice to assert() the pair and simplify things. I'll\nmake a note to look at that.\n\n\n> The \"%s '%s'\" here should really be marked for translation, but just\n> \"some reason '$path'\" is a pretty confusing message. This will emit\n> e.g.:\n\nI already have translations in the code that looks up the message,\nso doing it here for a pair of %s's felt wrong.\n\n> \n> \"bare repos are incompatible with fsmonitor '/some/path/to/repo'\"\n> \n> Since we always hand these to error maybe have the helper do e.g.:\n> \n> error(_(\"bare repository '%s' is incompatible with fsmonitor\"), path);\n> \n\nI'm wondering now if we should just drop the path from the message\nand use the error message from __get_reason_msg() as is. (It was\nuseful during debugging, but I could see it going away.)\n\n\n> I find the second-guessing in fsmonitor-settings.c really hard to\n> follow, i.e. how seemingly every function has some \"not loaded yet? load\n> it\" instead of a more typical \"init it\", \"use it\", \"free it\"\n> pattern. Including stuff like this:\n> \t\n> \tenum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n> \t{\n> \t if (!r)\n> \t r = the_repository;\n> \n> But anyway, seeing as we do try really hard to load the_repository (or a\n> repository) can't we use the_repository->gitdir etc. here instead of\n> xgetcwd(), or the_repository->worktree when non-bare?\n> \n\nI'll take a look and see if I can simplify it.\n\nThanks\nJeff\n"},{"id":"451239","messageId":"20220313104230.ctwbskywcq5jxv36@tb-raspi4","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2022-03-13T10:42:30Z","receivedAt":"2022-03-13T10:42:40Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"Hej Jeff,\n\nI tried your patch on both a newer Mac and an older machine (with HFS+)\nThe older machine doesn't have\nkFSEventStreamEventFlagItemCloned\nAs it is an enum, and not a #define, I ended up here:\n\n diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\n index 3332d3b779..fa172a05c4 100644\n --- a/compat/fsmonitor/fsm-listen-darwin.c\n +++ b/compat/fsmonitor/fsm-listen-darwin.c\n @@ -169,8 +169,6 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n strbuf_addstr(&msg, \"ItemXattrMod|\");\n if (flag & kFSEventStreamEventFlagOwnEvent)\n strbuf_addstr(&msg, \"OwnEvent|\");\n - if (flag & kFSEventStreamEventFlagItemCloned)\n - strbuf_addstr(&msg, \"ItemCloned|\");\n\n trace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n path, flag, msg.buf);\n @@ -221,8 +219,7 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n kFSEventStreamEventFlagItemModified |\n kFSEventStreamEventFlagItemRemoved |\n kFSEventStreamEventFlagItemRenamed |\n - kFSEventStreamEventFlagItemXattrMod |\n - kFSEventStreamEventFlagItemCloned;\n + kFSEventStreamEventFlagItemXattrMod ;\n\nOne other thing, I just add it here:\nThere is a new file, t/lib-unicode-nfc-nfd.sh, which helps us with this code:\ntest_lazy_prereq UNICODE_NFC_PRESERVED\n\nThe existing code uses a construct called\nUTF8_NFD_TO_NFC\n\nAnd now I have 2 questions:\n- Do we need the UNICODE_NFC_PRESERVED at all ?\n- And should the UTF8_NFD_TO_NFC better be called UTF8_NFC_TO_NFD,\n because that is what it checks.\n- Do we need the UNICODE_NFD_PRESERVED at all ?\n\nAs there are no non-UNICODE_NFD_PRESERVED filesystems, as far as I know.\nAnd the current code does no tests, just debug prints.\nI dunno.\n\nOn Tue, Mar 08, 2022 at 10:15:00PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> Here is V2 of part 3 of my builtin FSMonitor series.\n\n> [] I updated the daemon on MacOS to report both the NFC and NFD spellings of\n> a pathname when appropriate. This is a little more general than the\n> \"core.precomposeUnicode\" setting, since the daemon does not know how the\n> client has (or will have) it set when they make a query.\n>\n> [] I replaced my Unicode NFC/NFD test for MacOS to focus exclusively on\n> Unicode composition/decomposition sensitivity and to not confuse that with\n> case sensitivity.\n\nThat is a good thing.\n\n[snip]\n"},{"id":"451773","messageId":"2bb125d3-cef7-9d47-efa8-61a1aaba5316@jeffhostetler.com","threadId":"57414","inReplyTo":"20220313104230.ctwbskywcq5jxv36@tb-raspi4","subject":"Re: [PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-21T22:06:01Z","receivedAt":"2022-03-21T22:49:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/13/22 6:42 AM, Torsten Bögershausen wrote:\n> Hej Jeff,\n> \n> I tried your patch on both a newer Mac and an older machine (with HFS+)\n> The older machine doesn't have\n> kFSEventStreamEventFlagItemCloned\n> As it is an enum, and not a #define, I ended up here:\n> \n> diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\n> index 3332d3b779..fa172a05c4 100644\n> --- a/compat/fsmonitor/fsm-listen-darwin.c\n> +++ b/compat/fsmonitor/fsm-listen-darwin.c\n> @@ -169,8 +169,6 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n> strbuf_addstr(&msg, \"ItemXattrMod|\");\n> if (flag & kFSEventStreamEventFlagOwnEvent)\n> strbuf_addstr(&msg, \"OwnEvent|\");\n> - if (flag & kFSEventStreamEventFlagItemCloned)\n> - strbuf_addstr(&msg, \"ItemCloned|\");\n> \n> trace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n> path, flag, msg.buf);\n> @@ -221,8 +219,7 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n> kFSEventStreamEventFlagItemModified |\n> kFSEventStreamEventFlagItemRemoved |\n> kFSEventStreamEventFlagItemRenamed |\n> - kFSEventStreamEventFlagItemXattrMod |\n> - kFSEventStreamEventFlagItemCloned;\n> + kFSEventStreamEventFlagItemXattrMod ;\n\nIt looks like the ...Cloned bit was added to the SDK in 10.13 [1].\nAll the other bits were defined sometime between 10.5 and 10.10.\n\nI'll add something in V7 to guard that bit. I think 10.10 is old enough\nthat we don't need to special case those bits too.\n\nThanks,\nJeff\n\n[1] /Applications/Xcode.app/Contents/Developer/Platforms/ \\\n MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/ \\\n Library/Frameworks/CoreServices.framework/Frameworks/ \\\n FSEvents.framework/Versions/Current/Headers/FSEvents.h\n\n"},{"id":"451791","messageId":"1a64c0f7-ef54-9d61-f7fd-bd192c408ca0@jeffhostetler.com","threadId":"57414","inReplyTo":"20220313104230.ctwbskywcq5jxv36@tb-raspi4","subject":"Re: [PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-21T22:59:09Z","receivedAt":"2022-03-21T23:09:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/13/22 6:42 AM, Torsten Bögershausen wrote:\n> Hej Jeff,\n> \n[...]\n> \n> One other thing, I just add it here:\n> There is a new file, t/lib-unicode-nfc-nfd.sh, which helps us with this code:\n> test_lazy_prereq UNICODE_NFC_PRESERVED\n> \n> The existing code uses a construct called\n> UTF8_NFD_TO_NFC\n> \n> And now I have 2 questions:\n> - Do we need the UNICODE_NFC_PRESERVED at all ?\n> - And should the UTF8_NFD_TO_NFC better be called UTF8_NFC_TO_NFD,\n> because that is what it checks.\n> - Do we need the UNICODE_NFD_PRESERVED at all ?\n> \n> As there are no non-UNICODE_NFD_PRESERVED filesystems, as far as I know.\n> And the current code does no tests, just debug prints.\n> I dunno.\n\nI created t/lib-unicode-nfc-nfd.sh to help me understand\nthe issues. I found the existing UTF8_NFD_TO_NFC prereq\nconfusing (and yes it seemed poorly named).\n\nThe existing prereq returned the same answer on APFS, HFS+,\nand FAT32 (a thumbdrive). I know they behave differently\nand I found it odd that the prereq did not make any distinction.\n\nI was hesitant to rename the existing prereq because it is\ncurrently used by 5+ different tests and I didn't want to\nexpand the scope of my two already very large series.\n\nAlso, the existing prereq feels a little sloppy. It creates\na file in NFC and does a lstat in the NFD spelling. There\nare several ways that the OS and/or FS can lie to us. For\nexample, the prereq is satisfied on a FAT32 thumbdrive and\nwe know FAT32 doesn't do NFC-->NFD conversions. So I'd like\nto move away from that prereq definition at some point.\n\n\nMy new prereqs try to:\n\n(1) independently confirm whether there is aliasing happening\n at all (whether at the FS or OS layer).\n\n(2) determine if the actual on-disk spelling is altered by the\n FS (in both NFC and NFD cases).\n\n\nWe know that HFS+ does not preserve NFC spellings, but APFS\ndoes. (FAT32 also preserves NFC spelling under MacOS.)\nSo the UNICODE_NFC_PRESERVED lets me distinguish between HFS+\nand APFS/FAT32.\n\nI have not heard of any filesystems that convert NFD to NFC,\nso technically we don't need the UNICODE_NFD_PRESERVED prereq,\nbut then again until I tested that, it was unclear how MacOS\ndid the aliasing on APFS (and FAT32). On the basis of that\ntesting, we can say that MacOS -- at the MacOS layer -- is\nresponsible for the aliasing and that both NFC and NFD spellings\nare preserved on APFS and FAT32.\n\nSo I'd rather keep the 3 prereqs that I have now.\n\nThe ones marked _DOUBLE_ are currently extra. I have them to\nhelp study how code points with multiple combining characters\nare handled. I have prereqs for the basic double chars, but\nthere are several opportunities for weird edge cases (non-\ncanonical ordering and other collisions) that I don't want to\nget stuck on right now. So we might make more use of them in\nthe future.\n\n\nThat's too long of an answer, but hopefully that explains\nsome of my paranoia. :-)\n\nJeff\n\n> \n> On Tue, Mar 08, 2022 at 10:15:00PM +0000, Jeff Hostetler via GitGitGadget wrote:\n>> Here is V2 of part 3 of my builtin FSMonitor series.\n[...]\n"},{"id":"451821","messageId":"060401d83d79$f2540780$d6fc1680$@nexbridge.com","threadId":"57414","inReplyTo":"2bb125d3-cef7-9d47-efa8-61a1aaba5316@jeffhostetler.com","subject":"RE: [PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2022-03-21T23:18:11Z","receivedAt":"2022-03-21T23:24:20Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On March 21, 2022 6:06 PM, Jeff Hostetler wrote:\n>On 3/13/22 6:42 AM, Torsten Bögershausen wrote:\n>> Hej Jeff,\n>>\n>> I tried your patch on both a newer Mac and an older machine (with\n>> HFS+) The older machine doesn't have kFSEventStreamEventFlagItemCloned\n>> As it is an enum, and not a #define, I ended up here:\n>>\n>> diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-\n>listen-darwin.c\n>> index 3332d3b779..fa172a05c4 100644\n>> --- a/compat/fsmonitor/fsm-listen-darwin.c\n>> +++ b/compat/fsmonitor/fsm-listen-darwin.c\n>> @@ -169,8 +169,6 @@ static void log_flags_set(const char *path, const\n>FSEventStreamEventFlags flag)\n>> strbuf_addstr(&msg, \"ItemXattrMod|\");\n>> if (flag & kFSEventStreamEventFlagOwnEvent)\n>> strbuf_addstr(&msg, \"OwnEvent|\");\n>> - if (flag & kFSEventStreamEventFlagItemCloned)\n>> - strbuf_addstr(&msg, \"ItemCloned|\");\n>>\n>> trace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n>> path, flag, msg.buf);\n>> @@ -221,8 +219,7 @@ static int ef_ignore_xattr(const\n>FSEventStreamEventFlags ef)\n>> kFSEventStreamEventFlagItemModified |\n>> kFSEventStreamEventFlagItemRemoved |\n>> kFSEventStreamEventFlagItemRenamed |\n>> - kFSEventStreamEventFlagItemXattrMod |\n>> - kFSEventStreamEventFlagItemCloned;\n>> + kFSEventStreamEventFlagItemXattrMod ;\n>\n>It looks like the ...Cloned bit was added to the SDK in 10.13 [1].\n>All the other bits were defined sometime between 10.5 and 10.10.\n>\n>I'll add something in V7 to guard that bit. I think 10.10 is old enough that we don't\n>need to special case those bits too.\n\nI realize it is a bit late in the game, but would you consider a pre-hook and post-hook that automatically run with fsmonitor kicks off/terminates. I am thinking about use cases where this is integrated into more complex processes and it would be nice to have notifications of what fsmonitor is doing and when.\n\nThanks,\nRandall\n\n"},{"id":"451844","messageId":"e8ab7689-f6eb-9fda-e7aa-87fc2c4ee601@jeffhostetler.com","threadId":"57414","inReplyTo":"060401d83d79$f2540780$d6fc1680$@nexbridge.com","subject":"Re: [PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-22T14:10:49Z","receivedAt":"2022-03-22T14:10:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/21/22 7:18 PM, rsbecker@nexbridge.com wrote:\n> On March 21, 2022 6:06 PM, Jeff Hostetler wrote:\n>> On 3/13/22 6:42 AM, Torsten Bögershausen wrote:\n>>> Hej Jeff,\n>>>\n[...]\n>>\n>> It looks like the ...Cloned bit was added to the SDK in 10.13 [1].\n>> All the other bits were defined sometime between 10.5 and 10.10.\n>>\n>> I'll add something in V7 to guard that bit. I think 10.10 is old enough that we don't\n>> need to special case those bits too.\n> \n> I realize it is a bit late in the game, but would you consider a pre-hook and post-hook that automatically run with fsmonitor kicks off/terminates. I am thinking about use cases where this is integrated into more complex processes and it would be nice to have notifications of what fsmonitor is doing and when.\n> \n> Thanks,\n> Randall\n> \n\nI hadn't really considered having a pre/post hook for the daemon.\nI'm not opposed to it; I just hadn't thought about it.\n\nBy this I assume you mean something inside the fsmonitor--daemon\nprocess that invokes the hooks when it is starting/stopping.\nAs opposed to something in a client command (like status) before\nit implicitly started a daemon process. The latter method would\nnot give you post-hook events because the daemon usually outlives\nthe client command.\n\nPerhaps you could elaborate on what you would use these hooks for\nor how they would be helpful. It would be easy to add pre/post\nhooks in the main thread of the daemon. However, I worry about\nthe prehook slowing the startup of the daemon -- since the client\nstatus command might be waiting for it to become ready. I also\nhave a \"health\" thread in part3 that would be a candidate for\npre/post and any other periodic hooks that might be useful.\nBut again, before I suggest a design for this, it would be good\nto know what kind of things you would want to do with them.\n\nJeff\n"},{"id":"451845","messageId":"063f01d83df8$a3102e40$e9308ac0$@nexbridge.com","threadId":"57414","inReplyTo":"e8ab7689-f6eb-9fda-e7aa-87fc2c4ee601@jeffhostetler.com","subject":"RE: [PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2022-03-22T14:25:04Z","receivedAt":"2022-03-22T14:26:28Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On March 22, 2022 10:11 AM, Jeff Hostetler wrote:\n>On 3/21/22 7:18 PM, rsbecker@nexbridge.com wrote:\n>> On March 21, 2022 6:06 PM, Jeff Hostetler wrote:\n>>> On 3/13/22 6:42 AM, Torsten Bögershausen wrote:\n>>>> Hej Jeff,\n>>>>\n>[...]\n>>>\n>>> It looks like the ...Cloned bit was added to the SDK in 10.13 [1].\n>>> All the other bits were defined sometime between 10.5 and 10.10.\n>>>\n>>> I'll add something in V7 to guard that bit. I think 10.10 is old\n>>> enough that we don't need to special case those bits too.\n>>\n>> I realize it is a bit late in the game, but would you consider a pre-hook and post-\n>hook that automatically run with fsmonitor kicks off/terminates. I am thinking\n>about use cases where this is integrated into more complex processes and it\n>would be nice to have notifications of what fsmonitor is doing and when.\n>>\n>> Thanks,\n>> Randall\n>>\n>\n>I hadn't really considered having a pre/post hook for the daemon.\n>I'm not opposed to it; I just hadn't thought about it.\n>\n>By this I assume you mean something inside the fsmonitor--daemon process that\n>invokes the hooks when it is starting/stopping.\n>As opposed to something in a client command (like status) before it implicitly\n>started a daemon process. The latter method would not give you post-hook\n>events because the daemon usually outlives the client command.\n>\n>Perhaps you could elaborate on what you would use these hooks for or how they\n>would be helpful. It would be easy to add pre/post hooks in the main thread of\n>the daemon. However, I worry about the prehook slowing the startup of the\n>daemon -- since the client status command might be waiting for it to become\n>ready. I also have a \"health\" thread in part3 that would be a candidate for\n>pre/post and any other periodic hooks that might be useful.\n>But again, before I suggest a design for this, it would be good to know what kind of\n>things you would want to do with them.\n\nSome examples of what I have in mind. There are more, but this covers what I have in mind urgently:\n\n1. Setting up a lock file (semaphore) just before fsmonitor runs that will cause any scripts that might change the state of the repository on the fly to suspend until fsmonitor is done.\n2. Ensuring that in-flight scripts that do stuff are finished or not leaving the repo in a transitional state before fsmonitor runs - holding fsmonitor until the pre-hook finishes.\n3. Notifying syslog or some other paging system if something has gone horribly wrong - as in fsmonitor found something bad in the index.\n4. Clearing any semaphores created earlier (example 1).\n\nRegards,\nRandall\n\n"},{"id":"451847","messageId":"29f4b39b-1ad0-41fa-3470-c7afbeb613cf@jeffhostetler.com","threadId":"57414","inReplyTo":"063f01d83df8$a3102e40$e9308ac0$@nexbridge.com","subject":"Re: [PATCH v2 00/27] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-22T15:01:08Z","receivedAt":"2022-03-22T15:01:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/22/22 10:25 AM, rsbecker@nexbridge.com wrote:\n> On March 22, 2022 10:11 AM, Jeff Hostetler wrote:\n>> On 3/21/22 7:18 PM, rsbecker@nexbridge.com wrote:\n>>> On March 21, 2022 6:06 PM, Jeff Hostetler wrote:\n>>>> On 3/13/22 6:42 AM, Torsten Bögershausen wrote:\n>>>>> Hej Jeff,\n>>>>>\n>> [...]\n>>>>\n>>>> It looks like the ...Cloned bit was added to the SDK in 10.13 [1].\n>>>> All the other bits were defined sometime between 10.5 and 10.10.\n>>>>\n>>>> I'll add something in V7 to guard that bit. I think 10.10 is old\n>>>> enough that we don't need to special case those bits too.\n>>>\n>>> I realize it is a bit late in the game, but would you consider a pre-hook and post-\n>> hook that automatically run with fsmonitor kicks off/terminates. I am thinking\n>> about use cases where this is integrated into more complex processes and it\n>> would be nice to have notifications of what fsmonitor is doing and when.\n>>>\n>>> Thanks,\n>>> Randall\n>>>\n>>\n>> I hadn't really considered having a pre/post hook for the daemon.\n>> I'm not opposed to it; I just hadn't thought about it.\n>>\n>> By this I assume you mean something inside the fsmonitor--daemon process that\n>> invokes the hooks when it is starting/stopping.\n>> As opposed to something in a client command (like status) before it implicitly\n>> started a daemon process. The latter method would not give you post-hook\n>> events because the daemon usually outlives the client command.\n>>\n>> Perhaps you could elaborate on what you would use these hooks for or how they\n>> would be helpful. It would be easy to add pre/post hooks in the main thread of\n>> the daemon. However, I worry about the prehook slowing the startup of the\n>> daemon -- since the client status command might be waiting for it to become\n>> ready. I also have a \"health\" thread in part3 that would be a candidate for\n>> pre/post and any other periodic hooks that might be useful.\n>> But again, before I suggest a design for this, it would be good to know what kind of\n>> things you would want to do with them.\n> \n> Some examples of what I have in mind. There are more, but this covers what I have in mind urgently:\n> \n> 1. Setting up a lock file (semaphore) just before fsmonitor runs that will cause any scripts that might change the state of the repository on the fly to suspend until fsmonitor is done.\n\nThe builtin fsmonitor--daemon is a long-running process. It is\neither explicitly started by the user or implicitly started by\nclients commands, like \"git status\", lazily after it is enabled\nin the config. It runs until explicitly stopped (or the workdir\nis deleted).\n\nIt is designed to be running and watch the filesystem for changes.\n\nLater client commands can ask it for what has changed on disk\nsince the previous request (checkpoint). And the only way to\ncapture that info is to watch the file system as things happen.\n(Unless we have a really deep journal, but that often requires\nadmin access, so we don't use that.)\n\nThe fsmonitor daemon is unlike other subordinate commands in Git.\nFor example, \"git fetch\" might synchronously invoke \"git index-pack\"\nand communicate over the child's stdin/stdout. And that child is\nbound to a single parent process.\n\nWhen the daemon starts, it disassociates from the console and\nopens a socket or named pipe and listens for requests (REST-like) \ncommands. It is designed to respond to multiple clients concurrently\nand over a long time period -- like a daemon or service process.\n\n> 2. Ensuring that in-flight scripts that do stuff are finished or not leaving the repo in a transitional state before fsmonitor runs - holding fsmonitor until the pre-hook finishes.\n> 3. Notifying syslog or some other paging system if something has gone horribly wrong - as in fsmonitor found something bad in the index.\n\nfsmonitor doesn't read the index. it's only watching and summarizing\nfile system events. And can enumerate the list of changed paths between\ntwo checkpoints in response to a client request.\n\n\n> 4. Clearing any semaphores created earlier (example 1).\n\nThe daemon isn't designed to ever \"be done\" and hence there are no\non-disk lock files/semaphores. There might be some opportunities\nfor startup/shutdown to do some cleanup (temp files and the like),\nbut I think it is premature to talk about that right now.\n\nHope this helps,\nJeff\n\n\n> \n> Regards,\n> Randall\n> \n"},{"id":"451889","messageId":"779a15b38e8d48f13e7ba9bc460faa8cf6ea26f4.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 01/27] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:34Z","receivedAt":"2022-03-22T18:23:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach FSMonitor daemon on Windows to recognize shortname paths as\naliases of normal longname paths. FSMonitor clients, such as `git\nstatus`, should receive the longname spelling of changed files (when\npossible).\n\nSometimes we receive FS events using the shortname, such as when a CMD\nshell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\narrives using whatever combination of long and shortnames were used by\nthe other process. (Shortnames do seem to be case normalized,\nhowever.)\n\nUse Windows GetLongPathNameW() to try to map the pathname spelling in\nthe notification event into the normalized longname spelling. (This\ncan fail if the file/directory is deleted, moved, or renamed, because\nwe are asking the FS for the mapping in response to the event and\nafter it has already happened, but we try.)\n\nSpecial case the shortname spelling of \".git\" to avoid under-reporting\nthese events.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++-----\n t/t7527-builtin-fsmonitor.sh | 65 +++++\n 2 files changed, 374 insertions(+), 54 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 5b928ab66e5..3f1b68267bd 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -25,6 +25,9 @@ struct one_watch\n \tDWORD count;\n \n \tstruct strbuf path;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n+\tDWORD wpath_longname_len;\n+\n \tHANDLE hDir;\n \tHANDLE hEvent;\n \tOVERLAPPED overlapped;\n@@ -34,6 +37,21 @@ struct one_watch\n \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n \t */\n \tBOOL is_active;\n+\n+\t/*\n+\t * Are shortnames enabled on the containing drive? This is\n+\t * always true for \"C:/\" drives and usually never true for\n+\t * other drives.\n+\t *\n+\t * We only set this for the worktree because we only need to\n+\t * convert shortname paths to longname paths for items we send\n+\t * to clients. (We don't care about shortname expansion for\n+\t * paths inside a GITDIR because we never send them to\n+\t * clients.)\n+\t */\n+\tBOOL has_shortnames;\n+\tBOOL has_tilda;\n+\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n struct fsmonitor_daemon_backend_data\n@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n };\n \n /*\n- * Convert the WCHAR path from the notification into UTF8 and\n- * then normalize it.\n+ * Convert the WCHAR path from the event into UTF8 and normalize it.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n */\n-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n \t\t\t\t struct strbuf *normalized_path)\n {\n \tint reserve;\n \tint len = 0;\n \n \tstrbuf_reset(normalized_path);\n-\tif (!info->FileNameLength)\n+\tif (!wpath_len)\n \t\tgoto normalize;\n \n \t/*\n@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t * sequence of 2 UTF8 characters. That should let us\n \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n \t */\n-\treserve = info->FileNameLength + 1;\n+\treserve = 2 * wpath_len + 1;\n \tstrbuf_grow(normalized_path, reserve);\n \n \tfor (;;) {\n-\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n-\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n+\t\t\t\t\t wpath, wpath_len,\n \t\t\t\t\t normalized_path->buf,\n \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n \t\t\t\t\t NULL, NULL);\n@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n \t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n-\t\t\t GetLastError(),\n-\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n-\t\t\t info->FileName);\n+\t\t\t GetLastError(), (int)wpath_len, wpath);\n \t\t\treturn -1;\n \t\t}\n \n@@ -98,6 +115,152 @@ normalize:\n \treturn strbuf_normalize_path(normalized_path);\n }\n \n+/*\n+ * See if the worktree root directory has shortnames enabled.\n+ * This will help us decide if we need to do an expensive shortname\n+ * to longname conversion on every notification event.\n+ *\n+ * We do not want to create a file to test this, so we assume that the\n+ * root directory contains a \".git\" file or directory. (Our caller\n+ * only calls us for the worktree root, so this should be fine.)\n+ *\n+ * Remember the spelling of the shortname for \".git\" if it exists.\n+ */\n+static void check_for_shortnames(struct one_watch *watch)\n+{\n+\twchar_t buf_in[MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\twchar_t *last_slash = NULL;\n+\twchar_t *last_bslash = NULL;\n+\twchar_t *last;\n+\n+\t/* build L\"/.git\" */\n+\twcscpy(buf_in, watch->wpath_longname);\n+\twcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n+\n+\tif (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n+\t\treturn;\n+\n+\tlast_slash = wcsrchr(buf_out, L'/');\n+\tlast_bslash = wcsrchr(buf_out, L'\\\\');\n+\tif (last_slash > last_bslash)\n+\t\tlast = last_slash + 1;\n+\telse if (last_bslash)\n+\t\tlast = last_bslash + 1;\n+\telse\n+\t\tlast = buf_out;\n+\n+\tif (!wcscmp(last, L\".git\"))\n+\t\treturn;\n+\n+\twatch->has_shortnames = 1;\n+\twcsncpy(watch->dotgit_shortname, last,\n+\t\tARRAY_SIZE(watch->dotgit_shortname));\n+\n+\t/*\n+\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n+\t * we should be able to avoid shortname to longname mapping on\n+\t * every notification event if the source string does not\n+\t * contain a \"~\".\n+\t *\n+\t * However, the documentation for GetLongPathNameW() says\n+\t * that there are filesystems that don't follow that pattern\n+\t * and warns against this optimization.\n+\t *\n+\t * Lets test this.\n+\t */\n+\tif (wcschr(watch->dotgit_shortname, L'~'))\n+\t\twatch->has_tilda = 1;\n+}\n+\n+enum get_relative_result {\n+\tGRR_NO_CONVERSION_NEEDED,\n+\tGRR_HAVE_CONVERSION,\n+\tGRR_SHUTDOWN,\n+};\n+\n+/*\n+ * Info notification paths are relative to the root of the watch.\n+ * If our CWD is still at the root, then we can use relative paths\n+ * to convert from shortnames to longnames. If our process has a\n+ * different CWD, then we need to construct an absolute path, do\n+ * the conversion, and then return the root-relative portion.\n+ *\n+ * We use the longname form of the root as our basis and assume that\n+ * it already has a trailing slash.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n+ */\n+static enum get_relative_result get_relative_longname(\n+\tstruct one_watch *watch,\n+\tconst wchar_t *wpath, DWORD wpath_len,\n+\twchar_t *wpath_longname)\n+{\n+\twchar_t buf_in[2 * MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\tDWORD root_len;\n+\n+\t/* Build L\"/\" */\n+\troot_len = watch->wpath_longname_len;\n+\twcsncpy(buf_in, watch->wpath_longname, root_len);\n+\twcsncpy(buf_in + root_len, wpath, wpath_len);\n+\tbuf_in[root_len + wpath_len] = 0;\n+\n+\t/*\n+\t * We don't actually know if the source pathname is a\n+\t * shortname or a longname. This routine allows either to be\n+\t * given as input.\n+\t */\n+\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n+\t\t/*\n+\t\t * The shortname to longname conversion can fail for\n+\t\t * various reasons, for example if the file has been\n+\t\t * deleted. (That is, if we just received a\n+\t\t * delete-file notification event and the file is\n+\t\t * already gone, we can't ask the file system to\n+\t\t * lookup the longname for it. Likewise, for moves\n+\t\t * and renames where we are given the old name.)\n+\t\t *\n+\t\t * Since deleting or moving a file or directory by its\n+\t\t * shortname is rather obscure, I'm going ignore the\n+\t\t * failure and ask the caller to report the original\n+\t\t * relative path. This seems kinder than failing here\n+\t\t * and forcing a resync. Besides, forcing a resync on\n+\t\t * every file/directory delete would effectively\n+\t\t * cripple monitoring.\n+\t\t *\n+\t\t * We might revisit this in the future.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (!wcscmp(buf_in, buf_out)) {\n+\t\t/*\n+\t\t * The path does not have a shortname alias.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (wcsncmp(buf_in, buf_out, root_len)) {\n+\t\t/*\n+\t\t * The spelling of the root directory portion of the computed\n+\t\t * longname has changed. This should not happen. Basically,\n+\t\t * it means that we don't know where (without recomputing the\n+\t\t * longname of just the root directory) to split out the\n+\t\t * relative path. Since this should not happen, I'm just\n+\t\t * going to let this fail and force a shutdown (because all\n+\t\t * subsequent events are probably going to see the same\n+\t\t * mismatch).\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\t/* Return the worktree root-relative portion of the longname. */\n+\n+\twcscpy(wpath_longname, buf_out + root_len);\n+\treturn GRR_HAVE_CONVERSION;\n+}\n+\n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n@@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n-\twchar_t wpath[MAX_PATH];\n+\tDWORD len_longname;\n+\twchar_t wpath[MAX_PATH + 1];\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \tif (xutftowcs_path(wpath, path) < 0) {\n \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n@@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \t\treturn NULL;\n \t}\n \n+\tif (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n+\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n+\t\t GetLastError(), path);\n+\t\tCloseHandle(hDir);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen_longname = wcslen(wpath_longname);\n+\tif (wpath_longname[len_longname - 1] != L'/' &&\n+\t wpath_longname[len_longname - 1] != L'\\\\') {\n+\t\twpath_longname[len_longname++] = L'/';\n+\t\twpath_longname[len_longname] = 0;\n+\t}\n+\n \tCALLOC_ARRAY(watch, 1);\n \n \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n@@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tstrbuf_init(&watch->path, 0);\n \tstrbuf_addstr(&watch->path, path);\n \n+\twcscpy(watch->wpath_longname, wpath_longname);\n+\twatch->wpath_longname_len = len_longname;\n+\n \twatch->hDir = hDir;\n \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n \n@@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t,\n+\tDWORD info_action)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n+\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n \tstruct fsmonitor_batch *batch = NULL;\n \tconst char *p = watch->buffer;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \t/*\n \t * If the kernel gets more events than will fit in the kernel\n@@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n+\t\twchar_t *wpath = info->FileName;\n+\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n \t\tenum fsmonitor_path_type t;\n+\t\tenum get_relative_result grr;\n+\n+\t\tif (watch->has_shortnames) {\n+\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n+\t\t\t\t/*\n+\t\t\t\t * This event exactly matches the\n+\t\t\t\t * spelling of the shortname of\n+\t\t\t\t * \".git\", so we can skip some steps.\n+\t\t\t\t *\n+\t\t\t\t * (This case is odd because the user\n+\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n+\t\t\t\t * use the filesystem to map it back\n+\t\t\t\t * to \".git\".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_reset(&path);\n+\t\t\t\tstrbuf_addstr(&path, \".git\");\n+\t\t\t\tt = IS_DOT_GIT;\n+\t\t\t\tgoto process_it;\n+\t\t\t}\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n-\t\t\tgoto skip_this_path;\n-\n-\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n-\n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n+\t\t\tif (watch->has_tilda && !wcschr(wpath, L'~')) {\n+\t\t\t\t/*\n+\t\t\t\t * Shortnames on this filesystem have tildas\n+\t\t\t\t * and the notification path does not have\n+\t\t\t\t * one, so we assume that it is a longname.\n+\t\t\t\t */\n+\t\t\t\tgoto normalize_it;\n+\t\t\t}\n \n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n+\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n+\t\t\t\t\t\t wpath_longname);\n+\t\t\tswitch (grr) {\n+\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n+\t\t\t\tbreak;\n+\t\t\tcase GRR_HAVE_CONVERSION:\n+\t\t\t\twpath = wpath_longname;\n+\t\t\t\twpath_len = wcslen(wpath);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\tcase GRR_SHUTDOWN:\n \t\t\t\tgoto force_shutdown;\n \t\t\t}\n-\t\t\tbreak;\n+\t\t}\n \n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n+normalize_it:\n+\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n+\t\t\tgoto skip_this_path;\n \n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+process_it:\n+\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n+\t\t\t\t\t info->Action))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -382,6 +630,9 @@ force_shutdown:\n * Note that we DO NOT get filesystem events on the external \n * itself (it is not inside something that we are watching). In particular,\n * we do not get an event if the external is deleted.\n+ *\n+ * Also, we do not care about shortnames within the external , since\n+ * we never send these paths to clients.\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n@@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\tif (normalize_path_in_utf8(\n+\t\t\t info->FileName,\n+\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t &path) == -1)\n \t\t\tgoto skip_this_path;\n \n \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n@@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \tif (!data->watch_worktree)\n \t\tgoto failed;\n \n+\tcheck_for_shortnames(data->watch_worktree);\n+\n \tif (state->nr_paths_watching > 1) {\n \t\tdata->watch_gitdir = create_watch(state,\n \t\t\t\t\t\t state->path_gitdir_watch.buf);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 389ebf431c6..d28a74feeb9 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -173,6 +173,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon -C test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n+\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n+\n+\tstart_daemon -C test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"451890","messageId":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v2.git.1646777727.gitgitgadget@gmail.com","subject":"[PATCH v3 00/27] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:33Z","receivedAt":"2022-03-22T18:23:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V3 of Part 3 of my builtin FSMonitor series.\n\nI have addressed all of the feedback from Part 3 V2 and the mess that was\nPart 2.5 (now obsolete). This version builds upon the new V7 of Part 2\n(which includes 2.5).\n\nThis version includes: (1) fixup a few more \"_()\" calls in die() and error()\nmessages. (2) refactor how fsmonitor incompatibility error messages are\nformatted (3) make use of new \"start_daemon()\" function t7527 to reduce\nduplicated code. (4) improve documentation around Unicode/UTF8 testing on\nMacOS.\n\nHere is the range-diff from V2 to V3:\n\n 1: 34619e0652 ! 1: 779a15b38e fsm-listen-win32: handle shortnames\n @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI\n @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n goto normalize;\n if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n - error(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n + error(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n - GetLastError(),\n - (int)(info->FileNameLength / sizeof(WCHAR)),\n - info->FileName);\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'implicit daemon stop (rename\n +\n + git init test_implicit_1s &&\n +\n -+ start_daemon test_implicit_1s &&\n ++ start_daemon -C test_implicit_1s &&\n +\n + # renaming the .git directory will implicitly stop the daemon.\n + # this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'implicit daemon stop (rename\n + test_path_is_file test_implicit_1s2/GIT~1 &&\n + test_path_is_dir test_implicit_1s2/GIT~2 &&\n +\n -+ start_daemon test_implicit_1s2 &&\n ++ start_daemon -C test_implicit_1s2 &&\n +\n + # renaming the .git directory will implicitly stop the daemon.\n + # the rename-from FS Event will contain the shortname.\n 2: 3a0f30b849 ! 2: 11d4a17b69 t7527: test FSMonitor on repos with Unicode root paths\n @@ t/t7527-builtin-fsmonitor.sh: do\n + git -C \"$u\" add file1 &&\n + git -C \"$u\" config core.fsmonitor true &&\n +\n -+ start_daemon \"$u\" &&\n ++ start_daemon -C \"$u\" &&\n + git -C \"$u\" status >actual &&\n + grep \"new file: file1\" actual\n + '\n 3: 87d1c0b6f2 ! 3: 901fa32f6e t/helper/fsmonitor-client: create stress test\n @@ t/helper/test-fsmonitor-client.c: static int do_send_flush(void)\n + int nr_requests = 1;\n \n const char * const fsmonitor_client_usage[] = {\n - N_(\"test-helper fsmonitor-client query []\"),\n - N_(\"test-helper fsmonitor-client flush\"),\n -+ N_(\"test-helper fsmonitor-client hammer [] [] []\"),\n + \"test-tool fsmonitor-client query []\",\n + \"test-tool fsmonitor-client flush\",\n ++ \"test-tool fsmonitor-client hammer [] [] []\",\n NULL,\n };\n \n struct option options[] = {\n - OPT_STRING(0, \"token\", &token, N_(\"token\"),\n - N_(\"command token to send to the server\")),\n + OPT_STRING(0, \"token\", &token, \"token\",\n + \"command token to send to the server\"),\n +\n -+ OPT_INTEGER(0, \"threads\", &nr_threads, N_(\"number of client threads\")),\n -+ OPT_INTEGER(0, \"requests\", &nr_requests, N_(\"number of requests per thread\")),\n ++ OPT_INTEGER(0, \"threads\", &nr_threads, \"number of client threads\"),\n ++ OPT_INTEGER(0, \"requests\", &nr_requests, \"number of requests per thread\"),\n +\n OPT_END()\n };\n 4: 8c4f90ae4f ! 4: a8f0b2a525 fsmonitor-settings: bare repos are incompatible with FSMonitor\n @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar\n + prepare_repo_settings(the_repository);\n + fsm_settings__set_ipc(the_repository);\n +\n -+ if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {\n -+ const char *msg = fsm_settings__get_reason_msg(the_repository);\n -+\n -+ return error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n -+ }\n ++ if (fsm_settings__error_if_incompatible(the_repository))\n ++ return 1;\n +\n if (!strcmp(subcmd, \"start\"))\n return !!try_to_start_background_daemon();\n @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const\n if (fsmonitor > 0) {\n enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n +\n -+ if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {\n -+ const char *msg = fsm_settings__get_reason_msg(r);\n -+\n -+ return error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n -+ }\n ++ if (fsm_settings__error_if_incompatible(the_repository))\n ++ return 1;\n +\n if (fsm_mode == FSMONITOR_MODE_DISABLED) {\n - advise(_(\"core.fsmonitor is unset; \"\n - \"set it if you really want to \"\n + warning(_(\"core.fsmonitor is unset; \"\n + \"set it if you really want to \"\n \n ## fsmonitor-settings.c ##\n @@\n @@ fsmonitor-settings.c\n static void lookup_fsmonitor_settings(struct repository *r)\n {\n struct fsmonitor_settings *s;\n +@@ fsmonitor-settings.c: static void lookup_fsmonitor_settings(struct repository *r)\n + \n + CALLOC_ARRAY(s, 1);\n + s->mode = FSMONITOR_MODE_DISABLED;\n ++ s->reason = FSMONITOR_REASON_OK;\n + \n + r->settings.fsmonitor = s;\n + \n @@ fsmonitor-settings.c: void fsm_settings__set_ipc(struct repository *r)\n \n lookup_fsmonitor_settings(r);\n @@ fsmonitor-settings.c: void fsm_settings__set_disabled(struct repository *r)\n + return r->settings.fsmonitor->reason;\n +}\n +\n -+const char *fsm_settings__get_reason_msg(struct repository *r)\n ++int fsm_settings__error_if_incompatible(struct repository *r)\n +{\n + enum fsmonitor_reason reason = fsm_settings__get_reason(r);\n +\n + switch (reason) {\n + case FSMONITOR_REASON_OK:\n -+ return NULL;\n ++ return 0;\n +\n + case FSMONITOR_REASON_BARE:\n -+ return _(\"bare repos are incompatible with fsmonitor\");\n ++ error(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n ++ xgetcwd());\n ++ return 1;\n + }\n +\n -+ BUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\n ++ BUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n + reason);\n +}\n \n @@ fsmonitor-settings.h: void fsm_settings__set_disabled(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n -+const char *fsm_settings__get_reason_msg(struct repository *r);\n ++int fsm_settings__error_if_incompatible(struct repository *r);\n +\n struct fsmonitor_settings;\n \n @@ t/t7519-status-fsmonitor.sh: test_lazy_prereq UNTRACKED_CACHE '\n + test_must_fail \\\n + git -C ./bare-clone -c core.fsmonitor=foo \\\n + update-index --fsmonitor 2>actual &&\n -+ grep \"bare repos are incompatible with fsmonitor\" actual &&\n ++ grep \"bare repository .* is incompatible with fsmonitor\" actual &&\n +\n + test_must_fail \\\n + git -C ./bare-clone -c core.fsmonitor=true \\\n + update-index --fsmonitor 2>actual &&\n -+ grep \"bare repos are incompatible with fsmonitor\" actual\n ++ grep \"bare repository .* is incompatible with fsmonitor\" actual\n +'\n +\n +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n + test_when_finished \"rm -rf ./bare-clone actual\" &&\n + git init --bare bare-clone &&\n + test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n -+ grep \"bare repos are incompatible with fsmonitor\" actual\n ++ grep \"bare repository .* is incompatible with fsmonitor\" actual\n +'\n +\n test_expect_success 'setup' '\n 5: 6329328d18 ! 5: e32a8a7ea7 fsmonitor-settings: stub in platform-specific incompatibility checking\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor-settings: stub in platform-specific incompatibility checking\n + fsmonitor-settings: stub in Win32-specific incompatibility checking\n \n Extend generic incompatibility checkout with platform-specific\n mechanism. Stub in Win32 version.\n @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)\n \n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: const char *fsm_settings__get_reason_msg(struct repository *r);\n +@@ fsmonitor-settings.h: int fsm_settings__error_if_incompatible(struct repository *r);\n \n struct fsmonitor_settings;\n \n 6: fa9e86e7de ! 6: 5546339d96 fsmonitor-settings: VFS for Git virtual repos are incompatible\n @@ compat/fsmonitor/fsm-settings-win32.c\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)\n - \n - case FSMONITOR_REASON_BARE:\n - return _(\"bare repos are incompatible with fsmonitor\");\n +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n + error(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n + xgetcwd());\n + return 1;\n +\n + case FSMONITOR_REASON_VFS4GIT:\n -+ return _(\"virtual repos are incompatible with fsmonitor\");\n ++ error(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n ++ r->worktree);\n ++ return 1;\n }\n \n - BUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\n + BUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_mode {\n @@ fsmonitor-settings.h: enum fsmonitor_mode {\n \n ## t/t7519-status-fsmonitor.sh ##\n @@ t/t7519-status-fsmonitor.sh: test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n - grep \"bare repos are incompatible with fsmonitor\" actual\n + grep \"bare repository .* is incompatible with fsmonitor\" actual\n '\n \n +test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n @@ t/t7519-status-fsmonitor.sh: test_expect_success FSMONITOR_DAEMON 'run fsmonitor\n + test_must_fail git -C ./fake-virtual-clone \\\n + -c core.virtualfilesystem=true \\\n + fsmonitor--daemon run 2>actual &&\n -+ grep \"virtual repos are incompatible with fsmonitor\" actual\n ++ grep \"virtual repository .* is incompatible with fsmonitor\" actual\n +'\n +\n test_expect_success 'setup' '\n 7: c180241041 = 7: 1d2877efda fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: e3bfa0bd69 ! 8: 06d7f18676 fsmonitor-settings: remote repos on macOS are incompatible\n @@ compat/fsmonitor/fsm-settings-darwin.c\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)\n - case FSMONITOR_REASON_BARE:\n - return _(\"bare repos are incompatible with fsmonitor\");\n +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n + xgetcwd());\n + return 1;\n \n + case FSMONITOR_REASON_ERROR:\n -+ return _(\"repo incompatible with fsmonitor due to errors\");\n ++ error(_(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n ++ r->worktree);\n ++ return 1;\n +\n + case FSMONITOR_REASON_REMOTE:\n -+ return _(\"remote repos are incompatible with fsmonitor\");\n ++ error(_(\"remote repository '%s' is incompatible with fsmonitor\"),\n ++ r->worktree);\n ++ return 1;\n +\n case FSMONITOR_REASON_VFS4GIT:\n - return _(\"virtual repos are incompatible with fsmonitor\");\n - }\n + error(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n + r->worktree);\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_mode {\n 9: e32da3118f = 9: 5ca97f482d fsmonitor-settings: remote repos on Windows are incompatible\n27: e3e01677d9 ! 10: 6715143724 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n @@ compat/fsmonitor/fsm-settings-darwin.c: enum fsmonitor_reason fsm_os__incompatib\n \n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)\n - \n - case FSMONITOR_REASON_VFS4GIT:\n - return _(\"virtual repos are incompatible with fsmonitor\");\n +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n + error(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n + r->worktree);\n + return 1;\n +\n + case FSMONITOR_REASON_NOSOCKETS:\n -+ return _(\"repo filesystem does not support Unix sockets\");\n ++ error(_(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n ++ r->worktree);\n ++ return 1;\n }\n \n - BUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\n + BUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_reason {\n10: f63de4eda3 = 11: ed1f723130 unpack-trees: initialize fsmonitor_has_run_once in o->result\n11: fe305f5f28 = 12: 35c77b854b fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n12: c8f3e251b1 ! 13: a5affb359c fsmonitor--daemon: cd out of worktree root\n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)\n + */\n + home = getenv(\"HOME\");\n + if (home && *home && chdir(home))\n -+ die_errno(\"could not cd home '%s'\", home);\n ++ die_errno(_(\"could not cd home '%s'\"), home);\n +\n err = fsmonitor_run_daemon_1(&state);\n \n @@ compat/fsmonitor/fsm-listen-win32.c: static int recv_rdcw_watch(struct one_watch\n + * Shutdown if we get any error.\n */\n \n - error(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n + error(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\n \n ## fsmonitor--daemon.h ##\n @@ fsmonitor--daemon.h: struct fsmonitor_daemon_state {\n13: 71673be2da = 14: 087af5dfb6 fsmonitor--daemon: prepare for adding health thread\n14: 5387baaf5d ! 15: e78eb20c1b fsmonitor--daemon: rename listener thread related variables\n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)\n /* Prepare to (recursively) watch the directory. */\n \n ## compat/fsmonitor/fsm-listen-darwin.c ##\n -@@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef stream);\n +@@\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n @@ compat/fsmonitor/fsm-listen-darwin.c: int fsm_listen__ctor(struct fsmonitor_daem\n NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n @@ compat/fsmonitor/fsm-listen-darwin.c: int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n - error(\"Unable to create FSEventStream.\");\n + error(_(\"Unable to create FSEventStream.\"));\n \n - FREE_AND_NULL(state->backend_data);\n + FREE_AND_NULL(state->listen_data);\n15: f78e4ad87c = 16: 301fff5296 fsmonitor--daemon: stub in health thread\n16: bb72f911a0 = 17: c6b5bdd25e fsm-health-win32: add polling framework to monitor daemon health\n17: baf8c031a9 ! 18: 13d11713a8 fsm-health-win32: force shutdown daemon if worktree root moves\n @@ compat/fsmonitor/fsm-health-win32.c: struct fsm_health_data\n + return 0;\n +\n + default:\n -+ die(\"unhandled case in 'has_worktree_moved': %d\",\n ++ die(_(\"unhandled case in 'has_worktree_moved': %d\"),\n + (int)ctx);\n + }\n +\n18: 796b659139 = 19: 01c1a38c46 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n19: 2459192087 = 20: 0f0a5b5ca1 fsmonitor: optimize processing of directory events\n20: 06a3241385 ! 21: d8218d197a t7527: FSMonitor tests for directory moves\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '\n git -c core.fsmonitor=false add . &&\n test_tick &&\n git -c core.fsmonitor=false commit -m initial &&\n -@@ t/t7527-builtin-fsmonitor.sh: verify_status () {\n - echo HELLO AFTER\n +@@ t/t7527-builtin-fsmonitor.sh: directory_to_file () {\n + echo 1 >dir1\n }\n \n +move_directory_contents_deeper() {\n -+ mkdir T1/_new_\n ++ mkdir T1/_new_ &&\n + mv T1/[A-Z]* T1/_new_\n +}\n +\n21: 4b59013cad = 22: 79da369dcc t/perf/p7527: add perf test for builtin FSMonitor\n22: 524d449ed6 ! 23: 4ab4306ada fsmonitor: never set CE_FSMONITOR_VALID on submodules\n @@ t/t7527-builtin-fsmonitor.sh: do\n + git -C super submodule add ../sub ./dir_1/dir_2/sub &&\n + git -C super commit -m \"add sub\" &&\n +\n -+ start_daemon super &&\n ++ start_daemon -C super &&\n + git -C super config core.fsmonitor true &&\n + git -C super update-index --fsmonitor &&\n + git -C super status &&\n23: c7264decaf ! 24: 5d0fa19929 t7527: test FSMonitor on case insensitive+preserving file system\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule\" '\n +# test_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n +\n + git init test_insensitive &&\n -+ (\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/insensitive.trace\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n +\n -+ start_daemon test_insensitive\n -+ ) &&\n ++ start_daemon -C test_insensitive -tf \"$PWD/insensitive.trace\" &&\n +\n + mkdir -p test_insensitive/abc/def &&\n + echo xyz >test_insensitive/ABC/DEF/xyz &&\n24: 95b9d4210d = 25: 264397e8bd fsmonitor: on macOS also emit NFC spelling for NFD pathname\n25: 5a0c1b7a28 ! 26: e6b621fb76 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n @@ t/lib-unicode-nfc-nfd.sh (new)\n + ls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n +'\n +\n -+if test $unicode_debug = 1\n ++# The following is for debugging. I found it useful when\n ++# trying to understand the various (OS, FS) quirks WRT\n ++# Unicode and how composition/decomposition is handled.\n ++# For example, when trying to understand how (macOS, APFS)\n ++# and (macOS, HFS) and (macOS, FAT32) compare.\n ++#\n ++# It is rather noisy, so it is disabled by default.\n ++#\n ++if test \"$unicode_debug\" = \"true\"\n +then\n + if test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n + then\n26: a45c1fd300 ! 27: aa96a849ce t7527: test Unicode NFC/NFD handling on MacOS\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse\n egrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n -+unicode_debug=0\n ++# The variable \"unicode_debug\" is defined in the following library\n ++# script to dump information about how the (OS, FS) handles Unicode\n ++# composition. Uncomment the following line if you want to enable it.\n ++#\n ++# unicode_debug=true\n ++\n +. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n +\n +# See if the OS or filesystem does NFC/NFD aliasing/munging.\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse\n + test_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n +\n + git init test_unicode &&\n -+ (\n -+ GIT_TRACE_FSMONITOR=\"$(pwd)/unicode.trace\" &&\n -+ export GIT_TRACE_FSMONITOR &&\n +\n -+ start_daemon test_unicode\n -+ ) &&\n ++ start_daemon -C test_unicode -tf \"$PWD/unicode.trace\" &&\n +\n + # Create a directory using an NFC spelling.\n + #\n\n\nJeff Hostetler (27):\n fsm-listen-win32: handle shortnames\n t7527: test FSMonitor on repos with Unicode root paths\n t/helper/fsmonitor-client: create stress test\n fsmonitor-settings: bare repos are incompatible with FSMonitor\n fsmonitor-settings: stub in Win32-specific incompatibility checking\n fsmonitor-settings: VFS for Git virtual repos are incompatible\n fsmonitor-settings: stub in macOS-specific incompatibility checking\n fsmonitor-settings: remote repos on macOS are incompatible\n fsmonitor-settings: remote repos on Windows are incompatible\n fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n unpack-trees: initialize fsmonitor_has_run_once in o->result\n fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n fsmonitor--daemon: cd out of worktree root\n fsmonitor--daemon: prepare for adding health thread\n fsmonitor--daemon: rename listener thread related variables\n fsmonitor--daemon: stub in health thread\n fsm-health-win32: add polling framework to monitor daemon health\n fsm-health-win32: force shutdown daemon if worktree root moves\n fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n fsmonitor: optimize processing of directory events\n t7527: FSMonitor tests for directory moves\n t/perf/p7527: add perf test for builtin FSMonitor\n fsmonitor: never set CE_FSMONITOR_VALID on submodules\n t7527: test FSMonitor on case insensitive+preserving file system\n fsmonitor: on macOS also emit NFC spelling for NFD pathname\n t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n t7527: test Unicode NFC/NFD handling on MacOS\n\n Makefile | 19 +-\n builtin/fsmonitor--daemon.c | 104 ++++++-\n builtin/update-index.c | 4 +\n compat/fsmonitor/fsm-health-darwin.c | 24 ++\n compat/fsmonitor/fsm-health-win32.c | 278 +++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 +++\n compat/fsmonitor/fsm-listen-darwin.c | 122 ++++++--\n compat/fsmonitor/fsm-listen-win32.c | 413 ++++++++++++++++++++-----\n compat/fsmonitor/fsm-listen.h | 2 +-\n compat/fsmonitor/fsm-settings-darwin.c | 89 ++++++\n compat/fsmonitor/fsm-settings-win32.c | 137 ++++++++\n config.mak.uname | 5 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 11 +-\n fsmonitor-settings.c | 92 ++++++\n fsmonitor-settings.h | 29 ++\n fsmonitor.c | 73 ++++-\n fsmonitor.h | 11 +\n t/helper/test-fsmonitor-client.c | 106 +++++++\n t/lib-unicode-nfc-nfd.sh | 167 ++++++++++\n t/perf/p7527-builtin-fsmonitor.sh | 257 +++++++++++++++\n t/t7519-status-fsmonitor.sh | 32 ++\n t/t7527-builtin-fsmonitor.sh | 311 +++++++++++++++++++\n unpack-trees.c | 1 +\n 24 files changed, 2218 insertions(+), 124 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\n\nbase-commit: e6cf84dc8eb4933220187849b84e5cb96cda185f\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1143\n\nRange-diff vs v2:\n\n 1: 34619e0652b ! 1: 779a15b38e8 fsm-listen-win32: handle shortnames\n @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI\n @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n - \t\t\terror(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\",\n + \t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n -\t\t\t GetLastError(),\n -\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n -\t\t\t info->FileName);\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'implicit daemon stop (rename\n +\n +\tgit init test_implicit_1s &&\n +\n -+\tstart_daemon test_implicit_1s &&\n ++\tstart_daemon -C test_implicit_1s &&\n +\n +\t# renaming the .git directory will implicitly stop the daemon.\n +\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'implicit daemon stop (rename\n +\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n +\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n +\n -+\tstart_daemon test_implicit_1s2 &&\n ++\tstart_daemon -C test_implicit_1s2 &&\n +\n +\t# renaming the .git directory will implicitly stop the daemon.\n +\t# the rename-from FS Event will contain the shortname.\n 2: 3a0f30b849a ! 2: 11d4a17b692 t7527: test FSMonitor on repos with Unicode root paths\n @@ t/t7527-builtin-fsmonitor.sh: do\n +\t\tgit -C \"$u\" add file1 &&\n +\t\tgit -C \"$u\" config core.fsmonitor true &&\n +\n -+\t\tstart_daemon \"$u\" &&\n ++\t\tstart_daemon -C \"$u\" &&\n +\t\tgit -C \"$u\" status >actual &&\n +\t\tgrep \"new file: file1\" actual\n +\t'\n 3: 87d1c0b6f2a ! 3: 901fa32f6ea t/helper/fsmonitor-client: create stress test\n @@ t/helper/test-fsmonitor-client.c: static int do_send_flush(void)\n +\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n - \t\tN_(\"test-helper fsmonitor-client query []\"),\n - \t\tN_(\"test-helper fsmonitor-client flush\"),\n -+\t\tN_(\"test-helper fsmonitor-client hammer [] [] []\"),\n + \t\t\"test-tool fsmonitor-client query []\",\n + \t\t\"test-tool fsmonitor-client flush\",\n ++\t\t\"test-tool fsmonitor-client hammer [] [] []\",\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n - \t\tOPT_STRING(0, \"token\", &token, N_(\"token\"),\n - \t\t\t N_(\"command token to send to the server\")),\n + \t\tOPT_STRING(0, \"token\", &token, \"token\",\n + \t\t\t \"command token to send to the server\"),\n +\n -+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, N_(\"number of client threads\")),\n -+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, N_(\"number of requests per thread\")),\n ++\t\tOPT_INTEGER(0, \"threads\", &nr_threads, \"number of client threads\"),\n ++\t\tOPT_INTEGER(0, \"requests\", &nr_requests, \"number of requests per thread\"),\n +\n \t\tOPT_END()\n \t};\n 4: 8c4f90ae4fd ! 4: a8f0b2a5256 fsmonitor-settings: bare repos are incompatible with FSMonitor\n @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar\n +\tprepare_repo_settings(the_repository);\n +\tfsm_settings__set_ipc(the_repository);\n +\n -+\tif (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {\n -+\t\tconst char *msg = fsm_settings__get_reason_msg(the_repository);\n -+\n -+\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n -+\t}\n ++\tif (fsm_settings__error_if_incompatible(the_repository))\n ++\t\treturn 1;\n +\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const\n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n +\n -+\t\tif (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {\n -+\t\t\tconst char *msg = fsm_settings__get_reason_msg(r);\n -+\n -+\t\t\treturn error(\"%s '%s'\", msg ? msg : \"???\", xgetcwd());\n -+\t\t}\n ++\t\tif (fsm_settings__error_if_incompatible(the_repository))\n ++\t\t\treturn 1;\n +\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n - \t\t\tadvise(_(\"core.fsmonitor is unset; \"\n - \t\t\t\t \"set it if you really want to \"\n + \t\t\twarning(_(\"core.fsmonitor is unset; \"\n + \t\t\t\t\"set it if you really want to \"\n \n ## fsmonitor-settings.c ##\n @@\n @@ fsmonitor-settings.c\n static void lookup_fsmonitor_settings(struct repository *r)\n {\n \tstruct fsmonitor_settings *s;\n +@@ fsmonitor-settings.c: static void lookup_fsmonitor_settings(struct repository *r)\n + \n + \tCALLOC_ARRAY(s, 1);\n + \ts->mode = FSMONITOR_MODE_DISABLED;\n ++\ts->reason = FSMONITOR_REASON_OK;\n + \n + \tr->settings.fsmonitor = s;\n + \n @@ fsmonitor-settings.c: void fsm_settings__set_ipc(struct repository *r)\n \n \tlookup_fsmonitor_settings(r);\n @@ fsmonitor-settings.c: void fsm_settings__set_disabled(struct repository *r)\n +\treturn r->settings.fsmonitor->reason;\n +}\n +\n -+const char *fsm_settings__get_reason_msg(struct repository *r)\n ++int fsm_settings__error_if_incompatible(struct repository *r)\n +{\n +\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n +\n +\tswitch (reason) {\n +\tcase FSMONITOR_REASON_OK:\n -+\t\treturn NULL;\n ++\t\treturn 0;\n +\n +\tcase FSMONITOR_REASON_BARE:\n -+\t\treturn _(\"bare repos are incompatible with fsmonitor\");\n ++\t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n ++\t\t xgetcwd());\n ++\t\treturn 1;\n +\t}\n +\n -+\tBUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\n ++\tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n +\t reason);\n +}\n \n @@ fsmonitor-settings.h: void fsm_settings__set_disabled(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n -+const char *fsm_settings__get_reason_msg(struct repository *r);\n ++int fsm_settings__error_if_incompatible(struct repository *r);\n +\n struct fsmonitor_settings;\n \n @@ t/t7519-status-fsmonitor.sh: test_lazy_prereq UNTRACKED_CACHE '\n +\ttest_must_fail \\\n +\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n +\t\t\tupdate-index --fsmonitor 2>actual &&\n -+\tgrep \"bare repos are incompatible with fsmonitor\" actual &&\n ++\tgrep \"bare repository .* is incompatible with fsmonitor\" actual &&\n +\n +\ttest_must_fail \\\n +\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n +\t\t\tupdate-index --fsmonitor 2>actual &&\n -+\tgrep \"bare repos are incompatible with fsmonitor\" actual\n ++\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n +'\n +\n +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n +\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n +\tgit init --bare bare-clone &&\n +\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n -+\tgrep \"bare repos are incompatible with fsmonitor\" actual\n ++\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n +'\n +\n test_expect_success 'setup' '\n 5: 6329328d185 ! 5: e32a8a7ea7a fsmonitor-settings: stub in platform-specific incompatibility checking\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor-settings: stub in platform-specific incompatibility checking\n + fsmonitor-settings: stub in Win32-specific incompatibility checking\n \n Extend generic incompatibility checkout with platform-specific\n mechanism. Stub in Win32 version.\n @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)\n \n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: const char *fsm_settings__get_reason_msg(struct repository *r);\n +@@ fsmonitor-settings.h: int fsm_settings__error_if_incompatible(struct repository *r);\n \n struct fsmonitor_settings;\n \n 6: fa9e86e7de7 ! 6: 5546339d963 fsmonitor-settings: VFS for Git virtual repos are incompatible\n @@ compat/fsmonitor/fsm-settings-win32.c\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)\n - \n - \tcase FSMONITOR_REASON_BARE:\n - \t\treturn _(\"bare repos are incompatible with fsmonitor\");\n +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n + \t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n + \t\t xgetcwd());\n + \t\treturn 1;\n +\n +\tcase FSMONITOR_REASON_VFS4GIT:\n -+\t\treturn _(\"virtual repos are incompatible with fsmonitor\");\n ++\t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n ++\t\t r->worktree);\n ++\t\treturn 1;\n \t}\n \n - \tBUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\n + \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_mode {\n @@ fsmonitor-settings.h: enum fsmonitor_mode {\n \n ## t/t7519-status-fsmonitor.sh ##\n @@ t/t7519-status-fsmonitor.sh: test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n - \tgrep \"bare repos are incompatible with fsmonitor\" actual\n + \tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n '\n \n +test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n @@ t/t7519-status-fsmonitor.sh: test_expect_success FSMONITOR_DAEMON 'run fsmonitor\n +\ttest_must_fail git -C ./fake-virtual-clone \\\n +\t\t\t -c core.virtualfilesystem=true \\\n +\t\t\t fsmonitor--daemon run 2>actual &&\n -+\tgrep \"virtual repos are incompatible with fsmonitor\" actual\n ++\tgrep \"virtual repository .* is incompatible with fsmonitor\" actual\n +'\n +\n test_expect_success 'setup' '\n 7: c1802410410 = 7: 1d2877efda0 fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: e3bfa0bd69d ! 8: 06d7f18676d fsmonitor-settings: remote repos on macOS are incompatible\n @@ compat/fsmonitor/fsm-settings-darwin.c\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)\n - \tcase FSMONITOR_REASON_BARE:\n - \t\treturn _(\"bare repos are incompatible with fsmonitor\");\n +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n + \t\t xgetcwd());\n + \t\treturn 1;\n \n +\tcase FSMONITOR_REASON_ERROR:\n -+\t\treturn _(\"repo incompatible with fsmonitor due to errors\");\n ++\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n ++\t\t r->worktree);\n ++\t\treturn 1;\n +\n +\tcase FSMONITOR_REASON_REMOTE:\n -+\t\treturn _(\"remote repos are incompatible with fsmonitor\");\n ++\t\terror(_(\"remote repository '%s' is incompatible with fsmonitor\"),\n ++\t\t r->worktree);\n ++\t\treturn 1;\n +\n \tcase FSMONITOR_REASON_VFS4GIT:\n - \t\treturn _(\"virtual repos are incompatible with fsmonitor\");\n - \t}\n + \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n + \t\t r->worktree);\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_mode {\n 9: e32da3118fb = 9: 5ca97f482d0 fsmonitor-settings: remote repos on Windows are incompatible\n 27: e3e01677d93 ! 10: 67151437245 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n @@ compat/fsmonitor/fsm-settings-darwin.c: enum fsmonitor_reason fsm_os__incompatib\n \n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)\n - \n - \tcase FSMONITOR_REASON_VFS4GIT:\n - \t\treturn _(\"virtual repos are incompatible with fsmonitor\");\n +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n + \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n + \t\t r->worktree);\n + \t\treturn 1;\n +\n +\tcase FSMONITOR_REASON_NOSOCKETS:\n -+\t\treturn _(\"repo filesystem does not support Unix sockets\");\n ++\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n ++\t\t r->worktree);\n ++\t\treturn 1;\n \t}\n \n - \tBUG(\"Unhandled case in fsm_settings__get_reason_msg '%d'\",\n + \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_reason {\n 10: f63de4eda31 = 11: ed1f7231309 unpack-trees: initialize fsmonitor_has_run_once in o->result\n 11: fe305f5f287 = 12: 35c77b854bd fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n 12: c8f3e251b1f ! 13: a5affb359c4 fsmonitor--daemon: cd out of worktree root\n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)\n +\t */\n +\thome = getenv(\"HOME\");\n +\tif (home && *home && chdir(home))\n -+\t\tdie_errno(\"could not cd home '%s'\", home);\n ++\t\tdie_errno(_(\"could not cd home '%s'\"), home);\n +\n \terr = fsmonitor_run_daemon_1(&state);\n \n @@ compat/fsmonitor/fsm-listen-win32.c: static int recv_rdcw_watch(struct one_watch\n +\t * Shutdown if we get any error.\n \t */\n \n - \terror(\"GetOverlappedResult failed on '%s' [GLE %ld]\",\n + \terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\n \n ## fsmonitor--daemon.h ##\n @@ fsmonitor--daemon.h: struct fsmonitor_daemon_state {\n 13: 71673be2da5 = 14: 087af5dfb63 fsmonitor--daemon: prepare for adding health thread\n 14: 5387baaf5d7 ! 15: e78eb20c1bf fsmonitor--daemon: rename listener thread related variables\n @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)\n \t/* Prepare to (recursively) watch the directory. */\n \n ## compat/fsmonitor/fsm-listen-darwin.c ##\n -@@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef stream);\n +@@\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n @@ compat/fsmonitor/fsm-listen-darwin.c: int fsm_listen__ctor(struct fsmonitor_daem\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n @@ compat/fsmonitor/fsm-listen-darwin.c: int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n - \terror(\"Unable to create FSEventStream.\");\n + \terror(_(\"Unable to create FSEventStream.\"));\n \n -\tFREE_AND_NULL(state->backend_data);\n +\tFREE_AND_NULL(state->listen_data);\n 15: f78e4ad87c0 = 16: 301fff5296a fsmonitor--daemon: stub in health thread\n 16: bb72f911a05 = 17: c6b5bdd25e4 fsm-health-win32: add polling framework to monitor daemon health\n 17: baf8c031a97 ! 18: 13d11713a86 fsm-health-win32: force shutdown daemon if worktree root moves\n @@ compat/fsmonitor/fsm-health-win32.c: struct fsm_health_data\n +\t\treturn 0;\n +\n +\tdefault:\n -+\t\tdie(\"unhandled case in 'has_worktree_moved': %d\",\n ++\t\tdie(_(\"unhandled case in 'has_worktree_moved': %d\"),\n +\t\t (int)ctx);\n +\t}\n +\n 18: 796b6591393 = 19: 01c1a38c462 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n 19: 24591920878 = 20: 0f0a5b5ca16 fsmonitor: optimize processing of directory events\n 20: 06a32413854 ! 21: d8218d197ad t7527: FSMonitor tests for directory moves\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n -@@ t/t7527-builtin-fsmonitor.sh: verify_status () {\n - \techo HELLO AFTER\n +@@ t/t7527-builtin-fsmonitor.sh: directory_to_file () {\n + \techo 1 >dir1\n }\n \n +move_directory_contents_deeper() {\n -+\tmkdir T1/_new_\n ++\tmkdir T1/_new_ &&\n +\tmv T1/[A-Z]* T1/_new_\n +}\n +\n 21: 4b59013cadd = 22: 79da369dcce t/perf/p7527: add perf test for builtin FSMonitor\n 22: 524d449ed64 ! 23: 4ab4306adab fsmonitor: never set CE_FSMONITOR_VALID on submodules\n @@ t/t7527-builtin-fsmonitor.sh: do\n +\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n +\tgit -C super commit -m \"add sub\" &&\n +\n -+\tstart_daemon super &&\n ++\tstart_daemon -C super &&\n +\tgit -C super config core.fsmonitor true &&\n +\tgit -C super update-index --fsmonitor &&\n +\tgit -C super status &&\n 23: c7264decaf6 ! 24: 5d0fa19929d t7527: test FSMonitor on case insensitive+preserving file system\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule\" '\n +#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n +\n +\tgit init test_insensitive &&\n -+\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/insensitive.trace\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n +\n -+\t\tstart_daemon test_insensitive\n -+\t) &&\n ++\tstart_daemon -C test_insensitive -tf \"$PWD/insensitive.trace\" &&\n +\n +\tmkdir -p test_insensitive/abc/def &&\n +\techo xyz >test_insensitive/ABC/DEF/xyz &&\n 24: 95b9d4210d2 = 25: 264397e8bd4 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n 25: 5a0c1b7a287 ! 26: e6b621fb766 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n @@ t/lib-unicode-nfc-nfd.sh (new)\n +\tls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n +'\n +\n -+if test $unicode_debug = 1\n ++# The following is for debugging. I found it useful when\n ++# trying to understand the various (OS, FS) quirks WRT\n ++# Unicode and how composition/decomposition is handled.\n ++# For example, when trying to understand how (macOS, APFS)\n ++# and (macOS, HFS) and (macOS, FAT32) compare.\n ++#\n ++# It is rather noisy, so it is disabled by default.\n ++#\n ++if test \"$unicode_debug\" = \"true\"\n +then\n +\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n +\tthen\n 26: a45c1fd3000 ! 27: aa96a849ce4 t7527: test Unicode NFC/NFD handling on MacOS\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse\n \tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n -+unicode_debug=0\n ++# The variable \"unicode_debug\" is defined in the following library\n ++# script to dump information about how the (OS, FS) handles Unicode\n ++# composition. Uncomment the following line if you want to enable it.\n ++#\n ++# unicode_debug=true\n ++\n +. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n +\n +# See if the OS or filesystem does NFC/NFD aliasing/munging.\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse\n +\ttest_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n +\n +\tgit init test_unicode &&\n -+\t(\n -+\t\tGIT_TRACE_FSMONITOR=\"$(pwd)/unicode.trace\" &&\n -+\t\texport GIT_TRACE_FSMONITOR &&\n +\n -+\t\tstart_daemon test_unicode\n -+\t) &&\n ++\tstart_daemon -C test_unicode -tf \"$PWD/unicode.trace\" &&\n +\n +\t# Create a directory using an NFC spelling.\n +\t#\n\n-- \ngitgitgadget\n"},{"id":"451891","messageId":"11d4a17b69239bdce17a7b4d76a3491c29976c13.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 02/27] t7527: test FSMonitor on repos with Unicode root paths","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:35Z","receivedAt":"2022-03-22T18:23:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate some test repos with UTF8 characters in the pathname of the\nroot directory and verify that the builtin FSMonitor can watch them.\n\nThis test is mainly for Windows where we need to avoid `*A()`\nroutines.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex d28a74feeb9..429029fcadd 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -656,4 +656,27 @@ do\n \tdone\n done\n \n+# Test Unicode UTF-8 characters in the pathname of the working\n+# directory root. Use of \"*A()\" routines rather than \"*W()\" routines\n+# on Windows can sometimes lead to odd failures.\n+#\n+u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n+u2=$(printf \"u_e2_99_ab__\\xE2\\x99\\xAB\")\n+u_values=\"$u1 $u2\"\n+for u in $u_values\n+do\n+\ttest_expect_success \"Unicode in repo root path: $u\" '\n+\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n+\n+\t\tgit init \"$u\" &&\n+\t\techo 1 >\"$u\"/file1 &&\n+\t\tgit -C \"$u\" add file1 &&\n+\t\tgit -C \"$u\" config core.fsmonitor true &&\n+\n+\t\tstart_daemon -C \"$u\" &&\n+\t\tgit -C \"$u\" status >actual &&\n+\t\tgrep \"new file: file1\" actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"451892","messageId":"901fa32f6eacab7ba76aedbb020855c8f9718943.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 03/27] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:36Z","receivedAt":"2022-03-22T18:23:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a stress test to hammer on the fsmonitor daemon.\nCreate a client-side thread pool of n threads and have\neach of them make m requests as fast as they can.\n\nWe do not currently inspect the contents of the response.\nWe're only interested in placing a heavy request load on\nthe daemon.\n\nThis test is useful for interactive testing and various\nexperimentation. For example, to place additional load\non the daemon while another test is running. We currently\ndo not have a test script that actually uses this helper.\nWe might add such a test in the future.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++\n 1 file changed, 106 insertions(+)\n\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nindex 3062c8a3c2b..54a4856c48c 100644\n--- a/t/helper/test-fsmonitor-client.c\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -7,6 +7,8 @@\n #include \"cache.h\"\n #include \"parse-options.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2.h\"\n \n #ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n int cmd__fsmonitor_client(int argc, const char **argv)\n@@ -79,20 +81,121 @@ static int do_send_flush(void)\n \treturn 0;\n }\n \n+struct hammer_thread_data\n+{\n+\tpthread_t pthread_id;\n+\tint thread_nr;\n+\n+\tint nr_requests;\n+\tconst char *token;\n+\n+\tint sum_successful;\n+\tint sum_errors;\n+};\n+\n+static void *hammer_thread_proc(void *_hammer_thread_data)\n+{\n+\tstruct hammer_thread_data *data = _hammer_thread_data;\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint k;\n+\tint ret;\n+\n+\ttrace2_thread_start(\"hammer\");\n+\n+\tfor (k = 0; k < data->nr_requests; k++) {\n+\t\tstrbuf_reset(&answer);\n+\n+\t\tret = fsmonitor_ipc__send_query(data->token, &answer);\n+\t\tif (ret < 0)\n+\t\t\tdata->sum_errors++;\n+\t\telse\n+\t\t\tdata->sum_successful++;\n+\t}\n+\n+\tstrbuf_release(&answer);\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Start a pool of client threads that will each send a series of\n+ * commands to the daemon.\n+ *\n+ * The goal is to overload the daemon with a sustained series of\n+ * concurrent requests.\n+ */\n+static int do_hammer(const char *token, int nr_threads, int nr_requests)\n+{\n+\tstruct hammer_thread_data *data = NULL;\n+\tint k;\n+\tint sum_join_errors = 0;\n+\tint sum_commands = 0;\n+\tint sum_errors = 0;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\tif (nr_threads < 1)\n+\t\tnr_threads = 1;\n+\tif (nr_requests < 1)\n+\t\tnr_requests = 1;\n+\n+\tCALLOC_ARRAY(data, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\t\tp->thread_nr = k;\n+\t\tp->nr_requests = nr_requests;\n+\t\tp->token = token;\n+\n+\t\tif (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {\n+\t\t\twarning(\"failed to create thread[%d] skipping remainder\", k);\n+\t\t\tnr_threads = k;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\n+\t\tif (pthread_join(p->pthread_id, NULL))\n+\t\t\tsum_join_errors++;\n+\t\tsum_commands += p->sum_successful;\n+\t\tsum_errors += p->sum_errors;\n+\t}\n+\n+\tfprintf(stderr, \"HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\\n\",\n+\t\tnr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);\n+\n+\tfree(data);\n+\n+\t/*\n+\t * Return an error if any of the _send_query requests failed.\n+\t * We don't care about thread create/join errors.\n+\t */\n+\treturn sum_errors > 0;\n+}\n+\n int cmd__fsmonitor_client(int argc, const char **argv)\n {\n \tconst char *subcmd;\n \tconst char *token = NULL;\n+\tint nr_threads = 1;\n+\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n \t\t\"test-tool fsmonitor-client query []\",\n \t\t\"test-tool fsmonitor-client flush\",\n+\t\t\"test-tool fsmonitor-client hammer [] [] []\",\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n \t\tOPT_STRING(0, \"token\", &token, \"token\",\n \t\t\t \"command token to send to the server\"),\n+\n+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, \"number of client threads\"),\n+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, \"number of requests per thread\"),\n+\n \t\tOPT_END()\n \t};\n \n@@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)\n \tif (!strcmp(subcmd, \"flush\"))\n \t\treturn !!do_send_flush();\n \n+\tif (!strcmp(subcmd, \"hammer\"))\n+\t\treturn !!do_hammer(token, nr_threads, nr_requests);\n+\n \tdie(\"Unhandled subcommand: '%s'\", subcmd);\n }\n #endif\n-- \ngitgitgadget\n\n"},{"id":"451893","messageId":"a8f0b2a5256932e02469051ef5aa43445af5fc03.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:37Z","receivedAt":"2022-03-22T18:23:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a worktree, so there is nothing for the\ndaemon watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 ++++\n builtin/update-index.c | 4 +++\n fsmonitor-settings.c | 60 +++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 12 ++++++++\n t/t7519-status-fsmonitor.sh | 23 ++++++++++++++\n 5 files changed, 105 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex a30ecf6cfac..edca8a667ab 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1450,6 +1450,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\tfsm_settings__set_ipc(the_repository);\n+\n+\tif (fsm_settings__error_if_incompatible(the_repository))\n+\t\treturn 1;\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 876112abb21..d29048f16f2 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1237,6 +1237,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\t\tif (fsm_settings__error_if_incompatible(the_repository))\n+\t\t\treturn 1;\n+\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 757d230d538..86c09bd35fe 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -9,9 +9,33 @@\n */\n struct fsmonitor_settings {\n \tenum fsmonitor_mode mode;\n+\tenum fsmonitor_reason reason;\n \tchar *hook_path;\n };\n \n+static void set_incompatible(struct repository *r,\n+\t\t\t enum fsmonitor_reason reason)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\ts->reason = reason;\n+}\n+\n+static int check_for_incompatible(struct repository *r)\n+{\n+\tif (!r->worktree) {\n+\t\t/*\n+\t\t * Bare repositories don't have a working directory and\n+\t\t * therefore have nothing to watch.\n+\t\t */\n+\t\tset_incompatible(r, FSMONITOR_REASON_BARE);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static void lookup_fsmonitor_settings(struct repository *r)\n {\n \tstruct fsmonitor_settings *s;\n@@ -23,6 +47,7 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \n \tCALLOC_ARRAY(s, 1);\n \ts->mode = FSMONITOR_MODE_DISABLED;\n+\ts->reason = FSMONITOR_REASON_OK;\n \n \tr->settings.fsmonitor = s;\n \n@@ -86,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r)\n \n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n@@ -97,6 +125,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path)\n \n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n@@ -110,5 +141,34 @@ void fsm_settings__set_disabled(struct repository *r)\n \tlookup_fsmonitor_settings(r);\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n+\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\n+\n+int fsm_settings__error_if_incompatible(struct repository *r)\n+{\n+\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n+\n+\tswitch (reason) {\n+\tcase FSMONITOR_REASON_OK:\n+\t\treturn 0;\n+\n+\tcase FSMONITOR_REASON_BARE:\n+\t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n+\t\t xgetcwd());\n+\t\treturn 1;\n+\t}\n+\n+\tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n+\t reason);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a4c5d7b4889..8654edf33d8 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -4,11 +4,20 @@\n struct repository;\n \n enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n \tFSMONITOR_MODE_DISABLED = 0,\n \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n };\n \n+/*\n+ * Incompatibility reasons.\n+ */\n+enum fsmonitor_reason {\n+\tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n+\tFSMONITOR_REASON_BARE,\n+};\n+\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n@@ -16,6 +25,9 @@ void fsm_settings__set_disabled(struct repository *r);\n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n+int fsm_settings__error_if_incompatible(struct repository *r);\n+\n struct fsmonitor_settings;\n \n #endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex a6308acf006..9a8e21c5608 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '\n \ttest $ret -ne 1\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n+\tgit init --bare bare-clone &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n+\tgit init --bare bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"451894","messageId":"e32a8a7ea7ab88ab4a202559eba83fb1ad61d639.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 05/27] fsmonitor-settings: stub in Win32-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:38Z","receivedAt":"2022-03-22T18:23:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend generic incompatibility checkout with platform-specific\nmechanism. Stub in Win32 version.\n\nIn the existing fsmonitor-settings code we have a way to mark\ntypes of repos as incompatible with fsmonitor (whether via the\nhook and IPC APIs). For example, we do this for bare repos,\nsince there are no files to watch.\n\nExtend this exclusion mechanism for platform-specific reasons.\nThis commit just creates the framework and adds a stub for Win32.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 +++++++++++++\n compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++\n config.mak.uname | 4 ++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n fsmonitor-settings.c | 12 ++++++++++++\n fsmonitor-settings.h | 13 +++++++++++++\n 6 files changed, 54 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n\ndiff --git a/Makefile b/Makefile\nindex daa21bed6c3..93604fe8ef7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -475,6 +475,11 @@ all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n+# If your platform has OS-specific ways to tell if a repo is incompatible with\n+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n+# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n+# that implements the `fsm_os_settings__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n+ifdef FSMONITOR_OS_SETTINGS\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE\n ifdef FSMONITOR_DAEMON_BACKEND\n \t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n endif\n+ifdef FSMONITOR_OS_SETTINGS\n+\t@echo FSMONITOR_OS_SETTINGS=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_OS_SETTINGS)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 501970902da..cf224768ad6 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex ee0d7257b77..16705da2000 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 86c09bd35fe..8ff55f8c3fd 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r)\n \t\treturn 1;\n \t}\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+\t{\n+\t\tenum fsmonitor_reason reason;\n+\n+\t\treason = fsm_os__incompatible(r);\n+\t\tif (reason != FSMONITOR_REASON_OK) {\n+\t\t\tset_incompatible(r, reason);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+#endif\n+\n \treturn 0;\n }\n \ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 8654edf33d8..4b35f051fb1 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -30,4 +30,17 @@ int fsm_settings__error_if_incompatible(struct repository *r);\n \n struct fsmonitor_settings;\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+/*\n+ * Ask platform-specific code whether the repository is incompatible\n+ * with fsmonitor (both hook and ipc modes). For example, if the working\n+ * directory is on a remote volume and mounted via a technology that does\n+ * not support notification events, then we should not pretend to watch it.\n+ *\n+ * fsm_os__* routines should considered private to fsm_settings__\n+ * routines.\n+ */\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);\n+#endif /* HAVE_FSMONITOR_OS_SETTINGS */\n+\n #endif /* FSMONITOR_SETTINGS_H */\n-- \ngitgitgadget\n\n"},{"id":"451895","messageId":"5546339d963caf37828da4b32deb896162cd5db9.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:39Z","receivedAt":"2022-03-22T18:23:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nVFS for Git virtual repositories are incompatible with FSMonitor.\n\nVFS for Git is a downstream fork of Git. It contains its own custom\nfile system watcher that is aware of the virtualization. If a working\ndirectory is being managed by VFS for Git, we should not try to watch\nit because we may get incomplete results.\n\nWe do not know anything about how VFS for Git works, but we do\nknow that VFS for Git working directories contain a well-defined\nconfig setting. If it is set, mark the working directory as\nincompatible.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 5 +++++\n fsmonitor-settings.h | 1 +\n t/t7519-status-fsmonitor.sh | 9 +++++++++\n 4 files changed, 41 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 7fce32a3c5b..ee78bba38e3 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -3,7 +3,33 @@\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n \n+/*\n+ * VFS for Git is incompatible with FSMonitor.\n+ *\n+ * Granted, core Git does not know anything about VFS for Git and we\n+ * shouldn't make assumptions about a downstream feature, but users\n+ * can install both versions. And this can lead to incorrect results\n+ * from core Git commands. So, without bringing in any of the VFS for\n+ * Git code, do a simple config test for a published config setting.\n+ * (We do not look at the various *_TEST_* environment variables.)\n+ */\n+static enum fsmonitor_reason check_vfs4git(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n+\t\treturn FSMONITOR_REASON_VFS4GIT;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_vfs4git(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 8ff55f8c3fd..1efb6e17a20 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -179,6 +179,11 @@ int fsm_settings__error_if_incompatible(struct repository *r)\n \t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n \t\t xgetcwd());\n \t\treturn 1;\n+\n+\tcase FSMONITOR_REASON_VFS4GIT:\n+\t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 4b35f051fb1..6361fcbf6b0 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -16,6 +16,7 @@ enum fsmonitor_mode {\n enum fsmonitor_reason {\n \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 9a8e21c5608..156895f9437 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n \tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n '\n \n+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone actual\" &&\n+\tgit init fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"virtual repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"451896","messageId":"1d2877efda0fce9103598ad92c2cf7d618bb72a0.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:40Z","receivedAt":"2022-03-22T18:23:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++\n config.mak.uname | 1 +\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 13 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cf224768ad6..cf911d141f2 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)\n \tifndef NO_PTHREADS\n \tifndef NO_UNIX_SOCKETS\n \tFSMONITOR_DAEMON_BACKEND = darwin\n+\tFSMONITOR_OS_SETTINGS = darwin\n \tendif\n \tendif\n \ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 16705da2000..b8f9f7a0388 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"451897","messageId":"5ca97f482d06d61ff912a9b509e4c2884f01c8b9.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 09/27] fsmonitor-settings: remote repos on Windows are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:42Z","receivedAt":"2022-03-22T18:24:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on Windows and mark them as\nincompatible with FSMonitor.\n\nWith this `git fsmonitor--daemon run` will error out with a message like it\ndoes for bare repos.\n\nClient commands, such as `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++\n 1 file changed, 102 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex ee78bba38e3..907655720bb 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * VFS for Git is incompatible with FSMonitor.\n@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)\n \treturn FSMONITOR_REASON_OK;\n }\n \n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type dictates whether notification events are available at\n+ * all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ *\n+ * Notes for testing:\n+ *\n+ * (a) Windows allows a network share to be mapped to a drive letter.\n+ * (This is the normal method to access it.)\n+ *\n+ * $ NET USE Z: \\\\server\\share\n+ * $ git -C Z:/repo status\n+ *\n+ * (b) Windows allows a network share to be referenced WITHOUT mapping\n+ * it to drive letter.\n+ *\n+ * $ NET USE \\\\server\\share\\dir\n+ * $ git -C //server/share/repo status\n+ *\n+ * (c) Windows allows \"SUBST\" to create a fake drive mapping to an\n+ * arbitrary path (which may be remote)\n+ *\n+ * $ SUBST Q: Z:\\repo\n+ * $ git -C Q:/ status\n+ *\n+ * (d) Windows allows a directory symlink to be created on a local\n+ * file system that points to a remote repo.\n+ *\n+ * $ mklink /d ./link //server/share/repo\n+ * $ git -C ./link status\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\twchar_t wpath[MAX_PATH];\n+\twchar_t wfullpath[MAX_PATH];\n+\tsize_t wlen;\n+\tUINT driveType;\n+\n+\t/*\n+\t * Do everything in wide chars because the drive letter might be\n+\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n+\t */\n+\tif (xutftowcs_path(wpath, r->worktree) < 0)\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\t/*\n+\t * GetDriveTypeW() requires a final slash. We assume that the\n+\t * worktree pathname points to an actual directory.\n+\t */\n+\twlen = wcslen(wpath);\n+\tif (wpath[wlen - 1] != L'\\\\' && wpath[wlen - 1] != L'/') {\n+\t\twpath[wlen++] = L'\\\\';\n+\t\twpath[wlen] = 0;\n+\t}\n+\n+\t/*\n+\t * Normalize the path. If nothing else, this converts forward\n+\t * slashes to backslashes. This is essential to get GetDriveTypeW()\n+\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n+\t */\n+\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\tdriveType = GetDriveTypeW(wfullpath);\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"DriveType '%s' L'%ls' (%u)\",\n+\t\t\t r->worktree, wfullpath, driveType);\n+\n+\tif (driveType == DRIVE_REMOTE) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"check_remote('%s') true\",\n+\t\t\t\t r->worktree);\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n-- \ngitgitgadget\n\n"},{"id":"451898","messageId":"301fff5296a9b343dc1dd41b6a8c1eafc570141e.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 16/27] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:49Z","receivedAt":"2022-03-22T18:24:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate another thread to watch over the daemon process and\nautomatically shut it down if necessary.\n\nThis commit creates the basic framework for a \"health\" thread\nto monitor the daemon and/or the file system. Later commits\nwill add platform-specific code to do the actual work.\n\nThe \"health\" thread is intended to monitor conditions that\nwould be difficult to track inside the IPC thread pool and/or\nthe file system listener threads. For example, when there are\nfile system events outside of the watched worktree root or if\nwe want to have an idle-timeout auto-shutdown feature.\n\nThis commit creates the health thread itself, defines the thread-proc\nand sets up the thread's event loop. It integrates this new thread\ninto the existing IPC and Listener thread models.\n\nThis commit defines the API to the platform-specific code where all of\nthe monitoring will actually happen.\n\nThe platform-specific code for MacOS is just stubs. Meaning that the\nhealth thread will immediately exit on MacOS, but that is OK and\nexpected. Future work can define MacOS-specific monitoring.\n\nThe platform-specific code for Windows sets up enough of the\nWaitForMultipleObjects() machinery to watch for system and/or custom\nevents. Currently, the set of wait handles only includes our custom\nshutdown event (sent from our other theads). Later commits in this\nseries will extend the set of wait handles to monitor other\nconditions.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 6 ++-\n builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n contrib/buildsystems/CMakeLists.txt | 2 +\n fsmonitor--daemon.h | 4 ++\n 7 files changed, 192 insertions(+), 2 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n\ndiff --git a/Makefile b/Makefile\nindex 93604fe8ef7..5f1623baadd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -472,8 +472,9 @@ all::\n #\n # If your platform supports a built-in fsmonitor backend, set\n # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n-# `compat/fsmonitor/fsm-listen-.c` that implements the\n-# `fsm_listen__*()` routines.\n+# `compat/fsmonitor/fsm-listen-.c` and\n+# `compat/fsmonitor/fsm-health-.c` files\n+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n # If your platform has OS-specific ways to tell if a repo is incompatible with\n # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n@@ -1982,6 +1983,7 @@ endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n ifdef FSMONITOR_OS_SETTINGS\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex af91bc2fb4c..522904abf36 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n@@ -1137,6 +1138,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+static void *fsm_health__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-health\");\n+\n+\tfsm_health__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -1175,6 +1188,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint health_started = 0;\n \tint listener_started = 0;\n \tint err = 0;\n \n@@ -1202,6 +1216,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t}\n \tlistener_started = 1;\n \n+\t/*\n+\t * Start the health thread to watch over our process.\n+\t */\n+\tif (pthread_create(&state->health_thread, NULL,\n+\t\t\t fsm_health__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\terr = error(_(\"could not start fsmonitor health thread\"));\n+\t\tgoto cleanup;\n+\t}\n+\thealth_started = 1;\n+\n \t/*\n \t * The daemon is now fully functional in background threads.\n \t * Our primary thread should now just wait while the threads\n@@ -1224,10 +1249,17 @@ cleanup:\n \t\tpthread_join(state->listener_thread, NULL);\n \t}\n \n+\tif (health_started) {\n+\t\tfsm_health__stop_async(state);\n+\t\tpthread_join(state->health_thread, NULL);\n+\t}\n+\n \tif (err)\n \t\treturn err;\n \tif (state->listen_error_code)\n \t\treturn state->listen_error_code;\n+\tif (state->health_error_code)\n+\t\treturn state->health_error_code;\n \treturn 0;\n }\n \n@@ -1243,6 +1275,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.listen_error_code = 0;\n+\tstate.health_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n@@ -1322,6 +1355,11 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\tif (fsm_health__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize health thread\"));\n+\t\tgoto done;\n+\t}\n+\n \t/*\n \t * CD out of the worktree root directory.\n \t *\n@@ -1345,6 +1383,7 @@ done:\n \tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n+\tfsm_health__dtor(&state);\n \n \tipc_server_free(state.ipc_server_data);\n \ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nnew file mode 100644\nindex 00000000000..b9f709e8548\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -0,0 +1,24 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nnew file mode 100644\nindex 00000000000..94b1d020f25\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsm_health_data\n+{\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hHandles[1]; /* the array does not own these handles */\n+#define HEALTH_SHUTDOWN 0\n+\tint nr_handles; /* number of active event handles */\n+};\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_handles++;\n+\n+\tstate->health_data = data;\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tif (!state || !state->health_data)\n+\t\treturn;\n+\n+\tdata = state->health_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\n+\tFREE_AND_NULL(state->health_data);\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\n+\tfor (;;) {\n+\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n+\t\t\t\t\t\t data->hHandles,\n+\t\t\t\t\t\t FALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->health_error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+clean_shutdown:\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n+}\ndiff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\nnew file mode 100644\nindex 00000000000..45547ba9380\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health.h\n@@ -0,0 +1,47 @@\n+#ifndef FSM_HEALTH_H\n+#define FSM_HEALTH_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor health thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the health thread.\n+ * This will be called from the main thread AFTER joining the thread.\n+ */\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to monitor the\n+ * health of the daemon process. This will run in the health thread.\n+ *\n+ * The health thread should call `ipc_server_stop_async()` if it needs\n+ * to cause a shutdown. (It should NOT do so if it receives a shutdown\n+ * shutdown signal.)\n+ *\n+ * It should set `state->health_error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_health__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the health thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_HEALTH_H */\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex b8f9f7a0388..16ace43d1c7 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 2c6fa1a5d91..2102a5c9ff5 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n+struct fsm_health_data; /* opaque platform-specific data for health thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n+\tpthread_t health_thread;\n \tpthread_mutex_t main_lock;\n \n \tstruct strbuf path_worktree_watch;\n@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n \tstruct hashmap cookies;\n \n \tint listen_error_code;\n+\tint health_error_code;\n \tstruct fsm_listen_data *listen_data;\n+\tstruct fsm_health_data *health_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"451899","messageId":"c6b5bdd25e435d4a942ac72f23bb81f4d59861e1.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 17/27] fsm-health-win32: add polling framework to monitor daemon health","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:50Z","receivedAt":"2022-03-22T18:24:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend the Windows version of the \"health\" thread to periodically\ninspect the system and shutdown if warranted.\n\nThis commit updates the thread's wait loop to use a timeout and\ndefines a (currently empty) table of functions to poll the system.\n\nA later commit will add functions to the table to actually\ninspect the system.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-\n 1 file changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 94b1d020f25..24fc612bf02 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -4,6 +4,24 @@\n #include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n+/*\n+ * Every minute wake up and test our health.\n+ */\n+#define WAIT_FREQ_MS (60 * 1000)\n+\n+/*\n+ * State machine states for each of the interval functions\n+ * used for polling our health.\n+ */\n+enum interval_fn_ctx {\n+\tCTX_INIT = 0,\n+\tCTX_TERM,\n+\tCTX_TIMER\n+};\n+\n+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx);\n+\n struct fsm_health_data\n {\n \tHANDLE hEventShutdown;\n@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n \tFREE_AND_NULL(state->health_data);\n }\n \n+/*\n+ * A table of the polling functions.\n+ */\n+static interval_fn *table[] = {\n+\tNULL, /* must be last */\n+};\n+\n+/*\n+ * Call all of the polling functions in the table.\n+ * Shortcut and return first error.\n+ *\n+ * Return 0 if all succeeded.\n+ */\n+static int call_all(struct fsmonitor_daemon_state *state,\n+\t\t enum interval_fn_ctx ctx)\n+{\n+\tint k;\n+\n+\tfor (k = 0; table[k]; k++) {\n+\t\tint r = table[k](state, ctx);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n+\tint r;\n+\n+\tr = call_all(state, CTX_INIT);\n+\tif (r < 0)\n+\t\tgoto force_error_stop;\n+\tif (r > 0)\n+\t\tgoto force_shutdown;\n \n \tfor (;;) {\n \t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n \t\t\t\t\t\t data->hHandles,\n-\t\t\t\t\t\t FALSE, INFINITE);\n+\t\t\t\t\t\t FALSE, WAIT_FREQ_MS);\n \n \t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n \t\t\tgoto clean_shutdown;\n \n+\t\tif (dwWait == WAIT_TIMEOUT) {\n+\t\t\tr = call_all(state, CTX_TIMER);\n+\t\t\tif (r < 0)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (r > 0)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\terror(_(\"health thread wait failed [GLE %ld]\"),\n \t\t GetLastError());\n \t\tgoto force_error_stop;\n@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)\n \n force_error_stop:\n \tstate->health_error_code = -1;\n+force_shutdown:\n \tipc_server_stop_async(state->ipc_server_data);\n clean_shutdown:\n+\tcall_all(state, CTX_TERM);\n \treturn;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"451900","messageId":"01c1a38c46243666c8af0556d5d7588cf4203ada.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 19/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:52Z","receivedAt":"2022-03-22T18:24:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the listener thread to shutdown the daemon if the spelling of the\nworktree root directory changes.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 07113205a61..83d38e8ac6c 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tstrbuf_release(&msg);\n }\n \n+static int ef_is_root_changed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagRootChanged);\n+}\n+\n static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n {\n \treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n@@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_is_root_changed(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * The spelling of the pathname of the root directory\n+\t\t\t * has changed. This includes the name of the root\n+\t\t\t * directory itself or of any parent directory in the\n+\t\t\t * path.\n+\t\t\t *\n+\t\t\t * (There may be other conditions that throw this,\n+\t\t\t * but I couldn't find any information on it.)\n+\t\t\t *\n+\t\t\t * Force a shutdown now and avoid things getting\n+\t\t\t * out of sync. The Unix domain socket is inside\n+\t\t\t * the .git directory and a spelling change will make\n+\t\t\t * it hard for clients to rendezvous with us.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"event: root changed\");\n+\t\t\tgoto force_shutdown;\n+\t\t}\n+\n \t\tif (ef_ignore_xattr(event_flags[k])) {\n \t\t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n-- \ngitgitgadget\n\n"},{"id":"451901","messageId":"0f0a5b5ca1630985b9a91fa0e6c5fc951561679c.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 20/27] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:53Z","receivedAt":"2022-03-22T18:24:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to perform binary search over the cache-entries for a directory\nnotification and then linearly scan forward to find the immediate children.\n\nPreviously, when the FSMonitor reported a modified directory Git would\nperform a linear search on the entire cache-entry array for all\nentries matching that directory prefix and invalidate them. Since the\ncache-entry array is already sorted, we can use a binary search to\nfind the first matching entry and then only linearly walk forward and\ninvalidate entries until the prefix changes.\n\nAlso, the original code would invalidate anything having the same\ndirectory prefix. Since a directory event should only be received for\nitems that are immediately within the directory (and not within\nsub-directories of it), only invalidate those entries and not the\nwhole subtree.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 54 insertions(+), 17 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 292a6742b4f..e1229c289cf 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n-\tif (name[len - 1] == '/') {\n+\tint pos = index_name_pos(istate, name, len);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n+\t\t\t name, pos);\n \n+\tif (name[len - 1] == '/') {\n \t\t/*\n-\t\t * TODO We should binary search to find the first path with\n-\t\t * TODO this directory prefix. Then linearly update entries\n-\t\t * TODO while the prefix matches. Taking care to search without\n-\t\t * TODO the trailing slash -- because '/' sorts after a few\n-\t\t * TODO interesting special chars, like '.' and ' '.\n+\t\t * The daemon can decorate directory events, such as\n+\t\t * moves or renames, with a trailing slash if the OS\n+\t\t * FS Event contains sufficient information, such as\n+\t\t * MacOS.\n+\t\t *\n+\t\t * Use this to invalidate the entire cone under that\n+\t\t * directory.\n+\t\t *\n+\t\t * We do not expect an exact match because the index\n+\t\t * does not normally contain directory entries, so we\n+\t\t * start at the insertion point and scan.\n \t\t */\n+\t\tif (pos < 0)\n+\t\t\tpos = -pos - 1;\n \n \t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n-\t\t\t starts_with(istate->cache[i]->name, name))\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n-\t\t/* Need to remove the / from the path for the untracked cache */\n+\n+\t\t/*\n+\t\t * We need to remove the traling \"/\" from the path\n+\t\t * for the untracked cache.\n+\t\t */\n \t\tname[len - 1] = '\\0';\n+\t} else if (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\tint pos = index_name_pos(istate, name, strlen(name));\n-\n-\t\tif (pos >= 0) {\n-\t\t\tstruct cache_entry *ce = istate->cache[pos];\n-\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n \t}\n \n@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n \t * as it could be a new untracked file.\n \t */\n-\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"451902","messageId":"13d11713a869f62677ca2f281cc527020e616eec.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 18/27] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:51Z","receivedAt":"2022-03-22T18:24:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nForce shutdown fsmonitor daemon if the worktree root directory\nis moved, renamed, or deleted.\n\nUse Windows low-level GetFileInformationByHandle() to get and\ncompare the Windows system unique ID for the directory with a\ncached version when we started up. This lets us detect the\ncase where someone renames the directory that we are watching\nand then creates a new directory with the original pathname.\n\nThis is important because we are listening to a named pipe for\nrequests and they are stored in the Named Pipe File System (NPFS)\nwhich a kernel-resident pseudo filesystem not associated with\nthe actual NTFS directory.\n\nFor example, if the daemon was watching \"~/foo/\", it would have\na directory-watch handle on that directory and a named-pipe\nhandle for \"//./pipe/...foo\". Moving the directory to \"~/bar/\"\ndoes not invalidate the directory handle. (So the daemon would\nactually be watching \"~/bar\" but listening on \"//./pipe/...foo\".\nIf the user then does \"git init ~/foo\" and causes another daemon\nto start, the first daemon will still have ownership of the pipe\nand the second daemon instance will fail to start. \"git status\"\nclients in \"~/foo\" will ask \"//./pipe/...foo\" about changes and\nthe first daemon instance will tell them about \"~/bar\".\n\nThis commit causes the first daemon to shutdown if the system unique\nID for \"~/foo\" changes (changes from what it was when the daemon\nstarted). Shutdown occurs after a periodic poll. After the\nfirst daemon exits and releases the lock on the named pipe,\nsubsequent Git commands may cause another daemon to be started\non \"~/foo\". Similarly, a subsequent Git command may cause another\ndaemon to be started on \"~/bar\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++\n 1 file changed, 143 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 24fc612bf02..2ea08c1d4e8 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -29,8 +29,150 @@ struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n \tint nr_handles; /* number of active event handles */\n+\n+\tstruct wt_moved\n+\t{\n+\t\twchar_t wpath[MAX_PATH + 1];\n+\t\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\t} wt_moved;\n };\n \n+/*\n+ * Lookup the system unique ID for the path. This is as close as\n+ * we get to an inode number, but this also contains volume info,\n+ * so it is a little stronger.\n+ */\n+static int lookup_bhfi(wchar_t *wpath,\n+\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n+{\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileW(wpath, desired_access, share_mode, NULL,\n+\t\t\t OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] health thread could not open '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\treturn -1;\n+\t}\n+\n+\tif (!GetFileInformationByHandle(hDir, bhfi)) {\n+\t\terror(_(\"[GLE %ld] health thread getting BHFI for '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\tCloseHandle(hDir);\n+\t\treturn -1;\n+\t}\n+\n+\tCloseHandle(hDir);\n+\treturn 0;\n+}\n+\n+/*\n+ * Compare the relevant fields from two system unique IDs.\n+ * We use this to see if two different handles to the same\n+ * path actually refer to the same *instance* of the file\n+ * or directory.\n+ */\n+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n+\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n+{\n+\treturn (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&\n+\t\tbhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&\n+\t\tbhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);\n+}\n+\n+/*\n+ * Shutdown if the original worktree root directory been deleted,\n+ * moved, or renamed?\n+ *\n+ * Since the main thread did a \"chdir(getenv($HOME))\" and our CWD\n+ * is not in the worktree root directory and because the listener\n+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible\n+ * for the root directory to be moved or deleted while we are still\n+ * watching it. We want to detect that here and force a shutdown.\n+ *\n+ * Granted, a delete MAY cause some operations to fail, such as\n+ * GetOverlappedResult(), but it is not guaranteed. And because\n+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the\n+ * directory, not changes *ON* the directory, our watch will not\n+ * receive a delete event for it.\n+ *\n+ * A move/rename of the worktree root will also not generate an event.\n+ * And since the listener thread already has an open handle, it may\n+ * continue to receive events for events within the directory.\n+ * However, the pathname of the named-pipe was constructed using the\n+ * original location of the worktree root. (Remember named-pipes are\n+ * stored in the NPFS and not in the actual file system.) Clients\n+ * trying to talk to the worktree after the move/rename will not\n+ * reach our daemon process, since we're still listening on the\n+ * pipe with original path.\n+ *\n+ * Furthermore, if the user does something like:\n+ *\n+ * $ mv repo repo.old\n+ * $ git init repo\n+ *\n+ * A new daemon cannot be started in the new instance of \"repo\"\n+ * because the named-pipe is still being used by the daemon on\n+ * the original instance.\n+ *\n+ * So, detect move/rename/delete and shutdown. This should also\n+ * handle unsafe drive removal.\n+ *\n+ * We use the file system unique ID to distinguish the original\n+ * directory instance from a new instance and force a shutdown\n+ * if the unique ID changes.\n+ *\n+ * Since a worktree move/rename/delete/unmount doesn't happen\n+ * that often (and we can't get an immediate event anyway), we\n+ * use a timeout and periodically poll it.\n+ */\n+static int has_worktree_moved(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\tint r;\n+\n+\tswitch (ctx) {\n+\tcase CTX_TERM:\n+\t\treturn 0;\n+\n+\tcase CTX_INIT:\n+\t\tif (xutftowcs_path(data->wt_moved.wpath,\n+\t\t\t\t state->path_worktree_watch.buf) < 0) {\n+\t\t\terror(_(\"could not convert to wide characters: '%s'\"),\n+\t\t\t state->path_worktree_watch.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/*\n+\t\t * On the first call we lookup the unique sequence ID for\n+\t\t * the worktree root directory.\n+\t\t */\n+\t\treturn lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);\n+\n+\tcase CTX_TIMER:\n+\t\tr = lookup_bhfi(data->wt_moved.wpath, &bhfi);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t\tif (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {\n+\t\t\terror(_(\"BHFI changed '%ls'\"), data->wt_moved.wpath);\n+\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tdie(_(\"unhandled case in 'has_worktree_moved': %d\"),\n+\t\t (int)ctx);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+\n int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data;\n@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n * A table of the polling functions.\n */\n static interval_fn *table[] = {\n+\thas_worktree_moved,\n \tNULL, /* must be last */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"451903","messageId":"e78eb20c1bfeaceb58b5c3ad6d8d8a892c04394f.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 15/27] fsmonitor--daemon: rename listener thread related variables","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:48Z","receivedAt":"2022-03-22T18:24:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRename platform-specific listener thread related variables\nand data types as we prepare to add another backend thread\ntype.\n\n[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`\n[] `state->backend_data` becomes `state->listen_data`\n[] `state->error_code` becomes `state->listen_error_code`\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 +++---\n compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------\n compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++-------------\n compat/fsmonitor/fsm-listen.h | 2 +-\n fsmonitor--daemon.h | 6 +++---\n 5 files changed, 36 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 8033916aa63..af91bc2fb4c 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1226,8 +1226,8 @@ cleanup:\n \n \tif (err)\n \t\treturn err;\n-\tif (state->error_code)\n-\t\treturn state->error_code;\n+\tif (state->listen_error_code)\n+\t\treturn state->listen_error_code;\n \treturn 0;\n }\n \n@@ -1242,7 +1242,7 @@ static int fsmonitor_run_daemon(void)\n \thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n-\tstate.error_code = 0;\n+\tstate.listen_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 14105f45c18..07113205a61 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -27,7 +27,7 @@\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tCFStringRef cfsr_worktree_path;\n \tCFStringRef cfsr_gitdir_path;\n@@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t const FSEventStreamEventId event_ids[])\n {\n \tstruct fsmonitor_daemon_state *state = ctx;\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tchar **paths = (char **)event_paths;\n \tstruct fsmonitor_batch *batch = NULL;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tNULL,\n \t\tNULL\n \t};\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \tconst void *dir_array[2];\n \n \tCALLOC_ARRAY(data, 1);\n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \n \tdata->cfsr_worktree_path = CFStringCreateWithCString(\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n@@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n \terror(_(\"Unable to create FSEventStream.\"));\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tif (data->stream) {\n \t\tif (data->stream_started)\n@@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n \t\tFSEventStreamRelease(data->stream);\n \t}\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \tdata->shutdown_style = SHUTDOWN_EVENT;\n \n \tCFRunLoopStop(data->rl);\n@@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tdata->rl = CFRunLoopGetCurrent();\n \n@@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \n \tswitch (data->shutdown_style) {\n \tcase FORCE_ERROR_STOP:\n-\t\tstate->error_code = -1;\n+\t\tstate->listen_error_code = -1;\n \t\t/* fall thru */\n \tcase FORCE_SHUTDOWN:\n \t\tipc_server_stop_async(state->ipc_server_data);\n@@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \treturn;\n \n force_error_stop_without_loop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \tipc_server_stop_async(state->ipc_server_data);\n \treturn;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex c43d92b9620..be2d93f47b2 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -54,7 +54,7 @@ struct one_watch\n \twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tstruct one_watch *watch_worktree;\n \tstruct one_watch *watch_gitdir;\n@@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname(\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+\tSetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);\n }\n \n static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n@@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch)\n \tfree(watch);\n }\n \n-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+static int start_rdcw_watch(struct fsm_listen_data *data,\n \t\t\t struct one_watch *watch)\n {\n \tDWORD dwNotifyFilter =\n@@ -516,7 +516,7 @@ static int process_1_worktree_event(\n */\n static int process_worktree_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_worktree;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -646,7 +646,7 @@ force_shutdown:\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_gitdir;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -704,11 +704,11 @@ skip_this_path:\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tDWORD dwWait;\n \tint result;\n \n-\tstate->error_code = 0;\n+\tstate->listen_error_code = 0;\n \n \tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n \t\tgoto force_error_stop;\n@@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \t}\n \n force_error_stop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \n force_shutdown:\n \t/*\n@@ -790,7 +790,7 @@ clean_shutdown:\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \treturn 0;\n \n failed:\n@@ -836,16 +836,16 @@ failed:\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tCloseHandle(data->hEventShutdown);\n \tdestroy_watch(data->watch_worktree);\n \tdestroy_watch(data->watch_gitdir);\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nindex f0539349baf..41650bf8972 100644\n--- a/compat/fsmonitor/fsm-listen.h\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n * do so if the listener thread receives a normal shutdown signal from\n * the IPC layer.)\n *\n- * It should set `state->error_code` to -1 if the daemon should exit\n+ * It should set `state->listen_error_code` to -1 if the daemon should exit\n * with an error.\n */\n void fsm_listen__loop(struct fsmonitor_daemon_state *state);\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 223c2131b58..2c6fa1a5d91 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n */\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {\n \tint cookie_seq;\n \tstruct hashmap cookies;\n \n-\tint error_code;\n-\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\tint listen_error_code;\n+\tstruct fsm_listen_data *listen_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"451904","messageId":"087af5dfb6315849b0079d8f6df9bc2da7682653.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 14/27] fsmonitor--daemon: prepare for adding health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:47Z","receivedAt":"2022-03-22T18:24:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor daemon thread startup to make it easier to start\na third thread class to monitor the health of the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex f17d5ed84c8..8033916aa63 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1175,6 +1175,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint listener_started = 0;\n+\tint err = 0;\n \n \t/*\n \t * Start the IPC thread pool before the we've started the file\n@@ -1195,15 +1197,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \tif (pthread_create(&state->listener_thread, NULL,\n \t\t\t fsm_listen__thread_proc, state) < 0) {\n \t\tipc_server_stop_async(state->ipc_server_data);\n-\t\tipc_server_await(state->ipc_server_data);\n-\n-\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t\terr = error(_(\"could not start fsmonitor listener thread\"));\n+\t\tgoto cleanup;\n \t}\n+\tlistener_started = 1;\n \n \t/*\n \t * The daemon is now fully functional in background threads.\n+\t * Our primary thread should now just wait while the threads\n+\t * do all the work.\n+\t */\n+cleanup:\n+\t/*\n \t * Wait for the IPC thread pool to shutdown (whether by client\n-\t * request or from filesystem activity).\n+\t * request, from filesystem activity, or an error).\n \t */\n \tipc_server_await(state->ipc_server_data);\n \n@@ -1212,10 +1219,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * event from the IPC thread pool, but it doesn't hurt to tell\n \t * it again. And wait for it to shutdown.\n \t */\n-\tfsm_listen__stop_async(state);\n-\tpthread_join(state->listener_thread, NULL);\n+\tif (listener_started) {\n+\t\tfsm_listen__stop_async(state);\n+\t\tpthread_join(state->listener_thread, NULL);\n+\t}\n \n-\treturn state->error_code;\n+\tif (err)\n+\t\treturn err;\n+\tif (state->error_code)\n+\t\treturn state->error_code;\n+\treturn 0;\n }\n \n static int fsmonitor_run_daemon(void)\n-- \ngitgitgadget\n\n"},{"id":"451905","messageId":"6715143724586bd53de5070c61dba30dd8fe2bf8.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 10/27] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:43Z","receivedAt":"2022-03-22T18:24:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nOn MacOS mark repos on NTFS or FAT32 volumes as incompatible.\n\nThe builtin FSMonitor used Unix domain sockets on MacOS for IPC\nwith clients. These sockets are kept in the .git directory.\nUnix sockets are not supported by NTFS and FAT32, so the daemon\ncannot start up.\n\nTest for this during our compatibility checking so that client\ncommands do not keep trying to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---\n fsmonitor-settings.c | 5 +++++\n fsmonitor-settings.h | 1 +\n 3 files changed, 23 insertions(+), 3 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex fdd762bf79d..efc732c0f31 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -7,7 +7,7 @@\n #include \n \n /*\n- * Remote working directories are problematic for FSMonitor.\n+ * [1] Remote working directories are problematic for FSMonitor.\n *\n * The underlying file system on the server machine and/or the remote\n * mount type (NFS, SAMBA, etc.) dictates whether notification events\n@@ -40,8 +40,16 @@\n *\n * So (for now at least), mark remote working directories as\n * incompatible.\n+ *\n+ *\n+ * [2] FAT32 and NTFS working directories are problematic too.\n+ *\n+ * The builtin FSMonitor uses a Unix domain socket in the .git\n+ * directory for IPC. These Windows drive formats do not support\n+ * Unix domain sockets, so mark them as incompatible for the daemon.\n+ *\n */\n-static enum fsmonitor_reason check_remote(struct repository *r)\n+static enum fsmonitor_reason check_volume(struct repository *r)\n {\n \tstruct statfs fs;\n \n@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)\n \tif (!(fs.f_flags & MNT_LOCAL))\n \t\treturn FSMONITOR_REASON_REMOTE;\n \n+\tif (!strcmp(fs.f_fstypename, \"msdos\")) /* aka FAT32 */\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n+\tif (!strcmp(fs.f_fstypename, \"ntfs\"))\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \n@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n \n-\treason = check_remote(r);\n+\treason = check_volume(r);\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 0a1811ff004..60d5eaee497 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -194,6 +194,11 @@ int fsm_settings__error_if_incompatible(struct repository *r)\n \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t r->worktree);\n \t\treturn 1;\n+\n+\tcase FSMONITOR_REASON_NOSOCKETS:\n+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 34391b583b3..23d5676c8c8 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -19,6 +19,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n \tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n+\tFSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n-- \ngitgitgadget\n\n"},{"id":"451906","messageId":"06d7f18676d311b2ae246f078239739c5d1aa787.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 08/27] fsmonitor-settings: remote repos on macOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:41Z","receivedAt":"2022-03-22T18:24:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on macOS and mark them as\nincompatible with FSMonitor.\n\nWith this, `git fsmonitor--daemon run` will error out with a message\nlike it does for bare repos.\n\nClient commands, like `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 10 ++++\n fsmonitor-settings.h | 2 +\n 3 files changed, 78 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex 7fce32a3c5b..fdd762bf79d 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -2,8 +2,74 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n+#include \n+#include \n+\n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type (NFS, SAMBA, etc.) dictates whether notification events\n+ * are available at all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * For the builtin FSMonitor, we create the Unix domain socket for the\n+ * IPC in the .git directory. If the working directory is remote,\n+ * then the socket will be created on the remote file system. This\n+ * can fail if the remote file system does not support UDS file types\n+ * (e.g. smbfs to a Windows server) or if the remote kernel does not\n+ * allow a non-local process to bind() the socket. (These problems\n+ * could be fixed by moving the UDS out of the .git directory and to a\n+ * well-known local directory on the client machine, but care should\n+ * be taken to ensure that $HOME is actually local and not a managed\n+ * file share.)\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\tstruct statfs fs;\n+\n+\tif (statfs(r->worktree, &fs) == -1) {\n+\t\tint saved_errno = errno;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n+\t\t\t\t r->worktree, strerror(saved_errno));\n+\t\terrno = saved_errno;\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n+\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n+\n+\tif (!(fs.f_flags & MNT_LOCAL))\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 1efb6e17a20..0a1811ff004 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -180,6 +180,16 @@ int fsm_settings__error_if_incompatible(struct repository *r)\n \t\t xgetcwd());\n \t\treturn 1;\n \n+\tcase FSMONITOR_REASON_ERROR:\n+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n+\n+\tcase FSMONITOR_REASON_REMOTE:\n+\t\terror(_(\"remote repository '%s' is incompatible with fsmonitor\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n+\n \tcase FSMONITOR_REASON_VFS4GIT:\n \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t r->worktree);\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 6361fcbf6b0..34391b583b3 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -16,6 +16,8 @@ enum fsmonitor_mode {\n enum fsmonitor_reason {\n \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n+\tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"451907","messageId":"35c77b854bdb5ea3a339c9a57865d4d0debc4df5.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 12/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:45Z","receivedAt":"2022-03-22T18:24:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIgnore FSEvents resulting from `xattr` changes. Git does not care about\nxattr's or changes to xattr's, so don't waste time collecting these\nevents in the daemon nor transmitting them to clients.\n\nVarious security tools add xattrs to files and/or directories, such as\nto mark them as having been downloaded. We should ignore these events\nsince it doesn't affect the content of the file/directory or the normal\nmeta-data that Git cares about.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 0741fe834c3..14105f45c18 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tif (flag & kFSEventStreamEventFlagItemCloned)\n \t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n \n-\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n \t\t\t path, flag, msg.buf);\n \n \tstrbuf_release(&msg);\n@@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)\n \t\tef & kFSEventStreamEventFlagUserDropped);\n }\n \n+/*\n+ * If an `xattr` change is the only reason we received this event,\n+ * then silently ignore it. Git doesn't care about xattr's. We\n+ * have to be careful here because the kernel can combine multiple\n+ * events for a single path. And because events always have certain\n+ * bits set, such as `ItemIsFile` or `ItemIsDir`.\n+ *\n+ * Return 1 if we should ignore it.\n+ */\n+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n+{\n+\tstatic const FSEventStreamEventFlags mask =\n+\t\tkFSEventStreamEventFlagItemChangeOwner |\n+\t\tkFSEventStreamEventFlagItemCreated |\n+\t\tkFSEventStreamEventFlagItemFinderInfoMod |\n+\t\tkFSEventStreamEventFlagItemInodeMetaMod |\n+\t\tkFSEventStreamEventFlagItemModified |\n+\t\tkFSEventStreamEventFlagItemRemoved |\n+\t\tkFSEventStreamEventFlagItemRenamed |\n+\t\tkFSEventStreamEventFlagItemXattrMod |\n+\t\tkFSEventStreamEventFlagItemCloned;\n+\n+\treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n+}\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_ignore_xattr(event_flags[k])) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n+\t\t\t\t\t path_k, event_flags[k]);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n \n \t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-- \ngitgitgadget\n\n"},{"id":"451908","messageId":"ed1f7231309e5761aa0a434ae02458ce2b559eb2.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 11/27] unpack-trees: initialize fsmonitor_has_run_once in o->result","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:44Z","receivedAt":"2022-03-22T18:24:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInitialize `o->result.fsmonitor_has_run_once` based upon value\nin `o->src_index->fsmonitor_has_run_once` to prevent a second\nfsmonitor query during the tree traversal and possibly getting\na skewed view of the working directory.\n\nThe checkout code has already talked to the fsmonitor and the\ntraversal is updating the index as it traverses, so there is\nno need to query the fsmonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n unpack-trees.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..888cff81f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \to->result.fsmonitor_last_update =\n \t\txstrdup_or_null(o->src_index->fsmonitor_last_update);\n+\to->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;\n \n \t/*\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n-- \ngitgitgadget\n\n"},{"id":"451909","messageId":"a5affb359c4dee61b5cc2dc1e18c7bc49bc58131.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 13/27] fsmonitor--daemon: cd out of worktree root","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:46Z","receivedAt":"2022-03-22T18:24:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the fsmonitor--daemon to CD outside of the worktree\nbefore starting up.\n\nThe common Git startup mechanism causes the CWD of the daemon process\nto be in the root of the worktree. On Windows, this causes the daemon\nprocess to hold a locked handle on the CWD and prevents other\nprocesses from moving or deleting the worktree while the daemon is\nrunning.\n\nCD to HOME before entering main event loops.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++--\n compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------\n fsmonitor--daemon.h | 1 +\n 3 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex edca8a667ab..f17d5ed84c8 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1182,11 +1182,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * before we need it.\n \t */\n \tif (ipc_server_run_async(&state->ipc_server_data,\n-\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t state->path_ipc.buf, &ipc_opts,\n \t\t\t\t handle_client, state))\n \t\treturn error_errno(\n \t\t\t_(\"could not start IPC thread pool on '%s'\"),\n-\t\t\tfsmonitor_ipc__get_path());\n+\t\t\tstate->path_ipc.buf);\n \n \t/*\n \t * Start the fsmonitor listener thread to collect filesystem\n@@ -1221,6 +1221,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n static int fsmonitor_run_daemon(void)\n {\n \tstruct fsmonitor_daemon_state state;\n+\tconst char *home;\n \tint err;\n \n \tmemset(&state, 0, sizeof(state));\n@@ -1290,6 +1291,15 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_addch(&state.path_cookie_prefix, '/');\n \n+\t/*\n+\t * We create a named-pipe or unix domain socket inside of the\n+\t * \".git\" directory. (Well, on Windows, we base our named\n+\t * pipe in the NPFS on the absolute path of the git\n+\t * directory.)\n+\t */\n+\tstrbuf_init(&state.path_ipc, 0);\n+\tstrbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1299,6 +1309,23 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\t/*\n+\t * CD out of the worktree root directory.\n+\t *\n+\t * The common Git startup mechanism causes our CWD to be the\n+\t * root of the worktree. On Windows, this causes our process\n+\t * to hold a locked handle on the CWD. This prevents the\n+\t * worktree from being moved or deleted while the daemon is\n+\t * running.\n+\t *\n+\t * We assume that our FS and IPC listener threads have either\n+\t * opened all of the handles that they need or will do\n+\t * everything using absolute paths.\n+\t */\n+\thome = getenv(\"HOME\");\n+\tif (home && *home && chdir(home))\n+\t\tdie_errno(_(\"could not cd home '%s'\"), home);\n+\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n@@ -1311,6 +1338,7 @@ done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n \tstrbuf_release(&state.path_cookie_prefix);\n+\tstrbuf_release(&state.path_ipc);\n \n \treturn err;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 3f1b68267bd..c43d92b9620 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch)\n \t}\n \n \t/*\n-\t * NEEDSWORK: If an external is deleted, the above\n-\t * returns an error. I'm not sure that there's anything that\n-\t * we can do here other than failing -- the /.git\n-\t * link file would be broken anyway. We might try to check\n-\t * for that and return a better error message, but I'm not\n-\t * sure it is worth it.\n+\t * GetOverlappedResult() fails if the watched directory is\n+\t * deleted while we were waiting for an overlapped IO to\n+\t * complete. The documentation did not list specific errors,\n+\t * but I observed ERROR_ACCESS_DENIED (0x05) errors during\n+\t * testing.\n+\t *\n+\t * Note that we only get notificaiton events for events\n+\t * *within* the directory, not *on* the directory itself.\n+\t * (These might be properies of the parent directory, for\n+\t * example).\n+\t *\n+\t * NEEDSWORK: We might try to check for the deleted directory\n+\t * case and return a better error message, but I'm not sure it\n+\t * is worth it.\n+\t *\n+\t * Shutdown if we get any error.\n \t */\n \n \terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex bd09fffc176..223c2131b58 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n+\tstruct strbuf path_ipc;\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"451910","messageId":"d8218d197adb18d2034d21a0881cf05442c32680.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 21/27] t7527: FSMonitor tests for directory moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:54Z","receivedAt":"2022-03-22T18:24:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate unit tests to move a directory. Verify that `git status`\ngives the same result with and without FSMonitor enabled.\n\nNEEDSWORK: This test exposes a bug in the untracked-cache on\nWindows when FSMonitor is disabled. These are commented out\nfor the moment.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 39 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 429029fcadd..bc66d8285a3 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -281,6 +281,16 @@ test_expect_success 'setup' '\n \ttrace*\n \tEOF\n \n+\tmkdir -p T1/T2/T3/T4 &&\n+\techo 1 >T1/F1 &&\n+\techo 1 >T1/T2/F1 &&\n+\techo 1 >T1/T2/T3/F1 &&\n+\techo 1 >T1/T2/T3/T4/F1 &&\n+\techo 2 >T1/F2 &&\n+\techo 2 >T1/T2/F2 &&\n+\techo 2 >T1/T2/T3/F2 &&\n+\techo 2 >T1/T2/T3/T4/F2 &&\n+\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n@@ -363,6 +373,19 @@ directory_to_file () {\n \techo 1 >dir1\n }\n \n+move_directory_contents_deeper() {\n+\tmkdir T1/_new_ &&\n+\tmv T1/[A-Z]* T1/_new_\n+}\n+\n+move_directory_up() {\n+\tmv T1/T2/T3 T1\n+}\n+\n+move_directory() {\n+\tmv T1/T2/T3 T1/T2/NewT3\n+}\n+\n # The next few test cases confirm that our fsmonitor daemon sees each type\n # of OS filesystem notification that we care about. At this layer we just\n # ensure we are getting the OS notifications and do not try to confirm what\n@@ -645,6 +668,22 @@ do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n+\t\t# NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor\n+\t\t# is DISABLED. Turn off a few test that cause it problems until\n+\t\t# we can debug it.\n+\t\t#\n+\t\ttry_moves=\"true\"\n+\t\ttest_have_prereq UNTRACKED_CACHE,WINDOWS && \\\n+\t\t\ttest $uc_val = true && \\\n+\t\t\ttest $fsm_val = false && \\\n+\t\t\ttry_moves=\"false\"\n+\t\tif test $try_moves = true\n+\t\tthen\n+\t\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n+\t\t\tmatrix_try $uc_val $fsm_val move_directory_up\n+\t\t\tmatrix_try $uc_val $fsm_val move_directory\n+\t\tfi\n+\n \t\tif test $fsm_val = true\n \t\tthen\n \t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n-- \ngitgitgadget\n\n"},{"id":"451911","messageId":"79da369dcce69f14dab19116e23aaec36ca4d92a.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 22/27] t/perf/p7527: add perf test for builtin FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:55Z","receivedAt":"2022-03-22T18:24:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++\n 1 file changed, 257 insertions(+)\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\ndiff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..9338b9ea008\n--- /dev/null\n+++ b/t/perf/p7527-builtin-fsmonitor.sh\n@@ -0,0 +1,257 @@\n+#!/bin/sh\n+\n+test_description=\"Perf test for the builtin FSMonitor\"\n+\n+. ./perf-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+# Lie to perf-lib and ask for a new empty repo and avoid\n+# the complaints about GIT_PERF_REPO not being big enough\n+# the perf hit when GIT_PERF_LARGE_REPO is copied into\n+# the trash directory.\n+#\n+# NEEDSWORK: It would be nice if perf-lib had an option to\n+# \"borrow\" an existing large repo (especially for gigantic\n+# monorepos) and use it in-place. For now, fake it here.\n+#\n+test_perf_fresh_repo\n+\n+\n+# Use a generated synthetic monorepo. If it doesn't exist, we will\n+# generate it. If it does exist, we will put it in a known state\n+# before we start our timings.\n+#\n+PARAM_D=5\n+PARAM_W=10\n+PARAM_F=9\n+\n+PARAMS=\"$PARAM_D\".\"$PARAM_W\".\"$PARAM_F\"\n+\n+BALLAST_BR=p0006-ballast\n+export BALLAST_BR\n+\n+TMP_BR=tmp_br\n+export TMP_BR\n+\n+REPO=../repos/gen-many-files-\"$PARAMS\".git\n+export REPO\n+\n+if ! test -d $REPO\n+then\n+\t(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)\n+fi\n+\n+\n+enable_uc () {\n+\tgit -C $REPO config core.untrackedcache true\n+\tgit -C $REPO update-index --untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+disable_uc () {\n+\tgit -C $REPO config core.untrackedcache false\n+\tgit -C $REPO update-index --no-untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+start_fsm () {\n+\tgit -C $REPO fsmonitor--daemon start\n+\tgit -C $REPO fsmonitor--daemon status\n+\tgit -C $REPO config core.fsmonitor true\n+\tgit -C $REPO update-index --fsmonitor\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+stop_fsm () {\n+\tgit -C $REPO config --unset core.fsmonitor\n+\tgit -C $REPO update-index --no-fsmonitor\n+\ttest_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+\n+# Ensure that FSMonitor is turned off on the borrowed repo.\n+#\n+test_expect_success \"Setup borrowed repo (fsm+uc)\" \"\n+\tstop_fsm &&\n+\tdisable_uc\n+\"\n+\n+# Also ensure that it starts in a known state.\n+#\n+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time\n+# the ballast checkout, since only the first invocation does any work and the\n+# subsequent ones just print \"already on branch\" and quit, so the reported\n+# time is not useful.\n+#\n+# Create a temp branch and do all work relative to it so that we don't\n+# accidentially alter the real ballast branch.\n+#\n+test_expect_success \"Setup borrowed repo (temp ballast branch)\" \"\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO reset --hard &&\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tgit -C $REPO branch $TMP_BR $BALLAST_BR &&\n+\tgit -C $REPO checkout $TMP_BR\n+\"\n+\n+\n+echo Data >data.txt\n+\n+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With\n+# FSMonitor enabled, we can get a skewed view of status times, since\n+# the index MAY (or may not) be updated after the first invocation\n+# which will update the FSMonitor Token, so the subsequent invocations\n+# may get a smaller response from the daemon.\n+#\n+do_status () {\n+\tmsg=$1\n+\n+\ttest_perf \"$msg\" \"\n+\t\tgit -C $REPO status >/dev/null 2>&1\n+\t\"\n+}\n+\n+do_matrix () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tt=\"[uc $uc][fsm $fsm]\"\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_expect_success \"$t Setup matrix branch\" \"\n+\t\tgit -C $REPO clean -d -f &&\n+\t\tgit -C $REPO checkout $TMP_BR &&\n+\t\ttest_might_fail git -C $REPO branch -D $MATRIX_BR &&\n+\t\tgit -C $REPO branch $MATRIX_BR $TMP_BR &&\n+\t\tgit -C $REPO checkout $MATRIX_BR\n+\t\"\n+\n+\tif test $uc = true\n+\tthen\n+\t\tenable_uc\n+\telse\n+\t\tdisable_uc\n+\tfi\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstart_fsm\n+\telse\n+\t\tstop_fsm\n+\tfi\n+\n+\tdo_status \"$t status after checkout\"\n+\n+\t# Modify many files in the matrix branch.\n+\t# Stage them.\n+\t# Commit them.\n+\t# Rollback.\n+\t#\n+\ttest_expect_success \"$t modify tracked files\" \"\n+\t\tfind $REPO -name file1 -exec cp data.txt {} \\\\;\n+\t\"\n+\n+\tdo_status \"$t status after big change\"\n+\n+\t# Don't bother timing the \"add\" because _REPEAT_COUNT\n+\t# issue described above.\n+\t#\n+\ttest_expect_success \"$t add all\" \"\n+\t\tgit -C $REPO add -A\n+\t\"\n+\n+\tdo_status \"$t status after add all\"\n+\n+\ttest_expect_success \"$t add dot\" \"\n+\t\tgit -C $REPO add .\n+\t\"\n+\n+\tdo_status \"$t status after add dot\"\n+\n+\ttest_expect_success \"$t commit staged\" \"\n+\t\tgit -C $REPO commit -a -m data\n+\t\"\n+\n+\tdo_status \"$t status after commit\"\n+\n+\ttest_expect_success \"$t reset HEAD~1 hard\" \"\n+\t\tgit -C $REPO reset --hard HEAD~1 >/dev/null 2>&1\n+\t\"\n+\n+\tdo_status \"$t status after reset hard\"\n+\n+\t# Create some untracked files.\n+\t#\n+\ttest_expect_success \"$t create untracked files\" \"\n+\t\tcp -R $REPO/ballast/dir1 $REPO/ballast/xxx1\n+\t\"\n+\n+\tdo_status \"$t status after create untracked files\"\n+\n+\t# Remove the new untracked files.\n+\t#\n+\ttest_expect_success \"$t clean -df\" \"\n+\t\tgit -C $REPO clean -d -f\n+\t\"\n+\n+\tdo_status \"$t status after clean\"\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstop_fsm\n+\tfi\n+}\n+\n+# Begin testing each case in the matrix that we care about.\n+#\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\n+fsm_values=\"false true\"\n+\n+for uc_val in $uc_values\n+do\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tdo_matrix $uc_val $fsm_val\n+\tdone\n+done\n+\n+cleanup () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_might_fail git -C $REPO branch -D $MATRIX_BR\n+}\n+\n+\n+# We're borrowing this repo. We should leave it in a clean state.\n+#\n+test_expect_success \"Cleanup temp and matrix branches\" \"\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\tfor fsm_val in $fsm_values\n+\t\tdo\n+\t\t\tcleanup $uc_val $fsm_val\n+\t\tdone\n+\tdone\n+\"\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"451912","messageId":"4ab4306adab355cbad98457c34730c88197fd422.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 23/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:56Z","receivedAt":"2022-03-22T18:24:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNever set CE_FSMONITOR_VALID on the cache-entry of submodule\ndirectories.\n\nDuring a client command like 'git status', we may need to recurse\ninto each submodule to compute a status summary for the submodule.\nSince the purpose of the ce_flag is to let Git avoid scanning a\ncache-entry, setting the flag causes the recursive call to be\navoided and we report incorrect (no status) for the submodule.\n\nWe created an OS watch on the root directory of our working\ndirectory and we receive events for everything in the cone\nunder it. When submodules are present inside our working\ndirectory, we receive events for both our repo (the super) and\nany subs within it. Since our index doesn't have any information\nfor items within the submodules, we can't use those events.\n\nWe could try to truncate the paths of those events back to the\nsubmodule boundary and mark the GITLINK as dirty, but that\nfeels expensive since we would have to prefix compare every FS\nevent that we receive against a list of submodule roots. And\nit still wouldn't be sufficient to correctly report status on\nthe submodule, since we don't have any space in the cache-entry\nto cache the submodule's status (the 'SCMU' bits in porcelain\nV2 speak). That is, the CE_FSMONITOR_VALID bit just says that\nwe don't need to scan/inspect it because we already know the\nanswer -- it doesn't say that the item is clean -- and we\ndon't have space in the cache-entry to store those answers.\nSo we should always do the recursive scan.\n\nTherefore, we should never set the flag on GITLINK cache-entries.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 2 +\n fsmonitor.h | 11 +++++\n t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 106 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex e1229c289cf..57d6a483bee 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tif (fsmonitor_enabled) {\n \t\t\t/* Mark all entries valid */\n \t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\t\tif (S_ISGITLINK(istate->cache[i]->ce_mode))\n+\t\t\t\t\tcontinue;\n \t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n \t\t\t}\n \ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 3f41f653691..edf7ce5203b 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n * called any time the cache entry has been updated to reflect the\n * current state of the file on disk.\n+ *\n+ * However, never mark submodules as valid. When commands like \"git\n+ * status\" run they might need to recurse into the submodule (using a\n+ * child process) to get a summary of the submodule state. We don't\n+ * have (and don't want to create) the facility to translate every\n+ * FS event that we receive and that happens to be deep inside of a\n+ * submodule back to the submodule root, so we cannot correctly keep\n+ * track of this bit on the gitlink directory. Therefore, we never\n+ * set it on submodules.\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n \n \tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n \t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tif (S_ISGITLINK(ce->ce_mode))\n+\t\t\treturn;\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex bc66d8285a3..39efed42a69 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -718,4 +718,97 @@ do\n \t'\n done\n \n+# Test fsmonitor interaction with submodules.\n+#\n+# If we start the daemon in the super, it will see FS events for\n+# everything in the working directory cone and this includes any\n+# files/directories contained *within* the submodules.\n+#\n+# A `git status` at top level will get events for items within the\n+# submodule and ignore them, since they aren't named in the index\n+# of the super repo. This makes the fsmonitor response a little\n+# noisy, but it doesn't alter the correctness of the state of the\n+# super-proper.\n+#\n+# When we have submodules, `git status` normally does a recursive\n+# status on each of the submodules and adds a summary row for any\n+# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n+#\n+# It is therefore important that the top level status not be tricked\n+# by the FSMonitor response to skip those recursive calls.\n+\n+my_match_and_clean () {\n+\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n+\tgit -C super --no-optional-locks -c core.fsmonitor=false \\\n+\t\tstatus --porcelain=v2 >actual.without &&\n+\ttest_cmp actual.with actual.without &&\n+\n+\tgit -C super/dir_1/dir_2/sub reset --hard &&\n+\tgit -C super/dir_1/dir_2/sub clean -d -f\n+}\n+\n+test_expect_success \"Submodule\" '\n+\ttest_when_finished \"git -C super fsmonitor--daemon stop\" &&\n+\n+\tgit init \"super\" &&\n+\techo x >super/file_1 &&\n+\techo y >super/file_2 &&\n+\techo z >super/file_3 &&\n+\tmkdir super/dir_1 &&\n+\techo a >super/dir_1/file_11 &&\n+\techo b >super/dir_1/file_12 &&\n+\tmkdir super/dir_1/dir_2 &&\n+\techo a >super/dir_1/dir_2/file_21 &&\n+\techo b >super/dir_1/dir_2/file_22 &&\n+\tgit -C super add . &&\n+\tgit -C super commit -m \"initial super commit\" &&\n+\n+\tgit init \"sub\" &&\n+\techo x >sub/file_x &&\n+\techo y >sub/file_y &&\n+\techo z >sub/file_z &&\n+\tmkdir sub/dir_x &&\n+\techo a >sub/dir_x/file_a &&\n+\techo b >sub/dir_x/file_b &&\n+\tmkdir sub/dir_x/dir_y &&\n+\techo a >sub/dir_x/dir_y/file_a &&\n+\techo b >sub/dir_x/dir_y/file_b &&\n+\tgit -C sub add . &&\n+\tgit -C sub commit -m \"initial sub commit\" &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\tstart_daemon -C super &&\n+\tgit -C super config core.fsmonitor true &&\n+\tgit -C super update-index --fsmonitor &&\n+\tgit -C super status &&\n+\n+\t# Now run pairs of commands w/ and w/o FSMonitor while we make\n+\t# some dirt in the submodule and confirm matching output.\n+\n+\t# Completely clean status.\n+\tmy_match_and_clean &&\n+\n+\t# .M S..U\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M SC..\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tgit -C super/dir_1/dir_2/sub commit -m \"SC..\" &&\n+\tmy_match_and_clean\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"451913","messageId":"5d0fa19929d8b326786f967f8e7ba40b2bc834d2.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 24/27] t7527: test FSMonitor on case insensitive+preserving file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:57Z","receivedAt":"2022-03-22T18:24:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTest that FS events from the OS are received using the preserved,\non-disk spelling of files/directories rather than spelling used\nto make the change.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 39efed42a69..51cdf05e7ec 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -811,4 +811,40 @@ test_expect_success \"Submodule\" '\n \tmy_match_and_clean\n '\n \n+# On a case-insensitive file system, confirm that the daemon\n+# notices when the .git directory is moved/renamed/deleted\n+# regardless of how it is spelled in the the FS event.\n+# That is, does the FS event receive the spelling of the\n+# operation or does it receive the spelling preserved with\n+# the file/directory.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n+#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n+\n+\tgit init test_insensitive &&\n+\n+\tstart_daemon -C test_insensitive -tf \"$PWD/insensitive.trace\" &&\n+\n+\tmkdir -p test_insensitive/abc/def &&\n+\techo xyz >test_insensitive/ABC/DEF/xyz &&\n+\n+\ttest_path_is_dir test_insensitive/.git &&\n+\ttest_path_is_dir test_insensitive/.GIT &&\n+\n+\t# Rename .git using an alternate spelling to verify that that\n+\t# daemon detects it and automatically shuts down.\n+\tmv test_insensitive/.GIT test_insensitive/.FOO &&\n+\tsleep 1 &&\n+\tmv test_insensitive/.FOO test_insensitive/.git &&\n+\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\t# Verify that events were reported using on-disk spellings of the\n+\t# directories and files that we touched. We may or may not get a\n+\t# trailing slash on modified directories.\n+\t#\n+\tegrep \"^event: abc/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"451914","messageId":"264397e8bd4b53ecf1d260b99d1895b8d153f82b.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 25/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:58Z","receivedAt":"2022-03-22T18:25:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nEmit NFC or NFC and NFD spellings of pathnames on macOS.\n\nMacOS is Unicode composition insensitive, so NFC and NFD spellings are\ntreated as aliases and collide. While the spelling of pathnames in\nfilesystem events depends upon the underlying filesystem, such as\nAPFS, HFS+ or FAT32, the OS enforces such collisions regardless of\nfilesystem.\n\nTeach the daemon to always report the NFC spelling and to report\nthe NFD spelling when stored in that format on the disk.\n\nThis is slightly more general than \"core.precomposeUnicode\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 83d38e8ac6c..823cf63999e 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n \treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n }\n \n+/*\n+ * On MacOS we have to adjust for Unicode composition insensitivity\n+ * (where NFC and NFD spellings are not respected). The different\n+ * spellings are essentially aliases regardless of how the path is\n+ * actually stored on the disk.\n+ *\n+ * This is related to \"core.precomposeUnicode\" (which wants to try\n+ * to hide NFD completely and treat everything as NFC). Here, we\n+ * don't know what the value the client has (or will have) for this\n+ * config setting when they make a query, so assume the worst and\n+ * emit both when the OS gives us an NFD path.\n+ */\n+static void my_add_path(struct fsmonitor_batch *batch, const char *path)\n+{\n+\tchar *composed;\n+\n+\t/* add the NFC or NFD path as received from the OS */\n+\tfsmonitor_batch__add_path(batch, path);\n+\n+\t/* if NFD, also add the corresponding NFC spelling */\n+\tcomposed = (char *)precompose_string_if_needed(path);\n+\tif (!composed || composed == path)\n+\t\treturn;\n+\n+\tfsmonitor_batch__add_path(batch, composed);\n+\tfree(composed);\n+}\n+\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t\tmy_add_path(batch, rel);\n \t\t\t}\n \n \t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n@@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t\tmy_add_path(batch, tmp.buf);\n \t\t\t}\n \n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"451915","messageId":"e6b621fb766706691230631df4b27c64cf7106e8.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 26/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:22:59Z","receivedAt":"2022-03-22T18:25:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a set of prereqs to help understand how file names\nare handled by the filesystem when they contain NFC and NFD\nUnicode characters.\n\nSigned-off-by: Jeff Hostetler \n---\n t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 167 insertions(+)\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n\ndiff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\nnew file mode 100755\nindex 00000000000..cf9c26d1e22\n--- /dev/null\n+++ b/t/lib-unicode-nfc-nfd.sh\n@@ -0,0 +1,167 @@\n+# Help detect how Unicode NFC and NFD are handled on the filesystem.\n+\n+# A simple character that has a NFD form.\n+#\n+# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n+# UTF8(NFC): \\xc3 \\xa9\n+#\n+# NFD: U+0065 LATIN SMALL LETTER E\n+# U+0301 COMBINING ACUTE ACCENT\n+# UTF8(NFD): \\x65 + \\xcc \\x81\n+#\n+utf8_nfc=$(printf \"\\xc3\\xa9\")\n+utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n+\n+# Is the OS or the filesystem \"Unicode composition sensitive\"?\n+#\n+# That is, does the OS or the filesystem allow files to exist with\n+# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n+# tell us that the NFC and NFD forms are equivalent.\n+#\n+# This is or may be independent of what type of filesystem we have,\n+# since it might be handled by the OS at a layer above the FS.\n+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n+# collision, for example.\n+#\n+# This does not tell us how the Unicode pathname will be spelled\n+# on disk, but rather only that the two spelling \"collide\". We\n+# will examine the actual on disk spelling in a later prereq.\n+#\n+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${utf8_nfc} &&\n+\tmkdir trial_${utf8_nfd}\n+'\n+\n+# Is the spelling of an NFC pathname preserved on disk?\n+#\n+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n+# and on APFS, NFC paths are preserved. As we have established\n+# above, this is independent of \"composition sensitivity\".\n+#\n+# 0000000 63 5f c3 a9\n+#\n+# (/usr/bin/od output contains different amount of whitespace\n+# on different platforms, so we need the wildcards here.)\n+#\n+test_lazy_prereq UNICODE_NFC_PRESERVED '\n+\tmkdir c_${utf8_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *c3 *a9\"\n+'\n+\n+# Is the spelling of an NFD pathname preserved on disk?\n+#\n+# 0000000 64 5f 65 cc 81\n+#\n+test_lazy_prereq UNICODE_NFD_PRESERVED '\n+\tmkdir d_${utf8_nfd} &&\n+\tls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n+'\n+\tmkdir c_${utf8_nfc} &&\n+\tmkdir d_${utf8_nfd} &&\n+\n+# The following _DOUBLE_ forms are more for my curiosity,\n+# but there may be quirks lurking when there are multiple\n+# combining characters in non-canonical order.\n+\n+# Unicode also allows multiple combining characters\n+# that can be decomposed in pieces.\n+#\n+# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n+# UTF8(NFC): \\xe1 \\xbd \\xa7\n+#\n+# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n+# U+0342 COMBINING GREEK PERISPOMENI\n+# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n+#\n+# But U+1f61 decomposes into\n+# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n+# U+0314 COMBINING REVERSED COMMA ABOVE\n+# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n+#\n+# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n+#\n+# Note that I've used the canonical ordering of the\n+# combinining characters. It is also possible to\n+# swap them. My testing shows that that non-standard\n+# ordering also causes a collision in mkdir. However,\n+# the resulting names don't draw correctly on the\n+# terminal (implying that the on-disk format also has\n+# them out of order).\n+#\n+greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n+greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n+greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n+\n+# See if a double decomposition also collides.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${greek_nfc} &&\n+\tmkdir trial_${greek_nfd2}\n+'\n+\n+# See if the NFC spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n+\tmkdir c_${greek_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n+'\n+\n+# See if the NFD spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n+\tmkdir d_${greek_nfd2} &&\n+\tls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n+'\n+\n+# The following is for debugging. I found it useful when\n+# trying to understand the various (OS, FS) quirks WRT\n+# Unicode and how composition/decomposition is handled.\n+# For example, when trying to understand how (macOS, APFS)\n+# and (macOS, HFS) and (macOS, FAT32) compare.\n+#\n+# It is rather noisy, so it is disabled by default.\n+#\n+if test \"$unicode_debug\" = \"true\"\n+then\n+\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\techo NFC maintains original spelling.\n+\telse\n+\t\techo NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFD_PRESERVED\n+\tthen\n+\t\techo NFD maintains original spelling.\n+\telse\n+\t\techo NFD is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n+\tthen\n+\t\techo Double NFC maintains original spelling.\n+\telse\n+\t\techo Double NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n+\tthen\n+\t\techo Double NFD maintains original spelling.\n+\telse\n+\t\techo Double NFD is modified.\n+\tfi\n+fi\n-- \ngitgitgadget\n\n"},{"id":"451916","messageId":"aa96a849ce44056f142f9972052a2ff7c34e7401.1647973380.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v3 27/27] t7527: test Unicode NFC/NFD handling on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-22T18:23:00Z","receivedAt":"2022-03-22T18:25:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that the daemon reports events using the on-disk\nspelling for Unicode NFC/NFD characters. On APFS we still\nhave Unicode aliasing, so we cannot create two files that\nonly differ by NFC/NFD, but the on-disk format preserves\nthe spelling used to create the file. On HFS+ we also\nhave aliasing, but the path is always stored on disk in\nNFD.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 55 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 51cdf05e7ec..126bb2dc8de 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -847,4 +847,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n+# The variable \"unicode_debug\" is defined in the following library\n+# script to dump information about how the (OS, FS) handles Unicode\n+# composition. Uncomment the following line if you want to enable it.\n+#\n+# unicode_debug=true\n+\n+. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n+\n+# See if the OS or filesystem does NFC/NFD aliasing/munging.\n+#\n+# The daemon should err on the side of caution and send BOTH the\n+# NFC and NFD forms. It does not know the original spelling of\n+# the pathname (how the user thinks it should be spelled), so\n+# emit both and let the client decide (when necessary). This is\n+# similar to \"core.precomposeUnicode\".\n+#\n+test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n+\n+\tgit init test_unicode &&\n+\n+\tstart_daemon -C test_unicode -tf \"$PWD/unicode.trace\" &&\n+\n+\t# Create a directory using an NFC spelling.\n+\t#\n+\tmkdir test_unicode/nfc &&\n+\tmkdir test_unicode/nfc/c_${utf8_nfc} &&\n+\n+\t# Create a directory using an NFD spelling.\n+\t#\n+\tmkdir test_unicode/nfd &&\n+\tmkdir test_unicode/nfd/d_${utf8_nfd} &&\n+\n+\tgit -C test_unicode fsmonitor--daemon stop &&\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\t# We should have seen NFC event from OS.\n+\t\t# We should not have synthesized an NFD event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace &&\n+\t\tegrep -v \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace\n+\telse\n+\t\t# We should have seen NFD event from OS.\n+\t\t# We should have synthesized an NFC event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace &&\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace\n+\tfi &&\n+\n+\t# We assume UNICODE_NFD_PRESERVED.\n+\t# We should have seen explicit NFD from OS.\n+\t# We should have synthesized an NFC event.\n+\tegrep \"^event: nfd/d_${utf8_nfd}/?$\" ./unicode.trace &&\n+\tegrep \"^event: nfd/d_${utf8_nfc}/?$\" ./unicode.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"452154","messageId":"23bfb8c5165ab3a2f9613533d120586c10cec191.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 01/27] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:50:54Z","receivedAt":"2022-03-24T16:51:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach FSMonitor daemon on Windows to recognize shortname paths as\naliases of normal longname paths. FSMonitor clients, such as `git\nstatus`, should receive the longname spelling of changed files (when\npossible).\n\nSometimes we receive FS events using the shortname, such as when a CMD\nshell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\narrives using whatever combination of long and shortnames were used by\nthe other process. (Shortnames do seem to be case normalized,\nhowever.)\n\nUse Windows GetLongPathNameW() to try to map the pathname spelling in\nthe notification event into the normalized longname spelling. (This\ncan fail if the file/directory is deleted, moved, or renamed, because\nwe are asking the FS for the mapping in response to the event and\nafter it has already happened, but we try.)\n\nSpecial case the shortname spelling of \".git\" to avoid under-reporting\nthese events.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++-----\n t/t7527-builtin-fsmonitor.sh | 65 +++++\n 2 files changed, 374 insertions(+), 54 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 5b928ab66e5..3f1b68267bd 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -25,6 +25,9 @@ struct one_watch\n \tDWORD count;\n \n \tstruct strbuf path;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n+\tDWORD wpath_longname_len;\n+\n \tHANDLE hDir;\n \tHANDLE hEvent;\n \tOVERLAPPED overlapped;\n@@ -34,6 +37,21 @@ struct one_watch\n \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n \t */\n \tBOOL is_active;\n+\n+\t/*\n+\t * Are shortnames enabled on the containing drive? This is\n+\t * always true for \"C:/\" drives and usually never true for\n+\t * other drives.\n+\t *\n+\t * We only set this for the worktree because we only need to\n+\t * convert shortname paths to longname paths for items we send\n+\t * to clients. (We don't care about shortname expansion for\n+\t * paths inside a GITDIR because we never send them to\n+\t * clients.)\n+\t */\n+\tBOOL has_shortnames;\n+\tBOOL has_tilda;\n+\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n struct fsmonitor_daemon_backend_data\n@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n };\n \n /*\n- * Convert the WCHAR path from the notification into UTF8 and\n- * then normalize it.\n+ * Convert the WCHAR path from the event into UTF8 and normalize it.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n */\n-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n \t\t\t\t struct strbuf *normalized_path)\n {\n \tint reserve;\n \tint len = 0;\n \n \tstrbuf_reset(normalized_path);\n-\tif (!info->FileNameLength)\n+\tif (!wpath_len)\n \t\tgoto normalize;\n \n \t/*\n@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t * sequence of 2 UTF8 characters. That should let us\n \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n \t */\n-\treserve = info->FileNameLength + 1;\n+\treserve = 2 * wpath_len + 1;\n \tstrbuf_grow(normalized_path, reserve);\n \n \tfor (;;) {\n-\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n-\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n+\t\t\t\t\t wpath, wpath_len,\n \t\t\t\t\t normalized_path->buf,\n \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n \t\t\t\t\t NULL, NULL);\n@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n \t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n-\t\t\t GetLastError(),\n-\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n-\t\t\t info->FileName);\n+\t\t\t GetLastError(), (int)wpath_len, wpath);\n \t\t\treturn -1;\n \t\t}\n \n@@ -98,6 +115,152 @@ normalize:\n \treturn strbuf_normalize_path(normalized_path);\n }\n \n+/*\n+ * See if the worktree root directory has shortnames enabled.\n+ * This will help us decide if we need to do an expensive shortname\n+ * to longname conversion on every notification event.\n+ *\n+ * We do not want to create a file to test this, so we assume that the\n+ * root directory contains a \".git\" file or directory. (Our caller\n+ * only calls us for the worktree root, so this should be fine.)\n+ *\n+ * Remember the spelling of the shortname for \".git\" if it exists.\n+ */\n+static void check_for_shortnames(struct one_watch *watch)\n+{\n+\twchar_t buf_in[MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\twchar_t *last_slash = NULL;\n+\twchar_t *last_bslash = NULL;\n+\twchar_t *last;\n+\n+\t/* build L\"/.git\" */\n+\twcscpy(buf_in, watch->wpath_longname);\n+\twcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n+\n+\tif (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n+\t\treturn;\n+\n+\tlast_slash = wcsrchr(buf_out, L'/');\n+\tlast_bslash = wcsrchr(buf_out, L'\\\\');\n+\tif (last_slash > last_bslash)\n+\t\tlast = last_slash + 1;\n+\telse if (last_bslash)\n+\t\tlast = last_bslash + 1;\n+\telse\n+\t\tlast = buf_out;\n+\n+\tif (!wcscmp(last, L\".git\"))\n+\t\treturn;\n+\n+\twatch->has_shortnames = 1;\n+\twcsncpy(watch->dotgit_shortname, last,\n+\t\tARRAY_SIZE(watch->dotgit_shortname));\n+\n+\t/*\n+\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n+\t * we should be able to avoid shortname to longname mapping on\n+\t * every notification event if the source string does not\n+\t * contain a \"~\".\n+\t *\n+\t * However, the documentation for GetLongPathNameW() says\n+\t * that there are filesystems that don't follow that pattern\n+\t * and warns against this optimization.\n+\t *\n+\t * Lets test this.\n+\t */\n+\tif (wcschr(watch->dotgit_shortname, L'~'))\n+\t\twatch->has_tilda = 1;\n+}\n+\n+enum get_relative_result {\n+\tGRR_NO_CONVERSION_NEEDED,\n+\tGRR_HAVE_CONVERSION,\n+\tGRR_SHUTDOWN,\n+};\n+\n+/*\n+ * Info notification paths are relative to the root of the watch.\n+ * If our CWD is still at the root, then we can use relative paths\n+ * to convert from shortnames to longnames. If our process has a\n+ * different CWD, then we need to construct an absolute path, do\n+ * the conversion, and then return the root-relative portion.\n+ *\n+ * We use the longname form of the root as our basis and assume that\n+ * it already has a trailing slash.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n+ */\n+static enum get_relative_result get_relative_longname(\n+\tstruct one_watch *watch,\n+\tconst wchar_t *wpath, DWORD wpath_len,\n+\twchar_t *wpath_longname)\n+{\n+\twchar_t buf_in[2 * MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\tDWORD root_len;\n+\n+\t/* Build L\"/\" */\n+\troot_len = watch->wpath_longname_len;\n+\twcsncpy(buf_in, watch->wpath_longname, root_len);\n+\twcsncpy(buf_in + root_len, wpath, wpath_len);\n+\tbuf_in[root_len + wpath_len] = 0;\n+\n+\t/*\n+\t * We don't actually know if the source pathname is a\n+\t * shortname or a longname. This routine allows either to be\n+\t * given as input.\n+\t */\n+\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n+\t\t/*\n+\t\t * The shortname to longname conversion can fail for\n+\t\t * various reasons, for example if the file has been\n+\t\t * deleted. (That is, if we just received a\n+\t\t * delete-file notification event and the file is\n+\t\t * already gone, we can't ask the file system to\n+\t\t * lookup the longname for it. Likewise, for moves\n+\t\t * and renames where we are given the old name.)\n+\t\t *\n+\t\t * Since deleting or moving a file or directory by its\n+\t\t * shortname is rather obscure, I'm going ignore the\n+\t\t * failure and ask the caller to report the original\n+\t\t * relative path. This seems kinder than failing here\n+\t\t * and forcing a resync. Besides, forcing a resync on\n+\t\t * every file/directory delete would effectively\n+\t\t * cripple monitoring.\n+\t\t *\n+\t\t * We might revisit this in the future.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (!wcscmp(buf_in, buf_out)) {\n+\t\t/*\n+\t\t * The path does not have a shortname alias.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (wcsncmp(buf_in, buf_out, root_len)) {\n+\t\t/*\n+\t\t * The spelling of the root directory portion of the computed\n+\t\t * longname has changed. This should not happen. Basically,\n+\t\t * it means that we don't know where (without recomputing the\n+\t\t * longname of just the root directory) to split out the\n+\t\t * relative path. Since this should not happen, I'm just\n+\t\t * going to let this fail and force a shutdown (because all\n+\t\t * subsequent events are probably going to see the same\n+\t\t * mismatch).\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\t/* Return the worktree root-relative portion of the longname. */\n+\n+\twcscpy(wpath_longname, buf_out + root_len);\n+\treturn GRR_HAVE_CONVERSION;\n+}\n+\n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n@@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n-\twchar_t wpath[MAX_PATH];\n+\tDWORD len_longname;\n+\twchar_t wpath[MAX_PATH + 1];\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \tif (xutftowcs_path(wpath, path) < 0) {\n \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n@@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \t\treturn NULL;\n \t}\n \n+\tif (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n+\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n+\t\t GetLastError(), path);\n+\t\tCloseHandle(hDir);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen_longname = wcslen(wpath_longname);\n+\tif (wpath_longname[len_longname - 1] != L'/' &&\n+\t wpath_longname[len_longname - 1] != L'\\\\') {\n+\t\twpath_longname[len_longname++] = L'/';\n+\t\twpath_longname[len_longname] = 0;\n+\t}\n+\n \tCALLOC_ARRAY(watch, 1);\n \n \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n@@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tstrbuf_init(&watch->path, 0);\n \tstrbuf_addstr(&watch->path, path);\n \n+\twcscpy(watch->wpath_longname, wpath_longname);\n+\twatch->wpath_longname_len = len_longname;\n+\n \twatch->hDir = hDir;\n \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n \n@@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t,\n+\tDWORD info_action)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n+\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n \tstruct fsmonitor_batch *batch = NULL;\n \tconst char *p = watch->buffer;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \t/*\n \t * If the kernel gets more events than will fit in the kernel\n@@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n+\t\twchar_t *wpath = info->FileName;\n+\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n \t\tenum fsmonitor_path_type t;\n+\t\tenum get_relative_result grr;\n+\n+\t\tif (watch->has_shortnames) {\n+\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n+\t\t\t\t/*\n+\t\t\t\t * This event exactly matches the\n+\t\t\t\t * spelling of the shortname of\n+\t\t\t\t * \".git\", so we can skip some steps.\n+\t\t\t\t *\n+\t\t\t\t * (This case is odd because the user\n+\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n+\t\t\t\t * use the filesystem to map it back\n+\t\t\t\t * to \".git\".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_reset(&path);\n+\t\t\t\tstrbuf_addstr(&path, \".git\");\n+\t\t\t\tt = IS_DOT_GIT;\n+\t\t\t\tgoto process_it;\n+\t\t\t}\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n-\t\t\tgoto skip_this_path;\n-\n-\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n-\n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n+\t\t\tif (watch->has_tilda && !wcschr(wpath, L'~')) {\n+\t\t\t\t/*\n+\t\t\t\t * Shortnames on this filesystem have tildas\n+\t\t\t\t * and the notification path does not have\n+\t\t\t\t * one, so we assume that it is a longname.\n+\t\t\t\t */\n+\t\t\t\tgoto normalize_it;\n+\t\t\t}\n \n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n+\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n+\t\t\t\t\t\t wpath_longname);\n+\t\t\tswitch (grr) {\n+\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n+\t\t\t\tbreak;\n+\t\t\tcase GRR_HAVE_CONVERSION:\n+\t\t\t\twpath = wpath_longname;\n+\t\t\t\twpath_len = wcslen(wpath);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\tcase GRR_SHUTDOWN:\n \t\t\t\tgoto force_shutdown;\n \t\t\t}\n-\t\t\tbreak;\n+\t\t}\n \n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n+normalize_it:\n+\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n+\t\t\tgoto skip_this_path;\n \n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+process_it:\n+\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n+\t\t\t\t\t info->Action))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -382,6 +630,9 @@ force_shutdown:\n * Note that we DO NOT get filesystem events on the external \n * itself (it is not inside something that we are watching). In particular,\n * we do not get an event if the external is deleted.\n+ *\n+ * Also, we do not care about shortnames within the external , since\n+ * we never send these paths to clients.\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n@@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\tif (normalize_path_in_utf8(\n+\t\t\t info->FileName,\n+\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t &path) == -1)\n \t\t\tgoto skip_this_path;\n \n \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n@@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \tif (!data->watch_worktree)\n \t\tgoto failed;\n \n+\tcheck_for_shortnames(data->watch_worktree);\n+\n \tif (state->nr_paths_watching > 1) {\n \t\tdata->watch_gitdir = create_watch(state,\n \t\t\t\t\t\t state->path_gitdir_watch.buf);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1c40d8d2035..9004fbf2043 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -177,6 +177,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon -C test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n+\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n+\n+\tstart_daemon -C test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"452155","messageId":"4aade7b560a92a2215405001b108cf35101f6809.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 13/27] fsmonitor--daemon: cd out of worktree root","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:06Z","receivedAt":"2022-03-24T16:51:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the fsmonitor--daemon to CD outside of the worktree\nbefore starting up.\n\nThe common Git startup mechanism causes the CWD of the daemon process\nto be in the root of the worktree. On Windows, this causes the daemon\nprocess to hold a locked handle on the CWD and prevents other\nprocesses from moving or deleting the worktree while the daemon is\nrunning.\n\nCD to HOME before entering main event loops.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++--\n compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------\n fsmonitor--daemon.h | 1 +\n 3 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 50ae3cca575..1b7c757f583 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1181,11 +1181,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * before we need it.\n \t */\n \tif (ipc_server_run_async(&state->ipc_server_data,\n-\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t state->path_ipc.buf, &ipc_opts,\n \t\t\t\t handle_client, state))\n \t\treturn error_errno(\n \t\t\t_(\"could not start IPC thread pool on '%s'\"),\n-\t\t\tfsmonitor_ipc__get_path());\n+\t\t\tstate->path_ipc.buf);\n \n \t/*\n \t * Start the fsmonitor listener thread to collect filesystem\n@@ -1220,6 +1220,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n static int fsmonitor_run_daemon(void)\n {\n \tstruct fsmonitor_daemon_state state;\n+\tconst char *home;\n \tint err;\n \n \tmemset(&state, 0, sizeof(state));\n@@ -1289,6 +1290,15 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_addch(&state.path_cookie_prefix, '/');\n \n+\t/*\n+\t * We create a named-pipe or unix domain socket inside of the\n+\t * \".git\" directory. (Well, on Windows, we base our named\n+\t * pipe in the NPFS on the absolute path of the git\n+\t * directory.)\n+\t */\n+\tstrbuf_init(&state.path_ipc, 0);\n+\tstrbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1298,6 +1308,23 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\t/*\n+\t * CD out of the worktree root directory.\n+\t *\n+\t * The common Git startup mechanism causes our CWD to be the\n+\t * root of the worktree. On Windows, this causes our process\n+\t * to hold a locked handle on the CWD. This prevents the\n+\t * worktree from being moved or deleted while the daemon is\n+\t * running.\n+\t *\n+\t * We assume that our FS and IPC listener threads have either\n+\t * opened all of the handles that they need or will do\n+\t * everything using absolute paths.\n+\t */\n+\thome = getenv(\"HOME\");\n+\tif (home && *home && chdir(home))\n+\t\tdie_errno(_(\"could not cd home '%s'\"), home);\n+\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n@@ -1310,6 +1337,7 @@ done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n \tstrbuf_release(&state.path_cookie_prefix);\n+\tstrbuf_release(&state.path_ipc);\n \n \treturn err;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 3f1b68267bd..c43d92b9620 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch)\n \t}\n \n \t/*\n-\t * NEEDSWORK: If an external is deleted, the above\n-\t * returns an error. I'm not sure that there's anything that\n-\t * we can do here other than failing -- the /.git\n-\t * link file would be broken anyway. We might try to check\n-\t * for that and return a better error message, but I'm not\n-\t * sure it is worth it.\n+\t * GetOverlappedResult() fails if the watched directory is\n+\t * deleted while we were waiting for an overlapped IO to\n+\t * complete. The documentation did not list specific errors,\n+\t * but I observed ERROR_ACCESS_DENIED (0x05) errors during\n+\t * testing.\n+\t *\n+\t * Note that we only get notificaiton events for events\n+\t * *within* the directory, not *on* the directory itself.\n+\t * (These might be properies of the parent directory, for\n+\t * example).\n+\t *\n+\t * NEEDSWORK: We might try to check for the deleted directory\n+\t * case and return a better error message, but I'm not sure it\n+\t * is worth it.\n+\t *\n+\t * Shutdown if we get any error.\n \t */\n \n \terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex bd09fffc176..223c2131b58 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n+\tstruct strbuf path_ipc;\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"452156","messageId":"d8ebac2a9b2f0dc3c1857a36b75da7d80abf06d2.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 14/27] fsmonitor--daemon: prepare for adding health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:07Z","receivedAt":"2022-03-24T16:51:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor daemon thread startup to make it easier to start\na third thread class to monitor the health of the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1b7c757f583..14cd2d5eb52 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1174,6 +1174,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint listener_started = 0;\n+\tint err = 0;\n \n \t/*\n \t * Start the IPC thread pool before the we've started the file\n@@ -1194,15 +1196,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \tif (pthread_create(&state->listener_thread, NULL,\n \t\t\t fsm_listen__thread_proc, state) < 0) {\n \t\tipc_server_stop_async(state->ipc_server_data);\n-\t\tipc_server_await(state->ipc_server_data);\n-\n-\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t\terr = error(_(\"could not start fsmonitor listener thread\"));\n+\t\tgoto cleanup;\n \t}\n+\tlistener_started = 1;\n \n \t/*\n \t * The daemon is now fully functional in background threads.\n+\t * Our primary thread should now just wait while the threads\n+\t * do all the work.\n+\t */\n+cleanup:\n+\t/*\n \t * Wait for the IPC thread pool to shutdown (whether by client\n-\t * request or from filesystem activity).\n+\t * request, from filesystem activity, or an error).\n \t */\n \tipc_server_await(state->ipc_server_data);\n \n@@ -1211,10 +1218,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * event from the IPC thread pool, but it doesn't hurt to tell\n \t * it again. And wait for it to shutdown.\n \t */\n-\tfsm_listen__stop_async(state);\n-\tpthread_join(state->listener_thread, NULL);\n+\tif (listener_started) {\n+\t\tfsm_listen__stop_async(state);\n+\t\tpthread_join(state->listener_thread, NULL);\n+\t}\n \n-\treturn state->error_code;\n+\tif (err)\n+\t\treturn err;\n+\tif (state->error_code)\n+\t\treturn state->error_code;\n+\treturn 0;\n }\n \n static int fsmonitor_run_daemon(void)\n-- \ngitgitgadget\n\n"},{"id":"452157","messageId":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"[PATCH v4 00/27] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:50:53Z","receivedAt":"2022-03-24T16:51:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is V4 of Part 3 of my builtin FSMonitor series. This version has been\nupdated to depend upon V8 of Part 2.\n\n * [] Feedback on V7 of Part 2 caused a minor change in a unit test in Part\n 3, so I'm resending Part 3 so that we can test the complete feature.\n\n * [] I also changed in Part 2 how some of the untracked cache tests were\n handled when FSMonitor is disabled, so I've made the corresponding\n changes here.\n\nHere is the range-diff from V3 to V4:\n\n 1: 779a15b38e = 1: 23bfb8c516 fsm-listen-win32: handle shortnames\n 2: 11d4a17b69 = 2: d14f171460 t7527: test FSMonitor on repos with Unicode root paths\n 3: 901fa32f6e = 3: 4db2370d04 t/helper/fsmonitor-client: create stress test\n 4: a8f0b2a525 = 4: f2c0569c90 fsmonitor-settings: bare repos are incompatible with FSMonitor\n 5: e32a8a7ea7 = 5: b2599bb9d2 fsmonitor-settings: stub in Win32-specific incompatibility checking\n 6: 5546339d96 = 6: 9ad6d87ccc fsmonitor-settings: VFS for Git virtual repos are incompatible\n 7: 1d2877efda = 7: 7652c79ab3 fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: 06d7f18676 = 8: 2f2a523552 fsmonitor-settings: remote repos on macOS are incompatible\n 9: 5ca97f482d = 9: 0297d80388 fsmonitor-settings: remote repos on Windows are incompatible\n10: 6715143724 = 10: b6dfd609ad fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n11: ed1f723130 = 11: db5197b44b unpack-trees: initialize fsmonitor_has_run_once in o->result\n12: 35c77b854b = 12: 3f154d0251 fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n13: a5affb359c = 13: 4aade7b560 fsmonitor--daemon: cd out of worktree root\n14: 087af5dfb6 = 14: d8ebac2a9b fsmonitor--daemon: prepare for adding health thread\n15: e78eb20c1b = 15: 7fb0795e25 fsmonitor--daemon: rename listener thread related variables\n16: 301fff5296 = 16: e90adcd06d fsmonitor--daemon: stub in health thread\n17: c6b5bdd25e = 17: d9b91a998c fsm-health-win32: add polling framework to monitor daemon health\n18: 13d11713a8 = 18: 0e95ee0d01 fsm-health-win32: force shutdown daemon if worktree root moves\n19: 01c1a38c46 = 19: 48a590d202 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n20: 0f0a5b5ca1 = 20: 36ab239fd9 fsmonitor: optimize processing of directory events\n21: d8218d197a ! 21: 3010b22e69 t7527: FSMonitor tests for directory moves\n @@ Commit message\n Create unit tests to move a directory. Verify that `git status`\n gives the same result with and without FSMonitor enabled.\n \n - NEEDSWORK: This test exposes a bug in the untracked-cache on\n - Windows when FSMonitor is disabled. These are commented out\n - for the moment.\n -\n Signed-off-by: Jeff Hostetler \n \n ## t/t7527-builtin-fsmonitor.sh ##\n @@ t/t7527-builtin-fsmonitor.sh: do\n matrix_try $uc_val $fsm_val file_to_directory\n matrix_try $uc_val $fsm_val directory_to_file\n \n -+ # NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor\n -+ # is DISABLED. Turn off a few test that cause it problems until\n -+ # we can debug it.\n -+ #\n -+ try_moves=\"true\"\n -+ test_have_prereq UNTRACKED_CACHE,WINDOWS && \\\n -+ test $uc_val = true && \\\n -+ test $fsm_val = false && \\\n -+ try_moves=\"false\"\n -+ if test $try_moves = true\n -+ then\n -+ matrix_try $uc_val $fsm_val move_directory_contents_deeper\n -+ matrix_try $uc_val $fsm_val move_directory_up\n -+ matrix_try $uc_val $fsm_val move_directory\n -+ fi\n ++ matrix_try $uc_val $fsm_val move_directory_contents_deeper\n ++ matrix_try $uc_val $fsm_val move_directory_up\n ++ matrix_try $uc_val $fsm_val move_directory\n +\n if test $fsm_val = true\n then\n22: 79da369dcc = 22: 85cdb4d84f t/perf/p7527: add perf test for builtin FSMonitor\n23: 4ab4306ada = 23: 29063455c8 fsmonitor: never set CE_FSMONITOR_VALID on submodules\n24: 5d0fa19929 ! 24: 6e99f5e4f2 t7527: test FSMonitor on case insensitive+preserving file system\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule\" '\n +\n + git init test_insensitive &&\n +\n -+ start_daemon -C test_insensitive -tf \"$PWD/insensitive.trace\" &&\n ++ start_daemon -C test_insensitive --tf \"$PWD/insensitive.trace\" &&\n +\n + mkdir -p test_insensitive/abc/def &&\n + echo xyz >test_insensitive/ABC/DEF/xyz &&\n25: 264397e8bd = 25: cef7dbbaf0 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n26: e6b621fb76 = 26: bc2d5a7a93 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n27: aa96a849ce ! 27: 176c530c3f t7527: test Unicode NFC/NFD handling on MacOS\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse\n +\n + git init test_unicode &&\n +\n -+ start_daemon -C test_unicode -tf \"$PWD/unicode.trace\" &&\n ++ start_daemon -C test_unicode --tf \"$PWD/unicode.trace\" &&\n +\n + # Create a directory using an NFC spelling.\n + #\n\n\nJeff Hostetler (27):\n fsm-listen-win32: handle shortnames\n t7527: test FSMonitor on repos with Unicode root paths\n t/helper/fsmonitor-client: create stress test\n fsmonitor-settings: bare repos are incompatible with FSMonitor\n fsmonitor-settings: stub in Win32-specific incompatibility checking\n fsmonitor-settings: VFS for Git virtual repos are incompatible\n fsmonitor-settings: stub in macOS-specific incompatibility checking\n fsmonitor-settings: remote repos on macOS are incompatible\n fsmonitor-settings: remote repos on Windows are incompatible\n fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n unpack-trees: initialize fsmonitor_has_run_once in o->result\n fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n fsmonitor--daemon: cd out of worktree root\n fsmonitor--daemon: prepare for adding health thread\n fsmonitor--daemon: rename listener thread related variables\n fsmonitor--daemon: stub in health thread\n fsm-health-win32: add polling framework to monitor daemon health\n fsm-health-win32: force shutdown daemon if worktree root moves\n fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n fsmonitor: optimize processing of directory events\n t7527: FSMonitor tests for directory moves\n t/perf/p7527: add perf test for builtin FSMonitor\n fsmonitor: never set CE_FSMONITOR_VALID on submodules\n t7527: test FSMonitor on case insensitive+preserving file system\n fsmonitor: on macOS also emit NFC spelling for NFD pathname\n t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n t7527: test Unicode NFC/NFD handling on MacOS\n\n Makefile | 19 +-\n builtin/fsmonitor--daemon.c | 104 ++++++-\n builtin/update-index.c | 4 +\n compat/fsmonitor/fsm-health-darwin.c | 24 ++\n compat/fsmonitor/fsm-health-win32.c | 278 +++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 +++\n compat/fsmonitor/fsm-listen-darwin.c | 122 ++++++--\n compat/fsmonitor/fsm-listen-win32.c | 413 ++++++++++++++++++++-----\n compat/fsmonitor/fsm-listen.h | 2 +-\n compat/fsmonitor/fsm-settings-darwin.c | 89 ++++++\n compat/fsmonitor/fsm-settings-win32.c | 137 ++++++++\n config.mak.uname | 5 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 11 +-\n fsmonitor-settings.c | 92 ++++++\n fsmonitor-settings.h | 29 ++\n fsmonitor.c | 73 ++++-\n fsmonitor.h | 11 +\n t/helper/test-fsmonitor-client.c | 106 +++++++\n t/lib-unicode-nfc-nfd.sh | 167 ++++++++++\n t/perf/p7527-builtin-fsmonitor.sh | 257 +++++++++++++++\n t/t7519-status-fsmonitor.sh | 32 ++\n t/t7527-builtin-fsmonitor.sh | 299 ++++++++++++++++++\n unpack-trees.c | 1 +\n 24 files changed, 2206 insertions(+), 124 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\n\nbase-commit: f87a1eba693b297d049281fa2d0c21b573027347\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1143\n\nRange-diff vs v3:\n\n 1: 779a15b38e8 = 1: 23bfb8c5165 fsm-listen-win32: handle shortnames\n 2: 11d4a17b692 = 2: d14f1714604 t7527: test FSMonitor on repos with Unicode root paths\n 3: 901fa32f6ea = 3: 4db2370d046 t/helper/fsmonitor-client: create stress test\n 4: a8f0b2a5256 = 4: f2c0569c901 fsmonitor-settings: bare repos are incompatible with FSMonitor\n 5: e32a8a7ea7a = 5: b2599bb9d2e fsmonitor-settings: stub in Win32-specific incompatibility checking\n 6: 5546339d963 = 6: 9ad6d87ccce fsmonitor-settings: VFS for Git virtual repos are incompatible\n 7: 1d2877efda0 = 7: 7652c79ab35 fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: 06d7f18676d = 8: 2f2a5235522 fsmonitor-settings: remote repos on macOS are incompatible\n 9: 5ca97f482d0 = 9: 0297d80388a fsmonitor-settings: remote repos on Windows are incompatible\n 10: 67151437245 = 10: b6dfd609adb fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n 11: ed1f7231309 = 11: db5197b44bb unpack-trees: initialize fsmonitor_has_run_once in o->result\n 12: 35c77b854bd = 12: 3f154d02517 fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n 13: a5affb359c4 = 13: 4aade7b560a fsmonitor--daemon: cd out of worktree root\n 14: 087af5dfb63 = 14: d8ebac2a9b2 fsmonitor--daemon: prepare for adding health thread\n 15: e78eb20c1bf = 15: 7fb0795e25e fsmonitor--daemon: rename listener thread related variables\n 16: 301fff5296a = 16: e90adcd06db fsmonitor--daemon: stub in health thread\n 17: c6b5bdd25e4 = 17: d9b91a998ce fsm-health-win32: add polling framework to monitor daemon health\n 18: 13d11713a86 = 18: 0e95ee0d01b fsm-health-win32: force shutdown daemon if worktree root moves\n 19: 01c1a38c462 = 19: 48a590d2026 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n 20: 0f0a5b5ca16 = 20: 36ab239fd9a fsmonitor: optimize processing of directory events\n 21: d8218d197ad ! 21: 3010b22e690 t7527: FSMonitor tests for directory moves\n @@ Commit message\n Create unit tests to move a directory. Verify that `git status`\n gives the same result with and without FSMonitor enabled.\n \n - NEEDSWORK: This test exposes a bug in the untracked-cache on\n - Windows when FSMonitor is disabled. These are commented out\n - for the moment.\n -\n Signed-off-by: Jeff Hostetler \n \n ## t/t7527-builtin-fsmonitor.sh ##\n @@ t/t7527-builtin-fsmonitor.sh: do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n -+\t\t# NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor\n -+\t\t# is DISABLED. Turn off a few test that cause it problems until\n -+\t\t# we can debug it.\n -+\t\t#\n -+\t\ttry_moves=\"true\"\n -+\t\ttest_have_prereq UNTRACKED_CACHE,WINDOWS && \\\n -+\t\t\ttest $uc_val = true && \\\n -+\t\t\ttest $fsm_val = false && \\\n -+\t\t\ttry_moves=\"false\"\n -+\t\tif test $try_moves = true\n -+\t\tthen\n -+\t\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n -+\t\t\tmatrix_try $uc_val $fsm_val move_directory_up\n -+\t\t\tmatrix_try $uc_val $fsm_val move_directory\n -+\t\tfi\n ++\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n ++\t\tmatrix_try $uc_val $fsm_val move_directory_up\n ++\t\tmatrix_try $uc_val $fsm_val move_directory\n +\n \t\tif test $fsm_val = true\n \t\tthen\n 22: 79da369dcce = 22: 85cdb4d84f2 t/perf/p7527: add perf test for builtin FSMonitor\n 23: 4ab4306adab = 23: 29063455c83 fsmonitor: never set CE_FSMONITOR_VALID on submodules\n 24: 5d0fa19929d ! 24: 6e99f5e4f2a t7527: test FSMonitor on case insensitive+preserving file system\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule\" '\n +\n +\tgit init test_insensitive &&\n +\n -+\tstart_daemon -C test_insensitive -tf \"$PWD/insensitive.trace\" &&\n ++\tstart_daemon -C test_insensitive --tf \"$PWD/insensitive.trace\" &&\n +\n +\tmkdir -p test_insensitive/abc/def &&\n +\techo xyz >test_insensitive/ABC/DEF/xyz &&\n 25: 264397e8bd4 = 25: cef7dbbaf04 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n 26: e6b621fb766 = 26: bc2d5a7a930 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n 27: aa96a849ce4 ! 27: 176c530c3fa t7527: test Unicode NFC/NFD handling on MacOS\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse\n +\n +\tgit init test_unicode &&\n +\n -+\tstart_daemon -C test_unicode -tf \"$PWD/unicode.trace\" &&\n ++\tstart_daemon -C test_unicode --tf \"$PWD/unicode.trace\" &&\n +\n +\t# Create a directory using an NFC spelling.\n +\t#\n\n-- \ngitgitgadget\n"},{"id":"452160","messageId":"db5197b44bbd4ebc0c380aff481288157909120d.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 11/27] unpack-trees: initialize fsmonitor_has_run_once in o->result","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:04Z","receivedAt":"2022-03-24T16:52:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInitialize `o->result.fsmonitor_has_run_once` based upon value\nin `o->src_index->fsmonitor_has_run_once` to prevent a second\nfsmonitor query during the tree traversal and possibly getting\na skewed view of the working directory.\n\nThe checkout code has already talked to the fsmonitor and the\ntraversal is updating the index as it traverses, so there is\nno need to query the fsmonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n unpack-trees.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..888cff81f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \to->result.fsmonitor_last_update =\n \t\txstrdup_or_null(o->src_index->fsmonitor_last_update);\n+\to->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;\n \n \t/*\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n-- \ngitgitgadget\n\n"},{"id":"452163","messageId":"7fb0795e25e191ae0270eca3672fd3da60c2127b.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 15/27] fsmonitor--daemon: rename listener thread related variables","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:08Z","receivedAt":"2022-03-24T16:52:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRename platform-specific listener thread related variables\nand data types as we prepare to add another backend thread\ntype.\n\n[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`\n[] `state->backend_data` becomes `state->listen_data`\n[] `state->error_code` becomes `state->listen_error_code`\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 +++---\n compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------\n compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++-------------\n compat/fsmonitor/fsm-listen.h | 2 +-\n fsmonitor--daemon.h | 6 +++---\n 5 files changed, 36 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 14cd2d5eb52..d5def8faf90 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1225,8 +1225,8 @@ cleanup:\n \n \tif (err)\n \t\treturn err;\n-\tif (state->error_code)\n-\t\treturn state->error_code;\n+\tif (state->listen_error_code)\n+\t\treturn state->listen_error_code;\n \treturn 0;\n }\n \n@@ -1241,7 +1241,7 @@ static int fsmonitor_run_daemon(void)\n \thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n-\tstate.error_code = 0;\n+\tstate.listen_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 14105f45c18..07113205a61 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -27,7 +27,7 @@\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tCFStringRef cfsr_worktree_path;\n \tCFStringRef cfsr_gitdir_path;\n@@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t const FSEventStreamEventId event_ids[])\n {\n \tstruct fsmonitor_daemon_state *state = ctx;\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tchar **paths = (char **)event_paths;\n \tstruct fsmonitor_batch *batch = NULL;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tNULL,\n \t\tNULL\n \t};\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \tconst void *dir_array[2];\n \n \tCALLOC_ARRAY(data, 1);\n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \n \tdata->cfsr_worktree_path = CFStringCreateWithCString(\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n@@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n \terror(_(\"Unable to create FSEventStream.\"));\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tif (data->stream) {\n \t\tif (data->stream_started)\n@@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n \t\tFSEventStreamRelease(data->stream);\n \t}\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \tdata->shutdown_style = SHUTDOWN_EVENT;\n \n \tCFRunLoopStop(data->rl);\n@@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tdata->rl = CFRunLoopGetCurrent();\n \n@@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \n \tswitch (data->shutdown_style) {\n \tcase FORCE_ERROR_STOP:\n-\t\tstate->error_code = -1;\n+\t\tstate->listen_error_code = -1;\n \t\t/* fall thru */\n \tcase FORCE_SHUTDOWN:\n \t\tipc_server_stop_async(state->ipc_server_data);\n@@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \treturn;\n \n force_error_stop_without_loop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \tipc_server_stop_async(state->ipc_server_data);\n \treturn;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex c43d92b9620..be2d93f47b2 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -54,7 +54,7 @@ struct one_watch\n \twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tstruct one_watch *watch_worktree;\n \tstruct one_watch *watch_gitdir;\n@@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname(\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+\tSetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);\n }\n \n static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n@@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch)\n \tfree(watch);\n }\n \n-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+static int start_rdcw_watch(struct fsm_listen_data *data,\n \t\t\t struct one_watch *watch)\n {\n \tDWORD dwNotifyFilter =\n@@ -516,7 +516,7 @@ static int process_1_worktree_event(\n */\n static int process_worktree_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_worktree;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -646,7 +646,7 @@ force_shutdown:\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_gitdir;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -704,11 +704,11 @@ skip_this_path:\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tDWORD dwWait;\n \tint result;\n \n-\tstate->error_code = 0;\n+\tstate->listen_error_code = 0;\n \n \tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n \t\tgoto force_error_stop;\n@@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \t}\n \n force_error_stop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \n force_shutdown:\n \t/*\n@@ -790,7 +790,7 @@ clean_shutdown:\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \treturn 0;\n \n failed:\n@@ -836,16 +836,16 @@ failed:\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tCloseHandle(data->hEventShutdown);\n \tdestroy_watch(data->watch_worktree);\n \tdestroy_watch(data->watch_gitdir);\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nindex f0539349baf..41650bf8972 100644\n--- a/compat/fsmonitor/fsm-listen.h\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n * do so if the listener thread receives a normal shutdown signal from\n * the IPC layer.)\n *\n- * It should set `state->error_code` to -1 if the daemon should exit\n+ * It should set `state->listen_error_code` to -1 if the daemon should exit\n * with an error.\n */\n void fsm_listen__loop(struct fsmonitor_daemon_state *state);\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 223c2131b58..2c6fa1a5d91 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n */\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {\n \tint cookie_seq;\n \tstruct hashmap cookies;\n \n-\tint error_code;\n-\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\tint listen_error_code;\n+\tstruct fsm_listen_data *listen_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"452164","messageId":"9ad6d87ccced1020181a80bc7ed92e30d1b23ff3.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:50:59Z","receivedAt":"2022-03-24T16:52:14Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nVFS for Git virtual repositories are incompatible with FSMonitor.\n\nVFS for Git is a downstream fork of Git. It contains its own custom\nfile system watcher that is aware of the virtualization. If a working\ndirectory is being managed by VFS for Git, we should not try to watch\nit because we may get incomplete results.\n\nWe do not know anything about how VFS for Git works, but we do\nknow that VFS for Git working directories contain a well-defined\nconfig setting. If it is set, mark the working directory as\nincompatible.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 5 +++++\n fsmonitor-settings.h | 1 +\n t/t7519-status-fsmonitor.sh | 9 +++++++++\n 4 files changed, 41 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 7fce32a3c5b..ee78bba38e3 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -3,7 +3,33 @@\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n \n+/*\n+ * VFS for Git is incompatible with FSMonitor.\n+ *\n+ * Granted, core Git does not know anything about VFS for Git and we\n+ * shouldn't make assumptions about a downstream feature, but users\n+ * can install both versions. And this can lead to incorrect results\n+ * from core Git commands. So, without bringing in any of the VFS for\n+ * Git code, do a simple config test for a published config setting.\n+ * (We do not look at the various *_TEST_* environment variables.)\n+ */\n+static enum fsmonitor_reason check_vfs4git(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n+\t\treturn FSMONITOR_REASON_VFS4GIT;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_vfs4git(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 8ff55f8c3fd..1efb6e17a20 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -179,6 +179,11 @@ int fsm_settings__error_if_incompatible(struct repository *r)\n \t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n \t\t xgetcwd());\n \t\treturn 1;\n+\n+\tcase FSMONITOR_REASON_VFS4GIT:\n+\t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 4b35f051fb1..6361fcbf6b0 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -16,6 +16,7 @@ enum fsmonitor_mode {\n enum fsmonitor_reason {\n \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 9a8e21c5608..156895f9437 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n \tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n '\n \n+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone actual\" &&\n+\tgit init fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"virtual repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"452166","messageId":"e90adcd06db8134d595b598a33feb7fd848ad762.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 16/27] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:09Z","receivedAt":"2022-03-24T16:52:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate another thread to watch over the daemon process and\nautomatically shut it down if necessary.\n\nThis commit creates the basic framework for a \"health\" thread\nto monitor the daemon and/or the file system. Later commits\nwill add platform-specific code to do the actual work.\n\nThe \"health\" thread is intended to monitor conditions that\nwould be difficult to track inside the IPC thread pool and/or\nthe file system listener threads. For example, when there are\nfile system events outside of the watched worktree root or if\nwe want to have an idle-timeout auto-shutdown feature.\n\nThis commit creates the health thread itself, defines the thread-proc\nand sets up the thread's event loop. It integrates this new thread\ninto the existing IPC and Listener thread models.\n\nThis commit defines the API to the platform-specific code where all of\nthe monitoring will actually happen.\n\nThe platform-specific code for MacOS is just stubs. Meaning that the\nhealth thread will immediately exit on MacOS, but that is OK and\nexpected. Future work can define MacOS-specific monitoring.\n\nThe platform-specific code for Windows sets up enough of the\nWaitForMultipleObjects() machinery to watch for system and/or custom\nevents. Currently, the set of wait handles only includes our custom\nshutdown event (sent from our other theads). Later commits in this\nseries will extend the set of wait handles to monitor other\nconditions.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 6 ++-\n builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n contrib/buildsystems/CMakeLists.txt | 2 +\n fsmonitor--daemon.h | 4 ++\n 7 files changed, 192 insertions(+), 2 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n\ndiff --git a/Makefile b/Makefile\nindex 93604fe8ef7..5f1623baadd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -472,8 +472,9 @@ all::\n #\n # If your platform supports a built-in fsmonitor backend, set\n # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n-# `compat/fsmonitor/fsm-listen-.c` that implements the\n-# `fsm_listen__*()` routines.\n+# `compat/fsmonitor/fsm-listen-.c` and\n+# `compat/fsmonitor/fsm-health-.c` files\n+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n # If your platform has OS-specific ways to tell if a repo is incompatible with\n # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n@@ -1982,6 +1983,7 @@ endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n ifdef FSMONITOR_OS_SETTINGS\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex d5def8faf90..e22f53026e9 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n@@ -1136,6 +1137,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+static void *fsm_health__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-health\");\n+\n+\tfsm_health__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -1174,6 +1187,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint health_started = 0;\n \tint listener_started = 0;\n \tint err = 0;\n \n@@ -1201,6 +1215,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t}\n \tlistener_started = 1;\n \n+\t/*\n+\t * Start the health thread to watch over our process.\n+\t */\n+\tif (pthread_create(&state->health_thread, NULL,\n+\t\t\t fsm_health__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\terr = error(_(\"could not start fsmonitor health thread\"));\n+\t\tgoto cleanup;\n+\t}\n+\thealth_started = 1;\n+\n \t/*\n \t * The daemon is now fully functional in background threads.\n \t * Our primary thread should now just wait while the threads\n@@ -1223,10 +1248,17 @@ cleanup:\n \t\tpthread_join(state->listener_thread, NULL);\n \t}\n \n+\tif (health_started) {\n+\t\tfsm_health__stop_async(state);\n+\t\tpthread_join(state->health_thread, NULL);\n+\t}\n+\n \tif (err)\n \t\treturn err;\n \tif (state->listen_error_code)\n \t\treturn state->listen_error_code;\n+\tif (state->health_error_code)\n+\t\treturn state->health_error_code;\n \treturn 0;\n }\n \n@@ -1242,6 +1274,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.listen_error_code = 0;\n+\tstate.health_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n@@ -1321,6 +1354,11 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\tif (fsm_health__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize health thread\"));\n+\t\tgoto done;\n+\t}\n+\n \t/*\n \t * CD out of the worktree root directory.\n \t *\n@@ -1344,6 +1382,7 @@ done:\n \tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n+\tfsm_health__dtor(&state);\n \n \tipc_server_free(state.ipc_server_data);\n \ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nnew file mode 100644\nindex 00000000000..b9f709e8548\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -0,0 +1,24 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nnew file mode 100644\nindex 00000000000..94b1d020f25\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsm_health_data\n+{\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hHandles[1]; /* the array does not own these handles */\n+#define HEALTH_SHUTDOWN 0\n+\tint nr_handles; /* number of active event handles */\n+};\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_handles++;\n+\n+\tstate->health_data = data;\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tif (!state || !state->health_data)\n+\t\treturn;\n+\n+\tdata = state->health_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\n+\tFREE_AND_NULL(state->health_data);\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\n+\tfor (;;) {\n+\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n+\t\t\t\t\t\t data->hHandles,\n+\t\t\t\t\t\t FALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->health_error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+clean_shutdown:\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n+}\ndiff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\nnew file mode 100644\nindex 00000000000..45547ba9380\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health.h\n@@ -0,0 +1,47 @@\n+#ifndef FSM_HEALTH_H\n+#define FSM_HEALTH_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor health thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the health thread.\n+ * This will be called from the main thread AFTER joining the thread.\n+ */\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to monitor the\n+ * health of the daemon process. This will run in the health thread.\n+ *\n+ * The health thread should call `ipc_server_stop_async()` if it needs\n+ * to cause a shutdown. (It should NOT do so if it receives a shutdown\n+ * shutdown signal.)\n+ *\n+ * It should set `state->health_error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_health__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the health thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_HEALTH_H */\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex b8f9f7a0388..16ace43d1c7 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 2c6fa1a5d91..2102a5c9ff5 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n+struct fsm_health_data; /* opaque platform-specific data for health thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n+\tpthread_t health_thread;\n \tpthread_mutex_t main_lock;\n \n \tstruct strbuf path_worktree_watch;\n@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n \tstruct hashmap cookies;\n \n \tint listen_error_code;\n+\tint health_error_code;\n \tstruct fsm_listen_data *listen_data;\n+\tstruct fsm_health_data *health_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"452167","messageId":"d9b91a998ce0802b4f81dddcafb1f91e4746d028.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 17/27] fsm-health-win32: add polling framework to monitor daemon health","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:10Z","receivedAt":"2022-03-24T16:52:21Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend the Windows version of the \"health\" thread to periodically\ninspect the system and shutdown if warranted.\n\nThis commit updates the thread's wait loop to use a timeout and\ndefines a (currently empty) table of functions to poll the system.\n\nA later commit will add functions to the table to actually\ninspect the system.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-\n 1 file changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 94b1d020f25..24fc612bf02 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -4,6 +4,24 @@\n #include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n+/*\n+ * Every minute wake up and test our health.\n+ */\n+#define WAIT_FREQ_MS (60 * 1000)\n+\n+/*\n+ * State machine states for each of the interval functions\n+ * used for polling our health.\n+ */\n+enum interval_fn_ctx {\n+\tCTX_INIT = 0,\n+\tCTX_TERM,\n+\tCTX_TIMER\n+};\n+\n+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx);\n+\n struct fsm_health_data\n {\n \tHANDLE hEventShutdown;\n@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n \tFREE_AND_NULL(state->health_data);\n }\n \n+/*\n+ * A table of the polling functions.\n+ */\n+static interval_fn *table[] = {\n+\tNULL, /* must be last */\n+};\n+\n+/*\n+ * Call all of the polling functions in the table.\n+ * Shortcut and return first error.\n+ *\n+ * Return 0 if all succeeded.\n+ */\n+static int call_all(struct fsmonitor_daemon_state *state,\n+\t\t enum interval_fn_ctx ctx)\n+{\n+\tint k;\n+\n+\tfor (k = 0; table[k]; k++) {\n+\t\tint r = table[k](state, ctx);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n+\tint r;\n+\n+\tr = call_all(state, CTX_INIT);\n+\tif (r < 0)\n+\t\tgoto force_error_stop;\n+\tif (r > 0)\n+\t\tgoto force_shutdown;\n \n \tfor (;;) {\n \t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n \t\t\t\t\t\t data->hHandles,\n-\t\t\t\t\t\t FALSE, INFINITE);\n+\t\t\t\t\t\t FALSE, WAIT_FREQ_MS);\n \n \t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n \t\t\tgoto clean_shutdown;\n \n+\t\tif (dwWait == WAIT_TIMEOUT) {\n+\t\t\tr = call_all(state, CTX_TIMER);\n+\t\t\tif (r < 0)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (r > 0)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\terror(_(\"health thread wait failed [GLE %ld]\"),\n \t\t GetLastError());\n \t\tgoto force_error_stop;\n@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)\n \n force_error_stop:\n \tstate->health_error_code = -1;\n+force_shutdown:\n \tipc_server_stop_async(state->ipc_server_data);\n clean_shutdown:\n+\tcall_all(state, CTX_TERM);\n \treturn;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"452168","messageId":"0e95ee0d01b4d00d46c9d33087126746e7e55cab.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 18/27] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:11Z","receivedAt":"2022-03-24T16:52:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nForce shutdown fsmonitor daemon if the worktree root directory\nis moved, renamed, or deleted.\n\nUse Windows low-level GetFileInformationByHandle() to get and\ncompare the Windows system unique ID for the directory with a\ncached version when we started up. This lets us detect the\ncase where someone renames the directory that we are watching\nand then creates a new directory with the original pathname.\n\nThis is important because we are listening to a named pipe for\nrequests and they are stored in the Named Pipe File System (NPFS)\nwhich a kernel-resident pseudo filesystem not associated with\nthe actual NTFS directory.\n\nFor example, if the daemon was watching \"~/foo/\", it would have\na directory-watch handle on that directory and a named-pipe\nhandle for \"//./pipe/...foo\". Moving the directory to \"~/bar/\"\ndoes not invalidate the directory handle. (So the daemon would\nactually be watching \"~/bar\" but listening on \"//./pipe/...foo\".\nIf the user then does \"git init ~/foo\" and causes another daemon\nto start, the first daemon will still have ownership of the pipe\nand the second daemon instance will fail to start. \"git status\"\nclients in \"~/foo\" will ask \"//./pipe/...foo\" about changes and\nthe first daemon instance will tell them about \"~/bar\".\n\nThis commit causes the first daemon to shutdown if the system unique\nID for \"~/foo\" changes (changes from what it was when the daemon\nstarted). Shutdown occurs after a periodic poll. After the\nfirst daemon exits and releases the lock on the named pipe,\nsubsequent Git commands may cause another daemon to be started\non \"~/foo\". Similarly, a subsequent Git command may cause another\ndaemon to be started on \"~/bar\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++\n 1 file changed, 143 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 24fc612bf02..2ea08c1d4e8 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -29,8 +29,150 @@ struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n \tint nr_handles; /* number of active event handles */\n+\n+\tstruct wt_moved\n+\t{\n+\t\twchar_t wpath[MAX_PATH + 1];\n+\t\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\t} wt_moved;\n };\n \n+/*\n+ * Lookup the system unique ID for the path. This is as close as\n+ * we get to an inode number, but this also contains volume info,\n+ * so it is a little stronger.\n+ */\n+static int lookup_bhfi(wchar_t *wpath,\n+\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n+{\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileW(wpath, desired_access, share_mode, NULL,\n+\t\t\t OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] health thread could not open '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\treturn -1;\n+\t}\n+\n+\tif (!GetFileInformationByHandle(hDir, bhfi)) {\n+\t\terror(_(\"[GLE %ld] health thread getting BHFI for '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\tCloseHandle(hDir);\n+\t\treturn -1;\n+\t}\n+\n+\tCloseHandle(hDir);\n+\treturn 0;\n+}\n+\n+/*\n+ * Compare the relevant fields from two system unique IDs.\n+ * We use this to see if two different handles to the same\n+ * path actually refer to the same *instance* of the file\n+ * or directory.\n+ */\n+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n+\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n+{\n+\treturn (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&\n+\t\tbhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&\n+\t\tbhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);\n+}\n+\n+/*\n+ * Shutdown if the original worktree root directory been deleted,\n+ * moved, or renamed?\n+ *\n+ * Since the main thread did a \"chdir(getenv($HOME))\" and our CWD\n+ * is not in the worktree root directory and because the listener\n+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible\n+ * for the root directory to be moved or deleted while we are still\n+ * watching it. We want to detect that here and force a shutdown.\n+ *\n+ * Granted, a delete MAY cause some operations to fail, such as\n+ * GetOverlappedResult(), but it is not guaranteed. And because\n+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the\n+ * directory, not changes *ON* the directory, our watch will not\n+ * receive a delete event for it.\n+ *\n+ * A move/rename of the worktree root will also not generate an event.\n+ * And since the listener thread already has an open handle, it may\n+ * continue to receive events for events within the directory.\n+ * However, the pathname of the named-pipe was constructed using the\n+ * original location of the worktree root. (Remember named-pipes are\n+ * stored in the NPFS and not in the actual file system.) Clients\n+ * trying to talk to the worktree after the move/rename will not\n+ * reach our daemon process, since we're still listening on the\n+ * pipe with original path.\n+ *\n+ * Furthermore, if the user does something like:\n+ *\n+ * $ mv repo repo.old\n+ * $ git init repo\n+ *\n+ * A new daemon cannot be started in the new instance of \"repo\"\n+ * because the named-pipe is still being used by the daemon on\n+ * the original instance.\n+ *\n+ * So, detect move/rename/delete and shutdown. This should also\n+ * handle unsafe drive removal.\n+ *\n+ * We use the file system unique ID to distinguish the original\n+ * directory instance from a new instance and force a shutdown\n+ * if the unique ID changes.\n+ *\n+ * Since a worktree move/rename/delete/unmount doesn't happen\n+ * that often (and we can't get an immediate event anyway), we\n+ * use a timeout and periodically poll it.\n+ */\n+static int has_worktree_moved(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\tint r;\n+\n+\tswitch (ctx) {\n+\tcase CTX_TERM:\n+\t\treturn 0;\n+\n+\tcase CTX_INIT:\n+\t\tif (xutftowcs_path(data->wt_moved.wpath,\n+\t\t\t\t state->path_worktree_watch.buf) < 0) {\n+\t\t\terror(_(\"could not convert to wide characters: '%s'\"),\n+\t\t\t state->path_worktree_watch.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/*\n+\t\t * On the first call we lookup the unique sequence ID for\n+\t\t * the worktree root directory.\n+\t\t */\n+\t\treturn lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);\n+\n+\tcase CTX_TIMER:\n+\t\tr = lookup_bhfi(data->wt_moved.wpath, &bhfi);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t\tif (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {\n+\t\t\terror(_(\"BHFI changed '%ls'\"), data->wt_moved.wpath);\n+\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tdie(_(\"unhandled case in 'has_worktree_moved': %d\"),\n+\t\t (int)ctx);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+\n int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data;\n@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n * A table of the polling functions.\n */\n static interval_fn *table[] = {\n+\thas_worktree_moved,\n \tNULL, /* must be last */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"452169","messageId":"48a590d2026292254f29bc8f691c9397e2c5a33a.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 19/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:12Z","receivedAt":"2022-03-24T16:52:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the listener thread to shutdown the daemon if the spelling of the\nworktree root directory changes.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 07113205a61..83d38e8ac6c 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tstrbuf_release(&msg);\n }\n \n+static int ef_is_root_changed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagRootChanged);\n+}\n+\n static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n {\n \treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n@@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_is_root_changed(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * The spelling of the pathname of the root directory\n+\t\t\t * has changed. This includes the name of the root\n+\t\t\t * directory itself or of any parent directory in the\n+\t\t\t * path.\n+\t\t\t *\n+\t\t\t * (There may be other conditions that throw this,\n+\t\t\t * but I couldn't find any information on it.)\n+\t\t\t *\n+\t\t\t * Force a shutdown now and avoid things getting\n+\t\t\t * out of sync. The Unix domain socket is inside\n+\t\t\t * the .git directory and a spelling change will make\n+\t\t\t * it hard for clients to rendezvous with us.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"event: root changed\");\n+\t\t\tgoto force_shutdown;\n+\t\t}\n+\n \t\tif (ef_ignore_xattr(event_flags[k])) {\n \t\t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n-- \ngitgitgadget\n\n"},{"id":"452170","messageId":"36ab239fd9a2ee98b0ccaaf4d12bdd8950ffafeb.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 20/27] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:13Z","receivedAt":"2022-03-24T16:52:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to perform binary search over the cache-entries for a directory\nnotification and then linearly scan forward to find the immediate children.\n\nPreviously, when the FSMonitor reported a modified directory Git would\nperform a linear search on the entire cache-entry array for all\nentries matching that directory prefix and invalidate them. Since the\ncache-entry array is already sorted, we can use a binary search to\nfind the first matching entry and then only linearly walk forward and\ninvalidate entries until the prefix changes.\n\nAlso, the original code would invalidate anything having the same\ndirectory prefix. Since a directory event should only be received for\nitems that are immediately within the directory (and not within\nsub-directories of it), only invalidate those entries and not the\nwhole subtree.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 54 insertions(+), 17 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 292a6742b4f..e1229c289cf 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n-\tif (name[len - 1] == '/') {\n+\tint pos = index_name_pos(istate, name, len);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n+\t\t\t name, pos);\n \n+\tif (name[len - 1] == '/') {\n \t\t/*\n-\t\t * TODO We should binary search to find the first path with\n-\t\t * TODO this directory prefix. Then linearly update entries\n-\t\t * TODO while the prefix matches. Taking care to search without\n-\t\t * TODO the trailing slash -- because '/' sorts after a few\n-\t\t * TODO interesting special chars, like '.' and ' '.\n+\t\t * The daemon can decorate directory events, such as\n+\t\t * moves or renames, with a trailing slash if the OS\n+\t\t * FS Event contains sufficient information, such as\n+\t\t * MacOS.\n+\t\t *\n+\t\t * Use this to invalidate the entire cone under that\n+\t\t * directory.\n+\t\t *\n+\t\t * We do not expect an exact match because the index\n+\t\t * does not normally contain directory entries, so we\n+\t\t * start at the insertion point and scan.\n \t\t */\n+\t\tif (pos < 0)\n+\t\t\tpos = -pos - 1;\n \n \t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n-\t\t\t starts_with(istate->cache[i]->name, name))\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n-\t\t/* Need to remove the / from the path for the untracked cache */\n+\n+\t\t/*\n+\t\t * We need to remove the traling \"/\" from the path\n+\t\t * for the untracked cache.\n+\t\t */\n \t\tname[len - 1] = '\\0';\n+\t} else if (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\tint pos = index_name_pos(istate, name, strlen(name));\n-\n-\t\tif (pos >= 0) {\n-\t\t\tstruct cache_entry *ce = istate->cache[pos];\n-\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n \t}\n \n@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n \t * as it could be a new untracked file.\n \t */\n-\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"452171","messageId":"85cdb4d84f249bb741526997e46281b4ed63c7a1.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 22/27] t/perf/p7527: add perf test for builtin FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:15Z","receivedAt":"2022-03-24T16:52:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++\n 1 file changed, 257 insertions(+)\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\ndiff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..9338b9ea008\n--- /dev/null\n+++ b/t/perf/p7527-builtin-fsmonitor.sh\n@@ -0,0 +1,257 @@\n+#!/bin/sh\n+\n+test_description=\"Perf test for the builtin FSMonitor\"\n+\n+. ./perf-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+# Lie to perf-lib and ask for a new empty repo and avoid\n+# the complaints about GIT_PERF_REPO not being big enough\n+# the perf hit when GIT_PERF_LARGE_REPO is copied into\n+# the trash directory.\n+#\n+# NEEDSWORK: It would be nice if perf-lib had an option to\n+# \"borrow\" an existing large repo (especially for gigantic\n+# monorepos) and use it in-place. For now, fake it here.\n+#\n+test_perf_fresh_repo\n+\n+\n+# Use a generated synthetic monorepo. If it doesn't exist, we will\n+# generate it. If it does exist, we will put it in a known state\n+# before we start our timings.\n+#\n+PARAM_D=5\n+PARAM_W=10\n+PARAM_F=9\n+\n+PARAMS=\"$PARAM_D\".\"$PARAM_W\".\"$PARAM_F\"\n+\n+BALLAST_BR=p0006-ballast\n+export BALLAST_BR\n+\n+TMP_BR=tmp_br\n+export TMP_BR\n+\n+REPO=../repos/gen-many-files-\"$PARAMS\".git\n+export REPO\n+\n+if ! test -d $REPO\n+then\n+\t(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)\n+fi\n+\n+\n+enable_uc () {\n+\tgit -C $REPO config core.untrackedcache true\n+\tgit -C $REPO update-index --untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+disable_uc () {\n+\tgit -C $REPO config core.untrackedcache false\n+\tgit -C $REPO update-index --no-untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+start_fsm () {\n+\tgit -C $REPO fsmonitor--daemon start\n+\tgit -C $REPO fsmonitor--daemon status\n+\tgit -C $REPO config core.fsmonitor true\n+\tgit -C $REPO update-index --fsmonitor\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+stop_fsm () {\n+\tgit -C $REPO config --unset core.fsmonitor\n+\tgit -C $REPO update-index --no-fsmonitor\n+\ttest_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+\n+# Ensure that FSMonitor is turned off on the borrowed repo.\n+#\n+test_expect_success \"Setup borrowed repo (fsm+uc)\" \"\n+\tstop_fsm &&\n+\tdisable_uc\n+\"\n+\n+# Also ensure that it starts in a known state.\n+#\n+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time\n+# the ballast checkout, since only the first invocation does any work and the\n+# subsequent ones just print \"already on branch\" and quit, so the reported\n+# time is not useful.\n+#\n+# Create a temp branch and do all work relative to it so that we don't\n+# accidentially alter the real ballast branch.\n+#\n+test_expect_success \"Setup borrowed repo (temp ballast branch)\" \"\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO reset --hard &&\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tgit -C $REPO branch $TMP_BR $BALLAST_BR &&\n+\tgit -C $REPO checkout $TMP_BR\n+\"\n+\n+\n+echo Data >data.txt\n+\n+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With\n+# FSMonitor enabled, we can get a skewed view of status times, since\n+# the index MAY (or may not) be updated after the first invocation\n+# which will update the FSMonitor Token, so the subsequent invocations\n+# may get a smaller response from the daemon.\n+#\n+do_status () {\n+\tmsg=$1\n+\n+\ttest_perf \"$msg\" \"\n+\t\tgit -C $REPO status >/dev/null 2>&1\n+\t\"\n+}\n+\n+do_matrix () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tt=\"[uc $uc][fsm $fsm]\"\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_expect_success \"$t Setup matrix branch\" \"\n+\t\tgit -C $REPO clean -d -f &&\n+\t\tgit -C $REPO checkout $TMP_BR &&\n+\t\ttest_might_fail git -C $REPO branch -D $MATRIX_BR &&\n+\t\tgit -C $REPO branch $MATRIX_BR $TMP_BR &&\n+\t\tgit -C $REPO checkout $MATRIX_BR\n+\t\"\n+\n+\tif test $uc = true\n+\tthen\n+\t\tenable_uc\n+\telse\n+\t\tdisable_uc\n+\tfi\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstart_fsm\n+\telse\n+\t\tstop_fsm\n+\tfi\n+\n+\tdo_status \"$t status after checkout\"\n+\n+\t# Modify many files in the matrix branch.\n+\t# Stage them.\n+\t# Commit them.\n+\t# Rollback.\n+\t#\n+\ttest_expect_success \"$t modify tracked files\" \"\n+\t\tfind $REPO -name file1 -exec cp data.txt {} \\\\;\n+\t\"\n+\n+\tdo_status \"$t status after big change\"\n+\n+\t# Don't bother timing the \"add\" because _REPEAT_COUNT\n+\t# issue described above.\n+\t#\n+\ttest_expect_success \"$t add all\" \"\n+\t\tgit -C $REPO add -A\n+\t\"\n+\n+\tdo_status \"$t status after add all\"\n+\n+\ttest_expect_success \"$t add dot\" \"\n+\t\tgit -C $REPO add .\n+\t\"\n+\n+\tdo_status \"$t status after add dot\"\n+\n+\ttest_expect_success \"$t commit staged\" \"\n+\t\tgit -C $REPO commit -a -m data\n+\t\"\n+\n+\tdo_status \"$t status after commit\"\n+\n+\ttest_expect_success \"$t reset HEAD~1 hard\" \"\n+\t\tgit -C $REPO reset --hard HEAD~1 >/dev/null 2>&1\n+\t\"\n+\n+\tdo_status \"$t status after reset hard\"\n+\n+\t# Create some untracked files.\n+\t#\n+\ttest_expect_success \"$t create untracked files\" \"\n+\t\tcp -R $REPO/ballast/dir1 $REPO/ballast/xxx1\n+\t\"\n+\n+\tdo_status \"$t status after create untracked files\"\n+\n+\t# Remove the new untracked files.\n+\t#\n+\ttest_expect_success \"$t clean -df\" \"\n+\t\tgit -C $REPO clean -d -f\n+\t\"\n+\n+\tdo_status \"$t status after clean\"\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstop_fsm\n+\tfi\n+}\n+\n+# Begin testing each case in the matrix that we care about.\n+#\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\n+fsm_values=\"false true\"\n+\n+for uc_val in $uc_values\n+do\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tdo_matrix $uc_val $fsm_val\n+\tdone\n+done\n+\n+cleanup () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_might_fail git -C $REPO branch -D $MATRIX_BR\n+}\n+\n+\n+# We're borrowing this repo. We should leave it in a clean state.\n+#\n+test_expect_success \"Cleanup temp and matrix branches\" \"\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\tfor fsm_val in $fsm_values\n+\t\tdo\n+\t\t\tcleanup $uc_val $fsm_val\n+\t\tdone\n+\tdone\n+\"\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"452172","messageId":"29063455c8399e8226795c6ff609cff3c0ac96f3.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 23/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:16Z","receivedAt":"2022-03-24T16:52:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNever set CE_FSMONITOR_VALID on the cache-entry of submodule\ndirectories.\n\nDuring a client command like 'git status', we may need to recurse\ninto each submodule to compute a status summary for the submodule.\nSince the purpose of the ce_flag is to let Git avoid scanning a\ncache-entry, setting the flag causes the recursive call to be\navoided and we report incorrect (no status) for the submodule.\n\nWe created an OS watch on the root directory of our working\ndirectory and we receive events for everything in the cone\nunder it. When submodules are present inside our working\ndirectory, we receive events for both our repo (the super) and\nany subs within it. Since our index doesn't have any information\nfor items within the submodules, we can't use those events.\n\nWe could try to truncate the paths of those events back to the\nsubmodule boundary and mark the GITLINK as dirty, but that\nfeels expensive since we would have to prefix compare every FS\nevent that we receive against a list of submodule roots. And\nit still wouldn't be sufficient to correctly report status on\nthe submodule, since we don't have any space in the cache-entry\nto cache the submodule's status (the 'SCMU' bits in porcelain\nV2 speak). That is, the CE_FSMONITOR_VALID bit just says that\nwe don't need to scan/inspect it because we already know the\nanswer -- it doesn't say that the item is clean -- and we\ndon't have space in the cache-entry to store those answers.\nSo we should always do the recursive scan.\n\nTherefore, we should never set the flag on GITLINK cache-entries.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 2 +\n fsmonitor.h | 11 +++++\n t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 106 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex e1229c289cf..57d6a483bee 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tif (fsmonitor_enabled) {\n \t\t\t/* Mark all entries valid */\n \t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\t\tif (S_ISGITLINK(istate->cache[i]->ce_mode))\n+\t\t\t\t\tcontinue;\n \t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n \t\t\t}\n \ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 3f41f653691..edf7ce5203b 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n * called any time the cache entry has been updated to reflect the\n * current state of the file on disk.\n+ *\n+ * However, never mark submodules as valid. When commands like \"git\n+ * status\" run they might need to recurse into the submodule (using a\n+ * child process) to get a summary of the submodule state. We don't\n+ * have (and don't want to create) the facility to translate every\n+ * FS event that we receive and that happens to be deep inside of a\n+ * submodule back to the submodule root, so we cannot correctly keep\n+ * track of this bit on the gitlink directory. Therefore, we never\n+ * set it on submodules.\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n \n \tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n \t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tif (S_ISGITLINK(ce->ce_mode))\n+\t\t\treturn;\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 3f794fb9821..7de178a9bf8 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -732,4 +732,97 @@ do\n \t'\n done\n \n+# Test fsmonitor interaction with submodules.\n+#\n+# If we start the daemon in the super, it will see FS events for\n+# everything in the working directory cone and this includes any\n+# files/directories contained *within* the submodules.\n+#\n+# A `git status` at top level will get events for items within the\n+# submodule and ignore them, since they aren't named in the index\n+# of the super repo. This makes the fsmonitor response a little\n+# noisy, but it doesn't alter the correctness of the state of the\n+# super-proper.\n+#\n+# When we have submodules, `git status` normally does a recursive\n+# status on each of the submodules and adds a summary row for any\n+# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n+#\n+# It is therefore important that the top level status not be tricked\n+# by the FSMonitor response to skip those recursive calls.\n+\n+my_match_and_clean () {\n+\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n+\tgit -C super --no-optional-locks -c core.fsmonitor=false \\\n+\t\tstatus --porcelain=v2 >actual.without &&\n+\ttest_cmp actual.with actual.without &&\n+\n+\tgit -C super/dir_1/dir_2/sub reset --hard &&\n+\tgit -C super/dir_1/dir_2/sub clean -d -f\n+}\n+\n+test_expect_success \"Submodule\" '\n+\ttest_when_finished \"git -C super fsmonitor--daemon stop\" &&\n+\n+\tgit init \"super\" &&\n+\techo x >super/file_1 &&\n+\techo y >super/file_2 &&\n+\techo z >super/file_3 &&\n+\tmkdir super/dir_1 &&\n+\techo a >super/dir_1/file_11 &&\n+\techo b >super/dir_1/file_12 &&\n+\tmkdir super/dir_1/dir_2 &&\n+\techo a >super/dir_1/dir_2/file_21 &&\n+\techo b >super/dir_1/dir_2/file_22 &&\n+\tgit -C super add . &&\n+\tgit -C super commit -m \"initial super commit\" &&\n+\n+\tgit init \"sub\" &&\n+\techo x >sub/file_x &&\n+\techo y >sub/file_y &&\n+\techo z >sub/file_z &&\n+\tmkdir sub/dir_x &&\n+\techo a >sub/dir_x/file_a &&\n+\techo b >sub/dir_x/file_b &&\n+\tmkdir sub/dir_x/dir_y &&\n+\techo a >sub/dir_x/dir_y/file_a &&\n+\techo b >sub/dir_x/dir_y/file_b &&\n+\tgit -C sub add . &&\n+\tgit -C sub commit -m \"initial sub commit\" &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\tstart_daemon -C super &&\n+\tgit -C super config core.fsmonitor true &&\n+\tgit -C super update-index --fsmonitor &&\n+\tgit -C super status &&\n+\n+\t# Now run pairs of commands w/ and w/o FSMonitor while we make\n+\t# some dirt in the submodule and confirm matching output.\n+\n+\t# Completely clean status.\n+\tmy_match_and_clean &&\n+\n+\t# .M S..U\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M SC..\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tgit -C super/dir_1/dir_2/sub commit -m \"SC..\" &&\n+\tmy_match_and_clean\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"452173","messageId":"3010b22e6906a0a18128004abe7cd7eb2953d1f5.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 21/27] t7527: FSMonitor tests for directory moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:14Z","receivedAt":"2022-03-24T16:52:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate unit tests to move a directory. Verify that `git status`\ngives the same result with and without FSMonitor enabled.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 0509ffe30f2..3f794fb9821 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -285,6 +285,16 @@ test_expect_success 'setup' '\n \ttrace*\n \tEOF\n \n+\tmkdir -p T1/T2/T3/T4 &&\n+\techo 1 >T1/F1 &&\n+\techo 1 >T1/T2/F1 &&\n+\techo 1 >T1/T2/T3/F1 &&\n+\techo 1 >T1/T2/T3/T4/F1 &&\n+\techo 2 >T1/F2 &&\n+\techo 2 >T1/T2/F2 &&\n+\techo 2 >T1/T2/T3/F2 &&\n+\techo 2 >T1/T2/T3/T4/F2 &&\n+\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n@@ -367,6 +377,19 @@ directory_to_file () {\n \techo 1 >dir1\n }\n \n+move_directory_contents_deeper() {\n+\tmkdir T1/_new_ &&\n+\tmv T1/[A-Z]* T1/_new_\n+}\n+\n+move_directory_up() {\n+\tmv T1/T2/T3 T1\n+}\n+\n+move_directory() {\n+\tmv T1/T2/T3 T1/T2/NewT3\n+}\n+\n # The next few test cases confirm that our fsmonitor daemon sees each type\n # of OS filesystem notification that we care about. At this layer we just\n # ensure we are getting the OS notifications and do not try to confirm what\n@@ -671,6 +694,10 @@ do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n+\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n+\t\tmatrix_try $uc_val $fsm_val move_directory_up\n+\t\tmatrix_try $uc_val $fsm_val move_directory\n+\n \t\tif test $fsm_val = true\n \t\tthen\n \t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n-- \ngitgitgadget\n\n"},{"id":"452176","messageId":"b6dfd609adb732d3b50d3189f294831d8e6bb217.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 10/27] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:03Z","receivedAt":"2022-03-24T16:52:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nOn MacOS mark repos on NTFS or FAT32 volumes as incompatible.\n\nThe builtin FSMonitor used Unix domain sockets on MacOS for IPC\nwith clients. These sockets are kept in the .git directory.\nUnix sockets are not supported by NTFS and FAT32, so the daemon\ncannot start up.\n\nTest for this during our compatibility checking so that client\ncommands do not keep trying to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---\n fsmonitor-settings.c | 5 +++++\n fsmonitor-settings.h | 1 +\n 3 files changed, 23 insertions(+), 3 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex fdd762bf79d..efc732c0f31 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -7,7 +7,7 @@\n #include \n \n /*\n- * Remote working directories are problematic for FSMonitor.\n+ * [1] Remote working directories are problematic for FSMonitor.\n *\n * The underlying file system on the server machine and/or the remote\n * mount type (NFS, SAMBA, etc.) dictates whether notification events\n@@ -40,8 +40,16 @@\n *\n * So (for now at least), mark remote working directories as\n * incompatible.\n+ *\n+ *\n+ * [2] FAT32 and NTFS working directories are problematic too.\n+ *\n+ * The builtin FSMonitor uses a Unix domain socket in the .git\n+ * directory for IPC. These Windows drive formats do not support\n+ * Unix domain sockets, so mark them as incompatible for the daemon.\n+ *\n */\n-static enum fsmonitor_reason check_remote(struct repository *r)\n+static enum fsmonitor_reason check_volume(struct repository *r)\n {\n \tstruct statfs fs;\n \n@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)\n \tif (!(fs.f_flags & MNT_LOCAL))\n \t\treturn FSMONITOR_REASON_REMOTE;\n \n+\tif (!strcmp(fs.f_fstypename, \"msdos\")) /* aka FAT32 */\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n+\tif (!strcmp(fs.f_fstypename, \"ntfs\"))\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \n@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n \n-\treason = check_remote(r);\n+\treason = check_volume(r);\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 0a1811ff004..60d5eaee497 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -194,6 +194,11 @@ int fsm_settings__error_if_incompatible(struct repository *r)\n \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t r->worktree);\n \t\treturn 1;\n+\n+\tcase FSMONITOR_REASON_NOSOCKETS:\n+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 34391b583b3..23d5676c8c8 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -19,6 +19,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n \tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n+\tFSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n-- \ngitgitgadget\n\n"},{"id":"452177","messageId":"0297d80388a69bf52e19169ef1ce1b29688e4a94.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 09/27] fsmonitor-settings: remote repos on Windows are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:02Z","receivedAt":"2022-03-24T16:52:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on Windows and mark them as\nincompatible with FSMonitor.\n\nWith this `git fsmonitor--daemon run` will error out with a message like it\ndoes for bare repos.\n\nClient commands, such as `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++\n 1 file changed, 102 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex ee78bba38e3..907655720bb 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * VFS for Git is incompatible with FSMonitor.\n@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)\n \treturn FSMONITOR_REASON_OK;\n }\n \n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type dictates whether notification events are available at\n+ * all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ *\n+ * Notes for testing:\n+ *\n+ * (a) Windows allows a network share to be mapped to a drive letter.\n+ * (This is the normal method to access it.)\n+ *\n+ * $ NET USE Z: \\\\server\\share\n+ * $ git -C Z:/repo status\n+ *\n+ * (b) Windows allows a network share to be referenced WITHOUT mapping\n+ * it to drive letter.\n+ *\n+ * $ NET USE \\\\server\\share\\dir\n+ * $ git -C //server/share/repo status\n+ *\n+ * (c) Windows allows \"SUBST\" to create a fake drive mapping to an\n+ * arbitrary path (which may be remote)\n+ *\n+ * $ SUBST Q: Z:\\repo\n+ * $ git -C Q:/ status\n+ *\n+ * (d) Windows allows a directory symlink to be created on a local\n+ * file system that points to a remote repo.\n+ *\n+ * $ mklink /d ./link //server/share/repo\n+ * $ git -C ./link status\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\twchar_t wpath[MAX_PATH];\n+\twchar_t wfullpath[MAX_PATH];\n+\tsize_t wlen;\n+\tUINT driveType;\n+\n+\t/*\n+\t * Do everything in wide chars because the drive letter might be\n+\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n+\t */\n+\tif (xutftowcs_path(wpath, r->worktree) < 0)\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\t/*\n+\t * GetDriveTypeW() requires a final slash. We assume that the\n+\t * worktree pathname points to an actual directory.\n+\t */\n+\twlen = wcslen(wpath);\n+\tif (wpath[wlen - 1] != L'\\\\' && wpath[wlen - 1] != L'/') {\n+\t\twpath[wlen++] = L'\\\\';\n+\t\twpath[wlen] = 0;\n+\t}\n+\n+\t/*\n+\t * Normalize the path. If nothing else, this converts forward\n+\t * slashes to backslashes. This is essential to get GetDriveTypeW()\n+\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n+\t */\n+\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\tdriveType = GetDriveTypeW(wfullpath);\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"DriveType '%s' L'%ls' (%u)\",\n+\t\t\t r->worktree, wfullpath, driveType);\n+\n+\tif (driveType == DRIVE_REMOTE) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"check_remote('%s') true\",\n+\t\t\t\t r->worktree);\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n-- \ngitgitgadget\n\n"},{"id":"452178","messageId":"2f2a523552222d04b9ba5f309fc0f4037bda5b7e.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 08/27] fsmonitor-settings: remote repos on macOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:01Z","receivedAt":"2022-03-24T16:52:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on macOS and mark them as\nincompatible with FSMonitor.\n\nWith this, `git fsmonitor--daemon run` will error out with a message\nlike it does for bare repos.\n\nClient commands, like `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 10 ++++\n fsmonitor-settings.h | 2 +\n 3 files changed, 78 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex 7fce32a3c5b..fdd762bf79d 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -2,8 +2,74 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n+#include \n+#include \n+\n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type (NFS, SAMBA, etc.) dictates whether notification events\n+ * are available at all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * For the builtin FSMonitor, we create the Unix domain socket for the\n+ * IPC in the .git directory. If the working directory is remote,\n+ * then the socket will be created on the remote file system. This\n+ * can fail if the remote file system does not support UDS file types\n+ * (e.g. smbfs to a Windows server) or if the remote kernel does not\n+ * allow a non-local process to bind() the socket. (These problems\n+ * could be fixed by moving the UDS out of the .git directory and to a\n+ * well-known local directory on the client machine, but care should\n+ * be taken to ensure that $HOME is actually local and not a managed\n+ * file share.)\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\tstruct statfs fs;\n+\n+\tif (statfs(r->worktree, &fs) == -1) {\n+\t\tint saved_errno = errno;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n+\t\t\t\t r->worktree, strerror(saved_errno));\n+\t\terrno = saved_errno;\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n+\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n+\n+\tif (!(fs.f_flags & MNT_LOCAL))\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 1efb6e17a20..0a1811ff004 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -180,6 +180,16 @@ int fsm_settings__error_if_incompatible(struct repository *r)\n \t\t xgetcwd());\n \t\treturn 1;\n \n+\tcase FSMONITOR_REASON_ERROR:\n+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n+\n+\tcase FSMONITOR_REASON_REMOTE:\n+\t\terror(_(\"remote repository '%s' is incompatible with fsmonitor\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n+\n \tcase FSMONITOR_REASON_VFS4GIT:\n \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t r->worktree);\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 6361fcbf6b0..34391b583b3 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -16,6 +16,8 @@ enum fsmonitor_mode {\n enum fsmonitor_reason {\n \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n+\tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"452179","messageId":"7652c79ab35f509d1064c7c7a20fb4d593e725a6.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:00Z","receivedAt":"2022-03-24T16:52:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++\n config.mak.uname | 1 +\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 13 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cf224768ad6..cf911d141f2 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)\n \tifndef NO_PTHREADS\n \tifndef NO_UNIX_SOCKETS\n \tFSMONITOR_DAEMON_BACKEND = darwin\n+\tFSMONITOR_OS_SETTINGS = darwin\n \tendif\n \tendif\n \ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 16705da2000..b8f9f7a0388 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"452180","messageId":"f2c0569c9012a86f252562a9a906f6de37d0a236.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:50:57Z","receivedAt":"2022-03-24T16:52:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a worktree, so there is nothing for the\ndaemon watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 ++++\n builtin/update-index.c | 4 +++\n fsmonitor-settings.c | 60 +++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 12 ++++++++\n t/t7519-status-fsmonitor.sh | 23 ++++++++++++++\n 5 files changed, 105 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 46be55a4618..50ae3cca575 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1449,6 +1449,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\tfsm_settings__set_ipc(the_repository);\n+\n+\tif (fsm_settings__error_if_incompatible(the_repository))\n+\t\treturn 1;\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 876112abb21..d29048f16f2 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1237,6 +1237,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\t\tif (fsm_settings__error_if_incompatible(the_repository))\n+\t\t\treturn 1;\n+\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 757d230d538..86c09bd35fe 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -9,9 +9,33 @@\n */\n struct fsmonitor_settings {\n \tenum fsmonitor_mode mode;\n+\tenum fsmonitor_reason reason;\n \tchar *hook_path;\n };\n \n+static void set_incompatible(struct repository *r,\n+\t\t\t enum fsmonitor_reason reason)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\ts->reason = reason;\n+}\n+\n+static int check_for_incompatible(struct repository *r)\n+{\n+\tif (!r->worktree) {\n+\t\t/*\n+\t\t * Bare repositories don't have a working directory and\n+\t\t * therefore have nothing to watch.\n+\t\t */\n+\t\tset_incompatible(r, FSMONITOR_REASON_BARE);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static void lookup_fsmonitor_settings(struct repository *r)\n {\n \tstruct fsmonitor_settings *s;\n@@ -23,6 +47,7 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \n \tCALLOC_ARRAY(s, 1);\n \ts->mode = FSMONITOR_MODE_DISABLED;\n+\ts->reason = FSMONITOR_REASON_OK;\n \n \tr->settings.fsmonitor = s;\n \n@@ -86,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r)\n \n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n@@ -97,6 +125,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path)\n \n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n@@ -110,5 +141,34 @@ void fsm_settings__set_disabled(struct repository *r)\n \tlookup_fsmonitor_settings(r);\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n+\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\n+\n+int fsm_settings__error_if_incompatible(struct repository *r)\n+{\n+\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n+\n+\tswitch (reason) {\n+\tcase FSMONITOR_REASON_OK:\n+\t\treturn 0;\n+\n+\tcase FSMONITOR_REASON_BARE:\n+\t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n+\t\t xgetcwd());\n+\t\treturn 1;\n+\t}\n+\n+\tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n+\t reason);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a4c5d7b4889..8654edf33d8 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -4,11 +4,20 @@\n struct repository;\n \n enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n \tFSMONITOR_MODE_DISABLED = 0,\n \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n };\n \n+/*\n+ * Incompatibility reasons.\n+ */\n+enum fsmonitor_reason {\n+\tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n+\tFSMONITOR_REASON_BARE,\n+};\n+\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n@@ -16,6 +25,9 @@ void fsm_settings__set_disabled(struct repository *r);\n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n+int fsm_settings__error_if_incompatible(struct repository *r);\n+\n struct fsmonitor_settings;\n \n #endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex a6308acf006..9a8e21c5608 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '\n \ttest $ret -ne 1\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n+\tgit init --bare bare-clone &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n+\tgit init --bare bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"452181","messageId":"b2599bb9d2e265ab8e1b2b5297356176dad04166.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 05/27] fsmonitor-settings: stub in Win32-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:50:58Z","receivedAt":"2022-03-24T16:52:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend generic incompatibility checkout with platform-specific\nmechanism. Stub in Win32 version.\n\nIn the existing fsmonitor-settings code we have a way to mark\ntypes of repos as incompatible with fsmonitor (whether via the\nhook and IPC APIs). For example, we do this for bare repos,\nsince there are no files to watch.\n\nExtend this exclusion mechanism for platform-specific reasons.\nThis commit just creates the framework and adds a stub for Win32.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 +++++++++++++\n compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++\n config.mak.uname | 4 ++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n fsmonitor-settings.c | 12 ++++++++++++\n fsmonitor-settings.h | 13 +++++++++++++\n 6 files changed, 54 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n\ndiff --git a/Makefile b/Makefile\nindex daa21bed6c3..93604fe8ef7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -475,6 +475,11 @@ all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n+# If your platform has OS-specific ways to tell if a repo is incompatible with\n+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n+# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n+# that implements the `fsm_os_settings__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n+ifdef FSMONITOR_OS_SETTINGS\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE\n ifdef FSMONITOR_DAEMON_BACKEND\n \t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n endif\n+ifdef FSMONITOR_OS_SETTINGS\n+\t@echo FSMONITOR_OS_SETTINGS=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_OS_SETTINGS)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 501970902da..cf224768ad6 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex ee0d7257b77..16705da2000 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 86c09bd35fe..8ff55f8c3fd 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r)\n \t\treturn 1;\n \t}\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+\t{\n+\t\tenum fsmonitor_reason reason;\n+\n+\t\treason = fsm_os__incompatible(r);\n+\t\tif (reason != FSMONITOR_REASON_OK) {\n+\t\t\tset_incompatible(r, reason);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+#endif\n+\n \treturn 0;\n }\n \ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 8654edf33d8..4b35f051fb1 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -30,4 +30,17 @@ int fsm_settings__error_if_incompatible(struct repository *r);\n \n struct fsmonitor_settings;\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+/*\n+ * Ask platform-specific code whether the repository is incompatible\n+ * with fsmonitor (both hook and ipc modes). For example, if the working\n+ * directory is on a remote volume and mounted via a technology that does\n+ * not support notification events, then we should not pretend to watch it.\n+ *\n+ * fsm_os__* routines should considered private to fsm_settings__\n+ * routines.\n+ */\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);\n+#endif /* HAVE_FSMONITOR_OS_SETTINGS */\n+\n #endif /* FSMONITOR_SETTINGS_H */\n-- \ngitgitgadget\n\n"},{"id":"452182","messageId":"4db2370d046875a83ec0c249c2349df61b7ecc3c.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 03/27] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:50:56Z","receivedAt":"2022-03-24T16:52:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a stress test to hammer on the fsmonitor daemon.\nCreate a client-side thread pool of n threads and have\neach of them make m requests as fast as they can.\n\nWe do not currently inspect the contents of the response.\nWe're only interested in placing a heavy request load on\nthe daemon.\n\nThis test is useful for interactive testing and various\nexperimentation. For example, to place additional load\non the daemon while another test is running. We currently\ndo not have a test script that actually uses this helper.\nWe might add such a test in the future.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++\n 1 file changed, 106 insertions(+)\n\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nindex 3062c8a3c2b..54a4856c48c 100644\n--- a/t/helper/test-fsmonitor-client.c\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -7,6 +7,8 @@\n #include \"cache.h\"\n #include \"parse-options.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2.h\"\n \n #ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n int cmd__fsmonitor_client(int argc, const char **argv)\n@@ -79,20 +81,121 @@ static int do_send_flush(void)\n \treturn 0;\n }\n \n+struct hammer_thread_data\n+{\n+\tpthread_t pthread_id;\n+\tint thread_nr;\n+\n+\tint nr_requests;\n+\tconst char *token;\n+\n+\tint sum_successful;\n+\tint sum_errors;\n+};\n+\n+static void *hammer_thread_proc(void *_hammer_thread_data)\n+{\n+\tstruct hammer_thread_data *data = _hammer_thread_data;\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint k;\n+\tint ret;\n+\n+\ttrace2_thread_start(\"hammer\");\n+\n+\tfor (k = 0; k < data->nr_requests; k++) {\n+\t\tstrbuf_reset(&answer);\n+\n+\t\tret = fsmonitor_ipc__send_query(data->token, &answer);\n+\t\tif (ret < 0)\n+\t\t\tdata->sum_errors++;\n+\t\telse\n+\t\t\tdata->sum_successful++;\n+\t}\n+\n+\tstrbuf_release(&answer);\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Start a pool of client threads that will each send a series of\n+ * commands to the daemon.\n+ *\n+ * The goal is to overload the daemon with a sustained series of\n+ * concurrent requests.\n+ */\n+static int do_hammer(const char *token, int nr_threads, int nr_requests)\n+{\n+\tstruct hammer_thread_data *data = NULL;\n+\tint k;\n+\tint sum_join_errors = 0;\n+\tint sum_commands = 0;\n+\tint sum_errors = 0;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\tif (nr_threads < 1)\n+\t\tnr_threads = 1;\n+\tif (nr_requests < 1)\n+\t\tnr_requests = 1;\n+\n+\tCALLOC_ARRAY(data, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\t\tp->thread_nr = k;\n+\t\tp->nr_requests = nr_requests;\n+\t\tp->token = token;\n+\n+\t\tif (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {\n+\t\t\twarning(\"failed to create thread[%d] skipping remainder\", k);\n+\t\t\tnr_threads = k;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\n+\t\tif (pthread_join(p->pthread_id, NULL))\n+\t\t\tsum_join_errors++;\n+\t\tsum_commands += p->sum_successful;\n+\t\tsum_errors += p->sum_errors;\n+\t}\n+\n+\tfprintf(stderr, \"HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\\n\",\n+\t\tnr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);\n+\n+\tfree(data);\n+\n+\t/*\n+\t * Return an error if any of the _send_query requests failed.\n+\t * We don't care about thread create/join errors.\n+\t */\n+\treturn sum_errors > 0;\n+}\n+\n int cmd__fsmonitor_client(int argc, const char **argv)\n {\n \tconst char *subcmd;\n \tconst char *token = NULL;\n+\tint nr_threads = 1;\n+\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n \t\t\"test-tool fsmonitor-client query []\",\n \t\t\"test-tool fsmonitor-client flush\",\n+\t\t\"test-tool fsmonitor-client hammer [] [] []\",\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n \t\tOPT_STRING(0, \"token\", &token, \"token\",\n \t\t\t \"command token to send to the server\"),\n+\n+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, \"number of client threads\"),\n+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, \"number of requests per thread\"),\n+\n \t\tOPT_END()\n \t};\n \n@@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)\n \tif (!strcmp(subcmd, \"flush\"))\n \t\treturn !!do_send_flush();\n \n+\tif (!strcmp(subcmd, \"hammer\"))\n+\t\treturn !!do_hammer(token, nr_threads, nr_requests);\n+\n \tdie(\"Unhandled subcommand: '%s'\", subcmd);\n }\n #endif\n-- \ngitgitgadget\n\n"},{"id":"452183","messageId":"d14f17146042a01abd0bd69771428245fe89167d.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 02/27] t7527: test FSMonitor on repos with Unicode root paths","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:50:55Z","receivedAt":"2022-03-24T16:52:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate some test repos with UTF8 characters in the pathname of the\nroot directory and verify that the builtin FSMonitor can watch them.\n\nThis test is mainly for Windows where we need to avoid `*A()`\nroutines.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 9004fbf2043..0509ffe30f2 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -682,4 +682,27 @@ do\n \tdone\n done\n \n+# Test Unicode UTF-8 characters in the pathname of the working\n+# directory root. Use of \"*A()\" routines rather than \"*W()\" routines\n+# on Windows can sometimes lead to odd failures.\n+#\n+u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n+u2=$(printf \"u_e2_99_ab__\\xE2\\x99\\xAB\")\n+u_values=\"$u1 $u2\"\n+for u in $u_values\n+do\n+\ttest_expect_success \"Unicode in repo root path: $u\" '\n+\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n+\n+\t\tgit init \"$u\" &&\n+\t\techo 1 >\"$u\"/file1 &&\n+\t\tgit -C \"$u\" add file1 &&\n+\t\tgit -C \"$u\" config core.fsmonitor true &&\n+\n+\t\tstart_daemon -C \"$u\" &&\n+\t\tgit -C \"$u\" status >actual &&\n+\t\tgrep \"new file: file1\" actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"452184","messageId":"3f154d0251784436d1d90a2000cfc986df01f17f.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 12/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:05Z","receivedAt":"2022-03-24T16:52:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIgnore FSEvents resulting from `xattr` changes. Git does not care about\nxattr's or changes to xattr's, so don't waste time collecting these\nevents in the daemon nor transmitting them to clients.\n\nVarious security tools add xattrs to files and/or directories, such as\nto mark them as having been downloaded. We should ignore these events\nsince it doesn't affect the content of the file/directory or the normal\nmeta-data that Git cares about.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 0741fe834c3..14105f45c18 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tif (flag & kFSEventStreamEventFlagItemCloned)\n \t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n \n-\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n \t\t\t path, flag, msg.buf);\n \n \tstrbuf_release(&msg);\n@@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)\n \t\tef & kFSEventStreamEventFlagUserDropped);\n }\n \n+/*\n+ * If an `xattr` change is the only reason we received this event,\n+ * then silently ignore it. Git doesn't care about xattr's. We\n+ * have to be careful here because the kernel can combine multiple\n+ * events for a single path. And because events always have certain\n+ * bits set, such as `ItemIsFile` or `ItemIsDir`.\n+ *\n+ * Return 1 if we should ignore it.\n+ */\n+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n+{\n+\tstatic const FSEventStreamEventFlags mask =\n+\t\tkFSEventStreamEventFlagItemChangeOwner |\n+\t\tkFSEventStreamEventFlagItemCreated |\n+\t\tkFSEventStreamEventFlagItemFinderInfoMod |\n+\t\tkFSEventStreamEventFlagItemInodeMetaMod |\n+\t\tkFSEventStreamEventFlagItemModified |\n+\t\tkFSEventStreamEventFlagItemRemoved |\n+\t\tkFSEventStreamEventFlagItemRenamed |\n+\t\tkFSEventStreamEventFlagItemXattrMod |\n+\t\tkFSEventStreamEventFlagItemCloned;\n+\n+\treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n+}\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_ignore_xattr(event_flags[k])) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n+\t\t\t\t\t path_k, event_flags[k]);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n \n \t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-- \ngitgitgadget\n\n"},{"id":"452185","messageId":"6e99f5e4f2a97a3fc416c5b0d2014ec61f79721f.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 24/27] t7527: test FSMonitor on case insensitive+preserving file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:17Z","receivedAt":"2022-03-24T16:53:01Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTest that FS events from the OS are received using the preserved,\non-disk spelling of files/directories rather than spelling used\nto make the change.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 7de178a9bf8..0b92a9bc22f 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -825,4 +825,40 @@ test_expect_success \"Submodule\" '\n \tmy_match_and_clean\n '\n \n+# On a case-insensitive file system, confirm that the daemon\n+# notices when the .git directory is moved/renamed/deleted\n+# regardless of how it is spelled in the the FS event.\n+# That is, does the FS event receive the spelling of the\n+# operation or does it receive the spelling preserved with\n+# the file/directory.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n+#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n+\n+\tgit init test_insensitive &&\n+\n+\tstart_daemon -C test_insensitive --tf \"$PWD/insensitive.trace\" &&\n+\n+\tmkdir -p test_insensitive/abc/def &&\n+\techo xyz >test_insensitive/ABC/DEF/xyz &&\n+\n+\ttest_path_is_dir test_insensitive/.git &&\n+\ttest_path_is_dir test_insensitive/.GIT &&\n+\n+\t# Rename .git using an alternate spelling to verify that that\n+\t# daemon detects it and automatically shuts down.\n+\tmv test_insensitive/.GIT test_insensitive/.FOO &&\n+\tsleep 1 &&\n+\tmv test_insensitive/.FOO test_insensitive/.git &&\n+\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\t# Verify that events were reported using on-disk spellings of the\n+\t# directories and files that we touched. We may or may not get a\n+\t# trailing slash on modified directories.\n+\t#\n+\tegrep \"^event: abc/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"452186","messageId":"cef7dbbaf04b5fd2e4ac63888ec46ec0c6db26a6.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 25/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:18Z","receivedAt":"2022-03-24T16:53:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nEmit NFC or NFC and NFD spellings of pathnames on macOS.\n\nMacOS is Unicode composition insensitive, so NFC and NFD spellings are\ntreated as aliases and collide. While the spelling of pathnames in\nfilesystem events depends upon the underlying filesystem, such as\nAPFS, HFS+ or FAT32, the OS enforces such collisions regardless of\nfilesystem.\n\nTeach the daemon to always report the NFC spelling and to report\nthe NFD spelling when stored in that format on the disk.\n\nThis is slightly more general than \"core.precomposeUnicode\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 83d38e8ac6c..823cf63999e 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n \treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n }\n \n+/*\n+ * On MacOS we have to adjust for Unicode composition insensitivity\n+ * (where NFC and NFD spellings are not respected). The different\n+ * spellings are essentially aliases regardless of how the path is\n+ * actually stored on the disk.\n+ *\n+ * This is related to \"core.precomposeUnicode\" (which wants to try\n+ * to hide NFD completely and treat everything as NFC). Here, we\n+ * don't know what the value the client has (or will have) for this\n+ * config setting when they make a query, so assume the worst and\n+ * emit both when the OS gives us an NFD path.\n+ */\n+static void my_add_path(struct fsmonitor_batch *batch, const char *path)\n+{\n+\tchar *composed;\n+\n+\t/* add the NFC or NFD path as received from the OS */\n+\tfsmonitor_batch__add_path(batch, path);\n+\n+\t/* if NFD, also add the corresponding NFC spelling */\n+\tcomposed = (char *)precompose_string_if_needed(path);\n+\tif (!composed || composed == path)\n+\t\treturn;\n+\n+\tfsmonitor_batch__add_path(batch, composed);\n+\tfree(composed);\n+}\n+\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t\tmy_add_path(batch, rel);\n \t\t\t}\n \n \t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n@@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t\tmy_add_path(batch, tmp.buf);\n \t\t\t}\n \n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"452187","messageId":"bc2d5a7a9306c895f4a1105c08d86f24e5ea542c.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 26/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:19Z","receivedAt":"2022-03-24T16:53:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a set of prereqs to help understand how file names\nare handled by the filesystem when they contain NFC and NFD\nUnicode characters.\n\nSigned-off-by: Jeff Hostetler \n---\n t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 167 insertions(+)\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n\ndiff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\nnew file mode 100755\nindex 00000000000..cf9c26d1e22\n--- /dev/null\n+++ b/t/lib-unicode-nfc-nfd.sh\n@@ -0,0 +1,167 @@\n+# Help detect how Unicode NFC and NFD are handled on the filesystem.\n+\n+# A simple character that has a NFD form.\n+#\n+# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n+# UTF8(NFC): \\xc3 \\xa9\n+#\n+# NFD: U+0065 LATIN SMALL LETTER E\n+# U+0301 COMBINING ACUTE ACCENT\n+# UTF8(NFD): \\x65 + \\xcc \\x81\n+#\n+utf8_nfc=$(printf \"\\xc3\\xa9\")\n+utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n+\n+# Is the OS or the filesystem \"Unicode composition sensitive\"?\n+#\n+# That is, does the OS or the filesystem allow files to exist with\n+# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n+# tell us that the NFC and NFD forms are equivalent.\n+#\n+# This is or may be independent of what type of filesystem we have,\n+# since it might be handled by the OS at a layer above the FS.\n+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n+# collision, for example.\n+#\n+# This does not tell us how the Unicode pathname will be spelled\n+# on disk, but rather only that the two spelling \"collide\". We\n+# will examine the actual on disk spelling in a later prereq.\n+#\n+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${utf8_nfc} &&\n+\tmkdir trial_${utf8_nfd}\n+'\n+\n+# Is the spelling of an NFC pathname preserved on disk?\n+#\n+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n+# and on APFS, NFC paths are preserved. As we have established\n+# above, this is independent of \"composition sensitivity\".\n+#\n+# 0000000 63 5f c3 a9\n+#\n+# (/usr/bin/od output contains different amount of whitespace\n+# on different platforms, so we need the wildcards here.)\n+#\n+test_lazy_prereq UNICODE_NFC_PRESERVED '\n+\tmkdir c_${utf8_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *c3 *a9\"\n+'\n+\n+# Is the spelling of an NFD pathname preserved on disk?\n+#\n+# 0000000 64 5f 65 cc 81\n+#\n+test_lazy_prereq UNICODE_NFD_PRESERVED '\n+\tmkdir d_${utf8_nfd} &&\n+\tls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n+'\n+\tmkdir c_${utf8_nfc} &&\n+\tmkdir d_${utf8_nfd} &&\n+\n+# The following _DOUBLE_ forms are more for my curiosity,\n+# but there may be quirks lurking when there are multiple\n+# combining characters in non-canonical order.\n+\n+# Unicode also allows multiple combining characters\n+# that can be decomposed in pieces.\n+#\n+# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n+# UTF8(NFC): \\xe1 \\xbd \\xa7\n+#\n+# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n+# U+0342 COMBINING GREEK PERISPOMENI\n+# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n+#\n+# But U+1f61 decomposes into\n+# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n+# U+0314 COMBINING REVERSED COMMA ABOVE\n+# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n+#\n+# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n+#\n+# Note that I've used the canonical ordering of the\n+# combinining characters. It is also possible to\n+# swap them. My testing shows that that non-standard\n+# ordering also causes a collision in mkdir. However,\n+# the resulting names don't draw correctly on the\n+# terminal (implying that the on-disk format also has\n+# them out of order).\n+#\n+greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n+greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n+greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n+\n+# See if a double decomposition also collides.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${greek_nfc} &&\n+\tmkdir trial_${greek_nfd2}\n+'\n+\n+# See if the NFC spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n+\tmkdir c_${greek_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n+'\n+\n+# See if the NFD spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n+\tmkdir d_${greek_nfd2} &&\n+\tls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n+'\n+\n+# The following is for debugging. I found it useful when\n+# trying to understand the various (OS, FS) quirks WRT\n+# Unicode and how composition/decomposition is handled.\n+# For example, when trying to understand how (macOS, APFS)\n+# and (macOS, HFS) and (macOS, FAT32) compare.\n+#\n+# It is rather noisy, so it is disabled by default.\n+#\n+if test \"$unicode_debug\" = \"true\"\n+then\n+\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\techo NFC maintains original spelling.\n+\telse\n+\t\techo NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFD_PRESERVED\n+\tthen\n+\t\techo NFD maintains original spelling.\n+\telse\n+\t\techo NFD is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n+\tthen\n+\t\techo Double NFC maintains original spelling.\n+\telse\n+\t\techo Double NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n+\tthen\n+\t\techo Double NFD maintains original spelling.\n+\telse\n+\t\techo Double NFD is modified.\n+\tfi\n+fi\n-- \ngitgitgadget\n\n"},{"id":"452188","messageId":"176c530c3fa9a3ee7c6b2e55d440feaa09093480.1648140680.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v4 27/27] t7527: test Unicode NFC/NFD handling on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-24T16:51:20Z","receivedAt":"2022-03-24T16:53:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that the daemon reports events using the on-disk\nspelling for Unicode NFC/NFD characters. On APFS we still\nhave Unicode aliasing, so we cannot create two files that\nonly differ by NFC/NFD, but the on-disk format preserves\nthe spelling used to create the file. On HFS+ we also\nhave aliasing, but the path is always stored on disk in\nNFD.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 55 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 0b92a9bc22f..3f4e1c2ad7c 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -861,4 +861,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n+# The variable \"unicode_debug\" is defined in the following library\n+# script to dump information about how the (OS, FS) handles Unicode\n+# composition. Uncomment the following line if you want to enable it.\n+#\n+# unicode_debug=true\n+\n+. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n+\n+# See if the OS or filesystem does NFC/NFD aliasing/munging.\n+#\n+# The daemon should err on the side of caution and send BOTH the\n+# NFC and NFD forms. It does not know the original spelling of\n+# the pathname (how the user thinks it should be spelled), so\n+# emit both and let the client decide (when necessary). This is\n+# similar to \"core.precomposeUnicode\".\n+#\n+test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n+\n+\tgit init test_unicode &&\n+\n+\tstart_daemon -C test_unicode --tf \"$PWD/unicode.trace\" &&\n+\n+\t# Create a directory using an NFC spelling.\n+\t#\n+\tmkdir test_unicode/nfc &&\n+\tmkdir test_unicode/nfc/c_${utf8_nfc} &&\n+\n+\t# Create a directory using an NFD spelling.\n+\t#\n+\tmkdir test_unicode/nfd &&\n+\tmkdir test_unicode/nfd/d_${utf8_nfd} &&\n+\n+\tgit -C test_unicode fsmonitor--daemon stop &&\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\t# We should have seen NFC event from OS.\n+\t\t# We should not have synthesized an NFD event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace &&\n+\t\tegrep -v \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace\n+\telse\n+\t\t# We should have seen NFD event from OS.\n+\t\t# We should have synthesized an NFC event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace &&\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace\n+\tfi &&\n+\n+\t# We assume UNICODE_NFD_PRESERVED.\n+\t# We should have seen explicit NFD from OS.\n+\t# We should have synthesized an NFC event.\n+\tegrep \"^event: nfd/d_${utf8_nfd}/?$\" ./unicode.trace &&\n+\tegrep \"^event: nfd/d_${utf8_nfc}/?$\" ./unicode.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"452233","messageId":"xmqq4k3nf29t.fsf@gitster.g","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/27] Builtin FSMonitor Part 3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-24T21:50:54Z","receivedAt":"2022-03-24T21:50:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> Here is V4 of Part 3 of my builtin FSMonitor series. This version has been\n> updated to depend upon V8 of Part 2.\n\nThanks. Queued.\n\n"},{"id":"452257","messageId":"9c584888-2dd2-139a-cf41-b2974386d1f8@gmail.com","threadId":"57414","inReplyTo":"bc2d5a7a9306c895f4a1105c08d86f24e5ea542c.1648140680.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 26/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2022-03-25T08:59:09Z","receivedAt":"2022-03-25T08:59:18Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 24/03/22 23.51, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Create a set of prereqs to help understand how file names\n> are handled by the filesystem when they contain NFC and NFD\n> Unicode characters.\n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 167 insertions(+)\n> create mode 100755 t/lib-unicode-nfc-nfd.sh\n> \n> diff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\n> new file mode 100755\n> index 00000000000..cf9c26d1e22\n> --- /dev/null\n> +++ b/t/lib-unicode-nfc-nfd.sh\n> @@ -0,0 +1,167 @@\n> +# Help detect how Unicode NFC and NFD are handled on the filesystem.\n> +\n> +# A simple character that has a NFD form.\n> +#\n> +# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n> +# UTF8(NFC): \\xc3 \\xa9\n> +#\n> +# NFD: U+0065 LATIN SMALL LETTER E\n> +# U+0301 COMBINING ACUTE ACCENT\n> +# UTF8(NFD): \\x65 + \\xcc \\x81\n> +#\n> +utf8_nfc=$(printf \"\\xc3\\xa9\")\n> +utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n> +\n\nThe first nfc-nfd test subject (É) is simple case, right?\n\n> +# Is the OS or the filesystem \"Unicode composition sensitive\"?\n> +#\n> +# That is, does the OS or the filesystem allow files to exist with\n> +# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n> +# tell us that the NFC and NFD forms are equivalent.\n> +#\n> +# This is or may be independent of what type of filesystem we have,\n> +# since it might be handled by the OS at a layer above the FS.\n> +# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n> +# collision, for example.\n> +#\n> +# This does not tell us how the Unicode pathname will be spelled\n> +# on disk, but rather only that the two spelling \"collide\". We\n> +# will examine the actual on disk spelling in a later prereq.\n> +#\n> +test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n> +\tmkdir trial_${utf8_nfc} &&\n> +\tmkdir trial_${utf8_nfd}\n> +'\n> +\n> +# Is the spelling of an NFC pathname preserved on disk?\n> +#\n> +# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n> +# and on APFS, NFC paths are preserved. As we have established\n> +# above, this is independent of \"composition sensitivity\".\n> +#\n> +# 0000000 63 5f c3 a9\n> +#\n> +# (/usr/bin/od output contains different amount of whitespace\n> +# on different platforms, so we need the wildcards here.)\n> +#\n> +test_lazy_prereq UNICODE_NFC_PRESERVED '\n> +\tmkdir c_${utf8_nfc} &&\n> +\tls | od -t x1 | grep \"63 *5f *c3 *a9\"\n> +'\n> +\n> +# Is the spelling of an NFD pathname preserved on disk?\n> +#\n> +# 0000000 64 5f 65 cc 81\n> +#\n> +test_lazy_prereq UNICODE_NFD_PRESERVED '\n> +\tmkdir d_${utf8_nfd} &&\n> +\tls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n> +'\n> +\tmkdir c_${utf8_nfc} &&\n> +\tmkdir d_${utf8_nfd} &&\n> +\n> +# The following _DOUBLE_ forms are more for my curiosity,\n> +# but there may be quirks lurking when there are multiple\n> +# combining characters in non-canonical order.\n> +\n> +# Unicode also allows multiple combining characters\n> +# that can be decomposed in pieces.\n> +#\n> +# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n> +# UTF8(NFC): \\xe1 \\xbd \\xa7\n> +#\n> +# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n> +# U+0342 COMBINING GREEK PERISPOMENI\n> +# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n> +#\n> +# But U+1f61 decomposes into\n> +# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n> +# U+0314 COMBINING REVERSED COMMA ABOVE\n> +# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n> +#\n> +# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n> +#\n> +# Note that I've used the canonical ordering of the\n> +# combinining characters. It is also possible to\n> +# swap them. My testing shows that that non-standard\n> +# ordering also causes a collision in mkdir. However,\n> +# the resulting names don't draw correctly on the\n> +# terminal (implying that the on-disk format also has\n> +# them out of order).\n> +#\n> +greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n> +greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n> +greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n> +\n\nThe second test subject (greek) is more complex, right?\n\n> +# See if a double decomposition also collides.\n> +#\n> +test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n> +\tmkdir trial_${greek_nfc} &&\n> +\tmkdir trial_${greek_nfd2}\n> +'\n> +\n> +# See if the NFC spelling appears on the disk.\n> +#\n> +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n> +\tmkdir c_${greek_nfc} &&\n> +\tls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n> +'\n> +\n> +# See if the NFD spelling appears on the disk.\n> +#\n> +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n> +\tmkdir d_${greek_nfd2} &&\n> +\tls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n> +'\n> +\n> +# The following is for debugging. I found it useful when\n> +# trying to understand the various (OS, FS) quirks WRT\n> +# Unicode and how composition/decomposition is handled.\n> +# For example, when trying to understand how (macOS, APFS)\n> +# and (macOS, HFS) and (macOS, FAT32) compare.\n> +#\n> +# It is rather noisy, so it is disabled by default.\n> +#\n> +if test \"$unicode_debug\" = \"true\"\n> +then\n> +\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n> +\tthen\n> +\t\techo NFC and NFD are distinct on this OS/filesystem.\n> +\telse\n> +\t\techo NFC and NFD are aliases on this OS/filesystem.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_NFC_PRESERVED\n> +\tthen\n> +\t\techo NFC maintains original spelling.\n> +\telse\n> +\t\techo NFC is modified.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_NFD_PRESERVED\n> +\tthen\n> +\t\techo NFD maintains original spelling.\n> +\telse\n> +\t\techo NFD is modified.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n> +\tthen\n> +\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n> +\telse\n> +\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n> +\tthen\n> +\t\techo Double NFC maintains original spelling.\n> +\telse\n> +\t\techo Double NFC is modified.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n> +\tthen\n> +\t\techo Double NFD maintains original spelling.\n> +\telse\n> +\t\techo Double NFD is modified.\n> +\tfi\n> +fi\n\nIn general, this test is written from Mac OS perspective, but since we have\nGit users also from Linux, Windows, and other OSes, I'd like to see from these\nother perspective.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"452264","messageId":"c21da33f-30a9-a405-fb39-9aee2caf7675@github.com","threadId":"57414","inReplyTo":"9c584888-2dd2-139a-cf41-b2974386d1f8@gmail.com","subject":"Re: [PATCH v4 26/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-25T14:21:32Z","receivedAt":"2022-03-25T14:21:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/25/2022 4:59 AM, Bagas Sanjaya wrote:\n\n> In general, this test is written from Mac OS perspective, but since we have\n> Git users also from Linux, Windows, and other OSes, I'd like to see from these\n> other perspective.\n \nJeff can correct me if I'm wrong, but these tests are needed due to specific\nissues that come up on macOS filesystems. Windows does not have these problems\nand there is not a Linux implementation of the FS Monitor daemon (yet). Thus,\nthis macOS-specific inspection is intentional and correct.\n\nAny additional issues that might happen on a Linux filesystem should be\nadded when discovering them during that implementation.\n\nThanks,\n-Stolee\n"},{"id":"452270","messageId":"ba395b67-12a9-7fbb-7b0a-610ea055fe81@jeffhostetler.com","threadId":"57414","inReplyTo":"9c584888-2dd2-139a-cf41-b2974386d1f8@gmail.com","subject":"Re: [PATCH v4 26/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-25T14:38:12Z","receivedAt":"2022-03-25T14:38:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/25/22 4:59 AM, Bagas Sanjaya wrote:\n> On 24/03/22 23.51, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Create a set of prereqs to help understand how file names\n>> are handled by the filesystem when they contain NFC and NFD\n>> Unicode characters.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>>   t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++\n>>   1 file changed, 167 insertions(+)\n>>   create mode 100755 t/lib-unicode-nfc-nfd.sh\n>>\n>> diff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\n>> new file mode 100755\n>> index 00000000000..cf9c26d1e22\n>> --- /dev/null\n>> +++ b/t/lib-unicode-nfc-nfd.sh\n>> @@ -0,0 +1,167 @@\n>> +# Help detect how Unicode NFC and NFD are handled on the filesystem.\n>> +\n>> +# A simple character that has a NFD form.\n>> +#\n>> +# NFC:       U+00e9 LATIN SMALL LETTER E WITH ACUTE\n>> +# UTF8(NFC): \\xc3 \\xa9\n>> +#\n>> +# NFD:       U+0065 LATIN SMALL LETTER E\n>> +#            U+0301 COMBINING ACUTE ACCENT\n>> +# UTF8(NFD): \\x65  +  \\xcc \\x81\n>> +#\n>> +utf8_nfc=$(printf \"\\xc3\\xa9\")\n>> +utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n>> +\n> \n> The first nfc-nfd test subject (É) is simple case, right?\n\nYes, this first pair was to test basic composition sensitivity.\nSomething that we would expect to be munged on MacOS, but not elsewhere.\n\n> \n>> +# Is the OS or the filesystem \"Unicode composition sensitive\"?\n>> +#\n>> +# That is, does the OS or the filesystem allow files to exist with\n>> +# both the NFC and NFD spellings?  Or, does the OS/FS lie to us and\n>> +# tell us that the NFC and NFD forms are equivalent.\n>> +#\n>> +# This is or may be independent of what type of filesystem we have,\n>> +# since it might be handled by the OS at a layer above the FS.\n>> +# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n>> +# collision, for example.\n>> +#\n>> +# This does not tell us how the Unicode pathname will be spelled\n>> +# on disk, but rather only that the two spelling \"collide\".  We\n>> +# will examine the actual on disk spelling in a later prereq.\n>> +#\n>> +test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n>> +    mkdir trial_${utf8_nfc} &&\n>> +    mkdir trial_${utf8_nfd}\n>> +'\n>> +\n>> +# Is the spelling of an NFC pathname preserved on disk?\n>> +#\n>> +# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n>> +# and on APFS, NFC paths are preserved.  As we have established\n>> +# above, this is independent of \"composition sensitivity\".\n>> +#\n>> +# 0000000 63 5f c3 a9\n>> +#\n>> +# (/usr/bin/od output contains different amount of whitespace\n>> +# on different platforms, so we need the wildcards here.)\n>> +#\n>> +test_lazy_prereq UNICODE_NFC_PRESERVED '\n>> +    mkdir c_${utf8_nfc} &&\n>> +    ls | od -t x1 | grep \"63 *5f *c3 *a9\"\n>> +'\n>> +\n>> +# Is the spelling of an NFD pathname preserved on disk?\n>> +#\n>> +# 0000000 64 5f 65 cc 81\n>> +#\n>> +test_lazy_prereq UNICODE_NFD_PRESERVED '\n>> +    mkdir d_${utf8_nfd} &&\n>> +    ls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n>> +'\n>> +    mkdir c_${utf8_nfc} &&\n>> +    mkdir d_${utf8_nfd} &&\n>> +\n>> +# The following _DOUBLE_ forms are more for my curiosity,\n>> +# but there may be quirks lurking when there are multiple\n>> +# combining characters in non-canonical order.\n>> +\n>> +# Unicode also allows multiple combining characters\n>> +# that can be decomposed in pieces.\n>> +#\n>> +# NFC:        U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n>> +# UTF8(NFC):  \\xe1 \\xbd \\xa7\n>> +#\n>> +# NFD1:       U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n>> +#             U+0342 COMBINING GREEK PERISPOMENI\n>> +# UTF8(NFD1): \\xe1 \\xbd \\xa1  +  \\xcd \\x82\n>> +#\n>> +# But U+1f61 decomposes into\n>> +# NFD2:       U+03c9 GREEK SMALL LETTER OMEGA\n>> +#             U+0314 COMBINING REVERSED COMMA ABOVE\n>> +# UTF8(NFD2): \\xcf \\x89  +  \\xcc \\x94\n>> +#\n>> +# Yielding:   \\xcf \\x89  +  \\xcc \\x94  +  \\xcd \\x82\n>> +#\n>> +# Note that I've used the canonical ordering of the\n>> +# combinining characters.  It is also possible to\n>> +# swap them.  My testing shows that that non-standard\n>> +# ordering also causes a collision in mkdir.  However,\n>> +# the resulting names don't draw correctly on the\n>> +# terminal (implying that the on-disk format also has\n>> +# them out of order).\n>> +#\n>> +greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n>> +greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n>> +greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n>> +\n> \n> The second test subject (greek) is more complex, right?\n\nYes, the Greek character is more interesting because\nthere can be multiple combining characters for some\nGreek letters. (I don't speak Greek, so I don't know\nhow often these occur in practice, but Unicode defines\nand allows them, so I'm testing whether these longer\nsequences are also munged by the OS.\n\nFWIW, there are other cases the we could try here,\nsuch as putting the combining characters in the other\norder. Or testing with a partially composed/decomposed\nstring (nfd1). Or testing with the partially composed\n/decomposed using the swapped ordering. But I left these\nout since for now -- it seemed a bit too far off into the\nweeds.\n\n> \n>> +# See if a double decomposition also collides.\n>> +#\n>> +test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n>> +    mkdir trial_${greek_nfc} &&\n>> +    mkdir trial_${greek_nfd2}\n>> +'\n>> +\n>> +# See if the NFC spelling appears on the disk.\n>> +#\n>> +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n>> +    mkdir c_${greek_nfc} &&\n>> +    ls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n>> +'\n>> +\n>> +# See if the NFD spelling appears on the disk.\n>> +#\n>> +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n>> +    mkdir d_${greek_nfd2} &&\n>> +    ls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n>> +'\n>> +\n>> +# The following is for debugging. I found it useful when\n>> +# trying to understand the various (OS, FS) quirks WRT\n>> +# Unicode and how composition/decomposition is handled.\n>> +# For example, when trying to understand how (macOS, APFS)\n>> +# and (macOS, HFS) and (macOS, FAT32) compare.\n>> +#\n>> +# It is rather noisy, so it is disabled by default.\n>> +#\n>> +if test \"$unicode_debug\" = \"true\"\n>> +then\n>> +    if test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n>> +    then\n>> +        echo NFC and NFD are distinct on this OS/filesystem.\n>> +    else\n>> +        echo NFC and NFD are aliases on this OS/filesystem.\n>> +    fi\n>> +\n>> +    if test_have_prereq UNICODE_NFC_PRESERVED\n>> +    then\n>> +        echo NFC maintains original spelling.\n>> +    else\n>> +        echo NFC is modified.\n>> +    fi\n>> +\n>> +    if test_have_prereq UNICODE_NFD_PRESERVED\n>> +    then\n>> +        echo NFD maintains original spelling.\n>> +    else\n>> +        echo NFD is modified.\n>> +    fi\n>> +\n>> +    if test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n>> +    then\n>> +        echo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n>> +    else\n>> +        echo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n>> +    fi\n>> +\n>> +    if test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n>> +    then\n>> +        echo Double NFC maintains original spelling.\n>> +    else\n>> +        echo Double NFC is modified.\n>> +    fi\n>> +\n>> +    if test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n>> +    then\n>> +        echo Double NFD maintains original spelling.\n>> +    else\n>> +        echo Double NFD is modified.\n>> +    fi\n>> +fi\n> \n> In general, this test is written from Mac OS perspective, but since we have\n> Git users also from Linux, Windows, and other OSes, I'd like to see from \n> these\n> other perspective.\n> \n\nRight, MacOS does the NFC/NFD aliasing. What I have here proves\nthat they do it and how/if they alter the spelling of paths on disk.\nAnd we can check the values of these prereqs for APFS, HFS+, and FAT32\non MacOS.\n\nI don't know of any other OS that does similar artificial aliasing, so\nyeah, these tests are somewhat Mac-specific. However, they have also\nbeen run on Windows and Linux confirm that they do not do such aliasing.\n\nThere are other non-Mac tests that we could do -- mainly around\nmalformed UTF8 paths in a commit -- Linux treats paths as just 8-bit\ndata, so there is no guarantee that a tree created on Linux would have\nvalid UTF8 paths -- just 8-bit paths. Importing that onto Windows or\nMac could create issues, since Windows wants to use UCS2 with the *W()\nroutines and MacOS wants to do the UTF8 aliasing. But all of that is\noutside the scope of this patch series.\n\nIn this series, we're trying to watch the journal/event log and report\nthe changed paths. We want to use the on-disk spelling of the modified\nfiles, but augment that with the alternate alias spelling when we can.\n\nHope this helps,\nJeff\n\n"},{"id":"452477","messageId":"20220328143704.md3chsipj7iucook@tb-raspi4","threadId":"57414","inReplyTo":"pull.1143.v3.git.1647973380.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/27] Builtin FSMonitor Part 3","fromName":"Torsten B��gershausen","fromEmail":"tboegi@web.de","sentAt":"2022-03-28T14:37:04Z","receivedAt":"2022-03-28T14:37:27Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Tue, Mar 22, 2022 at 06:22:33PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> Here is V3 of Part 3 of my builtin FSMonitor series.\n>\n> I have addressed all of the feedback from Part 3 V2 and the mess that was\n> Part 2.5 (now obsolete). This version builds upon the new V7 of Part 2\n> (which includes 2.5).\n\nThanks for the patches. I think that my feedback is included.\n/Torsten\n"},{"id":"453922","messageId":"220419.86mtgh8kep.gmgdl@evledraar.gmail.com","threadId":"57414","inReplyTo":"f2c0569c9012a86f252562a9a906f6de37d0a236.1648140680.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-04-19T09:44:19Z","receivedAt":"2022-04-19T09:58:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 24 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n> index 46be55a4618..50ae3cca575 100644\n> --- a/builtin/fsmonitor--daemon.c\n> +++ b/builtin/fsmonitor--daemon.c\n> @@ -1449,6 +1449,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n> \t\t fsmonitor__ipc_threads);\n\nI think that structurally the way things are done in\nfsmonitor-settings.c make its use really hard to follow. E.g. here:\n\n> +\tprepare_repo_settings(the_repository);\n\nWe prep the repo, OK.\n\n> +\tfsm_settings__set_ipc(the_repository);\n\nSet IPC.\n\n> +\tif (fsm_settings__error_if_incompatible(the_repository))\n\nAnd here we'll error out if we're incompatible, and this is in the\ntop-level cmd_fsmonitor__daemon() function. All OK, except why didn't we\ncheck this before \"set IPC?\".\n\nAnyway, re-arranging some of the diff below:\n\n> @@ -86,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r)\n> \n> \tlookup_fsmonitor_settings(r);\n> \n> +\tif (check_for_incompatible(r))\n> +\t\treturn;\n> +\n> \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n> \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n> }\n\nHere in fsm_settings__set_ipc we return with a NOOP if we're not\ncompatible.\n\nThen:\n\n> +int fsm_settings__error_if_incompatible(struct repository *r)\n> +{\n> +\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n> +\n> +\tswitch (reason) {\n> +\tcase FSMONITOR_REASON_OK:\n> +\t\treturn 0;\n> +\n> +\tcase FSMONITOR_REASON_BARE:\n> +\t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n> +\t\t xgetcwd());\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n> +\t reason);\n> +}\n\nHere we'll call fsm_settings__get_reason() which does the same.\n\n> +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n> +{\n> +\tif (!r)\n> +\t\tr = the_repository;\n> +\n> +\tlookup_fsmonitor_settings(r);\n> +\n> +\treturn r->settings.fsmonitor->reason;\n> +}\n\nIs there a reason we can't skip this indirection when using the API like\nthis and e.g. do:\n\n\tenum fsmonitor_reason reason;\n\tprepare_repo_settings(the_repository);\n\treason = fsmonitor_check_for_incompatible(the_repository)\n if (reason != FSMONITOR_REASON_OK)\n \tdie(\"%s\", fsm_settings__get_reason_string(reason));\n\nThere's just two callers of this API in \"seen\", and neither need/want\nthis pattern where every method needs to lazy load itself, or the\nindirection where fsmonitor-settings.c needs to be used as a\nclearing-house for state management.\n\nMaybe I'm missing something, but why not make check_for_incompatible()\nnon-static and have the callers use that (and then it would return\n\"fsmonitor_reason\", not \"int\", the int return value being redundant to\nthe enum)>\n\n> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> index 876112abb21..d29048f16f2 100644\n> --- a/builtin/update-index.c\n> +++ b/builtin/update-index.c\n> @@ -1237,6 +1237,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n> \n> \tif (fsmonitor > 0) {\n> \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n> +\n> +\t\tif (fsm_settings__error_if_incompatible(the_repository))\n> +\t\t\treturn 1;\n> +\n> \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n> \t\t\twarning(_(\"core.fsmonitor is unset; \"\n> \t\t\t\t\"set it if you really want to \"\n\nThis looks like a bug, we knew before aquiring the lockfile that we\nweren't compatible, so why wait until here to error out? This seems to\nskip the rollback_lock_file(), so won't we leave a stale lock?\n"},{"id":"454002","messageId":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v4.git.1648140680.gitgitgadget@gmail.com","subject":"[PATCH v5 00/28] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:50Z","receivedAt":"2022-04-20T20:43:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 5 of FSMonitor part 3. This version:\n\n * refactored the submodule test in commit 23 to let me reuse parts of it in\n a later commit.\n\n * Added a new commit 28 to silence a stray warning when FSMonitor is\n implicitly started in a recursive git submodule absorbgitdirs call and\n receives the --super-prefix argument. The warning was harmless, but was\n confusing to users.\n\nThis warning was seen by a user in our experimental release of FSMonitor in\nGit-for-Windows.\n\n1: 23bfb8c516 = 1: 8b7c5f4e23 fsm-listen-win32: handle shortnames\n 2: d14f171460 = 2: 5b246bec24 t7527: test FSMonitor on repos with Unicode root paths\n 3: 4db2370d04 = 3: 8a474d6999 t/helper/fsmonitor-client: create stress test\n 4: f2c0569c90 = 4: 004b67b62e fsmonitor-settings: bare repos are incompatible with FSMonitor\n 5: b2599bb9d2 = 5: e1e55550c1 fsmonitor-settings: stub in Win32-specific incompatibility checking\n 6: 9ad6d87ccc = 6: 2d68fc9a46 fsmonitor-settings: VFS for Git virtual repos are incompatible\n 7: 7652c79ab3 = 7: 94ae2e424f fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: 2f2a523552 = 8: b2ca6c1b20 fsmonitor-settings: remote repos on macOS are incompatible\n 9: 0297d80388 = 9: a3cc4b3b16 fsmonitor-settings: remote repos on Windows are incompatible\n10: b6dfd609ad = 10: 8f1f484075 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n11: db5197b44b = 11: 8d48d9c562 unpack-trees: initialize fsmonitor_has_run_once in o->result\n12: 3f154d0251 = 12: 088c7b3334 fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n13: 4aade7b560 = 13: 00fab62666 fsmonitor--daemon: cd out of worktree root\n14: d8ebac2a9b = 14: 6552f51802 fsmonitor--daemon: prepare for adding health thread\n15: 7fb0795e25 = 15: f2bf07cd73 fsmonitor--daemon: rename listener thread related variables\n16: e90adcd06d = 16: 2a44f2eded fsmonitor--daemon: stub in health thread\n17: d9b91a998c = 17: 854fb5e365 fsm-health-win32: add polling framework to monitor daemon health\n18: 0e95ee0d01 = 18: 3af1fe0d61 fsm-health-win32: force shutdown daemon if worktree root moves\n19: 48a590d202 = 19: f1365cdd40 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n20: 36ab239fd9 = 20: 15698d64ed fsmonitor: optimize processing of directory events\n21: 3010b22e69 = 21: 9d0da8fc22 t7527: FSMonitor tests for directory moves\n22: 85cdb4d84f = 22: 040c00cfd6 t/perf/p7527: add perf test for builtin FSMonitor\n23: 29063455c8 ! 23: 5db241f7d2 fsmonitor: never set CE_FSMONITOR_VALID on submodules\n @@ t/t7527-builtin-fsmonitor.sh: do\n +# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n +#\n +# It is therefore important that the top level status not be tricked\n -+# by the FSMonitor response to skip those recursive calls.\n ++# by the FSMonitor response to skip those recursive calls. That is,\n ++# even if FSMonitor says that the mtime of the submodule directory\n ++# hasn't changed and it could be implicitly marked valid, we must\n ++# not take that shortcut. We need to force the recusion into the\n ++# submodule so that we get a summary of the status *within* the\n ++# submodule.\n ++\n ++create_super () {\n ++\tsuper=$1 &&\n ++\n ++\tgit init \"${super}\" &&\n ++\techo x >${super}/file_1 &&\n ++\techo y >${super}/file_2 &&\n ++\techo z >${super}/file_3 &&\n ++\tmkdir ${super}/dir_1 &&\n ++\techo a >${super}/dir_1/file_11 &&\n ++\techo b >${super}/dir_1/file_12 &&\n ++\tmkdir ${super}/dir_1/dir_2 &&\n ++\techo a >${super}/dir_1/dir_2/file_21 &&\n ++\techo b >${super}/dir_1/dir_2/file_22 &&\n ++\tgit -C ${super} add . &&\n ++\tgit -C ${super} commit -m \"initial ${super} commit\"\n ++}\n ++\n ++create_sub () {\n ++\tsub=$1 &&\n ++\n ++\tgit init \"${sub}\" &&\n ++\techo x >${sub}/file_x &&\n ++\techo y >${sub}/file_y &&\n ++\techo z >${sub}/file_z &&\n ++\tmkdir ${sub}/dir_x &&\n ++\techo a >${sub}/dir_x/file_a &&\n ++\techo b >${sub}/dir_x/file_b &&\n ++\tmkdir ${sub}/dir_x/dir_y &&\n ++\techo a >${sub}/dir_x/dir_y/file_a &&\n ++\techo b >${sub}/dir_x/dir_y/file_b &&\n ++\tgit -C ${sub} add . &&\n ++\tgit -C ${sub} commit -m \"initial ${sub} commit\"\n ++}\n +\n +my_match_and_clean () {\n +\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n @@ t/t7527-builtin-fsmonitor.sh: do\n +\tgit -C super/dir_1/dir_2/sub clean -d -f\n +}\n +\n -+test_expect_success \"Submodule\" '\n -+\ttest_when_finished \"git -C super fsmonitor--daemon stop\" &&\n -+\n -+\tgit init \"super\" &&\n -+\techo x >super/file_1 &&\n -+\techo y >super/file_2 &&\n -+\techo z >super/file_3 &&\n -+\tmkdir super/dir_1 &&\n -+\techo a >super/dir_1/file_11 &&\n -+\techo b >super/dir_1/file_12 &&\n -+\tmkdir super/dir_1/dir_2 &&\n -+\techo a >super/dir_1/dir_2/file_21 &&\n -+\techo b >super/dir_1/dir_2/file_22 &&\n -+\tgit -C super add . &&\n -+\tgit -C super commit -m \"initial super commit\" &&\n -+\n -+\tgit init \"sub\" &&\n -+\techo x >sub/file_x &&\n -+\techo y >sub/file_y &&\n -+\techo z >sub/file_z &&\n -+\tmkdir sub/dir_x &&\n -+\techo a >sub/dir_x/file_a &&\n -+\techo b >sub/dir_x/file_b &&\n -+\tmkdir sub/dir_x/dir_y &&\n -+\techo a >sub/dir_x/dir_y/file_a &&\n -+\techo b >sub/dir_x/dir_y/file_b &&\n -+\tgit -C sub add . &&\n -+\tgit -C sub commit -m \"initial sub commit\" &&\n ++test_expect_success \"Submodule always visited\" '\n ++\ttest_when_finished \"git -C super fsmonitor--daemon stop; \\\n ++\t\t\t rm -rf super; \\\n ++\t\t\t rm -rf sub\" &&\n ++\n ++\tcreate_super \"super\" &&\n ++\tcreate_sub \"sub\" &&\n +\n +\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n +\tgit -C super commit -m \"add sub\" &&\n24: 6e99f5e4f2 ! 24: 93de3707d2 t7527: test FSMonitor on case insensitive+preserving file system\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## t/t7527-builtin-fsmonitor.sh ##\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule\" '\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n my_match_and_clean\n '\n \n25: cef7dbbaf0 = 25: d890c2e2d9 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n26: bc2d5a7a93 = 26: 7c60623555 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n27: 176c530c3f = 27: 9724c41d18 t7527: test Unicode NFC/NFD handling on MacOS\n -: ---------- > 28: b8325fb7c7 fsmonitor--daemon: allow --super-prefix argument\n\n\nJeff Hostetler (28):\n fsm-listen-win32: handle shortnames\n t7527: test FSMonitor on repos with Unicode root paths\n t/helper/fsmonitor-client: create stress test\n fsmonitor-settings: bare repos are incompatible with FSMonitor\n fsmonitor-settings: stub in Win32-specific incompatibility checking\n fsmonitor-settings: VFS for Git virtual repos are incompatible\n fsmonitor-settings: stub in macOS-specific incompatibility checking\n fsmonitor-settings: remote repos on macOS are incompatible\n fsmonitor-settings: remote repos on Windows are incompatible\n fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n unpack-trees: initialize fsmonitor_has_run_once in o->result\n fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n fsmonitor--daemon: cd out of worktree root\n fsmonitor--daemon: prepare for adding health thread\n fsmonitor--daemon: rename listener thread related variables\n fsmonitor--daemon: stub in health thread\n fsm-health-win32: add polling framework to monitor daemon health\n fsm-health-win32: force shutdown daemon if worktree root moves\n fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n fsmonitor: optimize processing of directory events\n t7527: FSMonitor tests for directory moves\n t/perf/p7527: add perf test for builtin FSMonitor\n fsmonitor: never set CE_FSMONITOR_VALID on submodules\n t7527: test FSMonitor on case insensitive+preserving file system\n fsmonitor: on macOS also emit NFC spelling for NFD pathname\n t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n t7527: test Unicode NFC/NFD handling on MacOS\n fsmonitor--daemon: allow --super-prefix argument\n\n Makefile | 19 +-\n builtin/fsmonitor--daemon.c | 104 ++++++-\n builtin/update-index.c | 4 +\n compat/fsmonitor/fsm-health-darwin.c | 24 ++\n compat/fsmonitor/fsm-health-win32.c | 278 +++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 +++\n compat/fsmonitor/fsm-listen-darwin.c | 122 ++++++--\n compat/fsmonitor/fsm-listen-win32.c | 413 ++++++++++++++++++++-----\n compat/fsmonitor/fsm-listen.h | 2 +-\n compat/fsmonitor/fsm-settings-darwin.c | 89 ++++++\n compat/fsmonitor/fsm-settings-win32.c | 137 ++++++++\n config.mak.uname | 5 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 11 +-\n fsmonitor-settings.c | 92 ++++++\n fsmonitor-settings.h | 29 ++\n fsmonitor.c | 73 ++++-\n fsmonitor.h | 11 +\n git.c | 2 +-\n t/helper/test-fsmonitor-client.c | 106 +++++++\n t/lib-unicode-nfc-nfd.sh | 167 ++++++++++\n t/perf/p7527-builtin-fsmonitor.sh | 257 +++++++++++++++\n t/t7519-status-fsmonitor.sh | 32 ++\n t/t7527-builtin-fsmonitor.sh | 367 ++++++++++++++++++++++\n unpack-trees.c | 1 +\n 25 files changed, 2275 insertions(+), 125 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\n\nbase-commit: 5eb696daba2fe108d4d9ba2ccf4b357447ef9946\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/1143\n\nRange-diff vs v4:\n\n 1: 23bfb8c5165 = 1: 8b7c5f4e234 fsm-listen-win32: handle shortnames\n 2: d14f1714604 = 2: 5b246bec247 t7527: test FSMonitor on repos with Unicode root paths\n 3: 4db2370d046 = 3: 8a474d69999 t/helper/fsmonitor-client: create stress test\n 4: f2c0569c901 = 4: 004b67b62e3 fsmonitor-settings: bare repos are incompatible with FSMonitor\n 5: b2599bb9d2e = 5: e1e55550c10 fsmonitor-settings: stub in Win32-specific incompatibility checking\n 6: 9ad6d87ccce = 6: 2d68fc9a46a fsmonitor-settings: VFS for Git virtual repos are incompatible\n 7: 7652c79ab35 = 7: 94ae2e424f1 fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: 2f2a5235522 = 8: b2ca6c1b201 fsmonitor-settings: remote repos on macOS are incompatible\n 9: 0297d80388a = 9: a3cc4b3b16d fsmonitor-settings: remote repos on Windows are incompatible\n 10: b6dfd609adb = 10: 8f1f4840751 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n 11: db5197b44bb = 11: 8d48d9c5623 unpack-trees: initialize fsmonitor_has_run_once in o->result\n 12: 3f154d02517 = 12: 088c7b3334c fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n 13: 4aade7b560a = 13: 00fab626663 fsmonitor--daemon: cd out of worktree root\n 14: d8ebac2a9b2 = 14: 6552f51802b fsmonitor--daemon: prepare for adding health thread\n 15: 7fb0795e25e = 15: f2bf07cd739 fsmonitor--daemon: rename listener thread related variables\n 16: e90adcd06db = 16: 2a44f2eded1 fsmonitor--daemon: stub in health thread\n 17: d9b91a998ce = 17: 854fb5e3658 fsm-health-win32: add polling framework to monitor daemon health\n 18: 0e95ee0d01b = 18: 3af1fe0d61d fsm-health-win32: force shutdown daemon if worktree root moves\n 19: 48a590d2026 = 19: f1365cdd40c fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n 20: 36ab239fd9a = 20: 15698d64edd fsmonitor: optimize processing of directory events\n 21: 3010b22e690 = 21: 9d0da8fc22b t7527: FSMonitor tests for directory moves\n 22: 85cdb4d84f2 = 22: 040c00cfd6f t/perf/p7527: add perf test for builtin FSMonitor\n 23: 29063455c83 ! 23: 5db241f7d2f fsmonitor: never set CE_FSMONITOR_VALID on submodules\n @@ t/t7527-builtin-fsmonitor.sh: do\n +# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n +#\n +# It is therefore important that the top level status not be tricked\n -+# by the FSMonitor response to skip those recursive calls.\n ++# by the FSMonitor response to skip those recursive calls. That is,\n ++# even if FSMonitor says that the mtime of the submodule directory\n ++# hasn't changed and it could be implicitly marked valid, we must\n ++# not take that shortcut. We need to force the recusion into the\n ++# submodule so that we get a summary of the status *within* the\n ++# submodule.\n ++\n ++create_super () {\n ++\tsuper=$1 &&\n ++\n ++\tgit init \"${super}\" &&\n ++\techo x >${super}/file_1 &&\n ++\techo y >${super}/file_2 &&\n ++\techo z >${super}/file_3 &&\n ++\tmkdir ${super}/dir_1 &&\n ++\techo a >${super}/dir_1/file_11 &&\n ++\techo b >${super}/dir_1/file_12 &&\n ++\tmkdir ${super}/dir_1/dir_2 &&\n ++\techo a >${super}/dir_1/dir_2/file_21 &&\n ++\techo b >${super}/dir_1/dir_2/file_22 &&\n ++\tgit -C ${super} add . &&\n ++\tgit -C ${super} commit -m \"initial ${super} commit\"\n ++}\n ++\n ++create_sub () {\n ++\tsub=$1 &&\n ++\n ++\tgit init \"${sub}\" &&\n ++\techo x >${sub}/file_x &&\n ++\techo y >${sub}/file_y &&\n ++\techo z >${sub}/file_z &&\n ++\tmkdir ${sub}/dir_x &&\n ++\techo a >${sub}/dir_x/file_a &&\n ++\techo b >${sub}/dir_x/file_b &&\n ++\tmkdir ${sub}/dir_x/dir_y &&\n ++\techo a >${sub}/dir_x/dir_y/file_a &&\n ++\techo b >${sub}/dir_x/dir_y/file_b &&\n ++\tgit -C ${sub} add . &&\n ++\tgit -C ${sub} commit -m \"initial ${sub} commit\"\n ++}\n +\n +my_match_and_clean () {\n +\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n @@ t/t7527-builtin-fsmonitor.sh: do\n +\tgit -C super/dir_1/dir_2/sub clean -d -f\n +}\n +\n -+test_expect_success \"Submodule\" '\n -+\ttest_when_finished \"git -C super fsmonitor--daemon stop\" &&\n -+\n -+\tgit init \"super\" &&\n -+\techo x >super/file_1 &&\n -+\techo y >super/file_2 &&\n -+\techo z >super/file_3 &&\n -+\tmkdir super/dir_1 &&\n -+\techo a >super/dir_1/file_11 &&\n -+\techo b >super/dir_1/file_12 &&\n -+\tmkdir super/dir_1/dir_2 &&\n -+\techo a >super/dir_1/dir_2/file_21 &&\n -+\techo b >super/dir_1/dir_2/file_22 &&\n -+\tgit -C super add . &&\n -+\tgit -C super commit -m \"initial super commit\" &&\n -+\n -+\tgit init \"sub\" &&\n -+\techo x >sub/file_x &&\n -+\techo y >sub/file_y &&\n -+\techo z >sub/file_z &&\n -+\tmkdir sub/dir_x &&\n -+\techo a >sub/dir_x/file_a &&\n -+\techo b >sub/dir_x/file_b &&\n -+\tmkdir sub/dir_x/dir_y &&\n -+\techo a >sub/dir_x/dir_y/file_a &&\n -+\techo b >sub/dir_x/dir_y/file_b &&\n -+\tgit -C sub add . &&\n -+\tgit -C sub commit -m \"initial sub commit\" &&\n ++test_expect_success \"Submodule always visited\" '\n ++\ttest_when_finished \"git -C super fsmonitor--daemon stop; \\\n ++\t\t\t rm -rf super; \\\n ++\t\t\t rm -rf sub\" &&\n ++\n ++\tcreate_super \"super\" &&\n ++\tcreate_sub \"sub\" &&\n +\n +\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n +\tgit -C super commit -m \"add sub\" &&\n 24: 6e99f5e4f2a ! 24: 93de3707d26 t7527: test FSMonitor on case insensitive+preserving file system\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## t/t7527-builtin-fsmonitor.sh ##\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule\" '\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n \tmy_match_and_clean\n '\n \n 25: cef7dbbaf04 = 25: d890c2e2d97 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n 26: bc2d5a7a930 = 26: 7c606235557 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n 27: 176c530c3fa = 27: 9724c41d18d t7527: test Unicode NFC/NFD handling on MacOS\n -: ----------- > 28: b8325fb7c78 fsmonitor--daemon: allow --super-prefix argument\n\n-- \ngitgitgadget\n"},{"id":"454003","messageId":"8b7c5f4e234e5b139b640652fcfdafb2e24e9db8.1650487398.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 01/28] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:51Z","receivedAt":"2022-04-20T20:43:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach FSMonitor daemon on Windows to recognize shortname paths as\naliases of normal longname paths. FSMonitor clients, such as `git\nstatus`, should receive the longname spelling of changed files (when\npossible).\n\nSometimes we receive FS events using the shortname, such as when a CMD\nshell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\narrives using whatever combination of long and shortnames were used by\nthe other process. (Shortnames do seem to be case normalized,\nhowever.)\n\nUse Windows GetLongPathNameW() to try to map the pathname spelling in\nthe notification event into the normalized longname spelling. (This\ncan fail if the file/directory is deleted, moved, or renamed, because\nwe are asking the FS for the mapping in response to the event and\nafter it has already happened, but we try.)\n\nSpecial case the shortname spelling of \".git\" to avoid under-reporting\nthese events.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++-----\n t/t7527-builtin-fsmonitor.sh | 65 +++++\n 2 files changed, 374 insertions(+), 54 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 5b928ab66e5..3f1b68267bd 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -25,6 +25,9 @@ struct one_watch\n \tDWORD count;\n \n \tstruct strbuf path;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n+\tDWORD wpath_longname_len;\n+\n \tHANDLE hDir;\n \tHANDLE hEvent;\n \tOVERLAPPED overlapped;\n@@ -34,6 +37,21 @@ struct one_watch\n \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n \t */\n \tBOOL is_active;\n+\n+\t/*\n+\t * Are shortnames enabled on the containing drive? This is\n+\t * always true for \"C:/\" drives and usually never true for\n+\t * other drives.\n+\t *\n+\t * We only set this for the worktree because we only need to\n+\t * convert shortname paths to longname paths for items we send\n+\t * to clients. (We don't care about shortname expansion for\n+\t * paths inside a GITDIR because we never send them to\n+\t * clients.)\n+\t */\n+\tBOOL has_shortnames;\n+\tBOOL has_tilda;\n+\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n struct fsmonitor_daemon_backend_data\n@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n };\n \n /*\n- * Convert the WCHAR path from the notification into UTF8 and\n- * then normalize it.\n+ * Convert the WCHAR path from the event into UTF8 and normalize it.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n */\n-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n \t\t\t\t struct strbuf *normalized_path)\n {\n \tint reserve;\n \tint len = 0;\n \n \tstrbuf_reset(normalized_path);\n-\tif (!info->FileNameLength)\n+\tif (!wpath_len)\n \t\tgoto normalize;\n \n \t/*\n@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t * sequence of 2 UTF8 characters. That should let us\n \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n \t */\n-\treserve = info->FileNameLength + 1;\n+\treserve = 2 * wpath_len + 1;\n \tstrbuf_grow(normalized_path, reserve);\n \n \tfor (;;) {\n-\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n-\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n+\t\t\t\t\t wpath, wpath_len,\n \t\t\t\t\t normalized_path->buf,\n \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n \t\t\t\t\t NULL, NULL);\n@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n \t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n-\t\t\t GetLastError(),\n-\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n-\t\t\t info->FileName);\n+\t\t\t GetLastError(), (int)wpath_len, wpath);\n \t\t\treturn -1;\n \t\t}\n \n@@ -98,6 +115,152 @@ normalize:\n \treturn strbuf_normalize_path(normalized_path);\n }\n \n+/*\n+ * See if the worktree root directory has shortnames enabled.\n+ * This will help us decide if we need to do an expensive shortname\n+ * to longname conversion on every notification event.\n+ *\n+ * We do not want to create a file to test this, so we assume that the\n+ * root directory contains a \".git\" file or directory. (Our caller\n+ * only calls us for the worktree root, so this should be fine.)\n+ *\n+ * Remember the spelling of the shortname for \".git\" if it exists.\n+ */\n+static void check_for_shortnames(struct one_watch *watch)\n+{\n+\twchar_t buf_in[MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\twchar_t *last_slash = NULL;\n+\twchar_t *last_bslash = NULL;\n+\twchar_t *last;\n+\n+\t/* build L\"/.git\" */\n+\twcscpy(buf_in, watch->wpath_longname);\n+\twcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n+\n+\tif (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n+\t\treturn;\n+\n+\tlast_slash = wcsrchr(buf_out, L'/');\n+\tlast_bslash = wcsrchr(buf_out, L'\\\\');\n+\tif (last_slash > last_bslash)\n+\t\tlast = last_slash + 1;\n+\telse if (last_bslash)\n+\t\tlast = last_bslash + 1;\n+\telse\n+\t\tlast = buf_out;\n+\n+\tif (!wcscmp(last, L\".git\"))\n+\t\treturn;\n+\n+\twatch->has_shortnames = 1;\n+\twcsncpy(watch->dotgit_shortname, last,\n+\t\tARRAY_SIZE(watch->dotgit_shortname));\n+\n+\t/*\n+\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n+\t * we should be able to avoid shortname to longname mapping on\n+\t * every notification event if the source string does not\n+\t * contain a \"~\".\n+\t *\n+\t * However, the documentation for GetLongPathNameW() says\n+\t * that there are filesystems that don't follow that pattern\n+\t * and warns against this optimization.\n+\t *\n+\t * Lets test this.\n+\t */\n+\tif (wcschr(watch->dotgit_shortname, L'~'))\n+\t\twatch->has_tilda = 1;\n+}\n+\n+enum get_relative_result {\n+\tGRR_NO_CONVERSION_NEEDED,\n+\tGRR_HAVE_CONVERSION,\n+\tGRR_SHUTDOWN,\n+};\n+\n+/*\n+ * Info notification paths are relative to the root of the watch.\n+ * If our CWD is still at the root, then we can use relative paths\n+ * to convert from shortnames to longnames. If our process has a\n+ * different CWD, then we need to construct an absolute path, do\n+ * the conversion, and then return the root-relative portion.\n+ *\n+ * We use the longname form of the root as our basis and assume that\n+ * it already has a trailing slash.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n+ */\n+static enum get_relative_result get_relative_longname(\n+\tstruct one_watch *watch,\n+\tconst wchar_t *wpath, DWORD wpath_len,\n+\twchar_t *wpath_longname)\n+{\n+\twchar_t buf_in[2 * MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\tDWORD root_len;\n+\n+\t/* Build L\"/\" */\n+\troot_len = watch->wpath_longname_len;\n+\twcsncpy(buf_in, watch->wpath_longname, root_len);\n+\twcsncpy(buf_in + root_len, wpath, wpath_len);\n+\tbuf_in[root_len + wpath_len] = 0;\n+\n+\t/*\n+\t * We don't actually know if the source pathname is a\n+\t * shortname or a longname. This routine allows either to be\n+\t * given as input.\n+\t */\n+\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n+\t\t/*\n+\t\t * The shortname to longname conversion can fail for\n+\t\t * various reasons, for example if the file has been\n+\t\t * deleted. (That is, if we just received a\n+\t\t * delete-file notification event and the file is\n+\t\t * already gone, we can't ask the file system to\n+\t\t * lookup the longname for it. Likewise, for moves\n+\t\t * and renames where we are given the old name.)\n+\t\t *\n+\t\t * Since deleting or moving a file or directory by its\n+\t\t * shortname is rather obscure, I'm going ignore the\n+\t\t * failure and ask the caller to report the original\n+\t\t * relative path. This seems kinder than failing here\n+\t\t * and forcing a resync. Besides, forcing a resync on\n+\t\t * every file/directory delete would effectively\n+\t\t * cripple monitoring.\n+\t\t *\n+\t\t * We might revisit this in the future.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (!wcscmp(buf_in, buf_out)) {\n+\t\t/*\n+\t\t * The path does not have a shortname alias.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (wcsncmp(buf_in, buf_out, root_len)) {\n+\t\t/*\n+\t\t * The spelling of the root directory portion of the computed\n+\t\t * longname has changed. This should not happen. Basically,\n+\t\t * it means that we don't know where (without recomputing the\n+\t\t * longname of just the root directory) to split out the\n+\t\t * relative path. Since this should not happen, I'm just\n+\t\t * going to let this fail and force a shutdown (because all\n+\t\t * subsequent events are probably going to see the same\n+\t\t * mismatch).\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\t/* Return the worktree root-relative portion of the longname. */\n+\n+\twcscpy(wpath_longname, buf_out + root_len);\n+\treturn GRR_HAVE_CONVERSION;\n+}\n+\n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n@@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n-\twchar_t wpath[MAX_PATH];\n+\tDWORD len_longname;\n+\twchar_t wpath[MAX_PATH + 1];\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \tif (xutftowcs_path(wpath, path) < 0) {\n \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n@@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \t\treturn NULL;\n \t}\n \n+\tif (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n+\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n+\t\t GetLastError(), path);\n+\t\tCloseHandle(hDir);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen_longname = wcslen(wpath_longname);\n+\tif (wpath_longname[len_longname - 1] != L'/' &&\n+\t wpath_longname[len_longname - 1] != L'\\\\') {\n+\t\twpath_longname[len_longname++] = L'/';\n+\t\twpath_longname[len_longname] = 0;\n+\t}\n+\n \tCALLOC_ARRAY(watch, 1);\n \n \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n@@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tstrbuf_init(&watch->path, 0);\n \tstrbuf_addstr(&watch->path, path);\n \n+\twcscpy(watch->wpath_longname, wpath_longname);\n+\twatch->wpath_longname_len = len_longname;\n+\n \twatch->hDir = hDir;\n \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n \n@@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t,\n+\tDWORD info_action)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n+\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n \tstruct fsmonitor_batch *batch = NULL;\n \tconst char *p = watch->buffer;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \t/*\n \t * If the kernel gets more events than will fit in the kernel\n@@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n+\t\twchar_t *wpath = info->FileName;\n+\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n \t\tenum fsmonitor_path_type t;\n+\t\tenum get_relative_result grr;\n+\n+\t\tif (watch->has_shortnames) {\n+\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n+\t\t\t\t/*\n+\t\t\t\t * This event exactly matches the\n+\t\t\t\t * spelling of the shortname of\n+\t\t\t\t * \".git\", so we can skip some steps.\n+\t\t\t\t *\n+\t\t\t\t * (This case is odd because the user\n+\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n+\t\t\t\t * use the filesystem to map it back\n+\t\t\t\t * to \".git\".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_reset(&path);\n+\t\t\t\tstrbuf_addstr(&path, \".git\");\n+\t\t\t\tt = IS_DOT_GIT;\n+\t\t\t\tgoto process_it;\n+\t\t\t}\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n-\t\t\tgoto skip_this_path;\n-\n-\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n-\n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n+\t\t\tif (watch->has_tilda && !wcschr(wpath, L'~')) {\n+\t\t\t\t/*\n+\t\t\t\t * Shortnames on this filesystem have tildas\n+\t\t\t\t * and the notification path does not have\n+\t\t\t\t * one, so we assume that it is a longname.\n+\t\t\t\t */\n+\t\t\t\tgoto normalize_it;\n+\t\t\t}\n \n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n+\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n+\t\t\t\t\t\t wpath_longname);\n+\t\t\tswitch (grr) {\n+\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n+\t\t\t\tbreak;\n+\t\t\tcase GRR_HAVE_CONVERSION:\n+\t\t\t\twpath = wpath_longname;\n+\t\t\t\twpath_len = wcslen(wpath);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\tcase GRR_SHUTDOWN:\n \t\t\t\tgoto force_shutdown;\n \t\t\t}\n-\t\t\tbreak;\n+\t\t}\n \n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n+normalize_it:\n+\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n+\t\t\tgoto skip_this_path;\n \n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+process_it:\n+\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n+\t\t\t\t\t info->Action))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -382,6 +630,9 @@ force_shutdown:\n * Note that we DO NOT get filesystem events on the external \n * itself (it is not inside something that we are watching). In particular,\n * we do not get an event if the external is deleted.\n+ *\n+ * Also, we do not care about shortnames within the external , since\n+ * we never send these paths to clients.\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n@@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\tif (normalize_path_in_utf8(\n+\t\t\t info->FileName,\n+\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t &path) == -1)\n \t\t\tgoto skip_this_path;\n \n \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n@@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \tif (!data->watch_worktree)\n \t\tgoto failed;\n \n+\tcheck_for_shortnames(data->watch_worktree);\n+\n \tif (state->nr_paths_watching > 1) {\n \t\tdata->watch_gitdir = create_watch(state,\n \t\t\t\t\t\t state->path_gitdir_watch.buf);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex bd0c952a116..1be21785162 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -166,6 +166,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon -C test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n+\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n+\n+\tstart_daemon -C test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"454004","messageId":"5b246bec2470431898ddc74013f6fb325c2bc281.1650487398.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 02/28] t7527: test FSMonitor on repos with Unicode root paths","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:52Z","receivedAt":"2022-04-20T20:43:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate some test repos with UTF8 characters in the pathname of the\nroot directory and verify that the builtin FSMonitor can watch them.\n\nThis test is mainly for Windows where we need to avoid `*A()`\nroutines.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1be21785162..c9c7dd77e3c 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -671,4 +671,27 @@ do\n \tdone\n done\n \n+# Test Unicode UTF-8 characters in the pathname of the working\n+# directory root. Use of \"*A()\" routines rather than \"*W()\" routines\n+# on Windows can sometimes lead to odd failures.\n+#\n+u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n+u2=$(printf \"u_e2_99_ab__\\xE2\\x99\\xAB\")\n+u_values=\"$u1 $u2\"\n+for u in $u_values\n+do\n+\ttest_expect_success \"Unicode in repo root path: $u\" '\n+\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n+\n+\t\tgit init \"$u\" &&\n+\t\techo 1 >\"$u\"/file1 &&\n+\t\tgit -C \"$u\" add file1 &&\n+\t\tgit -C \"$u\" config core.fsmonitor true &&\n+\n+\t\tstart_daemon -C \"$u\" &&\n+\t\tgit -C \"$u\" status >actual &&\n+\t\tgrep \"new file: file1\" actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"454005","messageId":"8a474d69999514b343ff576ca1d96180c22b403d.1650487398.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 03/28] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:53Z","receivedAt":"2022-04-20T20:43:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a stress test to hammer on the fsmonitor daemon.\nCreate a client-side thread pool of n threads and have\neach of them make m requests as fast as they can.\n\nWe do not currently inspect the contents of the response.\nWe're only interested in placing a heavy request load on\nthe daemon.\n\nThis test is useful for interactive testing and various\nexperimentation. For example, to place additional load\non the daemon while another test is running. We currently\ndo not have a test script that actually uses this helper.\nWe might add such a test in the future.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++\n 1 file changed, 106 insertions(+)\n\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nindex 3062c8a3c2b..54a4856c48c 100644\n--- a/t/helper/test-fsmonitor-client.c\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -7,6 +7,8 @@\n #include \"cache.h\"\n #include \"parse-options.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2.h\"\n \n #ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n int cmd__fsmonitor_client(int argc, const char **argv)\n@@ -79,20 +81,121 @@ static int do_send_flush(void)\n \treturn 0;\n }\n \n+struct hammer_thread_data\n+{\n+\tpthread_t pthread_id;\n+\tint thread_nr;\n+\n+\tint nr_requests;\n+\tconst char *token;\n+\n+\tint sum_successful;\n+\tint sum_errors;\n+};\n+\n+static void *hammer_thread_proc(void *_hammer_thread_data)\n+{\n+\tstruct hammer_thread_data *data = _hammer_thread_data;\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint k;\n+\tint ret;\n+\n+\ttrace2_thread_start(\"hammer\");\n+\n+\tfor (k = 0; k < data->nr_requests; k++) {\n+\t\tstrbuf_reset(&answer);\n+\n+\t\tret = fsmonitor_ipc__send_query(data->token, &answer);\n+\t\tif (ret < 0)\n+\t\t\tdata->sum_errors++;\n+\t\telse\n+\t\t\tdata->sum_successful++;\n+\t}\n+\n+\tstrbuf_release(&answer);\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Start a pool of client threads that will each send a series of\n+ * commands to the daemon.\n+ *\n+ * The goal is to overload the daemon with a sustained series of\n+ * concurrent requests.\n+ */\n+static int do_hammer(const char *token, int nr_threads, int nr_requests)\n+{\n+\tstruct hammer_thread_data *data = NULL;\n+\tint k;\n+\tint sum_join_errors = 0;\n+\tint sum_commands = 0;\n+\tint sum_errors = 0;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\tif (nr_threads < 1)\n+\t\tnr_threads = 1;\n+\tif (nr_requests < 1)\n+\t\tnr_requests = 1;\n+\n+\tCALLOC_ARRAY(data, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\t\tp->thread_nr = k;\n+\t\tp->nr_requests = nr_requests;\n+\t\tp->token = token;\n+\n+\t\tif (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {\n+\t\t\twarning(\"failed to create thread[%d] skipping remainder\", k);\n+\t\t\tnr_threads = k;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\n+\t\tif (pthread_join(p->pthread_id, NULL))\n+\t\t\tsum_join_errors++;\n+\t\tsum_commands += p->sum_successful;\n+\t\tsum_errors += p->sum_errors;\n+\t}\n+\n+\tfprintf(stderr, \"HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\\n\",\n+\t\tnr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);\n+\n+\tfree(data);\n+\n+\t/*\n+\t * Return an error if any of the _send_query requests failed.\n+\t * We don't care about thread create/join errors.\n+\t */\n+\treturn sum_errors > 0;\n+}\n+\n int cmd__fsmonitor_client(int argc, const char **argv)\n {\n \tconst char *subcmd;\n \tconst char *token = NULL;\n+\tint nr_threads = 1;\n+\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n \t\t\"test-tool fsmonitor-client query []\",\n \t\t\"test-tool fsmonitor-client flush\",\n+\t\t\"test-tool fsmonitor-client hammer [] [] []\",\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n \t\tOPT_STRING(0, \"token\", &token, \"token\",\n \t\t\t \"command token to send to the server\"),\n+\n+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, \"number of client threads\"),\n+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, \"number of requests per thread\"),\n+\n \t\tOPT_END()\n \t};\n \n@@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)\n \tif (!strcmp(subcmd, \"flush\"))\n \t\treturn !!do_send_flush();\n \n+\tif (!strcmp(subcmd, \"hammer\"))\n+\t\treturn !!do_hammer(token, nr_threads, nr_requests);\n+\n \tdie(\"Unhandled subcommand: '%s'\", subcmd);\n }\n #endif\n-- \ngitgitgadget\n\n"},{"id":"454006","messageId":"004b67b62e3717c9daeb6dcda1e9d84d44fa4984.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 04/28] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:54Z","receivedAt":"2022-04-20T20:43:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a worktree, so there is nothing for the\ndaemon watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 ++++\n builtin/update-index.c | 4 +++\n fsmonitor-settings.c | 60 +++++++++++++++++++++++++++++++++++++\n fsmonitor-settings.h | 12 ++++++++\n t/t7519-status-fsmonitor.sh | 23 ++++++++++++++\n 5 files changed, 105 insertions(+)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 46be55a4618..50ae3cca575 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1449,6 +1449,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\tfsm_settings__set_ipc(the_repository);\n+\n+\tif (fsm_settings__error_if_incompatible(the_repository))\n+\t\treturn 1;\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 876112abb21..d29048f16f2 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1237,6 +1237,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\n+\t\tif (fsm_settings__error_if_incompatible(the_repository))\n+\t\t\treturn 1;\n+\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 757d230d538..86c09bd35fe 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -9,9 +9,33 @@\n */\n struct fsmonitor_settings {\n \tenum fsmonitor_mode mode;\n+\tenum fsmonitor_reason reason;\n \tchar *hook_path;\n };\n \n+static void set_incompatible(struct repository *r,\n+\t\t\t enum fsmonitor_reason reason)\n+{\n+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n+\n+\ts->mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\ts->reason = reason;\n+}\n+\n+static int check_for_incompatible(struct repository *r)\n+{\n+\tif (!r->worktree) {\n+\t\t/*\n+\t\t * Bare repositories don't have a working directory and\n+\t\t * therefore have nothing to watch.\n+\t\t */\n+\t\tset_incompatible(r, FSMONITOR_REASON_BARE);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static void lookup_fsmonitor_settings(struct repository *r)\n {\n \tstruct fsmonitor_settings *s;\n@@ -23,6 +47,7 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \n \tCALLOC_ARRAY(s, 1);\n \ts->mode = FSMONITOR_MODE_DISABLED;\n+\ts->reason = FSMONITOR_REASON_OK;\n \n \tr->settings.fsmonitor = s;\n \n@@ -86,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r)\n \n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n@@ -97,6 +125,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path)\n \n \tlookup_fsmonitor_settings(r);\n \n+\tif (check_for_incompatible(r))\n+\t\treturn;\n+\n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n@@ -110,5 +141,34 @@ void fsm_settings__set_disabled(struct repository *r)\n \tlookup_fsmonitor_settings(r);\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n+\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\n+\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\n+\n+int fsm_settings__error_if_incompatible(struct repository *r)\n+{\n+\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n+\n+\tswitch (reason) {\n+\tcase FSMONITOR_REASON_OK:\n+\t\treturn 0;\n+\n+\tcase FSMONITOR_REASON_BARE:\n+\t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n+\t\t xgetcwd());\n+\t\treturn 1;\n+\t}\n+\n+\tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n+\t reason);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a4c5d7b4889..8654edf33d8 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -4,11 +4,20 @@\n struct repository;\n \n enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n \tFSMONITOR_MODE_DISABLED = 0,\n \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n };\n \n+/*\n+ * Incompatibility reasons.\n+ */\n+enum fsmonitor_reason {\n+\tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n+\tFSMONITOR_REASON_BARE,\n+};\n+\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n@@ -16,6 +25,9 @@ void fsm_settings__set_disabled(struct repository *r);\n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n+int fsm_settings__error_if_incompatible(struct repository *r);\n+\n struct fsmonitor_settings;\n \n #endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex a6308acf006..9a8e21c5608 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '\n \ttest $ret -ne 1\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n+\tgit init --bare bare-clone &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n+\tgit init --bare bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"454007","messageId":"e1e55550c1050e0c202c90092a26f0c8ff31481d.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 05/28] fsmonitor-settings: stub in Win32-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:55Z","receivedAt":"2022-04-20T20:43:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend generic incompatibility checkout with platform-specific\nmechanism. Stub in Win32 version.\n\nIn the existing fsmonitor-settings code we have a way to mark\ntypes of repos as incompatible with fsmonitor (whether via the\nhook and IPC APIs). For example, we do this for bare repos,\nsince there are no files to watch.\n\nExtend this exclusion mechanism for platform-specific reasons.\nThis commit just creates the framework and adds a stub for Win32.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 +++++++++++++\n compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++\n config.mak.uname | 4 ++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n fsmonitor-settings.c | 12 ++++++++++++\n fsmonitor-settings.h | 13 +++++++++++++\n 6 files changed, 54 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n\ndiff --git a/Makefile b/Makefile\nindex daa21bed6c3..93604fe8ef7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -475,6 +475,11 @@ all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n+# If your platform has OS-specific ways to tell if a repo is incompatible with\n+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n+# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n+# that implements the `fsm_os_settings__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n+ifdef FSMONITOR_OS_SETTINGS\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE\n ifdef FSMONITOR_DAEMON_BACKEND\n \t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n endif\n+ifdef FSMONITOR_OS_SETTINGS\n+\t@echo FSMONITOR_OS_SETTINGS=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_OS_SETTINGS)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 501970902da..cf224768ad6 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex ee0d7257b77..16705da2000 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 86c09bd35fe..8ff55f8c3fd 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r)\n \t\treturn 1;\n \t}\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+\t{\n+\t\tenum fsmonitor_reason reason;\n+\n+\t\treason = fsm_os__incompatible(r);\n+\t\tif (reason != FSMONITOR_REASON_OK) {\n+\t\t\tset_incompatible(r, reason);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+#endif\n+\n \treturn 0;\n }\n \ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 8654edf33d8..4b35f051fb1 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -30,4 +30,17 @@ int fsm_settings__error_if_incompatible(struct repository *r);\n \n struct fsmonitor_settings;\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+/*\n+ * Ask platform-specific code whether the repository is incompatible\n+ * with fsmonitor (both hook and ipc modes). For example, if the working\n+ * directory is on a remote volume and mounted via a technology that does\n+ * not support notification events, then we should not pretend to watch it.\n+ *\n+ * fsm_os__* routines should considered private to fsm_settings__\n+ * routines.\n+ */\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);\n+#endif /* HAVE_FSMONITOR_OS_SETTINGS */\n+\n #endif /* FSMONITOR_SETTINGS_H */\n-- \ngitgitgadget\n\n"},{"id":"454008","messageId":"94ae2e424f183344a19c76cc5bb481b64be3f31f.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 07/28] fsmonitor-settings: stub in macOS-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:57Z","receivedAt":"2022-04-20T20:43:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++\n config.mak.uname | 1 +\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 13 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cf224768ad6..cf911d141f2 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)\n \tifndef NO_PTHREADS\n \tifndef NO_UNIX_SOCKETS\n \tFSMONITOR_DAEMON_BACKEND = darwin\n+\tFSMONITOR_OS_SETTINGS = darwin\n \tendif\n \tendif\n \ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 16705da2000..b8f9f7a0388 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"454009","messageId":"2d68fc9a46ad66177d686e889f191dd252f1c964.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 06/28] fsmonitor-settings: VFS for Git virtual repos are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:56Z","receivedAt":"2022-04-20T20:43:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nVFS for Git virtual repositories are incompatible with FSMonitor.\n\nVFS for Git is a downstream fork of Git. It contains its own custom\nfile system watcher that is aware of the virtualization. If a working\ndirectory is being managed by VFS for Git, we should not try to watch\nit because we may get incomplete results.\n\nWe do not know anything about how VFS for Git works, but we do\nknow that VFS for Git working directories contain a well-defined\nconfig setting. If it is set, mark the working directory as\nincompatible.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 5 +++++\n fsmonitor-settings.h | 1 +\n t/t7519-status-fsmonitor.sh | 9 +++++++++\n 4 files changed, 41 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 7fce32a3c5b..ee78bba38e3 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -3,7 +3,33 @@\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n \n+/*\n+ * VFS for Git is incompatible with FSMonitor.\n+ *\n+ * Granted, core Git does not know anything about VFS for Git and we\n+ * shouldn't make assumptions about a downstream feature, but users\n+ * can install both versions. And this can lead to incorrect results\n+ * from core Git commands. So, without bringing in any of the VFS for\n+ * Git code, do a simple config test for a published config setting.\n+ * (We do not look at the various *_TEST_* environment variables.)\n+ */\n+static enum fsmonitor_reason check_vfs4git(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n+\t\treturn FSMONITOR_REASON_VFS4GIT;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_vfs4git(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 8ff55f8c3fd..1efb6e17a20 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -179,6 +179,11 @@ int fsm_settings__error_if_incompatible(struct repository *r)\n \t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n \t\t xgetcwd());\n \t\treturn 1;\n+\n+\tcase FSMONITOR_REASON_VFS4GIT:\n+\t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 4b35f051fb1..6361fcbf6b0 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -16,6 +16,7 @@ enum fsmonitor_mode {\n enum fsmonitor_reason {\n \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 9a8e21c5608..156895f9437 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n \tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n '\n \n+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone actual\" &&\n+\tgit init fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"virtual repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"454010","messageId":"b2ca6c1b201a5d0e54497b9ce091bae6b205f2c7.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 08/28] fsmonitor-settings: remote repos on macOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:58Z","receivedAt":"2022-04-20T20:43:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on macOS and mark them as\nincompatible with FSMonitor.\n\nWith this, `git fsmonitor--daemon run` will error out with a message\nlike it does for bare repos.\n\nClient commands, like `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 10 ++++\n fsmonitor-settings.h | 2 +\n 3 files changed, 78 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex 7fce32a3c5b..fdd762bf79d 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -2,8 +2,74 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n+#include \n+#include \n+\n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type (NFS, SAMBA, etc.) dictates whether notification events\n+ * are available at all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * For the builtin FSMonitor, we create the Unix domain socket for the\n+ * IPC in the .git directory. If the working directory is remote,\n+ * then the socket will be created on the remote file system. This\n+ * can fail if the remote file system does not support UDS file types\n+ * (e.g. smbfs to a Windows server) or if the remote kernel does not\n+ * allow a non-local process to bind() the socket. (These problems\n+ * could be fixed by moving the UDS out of the .git directory and to a\n+ * well-known local directory on the client machine, but care should\n+ * be taken to ensure that $HOME is actually local and not a managed\n+ * file share.)\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\tstruct statfs fs;\n+\n+\tif (statfs(r->worktree, &fs) == -1) {\n+\t\tint saved_errno = errno;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n+\t\t\t\t r->worktree, strerror(saved_errno));\n+\t\terrno = saved_errno;\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n+\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n+\n+\tif (!(fs.f_flags & MNT_LOCAL))\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 1efb6e17a20..0a1811ff004 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -180,6 +180,16 @@ int fsm_settings__error_if_incompatible(struct repository *r)\n \t\t xgetcwd());\n \t\treturn 1;\n \n+\tcase FSMONITOR_REASON_ERROR:\n+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n+\n+\tcase FSMONITOR_REASON_REMOTE:\n+\t\terror(_(\"remote repository '%s' is incompatible with fsmonitor\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n+\n \tcase FSMONITOR_REASON_VFS4GIT:\n \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t r->worktree);\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 6361fcbf6b0..34391b583b3 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -16,6 +16,8 @@ enum fsmonitor_mode {\n enum fsmonitor_reason {\n \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n+\tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"454011","messageId":"a3cc4b3b16dc93c5c4378733a1f45f6a1a78be4a.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 09/28] fsmonitor-settings: remote repos on Windows are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:42:59Z","receivedAt":"2022-04-20T20:43:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on Windows and mark them as\nincompatible with FSMonitor.\n\nWith this `git fsmonitor--daemon run` will error out with a message like it\ndoes for bare repos.\n\nClient commands, such as `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++\n 1 file changed, 102 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex ee78bba38e3..907655720bb 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * VFS for Git is incompatible with FSMonitor.\n@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)\n \treturn FSMONITOR_REASON_OK;\n }\n \n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type dictates whether notification events are available at\n+ * all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ *\n+ * Notes for testing:\n+ *\n+ * (a) Windows allows a network share to be mapped to a drive letter.\n+ * (This is the normal method to access it.)\n+ *\n+ * $ NET USE Z: \\\\server\\share\n+ * $ git -C Z:/repo status\n+ *\n+ * (b) Windows allows a network share to be referenced WITHOUT mapping\n+ * it to drive letter.\n+ *\n+ * $ NET USE \\\\server\\share\\dir\n+ * $ git -C //server/share/repo status\n+ *\n+ * (c) Windows allows \"SUBST\" to create a fake drive mapping to an\n+ * arbitrary path (which may be remote)\n+ *\n+ * $ SUBST Q: Z:\\repo\n+ * $ git -C Q:/ status\n+ *\n+ * (d) Windows allows a directory symlink to be created on a local\n+ * file system that points to a remote repo.\n+ *\n+ * $ mklink /d ./link //server/share/repo\n+ * $ git -C ./link status\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\twchar_t wpath[MAX_PATH];\n+\twchar_t wfullpath[MAX_PATH];\n+\tsize_t wlen;\n+\tUINT driveType;\n+\n+\t/*\n+\t * Do everything in wide chars because the drive letter might be\n+\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n+\t */\n+\tif (xutftowcs_path(wpath, r->worktree) < 0)\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\t/*\n+\t * GetDriveTypeW() requires a final slash. We assume that the\n+\t * worktree pathname points to an actual directory.\n+\t */\n+\twlen = wcslen(wpath);\n+\tif (wpath[wlen - 1] != L'\\\\' && wpath[wlen - 1] != L'/') {\n+\t\twpath[wlen++] = L'\\\\';\n+\t\twpath[wlen] = 0;\n+\t}\n+\n+\t/*\n+\t * Normalize the path. If nothing else, this converts forward\n+\t * slashes to backslashes. This is essential to get GetDriveTypeW()\n+\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n+\t */\n+\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\tdriveType = GetDriveTypeW(wfullpath);\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"DriveType '%s' L'%ls' (%u)\",\n+\t\t\t r->worktree, wfullpath, driveType);\n+\n+\tif (driveType == DRIVE_REMOTE) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"check_remote('%s') true\",\n+\t\t\t\t r->worktree);\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n-- \ngitgitgadget\n\n"},{"id":"454012","messageId":"8f1f4840751892708de7f2e4ad14c13649943e25.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 10/28] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:00Z","receivedAt":"2022-04-20T20:44:01Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nOn MacOS mark repos on NTFS or FAT32 volumes as incompatible.\n\nThe builtin FSMonitor used Unix domain sockets on MacOS for IPC\nwith clients. These sockets are kept in the .git directory.\nUnix sockets are not supported by NTFS and FAT32, so the daemon\ncannot start up.\n\nTest for this during our compatibility checking so that client\ncommands do not keep trying to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---\n fsmonitor-settings.c | 5 +++++\n fsmonitor-settings.h | 1 +\n 3 files changed, 23 insertions(+), 3 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex fdd762bf79d..efc732c0f31 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -7,7 +7,7 @@\n #include \n \n /*\n- * Remote working directories are problematic for FSMonitor.\n+ * [1] Remote working directories are problematic for FSMonitor.\n *\n * The underlying file system on the server machine and/or the remote\n * mount type (NFS, SAMBA, etc.) dictates whether notification events\n@@ -40,8 +40,16 @@\n *\n * So (for now at least), mark remote working directories as\n * incompatible.\n+ *\n+ *\n+ * [2] FAT32 and NTFS working directories are problematic too.\n+ *\n+ * The builtin FSMonitor uses a Unix domain socket in the .git\n+ * directory for IPC. These Windows drive formats do not support\n+ * Unix domain sockets, so mark them as incompatible for the daemon.\n+ *\n */\n-static enum fsmonitor_reason check_remote(struct repository *r)\n+static enum fsmonitor_reason check_volume(struct repository *r)\n {\n \tstruct statfs fs;\n \n@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)\n \tif (!(fs.f_flags & MNT_LOCAL))\n \t\treturn FSMONITOR_REASON_REMOTE;\n \n+\tif (!strcmp(fs.f_fstypename, \"msdos\")) /* aka FAT32 */\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n+\tif (!strcmp(fs.f_fstypename, \"ntfs\"))\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \n@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n \n-\treason = check_remote(r);\n+\treason = check_volume(r);\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 0a1811ff004..60d5eaee497 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -194,6 +194,11 @@ int fsm_settings__error_if_incompatible(struct repository *r)\n \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t r->worktree);\n \t\treturn 1;\n+\n+\tcase FSMONITOR_REASON_NOSOCKETS:\n+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n+\t\t r->worktree);\n+\t\treturn 1;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 34391b583b3..23d5676c8c8 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -19,6 +19,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n \tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n+\tFSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n-- \ngitgitgadget\n\n"},{"id":"454013","messageId":"8d48d9c562369a29c1488dfcf7bd51a58158236a.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 11/28] unpack-trees: initialize fsmonitor_has_run_once in o->result","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:01Z","receivedAt":"2022-04-20T20:44:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInitialize `o->result.fsmonitor_has_run_once` based upon value\nin `o->src_index->fsmonitor_has_run_once` to prevent a second\nfsmonitor query during the tree traversal and possibly getting\na skewed view of the working directory.\n\nThe checkout code has already talked to the fsmonitor and the\ntraversal is updating the index as it traverses, so there is\nno need to query the fsmonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n unpack-trees.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..888cff81f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \to->result.fsmonitor_last_update =\n \t\txstrdup_or_null(o->src_index->fsmonitor_last_update);\n+\to->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;\n \n \t/*\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n-- \ngitgitgadget\n\n"},{"id":"454014","messageId":"088c7b3334c79679d605dd16dc90211c08320428.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 12/28] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:02Z","receivedAt":"2022-04-20T20:44:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIgnore FSEvents resulting from `xattr` changes. Git does not care about\nxattr's or changes to xattr's, so don't waste time collecting these\nevents in the daemon nor transmitting them to clients.\n\nVarious security tools add xattrs to files and/or directories, such as\nto mark them as having been downloaded. We should ignore these events\nsince it doesn't affect the content of the file/directory or the normal\nmeta-data that Git cares about.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 0741fe834c3..14105f45c18 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tif (flag & kFSEventStreamEventFlagItemCloned)\n \t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n \n-\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n \t\t\t path, flag, msg.buf);\n \n \tstrbuf_release(&msg);\n@@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)\n \t\tef & kFSEventStreamEventFlagUserDropped);\n }\n \n+/*\n+ * If an `xattr` change is the only reason we received this event,\n+ * then silently ignore it. Git doesn't care about xattr's. We\n+ * have to be careful here because the kernel can combine multiple\n+ * events for a single path. And because events always have certain\n+ * bits set, such as `ItemIsFile` or `ItemIsDir`.\n+ *\n+ * Return 1 if we should ignore it.\n+ */\n+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n+{\n+\tstatic const FSEventStreamEventFlags mask =\n+\t\tkFSEventStreamEventFlagItemChangeOwner |\n+\t\tkFSEventStreamEventFlagItemCreated |\n+\t\tkFSEventStreamEventFlagItemFinderInfoMod |\n+\t\tkFSEventStreamEventFlagItemInodeMetaMod |\n+\t\tkFSEventStreamEventFlagItemModified |\n+\t\tkFSEventStreamEventFlagItemRemoved |\n+\t\tkFSEventStreamEventFlagItemRenamed |\n+\t\tkFSEventStreamEventFlagItemXattrMod |\n+\t\tkFSEventStreamEventFlagItemCloned;\n+\n+\treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n+}\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_ignore_xattr(event_flags[k])) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n+\t\t\t\t\t path_k, event_flags[k]);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n \n \t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-- \ngitgitgadget\n\n"},{"id":"454015","messageId":"00fab626663292f8c82233a3a72b4f9e885de17b.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 13/28] fsmonitor--daemon: cd out of worktree root","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:03Z","receivedAt":"2022-04-20T20:44:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the fsmonitor--daemon to CD outside of the worktree\nbefore starting up.\n\nThe common Git startup mechanism causes the CWD of the daemon process\nto be in the root of the worktree. On Windows, this causes the daemon\nprocess to hold a locked handle on the CWD and prevents other\nprocesses from moving or deleting the worktree while the daemon is\nrunning.\n\nCD to HOME before entering main event loops.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++--\n compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------\n fsmonitor--daemon.h | 1 +\n 3 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 50ae3cca575..1b7c757f583 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1181,11 +1181,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * before we need it.\n \t */\n \tif (ipc_server_run_async(&state->ipc_server_data,\n-\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t state->path_ipc.buf, &ipc_opts,\n \t\t\t\t handle_client, state))\n \t\treturn error_errno(\n \t\t\t_(\"could not start IPC thread pool on '%s'\"),\n-\t\t\tfsmonitor_ipc__get_path());\n+\t\t\tstate->path_ipc.buf);\n \n \t/*\n \t * Start the fsmonitor listener thread to collect filesystem\n@@ -1220,6 +1220,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n static int fsmonitor_run_daemon(void)\n {\n \tstruct fsmonitor_daemon_state state;\n+\tconst char *home;\n \tint err;\n \n \tmemset(&state, 0, sizeof(state));\n@@ -1289,6 +1290,15 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_addch(&state.path_cookie_prefix, '/');\n \n+\t/*\n+\t * We create a named-pipe or unix domain socket inside of the\n+\t * \".git\" directory. (Well, on Windows, we base our named\n+\t * pipe in the NPFS on the absolute path of the git\n+\t * directory.)\n+\t */\n+\tstrbuf_init(&state.path_ipc, 0);\n+\tstrbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1298,6 +1308,23 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\t/*\n+\t * CD out of the worktree root directory.\n+\t *\n+\t * The common Git startup mechanism causes our CWD to be the\n+\t * root of the worktree. On Windows, this causes our process\n+\t * to hold a locked handle on the CWD. This prevents the\n+\t * worktree from being moved or deleted while the daemon is\n+\t * running.\n+\t *\n+\t * We assume that our FS and IPC listener threads have either\n+\t * opened all of the handles that they need or will do\n+\t * everything using absolute paths.\n+\t */\n+\thome = getenv(\"HOME\");\n+\tif (home && *home && chdir(home))\n+\t\tdie_errno(_(\"could not cd home '%s'\"), home);\n+\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n@@ -1310,6 +1337,7 @@ done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n \tstrbuf_release(&state.path_cookie_prefix);\n+\tstrbuf_release(&state.path_ipc);\n \n \treturn err;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 3f1b68267bd..c43d92b9620 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch)\n \t}\n \n \t/*\n-\t * NEEDSWORK: If an external is deleted, the above\n-\t * returns an error. I'm not sure that there's anything that\n-\t * we can do here other than failing -- the /.git\n-\t * link file would be broken anyway. We might try to check\n-\t * for that and return a better error message, but I'm not\n-\t * sure it is worth it.\n+\t * GetOverlappedResult() fails if the watched directory is\n+\t * deleted while we were waiting for an overlapped IO to\n+\t * complete. The documentation did not list specific errors,\n+\t * but I observed ERROR_ACCESS_DENIED (0x05) errors during\n+\t * testing.\n+\t *\n+\t * Note that we only get notificaiton events for events\n+\t * *within* the directory, not *on* the directory itself.\n+\t * (These might be properies of the parent directory, for\n+\t * example).\n+\t *\n+\t * NEEDSWORK: We might try to check for the deleted directory\n+\t * case and return a better error message, but I'm not sure it\n+\t * is worth it.\n+\t *\n+\t * Shutdown if we get any error.\n \t */\n \n \terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex bd09fffc176..223c2131b58 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n+\tstruct strbuf path_ipc;\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"454016","messageId":"6552f51802ba383f4c82866ce20d989749d47be3.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 14/28] fsmonitor--daemon: prepare for adding health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:04Z","receivedAt":"2022-04-20T20:44:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor daemon thread startup to make it easier to start\na third thread class to monitor the health of the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 1b7c757f583..14cd2d5eb52 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1174,6 +1174,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint listener_started = 0;\n+\tint err = 0;\n \n \t/*\n \t * Start the IPC thread pool before the we've started the file\n@@ -1194,15 +1196,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \tif (pthread_create(&state->listener_thread, NULL,\n \t\t\t fsm_listen__thread_proc, state) < 0) {\n \t\tipc_server_stop_async(state->ipc_server_data);\n-\t\tipc_server_await(state->ipc_server_data);\n-\n-\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t\terr = error(_(\"could not start fsmonitor listener thread\"));\n+\t\tgoto cleanup;\n \t}\n+\tlistener_started = 1;\n \n \t/*\n \t * The daemon is now fully functional in background threads.\n+\t * Our primary thread should now just wait while the threads\n+\t * do all the work.\n+\t */\n+cleanup:\n+\t/*\n \t * Wait for the IPC thread pool to shutdown (whether by client\n-\t * request or from filesystem activity).\n+\t * request, from filesystem activity, or an error).\n \t */\n \tipc_server_await(state->ipc_server_data);\n \n@@ -1211,10 +1218,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * event from the IPC thread pool, but it doesn't hurt to tell\n \t * it again. And wait for it to shutdown.\n \t */\n-\tfsm_listen__stop_async(state);\n-\tpthread_join(state->listener_thread, NULL);\n+\tif (listener_started) {\n+\t\tfsm_listen__stop_async(state);\n+\t\tpthread_join(state->listener_thread, NULL);\n+\t}\n \n-\treturn state->error_code;\n+\tif (err)\n+\t\treturn err;\n+\tif (state->error_code)\n+\t\treturn state->error_code;\n+\treturn 0;\n }\n \n static int fsmonitor_run_daemon(void)\n-- \ngitgitgadget\n\n"},{"id":"454017","messageId":"f2bf07cd7395abf4e3c79782b174c13201f2c5cc.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 15/28] fsmonitor--daemon: rename listener thread related variables","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:05Z","receivedAt":"2022-04-20T20:44:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRename platform-specific listener thread related variables\nand data types as we prepare to add another backend thread\ntype.\n\n[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`\n[] `state->backend_data` becomes `state->listen_data`\n[] `state->error_code` becomes `state->listen_error_code`\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 +++---\n compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------\n compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++-------------\n compat/fsmonitor/fsm-listen.h | 2 +-\n fsmonitor--daemon.h | 6 +++---\n 5 files changed, 36 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 14cd2d5eb52..d5def8faf90 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1225,8 +1225,8 @@ cleanup:\n \n \tif (err)\n \t\treturn err;\n-\tif (state->error_code)\n-\t\treturn state->error_code;\n+\tif (state->listen_error_code)\n+\t\treturn state->listen_error_code;\n \treturn 0;\n }\n \n@@ -1241,7 +1241,7 @@ static int fsmonitor_run_daemon(void)\n \thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n-\tstate.error_code = 0;\n+\tstate.listen_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 14105f45c18..07113205a61 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -27,7 +27,7 @@\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tCFStringRef cfsr_worktree_path;\n \tCFStringRef cfsr_gitdir_path;\n@@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t const FSEventStreamEventId event_ids[])\n {\n \tstruct fsmonitor_daemon_state *state = ctx;\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tchar **paths = (char **)event_paths;\n \tstruct fsmonitor_batch *batch = NULL;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tNULL,\n \t\tNULL\n \t};\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \tconst void *dir_array[2];\n \n \tCALLOC_ARRAY(data, 1);\n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \n \tdata->cfsr_worktree_path = CFStringCreateWithCString(\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n@@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n \terror(_(\"Unable to create FSEventStream.\"));\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tif (data->stream) {\n \t\tif (data->stream_started)\n@@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n \t\tFSEventStreamRelease(data->stream);\n \t}\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \tdata->shutdown_style = SHUTDOWN_EVENT;\n \n \tCFRunLoopStop(data->rl);\n@@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tdata->rl = CFRunLoopGetCurrent();\n \n@@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \n \tswitch (data->shutdown_style) {\n \tcase FORCE_ERROR_STOP:\n-\t\tstate->error_code = -1;\n+\t\tstate->listen_error_code = -1;\n \t\t/* fall thru */\n \tcase FORCE_SHUTDOWN:\n \t\tipc_server_stop_async(state->ipc_server_data);\n@@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \treturn;\n \n force_error_stop_without_loop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \tipc_server_stop_async(state->ipc_server_data);\n \treturn;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex c43d92b9620..be2d93f47b2 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -54,7 +54,7 @@ struct one_watch\n \twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tstruct one_watch *watch_worktree;\n \tstruct one_watch *watch_gitdir;\n@@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname(\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+\tSetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);\n }\n \n static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n@@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch)\n \tfree(watch);\n }\n \n-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+static int start_rdcw_watch(struct fsm_listen_data *data,\n \t\t\t struct one_watch *watch)\n {\n \tDWORD dwNotifyFilter =\n@@ -516,7 +516,7 @@ static int process_1_worktree_event(\n */\n static int process_worktree_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_worktree;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -646,7 +646,7 @@ force_shutdown:\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_gitdir;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -704,11 +704,11 @@ skip_this_path:\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tDWORD dwWait;\n \tint result;\n \n-\tstate->error_code = 0;\n+\tstate->listen_error_code = 0;\n \n \tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n \t\tgoto force_error_stop;\n@@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \t}\n \n force_error_stop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \n force_shutdown:\n \t/*\n@@ -790,7 +790,7 @@ clean_shutdown:\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \treturn 0;\n \n failed:\n@@ -836,16 +836,16 @@ failed:\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tCloseHandle(data->hEventShutdown);\n \tdestroy_watch(data->watch_worktree);\n \tdestroy_watch(data->watch_gitdir);\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nindex f0539349baf..41650bf8972 100644\n--- a/compat/fsmonitor/fsm-listen.h\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n * do so if the listener thread receives a normal shutdown signal from\n * the IPC layer.)\n *\n- * It should set `state->error_code` to -1 if the daemon should exit\n+ * It should set `state->listen_error_code` to -1 if the daemon should exit\n * with an error.\n */\n void fsm_listen__loop(struct fsmonitor_daemon_state *state);\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 223c2131b58..2c6fa1a5d91 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n */\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {\n \tint cookie_seq;\n \tstruct hashmap cookies;\n \n-\tint error_code;\n-\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\tint listen_error_code;\n+\tstruct fsm_listen_data *listen_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"454018","messageId":"2a44f2eded1eba45dc240ee743ff4e178aa72f4d.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 16/28] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:06Z","receivedAt":"2022-04-20T20:44:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate another thread to watch over the daemon process and\nautomatically shut it down if necessary.\n\nThis commit creates the basic framework for a \"health\" thread\nto monitor the daemon and/or the file system. Later commits\nwill add platform-specific code to do the actual work.\n\nThe \"health\" thread is intended to monitor conditions that\nwould be difficult to track inside the IPC thread pool and/or\nthe file system listener threads. For example, when there are\nfile system events outside of the watched worktree root or if\nwe want to have an idle-timeout auto-shutdown feature.\n\nThis commit creates the health thread itself, defines the thread-proc\nand sets up the thread's event loop. It integrates this new thread\ninto the existing IPC and Listener thread models.\n\nThis commit defines the API to the platform-specific code where all of\nthe monitoring will actually happen.\n\nThe platform-specific code for MacOS is just stubs. Meaning that the\nhealth thread will immediately exit on MacOS, but that is OK and\nexpected. Future work can define MacOS-specific monitoring.\n\nThe platform-specific code for Windows sets up enough of the\nWaitForMultipleObjects() machinery to watch for system and/or custom\nevents. Currently, the set of wait handles only includes our custom\nshutdown event (sent from our other theads). Later commits in this\nseries will extend the set of wait handles to monitor other\nconditions.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 6 ++-\n builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n contrib/buildsystems/CMakeLists.txt | 2 +\n fsmonitor--daemon.h | 4 ++\n 7 files changed, 192 insertions(+), 2 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n\ndiff --git a/Makefile b/Makefile\nindex 93604fe8ef7..5f1623baadd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -472,8 +472,9 @@ all::\n #\n # If your platform supports a built-in fsmonitor backend, set\n # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n-# `compat/fsmonitor/fsm-listen-.c` that implements the\n-# `fsm_listen__*()` routines.\n+# `compat/fsmonitor/fsm-listen-.c` and\n+# `compat/fsmonitor/fsm-health-.c` files\n+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n # If your platform has OS-specific ways to tell if a repo is incompatible with\n # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n@@ -1982,6 +1983,7 @@ endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n ifdef FSMONITOR_OS_SETTINGS\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex d5def8faf90..e22f53026e9 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n@@ -1136,6 +1137,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+static void *fsm_health__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-health\");\n+\n+\tfsm_health__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -1174,6 +1187,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint health_started = 0;\n \tint listener_started = 0;\n \tint err = 0;\n \n@@ -1201,6 +1215,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t}\n \tlistener_started = 1;\n \n+\t/*\n+\t * Start the health thread to watch over our process.\n+\t */\n+\tif (pthread_create(&state->health_thread, NULL,\n+\t\t\t fsm_health__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\terr = error(_(\"could not start fsmonitor health thread\"));\n+\t\tgoto cleanup;\n+\t}\n+\thealth_started = 1;\n+\n \t/*\n \t * The daemon is now fully functional in background threads.\n \t * Our primary thread should now just wait while the threads\n@@ -1223,10 +1248,17 @@ cleanup:\n \t\tpthread_join(state->listener_thread, NULL);\n \t}\n \n+\tif (health_started) {\n+\t\tfsm_health__stop_async(state);\n+\t\tpthread_join(state->health_thread, NULL);\n+\t}\n+\n \tif (err)\n \t\treturn err;\n \tif (state->listen_error_code)\n \t\treturn state->listen_error_code;\n+\tif (state->health_error_code)\n+\t\treturn state->health_error_code;\n \treturn 0;\n }\n \n@@ -1242,6 +1274,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.listen_error_code = 0;\n+\tstate.health_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n@@ -1321,6 +1354,11 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\tif (fsm_health__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize health thread\"));\n+\t\tgoto done;\n+\t}\n+\n \t/*\n \t * CD out of the worktree root directory.\n \t *\n@@ -1344,6 +1382,7 @@ done:\n \tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n+\tfsm_health__dtor(&state);\n \n \tipc_server_free(state.ipc_server_data);\n \ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nnew file mode 100644\nindex 00000000000..b9f709e8548\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -0,0 +1,24 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nnew file mode 100644\nindex 00000000000..94b1d020f25\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsm_health_data\n+{\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hHandles[1]; /* the array does not own these handles */\n+#define HEALTH_SHUTDOWN 0\n+\tint nr_handles; /* number of active event handles */\n+};\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_handles++;\n+\n+\tstate->health_data = data;\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tif (!state || !state->health_data)\n+\t\treturn;\n+\n+\tdata = state->health_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\n+\tFREE_AND_NULL(state->health_data);\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\n+\tfor (;;) {\n+\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n+\t\t\t\t\t\t data->hHandles,\n+\t\t\t\t\t\t FALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->health_error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+clean_shutdown:\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n+}\ndiff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\nnew file mode 100644\nindex 00000000000..45547ba9380\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health.h\n@@ -0,0 +1,47 @@\n+#ifndef FSM_HEALTH_H\n+#define FSM_HEALTH_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor health thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the health thread.\n+ * This will be called from the main thread AFTER joining the thread.\n+ */\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to monitor the\n+ * health of the daemon process. This will run in the health thread.\n+ *\n+ * The health thread should call `ipc_server_stop_async()` if it needs\n+ * to cause a shutdown. (It should NOT do so if it receives a shutdown\n+ * shutdown signal.)\n+ *\n+ * It should set `state->health_error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_health__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the health thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_HEALTH_H */\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex b8f9f7a0388..16ace43d1c7 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 2c6fa1a5d91..2102a5c9ff5 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n+struct fsm_health_data; /* opaque platform-specific data for health thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n+\tpthread_t health_thread;\n \tpthread_mutex_t main_lock;\n \n \tstruct strbuf path_worktree_watch;\n@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n \tstruct hashmap cookies;\n \n \tint listen_error_code;\n+\tint health_error_code;\n \tstruct fsm_listen_data *listen_data;\n+\tstruct fsm_health_data *health_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"454019","messageId":"854fb5e3658156f93f76915aaf3d4a194145b3fb.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 17/28] fsm-health-win32: add polling framework to monitor daemon health","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:07Z","receivedAt":"2022-04-20T20:44:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend the Windows version of the \"health\" thread to periodically\ninspect the system and shutdown if warranted.\n\nThis commit updates the thread's wait loop to use a timeout and\ndefines a (currently empty) table of functions to poll the system.\n\nA later commit will add functions to the table to actually\ninspect the system.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-\n 1 file changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 94b1d020f25..24fc612bf02 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -4,6 +4,24 @@\n #include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n+/*\n+ * Every minute wake up and test our health.\n+ */\n+#define WAIT_FREQ_MS (60 * 1000)\n+\n+/*\n+ * State machine states for each of the interval functions\n+ * used for polling our health.\n+ */\n+enum interval_fn_ctx {\n+\tCTX_INIT = 0,\n+\tCTX_TERM,\n+\tCTX_TIMER\n+};\n+\n+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx);\n+\n struct fsm_health_data\n {\n \tHANDLE hEventShutdown;\n@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n \tFREE_AND_NULL(state->health_data);\n }\n \n+/*\n+ * A table of the polling functions.\n+ */\n+static interval_fn *table[] = {\n+\tNULL, /* must be last */\n+};\n+\n+/*\n+ * Call all of the polling functions in the table.\n+ * Shortcut and return first error.\n+ *\n+ * Return 0 if all succeeded.\n+ */\n+static int call_all(struct fsmonitor_daemon_state *state,\n+\t\t enum interval_fn_ctx ctx)\n+{\n+\tint k;\n+\n+\tfor (k = 0; table[k]; k++) {\n+\t\tint r = table[k](state, ctx);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n+\tint r;\n+\n+\tr = call_all(state, CTX_INIT);\n+\tif (r < 0)\n+\t\tgoto force_error_stop;\n+\tif (r > 0)\n+\t\tgoto force_shutdown;\n \n \tfor (;;) {\n \t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n \t\t\t\t\t\t data->hHandles,\n-\t\t\t\t\t\t FALSE, INFINITE);\n+\t\t\t\t\t\t FALSE, WAIT_FREQ_MS);\n \n \t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n \t\t\tgoto clean_shutdown;\n \n+\t\tif (dwWait == WAIT_TIMEOUT) {\n+\t\t\tr = call_all(state, CTX_TIMER);\n+\t\t\tif (r < 0)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (r > 0)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\terror(_(\"health thread wait failed [GLE %ld]\"),\n \t\t GetLastError());\n \t\tgoto force_error_stop;\n@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)\n \n force_error_stop:\n \tstate->health_error_code = -1;\n+force_shutdown:\n \tipc_server_stop_async(state->ipc_server_data);\n clean_shutdown:\n+\tcall_all(state, CTX_TERM);\n \treturn;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"454020","messageId":"f1365cdd40cd625433ba5494b8f590e31f34a944.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 19/28] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:09Z","receivedAt":"2022-04-20T20:44:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the listener thread to shutdown the daemon if the spelling of the\nworktree root directory changes.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 07113205a61..83d38e8ac6c 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tstrbuf_release(&msg);\n }\n \n+static int ef_is_root_changed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagRootChanged);\n+}\n+\n static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n {\n \treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n@@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_is_root_changed(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * The spelling of the pathname of the root directory\n+\t\t\t * has changed. This includes the name of the root\n+\t\t\t * directory itself or of any parent directory in the\n+\t\t\t * path.\n+\t\t\t *\n+\t\t\t * (There may be other conditions that throw this,\n+\t\t\t * but I couldn't find any information on it.)\n+\t\t\t *\n+\t\t\t * Force a shutdown now and avoid things getting\n+\t\t\t * out of sync. The Unix domain socket is inside\n+\t\t\t * the .git directory and a spelling change will make\n+\t\t\t * it hard for clients to rendezvous with us.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"event: root changed\");\n+\t\t\tgoto force_shutdown;\n+\t\t}\n+\n \t\tif (ef_ignore_xattr(event_flags[k])) {\n \t\t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n-- \ngitgitgadget\n\n"},{"id":"454021","messageId":"5db241f7d2f654d46d4858ce9e9413032705abf3.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 23/28] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:13Z","receivedAt":"2022-04-20T20:44:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNever set CE_FSMONITOR_VALID on the cache-entry of submodule\ndirectories.\n\nDuring a client command like 'git status', we may need to recurse\ninto each submodule to compute a status summary for the submodule.\nSince the purpose of the ce_flag is to let Git avoid scanning a\ncache-entry, setting the flag causes the recursive call to be\navoided and we report incorrect (no status) for the submodule.\n\nWe created an OS watch on the root directory of our working\ndirectory and we receive events for everything in the cone\nunder it. When submodules are present inside our working\ndirectory, we receive events for both our repo (the super) and\nany subs within it. Since our index doesn't have any information\nfor items within the submodules, we can't use those events.\n\nWe could try to truncate the paths of those events back to the\nsubmodule boundary and mark the GITLINK as dirty, but that\nfeels expensive since we would have to prefix compare every FS\nevent that we receive against a list of submodule roots. And\nit still wouldn't be sufficient to correctly report status on\nthe submodule, since we don't have any space in the cache-entry\nto cache the submodule's status (the 'SCMU' bits in porcelain\nV2 speak). That is, the CE_FSMONITOR_VALID bit just says that\nwe don't need to scan/inspect it because we already know the\nanswer -- it doesn't say that the item is clean -- and we\ndon't have space in the cache-entry to store those answers.\nSo we should always do the recursive scan.\n\nTherefore, we should never set the flag on GITLINK cache-entries.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 2 +\n fsmonitor.h | 11 ++++\n t/t7527-builtin-fsmonitor.sh | 111 +++++++++++++++++++++++++++++++++++\n 3 files changed, 124 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex e1229c289cf..57d6a483bee 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tif (fsmonitor_enabled) {\n \t\t\t/* Mark all entries valid */\n \t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\t\tif (S_ISGITLINK(istate->cache[i]->ce_mode))\n+\t\t\t\t\tcontinue;\n \t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n \t\t\t}\n \ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 3f41f653691..edf7ce5203b 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n * called any time the cache entry has been updated to reflect the\n * current state of the file on disk.\n+ *\n+ * However, never mark submodules as valid. When commands like \"git\n+ * status\" run they might need to recurse into the submodule (using a\n+ * child process) to get a summary of the submodule state. We don't\n+ * have (and don't want to create) the facility to translate every\n+ * FS event that we receive and that happens to be deep inside of a\n+ * submodule back to the submodule root, so we cannot correctly keep\n+ * track of this bit on the gitlink directory. Therefore, we never\n+ * set it on submodules.\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n \n \tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n \t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tif (S_ISGITLINK(ce->ce_mode))\n+\t\t\treturn;\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex d0e681d008f..1cf0ffa5676 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -721,4 +721,115 @@ do\n \t'\n done\n \n+# Test fsmonitor interaction with submodules.\n+#\n+# If we start the daemon in the super, it will see FS events for\n+# everything in the working directory cone and this includes any\n+# files/directories contained *within* the submodules.\n+#\n+# A `git status` at top level will get events for items within the\n+# submodule and ignore them, since they aren't named in the index\n+# of the super repo. This makes the fsmonitor response a little\n+# noisy, but it doesn't alter the correctness of the state of the\n+# super-proper.\n+#\n+# When we have submodules, `git status` normally does a recursive\n+# status on each of the submodules and adds a summary row for any\n+# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n+#\n+# It is therefore important that the top level status not be tricked\n+# by the FSMonitor response to skip those recursive calls. That is,\n+# even if FSMonitor says that the mtime of the submodule directory\n+# hasn't changed and it could be implicitly marked valid, we must\n+# not take that shortcut. We need to force the recusion into the\n+# submodule so that we get a summary of the status *within* the\n+# submodule.\n+\n+create_super () {\n+\tsuper=$1 &&\n+\n+\tgit init \"${super}\" &&\n+\techo x >${super}/file_1 &&\n+\techo y >${super}/file_2 &&\n+\techo z >${super}/file_3 &&\n+\tmkdir ${super}/dir_1 &&\n+\techo a >${super}/dir_1/file_11 &&\n+\techo b >${super}/dir_1/file_12 &&\n+\tmkdir ${super}/dir_1/dir_2 &&\n+\techo a >${super}/dir_1/dir_2/file_21 &&\n+\techo b >${super}/dir_1/dir_2/file_22 &&\n+\tgit -C ${super} add . &&\n+\tgit -C ${super} commit -m \"initial ${super} commit\"\n+}\n+\n+create_sub () {\n+\tsub=$1 &&\n+\n+\tgit init \"${sub}\" &&\n+\techo x >${sub}/file_x &&\n+\techo y >${sub}/file_y &&\n+\techo z >${sub}/file_z &&\n+\tmkdir ${sub}/dir_x &&\n+\techo a >${sub}/dir_x/file_a &&\n+\techo b >${sub}/dir_x/file_b &&\n+\tmkdir ${sub}/dir_x/dir_y &&\n+\techo a >${sub}/dir_x/dir_y/file_a &&\n+\techo b >${sub}/dir_x/dir_y/file_b &&\n+\tgit -C ${sub} add . &&\n+\tgit -C ${sub} commit -m \"initial ${sub} commit\"\n+}\n+\n+my_match_and_clean () {\n+\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n+\tgit -C super --no-optional-locks -c core.fsmonitor=false \\\n+\t\tstatus --porcelain=v2 >actual.without &&\n+\ttest_cmp actual.with actual.without &&\n+\n+\tgit -C super/dir_1/dir_2/sub reset --hard &&\n+\tgit -C super/dir_1/dir_2/sub clean -d -f\n+}\n+\n+test_expect_success \"Submodule always visited\" '\n+\ttest_when_finished \"git -C super fsmonitor--daemon stop; \\\n+\t\t\t rm -rf super; \\\n+\t\t\t rm -rf sub\" &&\n+\n+\tcreate_super \"super\" &&\n+\tcreate_sub \"sub\" &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\tstart_daemon -C super &&\n+\tgit -C super config core.fsmonitor true &&\n+\tgit -C super update-index --fsmonitor &&\n+\tgit -C super status &&\n+\n+\t# Now run pairs of commands w/ and w/o FSMonitor while we make\n+\t# some dirt in the submodule and confirm matching output.\n+\n+\t# Completely clean status.\n+\tmy_match_and_clean &&\n+\n+\t# .M S..U\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M SC..\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tgit -C super/dir_1/dir_2/sub commit -m \"SC..\" &&\n+\tmy_match_and_clean\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"454022","messageId":"040c00cfd6f427cd686f9f64cec458dff1c26555.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 22/28] t/perf/p7527: add perf test for builtin FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:12Z","receivedAt":"2022-04-20T20:44:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++\n 1 file changed, 257 insertions(+)\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\ndiff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..9338b9ea008\n--- /dev/null\n+++ b/t/perf/p7527-builtin-fsmonitor.sh\n@@ -0,0 +1,257 @@\n+#!/bin/sh\n+\n+test_description=\"Perf test for the builtin FSMonitor\"\n+\n+. ./perf-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+# Lie to perf-lib and ask for a new empty repo and avoid\n+# the complaints about GIT_PERF_REPO not being big enough\n+# the perf hit when GIT_PERF_LARGE_REPO is copied into\n+# the trash directory.\n+#\n+# NEEDSWORK: It would be nice if perf-lib had an option to\n+# \"borrow\" an existing large repo (especially for gigantic\n+# monorepos) and use it in-place. For now, fake it here.\n+#\n+test_perf_fresh_repo\n+\n+\n+# Use a generated synthetic monorepo. If it doesn't exist, we will\n+# generate it. If it does exist, we will put it in a known state\n+# before we start our timings.\n+#\n+PARAM_D=5\n+PARAM_W=10\n+PARAM_F=9\n+\n+PARAMS=\"$PARAM_D\".\"$PARAM_W\".\"$PARAM_F\"\n+\n+BALLAST_BR=p0006-ballast\n+export BALLAST_BR\n+\n+TMP_BR=tmp_br\n+export TMP_BR\n+\n+REPO=../repos/gen-many-files-\"$PARAMS\".git\n+export REPO\n+\n+if ! test -d $REPO\n+then\n+\t(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)\n+fi\n+\n+\n+enable_uc () {\n+\tgit -C $REPO config core.untrackedcache true\n+\tgit -C $REPO update-index --untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+disable_uc () {\n+\tgit -C $REPO config core.untrackedcache false\n+\tgit -C $REPO update-index --no-untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+start_fsm () {\n+\tgit -C $REPO fsmonitor--daemon start\n+\tgit -C $REPO fsmonitor--daemon status\n+\tgit -C $REPO config core.fsmonitor true\n+\tgit -C $REPO update-index --fsmonitor\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+stop_fsm () {\n+\tgit -C $REPO config --unset core.fsmonitor\n+\tgit -C $REPO update-index --no-fsmonitor\n+\ttest_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+\n+# Ensure that FSMonitor is turned off on the borrowed repo.\n+#\n+test_expect_success \"Setup borrowed repo (fsm+uc)\" \"\n+\tstop_fsm &&\n+\tdisable_uc\n+\"\n+\n+# Also ensure that it starts in a known state.\n+#\n+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time\n+# the ballast checkout, since only the first invocation does any work and the\n+# subsequent ones just print \"already on branch\" and quit, so the reported\n+# time is not useful.\n+#\n+# Create a temp branch and do all work relative to it so that we don't\n+# accidentially alter the real ballast branch.\n+#\n+test_expect_success \"Setup borrowed repo (temp ballast branch)\" \"\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO reset --hard &&\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tgit -C $REPO branch $TMP_BR $BALLAST_BR &&\n+\tgit -C $REPO checkout $TMP_BR\n+\"\n+\n+\n+echo Data >data.txt\n+\n+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With\n+# FSMonitor enabled, we can get a skewed view of status times, since\n+# the index MAY (or may not) be updated after the first invocation\n+# which will update the FSMonitor Token, so the subsequent invocations\n+# may get a smaller response from the daemon.\n+#\n+do_status () {\n+\tmsg=$1\n+\n+\ttest_perf \"$msg\" \"\n+\t\tgit -C $REPO status >/dev/null 2>&1\n+\t\"\n+}\n+\n+do_matrix () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tt=\"[uc $uc][fsm $fsm]\"\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_expect_success \"$t Setup matrix branch\" \"\n+\t\tgit -C $REPO clean -d -f &&\n+\t\tgit -C $REPO checkout $TMP_BR &&\n+\t\ttest_might_fail git -C $REPO branch -D $MATRIX_BR &&\n+\t\tgit -C $REPO branch $MATRIX_BR $TMP_BR &&\n+\t\tgit -C $REPO checkout $MATRIX_BR\n+\t\"\n+\n+\tif test $uc = true\n+\tthen\n+\t\tenable_uc\n+\telse\n+\t\tdisable_uc\n+\tfi\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstart_fsm\n+\telse\n+\t\tstop_fsm\n+\tfi\n+\n+\tdo_status \"$t status after checkout\"\n+\n+\t# Modify many files in the matrix branch.\n+\t# Stage them.\n+\t# Commit them.\n+\t# Rollback.\n+\t#\n+\ttest_expect_success \"$t modify tracked files\" \"\n+\t\tfind $REPO -name file1 -exec cp data.txt {} \\\\;\n+\t\"\n+\n+\tdo_status \"$t status after big change\"\n+\n+\t# Don't bother timing the \"add\" because _REPEAT_COUNT\n+\t# issue described above.\n+\t#\n+\ttest_expect_success \"$t add all\" \"\n+\t\tgit -C $REPO add -A\n+\t\"\n+\n+\tdo_status \"$t status after add all\"\n+\n+\ttest_expect_success \"$t add dot\" \"\n+\t\tgit -C $REPO add .\n+\t\"\n+\n+\tdo_status \"$t status after add dot\"\n+\n+\ttest_expect_success \"$t commit staged\" \"\n+\t\tgit -C $REPO commit -a -m data\n+\t\"\n+\n+\tdo_status \"$t status after commit\"\n+\n+\ttest_expect_success \"$t reset HEAD~1 hard\" \"\n+\t\tgit -C $REPO reset --hard HEAD~1 >/dev/null 2>&1\n+\t\"\n+\n+\tdo_status \"$t status after reset hard\"\n+\n+\t# Create some untracked files.\n+\t#\n+\ttest_expect_success \"$t create untracked files\" \"\n+\t\tcp -R $REPO/ballast/dir1 $REPO/ballast/xxx1\n+\t\"\n+\n+\tdo_status \"$t status after create untracked files\"\n+\n+\t# Remove the new untracked files.\n+\t#\n+\ttest_expect_success \"$t clean -df\" \"\n+\t\tgit -C $REPO clean -d -f\n+\t\"\n+\n+\tdo_status \"$t status after clean\"\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstop_fsm\n+\tfi\n+}\n+\n+# Begin testing each case in the matrix that we care about.\n+#\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\n+fsm_values=\"false true\"\n+\n+for uc_val in $uc_values\n+do\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tdo_matrix $uc_val $fsm_val\n+\tdone\n+done\n+\n+cleanup () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_might_fail git -C $REPO branch -D $MATRIX_BR\n+}\n+\n+\n+# We're borrowing this repo. We should leave it in a clean state.\n+#\n+test_expect_success \"Cleanup temp and matrix branches\" \"\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\tfor fsm_val in $fsm_values\n+\t\tdo\n+\t\t\tcleanup $uc_val $fsm_val\n+\t\tdone\n+\tdone\n+\"\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"454023","messageId":"7c60623555770d6c9b2834262718fb183f483af1.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 26/28] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:16Z","receivedAt":"2022-04-20T20:44:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a set of prereqs to help understand how file names\nare handled by the filesystem when they contain NFC and NFD\nUnicode characters.\n\nSigned-off-by: Jeff Hostetler \n---\n t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 167 insertions(+)\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n\ndiff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\nnew file mode 100755\nindex 00000000000..cf9c26d1e22\n--- /dev/null\n+++ b/t/lib-unicode-nfc-nfd.sh\n@@ -0,0 +1,167 @@\n+# Help detect how Unicode NFC and NFD are handled on the filesystem.\n+\n+# A simple character that has a NFD form.\n+#\n+# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n+# UTF8(NFC): \\xc3 \\xa9\n+#\n+# NFD: U+0065 LATIN SMALL LETTER E\n+# U+0301 COMBINING ACUTE ACCENT\n+# UTF8(NFD): \\x65 + \\xcc \\x81\n+#\n+utf8_nfc=$(printf \"\\xc3\\xa9\")\n+utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n+\n+# Is the OS or the filesystem \"Unicode composition sensitive\"?\n+#\n+# That is, does the OS or the filesystem allow files to exist with\n+# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n+# tell us that the NFC and NFD forms are equivalent.\n+#\n+# This is or may be independent of what type of filesystem we have,\n+# since it might be handled by the OS at a layer above the FS.\n+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n+# collision, for example.\n+#\n+# This does not tell us how the Unicode pathname will be spelled\n+# on disk, but rather only that the two spelling \"collide\". We\n+# will examine the actual on disk spelling in a later prereq.\n+#\n+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${utf8_nfc} &&\n+\tmkdir trial_${utf8_nfd}\n+'\n+\n+# Is the spelling of an NFC pathname preserved on disk?\n+#\n+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n+# and on APFS, NFC paths are preserved. As we have established\n+# above, this is independent of \"composition sensitivity\".\n+#\n+# 0000000 63 5f c3 a9\n+#\n+# (/usr/bin/od output contains different amount of whitespace\n+# on different platforms, so we need the wildcards here.)\n+#\n+test_lazy_prereq UNICODE_NFC_PRESERVED '\n+\tmkdir c_${utf8_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *c3 *a9\"\n+'\n+\n+# Is the spelling of an NFD pathname preserved on disk?\n+#\n+# 0000000 64 5f 65 cc 81\n+#\n+test_lazy_prereq UNICODE_NFD_PRESERVED '\n+\tmkdir d_${utf8_nfd} &&\n+\tls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n+'\n+\tmkdir c_${utf8_nfc} &&\n+\tmkdir d_${utf8_nfd} &&\n+\n+# The following _DOUBLE_ forms are more for my curiosity,\n+# but there may be quirks lurking when there are multiple\n+# combining characters in non-canonical order.\n+\n+# Unicode also allows multiple combining characters\n+# that can be decomposed in pieces.\n+#\n+# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n+# UTF8(NFC): \\xe1 \\xbd \\xa7\n+#\n+# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n+# U+0342 COMBINING GREEK PERISPOMENI\n+# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n+#\n+# But U+1f61 decomposes into\n+# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n+# U+0314 COMBINING REVERSED COMMA ABOVE\n+# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n+#\n+# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n+#\n+# Note that I've used the canonical ordering of the\n+# combinining characters. It is also possible to\n+# swap them. My testing shows that that non-standard\n+# ordering also causes a collision in mkdir. However,\n+# the resulting names don't draw correctly on the\n+# terminal (implying that the on-disk format also has\n+# them out of order).\n+#\n+greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n+greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n+greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n+\n+# See if a double decomposition also collides.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${greek_nfc} &&\n+\tmkdir trial_${greek_nfd2}\n+'\n+\n+# See if the NFC spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n+\tmkdir c_${greek_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n+'\n+\n+# See if the NFD spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n+\tmkdir d_${greek_nfd2} &&\n+\tls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n+'\n+\n+# The following is for debugging. I found it useful when\n+# trying to understand the various (OS, FS) quirks WRT\n+# Unicode and how composition/decomposition is handled.\n+# For example, when trying to understand how (macOS, APFS)\n+# and (macOS, HFS) and (macOS, FAT32) compare.\n+#\n+# It is rather noisy, so it is disabled by default.\n+#\n+if test \"$unicode_debug\" = \"true\"\n+then\n+\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\techo NFC maintains original spelling.\n+\telse\n+\t\techo NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFD_PRESERVED\n+\tthen\n+\t\techo NFD maintains original spelling.\n+\telse\n+\t\techo NFD is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n+\tthen\n+\t\techo Double NFC maintains original spelling.\n+\telse\n+\t\techo Double NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n+\tthen\n+\t\techo Double NFD maintains original spelling.\n+\telse\n+\t\techo Double NFD is modified.\n+\tfi\n+fi\n-- \ngitgitgadget\n\n"},{"id":"454024","messageId":"d890c2e2d97046785efba2343d841e0cd62ccf64.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 25/28] fsmonitor: on macOS also emit NFC spelling for NFD pathname","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:15Z","receivedAt":"2022-04-20T20:44:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nEmit NFC or NFC and NFD spellings of pathnames on macOS.\n\nMacOS is Unicode composition insensitive, so NFC and NFD spellings are\ntreated as aliases and collide. While the spelling of pathnames in\nfilesystem events depends upon the underlying filesystem, such as\nAPFS, HFS+ or FAT32, the OS enforces such collisions regardless of\nfilesystem.\n\nTeach the daemon to always report the NFC spelling and to report\nthe NFD spelling when stored in that format on the disk.\n\nThis is slightly more general than \"core.precomposeUnicode\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 83d38e8ac6c..823cf63999e 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n \treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n }\n \n+/*\n+ * On MacOS we have to adjust for Unicode composition insensitivity\n+ * (where NFC and NFD spellings are not respected). The different\n+ * spellings are essentially aliases regardless of how the path is\n+ * actually stored on the disk.\n+ *\n+ * This is related to \"core.precomposeUnicode\" (which wants to try\n+ * to hide NFD completely and treat everything as NFC). Here, we\n+ * don't know what the value the client has (or will have) for this\n+ * config setting when they make a query, so assume the worst and\n+ * emit both when the OS gives us an NFD path.\n+ */\n+static void my_add_path(struct fsmonitor_batch *batch, const char *path)\n+{\n+\tchar *composed;\n+\n+\t/* add the NFC or NFD path as received from the OS */\n+\tfsmonitor_batch__add_path(batch, path);\n+\n+\t/* if NFD, also add the corresponding NFC spelling */\n+\tcomposed = (char *)precompose_string_if_needed(path);\n+\tif (!composed || composed == path)\n+\t\treturn;\n+\n+\tfsmonitor_batch__add_path(batch, composed);\n+\tfree(composed);\n+}\n+\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t\tmy_add_path(batch, rel);\n \t\t\t}\n \n \t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n@@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t\tmy_add_path(batch, tmp.buf);\n \t\t\t}\n \n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"454025","messageId":"9724c41d18d4429f8993c9cf83e7622eca079c46.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 27/28] t7527: test Unicode NFC/NFD handling on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:17Z","receivedAt":"2022-04-20T20:44:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that the daemon reports events using the on-disk\nspelling for Unicode NFC/NFD characters. On APFS we still\nhave Unicode aliasing, so we cannot create two files that\nonly differ by NFC/NFD, but the on-disk format preserves\nthe spelling used to create the file. On HFS+ we also\nhave aliasing, but the path is always stored on disk in\nNFD.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 55 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex a4f8008fea7..5476a01cbff 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -868,4 +868,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n+# The variable \"unicode_debug\" is defined in the following library\n+# script to dump information about how the (OS, FS) handles Unicode\n+# composition. Uncomment the following line if you want to enable it.\n+#\n+# unicode_debug=true\n+\n+. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n+\n+# See if the OS or filesystem does NFC/NFD aliasing/munging.\n+#\n+# The daemon should err on the side of caution and send BOTH the\n+# NFC and NFD forms. It does not know the original spelling of\n+# the pathname (how the user thinks it should be spelled), so\n+# emit both and let the client decide (when necessary). This is\n+# similar to \"core.precomposeUnicode\".\n+#\n+test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n+\n+\tgit init test_unicode &&\n+\n+\tstart_daemon -C test_unicode --tf \"$PWD/unicode.trace\" &&\n+\n+\t# Create a directory using an NFC spelling.\n+\t#\n+\tmkdir test_unicode/nfc &&\n+\tmkdir test_unicode/nfc/c_${utf8_nfc} &&\n+\n+\t# Create a directory using an NFD spelling.\n+\t#\n+\tmkdir test_unicode/nfd &&\n+\tmkdir test_unicode/nfd/d_${utf8_nfd} &&\n+\n+\tgit -C test_unicode fsmonitor--daemon stop &&\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\t# We should have seen NFC event from OS.\n+\t\t# We should not have synthesized an NFD event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace &&\n+\t\tegrep -v \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace\n+\telse\n+\t\t# We should have seen NFD event from OS.\n+\t\t# We should have synthesized an NFC event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace &&\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace\n+\tfi &&\n+\n+\t# We assume UNICODE_NFD_PRESERVED.\n+\t# We should have seen explicit NFD from OS.\n+\t# We should have synthesized an NFC event.\n+\tegrep \"^event: nfd/d_${utf8_nfd}/?$\" ./unicode.trace &&\n+\tegrep \"^event: nfd/d_${utf8_nfc}/?$\" ./unicode.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"454026","messageId":"15698d64edd0e65e5486a97024818a55c9e4295c.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 20/28] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:10Z","receivedAt":"2022-04-20T20:44:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to perform binary search over the cache-entries for a directory\nnotification and then linearly scan forward to find the immediate children.\n\nPreviously, when the FSMonitor reported a modified directory Git would\nperform a linear search on the entire cache-entry array for all\nentries matching that directory prefix and invalidate them. Since the\ncache-entry array is already sorted, we can use a binary search to\nfind the first matching entry and then only linearly walk forward and\ninvalidate entries until the prefix changes.\n\nAlso, the original code would invalidate anything having the same\ndirectory prefix. Since a directory event should only be received for\nitems that are immediately within the directory (and not within\nsub-directories of it), only invalidate those entries and not the\nwhole subtree.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 54 insertions(+), 17 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 292a6742b4f..e1229c289cf 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n-\tif (name[len - 1] == '/') {\n+\tint pos = index_name_pos(istate, name, len);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n+\t\t\t name, pos);\n \n+\tif (name[len - 1] == '/') {\n \t\t/*\n-\t\t * TODO We should binary search to find the first path with\n-\t\t * TODO this directory prefix. Then linearly update entries\n-\t\t * TODO while the prefix matches. Taking care to search without\n-\t\t * TODO the trailing slash -- because '/' sorts after a few\n-\t\t * TODO interesting special chars, like '.' and ' '.\n+\t\t * The daemon can decorate directory events, such as\n+\t\t * moves or renames, with a trailing slash if the OS\n+\t\t * FS Event contains sufficient information, such as\n+\t\t * MacOS.\n+\t\t *\n+\t\t * Use this to invalidate the entire cone under that\n+\t\t * directory.\n+\t\t *\n+\t\t * We do not expect an exact match because the index\n+\t\t * does not normally contain directory entries, so we\n+\t\t * start at the insertion point and scan.\n \t\t */\n+\t\tif (pos < 0)\n+\t\t\tpos = -pos - 1;\n \n \t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n-\t\t\t starts_with(istate->cache[i]->name, name))\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n-\t\t/* Need to remove the / from the path for the untracked cache */\n+\n+\t\t/*\n+\t\t * We need to remove the traling \"/\" from the path\n+\t\t * for the untracked cache.\n+\t\t */\n \t\tname[len - 1] = '\\0';\n+\t} else if (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\tint pos = index_name_pos(istate, name, strlen(name));\n-\n-\t\tif (pos >= 0) {\n-\t\t\tstruct cache_entry *ce = istate->cache[pos];\n-\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n \t}\n \n@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n \t * as it could be a new untracked file.\n \t */\n-\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"454027","messageId":"93de3707d26e1e4485a192b57cc4d2a9526318d5.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 24/28] t7527: test FSMonitor on case insensitive+preserving file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:14Z","receivedAt":"2022-04-20T20:44:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTest that FS events from the OS are received using the preserved,\non-disk spelling of files/directories rather than spelling used\nto make the change.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1cf0ffa5676..a4f8008fea7 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,4 +832,40 @@ test_expect_success \"Submodule always visited\" '\n \tmy_match_and_clean\n '\n \n+# On a case-insensitive file system, confirm that the daemon\n+# notices when the .git directory is moved/renamed/deleted\n+# regardless of how it is spelled in the the FS event.\n+# That is, does the FS event receive the spelling of the\n+# operation or does it receive the spelling preserved with\n+# the file/directory.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n+#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n+\n+\tgit init test_insensitive &&\n+\n+\tstart_daemon -C test_insensitive --tf \"$PWD/insensitive.trace\" &&\n+\n+\tmkdir -p test_insensitive/abc/def &&\n+\techo xyz >test_insensitive/ABC/DEF/xyz &&\n+\n+\ttest_path_is_dir test_insensitive/.git &&\n+\ttest_path_is_dir test_insensitive/.GIT &&\n+\n+\t# Rename .git using an alternate spelling to verify that that\n+\t# daemon detects it and automatically shuts down.\n+\tmv test_insensitive/.GIT test_insensitive/.FOO &&\n+\tsleep 1 &&\n+\tmv test_insensitive/.FOO test_insensitive/.git &&\n+\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\t# Verify that events were reported using on-disk spellings of the\n+\t# directories and files that we touched. We may or may not get a\n+\t# trailing slash on modified directories.\n+\t#\n+\tegrep \"^event: abc/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"454028","messageId":"9d0da8fc22bd713344da11bee41e6e8c2c644b7b.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 21/28] t7527: FSMonitor tests for directory moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:11Z","receivedAt":"2022-04-20T20:44:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate unit tests to move a directory. Verify that `git status`\ngives the same result with and without FSMonitor enabled.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex c9c7dd77e3c..d0e681d008f 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -274,6 +274,16 @@ test_expect_success 'setup' '\n \ttrace*\n \tEOF\n \n+\tmkdir -p T1/T2/T3/T4 &&\n+\techo 1 >T1/F1 &&\n+\techo 1 >T1/T2/F1 &&\n+\techo 1 >T1/T2/T3/F1 &&\n+\techo 1 >T1/T2/T3/T4/F1 &&\n+\techo 2 >T1/F2 &&\n+\techo 2 >T1/T2/F2 &&\n+\techo 2 >T1/T2/T3/F2 &&\n+\techo 2 >T1/T2/T3/T4/F2 &&\n+\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n@@ -356,6 +366,19 @@ directory_to_file () {\n \techo 1 >dir1\n }\n \n+move_directory_contents_deeper() {\n+\tmkdir T1/_new_ &&\n+\tmv T1/[A-Z]* T1/_new_\n+}\n+\n+move_directory_up() {\n+\tmv T1/T2/T3 T1\n+}\n+\n+move_directory() {\n+\tmv T1/T2/T3 T1/T2/NewT3\n+}\n+\n # The next few test cases confirm that our fsmonitor daemon sees each type\n # of OS filesystem notification that we care about. At this layer we just\n # ensure we are getting the OS notifications and do not try to confirm what\n@@ -660,6 +683,10 @@ do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n+\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n+\t\tmatrix_try $uc_val $fsm_val move_directory_up\n+\t\tmatrix_try $uc_val $fsm_val move_directory\n+\n \t\tif test $fsm_val = true\n \t\tthen\n \t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n-- \ngitgitgadget\n\n"},{"id":"454029","messageId":"3af1fe0d61d711b32ffa23a1816b4f0117244878.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 18/28] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:08Z","receivedAt":"2022-04-20T20:44:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nForce shutdown fsmonitor daemon if the worktree root directory\nis moved, renamed, or deleted.\n\nUse Windows low-level GetFileInformationByHandle() to get and\ncompare the Windows system unique ID for the directory with a\ncached version when we started up. This lets us detect the\ncase where someone renames the directory that we are watching\nand then creates a new directory with the original pathname.\n\nThis is important because we are listening to a named pipe for\nrequests and they are stored in the Named Pipe File System (NPFS)\nwhich a kernel-resident pseudo filesystem not associated with\nthe actual NTFS directory.\n\nFor example, if the daemon was watching \"~/foo/\", it would have\na directory-watch handle on that directory and a named-pipe\nhandle for \"//./pipe/...foo\". Moving the directory to \"~/bar/\"\ndoes not invalidate the directory handle. (So the daemon would\nactually be watching \"~/bar\" but listening on \"//./pipe/...foo\".\nIf the user then does \"git init ~/foo\" and causes another daemon\nto start, the first daemon will still have ownership of the pipe\nand the second daemon instance will fail to start. \"git status\"\nclients in \"~/foo\" will ask \"//./pipe/...foo\" about changes and\nthe first daemon instance will tell them about \"~/bar\".\n\nThis commit causes the first daemon to shutdown if the system unique\nID for \"~/foo\" changes (changes from what it was when the daemon\nstarted). Shutdown occurs after a periodic poll. After the\nfirst daemon exits and releases the lock on the named pipe,\nsubsequent Git commands may cause another daemon to be started\non \"~/foo\". Similarly, a subsequent Git command may cause another\ndaemon to be started on \"~/bar\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++\n 1 file changed, 143 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 24fc612bf02..2ea08c1d4e8 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -29,8 +29,150 @@ struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n \tint nr_handles; /* number of active event handles */\n+\n+\tstruct wt_moved\n+\t{\n+\t\twchar_t wpath[MAX_PATH + 1];\n+\t\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\t} wt_moved;\n };\n \n+/*\n+ * Lookup the system unique ID for the path. This is as close as\n+ * we get to an inode number, but this also contains volume info,\n+ * so it is a little stronger.\n+ */\n+static int lookup_bhfi(wchar_t *wpath,\n+\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n+{\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileW(wpath, desired_access, share_mode, NULL,\n+\t\t\t OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] health thread could not open '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\treturn -1;\n+\t}\n+\n+\tif (!GetFileInformationByHandle(hDir, bhfi)) {\n+\t\terror(_(\"[GLE %ld] health thread getting BHFI for '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\tCloseHandle(hDir);\n+\t\treturn -1;\n+\t}\n+\n+\tCloseHandle(hDir);\n+\treturn 0;\n+}\n+\n+/*\n+ * Compare the relevant fields from two system unique IDs.\n+ * We use this to see if two different handles to the same\n+ * path actually refer to the same *instance* of the file\n+ * or directory.\n+ */\n+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n+\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n+{\n+\treturn (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&\n+\t\tbhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&\n+\t\tbhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);\n+}\n+\n+/*\n+ * Shutdown if the original worktree root directory been deleted,\n+ * moved, or renamed?\n+ *\n+ * Since the main thread did a \"chdir(getenv($HOME))\" and our CWD\n+ * is not in the worktree root directory and because the listener\n+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible\n+ * for the root directory to be moved or deleted while we are still\n+ * watching it. We want to detect that here and force a shutdown.\n+ *\n+ * Granted, a delete MAY cause some operations to fail, such as\n+ * GetOverlappedResult(), but it is not guaranteed. And because\n+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the\n+ * directory, not changes *ON* the directory, our watch will not\n+ * receive a delete event for it.\n+ *\n+ * A move/rename of the worktree root will also not generate an event.\n+ * And since the listener thread already has an open handle, it may\n+ * continue to receive events for events within the directory.\n+ * However, the pathname of the named-pipe was constructed using the\n+ * original location of the worktree root. (Remember named-pipes are\n+ * stored in the NPFS and not in the actual file system.) Clients\n+ * trying to talk to the worktree after the move/rename will not\n+ * reach our daemon process, since we're still listening on the\n+ * pipe with original path.\n+ *\n+ * Furthermore, if the user does something like:\n+ *\n+ * $ mv repo repo.old\n+ * $ git init repo\n+ *\n+ * A new daemon cannot be started in the new instance of \"repo\"\n+ * because the named-pipe is still being used by the daemon on\n+ * the original instance.\n+ *\n+ * So, detect move/rename/delete and shutdown. This should also\n+ * handle unsafe drive removal.\n+ *\n+ * We use the file system unique ID to distinguish the original\n+ * directory instance from a new instance and force a shutdown\n+ * if the unique ID changes.\n+ *\n+ * Since a worktree move/rename/delete/unmount doesn't happen\n+ * that often (and we can't get an immediate event anyway), we\n+ * use a timeout and periodically poll it.\n+ */\n+static int has_worktree_moved(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\tint r;\n+\n+\tswitch (ctx) {\n+\tcase CTX_TERM:\n+\t\treturn 0;\n+\n+\tcase CTX_INIT:\n+\t\tif (xutftowcs_path(data->wt_moved.wpath,\n+\t\t\t\t state->path_worktree_watch.buf) < 0) {\n+\t\t\terror(_(\"could not convert to wide characters: '%s'\"),\n+\t\t\t state->path_worktree_watch.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/*\n+\t\t * On the first call we lookup the unique sequence ID for\n+\t\t * the worktree root directory.\n+\t\t */\n+\t\treturn lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);\n+\n+\tcase CTX_TIMER:\n+\t\tr = lookup_bhfi(data->wt_moved.wpath, &bhfi);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t\tif (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {\n+\t\t\terror(_(\"BHFI changed '%ls'\"), data->wt_moved.wpath);\n+\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tdie(_(\"unhandled case in 'has_worktree_moved': %d\"),\n+\t\t (int)ctx);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+\n int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data;\n@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n * A table of the polling functions.\n */\n static interval_fn *table[] = {\n+\thas_worktree_moved,\n \tNULL, /* must be last */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"454030","messageId":"b8325fb7c78d44dad44fd13ef5d9e81519dd8868.1650487399.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v5 28/28] fsmonitor--daemon: allow --super-prefix argument","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-20T20:43:18Z","receivedAt":"2022-04-20T20:44:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a test in t7527 to verify that we get a stray warning from\n`git fsmonitor--daemon start` when indirectly called from\n`git submodule absorbgitdirs`.\n\nUpdate `git fsmonitor--daemon` to take (and ignore) the `--super-prefix`\nargument to suppress the warning.\n\nWhen we have:\n\n1. a submodule with a `sub/.git/` directory (rather than a `sub/.git`\nfile).\n\n2. `core.fsmonitor` is turned on in the submodule, but the daemon is\nnot yet started in the submodule.\n\n3. and someone does a `git submodule absorbgitdirs` in the super.\n\nGit will recursively invoke `git submodule--helper absorb-git-dirs`\nin the submodule. This will read the index and may attempt to start\nthe fsmonitor--daemon with the `--super-prefix` argument.\n\n`git fsmonitor--daemon start` does not accept the `--super-prefix`\nargument and causes a warning to be issued.\n\nThis does not cause a problem because the `refresh_index()` code\nassumes a trivial response if the daemon does not start.\n\nThe net-net is a harmelss, but stray warning. Lets eliminate the\nwarning.\n\nSigned-off-by: Jeff Hostetler \n---\n git.c | 2 +-\n t/t7527-builtin-fsmonitor.sh | 50 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 51 insertions(+), 1 deletion(-)\n\ndiff --git a/git.c b/git.c\nindex 3d8e48cf555..e6fdac1f8e3 100644\n--- a/git.c\n+++ b/git.c\n@@ -537,7 +537,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n-\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, SUPPORT_SUPER_PREFIX | RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 5476a01cbff..237df6f3c4f 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,6 +832,56 @@ test_expect_success \"Submodule always visited\" '\n \tmy_match_and_clean\n '\n \n+# If a submodule has a `sub/.git/` directory (rather than a file\n+# pointing to the super's `.git/modules/sub`) and `core.fsmonitor`\n+# turned on in the submodule and the daemon is not yet started in\n+# the submodule, and someone does a `git submodule absorbgitdirs`\n+# in the super, Git will recursively invoke `git submodule--helper`\n+# to do the work and this may try to read the index. This will\n+# try to start the daemon in the submodule *and* pass (either\n+# directly or via inheritance) the `--super-prefix` arg to the\n+# `git fsmonitor--daemon start` command inside the submodule.\n+# This causes a warning because fsmonitor--daemon does take that\n+# global arg (see the table in git.c)\n+#\n+# This causes a warning when trying to start the daemon that is\n+# somewhat confusing. It does not seem to hurt anything because\n+# the fsmonitor code maps the query failure into a trivial response\n+# and does the work anyway.\n+#\n+# It would be nice to silence the warning, however.\n+\n+have_t2_error_event () {\n+\tlog=$1\n+\tmsg=\"fsmonitor--daemon doesnQt support --super-prefix\" &&\n+\n+\ttr '\\047' Q <$1 | grep -e \"$msg\"\n+}\n+\n+test_expect_success \"Stray Submodule super-prefix warning\" '\n+\ttest_when_finished \"rm -rf super; \\\n+\t\t\t rm -rf sub; \\\n+\t\t\t rm super-sub.trace\" &&\n+\n+\tcreate_super \"super\" &&\n+\tcreate_sub \"sub\" &&\n+\n+\t# Copy rather than submodule add so that we get a .git dir.\n+\tcp -R ./sub ./super/dir_1/dir_2/sub &&\n+\n+\tgit -C super/dir_1/dir_2/sub config core.fsmonitor true &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\ttest_path_is_dir super/dir_1/dir_2/sub/.git &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/super-sub.trace\" \\\n+\t\tgit -C super submodule absorbgitdirs &&\n+\n+\t! have_t2_error_event super-sub.trace\n+'\n+\n # On a case-insensitive file system, confirm that the daemon\n # notices when the .git directory is moved/renamed/deleted\n # regardless of how it is spelled in the the FS event.\n-- \ngitgitgadget\n"},{"id":"454041","messageId":"xmqqv8v3l3w1.fsf@gitster.g","threadId":"57414","inReplyTo":"5db241f7d2f654d46d4858ce9e9413032705abf3.1650487399.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 23/28] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-20T23:41:02Z","receivedAt":"2022-04-20T23:41:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> +create_super () {\n> +\tsuper=$1 &&\n> +\n> +\tgit init \"${super}\" &&\n\nIt is not wrong per-se, but a simple reference to a shell variable\nwithout magic interpolation like ${parameter-word} is easier to read\nwithout {} around the variable name, i.e.\n\n\tgit init \"$super\"\n\nan exception of course is when you want to suffix its value with\nalnum, i.e.\n\n\tfor d in \"$super\" \"${super}1\" \"$super\"2\n\tdo\n\t\t...\n\nand writing it as \"${super}1\" would probably be easier to see what\nis going on than \"$super\"2 notation.\n\n> +\techo x >${super}/file_1 &&\n> +\techo y >${super}/file_2 &&\n> +\techo z >${super}/file_3 &&\n\nCodingGuidelines still says that these redirection targets with\nvariable interpolation must be enclosed in double-quotes, i.e.\n\n\techo x >\"$super/file_1\" &&\n\n> +\tmkdir ${super}/dir_1 &&\n\nThe double quotes around \"${super}\" we saw on \"git init\" indicates\nthat the helper function wants to be prepared to handle a directory\npath with possibly $IFS whitespace characters in it correctly, so\nlet's make sure we are consistently prepared for such a parameter,\ni.e.\n\n\tmkdir \"$super/dir_1\" &&\n\nThe same applies to the rest of the script.\n\nThanks.\n"},{"id":"454043","messageId":"xmqqr15rl2vn.fsf@gitster.g","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 00/28] Builtin FSMonitor Part 3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-21T00:02:52Z","receivedAt":"2022-04-21T00:03:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> Here is version 5 of FSMonitor part 3. This version:\n>\n> * refactored the submodule test in commit 23 to let me reuse parts of it in\n> a later commit.\n>\n> * Added a new commit 28 to silence a stray warning when FSMonitor is\n> implicitly started in a recursive git submodule absorbgitdirs call and\n> receives the --super-prefix argument. The warning was harmless, but was\n> confusing to users.\n\nI've replaced the series and rebuilt 'seen' with it. After I push\nthe result out, I'll call it a day.\n\nThanks.\n"},{"id":"454262","messageId":"3a7d7f7b-dfba-9aa8-271f-6d12773668a4@jeffhostetler.com","threadId":"57414","inReplyTo":"220419.86mtgh8kep.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-04-22T14:47:55Z","receivedAt":"2022-04-22T14:47:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/19/22 5:44 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Mar 24 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n>> index 46be55a4618..50ae3cca575 100644\n>> --- a/builtin/fsmonitor--daemon.c\n>> +++ b/builtin/fsmonitor--daemon.c\n>> @@ -1449,6 +1449,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n>> \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n>> \t\t fsmonitor__ipc_threads);\n> \n> I think that structurally the way things are done in\n> fsmonitor-settings.c make its use really hard to follow. E.g. here:\n> \n>> +\tprepare_repo_settings(the_repository);\n> \n> We prep the repo, OK.\n> \n>> +\tfsm_settings__set_ipc(the_repository);\n> \n> Set IPC.\n> \n>> +\tif (fsm_settings__error_if_incompatible(the_repository))\n> \n> And here we'll error out if we're incompatible, and this is in the\n> top-level cmd_fsmonitor__daemon() function. All OK, except why didn't we\n> check this before \"set IPC?\".\n> \n> Anyway, re-arranging some of the diff below:\n> \n>> @@ -86,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r)\n>> \n>> \tlookup_fsmonitor_settings(r);\n>> \n>> +\tif (check_for_incompatible(r))\n>> +\t\treturn;\n>> +\n>> \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n>> \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n>> }\n> \n> Here in fsm_settings__set_ipc we return with a NOOP if we're not\n> compatible.\n> \n> Then:\n> \n>> +int fsm_settings__error_if_incompatible(struct repository *r)\n>> +{\n>> +\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n>> +\n>> +\tswitch (reason) {\n>> +\tcase FSMONITOR_REASON_OK:\n>> +\t\treturn 0;\n>> +\n>> +\tcase FSMONITOR_REASON_BARE:\n>> +\t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n>> +\t\t xgetcwd());\n>> +\t\treturn 1;\n>> +\t}\n>> +\n>> +\tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n>> +\t reason);\n>> +}\n> \n> Here we'll call fsm_settings__get_reason() which does the same.\n> \n>> +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n>> +{\n>> +\tif (!r)\n>> +\t\tr = the_repository;\n>> +\n>> +\tlookup_fsmonitor_settings(r);\n>> +\n>> +\treturn r->settings.fsmonitor->reason;\n>> +}\n> \n> Is there a reason we can't skip this indirection when using the API like\n> this and e.g. do:\n> \n> \tenum fsmonitor_reason reason;\n> \tprepare_repo_settings(the_repository);\n> \treason = fsmonitor_check_for_incompatible(the_repository)\n> if (reason != FSMONITOR_REASON_OK)\n> \tdie(\"%s\", fsm_settings__get_reason_string(reason));\n> \n> There's just two callers of this API in \"seen\", and neither need/want\n> this pattern where every method needs to lazy load itself, or the\n> indirection where fsmonitor-settings.c needs to be used as a\n> clearing-house for state management.\n> \n> Maybe I'm missing something, but why not make check_for_incompatible()\n> non-static and have the callers use that (and then it would return\n> \"fsmonitor_reason\", not \"int\", the int return value being redundant to\n> the enum)>\n\nI suppose we could rearrange things to hide less of the\nstate management. I'm not sure it matters one way or the\nother, but I'll give it a try and see if simplifies things.\n\n> \n>> diff --git a/builtin/update-index.c b/builtin/update-index.c\n>> index 876112abb21..d29048f16f2 100644\n>> --- a/builtin/update-index.c\n>> +++ b/builtin/update-index.c\n>> @@ -1237,6 +1237,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n>> \n>> \tif (fsmonitor > 0) {\n>> \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n>> +\n>> +\t\tif (fsm_settings__error_if_incompatible(the_repository))\n>> +\t\t\treturn 1;\n>> +\n>> \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n>> \t\t\twarning(_(\"core.fsmonitor is unset; \"\n>> \t\t\t\t\"set it if you really want to \"\n> \n> This looks like a bug, we knew before aquiring the lockfile that we\n> weren't compatible, so why wait until here to error out? This seems to\n> skip the rollback_lock_file(), so won't we leave a stale lock?\n> \n\nYes, good catch. The `return` here will bypass the rollback.\nHopefully, the above rearrangement will make this go away.\n\nI do have to wonder about the rest of this function. There are\nseveral `die()`, `exit()`, `usage()`, `BUG()`, and other `return`\nstatements after the index lock is taken that won't hit the\nrollback. Should those be investigated too? (I can't do that\nnow in the context of this series, though.)\n\nJeff\n"},{"id":"454277","messageId":"7cbabd84-0c4d-5517-0d5d-3f5725d3105d@jeffhostetler.com","threadId":"57414","inReplyTo":"xmqqv8v3l3w1.fsf@gitster.g","subject":"Re: [PATCH v5 23/28] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-04-22T20:47:09Z","receivedAt":"2022-04-22T22:01:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/20/22 7:41 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> +create_super () {\n>> +\tsuper=$1 &&\n>> +\n>> +\tgit init \"${super}\" &&\n> \n> It is not wrong per-se, but a simple reference to a shell variable\n> without magic interpolation like ${parameter-word} is easier to read\n> without {} around the variable name, i.e.\n> \n> \tgit init \"$super\"\n> \n> an exception of course is when you want to suffix its value with\n> alnum, i.e.\n> \n> \tfor d in \"$super\" \"${super}1\" \"$super\"2\n> \tdo\n> \t\t...\n> \n> and writing it as \"${super}1\" would probably be easier to see what\n> is going on than \"$super\"2 notation.\n> \n>> +\techo x >${super}/file_1 &&\n>> +\techo y >${super}/file_2 &&\n>> +\techo z >${super}/file_3 &&\n> \n> CodingGuidelines still says that these redirection targets with\n> variable interpolation must be enclosed in double-quotes, i.e.\n> \n> \techo x >\"$super/file_1\" &&\n> \n>> +\tmkdir ${super}/dir_1 &&\n> \n> The double quotes around \"${super}\" we saw on \"git init\" indicates\n> that the helper function wants to be prepared to handle a directory\n> path with possibly $IFS whitespace characters in it correctly, so\n> let's make sure we are consistently prepared for such a parameter,\n> i.e.\n> \n> \tmkdir \"$super/dir_1\" &&\n> \n> The same applies to the rest of the script.\n> \n> Thanks.\n> \n\ngood points. i'll fixup and resend.\nthanks!\n\njeff\n"},{"id":"454280","messageId":"8a474d69999514b343ff576ca1d96180c22b403d.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 03/28] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:29Z","receivedAt":"2022-04-22T22:35:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a stress test to hammer on the fsmonitor daemon.\nCreate a client-side thread pool of n threads and have\neach of them make m requests as fast as they can.\n\nWe do not currently inspect the contents of the response.\nWe're only interested in placing a heavy request load on\nthe daemon.\n\nThis test is useful for interactive testing and various\nexperimentation. For example, to place additional load\non the daemon while another test is running. We currently\ndo not have a test script that actually uses this helper.\nWe might add such a test in the future.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++\n 1 file changed, 106 insertions(+)\n\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nindex 3062c8a3c2b..54a4856c48c 100644\n--- a/t/helper/test-fsmonitor-client.c\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -7,6 +7,8 @@\n #include \"cache.h\"\n #include \"parse-options.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2.h\"\n \n #ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n int cmd__fsmonitor_client(int argc, const char **argv)\n@@ -79,20 +81,121 @@ static int do_send_flush(void)\n \treturn 0;\n }\n \n+struct hammer_thread_data\n+{\n+\tpthread_t pthread_id;\n+\tint thread_nr;\n+\n+\tint nr_requests;\n+\tconst char *token;\n+\n+\tint sum_successful;\n+\tint sum_errors;\n+};\n+\n+static void *hammer_thread_proc(void *_hammer_thread_data)\n+{\n+\tstruct hammer_thread_data *data = _hammer_thread_data;\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint k;\n+\tint ret;\n+\n+\ttrace2_thread_start(\"hammer\");\n+\n+\tfor (k = 0; k < data->nr_requests; k++) {\n+\t\tstrbuf_reset(&answer);\n+\n+\t\tret = fsmonitor_ipc__send_query(data->token, &answer);\n+\t\tif (ret < 0)\n+\t\t\tdata->sum_errors++;\n+\t\telse\n+\t\t\tdata->sum_successful++;\n+\t}\n+\n+\tstrbuf_release(&answer);\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Start a pool of client threads that will each send a series of\n+ * commands to the daemon.\n+ *\n+ * The goal is to overload the daemon with a sustained series of\n+ * concurrent requests.\n+ */\n+static int do_hammer(const char *token, int nr_threads, int nr_requests)\n+{\n+\tstruct hammer_thread_data *data = NULL;\n+\tint k;\n+\tint sum_join_errors = 0;\n+\tint sum_commands = 0;\n+\tint sum_errors = 0;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\tif (nr_threads < 1)\n+\t\tnr_threads = 1;\n+\tif (nr_requests < 1)\n+\t\tnr_requests = 1;\n+\n+\tCALLOC_ARRAY(data, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\t\tp->thread_nr = k;\n+\t\tp->nr_requests = nr_requests;\n+\t\tp->token = token;\n+\n+\t\tif (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {\n+\t\t\twarning(\"failed to create thread[%d] skipping remainder\", k);\n+\t\t\tnr_threads = k;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\n+\t\tif (pthread_join(p->pthread_id, NULL))\n+\t\t\tsum_join_errors++;\n+\t\tsum_commands += p->sum_successful;\n+\t\tsum_errors += p->sum_errors;\n+\t}\n+\n+\tfprintf(stderr, \"HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\\n\",\n+\t\tnr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);\n+\n+\tfree(data);\n+\n+\t/*\n+\t * Return an error if any of the _send_query requests failed.\n+\t * We don't care about thread create/join errors.\n+\t */\n+\treturn sum_errors > 0;\n+}\n+\n int cmd__fsmonitor_client(int argc, const char **argv)\n {\n \tconst char *subcmd;\n \tconst char *token = NULL;\n+\tint nr_threads = 1;\n+\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n \t\t\"test-tool fsmonitor-client query []\",\n \t\t\"test-tool fsmonitor-client flush\",\n+\t\t\"test-tool fsmonitor-client hammer [] [] []\",\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n \t\tOPT_STRING(0, \"token\", &token, \"token\",\n \t\t\t \"command token to send to the server\"),\n+\n+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, \"number of client threads\"),\n+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, \"number of requests per thread\"),\n+\n \t\tOPT_END()\n \t};\n \n@@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)\n \tif (!strcmp(subcmd, \"flush\"))\n \t\treturn !!do_send_flush();\n \n+\tif (!strcmp(subcmd, \"hammer\"))\n+\t\treturn !!do_hammer(token, nr_threads, nr_requests);\n+\n \tdie(\"Unhandled subcommand: '%s'\", subcmd);\n }\n #endif\n-- \ngitgitgadget\n\n"},{"id":"454281","messageId":"4b52083698cd556a05dea5ea8e18ba01ec923ffd.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 10/28] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:36Z","receivedAt":"2022-04-22T22:35:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nOn MacOS mark repos on NTFS or FAT32 volumes as incompatible.\n\nThe builtin FSMonitor used Unix domain sockets on MacOS for IPC\nwith clients. These sockets are kept in the .git directory.\nUnix sockets are not supported by NTFS and FAT32, so the daemon\ncannot start up.\n\nTest for this during our compatibility checking so that client\ncommands do not keep trying to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---\n fsmonitor-settings.c | 6 ++++++\n fsmonitor-settings.h | 1 +\n 3 files changed, 24 insertions(+), 3 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex fdd762bf79d..efc732c0f31 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -7,7 +7,7 @@\n #include \n \n /*\n- * Remote working directories are problematic for FSMonitor.\n+ * [1] Remote working directories are problematic for FSMonitor.\n *\n * The underlying file system on the server machine and/or the remote\n * mount type (NFS, SAMBA, etc.) dictates whether notification events\n@@ -40,8 +40,16 @@\n *\n * So (for now at least), mark remote working directories as\n * incompatible.\n+ *\n+ *\n+ * [2] FAT32 and NTFS working directories are problematic too.\n+ *\n+ * The builtin FSMonitor uses a Unix domain socket in the .git\n+ * directory for IPC. These Windows drive formats do not support\n+ * Unix domain sockets, so mark them as incompatible for the daemon.\n+ *\n */\n-static enum fsmonitor_reason check_remote(struct repository *r)\n+static enum fsmonitor_reason check_volume(struct repository *r)\n {\n \tstruct statfs fs;\n \n@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)\n \tif (!(fs.f_flags & MNT_LOCAL))\n \t\treturn FSMONITOR_REASON_REMOTE;\n \n+\tif (!strcmp(fs.f_fstypename, \"msdos\")) /* aka FAT32 */\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n+\tif (!strcmp(fs.f_fstypename, \"ntfs\"))\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \n@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n \n-\treason = check_remote(r);\n+\treason = check_volume(r);\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex d2fb0141f8e..658cb79da01 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -225,6 +225,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t\t r->worktree);\n \t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_NOSOCKETS:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex ed26ddd7ba9..2834296a1e3 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -20,6 +20,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n \tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n+\tFSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n-- \ngitgitgadget\n\n"},{"id":"454282","messageId":"5b246bec2470431898ddc74013f6fb325c2bc281.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 02/28] t7527: test FSMonitor on repos with Unicode root paths","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:28Z","receivedAt":"2022-04-22T22:35:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate some test repos with UTF8 characters in the pathname of the\nroot directory and verify that the builtin FSMonitor can watch them.\n\nThis test is mainly for Windows where we need to avoid `*A()`\nroutines.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1be21785162..c9c7dd77e3c 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -671,4 +671,27 @@ do\n \tdone\n done\n \n+# Test Unicode UTF-8 characters in the pathname of the working\n+# directory root. Use of \"*A()\" routines rather than \"*W()\" routines\n+# on Windows can sometimes lead to odd failures.\n+#\n+u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n+u2=$(printf \"u_e2_99_ab__\\xE2\\x99\\xAB\")\n+u_values=\"$u1 $u2\"\n+for u in $u_values\n+do\n+\ttest_expect_success \"Unicode in repo root path: $u\" '\n+\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n+\n+\t\tgit init \"$u\" &&\n+\t\techo 1 >\"$u\"/file1 &&\n+\t\tgit -C \"$u\" add file1 &&\n+\t\tgit -C \"$u\" config core.fsmonitor true &&\n+\n+\t\tstart_daemon -C \"$u\" &&\n+\t\tgit -C \"$u\" status >actual &&\n+\t\tgrep \"new file: file1\" actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"454283","messageId":"495e54049b4a6043206bc0a7a7fdc1cf0d761ec5.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 09/28] fsmonitor-settings: remote repos on Windows are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:35Z","receivedAt":"2022-04-22T22:35:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on Windows and mark them as\nincompatible with FSMonitor.\n\nWith this `git fsmonitor--daemon run` will error out with a message like it\ndoes for bare repos.\n\nClient commands, such as `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++\n 1 file changed, 102 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex ee78bba38e3..907655720bb 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * VFS for Git is incompatible with FSMonitor.\n@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)\n \treturn FSMONITOR_REASON_OK;\n }\n \n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type dictates whether notification events are available at\n+ * all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ *\n+ * Notes for testing:\n+ *\n+ * (a) Windows allows a network share to be mapped to a drive letter.\n+ * (This is the normal method to access it.)\n+ *\n+ * $ NET USE Z: \\\\server\\share\n+ * $ git -C Z:/repo status\n+ *\n+ * (b) Windows allows a network share to be referenced WITHOUT mapping\n+ * it to drive letter.\n+ *\n+ * $ NET USE \\\\server\\share\\dir\n+ * $ git -C //server/share/repo status\n+ *\n+ * (c) Windows allows \"SUBST\" to create a fake drive mapping to an\n+ * arbitrary path (which may be remote)\n+ *\n+ * $ SUBST Q: Z:\\repo\n+ * $ git -C Q:/ status\n+ *\n+ * (d) Windows allows a directory symlink to be created on a local\n+ * file system that points to a remote repo.\n+ *\n+ * $ mklink /d ./link //server/share/repo\n+ * $ git -C ./link status\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\twchar_t wpath[MAX_PATH];\n+\twchar_t wfullpath[MAX_PATH];\n+\tsize_t wlen;\n+\tUINT driveType;\n+\n+\t/*\n+\t * Do everything in wide chars because the drive letter might be\n+\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n+\t */\n+\tif (xutftowcs_path(wpath, r->worktree) < 0)\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\t/*\n+\t * GetDriveTypeW() requires a final slash. We assume that the\n+\t * worktree pathname points to an actual directory.\n+\t */\n+\twlen = wcslen(wpath);\n+\tif (wpath[wlen - 1] != L'\\\\' && wpath[wlen - 1] != L'/') {\n+\t\twpath[wlen++] = L'\\\\';\n+\t\twpath[wlen] = 0;\n+\t}\n+\n+\t/*\n+\t * Normalize the path. If nothing else, this converts forward\n+\t * slashes to backslashes. This is essential to get GetDriveTypeW()\n+\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n+\t */\n+\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\tdriveType = GetDriveTypeW(wfullpath);\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"DriveType '%s' L'%ls' (%u)\",\n+\t\t\t r->worktree, wfullpath, driveType);\n+\n+\tif (driveType == DRIVE_REMOTE) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"check_remote('%s') true\",\n+\t\t\t\t r->worktree);\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n-- \ngitgitgadget\n\n"},{"id":"454284","messageId":"2e225c3f4f279a71c2c0d5bbb5d65c147162de99.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 05/28] fsmonitor-settings: stub in Win32-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:31Z","receivedAt":"2022-04-22T22:35:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend generic incompatibility checkout with platform-specific\nmechanism. Stub in Win32 version.\n\nIn the existing fsmonitor-settings code we have a way to mark\ntypes of repos as incompatible with fsmonitor (whether via the\nhook and IPC APIs). For example, we do this for bare repos,\nsince there are no files to watch.\n\nExtend this exclusion mechanism for platform-specific reasons.\nThis commit just creates the framework and adds a stub for Win32.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 +++++++++++++\n compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++\n config.mak.uname | 4 ++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n fsmonitor-settings.c | 10 ++++++++++\n fsmonitor-settings.h | 13 +++++++++++++\n 6 files changed, 52 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n\ndiff --git a/Makefile b/Makefile\nindex daa21bed6c3..93604fe8ef7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -475,6 +475,11 @@ all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n+# If your platform has OS-specific ways to tell if a repo is incompatible with\n+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n+# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n+# that implements the `fsm_os_settings__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n+ifdef FSMONITOR_OS_SETTINGS\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE\n ifdef FSMONITOR_DAEMON_BACKEND\n \t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n endif\n+ifdef FSMONITOR_OS_SETTINGS\n+\t@echo FSMONITOR_OS_SETTINGS=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_OS_SETTINGS)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 501970902da..cf224768ad6 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex ee0d7257b77..16705da2000 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 7d3177d441a..f67db913f57 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -23,6 +23,16 @@ static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n \t\treturn FSMONITOR_REASON_BARE;\n \t}\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+\t{\n+\t\tenum fsmonitor_reason reason;\n+\n+\t\treason = fsm_os__incompatible(r);\n+\t\tif (reason != FSMONITOR_REASON_OK)\n+\t\t\treturn reason;\n+\t}\n+#endif\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 4c7592896e4..1694b7091a2 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -34,4 +34,17 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \n struct fsmonitor_settings;\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+/*\n+ * Ask platform-specific code whether the repository is incompatible\n+ * with fsmonitor (both hook and ipc modes). For example, if the working\n+ * directory is on a remote volume and mounted via a technology that does\n+ * not support notification events, then we should not pretend to watch it.\n+ *\n+ * fsm_os__* routines should considered private to fsm_settings__\n+ * routines.\n+ */\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);\n+#endif /* HAVE_FSMONITOR_OS_SETTINGS */\n+\n #endif /* FSMONITOR_SETTINGS_H */\n-- \ngitgitgadget\n\n"},{"id":"454285","messageId":"f4feb00ec2b9d6f80e5b57275eb8e141f013e253.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 12/28] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:38Z","receivedAt":"2022-04-22T22:35:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIgnore FSEvents resulting from `xattr` changes. Git does not care about\nxattr's or changes to xattr's, so don't waste time collecting these\nevents in the daemon nor transmitting them to clients.\n\nVarious security tools add xattrs to files and/or directories, such as\nto mark them as having been downloaded. We should ignore these events\nsince it doesn't affect the content of the file/directory or the normal\nmeta-data that Git cares about.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 0741fe834c3..14105f45c18 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tif (flag & kFSEventStreamEventFlagItemCloned)\n \t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n \n-\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n \t\t\t path, flag, msg.buf);\n \n \tstrbuf_release(&msg);\n@@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)\n \t\tef & kFSEventStreamEventFlagUserDropped);\n }\n \n+/*\n+ * If an `xattr` change is the only reason we received this event,\n+ * then silently ignore it. Git doesn't care about xattr's. We\n+ * have to be careful here because the kernel can combine multiple\n+ * events for a single path. And because events always have certain\n+ * bits set, such as `ItemIsFile` or `ItemIsDir`.\n+ *\n+ * Return 1 if we should ignore it.\n+ */\n+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n+{\n+\tstatic const FSEventStreamEventFlags mask =\n+\t\tkFSEventStreamEventFlagItemChangeOwner |\n+\t\tkFSEventStreamEventFlagItemCreated |\n+\t\tkFSEventStreamEventFlagItemFinderInfoMod |\n+\t\tkFSEventStreamEventFlagItemInodeMetaMod |\n+\t\tkFSEventStreamEventFlagItemModified |\n+\t\tkFSEventStreamEventFlagItemRemoved |\n+\t\tkFSEventStreamEventFlagItemRenamed |\n+\t\tkFSEventStreamEventFlagItemXattrMod |\n+\t\tkFSEventStreamEventFlagItemCloned;\n+\n+\treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n+}\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_ignore_xattr(event_flags[k])) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n+\t\t\t\t\t path_k, event_flags[k]);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n \n \t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-- \ngitgitgadget\n\n"},{"id":"454286","messageId":"1f5b772d42ae31a897a4966fe88714413826268c.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 08/28] fsmonitor-settings: remote repos on macOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:34Z","receivedAt":"2022-04-22T22:35:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on macOS and mark them as\nincompatible with FSMonitor.\n\nWith this, `git fsmonitor--daemon run` will error out with a message\nlike it does for bare repos.\n\nClient commands, like `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 12 +++++\n fsmonitor-settings.h | 2 +\n 3 files changed, 80 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex 7fce32a3c5b..fdd762bf79d 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -2,8 +2,74 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n+#include \n+#include \n+\n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type (NFS, SAMBA, etc.) dictates whether notification events\n+ * are available at all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * For the builtin FSMonitor, we create the Unix domain socket for the\n+ * IPC in the .git directory. If the working directory is remote,\n+ * then the socket will be created on the remote file system. This\n+ * can fail if the remote file system does not support UDS file types\n+ * (e.g. smbfs to a Windows server) or if the remote kernel does not\n+ * allow a non-local process to bind() the socket. (These problems\n+ * could be fixed by moving the UDS out of the .git directory and to a\n+ * well-known local directory on the client machine, but care should\n+ * be taken to ensure that $HOME is actually local and not a managed\n+ * file share.)\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\tstruct statfs fs;\n+\n+\tif (statfs(r->worktree, &fs) == -1) {\n+\t\tint saved_errno = errno;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n+\t\t\t\t r->worktree, strerror(saved_errno));\n+\t\terrno = saved_errno;\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n+\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n+\n+\tif (!(fs.f_flags & MNT_LOCAL))\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 600ae165ab1..d2fb0141f8e 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -208,6 +208,18 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t xgetcwd());\n \t\tgoto done;\n \n+\tcase FSMONITOR_REASON_ERROR:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_REMOTE:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"remote repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n+\n \tcase FSMONITOR_REASON_VFS4GIT:\n \t\tstrbuf_addf(&msg,\n \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 9e483c14513..ed26ddd7ba9 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -17,6 +17,8 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n \tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n+\tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"454287","messageId":"dbb983fd9d0473f3bac0b8873ac59a3b4c54d79b.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 13/28] fsmonitor--daemon: cd out of worktree root","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:39Z","receivedAt":"2022-04-22T22:35:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the fsmonitor--daemon to CD outside of the worktree\nbefore starting up.\n\nThe common Git startup mechanism causes the CWD of the daemon process\nto be in the root of the worktree. On Windows, this causes the daemon\nprocess to hold a locked handle on the CWD and prevents other\nprocesses from moving or deleting the worktree while the daemon is\nrunning.\n\nCD to HOME before entering main event loops.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++--\n compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------\n fsmonitor--daemon.h | 1 +\n 3 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 66b78a0353f..db297649daf 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1181,11 +1181,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * before we need it.\n \t */\n \tif (ipc_server_run_async(&state->ipc_server_data,\n-\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t state->path_ipc.buf, &ipc_opts,\n \t\t\t\t handle_client, state))\n \t\treturn error_errno(\n \t\t\t_(\"could not start IPC thread pool on '%s'\"),\n-\t\t\tfsmonitor_ipc__get_path());\n+\t\t\tstate->path_ipc.buf);\n \n \t/*\n \t * Start the fsmonitor listener thread to collect filesystem\n@@ -1220,6 +1220,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n static int fsmonitor_run_daemon(void)\n {\n \tstruct fsmonitor_daemon_state state;\n+\tconst char *home;\n \tint err;\n \n \tmemset(&state, 0, sizeof(state));\n@@ -1289,6 +1290,15 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_addch(&state.path_cookie_prefix, '/');\n \n+\t/*\n+\t * We create a named-pipe or unix domain socket inside of the\n+\t * \".git\" directory. (Well, on Windows, we base our named\n+\t * pipe in the NPFS on the absolute path of the git\n+\t * directory.)\n+\t */\n+\tstrbuf_init(&state.path_ipc, 0);\n+\tstrbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1298,6 +1308,23 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\t/*\n+\t * CD out of the worktree root directory.\n+\t *\n+\t * The common Git startup mechanism causes our CWD to be the\n+\t * root of the worktree. On Windows, this causes our process\n+\t * to hold a locked handle on the CWD. This prevents the\n+\t * worktree from being moved or deleted while the daemon is\n+\t * running.\n+\t *\n+\t * We assume that our FS and IPC listener threads have either\n+\t * opened all of the handles that they need or will do\n+\t * everything using absolute paths.\n+\t */\n+\thome = getenv(\"HOME\");\n+\tif (home && *home && chdir(home))\n+\t\tdie_errno(_(\"could not cd home '%s'\"), home);\n+\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n@@ -1310,6 +1337,7 @@ done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n \tstrbuf_release(&state.path_cookie_prefix);\n+\tstrbuf_release(&state.path_ipc);\n \n \treturn err;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 3f1b68267bd..c43d92b9620 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch)\n \t}\n \n \t/*\n-\t * NEEDSWORK: If an external is deleted, the above\n-\t * returns an error. I'm not sure that there's anything that\n-\t * we can do here other than failing -- the /.git\n-\t * link file would be broken anyway. We might try to check\n-\t * for that and return a better error message, but I'm not\n-\t * sure it is worth it.\n+\t * GetOverlappedResult() fails if the watched directory is\n+\t * deleted while we were waiting for an overlapped IO to\n+\t * complete. The documentation did not list specific errors,\n+\t * but I observed ERROR_ACCESS_DENIED (0x05) errors during\n+\t * testing.\n+\t *\n+\t * Note that we only get notificaiton events for events\n+\t * *within* the directory, not *on* the directory itself.\n+\t * (These might be properies of the parent directory, for\n+\t * example).\n+\t *\n+\t * NEEDSWORK: We might try to check for the deleted directory\n+\t * case and return a better error message, but I'm not sure it\n+\t * is worth it.\n+\t *\n+\t * Shutdown if we get any error.\n \t */\n \n \terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex bd09fffc176..223c2131b58 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n+\tstruct strbuf path_ipc;\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"454288","messageId":"32fc6ba74378cda7a5060d63066d16333765973a.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 16/28] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:42Z","receivedAt":"2022-04-22T22:36:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate another thread to watch over the daemon process and\nautomatically shut it down if necessary.\n\nThis commit creates the basic framework for a \"health\" thread\nto monitor the daemon and/or the file system. Later commits\nwill add platform-specific code to do the actual work.\n\nThe \"health\" thread is intended to monitor conditions that\nwould be difficult to track inside the IPC thread pool and/or\nthe file system listener threads. For example, when there are\nfile system events outside of the watched worktree root or if\nwe want to have an idle-timeout auto-shutdown feature.\n\nThis commit creates the health thread itself, defines the thread-proc\nand sets up the thread's event loop. It integrates this new thread\ninto the existing IPC and Listener thread models.\n\nThis commit defines the API to the platform-specific code where all of\nthe monitoring will actually happen.\n\nThe platform-specific code for MacOS is just stubs. Meaning that the\nhealth thread will immediately exit on MacOS, but that is OK and\nexpected. Future work can define MacOS-specific monitoring.\n\nThe platform-specific code for Windows sets up enough of the\nWaitForMultipleObjects() machinery to watch for system and/or custom\nevents. Currently, the set of wait handles only includes our custom\nshutdown event (sent from our other theads). Later commits in this\nseries will extend the set of wait handles to monitor other\nconditions.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 6 ++-\n builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n contrib/buildsystems/CMakeLists.txt | 2 +\n fsmonitor--daemon.h | 4 ++\n 7 files changed, 192 insertions(+), 2 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n\ndiff --git a/Makefile b/Makefile\nindex 93604fe8ef7..5f1623baadd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -472,8 +472,9 @@ all::\n #\n # If your platform supports a built-in fsmonitor backend, set\n # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n-# `compat/fsmonitor/fsm-listen-.c` that implements the\n-# `fsm_listen__*()` routines.\n+# `compat/fsmonitor/fsm-listen-.c` and\n+# `compat/fsmonitor/fsm-health-.c` files\n+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n # If your platform has OS-specific ways to tell if a repo is incompatible with\n # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n@@ -1982,6 +1983,7 @@ endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n ifdef FSMONITOR_OS_SETTINGS\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex b2f578b239a..2c109cf8b37 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n@@ -1136,6 +1137,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+static void *fsm_health__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-health\");\n+\n+\tfsm_health__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -1174,6 +1187,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint health_started = 0;\n \tint listener_started = 0;\n \tint err = 0;\n \n@@ -1201,6 +1215,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t}\n \tlistener_started = 1;\n \n+\t/*\n+\t * Start the health thread to watch over our process.\n+\t */\n+\tif (pthread_create(&state->health_thread, NULL,\n+\t\t\t fsm_health__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\terr = error(_(\"could not start fsmonitor health thread\"));\n+\t\tgoto cleanup;\n+\t}\n+\thealth_started = 1;\n+\n \t/*\n \t * The daemon is now fully functional in background threads.\n \t * Our primary thread should now just wait while the threads\n@@ -1223,10 +1248,17 @@ cleanup:\n \t\tpthread_join(state->listener_thread, NULL);\n \t}\n \n+\tif (health_started) {\n+\t\tfsm_health__stop_async(state);\n+\t\tpthread_join(state->health_thread, NULL);\n+\t}\n+\n \tif (err)\n \t\treturn err;\n \tif (state->listen_error_code)\n \t\treturn state->listen_error_code;\n+\tif (state->health_error_code)\n+\t\treturn state->health_error_code;\n \treturn 0;\n }\n \n@@ -1242,6 +1274,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.listen_error_code = 0;\n+\tstate.health_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n@@ -1321,6 +1354,11 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\tif (fsm_health__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize health thread\"));\n+\t\tgoto done;\n+\t}\n+\n \t/*\n \t * CD out of the worktree root directory.\n \t *\n@@ -1344,6 +1382,7 @@ done:\n \tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n+\tfsm_health__dtor(&state);\n \n \tipc_server_free(state.ipc_server_data);\n \ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nnew file mode 100644\nindex 00000000000..b9f709e8548\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -0,0 +1,24 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nnew file mode 100644\nindex 00000000000..94b1d020f25\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsm_health_data\n+{\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hHandles[1]; /* the array does not own these handles */\n+#define HEALTH_SHUTDOWN 0\n+\tint nr_handles; /* number of active event handles */\n+};\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_handles++;\n+\n+\tstate->health_data = data;\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tif (!state || !state->health_data)\n+\t\treturn;\n+\n+\tdata = state->health_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\n+\tFREE_AND_NULL(state->health_data);\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\n+\tfor (;;) {\n+\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n+\t\t\t\t\t\t data->hHandles,\n+\t\t\t\t\t\t FALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->health_error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+clean_shutdown:\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n+}\ndiff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\nnew file mode 100644\nindex 00000000000..45547ba9380\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health.h\n@@ -0,0 +1,47 @@\n+#ifndef FSM_HEALTH_H\n+#define FSM_HEALTH_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor health thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the health thread.\n+ * This will be called from the main thread AFTER joining the thread.\n+ */\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to monitor the\n+ * health of the daemon process. This will run in the health thread.\n+ *\n+ * The health thread should call `ipc_server_stop_async()` if it needs\n+ * to cause a shutdown. (It should NOT do so if it receives a shutdown\n+ * shutdown signal.)\n+ *\n+ * It should set `state->health_error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_health__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the health thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_HEALTH_H */\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex b8f9f7a0388..16ace43d1c7 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 2c6fa1a5d91..2102a5c9ff5 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n+struct fsm_health_data; /* opaque platform-specific data for health thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n+\tpthread_t health_thread;\n \tpthread_mutex_t main_lock;\n \n \tstruct strbuf path_worktree_watch;\n@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n \tstruct hashmap cookies;\n \n \tint listen_error_code;\n+\tint health_error_code;\n \tstruct fsm_listen_data *listen_data;\n+\tstruct fsm_health_data *health_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"454289","messageId":"8b7c5f4e234e5b139b640652fcfdafb2e24e9db8.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 01/28] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:27Z","receivedAt":"2022-04-22T22:36:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach FSMonitor daemon on Windows to recognize shortname paths as\naliases of normal longname paths. FSMonitor clients, such as `git\nstatus`, should receive the longname spelling of changed files (when\npossible).\n\nSometimes we receive FS events using the shortname, such as when a CMD\nshell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\narrives using whatever combination of long and shortnames were used by\nthe other process. (Shortnames do seem to be case normalized,\nhowever.)\n\nUse Windows GetLongPathNameW() to try to map the pathname spelling in\nthe notification event into the normalized longname spelling. (This\ncan fail if the file/directory is deleted, moved, or renamed, because\nwe are asking the FS for the mapping in response to the event and\nafter it has already happened, but we try.)\n\nSpecial case the shortname spelling of \".git\" to avoid under-reporting\nthese events.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++-----\n t/t7527-builtin-fsmonitor.sh | 65 +++++\n 2 files changed, 374 insertions(+), 54 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 5b928ab66e5..3f1b68267bd 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -25,6 +25,9 @@ struct one_watch\n \tDWORD count;\n \n \tstruct strbuf path;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n+\tDWORD wpath_longname_len;\n+\n \tHANDLE hDir;\n \tHANDLE hEvent;\n \tOVERLAPPED overlapped;\n@@ -34,6 +37,21 @@ struct one_watch\n \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n \t */\n \tBOOL is_active;\n+\n+\t/*\n+\t * Are shortnames enabled on the containing drive? This is\n+\t * always true for \"C:/\" drives and usually never true for\n+\t * other drives.\n+\t *\n+\t * We only set this for the worktree because we only need to\n+\t * convert shortname paths to longname paths for items we send\n+\t * to clients. (We don't care about shortname expansion for\n+\t * paths inside a GITDIR because we never send them to\n+\t * clients.)\n+\t */\n+\tBOOL has_shortnames;\n+\tBOOL has_tilda;\n+\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n struct fsmonitor_daemon_backend_data\n@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n };\n \n /*\n- * Convert the WCHAR path from the notification into UTF8 and\n- * then normalize it.\n+ * Convert the WCHAR path from the event into UTF8 and normalize it.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n */\n-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n \t\t\t\t struct strbuf *normalized_path)\n {\n \tint reserve;\n \tint len = 0;\n \n \tstrbuf_reset(normalized_path);\n-\tif (!info->FileNameLength)\n+\tif (!wpath_len)\n \t\tgoto normalize;\n \n \t/*\n@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t * sequence of 2 UTF8 characters. That should let us\n \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n \t */\n-\treserve = info->FileNameLength + 1;\n+\treserve = 2 * wpath_len + 1;\n \tstrbuf_grow(normalized_path, reserve);\n \n \tfor (;;) {\n-\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n-\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n+\t\t\t\t\t wpath, wpath_len,\n \t\t\t\t\t normalized_path->buf,\n \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n \t\t\t\t\t NULL, NULL);\n@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n \t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n-\t\t\t GetLastError(),\n-\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n-\t\t\t info->FileName);\n+\t\t\t GetLastError(), (int)wpath_len, wpath);\n \t\t\treturn -1;\n \t\t}\n \n@@ -98,6 +115,152 @@ normalize:\n \treturn strbuf_normalize_path(normalized_path);\n }\n \n+/*\n+ * See if the worktree root directory has shortnames enabled.\n+ * This will help us decide if we need to do an expensive shortname\n+ * to longname conversion on every notification event.\n+ *\n+ * We do not want to create a file to test this, so we assume that the\n+ * root directory contains a \".git\" file or directory. (Our caller\n+ * only calls us for the worktree root, so this should be fine.)\n+ *\n+ * Remember the spelling of the shortname for \".git\" if it exists.\n+ */\n+static void check_for_shortnames(struct one_watch *watch)\n+{\n+\twchar_t buf_in[MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\twchar_t *last_slash = NULL;\n+\twchar_t *last_bslash = NULL;\n+\twchar_t *last;\n+\n+\t/* build L\"/.git\" */\n+\twcscpy(buf_in, watch->wpath_longname);\n+\twcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n+\n+\tif (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n+\t\treturn;\n+\n+\tlast_slash = wcsrchr(buf_out, L'/');\n+\tlast_bslash = wcsrchr(buf_out, L'\\\\');\n+\tif (last_slash > last_bslash)\n+\t\tlast = last_slash + 1;\n+\telse if (last_bslash)\n+\t\tlast = last_bslash + 1;\n+\telse\n+\t\tlast = buf_out;\n+\n+\tif (!wcscmp(last, L\".git\"))\n+\t\treturn;\n+\n+\twatch->has_shortnames = 1;\n+\twcsncpy(watch->dotgit_shortname, last,\n+\t\tARRAY_SIZE(watch->dotgit_shortname));\n+\n+\t/*\n+\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n+\t * we should be able to avoid shortname to longname mapping on\n+\t * every notification event if the source string does not\n+\t * contain a \"~\".\n+\t *\n+\t * However, the documentation for GetLongPathNameW() says\n+\t * that there are filesystems that don't follow that pattern\n+\t * and warns against this optimization.\n+\t *\n+\t * Lets test this.\n+\t */\n+\tif (wcschr(watch->dotgit_shortname, L'~'))\n+\t\twatch->has_tilda = 1;\n+}\n+\n+enum get_relative_result {\n+\tGRR_NO_CONVERSION_NEEDED,\n+\tGRR_HAVE_CONVERSION,\n+\tGRR_SHUTDOWN,\n+};\n+\n+/*\n+ * Info notification paths are relative to the root of the watch.\n+ * If our CWD is still at the root, then we can use relative paths\n+ * to convert from shortnames to longnames. If our process has a\n+ * different CWD, then we need to construct an absolute path, do\n+ * the conversion, and then return the root-relative portion.\n+ *\n+ * We use the longname form of the root as our basis and assume that\n+ * it already has a trailing slash.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n+ */\n+static enum get_relative_result get_relative_longname(\n+\tstruct one_watch *watch,\n+\tconst wchar_t *wpath, DWORD wpath_len,\n+\twchar_t *wpath_longname)\n+{\n+\twchar_t buf_in[2 * MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\tDWORD root_len;\n+\n+\t/* Build L\"/\" */\n+\troot_len = watch->wpath_longname_len;\n+\twcsncpy(buf_in, watch->wpath_longname, root_len);\n+\twcsncpy(buf_in + root_len, wpath, wpath_len);\n+\tbuf_in[root_len + wpath_len] = 0;\n+\n+\t/*\n+\t * We don't actually know if the source pathname is a\n+\t * shortname or a longname. This routine allows either to be\n+\t * given as input.\n+\t */\n+\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n+\t\t/*\n+\t\t * The shortname to longname conversion can fail for\n+\t\t * various reasons, for example if the file has been\n+\t\t * deleted. (That is, if we just received a\n+\t\t * delete-file notification event and the file is\n+\t\t * already gone, we can't ask the file system to\n+\t\t * lookup the longname for it. Likewise, for moves\n+\t\t * and renames where we are given the old name.)\n+\t\t *\n+\t\t * Since deleting or moving a file or directory by its\n+\t\t * shortname is rather obscure, I'm going ignore the\n+\t\t * failure and ask the caller to report the original\n+\t\t * relative path. This seems kinder than failing here\n+\t\t * and forcing a resync. Besides, forcing a resync on\n+\t\t * every file/directory delete would effectively\n+\t\t * cripple monitoring.\n+\t\t *\n+\t\t * We might revisit this in the future.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (!wcscmp(buf_in, buf_out)) {\n+\t\t/*\n+\t\t * The path does not have a shortname alias.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (wcsncmp(buf_in, buf_out, root_len)) {\n+\t\t/*\n+\t\t * The spelling of the root directory portion of the computed\n+\t\t * longname has changed. This should not happen. Basically,\n+\t\t * it means that we don't know where (without recomputing the\n+\t\t * longname of just the root directory) to split out the\n+\t\t * relative path. Since this should not happen, I'm just\n+\t\t * going to let this fail and force a shutdown (because all\n+\t\t * subsequent events are probably going to see the same\n+\t\t * mismatch).\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\t/* Return the worktree root-relative portion of the longname. */\n+\n+\twcscpy(wpath_longname, buf_out + root_len);\n+\treturn GRR_HAVE_CONVERSION;\n+}\n+\n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n@@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n-\twchar_t wpath[MAX_PATH];\n+\tDWORD len_longname;\n+\twchar_t wpath[MAX_PATH + 1];\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \tif (xutftowcs_path(wpath, path) < 0) {\n \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n@@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \t\treturn NULL;\n \t}\n \n+\tif (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n+\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n+\t\t GetLastError(), path);\n+\t\tCloseHandle(hDir);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen_longname = wcslen(wpath_longname);\n+\tif (wpath_longname[len_longname - 1] != L'/' &&\n+\t wpath_longname[len_longname - 1] != L'\\\\') {\n+\t\twpath_longname[len_longname++] = L'/';\n+\t\twpath_longname[len_longname] = 0;\n+\t}\n+\n \tCALLOC_ARRAY(watch, 1);\n \n \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n@@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tstrbuf_init(&watch->path, 0);\n \tstrbuf_addstr(&watch->path, path);\n \n+\twcscpy(watch->wpath_longname, wpath_longname);\n+\twatch->wpath_longname_len = len_longname;\n+\n \twatch->hDir = hDir;\n \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n \n@@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t,\n+\tDWORD info_action)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n+\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n \tstruct fsmonitor_batch *batch = NULL;\n \tconst char *p = watch->buffer;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \t/*\n \t * If the kernel gets more events than will fit in the kernel\n@@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n+\t\twchar_t *wpath = info->FileName;\n+\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n \t\tenum fsmonitor_path_type t;\n+\t\tenum get_relative_result grr;\n+\n+\t\tif (watch->has_shortnames) {\n+\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n+\t\t\t\t/*\n+\t\t\t\t * This event exactly matches the\n+\t\t\t\t * spelling of the shortname of\n+\t\t\t\t * \".git\", so we can skip some steps.\n+\t\t\t\t *\n+\t\t\t\t * (This case is odd because the user\n+\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n+\t\t\t\t * use the filesystem to map it back\n+\t\t\t\t * to \".git\".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_reset(&path);\n+\t\t\t\tstrbuf_addstr(&path, \".git\");\n+\t\t\t\tt = IS_DOT_GIT;\n+\t\t\t\tgoto process_it;\n+\t\t\t}\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n-\t\t\tgoto skip_this_path;\n-\n-\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n-\n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n+\t\t\tif (watch->has_tilda && !wcschr(wpath, L'~')) {\n+\t\t\t\t/*\n+\t\t\t\t * Shortnames on this filesystem have tildas\n+\t\t\t\t * and the notification path does not have\n+\t\t\t\t * one, so we assume that it is a longname.\n+\t\t\t\t */\n+\t\t\t\tgoto normalize_it;\n+\t\t\t}\n \n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n+\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n+\t\t\t\t\t\t wpath_longname);\n+\t\t\tswitch (grr) {\n+\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n+\t\t\t\tbreak;\n+\t\t\tcase GRR_HAVE_CONVERSION:\n+\t\t\t\twpath = wpath_longname;\n+\t\t\t\twpath_len = wcslen(wpath);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\tcase GRR_SHUTDOWN:\n \t\t\t\tgoto force_shutdown;\n \t\t\t}\n-\t\t\tbreak;\n+\t\t}\n \n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n+normalize_it:\n+\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n+\t\t\tgoto skip_this_path;\n \n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+process_it:\n+\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n+\t\t\t\t\t info->Action))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -382,6 +630,9 @@ force_shutdown:\n * Note that we DO NOT get filesystem events on the external \n * itself (it is not inside something that we are watching). In particular,\n * we do not get an event if the external is deleted.\n+ *\n+ * Also, we do not care about shortnames within the external , since\n+ * we never send these paths to clients.\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n@@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\tif (normalize_path_in_utf8(\n+\t\t\t info->FileName,\n+\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t &path) == -1)\n \t\t\tgoto skip_this_path;\n \n \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n@@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \tif (!data->watch_worktree)\n \t\tgoto failed;\n \n+\tcheck_for_shortnames(data->watch_worktree);\n+\n \tif (state->nr_paths_watching > 1) {\n \t\tdata->watch_gitdir = create_watch(state,\n \t\t\t\t\t\t state->path_gitdir_watch.buf);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex bd0c952a116..1be21785162 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -166,6 +166,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon -C test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n+\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n+\n+\tstart_daemon -C test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"454290","messageId":"c50ed29a31045d99374eaddde1cb9229eae3b773.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 07/28] fsmonitor-settings: stub in macOS-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:33Z","receivedAt":"2022-04-22T22:36:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++\n config.mak.uname | 1 +\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 13 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cf224768ad6..cf911d141f2 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)\n \tifndef NO_PTHREADS\n \tifndef NO_UNIX_SOCKETS\n \tFSMONITOR_DAEMON_BACKEND = darwin\n+\tFSMONITOR_OS_SETTINGS = darwin\n \tendif\n \tendif\n \ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 16705da2000..b8f9f7a0388 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"454291","messageId":"d4a4263d379d2d0f3b61d57541161d8706bb96a7.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 11/28] unpack-trees: initialize fsmonitor_has_run_once in o->result","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:37Z","receivedAt":"2022-04-22T22:36:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInitialize `o->result.fsmonitor_has_run_once` based upon value\nin `o->src_index->fsmonitor_has_run_once` to prevent a second\nfsmonitor query during the tree traversal and possibly getting\na skewed view of the working directory.\n\nThe checkout code has already talked to the fsmonitor and the\ntraversal is updating the index as it traverses, so there is\nno need to query the fsmonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n unpack-trees.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..888cff81f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \to->result.fsmonitor_last_update =\n \t\txstrdup_or_null(o->src_index->fsmonitor_last_update);\n+\to->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;\n \n \t/*\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n-- \ngitgitgadget\n\n"},{"id":"454292","messageId":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v5.git.1650487398.gitgitgadget@gmail.com","subject":"[PATCH v6 00/28] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:26Z","receivedAt":"2022-04-22T22:36:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 6 of part 3 of FSMonitor.\n\nThis version addresses:\n\n 1. Junio's comments on V5 23/28 WRT shell variable references and quoting\n pathnames in the create_super and create_sub helper functions.\n\n 2. Ævar's comments on V4 4/27 (sorry I didn't see them until after I sent\n V5) WRT somewhat blurry logic around the fsmonitor-settings and\n detecting incompatible worktrees. I simplified things, but not to the\n level that Ævar was suggesting. For example, in builtin/update-index.c\n the suggestion was to detect incompatible FS before taking the lock on\n the index, but the lock is taken before the CL args are parsed (because\n update-index uses a custom version of parse_options_start()), so we\n don't know yet whether the user passed --fsmonitor until much later and\n that is what triggers the error/warning. I did replace the return 1 with\n a die() so hopefully, we'll release the lock on the index like all of\n the other errors in that function. I did try to better document the code\n in update-index.c and in builtin/fsmonitor--daemon.c to explain how\n things are supposed to work. So hopefully it'll be easier to review.\n\n 3. Also, in update-index and fsmonitor--daemon, I redid how the error\n messages are printed, so that I could use die() in the cmd_*() functions\n rather than having calls to error() hidden inside fsmonitor-settings.c.\n I think that helped with the above cleanup.\n\nHere is a range diff from V5. It is a little noisy because of the untangling\nwithin fsmonitor-settings.c and moving the error messages.\n\n 1: 8b7c5f4e23 = 1: 8b7c5f4e23 fsm-listen-win32: handle shortnames\n 2: 5b246bec24 = 2: 5b246bec24 t7527: test FSMonitor on repos with Unicode root paths\n 3: 8a474d6999 = 3: 8a474d6999 t/helper/fsmonitor-client: create stress test\n 4: 004b67b62e < -: ---------- fsmonitor-settings: bare repos are incompatible with FSMonitor\n -: ---------- > 4: 72b94acd5f fsmonitor-settings: bare repos are incompatible with FSMonitor\n 5: e1e55550c1 ! 5: 2e225c3f4f fsmonitor-settings: stub in Win32-specific incompatibility checking\n @@ contrib/buildsystems/CMakeLists.txt: if(SUPPORTS_SIMPLE_IPC)\n list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)\n - \t\treturn 1;\n +@@ fsmonitor-settings.c: static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n + \t\treturn FSMONITOR_REASON_BARE;\n }\n \n +#ifdef HAVE_FSMONITOR_OS_SETTINGS\n @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)\n +\t\tenum fsmonitor_reason reason;\n +\n +\t\treason = fsm_os__incompatible(r);\n -+\t\tif (reason != FSMONITOR_REASON_OK) {\n -+\t\t\tset_incompatible(r, reason);\n -+\t\t\treturn 1;\n -+\t\t}\n ++\t\tif (reason != FSMONITOR_REASON_OK)\n ++\t\t\treturn reason;\n +\t}\n +#endif\n +\n - \treturn 0;\n + \treturn FSMONITOR_REASON_OK;\n }\n \n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: int fsm_settings__error_if_incompatible(struct repository *r);\n +@@ fsmonitor-settings.h: char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \n struct fsmonitor_settings;\n \n 6: 2d68fc9a46 ! 6: e0d3bdf755 fsmonitor-settings: VFS for Git virtual repos are incompatible\n @@ compat/fsmonitor/fsm-settings-win32.c\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n - \t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n - \t\t xgetcwd());\n - \t\treturn 1;\n +@@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repository *r,\n + \t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n + \t\t\t xgetcwd());\n + \t\tgoto done;\n +\n +\tcase FSMONITOR_REASON_VFS4GIT:\n -+\t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n -+\t\t r->worktree);\n -+\t\treturn 1;\n ++\t\tstrbuf_addf(&msg,\n ++\t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n ++\t\t\t r->worktree);\n ++\t\tgoto done;\n }\n \n - \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n + \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: enum fsmonitor_mode {\n - enum fsmonitor_reason {\n - \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n +@@ fsmonitor-settings.h: enum fsmonitor_reason {\n + \tFSMONITOR_REASON_UNTESTED = 0,\n + \tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n FSMONITOR_REASON_BARE,\n +\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n 7: 94ae2e424f = 7: c50ed29a31 fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: b2ca6c1b20 ! 8: 1f5b772d42 fsmonitor-settings: remote repos on macOS are incompatible\n @@ compat/fsmonitor/fsm-settings-darwin.c\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n - \t\t xgetcwd());\n - \t\treturn 1;\n +@@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repository *r,\n + \t\t\t xgetcwd());\n + \t\tgoto done;\n \n +\tcase FSMONITOR_REASON_ERROR:\n -+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n -+\t\t r->worktree);\n -+\t\treturn 1;\n ++\t\tstrbuf_addf(&msg,\n ++\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n ++\t\t\t r->worktree);\n ++\t\tgoto done;\n +\n +\tcase FSMONITOR_REASON_REMOTE:\n -+\t\terror(_(\"remote repository '%s' is incompatible with fsmonitor\"),\n -+\t\t r->worktree);\n -+\t\treturn 1;\n ++\t\tstrbuf_addf(&msg,\n ++\t\t\t _(\"remote repository '%s' is incompatible with fsmonitor\"),\n ++\t\t\t r->worktree);\n ++\t\tgoto done;\n +\n case FSMONITOR_REASON_VFS4GIT:\n - \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n - \t\t r->worktree);\n + \t\tstrbuf_addf(&msg,\n + \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: enum fsmonitor_mode {\n - enum fsmonitor_reason {\n - \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n +@@ fsmonitor-settings.h: enum fsmonitor_reason {\n + \tFSMONITOR_REASON_UNTESTED = 0,\n + \tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n FSMONITOR_REASON_BARE,\n +\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n +\tFSMONITOR_REASON_REMOTE,\n 9: a3cc4b3b16 = 9: 495e54049b fsmonitor-settings: remote repos on Windows are incompatible\n10: 8f1f484075 ! 10: 4b52083698 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n @@ compat/fsmonitor/fsm-settings-darwin.c: enum fsmonitor_reason fsm_os__incompatib\n \n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n - \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n - \t\t r->worktree);\n - \t\treturn 1;\n +@@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repository *r,\n + \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n + \t\t\t r->worktree);\n + \t\tgoto done;\n +\n +\tcase FSMONITOR_REASON_NOSOCKETS:\n -+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n -+\t\t r->worktree);\n -+\t\treturn 1;\n ++\t\tstrbuf_addf(&msg,\n ++\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n ++\t\t\t r->worktree);\n ++\t\tgoto done;\n }\n \n - \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n + \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_reason {\n11: 8d48d9c562 = 11: d4a4263d37 unpack-trees: initialize fsmonitor_has_run_once in o->result\n12: 088c7b3334 = 12: f4feb00ec2 fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n13: 00fab62666 = 13: dbb983fd9d fsmonitor--daemon: cd out of worktree root\n14: 6552f51802 = 14: ae90b99ea9 fsmonitor--daemon: prepare for adding health thread\n15: f2bf07cd73 = 15: b6c5800095 fsmonitor--daemon: rename listener thread related variables\n16: 2a44f2eded = 16: 32fc6ba743 fsmonitor--daemon: stub in health thread\n17: 854fb5e365 = 17: 77bc037481 fsm-health-win32: add polling framework to monitor daemon health\n18: 3af1fe0d61 = 18: b06edd995e fsm-health-win32: force shutdown daemon if worktree root moves\n19: f1365cdd40 = 19: 1bd5f34624 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n20: 15698d64ed = 20: 48af0813de fsmonitor: optimize processing of directory events\n21: 9d0da8fc22 = 21: a9b35e770f t7527: FSMonitor tests for directory moves\n22: 040c00cfd6 = 22: 26308936af t/perf/p7527: add perf test for builtin FSMonitor\n23: 5db241f7d2 ! 23: d0e25f6bac fsmonitor: never set CE_FSMONITOR_VALID on submodules\n @@ t/t7527-builtin-fsmonitor.sh: do\n +# submodule.\n +\n +create_super () {\n -+\tsuper=$1 &&\n -+\n -+\tgit init \"${super}\" &&\n -+\techo x >${super}/file_1 &&\n -+\techo y >${super}/file_2 &&\n -+\techo z >${super}/file_3 &&\n -+\tmkdir ${super}/dir_1 &&\n -+\techo a >${super}/dir_1/file_11 &&\n -+\techo b >${super}/dir_1/file_12 &&\n -+\tmkdir ${super}/dir_1/dir_2 &&\n -+\techo a >${super}/dir_1/dir_2/file_21 &&\n -+\techo b >${super}/dir_1/dir_2/file_22 &&\n -+\tgit -C ${super} add . &&\n -+\tgit -C ${super} commit -m \"initial ${super} commit\"\n ++\tsuper=\"$1\" &&\n ++\n ++\tgit init \"$super\" &&\n ++\techo x >\"$super/file_1\" &&\n ++\techo y >\"$super/file_2\" &&\n ++\techo z >\"$super/file_3\" &&\n ++\tmkdir \"$super/dir_1\" &&\n ++\techo a >\"$super/dir_1/file_11\" &&\n ++\techo b >\"$super/dir_1/file_12\" &&\n ++\tmkdir \"$super/dir_1/dir_2\" &&\n ++\techo a >\"$super/dir_1/dir_2/file_21\" &&\n ++\techo b >\"$super/dir_1/dir_2/file_22\" &&\n ++\tgit -C \"$super\" add . &&\n ++\tgit -C \"$super\" commit -m \"initial $super commit\"\n +}\n +\n +create_sub () {\n -+\tsub=$1 &&\n -+\n -+\tgit init \"${sub}\" &&\n -+\techo x >${sub}/file_x &&\n -+\techo y >${sub}/file_y &&\n -+\techo z >${sub}/file_z &&\n -+\tmkdir ${sub}/dir_x &&\n -+\techo a >${sub}/dir_x/file_a &&\n -+\techo b >${sub}/dir_x/file_b &&\n -+\tmkdir ${sub}/dir_x/dir_y &&\n -+\techo a >${sub}/dir_x/dir_y/file_a &&\n -+\techo b >${sub}/dir_x/dir_y/file_b &&\n -+\tgit -C ${sub} add . &&\n -+\tgit -C ${sub} commit -m \"initial ${sub} commit\"\n ++\tsub=\"$1\" &&\n ++\n ++\tgit init \"$sub\" &&\n ++\techo x >\"$sub/file_x\" &&\n ++\techo y >\"$sub/file_y\" &&\n ++\techo z >\"$sub/file_z\" &&\n ++\tmkdir \"$sub/dir_x\" &&\n ++\techo a >\"$sub/dir_x/file_a\" &&\n ++\techo b >\"$sub/dir_x/file_b\" &&\n ++\tmkdir \"$sub/dir_x/dir_y\" &&\n ++\techo a >\"$sub/dir_x/dir_y/file_a\" &&\n ++\techo b >\"$sub/dir_x/dir_y/file_b\" &&\n ++\tgit -C \"$sub\" add . &&\n ++\tgit -C \"$sub\" commit -m \"initial $sub commit\"\n +}\n +\n +my_match_and_clean () {\n @@ t/t7527-builtin-fsmonitor.sh: do\n +\t\t\t rm -rf super; \\\n +\t\t\t rm -rf sub\" &&\n +\n -+\tcreate_super \"super\" &&\n -+\tcreate_sub \"sub\" &&\n ++\tcreate_super super &&\n ++\tcreate_sub sub &&\n +\n +\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n +\tgit -C super commit -m \"add sub\" &&\n24: 93de3707d2 = 24: 410dd2d292 t7527: test FSMonitor on case insensitive+preserving file system\n25: d890c2e2d9 = 25: cd7c55b0d3 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n26: 7c60623555 = 26: 8278f32c4d t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n27: 9724c41d18 = 27: 4efb3a4383 t7527: test Unicode NFC/NFD handling on MacOS\n28: b8325fb7c7 ! 28: df1b4f3a80 fsmonitor--daemon: allow --super-prefix argument\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n +\t\t\t rm -rf sub; \\\n +\t\t\t rm super-sub.trace\" &&\n +\n -+\tcreate_super \"super\" &&\n -+\tcreate_sub \"sub\" &&\n ++\tcreate_super super &&\n ++\tcreate_sub sub &&\n +\n +\t# Copy rather than submodule add so that we get a .git dir.\n +\tcp -R ./sub ./super/dir_1/dir_2/sub &&\n\n\nJeff Hostetler (28):\n fsm-listen-win32: handle shortnames\n t7527: test FSMonitor on repos with Unicode root paths\n t/helper/fsmonitor-client: create stress test\n fsmonitor-settings: bare repos are incompatible with FSMonitor\n fsmonitor-settings: stub in Win32-specific incompatibility checking\n fsmonitor-settings: VFS for Git virtual repos are incompatible\n fsmonitor-settings: stub in macOS-specific incompatibility checking\n fsmonitor-settings: remote repos on macOS are incompatible\n fsmonitor-settings: remote repos on Windows are incompatible\n fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n unpack-trees: initialize fsmonitor_has_run_once in o->result\n fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n fsmonitor--daemon: cd out of worktree root\n fsmonitor--daemon: prepare for adding health thread\n fsmonitor--daemon: rename listener thread related variables\n fsmonitor--daemon: stub in health thread\n fsm-health-win32: add polling framework to monitor daemon health\n fsm-health-win32: force shutdown daemon if worktree root moves\n fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n fsmonitor: optimize processing of directory events\n t7527: FSMonitor tests for directory moves\n t/perf/p7527: add perf test for builtin FSMonitor\n fsmonitor: never set CE_FSMONITOR_VALID on submodules\n t7527: test FSMonitor on case insensitive+preserving file system\n fsmonitor: on macOS also emit NFC spelling for NFD pathname\n t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n t7527: test Unicode NFC/NFD handling on MacOS\n fsmonitor--daemon: allow --super-prefix argument\n\n Makefile | 19 +-\n builtin/fsmonitor--daemon.c | 116 ++++++-\n builtin/update-index.c | 16 +\n compat/fsmonitor/fsm-health-darwin.c | 24 ++\n compat/fsmonitor/fsm-health-win32.c | 278 +++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 +++\n compat/fsmonitor/fsm-listen-darwin.c | 122 ++++++--\n compat/fsmonitor/fsm-listen-win32.c | 413 ++++++++++++++++++++-----\n compat/fsmonitor/fsm-listen.h | 2 +-\n compat/fsmonitor/fsm-settings-darwin.c | 89 ++++++\n compat/fsmonitor/fsm-settings-win32.c | 137 ++++++++\n config.mak.uname | 5 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 11 +-\n fsmonitor-settings.c | 167 ++++++++--\n fsmonitor-settings.h | 33 ++\n fsmonitor.c | 73 ++++-\n fsmonitor.h | 11 +\n git.c | 2 +-\n t/helper/test-fsmonitor-client.c | 106 +++++++\n t/lib-unicode-nfc-nfd.sh | 167 ++++++++++\n t/perf/p7527-builtin-fsmonitor.sh | 257 +++++++++++++++\n t/t7519-status-fsmonitor.sh | 32 ++\n t/t7527-builtin-fsmonitor.sh | 367 ++++++++++++++++++++++\n unpack-trees.c | 1 +\n 25 files changed, 2358 insertions(+), 145 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\n\nbase-commit: 5eb696daba2fe108d4d9ba2ccf4b357447ef9946\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/1143\n\nRange-diff vs v5:\n\n 1: 8b7c5f4e234 = 1: 8b7c5f4e234 fsm-listen-win32: handle shortnames\n 2: 5b246bec247 = 2: 5b246bec247 t7527: test FSMonitor on repos with Unicode root paths\n 3: 8a474d69999 = 3: 8a474d69999 t/helper/fsmonitor-client: create stress test\n 4: 004b67b62e3 ! 4: 72b94acd5fe fsmonitor-settings: bare repos are incompatible with FSMonitor\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## builtin/fsmonitor--daemon.c ##\n +@@ builtin/fsmonitor--daemon.c: static int try_to_start_background_daemon(void)\n + int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n + {\n + \tconst char *subcmd;\n ++\tenum fsmonitor_reason reason;\n + \tint detach_console = 0;\n + \n + \tstruct option options[] = {\n @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n +\tprepare_repo_settings(the_repository);\n ++\t/*\n ++\t * If the repo is fsmonitor-compatible, explicitly set IPC-mode\n ++\t * (without bothering to load the `core.fsmonitor` config settings).\n ++\t *\n ++\t * If the repo is not compatible, the repo-settings will be set to\n ++\t * incompatible rather than IPC, so we can use one of the __get\n ++\t * routines to detect the discrepancy.\n ++\t */\n +\tfsm_settings__set_ipc(the_repository);\n +\n -+\tif (fsm_settings__error_if_incompatible(the_repository))\n -+\t\treturn 1;\n ++\treason = fsm_settings__get_reason(the_repository);\n ++\tif (reason > FSMONITOR_REASON_OK)\n ++\t\tdie(\"%s\",\n ++\t\t fsm_settings__get_incompatible_msg(the_repository,\n ++\t\t\t\t\t\t reason));\n +\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n ++\t\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n +\n -+\t\tif (fsm_settings__error_if_incompatible(the_repository))\n -+\t\t\treturn 1;\n ++\t\t/*\n ++\t\t * The user wants to turn on FSMonitor using the command\n ++\t\t * line argument. (We don't know (or care) whether that\n ++\t\t * is the IPC or HOOK version.)\n ++\t\t *\n ++\t\t * Use one of the __get routines to force load the FSMonitor\n ++\t\t * config settings into the repo-settings. That will detect\n ++\t\t * whether the file system is compatible so that we can stop\n ++\t\t * here with a nice error message.\n ++\t\t */\n ++\t\tif (reason > FSMONITOR_REASON_OK)\n ++\t\t\tdie(\"%s\",\n ++\t\t\t fsm_settings__get_incompatible_msg(r, reason));\n +\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n @@ fsmonitor-settings.c\n \tchar *hook_path;\n };\n \n -+static void set_incompatible(struct repository *r,\n -+\t\t\t enum fsmonitor_reason reason)\n -+{\n -+\tstruct fsmonitor_settings *s = r->settings.fsmonitor;\n -+\n -+\ts->mode = FSMONITOR_MODE_INCOMPATIBLE;\n -+\ts->reason = reason;\n -+}\n -+\n -+static int check_for_incompatible(struct repository *r)\n +-static void lookup_fsmonitor_settings(struct repository *r)\n ++static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n +{\n +\tif (!r->worktree) {\n +\t\t/*\n +\t\t * Bare repositories don't have a working directory and\n +\t\t * therefore have nothing to watch.\n +\t\t */\n -+\t\tset_incompatible(r, FSMONITOR_REASON_BARE);\n -+\t\treturn 1;\n ++\t\treturn FSMONITOR_REASON_BARE;\n +\t}\n +\n -+\treturn 0;\n ++\treturn FSMONITOR_REASON_OK;\n +}\n +\n - static void lookup_fsmonitor_settings(struct repository *r)\n ++static struct fsmonitor_settings *alloc_settings(void)\n {\n \tstruct fsmonitor_settings *s;\n ++\n ++\tCALLOC_ARRAY(s, 1);\n ++\ts->mode = FSMONITOR_MODE_DISABLED;\n ++\ts->reason = FSMONITOR_REASON_UNTESTED;\n ++\n ++\treturn s;\n ++}\n ++\n ++static void lookup_fsmonitor_settings(struct repository *r)\n ++{\n + \tconst char *const_str;\n + \tint bool_value;\n + \n + \tif (r->settings.fsmonitor)\n + \t\treturn;\n + \n +-\tCALLOC_ARRAY(s, 1);\n +-\ts->mode = FSMONITOR_MODE_DISABLED;\n +-\n +-\tr->settings.fsmonitor = s;\n +-\n + \t/*\n + \t * Overload the existing \"core.fsmonitor\" config setting (which\n + \t * has historically been either unset or a hook pathname) to\n @@ fsmonitor-settings.c: static void lookup_fsmonitor_settings(struct repository *r)\n + \tcase 0: /* config value was set to */\n + \t\tif (bool_value)\n + \t\t\tfsm_settings__set_ipc(r);\n ++\t\telse\n ++\t\t\tfsm_settings__set_disabled(r);\n + \t\treturn;\n \n - \tCALLOC_ARRAY(s, 1);\n - \ts->mode = FSMONITOR_MODE_DISABLED;\n -+\ts->reason = FSMONITOR_REASON_OK;\n + \tcase 1: /* config value was unset */\n +@@ fsmonitor-settings.c: static void lookup_fsmonitor_settings(struct repository *r)\n + \t\treturn;\n + \t}\n \n - \tr->settings.fsmonitor = s;\n +-\tif (!const_str || !*const_str)\n +-\t\treturn;\n +-\n +-\tfsm_settings__set_hook(r, const_str);\n ++\tif (const_str && *const_str)\n ++\t\tfsm_settings__set_hook(r, const_str);\n ++\telse\n ++\t\tfsm_settings__set_disabled(r);\n + }\n \n -@@ fsmonitor-settings.c: void fsm_settings__set_ipc(struct repository *r)\n + enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n + {\n + \tif (!r)\n + \t\tr = the_repository;\n +-\n +-\tlookup_fsmonitor_settings(r);\n ++\tif (!r->settings.fsmonitor)\n ++\t\tlookup_fsmonitor_settings(r);\n \n - \tlookup_fsmonitor_settings(r);\n + \treturn r->settings.fsmonitor->mode;\n + }\n +@@ fsmonitor-settings.c: const char *fsm_settings__get_hook_path(struct repository *r)\n + {\n + \tif (!r)\n + \t\tr = the_repository;\n +-\n +-\tlookup_fsmonitor_settings(r);\n ++\tif (!r->settings.fsmonitor)\n ++\t\tlookup_fsmonitor_settings(r);\n \n -+\tif (check_for_incompatible(r))\n + \treturn r->settings.fsmonitor->hook_path;\n + }\n + \n + void fsm_settings__set_ipc(struct repository *r)\n + {\n ++\tenum fsmonitor_reason reason = check_for_incompatible(r);\n ++\n ++\tif (reason != FSMONITOR_REASON_OK) {\n ++\t\tfsm_settings__set_incompatible(r, reason);\n +\t\treturn;\n ++\t}\n +\n ++\t/*\n ++\t * Caller requested IPC explicitly, so avoid (possibly\n ++\t * recursive) config lookup.\n ++\t */\n + \tif (!r)\n + \t\tr = the_repository;\n +-\n +-\tlookup_fsmonitor_settings(r);\n ++\tif (!r->settings.fsmonitor)\n ++\t\tr->settings.fsmonitor = alloc_settings();\n + \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n ++\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n -@@ fsmonitor-settings.c: void fsm_settings__set_hook(struct repository *r, const char *path)\n - \n - \tlookup_fsmonitor_settings(r);\n \n -+\tif (check_for_incompatible(r))\n + void fsm_settings__set_hook(struct repository *r, const char *path)\n + {\n ++\tenum fsmonitor_reason reason = check_for_incompatible(r);\n ++\n ++\tif (reason != FSMONITOR_REASON_OK) {\n ++\t\tfsm_settings__set_incompatible(r, reason);\n +\t\treturn;\n ++\t}\n +\n ++\t/*\n ++\t * Caller requested hook explicitly, so avoid (possibly\n ++\t * recursive) config lookup.\n ++\t */\n + \tif (!r)\n + \t\tr = the_repository;\n +-\n +-\tlookup_fsmonitor_settings(r);\n ++\tif (!r->settings.fsmonitor)\n ++\t\tr->settings.fsmonitor = alloc_settings();\n + \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n ++\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n + }\n @@ fsmonitor-settings.c: void fsm_settings__set_disabled(struct repository *r)\n - \tlookup_fsmonitor_settings(r);\n + {\n + \tif (!r)\n + \t\tr = the_repository;\n +-\n +-\tlookup_fsmonitor_settings(r);\n ++\tif (!r->settings.fsmonitor)\n ++\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n +\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n ++\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n ++}\n ++\n ++void fsm_settings__set_incompatible(struct repository *r,\n ++\t\t\t\t enum fsmonitor_reason reason)\n ++{\n ++\tif (!r)\n ++\t\tr = the_repository;\n ++\tif (!r->settings.fsmonitor)\n ++\t\tr->settings.fsmonitor = alloc_settings();\n ++\n ++\tr->settings.fsmonitor->mode = FSMONITOR_MODE_INCOMPATIBLE;\n ++\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n +\n @@ fsmonitor-settings.c: void fsm_settings__set_disabled(struct repository *r)\n +{\n +\tif (!r)\n +\t\tr = the_repository;\n -+\n -+\tlookup_fsmonitor_settings(r);\n ++\tif (!r->settings.fsmonitor)\n ++\t\tlookup_fsmonitor_settings(r);\n +\n +\treturn r->settings.fsmonitor->reason;\n +}\n +\n -+int fsm_settings__error_if_incompatible(struct repository *r)\n ++char *fsm_settings__get_incompatible_msg(const struct repository *r,\n ++\t\t\t\t\t enum fsmonitor_reason reason)\n +{\n -+\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n ++\tstruct strbuf msg = STRBUF_INIT;\n +\n +\tswitch (reason) {\n ++\tcase FSMONITOR_REASON_UNTESTED:\n +\tcase FSMONITOR_REASON_OK:\n -+\t\treturn 0;\n ++\t\tgoto done;\n +\n +\tcase FSMONITOR_REASON_BARE:\n -+\t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n -+\t\t xgetcwd());\n -+\t\treturn 1;\n ++\t\tstrbuf_addf(&msg,\n ++\t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n ++\t\t\t xgetcwd());\n ++\t\tgoto done;\n +\t}\n +\n -+\tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n ++\tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n +\t reason);\n ++\n ++done:\n ++\treturn strbuf_detach(&msg, NULL);\n +}\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h\n + * Incompatibility reasons.\n + */\n +enum fsmonitor_reason {\n -+\tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n ++\tFSMONITOR_REASON_UNTESTED = 0,\n ++\tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n +\tFSMONITOR_REASON_BARE,\n +};\n +\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n -@@ fsmonitor-settings.h: void fsm_settings__set_disabled(struct repository *r);\n ++void fsm_settings__set_incompatible(struct repository *r,\n ++\t\t\t\t enum fsmonitor_reason reason);\n + \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n -+int fsm_settings__error_if_incompatible(struct repository *r);\n ++char *fsm_settings__get_incompatible_msg(const struct repository *r,\n ++\t\t\t\t\t enum fsmonitor_reason reason);\n +\n struct fsmonitor_settings;\n \n 5: e1e55550c10 ! 5: 2e225c3f4f2 fsmonitor-settings: stub in Win32-specific incompatibility checking\n @@ contrib/buildsystems/CMakeLists.txt: if(SUPPORTS_SIMPLE_IPC)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)\n - \t\treturn 1;\n +@@ fsmonitor-settings.c: static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n + \t\treturn FSMONITOR_REASON_BARE;\n \t}\n \n +#ifdef HAVE_FSMONITOR_OS_SETTINGS\n @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)\n +\t\tenum fsmonitor_reason reason;\n +\n +\t\treason = fsm_os__incompatible(r);\n -+\t\tif (reason != FSMONITOR_REASON_OK) {\n -+\t\t\tset_incompatible(r, reason);\n -+\t\t\treturn 1;\n -+\t\t}\n ++\t\tif (reason != FSMONITOR_REASON_OK)\n ++\t\t\treturn reason;\n +\t}\n +#endif\n +\n - \treturn 0;\n + \treturn FSMONITOR_REASON_OK;\n }\n \n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: int fsm_settings__error_if_incompatible(struct repository *r);\n +@@ fsmonitor-settings.h: char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \n struct fsmonitor_settings;\n \n 6: 2d68fc9a46a ! 6: e0d3bdf7556 fsmonitor-settings: VFS for Git virtual repos are incompatible\n @@ compat/fsmonitor/fsm-settings-win32.c\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n - \t\terror(_(\"bare repository '%s' is incompatible with fsmonitor\"),\n - \t\t xgetcwd());\n - \t\treturn 1;\n +@@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repository *r,\n + \t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n + \t\t\t xgetcwd());\n + \t\tgoto done;\n +\n +\tcase FSMONITOR_REASON_VFS4GIT:\n -+\t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n -+\t\t r->worktree);\n -+\t\treturn 1;\n ++\t\tstrbuf_addf(&msg,\n ++\t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n ++\t\t\t r->worktree);\n ++\t\tgoto done;\n \t}\n \n - \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n + \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: enum fsmonitor_mode {\n - enum fsmonitor_reason {\n - \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n +@@ fsmonitor-settings.h: enum fsmonitor_reason {\n + \tFSMONITOR_REASON_UNTESTED = 0,\n + \tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n +\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n 7: 94ae2e424f1 = 7: c50ed29a310 fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: b2ca6c1b201 ! 8: 1f5b772d42a fsmonitor-settings: remote repos on macOS are incompatible\n @@ compat/fsmonitor/fsm-settings-darwin.c\n }\n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n - \t\t xgetcwd());\n - \t\treturn 1;\n +@@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repository *r,\n + \t\t\t xgetcwd());\n + \t\tgoto done;\n \n +\tcase FSMONITOR_REASON_ERROR:\n -+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n -+\t\t r->worktree);\n -+\t\treturn 1;\n ++\t\tstrbuf_addf(&msg,\n ++\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n ++\t\t\t r->worktree);\n ++\t\tgoto done;\n +\n +\tcase FSMONITOR_REASON_REMOTE:\n -+\t\terror(_(\"remote repository '%s' is incompatible with fsmonitor\"),\n -+\t\t r->worktree);\n -+\t\treturn 1;\n ++\t\tstrbuf_addf(&msg,\n ++\t\t\t _(\"remote repository '%s' is incompatible with fsmonitor\"),\n ++\t\t\t r->worktree);\n ++\t\tgoto done;\n +\n \tcase FSMONITOR_REASON_VFS4GIT:\n - \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n - \t\t r->worktree);\n + \t\tstrbuf_addf(&msg,\n + \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \n ## fsmonitor-settings.h ##\n -@@ fsmonitor-settings.h: enum fsmonitor_mode {\n - enum fsmonitor_reason {\n - \tFSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */\n +@@ fsmonitor-settings.h: enum fsmonitor_reason {\n + \tFSMONITOR_REASON_UNTESTED = 0,\n + \tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n +\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n +\tFSMONITOR_REASON_REMOTE,\n 9: a3cc4b3b16d = 9: 495e54049b4 fsmonitor-settings: remote repos on Windows are incompatible\n 10: 8f1f4840751 ! 10: 4b52083698c fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n @@ compat/fsmonitor/fsm-settings-darwin.c: enum fsmonitor_reason fsm_os__incompatib\n \n \n ## fsmonitor-settings.c ##\n -@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)\n - \t\terror(_(\"virtual repository '%s' is incompatible with fsmonitor\"),\n - \t\t r->worktree);\n - \t\treturn 1;\n +@@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repository *r,\n + \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n + \t\t\t r->worktree);\n + \t\tgoto done;\n +\n +\tcase FSMONITOR_REASON_NOSOCKETS:\n -+\t\terror(_(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n -+\t\t r->worktree);\n -+\t\treturn 1;\n ++\t\tstrbuf_addf(&msg,\n ++\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n ++\t\t\t r->worktree);\n ++\t\tgoto done;\n \t}\n \n - \tBUG(\"Unhandled case in fsm_settings__error_if_incompatible: '%d'\",\n + \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n \n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_reason {\n 11: 8d48d9c5623 = 11: d4a4263d379 unpack-trees: initialize fsmonitor_has_run_once in o->result\n 12: 088c7b3334c = 12: f4feb00ec2b fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n 13: 00fab626663 = 13: dbb983fd9d0 fsmonitor--daemon: cd out of worktree root\n 14: 6552f51802b = 14: ae90b99ea9b fsmonitor--daemon: prepare for adding health thread\n 15: f2bf07cd739 = 15: b6c5800095f fsmonitor--daemon: rename listener thread related variables\n 16: 2a44f2eded1 = 16: 32fc6ba7437 fsmonitor--daemon: stub in health thread\n 17: 854fb5e3658 = 17: 77bc037481a fsm-health-win32: add polling framework to monitor daemon health\n 18: 3af1fe0d61d = 18: b06edd995ea fsm-health-win32: force shutdown daemon if worktree root moves\n 19: f1365cdd40c = 19: 1bd5f346248 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n 20: 15698d64edd = 20: 48af0813dec fsmonitor: optimize processing of directory events\n 21: 9d0da8fc22b = 21: a9b35e770f3 t7527: FSMonitor tests for directory moves\n 22: 040c00cfd6f = 22: 26308936af9 t/perf/p7527: add perf test for builtin FSMonitor\n 23: 5db241f7d2f ! 23: d0e25f6bac6 fsmonitor: never set CE_FSMONITOR_VALID on submodules\n @@ t/t7527-builtin-fsmonitor.sh: do\n +# submodule.\n +\n +create_super () {\n -+\tsuper=$1 &&\n -+\n -+\tgit init \"${super}\" &&\n -+\techo x >${super}/file_1 &&\n -+\techo y >${super}/file_2 &&\n -+\techo z >${super}/file_3 &&\n -+\tmkdir ${super}/dir_1 &&\n -+\techo a >${super}/dir_1/file_11 &&\n -+\techo b >${super}/dir_1/file_12 &&\n -+\tmkdir ${super}/dir_1/dir_2 &&\n -+\techo a >${super}/dir_1/dir_2/file_21 &&\n -+\techo b >${super}/dir_1/dir_2/file_22 &&\n -+\tgit -C ${super} add . &&\n -+\tgit -C ${super} commit -m \"initial ${super} commit\"\n ++\tsuper=\"$1\" &&\n ++\n ++\tgit init \"$super\" &&\n ++\techo x >\"$super/file_1\" &&\n ++\techo y >\"$super/file_2\" &&\n ++\techo z >\"$super/file_3\" &&\n ++\tmkdir \"$super/dir_1\" &&\n ++\techo a >\"$super/dir_1/file_11\" &&\n ++\techo b >\"$super/dir_1/file_12\" &&\n ++\tmkdir \"$super/dir_1/dir_2\" &&\n ++\techo a >\"$super/dir_1/dir_2/file_21\" &&\n ++\techo b >\"$super/dir_1/dir_2/file_22\" &&\n ++\tgit -C \"$super\" add . &&\n ++\tgit -C \"$super\" commit -m \"initial $super commit\"\n +}\n +\n +create_sub () {\n -+\tsub=$1 &&\n -+\n -+\tgit init \"${sub}\" &&\n -+\techo x >${sub}/file_x &&\n -+\techo y >${sub}/file_y &&\n -+\techo z >${sub}/file_z &&\n -+\tmkdir ${sub}/dir_x &&\n -+\techo a >${sub}/dir_x/file_a &&\n -+\techo b >${sub}/dir_x/file_b &&\n -+\tmkdir ${sub}/dir_x/dir_y &&\n -+\techo a >${sub}/dir_x/dir_y/file_a &&\n -+\techo b >${sub}/dir_x/dir_y/file_b &&\n -+\tgit -C ${sub} add . &&\n -+\tgit -C ${sub} commit -m \"initial ${sub} commit\"\n ++\tsub=\"$1\" &&\n ++\n ++\tgit init \"$sub\" &&\n ++\techo x >\"$sub/file_x\" &&\n ++\techo y >\"$sub/file_y\" &&\n ++\techo z >\"$sub/file_z\" &&\n ++\tmkdir \"$sub/dir_x\" &&\n ++\techo a >\"$sub/dir_x/file_a\" &&\n ++\techo b >\"$sub/dir_x/file_b\" &&\n ++\tmkdir \"$sub/dir_x/dir_y\" &&\n ++\techo a >\"$sub/dir_x/dir_y/file_a\" &&\n ++\techo b >\"$sub/dir_x/dir_y/file_b\" &&\n ++\tgit -C \"$sub\" add . &&\n ++\tgit -C \"$sub\" commit -m \"initial $sub commit\"\n +}\n +\n +my_match_and_clean () {\n @@ t/t7527-builtin-fsmonitor.sh: do\n +\t\t\t rm -rf super; \\\n +\t\t\t rm -rf sub\" &&\n +\n -+\tcreate_super \"super\" &&\n -+\tcreate_sub \"sub\" &&\n ++\tcreate_super super &&\n ++\tcreate_sub sub &&\n +\n +\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n +\tgit -C super commit -m \"add sub\" &&\n 24: 93de3707d26 = 24: 410dd2d2920 t7527: test FSMonitor on case insensitive+preserving file system\n 25: d890c2e2d97 = 25: cd7c55b0d38 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n 26: 7c606235557 = 26: 8278f32c4d8 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n 27: 9724c41d18d = 27: 4efb3a43838 t7527: test Unicode NFC/NFD handling on MacOS\n 28: b8325fb7c78 ! 28: df1b4f3a80f fsmonitor--daemon: allow --super-prefix argument\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n +\t\t\t rm -rf sub; \\\n +\t\t\t rm super-sub.trace\" &&\n +\n -+\tcreate_super \"super\" &&\n -+\tcreate_sub \"sub\" &&\n ++\tcreate_super super &&\n ++\tcreate_sub sub &&\n +\n +\t# Copy rather than submodule add so that we get a .git dir.\n +\tcp -R ./sub ./super/dir_1/dir_2/sub &&\n\n-- \ngitgitgadget\n"},{"id":"454293","messageId":"b6c5800095fcd6ac77ff9b355df683cd3ee4859e.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 15/28] fsmonitor--daemon: rename listener thread related variables","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:41Z","receivedAt":"2022-04-22T22:36:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRename platform-specific listener thread related variables\nand data types as we prepare to add another backend thread\ntype.\n\n[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`\n[] `state->backend_data` becomes `state->listen_data`\n[] `state->error_code` becomes `state->listen_error_code`\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 +++---\n compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------\n compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++-------------\n compat/fsmonitor/fsm-listen.h | 2 +-\n fsmonitor--daemon.h | 6 +++---\n 5 files changed, 36 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 90fa9d09efb..b2f578b239a 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1225,8 +1225,8 @@ cleanup:\n \n \tif (err)\n \t\treturn err;\n-\tif (state->error_code)\n-\t\treturn state->error_code;\n+\tif (state->listen_error_code)\n+\t\treturn state->listen_error_code;\n \treturn 0;\n }\n \n@@ -1241,7 +1241,7 @@ static int fsmonitor_run_daemon(void)\n \thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n-\tstate.error_code = 0;\n+\tstate.listen_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 14105f45c18..07113205a61 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -27,7 +27,7 @@\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tCFStringRef cfsr_worktree_path;\n \tCFStringRef cfsr_gitdir_path;\n@@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t const FSEventStreamEventId event_ids[])\n {\n \tstruct fsmonitor_daemon_state *state = ctx;\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tchar **paths = (char **)event_paths;\n \tstruct fsmonitor_batch *batch = NULL;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tNULL,\n \t\tNULL\n \t};\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \tconst void *dir_array[2];\n \n \tCALLOC_ARRAY(data, 1);\n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \n \tdata->cfsr_worktree_path = CFStringCreateWithCString(\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n@@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n \terror(_(\"Unable to create FSEventStream.\"));\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tif (data->stream) {\n \t\tif (data->stream_started)\n@@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n \t\tFSEventStreamRelease(data->stream);\n \t}\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \tdata->shutdown_style = SHUTDOWN_EVENT;\n \n \tCFRunLoopStop(data->rl);\n@@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tdata->rl = CFRunLoopGetCurrent();\n \n@@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \n \tswitch (data->shutdown_style) {\n \tcase FORCE_ERROR_STOP:\n-\t\tstate->error_code = -1;\n+\t\tstate->listen_error_code = -1;\n \t\t/* fall thru */\n \tcase FORCE_SHUTDOWN:\n \t\tipc_server_stop_async(state->ipc_server_data);\n@@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \treturn;\n \n force_error_stop_without_loop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \tipc_server_stop_async(state->ipc_server_data);\n \treturn;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex c43d92b9620..be2d93f47b2 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -54,7 +54,7 @@ struct one_watch\n \twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tstruct one_watch *watch_worktree;\n \tstruct one_watch *watch_gitdir;\n@@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname(\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+\tSetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);\n }\n \n static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n@@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch)\n \tfree(watch);\n }\n \n-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+static int start_rdcw_watch(struct fsm_listen_data *data,\n \t\t\t struct one_watch *watch)\n {\n \tDWORD dwNotifyFilter =\n@@ -516,7 +516,7 @@ static int process_1_worktree_event(\n */\n static int process_worktree_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_worktree;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -646,7 +646,7 @@ force_shutdown:\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_gitdir;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -704,11 +704,11 @@ skip_this_path:\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tDWORD dwWait;\n \tint result;\n \n-\tstate->error_code = 0;\n+\tstate->listen_error_code = 0;\n \n \tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n \t\tgoto force_error_stop;\n@@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \t}\n \n force_error_stop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \n force_shutdown:\n \t/*\n@@ -790,7 +790,7 @@ clean_shutdown:\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \treturn 0;\n \n failed:\n@@ -836,16 +836,16 @@ failed:\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tCloseHandle(data->hEventShutdown);\n \tdestroy_watch(data->watch_worktree);\n \tdestroy_watch(data->watch_gitdir);\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nindex f0539349baf..41650bf8972 100644\n--- a/compat/fsmonitor/fsm-listen.h\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n * do so if the listener thread receives a normal shutdown signal from\n * the IPC layer.)\n *\n- * It should set `state->error_code` to -1 if the daemon should exit\n+ * It should set `state->listen_error_code` to -1 if the daemon should exit\n * with an error.\n */\n void fsm_listen__loop(struct fsmonitor_daemon_state *state);\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 223c2131b58..2c6fa1a5d91 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n */\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {\n \tint cookie_seq;\n \tstruct hashmap cookies;\n \n-\tint error_code;\n-\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\tint listen_error_code;\n+\tstruct fsm_listen_data *listen_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"454294","messageId":"77bc037481a89d69a2c038e5ff0c6d3d5ac656f1.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 17/28] fsm-health-win32: add polling framework to monitor daemon health","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:43Z","receivedAt":"2022-04-22T22:36:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend the Windows version of the \"health\" thread to periodically\ninspect the system and shutdown if warranted.\n\nThis commit updates the thread's wait loop to use a timeout and\ndefines a (currently empty) table of functions to poll the system.\n\nA later commit will add functions to the table to actually\ninspect the system.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-\n 1 file changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 94b1d020f25..24fc612bf02 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -4,6 +4,24 @@\n #include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n+/*\n+ * Every minute wake up and test our health.\n+ */\n+#define WAIT_FREQ_MS (60 * 1000)\n+\n+/*\n+ * State machine states for each of the interval functions\n+ * used for polling our health.\n+ */\n+enum interval_fn_ctx {\n+\tCTX_INIT = 0,\n+\tCTX_TERM,\n+\tCTX_TIMER\n+};\n+\n+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx);\n+\n struct fsm_health_data\n {\n \tHANDLE hEventShutdown;\n@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n \tFREE_AND_NULL(state->health_data);\n }\n \n+/*\n+ * A table of the polling functions.\n+ */\n+static interval_fn *table[] = {\n+\tNULL, /* must be last */\n+};\n+\n+/*\n+ * Call all of the polling functions in the table.\n+ * Shortcut and return first error.\n+ *\n+ * Return 0 if all succeeded.\n+ */\n+static int call_all(struct fsmonitor_daemon_state *state,\n+\t\t enum interval_fn_ctx ctx)\n+{\n+\tint k;\n+\n+\tfor (k = 0; table[k]; k++) {\n+\t\tint r = table[k](state, ctx);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n+\tint r;\n+\n+\tr = call_all(state, CTX_INIT);\n+\tif (r < 0)\n+\t\tgoto force_error_stop;\n+\tif (r > 0)\n+\t\tgoto force_shutdown;\n \n \tfor (;;) {\n \t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n \t\t\t\t\t\t data->hHandles,\n-\t\t\t\t\t\t FALSE, INFINITE);\n+\t\t\t\t\t\t FALSE, WAIT_FREQ_MS);\n \n \t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n \t\t\tgoto clean_shutdown;\n \n+\t\tif (dwWait == WAIT_TIMEOUT) {\n+\t\t\tr = call_all(state, CTX_TIMER);\n+\t\t\tif (r < 0)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (r > 0)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\terror(_(\"health thread wait failed [GLE %ld]\"),\n \t\t GetLastError());\n \t\tgoto force_error_stop;\n@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)\n \n force_error_stop:\n \tstate->health_error_code = -1;\n+force_shutdown:\n \tipc_server_stop_async(state->ipc_server_data);\n clean_shutdown:\n+\tcall_all(state, CTX_TERM);\n \treturn;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"454295","messageId":"e0d3bdf7556f8315378c09f6d741a5f56c01c40c.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 06/28] fsmonitor-settings: VFS for Git virtual repos are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:32Z","receivedAt":"2022-04-22T22:36:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nVFS for Git virtual repositories are incompatible with FSMonitor.\n\nVFS for Git is a downstream fork of Git. It contains its own custom\nfile system watcher that is aware of the virtualization. If a working\ndirectory is being managed by VFS for Git, we should not try to watch\nit because we may get incomplete results.\n\nWe do not know anything about how VFS for Git works, but we do\nknow that VFS for Git working directories contain a well-defined\nconfig setting. If it is set, mark the working directory as\nincompatible.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 6 ++++++\n fsmonitor-settings.h | 1 +\n t/t7519-status-fsmonitor.sh | 9 +++++++++\n 4 files changed, 42 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 7fce32a3c5b..ee78bba38e3 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -3,7 +3,33 @@\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n \n+/*\n+ * VFS for Git is incompatible with FSMonitor.\n+ *\n+ * Granted, core Git does not know anything about VFS for Git and we\n+ * shouldn't make assumptions about a downstream feature, but users\n+ * can install both versions. And this can lead to incorrect results\n+ * from core Git commands. So, without bringing in any of the VFS for\n+ * Git code, do a simple config test for a published config setting.\n+ * (We do not look at the various *_TEST_* environment variables.)\n+ */\n+static enum fsmonitor_reason check_vfs4git(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n+\t\treturn FSMONITOR_REASON_VFS4GIT;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_vfs4git(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex f67db913f57..600ae165ab1 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -207,6 +207,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n \t\t\t xgetcwd());\n \t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_VFS4GIT:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 1694b7091a2..9e483c14513 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -17,6 +17,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n \tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 9a8e21c5608..156895f9437 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n \tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n '\n \n+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone actual\" &&\n+\tgit init fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"virtual repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"454296","messageId":"72b94acd5fe87eaa1dba9bfb66a7a738a0b46a91.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 04/28] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:30Z","receivedAt":"2022-04-22T22:36:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a worktree, so there is nothing for the\ndaemon watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 18 +++++\n builtin/update-index.c | 16 +++++\n fsmonitor-settings.c | 133 ++++++++++++++++++++++++++++++------\n fsmonitor-settings.h | 16 +++++\n t/t7519-status-fsmonitor.sh | 23 +++++++\n 5 files changed, 186 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 46be55a4618..66b78a0353f 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1423,6 +1423,7 @@ static int try_to_start_background_daemon(void)\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tenum fsmonitor_reason reason;\n \tint detach_console = 0;\n \n \tstruct option options[] = {\n@@ -1449,6 +1450,23 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\t/*\n+\t * If the repo is fsmonitor-compatible, explicitly set IPC-mode\n+\t * (without bothering to load the `core.fsmonitor` config settings).\n+\t *\n+\t * If the repo is not compatible, the repo-settings will be set to\n+\t * incompatible rather than IPC, so we can use one of the __get\n+\t * routines to detect the discrepancy.\n+\t */\n+\tfsm_settings__set_ipc(the_repository);\n+\n+\treason = fsm_settings__get_reason(the_repository);\n+\tif (reason > FSMONITOR_REASON_OK)\n+\t\tdie(\"%s\",\n+\t\t fsm_settings__get_incompatible_msg(the_repository,\n+\t\t\t\t\t\t reason));\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 876112abb21..01ed4c4976b 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1237,6 +1237,22 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n+\n+\t\t/*\n+\t\t * The user wants to turn on FSMonitor using the command\n+\t\t * line argument. (We don't know (or care) whether that\n+\t\t * is the IPC or HOOK version.)\n+\t\t *\n+\t\t * Use one of the __get routines to force load the FSMonitor\n+\t\t * config settings into the repo-settings. That will detect\n+\t\t * whether the file system is compatible so that we can stop\n+\t\t * here with a nice error message.\n+\t\t */\n+\t\tif (reason > FSMONITOR_REASON_OK)\n+\t\t\tdie(\"%s\",\n+\t\t\t fsm_settings__get_incompatible_msg(r, reason));\n+\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 757d230d538..7d3177d441a 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -9,23 +9,42 @@\n */\n struct fsmonitor_settings {\n \tenum fsmonitor_mode mode;\n+\tenum fsmonitor_reason reason;\n \tchar *hook_path;\n };\n \n-static void lookup_fsmonitor_settings(struct repository *r)\n+static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n+{\n+\tif (!r->worktree) {\n+\t\t/*\n+\t\t * Bare repositories don't have a working directory and\n+\t\t * therefore have nothing to watch.\n+\t\t */\n+\t\treturn FSMONITOR_REASON_BARE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n+static struct fsmonitor_settings *alloc_settings(void)\n {\n \tstruct fsmonitor_settings *s;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\ts->reason = FSMONITOR_REASON_UNTESTED;\n+\n+\treturn s;\n+}\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n \tconst char *const_str;\n \tint bool_value;\n \n \tif (r->settings.fsmonitor)\n \t\treturn;\n \n-\tCALLOC_ARRAY(s, 1);\n-\ts->mode = FSMONITOR_MODE_DISABLED;\n-\n-\tr->settings.fsmonitor = s;\n-\n \t/*\n \t * Overload the existing \"core.fsmonitor\" config setting (which\n \t * has historically been either unset or a hook pathname) to\n@@ -38,6 +57,8 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \tcase 0: /* config value was set to */\n \t\tif (bool_value)\n \t\t\tfsm_settings__set_ipc(r);\n+\t\telse\n+\t\t\tfsm_settings__set_disabled(r);\n \t\treturn;\n \n \tcase 1: /* config value was unset */\n@@ -53,18 +74,18 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \t\treturn;\n \t}\n \n-\tif (!const_str || !*const_str)\n-\t\treturn;\n-\n-\tfsm_settings__set_hook(r, const_str);\n+\tif (const_str && *const_str)\n+\t\tfsm_settings__set_hook(r, const_str);\n+\telse\n+\t\tfsm_settings__set_disabled(r);\n }\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n \n \treturn r->settings.fsmonitor->mode;\n }\n@@ -73,31 +94,55 @@ const char *fsm_settings__get_hook_path(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n \n \treturn r->settings.fsmonitor->hook_path;\n }\n \n void fsm_settings__set_ipc(struct repository *r)\n {\n+\tenum fsmonitor_reason reason = check_for_incompatible(r);\n+\n+\tif (reason != FSMONITOR_REASON_OK) {\n+\t\tfsm_settings__set_incompatible(r, reason);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Caller requested IPC explicitly, so avoid (possibly\n+\t * recursive) config lookup.\n+\t */\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n \n void fsm_settings__set_hook(struct repository *r, const char *path)\n {\n+\tenum fsmonitor_reason reason = check_for_incompatible(r);\n+\n+\tif (reason != FSMONITOR_REASON_OK) {\n+\t\tfsm_settings__set_incompatible(r, reason);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Caller requested hook explicitly, so avoid (possibly\n+\t * recursive) config lookup.\n+\t */\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n }\n@@ -106,9 +151,57 @@ void fsm_settings__set_disabled(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\n+\n+void fsm_settings__set_incompatible(struct repository *r,\n+\t\t\t\t enum fsmonitor_reason reason)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n+\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\n+\n+char *fsm_settings__get_incompatible_msg(const struct repository *r,\n+\t\t\t\t\t enum fsmonitor_reason reason)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tswitch (reason) {\n+\tcase FSMONITOR_REASON_UNTESTED:\n+\tcase FSMONITOR_REASON_OK:\n+\t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_BARE:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t xgetcwd());\n+\t\tgoto done;\n+\t}\n+\n+\tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n+\t reason);\n+\n+done:\n+\treturn strbuf_detach(&msg, NULL);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a4c5d7b4889..4c7592896e4 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -4,18 +4,34 @@\n struct repository;\n \n enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n \tFSMONITOR_MODE_DISABLED = 0,\n \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n };\n \n+/*\n+ * Incompatibility reasons.\n+ */\n+enum fsmonitor_reason {\n+\tFSMONITOR_REASON_UNTESTED = 0,\n+\tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n+\tFSMONITOR_REASON_BARE,\n+};\n+\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n+void fsm_settings__set_incompatible(struct repository *r,\n+\t\t\t\t enum fsmonitor_reason reason);\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n+char *fsm_settings__get_incompatible_msg(const struct repository *r,\n+\t\t\t\t\t enum fsmonitor_reason reason);\n+\n struct fsmonitor_settings;\n \n #endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex a6308acf006..9a8e21c5608 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '\n \ttest $ret -ne 1\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n+\tgit init --bare bare-clone &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n+\tgit init --bare bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"454297","messageId":"ae90b99ea9b821cf9dbf307a56b3000578cb3391.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 14/28] fsmonitor--daemon: prepare for adding health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:40Z","receivedAt":"2022-04-22T22:36:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor daemon thread startup to make it easier to start\na third thread class to monitor the health of the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex db297649daf..90fa9d09efb 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1174,6 +1174,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint listener_started = 0;\n+\tint err = 0;\n \n \t/*\n \t * Start the IPC thread pool before the we've started the file\n@@ -1194,15 +1196,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \tif (pthread_create(&state->listener_thread, NULL,\n \t\t\t fsm_listen__thread_proc, state) < 0) {\n \t\tipc_server_stop_async(state->ipc_server_data);\n-\t\tipc_server_await(state->ipc_server_data);\n-\n-\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t\terr = error(_(\"could not start fsmonitor listener thread\"));\n+\t\tgoto cleanup;\n \t}\n+\tlistener_started = 1;\n \n \t/*\n \t * The daemon is now fully functional in background threads.\n+\t * Our primary thread should now just wait while the threads\n+\t * do all the work.\n+\t */\n+cleanup:\n+\t/*\n \t * Wait for the IPC thread pool to shutdown (whether by client\n-\t * request or from filesystem activity).\n+\t * request, from filesystem activity, or an error).\n \t */\n \tipc_server_await(state->ipc_server_data);\n \n@@ -1211,10 +1218,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * event from the IPC thread pool, but it doesn't hurt to tell\n \t * it again. And wait for it to shutdown.\n \t */\n-\tfsm_listen__stop_async(state);\n-\tpthread_join(state->listener_thread, NULL);\n+\tif (listener_started) {\n+\t\tfsm_listen__stop_async(state);\n+\t\tpthread_join(state->listener_thread, NULL);\n+\t}\n \n-\treturn state->error_code;\n+\tif (err)\n+\t\treturn err;\n+\tif (state->error_code)\n+\t\treturn state->error_code;\n+\treturn 0;\n }\n \n static int fsmonitor_run_daemon(void)\n-- \ngitgitgadget\n\n"},{"id":"454298","messageId":"df1b4f3a80f812a284cff4caeb0ab2ca75e38c2f.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 28/28] fsmonitor--daemon: allow --super-prefix argument","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:54Z","receivedAt":"2022-04-22T22:36:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a test in t7527 to verify that we get a stray warning from\n`git fsmonitor--daemon start` when indirectly called from\n`git submodule absorbgitdirs`.\n\nUpdate `git fsmonitor--daemon` to take (and ignore) the `--super-prefix`\nargument to suppress the warning.\n\nWhen we have:\n\n1. a submodule with a `sub/.git/` directory (rather than a `sub/.git`\nfile).\n\n2. `core.fsmonitor` is turned on in the submodule, but the daemon is\nnot yet started in the submodule.\n\n3. and someone does a `git submodule absorbgitdirs` in the super.\n\nGit will recursively invoke `git submodule--helper absorb-git-dirs`\nin the submodule. This will read the index and may attempt to start\nthe fsmonitor--daemon with the `--super-prefix` argument.\n\n`git fsmonitor--daemon start` does not accept the `--super-prefix`\nargument and causes a warning to be issued.\n\nThis does not cause a problem because the `refresh_index()` code\nassumes a trivial response if the daemon does not start.\n\nThe net-net is a harmelss, but stray warning. Lets eliminate the\nwarning.\n\nSigned-off-by: Jeff Hostetler \n---\n git.c | 2 +-\n t/t7527-builtin-fsmonitor.sh | 50 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 51 insertions(+), 1 deletion(-)\n\ndiff --git a/git.c b/git.c\nindex 3d8e48cf555..e6fdac1f8e3 100644\n--- a/git.c\n+++ b/git.c\n@@ -537,7 +537,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n-\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, SUPPORT_SUPER_PREFIX | RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex ab29cca535d..f6e1d2da6d6 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,6 +832,56 @@ test_expect_success \"Submodule always visited\" '\n \tmy_match_and_clean\n '\n \n+# If a submodule has a `sub/.git/` directory (rather than a file\n+# pointing to the super's `.git/modules/sub`) and `core.fsmonitor`\n+# turned on in the submodule and the daemon is not yet started in\n+# the submodule, and someone does a `git submodule absorbgitdirs`\n+# in the super, Git will recursively invoke `git submodule--helper`\n+# to do the work and this may try to read the index. This will\n+# try to start the daemon in the submodule *and* pass (either\n+# directly or via inheritance) the `--super-prefix` arg to the\n+# `git fsmonitor--daemon start` command inside the submodule.\n+# This causes a warning because fsmonitor--daemon does take that\n+# global arg (see the table in git.c)\n+#\n+# This causes a warning when trying to start the daemon that is\n+# somewhat confusing. It does not seem to hurt anything because\n+# the fsmonitor code maps the query failure into a trivial response\n+# and does the work anyway.\n+#\n+# It would be nice to silence the warning, however.\n+\n+have_t2_error_event () {\n+\tlog=$1\n+\tmsg=\"fsmonitor--daemon doesnQt support --super-prefix\" &&\n+\n+\ttr '\\047' Q <$1 | grep -e \"$msg\"\n+}\n+\n+test_expect_success \"Stray Submodule super-prefix warning\" '\n+\ttest_when_finished \"rm -rf super; \\\n+\t\t\t rm -rf sub; \\\n+\t\t\t rm super-sub.trace\" &&\n+\n+\tcreate_super super &&\n+\tcreate_sub sub &&\n+\n+\t# Copy rather than submodule add so that we get a .git dir.\n+\tcp -R ./sub ./super/dir_1/dir_2/sub &&\n+\n+\tgit -C super/dir_1/dir_2/sub config core.fsmonitor true &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\ttest_path_is_dir super/dir_1/dir_2/sub/.git &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/super-sub.trace\" \\\n+\t\tgit -C super submodule absorbgitdirs &&\n+\n+\t! have_t2_error_event super-sub.trace\n+'\n+\n # On a case-insensitive file system, confirm that the daemon\n # notices when the .git directory is moved/renamed/deleted\n # regardless of how it is spelled in the the FS event.\n-- \ngitgitgadget\n"},{"id":"454299","messageId":"4efb3a438389249b3d9f9b388d7f04d74aaa2f92.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 27/28] t7527: test Unicode NFC/NFD handling on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:53Z","receivedAt":"2022-04-22T22:36:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that the daemon reports events using the on-disk\nspelling for Unicode NFC/NFD characters. On APFS we still\nhave Unicode aliasing, so we cannot create two files that\nonly differ by NFC/NFD, but the on-disk format preserves\nthe spelling used to create the file. On HFS+ we also\nhave aliasing, but the path is always stored on disk in\nNFD.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 55 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1cdbacdbfb0..ab29cca535d 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -868,4 +868,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n+# The variable \"unicode_debug\" is defined in the following library\n+# script to dump information about how the (OS, FS) handles Unicode\n+# composition. Uncomment the following line if you want to enable it.\n+#\n+# unicode_debug=true\n+\n+. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n+\n+# See if the OS or filesystem does NFC/NFD aliasing/munging.\n+#\n+# The daemon should err on the side of caution and send BOTH the\n+# NFC and NFD forms. It does not know the original spelling of\n+# the pathname (how the user thinks it should be spelled), so\n+# emit both and let the client decide (when necessary). This is\n+# similar to \"core.precomposeUnicode\".\n+#\n+test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n+\n+\tgit init test_unicode &&\n+\n+\tstart_daemon -C test_unicode --tf \"$PWD/unicode.trace\" &&\n+\n+\t# Create a directory using an NFC spelling.\n+\t#\n+\tmkdir test_unicode/nfc &&\n+\tmkdir test_unicode/nfc/c_${utf8_nfc} &&\n+\n+\t# Create a directory using an NFD spelling.\n+\t#\n+\tmkdir test_unicode/nfd &&\n+\tmkdir test_unicode/nfd/d_${utf8_nfd} &&\n+\n+\tgit -C test_unicode fsmonitor--daemon stop &&\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\t# We should have seen NFC event from OS.\n+\t\t# We should not have synthesized an NFD event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace &&\n+\t\tegrep -v \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace\n+\telse\n+\t\t# We should have seen NFD event from OS.\n+\t\t# We should have synthesized an NFC event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace &&\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace\n+\tfi &&\n+\n+\t# We assume UNICODE_NFD_PRESERVED.\n+\t# We should have seen explicit NFD from OS.\n+\t# We should have synthesized an NFC event.\n+\tegrep \"^event: nfd/d_${utf8_nfd}/?$\" ./unicode.trace &&\n+\tegrep \"^event: nfd/d_${utf8_nfc}/?$\" ./unicode.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"454300","messageId":"8278f32c4d894d4930b9f1f70f3aa01679e2011e.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 26/28] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:52Z","receivedAt":"2022-04-22T22:36:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a set of prereqs to help understand how file names\nare handled by the filesystem when they contain NFC and NFD\nUnicode characters.\n\nSigned-off-by: Jeff Hostetler \n---\n t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 167 insertions(+)\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n\ndiff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\nnew file mode 100755\nindex 00000000000..cf9c26d1e22\n--- /dev/null\n+++ b/t/lib-unicode-nfc-nfd.sh\n@@ -0,0 +1,167 @@\n+# Help detect how Unicode NFC and NFD are handled on the filesystem.\n+\n+# A simple character that has a NFD form.\n+#\n+# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n+# UTF8(NFC): \\xc3 \\xa9\n+#\n+# NFD: U+0065 LATIN SMALL LETTER E\n+# U+0301 COMBINING ACUTE ACCENT\n+# UTF8(NFD): \\x65 + \\xcc \\x81\n+#\n+utf8_nfc=$(printf \"\\xc3\\xa9\")\n+utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n+\n+# Is the OS or the filesystem \"Unicode composition sensitive\"?\n+#\n+# That is, does the OS or the filesystem allow files to exist with\n+# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n+# tell us that the NFC and NFD forms are equivalent.\n+#\n+# This is or may be independent of what type of filesystem we have,\n+# since it might be handled by the OS at a layer above the FS.\n+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n+# collision, for example.\n+#\n+# This does not tell us how the Unicode pathname will be spelled\n+# on disk, but rather only that the two spelling \"collide\". We\n+# will examine the actual on disk spelling in a later prereq.\n+#\n+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${utf8_nfc} &&\n+\tmkdir trial_${utf8_nfd}\n+'\n+\n+# Is the spelling of an NFC pathname preserved on disk?\n+#\n+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n+# and on APFS, NFC paths are preserved. As we have established\n+# above, this is independent of \"composition sensitivity\".\n+#\n+# 0000000 63 5f c3 a9\n+#\n+# (/usr/bin/od output contains different amount of whitespace\n+# on different platforms, so we need the wildcards here.)\n+#\n+test_lazy_prereq UNICODE_NFC_PRESERVED '\n+\tmkdir c_${utf8_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *c3 *a9\"\n+'\n+\n+# Is the spelling of an NFD pathname preserved on disk?\n+#\n+# 0000000 64 5f 65 cc 81\n+#\n+test_lazy_prereq UNICODE_NFD_PRESERVED '\n+\tmkdir d_${utf8_nfd} &&\n+\tls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n+'\n+\tmkdir c_${utf8_nfc} &&\n+\tmkdir d_${utf8_nfd} &&\n+\n+# The following _DOUBLE_ forms are more for my curiosity,\n+# but there may be quirks lurking when there are multiple\n+# combining characters in non-canonical order.\n+\n+# Unicode also allows multiple combining characters\n+# that can be decomposed in pieces.\n+#\n+# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n+# UTF8(NFC): \\xe1 \\xbd \\xa7\n+#\n+# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n+# U+0342 COMBINING GREEK PERISPOMENI\n+# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n+#\n+# But U+1f61 decomposes into\n+# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n+# U+0314 COMBINING REVERSED COMMA ABOVE\n+# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n+#\n+# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n+#\n+# Note that I've used the canonical ordering of the\n+# combinining characters. It is also possible to\n+# swap them. My testing shows that that non-standard\n+# ordering also causes a collision in mkdir. However,\n+# the resulting names don't draw correctly on the\n+# terminal (implying that the on-disk format also has\n+# them out of order).\n+#\n+greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n+greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n+greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n+\n+# See if a double decomposition also collides.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${greek_nfc} &&\n+\tmkdir trial_${greek_nfd2}\n+'\n+\n+# See if the NFC spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n+\tmkdir c_${greek_nfc} &&\n+\tls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n+'\n+\n+# See if the NFD spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n+\tmkdir d_${greek_nfd2} &&\n+\tls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n+'\n+\n+# The following is for debugging. I found it useful when\n+# trying to understand the various (OS, FS) quirks WRT\n+# Unicode and how composition/decomposition is handled.\n+# For example, when trying to understand how (macOS, APFS)\n+# and (macOS, HFS) and (macOS, FAT32) compare.\n+#\n+# It is rather noisy, so it is disabled by default.\n+#\n+if test \"$unicode_debug\" = \"true\"\n+then\n+\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\techo NFC maintains original spelling.\n+\telse\n+\t\techo NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFD_PRESERVED\n+\tthen\n+\t\techo NFD maintains original spelling.\n+\telse\n+\t\techo NFD is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n+\tthen\n+\t\techo Double NFC maintains original spelling.\n+\telse\n+\t\techo Double NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n+\tthen\n+\t\techo Double NFD maintains original spelling.\n+\telse\n+\t\techo Double NFD is modified.\n+\tfi\n+fi\n-- \ngitgitgadget\n\n"},{"id":"454301","messageId":"cd7c55b0d389932535ab1a6843bfba0a0440cda5.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 25/28] fsmonitor: on macOS also emit NFC spelling for NFD pathname","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:51Z","receivedAt":"2022-04-22T22:36:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nEmit NFC or NFC and NFD spellings of pathnames on macOS.\n\nMacOS is Unicode composition insensitive, so NFC and NFD spellings are\ntreated as aliases and collide. While the spelling of pathnames in\nfilesystem events depends upon the underlying filesystem, such as\nAPFS, HFS+ or FAT32, the OS enforces such collisions regardless of\nfilesystem.\n\nTeach the daemon to always report the NFC spelling and to report\nthe NFD spelling when stored in that format on the disk.\n\nThis is slightly more general than \"core.precomposeUnicode\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 83d38e8ac6c..823cf63999e 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n \treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n }\n \n+/*\n+ * On MacOS we have to adjust for Unicode composition insensitivity\n+ * (where NFC and NFD spellings are not respected). The different\n+ * spellings are essentially aliases regardless of how the path is\n+ * actually stored on the disk.\n+ *\n+ * This is related to \"core.precomposeUnicode\" (which wants to try\n+ * to hide NFD completely and treat everything as NFC). Here, we\n+ * don't know what the value the client has (or will have) for this\n+ * config setting when they make a query, so assume the worst and\n+ * emit both when the OS gives us an NFD path.\n+ */\n+static void my_add_path(struct fsmonitor_batch *batch, const char *path)\n+{\n+\tchar *composed;\n+\n+\t/* add the NFC or NFD path as received from the OS */\n+\tfsmonitor_batch__add_path(batch, path);\n+\n+\t/* if NFD, also add the corresponding NFC spelling */\n+\tcomposed = (char *)precompose_string_if_needed(path);\n+\tif (!composed || composed == path)\n+\t\treturn;\n+\n+\tfsmonitor_batch__add_path(batch, composed);\n+\tfree(composed);\n+}\n+\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t\tmy_add_path(batch, rel);\n \t\t\t}\n \n \t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n@@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t\tmy_add_path(batch, tmp.buf);\n \t\t\t}\n \n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"454302","messageId":"410dd2d2920eb6086378119765737dda7f3c7c29.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 24/28] t7527: test FSMonitor on case insensitive+preserving file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:50Z","receivedAt":"2022-04-22T22:36:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTest that FS events from the OS are received using the preserved,\non-disk spelling of files/directories rather than spelling used\nto make the change.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 4c49ae5a684..1cdbacdbfb0 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,4 +832,40 @@ test_expect_success \"Submodule always visited\" '\n \tmy_match_and_clean\n '\n \n+# On a case-insensitive file system, confirm that the daemon\n+# notices when the .git directory is moved/renamed/deleted\n+# regardless of how it is spelled in the the FS event.\n+# That is, does the FS event receive the spelling of the\n+# operation or does it receive the spelling preserved with\n+# the file/directory.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n+#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n+\n+\tgit init test_insensitive &&\n+\n+\tstart_daemon -C test_insensitive --tf \"$PWD/insensitive.trace\" &&\n+\n+\tmkdir -p test_insensitive/abc/def &&\n+\techo xyz >test_insensitive/ABC/DEF/xyz &&\n+\n+\ttest_path_is_dir test_insensitive/.git &&\n+\ttest_path_is_dir test_insensitive/.GIT &&\n+\n+\t# Rename .git using an alternate spelling to verify that that\n+\t# daemon detects it and automatically shuts down.\n+\tmv test_insensitive/.GIT test_insensitive/.FOO &&\n+\tsleep 1 &&\n+\tmv test_insensitive/.FOO test_insensitive/.git &&\n+\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\t# Verify that events were reported using on-disk spellings of the\n+\t# directories and files that we touched. We may or may not get a\n+\t# trailing slash on modified directories.\n+\t#\n+\tegrep \"^event: abc/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"454303","messageId":"d0e25f6bac663e9ae4d63322f102378dd2ecba84.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 23/28] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:49Z","receivedAt":"2022-04-22T22:36:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNever set CE_FSMONITOR_VALID on the cache-entry of submodule\ndirectories.\n\nDuring a client command like 'git status', we may need to recurse\ninto each submodule to compute a status summary for the submodule.\nSince the purpose of the ce_flag is to let Git avoid scanning a\ncache-entry, setting the flag causes the recursive call to be\navoided and we report incorrect (no status) for the submodule.\n\nWe created an OS watch on the root directory of our working\ndirectory and we receive events for everything in the cone\nunder it. When submodules are present inside our working\ndirectory, we receive events for both our repo (the super) and\nany subs within it. Since our index doesn't have any information\nfor items within the submodules, we can't use those events.\n\nWe could try to truncate the paths of those events back to the\nsubmodule boundary and mark the GITLINK as dirty, but that\nfeels expensive since we would have to prefix compare every FS\nevent that we receive against a list of submodule roots. And\nit still wouldn't be sufficient to correctly report status on\nthe submodule, since we don't have any space in the cache-entry\nto cache the submodule's status (the 'SCMU' bits in porcelain\nV2 speak). That is, the CE_FSMONITOR_VALID bit just says that\nwe don't need to scan/inspect it because we already know the\nanswer -- it doesn't say that the item is clean -- and we\ndon't have space in the cache-entry to store those answers.\nSo we should always do the recursive scan.\n\nTherefore, we should never set the flag on GITLINK cache-entries.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 2 +\n fsmonitor.h | 11 ++++\n t/t7527-builtin-fsmonitor.sh | 111 +++++++++++++++++++++++++++++++++++\n 3 files changed, 124 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex e1229c289cf..57d6a483bee 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tif (fsmonitor_enabled) {\n \t\t\t/* Mark all entries valid */\n \t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\t\tif (S_ISGITLINK(istate->cache[i]->ce_mode))\n+\t\t\t\t\tcontinue;\n \t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n \t\t\t}\n \ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 3f41f653691..edf7ce5203b 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n * called any time the cache entry has been updated to reflect the\n * current state of the file on disk.\n+ *\n+ * However, never mark submodules as valid. When commands like \"git\n+ * status\" run they might need to recurse into the submodule (using a\n+ * child process) to get a summary of the submodule state. We don't\n+ * have (and don't want to create) the facility to translate every\n+ * FS event that we receive and that happens to be deep inside of a\n+ * submodule back to the submodule root, so we cannot correctly keep\n+ * track of this bit on the gitlink directory. Therefore, we never\n+ * set it on submodules.\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n \n \tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n \t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tif (S_ISGITLINK(ce->ce_mode))\n+\t\t\treturn;\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex d0e681d008f..4c49ae5a684 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -721,4 +721,115 @@ do\n \t'\n done\n \n+# Test fsmonitor interaction with submodules.\n+#\n+# If we start the daemon in the super, it will see FS events for\n+# everything in the working directory cone and this includes any\n+# files/directories contained *within* the submodules.\n+#\n+# A `git status` at top level will get events for items within the\n+# submodule and ignore them, since they aren't named in the index\n+# of the super repo. This makes the fsmonitor response a little\n+# noisy, but it doesn't alter the correctness of the state of the\n+# super-proper.\n+#\n+# When we have submodules, `git status` normally does a recursive\n+# status on each of the submodules and adds a summary row for any\n+# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n+#\n+# It is therefore important that the top level status not be tricked\n+# by the FSMonitor response to skip those recursive calls. That is,\n+# even if FSMonitor says that the mtime of the submodule directory\n+# hasn't changed and it could be implicitly marked valid, we must\n+# not take that shortcut. We need to force the recusion into the\n+# submodule so that we get a summary of the status *within* the\n+# submodule.\n+\n+create_super () {\n+\tsuper=\"$1\" &&\n+\n+\tgit init \"$super\" &&\n+\techo x >\"$super/file_1\" &&\n+\techo y >\"$super/file_2\" &&\n+\techo z >\"$super/file_3\" &&\n+\tmkdir \"$super/dir_1\" &&\n+\techo a >\"$super/dir_1/file_11\" &&\n+\techo b >\"$super/dir_1/file_12\" &&\n+\tmkdir \"$super/dir_1/dir_2\" &&\n+\techo a >\"$super/dir_1/dir_2/file_21\" &&\n+\techo b >\"$super/dir_1/dir_2/file_22\" &&\n+\tgit -C \"$super\" add . &&\n+\tgit -C \"$super\" commit -m \"initial $super commit\"\n+}\n+\n+create_sub () {\n+\tsub=\"$1\" &&\n+\n+\tgit init \"$sub\" &&\n+\techo x >\"$sub/file_x\" &&\n+\techo y >\"$sub/file_y\" &&\n+\techo z >\"$sub/file_z\" &&\n+\tmkdir \"$sub/dir_x\" &&\n+\techo a >\"$sub/dir_x/file_a\" &&\n+\techo b >\"$sub/dir_x/file_b\" &&\n+\tmkdir \"$sub/dir_x/dir_y\" &&\n+\techo a >\"$sub/dir_x/dir_y/file_a\" &&\n+\techo b >\"$sub/dir_x/dir_y/file_b\" &&\n+\tgit -C \"$sub\" add . &&\n+\tgit -C \"$sub\" commit -m \"initial $sub commit\"\n+}\n+\n+my_match_and_clean () {\n+\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n+\tgit -C super --no-optional-locks -c core.fsmonitor=false \\\n+\t\tstatus --porcelain=v2 >actual.without &&\n+\ttest_cmp actual.with actual.without &&\n+\n+\tgit -C super/dir_1/dir_2/sub reset --hard &&\n+\tgit -C super/dir_1/dir_2/sub clean -d -f\n+}\n+\n+test_expect_success \"Submodule always visited\" '\n+\ttest_when_finished \"git -C super fsmonitor--daemon stop; \\\n+\t\t\t rm -rf super; \\\n+\t\t\t rm -rf sub\" &&\n+\n+\tcreate_super super &&\n+\tcreate_sub sub &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\tstart_daemon -C super &&\n+\tgit -C super config core.fsmonitor true &&\n+\tgit -C super update-index --fsmonitor &&\n+\tgit -C super status &&\n+\n+\t# Now run pairs of commands w/ and w/o FSMonitor while we make\n+\t# some dirt in the submodule and confirm matching output.\n+\n+\t# Completely clean status.\n+\tmy_match_and_clean &&\n+\n+\t# .M S..U\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M SC..\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tgit -C super/dir_1/dir_2/sub commit -m \"SC..\" &&\n+\tmy_match_and_clean\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"454304","messageId":"26308936af90e2e01180572fd25ac2239a3d20da.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 22/28] t/perf/p7527: add perf test for builtin FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:48Z","receivedAt":"2022-04-22T22:36:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++\n 1 file changed, 257 insertions(+)\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\ndiff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..9338b9ea008\n--- /dev/null\n+++ b/t/perf/p7527-builtin-fsmonitor.sh\n@@ -0,0 +1,257 @@\n+#!/bin/sh\n+\n+test_description=\"Perf test for the builtin FSMonitor\"\n+\n+. ./perf-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+# Lie to perf-lib and ask for a new empty repo and avoid\n+# the complaints about GIT_PERF_REPO not being big enough\n+# the perf hit when GIT_PERF_LARGE_REPO is copied into\n+# the trash directory.\n+#\n+# NEEDSWORK: It would be nice if perf-lib had an option to\n+# \"borrow\" an existing large repo (especially for gigantic\n+# monorepos) and use it in-place. For now, fake it here.\n+#\n+test_perf_fresh_repo\n+\n+\n+# Use a generated synthetic monorepo. If it doesn't exist, we will\n+# generate it. If it does exist, we will put it in a known state\n+# before we start our timings.\n+#\n+PARAM_D=5\n+PARAM_W=10\n+PARAM_F=9\n+\n+PARAMS=\"$PARAM_D\".\"$PARAM_W\".\"$PARAM_F\"\n+\n+BALLAST_BR=p0006-ballast\n+export BALLAST_BR\n+\n+TMP_BR=tmp_br\n+export TMP_BR\n+\n+REPO=../repos/gen-many-files-\"$PARAMS\".git\n+export REPO\n+\n+if ! test -d $REPO\n+then\n+\t(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)\n+fi\n+\n+\n+enable_uc () {\n+\tgit -C $REPO config core.untrackedcache true\n+\tgit -C $REPO update-index --untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+disable_uc () {\n+\tgit -C $REPO config core.untrackedcache false\n+\tgit -C $REPO update-index --no-untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+start_fsm () {\n+\tgit -C $REPO fsmonitor--daemon start\n+\tgit -C $REPO fsmonitor--daemon status\n+\tgit -C $REPO config core.fsmonitor true\n+\tgit -C $REPO update-index --fsmonitor\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+stop_fsm () {\n+\tgit -C $REPO config --unset core.fsmonitor\n+\tgit -C $REPO update-index --no-fsmonitor\n+\ttest_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+\n+# Ensure that FSMonitor is turned off on the borrowed repo.\n+#\n+test_expect_success \"Setup borrowed repo (fsm+uc)\" \"\n+\tstop_fsm &&\n+\tdisable_uc\n+\"\n+\n+# Also ensure that it starts in a known state.\n+#\n+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time\n+# the ballast checkout, since only the first invocation does any work and the\n+# subsequent ones just print \"already on branch\" and quit, so the reported\n+# time is not useful.\n+#\n+# Create a temp branch and do all work relative to it so that we don't\n+# accidentially alter the real ballast branch.\n+#\n+test_expect_success \"Setup borrowed repo (temp ballast branch)\" \"\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO reset --hard &&\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tgit -C $REPO branch $TMP_BR $BALLAST_BR &&\n+\tgit -C $REPO checkout $TMP_BR\n+\"\n+\n+\n+echo Data >data.txt\n+\n+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With\n+# FSMonitor enabled, we can get a skewed view of status times, since\n+# the index MAY (or may not) be updated after the first invocation\n+# which will update the FSMonitor Token, so the subsequent invocations\n+# may get a smaller response from the daemon.\n+#\n+do_status () {\n+\tmsg=$1\n+\n+\ttest_perf \"$msg\" \"\n+\t\tgit -C $REPO status >/dev/null 2>&1\n+\t\"\n+}\n+\n+do_matrix () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tt=\"[uc $uc][fsm $fsm]\"\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_expect_success \"$t Setup matrix branch\" \"\n+\t\tgit -C $REPO clean -d -f &&\n+\t\tgit -C $REPO checkout $TMP_BR &&\n+\t\ttest_might_fail git -C $REPO branch -D $MATRIX_BR &&\n+\t\tgit -C $REPO branch $MATRIX_BR $TMP_BR &&\n+\t\tgit -C $REPO checkout $MATRIX_BR\n+\t\"\n+\n+\tif test $uc = true\n+\tthen\n+\t\tenable_uc\n+\telse\n+\t\tdisable_uc\n+\tfi\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstart_fsm\n+\telse\n+\t\tstop_fsm\n+\tfi\n+\n+\tdo_status \"$t status after checkout\"\n+\n+\t# Modify many files in the matrix branch.\n+\t# Stage them.\n+\t# Commit them.\n+\t# Rollback.\n+\t#\n+\ttest_expect_success \"$t modify tracked files\" \"\n+\t\tfind $REPO -name file1 -exec cp data.txt {} \\\\;\n+\t\"\n+\n+\tdo_status \"$t status after big change\"\n+\n+\t# Don't bother timing the \"add\" because _REPEAT_COUNT\n+\t# issue described above.\n+\t#\n+\ttest_expect_success \"$t add all\" \"\n+\t\tgit -C $REPO add -A\n+\t\"\n+\n+\tdo_status \"$t status after add all\"\n+\n+\ttest_expect_success \"$t add dot\" \"\n+\t\tgit -C $REPO add .\n+\t\"\n+\n+\tdo_status \"$t status after add dot\"\n+\n+\ttest_expect_success \"$t commit staged\" \"\n+\t\tgit -C $REPO commit -a -m data\n+\t\"\n+\n+\tdo_status \"$t status after commit\"\n+\n+\ttest_expect_success \"$t reset HEAD~1 hard\" \"\n+\t\tgit -C $REPO reset --hard HEAD~1 >/dev/null 2>&1\n+\t\"\n+\n+\tdo_status \"$t status after reset hard\"\n+\n+\t# Create some untracked files.\n+\t#\n+\ttest_expect_success \"$t create untracked files\" \"\n+\t\tcp -R $REPO/ballast/dir1 $REPO/ballast/xxx1\n+\t\"\n+\n+\tdo_status \"$t status after create untracked files\"\n+\n+\t# Remove the new untracked files.\n+\t#\n+\ttest_expect_success \"$t clean -df\" \"\n+\t\tgit -C $REPO clean -d -f\n+\t\"\n+\n+\tdo_status \"$t status after clean\"\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstop_fsm\n+\tfi\n+}\n+\n+# Begin testing each case in the matrix that we care about.\n+#\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\n+fsm_values=\"false true\"\n+\n+for uc_val in $uc_values\n+do\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tdo_matrix $uc_val $fsm_val\n+\tdone\n+done\n+\n+cleanup () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_might_fail git -C $REPO branch -D $MATRIX_BR\n+}\n+\n+\n+# We're borrowing this repo. We should leave it in a clean state.\n+#\n+test_expect_success \"Cleanup temp and matrix branches\" \"\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\tfor fsm_val in $fsm_values\n+\t\tdo\n+\t\t\tcleanup $uc_val $fsm_val\n+\t\tdone\n+\tdone\n+\"\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"454305","messageId":"a9b35e770f32dce0443c15a11caabc3c8e15faaa.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 21/28] t7527: FSMonitor tests for directory moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:47Z","receivedAt":"2022-04-22T22:36:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate unit tests to move a directory. Verify that `git status`\ngives the same result with and without FSMonitor enabled.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex c9c7dd77e3c..d0e681d008f 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -274,6 +274,16 @@ test_expect_success 'setup' '\n \ttrace*\n \tEOF\n \n+\tmkdir -p T1/T2/T3/T4 &&\n+\techo 1 >T1/F1 &&\n+\techo 1 >T1/T2/F1 &&\n+\techo 1 >T1/T2/T3/F1 &&\n+\techo 1 >T1/T2/T3/T4/F1 &&\n+\techo 2 >T1/F2 &&\n+\techo 2 >T1/T2/F2 &&\n+\techo 2 >T1/T2/T3/F2 &&\n+\techo 2 >T1/T2/T3/T4/F2 &&\n+\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n@@ -356,6 +366,19 @@ directory_to_file () {\n \techo 1 >dir1\n }\n \n+move_directory_contents_deeper() {\n+\tmkdir T1/_new_ &&\n+\tmv T1/[A-Z]* T1/_new_\n+}\n+\n+move_directory_up() {\n+\tmv T1/T2/T3 T1\n+}\n+\n+move_directory() {\n+\tmv T1/T2/T3 T1/T2/NewT3\n+}\n+\n # The next few test cases confirm that our fsmonitor daemon sees each type\n # of OS filesystem notification that we care about. At this layer we just\n # ensure we are getting the OS notifications and do not try to confirm what\n@@ -660,6 +683,10 @@ do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n+\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n+\t\tmatrix_try $uc_val $fsm_val move_directory_up\n+\t\tmatrix_try $uc_val $fsm_val move_directory\n+\n \t\tif test $fsm_val = true\n \t\tthen\n \t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n-- \ngitgitgadget\n\n"},{"id":"454306","messageId":"48af0813deccab924d3591b4df025b17bf309260.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 20/28] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:46Z","receivedAt":"2022-04-22T22:36:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to perform binary search over the cache-entries for a directory\nnotification and then linearly scan forward to find the immediate children.\n\nPreviously, when the FSMonitor reported a modified directory Git would\nperform a linear search on the entire cache-entry array for all\nentries matching that directory prefix and invalidate them. Since the\ncache-entry array is already sorted, we can use a binary search to\nfind the first matching entry and then only linearly walk forward and\ninvalidate entries until the prefix changes.\n\nAlso, the original code would invalidate anything having the same\ndirectory prefix. Since a directory event should only be received for\nitems that are immediately within the directory (and not within\nsub-directories of it), only invalidate those entries and not the\nwhole subtree.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 54 insertions(+), 17 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 292a6742b4f..e1229c289cf 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n-\tif (name[len - 1] == '/') {\n+\tint pos = index_name_pos(istate, name, len);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n+\t\t\t name, pos);\n \n+\tif (name[len - 1] == '/') {\n \t\t/*\n-\t\t * TODO We should binary search to find the first path with\n-\t\t * TODO this directory prefix. Then linearly update entries\n-\t\t * TODO while the prefix matches. Taking care to search without\n-\t\t * TODO the trailing slash -- because '/' sorts after a few\n-\t\t * TODO interesting special chars, like '.' and ' '.\n+\t\t * The daemon can decorate directory events, such as\n+\t\t * moves or renames, with a trailing slash if the OS\n+\t\t * FS Event contains sufficient information, such as\n+\t\t * MacOS.\n+\t\t *\n+\t\t * Use this to invalidate the entire cone under that\n+\t\t * directory.\n+\t\t *\n+\t\t * We do not expect an exact match because the index\n+\t\t * does not normally contain directory entries, so we\n+\t\t * start at the insertion point and scan.\n \t\t */\n+\t\tif (pos < 0)\n+\t\t\tpos = -pos - 1;\n \n \t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n-\t\t\t starts_with(istate->cache[i]->name, name))\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n-\t\t/* Need to remove the / from the path for the untracked cache */\n+\n+\t\t/*\n+\t\t * We need to remove the traling \"/\" from the path\n+\t\t * for the untracked cache.\n+\t\t */\n \t\tname[len - 1] = '\\0';\n+\t} else if (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\tint pos = index_name_pos(istate, name, strlen(name));\n-\n-\t\tif (pos >= 0) {\n-\t\t\tstruct cache_entry *ce = istate->cache[pos];\n-\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n \t}\n \n@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n \t * as it could be a new untracked file.\n \t */\n-\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"454307","messageId":"1bd5f346248dad90ab6ac4cea18149d4bb2d2c56.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 19/28] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:45Z","receivedAt":"2022-04-22T22:36:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the listener thread to shutdown the daemon if the spelling of the\nworktree root directory changes.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 07113205a61..83d38e8ac6c 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tstrbuf_release(&msg);\n }\n \n+static int ef_is_root_changed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagRootChanged);\n+}\n+\n static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n {\n \treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n@@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_is_root_changed(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * The spelling of the pathname of the root directory\n+\t\t\t * has changed. This includes the name of the root\n+\t\t\t * directory itself or of any parent directory in the\n+\t\t\t * path.\n+\t\t\t *\n+\t\t\t * (There may be other conditions that throw this,\n+\t\t\t * but I couldn't find any information on it.)\n+\t\t\t *\n+\t\t\t * Force a shutdown now and avoid things getting\n+\t\t\t * out of sync. The Unix domain socket is inside\n+\t\t\t * the .git directory and a spelling change will make\n+\t\t\t * it hard for clients to rendezvous with us.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"event: root changed\");\n+\t\t\tgoto force_shutdown;\n+\t\t}\n+\n \t\tif (ef_ignore_xattr(event_flags[k])) {\n \t\t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n-- \ngitgitgadget\n\n"},{"id":"454308","messageId":"b06edd995eac126769bf970bc894debd4eb3452d.1650662994.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v6 18/28] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-22T21:29:44Z","receivedAt":"2022-04-22T22:37:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nForce shutdown fsmonitor daemon if the worktree root directory\nis moved, renamed, or deleted.\n\nUse Windows low-level GetFileInformationByHandle() to get and\ncompare the Windows system unique ID for the directory with a\ncached version when we started up. This lets us detect the\ncase where someone renames the directory that we are watching\nand then creates a new directory with the original pathname.\n\nThis is important because we are listening to a named pipe for\nrequests and they are stored in the Named Pipe File System (NPFS)\nwhich a kernel-resident pseudo filesystem not associated with\nthe actual NTFS directory.\n\nFor example, if the daemon was watching \"~/foo/\", it would have\na directory-watch handle on that directory and a named-pipe\nhandle for \"//./pipe/...foo\". Moving the directory to \"~/bar/\"\ndoes not invalidate the directory handle. (So the daemon would\nactually be watching \"~/bar\" but listening on \"//./pipe/...foo\".\nIf the user then does \"git init ~/foo\" and causes another daemon\nto start, the first daemon will still have ownership of the pipe\nand the second daemon instance will fail to start. \"git status\"\nclients in \"~/foo\" will ask \"//./pipe/...foo\" about changes and\nthe first daemon instance will tell them about \"~/bar\".\n\nThis commit causes the first daemon to shutdown if the system unique\nID for \"~/foo\" changes (changes from what it was when the daemon\nstarted). Shutdown occurs after a periodic poll. After the\nfirst daemon exits and releases the lock on the named pipe,\nsubsequent Git commands may cause another daemon to be started\non \"~/foo\". Similarly, a subsequent Git command may cause another\ndaemon to be started on \"~/bar\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++\n 1 file changed, 143 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 24fc612bf02..2ea08c1d4e8 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -29,8 +29,150 @@ struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n \tint nr_handles; /* number of active event handles */\n+\n+\tstruct wt_moved\n+\t{\n+\t\twchar_t wpath[MAX_PATH + 1];\n+\t\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\t} wt_moved;\n };\n \n+/*\n+ * Lookup the system unique ID for the path. This is as close as\n+ * we get to an inode number, but this also contains volume info,\n+ * so it is a little stronger.\n+ */\n+static int lookup_bhfi(wchar_t *wpath,\n+\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n+{\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileW(wpath, desired_access, share_mode, NULL,\n+\t\t\t OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] health thread could not open '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\treturn -1;\n+\t}\n+\n+\tif (!GetFileInformationByHandle(hDir, bhfi)) {\n+\t\terror(_(\"[GLE %ld] health thread getting BHFI for '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\tCloseHandle(hDir);\n+\t\treturn -1;\n+\t}\n+\n+\tCloseHandle(hDir);\n+\treturn 0;\n+}\n+\n+/*\n+ * Compare the relevant fields from two system unique IDs.\n+ * We use this to see if two different handles to the same\n+ * path actually refer to the same *instance* of the file\n+ * or directory.\n+ */\n+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n+\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n+{\n+\treturn (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&\n+\t\tbhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&\n+\t\tbhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);\n+}\n+\n+/*\n+ * Shutdown if the original worktree root directory been deleted,\n+ * moved, or renamed?\n+ *\n+ * Since the main thread did a \"chdir(getenv($HOME))\" and our CWD\n+ * is not in the worktree root directory and because the listener\n+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible\n+ * for the root directory to be moved or deleted while we are still\n+ * watching it. We want to detect that here and force a shutdown.\n+ *\n+ * Granted, a delete MAY cause some operations to fail, such as\n+ * GetOverlappedResult(), but it is not guaranteed. And because\n+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the\n+ * directory, not changes *ON* the directory, our watch will not\n+ * receive a delete event for it.\n+ *\n+ * A move/rename of the worktree root will also not generate an event.\n+ * And since the listener thread already has an open handle, it may\n+ * continue to receive events for events within the directory.\n+ * However, the pathname of the named-pipe was constructed using the\n+ * original location of the worktree root. (Remember named-pipes are\n+ * stored in the NPFS and not in the actual file system.) Clients\n+ * trying to talk to the worktree after the move/rename will not\n+ * reach our daemon process, since we're still listening on the\n+ * pipe with original path.\n+ *\n+ * Furthermore, if the user does something like:\n+ *\n+ * $ mv repo repo.old\n+ * $ git init repo\n+ *\n+ * A new daemon cannot be started in the new instance of \"repo\"\n+ * because the named-pipe is still being used by the daemon on\n+ * the original instance.\n+ *\n+ * So, detect move/rename/delete and shutdown. This should also\n+ * handle unsafe drive removal.\n+ *\n+ * We use the file system unique ID to distinguish the original\n+ * directory instance from a new instance and force a shutdown\n+ * if the unique ID changes.\n+ *\n+ * Since a worktree move/rename/delete/unmount doesn't happen\n+ * that often (and we can't get an immediate event anyway), we\n+ * use a timeout and periodically poll it.\n+ */\n+static int has_worktree_moved(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\tint r;\n+\n+\tswitch (ctx) {\n+\tcase CTX_TERM:\n+\t\treturn 0;\n+\n+\tcase CTX_INIT:\n+\t\tif (xutftowcs_path(data->wt_moved.wpath,\n+\t\t\t\t state->path_worktree_watch.buf) < 0) {\n+\t\t\terror(_(\"could not convert to wide characters: '%s'\"),\n+\t\t\t state->path_worktree_watch.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/*\n+\t\t * On the first call we lookup the unique sequence ID for\n+\t\t * the worktree root directory.\n+\t\t */\n+\t\treturn lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);\n+\n+\tcase CTX_TIMER:\n+\t\tr = lookup_bhfi(data->wt_moved.wpath, &bhfi);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t\tif (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {\n+\t\t\terror(_(\"BHFI changed '%ls'\"), data->wt_moved.wpath);\n+\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tdie(_(\"unhandled case in 'has_worktree_moved': %d\"),\n+\t\t (int)ctx);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+\n int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data;\n@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n * A table of the polling functions.\n */\n static interval_fn *table[] = {\n+\thas_worktree_moved,\n \tNULL, /* must be last */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"455197","messageId":"nycvar.QRO.7.76.6.2205121617460.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"8b7c5f4e234e5b139b640652fcfdafb2e24e9db8.1650662994.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 01/28] fsm-listen-win32: handle shortnames","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-12T14:20:49Z","receivedAt":"2022-05-12T14:21:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Teach FSMonitor daemon on Windows to recognize shortname paths as\n> aliases of normal longname paths. FSMonitor clients, such as `git\n> status`, should receive the longname spelling of changed files (when\n> possible).\n>\n> Sometimes we receive FS events using the shortname, such as when a CMD\n> shell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\n> arrives using whatever combination of long and shortnames were used by\n> the other process. (Shortnames do seem to be case normalized,\n> however.)\n>\n> Use Windows GetLongPathNameW() to try to map the pathname spelling in\n> the notification event into the normalized longname spelling. (This\n> can fail if the file/directory is deleted, moved, or renamed, because\n> we are asking the FS for the mapping in response to the event and\n> after it has already happened, but we try.)\n>\n> Special case the shortname spelling of \".git\" to avoid under-reporting\n> these events.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++-----\n> t/t7527-builtin-fsmonitor.sh | 65 +++++\n> 2 files changed, 374 insertions(+), 54 deletions(-)\n>\n> diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\n> index 5b928ab66e5..3f1b68267bd 100644\n> --- a/compat/fsmonitor/fsm-listen-win32.c\n> +++ b/compat/fsmonitor/fsm-listen-win32.c\n> @@ -25,6 +25,9 @@ struct one_watch\n> \tDWORD count;\n>\n> \tstruct strbuf path;\n> +\twchar_t wpath_longname[MAX_PATH + 1];\n> +\tDWORD wpath_longname_len;\n> +\n> \tHANDLE hDir;\n> \tHANDLE hEvent;\n> \tOVERLAPPED overlapped;\n> @@ -34,6 +37,21 @@ struct one_watch\n> \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n> \t */\n> \tBOOL is_active;\n> +\n> +\t/*\n> +\t * Are shortnames enabled on the containing drive? This is\n> +\t * always true for \"C:/\" drives and usually never true for\n> +\t * other drives.\n> +\t *\n> +\t * We only set this for the worktree because we only need to\n> +\t * convert shortname paths to longname paths for items we send\n> +\t * to clients. (We don't care about shortname expansion for\n> +\t * paths inside a GITDIR because we never send them to\n> +\t * clients.)\n> +\t */\n> +\tBOOL has_shortnames;\n> +\tBOOL has_tilda;\n\nAs much as I am a fan of Tilda Swinson, the thing to which we're referring\nhere is a tilde (with an \"e\" in the end, not an \"a\").\n\n> +\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n> };\n>\n> struct fsmonitor_daemon_backend_data\n> @@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n> };\n>\n> /*\n> - * Convert the WCHAR path from the notification into UTF8 and\n> - * then normalize it.\n> + * Convert the WCHAR path from the event into UTF8 and normalize it.\n> + *\n> + * `wpath_len` is in WCHARS not bytes.\n> */\n> -static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n> +static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n> \t\t\t\t struct strbuf *normalized_path)\n> {\n> \tint reserve;\n> \tint len = 0;\n>\n> \tstrbuf_reset(normalized_path);\n> -\tif (!info->FileNameLength)\n> +\tif (!wpath_len)\n> \t\tgoto normalize;\n>\n> \t/*\n> @@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n> \t * sequence of 2 UTF8 characters. That should let us\n> \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n> \t */\n> -\treserve = info->FileNameLength + 1;\n> +\treserve = 2 * wpath_len + 1;\n> \tstrbuf_grow(normalized_path, reserve);\n>\n> \tfor (;;) {\n> -\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n> -\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n> +\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n> +\t\t\t\t\t wpath, wpath_len,\n> \t\t\t\t\t normalized_path->buf,\n> \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n> \t\t\t\t\t NULL, NULL);\n> @@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n> \t\t\tgoto normalize;\n> \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n> \t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n> -\t\t\t GetLastError(),\n> -\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n> -\t\t\t info->FileName);\n> +\t\t\t GetLastError(), (int)wpath_len, wpath);\n> \t\t\treturn -1;\n> \t\t}\n>\n> @@ -98,6 +115,152 @@ normalize:\n> \treturn strbuf_normalize_path(normalized_path);\n> }\n>\n> +/*\n> + * See if the worktree root directory has shortnames enabled.\n> + * This will help us decide if we need to do an expensive shortname\n> + * to longname conversion on every notification event.\n> + *\n> + * We do not want to create a file to test this, so we assume that the\n> + * root directory contains a \".git\" file or directory. (Our caller\n> + * only calls us for the worktree root, so this should be fine.)\n> + *\n> + * Remember the spelling of the shortname for \".git\" if it exists.\n> + */\n> +static void check_for_shortnames(struct one_watch *watch)\n> +{\n> +\twchar_t buf_in[MAX_PATH + 1];\n> +\twchar_t buf_out[MAX_PATH + 1];\n> +\twchar_t *last_slash = NULL;\n> +\twchar_t *last_bslash = NULL;\n> +\twchar_t *last;\n> +\n> +\t/* build L\"/.git\" */\n> +\twcscpy(buf_in, watch->wpath_longname);\n> +\twcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n\nCould you use `wcsncpy()` here (with the appropriate length designed not\nto overrun the `buf_in` buffer?\n\nOr even better: use `swprintf()` (which has a `count` parameter)? The\nperformance impact should be negligible because we only do this once,\nright?\n\n> +\n> +\tif (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n> +\t\treturn;\n> +\n> +\tlast_slash = wcsrchr(buf_out, L'/');\n> +\tlast_bslash = wcsrchr(buf_out, L'\\\\');\n> +\tif (last_slash > last_bslash)\n> +\t\tlast = last_slash + 1;\n> +\telse if (last_bslash)\n> +\t\tlast = last_bslash + 1;\n> +\telse\n> +\t\tlast = buf_out;\n\nWhile this is all correct, I would find it clearer to write this as\nfollowing:\n\n\tfor (filename = p = buf_out; *p; p++)\n\t\t/* We can be sure that `buf_out` does not end in a slash */\n\t\tif (*p == L'/' || *p == '\\\\')\n\t\t\tfilename = p + 1;\n\n> +\n> +\tif (!wcscmp(last, L\".git\"))\n> +\t\treturn;\n> +\n> +\twatch->has_shortnames = 1;\n> +\twcsncpy(watch->dotgit_shortname, last,\n> +\t\tARRAY_SIZE(watch->dotgit_shortname));\n> +\n> +\t/*\n> +\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n> +\t * we should be able to avoid shortname to longname mapping on\n> +\t * every notification event if the source string does not\n> +\t * contain a \"~\".\n> +\t *\n> +\t * However, the documentation for GetLongPathNameW() says\n> +\t * that there are filesystems that don't follow that pattern\n> +\t * and warns against this optimization.\n> +\t *\n> +\t * Lets test this.\n> +\t */\n> +\tif (wcschr(watch->dotgit_shortname, L'~'))\n> +\t\twatch->has_tilda = 1;\n> +}\n> +\n> +enum get_relative_result {\n> +\tGRR_NO_CONVERSION_NEEDED,\n> +\tGRR_HAVE_CONVERSION,\n> +\tGRR_SHUTDOWN,\n> +};\n> +\n> +/*\n> + * Info notification paths are relative to the root of the watch.\n> + * If our CWD is still at the root, then we can use relative paths\n> + * to convert from shortnames to longnames. If our process has a\n> + * different CWD, then we need to construct an absolute path, do\n> + * the conversion, and then return the root-relative portion.\n> + *\n> + * We use the longname form of the root as our basis and assume that\n> + * it already has a trailing slash.\n> + *\n> + * `wpath_len` is in WCHARS not bytes.\n> + */\n> +static enum get_relative_result get_relative_longname(\n> +\tstruct one_watch *watch,\n> +\tconst wchar_t *wpath, DWORD wpath_len,\n> +\twchar_t *wpath_longname)\n> +{\n> +\twchar_t buf_in[2 * MAX_PATH + 1];\n> +\twchar_t buf_out[MAX_PATH + 1];\n> +\tDWORD root_len;\n> +\n> +\t/* Build L\"/\" */\n> +\troot_len = watch->wpath_longname_len;\n> +\twcsncpy(buf_in, watch->wpath_longname, root_len);\n> +\twcsncpy(buf_in + root_len, wpath, wpath_len);\n\nHere, too, I would like to have a check to prevent an overrun. Maybe\n`swprintf()` again? I guess we could invent `xswprintf()` which would\nreturn an error if the return value is -1 or if it used up the entire\nbuffer (i.e. if it overran).\n\n> +\tbuf_in[root_len + wpath_len] = 0;\n> +\n> +\t/*\n> +\t * We don't actually know if the source pathname is a\n> +\t * shortname or a longname. This routine allows either to be\n> +\t * given as input.\n> +\t */\n> +\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n> +\t\t/*\n> +\t\t * The shortname to longname conversion can fail for\n> +\t\t * various reasons, for example if the file has been\n> +\t\t * deleted. (That is, if we just received a\n> +\t\t * delete-file notification event and the file is\n> +\t\t * already gone, we can't ask the file system to\n> +\t\t * lookup the longname for it. Likewise, for moves\n> +\t\t * and renames where we are given the old name.)\n> +\t\t *\n> +\t\t * Since deleting or moving a file or directory by its\n> +\t\t * shortname is rather obscure, I'm going ignore the\n> +\t\t * failure and ask the caller to report the original\n> +\t\t * relative path. This seems kinder than failing here\n> +\t\t * and forcing a resync. Besides, forcing a resync on\n> +\t\t * every file/directory delete would effectively\n> +\t\t * cripple monitoring.\n> +\t\t *\n> +\t\t * We might revisit this in the future.\n> +\t\t */\n> +\t\treturn GRR_NO_CONVERSION_NEEDED;\n> +\t}\n> +\n> +\tif (!wcscmp(buf_in, buf_out)) {\n> +\t\t/*\n> +\t\t * The path does not have a shortname alias.\n> +\t\t */\n> +\t\treturn GRR_NO_CONVERSION_NEEDED;\n> +\t}\n> +\n> +\tif (wcsncmp(buf_in, buf_out, root_len)) {\n> +\t\t/*\n> +\t\t * The spelling of the root directory portion of the computed\n> +\t\t * longname has changed. This should not happen. Basically,\n> +\t\t * it means that we don't know where (without recomputing the\n> +\t\t * longname of just the root directory) to split out the\n> +\t\t * relative path. Since this should not happen, I'm just\n> +\t\t * going to let this fail and force a shutdown (because all\n> +\t\t * subsequent events are probably going to see the same\n> +\t\t * mismatch).\n> +\t\t */\n> +\t\treturn GRR_SHUTDOWN;\n> +\t}\n> +\n> +\t/* Return the worktree root-relative portion of the longname. */\n> +\n> +\twcscpy(wpath_longname, buf_out + root_len);\n> +\treturn GRR_HAVE_CONVERSION;\n> +}\n> +\n> void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n> {\n> \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n> @@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n> \tDWORD share_mode =\n> \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n> \tHANDLE hDir;\n> -\twchar_t wpath[MAX_PATH];\n> +\tDWORD len_longname;\n> +\twchar_t wpath[MAX_PATH + 1];\n> +\twchar_t wpath_longname[MAX_PATH + 1];\n>\n> \tif (xutftowcs_path(wpath, path) < 0) {\n> \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n> @@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n> \t\treturn NULL;\n> \t}\n>\n> +\tif (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n> +\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n> +\t\t GetLastError(), path);\n> +\t\tCloseHandle(hDir);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\tlen_longname = wcslen(wpath_longname);\n\nLet's assign the return value of `GetLongPathNameW()` to `len_longname`,\nin case of success it contains the number of characters, too.\n\nThe rest of the patch looks good to me!\n\nThank you,\nDscho\n\n> +\tif (wpath_longname[len_longname - 1] != L'/' &&\n> +\t wpath_longname[len_longname - 1] != L'\\\\') {\n> +\t\twpath_longname[len_longname++] = L'/';\n> +\t\twpath_longname[len_longname] = 0;\n> +\t}\n> +\n> \tCALLOC_ARRAY(watch, 1);\n>\n> \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n> @@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n> \tstrbuf_init(&watch->path, 0);\n> \tstrbuf_addstr(&watch->path, path);\n>\n> +\twcscpy(watch->wpath_longname, wpath_longname);\n> +\twatch->wpath_longname_len = len_longname;\n> +\n> \twatch->hDir = hDir;\n> \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n>\n> @@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n> \twatch->is_active = FALSE;\n> }\n>\n> +/*\n> + * Process a single relative pathname event.\n> + * Return 1 if we should shutdown.\n> + */\n> +static int process_1_worktree_event(\n> +\tstruct string_list *cookie_list,\n> +\tstruct fsmonitor_batch **batch,\n> +\tconst struct strbuf *path,\n> +\tenum fsmonitor_path_type t,\n> +\tDWORD info_action)\n> +{\n> +\tconst char *slash;\n> +\n> +\tswitch (t) {\n> +\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n> +\t\t/* special case cookie files within .git */\n> +\n> +\t\t/* Use just the filename of the cookie file. */\n> +\t\tslash = find_last_dir_sep(path->buf);\n> +\t\tstring_list_append(cookie_list,\n> +\t\t\t\t slash ? slash + 1 : path->buf);\n> +\t\tbreak;\n> +\n> +\tcase IS_INSIDE_DOT_GIT:\n> +\t\t/* ignore everything inside of \"/.git/\" */\n> +\t\tbreak;\n> +\n> +\tcase IS_DOT_GIT:\n> +\t\t/* \"/.git\" was deleted (or renamed away) */\n> +\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n> +\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n> +\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n> +\t\t\t\t\t \"fsm-listen/dotgit\",\n> +\t\t\t\t\t \"removed\");\n> +\t\t\treturn 1;\n> +\t\t}\n> +\t\tbreak;\n> +\n> +\tcase IS_WORKDIR_PATH:\n> +\t\t/* queue normal pathname */\n> +\t\tif (!*batch)\n> +\t\t\t*batch = fsmonitor_batch__new();\n> +\t\tfsmonitor_batch__add_path(*batch, path->buf);\n> +\t\tbreak;\n> +\n> +\tcase IS_GITDIR:\n> +\tcase IS_INSIDE_GITDIR:\n> +\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n> +\tdefault:\n> +\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n> +\t\t t, path->buf);\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> /*\n> * Process filesystem events that happen anywhere (recursively) under the\n> * root directory. For a normal working directory, this includes\n> @@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n> \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n> \tstruct fsmonitor_batch *batch = NULL;\n> \tconst char *p = watch->buffer;\n> +\twchar_t wpath_longname[MAX_PATH + 1];\n>\n> \t/*\n> \t * If the kernel gets more events than will fit in the kernel\n> @@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n> \t */\n> \tfor (;;) {\n> \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n> -\t\tconst char *slash;\n> +\t\twchar_t *wpath = info->FileName;\n> +\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n> \t\tenum fsmonitor_path_type t;\n> +\t\tenum get_relative_result grr;\n> +\n> +\t\tif (watch->has_shortnames) {\n> +\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n> +\t\t\t\t/*\n> +\t\t\t\t * This event exactly matches the\n> +\t\t\t\t * spelling of the shortname of\n> +\t\t\t\t * \".git\", so we can skip some steps.\n> +\t\t\t\t *\n> +\t\t\t\t * (This case is odd because the user\n> +\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n> +\t\t\t\t * use the filesystem to map it back\n> +\t\t\t\t * to \".git\".)\n> +\t\t\t\t */\n> +\t\t\t\tstrbuf_reset(&path);\n> +\t\t\t\tstrbuf_addstr(&path, \".git\");\n> +\t\t\t\tt = IS_DOT_GIT;\n> +\t\t\t\tgoto process_it;\n> +\t\t\t}\n>\n> -\t\tstrbuf_reset(&path);\n> -\t\tif (normalize_path_in_utf8(info, &path) == -1)\n> -\t\t\tgoto skip_this_path;\n> -\n> -\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n> -\n> -\t\tswitch (t) {\n> -\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n> -\t\t\t/* special case cookie files within .git */\n> -\n> -\t\t\t/* Use just the filename of the cookie file. */\n> -\t\t\tslash = find_last_dir_sep(path.buf);\n> -\t\t\tstring_list_append(&cookie_list,\n> -\t\t\t\t\t slash ? slash + 1 : path.buf);\n> -\t\t\tbreak;\n> -\n> -\t\tcase IS_INSIDE_DOT_GIT:\n> -\t\t\t/* ignore everything inside of \"/.git/\" */\n> -\t\t\tbreak;\n> +\t\t\tif (watch->has_tilda && !wcschr(wpath, L'~')) {\n> +\t\t\t\t/*\n> +\t\t\t\t * Shortnames on this filesystem have tildas\n> +\t\t\t\t * and the notification path does not have\n> +\t\t\t\t * one, so we assume that it is a longname.\n> +\t\t\t\t */\n> +\t\t\t\tgoto normalize_it;\n> +\t\t\t}\n>\n> -\t\tcase IS_DOT_GIT:\n> -\t\t\t/* \"/.git\" was deleted (or renamed away) */\n> -\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n> -\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n> -\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n> -\t\t\t\t\t\t \"fsm-listen/dotgit\",\n> -\t\t\t\t\t\t \"removed\");\n> +\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n> +\t\t\t\t\t\t wpath_longname);\n> +\t\t\tswitch (grr) {\n> +\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n> +\t\t\t\tbreak;\n> +\t\t\tcase GRR_HAVE_CONVERSION:\n> +\t\t\t\twpath = wpath_longname;\n> +\t\t\t\twpath_len = wcslen(wpath);\n> +\t\t\t\tbreak;\n> +\t\t\tdefault:\n> +\t\t\tcase GRR_SHUTDOWN:\n> \t\t\t\tgoto force_shutdown;\n> \t\t\t}\n> -\t\t\tbreak;\n> +\t\t}\n>\n> -\t\tcase IS_WORKDIR_PATH:\n> -\t\t\t/* queue normal pathname */\n> -\t\t\tif (!batch)\n> -\t\t\t\tbatch = fsmonitor_batch__new();\n> -\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n> -\t\t\tbreak;\n> +normalize_it:\n> +\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n> +\t\t\tgoto skip_this_path;\n>\n> -\t\tcase IS_GITDIR:\n> -\t\tcase IS_INSIDE_GITDIR:\n> -\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n> -\t\tdefault:\n> -\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n> -\t\t\t t, path.buf);\n> -\t\t}\n> +\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n> +\n> +process_it:\n> +\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n> +\t\t\t\t\t info->Action))\n> +\t\t\tgoto force_shutdown;\n>\n> skip_this_path:\n> \t\tif (!info->NextEntryOffset)\n> @@ -382,6 +630,9 @@ force_shutdown:\n> * Note that we DO NOT get filesystem events on the external \n> * itself (it is not inside something that we are watching). In particular,\n> * we do not get an event if the external is deleted.\n> + *\n> + * Also, we do not care about shortnames within the external , since\n> + * we never send these paths to clients.\n> */\n> static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n> {\n> @@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n> \t\tconst char *slash;\n> \t\tenum fsmonitor_path_type t;\n>\n> -\t\tstrbuf_reset(&path);\n> -\t\tif (normalize_path_in_utf8(info, &path) == -1)\n> +\t\tif (normalize_path_in_utf8(\n> +\t\t\t info->FileName,\n> +\t\t\t info->FileNameLength / sizeof(WCHAR),\n> +\t\t\t &path) == -1)\n> \t\t\tgoto skip_this_path;\n>\n> \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n> @@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n> \tif (!data->watch_worktree)\n> \t\tgoto failed;\n>\n> +\tcheck_for_shortnames(data->watch_worktree);\n> +\n> \tif (state->nr_paths_watching > 1) {\n> \t\tdata->watch_gitdir = create_watch(state,\n> \t\t\t\t\t\t state->path_gitdir_watch.buf);\n> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n> index bd0c952a116..1be21785162 100755\n> --- a/t/t7527-builtin-fsmonitor.sh\n> +++ b/t/t7527-builtin-fsmonitor.sh\n> @@ -166,6 +166,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n> \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n> '\n>\n> +# File systems on Windows may or may not have shortnames.\n> +# This is a volume-specific setting on modern systems.\n> +# \"C:/\" drives are required to have them enabled. Other\n> +# hard drives default to disabled.\n> +#\n> +# This is a crude test to see if shortnames are enabled\n> +# on the volume containing the test directory. It is\n> +# crude, but it does not require elevation like `fsutil`.\n> +#\n> +test_lazy_prereq SHORTNAMES '\n> +\tmkdir .foo &&\n> +\ttest -d \"FOO~1\"\n> +'\n> +\n> +# Here we assume that the shortname of \".git\" is \"GIT~1\".\n> +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n> +\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n> +\n> +\tgit init test_implicit_1s &&\n> +\n> +\tstart_daemon -C test_implicit_1s &&\n> +\n> +\t# renaming the .git directory will implicitly stop the daemon.\n> +\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n> +\t# the rename-from FS Event will contain the shortname.\n> +\t#\n> +\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n> +\n> +\tsleep 1 &&\n> +\t# put it back so that our status will not crawl out to our\n> +\t# parent directory.\n> +\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n> +\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n> +\n> +\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n> +'\n> +\n> +# Here we first create a file with LONGNAME of \"GIT~1\" before\n> +# we create the repo. This will cause the shortname of \".git\"\n> +# to be \"GIT~2\".\n> +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n> +\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n> +\n> +\tmkdir test_implicit_1s2 &&\n> +\techo HELLO >test_implicit_1s2/GIT~1 &&\n> +\tgit init test_implicit_1s2 &&\n> +\n> +\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n> +\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n> +\n> +\tstart_daemon -C test_implicit_1s2 &&\n> +\n> +\t# renaming the .git directory will implicitly stop the daemon.\n> +\t# the rename-from FS Event will contain the shortname.\n> +\t#\n> +\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n> +\n> +\tsleep 1 &&\n> +\t# put it back so that our status will not crawl out to our\n> +\t# parent directory.\n> +\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n> +\n> +\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n> +'\n> +\n> test_expect_success 'cannot start multiple daemons' '\n> \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n>\n> --\n> gitgitgadget\n>\n>\n"},{"id":"455198","messageId":"nycvar.QRO.7.76.6.2205121620540.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"72b94acd5fe87eaa1dba9bfb66a7a738a0b46a91.1650662994.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 04/28] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-12T14:21:57Z","receivedAt":"2022-05-12T14:22:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Bare repos do not have a worktree, so there is nothing for the\n> daemon watch.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> builtin/fsmonitor--daemon.c | 18 +++++\n> builtin/update-index.c | 16 +++++\n> fsmonitor-settings.c | 133 ++++++++++++++++++++++++++++++------\n> fsmonitor-settings.h | 16 +++++\n> t/t7519-status-fsmonitor.sh | 23 +++++++\n> 5 files changed, 186 insertions(+), 20 deletions(-)\n>\n> diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n> index 46be55a4618..66b78a0353f 100644\n> --- a/builtin/fsmonitor--daemon.c\n> +++ b/builtin/fsmonitor--daemon.c\n> @@ -1423,6 +1423,7 @@ static int try_to_start_background_daemon(void)\n> int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char *subcmd;\n> +\tenum fsmonitor_reason reason;\n> \tint detach_console = 0;\n>\n> \tstruct option options[] = {\n> @@ -1449,6 +1450,23 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n> \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n> \t\t fsmonitor__ipc_threads);\n>\n> +\tprepare_repo_settings(the_repository);\n> +\t/*\n> +\t * If the repo is fsmonitor-compatible, explicitly set IPC-mode\n> +\t * (without bothering to load the `core.fsmonitor` config settings).\n> +\t *\n> +\t * If the repo is not compatible, the repo-settings will be set to\n> +\t * incompatible rather than IPC, so we can use one of the __get\n> +\t * routines to detect the discrepancy.\n> +\t */\n> +\tfsm_settings__set_ipc(the_repository);\n> +\n> +\treason = fsm_settings__get_reason(the_repository);\n> +\tif (reason > FSMONITOR_REASON_OK)\n> +\t\tdie(\"%s\",\n> +\t\t fsm_settings__get_incompatible_msg(the_repository,\n> +\t\t\t\t\t\t reason));\n> +\n> \tif (!strcmp(subcmd, \"start\"))\n> \t\treturn !!try_to_start_background_daemon();\n>\n> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> index 876112abb21..01ed4c4976b 100644\n> --- a/builtin/update-index.c\n> +++ b/builtin/update-index.c\n> @@ -1237,6 +1237,22 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n>\n> \tif (fsmonitor > 0) {\n> \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n> +\t\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n> +\n> +\t\t/*\n> +\t\t * The user wants to turn on FSMonitor using the command\n> +\t\t * line argument. (We don't know (or care) whether that\n> +\t\t * is the IPC or HOOK version.)\n> +\t\t *\n> +\t\t * Use one of the __get routines to force load the FSMonitor\n> +\t\t * config settings into the repo-settings. That will detect\n> +\t\t * whether the file system is compatible so that we can stop\n> +\t\t * here with a nice error message.\n> +\t\t */\n> +\t\tif (reason > FSMONITOR_REASON_OK)\n> +\t\t\tdie(\"%s\",\n> +\t\t\t fsm_settings__get_incompatible_msg(r, reason));\n> +\n> \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n> \t\t\twarning(_(\"core.fsmonitor is unset; \"\n> \t\t\t\t\"set it if you really want to \"\n> diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\n> index 757d230d538..7d3177d441a 100644\n> --- a/fsmonitor-settings.c\n> +++ b/fsmonitor-settings.c\n> @@ -9,23 +9,42 @@\n> */\n> struct fsmonitor_settings {\n> \tenum fsmonitor_mode mode;\n> +\tenum fsmonitor_reason reason;\n> \tchar *hook_path;\n> };\n>\n> -static void lookup_fsmonitor_settings(struct repository *r)\n> +static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n> +{\n> +\tif (!r->worktree) {\n> +\t\t/*\n> +\t\t * Bare repositories don't have a working directory and\n> +\t\t * therefore have nothing to watch.\n> +\t\t */\n> +\t\treturn FSMONITOR_REASON_BARE;\n> +\t}\n> +\n> +\treturn FSMONITOR_REASON_OK;\n> +}\n> +\n> +static struct fsmonitor_settings *alloc_settings(void)\n> {\n> \tstruct fsmonitor_settings *s;\n> +\n> +\tCALLOC_ARRAY(s, 1);\n> +\ts->mode = FSMONITOR_MODE_DISABLED;\n> +\ts->reason = FSMONITOR_REASON_UNTESTED;\n> +\n> +\treturn s;\n> +}\n> +\n> +static void lookup_fsmonitor_settings(struct repository *r)\n> +{\n> \tconst char *const_str;\n> \tint bool_value;\n>\n> \tif (r->settings.fsmonitor)\n> \t\treturn;\n>\n> -\tCALLOC_ARRAY(s, 1);\n> -\ts->mode = FSMONITOR_MODE_DISABLED;\n> -\n> -\tr->settings.fsmonitor = s;\n> -\n> \t/*\n> \t * Overload the existing \"core.fsmonitor\" config setting (which\n> \t * has historically been either unset or a hook pathname) to\n> @@ -38,6 +57,8 @@ static void lookup_fsmonitor_settings(struct repository *r)\n> \tcase 0: /* config value was set to */\n> \t\tif (bool_value)\n> \t\t\tfsm_settings__set_ipc(r);\n> +\t\telse\n> +\t\t\tfsm_settings__set_disabled(r);\n> \t\treturn;\n>\n> \tcase 1: /* config value was unset */\n> @@ -53,18 +74,18 @@ static void lookup_fsmonitor_settings(struct repository *r)\n> \t\treturn;\n> \t}\n>\n> -\tif (!const_str || !*const_str)\n> -\t\treturn;\n> -\n> -\tfsm_settings__set_hook(r, const_str);\n> +\tif (const_str && *const_str)\n> +\t\tfsm_settings__set_hook(r, const_str);\n> +\telse\n> +\t\tfsm_settings__set_disabled(r);\n> }\n>\n> enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n> {\n> \tif (!r)\n> \t\tr = the_repository;\n> -\n> -\tlookup_fsmonitor_settings(r);\n> +\tif (!r->settings.fsmonitor)\n> +\t\tlookup_fsmonitor_settings(r);\n>\n> \treturn r->settings.fsmonitor->mode;\n> }\n> @@ -73,31 +94,55 @@ const char *fsm_settings__get_hook_path(struct repository *r)\n> {\n> \tif (!r)\n> \t\tr = the_repository;\n> -\n> -\tlookup_fsmonitor_settings(r);\n> +\tif (!r->settings.fsmonitor)\n> +\t\tlookup_fsmonitor_settings(r);\n>\n> \treturn r->settings.fsmonitor->hook_path;\n> }\n>\n> void fsm_settings__set_ipc(struct repository *r)\n> {\n> +\tenum fsmonitor_reason reason = check_for_incompatible(r);\n> +\n> +\tif (reason != FSMONITOR_REASON_OK) {\n> +\t\tfsm_settings__set_incompatible(r, reason);\n> +\t\treturn;\n> +\t}\n> +\n> +\t/*\n> +\t * Caller requested IPC explicitly, so avoid (possibly\n> +\t * recursive) config lookup.\n> +\t */\n> \tif (!r)\n> \t\tr = the_repository;\n> -\n> -\tlookup_fsmonitor_settings(r);\n> +\tif (!r->settings.fsmonitor)\n> +\t\tr->settings.fsmonitor = alloc_settings();\n>\n> \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n> +\tr->settings.fsmonitor->reason = reason;\n> \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n> }\n>\n> void fsm_settings__set_hook(struct repository *r, const char *path)\n> {\n> +\tenum fsmonitor_reason reason = check_for_incompatible(r);\n> +\n> +\tif (reason != FSMONITOR_REASON_OK) {\n> +\t\tfsm_settings__set_incompatible(r, reason);\n> +\t\treturn;\n> +\t}\n> +\n> +\t/*\n> +\t * Caller requested hook explicitly, so avoid (possibly\n> +\t * recursive) config lookup.\n> +\t */\n> \tif (!r)\n> \t\tr = the_repository;\n> -\n> -\tlookup_fsmonitor_settings(r);\n> +\tif (!r->settings.fsmonitor)\n> +\t\tr->settings.fsmonitor = alloc_settings();\n>\n> \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n> +\tr->settings.fsmonitor->reason = reason;\n> \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n> \tr->settings.fsmonitor->hook_path = strdup(path);\n> }\n> @@ -106,9 +151,57 @@ void fsm_settings__set_disabled(struct repository *r)\n> {\n> \tif (!r)\n> \t\tr = the_repository;\n> -\n> -\tlookup_fsmonitor_settings(r);\n> +\tif (!r->settings.fsmonitor)\n> +\t\tr->settings.fsmonitor = alloc_settings();\n>\n> \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n> +\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n> +\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n> +}\n> +\n> +void fsm_settings__set_incompatible(struct repository *r,\n> +\t\t\t\t enum fsmonitor_reason reason)\n> +{\n> +\tif (!r)\n> +\t\tr = the_repository;\n> +\tif (!r->settings.fsmonitor)\n> +\t\tr->settings.fsmonitor = alloc_settings();\n> +\n> +\tr->settings.fsmonitor->mode = FSMONITOR_MODE_INCOMPATIBLE;\n> +\tr->settings.fsmonitor->reason = reason;\n> \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n> }\n> +\n> +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n> +{\n> +\tif (!r)\n> +\t\tr = the_repository;\n> +\tif (!r->settings.fsmonitor)\n> +\t\tlookup_fsmonitor_settings(r);\n> +\n> +\treturn r->settings.fsmonitor->reason;\n> +}\n> +\n> +char *fsm_settings__get_incompatible_msg(const struct repository *r,\n> +\t\t\t\t\t enum fsmonitor_reason reason)\n> +{\n> +\tstruct strbuf msg = STRBUF_INIT;\n> +\n> +\tswitch (reason) {\n> +\tcase FSMONITOR_REASON_UNTESTED:\n> +\tcase FSMONITOR_REASON_OK:\n> +\t\tgoto done;\n> +\n> +\tcase FSMONITOR_REASON_BARE:\n> +\t\tstrbuf_addf(&msg,\n> +\t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n> +\t\t\t xgetcwd());\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n> +\t reason);\n> +\n> +done:\n> +\treturn strbuf_detach(&msg, NULL);\n> +}\n> diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\n> index a4c5d7b4889..4c7592896e4 100644\n> --- a/fsmonitor-settings.h\n> +++ b/fsmonitor-settings.h\n> @@ -4,18 +4,34 @@\n> struct repository;\n>\n> enum fsmonitor_mode {\n> +\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n> \tFSMONITOR_MODE_DISABLED = 0,\n> \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n> \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n> };\n>\n> +/*\n> + * Incompatibility reasons.\n> + */\n> +enum fsmonitor_reason {\n> +\tFSMONITOR_REASON_UNTESTED = 0,\n> +\tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n\ns/disbled/disabled/\n\nI usually try to avoid pointing out speling erors, but I kind of wanted to\nshow that I really studied this patch series for a couple hours.\n\nThe patch looks good to me,\nDscho\n\n> +\tFSMONITOR_REASON_BARE,\n> +};\n> +\n> void fsm_settings__set_ipc(struct repository *r);\n> void fsm_settings__set_hook(struct repository *r, const char *path);\n> void fsm_settings__set_disabled(struct repository *r);\n> +void fsm_settings__set_incompatible(struct repository *r,\n> +\t\t\t\t enum fsmonitor_reason reason);\n>\n> enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n> const char *fsm_settings__get_hook_path(struct repository *r);\n>\n> +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n> +char *fsm_settings__get_incompatible_msg(const struct repository *r,\n> +\t\t\t\t\t enum fsmonitor_reason reason);\n> +\n> struct fsmonitor_settings;\n>\n> #endif /* FSMONITOR_SETTINGS_H */\n> diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\n> index a6308acf006..9a8e21c5608 100755\n> --- a/t/t7519-status-fsmonitor.sh\n> +++ b/t/t7519-status-fsmonitor.sh\n> @@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '\n> \ttest $ret -ne 1\n> '\n>\n> +# Test that we detect and disallow repos that are incompatible with FSMonitor.\n> +test_expect_success 'incompatible bare repo' '\n> +\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n> +\tgit init --bare bare-clone &&\n> +\n> +\ttest_must_fail \\\n> +\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n> +\t\t\tupdate-index --fsmonitor 2>actual &&\n> +\tgrep \"bare repository .* is incompatible with fsmonitor\" actual &&\n> +\n> +\ttest_must_fail \\\n> +\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n> +\t\t\tupdate-index --fsmonitor 2>actual &&\n> +\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n> +'\n> +\n> +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n> +\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n> +\tgit init --bare bare-clone &&\n> +\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n> +\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n> +'\n> +\n> test_expect_success 'setup' '\n> \tmkdir -p .git/hooks &&\n> \t: >tracked &&\n> --\n> gitgitgadget\n>\n>\n"},{"id":"455202","messageId":"nycvar.QRO.7.76.6.2205121622030.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"32fc6ba74378cda7a5060d63066d16333765973a.1650662994.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 16/28] fsmonitor--daemon: stub in health thread","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-12T15:05:11Z","receivedAt":"2022-05-12T15:05:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Create another thread to watch over the daemon process and\n> automatically shut it down if necessary.\n>\n> This commit creates the basic framework for a \"health\" thread\n> to monitor the daemon and/or the file system. Later commits\n> will add platform-specific code to do the actual work.\n>\n> The \"health\" thread is intended to monitor conditions that\n> would be difficult to track inside the IPC thread pool and/or\n> the file system listener threads. For example, when there are\n> file system events outside of the watched worktree root or if\n> we want to have an idle-timeout auto-shutdown feature.\n>\n> This commit creates the health thread itself, defines the thread-proc\n> and sets up the thread's event loop. It integrates this new thread\n> into the existing IPC and Listener thread models.\n>\n> This commit defines the API to the platform-specific code where all of\n> the monitoring will actually happen.\n>\n> The platform-specific code for MacOS is just stubs. Meaning that the\n> health thread will immediately exit on MacOS, but that is OK and\n> expected. Future work can define MacOS-specific monitoring.\n>\n> The platform-specific code for Windows sets up enough of the\n> WaitForMultipleObjects() machinery to watch for system and/or custom\n> events. Currently, the set of wait handles only includes our custom\n> shutdown event (sent from our other theads). Later commits in this\n> series will extend the set of wait handles to monitor other\n> conditions.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> Makefile | 6 ++-\n> builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n> compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n> compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n> compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n> contrib/buildsystems/CMakeLists.txt | 2 +\n> fsmonitor--daemon.h | 4 ++\n> 7 files changed, 192 insertions(+), 2 deletions(-)\n> create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n> create mode 100644 compat/fsmonitor/fsm-health-win32.c\n> create mode 100644 compat/fsmonitor/fsm-health.h\n>\n> diff --git a/Makefile b/Makefile\n> index 93604fe8ef7..5f1623baadd 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -472,8 +472,9 @@ all::\n> #\n> # If your platform supports a built-in fsmonitor backend, set\n> # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n> -# `compat/fsmonitor/fsm-listen-.c` that implements the\n> -# `fsm_listen__*()` routines.\n> +# `compat/fsmonitor/fsm-listen-.c` and\n> +# `compat/fsmonitor/fsm-health-.c` files\n> +# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n> #\n> # If your platform has OS-specific ways to tell if a repo is incompatible with\n> # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n> @@ -1982,6 +1983,7 @@ endif\n> ifdef FSMONITOR_DAEMON_BACKEND\n> \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n> \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n> +\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n> endif\n>\n> ifdef FSMONITOR_OS_SETTINGS\n> diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\n> index b2f578b239a..2c109cf8b37 100644\n> --- a/builtin/fsmonitor--daemon.c\n> +++ b/builtin/fsmonitor--daemon.c\n> @@ -3,6 +3,7 @@\n> #include \"parse-options.h\"\n> #include \"fsmonitor.h\"\n> #include \"fsmonitor-ipc.h\"\n> +#include \"compat/fsmonitor/fsm-health.h\"\n> #include \"compat/fsmonitor/fsm-listen.h\"\n> #include \"fsmonitor--daemon.h\"\n> #include \"simple-ipc.h\"\n> @@ -1136,6 +1137,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n> \tpthread_mutex_unlock(&state->main_lock);\n> }\n>\n> +static void *fsm_health__thread_proc(void *_state)\n> +{\n> +\tstruct fsmonitor_daemon_state *state = _state;\n> +\n> +\ttrace2_thread_start(\"fsm-health\");\n> +\n> +\tfsm_health__loop(state);\n> +\n> +\ttrace2_thread_exit();\n> +\treturn NULL;\n> +}\n> +\n> static void *fsm_listen__thread_proc(void *_state)\n> {\n> \tstruct fsmonitor_daemon_state *state = _state;\n> @@ -1174,6 +1187,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n> \t\t */\n> \t\t.uds_disallow_chdir = 0\n> \t};\n> +\tint health_started = 0;\n> \tint listener_started = 0;\n> \tint err = 0;\n>\n> @@ -1201,6 +1215,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n> \t}\n> \tlistener_started = 1;\n>\n> +\t/*\n> +\t * Start the health thread to watch over our process.\n> +\t */\n> +\tif (pthread_create(&state->health_thread, NULL,\n> +\t\t\t fsm_health__thread_proc, state) < 0) {\n> +\t\tipc_server_stop_async(state->ipc_server_data);\n> +\t\terr = error(_(\"could not start fsmonitor health thread\"));\n> +\t\tgoto cleanup;\n> +\t}\n> +\thealth_started = 1;\n> +\n> \t/*\n> \t * The daemon is now fully functional in background threads.\n> \t * Our primary thread should now just wait while the threads\n> @@ -1223,10 +1248,17 @@ cleanup:\n> \t\tpthread_join(state->listener_thread, NULL);\n> \t}\n>\n> +\tif (health_started) {\n> +\t\tfsm_health__stop_async(state);\n> +\t\tpthread_join(state->health_thread, NULL);\n> +\t}\n> +\n> \tif (err)\n> \t\treturn err;\n> \tif (state->listen_error_code)\n> \t\treturn state->listen_error_code;\n> +\tif (state->health_error_code)\n> +\t\treturn state->health_error_code;\n> \treturn 0;\n> }\n>\n> @@ -1242,6 +1274,7 @@ static int fsmonitor_run_daemon(void)\n> \tpthread_mutex_init(&state.main_lock, NULL);\n> \tpthread_cond_init(&state.cookies_cond, NULL);\n> \tstate.listen_error_code = 0;\n> +\tstate.health_error_code = 0;\n> \tstate.current_token_data = fsmonitor_new_token_data();\n>\n> \t/* Prepare to (recursively) watch the directory. */\n> @@ -1321,6 +1354,11 @@ static int fsmonitor_run_daemon(void)\n> \t\tgoto done;\n> \t}\n>\n> +\tif (fsm_health__ctor(&state)) {\n> +\t\terr = error(_(\"could not initialize health thread\"));\n> +\t\tgoto done;\n> +\t}\n> +\n> \t/*\n> \t * CD out of the worktree root directory.\n> \t *\n> @@ -1344,6 +1382,7 @@ done:\n> \tpthread_cond_destroy(&state.cookies_cond);\n> \tpthread_mutex_destroy(&state.main_lock);\n> \tfsm_listen__dtor(&state);\n> +\tfsm_health__dtor(&state);\n>\n> \tipc_server_free(state.ipc_server_data);\n>\n> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\n> new file mode 100644\n> index 00000000000..b9f709e8548\n> --- /dev/null\n> +++ b/compat/fsmonitor/fsm-health-darwin.c\n> @@ -0,0 +1,24 @@\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"fsmonitor.h\"\n> +#include \"fsm-health.h\"\n> +#include \"fsmonitor--daemon.h\"\n> +\n> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n> +{\n> +\treturn 0;\n> +}\n> +\n> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n> +{\n> +\treturn;\n> +}\n> +\n> +void fsm_health__loop(struct fsmonitor_daemon_state *state)\n> +{\n> +\treturn;\n> +}\n> +\n> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n> +{\n> +}\n> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\n> new file mode 100644\n> index 00000000000..94b1d020f25\n> --- /dev/null\n> +++ b/compat/fsmonitor/fsm-health-win32.c\n> @@ -0,0 +1,72 @@\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"fsmonitor.h\"\n> +#include \"fsm-health.h\"\n> +#include \"fsmonitor--daemon.h\"\n> +\n> +struct fsm_health_data\n> +{\n> +\tHANDLE hEventShutdown;\n> +\n> +\tHANDLE hHandles[1]; /* the array does not own these handles */\n> +#define HEALTH_SHUTDOWN 0\n\nHow about defining `HANDLE hHandles[HEALTH_SHUTDOWN + 1]` to indicate that\nthe constant is used as an offset into `hHandles`?\n\n> +\tint nr_handles; /* number of active event handles */\n> +};\n> +\n> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n> +{\n> +\tstruct fsm_health_data *data;\n> +\n> +\tCALLOC_ARRAY(data, 1);\n\nI _guess_ that this is okay, even if `data` is not actually an array. But\nit's a convenient construct to get the parameters right.\n\nThank you!\nDscho\n\n> +\n> +\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n> +\n> +\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n> +\tdata->nr_handles++;\n> +\n> +\tstate->health_data = data;\n> +\treturn 0;\n> +}\n> +\n> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n> +{\n> +\tstruct fsm_health_data *data;\n> +\n> +\tif (!state || !state->health_data)\n> +\t\treturn;\n> +\n> +\tdata = state->health_data;\n> +\n> +\tCloseHandle(data->hEventShutdown);\n> +\n> +\tFREE_AND_NULL(state->health_data);\n> +}\n> +\n> +void fsm_health__loop(struct fsmonitor_daemon_state *state)\n> +{\n> +\tstruct fsm_health_data *data = state->health_data;\n> +\n> +\tfor (;;) {\n> +\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n> +\t\t\t\t\t\t data->hHandles,\n> +\t\t\t\t\t\t FALSE, INFINITE);\n> +\n> +\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n> +\t\t\tgoto clean_shutdown;\n> +\n> +\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n> +\t\t GetLastError());\n> +\t\tgoto force_error_stop;\n> +\t}\n> +\n> +force_error_stop:\n> +\tstate->health_error_code = -1;\n> +\tipc_server_stop_async(state->ipc_server_data);\n> +clean_shutdown:\n> +\treturn;\n> +}\n> +\n> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n> +{\n> +\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n> +}\n> diff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\n> new file mode 100644\n> index 00000000000..45547ba9380\n> --- /dev/null\n> +++ b/compat/fsmonitor/fsm-health.h\n> @@ -0,0 +1,47 @@\n> +#ifndef FSM_HEALTH_H\n> +#define FSM_HEALTH_H\n> +\n> +/* This needs to be implemented by each backend */\n> +\n> +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n> +\n> +struct fsmonitor_daemon_state;\n> +\n> +/*\n> + * Initialize platform-specific data for the fsmonitor health thread.\n> + * This will be called from the main thread PRIOR to staring the\n> + * thread.\n> + *\n> + * Returns 0 if successful.\n> + * Returns -1 otherwise.\n> + */\n> +int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n> +\n> +/*\n> + * Cleanup platform-specific data for the health thread.\n> + * This will be called from the main thread AFTER joining the thread.\n> + */\n> +void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n> +\n> +/*\n> + * The main body of the platform-specific event loop to monitor the\n> + * health of the daemon process. This will run in the health thread.\n> + *\n> + * The health thread should call `ipc_server_stop_async()` if it needs\n> + * to cause a shutdown. (It should NOT do so if it receives a shutdown\n> + * shutdown signal.)\n> + *\n> + * It should set `state->health_error_code` to -1 if the daemon should exit\n> + * with an error.\n> + */\n> +void fsm_health__loop(struct fsmonitor_daemon_state *state);\n> +\n> +/*\n> + * Gently request that the health thread shutdown.\n> + * It does not wait for it to stop. The caller should do a JOIN\n> + * to wait for it.\n> + */\n> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n> +\n> +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n> +#endif /* FSM_HEALTH_H */\n> diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\n> index b8f9f7a0388..16ace43d1c7 100644\n> --- a/contrib/buildsystems/CMakeLists.txt\n> +++ b/contrib/buildsystems/CMakeLists.txt\n> @@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n> \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n> \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n> \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n> +\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n>\n> \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n> \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n> \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n> \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n> \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n> +\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n>\n> \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n> \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n> diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\n> index 2c6fa1a5d91..2102a5c9ff5 100644\n> --- a/fsmonitor--daemon.h\n> +++ b/fsmonitor--daemon.h\n> @@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n> void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n>\n> struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n> +struct fsm_health_data; /* opaque platform-specific data for health thread */\n>\n> struct fsmonitor_daemon_state {\n> \tpthread_t listener_thread;\n> +\tpthread_t health_thread;\n> \tpthread_mutex_t main_lock;\n>\n> \tstruct strbuf path_worktree_watch;\n> @@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n> \tstruct hashmap cookies;\n>\n> \tint listen_error_code;\n> +\tint health_error_code;\n> \tstruct fsm_listen_data *listen_data;\n> +\tstruct fsm_health_data *health_data;\n>\n> \tstruct ipc_server_data *ipc_server_data;\n> \tstruct strbuf path_ipc;\n> --\n> gitgitgadget\n>\n>\n"},{"id":"455203","messageId":"nycvar.QRO.7.76.6.2205121708400.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"48af0813deccab924d3591b4df025b17bf309260.1650662994.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 20/28] fsmonitor: optimize processing of directory events","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-12T15:08:48Z","receivedAt":"2022-05-12T15:09:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Teach Git to perform binary search over the cache-entries for a directory\n> notification and then linearly scan forward to find the immediate children.\n>\n> Previously, when the FSMonitor reported a modified directory Git would\n> perform a linear search on the entire cache-entry array for all\n> entries matching that directory prefix and invalidate them. Since the\n> cache-entry array is already sorted, we can use a binary search to\n> find the first matching entry and then only linearly walk forward and\n> invalidate entries until the prefix changes.\n>\n> Also, the original code would invalidate anything having the same\n> directory prefix. Since a directory event should only be received for\n> items that are immediately within the directory (and not within\n> sub-directories of it), only invalidate those entries and not the\n> whole subtree.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n> 1 file changed, 54 insertions(+), 17 deletions(-)\n>\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 292a6742b4f..e1229c289cf 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> {\n> \tint i, len = strlen(name);\n> -\tif (name[len - 1] == '/') {\n> +\tint pos = index_name_pos(istate, name, len);\n> +\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n> +\t\t\t name, pos);\n>\n> +\tif (name[len - 1] == '/') {\n> \t\t/*\n> -\t\t * TODO We should binary search to find the first path with\n> -\t\t * TODO this directory prefix. Then linearly update entries\n> -\t\t * TODO while the prefix matches. Taking care to search without\n> -\t\t * TODO the trailing slash -- because '/' sorts after a few\n> -\t\t * TODO interesting special chars, like '.' and ' '.\n> +\t\t * The daemon can decorate directory events, such as\n> +\t\t * moves or renames, with a trailing slash if the OS\n> +\t\t * FS Event contains sufficient information, such as\n> +\t\t * MacOS.\n> +\t\t *\n> +\t\t * Use this to invalidate the entire cone under that\n> +\t\t * directory.\n> +\t\t *\n> +\t\t * We do not expect an exact match because the index\n> +\t\t * does not normally contain directory entries, so we\n> +\t\t * start at the insertion point and scan.\n> \t\t */\n> +\t\tif (pos < 0)\n> +\t\t\tpos = -pos - 1;\n>\n> \t\t/* Mark all entries for the folder invalid */\n> -\t\tfor (i = 0; i < istate->cache_nr; i++) {\n> -\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n> -\t\t\t starts_with(istate->cache[i]->name, name))\n> -\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> +\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> +\t\t\t\tbreak;\n> +\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> \t\t}\n> -\t\t/* Need to remove the / from the path for the untracked cache */\n> +\n> +\t\t/*\n> +\t\t * We need to remove the traling \"/\" from the path\n> +\t\t * for the untracked cache.\n> +\t\t */\n> \t\tname[len - 1] = '\\0';\n> +\t} else if (pos >= 0) {\n> +\t\t/*\n> +\t\t * We have an exact match for this path and can just\n> +\t\t * invalidate it.\n> +\t\t */\n> +\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> \t} else {\n> -\t\tint pos = index_name_pos(istate, name, strlen(name));\n> -\n> -\t\tif (pos >= 0) {\n> -\t\t\tstruct cache_entry *ce = istate->cache[pos];\n> -\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\t/*\n> +\t\t * The path is not a tracked file -or- it is a\n> +\t\t * directory event on a platform that cannot\n> +\t\t * distinguish between file and directory events in\n> +\t\t * the event handler, such as Windows.\n> +\t\t *\n> +\t\t * Scan as if it is a directory and invalidate the\n> +\t\t * cone under it. (But remember to ignore items\n> +\t\t * between \"name\" and \"name/\", such as \"name-\" and\n> +\t\t * \"name.\".\n> +\t\t */\n> +\t\tpos = -pos - 1;\n> +\n> +\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> +\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> +\t\t\t\tbreak;\n> +\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n\nNice attention to detail casting `istate->cache[i]->name[len]` to\n`(unsigned char)` before comparing to '/'!\n\n> +\t\t\t\tbreak;\n> +\t\t\tif (istate->cache[i]->name[len] == '/')\n> +\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> \t\t}\n> \t}\n>\n> @@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \t * Mark the untracked cache dirty even if it wasn't found in the index\n> \t * as it could be a new untracked file.\n> \t */\n> -\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n\nDid you mean to remove this statement in this patch? Not a big issue, but\nI wonder what the rationale for it is, and since I have an inquisitive\nmind, I figured I'd just ask.\n\nThanks,\nDscho\n\n> \tuntracked_cache_invalidate_path(istate, name, 0);\n> }\n>\n> --\n> gitgitgadget\n>\n>\n"},{"id":"455204","messageId":"nycvar.QRO.7.76.6.2205121709210.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"d0e25f6bac663e9ae4d63322f102378dd2ecba84.1650662994.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 23/28] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-12T15:11:15Z","receivedAt":"2022-05-12T15:11:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Never set CE_FSMONITOR_VALID on the cache-entry of submodule\n> directories.\n>\n> During a client command like 'git status', we may need to recurse\n> into each submodule to compute a status summary for the submodule.\n> Since the purpose of the ce_flag is to let Git avoid scanning a\n> cache-entry, setting the flag causes the recursive call to be\n> avoided and we report incorrect (no status) for the submodule.\n>\n> We created an OS watch on the root directory of our working\n> directory and we receive events for everything in the cone\n> under it. When submodules are present inside our working\n> directory, we receive events for both our repo (the super) and\n> any subs within it. Since our index doesn't have any information\n> for items within the submodules, we can't use those events.\n>\n> We could try to truncate the paths of those events back to the\n> submodule boundary and mark the GITLINK as dirty, but that\n> feels expensive since we would have to prefix compare every FS\n> event that we receive against a list of submodule roots. And\n> it still wouldn't be sufficient to correctly report status on\n> the submodule, since we don't have any space in the cache-entry\n> to cache the submodule's status (the 'SCMU' bits in porcelain\n> V2 speak). That is, the CE_FSMONITOR_VALID bit just says that\n> we don't need to scan/inspect it because we already know the\n> answer -- it doesn't say that the item is clean -- and we\n> don't have space in the cache-entry to store those answers.\n> So we should always do the recursive scan.\n>\n> Therefore, we should never set the flag on GITLINK cache-entries.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 2 +\n> fsmonitor.h | 11 ++++\n> t/t7527-builtin-fsmonitor.sh | 111 +++++++++++++++++++++++++++++++++++\n> 3 files changed, 124 insertions(+)\n>\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index e1229c289cf..57d6a483bee 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)\n> \t\tif (fsmonitor_enabled) {\n> \t\t\t/* Mark all entries valid */\n> \t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n> +\t\t\t\tif (S_ISGITLINK(istate->cache[i]->ce_mode))\n> +\t\t\t\t\tcontinue;\n> \t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n> \t\t\t}\n>\n> diff --git a/fsmonitor.h b/fsmonitor.h\n> index 3f41f653691..edf7ce5203b 100644\n> --- a/fsmonitor.h\n> +++ b/fsmonitor.h\n> @@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n> * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n> * called any time the cache entry has been updated to reflect the\n> * current state of the file on disk.\n> + *\n> + * However, never mark submodules as valid. When commands like \"git\n> + * status\" run they might need to recurse into the submodule (using a\n> + * child process) to get a summary of the submodule state. We don't\n> + * have (and don't want to create) the facility to translate every\n> + * FS event that we receive and that happens to be deep inside of a\n> + * submodule back to the submodule root, so we cannot correctly keep\n> + * track of this bit on the gitlink directory. Therefore, we never\n> + * set it on submodules.\n> */\n> static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n> {\n> @@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n>\n> \tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n> \t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n> +\t\tif (S_ISGITLINK(ce->ce_mode))\n> +\t\t\treturn;\n> \t\tistate->cache_changed = 1;\n> \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n> \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\n> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n> index d0e681d008f..4c49ae5a684 100755\n> --- a/t/t7527-builtin-fsmonitor.sh\n> +++ b/t/t7527-builtin-fsmonitor.sh\n> @@ -721,4 +721,115 @@ do\n> \t'\n> done\n>\n> +# Test fsmonitor interaction with submodules.\n> +#\n> +# If we start the daemon in the super, it will see FS events for\n> +# everything in the working directory cone and this includes any\n> +# files/directories contained *within* the submodules.\n> +#\n> +# A `git status` at top level will get events for items within the\n> +# submodule and ignore them, since they aren't named in the index\n> +# of the super repo. This makes the fsmonitor response a little\n> +# noisy, but it doesn't alter the correctness of the state of the\n> +# super-proper.\n> +#\n> +# When we have submodules, `git status` normally does a recursive\n> +# status on each of the submodules and adds a summary row for any\n> +# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n> +#\n> +# It is therefore important that the top level status not be tricked\n> +# by the FSMonitor response to skip those recursive calls. That is,\n> +# even if FSMonitor says that the mtime of the submodule directory\n> +# hasn't changed and it could be implicitly marked valid, we must\n> +# not take that shortcut. We need to force the recusion into the\n> +# submodule so that we get a summary of the status *within* the\n> +# submodule.\n> +\n> +create_super () {\n> +\tsuper=\"$1\" &&\n> +\n> +\tgit init \"$super\" &&\n> +\techo x >\"$super/file_1\" &&\n> +\techo y >\"$super/file_2\" &&\n> +\techo z >\"$super/file_3\" &&\n> +\tmkdir \"$super/dir_1\" &&\n> +\techo a >\"$super/dir_1/file_11\" &&\n> +\techo b >\"$super/dir_1/file_12\" &&\n> +\tmkdir \"$super/dir_1/dir_2\" &&\n> +\techo a >\"$super/dir_1/dir_2/file_21\" &&\n> +\techo b >\"$super/dir_1/dir_2/file_22\" &&\n> +\tgit -C \"$super\" add . &&\n> +\tgit -C \"$super\" commit -m \"initial $super commit\"\n> +}\n> +\n> +create_sub () {\n> +\tsub=\"$1\" &&\n> +\n> +\tgit init \"$sub\" &&\n> +\techo x >\"$sub/file_x\" &&\n> +\techo y >\"$sub/file_y\" &&\n> +\techo z >\"$sub/file_z\" &&\n> +\tmkdir \"$sub/dir_x\" &&\n> +\techo a >\"$sub/dir_x/file_a\" &&\n> +\techo b >\"$sub/dir_x/file_b\" &&\n> +\tmkdir \"$sub/dir_x/dir_y\" &&\n> +\techo a >\"$sub/dir_x/dir_y/file_a\" &&\n> +\techo b >\"$sub/dir_x/dir_y/file_b\" &&\n> +\tgit -C \"$sub\" add . &&\n> +\tgit -C \"$sub\" commit -m \"initial $sub commit\"\n> +}\n> +\n> +my_match_and_clean () {\n> +\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n> +\tgit -C super --no-optional-locks -c core.fsmonitor=false \\\n> +\t\tstatus --porcelain=v2 >actual.without &&\n> +\ttest_cmp actual.with actual.without &&\n> +\n> +\tgit -C super/dir_1/dir_2/sub reset --hard &&\n> +\tgit -C super/dir_1/dir_2/sub clean -d -f\n> +}\n> +\n> +test_expect_success \"Submodule always visited\" '\n\nI almost feel bad offering this nit: could you use single-quotes, and\nstart with a lower-case `s`?\n\nThanks,\nDscho\n\n> +\ttest_when_finished \"git -C super fsmonitor--daemon stop; \\\n> +\t\t\t rm -rf super; \\\n> +\t\t\t rm -rf sub\" &&\n> +\n> +\tcreate_super super &&\n> +\tcreate_sub sub &&\n> +\n> +\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n> +\tgit -C super commit -m \"add sub\" &&\n> +\n> +\tstart_daemon -C super &&\n> +\tgit -C super config core.fsmonitor true &&\n> +\tgit -C super update-index --fsmonitor &&\n> +\tgit -C super status &&\n> +\n> +\t# Now run pairs of commands w/ and w/o FSMonitor while we make\n> +\t# some dirt in the submodule and confirm matching output.\n> +\n> +\t# Completely clean status.\n> +\tmy_match_and_clean &&\n> +\n> +\t# .M S..U\n> +\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&\n> +\tmy_match_and_clean &&\n> +\n> +\t# .M S.M.\n> +\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&\n> +\tgit -C super/dir_1/dir_2/sub add . &&\n> +\tmy_match_and_clean &&\n> +\n> +\t# .M S.M.\n> +\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n> +\tgit -C super/dir_1/dir_2/sub add . &&\n> +\tmy_match_and_clean &&\n> +\n> +\t# .M SC..\n> +\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n> +\tgit -C super/dir_1/dir_2/sub add . &&\n> +\tgit -C super/dir_1/dir_2/sub commit -m \"SC..\" &&\n> +\tmy_match_and_clean\n> +'\n> +\n> test_done\n> --\n> gitgitgadget\n>\n>\n"},{"id":"455205","messageId":"nycvar.QRO.7.76.6.2205121711300.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"8278f32c4d894d4930b9f1f70f3aa01679e2011e.1650662994.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 26/28] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-12T15:26:19Z","receivedAt":"2022-05-12T15:26:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler \n>\n> Create a set of prereqs to help understand how file names\n> are handled by the filesystem when they contain NFC and NFD\n> Unicode characters.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 167 insertions(+)\n> create mode 100755 t/lib-unicode-nfc-nfd.sh\n>\n> diff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\n> new file mode 100755\n> index 00000000000..cf9c26d1e22\n> --- /dev/null\n> +++ b/t/lib-unicode-nfc-nfd.sh\n> @@ -0,0 +1,167 @@\n> +# Help detect how Unicode NFC and NFD are handled on the filesystem.\n> +\n> +# A simple character that has a NFD form.\n> +#\n> +# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n> +# UTF8(NFC): \\xc3 \\xa9\n> +#\n> +# NFD: U+0065 LATIN SMALL LETTER E\n> +# U+0301 COMBINING ACUTE ACCENT\n> +# UTF8(NFD): \\x65 + \\xcc \\x81\n> +#\n> +utf8_nfc=$(printf \"\\xc3\\xa9\")\n> +utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n> +\n> +# Is the OS or the filesystem \"Unicode composition sensitive\"?\n> +#\n> +# That is, does the OS or the filesystem allow files to exist with\n> +# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n> +# tell us that the NFC and NFD forms are equivalent.\n> +#\n> +# This is or may be independent of what type of filesystem we have,\n> +# since it might be handled by the OS at a layer above the FS.\n> +# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n> +# collision, for example.\n> +#\n> +# This does not tell us how the Unicode pathname will be spelled\n> +# on disk, but rather only that the two spelling \"collide\". We\n> +# will examine the actual on disk spelling in a later prereq.\n> +#\n> +test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n> +\tmkdir trial_${utf8_nfc} &&\n> +\tmkdir trial_${utf8_nfd}\n> +'\n> +\n> +# Is the spelling of an NFC pathname preserved on disk?\n> +#\n> +# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n> +# and on APFS, NFC paths are preserved. As we have established\n> +# above, this is independent of \"composition sensitivity\".\n> +#\n> +# 0000000 63 5f c3 a9\n> +#\n> +# (/usr/bin/od output contains different amount of whitespace\n> +# on different platforms, so we need the wildcards here.)\n> +#\n> +test_lazy_prereq UNICODE_NFC_PRESERVED '\n> +\tmkdir c_${utf8_nfc} &&\n> +\tls | od -t x1 | grep \"63 *5f *c3 *a9\"\n\nAs far as I can see, this would be the first usage of `od` in the test\nsuite. I'd actually like to reduce our dependency on Unix-y tools, not\nincrease it.\n\nOne thing we could do would be to imitate t4030, and introduce a shell\nfunction that calls Perl, something like:\n\n\tbin2hex () {\n\t\tperl -e '\n\t\t\t$/ = undef;\n\t\t\t$_ = <>;\n\t\t\ts/./sprintf(\"%02x \", ord($&))/ge;\n\t\t\tprint $_\n\t\t'\n\t}\n\nBut it is a thorn in my side for quite a few years already that we\n_require_ Perl, even in NO_PERL builds.\n\nSo maybe a much better idea would be to introduce a small helper in\n`t/helper/` that converts binary data on stdin to hex on stdout? Something\nlike this:\n\n-- snip --\nFrom bee2a3c43c90683b3e86e1739361570cce76d382 Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin \nDate: Thu, 12 May 2022 17:24:50 +0200\nSubject: [PATCH] tests: add a helped to print a hexdump\n\nSigned-off-by: Johannes Schindelin \n---\n Makefile | 1 +\n t/helper/test-hexdump.c | 24 ++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 27 insertions(+)\n create mode 100644 t/helper/test-hexdump.c\n\ndiff --git a/Makefile b/Makefile\nindex 4a23508d16f..fc262f99a1f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -708,6 +708,7 @@ TEST_BUILTINS_OBJS += test-getcwd.o\n TEST_BUILTINS_OBJS += test-hash-speed.o\n TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n+TEST_BUILTINS_OBJS += test-hexdump.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\n TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\ndiff --git a/t/helper/test-hexdump.c b/t/helper/test-hexdump.c\nnew file mode 100644\nindex 00000000000..13f154d9fa7\n--- /dev/null\n+++ b/t/helper/test-hexdump.c\n@@ -0,0 +1,24 @@\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Read stdin and print a hexdump to stdout.\n+ */\n+int cmd__hexdump(int argc, const char **argv)\n+{\n+\tchar buf[1024];\n+\tssize_t i, len;\n+\n+\tfor (;;) {\n+\t\tlen = xread(0, buf, sizeof(buf));\n+\t\tif (len < 0)\n+\t\t\tdie_errno(\"failure reading stdin\");\n+\t\tif (!len)\n+\t\t\tbreak;\n+\n+\t\tfor (i = 0; i < len; i++)\n+\t\t\tprintf(\"%02x \", buf[i]);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 3ce5585e53a..44bd8269a07 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -35,6 +35,7 @@ static struct test_cmd cmds[] = {\n \t{ \"genzeros\", cmd__genzeros },\n \t{ \"getcwd\", cmd__getcwd },\n \t{ \"hashmap\", cmd__hashmap },\n+\t{ \"hexdump\", cmd__hexdump },\n \t{ \"hash-speed\", cmd__hash_speed },\n \t{ \"index-version\", cmd__index_version },\n \t{ \"json-writer\", cmd__json_writer },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9f0f5228508..8ec30136913 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -25,6 +25,7 @@ int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n+int cmd__hexdump(int argc, const char **argv);\n int cmd__hash_speed(int argc, const char **argv);\n int cmd__index_version(int argc, const char **argv);\n int cmd__json_writer(int argc, const char **argv);\n-- snap --\n\nOther than the `od` usage, this patch looks good to me.\n\nThank you very much for driving FSMonitor forward!\nDscho\n\n> +'\n> +\n> +# Is the spelling of an NFD pathname preserved on disk?\n> +#\n> +# 0000000 64 5f 65 cc 81\n> +#\n> +test_lazy_prereq UNICODE_NFD_PRESERVED '\n> +\tmkdir d_${utf8_nfd} &&\n> +\tls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n> +'\n> +\tmkdir c_${utf8_nfc} &&\n> +\tmkdir d_${utf8_nfd} &&\n> +\n> +# The following _DOUBLE_ forms are more for my curiosity,\n> +# but there may be quirks lurking when there are multiple\n> +# combining characters in non-canonical order.\n> +\n> +# Unicode also allows multiple combining characters\n> +# that can be decomposed in pieces.\n> +#\n> +# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n> +# UTF8(NFC): \\xe1 \\xbd \\xa7\n> +#\n> +# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n> +# U+0342 COMBINING GREEK PERISPOMENI\n> +# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n> +#\n> +# But U+1f61 decomposes into\n> +# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n> +# U+0314 COMBINING REVERSED COMMA ABOVE\n> +# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n> +#\n> +# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n> +#\n> +# Note that I've used the canonical ordering of the\n> +# combinining characters. It is also possible to\n> +# swap them. My testing shows that that non-standard\n> +# ordering also causes a collision in mkdir. However,\n> +# the resulting names don't draw correctly on the\n> +# terminal (implying that the on-disk format also has\n> +# them out of order).\n> +#\n> +greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n> +greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n> +greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n> +\n> +# See if a double decomposition also collides.\n> +#\n> +test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n> +\tmkdir trial_${greek_nfc} &&\n> +\tmkdir trial_${greek_nfd2}\n> +'\n> +\n> +# See if the NFC spelling appears on the disk.\n> +#\n> +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n> +\tmkdir c_${greek_nfc} &&\n> +\tls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n> +'\n> +\n> +# See if the NFD spelling appears on the disk.\n> +#\n> +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n> +\tmkdir d_${greek_nfd2} &&\n> +\tls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n> +'\n> +\n> +# The following is for debugging. I found it useful when\n> +# trying to understand the various (OS, FS) quirks WRT\n> +# Unicode and how composition/decomposition is handled.\n> +# For example, when trying to understand how (macOS, APFS)\n> +# and (macOS, HFS) and (macOS, FAT32) compare.\n> +#\n> +# It is rather noisy, so it is disabled by default.\n> +#\n> +if test \"$unicode_debug\" = \"true\"\n> +then\n> +\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n> +\tthen\n> +\t\techo NFC and NFD are distinct on this OS/filesystem.\n> +\telse\n> +\t\techo NFC and NFD are aliases on this OS/filesystem.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_NFC_PRESERVED\n> +\tthen\n> +\t\techo NFC maintains original spelling.\n> +\telse\n> +\t\techo NFC is modified.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_NFD_PRESERVED\n> +\tthen\n> +\t\techo NFD maintains original spelling.\n> +\telse\n> +\t\techo NFD is modified.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n> +\tthen\n> +\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n> +\telse\n> +\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n> +\tthen\n> +\t\techo Double NFC maintains original spelling.\n> +\telse\n> +\t\techo Double NFC is modified.\n> +\tfi\n> +\n> +\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n> +\tthen\n> +\t\techo Double NFD maintains original spelling.\n> +\telse\n> +\t\techo Double NFD is modified.\n> +\tfi\n> +fi\n> --\n> gitgitgadget\n>\n>\n"},{"id":"455206","messageId":"nycvar.QRO.7.76.6.2205121726260.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 00/28] Builtin FSMonitor Part 3","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-12T15:35:16Z","receivedAt":"2022-05-12T15:35:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> Here is version 6 of part 3 of FSMonitor.\n>\n> This version addresses:\n>\n> 1. Junio's comments on V5 23/28 WRT shell variable references and quoting\n> pathnames in the create_super and create_sub helper functions.\n>\n> 2. Ævar's comments on V4 4/27 (sorry I didn't see them until after I sent\n> V5) WRT somewhat blurry logic around the fsmonitor-settings and\n> detecting incompatible worktrees. I simplified things, but not to the\n> level that Ævar was suggesting. For example, in builtin/update-index.c\n> the suggestion was to detect incompatible FS before taking the lock on\n> the index, but the lock is taken before the CL args are parsed (because\n> update-index uses a custom version of parse_options_start()), so we\n> don't know yet whether the user passed --fsmonitor until much later and\n> that is what triggers the error/warning. I did replace the return 1 with\n> a die() so hopefully, we'll release the lock on the index like all of\n> the other errors in that function. I did try to better document the code\n> in update-index.c and in builtin/fsmonitor--daemon.c to explain how\n> things are supposed to work. So hopefully it'll be easier to review.\n>\n> 3. Also, in update-index and fsmonitor--daemon, I redid how the error\n> messages are printed, so that I could use die() in the cmd_*() functions\n> rather than having calls to error() hidden inside fsmonitor-settings.c.\n> I think that helped with the above cleanup.\n\nThank you _so_ much for keeping on the ball. I do see how much effort you\nhad to put into FSMonitor, what with three large patch series, plenty of\nreviews that necessitated plenty of iterations, but I heard from a couple\nof sides now just how important this feature is for users who work with\nlarge repositories.\n\nYour work truly has a great impact on Git users!\n\nI offered a couple of suggestions in my replies to individual patches,\nnothing majorly critical (except maybe the `wcscpy()`/`wcsncpy()` calls\nthat _might_ overrun their buffer in cornercases).\n\nHopefully you find the suggestions useful rather than annoying at this\nlate stage (you're already at iteration 6 of the third large patch series,\nafter all). I just didn't want us to run into any big surprises like when\nthe second FSMonitor patch series was integrated into `next` (finally!!!),\nonly to be dropped and needing to be replaced by yet another iteration\n(not so yay :-( ).\n\nAfter studying your patches for a few hours and then writing up my review\n(only being interrupted _once_ today, briefly, yay!), I am fairly happy\neven with the current shape of the series, and if you want to address my\nsuggestions and send out a seventh iteration of the patch series, I am\ncertain that it is ready for `next`.\n\nAgain, thank you _so much_ for keeping up the good work,\nDscho\n"},{"id":"455393","messageId":"59cfe53d-4529-b594-014c-ade72ba86c41@jeffhostetler.com","threadId":"57414","inReplyTo":"nycvar.QRO.7.76.6.2205121617460.352@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v6 01/28] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-05-17T19:26:56Z","receivedAt":"2022-05-17T19:27:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/12/22 10:20 AM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> On Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Teach FSMonitor daemon on Windows to recognize shortname paths as\n>> aliases of normal longname paths. FSMonitor clients, such as `git\n>> status`, should receive the longname spelling of changed files (when\n>> possible).\n>>\n[...]\n\n>> +/*\n>> + * See if the worktree root directory has shortnames enabled.\n>> + * This will help us decide if we need to do an expensive shortname\n>> + * to longname conversion on every notification event.\n>> + *\n>> + * We do not want to create a file to test this, so we assume that the\n>> + * root directory contains a \".git\" file or directory. (Our caller\n>> + * only calls us for the worktree root, so this should be fine.)\n>> + *\n>> + * Remember the spelling of the shortname for \".git\" if it exists.\n>> + */\n>> +static void check_for_shortnames(struct one_watch *watch)\n>> +{\n>> +\twchar_t buf_in[MAX_PATH + 1];\n>> +\twchar_t buf_out[MAX_PATH + 1];\n>> +\twchar_t *last_slash = NULL;\n>> +\twchar_t *last_bslash = NULL;\n>> +\twchar_t *last;\n>> +\n>> +\t/* build L\"/.git\" */\n>> +\twcscpy(buf_in, watch->wpath_longname);\n>> +\twcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n> \n> Could you use `wcsncpy()` here (with the appropriate length designed not\n> to overrun the `buf_in` buffer?\n> \n> Or even better: use `swprintf()` (which has a `count` parameter)? The\n> performance impact should be negligible because we only do this once,\n> right?\n\nThe RHS is a MAX_PATH buffer, so I don't think it was exploitable,\nbut yes it is good to make it explicit. Good catch. Thanks.\n\n\n>> +\n>> +\tif (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n>> +\t\treturn;\n>> +\n>> +\tlast_slash = wcsrchr(buf_out, L'/');\n>> +\tlast_bslash = wcsrchr(buf_out, L'\\\\');\n>> +\tif (last_slash > last_bslash)\n>> +\t\tlast = last_slash + 1;\n>> +\telse if (last_bslash)\n>> +\t\tlast = last_bslash + 1;\n>> +\telse\n>> +\t\tlast = buf_out;\n> \n> While this is all correct, I would find it clearer to write this as\n> following:\n> \n> \tfor (filename = p = buf_out; *p; p++)\n> \t\t/* We can be sure that `buf_out` does not end in a slash */\n> \t\tif (*p == L'/' || *p == '\\\\')\n> \t\t\tfilename = p + 1;\n\nsure.\n\n> \n>> +\n>> +\tif (!wcscmp(last, L\".git\"))\n>> +\t\treturn;\n>> +\n>> +\twatch->has_shortnames = 1;\n>> +\twcsncpy(watch->dotgit_shortname, last,\n>> +\t\tARRAY_SIZE(watch->dotgit_shortname));\n>> +\n>> +\t/*\n>> +\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n>> +\t * we should be able to avoid shortname to longname mapping on\n>> +\t * every notification event if the source string does not\n>> +\t * contain a \"~\".\n>> +\t *\n>> +\t * However, the documentation for GetLongPathNameW() says\n>> +\t * that there are filesystems that don't follow that pattern\n>> +\t * and warns against this optimization.\n>> +\t *\n>> +\t * Lets test this.\n>> +\t */\n>> +\tif (wcschr(watch->dotgit_shortname, L'~'))\n>> +\t\twatch->has_tilda = 1;\n>> +}\n>> +\n>> +enum get_relative_result {\n>> +\tGRR_NO_CONVERSION_NEEDED,\n>> +\tGRR_HAVE_CONVERSION,\n>> +\tGRR_SHUTDOWN,\n>> +};\n>> +\n>> +/*\n>> + * Info notification paths are relative to the root of the watch.\n>> + * If our CWD is still at the root, then we can use relative paths\n>> + * to convert from shortnames to longnames. If our process has a\n>> + * different CWD, then we need to construct an absolute path, do\n>> + * the conversion, and then return the root-relative portion.\n>> + *\n>> + * We use the longname form of the root as our basis and assume that\n>> + * it already has a trailing slash.\n>> + *\n>> + * `wpath_len` is in WCHARS not bytes.\n>> + */\n>> +static enum get_relative_result get_relative_longname(\n>> +\tstruct one_watch *watch,\n>> +\tconst wchar_t *wpath, DWORD wpath_len,\n>> +\twchar_t *wpath_longname)\n>> +{\n>> +\twchar_t buf_in[2 * MAX_PATH + 1];\n>> +\twchar_t buf_out[MAX_PATH + 1];\n>> +\tDWORD root_len;\n>> +\n>> +\t/* Build L\"/\" */\n>> +\troot_len = watch->wpath_longname_len;\n>> +\twcsncpy(buf_in, watch->wpath_longname, root_len);\n>> +\twcsncpy(buf_in + root_len, wpath, wpath_len);\n> \n> Here, too, I would like to have a check to prevent an overrun. Maybe\n> `swprintf()` again? I guess we could invent `xswprintf()` which would\n> return an error if the return value is -1 or if it used up the entire\n> buffer (i.e. if it overran).\n\nThe relative portion is not necessarily null terminated. It comes\nfrom the FILE_NOTIFY_INFOMATION buffer from the kernel. We could\nuse swprintf() here, to prevent the overflow, but we might miss the\nfact that it was truncated. I'll add checks to make sure the sum\nis within limits or give up.\n\n> \n>> +\tbuf_in[root_len + wpath_len] = 0;\n>> +\n>> +\t/*\n>> +\t * We don't actually know if the source pathname is a\n>> +\t * shortname or a longname. This routine allows either to be\n>> +\t * given as input.\n>> +\t */\n>> +\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n>> +\t\t/*\n>> +\t\t * The shortname to longname conversion can fail for\n>> +\t\t * various reasons, for example if the file has been\n>> +\t\t * deleted. (That is, if we just received a\n>> +\t\t * delete-file notification event and the file is\n>> +\t\t * already gone, we can't ask the file system to\n>> +\t\t * lookup the longname for it. Likewise, for moves\n>> +\t\t * and renames where we are given the old name.)\n>> +\t\t *\n>> +\t\t * Since deleting or moving a file or directory by its\n>> +\t\t * shortname is rather obscure, I'm going ignore the\n>> +\t\t * failure and ask the caller to report the original\n>> +\t\t * relative path. This seems kinder than failing here\n>> +\t\t * and forcing a resync. Besides, forcing a resync on\n>> +\t\t * every file/directory delete would effectively\n>> +\t\t * cripple monitoring.\n>> +\t\t *\n>> +\t\t * We might revisit this in the future.\n>> +\t\t */\n>> +\t\treturn GRR_NO_CONVERSION_NEEDED;\n>> +\t}\n>> +\n>> +\tif (!wcscmp(buf_in, buf_out)) {\n>> +\t\t/*\n>> +\t\t * The path does not have a shortname alias.\n>> +\t\t */\n>> +\t\treturn GRR_NO_CONVERSION_NEEDED;\n>> +\t}\n>> +\n>> +\tif (wcsncmp(buf_in, buf_out, root_len)) {\n>> +\t\t/*\n>> +\t\t * The spelling of the root directory portion of the computed\n>> +\t\t * longname has changed. This should not happen. Basically,\n>> +\t\t * it means that we don't know where (without recomputing the\n>> +\t\t * longname of just the root directory) to split out the\n>> +\t\t * relative path. Since this should not happen, I'm just\n>> +\t\t * going to let this fail and force a shutdown (because all\n>> +\t\t * subsequent events are probably going to see the same\n>> +\t\t * mismatch).\n>> +\t\t */\n>> +\t\treturn GRR_SHUTDOWN;\n>> +\t}\n>> +\n>> +\t/* Return the worktree root-relative portion of the longname. */\n>> +\n>> +\twcscpy(wpath_longname, buf_out + root_len);\n>> +\treturn GRR_HAVE_CONVERSION;\n>> +}\n>> +\n>> void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n>> {\n>> \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n>> @@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n>> \tDWORD share_mode =\n>> \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n>> \tHANDLE hDir;\n>> -\twchar_t wpath[MAX_PATH];\n>> +\tDWORD len_longname;\n>> +\twchar_t wpath[MAX_PATH + 1];\n>> +\twchar_t wpath_longname[MAX_PATH + 1];\n>>\n>> \tif (xutftowcs_path(wpath, path) < 0) {\n>> \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n>> @@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n>> \t\treturn NULL;\n>> \t}\n>>\n>> +\tif (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n>> +\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n>> +\t\t GetLastError(), path);\n>> +\t\tCloseHandle(hDir);\n>> +\t\treturn NULL;\n>> +\t}\n>> +\n>> +\tlen_longname = wcslen(wpath_longname);\n> \n> Let's assign the return value of `GetLongPathNameW()` to `len_longname`,\n> in case of success it contains the number of characters, too.\n\nGood idea. Thanks!\n\n> \n> The rest of the patch looks good to me!\n> \n> Thank you,\n> Dscho\n\nThanks for you attention to detail here.\nJeff\n"},{"id":"455394","messageId":"d2c909df-44f6-bc2c-ebbc-13ea51cf62fe@jeffhostetler.com","threadId":"57414","inReplyTo":"nycvar.QRO.7.76.6.2205121622030.352@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v6 16/28] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-05-17T19:48:51Z","receivedAt":"2022-05-17T19:48:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/12/22 11:05 AM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> On Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Create another thread to watch over the daemon process and\n>> automatically shut it down if necessary.\n>>\n[...]\n\n>> The platform-specific code for Windows sets up enough of the\n>> WaitForMultipleObjects() machinery to watch for system and/or custom\n>> events. Currently, the set of wait handles only includes our custom\n>> shutdown event (sent from our other theads). Later commits in this\n>> series will extend the set of wait handles to monitor other\n>> conditions.\n[...]\n\n>> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\n>> new file mode 100644\n>> index 00000000000..94b1d020f25\n>> --- /dev/null\n>> +++ b/compat/fsmonitor/fsm-health-win32.c\n>> @@ -0,0 +1,72 @@\n>> +#include \"cache.h\"\n>> +#include \"config.h\"\n>> +#include \"fsmonitor.h\"\n>> +#include \"fsm-health.h\"\n>> +#include \"fsmonitor--daemon.h\"\n>> +\n>> +struct fsm_health_data\n>> +{\n>> +\tHANDLE hEventShutdown;\n>> +\n>> +\tHANDLE hHandles[1]; /* the array does not own these handles */\n>> +#define HEALTH_SHUTDOWN 0\n> \n> How about defining `HANDLE hHandles[HEALTH_SHUTDOWN + 1]` to indicate that\n> the constant is used as an offset into `hHandles`?\n> \n>> +\tint nr_handles; /* number of active event handles */\n\nI think I'd like to keep this one as is. It matches the style that\nI used in `fsm-listen-win32.c` where I have 3 listener handles.\n\nGranted, it does look a little odd when there is only 1 handle in the\narray. But the idea was to allow new handles to be added as we want\nto watch more things.\n\nIt might be clearer (in both of them) to define the array in\nterms of an enum rather than a local list of #define's.\nBut I'm not sure it matters.\n\nJeff\n\n\n"},{"id":"455395","messageId":"c716b1d6-a0a8-6d03-dfe5-70613e4ed942@jeffhostetler.com","threadId":"57414","inReplyTo":"nycvar.QRO.7.76.6.2205121708400.352@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v6 20/28] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-05-17T20:17:39Z","receivedAt":"2022-05-17T20:17:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/12/22 11:08 AM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> On Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> Teach Git to perform binary search over the cache-entries for a directory\n>> notification and then linearly scan forward to find the immediate children.\n>>\n[...]\n\n>> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n>> {\n>> \tint i, len = strlen(name);\n>> -\tif (name[len - 1] == '/') {\n>> +\tint pos = index_name_pos(istate, name, len);\n>> +\n>> +\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n>> +\t\t\t name, pos);\n>>\n[...]\n\n>> +\tif (name[len - 1] == '/') {\n[...]\n>> \t}\n\n>> @@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n>> \t * Mark the untracked cache dirty even if it wasn't found in the index\n>> \t * as it could be a new untracked file.\n>> \t */\n>> -\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n> \n> Did you mean to remove this statement in this patch? Not a big issue, but\n> I wonder what the rationale for it is, and since I have an inquisitive\n> mind, I figured I'd just ask.\n\nI just moved it to the top of the function. That lets me see `name`\nbefore it is modified in one of the else arms (it was helpful to see\nwhether the daemon sent a trailing slash or not). And I also wanted\nto see the computed value of `pos` (before the \"-pos - 1\" tricks).\n\nJeff\n"},{"id":"455397","messageId":"adaffdd2-9294-0ddb-5eac-d8b90ae13f20@jeffhostetler.com","threadId":"57414","inReplyTo":"nycvar.QRO.7.76.6.2205121711300.352@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v6 26/28] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-05-17T21:14:11Z","receivedAt":"2022-05-17T21:14:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/12/22 11:26 AM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> On Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler \n>>\n[...]\n>> +#\n>> +test_lazy_prereq UNICODE_NFC_PRESERVED '\n>> +\tmkdir c_${utf8_nfc} &&\n>> +\tls | od -t x1 | grep \"63 *5f *c3 *a9\"\n> \n> As far as I can see, this would be the first usage of `od` in the test\n> suite. I'd actually like to reduce our dependency on Unix-y tools, not\n> increase it.\n> \n> One thing we could do would be to imitate t4030, and introduce a shell\n> function that calls Perl, something like:\n> \n> \tbin2hex () {\n> \t\tperl -e '\n> \t\t\t$/ = undef;\n> \t\t\t$_ = <>;\n> \t\t\ts/./sprintf(\"%02x \", ord($&))/ge;\n> \t\t\tprint $_\n> \t\t'\n> \t}\n> \n> But it is a thorn in my side for quite a few years already that we\n> _require_ Perl, even in NO_PERL builds.\n> \n> So maybe a much better idea would be to introduce a small helper in\n> `t/helper/` that converts binary data on stdin to hex on stdout? Something\n> like this:\n> \n\nYeah, lets add the hexdump helper.\n\nThanks\nJeff\n"},{"id":"455817","messageId":"1bf2e36b6adb127dd1654bba0b6ba84f626535a0.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 02/30] t7527: test FSMonitor on repos with Unicode root paths","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:17Z","receivedAt":"2022-05-23T20:12:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate some test repos with UTF8 characters in the pathname of the\nroot directory and verify that the builtin FSMonitor can watch them.\n\nThis test is mainly for Windows where we need to avoid `*A()`\nroutines.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1be21785162..12655958e71 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -671,4 +671,27 @@ do\n \tdone\n done\n \n+# Test Unicode UTF-8 characters in the pathname of the working\n+# directory root. Use of \"*A()\" routines rather than \"*W()\" routines\n+# on Windows can sometimes lead to odd failures.\n+#\n+u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n+u2=$(printf \"u_e2_99_ab__\\xE2\\x99\\xAB\")\n+u_values=\"$u1 $u2\"\n+for u in $u_values\n+do\n+\ttest_expect_success \"unicode in repo root path: $u\" '\n+\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n+\n+\t\tgit init \"$u\" &&\n+\t\techo 1 >\"$u\"/file1 &&\n+\t\tgit -C \"$u\" add file1 &&\n+\t\tgit -C \"$u\" config core.fsmonitor true &&\n+\n+\t\tstart_daemon -C \"$u\" &&\n+\t\tgit -C \"$u\" status >actual &&\n+\t\tgrep \"new file: file1\" actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"455818","messageId":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v6.git.1650662994.gitgitgadget@gmail.com","subject":"[PATCH v7 00/30] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:15Z","receivedAt":"2022-05-23T20:13:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 7 of part 3 of FSMonitor.\n\nThis version mainly addresses comments received on V6.\n\n 1. A few typos.\n 2. Cleanup of some wide-char string manipulations in Windows code.\n 3. Fix some coding-style issues in the unit tests: don't capitalize test\n case description and single-quote the descriptions when possible.\n 4. Add t/helper/test-hexdump.c and use that in t7527 rather than\n /usr/bin/od in the Unicode/UTF8 compatibility tests for macOS.\n 5. Refactor the implicit daemon shutdown tests to retry a few times rather\n than a fixed sleep. This helps avoid races in the CI build runs.\n\nHere is a range-diff against V6:\n\n 1: 8b7c5f4e23 ! 1: 26144c5865 fsm-listen-win32: handle shortnames\n @@ compat/fsmonitor/fsm-listen-win32.c: struct one_watch\n + * clients.)\n + */\n + BOOL has_shortnames;\n -+ BOOL has_tilda;\n ++ BOOL has_tilde;\n + wchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI\n +{\n + wchar_t buf_in[MAX_PATH + 1];\n + wchar_t buf_out[MAX_PATH + 1];\n -+ wchar_t *last_slash = NULL;\n -+ wchar_t *last_bslash = NULL;\n + wchar_t *last;\n ++ wchar_t *p;\n +\n + /* build L\"/.git\" */\n -+ wcscpy(buf_in, watch->wpath_longname);\n -+ wcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n ++ swprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%s.git\",\n ++ watch->wpath_longname);\n +\n -+ if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n ++ if (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)))\n + return;\n +\n -+ last_slash = wcsrchr(buf_out, L'/');\n -+ last_bslash = wcsrchr(buf_out, L'\\\\');\n -+ if (last_slash > last_bslash)\n -+ last = last_slash + 1;\n -+ else if (last_bslash)\n -+ last = last_bslash + 1;\n -+ else\n -+ last = buf_out;\n ++ /*\n ++ * Get the final filename component of the shortpath.\n ++ * We know that the path does not have a final slash.\n ++ */\n ++ for (last = p = buf_out; *p; p++)\n ++ if (*p == L'/' || *p == '\\\\')\n ++ last = p + 1;\n +\n + if (!wcscmp(last, L\".git\"))\n + return;\n @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI\n + * Lets test this.\n + */\n + if (wcschr(watch->dotgit_shortname, L'~'))\n -+ watch->has_tilda = 1;\n ++ watch->has_tilde = 1;\n +}\n +\n +enum get_relative_result {\n @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI\n +static enum get_relative_result get_relative_longname(\n + struct one_watch *watch,\n + const wchar_t *wpath, DWORD wpath_len,\n -+ wchar_t *wpath_longname)\n ++ wchar_t *wpath_longname, size_t bufsize_wpath_longname)\n +{\n + wchar_t buf_in[2 * MAX_PATH + 1];\n + wchar_t buf_out[MAX_PATH + 1];\n + DWORD root_len;\n ++ DWORD out_len;\n +\n -+ /* Build L\"/\" */\n ++ /*\n ++ * Build L\"/\"\n ++ * Note that the might not be null terminated\n ++ * so we avoid swprintf() constructions.\n ++ */\n + root_len = watch->wpath_longname_len;\n ++ if (root_len + wpath_len >= ARRAY_SIZE(buf_in)) {\n ++ /*\n ++ * This should not happen. We cannot append the observed\n ++ * relative path onto the end of the worktree root path\n ++ * without overflowing the buffer. Just give up.\n ++ */\n ++ return GRR_SHUTDOWN;\n ++ }\n + wcsncpy(buf_in, watch->wpath_longname, root_len);\n + wcsncpy(buf_in + root_len, wpath, wpath_len);\n + buf_in[root_len + wpath_len] = 0;\n +\n + /*\n + * We don't actually know if the source pathname is a\n -+ * shortname or a longname. This routine allows either to be\n -+ * given as input.\n ++ * shortname or a longname. This Windows routine allows\n ++ * either to be given as input.\n + */\n -+ if (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n ++ out_len = GetLongPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out));\n ++ if (!out_len) {\n + /*\n + * The shortname to longname conversion can fail for\n + * various reasons, for example if the file has been\n @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI\n + return GRR_SHUTDOWN;\n + }\n +\n ++ if (out_len - root_len >= bufsize_wpath_longname) {\n ++ /*\n ++ * This should not happen. We cannot copy the root-relative\n ++ * portion of the path into the provided buffer without an\n ++ * overrun. Just give up.\n ++ */\n ++ return GRR_SHUTDOWN;\n ++ }\n ++\n + /* Return the worktree root-relative portion of the longname. */\n +\n + wcscpy(wpath_longname, buf_out + root_len);\n @@ compat/fsmonitor/fsm-listen-win32.c: static struct one_watch *create_watch(struc\n return NULL;\n }\n \n -+ if (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n ++ len_longname = GetLongPathNameW(wpath, wpath_longname,\n ++ ARRAY_SIZE(wpath_longname));\n ++ if (!len_longname) {\n + error(_(\"[GLE %ld] could not get longname of '%s'\"),\n + GetLastError(), path);\n + CloseHandle(hDir);\n + return NULL;\n + }\n +\n -+ len_longname = wcslen(wpath_longname);\n + if (wpath_longname[len_longname - 1] != L'/' &&\n + wpath_longname[len_longname - 1] != L'\\\\') {\n + wpath_longname[len_longname++] = L'/';\n @@ compat/fsmonitor/fsm-listen-win32.c: static int process_worktree_events(struct f\n - case IS_INSIDE_DOT_GIT:\n - /* ignore everything inside of \"/.git/\" */\n - break;\n -+ if (watch->has_tilda && !wcschr(wpath, L'~')) {\n ++ if (watch->has_tilde && !wcschr(wpath, L'~')) {\n + /*\n -+ * Shortnames on this filesystem have tildas\n ++ * Shortnames on this filesystem have tildes\n + * and the notification path does not have\n + * one, so we assume that it is a longname.\n + */\n @@ compat/fsmonitor/fsm-listen-win32.c: static int process_worktree_events(struct f\n - \"fsm-listen/dotgit\",\n - \"removed\");\n + grr = get_relative_longname(watch, wpath, wpath_len,\n -+ wpath_longname);\n ++ wpath_longname,\n ++ ARRAY_SIZE(wpath_longname));\n + switch (grr) {\n + case GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n + break;\n 2: 5b246bec24 ! 2: 1bf2e36b6a t7527: test FSMonitor on repos with Unicode root paths\n @@ t/t7527-builtin-fsmonitor.sh: do\n +u_values=\"$u1 $u2\"\n +for u in $u_values\n +do\n -+ test_expect_success \"Unicode in repo root path: $u\" '\n ++ test_expect_success \"unicode in repo root path: $u\" '\n + test_when_finished \"stop_daemon_delete_repo $u\" &&\n +\n + git init \"$u\" &&\n 3: 8a474d6999 = 3: 4bca494bb2 t/helper/fsmonitor-client: create stress test\n 4: 72b94acd5f ! 4: 663deabc3f fsmonitor-settings: bare repos are incompatible with FSMonitor\n @@ fsmonitor-settings.h\n + */\n +enum fsmonitor_reason {\n + FSMONITOR_REASON_UNTESTED = 0,\n -+ FSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n ++ FSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n + FSMONITOR_REASON_BARE,\n +};\n +\n 5: 2e225c3f4f = 5: 7cb0180a1e fsmonitor-settings: stub in Win32-specific incompatibility checking\n 6: e0d3bdf755 ! 6: 9774faddc4 fsmonitor-settings: VFS for Git virtual repos are incompatible\n @@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repo\n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_reason {\n FSMONITOR_REASON_UNTESTED = 0,\n - FSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n + FSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n FSMONITOR_REASON_BARE,\n + FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n 7: c50ed29a31 = 7: f7ef7dcffc fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: 1f5b772d42 ! 8: dc2dfd6793 fsmonitor-settings: remote repos on macOS are incompatible\n @@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repo\n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_reason {\n FSMONITOR_REASON_UNTESTED = 0,\n - FSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n + FSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n FSMONITOR_REASON_BARE,\n + FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n + FSMONITOR_REASON_REMOTE,\n 9: 495e54049b = 9: 5627038aaa fsmonitor-settings: remote repos on Windows are incompatible\n10: 4b52083698 = 10: 9a12cc78b5 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n11: d4a4263d37 = 11: aaff000cec unpack-trees: initialize fsmonitor_has_run_once in o->result\n12: f4feb00ec2 = 12: 4f2b15d3d1 fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n13: dbb983fd9d = 13: 427dec412a fsmonitor--daemon: cd out of worktree root\n14: ae90b99ea9 = 14: 51b266b06e fsmonitor--daemon: prepare for adding health thread\n15: b6c5800095 = 15: 594e0ae243 fsmonitor--daemon: rename listener thread related variables\n16: 32fc6ba743 = 16: c2b5c02ed3 fsmonitor--daemon: stub in health thread\n17: 77bc037481 = 17: 46a5ae2a63 fsm-health-win32: add polling framework to monitor daemon health\n18: b06edd995e = 18: 7cf1be5f8e fsm-health-win32: force shutdown daemon if worktree root moves\n19: 1bd5f34624 = 19: 95cf1299d4 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n20: 48af0813de = 20: b020bfb456 fsmonitor: optimize processing of directory events\n21: a9b35e770f = 21: d058d7e0c0 t7527: FSMonitor tests for directory moves\n22: 26308936af = 22: f5dac28681 t/perf/p7527: add perf test for builtin FSMonitor\n23: d0e25f6bac ! 23: 92f5c0d2c8 fsmonitor: never set CE_FSMONITOR_VALID on submodules\n @@ t/t7527-builtin-fsmonitor.sh: do\n + git -C super/dir_1/dir_2/sub clean -d -f\n +}\n +\n -+test_expect_success \"Submodule always visited\" '\n ++test_expect_success 'submodule always visited' '\n + test_when_finished \"git -C super fsmonitor--daemon stop; \\\n + rm -rf super; \\\n + rm -rf sub\" &&\n24: 410dd2d292 ! 24: 40b80adbb3 t7527: test FSMonitor on case insensitive+preserving file system\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## t/t7527-builtin-fsmonitor.sh ##\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'submodule always visited' '\n my_match_and_clean\n '\n \n25: cd7c55b0d3 = 25: b93f064269 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n -: ---------- > 26: 6f2e935f14 t/helper/hexdump: add helper to print hexdump of stdin\n26: 8278f32c4d ! 27: 6a83086995 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n @@ t/lib-unicode-nfc-nfd.sh (new)\n +# and on APFS, NFC paths are preserved. As we have established\n +# above, this is independent of \"composition sensitivity\".\n +#\n -+# 0000000 63 5f c3 a9\n -+#\n -+# (/usr/bin/od output contains different amount of whitespace\n -+# on different platforms, so we need the wildcards here.)\n -+#\n +test_lazy_prereq UNICODE_NFC_PRESERVED '\n + mkdir c_${utf8_nfc} &&\n -+ ls | od -t x1 | grep \"63 *5f *c3 *a9\"\n ++ ls | test-tool hexdump | grep \"63 5f c3 a9\"\n +'\n +\n +# Is the spelling of an NFD pathname preserved on disk?\n +#\n -+# 0000000 64 5f 65 cc 81\n -+#\n +test_lazy_prereq UNICODE_NFD_PRESERVED '\n + mkdir d_${utf8_nfd} &&\n -+ ls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n ++ ls | test-tool hexdump | grep \"64 5f 65 cc 81\"\n +'\n -+ mkdir c_${utf8_nfc} &&\n -+ mkdir d_${utf8_nfd} &&\n +\n +# The following _DOUBLE_ forms are more for my curiosity,\n +# but there may be quirks lurking when there are multiple\n @@ t/lib-unicode-nfc-nfd.sh (new)\n +#\n +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n + mkdir c_${greek_nfc} &&\n -+ ls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n ++ ls | test-tool hexdump | grep \"63 5f e1 bd a7\"\n +'\n +\n +# See if the NFD spelling appears on the disk.\n +#\n +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n + mkdir d_${greek_nfd2} &&\n -+ ls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n ++ ls | test-tool hexdump | grep \"64 5f cf 89 cc 94 cd 82\"\n +'\n +\n +# The following is for debugging. I found it useful when\n27: 4efb3a4383 = 28: f9a7869d20 t7527: test Unicode NFC/NFD handling on MacOS\n28: df1b4f3a80 ! 29: 9fc7c97092 fsmonitor--daemon: allow --super-prefix argument\n @@ git.c: static struct cmd_struct commands[] = {\n { \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n \n ## t/t7527-builtin-fsmonitor.sh ##\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'submodule always visited' '\n my_match_and_clean\n '\n \n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n + tr '\\047' Q <$1 | grep -e \"$msg\"\n +}\n +\n -+test_expect_success \"Stray Submodule super-prefix warning\" '\n ++test_expect_success \"stray submodule super-prefix warning\" '\n + test_when_finished \"rm -rf super; \\\n + rm -rf sub; \\\n + rm super-sub.trace\" &&\n -: ---------- > 30: ca833ecc7a t7527: improve implicit shutdown testing in fsmonitor--daemon\n\n\nJeff Hostetler (30):\n fsm-listen-win32: handle shortnames\n t7527: test FSMonitor on repos with Unicode root paths\n t/helper/fsmonitor-client: create stress test\n fsmonitor-settings: bare repos are incompatible with FSMonitor\n fsmonitor-settings: stub in Win32-specific incompatibility checking\n fsmonitor-settings: VFS for Git virtual repos are incompatible\n fsmonitor-settings: stub in macOS-specific incompatibility checking\n fsmonitor-settings: remote repos on macOS are incompatible\n fsmonitor-settings: remote repos on Windows are incompatible\n fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n unpack-trees: initialize fsmonitor_has_run_once in o->result\n fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n fsmonitor--daemon: cd out of worktree root\n fsmonitor--daemon: prepare for adding health thread\n fsmonitor--daemon: rename listener thread related variables\n fsmonitor--daemon: stub in health thread\n fsm-health-win32: add polling framework to monitor daemon health\n fsm-health-win32: force shutdown daemon if worktree root moves\n fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n fsmonitor: optimize processing of directory events\n t7527: FSMonitor tests for directory moves\n t/perf/p7527: add perf test for builtin FSMonitor\n fsmonitor: never set CE_FSMONITOR_VALID on submodules\n t7527: test FSMonitor on case insensitive+preserving file system\n fsmonitor: on macOS also emit NFC spelling for NFD pathname\n t/helper/hexdump: add helper to print hexdump of stdin\n t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n t7527: test Unicode NFC/NFD handling on MacOS\n fsmonitor--daemon: allow --super-prefix argument\n t7527: improve implicit shutdown testing in fsmonitor--daemon\n\n Makefile | 20 +-\n builtin/fsmonitor--daemon.c | 116 ++++++-\n builtin/update-index.c | 16 +\n compat/fsmonitor/fsm-health-darwin.c | 24 ++\n compat/fsmonitor/fsm-health-win32.c | 278 ++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 +++\n compat/fsmonitor/fsm-listen-darwin.c | 122 ++++++-\n compat/fsmonitor/fsm-listen-win32.c | 436 ++++++++++++++++++++-----\n compat/fsmonitor/fsm-listen.h | 2 +-\n compat/fsmonitor/fsm-settings-darwin.c | 89 +++++\n compat/fsmonitor/fsm-settings-win32.c | 137 ++++++++\n config.mak.uname | 5 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 11 +-\n fsmonitor-settings.c | 167 ++++++++--\n fsmonitor-settings.h | 33 ++\n fsmonitor.c | 73 ++++-\n fsmonitor.h | 11 +\n git.c | 2 +-\n t/helper/test-fsmonitor-client.c | 106 ++++++\n t/helper/test-hexdump.c | 24 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/lib-unicode-nfc-nfd.sh | 158 +++++++++\n t/perf/p7527-builtin-fsmonitor.sh | 257 +++++++++++++++\n t/t7519-status-fsmonitor.sh | 32 ++\n t/t7527-builtin-fsmonitor.sh | 401 ++++++++++++++++++++++-\n unpack-trees.c | 1 +\n 28 files changed, 2429 insertions(+), 149 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n create mode 100644 t/helper/test-hexdump.c\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\n\nbase-commit: 5eb696daba2fe108d4d9ba2ccf4b357447ef9946\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v7\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v7\nPull-Request: https://github.com/gitgitgadget/git/pull/1143\n\nRange-diff vs v6:\n\n 1: 8b7c5f4e234 ! 1: 26144c58659 fsm-listen-win32: handle shortnames\n @@ compat/fsmonitor/fsm-listen-win32.c: struct one_watch\n +\t * clients.)\n +\t */\n +\tBOOL has_shortnames;\n -+\tBOOL has_tilda;\n ++\tBOOL has_tilde;\n +\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n @@ compat/fsmonitor/fsm-listen-win32.c: normalize:\n +{\n +\twchar_t buf_in[MAX_PATH + 1];\n +\twchar_t buf_out[MAX_PATH + 1];\n -+\twchar_t *last_slash = NULL;\n -+\twchar_t *last_bslash = NULL;\n +\twchar_t *last;\n ++\twchar_t *p;\n +\n +\t/* build L\"/.git\" */\n -+\twcscpy(buf_in, watch->wpath_longname);\n -+\twcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n ++\tswprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%s.git\",\n ++\t\t watch->wpath_longname);\n +\n -+\tif (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n ++\tif (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)))\n +\t\treturn;\n +\n -+\tlast_slash = wcsrchr(buf_out, L'/');\n -+\tlast_bslash = wcsrchr(buf_out, L'\\\\');\n -+\tif (last_slash > last_bslash)\n -+\t\tlast = last_slash + 1;\n -+\telse if (last_bslash)\n -+\t\tlast = last_bslash + 1;\n -+\telse\n -+\t\tlast = buf_out;\n ++\t/*\n ++\t * Get the final filename component of the shortpath.\n ++\t * We know that the path does not have a final slash.\n ++\t */\n ++\tfor (last = p = buf_out; *p; p++)\n ++\t\tif (*p == L'/' || *p == '\\\\')\n ++\t\t\tlast = p + 1;\n +\n +\tif (!wcscmp(last, L\".git\"))\n +\t\treturn;\n @@ compat/fsmonitor/fsm-listen-win32.c: normalize:\n +\t * Lets test this.\n +\t */\n +\tif (wcschr(watch->dotgit_shortname, L'~'))\n -+\t\twatch->has_tilda = 1;\n ++\t\twatch->has_tilde = 1;\n +}\n +\n +enum get_relative_result {\n @@ compat/fsmonitor/fsm-listen-win32.c: normalize:\n +static enum get_relative_result get_relative_longname(\n +\tstruct one_watch *watch,\n +\tconst wchar_t *wpath, DWORD wpath_len,\n -+\twchar_t *wpath_longname)\n ++\twchar_t *wpath_longname, size_t bufsize_wpath_longname)\n +{\n +\twchar_t buf_in[2 * MAX_PATH + 1];\n +\twchar_t buf_out[MAX_PATH + 1];\n +\tDWORD root_len;\n ++\tDWORD out_len;\n +\n -+\t/* Build L\"/\" */\n ++\t/*\n ++\t * Build L\"/\"\n ++\t * Note that the might not be null terminated\n ++\t * so we avoid swprintf() constructions.\n ++\t */\n +\troot_len = watch->wpath_longname_len;\n ++\tif (root_len + wpath_len >= ARRAY_SIZE(buf_in)) {\n ++\t\t/*\n ++\t\t * This should not happen. We cannot append the observed\n ++\t\t * relative path onto the end of the worktree root path\n ++\t\t * without overflowing the buffer. Just give up.\n ++\t\t */\n ++\t\treturn GRR_SHUTDOWN;\n ++\t}\n +\twcsncpy(buf_in, watch->wpath_longname, root_len);\n +\twcsncpy(buf_in + root_len, wpath, wpath_len);\n +\tbuf_in[root_len + wpath_len] = 0;\n +\n +\t/*\n +\t * We don't actually know if the source pathname is a\n -+\t * shortname or a longname. This routine allows either to be\n -+\t * given as input.\n ++\t * shortname or a longname. This Windows routine allows\n ++\t * either to be given as input.\n +\t */\n -+\tif (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {\n ++\tout_len = GetLongPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out));\n ++\tif (!out_len) {\n +\t\t/*\n +\t\t * The shortname to longname conversion can fail for\n +\t\t * various reasons, for example if the file has been\n @@ compat/fsmonitor/fsm-listen-win32.c: normalize:\n +\t\treturn GRR_SHUTDOWN;\n +\t}\n +\n ++\tif (out_len - root_len >= bufsize_wpath_longname) {\n ++\t\t/*\n ++\t\t * This should not happen. We cannot copy the root-relative\n ++\t\t * portion of the path into the provided buffer without an\n ++\t\t * overrun. Just give up.\n ++\t\t */\n ++\t\treturn GRR_SHUTDOWN;\n ++\t}\n ++\n +\t/* Return the worktree root-relative portion of the longname. */\n +\n +\twcscpy(wpath_longname, buf_out + root_len);\n @@ compat/fsmonitor/fsm-listen-win32.c: static struct one_watch *create_watch(struc\n \t\treturn NULL;\n \t}\n \n -+\tif (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {\n ++\tlen_longname = GetLongPathNameW(wpath, wpath_longname,\n ++\t\t\t\t\tARRAY_SIZE(wpath_longname));\n ++\tif (!len_longname) {\n +\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n +\t\t GetLastError(), path);\n +\t\tCloseHandle(hDir);\n +\t\treturn NULL;\n +\t}\n +\n -+\tlen_longname = wcslen(wpath_longname);\n +\tif (wpath_longname[len_longname - 1] != L'/' &&\n +\t wpath_longname[len_longname - 1] != L'\\\\') {\n +\t\twpath_longname[len_longname++] = L'/';\n @@ compat/fsmonitor/fsm-listen-win32.c: static int process_worktree_events(struct f\n -\t\tcase IS_INSIDE_DOT_GIT:\n -\t\t\t/* ignore everything inside of \"/.git/\" */\n -\t\t\tbreak;\n -+\t\t\tif (watch->has_tilda && !wcschr(wpath, L'~')) {\n ++\t\t\tif (watch->has_tilde && !wcschr(wpath, L'~')) {\n +\t\t\t\t/*\n -+\t\t\t\t * Shortnames on this filesystem have tildas\n ++\t\t\t\t * Shortnames on this filesystem have tildes\n +\t\t\t\t * and the notification path does not have\n +\t\t\t\t * one, so we assume that it is a longname.\n +\t\t\t\t */\n @@ compat/fsmonitor/fsm-listen-win32.c: static int process_worktree_events(struct f\n -\t\t\t\t\t\t \"fsm-listen/dotgit\",\n -\t\t\t\t\t\t \"removed\");\n +\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n -+\t\t\t\t\t\t wpath_longname);\n ++\t\t\t\t\t\t wpath_longname,\n ++\t\t\t\t\t\t ARRAY_SIZE(wpath_longname));\n +\t\t\tswitch (grr) {\n +\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n +\t\t\t\tbreak;\n 2: 5b246bec247 ! 2: 1bf2e36b6ad t7527: test FSMonitor on repos with Unicode root paths\n @@ t/t7527-builtin-fsmonitor.sh: do\n +u_values=\"$u1 $u2\"\n +for u in $u_values\n +do\n -+\ttest_expect_success \"Unicode in repo root path: $u\" '\n ++\ttest_expect_success \"unicode in repo root path: $u\" '\n +\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n +\n +\t\tgit init \"$u\" &&\n 3: 8a474d69999 = 3: 4bca494bb22 t/helper/fsmonitor-client: create stress test\n 4: 72b94acd5fe ! 4: 663deabc3f6 fsmonitor-settings: bare repos are incompatible with FSMonitor\n @@ fsmonitor-settings.h\n + */\n +enum fsmonitor_reason {\n +\tFSMONITOR_REASON_UNTESTED = 0,\n -+\tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n ++\tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n +\tFSMONITOR_REASON_BARE,\n +};\n +\n 5: 2e225c3f4f2 = 5: 7cb0180a1ed fsmonitor-settings: stub in Win32-specific incompatibility checking\n 6: e0d3bdf7556 ! 6: 9774faddc45 fsmonitor-settings: VFS for Git virtual repos are incompatible\n @@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repo\n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n - \tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n + \tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n \tFSMONITOR_REASON_BARE,\n +\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n 7: c50ed29a310 = 7: f7ef7dcffc8 fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: 1f5b772d42a ! 8: dc2dfd67931 fsmonitor-settings: remote repos on macOS are incompatible\n @@ fsmonitor-settings.c: char *fsm_settings__get_incompatible_msg(const struct repo\n ## fsmonitor-settings.h ##\n @@ fsmonitor-settings.h: enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n - \tFSMONITOR_REASON_OK, /* no incompatibility or when disbled */\n + \tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n \tFSMONITOR_REASON_BARE,\n +\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n +\tFSMONITOR_REASON_REMOTE,\n 9: 495e54049b4 = 9: 5627038aaa3 fsmonitor-settings: remote repos on Windows are incompatible\n 10: 4b52083698c = 10: 9a12cc78b5d fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n 11: d4a4263d379 = 11: aaff000cecb unpack-trees: initialize fsmonitor_has_run_once in o->result\n 12: f4feb00ec2b = 12: 4f2b15d3d1f fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n 13: dbb983fd9d0 = 13: 427dec412a5 fsmonitor--daemon: cd out of worktree root\n 14: ae90b99ea9b = 14: 51b266b06e1 fsmonitor--daemon: prepare for adding health thread\n 15: b6c5800095f = 15: 594e0ae243d fsmonitor--daemon: rename listener thread related variables\n 16: 32fc6ba7437 = 16: c2b5c02ed38 fsmonitor--daemon: stub in health thread\n 17: 77bc037481a = 17: 46a5ae2a635 fsm-health-win32: add polling framework to monitor daemon health\n 18: b06edd995ea = 18: 7cf1be5f8e2 fsm-health-win32: force shutdown daemon if worktree root moves\n 19: 1bd5f346248 = 19: 95cf1299d44 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n 20: 48af0813dec = 20: b020bfb4568 fsmonitor: optimize processing of directory events\n 21: a9b35e770f3 = 21: d058d7e0c08 t7527: FSMonitor tests for directory moves\n 22: 26308936af9 = 22: f5dac286812 t/perf/p7527: add perf test for builtin FSMonitor\n 23: d0e25f6bac6 ! 23: 92f5c0d2c8b fsmonitor: never set CE_FSMONITOR_VALID on submodules\n @@ t/t7527-builtin-fsmonitor.sh: do\n +\tgit -C super/dir_1/dir_2/sub clean -d -f\n +}\n +\n -+test_expect_success \"Submodule always visited\" '\n ++test_expect_success 'submodule always visited' '\n +\ttest_when_finished \"git -C super fsmonitor--daemon stop; \\\n +\t\t\t rm -rf super; \\\n +\t\t\t rm -rf sub\" &&\n 24: 410dd2d2920 ! 24: 40b80adbb31 t7527: test FSMonitor on case insensitive+preserving file system\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## t/t7527-builtin-fsmonitor.sh ##\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'submodule always visited' '\n \tmy_match_and_clean\n '\n \n 25: cd7c55b0d38 = 25: b93f0642699 fsmonitor: on macOS also emit NFC spelling for NFD pathname\n -: ----------- > 26: 6f2e935f148 t/helper/hexdump: add helper to print hexdump of stdin\n 26: 8278f32c4d8 ! 27: 6a830869954 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n @@ t/lib-unicode-nfc-nfd.sh (new)\n +# and on APFS, NFC paths are preserved. As we have established\n +# above, this is independent of \"composition sensitivity\".\n +#\n -+# 0000000 63 5f c3 a9\n -+#\n -+# (/usr/bin/od output contains different amount of whitespace\n -+# on different platforms, so we need the wildcards here.)\n -+#\n +test_lazy_prereq UNICODE_NFC_PRESERVED '\n +\tmkdir c_${utf8_nfc} &&\n -+\tls | od -t x1 | grep \"63 *5f *c3 *a9\"\n ++\tls | test-tool hexdump | grep \"63 5f c3 a9\"\n +'\n +\n +# Is the spelling of an NFD pathname preserved on disk?\n +#\n -+# 0000000 64 5f 65 cc 81\n -+#\n +test_lazy_prereq UNICODE_NFD_PRESERVED '\n +\tmkdir d_${utf8_nfd} &&\n -+\tls | od -t x1 | grep \"64 *5f *65 *cc *81\"\n ++\tls | test-tool hexdump | grep \"64 5f 65 cc 81\"\n +'\n -+\tmkdir c_${utf8_nfc} &&\n -+\tmkdir d_${utf8_nfd} &&\n +\n +# The following _DOUBLE_ forms are more for my curiosity,\n +# but there may be quirks lurking when there are multiple\n @@ t/lib-unicode-nfc-nfd.sh (new)\n +#\n +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n +\tmkdir c_${greek_nfc} &&\n -+\tls | od -t x1 | grep \"63 *5f *e1 *bd *a7\"\n ++\tls | test-tool hexdump | grep \"63 5f e1 bd a7\"\n +'\n +\n +# See if the NFD spelling appears on the disk.\n +#\n +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n +\tmkdir d_${greek_nfd2} &&\n -+\tls | od -t x1 | grep \"64 *5f *cf *89 *cc *94 *cd *82\"\n ++\tls | test-tool hexdump | grep \"64 5f cf 89 cc 94 cd 82\"\n +'\n +\n +# The following is for debugging. I found it useful when\n 27: 4efb3a43838 = 28: f9a7869d202 t7527: test Unicode NFC/NFD handling on MacOS\n 28: df1b4f3a80f ! 29: 9fc7c970929 fsmonitor--daemon: allow --super-prefix argument\n @@ git.c: static struct cmd_struct commands[] = {\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\n \n ## t/t7527-builtin-fsmonitor.sh ##\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'submodule always visited' '\n \tmy_match_and_clean\n '\n \n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success \"Submodule always visited\" '\n +\ttr '\\047' Q <$1 | grep -e \"$msg\"\n +}\n +\n -+test_expect_success \"Stray Submodule super-prefix warning\" '\n ++test_expect_success \"stray submodule super-prefix warning\" '\n +\ttest_when_finished \"rm -rf super; \\\n +\t\t\t rm -rf sub; \\\n +\t\t\t rm super-sub.trace\" &&\n -: ----------- > 30: ca833ecc7a1 t7527: improve implicit shutdown testing in fsmonitor--daemon\n\n-- \ngitgitgadget\n"},{"id":"455819","messageId":"26144c586599a17c05f27192c3453e6f44c56d3a.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 01/30] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:16Z","receivedAt":"2022-05-23T20:13:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach FSMonitor daemon on Windows to recognize shortname paths as\naliases of normal longname paths. FSMonitor clients, such as `git\nstatus`, should receive the longname spelling of changed files (when\npossible).\n\nSometimes we receive FS events using the shortname, such as when a CMD\nshell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\narrives using whatever combination of long and shortnames were used by\nthe other process. (Shortnames do seem to be case normalized,\nhowever.)\n\nUse Windows GetLongPathNameW() to try to map the pathname spelling in\nthe notification event into the normalized longname spelling. (This\ncan fail if the file/directory is deleted, moved, or renamed, because\nwe are asking the FS for the mapping in response to the event and\nafter it has already happened, but we try.)\n\nSpecial case the shortname spelling of \".git\" to avoid under-reporting\nthese events.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 386 ++++++++++++++++++++++++----\n t/t7527-builtin-fsmonitor.sh | 65 +++++\n 2 files changed, 397 insertions(+), 54 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 5b928ab66e5..0a86aea3f7e 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -25,6 +25,9 @@ struct one_watch\n \tDWORD count;\n \n \tstruct strbuf path;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n+\tDWORD wpath_longname_len;\n+\n \tHANDLE hDir;\n \tHANDLE hEvent;\n \tOVERLAPPED overlapped;\n@@ -34,6 +37,21 @@ struct one_watch\n \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n \t */\n \tBOOL is_active;\n+\n+\t/*\n+\t * Are shortnames enabled on the containing drive? This is\n+\t * always true for \"C:/\" drives and usually never true for\n+\t * other drives.\n+\t *\n+\t * We only set this for the worktree because we only need to\n+\t * convert shortname paths to longname paths for items we send\n+\t * to clients. (We don't care about shortname expansion for\n+\t * paths inside a GITDIR because we never send them to\n+\t * clients.)\n+\t */\n+\tBOOL has_shortnames;\n+\tBOOL has_tilde;\n+\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n struct fsmonitor_daemon_backend_data\n@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n };\n \n /*\n- * Convert the WCHAR path from the notification into UTF8 and\n- * then normalize it.\n+ * Convert the WCHAR path from the event into UTF8 and normalize it.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n */\n-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n \t\t\t\t struct strbuf *normalized_path)\n {\n \tint reserve;\n \tint len = 0;\n \n \tstrbuf_reset(normalized_path);\n-\tif (!info->FileNameLength)\n+\tif (!wpath_len)\n \t\tgoto normalize;\n \n \t/*\n@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t * sequence of 2 UTF8 characters. That should let us\n \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n \t */\n-\treserve = info->FileNameLength + 1;\n+\treserve = 2 * wpath_len + 1;\n \tstrbuf_grow(normalized_path, reserve);\n \n \tfor (;;) {\n-\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n-\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n+\t\t\t\t\t wpath, wpath_len,\n \t\t\t\t\t normalized_path->buf,\n \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n \t\t\t\t\t NULL, NULL);\n@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n \t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n-\t\t\t GetLastError(),\n-\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n-\t\t\t info->FileName);\n+\t\t\t GetLastError(), (int)wpath_len, wpath);\n \t\t\treturn -1;\n \t\t}\n \n@@ -98,6 +115,173 @@ normalize:\n \treturn strbuf_normalize_path(normalized_path);\n }\n \n+/*\n+ * See if the worktree root directory has shortnames enabled.\n+ * This will help us decide if we need to do an expensive shortname\n+ * to longname conversion on every notification event.\n+ *\n+ * We do not want to create a file to test this, so we assume that the\n+ * root directory contains a \".git\" file or directory. (Our caller\n+ * only calls us for the worktree root, so this should be fine.)\n+ *\n+ * Remember the spelling of the shortname for \".git\" if it exists.\n+ */\n+static void check_for_shortnames(struct one_watch *watch)\n+{\n+\twchar_t buf_in[MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\twchar_t *last;\n+\twchar_t *p;\n+\n+\t/* build L\"/.git\" */\n+\tswprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%s.git\",\n+\t\t watch->wpath_longname);\n+\n+\tif (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)))\n+\t\treturn;\n+\n+\t/*\n+\t * Get the final filename component of the shortpath.\n+\t * We know that the path does not have a final slash.\n+\t */\n+\tfor (last = p = buf_out; *p; p++)\n+\t\tif (*p == L'/' || *p == '\\\\')\n+\t\t\tlast = p + 1;\n+\n+\tif (!wcscmp(last, L\".git\"))\n+\t\treturn;\n+\n+\twatch->has_shortnames = 1;\n+\twcsncpy(watch->dotgit_shortname, last,\n+\t\tARRAY_SIZE(watch->dotgit_shortname));\n+\n+\t/*\n+\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n+\t * we should be able to avoid shortname to longname mapping on\n+\t * every notification event if the source string does not\n+\t * contain a \"~\".\n+\t *\n+\t * However, the documentation for GetLongPathNameW() says\n+\t * that there are filesystems that don't follow that pattern\n+\t * and warns against this optimization.\n+\t *\n+\t * Lets test this.\n+\t */\n+\tif (wcschr(watch->dotgit_shortname, L'~'))\n+\t\twatch->has_tilde = 1;\n+}\n+\n+enum get_relative_result {\n+\tGRR_NO_CONVERSION_NEEDED,\n+\tGRR_HAVE_CONVERSION,\n+\tGRR_SHUTDOWN,\n+};\n+\n+/*\n+ * Info notification paths are relative to the root of the watch.\n+ * If our CWD is still at the root, then we can use relative paths\n+ * to convert from shortnames to longnames. If our process has a\n+ * different CWD, then we need to construct an absolute path, do\n+ * the conversion, and then return the root-relative portion.\n+ *\n+ * We use the longname form of the root as our basis and assume that\n+ * it already has a trailing slash.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n+ */\n+static enum get_relative_result get_relative_longname(\n+\tstruct one_watch *watch,\n+\tconst wchar_t *wpath, DWORD wpath_len,\n+\twchar_t *wpath_longname, size_t bufsize_wpath_longname)\n+{\n+\twchar_t buf_in[2 * MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\tDWORD root_len;\n+\tDWORD out_len;\n+\n+\t/*\n+\t * Build L\"/\"\n+\t * Note that the might not be null terminated\n+\t * so we avoid swprintf() constructions.\n+\t */\n+\troot_len = watch->wpath_longname_len;\n+\tif (root_len + wpath_len >= ARRAY_SIZE(buf_in)) {\n+\t\t/*\n+\t\t * This should not happen. We cannot append the observed\n+\t\t * relative path onto the end of the worktree root path\n+\t\t * without overflowing the buffer. Just give up.\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\twcsncpy(buf_in, watch->wpath_longname, root_len);\n+\twcsncpy(buf_in + root_len, wpath, wpath_len);\n+\tbuf_in[root_len + wpath_len] = 0;\n+\n+\t/*\n+\t * We don't actually know if the source pathname is a\n+\t * shortname or a longname. This Windows routine allows\n+\t * either to be given as input.\n+\t */\n+\tout_len = GetLongPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out));\n+\tif (!out_len) {\n+\t\t/*\n+\t\t * The shortname to longname conversion can fail for\n+\t\t * various reasons, for example if the file has been\n+\t\t * deleted. (That is, if we just received a\n+\t\t * delete-file notification event and the file is\n+\t\t * already gone, we can't ask the file system to\n+\t\t * lookup the longname for it. Likewise, for moves\n+\t\t * and renames where we are given the old name.)\n+\t\t *\n+\t\t * Since deleting or moving a file or directory by its\n+\t\t * shortname is rather obscure, I'm going ignore the\n+\t\t * failure and ask the caller to report the original\n+\t\t * relative path. This seems kinder than failing here\n+\t\t * and forcing a resync. Besides, forcing a resync on\n+\t\t * every file/directory delete would effectively\n+\t\t * cripple monitoring.\n+\t\t *\n+\t\t * We might revisit this in the future.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (!wcscmp(buf_in, buf_out)) {\n+\t\t/*\n+\t\t * The path does not have a shortname alias.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (wcsncmp(buf_in, buf_out, root_len)) {\n+\t\t/*\n+\t\t * The spelling of the root directory portion of the computed\n+\t\t * longname has changed. This should not happen. Basically,\n+\t\t * it means that we don't know where (without recomputing the\n+\t\t * longname of just the root directory) to split out the\n+\t\t * relative path. Since this should not happen, I'm just\n+\t\t * going to let this fail and force a shutdown (because all\n+\t\t * subsequent events are probably going to see the same\n+\t\t * mismatch).\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\tif (out_len - root_len >= bufsize_wpath_longname) {\n+\t\t/*\n+\t\t * This should not happen. We cannot copy the root-relative\n+\t\t * portion of the path into the provided buffer without an\n+\t\t * overrun. Just give up.\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\t/* Return the worktree root-relative portion of the longname. */\n+\n+\twcscpy(wpath_longname, buf_out + root_len);\n+\treturn GRR_HAVE_CONVERSION;\n+}\n+\n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n@@ -111,7 +295,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n-\twchar_t wpath[MAX_PATH];\n+\tDWORD len_longname;\n+\twchar_t wpath[MAX_PATH + 1];\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \tif (xutftowcs_path(wpath, path) < 0) {\n \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n@@ -128,6 +314,21 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \t\treturn NULL;\n \t}\n \n+\tlen_longname = GetLongPathNameW(wpath, wpath_longname,\n+\t\t\t\t\tARRAY_SIZE(wpath_longname));\n+\tif (!len_longname) {\n+\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n+\t\t GetLastError(), path);\n+\t\tCloseHandle(hDir);\n+\t\treturn NULL;\n+\t}\n+\n+\tif (wpath_longname[len_longname - 1] != L'/' &&\n+\t wpath_longname[len_longname - 1] != L'\\\\') {\n+\t\twpath_longname[len_longname++] = L'/';\n+\t\twpath_longname[len_longname] = 0;\n+\t}\n+\n \tCALLOC_ARRAY(watch, 1);\n \n \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n@@ -135,6 +336,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tstrbuf_init(&watch->path, 0);\n \tstrbuf_addstr(&watch->path, path);\n \n+\twcscpy(watch->wpath_longname, wpath_longname);\n+\twatch->wpath_longname_len = len_longname;\n+\n \twatch->hDir = hDir;\n \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n \n@@ -258,6 +462,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t,\n+\tDWORD info_action)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n+\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -274,6 +534,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n \tstruct fsmonitor_batch *batch = NULL;\n \tconst char *p = watch->buffer;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \t/*\n \t * If the kernel gets more events than will fit in the kernel\n@@ -306,54 +567,64 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n+\t\twchar_t *wpath = info->FileName;\n+\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n \t\tenum fsmonitor_path_type t;\n+\t\tenum get_relative_result grr;\n+\n+\t\tif (watch->has_shortnames) {\n+\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n+\t\t\t\t/*\n+\t\t\t\t * This event exactly matches the\n+\t\t\t\t * spelling of the shortname of\n+\t\t\t\t * \".git\", so we can skip some steps.\n+\t\t\t\t *\n+\t\t\t\t * (This case is odd because the user\n+\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n+\t\t\t\t * use the filesystem to map it back\n+\t\t\t\t * to \".git\".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_reset(&path);\n+\t\t\t\tstrbuf_addstr(&path, \".git\");\n+\t\t\t\tt = IS_DOT_GIT;\n+\t\t\t\tgoto process_it;\n+\t\t\t}\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n-\t\t\tgoto skip_this_path;\n-\n-\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n-\n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n+\t\t\tif (watch->has_tilde && !wcschr(wpath, L'~')) {\n+\t\t\t\t/*\n+\t\t\t\t * Shortnames on this filesystem have tildes\n+\t\t\t\t * and the notification path does not have\n+\t\t\t\t * one, so we assume that it is a longname.\n+\t\t\t\t */\n+\t\t\t\tgoto normalize_it;\n+\t\t\t}\n \n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n+\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n+\t\t\t\t\t\t wpath_longname,\n+\t\t\t\t\t\t ARRAY_SIZE(wpath_longname));\n+\t\t\tswitch (grr) {\n+\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n+\t\t\t\tbreak;\n+\t\t\tcase GRR_HAVE_CONVERSION:\n+\t\t\t\twpath = wpath_longname;\n+\t\t\t\twpath_len = wcslen(wpath);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\tcase GRR_SHUTDOWN:\n \t\t\t\tgoto force_shutdown;\n \t\t\t}\n-\t\t\tbreak;\n+\t\t}\n \n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n+normalize_it:\n+\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n+\t\t\tgoto skip_this_path;\n \n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+process_it:\n+\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n+\t\t\t\t\t info->Action))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -382,6 +653,9 @@ force_shutdown:\n * Note that we DO NOT get filesystem events on the external \n * itself (it is not inside something that we are watching). In particular,\n * we do not get an event if the external is deleted.\n+ *\n+ * Also, we do not care about shortnames within the external , since\n+ * we never send these paths to clients.\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n@@ -403,8 +677,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\tif (normalize_path_in_utf8(\n+\t\t\t info->FileName,\n+\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t &path) == -1)\n \t\t\tgoto skip_this_path;\n \n \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n@@ -538,6 +814,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \tif (!data->watch_worktree)\n \t\tgoto failed;\n \n+\tcheck_for_shortnames(data->watch_worktree);\n+\n \tif (state->nr_paths_watching > 1) {\n \t\tdata->watch_gitdir = create_watch(state,\n \t\t\t\t\t\t state->path_gitdir_watch.buf);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex bd0c952a116..1be21785162 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -166,6 +166,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon -C test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n+\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n+\n+\tstart_daemon -C test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"455820","messageId":"4bca494bb22365fd9ccc59f6470c5124f74caf84.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 03/30] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:18Z","receivedAt":"2022-05-23T20:13:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a stress test to hammer on the fsmonitor daemon.\nCreate a client-side thread pool of n threads and have\neach of them make m requests as fast as they can.\n\nWe do not currently inspect the contents of the response.\nWe're only interested in placing a heavy request load on\nthe daemon.\n\nThis test is useful for interactive testing and various\nexperimentation. For example, to place additional load\non the daemon while another test is running. We currently\ndo not have a test script that actually uses this helper.\nWe might add such a test in the future.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++\n 1 file changed, 106 insertions(+)\n\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nindex 3062c8a3c2b..54a4856c48c 100644\n--- a/t/helper/test-fsmonitor-client.c\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -7,6 +7,8 @@\n #include \"cache.h\"\n #include \"parse-options.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2.h\"\n \n #ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n int cmd__fsmonitor_client(int argc, const char **argv)\n@@ -79,20 +81,121 @@ static int do_send_flush(void)\n \treturn 0;\n }\n \n+struct hammer_thread_data\n+{\n+\tpthread_t pthread_id;\n+\tint thread_nr;\n+\n+\tint nr_requests;\n+\tconst char *token;\n+\n+\tint sum_successful;\n+\tint sum_errors;\n+};\n+\n+static void *hammer_thread_proc(void *_hammer_thread_data)\n+{\n+\tstruct hammer_thread_data *data = _hammer_thread_data;\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint k;\n+\tint ret;\n+\n+\ttrace2_thread_start(\"hammer\");\n+\n+\tfor (k = 0; k < data->nr_requests; k++) {\n+\t\tstrbuf_reset(&answer);\n+\n+\t\tret = fsmonitor_ipc__send_query(data->token, &answer);\n+\t\tif (ret < 0)\n+\t\t\tdata->sum_errors++;\n+\t\telse\n+\t\t\tdata->sum_successful++;\n+\t}\n+\n+\tstrbuf_release(&answer);\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Start a pool of client threads that will each send a series of\n+ * commands to the daemon.\n+ *\n+ * The goal is to overload the daemon with a sustained series of\n+ * concurrent requests.\n+ */\n+static int do_hammer(const char *token, int nr_threads, int nr_requests)\n+{\n+\tstruct hammer_thread_data *data = NULL;\n+\tint k;\n+\tint sum_join_errors = 0;\n+\tint sum_commands = 0;\n+\tint sum_errors = 0;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\tif (nr_threads < 1)\n+\t\tnr_threads = 1;\n+\tif (nr_requests < 1)\n+\t\tnr_requests = 1;\n+\n+\tCALLOC_ARRAY(data, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\t\tp->thread_nr = k;\n+\t\tp->nr_requests = nr_requests;\n+\t\tp->token = token;\n+\n+\t\tif (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {\n+\t\t\twarning(\"failed to create thread[%d] skipping remainder\", k);\n+\t\t\tnr_threads = k;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\n+\t\tif (pthread_join(p->pthread_id, NULL))\n+\t\t\tsum_join_errors++;\n+\t\tsum_commands += p->sum_successful;\n+\t\tsum_errors += p->sum_errors;\n+\t}\n+\n+\tfprintf(stderr, \"HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\\n\",\n+\t\tnr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);\n+\n+\tfree(data);\n+\n+\t/*\n+\t * Return an error if any of the _send_query requests failed.\n+\t * We don't care about thread create/join errors.\n+\t */\n+\treturn sum_errors > 0;\n+}\n+\n int cmd__fsmonitor_client(int argc, const char **argv)\n {\n \tconst char *subcmd;\n \tconst char *token = NULL;\n+\tint nr_threads = 1;\n+\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n \t\t\"test-tool fsmonitor-client query []\",\n \t\t\"test-tool fsmonitor-client flush\",\n+\t\t\"test-tool fsmonitor-client hammer [] [] []\",\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n \t\tOPT_STRING(0, \"token\", &token, \"token\",\n \t\t\t \"command token to send to the server\"),\n+\n+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, \"number of client threads\"),\n+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, \"number of requests per thread\"),\n+\n \t\tOPT_END()\n \t};\n \n@@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)\n \tif (!strcmp(subcmd, \"flush\"))\n \t\treturn !!do_send_flush();\n \n+\tif (!strcmp(subcmd, \"hammer\"))\n+\t\treturn !!do_hammer(token, nr_threads, nr_requests);\n+\n \tdie(\"Unhandled subcommand: '%s'\", subcmd);\n }\n #endif\n-- \ngitgitgadget\n\n"},{"id":"455821","messageId":"663deabc3f67c25fb6888871b779b0c8f25ac75b.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 04/30] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:19Z","receivedAt":"2022-05-23T20:13:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a worktree, so there is nothing for the\ndaemon watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 18 +++++\n builtin/update-index.c | 16 +++++\n fsmonitor-settings.c | 133 ++++++++++++++++++++++++++++++------\n fsmonitor-settings.h | 16 +++++\n t/t7519-status-fsmonitor.sh | 23 +++++++\n 5 files changed, 186 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 46be55a4618..66b78a0353f 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1423,6 +1423,7 @@ static int try_to_start_background_daemon(void)\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tenum fsmonitor_reason reason;\n \tint detach_console = 0;\n \n \tstruct option options[] = {\n@@ -1449,6 +1450,23 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\t/*\n+\t * If the repo is fsmonitor-compatible, explicitly set IPC-mode\n+\t * (without bothering to load the `core.fsmonitor` config settings).\n+\t *\n+\t * If the repo is not compatible, the repo-settings will be set to\n+\t * incompatible rather than IPC, so we can use one of the __get\n+\t * routines to detect the discrepancy.\n+\t */\n+\tfsm_settings__set_ipc(the_repository);\n+\n+\treason = fsm_settings__get_reason(the_repository);\n+\tif (reason > FSMONITOR_REASON_OK)\n+\t\tdie(\"%s\",\n+\t\t fsm_settings__get_incompatible_msg(the_repository,\n+\t\t\t\t\t\t reason));\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 876112abb21..01ed4c4976b 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1237,6 +1237,22 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n+\n+\t\t/*\n+\t\t * The user wants to turn on FSMonitor using the command\n+\t\t * line argument. (We don't know (or care) whether that\n+\t\t * is the IPC or HOOK version.)\n+\t\t *\n+\t\t * Use one of the __get routines to force load the FSMonitor\n+\t\t * config settings into the repo-settings. That will detect\n+\t\t * whether the file system is compatible so that we can stop\n+\t\t * here with a nice error message.\n+\t\t */\n+\t\tif (reason > FSMONITOR_REASON_OK)\n+\t\t\tdie(\"%s\",\n+\t\t\t fsm_settings__get_incompatible_msg(r, reason));\n+\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 757d230d538..7d3177d441a 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -9,23 +9,42 @@\n */\n struct fsmonitor_settings {\n \tenum fsmonitor_mode mode;\n+\tenum fsmonitor_reason reason;\n \tchar *hook_path;\n };\n \n-static void lookup_fsmonitor_settings(struct repository *r)\n+static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n+{\n+\tif (!r->worktree) {\n+\t\t/*\n+\t\t * Bare repositories don't have a working directory and\n+\t\t * therefore have nothing to watch.\n+\t\t */\n+\t\treturn FSMONITOR_REASON_BARE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n+static struct fsmonitor_settings *alloc_settings(void)\n {\n \tstruct fsmonitor_settings *s;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\ts->reason = FSMONITOR_REASON_UNTESTED;\n+\n+\treturn s;\n+}\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n \tconst char *const_str;\n \tint bool_value;\n \n \tif (r->settings.fsmonitor)\n \t\treturn;\n \n-\tCALLOC_ARRAY(s, 1);\n-\ts->mode = FSMONITOR_MODE_DISABLED;\n-\n-\tr->settings.fsmonitor = s;\n-\n \t/*\n \t * Overload the existing \"core.fsmonitor\" config setting (which\n \t * has historically been either unset or a hook pathname) to\n@@ -38,6 +57,8 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \tcase 0: /* config value was set to */\n \t\tif (bool_value)\n \t\t\tfsm_settings__set_ipc(r);\n+\t\telse\n+\t\t\tfsm_settings__set_disabled(r);\n \t\treturn;\n \n \tcase 1: /* config value was unset */\n@@ -53,18 +74,18 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \t\treturn;\n \t}\n \n-\tif (!const_str || !*const_str)\n-\t\treturn;\n-\n-\tfsm_settings__set_hook(r, const_str);\n+\tif (const_str && *const_str)\n+\t\tfsm_settings__set_hook(r, const_str);\n+\telse\n+\t\tfsm_settings__set_disabled(r);\n }\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n \n \treturn r->settings.fsmonitor->mode;\n }\n@@ -73,31 +94,55 @@ const char *fsm_settings__get_hook_path(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n \n \treturn r->settings.fsmonitor->hook_path;\n }\n \n void fsm_settings__set_ipc(struct repository *r)\n {\n+\tenum fsmonitor_reason reason = check_for_incompatible(r);\n+\n+\tif (reason != FSMONITOR_REASON_OK) {\n+\t\tfsm_settings__set_incompatible(r, reason);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Caller requested IPC explicitly, so avoid (possibly\n+\t * recursive) config lookup.\n+\t */\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n \n void fsm_settings__set_hook(struct repository *r, const char *path)\n {\n+\tenum fsmonitor_reason reason = check_for_incompatible(r);\n+\n+\tif (reason != FSMONITOR_REASON_OK) {\n+\t\tfsm_settings__set_incompatible(r, reason);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Caller requested hook explicitly, so avoid (possibly\n+\t * recursive) config lookup.\n+\t */\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n }\n@@ -106,9 +151,57 @@ void fsm_settings__set_disabled(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\n+\n+void fsm_settings__set_incompatible(struct repository *r,\n+\t\t\t\t enum fsmonitor_reason reason)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n+\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\n+\n+char *fsm_settings__get_incompatible_msg(const struct repository *r,\n+\t\t\t\t\t enum fsmonitor_reason reason)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tswitch (reason) {\n+\tcase FSMONITOR_REASON_UNTESTED:\n+\tcase FSMONITOR_REASON_OK:\n+\t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_BARE:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t xgetcwd());\n+\t\tgoto done;\n+\t}\n+\n+\tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n+\t reason);\n+\n+done:\n+\treturn strbuf_detach(&msg, NULL);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a4c5d7b4889..8d9331c0c0a 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -4,18 +4,34 @@\n struct repository;\n \n enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n \tFSMONITOR_MODE_DISABLED = 0,\n \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n };\n \n+/*\n+ * Incompatibility reasons.\n+ */\n+enum fsmonitor_reason {\n+\tFSMONITOR_REASON_UNTESTED = 0,\n+\tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n+\tFSMONITOR_REASON_BARE,\n+};\n+\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n+void fsm_settings__set_incompatible(struct repository *r,\n+\t\t\t\t enum fsmonitor_reason reason);\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n+char *fsm_settings__get_incompatible_msg(const struct repository *r,\n+\t\t\t\t\t enum fsmonitor_reason reason);\n+\n struct fsmonitor_settings;\n \n #endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex a6308acf006..9a8e21c5608 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '\n \ttest $ret -ne 1\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n+\tgit init --bare bare-clone &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n+\tgit init --bare bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"455822","messageId":"7cb0180a1ed95f9ebecc605cd1d968bced46d375.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 05/30] fsmonitor-settings: stub in Win32-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:20Z","receivedAt":"2022-05-23T20:13:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend generic incompatibility checkout with platform-specific\nmechanism. Stub in Win32 version.\n\nIn the existing fsmonitor-settings code we have a way to mark\ntypes of repos as incompatible with fsmonitor (whether via the\nhook and IPC APIs). For example, we do this for bare repos,\nsince there are no files to watch.\n\nExtend this exclusion mechanism for platform-specific reasons.\nThis commit just creates the framework and adds a stub for Win32.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 +++++++++++++\n compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++\n config.mak.uname | 4 ++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n fsmonitor-settings.c | 10 ++++++++++\n fsmonitor-settings.h | 13 +++++++++++++\n 6 files changed, 52 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n\ndiff --git a/Makefile b/Makefile\nindex daa21bed6c3..93604fe8ef7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -475,6 +475,11 @@ all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n+# If your platform has OS-specific ways to tell if a repo is incompatible with\n+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n+# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n+# that implements the `fsm_os_settings__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n+ifdef FSMONITOR_OS_SETTINGS\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE\n ifdef FSMONITOR_DAEMON_BACKEND\n \t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n endif\n+ifdef FSMONITOR_OS_SETTINGS\n+\t@echo FSMONITOR_OS_SETTINGS=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_OS_SETTINGS)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 501970902da..cf224768ad6 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex ee0d7257b77..16705da2000 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 7d3177d441a..f67db913f57 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -23,6 +23,16 @@ static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n \t\treturn FSMONITOR_REASON_BARE;\n \t}\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+\t{\n+\t\tenum fsmonitor_reason reason;\n+\n+\t\treason = fsm_os__incompatible(r);\n+\t\tif (reason != FSMONITOR_REASON_OK)\n+\t\t\treturn reason;\n+\t}\n+#endif\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 8d9331c0c0a..6cb0d8e7d9f 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -34,4 +34,17 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \n struct fsmonitor_settings;\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+/*\n+ * Ask platform-specific code whether the repository is incompatible\n+ * with fsmonitor (both hook and ipc modes). For example, if the working\n+ * directory is on a remote volume and mounted via a technology that does\n+ * not support notification events, then we should not pretend to watch it.\n+ *\n+ * fsm_os__* routines should considered private to fsm_settings__\n+ * routines.\n+ */\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);\n+#endif /* HAVE_FSMONITOR_OS_SETTINGS */\n+\n #endif /* FSMONITOR_SETTINGS_H */\n-- \ngitgitgadget\n\n"},{"id":"455823","messageId":"9774faddc45cd33362aeb1e3ea80125bc496cbc9.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 06/30] fsmonitor-settings: VFS for Git virtual repos are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:21Z","receivedAt":"2022-05-23T20:13:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nVFS for Git virtual repositories are incompatible with FSMonitor.\n\nVFS for Git is a downstream fork of Git. It contains its own custom\nfile system watcher that is aware of the virtualization. If a working\ndirectory is being managed by VFS for Git, we should not try to watch\nit because we may get incomplete results.\n\nWe do not know anything about how VFS for Git works, but we do\nknow that VFS for Git working directories contain a well-defined\nconfig setting. If it is set, mark the working directory as\nincompatible.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 6 ++++++\n fsmonitor-settings.h | 1 +\n t/t7519-status-fsmonitor.sh | 9 +++++++++\n 4 files changed, 42 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 7fce32a3c5b..ee78bba38e3 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -3,7 +3,33 @@\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n \n+/*\n+ * VFS for Git is incompatible with FSMonitor.\n+ *\n+ * Granted, core Git does not know anything about VFS for Git and we\n+ * shouldn't make assumptions about a downstream feature, but users\n+ * can install both versions. And this can lead to incorrect results\n+ * from core Git commands. So, without bringing in any of the VFS for\n+ * Git code, do a simple config test for a published config setting.\n+ * (We do not look at the various *_TEST_* environment variables.)\n+ */\n+static enum fsmonitor_reason check_vfs4git(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n+\t\treturn FSMONITOR_REASON_VFS4GIT;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_vfs4git(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex f67db913f57..600ae165ab1 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -207,6 +207,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n \t\t\t xgetcwd());\n \t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_VFS4GIT:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 6cb0d8e7d9f..a48802cde9c 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -17,6 +17,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n \tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 9a8e21c5608..156895f9437 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n \tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n '\n \n+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone actual\" &&\n+\tgit init fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"virtual repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"455824","messageId":"dc2dfd679312a71ef0613857860b4ad426913471.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 08/30] fsmonitor-settings: remote repos on macOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:23Z","receivedAt":"2022-05-23T20:13:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on macOS and mark them as\nincompatible with FSMonitor.\n\nWith this, `git fsmonitor--daemon run` will error out with a message\nlike it does for bare repos.\n\nClient commands, like `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 12 +++++\n fsmonitor-settings.h | 2 +\n 3 files changed, 80 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex 7fce32a3c5b..fdd762bf79d 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -2,8 +2,74 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n+#include \n+#include \n+\n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type (NFS, SAMBA, etc.) dictates whether notification events\n+ * are available at all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * For the builtin FSMonitor, we create the Unix domain socket for the\n+ * IPC in the .git directory. If the working directory is remote,\n+ * then the socket will be created on the remote file system. This\n+ * can fail if the remote file system does not support UDS file types\n+ * (e.g. smbfs to a Windows server) or if the remote kernel does not\n+ * allow a non-local process to bind() the socket. (These problems\n+ * could be fixed by moving the UDS out of the .git directory and to a\n+ * well-known local directory on the client machine, but care should\n+ * be taken to ensure that $HOME is actually local and not a managed\n+ * file share.)\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\tstruct statfs fs;\n+\n+\tif (statfs(r->worktree, &fs) == -1) {\n+\t\tint saved_errno = errno;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n+\t\t\t\t r->worktree, strerror(saved_errno));\n+\t\terrno = saved_errno;\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n+\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n+\n+\tif (!(fs.f_flags & MNT_LOCAL))\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 600ae165ab1..d2fb0141f8e 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -208,6 +208,18 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t xgetcwd());\n \t\tgoto done;\n \n+\tcase FSMONITOR_REASON_ERROR:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_REMOTE:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"remote repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n+\n \tcase FSMONITOR_REASON_VFS4GIT:\n \t\tstrbuf_addf(&msg,\n \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a48802cde9c..afd1b3874ac 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -17,6 +17,8 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n \tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n+\tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"455825","messageId":"5627038aaa3e21c9d8607accfd3b2a46fa60e7fb.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 09/30] fsmonitor-settings: remote repos on Windows are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:24Z","receivedAt":"2022-05-23T20:13:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on Windows and mark them as\nincompatible with FSMonitor.\n\nWith this `git fsmonitor--daemon run` will error out with a message like it\ndoes for bare repos.\n\nClient commands, such as `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++\n 1 file changed, 102 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex ee78bba38e3..907655720bb 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * VFS for Git is incompatible with FSMonitor.\n@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)\n \treturn FSMONITOR_REASON_OK;\n }\n \n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type dictates whether notification events are available at\n+ * all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ *\n+ * Notes for testing:\n+ *\n+ * (a) Windows allows a network share to be mapped to a drive letter.\n+ * (This is the normal method to access it.)\n+ *\n+ * $ NET USE Z: \\\\server\\share\n+ * $ git -C Z:/repo status\n+ *\n+ * (b) Windows allows a network share to be referenced WITHOUT mapping\n+ * it to drive letter.\n+ *\n+ * $ NET USE \\\\server\\share\\dir\n+ * $ git -C //server/share/repo status\n+ *\n+ * (c) Windows allows \"SUBST\" to create a fake drive mapping to an\n+ * arbitrary path (which may be remote)\n+ *\n+ * $ SUBST Q: Z:\\repo\n+ * $ git -C Q:/ status\n+ *\n+ * (d) Windows allows a directory symlink to be created on a local\n+ * file system that points to a remote repo.\n+ *\n+ * $ mklink /d ./link //server/share/repo\n+ * $ git -C ./link status\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\twchar_t wpath[MAX_PATH];\n+\twchar_t wfullpath[MAX_PATH];\n+\tsize_t wlen;\n+\tUINT driveType;\n+\n+\t/*\n+\t * Do everything in wide chars because the drive letter might be\n+\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n+\t */\n+\tif (xutftowcs_path(wpath, r->worktree) < 0)\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\t/*\n+\t * GetDriveTypeW() requires a final slash. We assume that the\n+\t * worktree pathname points to an actual directory.\n+\t */\n+\twlen = wcslen(wpath);\n+\tif (wpath[wlen - 1] != L'\\\\' && wpath[wlen - 1] != L'/') {\n+\t\twpath[wlen++] = L'\\\\';\n+\t\twpath[wlen] = 0;\n+\t}\n+\n+\t/*\n+\t * Normalize the path. If nothing else, this converts forward\n+\t * slashes to backslashes. This is essential to get GetDriveTypeW()\n+\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n+\t */\n+\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\tdriveType = GetDriveTypeW(wfullpath);\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"DriveType '%s' L'%ls' (%u)\",\n+\t\t\t r->worktree, wfullpath, driveType);\n+\n+\tif (driveType == DRIVE_REMOTE) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"check_remote('%s') true\",\n+\t\t\t\t r->worktree);\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n-- \ngitgitgadget\n\n"},{"id":"455826","messageId":"f7ef7dcffc88f95dcf8fbca4d6e24648d52152c7.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 07/30] fsmonitor-settings: stub in macOS-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:22Z","receivedAt":"2022-05-23T20:13:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++\n config.mak.uname | 1 +\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 13 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cf224768ad6..cf911d141f2 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)\n \tifndef NO_PTHREADS\n \tifndef NO_UNIX_SOCKETS\n \tFSMONITOR_DAEMON_BACKEND = darwin\n+\tFSMONITOR_OS_SETTINGS = darwin\n \tendif\n \tendif\n \ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 16705da2000..b8f9f7a0388 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"455827","messageId":"aaff000cecb43550e0974bfda117e1c065ab8203.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 11/30] unpack-trees: initialize fsmonitor_has_run_once in o->result","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:26Z","receivedAt":"2022-05-23T20:13:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInitialize `o->result.fsmonitor_has_run_once` based upon value\nin `o->src_index->fsmonitor_has_run_once` to prevent a second\nfsmonitor query during the tree traversal and possibly getting\na skewed view of the working directory.\n\nThe checkout code has already talked to the fsmonitor and the\ntraversal is updating the index as it traverses, so there is\nno need to query the fsmonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n unpack-trees.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..888cff81f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \to->result.fsmonitor_last_update =\n \t\txstrdup_or_null(o->src_index->fsmonitor_last_update);\n+\to->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;\n \n \t/*\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n-- \ngitgitgadget\n\n"},{"id":"455828","messageId":"4f2b15d3d1f10eaec8e485b84281eb4cca207a69.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 12/30] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:27Z","receivedAt":"2022-05-23T20:14:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIgnore FSEvents resulting from `xattr` changes. Git does not care about\nxattr's or changes to xattr's, so don't waste time collecting these\nevents in the daemon nor transmitting them to clients.\n\nVarious security tools add xattrs to files and/or directories, such as\nto mark them as having been downloaded. We should ignore these events\nsince it doesn't affect the content of the file/directory or the normal\nmeta-data that Git cares about.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 0741fe834c3..14105f45c18 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tif (flag & kFSEventStreamEventFlagItemCloned)\n \t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n \n-\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n \t\t\t path, flag, msg.buf);\n \n \tstrbuf_release(&msg);\n@@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)\n \t\tef & kFSEventStreamEventFlagUserDropped);\n }\n \n+/*\n+ * If an `xattr` change is the only reason we received this event,\n+ * then silently ignore it. Git doesn't care about xattr's. We\n+ * have to be careful here because the kernel can combine multiple\n+ * events for a single path. And because events always have certain\n+ * bits set, such as `ItemIsFile` or `ItemIsDir`.\n+ *\n+ * Return 1 if we should ignore it.\n+ */\n+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n+{\n+\tstatic const FSEventStreamEventFlags mask =\n+\t\tkFSEventStreamEventFlagItemChangeOwner |\n+\t\tkFSEventStreamEventFlagItemCreated |\n+\t\tkFSEventStreamEventFlagItemFinderInfoMod |\n+\t\tkFSEventStreamEventFlagItemInodeMetaMod |\n+\t\tkFSEventStreamEventFlagItemModified |\n+\t\tkFSEventStreamEventFlagItemRemoved |\n+\t\tkFSEventStreamEventFlagItemRenamed |\n+\t\tkFSEventStreamEventFlagItemXattrMod |\n+\t\tkFSEventStreamEventFlagItemCloned;\n+\n+\treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n+}\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_ignore_xattr(event_flags[k])) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n+\t\t\t\t\t path_k, event_flags[k]);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n \n \t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-- \ngitgitgadget\n\n"},{"id":"455829","messageId":"594e0ae243de868ff18f94aafc4358d5801b88c5.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 15/30] fsmonitor--daemon: rename listener thread related variables","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:30Z","receivedAt":"2022-05-23T20:14:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRename platform-specific listener thread related variables\nand data types as we prepare to add another backend thread\ntype.\n\n[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`\n[] `state->backend_data` becomes `state->listen_data`\n[] `state->error_code` becomes `state->listen_error_code`\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 +++---\n compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------\n compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++-------------\n compat/fsmonitor/fsm-listen.h | 2 +-\n fsmonitor--daemon.h | 6 +++---\n 5 files changed, 36 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 90fa9d09efb..b2f578b239a 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1225,8 +1225,8 @@ cleanup:\n \n \tif (err)\n \t\treturn err;\n-\tif (state->error_code)\n-\t\treturn state->error_code;\n+\tif (state->listen_error_code)\n+\t\treturn state->listen_error_code;\n \treturn 0;\n }\n \n@@ -1241,7 +1241,7 @@ static int fsmonitor_run_daemon(void)\n \thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n-\tstate.error_code = 0;\n+\tstate.listen_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 14105f45c18..07113205a61 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -27,7 +27,7 @@\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tCFStringRef cfsr_worktree_path;\n \tCFStringRef cfsr_gitdir_path;\n@@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t const FSEventStreamEventId event_ids[])\n {\n \tstruct fsmonitor_daemon_state *state = ctx;\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tchar **paths = (char **)event_paths;\n \tstruct fsmonitor_batch *batch = NULL;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tNULL,\n \t\tNULL\n \t};\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \tconst void *dir_array[2];\n \n \tCALLOC_ARRAY(data, 1);\n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \n \tdata->cfsr_worktree_path = CFStringCreateWithCString(\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n@@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n \terror(_(\"Unable to create FSEventStream.\"));\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tif (data->stream) {\n \t\tif (data->stream_started)\n@@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n \t\tFSEventStreamRelease(data->stream);\n \t}\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \tdata->shutdown_style = SHUTDOWN_EVENT;\n \n \tCFRunLoopStop(data->rl);\n@@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tdata->rl = CFRunLoopGetCurrent();\n \n@@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \n \tswitch (data->shutdown_style) {\n \tcase FORCE_ERROR_STOP:\n-\t\tstate->error_code = -1;\n+\t\tstate->listen_error_code = -1;\n \t\t/* fall thru */\n \tcase FORCE_SHUTDOWN:\n \t\tipc_server_stop_async(state->ipc_server_data);\n@@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \treturn;\n \n force_error_stop_without_loop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \tipc_server_stop_async(state->ipc_server_data);\n \treturn;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 7f48306df53..2943632c771 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -54,7 +54,7 @@ struct one_watch\n \twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tstruct one_watch *watch_worktree;\n \tstruct one_watch *watch_gitdir;\n@@ -284,7 +284,7 @@ static enum get_relative_result get_relative_longname(\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+\tSetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);\n }\n \n static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n@@ -359,7 +359,7 @@ static void destroy_watch(struct one_watch *watch)\n \tfree(watch);\n }\n \n-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+static int start_rdcw_watch(struct fsm_listen_data *data,\n \t\t\t struct one_watch *watch)\n {\n \tDWORD dwNotifyFilter =\n@@ -538,7 +538,7 @@ static int process_1_worktree_event(\n */\n static int process_worktree_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_worktree;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -669,7 +669,7 @@ force_shutdown:\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_gitdir;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -727,11 +727,11 @@ skip_this_path:\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tDWORD dwWait;\n \tint result;\n \n-\tstate->error_code = 0;\n+\tstate->listen_error_code = 0;\n \n \tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n \t\tgoto force_error_stop;\n@@ -796,7 +796,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \t}\n \n force_error_stop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \n force_shutdown:\n \t/*\n@@ -813,7 +813,7 @@ clean_shutdown:\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -846,7 +846,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \treturn 0;\n \n failed:\n@@ -859,16 +859,16 @@ failed:\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tCloseHandle(data->hEventShutdown);\n \tdestroy_watch(data->watch_worktree);\n \tdestroy_watch(data->watch_gitdir);\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nindex f0539349baf..41650bf8972 100644\n--- a/compat/fsmonitor/fsm-listen.h\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n * do so if the listener thread receives a normal shutdown signal from\n * the IPC layer.)\n *\n- * It should set `state->error_code` to -1 if the daemon should exit\n+ * It should set `state->listen_error_code` to -1 if the daemon should exit\n * with an error.\n */\n void fsm_listen__loop(struct fsmonitor_daemon_state *state);\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 223c2131b58..2c6fa1a5d91 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n */\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {\n \tint cookie_seq;\n \tstruct hashmap cookies;\n \n-\tint error_code;\n-\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\tint listen_error_code;\n+\tstruct fsm_listen_data *listen_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"455830","messageId":"c2b5c02ed3839902ed2734418fc493dd4b7e0725.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 16/30] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:31Z","receivedAt":"2022-05-23T20:14:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate another thread to watch over the daemon process and\nautomatically shut it down if necessary.\n\nThis commit creates the basic framework for a \"health\" thread\nto monitor the daemon and/or the file system. Later commits\nwill add platform-specific code to do the actual work.\n\nThe \"health\" thread is intended to monitor conditions that\nwould be difficult to track inside the IPC thread pool and/or\nthe file system listener threads. For example, when there are\nfile system events outside of the watched worktree root or if\nwe want to have an idle-timeout auto-shutdown feature.\n\nThis commit creates the health thread itself, defines the thread-proc\nand sets up the thread's event loop. It integrates this new thread\ninto the existing IPC and Listener thread models.\n\nThis commit defines the API to the platform-specific code where all of\nthe monitoring will actually happen.\n\nThe platform-specific code for MacOS is just stubs. Meaning that the\nhealth thread will immediately exit on MacOS, but that is OK and\nexpected. Future work can define MacOS-specific monitoring.\n\nThe platform-specific code for Windows sets up enough of the\nWaitForMultipleObjects() machinery to watch for system and/or custom\nevents. Currently, the set of wait handles only includes our custom\nshutdown event (sent from our other theads). Later commits in this\nseries will extend the set of wait handles to monitor other\nconditions.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 6 ++-\n builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n contrib/buildsystems/CMakeLists.txt | 2 +\n fsmonitor--daemon.h | 4 ++\n 7 files changed, 192 insertions(+), 2 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n\ndiff --git a/Makefile b/Makefile\nindex 93604fe8ef7..5f1623baadd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -472,8 +472,9 @@ all::\n #\n # If your platform supports a built-in fsmonitor backend, set\n # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n-# `compat/fsmonitor/fsm-listen-.c` that implements the\n-# `fsm_listen__*()` routines.\n+# `compat/fsmonitor/fsm-listen-.c` and\n+# `compat/fsmonitor/fsm-health-.c` files\n+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n # If your platform has OS-specific ways to tell if a repo is incompatible with\n # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n@@ -1982,6 +1983,7 @@ endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n ifdef FSMONITOR_OS_SETTINGS\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex b2f578b239a..2c109cf8b37 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n@@ -1136,6 +1137,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+static void *fsm_health__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-health\");\n+\n+\tfsm_health__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -1174,6 +1187,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint health_started = 0;\n \tint listener_started = 0;\n \tint err = 0;\n \n@@ -1201,6 +1215,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t}\n \tlistener_started = 1;\n \n+\t/*\n+\t * Start the health thread to watch over our process.\n+\t */\n+\tif (pthread_create(&state->health_thread, NULL,\n+\t\t\t fsm_health__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\terr = error(_(\"could not start fsmonitor health thread\"));\n+\t\tgoto cleanup;\n+\t}\n+\thealth_started = 1;\n+\n \t/*\n \t * The daemon is now fully functional in background threads.\n \t * Our primary thread should now just wait while the threads\n@@ -1223,10 +1248,17 @@ cleanup:\n \t\tpthread_join(state->listener_thread, NULL);\n \t}\n \n+\tif (health_started) {\n+\t\tfsm_health__stop_async(state);\n+\t\tpthread_join(state->health_thread, NULL);\n+\t}\n+\n \tif (err)\n \t\treturn err;\n \tif (state->listen_error_code)\n \t\treturn state->listen_error_code;\n+\tif (state->health_error_code)\n+\t\treturn state->health_error_code;\n \treturn 0;\n }\n \n@@ -1242,6 +1274,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.listen_error_code = 0;\n+\tstate.health_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n@@ -1321,6 +1354,11 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\tif (fsm_health__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize health thread\"));\n+\t\tgoto done;\n+\t}\n+\n \t/*\n \t * CD out of the worktree root directory.\n \t *\n@@ -1344,6 +1382,7 @@ done:\n \tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n+\tfsm_health__dtor(&state);\n \n \tipc_server_free(state.ipc_server_data);\n \ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nnew file mode 100644\nindex 00000000000..b9f709e8548\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -0,0 +1,24 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nnew file mode 100644\nindex 00000000000..94b1d020f25\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsm_health_data\n+{\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hHandles[1]; /* the array does not own these handles */\n+#define HEALTH_SHUTDOWN 0\n+\tint nr_handles; /* number of active event handles */\n+};\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_handles++;\n+\n+\tstate->health_data = data;\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tif (!state || !state->health_data)\n+\t\treturn;\n+\n+\tdata = state->health_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\n+\tFREE_AND_NULL(state->health_data);\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\n+\tfor (;;) {\n+\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n+\t\t\t\t\t\t data->hHandles,\n+\t\t\t\t\t\t FALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->health_error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+clean_shutdown:\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n+}\ndiff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\nnew file mode 100644\nindex 00000000000..45547ba9380\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health.h\n@@ -0,0 +1,47 @@\n+#ifndef FSM_HEALTH_H\n+#define FSM_HEALTH_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor health thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the health thread.\n+ * This will be called from the main thread AFTER joining the thread.\n+ */\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to monitor the\n+ * health of the daemon process. This will run in the health thread.\n+ *\n+ * The health thread should call `ipc_server_stop_async()` if it needs\n+ * to cause a shutdown. (It should NOT do so if it receives a shutdown\n+ * shutdown signal.)\n+ *\n+ * It should set `state->health_error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_health__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the health thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_HEALTH_H */\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex b8f9f7a0388..16ace43d1c7 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 2c6fa1a5d91..2102a5c9ff5 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n+struct fsm_health_data; /* opaque platform-specific data for health thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n+\tpthread_t health_thread;\n \tpthread_mutex_t main_lock;\n \n \tstruct strbuf path_worktree_watch;\n@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n \tstruct hashmap cookies;\n \n \tint listen_error_code;\n+\tint health_error_code;\n \tstruct fsm_listen_data *listen_data;\n+\tstruct fsm_health_data *health_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"455831","messageId":"46a5ae2a6353e1da3c3ba84c9a2b5e03d6eeae44.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 17/30] fsm-health-win32: add polling framework to monitor daemon health","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:32Z","receivedAt":"2022-05-23T20:14:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend the Windows version of the \"health\" thread to periodically\ninspect the system and shutdown if warranted.\n\nThis commit updates the thread's wait loop to use a timeout and\ndefines a (currently empty) table of functions to poll the system.\n\nA later commit will add functions to the table to actually\ninspect the system.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-\n 1 file changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 94b1d020f25..24fc612bf02 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -4,6 +4,24 @@\n #include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n+/*\n+ * Every minute wake up and test our health.\n+ */\n+#define WAIT_FREQ_MS (60 * 1000)\n+\n+/*\n+ * State machine states for each of the interval functions\n+ * used for polling our health.\n+ */\n+enum interval_fn_ctx {\n+\tCTX_INIT = 0,\n+\tCTX_TERM,\n+\tCTX_TIMER\n+};\n+\n+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx);\n+\n struct fsm_health_data\n {\n \tHANDLE hEventShutdown;\n@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n \tFREE_AND_NULL(state->health_data);\n }\n \n+/*\n+ * A table of the polling functions.\n+ */\n+static interval_fn *table[] = {\n+\tNULL, /* must be last */\n+};\n+\n+/*\n+ * Call all of the polling functions in the table.\n+ * Shortcut and return first error.\n+ *\n+ * Return 0 if all succeeded.\n+ */\n+static int call_all(struct fsmonitor_daemon_state *state,\n+\t\t enum interval_fn_ctx ctx)\n+{\n+\tint k;\n+\n+\tfor (k = 0; table[k]; k++) {\n+\t\tint r = table[k](state, ctx);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n+\tint r;\n+\n+\tr = call_all(state, CTX_INIT);\n+\tif (r < 0)\n+\t\tgoto force_error_stop;\n+\tif (r > 0)\n+\t\tgoto force_shutdown;\n \n \tfor (;;) {\n \t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n \t\t\t\t\t\t data->hHandles,\n-\t\t\t\t\t\t FALSE, INFINITE);\n+\t\t\t\t\t\t FALSE, WAIT_FREQ_MS);\n \n \t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n \t\t\tgoto clean_shutdown;\n \n+\t\tif (dwWait == WAIT_TIMEOUT) {\n+\t\t\tr = call_all(state, CTX_TIMER);\n+\t\t\tif (r < 0)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (r > 0)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\terror(_(\"health thread wait failed [GLE %ld]\"),\n \t\t GetLastError());\n \t\tgoto force_error_stop;\n@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)\n \n force_error_stop:\n \tstate->health_error_code = -1;\n+force_shutdown:\n \tipc_server_stop_async(state->ipc_server_data);\n clean_shutdown:\n+\tcall_all(state, CTX_TERM);\n \treturn;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"455832","messageId":"51b266b06e1739f09d785787c93fc3e346f47332.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 14/30] fsmonitor--daemon: prepare for adding health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:29Z","receivedAt":"2022-05-23T20:14:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor daemon thread startup to make it easier to start\na third thread class to monitor the health of the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex db297649daf..90fa9d09efb 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1174,6 +1174,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint listener_started = 0;\n+\tint err = 0;\n \n \t/*\n \t * Start the IPC thread pool before the we've started the file\n@@ -1194,15 +1196,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \tif (pthread_create(&state->listener_thread, NULL,\n \t\t\t fsm_listen__thread_proc, state) < 0) {\n \t\tipc_server_stop_async(state->ipc_server_data);\n-\t\tipc_server_await(state->ipc_server_data);\n-\n-\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t\terr = error(_(\"could not start fsmonitor listener thread\"));\n+\t\tgoto cleanup;\n \t}\n+\tlistener_started = 1;\n \n \t/*\n \t * The daemon is now fully functional in background threads.\n+\t * Our primary thread should now just wait while the threads\n+\t * do all the work.\n+\t */\n+cleanup:\n+\t/*\n \t * Wait for the IPC thread pool to shutdown (whether by client\n-\t * request or from filesystem activity).\n+\t * request, from filesystem activity, or an error).\n \t */\n \tipc_server_await(state->ipc_server_data);\n \n@@ -1211,10 +1218,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * event from the IPC thread pool, but it doesn't hurt to tell\n \t * it again. And wait for it to shutdown.\n \t */\n-\tfsm_listen__stop_async(state);\n-\tpthread_join(state->listener_thread, NULL);\n+\tif (listener_started) {\n+\t\tfsm_listen__stop_async(state);\n+\t\tpthread_join(state->listener_thread, NULL);\n+\t}\n \n-\treturn state->error_code;\n+\tif (err)\n+\t\treturn err;\n+\tif (state->error_code)\n+\t\treturn state->error_code;\n+\treturn 0;\n }\n \n static int fsmonitor_run_daemon(void)\n-- \ngitgitgadget\n\n"},{"id":"455833","messageId":"427dec412a559bffab789126ff8fe34eaef346fd.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 13/30] fsmonitor--daemon: cd out of worktree root","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:28Z","receivedAt":"2022-05-23T20:14:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the fsmonitor--daemon to CD outside of the worktree\nbefore starting up.\n\nThe common Git startup mechanism causes the CWD of the daemon process\nto be in the root of the worktree. On Windows, this causes the daemon\nprocess to hold a locked handle on the CWD and prevents other\nprocesses from moving or deleting the worktree while the daemon is\nrunning.\n\nCD to HOME before entering main event loops.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++--\n compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------\n fsmonitor--daemon.h | 1 +\n 3 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 66b78a0353f..db297649daf 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1181,11 +1181,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * before we need it.\n \t */\n \tif (ipc_server_run_async(&state->ipc_server_data,\n-\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t state->path_ipc.buf, &ipc_opts,\n \t\t\t\t handle_client, state))\n \t\treturn error_errno(\n \t\t\t_(\"could not start IPC thread pool on '%s'\"),\n-\t\t\tfsmonitor_ipc__get_path());\n+\t\t\tstate->path_ipc.buf);\n \n \t/*\n \t * Start the fsmonitor listener thread to collect filesystem\n@@ -1220,6 +1220,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n static int fsmonitor_run_daemon(void)\n {\n \tstruct fsmonitor_daemon_state state;\n+\tconst char *home;\n \tint err;\n \n \tmemset(&state, 0, sizeof(state));\n@@ -1289,6 +1290,15 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_addch(&state.path_cookie_prefix, '/');\n \n+\t/*\n+\t * We create a named-pipe or unix domain socket inside of the\n+\t * \".git\" directory. (Well, on Windows, we base our named\n+\t * pipe in the NPFS on the absolute path of the git\n+\t * directory.)\n+\t */\n+\tstrbuf_init(&state.path_ipc, 0);\n+\tstrbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1298,6 +1308,23 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\t/*\n+\t * CD out of the worktree root directory.\n+\t *\n+\t * The common Git startup mechanism causes our CWD to be the\n+\t * root of the worktree. On Windows, this causes our process\n+\t * to hold a locked handle on the CWD. This prevents the\n+\t * worktree from being moved or deleted while the daemon is\n+\t * running.\n+\t *\n+\t * We assume that our FS and IPC listener threads have either\n+\t * opened all of the handles that they need or will do\n+\t * everything using absolute paths.\n+\t */\n+\thome = getenv(\"HOME\");\n+\tif (home && *home && chdir(home))\n+\t\tdie_errno(_(\"could not cd home '%s'\"), home);\n+\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n@@ -1310,6 +1337,7 @@ done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n \tstrbuf_release(&state.path_cookie_prefix);\n+\tstrbuf_release(&state.path_ipc);\n \n \treturn err;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 0a86aea3f7e..7f48306df53 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -424,12 +424,22 @@ static int recv_rdcw_watch(struct one_watch *watch)\n \t}\n \n \t/*\n-\t * NEEDSWORK: If an external is deleted, the above\n-\t * returns an error. I'm not sure that there's anything that\n-\t * we can do here other than failing -- the /.git\n-\t * link file would be broken anyway. We might try to check\n-\t * for that and return a better error message, but I'm not\n-\t * sure it is worth it.\n+\t * GetOverlappedResult() fails if the watched directory is\n+\t * deleted while we were waiting for an overlapped IO to\n+\t * complete. The documentation did not list specific errors,\n+\t * but I observed ERROR_ACCESS_DENIED (0x05) errors during\n+\t * testing.\n+\t *\n+\t * Note that we only get notificaiton events for events\n+\t * *within* the directory, not *on* the directory itself.\n+\t * (These might be properies of the parent directory, for\n+\t * example).\n+\t *\n+\t * NEEDSWORK: We might try to check for the deleted directory\n+\t * case and return a better error message, but I'm not sure it\n+\t * is worth it.\n+\t *\n+\t * Shutdown if we get any error.\n \t */\n \n \terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex bd09fffc176..223c2131b58 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n+\tstruct strbuf path_ipc;\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"455834","messageId":"9a12cc78b5d16a05d4ed76cdb2a1f634ed32ef8a.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 10/30] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:25Z","receivedAt":"2022-05-23T20:14:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nOn MacOS mark repos on NTFS or FAT32 volumes as incompatible.\n\nThe builtin FSMonitor used Unix domain sockets on MacOS for IPC\nwith clients. These sockets are kept in the .git directory.\nUnix sockets are not supported by NTFS and FAT32, so the daemon\ncannot start up.\n\nTest for this during our compatibility checking so that client\ncommands do not keep trying to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---\n fsmonitor-settings.c | 6 ++++++\n fsmonitor-settings.h | 1 +\n 3 files changed, 24 insertions(+), 3 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex fdd762bf79d..efc732c0f31 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -7,7 +7,7 @@\n #include \n \n /*\n- * Remote working directories are problematic for FSMonitor.\n+ * [1] Remote working directories are problematic for FSMonitor.\n *\n * The underlying file system on the server machine and/or the remote\n * mount type (NFS, SAMBA, etc.) dictates whether notification events\n@@ -40,8 +40,16 @@\n *\n * So (for now at least), mark remote working directories as\n * incompatible.\n+ *\n+ *\n+ * [2] FAT32 and NTFS working directories are problematic too.\n+ *\n+ * The builtin FSMonitor uses a Unix domain socket in the .git\n+ * directory for IPC. These Windows drive formats do not support\n+ * Unix domain sockets, so mark them as incompatible for the daemon.\n+ *\n */\n-static enum fsmonitor_reason check_remote(struct repository *r)\n+static enum fsmonitor_reason check_volume(struct repository *r)\n {\n \tstruct statfs fs;\n \n@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)\n \tif (!(fs.f_flags & MNT_LOCAL))\n \t\treturn FSMONITOR_REASON_REMOTE;\n \n+\tif (!strcmp(fs.f_fstypename, \"msdos\")) /* aka FAT32 */\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n+\tif (!strcmp(fs.f_fstypename, \"ntfs\"))\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \n@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n \n-\treason = check_remote(r);\n+\treason = check_volume(r);\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex d2fb0141f8e..658cb79da01 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -225,6 +225,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t\t r->worktree);\n \t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_NOSOCKETS:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex afd1b3874ac..d9c2605197f 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -20,6 +20,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n \tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n+\tFSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n-- \ngitgitgadget\n\n"},{"id":"455835","messageId":"7cf1be5f8e2c644a5bce9eb4484de2f6c82af359.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 18/30] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:33Z","receivedAt":"2022-05-23T20:14:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nForce shutdown fsmonitor daemon if the worktree root directory\nis moved, renamed, or deleted.\n\nUse Windows low-level GetFileInformationByHandle() to get and\ncompare the Windows system unique ID for the directory with a\ncached version when we started up. This lets us detect the\ncase where someone renames the directory that we are watching\nand then creates a new directory with the original pathname.\n\nThis is important because we are listening to a named pipe for\nrequests and they are stored in the Named Pipe File System (NPFS)\nwhich a kernel-resident pseudo filesystem not associated with\nthe actual NTFS directory.\n\nFor example, if the daemon was watching \"~/foo/\", it would have\na directory-watch handle on that directory and a named-pipe\nhandle for \"//./pipe/...foo\". Moving the directory to \"~/bar/\"\ndoes not invalidate the directory handle. (So the daemon would\nactually be watching \"~/bar\" but listening on \"//./pipe/...foo\".\nIf the user then does \"git init ~/foo\" and causes another daemon\nto start, the first daemon will still have ownership of the pipe\nand the second daemon instance will fail to start. \"git status\"\nclients in \"~/foo\" will ask \"//./pipe/...foo\" about changes and\nthe first daemon instance will tell them about \"~/bar\".\n\nThis commit causes the first daemon to shutdown if the system unique\nID for \"~/foo\" changes (changes from what it was when the daemon\nstarted). Shutdown occurs after a periodic poll. After the\nfirst daemon exits and releases the lock on the named pipe,\nsubsequent Git commands may cause another daemon to be started\non \"~/foo\". Similarly, a subsequent Git command may cause another\ndaemon to be started on \"~/bar\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++\n 1 file changed, 143 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 24fc612bf02..2ea08c1d4e8 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -29,8 +29,150 @@ struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n \tint nr_handles; /* number of active event handles */\n+\n+\tstruct wt_moved\n+\t{\n+\t\twchar_t wpath[MAX_PATH + 1];\n+\t\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\t} wt_moved;\n };\n \n+/*\n+ * Lookup the system unique ID for the path. This is as close as\n+ * we get to an inode number, but this also contains volume info,\n+ * so it is a little stronger.\n+ */\n+static int lookup_bhfi(wchar_t *wpath,\n+\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n+{\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileW(wpath, desired_access, share_mode, NULL,\n+\t\t\t OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] health thread could not open '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\treturn -1;\n+\t}\n+\n+\tif (!GetFileInformationByHandle(hDir, bhfi)) {\n+\t\terror(_(\"[GLE %ld] health thread getting BHFI for '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\tCloseHandle(hDir);\n+\t\treturn -1;\n+\t}\n+\n+\tCloseHandle(hDir);\n+\treturn 0;\n+}\n+\n+/*\n+ * Compare the relevant fields from two system unique IDs.\n+ * We use this to see if two different handles to the same\n+ * path actually refer to the same *instance* of the file\n+ * or directory.\n+ */\n+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n+\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n+{\n+\treturn (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&\n+\t\tbhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&\n+\t\tbhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);\n+}\n+\n+/*\n+ * Shutdown if the original worktree root directory been deleted,\n+ * moved, or renamed?\n+ *\n+ * Since the main thread did a \"chdir(getenv($HOME))\" and our CWD\n+ * is not in the worktree root directory and because the listener\n+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible\n+ * for the root directory to be moved or deleted while we are still\n+ * watching it. We want to detect that here and force a shutdown.\n+ *\n+ * Granted, a delete MAY cause some operations to fail, such as\n+ * GetOverlappedResult(), but it is not guaranteed. And because\n+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the\n+ * directory, not changes *ON* the directory, our watch will not\n+ * receive a delete event for it.\n+ *\n+ * A move/rename of the worktree root will also not generate an event.\n+ * And since the listener thread already has an open handle, it may\n+ * continue to receive events for events within the directory.\n+ * However, the pathname of the named-pipe was constructed using the\n+ * original location of the worktree root. (Remember named-pipes are\n+ * stored in the NPFS and not in the actual file system.) Clients\n+ * trying to talk to the worktree after the move/rename will not\n+ * reach our daemon process, since we're still listening on the\n+ * pipe with original path.\n+ *\n+ * Furthermore, if the user does something like:\n+ *\n+ * $ mv repo repo.old\n+ * $ git init repo\n+ *\n+ * A new daemon cannot be started in the new instance of \"repo\"\n+ * because the named-pipe is still being used by the daemon on\n+ * the original instance.\n+ *\n+ * So, detect move/rename/delete and shutdown. This should also\n+ * handle unsafe drive removal.\n+ *\n+ * We use the file system unique ID to distinguish the original\n+ * directory instance from a new instance and force a shutdown\n+ * if the unique ID changes.\n+ *\n+ * Since a worktree move/rename/delete/unmount doesn't happen\n+ * that often (and we can't get an immediate event anyway), we\n+ * use a timeout and periodically poll it.\n+ */\n+static int has_worktree_moved(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\tint r;\n+\n+\tswitch (ctx) {\n+\tcase CTX_TERM:\n+\t\treturn 0;\n+\n+\tcase CTX_INIT:\n+\t\tif (xutftowcs_path(data->wt_moved.wpath,\n+\t\t\t\t state->path_worktree_watch.buf) < 0) {\n+\t\t\terror(_(\"could not convert to wide characters: '%s'\"),\n+\t\t\t state->path_worktree_watch.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/*\n+\t\t * On the first call we lookup the unique sequence ID for\n+\t\t * the worktree root directory.\n+\t\t */\n+\t\treturn lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);\n+\n+\tcase CTX_TIMER:\n+\t\tr = lookup_bhfi(data->wt_moved.wpath, &bhfi);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t\tif (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {\n+\t\t\terror(_(\"BHFI changed '%ls'\"), data->wt_moved.wpath);\n+\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tdie(_(\"unhandled case in 'has_worktree_moved': %d\"),\n+\t\t (int)ctx);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+\n int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data;\n@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n * A table of the polling functions.\n */\n static interval_fn *table[] = {\n+\thas_worktree_moved,\n \tNULL, /* must be last */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"455836","messageId":"d058d7e0c0829bed9df266c4f6c9f12c30caad4d.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 21/30] t7527: FSMonitor tests for directory moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:36Z","receivedAt":"2022-05-23T20:14:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate unit tests to move a directory. Verify that `git status`\ngives the same result with and without FSMonitor enabled.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 12655958e71..3bc335b891d 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -274,6 +274,16 @@ test_expect_success 'setup' '\n \ttrace*\n \tEOF\n \n+\tmkdir -p T1/T2/T3/T4 &&\n+\techo 1 >T1/F1 &&\n+\techo 1 >T1/T2/F1 &&\n+\techo 1 >T1/T2/T3/F1 &&\n+\techo 1 >T1/T2/T3/T4/F1 &&\n+\techo 2 >T1/F2 &&\n+\techo 2 >T1/T2/F2 &&\n+\techo 2 >T1/T2/T3/F2 &&\n+\techo 2 >T1/T2/T3/T4/F2 &&\n+\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n@@ -356,6 +366,19 @@ directory_to_file () {\n \techo 1 >dir1\n }\n \n+move_directory_contents_deeper() {\n+\tmkdir T1/_new_ &&\n+\tmv T1/[A-Z]* T1/_new_\n+}\n+\n+move_directory_up() {\n+\tmv T1/T2/T3 T1\n+}\n+\n+move_directory() {\n+\tmv T1/T2/T3 T1/T2/NewT3\n+}\n+\n # The next few test cases confirm that our fsmonitor daemon sees each type\n # of OS filesystem notification that we care about. At this layer we just\n # ensure we are getting the OS notifications and do not try to confirm what\n@@ -660,6 +683,10 @@ do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n+\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n+\t\tmatrix_try $uc_val $fsm_val move_directory_up\n+\t\tmatrix_try $uc_val $fsm_val move_directory\n+\n \t\tif test $fsm_val = true\n \t\tthen\n \t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n-- \ngitgitgadget\n\n"},{"id":"455837","messageId":"f5dac2868124b57a68a5b7594f02e934c63f7e95.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 22/30] t/perf/p7527: add perf test for builtin FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:37Z","receivedAt":"2022-05-23T20:14:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++\n 1 file changed, 257 insertions(+)\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\ndiff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..9338b9ea008\n--- /dev/null\n+++ b/t/perf/p7527-builtin-fsmonitor.sh\n@@ -0,0 +1,257 @@\n+#!/bin/sh\n+\n+test_description=\"Perf test for the builtin FSMonitor\"\n+\n+. ./perf-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+# Lie to perf-lib and ask for a new empty repo and avoid\n+# the complaints about GIT_PERF_REPO not being big enough\n+# the perf hit when GIT_PERF_LARGE_REPO is copied into\n+# the trash directory.\n+#\n+# NEEDSWORK: It would be nice if perf-lib had an option to\n+# \"borrow\" an existing large repo (especially for gigantic\n+# monorepos) and use it in-place. For now, fake it here.\n+#\n+test_perf_fresh_repo\n+\n+\n+# Use a generated synthetic monorepo. If it doesn't exist, we will\n+# generate it. If it does exist, we will put it in a known state\n+# before we start our timings.\n+#\n+PARAM_D=5\n+PARAM_W=10\n+PARAM_F=9\n+\n+PARAMS=\"$PARAM_D\".\"$PARAM_W\".\"$PARAM_F\"\n+\n+BALLAST_BR=p0006-ballast\n+export BALLAST_BR\n+\n+TMP_BR=tmp_br\n+export TMP_BR\n+\n+REPO=../repos/gen-many-files-\"$PARAMS\".git\n+export REPO\n+\n+if ! test -d $REPO\n+then\n+\t(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)\n+fi\n+\n+\n+enable_uc () {\n+\tgit -C $REPO config core.untrackedcache true\n+\tgit -C $REPO update-index --untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+disable_uc () {\n+\tgit -C $REPO config core.untrackedcache false\n+\tgit -C $REPO update-index --no-untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+start_fsm () {\n+\tgit -C $REPO fsmonitor--daemon start\n+\tgit -C $REPO fsmonitor--daemon status\n+\tgit -C $REPO config core.fsmonitor true\n+\tgit -C $REPO update-index --fsmonitor\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+stop_fsm () {\n+\tgit -C $REPO config --unset core.fsmonitor\n+\tgit -C $REPO update-index --no-fsmonitor\n+\ttest_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+\n+# Ensure that FSMonitor is turned off on the borrowed repo.\n+#\n+test_expect_success \"Setup borrowed repo (fsm+uc)\" \"\n+\tstop_fsm &&\n+\tdisable_uc\n+\"\n+\n+# Also ensure that it starts in a known state.\n+#\n+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time\n+# the ballast checkout, since only the first invocation does any work and the\n+# subsequent ones just print \"already on branch\" and quit, so the reported\n+# time is not useful.\n+#\n+# Create a temp branch and do all work relative to it so that we don't\n+# accidentially alter the real ballast branch.\n+#\n+test_expect_success \"Setup borrowed repo (temp ballast branch)\" \"\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO reset --hard &&\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tgit -C $REPO branch $TMP_BR $BALLAST_BR &&\n+\tgit -C $REPO checkout $TMP_BR\n+\"\n+\n+\n+echo Data >data.txt\n+\n+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With\n+# FSMonitor enabled, we can get a skewed view of status times, since\n+# the index MAY (or may not) be updated after the first invocation\n+# which will update the FSMonitor Token, so the subsequent invocations\n+# may get a smaller response from the daemon.\n+#\n+do_status () {\n+\tmsg=$1\n+\n+\ttest_perf \"$msg\" \"\n+\t\tgit -C $REPO status >/dev/null 2>&1\n+\t\"\n+}\n+\n+do_matrix () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tt=\"[uc $uc][fsm $fsm]\"\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_expect_success \"$t Setup matrix branch\" \"\n+\t\tgit -C $REPO clean -d -f &&\n+\t\tgit -C $REPO checkout $TMP_BR &&\n+\t\ttest_might_fail git -C $REPO branch -D $MATRIX_BR &&\n+\t\tgit -C $REPO branch $MATRIX_BR $TMP_BR &&\n+\t\tgit -C $REPO checkout $MATRIX_BR\n+\t\"\n+\n+\tif test $uc = true\n+\tthen\n+\t\tenable_uc\n+\telse\n+\t\tdisable_uc\n+\tfi\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstart_fsm\n+\telse\n+\t\tstop_fsm\n+\tfi\n+\n+\tdo_status \"$t status after checkout\"\n+\n+\t# Modify many files in the matrix branch.\n+\t# Stage them.\n+\t# Commit them.\n+\t# Rollback.\n+\t#\n+\ttest_expect_success \"$t modify tracked files\" \"\n+\t\tfind $REPO -name file1 -exec cp data.txt {} \\\\;\n+\t\"\n+\n+\tdo_status \"$t status after big change\"\n+\n+\t# Don't bother timing the \"add\" because _REPEAT_COUNT\n+\t# issue described above.\n+\t#\n+\ttest_expect_success \"$t add all\" \"\n+\t\tgit -C $REPO add -A\n+\t\"\n+\n+\tdo_status \"$t status after add all\"\n+\n+\ttest_expect_success \"$t add dot\" \"\n+\t\tgit -C $REPO add .\n+\t\"\n+\n+\tdo_status \"$t status after add dot\"\n+\n+\ttest_expect_success \"$t commit staged\" \"\n+\t\tgit -C $REPO commit -a -m data\n+\t\"\n+\n+\tdo_status \"$t status after commit\"\n+\n+\ttest_expect_success \"$t reset HEAD~1 hard\" \"\n+\t\tgit -C $REPO reset --hard HEAD~1 >/dev/null 2>&1\n+\t\"\n+\n+\tdo_status \"$t status after reset hard\"\n+\n+\t# Create some untracked files.\n+\t#\n+\ttest_expect_success \"$t create untracked files\" \"\n+\t\tcp -R $REPO/ballast/dir1 $REPO/ballast/xxx1\n+\t\"\n+\n+\tdo_status \"$t status after create untracked files\"\n+\n+\t# Remove the new untracked files.\n+\t#\n+\ttest_expect_success \"$t clean -df\" \"\n+\t\tgit -C $REPO clean -d -f\n+\t\"\n+\n+\tdo_status \"$t status after clean\"\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstop_fsm\n+\tfi\n+}\n+\n+# Begin testing each case in the matrix that we care about.\n+#\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\n+fsm_values=\"false true\"\n+\n+for uc_val in $uc_values\n+do\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tdo_matrix $uc_val $fsm_val\n+\tdone\n+done\n+\n+cleanup () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_might_fail git -C $REPO branch -D $MATRIX_BR\n+}\n+\n+\n+# We're borrowing this repo. We should leave it in a clean state.\n+#\n+test_expect_success \"Cleanup temp and matrix branches\" \"\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\tfor fsm_val in $fsm_values\n+\t\tdo\n+\t\t\tcleanup $uc_val $fsm_val\n+\t\tdone\n+\tdone\n+\"\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"455838","messageId":"40b80adbb31e866afb2b74b681695d3dadffc8df.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 24/30] t7527: test FSMonitor on case insensitive+preserving file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:39Z","receivedAt":"2022-05-23T20:14:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTest that FS events from the OS are received using the preserved,\non-disk spelling of files/directories rather than spelling used\nto make the change.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex cf4fb72c3f0..fbb7d6aef6e 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,4 +832,40 @@ test_expect_success 'submodule always visited' '\n \tmy_match_and_clean\n '\n \n+# On a case-insensitive file system, confirm that the daemon\n+# notices when the .git directory is moved/renamed/deleted\n+# regardless of how it is spelled in the the FS event.\n+# That is, does the FS event receive the spelling of the\n+# operation or does it receive the spelling preserved with\n+# the file/directory.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n+#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n+\n+\tgit init test_insensitive &&\n+\n+\tstart_daemon -C test_insensitive --tf \"$PWD/insensitive.trace\" &&\n+\n+\tmkdir -p test_insensitive/abc/def &&\n+\techo xyz >test_insensitive/ABC/DEF/xyz &&\n+\n+\ttest_path_is_dir test_insensitive/.git &&\n+\ttest_path_is_dir test_insensitive/.GIT &&\n+\n+\t# Rename .git using an alternate spelling to verify that that\n+\t# daemon detects it and automatically shuts down.\n+\tmv test_insensitive/.GIT test_insensitive/.FOO &&\n+\tsleep 1 &&\n+\tmv test_insensitive/.FOO test_insensitive/.git &&\n+\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\t# Verify that events were reported using on-disk spellings of the\n+\t# directories and files that we touched. We may or may not get a\n+\t# trailing slash on modified directories.\n+\t#\n+\tegrep \"^event: abc/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"455839","messageId":"f9a7869d202b5c3fc953967ab6e9e032e7450e79.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 28/30] t7527: test Unicode NFC/NFD handling on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:43Z","receivedAt":"2022-05-23T20:14:58Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that the daemon reports events using the on-disk\nspelling for Unicode NFC/NFD characters. On APFS we still\nhave Unicode aliasing, so we cannot create two files that\nonly differ by NFC/NFD, but the on-disk format preserves\nthe spelling used to create the file. On HFS+ we also\nhave aliasing, but the path is always stored on disk in\nNFD.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 55 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex fbb7d6aef6e..9edae3ed830 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -868,4 +868,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n+# The variable \"unicode_debug\" is defined in the following library\n+# script to dump information about how the (OS, FS) handles Unicode\n+# composition. Uncomment the following line if you want to enable it.\n+#\n+# unicode_debug=true\n+\n+. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n+\n+# See if the OS or filesystem does NFC/NFD aliasing/munging.\n+#\n+# The daemon should err on the side of caution and send BOTH the\n+# NFC and NFD forms. It does not know the original spelling of\n+# the pathname (how the user thinks it should be spelled), so\n+# emit both and let the client decide (when necessary). This is\n+# similar to \"core.precomposeUnicode\".\n+#\n+test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n+\n+\tgit init test_unicode &&\n+\n+\tstart_daemon -C test_unicode --tf \"$PWD/unicode.trace\" &&\n+\n+\t# Create a directory using an NFC spelling.\n+\t#\n+\tmkdir test_unicode/nfc &&\n+\tmkdir test_unicode/nfc/c_${utf8_nfc} &&\n+\n+\t# Create a directory using an NFD spelling.\n+\t#\n+\tmkdir test_unicode/nfd &&\n+\tmkdir test_unicode/nfd/d_${utf8_nfd} &&\n+\n+\tgit -C test_unicode fsmonitor--daemon stop &&\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\t# We should have seen NFC event from OS.\n+\t\t# We should not have synthesized an NFD event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace &&\n+\t\tegrep -v \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace\n+\telse\n+\t\t# We should have seen NFD event from OS.\n+\t\t# We should have synthesized an NFC event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace &&\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace\n+\tfi &&\n+\n+\t# We assume UNICODE_NFD_PRESERVED.\n+\t# We should have seen explicit NFD from OS.\n+\t# We should have synthesized an NFC event.\n+\tegrep \"^event: nfd/d_${utf8_nfd}/?$\" ./unicode.trace &&\n+\tegrep \"^event: nfd/d_${utf8_nfc}/?$\" ./unicode.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"455840","messageId":"b020bfb45683c1c36b8310b6cfe9d29b8876f358.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 20/30] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:35Z","receivedAt":"2022-05-23T20:15:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to perform binary search over the cache-entries for a directory\nnotification and then linearly scan forward to find the immediate children.\n\nPreviously, when the FSMonitor reported a modified directory Git would\nperform a linear search on the entire cache-entry array for all\nentries matching that directory prefix and invalidate them. Since the\ncache-entry array is already sorted, we can use a binary search to\nfind the first matching entry and then only linearly walk forward and\ninvalidate entries until the prefix changes.\n\nAlso, the original code would invalidate anything having the same\ndirectory prefix. Since a directory event should only be received for\nitems that are immediately within the directory (and not within\nsub-directories of it), only invalidate those entries and not the\nwhole subtree.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 54 insertions(+), 17 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 292a6742b4f..e1229c289cf 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n-\tif (name[len - 1] == '/') {\n+\tint pos = index_name_pos(istate, name, len);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n+\t\t\t name, pos);\n \n+\tif (name[len - 1] == '/') {\n \t\t/*\n-\t\t * TODO We should binary search to find the first path with\n-\t\t * TODO this directory prefix. Then linearly update entries\n-\t\t * TODO while the prefix matches. Taking care to search without\n-\t\t * TODO the trailing slash -- because '/' sorts after a few\n-\t\t * TODO interesting special chars, like '.' and ' '.\n+\t\t * The daemon can decorate directory events, such as\n+\t\t * moves or renames, with a trailing slash if the OS\n+\t\t * FS Event contains sufficient information, such as\n+\t\t * MacOS.\n+\t\t *\n+\t\t * Use this to invalidate the entire cone under that\n+\t\t * directory.\n+\t\t *\n+\t\t * We do not expect an exact match because the index\n+\t\t * does not normally contain directory entries, so we\n+\t\t * start at the insertion point and scan.\n \t\t */\n+\t\tif (pos < 0)\n+\t\t\tpos = -pos - 1;\n \n \t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n-\t\t\t starts_with(istate->cache[i]->name, name))\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n-\t\t/* Need to remove the / from the path for the untracked cache */\n+\n+\t\t/*\n+\t\t * We need to remove the traling \"/\" from the path\n+\t\t * for the untracked cache.\n+\t\t */\n \t\tname[len - 1] = '\\0';\n+\t} else if (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\tint pos = index_name_pos(istate, name, strlen(name));\n-\n-\t\tif (pos >= 0) {\n-\t\t\tstruct cache_entry *ce = istate->cache[pos];\n-\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n \t}\n \n@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n \t * as it could be a new untracked file.\n \t */\n-\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"455841","messageId":"6f2e935f148e826609153378751c04807858e76c.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:41Z","receivedAt":"2022-05-23T20:15:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n t/helper/test-hexdump.c | 24 ++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 27 insertions(+)\n create mode 100644 t/helper/test-hexdump.c\n\ndiff --git a/Makefile b/Makefile\nindex 5f1623baadd..5afa194aac6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -729,6 +729,7 @@ TEST_BUILTINS_OBJS += test-getcwd.o\n TEST_BUILTINS_OBJS += test-hash-speed.o\n TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n+TEST_BUILTINS_OBJS += test-hexdump.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\n TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\ndiff --git a/t/helper/test-hexdump.c b/t/helper/test-hexdump.c\nnew file mode 100644\nindex 00000000000..f1e0a0fabf3\n--- /dev/null\n+++ b/t/helper/test-hexdump.c\n@@ -0,0 +1,24 @@\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Read stdin and print a hexdump to stdout.\n+ */\n+int cmd__hexdump(int argc, const char **argv)\n+{\n+\tchar buf[1024];\n+\tssize_t i, len;\n+\n+\tfor (;;) {\n+\t\tlen = xread(0, buf, sizeof(buf));\n+\t\tif (len < 0)\n+\t\t\tdie_errno(\"failure reading stdin\");\n+\t\tif (!len)\n+\t\t\tbreak;\n+\n+\t\tfor (i = 0; i < len; i++)\n+\t\t\tprintf(\"%02x \", (unsigned char)buf[i]);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 0424f7adf5d..88c4b28cdfa 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -38,6 +38,7 @@ static struct test_cmd cmds[] = {\n \t{ \"getcwd\", cmd__getcwd },\n \t{ \"hashmap\", cmd__hashmap },\n \t{ \"hash-speed\", cmd__hash_speed },\n+\t{ \"hexdump\", cmd__hexdump },\n \t{ \"index-version\", cmd__index_version },\n \t{ \"json-writer\", cmd__json_writer },\n \t{ \"lazy-init-name-hash\", cmd__lazy_init_name_hash },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex c876e8246fb..511f6251bf5 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -29,6 +29,7 @@ int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n int cmd__hash_speed(int argc, const char **argv);\n+int cmd__hexdump(int argc, const char **argv);\n int cmd__index_version(int argc, const char **argv);\n int cmd__json_writer(int argc, const char **argv);\n int cmd__lazy_init_name_hash(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"455842","messageId":"95cf1299d445696630d8c0850a69dc2c0ee040ce.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 19/30] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:34Z","receivedAt":"2022-05-23T20:15:08Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the listener thread to shutdown the daemon if the spelling of the\nworktree root directory changes.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 07113205a61..83d38e8ac6c 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tstrbuf_release(&msg);\n }\n \n+static int ef_is_root_changed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagRootChanged);\n+}\n+\n static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n {\n \treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n@@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_is_root_changed(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * The spelling of the pathname of the root directory\n+\t\t\t * has changed. This includes the name of the root\n+\t\t\t * directory itself or of any parent directory in the\n+\t\t\t * path.\n+\t\t\t *\n+\t\t\t * (There may be other conditions that throw this,\n+\t\t\t * but I couldn't find any information on it.)\n+\t\t\t *\n+\t\t\t * Force a shutdown now and avoid things getting\n+\t\t\t * out of sync. The Unix domain socket is inside\n+\t\t\t * the .git directory and a spelling change will make\n+\t\t\t * it hard for clients to rendezvous with us.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"event: root changed\");\n+\t\t\tgoto force_shutdown;\n+\t\t}\n+\n \t\tif (ef_ignore_xattr(event_flags[k])) {\n \t\t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n-- \ngitgitgadget\n\n"},{"id":"455843","messageId":"92f5c0d2c8b1a69aec4325ab35193fe257d481f1.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 23/30] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:38Z","receivedAt":"2022-05-23T20:15:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNever set CE_FSMONITOR_VALID on the cache-entry of submodule\ndirectories.\n\nDuring a client command like 'git status', we may need to recurse\ninto each submodule to compute a status summary for the submodule.\nSince the purpose of the ce_flag is to let Git avoid scanning a\ncache-entry, setting the flag causes the recursive call to be\navoided and we report incorrect (no status) for the submodule.\n\nWe created an OS watch on the root directory of our working\ndirectory and we receive events for everything in the cone\nunder it. When submodules are present inside our working\ndirectory, we receive events for both our repo (the super) and\nany subs within it. Since our index doesn't have any information\nfor items within the submodules, we can't use those events.\n\nWe could try to truncate the paths of those events back to the\nsubmodule boundary and mark the GITLINK as dirty, but that\nfeels expensive since we would have to prefix compare every FS\nevent that we receive against a list of submodule roots. And\nit still wouldn't be sufficient to correctly report status on\nthe submodule, since we don't have any space in the cache-entry\nto cache the submodule's status (the 'SCMU' bits in porcelain\nV2 speak). That is, the CE_FSMONITOR_VALID bit just says that\nwe don't need to scan/inspect it because we already know the\nanswer -- it doesn't say that the item is clean -- and we\ndon't have space in the cache-entry to store those answers.\nSo we should always do the recursive scan.\n\nTherefore, we should never set the flag on GITLINK cache-entries.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 2 +\n fsmonitor.h | 11 ++++\n t/t7527-builtin-fsmonitor.sh | 111 +++++++++++++++++++++++++++++++++++\n 3 files changed, 124 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex e1229c289cf..57d6a483bee 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tif (fsmonitor_enabled) {\n \t\t\t/* Mark all entries valid */\n \t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\t\tif (S_ISGITLINK(istate->cache[i]->ce_mode))\n+\t\t\t\t\tcontinue;\n \t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n \t\t\t}\n \ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 3f41f653691..edf7ce5203b 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n * called any time the cache entry has been updated to reflect the\n * current state of the file on disk.\n+ *\n+ * However, never mark submodules as valid. When commands like \"git\n+ * status\" run they might need to recurse into the submodule (using a\n+ * child process) to get a summary of the submodule state. We don't\n+ * have (and don't want to create) the facility to translate every\n+ * FS event that we receive and that happens to be deep inside of a\n+ * submodule back to the submodule root, so we cannot correctly keep\n+ * track of this bit on the gitlink directory. Therefore, we never\n+ * set it on submodules.\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n \n \tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n \t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tif (S_ISGITLINK(ce->ce_mode))\n+\t\t\treturn;\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 3bc335b891d..cf4fb72c3f0 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -721,4 +721,115 @@ do\n \t'\n done\n \n+# Test fsmonitor interaction with submodules.\n+#\n+# If we start the daemon in the super, it will see FS events for\n+# everything in the working directory cone and this includes any\n+# files/directories contained *within* the submodules.\n+#\n+# A `git status` at top level will get events for items within the\n+# submodule and ignore them, since they aren't named in the index\n+# of the super repo. This makes the fsmonitor response a little\n+# noisy, but it doesn't alter the correctness of the state of the\n+# super-proper.\n+#\n+# When we have submodules, `git status` normally does a recursive\n+# status on each of the submodules and adds a summary row for any\n+# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n+#\n+# It is therefore important that the top level status not be tricked\n+# by the FSMonitor response to skip those recursive calls. That is,\n+# even if FSMonitor says that the mtime of the submodule directory\n+# hasn't changed and it could be implicitly marked valid, we must\n+# not take that shortcut. We need to force the recusion into the\n+# submodule so that we get a summary of the status *within* the\n+# submodule.\n+\n+create_super () {\n+\tsuper=\"$1\" &&\n+\n+\tgit init \"$super\" &&\n+\techo x >\"$super/file_1\" &&\n+\techo y >\"$super/file_2\" &&\n+\techo z >\"$super/file_3\" &&\n+\tmkdir \"$super/dir_1\" &&\n+\techo a >\"$super/dir_1/file_11\" &&\n+\techo b >\"$super/dir_1/file_12\" &&\n+\tmkdir \"$super/dir_1/dir_2\" &&\n+\techo a >\"$super/dir_1/dir_2/file_21\" &&\n+\techo b >\"$super/dir_1/dir_2/file_22\" &&\n+\tgit -C \"$super\" add . &&\n+\tgit -C \"$super\" commit -m \"initial $super commit\"\n+}\n+\n+create_sub () {\n+\tsub=\"$1\" &&\n+\n+\tgit init \"$sub\" &&\n+\techo x >\"$sub/file_x\" &&\n+\techo y >\"$sub/file_y\" &&\n+\techo z >\"$sub/file_z\" &&\n+\tmkdir \"$sub/dir_x\" &&\n+\techo a >\"$sub/dir_x/file_a\" &&\n+\techo b >\"$sub/dir_x/file_b\" &&\n+\tmkdir \"$sub/dir_x/dir_y\" &&\n+\techo a >\"$sub/dir_x/dir_y/file_a\" &&\n+\techo b >\"$sub/dir_x/dir_y/file_b\" &&\n+\tgit -C \"$sub\" add . &&\n+\tgit -C \"$sub\" commit -m \"initial $sub commit\"\n+}\n+\n+my_match_and_clean () {\n+\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n+\tgit -C super --no-optional-locks -c core.fsmonitor=false \\\n+\t\tstatus --porcelain=v2 >actual.without &&\n+\ttest_cmp actual.with actual.without &&\n+\n+\tgit -C super/dir_1/dir_2/sub reset --hard &&\n+\tgit -C super/dir_1/dir_2/sub clean -d -f\n+}\n+\n+test_expect_success 'submodule always visited' '\n+\ttest_when_finished \"git -C super fsmonitor--daemon stop; \\\n+\t\t\t rm -rf super; \\\n+\t\t\t rm -rf sub\" &&\n+\n+\tcreate_super super &&\n+\tcreate_sub sub &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\tstart_daemon -C super &&\n+\tgit -C super config core.fsmonitor true &&\n+\tgit -C super update-index --fsmonitor &&\n+\tgit -C super status &&\n+\n+\t# Now run pairs of commands w/ and w/o FSMonitor while we make\n+\t# some dirt in the submodule and confirm matching output.\n+\n+\t# Completely clean status.\n+\tmy_match_and_clean &&\n+\n+\t# .M S..U\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M SC..\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tgit -C super/dir_1/dir_2/sub commit -m \"SC..\" &&\n+\tmy_match_and_clean\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"455844","messageId":"9fc7c970929d025f023666ce5d8432a3bcfa1714.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 29/30] fsmonitor--daemon: allow --super-prefix argument","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:44Z","receivedAt":"2022-05-23T20:15:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a test in t7527 to verify that we get a stray warning from\n`git fsmonitor--daemon start` when indirectly called from\n`git submodule absorbgitdirs`.\n\nUpdate `git fsmonitor--daemon` to take (and ignore) the `--super-prefix`\nargument to suppress the warning.\n\nWhen we have:\n\n1. a submodule with a `sub/.git/` directory (rather than a `sub/.git`\nfile).\n\n2. `core.fsmonitor` is turned on in the submodule, but the daemon is\nnot yet started in the submodule.\n\n3. and someone does a `git submodule absorbgitdirs` in the super.\n\nGit will recursively invoke `git submodule--helper absorb-git-dirs`\nin the submodule. This will read the index and may attempt to start\nthe fsmonitor--daemon with the `--super-prefix` argument.\n\n`git fsmonitor--daemon start` does not accept the `--super-prefix`\nargument and causes a warning to be issued.\n\nThis does not cause a problem because the `refresh_index()` code\nassumes a trivial response if the daemon does not start.\n\nThe net-net is a harmelss, but stray warning. Lets eliminate the\nwarning.\n\nSigned-off-by: Jeff Hostetler \n---\n git.c | 2 +-\n t/t7527-builtin-fsmonitor.sh | 50 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 51 insertions(+), 1 deletion(-)\n\ndiff --git a/git.c b/git.c\nindex 3d8e48cf555..e6fdac1f8e3 100644\n--- a/git.c\n+++ b/git.c\n@@ -537,7 +537,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n-\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, SUPPORT_SUPER_PREFIX | RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 9edae3ed830..19edc96fd4d 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,6 +832,56 @@ test_expect_success 'submodule always visited' '\n \tmy_match_and_clean\n '\n \n+# If a submodule has a `sub/.git/` directory (rather than a file\n+# pointing to the super's `.git/modules/sub`) and `core.fsmonitor`\n+# turned on in the submodule and the daemon is not yet started in\n+# the submodule, and someone does a `git submodule absorbgitdirs`\n+# in the super, Git will recursively invoke `git submodule--helper`\n+# to do the work and this may try to read the index. This will\n+# try to start the daemon in the submodule *and* pass (either\n+# directly or via inheritance) the `--super-prefix` arg to the\n+# `git fsmonitor--daemon start` command inside the submodule.\n+# This causes a warning because fsmonitor--daemon does take that\n+# global arg (see the table in git.c)\n+#\n+# This causes a warning when trying to start the daemon that is\n+# somewhat confusing. It does not seem to hurt anything because\n+# the fsmonitor code maps the query failure into a trivial response\n+# and does the work anyway.\n+#\n+# It would be nice to silence the warning, however.\n+\n+have_t2_error_event () {\n+\tlog=$1\n+\tmsg=\"fsmonitor--daemon doesnQt support --super-prefix\" &&\n+\n+\ttr '\\047' Q <$1 | grep -e \"$msg\"\n+}\n+\n+test_expect_success \"stray submodule super-prefix warning\" '\n+\ttest_when_finished \"rm -rf super; \\\n+\t\t\t rm -rf sub; \\\n+\t\t\t rm super-sub.trace\" &&\n+\n+\tcreate_super super &&\n+\tcreate_sub sub &&\n+\n+\t# Copy rather than submodule add so that we get a .git dir.\n+\tcp -R ./sub ./super/dir_1/dir_2/sub &&\n+\n+\tgit -C super/dir_1/dir_2/sub config core.fsmonitor true &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\ttest_path_is_dir super/dir_1/dir_2/sub/.git &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/super-sub.trace\" \\\n+\t\tgit -C super submodule absorbgitdirs &&\n+\n+\t! have_t2_error_event super-sub.trace\n+'\n+\n # On a case-insensitive file system, confirm that the daemon\n # notices when the .git directory is moved/renamed/deleted\n # regardless of how it is spelled in the the FS event.\n-- \ngitgitgadget\n\n"},{"id":"455845","messageId":"b93f0642699dcacd23334803dce023229a4228a5.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 25/30] fsmonitor: on macOS also emit NFC spelling for NFD pathname","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:40Z","receivedAt":"2022-05-23T20:15:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nEmit NFC or NFC and NFD spellings of pathnames on macOS.\n\nMacOS is Unicode composition insensitive, so NFC and NFD spellings are\ntreated as aliases and collide. While the spelling of pathnames in\nfilesystem events depends upon the underlying filesystem, such as\nAPFS, HFS+ or FAT32, the OS enforces such collisions regardless of\nfilesystem.\n\nTeach the daemon to always report the NFC spelling and to report\nthe NFD spelling when stored in that format on the disk.\n\nThis is slightly more general than \"core.precomposeUnicode\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 83d38e8ac6c..823cf63999e 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n \treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n }\n \n+/*\n+ * On MacOS we have to adjust for Unicode composition insensitivity\n+ * (where NFC and NFD spellings are not respected). The different\n+ * spellings are essentially aliases regardless of how the path is\n+ * actually stored on the disk.\n+ *\n+ * This is related to \"core.precomposeUnicode\" (which wants to try\n+ * to hide NFD completely and treat everything as NFC). Here, we\n+ * don't know what the value the client has (or will have) for this\n+ * config setting when they make a query, so assume the worst and\n+ * emit both when the OS gives us an NFD path.\n+ */\n+static void my_add_path(struct fsmonitor_batch *batch, const char *path)\n+{\n+\tchar *composed;\n+\n+\t/* add the NFC or NFD path as received from the OS */\n+\tfsmonitor_batch__add_path(batch, path);\n+\n+\t/* if NFD, also add the corresponding NFC spelling */\n+\tcomposed = (char *)precompose_string_if_needed(path);\n+\tif (!composed || composed == path)\n+\t\treturn;\n+\n+\tfsmonitor_batch__add_path(batch, composed);\n+\tfree(composed);\n+}\n+\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t\tmy_add_path(batch, rel);\n \t\t\t}\n \n \t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n@@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t\tmy_add_path(batch, tmp.buf);\n \t\t\t}\n \n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"455846","messageId":"6a8308699543faaea760d4605babe50a0e478f41.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 27/30] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:42Z","receivedAt":"2022-05-23T20:15:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a set of prereqs to help understand how file names\nare handled by the filesystem when they contain NFC and NFD\nUnicode characters.\n\nSigned-off-by: Jeff Hostetler \n---\n t/lib-unicode-nfc-nfd.sh | 158 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 158 insertions(+)\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n\ndiff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\nnew file mode 100755\nindex 00000000000..76c6fbc0ec2\n--- /dev/null\n+++ b/t/lib-unicode-nfc-nfd.sh\n@@ -0,0 +1,158 @@\n+# Help detect how Unicode NFC and NFD are handled on the filesystem.\n+\n+# A simple character that has a NFD form.\n+#\n+# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n+# UTF8(NFC): \\xc3 \\xa9\n+#\n+# NFD: U+0065 LATIN SMALL LETTER E\n+# U+0301 COMBINING ACUTE ACCENT\n+# UTF8(NFD): \\x65 + \\xcc \\x81\n+#\n+utf8_nfc=$(printf \"\\xc3\\xa9\")\n+utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n+\n+# Is the OS or the filesystem \"Unicode composition sensitive\"?\n+#\n+# That is, does the OS or the filesystem allow files to exist with\n+# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n+# tell us that the NFC and NFD forms are equivalent.\n+#\n+# This is or may be independent of what type of filesystem we have,\n+# since it might be handled by the OS at a layer above the FS.\n+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n+# collision, for example.\n+#\n+# This does not tell us how the Unicode pathname will be spelled\n+# on disk, but rather only that the two spelling \"collide\". We\n+# will examine the actual on disk spelling in a later prereq.\n+#\n+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${utf8_nfc} &&\n+\tmkdir trial_${utf8_nfd}\n+'\n+\n+# Is the spelling of an NFC pathname preserved on disk?\n+#\n+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n+# and on APFS, NFC paths are preserved. As we have established\n+# above, this is independent of \"composition sensitivity\".\n+#\n+test_lazy_prereq UNICODE_NFC_PRESERVED '\n+\tmkdir c_${utf8_nfc} &&\n+\tls | test-tool hexdump | grep \"63 5f c3 a9\"\n+'\n+\n+# Is the spelling of an NFD pathname preserved on disk?\n+#\n+test_lazy_prereq UNICODE_NFD_PRESERVED '\n+\tmkdir d_${utf8_nfd} &&\n+\tls | test-tool hexdump | grep \"64 5f 65 cc 81\"\n+'\n+\n+# The following _DOUBLE_ forms are more for my curiosity,\n+# but there may be quirks lurking when there are multiple\n+# combining characters in non-canonical order.\n+\n+# Unicode also allows multiple combining characters\n+# that can be decomposed in pieces.\n+#\n+# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n+# UTF8(NFC): \\xe1 \\xbd \\xa7\n+#\n+# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n+# U+0342 COMBINING GREEK PERISPOMENI\n+# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n+#\n+# But U+1f61 decomposes into\n+# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n+# U+0314 COMBINING REVERSED COMMA ABOVE\n+# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n+#\n+# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n+#\n+# Note that I've used the canonical ordering of the\n+# combinining characters. It is also possible to\n+# swap them. My testing shows that that non-standard\n+# ordering also causes a collision in mkdir. However,\n+# the resulting names don't draw correctly on the\n+# terminal (implying that the on-disk format also has\n+# them out of order).\n+#\n+greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n+greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n+greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n+\n+# See if a double decomposition also collides.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${greek_nfc} &&\n+\tmkdir trial_${greek_nfd2}\n+'\n+\n+# See if the NFC spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n+\tmkdir c_${greek_nfc} &&\n+\tls | test-tool hexdump | grep \"63 5f e1 bd a7\"\n+'\n+\n+# See if the NFD spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n+\tmkdir d_${greek_nfd2} &&\n+\tls | test-tool hexdump | grep \"64 5f cf 89 cc 94 cd 82\"\n+'\n+\n+# The following is for debugging. I found it useful when\n+# trying to understand the various (OS, FS) quirks WRT\n+# Unicode and how composition/decomposition is handled.\n+# For example, when trying to understand how (macOS, APFS)\n+# and (macOS, HFS) and (macOS, FAT32) compare.\n+#\n+# It is rather noisy, so it is disabled by default.\n+#\n+if test \"$unicode_debug\" = \"true\"\n+then\n+\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\techo NFC maintains original spelling.\n+\telse\n+\t\techo NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFD_PRESERVED\n+\tthen\n+\t\techo NFD maintains original spelling.\n+\telse\n+\t\techo NFD is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n+\tthen\n+\t\techo Double NFC maintains original spelling.\n+\telse\n+\t\techo Double NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n+\tthen\n+\t\techo Double NFD maintains original spelling.\n+\telse\n+\t\techo Double NFD is modified.\n+\tfi\n+fi\n-- \ngitgitgadget\n\n"},{"id":"455847","messageId":"ca833ecc7a17925d3568572f183ba4fc0833a54b.1653336765.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v7 30/30] t7527: improve implicit shutdown testing in fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-23T20:12:45Z","receivedAt":"2022-05-23T20:15:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor the tests that exercise implicit shutdown cases\nto make them more robust and less racy.\n\nThe fsmonitor--daemon will implicitly shutdown in a variety\nof situations, such as when the \".git\" directory is deleted\nor renamed.\n\nThe existing tests would delete or rename the directory, sleep\nfor one second, and then check the status of the daemon. This\nis racy, since the client/status command has no way to sync\nwith the daemon. This was noticed occasionally on very slow\nCI build machines where it would cause a random test to fail.\n\nReplace the simple sleep with a sleep-and-retry loop.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 54 ++++++++++++++++++++++++++----------\n 1 file changed, 40 insertions(+), 14 deletions(-)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 19edc96fd4d..56c0dfffea6 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -124,6 +124,36 @@ test_expect_success 'implicit daemon start' '\n \ttest_must_fail git -C test_implicit fsmonitor--daemon status\n '\n \n+# Verify that the daemon has shutdown. Spin a few seconds to\n+# make the test a little more robust during CI testing.\n+#\n+# We're looking for an implicit shutdown, such as when we delete or\n+# rename the \".git\" directory. Our delete/rename will cause a file\n+# system event that the daemon will see and the daemon will\n+# auto-shutdown as soon as it sees it. But this is racy with our `git\n+# fsmonitor--daemon status` commands (and we cannot use a cookie file\n+# here to help us). So spin a little and give the daemon a chance to\n+# see the event. (This is primarily for underpowered CI build/test\n+# machines (where it might take a moment to wake and reschedule the\n+# daemon process) to avoid false alarms during test runs.)\n+#\n+IMPLICIT_TIMEOUT=5\n+\n+verify_implicit_shutdown () {\n+\tr=$1 &&\n+\n+\tk=0 &&\n+\twhile test \"$k\" -lt $IMPLICIT_TIMEOUT\n+\tdo\n+\t\tgit -C $r fsmonitor--daemon status || return 0\n+\n+\t\tsleep 1\n+\t\tk=$(( $k + 1 ))\n+\tdone &&\n+\n+\treturn 1\n+}\n+\n test_expect_success 'implicit daemon stop (delete .git)' '\n \ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n \n@@ -142,10 +172,9 @@ test_expect_success 'implicit daemon stop (delete .git)' '\n \t# This would make the test result dependent upon whether we\n \t# were using fsmonitor on our development worktree.\n \t#\n-\tsleep 1 &&\n \tmkdir test_implicit_1/.git &&\n \n-\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_1\n '\n \n test_expect_success 'implicit daemon stop (rename .git)' '\n@@ -160,10 +189,9 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \n \t# See [1] above.\n \t#\n-\tsleep 1 &&\n \tmkdir test_implicit_2/.git &&\n \n-\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_2\n '\n \n # File systems on Windows may or may not have shortnames.\n@@ -194,13 +222,11 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n \t#\n \tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n \n-\tsleep 1 &&\n-\t# put it back so that our status will not crawl out to our\n-\t# parent directory.\n+\t# See [1] above.\n \t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n \tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n \n-\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_1s\n '\n \n # Here we first create a file with LONGNAME of \"GIT~1\" before\n@@ -223,12 +249,10 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n \t#\n \tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n \n-\tsleep 1 &&\n-\t# put it back so that our status will not crawl out to our\n-\t# parent directory.\n+\t# See [1] above.\n \tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n \n-\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_1s2\n '\n \n test_expect_success 'cannot start multiple daemons' '\n@@ -905,9 +929,11 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \t# Rename .git using an alternate spelling to verify that that\n \t# daemon detects it and automatically shuts down.\n \tmv test_insensitive/.GIT test_insensitive/.FOO &&\n-\tsleep 1 &&\n+\n+\t# See [1] above.\n \tmv test_insensitive/.FOO test_insensitive/.git &&\n-\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\tverify_implicit_shutdown test_insensitive &&\n \n \t# Verify that events were reported using on-disk spellings of the\n \t# directories and files that we touched. We may or may not get a\n-- \ngitgitgadget\n"},{"id":"455853","messageId":"xmqqk0acosks.fsf@gitster.g","threadId":"57414","inReplyTo":"6f2e935f148e826609153378751c04807858e76c.1653336765.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-23T21:19:15Z","receivedAt":"2022-05-23T21:19:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> +int cmd__hexdump(int argc, const char **argv)\n> +{\n> +\tchar buf[1024];\n> +\tssize_t i, len;\n> +\n> +\tfor (;;) {\n> +\t\tlen = xread(0, buf, sizeof(buf));\n> +\t\tif (len < 0)\n> +\t\t\tdie_errno(\"failure reading stdin\");\n> +\t\tif (!len)\n> +\t\t\tbreak;\n> +\n> +\t\tfor (i = 0; i < len; i++)\n> +\t\t\tprintf(\"%02x \", (unsigned char)buf[i]);\n> +\t}\n> +\n> +\treturn 0;\n> +}\n\nIt is meant to be consumed by machine, so I do not think we would\nmind too much about a single long line, but given that consumers\ninclude \"grep\", it would probably be better to avoid emitting an\nincomplete line, especially since addition of this tool is all about\nportability across platforms.\n\nAn extra putchar('\\n'); after the loop would fix it easily.\n\nThanks.\n\n> diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n> index 0424f7adf5d..88c4b28cdfa 100644\n> --- a/t/helper/test-tool.c\n> +++ b/t/helper/test-tool.c\n> @@ -38,6 +38,7 @@ static struct test_cmd cmds[] = {\n> \t{ \"getcwd\", cmd__getcwd },\n> \t{ \"hashmap\", cmd__hashmap },\n> \t{ \"hash-speed\", cmd__hash_speed },\n> +\t{ \"hexdump\", cmd__hexdump },\n> \t{ \"index-version\", cmd__index_version },\n> \t{ \"json-writer\", cmd__json_writer },\n> \t{ \"lazy-init-name-hash\", cmd__lazy_init_name_hash },\n> diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n> index c876e8246fb..511f6251bf5 100644\n> --- a/t/helper/test-tool.h\n> +++ b/t/helper/test-tool.h\n> @@ -29,6 +29,7 @@ int cmd__genzeros(int argc, const char **argv);\n> int cmd__getcwd(int argc, const char **argv);\n> int cmd__hashmap(int argc, const char **argv);\n> int cmd__hash_speed(int argc, const char **argv);\n> +int cmd__hexdump(int argc, const char **argv);\n> int cmd__index_version(int argc, const char **argv);\n> int cmd__json_writer(int argc, const char **argv);\n> int cmd__lazy_init_name_hash(int argc, const char **argv);\n"},{"id":"455854","messageId":"xmqq5ylworwy.fsf@gitster.g","threadId":"57414","inReplyTo":"6a8308699543faaea760d4605babe50a0e478f41.1653336765.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 27/30] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-23T21:33:33Z","receivedAt":"2022-05-23T21:33:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> +\tls | test-tool hexdump | grep \"63 5f c3 a9\"\n\nA few comments:\n\n * Not folding output lines at arbitrary place like \"od\", \"hd\",\n etc. does, is a good design decision made by \"hexdump\" here.\n Depending on where in the pathname the 4-byte sequence appears,\n tools from other people may split the sequence across output\n lines, making grep ineffective. But our hexdump would work fine\n here.\n\n * For the narrow purpose of the tests in this script, output that\n is a single long line produced by hexdump might be sufficient,\n but I wonder if it makes the tool more useful if we at least\n placed the hexified output for each line on separate output\n lines.\n\n * Purist in us may find it a bit disturbing that exit status from\n test-tool is hidden by the pipe. I do not care too deeply about\n it, as it is very unlikely that we care about segfault after\n hexdump successfully shows the substring the downstream grep is\n looking for, but it does make us feel dirty.\n\nA devil's advocate suggestion is to go in the completely opposite\nside of the spectrum. Perhaps if we are willing to limit the tool's\nutility to the tests done in this script file, it might be a good\nidea to combine the latter two elements in the pipeline, i.e.\n\n\tls | test-tool hexgrep 63 5f c3 a9\n\nthat exits with 0 when the output from \"ls\" has the 4-byte sequence,\nexits with 1 when it does not, and exits with 139 when it segfauls ;-)\n"},{"id":"455888","messageId":"nycvar.QRO.7.76.6.2205241352130.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"d2c909df-44f6-bc2c-ebbc-13ea51cf62fe@jeffhostetler.com","subject":"Re: [PATCH v6 16/28] fsmonitor--daemon: stub in health thread","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-24T11:52:51Z","receivedAt":"2022-05-24T11:53:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Tue, 17 May 2022, Jeff Hostetler wrote:\n\n> On 5/12/22 11:05 AM, Johannes Schindelin wrote:\n> >\n> > On Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n> >\n> > > From: Jeff Hostetler \n> > >\n> > > Create another thread to watch over the daemon process and\n> > > automatically shut it down if necessary.\n> > >\n> [...]\n>\n> > > The platform-specific code for Windows sets up enough of the\n> > > WaitForMultipleObjects() machinery to watch for system and/or custom\n> > > events. Currently, the set of wait handles only includes our custom\n> > > shutdown event (sent from our other theads). Later commits in this\n> > > series will extend the set of wait handles to monitor other\n> > > conditions.\n> [...]\n>\n> > > diff --git a/compat/fsmonitor/fsm-health-win32.c\n> > > b/compat/fsmonitor/fsm-health-win32.c\n> > > new file mode 100644\n> > > index 00000000000..94b1d020f25\n> > > --- /dev/null\n> > > +++ b/compat/fsmonitor/fsm-health-win32.c\n> > > @@ -0,0 +1,72 @@\n> > > +#include \"cache.h\"\n> > > +#include \"config.h\"\n> > > +#include \"fsmonitor.h\"\n> > > +#include \"fsm-health.h\"\n> > > +#include \"fsmonitor--daemon.h\"\n> > > +\n> > > +struct fsm_health_data\n> > > +{\n> > > +\tHANDLE hEventShutdown;\n> > > +\n> > > +\tHANDLE hHandles[1]; /* the array does not own these handles */\n> > > +#define HEALTH_SHUTDOWN 0\n> >\n> > How about defining `HANDLE hHandles[HEALTH_SHUTDOWN + 1]` to indicate that\n> > the constant is used as an offset into `hHandles`?\n> >\n> > > +\tint nr_handles; /* number of active event handles */\n>\n> I think I'd like to keep this one as is. It matches the style that\n> I used in `fsm-listen-win32.c` where I have 3 listener handles.\n>\n> Granted, it does look a little odd when there is only 1 handle in the\n> array. But the idea was to allow new handles to be added as we want\n> to watch more things.\n>\n> It might be clearer (in both of them) to define the array in\n> terms of an enum rather than a local list of #define's.\n> But I'm not sure it matters.\n\nYour explanation makes sense to me, I have no objections against keeping\nthe code as-is.\n\nThanks,\nDscho\n"},{"id":"455889","messageId":"nycvar.QRO.7.76.6.2205241353030.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"c716b1d6-a0a8-6d03-dfe5-70613e4ed942@jeffhostetler.com","subject":"Re: [PATCH v6 20/28] fsmonitor: optimize processing of directory events","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-24T11:53:38Z","receivedAt":"2022-05-24T11:54:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Tue, 17 May 2022, Jeff Hostetler wrote:\n\n> On 5/12/22 11:08 AM, Johannes Schindelin wrote:\n> >\n> > On Fri, 22 Apr 2022, Jeff Hostetler via GitGitGadget wrote:\n> >\n> > > From: Jeff Hostetler \n> > >\n> > > Teach Git to perform binary search over the cache-entries for a directory\n> > > notification and then linearly scan forward to find the immediate\n> > > children.\n> > >\n> [...]\n>\n> > > static void fsmonitor_refresh_callback(struct index_state *istate, char\n> > > *name)\n> > > {\n> > > \tint i, len = strlen(name);\n> > > -\tif (name[len - 1] == '/') {\n> > > +\tint pos = index_name_pos(istate, name, len);\n> > > +\n> > > +\ttrace_printf_key(&trace_fsmonitor,\n> > > +\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n> > > +\t\t\t name, pos);\n> > >\n> [...]\n>\n> > > +\tif (name[len - 1] == '/') {\n> [...]\n> > > }\n>\n> > > @@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct\n> > > index_state *istate, char *name)\n> > > * Mark the untracked cache dirty even if it wasn't found in the index\n> > > * as it could be a new untracked file.\n> > > */\n> > > -\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\",\n> > > name);\n> >\n> > Did you mean to remove this statement in this patch? Not a big issue, but\n> > I wonder what the rationale for it is, and since I have an inquisitive\n> > mind, I figured I'd just ask.\n>\n> I just moved it to the top of the function. That lets me see `name`\n> before it is modified in one of the else arms (it was helpful to see\n> whether the daemon sent a trailing slash or not). And I also wanted\n> to see the computed value of `pos` (before the \"-pos - 1\" tricks).\n\nOh, I missed that it was moved rather than removed. Sorry for that!\n\nThank you,\nDscho\n"},{"id":"455890","messageId":"nycvar.QRO.7.76.6.2205241357410.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 00/30] Builtin FSMonitor Part 3","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-24T12:00:00Z","receivedAt":"2022-05-24T12:00:21Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Mon, 23 May 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> 1: 8b7c5f4e23 ! 1: 26144c5865 fsm-listen-win32: handle shortnames\n> [...]\n> @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI\n> +{\n> + wchar_t buf_in[MAX_PATH + 1];\n> + wchar_t buf_out[MAX_PATH + 1];\n> -+ wchar_t *last_slash = NULL;\n> -+ wchar_t *last_bslash = NULL;\n> + wchar_t *last;\n> ++ wchar_t *p;\n> +\n> + /* build L\"/.git\" */\n> -+ wcscpy(buf_in, watch->wpath_longname);\n> -+ wcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n> ++ swprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%s.git\",\n> ++ watch->wpath_longname);\n> +\n> -+ if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n> ++ if (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)))\n\nNice touch using `ARRAY_SIZE()` here!\n\nThe changes look good to me, from my side this is good to go.\n\nThank you so much!\nDscho\n"},{"id":"455891","messageId":"nycvar.QRO.7.76.6.2205241400200.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"xmqq5ylworwy.fsf@gitster.g","subject":"Re: [PATCH v7 27/30] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-24T12:14:12Z","receivedAt":"2022-05-24T12:14:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 23 May 2022, Junio C Hamano wrote:\n\n> \"Jeff Hostetler via GitGitGadget\" writes:\n>\n> A devil's advocate suggestion is to go in the completely opposite\n> side of the spectrum. Perhaps if we are willing to limit the tool's\n> utility to the tests done in this script file, it might be a good\n> idea to combine the latter two elements in the pipeline, i.e.\n>\n> \tls | test-tool hexgrep 63 5f c3 a9\n>\n> that exits with 0 when the output from \"ls\" has the 4-byte sequence,\n> exits with 1 when it does not, and exits with 139 when it segfauls ;-)\n\nI like the idea, but from what I recall of the Knuth-Pratt algorithm\n[*1*], the implementation might get a bit more involved than the current\n`test-hexdump.c`. With non-repetitive patterns like you wrote above, you\ncan simply re-set the needle's offset to 0 if a mismatch was seen. It's\npartially-repetitive patterns such as `01 02 01 02 01 ff` that make things\ntricker: After encountering a `01 02 01 02 01`, if the next character is a\n`02`, we must not reset the needle's offset completely, as the next two\ncharacters might be `01 ff`, i.e. a match.\n\nSince the purpose of this already-long, already well-iterated patch series\nis not necessarily to improve the test suite in such an involved manner,\nit should be left as an excercise for another patch series whose purpose\n_is_ to improve Git's test framework.\n\nDon't get me wrong, I am very much in favor of that `hexgrep` idea. Just\nin its own, dedicated patch series.\n\nCiao,\nDscho\n\nFootnote *1*: I actually had to look at Wikipedia page at\nhttps://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm\nto realize that I did unjustice to James Morris by forgetting that they\nhad discovered the algorithm independently.\n"},{"id":"455892","messageId":"nycvar.QRO.7.76.6.2205241415040.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"xmqqk0acosks.fsf@gitster.g","subject":"Re: [PATCH v7 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-24T12:16:07Z","receivedAt":"2022-05-24T12:16:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 23 May 2022, Junio C Hamano wrote:\n\n> \"Jeff Hostetler via GitGitGadget\" writes:\n>\n> > +int cmd__hexdump(int argc, const char **argv)\n> > +{\n> > +\tchar buf[1024];\n> > +\tssize_t i, len;\n> > +\n> > +\tfor (;;) {\n> > +\t\tlen = xread(0, buf, sizeof(buf));\n> > +\t\tif (len < 0)\n> > +\t\t\tdie_errno(\"failure reading stdin\");\n> > +\t\tif (!len)\n> > +\t\t\tbreak;\n> > +\n> > +\t\tfor (i = 0; i < len; i++)\n> > +\t\t\tprintf(\"%02x \", (unsigned char)buf[i]);\n> > +\t}\n> > +\n> > +\treturn 0;\n> > +}\n>\n> It is meant to be consumed by machine, so I do not think we would\n> mind too much about a single long line, but given that consumers\n> include \"grep\", it would probably be better to avoid emitting an\n> incomplete line, especially since addition of this tool is all about\n> portability across platforms.\n\nDo you know of any `grep` implementation that has problems with text\nmissing the usual trailing newlines?\n\nCiao,\nDscho\n"},{"id":"455906","messageId":"1fe2e843-20d1-4ea9-7ea2-c05330da17c2@jeffhostetler.com","threadId":"57414","inReplyTo":"xmqqk0acosks.fsf@gitster.g","subject":"Re: [PATCH v7 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-05-24T14:44:07Z","receivedAt":"2022-05-24T14:44:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/23/22 5:19 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> +int cmd__hexdump(int argc, const char **argv)\n>> +{\n>> +\tchar buf[1024];\n>> +\tssize_t i, len;\n>> +\n>> +\tfor (;;) {\n>> +\t\tlen = xread(0, buf, sizeof(buf));\n>> +\t\tif (len < 0)\n>> +\t\t\tdie_errno(\"failure reading stdin\");\n>> +\t\tif (!len)\n>> +\t\t\tbreak;\n>> +\n>> +\t\tfor (i = 0; i < len; i++)\n>> +\t\t\tprintf(\"%02x \", (unsigned char)buf[i]);\n>> +\t}\n>> +\n>> +\treturn 0;\n>> +}\n> \n> It is meant to be consumed by machine, so I do not think we would\n> mind too much about a single long line, but given that consumers\n> include \"grep\", it would probably be better to avoid emitting an\n> incomplete line, especially since addition of this tool is all about\n> portability across platforms.\n> \n> An extra putchar('\\n'); after the loop would fix it easily.\n\nYes, I should have added a final LF. I was more focused\non cleaning up the test cases.\n\nWould you prefer a send a V8 or would you be willing\nto push a fixup commit on top?\n\nJeff\n"},{"id":"455908","messageId":"cef019a1-47c1-8f45-5018-bf0c2b4bfad4@jeffhostetler.com","threadId":"57414","inReplyTo":"xmqq5ylworwy.fsf@gitster.g","subject":"Re: [PATCH v7 27/30] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-05-24T15:06:03Z","receivedAt":"2022-05-24T15:06:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/23/22 5:33 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> +\tls | test-tool hexdump | grep \"63 5f c3 a9\"\n> \n> A few comments:\n> \n> * Not folding output lines at arbitrary place like \"od\", \"hd\",\n> etc. does, is a good design decision made by \"hexdump\" here.\n> Depending on where in the pathname the 4-byte sequence appears,\n> tools from other people may split the sequence across output\n> lines, making grep ineffective. But our hexdump would work fine\n> here.\n> \n> * For the narrow purpose of the tests in this script, output that\n> is a single long line produced by hexdump might be sufficient,\n> but I wonder if it makes the tool more useful if we at least\n> placed the hexified output for each line on separate output\n> lines.\n\n\nYeah, having tools arbitrarily wrap every 16 or whatever bytes\n(and including offset line prefixes) makes it difficult to use\nwhen looking for specific patterns that might span a boundary.\n\nI could see having a command line option to emit a '\\n' (in addition\nto or in place of) each LF in the input. I suppose it depends on the\ntype of data we are dumping. (That also gets into issues about CRLFs,\nhowever.)\n\nI'm using hexdump for unicode text here, soit could make sense. But\nif I were using it to dump .git/index it wouldn't.\n\nSo having the default be one very long line is a good start.\nWe can teach it more later.\n\n> \n> * Purist in us may find it a bit disturbing that exit status from\n> test-tool is hidden by the pipe. I do not care too deeply about\n> it, as it is very unlikely that we care about segfault after\n> hexdump successfully shows the substring the downstream grep is\n> looking for, but it does make us feel dirty.\n\nGiven the simplicity of the current version of the helper, I'm not\nreally worried about such problems. I suppose that we could do the\nusual trick of writing the hex dump to a file and grepping it, but\nI'm not sure it's worth the bother right now.\n\n> \n> A devil's advocate suggestion is to go in the completely opposite\n> side of the spectrum. Perhaps if we are willing to limit the tool's\n> utility to the tests done in this script file, it might be a good\n> idea to combine the latter two elements in the pipeline, i.e.\n> \n> \tls | test-tool hexgrep 63 5f c3 a9\n> \n> that exits with 0 when the output from \"ls\" has the 4-byte sequence,\n> exits with 1 when it does not, and exits with 139 when it segfauls ;-)\n> \n\nI was a little afraid to suggest a hex version of grep. That would\nbe interesting project to work on, but has lots of hard problems in\nit and is too much to tack on to this series. Johannes raises some\ninteresting questions in a later response in this thread that suggest\nthat this could be a seriously non-trivial task. So again, I'd like\nto not attempt this.\n\nThanks\nJeff\n"},{"id":"455909","messageId":"dc1bd9f6-f83f-5cb1-9b0c-68d28a4cc216@jeffhostetler.com","threadId":"57414","inReplyTo":"nycvar.QRO.7.76.6.2205241357410.352@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v7 00/30] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-05-24T15:07:47Z","receivedAt":"2022-05-24T15:07:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/24/22 8:00 AM, Johannes Schindelin wrote:\n> Hi Jeff,\n> \n> On Mon, 23 May 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> 1: 8b7c5f4e23 ! 1: 26144c5865 fsm-listen-win32: handle shortnames\n>> [...]\n>> @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI\n>> +{\n>> + wchar_t buf_in[MAX_PATH + 1];\n>> + wchar_t buf_out[MAX_PATH + 1];\n>> -+ wchar_t *last_slash = NULL;\n>> -+ wchar_t *last_bslash = NULL;\n>> + wchar_t *last;\n>> ++ wchar_t *p;\n>> +\n>> + /* build L\"/.git\" */\n>> -+ wcscpy(buf_in, watch->wpath_longname);\n>> -+ wcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n>> ++ swprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%s.git\",\n>> ++ watch->wpath_longname);\n>> +\n>> -+ if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n>> ++ if (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)))\n> \n> Nice touch using `ARRAY_SIZE()` here!\n\nThanks. And hopefully it will make the GFW downstream MAX_LONG_PATH\nfixups easier too.\n\n> \n> The changes look good to me, from my side this is good to go.\n> \n> Thank you so much!\n> Dscho\n> \n\nThanks for all your reviews.\nJeff\n"},{"id":"455944","messageId":"xmqqbkvmhfoc.fsf@gitster.g","threadId":"57414","inReplyTo":"nycvar.QRO.7.76.6.2205241415040.352@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v7 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-24T19:52:03Z","receivedAt":"2022-05-24T19:52:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Do you know of any `grep` implementation that has problems with text\n> missing the usual trailing newlines?\n\nI recall that we had reports on BSD variants, but please do not\nquote me on that. Perhaps all the BSD variants are good now, or\nperhaps some aren't.\n\nIn any case, I do not take the \"if it works on Windows and Linux, we\ndo not care about the rest of the world\" world view, so finding the\nanswer to that question unfortunately does not give much input to\nthe issue in either way.\n\nAnd in this particular case, it is much simpler to mak sure that the\nfile does not end in an incomplete line than us exchanging e-mails\nback and forth, so that would be the most economical solution I\nwould prefer.\n\nThanks.\n\n"},{"id":"455945","messageId":"xmqq7d6ahfl3.fsf@gitster.g","threadId":"57414","inReplyTo":"1fe2e843-20d1-4ea9-7ea2-c05330da17c2@jeffhostetler.com","subject":"Re: [PATCH v7 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-24T19:54:00Z","receivedAt":"2022-05-24T19:54:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> Would you prefer a send a V8 or would you be willing\n> to push a fixup commit on top?\n\nIf \"test-tool hexgrep\" does not turn out to be a better solution,\nand if there is no other changes needed, then I do not mind locally\namending, but I'd rather avoid touching a middle step in multi dozen\npatch series myself if I can.\n\nThanks.\n"},{"id":"456040","messageId":"nycvar.QRO.7.76.6.2205251220260.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"xmqqbkvmhfoc.fsf@gitster.g","subject":"Re: [PATCH v7 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-25T10:21:45Z","receivedAt":"2022-05-25T10:22:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 24 May 2022, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > Do you know of any `grep` implementation that has problems with text\n> > missing the usual trailing newlines?\n>\n> I recall that we had reports on BSD variants, but please do not\n> quote me on that. Perhaps all the BSD variants are good now, or\n> perhaps some aren't.\n\nI did not recall any such reports, but take your word for it.\n\n> In any case, I do not take the \"if it works on Windows and Linux, we\n> do not care about the rest of the world\" world view,\n\nJust in case it was unclear to you: we're on the same page here.\n\n> so finding the answer to that question unfortunately does not give much\n> input to the issue in either way.\n>\n> And in this particular case, it is much simpler to mak sure that the\n> file does not end in an incomplete line than us exchanging e-mails\n> back and forth, so that would be the most economical solution I\n> would prefer.\n\nIndeed, the trailing newline is easily added and fixes a known issue, so\nI'm all for it.\n\nCiao,\nDscho\n"},{"id":"456041","messageId":"nycvar.QRO.7.76.6.2205251223180.352@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"dc1bd9f6-f83f-5cb1-9b0c-68d28a4cc216@jeffhostetler.com","subject":"Re: [PATCH v7 00/30] Builtin FSMonitor Part 3","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-05-25T10:23:48Z","receivedAt":"2022-05-25T10:24:09Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Tue, 24 May 2022, Jeff Hostetler wrote:\n\n> On 5/24/22 8:00 AM, Johannes Schindelin wrote:\n> >\n> > On Mon, 23 May 2022, Jeff Hostetler via GitGitGadget wrote:\n> >\n> > > 1: 8b7c5f4e23 ! 1: 26144c5865 fsm-listen-win32: handle shortnames\n> > > [...]\n> > > @@ compat/fsmonitor/fsm-listen-win32.c: static int\n> > > normalize_path_in_utf8(FILE_NOTI\n> > > +{\n> > > + wchar_t buf_in[MAX_PATH + 1];\n> > > + wchar_t buf_out[MAX_PATH + 1];\n> > > -+ wchar_t *last_slash = NULL;\n> > > -+ wchar_t *last_bslash = NULL;\n> > > + wchar_t *last;\n> > > ++ wchar_t *p;\n> > > +\n> > > + /* build L\"/.git\" */\n> > > -+ wcscpy(buf_in, watch->wpath_longname);\n> > > -+ wcscpy(buf_in + watch->wpath_longname_len, L\".git\");\n> > > ++ swprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%s.git\",\n> > > ++ watch->wpath_longname);\n> > > +\n> > > -+ if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))\n> > > ++ if (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)))\n> >\n> > Nice touch using `ARRAY_SIZE()` here!\n>\n> Thanks. And hopefully it will make the GFW downstream MAX_LONG_PATH\n> fixups easier too.\n\nAs a matter of fact, it already did!\n\nThank you,\nDscho\n"},{"id":"456067","messageId":"bb500f0f-7608-e965-1616-6d23d81ab7c9@jeffhostetler.com","threadId":"57414","inReplyTo":"xmqq7d6ahfl3.fsf@gitster.g","subject":"Re: [PATCH v7 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-05-25T13:45:46Z","receivedAt":"2022-05-25T13:45:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/24/22 3:54 PM, Junio C Hamano wrote:\n> Jeff Hostetler writes:\n> \n>> Would you prefer a send a V8 or would you be willing\n>> to push a fixup commit on top?\n> \n> If \"test-tool hexgrep\" does not turn out to be a better solution,\n> and if there is no other changes needed, then I do not mind locally\n> amending, but I'd rather avoid touching a middle step in multi dozen\n> patch series myself if I can.\n> \n> Thanks.\n> \n\nThat's fine. I'll fix it and send V8.\nThanks\nJeff\n"},{"id":"456070","messageId":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v7.git.1653336765.gitgitgadget@gmail.com","subject":"[PATCH v8 00/30] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:22Z","receivedAt":"2022-05-25T15:01:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 8 of part 3 of FSMonitor.\n\nThis version addresses the new t/helper/test-hexdump utility to emit an LF\nat the end. I also updated the test scripts to write hexdump output to a\nfile and then grep that file to avoid having the tool being in the middle of\na pipeline.\n\nJeff Hostetler (30):\n fsm-listen-win32: handle shortnames\n t7527: test FSMonitor on repos with Unicode root paths\n t/helper/fsmonitor-client: create stress test\n fsmonitor-settings: bare repos are incompatible with FSMonitor\n fsmonitor-settings: stub in Win32-specific incompatibility checking\n fsmonitor-settings: VFS for Git virtual repos are incompatible\n fsmonitor-settings: stub in macOS-specific incompatibility checking\n fsmonitor-settings: remote repos on macOS are incompatible\n fsmonitor-settings: remote repos on Windows are incompatible\n fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n unpack-trees: initialize fsmonitor_has_run_once in o->result\n fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n fsmonitor--daemon: cd out of worktree root\n fsmonitor--daemon: prepare for adding health thread\n fsmonitor--daemon: rename listener thread related variables\n fsmonitor--daemon: stub in health thread\n fsm-health-win32: add polling framework to monitor daemon health\n fsm-health-win32: force shutdown daemon if worktree root moves\n fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n fsmonitor: optimize processing of directory events\n t7527: FSMonitor tests for directory moves\n t/perf/p7527: add perf test for builtin FSMonitor\n fsmonitor: never set CE_FSMONITOR_VALID on submodules\n t7527: test FSMonitor on case insensitive+preserving file system\n fsmonitor: on macOS also emit NFC spelling for NFD pathname\n t/helper/hexdump: add helper to print hexdump of stdin\n t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n t7527: test Unicode NFC/NFD handling on MacOS\n fsmonitor--daemon: allow --super-prefix argument\n t7527: improve implicit shutdown testing in fsmonitor--daemon\n\n Makefile | 20 +-\n builtin/fsmonitor--daemon.c | 116 ++++++-\n builtin/update-index.c | 16 +\n compat/fsmonitor/fsm-health-darwin.c | 24 ++\n compat/fsmonitor/fsm-health-win32.c | 278 ++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 +++\n compat/fsmonitor/fsm-listen-darwin.c | 122 ++++++-\n compat/fsmonitor/fsm-listen-win32.c | 436 ++++++++++++++++++++-----\n compat/fsmonitor/fsm-listen.h | 2 +-\n compat/fsmonitor/fsm-settings-darwin.c | 89 +++++\n compat/fsmonitor/fsm-settings-win32.c | 137 ++++++++\n config.mak.uname | 5 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 11 +-\n fsmonitor-settings.c | 167 ++++++++--\n fsmonitor-settings.h | 33 ++\n fsmonitor.c | 73 ++++-\n fsmonitor.h | 11 +\n git.c | 2 +-\n t/helper/test-fsmonitor-client.c | 106 ++++++\n t/helper/test-hexdump.c | 30 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/lib-unicode-nfc-nfd.sh | 162 +++++++++\n t/perf/p7527-builtin-fsmonitor.sh | 257 +++++++++++++++\n t/t7519-status-fsmonitor.sh | 32 ++\n t/t7527-builtin-fsmonitor.sh | 401 ++++++++++++++++++++++-\n unpack-trees.c | 1 +\n 28 files changed, 2439 insertions(+), 149 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n create mode 100644 t/helper/test-hexdump.c\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\n\nbase-commit: 5eb696daba2fe108d4d9ba2ccf4b357447ef9946\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v8\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v8\nPull-Request: https://github.com/gitgitgadget/git/pull/1143\n\nRange-diff vs v7:\n\n 1: 26144c58659 = 1: 26144c58659 fsm-listen-win32: handle shortnames\n 2: 1bf2e36b6ad = 2: 1bf2e36b6ad t7527: test FSMonitor on repos with Unicode root paths\n 3: 4bca494bb22 = 3: 4bca494bb22 t/helper/fsmonitor-client: create stress test\n 4: 663deabc3f6 = 4: 663deabc3f6 fsmonitor-settings: bare repos are incompatible with FSMonitor\n 5: 7cb0180a1ed = 5: 7cb0180a1ed fsmonitor-settings: stub in Win32-specific incompatibility checking\n 6: 9774faddc45 = 6: 9774faddc45 fsmonitor-settings: VFS for Git virtual repos are incompatible\n 7: f7ef7dcffc8 = 7: f7ef7dcffc8 fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: dc2dfd67931 = 8: dc2dfd67931 fsmonitor-settings: remote repos on macOS are incompatible\n 9: 5627038aaa3 = 9: 5627038aaa3 fsmonitor-settings: remote repos on Windows are incompatible\n 10: 9a12cc78b5d = 10: 9a12cc78b5d fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n 11: aaff000cecb = 11: aaff000cecb unpack-trees: initialize fsmonitor_has_run_once in o->result\n 12: 4f2b15d3d1f = 12: 4f2b15d3d1f fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n 13: 427dec412a5 = 13: 427dec412a5 fsmonitor--daemon: cd out of worktree root\n 14: 51b266b06e1 = 14: 51b266b06e1 fsmonitor--daemon: prepare for adding health thread\n 15: 594e0ae243d = 15: 594e0ae243d fsmonitor--daemon: rename listener thread related variables\n 16: c2b5c02ed38 = 16: c2b5c02ed38 fsmonitor--daemon: stub in health thread\n 17: 46a5ae2a635 = 17: 46a5ae2a635 fsm-health-win32: add polling framework to monitor daemon health\n 18: 7cf1be5f8e2 = 18: 7cf1be5f8e2 fsm-health-win32: force shutdown daemon if worktree root moves\n 19: 95cf1299d44 = 19: 95cf1299d44 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n 20: b020bfb4568 = 20: b020bfb4568 fsmonitor: optimize processing of directory events\n 21: d058d7e0c08 = 21: d058d7e0c08 t7527: FSMonitor tests for directory moves\n 22: f5dac286812 = 22: f5dac286812 t/perf/p7527: add perf test for builtin FSMonitor\n 23: 92f5c0d2c8b = 23: 92f5c0d2c8b fsmonitor: never set CE_FSMONITOR_VALID on submodules\n 24: 40b80adbb31 = 24: 40b80adbb31 t7527: test FSMonitor on case insensitive+preserving file system\n 25: b93f0642699 = 25: ea19a06e8cb fsmonitor: on macOS also emit NFC spelling for NFD pathname\n 26: 6f2e935f148 ! 26: 66a01db4739 t/helper/hexdump: add helper to print hexdump of stdin\n @@\n ## Metadata ##\n -Author: Jeff Hostetler \n +Author: Jeff Hostetler \n \n ## Commit message ##\n t/helper/hexdump: add helper to print hexdump of stdin\n \n Co-authored-by: Johannes Schindelin \n Signed-off-by: Johannes Schindelin \n - Signed-off-by: Jeff Hostetler \n + Signed-off-by: Jeff Hostetler \n \n ## Makefile ##\n @@ Makefile: TEST_BUILTINS_OBJS += test-getcwd.o\n @@ t/helper/test-hexdump.c (new)\n +{\n +\tchar buf[1024];\n +\tssize_t i, len;\n ++\tint have_data = 0;\n +\n +\tfor (;;) {\n +\t\tlen = xread(0, buf, sizeof(buf));\n @@ t/helper/test-hexdump.c (new)\n +\t\tif (!len)\n +\t\t\tbreak;\n +\n ++\t\thave_data = 1;\n ++\n +\t\tfor (i = 0; i < len; i++)\n +\t\t\tprintf(\"%02x \", (unsigned char)buf[i]);\n +\t}\n +\n ++\tif (have_data)\n ++\t\tputchar('\\n');\n ++\n +\treturn 0;\n +}\n \n 27: 6a830869954 ! 27: 25c6066eddc t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n @@ t/lib-unicode-nfc-nfd.sh (new)\n +#\n +test_lazy_prereq UNICODE_NFC_PRESERVED '\n +\tmkdir c_${utf8_nfc} &&\n -+\tls | test-tool hexdump | grep \"63 5f c3 a9\"\n ++\tls | test-tool hexdump >dump &&\n ++\tgrep \"63 5f c3 a9\" dump\n +'\n +\n +# Is the spelling of an NFD pathname preserved on disk?\n +#\n +test_lazy_prereq UNICODE_NFD_PRESERVED '\n +\tmkdir d_${utf8_nfd} &&\n -+\tls | test-tool hexdump | grep \"64 5f 65 cc 81\"\n ++\tls | test-tool hexdump >dump &&\n ++\tgrep \"64 5f 65 cc 81\" dump\n +'\n +\n +# The following _DOUBLE_ forms are more for my curiosity,\n @@ t/lib-unicode-nfc-nfd.sh (new)\n +#\n +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n +\tmkdir c_${greek_nfc} &&\n -+\tls | test-tool hexdump | grep \"63 5f e1 bd a7\"\n ++\tls | test-tool hexdump >dump &&\n ++\tgrep \"63 5f e1 bd a7\" dump\n +'\n +\n +# See if the NFD spelling appears on the disk.\n +#\n +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n +\tmkdir d_${greek_nfd2} &&\n -+\tls | test-tool hexdump | grep \"64 5f cf 89 cc 94 cd 82\"\n ++\tls | test-tool hexdump >dump &&\n ++\tgrep \"64 5f cf 89 cc 94 cd 82\" dump\n +'\n +\n +# The following is for debugging. I found it useful when\n 28: f9a7869d202 = 28: fc3a0e7847f t7527: test Unicode NFC/NFD handling on MacOS\n 29: 9fc7c970929 = 29: 25676ca4ec2 fsmonitor--daemon: allow --super-prefix argument\n 30: ca833ecc7a1 ! 30: d70d2545a5a t7527: improve implicit shutdown testing in fsmonitor--daemon\n @@\n ## Metadata ##\n -Author: Jeff Hostetler \n +Author: Jeff Hostetler \n \n ## Commit message ##\n t7527: improve implicit shutdown testing in fsmonitor--daemon\n @@ Commit message\n \n Replace the simple sleep with a sleep-and-retry loop.\n \n - Signed-off-by: Jeff Hostetler \n + Signed-off-by: Jeff Hostetler \n \n ## t/t7527-builtin-fsmonitor.sh ##\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'implicit daemon start' '\n\n-- \ngitgitgadget\n"},{"id":"456071","messageId":"26144c586599a17c05f27192c3453e6f44c56d3a.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 01/30] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:23Z","receivedAt":"2022-05-25T15:01:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach FSMonitor daemon on Windows to recognize shortname paths as\naliases of normal longname paths. FSMonitor clients, such as `git\nstatus`, should receive the longname spelling of changed files (when\npossible).\n\nSometimes we receive FS events using the shortname, such as when a CMD\nshell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\narrives using whatever combination of long and shortnames were used by\nthe other process. (Shortnames do seem to be case normalized,\nhowever.)\n\nUse Windows GetLongPathNameW() to try to map the pathname spelling in\nthe notification event into the normalized longname spelling. (This\ncan fail if the file/directory is deleted, moved, or renamed, because\nwe are asking the FS for the mapping in response to the event and\nafter it has already happened, but we try.)\n\nSpecial case the shortname spelling of \".git\" to avoid under-reporting\nthese events.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 386 ++++++++++++++++++++++++----\n t/t7527-builtin-fsmonitor.sh | 65 +++++\n 2 files changed, 397 insertions(+), 54 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 5b928ab66e5..0a86aea3f7e 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -25,6 +25,9 @@ struct one_watch\n \tDWORD count;\n \n \tstruct strbuf path;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n+\tDWORD wpath_longname_len;\n+\n \tHANDLE hDir;\n \tHANDLE hEvent;\n \tOVERLAPPED overlapped;\n@@ -34,6 +37,21 @@ struct one_watch\n \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n \t */\n \tBOOL is_active;\n+\n+\t/*\n+\t * Are shortnames enabled on the containing drive? This is\n+\t * always true for \"C:/\" drives and usually never true for\n+\t * other drives.\n+\t *\n+\t * We only set this for the worktree because we only need to\n+\t * convert shortname paths to longname paths for items we send\n+\t * to clients. (We don't care about shortname expansion for\n+\t * paths inside a GITDIR because we never send them to\n+\t * clients.)\n+\t */\n+\tBOOL has_shortnames;\n+\tBOOL has_tilde;\n+\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n struct fsmonitor_daemon_backend_data\n@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n };\n \n /*\n- * Convert the WCHAR path from the notification into UTF8 and\n- * then normalize it.\n+ * Convert the WCHAR path from the event into UTF8 and normalize it.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n */\n-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n \t\t\t\t struct strbuf *normalized_path)\n {\n \tint reserve;\n \tint len = 0;\n \n \tstrbuf_reset(normalized_path);\n-\tif (!info->FileNameLength)\n+\tif (!wpath_len)\n \t\tgoto normalize;\n \n \t/*\n@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t * sequence of 2 UTF8 characters. That should let us\n \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n \t */\n-\treserve = info->FileNameLength + 1;\n+\treserve = 2 * wpath_len + 1;\n \tstrbuf_grow(normalized_path, reserve);\n \n \tfor (;;) {\n-\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n-\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n+\t\t\t\t\t wpath, wpath_len,\n \t\t\t\t\t normalized_path->buf,\n \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n \t\t\t\t\t NULL, NULL);\n@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n \t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n-\t\t\t GetLastError(),\n-\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n-\t\t\t info->FileName);\n+\t\t\t GetLastError(), (int)wpath_len, wpath);\n \t\t\treturn -1;\n \t\t}\n \n@@ -98,6 +115,173 @@ normalize:\n \treturn strbuf_normalize_path(normalized_path);\n }\n \n+/*\n+ * See if the worktree root directory has shortnames enabled.\n+ * This will help us decide if we need to do an expensive shortname\n+ * to longname conversion on every notification event.\n+ *\n+ * We do not want to create a file to test this, so we assume that the\n+ * root directory contains a \".git\" file or directory. (Our caller\n+ * only calls us for the worktree root, so this should be fine.)\n+ *\n+ * Remember the spelling of the shortname for \".git\" if it exists.\n+ */\n+static void check_for_shortnames(struct one_watch *watch)\n+{\n+\twchar_t buf_in[MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\twchar_t *last;\n+\twchar_t *p;\n+\n+\t/* build L\"/.git\" */\n+\tswprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%s.git\",\n+\t\t watch->wpath_longname);\n+\n+\tif (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)))\n+\t\treturn;\n+\n+\t/*\n+\t * Get the final filename component of the shortpath.\n+\t * We know that the path does not have a final slash.\n+\t */\n+\tfor (last = p = buf_out; *p; p++)\n+\t\tif (*p == L'/' || *p == '\\\\')\n+\t\t\tlast = p + 1;\n+\n+\tif (!wcscmp(last, L\".git\"))\n+\t\treturn;\n+\n+\twatch->has_shortnames = 1;\n+\twcsncpy(watch->dotgit_shortname, last,\n+\t\tARRAY_SIZE(watch->dotgit_shortname));\n+\n+\t/*\n+\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n+\t * we should be able to avoid shortname to longname mapping on\n+\t * every notification event if the source string does not\n+\t * contain a \"~\".\n+\t *\n+\t * However, the documentation for GetLongPathNameW() says\n+\t * that there are filesystems that don't follow that pattern\n+\t * and warns against this optimization.\n+\t *\n+\t * Lets test this.\n+\t */\n+\tif (wcschr(watch->dotgit_shortname, L'~'))\n+\t\twatch->has_tilde = 1;\n+}\n+\n+enum get_relative_result {\n+\tGRR_NO_CONVERSION_NEEDED,\n+\tGRR_HAVE_CONVERSION,\n+\tGRR_SHUTDOWN,\n+};\n+\n+/*\n+ * Info notification paths are relative to the root of the watch.\n+ * If our CWD is still at the root, then we can use relative paths\n+ * to convert from shortnames to longnames. If our process has a\n+ * different CWD, then we need to construct an absolute path, do\n+ * the conversion, and then return the root-relative portion.\n+ *\n+ * We use the longname form of the root as our basis and assume that\n+ * it already has a trailing slash.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n+ */\n+static enum get_relative_result get_relative_longname(\n+\tstruct one_watch *watch,\n+\tconst wchar_t *wpath, DWORD wpath_len,\n+\twchar_t *wpath_longname, size_t bufsize_wpath_longname)\n+{\n+\twchar_t buf_in[2 * MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\tDWORD root_len;\n+\tDWORD out_len;\n+\n+\t/*\n+\t * Build L\"/\"\n+\t * Note that the might not be null terminated\n+\t * so we avoid swprintf() constructions.\n+\t */\n+\troot_len = watch->wpath_longname_len;\n+\tif (root_len + wpath_len >= ARRAY_SIZE(buf_in)) {\n+\t\t/*\n+\t\t * This should not happen. We cannot append the observed\n+\t\t * relative path onto the end of the worktree root path\n+\t\t * without overflowing the buffer. Just give up.\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\twcsncpy(buf_in, watch->wpath_longname, root_len);\n+\twcsncpy(buf_in + root_len, wpath, wpath_len);\n+\tbuf_in[root_len + wpath_len] = 0;\n+\n+\t/*\n+\t * We don't actually know if the source pathname is a\n+\t * shortname or a longname. This Windows routine allows\n+\t * either to be given as input.\n+\t */\n+\tout_len = GetLongPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out));\n+\tif (!out_len) {\n+\t\t/*\n+\t\t * The shortname to longname conversion can fail for\n+\t\t * various reasons, for example if the file has been\n+\t\t * deleted. (That is, if we just received a\n+\t\t * delete-file notification event and the file is\n+\t\t * already gone, we can't ask the file system to\n+\t\t * lookup the longname for it. Likewise, for moves\n+\t\t * and renames where we are given the old name.)\n+\t\t *\n+\t\t * Since deleting or moving a file or directory by its\n+\t\t * shortname is rather obscure, I'm going ignore the\n+\t\t * failure and ask the caller to report the original\n+\t\t * relative path. This seems kinder than failing here\n+\t\t * and forcing a resync. Besides, forcing a resync on\n+\t\t * every file/directory delete would effectively\n+\t\t * cripple monitoring.\n+\t\t *\n+\t\t * We might revisit this in the future.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (!wcscmp(buf_in, buf_out)) {\n+\t\t/*\n+\t\t * The path does not have a shortname alias.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (wcsncmp(buf_in, buf_out, root_len)) {\n+\t\t/*\n+\t\t * The spelling of the root directory portion of the computed\n+\t\t * longname has changed. This should not happen. Basically,\n+\t\t * it means that we don't know where (without recomputing the\n+\t\t * longname of just the root directory) to split out the\n+\t\t * relative path. Since this should not happen, I'm just\n+\t\t * going to let this fail and force a shutdown (because all\n+\t\t * subsequent events are probably going to see the same\n+\t\t * mismatch).\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\tif (out_len - root_len >= bufsize_wpath_longname) {\n+\t\t/*\n+\t\t * This should not happen. We cannot copy the root-relative\n+\t\t * portion of the path into the provided buffer without an\n+\t\t * overrun. Just give up.\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\t/* Return the worktree root-relative portion of the longname. */\n+\n+\twcscpy(wpath_longname, buf_out + root_len);\n+\treturn GRR_HAVE_CONVERSION;\n+}\n+\n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n@@ -111,7 +295,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n-\twchar_t wpath[MAX_PATH];\n+\tDWORD len_longname;\n+\twchar_t wpath[MAX_PATH + 1];\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \tif (xutftowcs_path(wpath, path) < 0) {\n \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n@@ -128,6 +314,21 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \t\treturn NULL;\n \t}\n \n+\tlen_longname = GetLongPathNameW(wpath, wpath_longname,\n+\t\t\t\t\tARRAY_SIZE(wpath_longname));\n+\tif (!len_longname) {\n+\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n+\t\t GetLastError(), path);\n+\t\tCloseHandle(hDir);\n+\t\treturn NULL;\n+\t}\n+\n+\tif (wpath_longname[len_longname - 1] != L'/' &&\n+\t wpath_longname[len_longname - 1] != L'\\\\') {\n+\t\twpath_longname[len_longname++] = L'/';\n+\t\twpath_longname[len_longname] = 0;\n+\t}\n+\n \tCALLOC_ARRAY(watch, 1);\n \n \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n@@ -135,6 +336,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tstrbuf_init(&watch->path, 0);\n \tstrbuf_addstr(&watch->path, path);\n \n+\twcscpy(watch->wpath_longname, wpath_longname);\n+\twatch->wpath_longname_len = len_longname;\n+\n \twatch->hDir = hDir;\n \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n \n@@ -258,6 +462,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t,\n+\tDWORD info_action)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n+\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -274,6 +534,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n \tstruct fsmonitor_batch *batch = NULL;\n \tconst char *p = watch->buffer;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \t/*\n \t * If the kernel gets more events than will fit in the kernel\n@@ -306,54 +567,64 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n+\t\twchar_t *wpath = info->FileName;\n+\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n \t\tenum fsmonitor_path_type t;\n+\t\tenum get_relative_result grr;\n+\n+\t\tif (watch->has_shortnames) {\n+\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n+\t\t\t\t/*\n+\t\t\t\t * This event exactly matches the\n+\t\t\t\t * spelling of the shortname of\n+\t\t\t\t * \".git\", so we can skip some steps.\n+\t\t\t\t *\n+\t\t\t\t * (This case is odd because the user\n+\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n+\t\t\t\t * use the filesystem to map it back\n+\t\t\t\t * to \".git\".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_reset(&path);\n+\t\t\t\tstrbuf_addstr(&path, \".git\");\n+\t\t\t\tt = IS_DOT_GIT;\n+\t\t\t\tgoto process_it;\n+\t\t\t}\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n-\t\t\tgoto skip_this_path;\n-\n-\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n-\n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n+\t\t\tif (watch->has_tilde && !wcschr(wpath, L'~')) {\n+\t\t\t\t/*\n+\t\t\t\t * Shortnames on this filesystem have tildes\n+\t\t\t\t * and the notification path does not have\n+\t\t\t\t * one, so we assume that it is a longname.\n+\t\t\t\t */\n+\t\t\t\tgoto normalize_it;\n+\t\t\t}\n \n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n+\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n+\t\t\t\t\t\t wpath_longname,\n+\t\t\t\t\t\t ARRAY_SIZE(wpath_longname));\n+\t\t\tswitch (grr) {\n+\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n+\t\t\t\tbreak;\n+\t\t\tcase GRR_HAVE_CONVERSION:\n+\t\t\t\twpath = wpath_longname;\n+\t\t\t\twpath_len = wcslen(wpath);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\tcase GRR_SHUTDOWN:\n \t\t\t\tgoto force_shutdown;\n \t\t\t}\n-\t\t\tbreak;\n+\t\t}\n \n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n+normalize_it:\n+\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n+\t\t\tgoto skip_this_path;\n \n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+process_it:\n+\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n+\t\t\t\t\t info->Action))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -382,6 +653,9 @@ force_shutdown:\n * Note that we DO NOT get filesystem events on the external \n * itself (it is not inside something that we are watching). In particular,\n * we do not get an event if the external is deleted.\n+ *\n+ * Also, we do not care about shortnames within the external , since\n+ * we never send these paths to clients.\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n@@ -403,8 +677,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\tif (normalize_path_in_utf8(\n+\t\t\t info->FileName,\n+\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t &path) == -1)\n \t\t\tgoto skip_this_path;\n \n \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n@@ -538,6 +814,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \tif (!data->watch_worktree)\n \t\tgoto failed;\n \n+\tcheck_for_shortnames(data->watch_worktree);\n+\n \tif (state->nr_paths_watching > 1) {\n \t\tdata->watch_gitdir = create_watch(state,\n \t\t\t\t\t\t state->path_gitdir_watch.buf);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex bd0c952a116..1be21785162 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -166,6 +166,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon -C test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n+\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n+\n+\tstart_daemon -C test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"456072","messageId":"1bf2e36b6adb127dd1654bba0b6ba84f626535a0.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 02/30] t7527: test FSMonitor on repos with Unicode root paths","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:24Z","receivedAt":"2022-05-25T15:01:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate some test repos with UTF8 characters in the pathname of the\nroot directory and verify that the builtin FSMonitor can watch them.\n\nThis test is mainly for Windows where we need to avoid `*A()`\nroutines.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1be21785162..12655958e71 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -671,4 +671,27 @@ do\n \tdone\n done\n \n+# Test Unicode UTF-8 characters in the pathname of the working\n+# directory root. Use of \"*A()\" routines rather than \"*W()\" routines\n+# on Windows can sometimes lead to odd failures.\n+#\n+u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n+u2=$(printf \"u_e2_99_ab__\\xE2\\x99\\xAB\")\n+u_values=\"$u1 $u2\"\n+for u in $u_values\n+do\n+\ttest_expect_success \"unicode in repo root path: $u\" '\n+\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n+\n+\t\tgit init \"$u\" &&\n+\t\techo 1 >\"$u\"/file1 &&\n+\t\tgit -C \"$u\" add file1 &&\n+\t\tgit -C \"$u\" config core.fsmonitor true &&\n+\n+\t\tstart_daemon -C \"$u\" &&\n+\t\tgit -C \"$u\" status >actual &&\n+\t\tgrep \"new file: file1\" actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"456073","messageId":"4bca494bb22365fd9ccc59f6470c5124f74caf84.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 03/30] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:25Z","receivedAt":"2022-05-25T15:01:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a stress test to hammer on the fsmonitor daemon.\nCreate a client-side thread pool of n threads and have\neach of them make m requests as fast as they can.\n\nWe do not currently inspect the contents of the response.\nWe're only interested in placing a heavy request load on\nthe daemon.\n\nThis test is useful for interactive testing and various\nexperimentation. For example, to place additional load\non the daemon while another test is running. We currently\ndo not have a test script that actually uses this helper.\nWe might add such a test in the future.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++\n 1 file changed, 106 insertions(+)\n\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nindex 3062c8a3c2b..54a4856c48c 100644\n--- a/t/helper/test-fsmonitor-client.c\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -7,6 +7,8 @@\n #include \"cache.h\"\n #include \"parse-options.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2.h\"\n \n #ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n int cmd__fsmonitor_client(int argc, const char **argv)\n@@ -79,20 +81,121 @@ static int do_send_flush(void)\n \treturn 0;\n }\n \n+struct hammer_thread_data\n+{\n+\tpthread_t pthread_id;\n+\tint thread_nr;\n+\n+\tint nr_requests;\n+\tconst char *token;\n+\n+\tint sum_successful;\n+\tint sum_errors;\n+};\n+\n+static void *hammer_thread_proc(void *_hammer_thread_data)\n+{\n+\tstruct hammer_thread_data *data = _hammer_thread_data;\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint k;\n+\tint ret;\n+\n+\ttrace2_thread_start(\"hammer\");\n+\n+\tfor (k = 0; k < data->nr_requests; k++) {\n+\t\tstrbuf_reset(&answer);\n+\n+\t\tret = fsmonitor_ipc__send_query(data->token, &answer);\n+\t\tif (ret < 0)\n+\t\t\tdata->sum_errors++;\n+\t\telse\n+\t\t\tdata->sum_successful++;\n+\t}\n+\n+\tstrbuf_release(&answer);\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Start a pool of client threads that will each send a series of\n+ * commands to the daemon.\n+ *\n+ * The goal is to overload the daemon with a sustained series of\n+ * concurrent requests.\n+ */\n+static int do_hammer(const char *token, int nr_threads, int nr_requests)\n+{\n+\tstruct hammer_thread_data *data = NULL;\n+\tint k;\n+\tint sum_join_errors = 0;\n+\tint sum_commands = 0;\n+\tint sum_errors = 0;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\tif (nr_threads < 1)\n+\t\tnr_threads = 1;\n+\tif (nr_requests < 1)\n+\t\tnr_requests = 1;\n+\n+\tCALLOC_ARRAY(data, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\t\tp->thread_nr = k;\n+\t\tp->nr_requests = nr_requests;\n+\t\tp->token = token;\n+\n+\t\tif (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {\n+\t\t\twarning(\"failed to create thread[%d] skipping remainder\", k);\n+\t\t\tnr_threads = k;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\n+\t\tif (pthread_join(p->pthread_id, NULL))\n+\t\t\tsum_join_errors++;\n+\t\tsum_commands += p->sum_successful;\n+\t\tsum_errors += p->sum_errors;\n+\t}\n+\n+\tfprintf(stderr, \"HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\\n\",\n+\t\tnr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);\n+\n+\tfree(data);\n+\n+\t/*\n+\t * Return an error if any of the _send_query requests failed.\n+\t * We don't care about thread create/join errors.\n+\t */\n+\treturn sum_errors > 0;\n+}\n+\n int cmd__fsmonitor_client(int argc, const char **argv)\n {\n \tconst char *subcmd;\n \tconst char *token = NULL;\n+\tint nr_threads = 1;\n+\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n \t\t\"test-tool fsmonitor-client query []\",\n \t\t\"test-tool fsmonitor-client flush\",\n+\t\t\"test-tool fsmonitor-client hammer [] [] []\",\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n \t\tOPT_STRING(0, \"token\", &token, \"token\",\n \t\t\t \"command token to send to the server\"),\n+\n+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, \"number of client threads\"),\n+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, \"number of requests per thread\"),\n+\n \t\tOPT_END()\n \t};\n \n@@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)\n \tif (!strcmp(subcmd, \"flush\"))\n \t\treturn !!do_send_flush();\n \n+\tif (!strcmp(subcmd, \"hammer\"))\n+\t\treturn !!do_hammer(token, nr_threads, nr_requests);\n+\n \tdie(\"Unhandled subcommand: '%s'\", subcmd);\n }\n #endif\n-- \ngitgitgadget\n\n"},{"id":"456074","messageId":"663deabc3f67c25fb6888871b779b0c8f25ac75b.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 04/30] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:26Z","receivedAt":"2022-05-25T15:01:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a worktree, so there is nothing for the\ndaemon watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 18 +++++\n builtin/update-index.c | 16 +++++\n fsmonitor-settings.c | 133 ++++++++++++++++++++++++++++++------\n fsmonitor-settings.h | 16 +++++\n t/t7519-status-fsmonitor.sh | 23 +++++++\n 5 files changed, 186 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 46be55a4618..66b78a0353f 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1423,6 +1423,7 @@ static int try_to_start_background_daemon(void)\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tenum fsmonitor_reason reason;\n \tint detach_console = 0;\n \n \tstruct option options[] = {\n@@ -1449,6 +1450,23 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\t/*\n+\t * If the repo is fsmonitor-compatible, explicitly set IPC-mode\n+\t * (without bothering to load the `core.fsmonitor` config settings).\n+\t *\n+\t * If the repo is not compatible, the repo-settings will be set to\n+\t * incompatible rather than IPC, so we can use one of the __get\n+\t * routines to detect the discrepancy.\n+\t */\n+\tfsm_settings__set_ipc(the_repository);\n+\n+\treason = fsm_settings__get_reason(the_repository);\n+\tif (reason > FSMONITOR_REASON_OK)\n+\t\tdie(\"%s\",\n+\t\t fsm_settings__get_incompatible_msg(the_repository,\n+\t\t\t\t\t\t reason));\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 876112abb21..01ed4c4976b 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1237,6 +1237,22 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n+\n+\t\t/*\n+\t\t * The user wants to turn on FSMonitor using the command\n+\t\t * line argument. (We don't know (or care) whether that\n+\t\t * is the IPC or HOOK version.)\n+\t\t *\n+\t\t * Use one of the __get routines to force load the FSMonitor\n+\t\t * config settings into the repo-settings. That will detect\n+\t\t * whether the file system is compatible so that we can stop\n+\t\t * here with a nice error message.\n+\t\t */\n+\t\tif (reason > FSMONITOR_REASON_OK)\n+\t\t\tdie(\"%s\",\n+\t\t\t fsm_settings__get_incompatible_msg(r, reason));\n+\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 757d230d538..7d3177d441a 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -9,23 +9,42 @@\n */\n struct fsmonitor_settings {\n \tenum fsmonitor_mode mode;\n+\tenum fsmonitor_reason reason;\n \tchar *hook_path;\n };\n \n-static void lookup_fsmonitor_settings(struct repository *r)\n+static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n+{\n+\tif (!r->worktree) {\n+\t\t/*\n+\t\t * Bare repositories don't have a working directory and\n+\t\t * therefore have nothing to watch.\n+\t\t */\n+\t\treturn FSMONITOR_REASON_BARE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n+static struct fsmonitor_settings *alloc_settings(void)\n {\n \tstruct fsmonitor_settings *s;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\ts->reason = FSMONITOR_REASON_UNTESTED;\n+\n+\treturn s;\n+}\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n \tconst char *const_str;\n \tint bool_value;\n \n \tif (r->settings.fsmonitor)\n \t\treturn;\n \n-\tCALLOC_ARRAY(s, 1);\n-\ts->mode = FSMONITOR_MODE_DISABLED;\n-\n-\tr->settings.fsmonitor = s;\n-\n \t/*\n \t * Overload the existing \"core.fsmonitor\" config setting (which\n \t * has historically been either unset or a hook pathname) to\n@@ -38,6 +57,8 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \tcase 0: /* config value was set to */\n \t\tif (bool_value)\n \t\t\tfsm_settings__set_ipc(r);\n+\t\telse\n+\t\t\tfsm_settings__set_disabled(r);\n \t\treturn;\n \n \tcase 1: /* config value was unset */\n@@ -53,18 +74,18 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \t\treturn;\n \t}\n \n-\tif (!const_str || !*const_str)\n-\t\treturn;\n-\n-\tfsm_settings__set_hook(r, const_str);\n+\tif (const_str && *const_str)\n+\t\tfsm_settings__set_hook(r, const_str);\n+\telse\n+\t\tfsm_settings__set_disabled(r);\n }\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n \n \treturn r->settings.fsmonitor->mode;\n }\n@@ -73,31 +94,55 @@ const char *fsm_settings__get_hook_path(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n \n \treturn r->settings.fsmonitor->hook_path;\n }\n \n void fsm_settings__set_ipc(struct repository *r)\n {\n+\tenum fsmonitor_reason reason = check_for_incompatible(r);\n+\n+\tif (reason != FSMONITOR_REASON_OK) {\n+\t\tfsm_settings__set_incompatible(r, reason);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Caller requested IPC explicitly, so avoid (possibly\n+\t * recursive) config lookup.\n+\t */\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n \n void fsm_settings__set_hook(struct repository *r, const char *path)\n {\n+\tenum fsmonitor_reason reason = check_for_incompatible(r);\n+\n+\tif (reason != FSMONITOR_REASON_OK) {\n+\t\tfsm_settings__set_incompatible(r, reason);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Caller requested hook explicitly, so avoid (possibly\n+\t * recursive) config lookup.\n+\t */\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n }\n@@ -106,9 +151,57 @@ void fsm_settings__set_disabled(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\n+\n+void fsm_settings__set_incompatible(struct repository *r,\n+\t\t\t\t enum fsmonitor_reason reason)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n+\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\n+\n+char *fsm_settings__get_incompatible_msg(const struct repository *r,\n+\t\t\t\t\t enum fsmonitor_reason reason)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tswitch (reason) {\n+\tcase FSMONITOR_REASON_UNTESTED:\n+\tcase FSMONITOR_REASON_OK:\n+\t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_BARE:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t xgetcwd());\n+\t\tgoto done;\n+\t}\n+\n+\tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n+\t reason);\n+\n+done:\n+\treturn strbuf_detach(&msg, NULL);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a4c5d7b4889..8d9331c0c0a 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -4,18 +4,34 @@\n struct repository;\n \n enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n \tFSMONITOR_MODE_DISABLED = 0,\n \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n };\n \n+/*\n+ * Incompatibility reasons.\n+ */\n+enum fsmonitor_reason {\n+\tFSMONITOR_REASON_UNTESTED = 0,\n+\tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n+\tFSMONITOR_REASON_BARE,\n+};\n+\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n+void fsm_settings__set_incompatible(struct repository *r,\n+\t\t\t\t enum fsmonitor_reason reason);\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n+char *fsm_settings__get_incompatible_msg(const struct repository *r,\n+\t\t\t\t\t enum fsmonitor_reason reason);\n+\n struct fsmonitor_settings;\n \n #endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex a6308acf006..9a8e21c5608 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '\n \ttest $ret -ne 1\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n+\tgit init --bare bare-clone &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n+\tgit init --bare bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"456075","messageId":"7cb0180a1ed95f9ebecc605cd1d968bced46d375.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 05/30] fsmonitor-settings: stub in Win32-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:27Z","receivedAt":"2022-05-25T15:01:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend generic incompatibility checkout with platform-specific\nmechanism. Stub in Win32 version.\n\nIn the existing fsmonitor-settings code we have a way to mark\ntypes of repos as incompatible with fsmonitor (whether via the\nhook and IPC APIs). For example, we do this for bare repos,\nsince there are no files to watch.\n\nExtend this exclusion mechanism for platform-specific reasons.\nThis commit just creates the framework and adds a stub for Win32.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 +++++++++++++\n compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++\n config.mak.uname | 4 ++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n fsmonitor-settings.c | 10 ++++++++++\n fsmonitor-settings.h | 13 +++++++++++++\n 6 files changed, 52 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n\ndiff --git a/Makefile b/Makefile\nindex daa21bed6c3..93604fe8ef7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -475,6 +475,11 @@ all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n+# If your platform has OS-specific ways to tell if a repo is incompatible with\n+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n+# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n+# that implements the `fsm_os_settings__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n+ifdef FSMONITOR_OS_SETTINGS\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE\n ifdef FSMONITOR_DAEMON_BACKEND\n \t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n endif\n+ifdef FSMONITOR_OS_SETTINGS\n+\t@echo FSMONITOR_OS_SETTINGS=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_OS_SETTINGS)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 501970902da..cf224768ad6 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex ee0d7257b77..16705da2000 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 7d3177d441a..f67db913f57 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -23,6 +23,16 @@ static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n \t\treturn FSMONITOR_REASON_BARE;\n \t}\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+\t{\n+\t\tenum fsmonitor_reason reason;\n+\n+\t\treason = fsm_os__incompatible(r);\n+\t\tif (reason != FSMONITOR_REASON_OK)\n+\t\t\treturn reason;\n+\t}\n+#endif\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 8d9331c0c0a..6cb0d8e7d9f 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -34,4 +34,17 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \n struct fsmonitor_settings;\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+/*\n+ * Ask platform-specific code whether the repository is incompatible\n+ * with fsmonitor (both hook and ipc modes). For example, if the working\n+ * directory is on a remote volume and mounted via a technology that does\n+ * not support notification events, then we should not pretend to watch it.\n+ *\n+ * fsm_os__* routines should considered private to fsm_settings__\n+ * routines.\n+ */\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);\n+#endif /* HAVE_FSMONITOR_OS_SETTINGS */\n+\n #endif /* FSMONITOR_SETTINGS_H */\n-- \ngitgitgadget\n\n"},{"id":"456076","messageId":"9774faddc45cd33362aeb1e3ea80125bc496cbc9.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 06/30] fsmonitor-settings: VFS for Git virtual repos are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:28Z","receivedAt":"2022-05-25T15:01:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nVFS for Git virtual repositories are incompatible with FSMonitor.\n\nVFS for Git is a downstream fork of Git. It contains its own custom\nfile system watcher that is aware of the virtualization. If a working\ndirectory is being managed by VFS for Git, we should not try to watch\nit because we may get incomplete results.\n\nWe do not know anything about how VFS for Git works, but we do\nknow that VFS for Git working directories contain a well-defined\nconfig setting. If it is set, mark the working directory as\nincompatible.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 6 ++++++\n fsmonitor-settings.h | 1 +\n t/t7519-status-fsmonitor.sh | 9 +++++++++\n 4 files changed, 42 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 7fce32a3c5b..ee78bba38e3 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -3,7 +3,33 @@\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n \n+/*\n+ * VFS for Git is incompatible with FSMonitor.\n+ *\n+ * Granted, core Git does not know anything about VFS for Git and we\n+ * shouldn't make assumptions about a downstream feature, but users\n+ * can install both versions. And this can lead to incorrect results\n+ * from core Git commands. So, without bringing in any of the VFS for\n+ * Git code, do a simple config test for a published config setting.\n+ * (We do not look at the various *_TEST_* environment variables.)\n+ */\n+static enum fsmonitor_reason check_vfs4git(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n+\t\treturn FSMONITOR_REASON_VFS4GIT;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_vfs4git(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex f67db913f57..600ae165ab1 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -207,6 +207,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n \t\t\t xgetcwd());\n \t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_VFS4GIT:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 6cb0d8e7d9f..a48802cde9c 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -17,6 +17,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n \tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 9a8e21c5608..156895f9437 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n \tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n '\n \n+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone actual\" &&\n+\tgit init fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"virtual repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"456077","messageId":"5627038aaa3e21c9d8607accfd3b2a46fa60e7fb.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 09/30] fsmonitor-settings: remote repos on Windows are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:31Z","receivedAt":"2022-05-25T15:01:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on Windows and mark them as\nincompatible with FSMonitor.\n\nWith this `git fsmonitor--daemon run` will error out with a message like it\ndoes for bare repos.\n\nClient commands, such as `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++\n 1 file changed, 102 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex ee78bba38e3..907655720bb 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * VFS for Git is incompatible with FSMonitor.\n@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)\n \treturn FSMONITOR_REASON_OK;\n }\n \n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type dictates whether notification events are available at\n+ * all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ *\n+ * Notes for testing:\n+ *\n+ * (a) Windows allows a network share to be mapped to a drive letter.\n+ * (This is the normal method to access it.)\n+ *\n+ * $ NET USE Z: \\\\server\\share\n+ * $ git -C Z:/repo status\n+ *\n+ * (b) Windows allows a network share to be referenced WITHOUT mapping\n+ * it to drive letter.\n+ *\n+ * $ NET USE \\\\server\\share\\dir\n+ * $ git -C //server/share/repo status\n+ *\n+ * (c) Windows allows \"SUBST\" to create a fake drive mapping to an\n+ * arbitrary path (which may be remote)\n+ *\n+ * $ SUBST Q: Z:\\repo\n+ * $ git -C Q:/ status\n+ *\n+ * (d) Windows allows a directory symlink to be created on a local\n+ * file system that points to a remote repo.\n+ *\n+ * $ mklink /d ./link //server/share/repo\n+ * $ git -C ./link status\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\twchar_t wpath[MAX_PATH];\n+\twchar_t wfullpath[MAX_PATH];\n+\tsize_t wlen;\n+\tUINT driveType;\n+\n+\t/*\n+\t * Do everything in wide chars because the drive letter might be\n+\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n+\t */\n+\tif (xutftowcs_path(wpath, r->worktree) < 0)\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\t/*\n+\t * GetDriveTypeW() requires a final slash. We assume that the\n+\t * worktree pathname points to an actual directory.\n+\t */\n+\twlen = wcslen(wpath);\n+\tif (wpath[wlen - 1] != L'\\\\' && wpath[wlen - 1] != L'/') {\n+\t\twpath[wlen++] = L'\\\\';\n+\t\twpath[wlen] = 0;\n+\t}\n+\n+\t/*\n+\t * Normalize the path. If nothing else, this converts forward\n+\t * slashes to backslashes. This is essential to get GetDriveTypeW()\n+\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n+\t */\n+\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\tdriveType = GetDriveTypeW(wfullpath);\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"DriveType '%s' L'%ls' (%u)\",\n+\t\t\t r->worktree, wfullpath, driveType);\n+\n+\tif (driveType == DRIVE_REMOTE) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"check_remote('%s') true\",\n+\t\t\t\t r->worktree);\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n-- \ngitgitgadget\n\n"},{"id":"456078","messageId":"dc2dfd679312a71ef0613857860b4ad426913471.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 08/30] fsmonitor-settings: remote repos on macOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:30Z","receivedAt":"2022-05-25T15:01:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on macOS and mark them as\nincompatible with FSMonitor.\n\nWith this, `git fsmonitor--daemon run` will error out with a message\nlike it does for bare repos.\n\nClient commands, like `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 12 +++++\n fsmonitor-settings.h | 2 +\n 3 files changed, 80 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex 7fce32a3c5b..fdd762bf79d 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -2,8 +2,74 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n+#include \n+#include \n+\n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type (NFS, SAMBA, etc.) dictates whether notification events\n+ * are available at all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * For the builtin FSMonitor, we create the Unix domain socket for the\n+ * IPC in the .git directory. If the working directory is remote,\n+ * then the socket will be created on the remote file system. This\n+ * can fail if the remote file system does not support UDS file types\n+ * (e.g. smbfs to a Windows server) or if the remote kernel does not\n+ * allow a non-local process to bind() the socket. (These problems\n+ * could be fixed by moving the UDS out of the .git directory and to a\n+ * well-known local directory on the client machine, but care should\n+ * be taken to ensure that $HOME is actually local and not a managed\n+ * file share.)\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\tstruct statfs fs;\n+\n+\tif (statfs(r->worktree, &fs) == -1) {\n+\t\tint saved_errno = errno;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n+\t\t\t\t r->worktree, strerror(saved_errno));\n+\t\terrno = saved_errno;\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n+\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n+\n+\tif (!(fs.f_flags & MNT_LOCAL))\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 600ae165ab1..d2fb0141f8e 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -208,6 +208,18 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t xgetcwd());\n \t\tgoto done;\n \n+\tcase FSMONITOR_REASON_ERROR:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_REMOTE:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"remote repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n+\n \tcase FSMONITOR_REASON_VFS4GIT:\n \t\tstrbuf_addf(&msg,\n \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a48802cde9c..afd1b3874ac 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -17,6 +17,8 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n \tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n+\tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"456079","messageId":"aaff000cecb43550e0974bfda117e1c065ab8203.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 11/30] unpack-trees: initialize fsmonitor_has_run_once in o->result","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:33Z","receivedAt":"2022-05-25T15:01:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInitialize `o->result.fsmonitor_has_run_once` based upon value\nin `o->src_index->fsmonitor_has_run_once` to prevent a second\nfsmonitor query during the tree traversal and possibly getting\na skewed view of the working directory.\n\nThe checkout code has already talked to the fsmonitor and the\ntraversal is updating the index as it traverses, so there is\nno need to query the fsmonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n unpack-trees.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..888cff81f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \to->result.fsmonitor_last_update =\n \t\txstrdup_or_null(o->src_index->fsmonitor_last_update);\n+\to->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;\n \n \t/*\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n-- \ngitgitgadget\n\n"},{"id":"456080","messageId":"9a12cc78b5d16a05d4ed76cdb2a1f634ed32ef8a.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 10/30] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:32Z","receivedAt":"2022-05-25T15:01:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nOn MacOS mark repos on NTFS or FAT32 volumes as incompatible.\n\nThe builtin FSMonitor used Unix domain sockets on MacOS for IPC\nwith clients. These sockets are kept in the .git directory.\nUnix sockets are not supported by NTFS and FAT32, so the daemon\ncannot start up.\n\nTest for this during our compatibility checking so that client\ncommands do not keep trying to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---\n fsmonitor-settings.c | 6 ++++++\n fsmonitor-settings.h | 1 +\n 3 files changed, 24 insertions(+), 3 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex fdd762bf79d..efc732c0f31 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -7,7 +7,7 @@\n #include \n \n /*\n- * Remote working directories are problematic for FSMonitor.\n+ * [1] Remote working directories are problematic for FSMonitor.\n *\n * The underlying file system on the server machine and/or the remote\n * mount type (NFS, SAMBA, etc.) dictates whether notification events\n@@ -40,8 +40,16 @@\n *\n * So (for now at least), mark remote working directories as\n * incompatible.\n+ *\n+ *\n+ * [2] FAT32 and NTFS working directories are problematic too.\n+ *\n+ * The builtin FSMonitor uses a Unix domain socket in the .git\n+ * directory for IPC. These Windows drive formats do not support\n+ * Unix domain sockets, so mark them as incompatible for the daemon.\n+ *\n */\n-static enum fsmonitor_reason check_remote(struct repository *r)\n+static enum fsmonitor_reason check_volume(struct repository *r)\n {\n \tstruct statfs fs;\n \n@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)\n \tif (!(fs.f_flags & MNT_LOCAL))\n \t\treturn FSMONITOR_REASON_REMOTE;\n \n+\tif (!strcmp(fs.f_fstypename, \"msdos\")) /* aka FAT32 */\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n+\tif (!strcmp(fs.f_fstypename, \"ntfs\"))\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \n@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n \n-\treason = check_remote(r);\n+\treason = check_volume(r);\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex d2fb0141f8e..658cb79da01 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -225,6 +225,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t\t r->worktree);\n \t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_NOSOCKETS:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex afd1b3874ac..d9c2605197f 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -20,6 +20,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n \tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n+\tFSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n-- \ngitgitgadget\n\n"},{"id":"456081","messageId":"4f2b15d3d1f10eaec8e485b84281eb4cca207a69.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 12/30] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:34Z","receivedAt":"2022-05-25T15:01:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIgnore FSEvents resulting from `xattr` changes. Git does not care about\nxattr's or changes to xattr's, so don't waste time collecting these\nevents in the daemon nor transmitting them to clients.\n\nVarious security tools add xattrs to files and/or directories, such as\nto mark them as having been downloaded. We should ignore these events\nsince it doesn't affect the content of the file/directory or the normal\nmeta-data that Git cares about.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 0741fe834c3..14105f45c18 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tif (flag & kFSEventStreamEventFlagItemCloned)\n \t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n \n-\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n \t\t\t path, flag, msg.buf);\n \n \tstrbuf_release(&msg);\n@@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)\n \t\tef & kFSEventStreamEventFlagUserDropped);\n }\n \n+/*\n+ * If an `xattr` change is the only reason we received this event,\n+ * then silently ignore it. Git doesn't care about xattr's. We\n+ * have to be careful here because the kernel can combine multiple\n+ * events for a single path. And because events always have certain\n+ * bits set, such as `ItemIsFile` or `ItemIsDir`.\n+ *\n+ * Return 1 if we should ignore it.\n+ */\n+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n+{\n+\tstatic const FSEventStreamEventFlags mask =\n+\t\tkFSEventStreamEventFlagItemChangeOwner |\n+\t\tkFSEventStreamEventFlagItemCreated |\n+\t\tkFSEventStreamEventFlagItemFinderInfoMod |\n+\t\tkFSEventStreamEventFlagItemInodeMetaMod |\n+\t\tkFSEventStreamEventFlagItemModified |\n+\t\tkFSEventStreamEventFlagItemRemoved |\n+\t\tkFSEventStreamEventFlagItemRenamed |\n+\t\tkFSEventStreamEventFlagItemXattrMod |\n+\t\tkFSEventStreamEventFlagItemCloned;\n+\n+\treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n+}\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_ignore_xattr(event_flags[k])) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n+\t\t\t\t\t path_k, event_flags[k]);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n \n \t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-- \ngitgitgadget\n\n"},{"id":"456082","messageId":"c2b5c02ed3839902ed2734418fc493dd4b7e0725.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 16/30] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:38Z","receivedAt":"2022-05-25T15:01:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate another thread to watch over the daemon process and\nautomatically shut it down if necessary.\n\nThis commit creates the basic framework for a \"health\" thread\nto monitor the daemon and/or the file system. Later commits\nwill add platform-specific code to do the actual work.\n\nThe \"health\" thread is intended to monitor conditions that\nwould be difficult to track inside the IPC thread pool and/or\nthe file system listener threads. For example, when there are\nfile system events outside of the watched worktree root or if\nwe want to have an idle-timeout auto-shutdown feature.\n\nThis commit creates the health thread itself, defines the thread-proc\nand sets up the thread's event loop. It integrates this new thread\ninto the existing IPC and Listener thread models.\n\nThis commit defines the API to the platform-specific code where all of\nthe monitoring will actually happen.\n\nThe platform-specific code for MacOS is just stubs. Meaning that the\nhealth thread will immediately exit on MacOS, but that is OK and\nexpected. Future work can define MacOS-specific monitoring.\n\nThe platform-specific code for Windows sets up enough of the\nWaitForMultipleObjects() machinery to watch for system and/or custom\nevents. Currently, the set of wait handles only includes our custom\nshutdown event (sent from our other theads). Later commits in this\nseries will extend the set of wait handles to monitor other\nconditions.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 6 ++-\n builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n contrib/buildsystems/CMakeLists.txt | 2 +\n fsmonitor--daemon.h | 4 ++\n 7 files changed, 192 insertions(+), 2 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n\ndiff --git a/Makefile b/Makefile\nindex 93604fe8ef7..5f1623baadd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -472,8 +472,9 @@ all::\n #\n # If your platform supports a built-in fsmonitor backend, set\n # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n-# `compat/fsmonitor/fsm-listen-.c` that implements the\n-# `fsm_listen__*()` routines.\n+# `compat/fsmonitor/fsm-listen-.c` and\n+# `compat/fsmonitor/fsm-health-.c` files\n+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n # If your platform has OS-specific ways to tell if a repo is incompatible with\n # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n@@ -1982,6 +1983,7 @@ endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n ifdef FSMONITOR_OS_SETTINGS\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex b2f578b239a..2c109cf8b37 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n@@ -1136,6 +1137,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+static void *fsm_health__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-health\");\n+\n+\tfsm_health__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -1174,6 +1187,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint health_started = 0;\n \tint listener_started = 0;\n \tint err = 0;\n \n@@ -1201,6 +1215,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t}\n \tlistener_started = 1;\n \n+\t/*\n+\t * Start the health thread to watch over our process.\n+\t */\n+\tif (pthread_create(&state->health_thread, NULL,\n+\t\t\t fsm_health__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\terr = error(_(\"could not start fsmonitor health thread\"));\n+\t\tgoto cleanup;\n+\t}\n+\thealth_started = 1;\n+\n \t/*\n \t * The daemon is now fully functional in background threads.\n \t * Our primary thread should now just wait while the threads\n@@ -1223,10 +1248,17 @@ cleanup:\n \t\tpthread_join(state->listener_thread, NULL);\n \t}\n \n+\tif (health_started) {\n+\t\tfsm_health__stop_async(state);\n+\t\tpthread_join(state->health_thread, NULL);\n+\t}\n+\n \tif (err)\n \t\treturn err;\n \tif (state->listen_error_code)\n \t\treturn state->listen_error_code;\n+\tif (state->health_error_code)\n+\t\treturn state->health_error_code;\n \treturn 0;\n }\n \n@@ -1242,6 +1274,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.listen_error_code = 0;\n+\tstate.health_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n@@ -1321,6 +1354,11 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\tif (fsm_health__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize health thread\"));\n+\t\tgoto done;\n+\t}\n+\n \t/*\n \t * CD out of the worktree root directory.\n \t *\n@@ -1344,6 +1382,7 @@ done:\n \tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n+\tfsm_health__dtor(&state);\n \n \tipc_server_free(state.ipc_server_data);\n \ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nnew file mode 100644\nindex 00000000000..b9f709e8548\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -0,0 +1,24 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nnew file mode 100644\nindex 00000000000..94b1d020f25\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsm_health_data\n+{\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hHandles[1]; /* the array does not own these handles */\n+#define HEALTH_SHUTDOWN 0\n+\tint nr_handles; /* number of active event handles */\n+};\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_handles++;\n+\n+\tstate->health_data = data;\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tif (!state || !state->health_data)\n+\t\treturn;\n+\n+\tdata = state->health_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\n+\tFREE_AND_NULL(state->health_data);\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\n+\tfor (;;) {\n+\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n+\t\t\t\t\t\t data->hHandles,\n+\t\t\t\t\t\t FALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->health_error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+clean_shutdown:\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n+}\ndiff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\nnew file mode 100644\nindex 00000000000..45547ba9380\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health.h\n@@ -0,0 +1,47 @@\n+#ifndef FSM_HEALTH_H\n+#define FSM_HEALTH_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor health thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the health thread.\n+ * This will be called from the main thread AFTER joining the thread.\n+ */\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to monitor the\n+ * health of the daemon process. This will run in the health thread.\n+ *\n+ * The health thread should call `ipc_server_stop_async()` if it needs\n+ * to cause a shutdown. (It should NOT do so if it receives a shutdown\n+ * shutdown signal.)\n+ *\n+ * It should set `state->health_error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_health__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the health thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_HEALTH_H */\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex b8f9f7a0388..16ace43d1c7 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 2c6fa1a5d91..2102a5c9ff5 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n+struct fsm_health_data; /* opaque platform-specific data for health thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n+\tpthread_t health_thread;\n \tpthread_mutex_t main_lock;\n \n \tstruct strbuf path_worktree_watch;\n@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n \tstruct hashmap cookies;\n \n \tint listen_error_code;\n+\tint health_error_code;\n \tstruct fsm_listen_data *listen_data;\n+\tstruct fsm_health_data *health_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"456083","messageId":"46a5ae2a6353e1da3c3ba84c9a2b5e03d6eeae44.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 17/30] fsm-health-win32: add polling framework to monitor daemon health","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:39Z","receivedAt":"2022-05-25T15:01:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend the Windows version of the \"health\" thread to periodically\ninspect the system and shutdown if warranted.\n\nThis commit updates the thread's wait loop to use a timeout and\ndefines a (currently empty) table of functions to poll the system.\n\nA later commit will add functions to the table to actually\ninspect the system.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-\n 1 file changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 94b1d020f25..24fc612bf02 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -4,6 +4,24 @@\n #include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n+/*\n+ * Every minute wake up and test our health.\n+ */\n+#define WAIT_FREQ_MS (60 * 1000)\n+\n+/*\n+ * State machine states for each of the interval functions\n+ * used for polling our health.\n+ */\n+enum interval_fn_ctx {\n+\tCTX_INIT = 0,\n+\tCTX_TERM,\n+\tCTX_TIMER\n+};\n+\n+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx);\n+\n struct fsm_health_data\n {\n \tHANDLE hEventShutdown;\n@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n \tFREE_AND_NULL(state->health_data);\n }\n \n+/*\n+ * A table of the polling functions.\n+ */\n+static interval_fn *table[] = {\n+\tNULL, /* must be last */\n+};\n+\n+/*\n+ * Call all of the polling functions in the table.\n+ * Shortcut and return first error.\n+ *\n+ * Return 0 if all succeeded.\n+ */\n+static int call_all(struct fsmonitor_daemon_state *state,\n+\t\t enum interval_fn_ctx ctx)\n+{\n+\tint k;\n+\n+\tfor (k = 0; table[k]; k++) {\n+\t\tint r = table[k](state, ctx);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n+\tint r;\n+\n+\tr = call_all(state, CTX_INIT);\n+\tif (r < 0)\n+\t\tgoto force_error_stop;\n+\tif (r > 0)\n+\t\tgoto force_shutdown;\n \n \tfor (;;) {\n \t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n \t\t\t\t\t\t data->hHandles,\n-\t\t\t\t\t\t FALSE, INFINITE);\n+\t\t\t\t\t\t FALSE, WAIT_FREQ_MS);\n \n \t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n \t\t\tgoto clean_shutdown;\n \n+\t\tif (dwWait == WAIT_TIMEOUT) {\n+\t\t\tr = call_all(state, CTX_TIMER);\n+\t\t\tif (r < 0)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (r > 0)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\terror(_(\"health thread wait failed [GLE %ld]\"),\n \t\t GetLastError());\n \t\tgoto force_error_stop;\n@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)\n \n force_error_stop:\n \tstate->health_error_code = -1;\n+force_shutdown:\n \tipc_server_stop_async(state->ipc_server_data);\n clean_shutdown:\n+\tcall_all(state, CTX_TERM);\n \treturn;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"456084","messageId":"51b266b06e1739f09d785787c93fc3e346f47332.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 14/30] fsmonitor--daemon: prepare for adding health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:36Z","receivedAt":"2022-05-25T15:01:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor daemon thread startup to make it easier to start\na third thread class to monitor the health of the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex db297649daf..90fa9d09efb 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1174,6 +1174,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint listener_started = 0;\n+\tint err = 0;\n \n \t/*\n \t * Start the IPC thread pool before the we've started the file\n@@ -1194,15 +1196,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \tif (pthread_create(&state->listener_thread, NULL,\n \t\t\t fsm_listen__thread_proc, state) < 0) {\n \t\tipc_server_stop_async(state->ipc_server_data);\n-\t\tipc_server_await(state->ipc_server_data);\n-\n-\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t\terr = error(_(\"could not start fsmonitor listener thread\"));\n+\t\tgoto cleanup;\n \t}\n+\tlistener_started = 1;\n \n \t/*\n \t * The daemon is now fully functional in background threads.\n+\t * Our primary thread should now just wait while the threads\n+\t * do all the work.\n+\t */\n+cleanup:\n+\t/*\n \t * Wait for the IPC thread pool to shutdown (whether by client\n-\t * request or from filesystem activity).\n+\t * request, from filesystem activity, or an error).\n \t */\n \tipc_server_await(state->ipc_server_data);\n \n@@ -1211,10 +1218,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * event from the IPC thread pool, but it doesn't hurt to tell\n \t * it again. And wait for it to shutdown.\n \t */\n-\tfsm_listen__stop_async(state);\n-\tpthread_join(state->listener_thread, NULL);\n+\tif (listener_started) {\n+\t\tfsm_listen__stop_async(state);\n+\t\tpthread_join(state->listener_thread, NULL);\n+\t}\n \n-\treturn state->error_code;\n+\tif (err)\n+\t\treturn err;\n+\tif (state->error_code)\n+\t\treturn state->error_code;\n+\treturn 0;\n }\n \n static int fsmonitor_run_daemon(void)\n-- \ngitgitgadget\n\n"},{"id":"456085","messageId":"594e0ae243de868ff18f94aafc4358d5801b88c5.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 15/30] fsmonitor--daemon: rename listener thread related variables","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:37Z","receivedAt":"2022-05-25T15:01:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRename platform-specific listener thread related variables\nand data types as we prepare to add another backend thread\ntype.\n\n[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`\n[] `state->backend_data` becomes `state->listen_data`\n[] `state->error_code` becomes `state->listen_error_code`\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 +++---\n compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------\n compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++-------------\n compat/fsmonitor/fsm-listen.h | 2 +-\n fsmonitor--daemon.h | 6 +++---\n 5 files changed, 36 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 90fa9d09efb..b2f578b239a 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1225,8 +1225,8 @@ cleanup:\n \n \tif (err)\n \t\treturn err;\n-\tif (state->error_code)\n-\t\treturn state->error_code;\n+\tif (state->listen_error_code)\n+\t\treturn state->listen_error_code;\n \treturn 0;\n }\n \n@@ -1241,7 +1241,7 @@ static int fsmonitor_run_daemon(void)\n \thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n-\tstate.error_code = 0;\n+\tstate.listen_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 14105f45c18..07113205a61 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -27,7 +27,7 @@\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tCFStringRef cfsr_worktree_path;\n \tCFStringRef cfsr_gitdir_path;\n@@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t const FSEventStreamEventId event_ids[])\n {\n \tstruct fsmonitor_daemon_state *state = ctx;\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tchar **paths = (char **)event_paths;\n \tstruct fsmonitor_batch *batch = NULL;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tNULL,\n \t\tNULL\n \t};\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \tconst void *dir_array[2];\n \n \tCALLOC_ARRAY(data, 1);\n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \n \tdata->cfsr_worktree_path = CFStringCreateWithCString(\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n@@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n \terror(_(\"Unable to create FSEventStream.\"));\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tif (data->stream) {\n \t\tif (data->stream_started)\n@@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n \t\tFSEventStreamRelease(data->stream);\n \t}\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \tdata->shutdown_style = SHUTDOWN_EVENT;\n \n \tCFRunLoopStop(data->rl);\n@@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tdata->rl = CFRunLoopGetCurrent();\n \n@@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \n \tswitch (data->shutdown_style) {\n \tcase FORCE_ERROR_STOP:\n-\t\tstate->error_code = -1;\n+\t\tstate->listen_error_code = -1;\n \t\t/* fall thru */\n \tcase FORCE_SHUTDOWN:\n \t\tipc_server_stop_async(state->ipc_server_data);\n@@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \treturn;\n \n force_error_stop_without_loop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \tipc_server_stop_async(state->ipc_server_data);\n \treturn;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 7f48306df53..2943632c771 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -54,7 +54,7 @@ struct one_watch\n \twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tstruct one_watch *watch_worktree;\n \tstruct one_watch *watch_gitdir;\n@@ -284,7 +284,7 @@ static enum get_relative_result get_relative_longname(\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+\tSetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);\n }\n \n static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n@@ -359,7 +359,7 @@ static void destroy_watch(struct one_watch *watch)\n \tfree(watch);\n }\n \n-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+static int start_rdcw_watch(struct fsm_listen_data *data,\n \t\t\t struct one_watch *watch)\n {\n \tDWORD dwNotifyFilter =\n@@ -538,7 +538,7 @@ static int process_1_worktree_event(\n */\n static int process_worktree_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_worktree;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -669,7 +669,7 @@ force_shutdown:\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_gitdir;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -727,11 +727,11 @@ skip_this_path:\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tDWORD dwWait;\n \tint result;\n \n-\tstate->error_code = 0;\n+\tstate->listen_error_code = 0;\n \n \tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n \t\tgoto force_error_stop;\n@@ -796,7 +796,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \t}\n \n force_error_stop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \n force_shutdown:\n \t/*\n@@ -813,7 +813,7 @@ clean_shutdown:\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -846,7 +846,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \treturn 0;\n \n failed:\n@@ -859,16 +859,16 @@ failed:\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tCloseHandle(data->hEventShutdown);\n \tdestroy_watch(data->watch_worktree);\n \tdestroy_watch(data->watch_gitdir);\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nindex f0539349baf..41650bf8972 100644\n--- a/compat/fsmonitor/fsm-listen.h\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n * do so if the listener thread receives a normal shutdown signal from\n * the IPC layer.)\n *\n- * It should set `state->error_code` to -1 if the daemon should exit\n+ * It should set `state->listen_error_code` to -1 if the daemon should exit\n * with an error.\n */\n void fsm_listen__loop(struct fsmonitor_daemon_state *state);\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 223c2131b58..2c6fa1a5d91 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n */\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {\n \tint cookie_seq;\n \tstruct hashmap cookies;\n \n-\tint error_code;\n-\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\tint listen_error_code;\n+\tstruct fsm_listen_data *listen_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"456086","messageId":"427dec412a559bffab789126ff8fe34eaef346fd.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 13/30] fsmonitor--daemon: cd out of worktree root","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:35Z","receivedAt":"2022-05-25T15:02:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the fsmonitor--daemon to CD outside of the worktree\nbefore starting up.\n\nThe common Git startup mechanism causes the CWD of the daemon process\nto be in the root of the worktree. On Windows, this causes the daemon\nprocess to hold a locked handle on the CWD and prevents other\nprocesses from moving or deleting the worktree while the daemon is\nrunning.\n\nCD to HOME before entering main event loops.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++--\n compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------\n fsmonitor--daemon.h | 1 +\n 3 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 66b78a0353f..db297649daf 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1181,11 +1181,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * before we need it.\n \t */\n \tif (ipc_server_run_async(&state->ipc_server_data,\n-\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t state->path_ipc.buf, &ipc_opts,\n \t\t\t\t handle_client, state))\n \t\treturn error_errno(\n \t\t\t_(\"could not start IPC thread pool on '%s'\"),\n-\t\t\tfsmonitor_ipc__get_path());\n+\t\t\tstate->path_ipc.buf);\n \n \t/*\n \t * Start the fsmonitor listener thread to collect filesystem\n@@ -1220,6 +1220,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n static int fsmonitor_run_daemon(void)\n {\n \tstruct fsmonitor_daemon_state state;\n+\tconst char *home;\n \tint err;\n \n \tmemset(&state, 0, sizeof(state));\n@@ -1289,6 +1290,15 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_addch(&state.path_cookie_prefix, '/');\n \n+\t/*\n+\t * We create a named-pipe or unix domain socket inside of the\n+\t * \".git\" directory. (Well, on Windows, we base our named\n+\t * pipe in the NPFS on the absolute path of the git\n+\t * directory.)\n+\t */\n+\tstrbuf_init(&state.path_ipc, 0);\n+\tstrbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1298,6 +1308,23 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\t/*\n+\t * CD out of the worktree root directory.\n+\t *\n+\t * The common Git startup mechanism causes our CWD to be the\n+\t * root of the worktree. On Windows, this causes our process\n+\t * to hold a locked handle on the CWD. This prevents the\n+\t * worktree from being moved or deleted while the daemon is\n+\t * running.\n+\t *\n+\t * We assume that our FS and IPC listener threads have either\n+\t * opened all of the handles that they need or will do\n+\t * everything using absolute paths.\n+\t */\n+\thome = getenv(\"HOME\");\n+\tif (home && *home && chdir(home))\n+\t\tdie_errno(_(\"could not cd home '%s'\"), home);\n+\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n@@ -1310,6 +1337,7 @@ done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n \tstrbuf_release(&state.path_cookie_prefix);\n+\tstrbuf_release(&state.path_ipc);\n \n \treturn err;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 0a86aea3f7e..7f48306df53 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -424,12 +424,22 @@ static int recv_rdcw_watch(struct one_watch *watch)\n \t}\n \n \t/*\n-\t * NEEDSWORK: If an external is deleted, the above\n-\t * returns an error. I'm not sure that there's anything that\n-\t * we can do here other than failing -- the /.git\n-\t * link file would be broken anyway. We might try to check\n-\t * for that and return a better error message, but I'm not\n-\t * sure it is worth it.\n+\t * GetOverlappedResult() fails if the watched directory is\n+\t * deleted while we were waiting for an overlapped IO to\n+\t * complete. The documentation did not list specific errors,\n+\t * but I observed ERROR_ACCESS_DENIED (0x05) errors during\n+\t * testing.\n+\t *\n+\t * Note that we only get notificaiton events for events\n+\t * *within* the directory, not *on* the directory itself.\n+\t * (These might be properies of the parent directory, for\n+\t * example).\n+\t *\n+\t * NEEDSWORK: We might try to check for the deleted directory\n+\t * case and return a better error message, but I'm not sure it\n+\t * is worth it.\n+\t *\n+\t * Shutdown if we get any error.\n \t */\n \n \terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex bd09fffc176..223c2131b58 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n+\tstruct strbuf path_ipc;\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"456087","messageId":"f7ef7dcffc88f95dcf8fbca4d6e24648d52152c7.1653490852.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 07/30] fsmonitor-settings: stub in macOS-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:29Z","receivedAt":"2022-05-25T15:02:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++\n config.mak.uname | 1 +\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 13 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cf224768ad6..cf911d141f2 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)\n \tifndef NO_PTHREADS\n \tifndef NO_UNIX_SOCKETS\n \tFSMONITOR_DAEMON_BACKEND = darwin\n+\tFSMONITOR_OS_SETTINGS = darwin\n \tendif\n \tendif\n \ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 16705da2000..b8f9f7a0388 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"456088","messageId":"7cf1be5f8e2c644a5bce9eb4484de2f6c82af359.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 18/30] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:40Z","receivedAt":"2022-05-25T15:02:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nForce shutdown fsmonitor daemon if the worktree root directory\nis moved, renamed, or deleted.\n\nUse Windows low-level GetFileInformationByHandle() to get and\ncompare the Windows system unique ID for the directory with a\ncached version when we started up. This lets us detect the\ncase where someone renames the directory that we are watching\nand then creates a new directory with the original pathname.\n\nThis is important because we are listening to a named pipe for\nrequests and they are stored in the Named Pipe File System (NPFS)\nwhich a kernel-resident pseudo filesystem not associated with\nthe actual NTFS directory.\n\nFor example, if the daemon was watching \"~/foo/\", it would have\na directory-watch handle on that directory and a named-pipe\nhandle for \"//./pipe/...foo\". Moving the directory to \"~/bar/\"\ndoes not invalidate the directory handle. (So the daemon would\nactually be watching \"~/bar\" but listening on \"//./pipe/...foo\".\nIf the user then does \"git init ~/foo\" and causes another daemon\nto start, the first daemon will still have ownership of the pipe\nand the second daemon instance will fail to start. \"git status\"\nclients in \"~/foo\" will ask \"//./pipe/...foo\" about changes and\nthe first daemon instance will tell them about \"~/bar\".\n\nThis commit causes the first daemon to shutdown if the system unique\nID for \"~/foo\" changes (changes from what it was when the daemon\nstarted). Shutdown occurs after a periodic poll. After the\nfirst daemon exits and releases the lock on the named pipe,\nsubsequent Git commands may cause another daemon to be started\non \"~/foo\". Similarly, a subsequent Git command may cause another\ndaemon to be started on \"~/bar\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++\n 1 file changed, 143 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 24fc612bf02..2ea08c1d4e8 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -29,8 +29,150 @@ struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n \tint nr_handles; /* number of active event handles */\n+\n+\tstruct wt_moved\n+\t{\n+\t\twchar_t wpath[MAX_PATH + 1];\n+\t\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\t} wt_moved;\n };\n \n+/*\n+ * Lookup the system unique ID for the path. This is as close as\n+ * we get to an inode number, but this also contains volume info,\n+ * so it is a little stronger.\n+ */\n+static int lookup_bhfi(wchar_t *wpath,\n+\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n+{\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileW(wpath, desired_access, share_mode, NULL,\n+\t\t\t OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] health thread could not open '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\treturn -1;\n+\t}\n+\n+\tif (!GetFileInformationByHandle(hDir, bhfi)) {\n+\t\terror(_(\"[GLE %ld] health thread getting BHFI for '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\tCloseHandle(hDir);\n+\t\treturn -1;\n+\t}\n+\n+\tCloseHandle(hDir);\n+\treturn 0;\n+}\n+\n+/*\n+ * Compare the relevant fields from two system unique IDs.\n+ * We use this to see if two different handles to the same\n+ * path actually refer to the same *instance* of the file\n+ * or directory.\n+ */\n+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n+\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n+{\n+\treturn (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&\n+\t\tbhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&\n+\t\tbhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);\n+}\n+\n+/*\n+ * Shutdown if the original worktree root directory been deleted,\n+ * moved, or renamed?\n+ *\n+ * Since the main thread did a \"chdir(getenv($HOME))\" and our CWD\n+ * is not in the worktree root directory and because the listener\n+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible\n+ * for the root directory to be moved or deleted while we are still\n+ * watching it. We want to detect that here and force a shutdown.\n+ *\n+ * Granted, a delete MAY cause some operations to fail, such as\n+ * GetOverlappedResult(), but it is not guaranteed. And because\n+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the\n+ * directory, not changes *ON* the directory, our watch will not\n+ * receive a delete event for it.\n+ *\n+ * A move/rename of the worktree root will also not generate an event.\n+ * And since the listener thread already has an open handle, it may\n+ * continue to receive events for events within the directory.\n+ * However, the pathname of the named-pipe was constructed using the\n+ * original location of the worktree root. (Remember named-pipes are\n+ * stored in the NPFS and not in the actual file system.) Clients\n+ * trying to talk to the worktree after the move/rename will not\n+ * reach our daemon process, since we're still listening on the\n+ * pipe with original path.\n+ *\n+ * Furthermore, if the user does something like:\n+ *\n+ * $ mv repo repo.old\n+ * $ git init repo\n+ *\n+ * A new daemon cannot be started in the new instance of \"repo\"\n+ * because the named-pipe is still being used by the daemon on\n+ * the original instance.\n+ *\n+ * So, detect move/rename/delete and shutdown. This should also\n+ * handle unsafe drive removal.\n+ *\n+ * We use the file system unique ID to distinguish the original\n+ * directory instance from a new instance and force a shutdown\n+ * if the unique ID changes.\n+ *\n+ * Since a worktree move/rename/delete/unmount doesn't happen\n+ * that often (and we can't get an immediate event anyway), we\n+ * use a timeout and periodically poll it.\n+ */\n+static int has_worktree_moved(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\tint r;\n+\n+\tswitch (ctx) {\n+\tcase CTX_TERM:\n+\t\treturn 0;\n+\n+\tcase CTX_INIT:\n+\t\tif (xutftowcs_path(data->wt_moved.wpath,\n+\t\t\t\t state->path_worktree_watch.buf) < 0) {\n+\t\t\terror(_(\"could not convert to wide characters: '%s'\"),\n+\t\t\t state->path_worktree_watch.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/*\n+\t\t * On the first call we lookup the unique sequence ID for\n+\t\t * the worktree root directory.\n+\t\t */\n+\t\treturn lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);\n+\n+\tcase CTX_TIMER:\n+\t\tr = lookup_bhfi(data->wt_moved.wpath, &bhfi);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t\tif (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {\n+\t\t\terror(_(\"BHFI changed '%ls'\"), data->wt_moved.wpath);\n+\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tdie(_(\"unhandled case in 'has_worktree_moved': %d\"),\n+\t\t (int)ctx);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+\n int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data;\n@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n * A table of the polling functions.\n */\n static interval_fn *table[] = {\n+\thas_worktree_moved,\n \tNULL, /* must be last */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"456089","messageId":"d058d7e0c0829bed9df266c4f6c9f12c30caad4d.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 21/30] t7527: FSMonitor tests for directory moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:43Z","receivedAt":"2022-05-25T15:02:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate unit tests to move a directory. Verify that `git status`\ngives the same result with and without FSMonitor enabled.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 12655958e71..3bc335b891d 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -274,6 +274,16 @@ test_expect_success 'setup' '\n \ttrace*\n \tEOF\n \n+\tmkdir -p T1/T2/T3/T4 &&\n+\techo 1 >T1/F1 &&\n+\techo 1 >T1/T2/F1 &&\n+\techo 1 >T1/T2/T3/F1 &&\n+\techo 1 >T1/T2/T3/T4/F1 &&\n+\techo 2 >T1/F2 &&\n+\techo 2 >T1/T2/F2 &&\n+\techo 2 >T1/T2/T3/F2 &&\n+\techo 2 >T1/T2/T3/T4/F2 &&\n+\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n@@ -356,6 +366,19 @@ directory_to_file () {\n \techo 1 >dir1\n }\n \n+move_directory_contents_deeper() {\n+\tmkdir T1/_new_ &&\n+\tmv T1/[A-Z]* T1/_new_\n+}\n+\n+move_directory_up() {\n+\tmv T1/T2/T3 T1\n+}\n+\n+move_directory() {\n+\tmv T1/T2/T3 T1/T2/NewT3\n+}\n+\n # The next few test cases confirm that our fsmonitor daemon sees each type\n # of OS filesystem notification that we care about. At this layer we just\n # ensure we are getting the OS notifications and do not try to confirm what\n@@ -660,6 +683,10 @@ do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n+\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n+\t\tmatrix_try $uc_val $fsm_val move_directory_up\n+\t\tmatrix_try $uc_val $fsm_val move_directory\n+\n \t\tif test $fsm_val = true\n \t\tthen\n \t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n-- \ngitgitgadget\n\n"},{"id":"456090","messageId":"b020bfb45683c1c36b8310b6cfe9d29b8876f358.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 20/30] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:42Z","receivedAt":"2022-05-25T15:02:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to perform binary search over the cache-entries for a directory\nnotification and then linearly scan forward to find the immediate children.\n\nPreviously, when the FSMonitor reported a modified directory Git would\nperform a linear search on the entire cache-entry array for all\nentries matching that directory prefix and invalidate them. Since the\ncache-entry array is already sorted, we can use a binary search to\nfind the first matching entry and then only linearly walk forward and\ninvalidate entries until the prefix changes.\n\nAlso, the original code would invalidate anything having the same\ndirectory prefix. Since a directory event should only be received for\nitems that are immediately within the directory (and not within\nsub-directories of it), only invalidate those entries and not the\nwhole subtree.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 54 insertions(+), 17 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 292a6742b4f..e1229c289cf 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n-\tif (name[len - 1] == '/') {\n+\tint pos = index_name_pos(istate, name, len);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n+\t\t\t name, pos);\n \n+\tif (name[len - 1] == '/') {\n \t\t/*\n-\t\t * TODO We should binary search to find the first path with\n-\t\t * TODO this directory prefix. Then linearly update entries\n-\t\t * TODO while the prefix matches. Taking care to search without\n-\t\t * TODO the trailing slash -- because '/' sorts after a few\n-\t\t * TODO interesting special chars, like '.' and ' '.\n+\t\t * The daemon can decorate directory events, such as\n+\t\t * moves or renames, with a trailing slash if the OS\n+\t\t * FS Event contains sufficient information, such as\n+\t\t * MacOS.\n+\t\t *\n+\t\t * Use this to invalidate the entire cone under that\n+\t\t * directory.\n+\t\t *\n+\t\t * We do not expect an exact match because the index\n+\t\t * does not normally contain directory entries, so we\n+\t\t * start at the insertion point and scan.\n \t\t */\n+\t\tif (pos < 0)\n+\t\t\tpos = -pos - 1;\n \n \t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n-\t\t\t starts_with(istate->cache[i]->name, name))\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n-\t\t/* Need to remove the / from the path for the untracked cache */\n+\n+\t\t/*\n+\t\t * We need to remove the traling \"/\" from the path\n+\t\t * for the untracked cache.\n+\t\t */\n \t\tname[len - 1] = '\\0';\n+\t} else if (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\tint pos = index_name_pos(istate, name, strlen(name));\n-\n-\t\tif (pos >= 0) {\n-\t\t\tstruct cache_entry *ce = istate->cache[pos];\n-\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n \t}\n \n@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n \t * as it could be a new untracked file.\n \t */\n-\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"456091","messageId":"95cf1299d445696630d8c0850a69dc2c0ee040ce.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 19/30] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:41Z","receivedAt":"2022-05-25T15:02:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the listener thread to shutdown the daemon if the spelling of the\nworktree root directory changes.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 07113205a61..83d38e8ac6c 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tstrbuf_release(&msg);\n }\n \n+static int ef_is_root_changed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagRootChanged);\n+}\n+\n static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n {\n \treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n@@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_is_root_changed(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * The spelling of the pathname of the root directory\n+\t\t\t * has changed. This includes the name of the root\n+\t\t\t * directory itself or of any parent directory in the\n+\t\t\t * path.\n+\t\t\t *\n+\t\t\t * (There may be other conditions that throw this,\n+\t\t\t * but I couldn't find any information on it.)\n+\t\t\t *\n+\t\t\t * Force a shutdown now and avoid things getting\n+\t\t\t * out of sync. The Unix domain socket is inside\n+\t\t\t * the .git directory and a spelling change will make\n+\t\t\t * it hard for clients to rendezvous with us.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"event: root changed\");\n+\t\t\tgoto force_shutdown;\n+\t\t}\n+\n \t\tif (ef_ignore_xattr(event_flags[k])) {\n \t\t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n-- \ngitgitgadget\n\n"},{"id":"456092","messageId":"f5dac2868124b57a68a5b7594f02e934c63f7e95.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 22/30] t/perf/p7527: add perf test for builtin FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:44Z","receivedAt":"2022-05-25T15:02:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++\n 1 file changed, 257 insertions(+)\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\ndiff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..9338b9ea008\n--- /dev/null\n+++ b/t/perf/p7527-builtin-fsmonitor.sh\n@@ -0,0 +1,257 @@\n+#!/bin/sh\n+\n+test_description=\"Perf test for the builtin FSMonitor\"\n+\n+. ./perf-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+# Lie to perf-lib and ask for a new empty repo and avoid\n+# the complaints about GIT_PERF_REPO not being big enough\n+# the perf hit when GIT_PERF_LARGE_REPO is copied into\n+# the trash directory.\n+#\n+# NEEDSWORK: It would be nice if perf-lib had an option to\n+# \"borrow\" an existing large repo (especially for gigantic\n+# monorepos) and use it in-place. For now, fake it here.\n+#\n+test_perf_fresh_repo\n+\n+\n+# Use a generated synthetic monorepo. If it doesn't exist, we will\n+# generate it. If it does exist, we will put it in a known state\n+# before we start our timings.\n+#\n+PARAM_D=5\n+PARAM_W=10\n+PARAM_F=9\n+\n+PARAMS=\"$PARAM_D\".\"$PARAM_W\".\"$PARAM_F\"\n+\n+BALLAST_BR=p0006-ballast\n+export BALLAST_BR\n+\n+TMP_BR=tmp_br\n+export TMP_BR\n+\n+REPO=../repos/gen-many-files-\"$PARAMS\".git\n+export REPO\n+\n+if ! test -d $REPO\n+then\n+\t(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)\n+fi\n+\n+\n+enable_uc () {\n+\tgit -C $REPO config core.untrackedcache true\n+\tgit -C $REPO update-index --untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+disable_uc () {\n+\tgit -C $REPO config core.untrackedcache false\n+\tgit -C $REPO update-index --no-untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+start_fsm () {\n+\tgit -C $REPO fsmonitor--daemon start\n+\tgit -C $REPO fsmonitor--daemon status\n+\tgit -C $REPO config core.fsmonitor true\n+\tgit -C $REPO update-index --fsmonitor\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+stop_fsm () {\n+\tgit -C $REPO config --unset core.fsmonitor\n+\tgit -C $REPO update-index --no-fsmonitor\n+\ttest_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+\n+# Ensure that FSMonitor is turned off on the borrowed repo.\n+#\n+test_expect_success \"Setup borrowed repo (fsm+uc)\" \"\n+\tstop_fsm &&\n+\tdisable_uc\n+\"\n+\n+# Also ensure that it starts in a known state.\n+#\n+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time\n+# the ballast checkout, since only the first invocation does any work and the\n+# subsequent ones just print \"already on branch\" and quit, so the reported\n+# time is not useful.\n+#\n+# Create a temp branch and do all work relative to it so that we don't\n+# accidentially alter the real ballast branch.\n+#\n+test_expect_success \"Setup borrowed repo (temp ballast branch)\" \"\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO reset --hard &&\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tgit -C $REPO branch $TMP_BR $BALLAST_BR &&\n+\tgit -C $REPO checkout $TMP_BR\n+\"\n+\n+\n+echo Data >data.txt\n+\n+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With\n+# FSMonitor enabled, we can get a skewed view of status times, since\n+# the index MAY (or may not) be updated after the first invocation\n+# which will update the FSMonitor Token, so the subsequent invocations\n+# may get a smaller response from the daemon.\n+#\n+do_status () {\n+\tmsg=$1\n+\n+\ttest_perf \"$msg\" \"\n+\t\tgit -C $REPO status >/dev/null 2>&1\n+\t\"\n+}\n+\n+do_matrix () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tt=\"[uc $uc][fsm $fsm]\"\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_expect_success \"$t Setup matrix branch\" \"\n+\t\tgit -C $REPO clean -d -f &&\n+\t\tgit -C $REPO checkout $TMP_BR &&\n+\t\ttest_might_fail git -C $REPO branch -D $MATRIX_BR &&\n+\t\tgit -C $REPO branch $MATRIX_BR $TMP_BR &&\n+\t\tgit -C $REPO checkout $MATRIX_BR\n+\t\"\n+\n+\tif test $uc = true\n+\tthen\n+\t\tenable_uc\n+\telse\n+\t\tdisable_uc\n+\tfi\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstart_fsm\n+\telse\n+\t\tstop_fsm\n+\tfi\n+\n+\tdo_status \"$t status after checkout\"\n+\n+\t# Modify many files in the matrix branch.\n+\t# Stage them.\n+\t# Commit them.\n+\t# Rollback.\n+\t#\n+\ttest_expect_success \"$t modify tracked files\" \"\n+\t\tfind $REPO -name file1 -exec cp data.txt {} \\\\;\n+\t\"\n+\n+\tdo_status \"$t status after big change\"\n+\n+\t# Don't bother timing the \"add\" because _REPEAT_COUNT\n+\t# issue described above.\n+\t#\n+\ttest_expect_success \"$t add all\" \"\n+\t\tgit -C $REPO add -A\n+\t\"\n+\n+\tdo_status \"$t status after add all\"\n+\n+\ttest_expect_success \"$t add dot\" \"\n+\t\tgit -C $REPO add .\n+\t\"\n+\n+\tdo_status \"$t status after add dot\"\n+\n+\ttest_expect_success \"$t commit staged\" \"\n+\t\tgit -C $REPO commit -a -m data\n+\t\"\n+\n+\tdo_status \"$t status after commit\"\n+\n+\ttest_expect_success \"$t reset HEAD~1 hard\" \"\n+\t\tgit -C $REPO reset --hard HEAD~1 >/dev/null 2>&1\n+\t\"\n+\n+\tdo_status \"$t status after reset hard\"\n+\n+\t# Create some untracked files.\n+\t#\n+\ttest_expect_success \"$t create untracked files\" \"\n+\t\tcp -R $REPO/ballast/dir1 $REPO/ballast/xxx1\n+\t\"\n+\n+\tdo_status \"$t status after create untracked files\"\n+\n+\t# Remove the new untracked files.\n+\t#\n+\ttest_expect_success \"$t clean -df\" \"\n+\t\tgit -C $REPO clean -d -f\n+\t\"\n+\n+\tdo_status \"$t status after clean\"\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstop_fsm\n+\tfi\n+}\n+\n+# Begin testing each case in the matrix that we care about.\n+#\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\n+fsm_values=\"false true\"\n+\n+for uc_val in $uc_values\n+do\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tdo_matrix $uc_val $fsm_val\n+\tdone\n+done\n+\n+cleanup () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_might_fail git -C $REPO branch -D $MATRIX_BR\n+}\n+\n+\n+# We're borrowing this repo. We should leave it in a clean state.\n+#\n+test_expect_success \"Cleanup temp and matrix branches\" \"\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\tfor fsm_val in $fsm_values\n+\t\tdo\n+\t\t\tcleanup $uc_val $fsm_val\n+\t\tdone\n+\tdone\n+\"\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"456093","messageId":"92f5c0d2c8b1a69aec4325ab35193fe257d481f1.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 23/30] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:45Z","receivedAt":"2022-05-25T15:02:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNever set CE_FSMONITOR_VALID on the cache-entry of submodule\ndirectories.\n\nDuring a client command like 'git status', we may need to recurse\ninto each submodule to compute a status summary for the submodule.\nSince the purpose of the ce_flag is to let Git avoid scanning a\ncache-entry, setting the flag causes the recursive call to be\navoided and we report incorrect (no status) for the submodule.\n\nWe created an OS watch on the root directory of our working\ndirectory and we receive events for everything in the cone\nunder it. When submodules are present inside our working\ndirectory, we receive events for both our repo (the super) and\nany subs within it. Since our index doesn't have any information\nfor items within the submodules, we can't use those events.\n\nWe could try to truncate the paths of those events back to the\nsubmodule boundary and mark the GITLINK as dirty, but that\nfeels expensive since we would have to prefix compare every FS\nevent that we receive against a list of submodule roots. And\nit still wouldn't be sufficient to correctly report status on\nthe submodule, since we don't have any space in the cache-entry\nto cache the submodule's status (the 'SCMU' bits in porcelain\nV2 speak). That is, the CE_FSMONITOR_VALID bit just says that\nwe don't need to scan/inspect it because we already know the\nanswer -- it doesn't say that the item is clean -- and we\ndon't have space in the cache-entry to store those answers.\nSo we should always do the recursive scan.\n\nTherefore, we should never set the flag on GITLINK cache-entries.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 2 +\n fsmonitor.h | 11 ++++\n t/t7527-builtin-fsmonitor.sh | 111 +++++++++++++++++++++++++++++++++++\n 3 files changed, 124 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex e1229c289cf..57d6a483bee 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tif (fsmonitor_enabled) {\n \t\t\t/* Mark all entries valid */\n \t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\t\tif (S_ISGITLINK(istate->cache[i]->ce_mode))\n+\t\t\t\t\tcontinue;\n \t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n \t\t\t}\n \ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 3f41f653691..edf7ce5203b 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n * called any time the cache entry has been updated to reflect the\n * current state of the file on disk.\n+ *\n+ * However, never mark submodules as valid. When commands like \"git\n+ * status\" run they might need to recurse into the submodule (using a\n+ * child process) to get a summary of the submodule state. We don't\n+ * have (and don't want to create) the facility to translate every\n+ * FS event that we receive and that happens to be deep inside of a\n+ * submodule back to the submodule root, so we cannot correctly keep\n+ * track of this bit on the gitlink directory. Therefore, we never\n+ * set it on submodules.\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n \n \tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n \t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tif (S_ISGITLINK(ce->ce_mode))\n+\t\t\treturn;\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 3bc335b891d..cf4fb72c3f0 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -721,4 +721,115 @@ do\n \t'\n done\n \n+# Test fsmonitor interaction with submodules.\n+#\n+# If we start the daemon in the super, it will see FS events for\n+# everything in the working directory cone and this includes any\n+# files/directories contained *within* the submodules.\n+#\n+# A `git status` at top level will get events for items within the\n+# submodule and ignore them, since they aren't named in the index\n+# of the super repo. This makes the fsmonitor response a little\n+# noisy, but it doesn't alter the correctness of the state of the\n+# super-proper.\n+#\n+# When we have submodules, `git status` normally does a recursive\n+# status on each of the submodules and adds a summary row for any\n+# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n+#\n+# It is therefore important that the top level status not be tricked\n+# by the FSMonitor response to skip those recursive calls. That is,\n+# even if FSMonitor says that the mtime of the submodule directory\n+# hasn't changed and it could be implicitly marked valid, we must\n+# not take that shortcut. We need to force the recusion into the\n+# submodule so that we get a summary of the status *within* the\n+# submodule.\n+\n+create_super () {\n+\tsuper=\"$1\" &&\n+\n+\tgit init \"$super\" &&\n+\techo x >\"$super/file_1\" &&\n+\techo y >\"$super/file_2\" &&\n+\techo z >\"$super/file_3\" &&\n+\tmkdir \"$super/dir_1\" &&\n+\techo a >\"$super/dir_1/file_11\" &&\n+\techo b >\"$super/dir_1/file_12\" &&\n+\tmkdir \"$super/dir_1/dir_2\" &&\n+\techo a >\"$super/dir_1/dir_2/file_21\" &&\n+\techo b >\"$super/dir_1/dir_2/file_22\" &&\n+\tgit -C \"$super\" add . &&\n+\tgit -C \"$super\" commit -m \"initial $super commit\"\n+}\n+\n+create_sub () {\n+\tsub=\"$1\" &&\n+\n+\tgit init \"$sub\" &&\n+\techo x >\"$sub/file_x\" &&\n+\techo y >\"$sub/file_y\" &&\n+\techo z >\"$sub/file_z\" &&\n+\tmkdir \"$sub/dir_x\" &&\n+\techo a >\"$sub/dir_x/file_a\" &&\n+\techo b >\"$sub/dir_x/file_b\" &&\n+\tmkdir \"$sub/dir_x/dir_y\" &&\n+\techo a >\"$sub/dir_x/dir_y/file_a\" &&\n+\techo b >\"$sub/dir_x/dir_y/file_b\" &&\n+\tgit -C \"$sub\" add . &&\n+\tgit -C \"$sub\" commit -m \"initial $sub commit\"\n+}\n+\n+my_match_and_clean () {\n+\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n+\tgit -C super --no-optional-locks -c core.fsmonitor=false \\\n+\t\tstatus --porcelain=v2 >actual.without &&\n+\ttest_cmp actual.with actual.without &&\n+\n+\tgit -C super/dir_1/dir_2/sub reset --hard &&\n+\tgit -C super/dir_1/dir_2/sub clean -d -f\n+}\n+\n+test_expect_success 'submodule always visited' '\n+\ttest_when_finished \"git -C super fsmonitor--daemon stop; \\\n+\t\t\t rm -rf super; \\\n+\t\t\t rm -rf sub\" &&\n+\n+\tcreate_super super &&\n+\tcreate_sub sub &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\tstart_daemon -C super &&\n+\tgit -C super config core.fsmonitor true &&\n+\tgit -C super update-index --fsmonitor &&\n+\tgit -C super status &&\n+\n+\t# Now run pairs of commands w/ and w/o FSMonitor while we make\n+\t# some dirt in the submodule and confirm matching output.\n+\n+\t# Completely clean status.\n+\tmy_match_and_clean &&\n+\n+\t# .M S..U\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M SC..\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tgit -C super/dir_1/dir_2/sub commit -m \"SC..\" &&\n+\tmy_match_and_clean\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"456094","messageId":"ea19a06e8cb263a5a33a099c1782fb6d535d571f.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 25/30] fsmonitor: on macOS also emit NFC spelling for NFD pathname","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:47Z","receivedAt":"2022-05-25T15:02:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nEmit NFC or NFC and NFD spellings of pathnames on macOS.\n\nMacOS is Unicode composition insensitive, so NFC and NFD spellings are\ntreated as aliases and collide. While the spelling of pathnames in\nfilesystem events depends upon the underlying filesystem, such as\nAPFS, HFS+ or FAT32, the OS enforces such collisions regardless of\nfilesystem.\n\nTeach the daemon to always report the NFC spelling and to report\nthe NFD spelling when stored in that format on the disk.\n\nThis is slightly more general than \"core.precomposeUnicode\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 83d38e8ac6c..823cf63999e 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n \treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n }\n \n+/*\n+ * On MacOS we have to adjust for Unicode composition insensitivity\n+ * (where NFC and NFD spellings are not respected). The different\n+ * spellings are essentially aliases regardless of how the path is\n+ * actually stored on the disk.\n+ *\n+ * This is related to \"core.precomposeUnicode\" (which wants to try\n+ * to hide NFD completely and treat everything as NFC). Here, we\n+ * don't know what the value the client has (or will have) for this\n+ * config setting when they make a query, so assume the worst and\n+ * emit both when the OS gives us an NFD path.\n+ */\n+static void my_add_path(struct fsmonitor_batch *batch, const char *path)\n+{\n+\tchar *composed;\n+\n+\t/* add the NFC or NFD path as received from the OS */\n+\tfsmonitor_batch__add_path(batch, path);\n+\n+\t/* if NFD, also add the corresponding NFC spelling */\n+\tcomposed = (char *)precompose_string_if_needed(path);\n+\tif (!composed || composed == path)\n+\t\treturn;\n+\n+\tfsmonitor_batch__add_path(batch, composed);\n+\tfree(composed);\n+}\n+\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t\tmy_add_path(batch, rel);\n \t\t\t}\n \n \t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n@@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t\tmy_add_path(batch, tmp.buf);\n \t\t\t}\n \n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"456095","messageId":"40b80adbb31e866afb2b74b681695d3dadffc8df.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 24/30] t7527: test FSMonitor on case insensitive+preserving file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:46Z","receivedAt":"2022-05-25T15:02:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTest that FS events from the OS are received using the preserved,\non-disk spelling of files/directories rather than spelling used\nto make the change.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex cf4fb72c3f0..fbb7d6aef6e 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,4 +832,40 @@ test_expect_success 'submodule always visited' '\n \tmy_match_and_clean\n '\n \n+# On a case-insensitive file system, confirm that the daemon\n+# notices when the .git directory is moved/renamed/deleted\n+# regardless of how it is spelled in the the FS event.\n+# That is, does the FS event receive the spelling of the\n+# operation or does it receive the spelling preserved with\n+# the file/directory.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n+#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n+\n+\tgit init test_insensitive &&\n+\n+\tstart_daemon -C test_insensitive --tf \"$PWD/insensitive.trace\" &&\n+\n+\tmkdir -p test_insensitive/abc/def &&\n+\techo xyz >test_insensitive/ABC/DEF/xyz &&\n+\n+\ttest_path_is_dir test_insensitive/.git &&\n+\ttest_path_is_dir test_insensitive/.GIT &&\n+\n+\t# Rename .git using an alternate spelling to verify that that\n+\t# daemon detects it and automatically shuts down.\n+\tmv test_insensitive/.GIT test_insensitive/.FOO &&\n+\tsleep 1 &&\n+\tmv test_insensitive/.FOO test_insensitive/.git &&\n+\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\t# Verify that events were reported using on-disk spellings of the\n+\t# directories and files that we touched. We may or may not get a\n+\t# trailing slash on modified directories.\n+\t#\n+\tegrep \"^event: abc/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"456096","messageId":"25c6066eddc725c7d4c233620fca82d3638b394e.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 27/30] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:49Z","receivedAt":"2022-05-25T15:02:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a set of prereqs to help understand how file names\nare handled by the filesystem when they contain NFC and NFD\nUnicode characters.\n\nSigned-off-by: Jeff Hostetler \n---\n t/lib-unicode-nfc-nfd.sh | 162 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 162 insertions(+)\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n\ndiff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\nnew file mode 100755\nindex 00000000000..22232247efc\n--- /dev/null\n+++ b/t/lib-unicode-nfc-nfd.sh\n@@ -0,0 +1,162 @@\n+# Help detect how Unicode NFC and NFD are handled on the filesystem.\n+\n+# A simple character that has a NFD form.\n+#\n+# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n+# UTF8(NFC): \\xc3 \\xa9\n+#\n+# NFD: U+0065 LATIN SMALL LETTER E\n+# U+0301 COMBINING ACUTE ACCENT\n+# UTF8(NFD): \\x65 + \\xcc \\x81\n+#\n+utf8_nfc=$(printf \"\\xc3\\xa9\")\n+utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n+\n+# Is the OS or the filesystem \"Unicode composition sensitive\"?\n+#\n+# That is, does the OS or the filesystem allow files to exist with\n+# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n+# tell us that the NFC and NFD forms are equivalent.\n+#\n+# This is or may be independent of what type of filesystem we have,\n+# since it might be handled by the OS at a layer above the FS.\n+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n+# collision, for example.\n+#\n+# This does not tell us how the Unicode pathname will be spelled\n+# on disk, but rather only that the two spelling \"collide\". We\n+# will examine the actual on disk spelling in a later prereq.\n+#\n+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${utf8_nfc} &&\n+\tmkdir trial_${utf8_nfd}\n+'\n+\n+# Is the spelling of an NFC pathname preserved on disk?\n+#\n+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n+# and on APFS, NFC paths are preserved. As we have established\n+# above, this is independent of \"composition sensitivity\".\n+#\n+test_lazy_prereq UNICODE_NFC_PRESERVED '\n+\tmkdir c_${utf8_nfc} &&\n+\tls | test-tool hexdump >dump &&\n+\tgrep \"63 5f c3 a9\" dump\n+'\n+\n+# Is the spelling of an NFD pathname preserved on disk?\n+#\n+test_lazy_prereq UNICODE_NFD_PRESERVED '\n+\tmkdir d_${utf8_nfd} &&\n+\tls | test-tool hexdump >dump &&\n+\tgrep \"64 5f 65 cc 81\" dump\n+'\n+\n+# The following _DOUBLE_ forms are more for my curiosity,\n+# but there may be quirks lurking when there are multiple\n+# combining characters in non-canonical order.\n+\n+# Unicode also allows multiple combining characters\n+# that can be decomposed in pieces.\n+#\n+# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n+# UTF8(NFC): \\xe1 \\xbd \\xa7\n+#\n+# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n+# U+0342 COMBINING GREEK PERISPOMENI\n+# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n+#\n+# But U+1f61 decomposes into\n+# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n+# U+0314 COMBINING REVERSED COMMA ABOVE\n+# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n+#\n+# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n+#\n+# Note that I've used the canonical ordering of the\n+# combinining characters. It is also possible to\n+# swap them. My testing shows that that non-standard\n+# ordering also causes a collision in mkdir. However,\n+# the resulting names don't draw correctly on the\n+# terminal (implying that the on-disk format also has\n+# them out of order).\n+#\n+greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n+greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n+greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n+\n+# See if a double decomposition also collides.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${greek_nfc} &&\n+\tmkdir trial_${greek_nfd2}\n+'\n+\n+# See if the NFC spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n+\tmkdir c_${greek_nfc} &&\n+\tls | test-tool hexdump >dump &&\n+\tgrep \"63 5f e1 bd a7\" dump\n+'\n+\n+# See if the NFD spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n+\tmkdir d_${greek_nfd2} &&\n+\tls | test-tool hexdump >dump &&\n+\tgrep \"64 5f cf 89 cc 94 cd 82\" dump\n+'\n+\n+# The following is for debugging. I found it useful when\n+# trying to understand the various (OS, FS) quirks WRT\n+# Unicode and how composition/decomposition is handled.\n+# For example, when trying to understand how (macOS, APFS)\n+# and (macOS, HFS) and (macOS, FAT32) compare.\n+#\n+# It is rather noisy, so it is disabled by default.\n+#\n+if test \"$unicode_debug\" = \"true\"\n+then\n+\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\techo NFC maintains original spelling.\n+\telse\n+\t\techo NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFD_PRESERVED\n+\tthen\n+\t\techo NFD maintains original spelling.\n+\telse\n+\t\techo NFD is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n+\tthen\n+\t\techo Double NFC maintains original spelling.\n+\telse\n+\t\techo Double NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n+\tthen\n+\t\techo Double NFD maintains original spelling.\n+\telse\n+\t\techo Double NFD is modified.\n+\tfi\n+fi\n-- \ngitgitgadget\n\n"},{"id":"456097","messageId":"66a01db47395a86cce6c0e427779c75d02c37757.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:48Z","receivedAt":"2022-05-25T15:02:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n t/helper/test-hexdump.c | 30 ++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 33 insertions(+)\n create mode 100644 t/helper/test-hexdump.c\n\ndiff --git a/Makefile b/Makefile\nindex 5f1623baadd..5afa194aac6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -729,6 +729,7 @@ TEST_BUILTINS_OBJS += test-getcwd.o\n TEST_BUILTINS_OBJS += test-hash-speed.o\n TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n+TEST_BUILTINS_OBJS += test-hexdump.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\n TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\ndiff --git a/t/helper/test-hexdump.c b/t/helper/test-hexdump.c\nnew file mode 100644\nindex 00000000000..811e89c1bcb\n--- /dev/null\n+++ b/t/helper/test-hexdump.c\n@@ -0,0 +1,30 @@\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Read stdin and print a hexdump to stdout.\n+ */\n+int cmd__hexdump(int argc, const char **argv)\n+{\n+\tchar buf[1024];\n+\tssize_t i, len;\n+\tint have_data = 0;\n+\n+\tfor (;;) {\n+\t\tlen = xread(0, buf, sizeof(buf));\n+\t\tif (len < 0)\n+\t\t\tdie_errno(\"failure reading stdin\");\n+\t\tif (!len)\n+\t\t\tbreak;\n+\n+\t\thave_data = 1;\n+\n+\t\tfor (i = 0; i < len; i++)\n+\t\t\tprintf(\"%02x \", (unsigned char)buf[i]);\n+\t}\n+\n+\tif (have_data)\n+\t\tputchar('\\n');\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 0424f7adf5d..88c4b28cdfa 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -38,6 +38,7 @@ static struct test_cmd cmds[] = {\n \t{ \"getcwd\", cmd__getcwd },\n \t{ \"hashmap\", cmd__hashmap },\n \t{ \"hash-speed\", cmd__hash_speed },\n+\t{ \"hexdump\", cmd__hexdump },\n \t{ \"index-version\", cmd__index_version },\n \t{ \"json-writer\", cmd__json_writer },\n \t{ \"lazy-init-name-hash\", cmd__lazy_init_name_hash },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex c876e8246fb..511f6251bf5 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -29,6 +29,7 @@ int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n int cmd__hash_speed(int argc, const char **argv);\n+int cmd__hexdump(int argc, const char **argv);\n int cmd__index_version(int argc, const char **argv);\n int cmd__json_writer(int argc, const char **argv);\n int cmd__lazy_init_name_hash(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"456098","messageId":"fc3a0e7847ff62a30d71fd48817740651e0bdbe0.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 28/30] t7527: test Unicode NFC/NFD handling on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:50Z","receivedAt":"2022-05-25T15:02:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that the daemon reports events using the on-disk\nspelling for Unicode NFC/NFD characters. On APFS we still\nhave Unicode aliasing, so we cannot create two files that\nonly differ by NFC/NFD, but the on-disk format preserves\nthe spelling used to create the file. On HFS+ we also\nhave aliasing, but the path is always stored on disk in\nNFD.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 55 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex fbb7d6aef6e..9edae3ed830 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -868,4 +868,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n+# The variable \"unicode_debug\" is defined in the following library\n+# script to dump information about how the (OS, FS) handles Unicode\n+# composition. Uncomment the following line if you want to enable it.\n+#\n+# unicode_debug=true\n+\n+. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n+\n+# See if the OS or filesystem does NFC/NFD aliasing/munging.\n+#\n+# The daemon should err on the side of caution and send BOTH the\n+# NFC and NFD forms. It does not know the original spelling of\n+# the pathname (how the user thinks it should be spelled), so\n+# emit both and let the client decide (when necessary). This is\n+# similar to \"core.precomposeUnicode\".\n+#\n+test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n+\n+\tgit init test_unicode &&\n+\n+\tstart_daemon -C test_unicode --tf \"$PWD/unicode.trace\" &&\n+\n+\t# Create a directory using an NFC spelling.\n+\t#\n+\tmkdir test_unicode/nfc &&\n+\tmkdir test_unicode/nfc/c_${utf8_nfc} &&\n+\n+\t# Create a directory using an NFD spelling.\n+\t#\n+\tmkdir test_unicode/nfd &&\n+\tmkdir test_unicode/nfd/d_${utf8_nfd} &&\n+\n+\tgit -C test_unicode fsmonitor--daemon stop &&\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\t# We should have seen NFC event from OS.\n+\t\t# We should not have synthesized an NFD event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace &&\n+\t\tegrep -v \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace\n+\telse\n+\t\t# We should have seen NFD event from OS.\n+\t\t# We should have synthesized an NFC event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace &&\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace\n+\tfi &&\n+\n+\t# We assume UNICODE_NFD_PRESERVED.\n+\t# We should have seen explicit NFD from OS.\n+\t# We should have synthesized an NFC event.\n+\tegrep \"^event: nfd/d_${utf8_nfd}/?$\" ./unicode.trace &&\n+\tegrep \"^event: nfd/d_${utf8_nfc}/?$\" ./unicode.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"456099","messageId":"d70d2545a5a97efa226a7796e317c4447e81bf96.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 30/30] t7527: improve implicit shutdown testing in fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:52Z","receivedAt":"2022-05-25T15:02:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor the tests that exercise implicit shutdown cases\nto make them more robust and less racy.\n\nThe fsmonitor--daemon will implicitly shutdown in a variety\nof situations, such as when the \".git\" directory is deleted\nor renamed.\n\nThe existing tests would delete or rename the directory, sleep\nfor one second, and then check the status of the daemon. This\nis racy, since the client/status command has no way to sync\nwith the daemon. This was noticed occasionally on very slow\nCI build machines where it would cause a random test to fail.\n\nReplace the simple sleep with a sleep-and-retry loop.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 54 ++++++++++++++++++++++++++----------\n 1 file changed, 40 insertions(+), 14 deletions(-)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 19edc96fd4d..56c0dfffea6 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -124,6 +124,36 @@ test_expect_success 'implicit daemon start' '\n \ttest_must_fail git -C test_implicit fsmonitor--daemon status\n '\n \n+# Verify that the daemon has shutdown. Spin a few seconds to\n+# make the test a little more robust during CI testing.\n+#\n+# We're looking for an implicit shutdown, such as when we delete or\n+# rename the \".git\" directory. Our delete/rename will cause a file\n+# system event that the daemon will see and the daemon will\n+# auto-shutdown as soon as it sees it. But this is racy with our `git\n+# fsmonitor--daemon status` commands (and we cannot use a cookie file\n+# here to help us). So spin a little and give the daemon a chance to\n+# see the event. (This is primarily for underpowered CI build/test\n+# machines (where it might take a moment to wake and reschedule the\n+# daemon process) to avoid false alarms during test runs.)\n+#\n+IMPLICIT_TIMEOUT=5\n+\n+verify_implicit_shutdown () {\n+\tr=$1 &&\n+\n+\tk=0 &&\n+\twhile test \"$k\" -lt $IMPLICIT_TIMEOUT\n+\tdo\n+\t\tgit -C $r fsmonitor--daemon status || return 0\n+\n+\t\tsleep 1\n+\t\tk=$(( $k + 1 ))\n+\tdone &&\n+\n+\treturn 1\n+}\n+\n test_expect_success 'implicit daemon stop (delete .git)' '\n \ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n \n@@ -142,10 +172,9 @@ test_expect_success 'implicit daemon stop (delete .git)' '\n \t# This would make the test result dependent upon whether we\n \t# were using fsmonitor on our development worktree.\n \t#\n-\tsleep 1 &&\n \tmkdir test_implicit_1/.git &&\n \n-\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_1\n '\n \n test_expect_success 'implicit daemon stop (rename .git)' '\n@@ -160,10 +189,9 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \n \t# See [1] above.\n \t#\n-\tsleep 1 &&\n \tmkdir test_implicit_2/.git &&\n \n-\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_2\n '\n \n # File systems on Windows may or may not have shortnames.\n@@ -194,13 +222,11 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n \t#\n \tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n \n-\tsleep 1 &&\n-\t# put it back so that our status will not crawl out to our\n-\t# parent directory.\n+\t# See [1] above.\n \t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n \tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n \n-\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_1s\n '\n \n # Here we first create a file with LONGNAME of \"GIT~1\" before\n@@ -223,12 +249,10 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n \t#\n \tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n \n-\tsleep 1 &&\n-\t# put it back so that our status will not crawl out to our\n-\t# parent directory.\n+\t# See [1] above.\n \tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n \n-\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_1s2\n '\n \n test_expect_success 'cannot start multiple daemons' '\n@@ -905,9 +929,11 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \t# Rename .git using an alternate spelling to verify that that\n \t# daemon detects it and automatically shuts down.\n \tmv test_insensitive/.GIT test_insensitive/.FOO &&\n-\tsleep 1 &&\n+\n+\t# See [1] above.\n \tmv test_insensitive/.FOO test_insensitive/.git &&\n-\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\tverify_implicit_shutdown test_insensitive &&\n \n \t# Verify that events were reported using on-disk spellings of the\n \t# directories and files that we touched. We may or may not get a\n-- \ngitgitgadget\n"},{"id":"456100","messageId":"25676ca4ec2bdaa6fdaf265dcdfe39b06d4d7bfe.1653490853.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v8 29/30] fsmonitor--daemon: allow --super-prefix argument","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-25T15:00:51Z","receivedAt":"2022-05-25T15:02:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a test in t7527 to verify that we get a stray warning from\n`git fsmonitor--daemon start` when indirectly called from\n`git submodule absorbgitdirs`.\n\nUpdate `git fsmonitor--daemon` to take (and ignore) the `--super-prefix`\nargument to suppress the warning.\n\nWhen we have:\n\n1. a submodule with a `sub/.git/` directory (rather than a `sub/.git`\nfile).\n\n2. `core.fsmonitor` is turned on in the submodule, but the daemon is\nnot yet started in the submodule.\n\n3. and someone does a `git submodule absorbgitdirs` in the super.\n\nGit will recursively invoke `git submodule--helper absorb-git-dirs`\nin the submodule. This will read the index and may attempt to start\nthe fsmonitor--daemon with the `--super-prefix` argument.\n\n`git fsmonitor--daemon start` does not accept the `--super-prefix`\nargument and causes a warning to be issued.\n\nThis does not cause a problem because the `refresh_index()` code\nassumes a trivial response if the daemon does not start.\n\nThe net-net is a harmelss, but stray warning. Lets eliminate the\nwarning.\n\nSigned-off-by: Jeff Hostetler \n---\n git.c | 2 +-\n t/t7527-builtin-fsmonitor.sh | 50 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 51 insertions(+), 1 deletion(-)\n\ndiff --git a/git.c b/git.c\nindex 3d8e48cf555..e6fdac1f8e3 100644\n--- a/git.c\n+++ b/git.c\n@@ -537,7 +537,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n-\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, SUPPORT_SUPER_PREFIX | RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 9edae3ed830..19edc96fd4d 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,6 +832,56 @@ test_expect_success 'submodule always visited' '\n \tmy_match_and_clean\n '\n \n+# If a submodule has a `sub/.git/` directory (rather than a file\n+# pointing to the super's `.git/modules/sub`) and `core.fsmonitor`\n+# turned on in the submodule and the daemon is not yet started in\n+# the submodule, and someone does a `git submodule absorbgitdirs`\n+# in the super, Git will recursively invoke `git submodule--helper`\n+# to do the work and this may try to read the index. This will\n+# try to start the daemon in the submodule *and* pass (either\n+# directly or via inheritance) the `--super-prefix` arg to the\n+# `git fsmonitor--daemon start` command inside the submodule.\n+# This causes a warning because fsmonitor--daemon does take that\n+# global arg (see the table in git.c)\n+#\n+# This causes a warning when trying to start the daemon that is\n+# somewhat confusing. It does not seem to hurt anything because\n+# the fsmonitor code maps the query failure into a trivial response\n+# and does the work anyway.\n+#\n+# It would be nice to silence the warning, however.\n+\n+have_t2_error_event () {\n+\tlog=$1\n+\tmsg=\"fsmonitor--daemon doesnQt support --super-prefix\" &&\n+\n+\ttr '\\047' Q <$1 | grep -e \"$msg\"\n+}\n+\n+test_expect_success \"stray submodule super-prefix warning\" '\n+\ttest_when_finished \"rm -rf super; \\\n+\t\t\t rm -rf sub; \\\n+\t\t\t rm super-sub.trace\" &&\n+\n+\tcreate_super super &&\n+\tcreate_sub sub &&\n+\n+\t# Copy rather than submodule add so that we get a .git dir.\n+\tcp -R ./sub ./super/dir_1/dir_2/sub &&\n+\n+\tgit -C super/dir_1/dir_2/sub config core.fsmonitor true &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\ttest_path_is_dir super/dir_1/dir_2/sub/.git &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/super-sub.trace\" \\\n+\t\tgit -C super submodule absorbgitdirs &&\n+\n+\t! have_t2_error_event super-sub.trace\n+'\n+\n # On a case-insensitive file system, confirm that the daemon\n # notices when the .git directory is moved/renamed/deleted\n # regardless of how it is spelled in the the FS event.\n-- \ngitgitgadget\n\n"},{"id":"456106","messageId":"xmqqy1yp8ta2.fsf@gitster.g","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 00/30] Builtin FSMonitor Part 3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-25T16:35:17Z","receivedAt":"2022-05-25T16:35:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> This version addresses the new t/helper/test-hexdump utility to emit an LF\n> at the end. I also updated the test scripts to write hexdump output to a\n> file and then grep that file to avoid having the tool being in the middle of\n> a pipeline.\n\nhttps://github.com/git/git/runs/6583917870 shows t7527 is not happy\nin \"win test (9)\" job with the previous version. It is unclear if\nthat has been addressed, but we'll see soon.\n\nWill replace. Thanks.\n"},{"id":"456110","messageId":"5224e941-af0d-d478-00f0-cf6639e3a726@jeffhostetler.com","threadId":"57414","inReplyTo":"xmqqy1yp8ta2.fsf@gitster.g","subject":"Re: [PATCH v8 00/30] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-05-25T17:29:05Z","receivedAt":"2022-05-25T17:29:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/25/22 12:35 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> This version addresses the new t/helper/test-hexdump utility to emit an LF\n>> at the end. I also updated the test scripts to write hexdump output to a\n>> file and then grep that file to avoid having the tool being in the middle of\n>> a pipeline.\n> \n> https://github.com/git/git/runs/6583917870 shows t7527 is not happy\n> in \"win test (9)\" job with the previous version. It is unclear if\n> that has been addressed, but we'll see soon.\n> \n> Will replace. Thanks.\n> \n\nI'll investigate. Thanks.\n\nAn earlier version had a \"mv .git foo; sleep 1; mv foo .git\"\nand then tested to see if the daemon had noticed the change\nand automatically shutting down. That was a little racy on\nvery slow build machines (well, that was the only place that\nit was seen). So added `verify_implicit_shutdown()` to try\nto make it less sensitive. Looks like it might need a little\nmore help.\n\nJeff\n"},{"id":"456234","messageId":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v8.git.1653490852.gitgitgadget@gmail.com","subject":"[PATCH v9 00/30] Builtin FSMonitor Part 3","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:46:54Z","receivedAt":"2022-05-26T21:49:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 9 of part 3 of FSMonitor.\n\nThis version addresses the test failure on t7527.[56] when deleting/renaming\nthe .git directory using the \"GIT~1\" spelling. Between v6 and v7, I changed\nthe way check_for_shortnames() constructed the wchar_t path to .git from a\npair of wcscpy() calls to a swprintf() call. However, I used a \"%s\" by\nmistake rather than a \"%ls\".\n\nThis caused a non-portable behavior. The tests passed with MSVC and with GCC\n10.1.0 on my laptop, but failed under GCC on one of the CI build machines.\n\nThis was partially hidden by CI machines that have GCC 12 and that fail to\ncompile Git without the fixes for GCC 12.x.\n\nJeff Hostetler (30):\n fsm-listen-win32: handle shortnames\n t7527: test FSMonitor on repos with Unicode root paths\n t/helper/fsmonitor-client: create stress test\n fsmonitor-settings: bare repos are incompatible with FSMonitor\n fsmonitor-settings: stub in Win32-specific incompatibility checking\n fsmonitor-settings: VFS for Git virtual repos are incompatible\n fsmonitor-settings: stub in macOS-specific incompatibility checking\n fsmonitor-settings: remote repos on macOS are incompatible\n fsmonitor-settings: remote repos on Windows are incompatible\n fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n unpack-trees: initialize fsmonitor_has_run_once in o->result\n fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n fsmonitor--daemon: cd out of worktree root\n fsmonitor--daemon: prepare for adding health thread\n fsmonitor--daemon: rename listener thread related variables\n fsmonitor--daemon: stub in health thread\n fsm-health-win32: add polling framework to monitor daemon health\n fsm-health-win32: force shutdown daemon if worktree root moves\n fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n fsmonitor: optimize processing of directory events\n t7527: FSMonitor tests for directory moves\n t/perf/p7527: add perf test for builtin FSMonitor\n fsmonitor: never set CE_FSMONITOR_VALID on submodules\n t7527: test FSMonitor on case insensitive+preserving file system\n fsmonitor: on macOS also emit NFC spelling for NFD pathname\n t/helper/hexdump: add helper to print hexdump of stdin\n t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n t7527: test Unicode NFC/NFD handling on MacOS\n fsmonitor--daemon: allow --super-prefix argument\n t7527: improve implicit shutdown testing in fsmonitor--daemon\n\n Makefile | 20 +-\n builtin/fsmonitor--daemon.c | 116 ++++++-\n builtin/update-index.c | 16 +\n compat/fsmonitor/fsm-health-darwin.c | 24 ++\n compat/fsmonitor/fsm-health-win32.c | 278 ++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 +++\n compat/fsmonitor/fsm-listen-darwin.c | 122 ++++++-\n compat/fsmonitor/fsm-listen-win32.c | 436 ++++++++++++++++++++-----\n compat/fsmonitor/fsm-listen.h | 2 +-\n compat/fsmonitor/fsm-settings-darwin.c | 89 +++++\n compat/fsmonitor/fsm-settings-win32.c | 137 ++++++++\n config.mak.uname | 5 +\n contrib/buildsystems/CMakeLists.txt | 8 +\n fsmonitor--daemon.h | 11 +-\n fsmonitor-settings.c | 167 ++++++++--\n fsmonitor-settings.h | 33 ++\n fsmonitor.c | 73 ++++-\n fsmonitor.h | 11 +\n git.c | 2 +-\n t/helper/test-fsmonitor-client.c | 106 ++++++\n t/helper/test-hexdump.c | 30 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/lib-unicode-nfc-nfd.sh | 162 +++++++++\n t/perf/p7527-builtin-fsmonitor.sh | 257 +++++++++++++++\n t/t7519-status-fsmonitor.sh | 32 ++\n t/t7527-builtin-fsmonitor.sh | 401 ++++++++++++++++++++++-\n unpack-trees.c | 1 +\n 28 files changed, 2439 insertions(+), 149 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n create mode 100644 t/helper/test-hexdump.c\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\n\nbase-commit: 5eb696daba2fe108d4d9ba2ccf4b357447ef9946\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v9\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v9\nPull-Request: https://github.com/gitgitgadget/git/pull/1143\n\nRange-diff vs v8:\n\n 1: 26144c58659 ! 1: 9353e5863dc fsm-listen-win32: handle shortnames\n @@ compat/fsmonitor/fsm-listen-win32.c: normalize:\n +\twchar_t *p;\n +\n +\t/* build L\"/.git\" */\n -+\tswprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%s.git\",\n ++\tswprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%ls.git\",\n +\t\t watch->wpath_longname);\n +\n +\tif (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)))\n 2: 1bf2e36b6ad = 2: 2ae8a02ebe0 t7527: test FSMonitor on repos with Unicode root paths\n 3: 4bca494bb22 = 3: 6f8baf5b723 t/helper/fsmonitor-client: create stress test\n 4: 663deabc3f6 = 4: 195f90cc9ef fsmonitor-settings: bare repos are incompatible with FSMonitor\n 5: 7cb0180a1ed = 5: 0b182569e11 fsmonitor-settings: stub in Win32-specific incompatibility checking\n 6: 9774faddc45 = 6: 06d49b0f230 fsmonitor-settings: VFS for Git virtual repos are incompatible\n 7: f7ef7dcffc8 = 7: 2b6e8b0b241 fsmonitor-settings: stub in macOS-specific incompatibility checking\n 8: dc2dfd67931 = 8: 8068b3bebee fsmonitor-settings: remote repos on macOS are incompatible\n 9: 5627038aaa3 = 9: a5c40a4464d fsmonitor-settings: remote repos on Windows are incompatible\n 10: 9a12cc78b5d = 10: d2569ed3211 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible\n 11: aaff000cecb = 11: 27f360b3336 unpack-trees: initialize fsmonitor_has_run_once in o->result\n 12: 4f2b15d3d1f = 12: ad8f65efe0d fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS\n 13: 427dec412a5 = 13: 7e302958cd4 fsmonitor--daemon: cd out of worktree root\n 14: 51b266b06e1 = 14: 6b271866f67 fsmonitor--daemon: prepare for adding health thread\n 15: 594e0ae243d = 15: df1061c0ff5 fsmonitor--daemon: rename listener thread related variables\n 16: c2b5c02ed38 = 16: e255a5b7104 fsmonitor--daemon: stub in health thread\n 17: 46a5ae2a635 = 17: f710d305dd4 fsm-health-win32: add polling framework to monitor daemon health\n 18: 7cf1be5f8e2 = 18: b3f5a945d3e fsm-health-win32: force shutdown daemon if worktree root moves\n 19: 95cf1299d44 = 19: d8949ab5df6 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed\n 20: b020bfb4568 = 20: 8fa7f4fa9be fsmonitor: optimize processing of directory events\n 21: d058d7e0c08 = 21: 6976f1d45ea t7527: FSMonitor tests for directory moves\n 22: f5dac286812 = 22: 81559df45ab t/perf/p7527: add perf test for builtin FSMonitor\n 23: 92f5c0d2c8b = 23: d0c8fecd1a0 fsmonitor: never set CE_FSMONITOR_VALID on submodules\n 24: 40b80adbb31 = 24: 41f8cbc2ae4 t7527: test FSMonitor on case insensitive+preserving file system\n 25: ea19a06e8cb = 25: c8c4c22360c fsmonitor: on macOS also emit NFC spelling for NFD pathname\n 26: 66a01db4739 = 26: 1612dcba503 t/helper/hexdump: add helper to print hexdump of stdin\n 27: 25c6066eddc = 27: e5a7f05d9d4 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd\n 28: fc3a0e7847f = 28: b27dc48a650 t7527: test Unicode NFC/NFD handling on MacOS\n 29: 25676ca4ec2 = 29: 2905b3bb59e fsmonitor--daemon: allow --super-prefix argument\n 30: d70d2545a5a = 30: 2f0dea304f0 t7527: improve implicit shutdown testing in fsmonitor--daemon\n\n-- \ngitgitgadget\n"},{"id":"456235","messageId":"9353e5863dc758bba4e23825b67e462373246a42.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 01/30] fsm-listen-win32: handle shortnames","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:46:55Z","receivedAt":"2022-05-26T21:49:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach FSMonitor daemon on Windows to recognize shortname paths as\naliases of normal longname paths. FSMonitor clients, such as `git\nstatus`, should receive the longname spelling of changed files (when\npossible).\n\nSometimes we receive FS events using the shortname, such as when a CMD\nshell runs \"RENAME GIT~1 FOO\" or \"RMDIR GIT~1\". The FS notification\narrives using whatever combination of long and shortnames were used by\nthe other process. (Shortnames do seem to be case normalized,\nhowever.)\n\nUse Windows GetLongPathNameW() to try to map the pathname spelling in\nthe notification event into the normalized longname spelling. (This\ncan fail if the file/directory is deleted, moved, or renamed, because\nwe are asking the FS for the mapping in response to the event and\nafter it has already happened, but we try.)\n\nSpecial case the shortname spelling of \".git\" to avoid under-reporting\nthese events.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-win32.c | 386 ++++++++++++++++++++++++----\n t/t7527-builtin-fsmonitor.sh | 65 +++++\n 2 files changed, 397 insertions(+), 54 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 5b928ab66e5..4f46bd1d0a6 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -25,6 +25,9 @@ struct one_watch\n \tDWORD count;\n \n \tstruct strbuf path;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n+\tDWORD wpath_longname_len;\n+\n \tHANDLE hDir;\n \tHANDLE hEvent;\n \tOVERLAPPED overlapped;\n@@ -34,6 +37,21 @@ struct one_watch\n \t * need to later call GetOverlappedResult() and possibly CancelIoEx().\n \t */\n \tBOOL is_active;\n+\n+\t/*\n+\t * Are shortnames enabled on the containing drive? This is\n+\t * always true for \"C:/\" drives and usually never true for\n+\t * other drives.\n+\t *\n+\t * We only set this for the worktree because we only need to\n+\t * convert shortname paths to longname paths for items we send\n+\t * to clients. (We don't care about shortname expansion for\n+\t * paths inside a GITDIR because we never send them to\n+\t * clients.)\n+\t */\n+\tBOOL has_shortnames;\n+\tBOOL has_tilde;\n+\twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n struct fsmonitor_daemon_backend_data\n@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data\n };\n \n /*\n- * Convert the WCHAR path from the notification into UTF8 and\n- * then normalize it.\n+ * Convert the WCHAR path from the event into UTF8 and normalize it.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n */\n-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,\n \t\t\t\t struct strbuf *normalized_path)\n {\n \tint reserve;\n \tint len = 0;\n \n \tstrbuf_reset(normalized_path);\n-\tif (!info->FileNameLength)\n+\tif (!wpath_len)\n \t\tgoto normalize;\n \n \t/*\n@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t * sequence of 2 UTF8 characters. That should let us\n \t * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.\n \t */\n-\treserve = info->FileNameLength + 1;\n+\treserve = 2 * wpath_len + 1;\n \tstrbuf_grow(normalized_path, reserve);\n \n \tfor (;;) {\n-\t\tlen = WideCharToMultiByte(CP_UTF8, 0, info->FileName,\n-\t\t\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\tlen = WideCharToMultiByte(CP_UTF8, 0,\n+\t\t\t\t\t wpath, wpath_len,\n \t\t\t\t\t normalized_path->buf,\n \t\t\t\t\t strbuf_avail(normalized_path) - 1,\n \t\t\t\t\t NULL, NULL);\n@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,\n \t\t\tgoto normalize;\n \t\tif (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {\n \t\t\terror(_(\"[GLE %ld] could not convert path to UTF-8: '%.*ls'\"),\n-\t\t\t GetLastError(),\n-\t\t\t (int)(info->FileNameLength / sizeof(WCHAR)),\n-\t\t\t info->FileName);\n+\t\t\t GetLastError(), (int)wpath_len, wpath);\n \t\t\treturn -1;\n \t\t}\n \n@@ -98,6 +115,173 @@ normalize:\n \treturn strbuf_normalize_path(normalized_path);\n }\n \n+/*\n+ * See if the worktree root directory has shortnames enabled.\n+ * This will help us decide if we need to do an expensive shortname\n+ * to longname conversion on every notification event.\n+ *\n+ * We do not want to create a file to test this, so we assume that the\n+ * root directory contains a \".git\" file or directory. (Our caller\n+ * only calls us for the worktree root, so this should be fine.)\n+ *\n+ * Remember the spelling of the shortname for \".git\" if it exists.\n+ */\n+static void check_for_shortnames(struct one_watch *watch)\n+{\n+\twchar_t buf_in[MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\twchar_t *last;\n+\twchar_t *p;\n+\n+\t/* build L\"/.git\" */\n+\tswprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L\"%ls.git\",\n+\t\t watch->wpath_longname);\n+\n+\tif (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)))\n+\t\treturn;\n+\n+\t/*\n+\t * Get the final filename component of the shortpath.\n+\t * We know that the path does not have a final slash.\n+\t */\n+\tfor (last = p = buf_out; *p; p++)\n+\t\tif (*p == L'/' || *p == '\\\\')\n+\t\t\tlast = p + 1;\n+\n+\tif (!wcscmp(last, L\".git\"))\n+\t\treturn;\n+\n+\twatch->has_shortnames = 1;\n+\twcsncpy(watch->dotgit_shortname, last,\n+\t\tARRAY_SIZE(watch->dotgit_shortname));\n+\n+\t/*\n+\t * The shortname for \".git\" is usually of the form \"GIT~1\", so\n+\t * we should be able to avoid shortname to longname mapping on\n+\t * every notification event if the source string does not\n+\t * contain a \"~\".\n+\t *\n+\t * However, the documentation for GetLongPathNameW() says\n+\t * that there are filesystems that don't follow that pattern\n+\t * and warns against this optimization.\n+\t *\n+\t * Lets test this.\n+\t */\n+\tif (wcschr(watch->dotgit_shortname, L'~'))\n+\t\twatch->has_tilde = 1;\n+}\n+\n+enum get_relative_result {\n+\tGRR_NO_CONVERSION_NEEDED,\n+\tGRR_HAVE_CONVERSION,\n+\tGRR_SHUTDOWN,\n+};\n+\n+/*\n+ * Info notification paths are relative to the root of the watch.\n+ * If our CWD is still at the root, then we can use relative paths\n+ * to convert from shortnames to longnames. If our process has a\n+ * different CWD, then we need to construct an absolute path, do\n+ * the conversion, and then return the root-relative portion.\n+ *\n+ * We use the longname form of the root as our basis and assume that\n+ * it already has a trailing slash.\n+ *\n+ * `wpath_len` is in WCHARS not bytes.\n+ */\n+static enum get_relative_result get_relative_longname(\n+\tstruct one_watch *watch,\n+\tconst wchar_t *wpath, DWORD wpath_len,\n+\twchar_t *wpath_longname, size_t bufsize_wpath_longname)\n+{\n+\twchar_t buf_in[2 * MAX_PATH + 1];\n+\twchar_t buf_out[MAX_PATH + 1];\n+\tDWORD root_len;\n+\tDWORD out_len;\n+\n+\t/*\n+\t * Build L\"/\"\n+\t * Note that the might not be null terminated\n+\t * so we avoid swprintf() constructions.\n+\t */\n+\troot_len = watch->wpath_longname_len;\n+\tif (root_len + wpath_len >= ARRAY_SIZE(buf_in)) {\n+\t\t/*\n+\t\t * This should not happen. We cannot append the observed\n+\t\t * relative path onto the end of the worktree root path\n+\t\t * without overflowing the buffer. Just give up.\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\twcsncpy(buf_in, watch->wpath_longname, root_len);\n+\twcsncpy(buf_in + root_len, wpath, wpath_len);\n+\tbuf_in[root_len + wpath_len] = 0;\n+\n+\t/*\n+\t * We don't actually know if the source pathname is a\n+\t * shortname or a longname. This Windows routine allows\n+\t * either to be given as input.\n+\t */\n+\tout_len = GetLongPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out));\n+\tif (!out_len) {\n+\t\t/*\n+\t\t * The shortname to longname conversion can fail for\n+\t\t * various reasons, for example if the file has been\n+\t\t * deleted. (That is, if we just received a\n+\t\t * delete-file notification event and the file is\n+\t\t * already gone, we can't ask the file system to\n+\t\t * lookup the longname for it. Likewise, for moves\n+\t\t * and renames where we are given the old name.)\n+\t\t *\n+\t\t * Since deleting or moving a file or directory by its\n+\t\t * shortname is rather obscure, I'm going ignore the\n+\t\t * failure and ask the caller to report the original\n+\t\t * relative path. This seems kinder than failing here\n+\t\t * and forcing a resync. Besides, forcing a resync on\n+\t\t * every file/directory delete would effectively\n+\t\t * cripple monitoring.\n+\t\t *\n+\t\t * We might revisit this in the future.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (!wcscmp(buf_in, buf_out)) {\n+\t\t/*\n+\t\t * The path does not have a shortname alias.\n+\t\t */\n+\t\treturn GRR_NO_CONVERSION_NEEDED;\n+\t}\n+\n+\tif (wcsncmp(buf_in, buf_out, root_len)) {\n+\t\t/*\n+\t\t * The spelling of the root directory portion of the computed\n+\t\t * longname has changed. This should not happen. Basically,\n+\t\t * it means that we don't know where (without recomputing the\n+\t\t * longname of just the root directory) to split out the\n+\t\t * relative path. Since this should not happen, I'm just\n+\t\t * going to let this fail and force a shutdown (because all\n+\t\t * subsequent events are probably going to see the same\n+\t\t * mismatch).\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\tif (out_len - root_len >= bufsize_wpath_longname) {\n+\t\t/*\n+\t\t * This should not happen. We cannot copy the root-relative\n+\t\t * portion of the path into the provided buffer without an\n+\t\t * overrun. Just give up.\n+\t\t */\n+\t\treturn GRR_SHUTDOWN;\n+\t}\n+\n+\t/* Return the worktree root-relative portion of the longname. */\n+\n+\twcscpy(wpath_longname, buf_out + root_len);\n+\treturn GRR_HAVE_CONVERSION;\n+}\n+\n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n \tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n@@ -111,7 +295,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tDWORD share_mode =\n \t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n \tHANDLE hDir;\n-\twchar_t wpath[MAX_PATH];\n+\tDWORD len_longname;\n+\twchar_t wpath[MAX_PATH + 1];\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \tif (xutftowcs_path(wpath, path) < 0) {\n \t\terror(_(\"could not convert to wide characters: '%s'\"), path);\n@@ -128,6 +314,21 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \t\treturn NULL;\n \t}\n \n+\tlen_longname = GetLongPathNameW(wpath, wpath_longname,\n+\t\t\t\t\tARRAY_SIZE(wpath_longname));\n+\tif (!len_longname) {\n+\t\terror(_(\"[GLE %ld] could not get longname of '%s'\"),\n+\t\t GetLastError(), path);\n+\t\tCloseHandle(hDir);\n+\t\treturn NULL;\n+\t}\n+\n+\tif (wpath_longname[len_longname - 1] != L'/' &&\n+\t wpath_longname[len_longname - 1] != L'\\\\') {\n+\t\twpath_longname[len_longname++] = L'/';\n+\t\twpath_longname[len_longname] = 0;\n+\t}\n+\n \tCALLOC_ARRAY(watch, 1);\n \n \twatch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */\n@@ -135,6 +336,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n \tstrbuf_init(&watch->path, 0);\n \tstrbuf_addstr(&watch->path, path);\n \n+\twcscpy(watch->wpath_longname, wpath_longname);\n+\twatch->wpath_longname_len = len_longname;\n+\n \twatch->hDir = hDir;\n \twatch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);\n \n@@ -258,6 +462,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)\n \twatch->is_active = FALSE;\n }\n \n+/*\n+ * Process a single relative pathname event.\n+ * Return 1 if we should shutdown.\n+ */\n+static int process_1_worktree_event(\n+\tstruct string_list *cookie_list,\n+\tstruct fsmonitor_batch **batch,\n+\tconst struct strbuf *path,\n+\tenum fsmonitor_path_type t,\n+\tDWORD info_action)\n+{\n+\tconst char *slash;\n+\n+\tswitch (t) {\n+\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n+\t\t/* special case cookie files within .git */\n+\n+\t\t/* Use just the filename of the cookie file. */\n+\t\tslash = find_last_dir_sep(path->buf);\n+\t\tstring_list_append(cookie_list,\n+\t\t\t\t slash ? slash + 1 : path->buf);\n+\t\tbreak;\n+\n+\tcase IS_INSIDE_DOT_GIT:\n+\t\t/* ignore everything inside of \"/.git/\" */\n+\t\tbreak;\n+\n+\tcase IS_DOT_GIT:\n+\t\t/* \"/.git\" was deleted (or renamed away) */\n+\t\tif ((info_action == FILE_ACTION_REMOVED) ||\n+\t\t (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {\n+\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n+\t\t\t\t\t \"fsm-listen/dotgit\",\n+\t\t\t\t\t \"removed\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tbreak;\n+\n+\tcase IS_WORKDIR_PATH:\n+\t\t/* queue normal pathname */\n+\t\tif (!*batch)\n+\t\t\t*batch = fsmonitor_batch__new();\n+\t\tfsmonitor_batch__add_path(*batch, path->buf);\n+\t\tbreak;\n+\n+\tcase IS_GITDIR:\n+\tcase IS_INSIDE_GITDIR:\n+\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n+\tdefault:\n+\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n+\t\t t, path->buf);\n+\t}\n+\n+\treturn 0;\n+}\n+\n /*\n * Process filesystem events that happen anywhere (recursively) under the\n * root directory. For a normal working directory, this includes\n@@ -274,6 +534,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n \tstruct fsmonitor_batch *batch = NULL;\n \tconst char *p = watch->buffer;\n+\twchar_t wpath_longname[MAX_PATH + 1];\n \n \t/*\n \t * If the kernel gets more events than will fit in the kernel\n@@ -306,54 +567,64 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)\n \t */\n \tfor (;;) {\n \t\tFILE_NOTIFY_INFORMATION *info = (void *)p;\n-\t\tconst char *slash;\n+\t\twchar_t *wpath = info->FileName;\n+\t\tDWORD wpath_len = info->FileNameLength / sizeof(WCHAR);\n \t\tenum fsmonitor_path_type t;\n+\t\tenum get_relative_result grr;\n+\n+\t\tif (watch->has_shortnames) {\n+\t\t\tif (!wcscmp(wpath, watch->dotgit_shortname)) {\n+\t\t\t\t/*\n+\t\t\t\t * This event exactly matches the\n+\t\t\t\t * spelling of the shortname of\n+\t\t\t\t * \".git\", so we can skip some steps.\n+\t\t\t\t *\n+\t\t\t\t * (This case is odd because the user\n+\t\t\t\t * can \"rm -rf GIT~1\" and we cannot\n+\t\t\t\t * use the filesystem to map it back\n+\t\t\t\t * to \".git\".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_reset(&path);\n+\t\t\t\tstrbuf_addstr(&path, \".git\");\n+\t\t\t\tt = IS_DOT_GIT;\n+\t\t\t\tgoto process_it;\n+\t\t\t}\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n-\t\t\tgoto skip_this_path;\n-\n-\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n-\n-\t\tswitch (t) {\n-\t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-\t\t\t/* special case cookie files within .git */\n-\n-\t\t\t/* Use just the filename of the cookie file. */\n-\t\t\tslash = find_last_dir_sep(path.buf);\n-\t\t\tstring_list_append(&cookie_list,\n-\t\t\t\t\t slash ? slash + 1 : path.buf);\n-\t\t\tbreak;\n-\n-\t\tcase IS_INSIDE_DOT_GIT:\n-\t\t\t/* ignore everything inside of \"/.git/\" */\n-\t\t\tbreak;\n+\t\t\tif (watch->has_tilde && !wcschr(wpath, L'~')) {\n+\t\t\t\t/*\n+\t\t\t\t * Shortnames on this filesystem have tildes\n+\t\t\t\t * and the notification path does not have\n+\t\t\t\t * one, so we assume that it is a longname.\n+\t\t\t\t */\n+\t\t\t\tgoto normalize_it;\n+\t\t\t}\n \n-\t\tcase IS_DOT_GIT:\n-\t\t\t/* \"/.git\" was deleted (or renamed away) */\n-\t\t\tif ((info->Action == FILE_ACTION_REMOVED) ||\n-\t\t\t (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {\n-\t\t\t\ttrace2_data_string(\"fsmonitor\", NULL,\n-\t\t\t\t\t\t \"fsm-listen/dotgit\",\n-\t\t\t\t\t\t \"removed\");\n+\t\t\tgrr = get_relative_longname(watch, wpath, wpath_len,\n+\t\t\t\t\t\t wpath_longname,\n+\t\t\t\t\t\t ARRAY_SIZE(wpath_longname));\n+\t\t\tswitch (grr) {\n+\t\t\tcase GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */\n+\t\t\t\tbreak;\n+\t\t\tcase GRR_HAVE_CONVERSION:\n+\t\t\t\twpath = wpath_longname;\n+\t\t\t\twpath_len = wcslen(wpath);\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\tcase GRR_SHUTDOWN:\n \t\t\t\tgoto force_shutdown;\n \t\t\t}\n-\t\t\tbreak;\n+\t\t}\n \n-\t\tcase IS_WORKDIR_PATH:\n-\t\t\t/* queue normal pathname */\n-\t\t\tif (!batch)\n-\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\tfsmonitor_batch__add_path(batch, path.buf);\n-\t\t\tbreak;\n+normalize_it:\n+\t\tif (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)\n+\t\t\tgoto skip_this_path;\n \n-\t\tcase IS_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR:\n-\t\tcase IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:\n-\t\tdefault:\n-\t\t\tBUG(\"unexpected path classification '%d' for '%s'\",\n-\t\t\t t, path.buf);\n-\t\t}\n+\t\tt = fsmonitor_classify_path_workdir_relative(path.buf);\n+\n+process_it:\n+\t\tif (process_1_worktree_event(&cookie_list, &batch, &path, t,\n+\t\t\t\t\t info->Action))\n+\t\t\tgoto force_shutdown;\n \n skip_this_path:\n \t\tif (!info->NextEntryOffset)\n@@ -382,6 +653,9 @@ force_shutdown:\n * Note that we DO NOT get filesystem events on the external \n * itself (it is not inside something that we are watching). In particular,\n * we do not get an event if the external is deleted.\n+ *\n+ * Also, we do not care about shortnames within the external , since\n+ * we never send these paths to clients.\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n@@ -403,8 +677,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n \t\tconst char *slash;\n \t\tenum fsmonitor_path_type t;\n \n-\t\tstrbuf_reset(&path);\n-\t\tif (normalize_path_in_utf8(info, &path) == -1)\n+\t\tif (normalize_path_in_utf8(\n+\t\t\t info->FileName,\n+\t\t\t info->FileNameLength / sizeof(WCHAR),\n+\t\t\t &path) == -1)\n \t\t\tgoto skip_this_path;\n \n \t\tt = fsmonitor_classify_path_gitdir_relative(path.buf);\n@@ -538,6 +814,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \tif (!data->watch_worktree)\n \t\tgoto failed;\n \n+\tcheck_for_shortnames(data->watch_worktree);\n+\n \tif (state->nr_paths_watching > 1) {\n \t\tdata->watch_gitdir = create_watch(state,\n \t\t\t\t\t\t state->path_gitdir_watch.buf);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex bd0c952a116..1be21785162 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -166,6 +166,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n '\n \n+# File systems on Windows may or may not have shortnames.\n+# This is a volume-specific setting on modern systems.\n+# \"C:/\" drives are required to have them enabled. Other\n+# hard drives default to disabled.\n+#\n+# This is a crude test to see if shortnames are enabled\n+# on the volume containing the test directory. It is\n+# crude, but it does not require elevation like `fsutil`.\n+#\n+test_lazy_prereq SHORTNAMES '\n+\tmkdir .foo &&\n+\ttest -d \"FOO~1\"\n+'\n+\n+# Here we assume that the shortname of \".git\" is \"GIT~1\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s\" &&\n+\n+\tgit init test_implicit_1s &&\n+\n+\tstart_daemon -C test_implicit_1s &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n+\tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+'\n+\n+# Here we first create a file with LONGNAME of \"GIT~1\" before\n+# we create the repo. This will cause the shortname of \".git\"\n+# to be \"GIT~2\".\n+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_implicit_1s2\" &&\n+\n+\tmkdir test_implicit_1s2 &&\n+\techo HELLO >test_implicit_1s2/GIT~1 &&\n+\tgit init test_implicit_1s2 &&\n+\n+\ttest_path_is_file test_implicit_1s2/GIT~1 &&\n+\ttest_path_is_dir test_implicit_1s2/GIT~2 &&\n+\n+\tstart_daemon -C test_implicit_1s2 &&\n+\n+\t# renaming the .git directory will implicitly stop the daemon.\n+\t# the rename-from FS Event will contain the shortname.\n+\t#\n+\tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n+\n+\tsleep 1 &&\n+\t# put it back so that our status will not crawl out to our\n+\t# parent directory.\n+\tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n+\n+\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+'\n+\n test_expect_success 'cannot start multiple daemons' '\n \ttest_when_finished \"stop_daemon_delete_repo test_multiple\" &&\n \n-- \ngitgitgadget\n\n"},{"id":"456236","messageId":"06d49b0f230a72f4759a13e5ff213917c4fe7e00.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 06/30] fsmonitor-settings: VFS for Git virtual repos are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:00Z","receivedAt":"2022-05-26T21:49:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nVFS for Git virtual repositories are incompatible with FSMonitor.\n\nVFS for Git is a downstream fork of Git. It contains its own custom\nfile system watcher that is aware of the virtualization. If a working\ndirectory is being managed by VFS for Git, we should not try to watch\nit because we may get incomplete results.\n\nWe do not know anything about how VFS for Git works, but we do\nknow that VFS for Git working directories contain a well-defined\nconfig setting. If it is set, mark the working directory as\nincompatible.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 6 ++++++\n fsmonitor-settings.h | 1 +\n t/t7519-status-fsmonitor.sh | 9 +++++++++\n 4 files changed, 42 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex 7fce32a3c5b..ee78bba38e3 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -3,7 +3,33 @@\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n \n+/*\n+ * VFS for Git is incompatible with FSMonitor.\n+ *\n+ * Granted, core Git does not know anything about VFS for Git and we\n+ * shouldn't make assumptions about a downstream feature, but users\n+ * can install both versions. And this can lead to incorrect results\n+ * from core Git commands. So, without bringing in any of the VFS for\n+ * Git code, do a simple config test for a published config setting.\n+ * (We do not look at the various *_TEST_* environment variables.)\n+ */\n+static enum fsmonitor_reason check_vfs4git(struct repository *r)\n+{\n+\tconst char *const_str;\n+\n+\tif (!repo_config_get_value(r, \"core.virtualfilesystem\", &const_str))\n+\t\treturn FSMONITOR_REASON_VFS4GIT;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_vfs4git(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex f67db913f57..600ae165ab1 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -207,6 +207,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n \t\t\t xgetcwd());\n \t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_VFS4GIT:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 6cb0d8e7d9f..a48802cde9c 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -17,6 +17,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n \tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 9a8e21c5608..156895f9437 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n \tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n '\n \n+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '\n+\ttest_when_finished \"rm -rf ./fake-virtual-clone actual\" &&\n+\tgit init fake-virtual-clone &&\n+\ttest_must_fail git -C ./fake-virtual-clone \\\n+\t\t\t -c core.virtualfilesystem=true \\\n+\t\t\t fsmonitor--daemon run 2>actual &&\n+\tgrep \"virtual repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"456237","messageId":"6f8baf5b723c4eba2240ebb2abaf8124e315322d.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 03/30] t/helper/fsmonitor-client: create stress test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:46:57Z","receivedAt":"2022-05-26T21:49:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a stress test to hammer on the fsmonitor daemon.\nCreate a client-side thread pool of n threads and have\neach of them make m requests as fast as they can.\n\nWe do not currently inspect the contents of the response.\nWe're only interested in placing a heavy request load on\nthe daemon.\n\nThis test is useful for interactive testing and various\nexperimentation. For example, to place additional load\non the daemon while another test is running. We currently\ndo not have a test script that actually uses this helper.\nWe might add such a test in the future.\n\nSigned-off-by: Jeff Hostetler \n---\n t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++\n 1 file changed, 106 insertions(+)\n\ndiff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c\nindex 3062c8a3c2b..54a4856c48c 100644\n--- a/t/helper/test-fsmonitor-client.c\n+++ b/t/helper/test-fsmonitor-client.c\n@@ -7,6 +7,8 @@\n #include \"cache.h\"\n #include \"parse-options.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2.h\"\n \n #ifndef HAVE_FSMONITOR_DAEMON_BACKEND\n int cmd__fsmonitor_client(int argc, const char **argv)\n@@ -79,20 +81,121 @@ static int do_send_flush(void)\n \treturn 0;\n }\n \n+struct hammer_thread_data\n+{\n+\tpthread_t pthread_id;\n+\tint thread_nr;\n+\n+\tint nr_requests;\n+\tconst char *token;\n+\n+\tint sum_successful;\n+\tint sum_errors;\n+};\n+\n+static void *hammer_thread_proc(void *_hammer_thread_data)\n+{\n+\tstruct hammer_thread_data *data = _hammer_thread_data;\n+\tstruct strbuf answer = STRBUF_INIT;\n+\tint k;\n+\tint ret;\n+\n+\ttrace2_thread_start(\"hammer\");\n+\n+\tfor (k = 0; k < data->nr_requests; k++) {\n+\t\tstrbuf_reset(&answer);\n+\n+\t\tret = fsmonitor_ipc__send_query(data->token, &answer);\n+\t\tif (ret < 0)\n+\t\t\tdata->sum_errors++;\n+\t\telse\n+\t\t\tdata->sum_successful++;\n+\t}\n+\n+\tstrbuf_release(&answer);\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Start a pool of client threads that will each send a series of\n+ * commands to the daemon.\n+ *\n+ * The goal is to overload the daemon with a sustained series of\n+ * concurrent requests.\n+ */\n+static int do_hammer(const char *token, int nr_threads, int nr_requests)\n+{\n+\tstruct hammer_thread_data *data = NULL;\n+\tint k;\n+\tint sum_join_errors = 0;\n+\tint sum_commands = 0;\n+\tint sum_errors = 0;\n+\n+\tif (!token || !*token)\n+\t\ttoken = get_token_from_index();\n+\tif (nr_threads < 1)\n+\t\tnr_threads = 1;\n+\tif (nr_requests < 1)\n+\t\tnr_requests = 1;\n+\n+\tCALLOC_ARRAY(data, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\t\tp->thread_nr = k;\n+\t\tp->nr_requests = nr_requests;\n+\t\tp->token = token;\n+\n+\t\tif (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {\n+\t\t\twarning(\"failed to create thread[%d] skipping remainder\", k);\n+\t\t\tnr_threads = k;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tstruct hammer_thread_data *p = &data[k];\n+\n+\t\tif (pthread_join(p->pthread_id, NULL))\n+\t\t\tsum_join_errors++;\n+\t\tsum_commands += p->sum_successful;\n+\t\tsum_errors += p->sum_errors;\n+\t}\n+\n+\tfprintf(stderr, \"HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\\n\",\n+\t\tnr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);\n+\n+\tfree(data);\n+\n+\t/*\n+\t * Return an error if any of the _send_query requests failed.\n+\t * We don't care about thread create/join errors.\n+\t */\n+\treturn sum_errors > 0;\n+}\n+\n int cmd__fsmonitor_client(int argc, const char **argv)\n {\n \tconst char *subcmd;\n \tconst char *token = NULL;\n+\tint nr_threads = 1;\n+\tint nr_requests = 1;\n \n \tconst char * const fsmonitor_client_usage[] = {\n \t\t\"test-tool fsmonitor-client query []\",\n \t\t\"test-tool fsmonitor-client flush\",\n+\t\t\"test-tool fsmonitor-client hammer [] [] []\",\n \t\tNULL,\n \t};\n \n \tstruct option options[] = {\n \t\tOPT_STRING(0, \"token\", &token, \"token\",\n \t\t\t \"command token to send to the server\"),\n+\n+\t\tOPT_INTEGER(0, \"threads\", &nr_threads, \"number of client threads\"),\n+\t\tOPT_INTEGER(0, \"requests\", &nr_requests, \"number of requests per thread\"),\n+\n \t\tOPT_END()\n \t};\n \n@@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)\n \tif (!strcmp(subcmd, \"flush\"))\n \t\treturn !!do_send_flush();\n \n+\tif (!strcmp(subcmd, \"hammer\"))\n+\t\treturn !!do_hammer(token, nr_threads, nr_requests);\n+\n \tdie(\"Unhandled subcommand: '%s'\", subcmd);\n }\n #endif\n-- \ngitgitgadget\n\n"},{"id":"456238","messageId":"2ae8a02ebe002afcd870c9e23cd0c5f414855575.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 02/30] t7527: test FSMonitor on repos with Unicode root paths","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:46:56Z","receivedAt":"2022-05-26T21:49:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate some test repos with UTF8 characters in the pathname of the\nroot directory and verify that the builtin FSMonitor can watch them.\n\nThis test is mainly for Windows where we need to avoid `*A()`\nroutines.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 1be21785162..12655958e71 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -671,4 +671,27 @@ do\n \tdone\n done\n \n+# Test Unicode UTF-8 characters in the pathname of the working\n+# directory root. Use of \"*A()\" routines rather than \"*W()\" routines\n+# on Windows can sometimes lead to odd failures.\n+#\n+u1=$(printf \"u_c3_a6__\\xC3\\xA6\")\n+u2=$(printf \"u_e2_99_ab__\\xE2\\x99\\xAB\")\n+u_values=\"$u1 $u2\"\n+for u in $u_values\n+do\n+\ttest_expect_success \"unicode in repo root path: $u\" '\n+\t\ttest_when_finished \"stop_daemon_delete_repo $u\" &&\n+\n+\t\tgit init \"$u\" &&\n+\t\techo 1 >\"$u\"/file1 &&\n+\t\tgit -C \"$u\" add file1 &&\n+\t\tgit -C \"$u\" config core.fsmonitor true &&\n+\n+\t\tstart_daemon -C \"$u\" &&\n+\t\tgit -C \"$u\" status >actual &&\n+\t\tgrep \"new file: file1\" actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"456239","messageId":"195f90cc9ef85cae95f084d204b3648da9c51d87.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 04/30] fsmonitor-settings: bare repos are incompatible with FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:46:58Z","receivedAt":"2022-05-26T21:49:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nBare repos do not have a worktree, so there is nothing for the\ndaemon watch.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 18 +++++\n builtin/update-index.c | 16 +++++\n fsmonitor-settings.c | 133 ++++++++++++++++++++++++++++++------\n fsmonitor-settings.h | 16 +++++\n t/t7519-status-fsmonitor.sh | 23 +++++++\n 5 files changed, 186 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 46be55a4618..66b78a0353f 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1423,6 +1423,7 @@ static int try_to_start_background_daemon(void)\n int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n {\n \tconst char *subcmd;\n+\tenum fsmonitor_reason reason;\n \tint detach_console = 0;\n \n \tstruct option options[] = {\n@@ -1449,6 +1450,23 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"invalid 'ipc-threads' value (%d)\"),\n \t\t fsmonitor__ipc_threads);\n \n+\tprepare_repo_settings(the_repository);\n+\t/*\n+\t * If the repo is fsmonitor-compatible, explicitly set IPC-mode\n+\t * (without bothering to load the `core.fsmonitor` config settings).\n+\t *\n+\t * If the repo is not compatible, the repo-settings will be set to\n+\t * incompatible rather than IPC, so we can use one of the __get\n+\t * routines to detect the discrepancy.\n+\t */\n+\tfsm_settings__set_ipc(the_repository);\n+\n+\treason = fsm_settings__get_reason(the_repository);\n+\tif (reason > FSMONITOR_REASON_OK)\n+\t\tdie(\"%s\",\n+\t\t fsm_settings__get_incompatible_msg(the_repository,\n+\t\t\t\t\t\t reason));\n+\n \tif (!strcmp(subcmd, \"start\"))\n \t\treturn !!try_to_start_background_daemon();\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 876112abb21..01ed4c4976b 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1237,6 +1237,22 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tif (fsmonitor > 0) {\n \t\tenum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);\n+\t\tenum fsmonitor_reason reason = fsm_settings__get_reason(r);\n+\n+\t\t/*\n+\t\t * The user wants to turn on FSMonitor using the command\n+\t\t * line argument. (We don't know (or care) whether that\n+\t\t * is the IPC or HOOK version.)\n+\t\t *\n+\t\t * Use one of the __get routines to force load the FSMonitor\n+\t\t * config settings into the repo-settings. That will detect\n+\t\t * whether the file system is compatible so that we can stop\n+\t\t * here with a nice error message.\n+\t\t */\n+\t\tif (reason > FSMONITOR_REASON_OK)\n+\t\t\tdie(\"%s\",\n+\t\t\t fsm_settings__get_incompatible_msg(r, reason));\n+\n \t\tif (fsm_mode == FSMONITOR_MODE_DISABLED) {\n \t\t\twarning(_(\"core.fsmonitor is unset; \"\n \t\t\t\t\"set it if you really want to \"\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 757d230d538..7d3177d441a 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -9,23 +9,42 @@\n */\n struct fsmonitor_settings {\n \tenum fsmonitor_mode mode;\n+\tenum fsmonitor_reason reason;\n \tchar *hook_path;\n };\n \n-static void lookup_fsmonitor_settings(struct repository *r)\n+static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n+{\n+\tif (!r->worktree) {\n+\t\t/*\n+\t\t * Bare repositories don't have a working directory and\n+\t\t * therefore have nothing to watch.\n+\t\t */\n+\t\treturn FSMONITOR_REASON_BARE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n+static struct fsmonitor_settings *alloc_settings(void)\n {\n \tstruct fsmonitor_settings *s;\n+\n+\tCALLOC_ARRAY(s, 1);\n+\ts->mode = FSMONITOR_MODE_DISABLED;\n+\ts->reason = FSMONITOR_REASON_UNTESTED;\n+\n+\treturn s;\n+}\n+\n+static void lookup_fsmonitor_settings(struct repository *r)\n+{\n \tconst char *const_str;\n \tint bool_value;\n \n \tif (r->settings.fsmonitor)\n \t\treturn;\n \n-\tCALLOC_ARRAY(s, 1);\n-\ts->mode = FSMONITOR_MODE_DISABLED;\n-\n-\tr->settings.fsmonitor = s;\n-\n \t/*\n \t * Overload the existing \"core.fsmonitor\" config setting (which\n \t * has historically been either unset or a hook pathname) to\n@@ -38,6 +57,8 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \tcase 0: /* config value was set to */\n \t\tif (bool_value)\n \t\t\tfsm_settings__set_ipc(r);\n+\t\telse\n+\t\t\tfsm_settings__set_disabled(r);\n \t\treturn;\n \n \tcase 1: /* config value was unset */\n@@ -53,18 +74,18 @@ static void lookup_fsmonitor_settings(struct repository *r)\n \t\treturn;\n \t}\n \n-\tif (!const_str || !*const_str)\n-\t\treturn;\n-\n-\tfsm_settings__set_hook(r, const_str);\n+\tif (const_str && *const_str)\n+\t\tfsm_settings__set_hook(r, const_str);\n+\telse\n+\t\tfsm_settings__set_disabled(r);\n }\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n \n \treturn r->settings.fsmonitor->mode;\n }\n@@ -73,31 +94,55 @@ const char *fsm_settings__get_hook_path(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n \n \treturn r->settings.fsmonitor->hook_path;\n }\n \n void fsm_settings__set_ipc(struct repository *r)\n {\n+\tenum fsmonitor_reason reason = check_for_incompatible(r);\n+\n+\tif (reason != FSMONITOR_REASON_OK) {\n+\t\tfsm_settings__set_incompatible(r, reason);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Caller requested IPC explicitly, so avoid (possibly\n+\t * recursive) config lookup.\n+\t */\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n \n void fsm_settings__set_hook(struct repository *r, const char *path)\n {\n+\tenum fsmonitor_reason reason = check_for_incompatible(r);\n+\n+\tif (reason != FSMONITOR_REASON_OK) {\n+\t\tfsm_settings__set_incompatible(r, reason);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Caller requested hook explicitly, so avoid (possibly\n+\t * recursive) config lookup.\n+\t */\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n \tr->settings.fsmonitor->hook_path = strdup(path);\n }\n@@ -106,9 +151,57 @@ void fsm_settings__set_disabled(struct repository *r)\n {\n \tif (!r)\n \t\tr = the_repository;\n-\n-\tlookup_fsmonitor_settings(r);\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n \n \tr->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;\n+\tr->settings.fsmonitor->reason = FSMONITOR_REASON_OK;\n+\tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n+}\n+\n+void fsm_settings__set_incompatible(struct repository *r,\n+\t\t\t\t enum fsmonitor_reason reason)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\tif (!r->settings.fsmonitor)\n+\t\tr->settings.fsmonitor = alloc_settings();\n+\n+\tr->settings.fsmonitor->mode = FSMONITOR_MODE_INCOMPATIBLE;\n+\tr->settings.fsmonitor->reason = reason;\n \tFREE_AND_NULL(r->settings.fsmonitor->hook_path);\n }\n+\n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)\n+{\n+\tif (!r)\n+\t\tr = the_repository;\n+\tif (!r->settings.fsmonitor)\n+\t\tlookup_fsmonitor_settings(r);\n+\n+\treturn r->settings.fsmonitor->reason;\n+}\n+\n+char *fsm_settings__get_incompatible_msg(const struct repository *r,\n+\t\t\t\t\t enum fsmonitor_reason reason)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tswitch (reason) {\n+\tcase FSMONITOR_REASON_UNTESTED:\n+\tcase FSMONITOR_REASON_OK:\n+\t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_BARE:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"bare repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t xgetcwd());\n+\t\tgoto done;\n+\t}\n+\n+\tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\n+\t reason);\n+\n+done:\n+\treturn strbuf_detach(&msg, NULL);\n+}\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a4c5d7b4889..8d9331c0c0a 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -4,18 +4,34 @@\n struct repository;\n \n enum fsmonitor_mode {\n+\tFSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */\n \tFSMONITOR_MODE_DISABLED = 0,\n \tFSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */\n \tFSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */\n };\n \n+/*\n+ * Incompatibility reasons.\n+ */\n+enum fsmonitor_reason {\n+\tFSMONITOR_REASON_UNTESTED = 0,\n+\tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n+\tFSMONITOR_REASON_BARE,\n+};\n+\n void fsm_settings__set_ipc(struct repository *r);\n void fsm_settings__set_hook(struct repository *r, const char *path);\n void fsm_settings__set_disabled(struct repository *r);\n+void fsm_settings__set_incompatible(struct repository *r,\n+\t\t\t\t enum fsmonitor_reason reason);\n \n enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);\n const char *fsm_settings__get_hook_path(struct repository *r);\n \n+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);\n+char *fsm_settings__get_incompatible_msg(const struct repository *r,\n+\t\t\t\t\t enum fsmonitor_reason reason);\n+\n struct fsmonitor_settings;\n \n #endif /* FSMONITOR_SETTINGS_H */\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex a6308acf006..9a8e21c5608 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '\n \ttest $ret -ne 1\n '\n \n+# Test that we detect and disallow repos that are incompatible with FSMonitor.\n+test_expect_success 'incompatible bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual expect\" &&\n+\tgit init --bare bare-clone &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=foo \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual &&\n+\n+\ttest_must_fail \\\n+\t\tgit -C ./bare-clone -c core.fsmonitor=true \\\n+\t\t\tupdate-index --fsmonitor 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '\n+\ttest_when_finished \"rm -rf ./bare-clone actual\" &&\n+\tgit init --bare bare-clone &&\n+\ttest_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&\n+\tgrep \"bare repository .* is incompatible with fsmonitor\" actual\n+'\n+\n test_expect_success 'setup' '\n \tmkdir -p .git/hooks &&\n \t: >tracked &&\n-- \ngitgitgadget\n\n"},{"id":"456240","messageId":"0b182569e1135265fb54d2592fa2bc0db2c1312b.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 05/30] fsmonitor-settings: stub in Win32-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:46:59Z","receivedAt":"2022-05-26T21:49:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend generic incompatibility checkout with platform-specific\nmechanism. Stub in Win32 version.\n\nIn the existing fsmonitor-settings code we have a way to mark\ntypes of repos as incompatible with fsmonitor (whether via the\nhook and IPC APIs). For example, we do this for bare repos,\nsince there are no files to watch.\n\nExtend this exclusion mechanism for platform-specific reasons.\nThis commit just creates the framework and adds a stub for Win32.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 13 +++++++++++++\n compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++\n config.mak.uname | 4 ++++\n contrib/buildsystems/CMakeLists.txt | 3 +++\n fsmonitor-settings.c | 10 ++++++++++\n fsmonitor-settings.h | 13 +++++++++++++\n 6 files changed, 52 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-win32.c\n\ndiff --git a/Makefile b/Makefile\nindex daa21bed6c3..93604fe8ef7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -475,6 +475,11 @@ all::\n # `compat/fsmonitor/fsm-listen-.c` that implements the\n # `fsm_listen__*()` routines.\n #\n+# If your platform has OS-specific ways to tell if a repo is incompatible with\n+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n+# to the \"\" of the corresponding `compat/fsmonitor/fsm-settings-.c`\n+# that implements the `fsm_os_settings__*()` routines.\n+#\n # Define DEVELOPER to enable more compiler warnings. Compiler version\n # and family are auto detected, but could be overridden by defining\n # COMPILER_FEATURES (see config.mak.dev). You can still set\n@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n+ifdef FSMONITOR_OS_SETTINGS\n+\tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o\n+endif\n+\n ifeq ($(TCLTK_PATH),)\n NO_TCLTK = NoThanks\n endif\n@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE\n ifdef FSMONITOR_DAEMON_BACKEND\n \t@echo FSMONITOR_DAEMON_BACKEND=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_DAEMON_BACKEND)))'\\' >>$@+\n endif\n+ifdef FSMONITOR_OS_SETTINGS\n+\t@echo FSMONITOR_OS_SETTINGS=\\''$(subst ','\\'',$(subst ','\\'',$(FSMONITOR_OS_SETTINGS)))'\\' >>$@+\n+endif\n ifdef TEST_OUTPUT_DIRECTORY\n \t@echo TEST_OUTPUT_DIRECTORY=\\''$(subst ','\\'',$(subst ','\\'',$(TEST_OUTPUT_DIRECTORY)))'\\' >>$@+\n endif\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 501970902da..cf224768ad6 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tNO_SVN_TESTS = YesPlease\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)\n \t# These are always available, so we do not have to conditionally\n \t# support it.\n \tFSMONITOR_DAEMON_BACKEND = win32\n+\tFSMONITOR_OS_SETTINGS = win32\n+\n \tRUNTIME_PREFIX = YesPlease\n \tHAVE_WPGMPTR = YesWeDo\n \tNO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex ee0d7257b77..16705da2000 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 7d3177d441a..f67db913f57 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -23,6 +23,16 @@ static enum fsmonitor_reason check_for_incompatible(struct repository *r)\n \t\treturn FSMONITOR_REASON_BARE;\n \t}\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+\t{\n+\t\tenum fsmonitor_reason reason;\n+\n+\t\treason = fsm_os__incompatible(r);\n+\t\tif (reason != FSMONITOR_REASON_OK)\n+\t\t\treturn reason;\n+\t}\n+#endif\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex 8d9331c0c0a..6cb0d8e7d9f 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -34,4 +34,17 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \n struct fsmonitor_settings;\n \n+#ifdef HAVE_FSMONITOR_OS_SETTINGS\n+/*\n+ * Ask platform-specific code whether the repository is incompatible\n+ * with fsmonitor (both hook and ipc modes). For example, if the working\n+ * directory is on a remote volume and mounted via a technology that does\n+ * not support notification events, then we should not pretend to watch it.\n+ *\n+ * fsm_os__* routines should considered private to fsm_settings__\n+ * routines.\n+ */\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);\n+#endif /* HAVE_FSMONITOR_OS_SETTINGS */\n+\n #endif /* FSMONITOR_SETTINGS_H */\n-- \ngitgitgadget\n\n"},{"id":"456241","messageId":"2b6e8b0b241314e3f5d4dde9d81fb8ccc3c854c6.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 07/30] fsmonitor-settings: stub in macOS-specific incompatibility checking","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:01Z","receivedAt":"2022-05-26T21:49:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++\n config.mak.uname | 1 +\n contrib/buildsystems/CMakeLists.txt | 3 +++\n 3 files changed, 13 insertions(+)\n create mode 100644 compat/fsmonitor/fsm-settings-darwin.c\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nnew file mode 100644\nindex 00000000000..7fce32a3c5b\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -0,0 +1,9 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"repository.h\"\n+#include \"fsmonitor-settings.h\"\n+\n+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n+{\n+\treturn FSMONITOR_REASON_OK;\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cf224768ad6..cf911d141f2 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)\n \tifndef NO_PTHREADS\n \tifndef NO_UNIX_SOCKETS\n \tFSMONITOR_DAEMON_BACKEND = darwin\n+\tFSMONITOR_OS_SETTINGS = darwin\n \tendif\n \tendif\n \ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 16705da2000..b8f9f7a0388 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\n+\t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\n \tendif()\n endif()\n \n-- \ngitgitgadget\n\n"},{"id":"456242","messageId":"8068b3bebee45056d9b09ab71ac4e9d4b013e0bd.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 08/30] fsmonitor-settings: remote repos on macOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:02Z","receivedAt":"2022-05-26T21:49:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on macOS and mark them as\nincompatible with FSMonitor.\n\nWith this, `git fsmonitor--daemon run` will error out with a message\nlike it does for bare repos.\n\nClient commands, like `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++\n fsmonitor-settings.c | 12 +++++\n fsmonitor-settings.h | 2 +\n 3 files changed, 80 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex 7fce32a3c5b..fdd762bf79d 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -2,8 +2,74 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n+#include \n+#include \n+\n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type (NFS, SAMBA, etc.) dictates whether notification events\n+ * are available at all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * For the builtin FSMonitor, we create the Unix domain socket for the\n+ * IPC in the .git directory. If the working directory is remote,\n+ * then the socket will be created on the remote file system. This\n+ * can fail if the remote file system does not support UDS file types\n+ * (e.g. smbfs to a Windows server) or if the remote kernel does not\n+ * allow a non-local process to bind() the socket. (These problems\n+ * could be fixed by moving the UDS out of the .git directory and to a\n+ * well-known local directory on the client machine, but care should\n+ * be taken to ensure that $HOME is actually local and not a managed\n+ * file share.)\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\tstruct statfs fs;\n+\n+\tif (statfs(r->worktree, &fs) == -1) {\n+\t\tint saved_errno = errno;\n+\t\ttrace_printf_key(&trace_fsmonitor, \"statfs('%s') failed: %s\",\n+\t\t\t\t r->worktree, strerror(saved_errno));\n+\t\terrno = saved_errno;\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"statfs('%s') [type 0x%08x][flags 0x%08x] '%s'\",\n+\t\t\t r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);\n+\n+\tif (!(fs.f_flags & MNT_LOCAL))\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n \n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n+\tenum fsmonitor_reason reason;\n+\n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex 600ae165ab1..d2fb0141f8e 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -208,6 +208,18 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t xgetcwd());\n \t\tgoto done;\n \n+\tcase FSMONITOR_REASON_ERROR:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to errors\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_REMOTE:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"remote repository '%s' is incompatible with fsmonitor\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n+\n \tcase FSMONITOR_REASON_VFS4GIT:\n \t\tstrbuf_addf(&msg,\n \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex a48802cde9c..afd1b3874ac 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -17,6 +17,8 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_UNTESTED = 0,\n \tFSMONITOR_REASON_OK, /* no incompatibility or when disabled */\n \tFSMONITOR_REASON_BARE,\n+\tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n+\tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"456243","messageId":"a5c40a4464d8327f091c85da02b1fb01067bdafd.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 09/30] fsmonitor-settings: remote repos on Windows are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:03Z","receivedAt":"2022-05-26T21:49:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to detect remote working directories on Windows and mark them as\nincompatible with FSMonitor.\n\nWith this `git fsmonitor--daemon run` will error out with a message like it\ndoes for bare repos.\n\nClient commands, such as `git status`, will not attempt to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++\n 1 file changed, 102 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c\nindex ee78bba38e3..907655720bb 100644\n--- a/compat/fsmonitor/fsm-settings-win32.c\n+++ b/compat/fsmonitor/fsm-settings-win32.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"repository.h\"\n #include \"fsmonitor-settings.h\"\n+#include \"fsmonitor.h\"\n \n /*\n * VFS for Git is incompatible with FSMonitor.\n@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)\n \treturn FSMONITOR_REASON_OK;\n }\n \n+/*\n+ * Remote working directories are problematic for FSMonitor.\n+ *\n+ * The underlying file system on the server machine and/or the remote\n+ * mount type dictates whether notification events are available at\n+ * all to remote client machines.\n+ *\n+ * Kernel differences between the server and client machines also\n+ * dictate the how (buffering, frequency, de-dup) the events are\n+ * delivered to client machine processes.\n+ *\n+ * A client machine (such as a laptop) may choose to suspend/resume\n+ * and it is unclear (without lots of testing) whether the watcher can\n+ * resync after a resume. We might be able to treat this as a normal\n+ * \"events were dropped by the kernel\" event and do our normal \"flush\n+ * and resync\" --or-- we might need to close the existing (zombie?)\n+ * notification fd and create a new one.\n+ *\n+ * In theory, the above issues need to be addressed whether we are\n+ * using the Hook or IPC API.\n+ *\n+ * So (for now at least), mark remote working directories as\n+ * incompatible.\n+ *\n+ * Notes for testing:\n+ *\n+ * (a) Windows allows a network share to be mapped to a drive letter.\n+ * (This is the normal method to access it.)\n+ *\n+ * $ NET USE Z: \\\\server\\share\n+ * $ git -C Z:/repo status\n+ *\n+ * (b) Windows allows a network share to be referenced WITHOUT mapping\n+ * it to drive letter.\n+ *\n+ * $ NET USE \\\\server\\share\\dir\n+ * $ git -C //server/share/repo status\n+ *\n+ * (c) Windows allows \"SUBST\" to create a fake drive mapping to an\n+ * arbitrary path (which may be remote)\n+ *\n+ * $ SUBST Q: Z:\\repo\n+ * $ git -C Q:/ status\n+ *\n+ * (d) Windows allows a directory symlink to be created on a local\n+ * file system that points to a remote repo.\n+ *\n+ * $ mklink /d ./link //server/share/repo\n+ * $ git -C ./link status\n+ */\n+static enum fsmonitor_reason check_remote(struct repository *r)\n+{\n+\twchar_t wpath[MAX_PATH];\n+\twchar_t wfullpath[MAX_PATH];\n+\tsize_t wlen;\n+\tUINT driveType;\n+\n+\t/*\n+\t * Do everything in wide chars because the drive letter might be\n+\t * a multi-byte sequence. See win32_has_dos_drive_prefix().\n+\t */\n+\tif (xutftowcs_path(wpath, r->worktree) < 0)\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\t/*\n+\t * GetDriveTypeW() requires a final slash. We assume that the\n+\t * worktree pathname points to an actual directory.\n+\t */\n+\twlen = wcslen(wpath);\n+\tif (wpath[wlen - 1] != L'\\\\' && wpath[wlen - 1] != L'/') {\n+\t\twpath[wlen++] = L'\\\\';\n+\t\twpath[wlen] = 0;\n+\t}\n+\n+\t/*\n+\t * Normalize the path. If nothing else, this converts forward\n+\t * slashes to backslashes. This is essential to get GetDriveTypeW()\n+\t * correctly handle some UNC \"\\\\server\\share\\...\" paths.\n+\t */\n+\tif (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))\n+\t\treturn FSMONITOR_REASON_ERROR;\n+\n+\tdriveType = GetDriveTypeW(wfullpath);\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"DriveType '%s' L'%ls' (%u)\",\n+\t\t\t r->worktree, wfullpath, driveType);\n+\n+\tif (driveType == DRIVE_REMOTE) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"check_remote('%s') true\",\n+\t\t\t\t r->worktree);\n+\t\treturn FSMONITOR_REASON_REMOTE;\n+\t}\n+\n+\treturn FSMONITOR_REASON_OK;\n+}\n+\n enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \n+\treason = check_remote(r);\n+\tif (reason != FSMONITOR_REASON_OK)\n+\t\treturn reason;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n-- \ngitgitgadget\n\n"},{"id":"456244","messageId":"d2569ed32113801ab3bfb969f7790a10fd4478bc.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 10/30] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:04Z","receivedAt":"2022-05-26T21:49:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nOn MacOS mark repos on NTFS or FAT32 volumes as incompatible.\n\nThe builtin FSMonitor used Unix domain sockets on MacOS for IPC\nwith clients. These sockets are kept in the .git directory.\nUnix sockets are not supported by NTFS and FAT32, so the daemon\ncannot start up.\n\nTest for this during our compatibility checking so that client\ncommands do not keep trying to start the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---\n fsmonitor-settings.c | 6 ++++++\n fsmonitor-settings.h | 1 +\n 3 files changed, 24 insertions(+), 3 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c\nindex fdd762bf79d..efc732c0f31 100644\n--- a/compat/fsmonitor/fsm-settings-darwin.c\n+++ b/compat/fsmonitor/fsm-settings-darwin.c\n@@ -7,7 +7,7 @@\n #include \n \n /*\n- * Remote working directories are problematic for FSMonitor.\n+ * [1] Remote working directories are problematic for FSMonitor.\n *\n * The underlying file system on the server machine and/or the remote\n * mount type (NFS, SAMBA, etc.) dictates whether notification events\n@@ -40,8 +40,16 @@\n *\n * So (for now at least), mark remote working directories as\n * incompatible.\n+ *\n+ *\n+ * [2] FAT32 and NTFS working directories are problematic too.\n+ *\n+ * The builtin FSMonitor uses a Unix domain socket in the .git\n+ * directory for IPC. These Windows drive formats do not support\n+ * Unix domain sockets, so mark them as incompatible for the daemon.\n+ *\n */\n-static enum fsmonitor_reason check_remote(struct repository *r)\n+static enum fsmonitor_reason check_volume(struct repository *r)\n {\n \tstruct statfs fs;\n \n@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)\n \tif (!(fs.f_flags & MNT_LOCAL))\n \t\treturn FSMONITOR_REASON_REMOTE;\n \n+\tif (!strcmp(fs.f_fstypename, \"msdos\")) /* aka FAT32 */\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n+\tif (!strcmp(fs.f_fstypename, \"ntfs\"))\n+\t\treturn FSMONITOR_REASON_NOSOCKETS;\n+\n \treturn FSMONITOR_REASON_OK;\n }\n \n@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)\n {\n \tenum fsmonitor_reason reason;\n \n-\treason = check_remote(r);\n+\treason = check_volume(r);\n \tif (reason != FSMONITOR_REASON_OK)\n \t\treturn reason;\n \ndiff --git a/fsmonitor-settings.c b/fsmonitor-settings.c\nindex d2fb0141f8e..658cb79da01 100644\n--- a/fsmonitor-settings.c\n+++ b/fsmonitor-settings.c\n@@ -225,6 +225,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,\n \t\t\t _(\"virtual repository '%s' is incompatible with fsmonitor\"),\n \t\t\t r->worktree);\n \t\tgoto done;\n+\n+\tcase FSMONITOR_REASON_NOSOCKETS:\n+\t\tstrbuf_addf(&msg,\n+\t\t\t _(\"repository '%s' is incompatible with fsmonitor due to lack of Unix sockets\"),\n+\t\t\t r->worktree);\n+\t\tgoto done;\n \t}\n \n \tBUG(\"Unhandled case in fsm_settings__get_incompatible_msg: '%d'\",\ndiff --git a/fsmonitor-settings.h b/fsmonitor-settings.h\nindex afd1b3874ac..d9c2605197f 100644\n--- a/fsmonitor-settings.h\n+++ b/fsmonitor-settings.h\n@@ -20,6 +20,7 @@ enum fsmonitor_reason {\n \tFSMONITOR_REASON_ERROR, /* FS error probing for compatibility */\n \tFSMONITOR_REASON_REMOTE,\n \tFSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */\n+\tFSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */\n };\n \n void fsm_settings__set_ipc(struct repository *r);\n-- \ngitgitgadget\n\n"},{"id":"456245","messageId":"7e302958cd4cc7b229e2e7e895e2af25b3051d19.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 13/30] fsmonitor--daemon: cd out of worktree root","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:07Z","receivedAt":"2022-05-26T21:49:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the fsmonitor--daemon to CD outside of the worktree\nbefore starting up.\n\nThe common Git startup mechanism causes the CWD of the daemon process\nto be in the root of the worktree. On Windows, this causes the daemon\nprocess to hold a locked handle on the CWD and prevents other\nprocesses from moving or deleting the worktree while the daemon is\nrunning.\n\nCD to HOME before entering main event loops.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++--\n compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------\n fsmonitor--daemon.h | 1 +\n 3 files changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 66b78a0353f..db297649daf 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1181,11 +1181,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * before we need it.\n \t */\n \tif (ipc_server_run_async(&state->ipc_server_data,\n-\t\t\t\t fsmonitor_ipc__get_path(), &ipc_opts,\n+\t\t\t\t state->path_ipc.buf, &ipc_opts,\n \t\t\t\t handle_client, state))\n \t\treturn error_errno(\n \t\t\t_(\"could not start IPC thread pool on '%s'\"),\n-\t\t\tfsmonitor_ipc__get_path());\n+\t\t\tstate->path_ipc.buf);\n \n \t/*\n \t * Start the fsmonitor listener thread to collect filesystem\n@@ -1220,6 +1220,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n static int fsmonitor_run_daemon(void)\n {\n \tstruct fsmonitor_daemon_state state;\n+\tconst char *home;\n \tint err;\n \n \tmemset(&state, 0, sizeof(state));\n@@ -1289,6 +1290,15 @@ static int fsmonitor_run_daemon(void)\n \n \tstrbuf_addch(&state.path_cookie_prefix, '/');\n \n+\t/*\n+\t * We create a named-pipe or unix domain socket inside of the\n+\t * \".git\" directory. (Well, on Windows, we base our named\n+\t * pipe in the NPFS on the absolute path of the git\n+\t * directory.)\n+\t */\n+\tstrbuf_init(&state.path_ipc, 0);\n+\tstrbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));\n+\n \t/*\n \t * Confirm that we can create platform-specific resources for the\n \t * filesystem listener before we bother starting all the threads.\n@@ -1298,6 +1308,23 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\t/*\n+\t * CD out of the worktree root directory.\n+\t *\n+\t * The common Git startup mechanism causes our CWD to be the\n+\t * root of the worktree. On Windows, this causes our process\n+\t * to hold a locked handle on the CWD. This prevents the\n+\t * worktree from being moved or deleted while the daemon is\n+\t * running.\n+\t *\n+\t * We assume that our FS and IPC listener threads have either\n+\t * opened all of the handles that they need or will do\n+\t * everything using absolute paths.\n+\t */\n+\thome = getenv(\"HOME\");\n+\tif (home && *home && chdir(home))\n+\t\tdie_errno(_(\"could not cd home '%s'\"), home);\n+\n \terr = fsmonitor_run_daemon_1(&state);\n \n done:\n@@ -1310,6 +1337,7 @@ done:\n \tstrbuf_release(&state.path_worktree_watch);\n \tstrbuf_release(&state.path_gitdir_watch);\n \tstrbuf_release(&state.path_cookie_prefix);\n+\tstrbuf_release(&state.path_ipc);\n \n \treturn err;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 4f46bd1d0a6..35f2fb95784 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -424,12 +424,22 @@ static int recv_rdcw_watch(struct one_watch *watch)\n \t}\n \n \t/*\n-\t * NEEDSWORK: If an external is deleted, the above\n-\t * returns an error. I'm not sure that there's anything that\n-\t * we can do here other than failing -- the /.git\n-\t * link file would be broken anyway. We might try to check\n-\t * for that and return a better error message, but I'm not\n-\t * sure it is worth it.\n+\t * GetOverlappedResult() fails if the watched directory is\n+\t * deleted while we were waiting for an overlapped IO to\n+\t * complete. The documentation did not list specific errors,\n+\t * but I observed ERROR_ACCESS_DENIED (0x05) errors during\n+\t * testing.\n+\t *\n+\t * Note that we only get notificaiton events for events\n+\t * *within* the directory, not *on* the directory itself.\n+\t * (These might be properies of the parent directory, for\n+\t * example).\n+\t *\n+\t * NEEDSWORK: We might try to check for the deleted directory\n+\t * case and return a better error message, but I'm not sure it\n+\t * is worth it.\n+\t *\n+\t * Shutdown if we get any error.\n \t */\n \n \terror(_(\"GetOverlappedResult failed on '%s' [GLE %ld]\"),\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex bd09fffc176..223c2131b58 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {\n \tstruct fsmonitor_daemon_backend_data *backend_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n+\tstruct strbuf path_ipc;\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"456246","messageId":"27f360b33364d997ee7e31b0b0c6704fad088616.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 11/30] unpack-trees: initialize fsmonitor_has_run_once in o->result","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:05Z","receivedAt":"2022-05-26T21:49:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nInitialize `o->result.fsmonitor_has_run_once` based upon value\nin `o->src_index->fsmonitor_has_run_once` to prevent a second\nfsmonitor query during the tree traversal and possibly getting\na skewed view of the working directory.\n\nThe checkout code has already talked to the fsmonitor and the\ntraversal is updating the index as it traverses, so there is\nno need to query the fsmonitor.\n\nSigned-off-by: Jeff Hostetler \n---\n unpack-trees.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..888cff81f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \to->result.fsmonitor_last_update =\n \t\txstrdup_or_null(o->src_index->fsmonitor_last_update);\n+\to->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;\n \n \t/*\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n-- \ngitgitgadget\n\n"},{"id":"456247","messageId":"ad8f65efe0d773fa2f14423276a8aaef778df993.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 12/30] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:06Z","receivedAt":"2022-05-26T21:49:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIgnore FSEvents resulting from `xattr` changes. Git does not care about\nxattr's or changes to xattr's, so don't waste time collecting these\nevents in the daemon nor transmitting them to clients.\n\nVarious security tools add xattrs to files and/or directories, such as\nto mark them as having been downloaded. We should ignore these events\nsince it doesn't affect the content of the file/directory or the normal\nmeta-data that Git cares about.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 0741fe834c3..14105f45c18 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tif (flag & kFSEventStreamEventFlagItemCloned)\n \t\tstrbuf_addstr(&msg, \"ItemCloned|\");\n \n-\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=%u %s\",\n+\ttrace_printf_key(&trace_fsmonitor, \"fsevent: '%s', flags=0x%x %s\",\n \t\t\t path, flag, msg.buf);\n \n \tstrbuf_release(&msg);\n@@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)\n \t\tef & kFSEventStreamEventFlagUserDropped);\n }\n \n+/*\n+ * If an `xattr` change is the only reason we received this event,\n+ * then silently ignore it. Git doesn't care about xattr's. We\n+ * have to be careful here because the kernel can combine multiple\n+ * events for a single path. And because events always have certain\n+ * bits set, such as `ItemIsFile` or `ItemIsDir`.\n+ *\n+ * Return 1 if we should ignore it.\n+ */\n+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n+{\n+\tstatic const FSEventStreamEventFlags mask =\n+\t\tkFSEventStreamEventFlagItemChangeOwner |\n+\t\tkFSEventStreamEventFlagItemCreated |\n+\t\tkFSEventStreamEventFlagItemFinderInfoMod |\n+\t\tkFSEventStreamEventFlagItemInodeMetaMod |\n+\t\tkFSEventStreamEventFlagItemModified |\n+\t\tkFSEventStreamEventFlagItemRemoved |\n+\t\tkFSEventStreamEventFlagItemRenamed |\n+\t\tkFSEventStreamEventFlagItemXattrMod |\n+\t\tkFSEventStreamEventFlagItemCloned;\n+\n+\treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n+}\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_ignore_xattr(event_flags[k])) {\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n+\t\t\t\t\t path_k, event_flags[k]);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tswitch (fsmonitor_classify_path_absolute(state, path_k)) {\n \n \t\tcase IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:\n-- \ngitgitgadget\n\n"},{"id":"456248","messageId":"df1061c0ff5bd8843b3f05344014955b299f29f1.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 15/30] fsmonitor--daemon: rename listener thread related variables","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:09Z","receivedAt":"2022-05-26T21:49:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRename platform-specific listener thread related variables\nand data types as we prepare to add another backend thread\ntype.\n\n[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`\n[] `state->backend_data` becomes `state->listen_data`\n[] `state->error_code` becomes `state->listen_error_code`\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 6 +++---\n compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------\n compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++-------------\n compat/fsmonitor/fsm-listen.h | 2 +-\n fsmonitor--daemon.h | 6 +++---\n 5 files changed, 36 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex 90fa9d09efb..b2f578b239a 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1225,8 +1225,8 @@ cleanup:\n \n \tif (err)\n \t\treturn err;\n-\tif (state->error_code)\n-\t\treturn state->error_code;\n+\tif (state->listen_error_code)\n+\t\treturn state->listen_error_code;\n \treturn 0;\n }\n \n@@ -1241,7 +1241,7 @@ static int fsmonitor_run_daemon(void)\n \thashmap_init(&state.cookies, cookies_cmp, NULL, 0);\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n-\tstate.error_code = 0;\n+\tstate.listen_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 14105f45c18..07113205a61 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -27,7 +27,7 @@\n #include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tCFStringRef cfsr_worktree_path;\n \tCFStringRef cfsr_gitdir_path;\n@@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t const FSEventStreamEventId event_ids[])\n {\n \tstruct fsmonitor_daemon_state *state = ctx;\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tchar **paths = (char **)event_paths;\n \tstruct fsmonitor_batch *batch = NULL;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tNULL,\n \t\tNULL\n \t};\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \tconst void *dir_array[2];\n \n \tCALLOC_ARRAY(data, 1);\n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \n \tdata->cfsr_worktree_path = CFStringCreateWithCString(\n \t\tNULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);\n@@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n failed:\n \terror(_(\"Unable to create FSEventStream.\"));\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n \treturn -1;\n }\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tif (data->stream) {\n \t\tif (data->stream_started)\n@@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n \t\tFSEventStreamRelease(data->stream);\n \t}\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \tdata->shutdown_style = SHUTDOWN_EVENT;\n \n \tCFRunLoopStop(data->rl);\n@@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tdata->rl = CFRunLoopGetCurrent();\n \n@@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \n \tswitch (data->shutdown_style) {\n \tcase FORCE_ERROR_STOP:\n-\t\tstate->error_code = -1;\n+\t\tstate->listen_error_code = -1;\n \t\t/* fall thru */\n \tcase FORCE_SHUTDOWN:\n \t\tipc_server_stop_async(state->ipc_server_data);\n@@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \treturn;\n \n force_error_stop_without_loop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \tipc_server_stop_async(state->ipc_server_data);\n \treturn;\n }\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 35f2fb95784..03df8d951b8 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -54,7 +54,7 @@ struct one_watch\n \twchar_t dotgit_shortname[16]; /* for 8.3 name */\n };\n \n-struct fsmonitor_daemon_backend_data\n+struct fsm_listen_data\n {\n \tstruct one_watch *watch_worktree;\n \tstruct one_watch *watch_gitdir;\n@@ -284,7 +284,7 @@ static enum get_relative_result get_relative_longname(\n \n void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)\n {\n-\tSetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);\n+\tSetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);\n }\n \n static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,\n@@ -359,7 +359,7 @@ static void destroy_watch(struct one_watch *watch)\n \tfree(watch);\n }\n \n-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,\n+static int start_rdcw_watch(struct fsm_listen_data *data,\n \t\t\t struct one_watch *watch)\n {\n \tDWORD dwNotifyFilter =\n@@ -538,7 +538,7 @@ static int process_1_worktree_event(\n */\n static int process_worktree_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_worktree;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -669,7 +669,7 @@ force_shutdown:\n */\n static int process_gitdir_events(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tstruct one_watch *watch = data->watch_gitdir;\n \tstruct strbuf path = STRBUF_INIT;\n \tstruct string_list cookie_list = STRING_LIST_INIT_DUP;\n@@ -727,11 +727,11 @@ skip_this_path:\n \n void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data = state->backend_data;\n+\tstruct fsm_listen_data *data = state->listen_data;\n \tDWORD dwWait;\n \tint result;\n \n-\tstate->error_code = 0;\n+\tstate->listen_error_code = 0;\n \n \tif (start_rdcw_watch(data, data->watch_worktree) == -1)\n \t\tgoto force_error_stop;\n@@ -796,7 +796,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)\n \t}\n \n force_error_stop:\n-\tstate->error_code = -1;\n+\tstate->listen_error_code = -1;\n \n force_shutdown:\n \t/*\n@@ -813,7 +813,7 @@ clean_shutdown:\n \n int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n \n@@ -846,7 +846,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)\n \t\tdata->nr_listener_handles++;\n \t}\n \n-\tstate->backend_data = data;\n+\tstate->listen_data = data;\n \treturn 0;\n \n failed:\n@@ -859,16 +859,16 @@ failed:\n \n void fsm_listen__dtor(struct fsmonitor_daemon_state *state)\n {\n-\tstruct fsmonitor_daemon_backend_data *data;\n+\tstruct fsm_listen_data *data;\n \n-\tif (!state || !state->backend_data)\n+\tif (!state || !state->listen_data)\n \t\treturn;\n \n-\tdata = state->backend_data;\n+\tdata = state->listen_data;\n \n \tCloseHandle(data->hEventShutdown);\n \tdestroy_watch(data->watch_worktree);\n \tdestroy_watch(data->watch_gitdir);\n \n-\tFREE_AND_NULL(state->backend_data);\n+\tFREE_AND_NULL(state->listen_data);\n }\ndiff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h\nindex f0539349baf..41650bf8972 100644\n--- a/compat/fsmonitor/fsm-listen.h\n+++ b/compat/fsmonitor/fsm-listen.h\n@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);\n * do so if the listener thread receives a normal shutdown signal from\n * the IPC layer.)\n *\n- * It should set `state->error_code` to -1 if the daemon should exit\n+ * It should set `state->listen_error_code` to -1 if the daemon should exit\n * with an error.\n */\n void fsm_listen__loop(struct fsmonitor_daemon_state *state);\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 223c2131b58..2c6fa1a5d91 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n */\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */\n+struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {\n \tint cookie_seq;\n \tstruct hashmap cookies;\n \n-\tint error_code;\n-\tstruct fsmonitor_daemon_backend_data *backend_data;\n+\tint listen_error_code;\n+\tstruct fsm_listen_data *listen_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"456249","messageId":"6b271866f67ef6c7f3ffe76914f950c81aae053c.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 14/30] fsmonitor--daemon: prepare for adding health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:08Z","receivedAt":"2022-05-26T21:50:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor daemon thread startup to make it easier to start\na third thread class to monitor the health of the daemon.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------\n 1 file changed, 20 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex db297649daf..90fa9d09efb 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -1174,6 +1174,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint listener_started = 0;\n+\tint err = 0;\n \n \t/*\n \t * Start the IPC thread pool before the we've started the file\n@@ -1194,15 +1196,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \tif (pthread_create(&state->listener_thread, NULL,\n \t\t\t fsm_listen__thread_proc, state) < 0) {\n \t\tipc_server_stop_async(state->ipc_server_data);\n-\t\tipc_server_await(state->ipc_server_data);\n-\n-\t\treturn error(_(\"could not start fsmonitor listener thread\"));\n+\t\terr = error(_(\"could not start fsmonitor listener thread\"));\n+\t\tgoto cleanup;\n \t}\n+\tlistener_started = 1;\n \n \t/*\n \t * The daemon is now fully functional in background threads.\n+\t * Our primary thread should now just wait while the threads\n+\t * do all the work.\n+\t */\n+cleanup:\n+\t/*\n \t * Wait for the IPC thread pool to shutdown (whether by client\n-\t * request or from filesystem activity).\n+\t * request, from filesystem activity, or an error).\n \t */\n \tipc_server_await(state->ipc_server_data);\n \n@@ -1211,10 +1218,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t * event from the IPC thread pool, but it doesn't hurt to tell\n \t * it again. And wait for it to shutdown.\n \t */\n-\tfsm_listen__stop_async(state);\n-\tpthread_join(state->listener_thread, NULL);\n+\tif (listener_started) {\n+\t\tfsm_listen__stop_async(state);\n+\t\tpthread_join(state->listener_thread, NULL);\n+\t}\n \n-\treturn state->error_code;\n+\tif (err)\n+\t\treturn err;\n+\tif (state->error_code)\n+\t\treturn state->error_code;\n+\treturn 0;\n }\n \n static int fsmonitor_run_daemon(void)\n-- \ngitgitgadget\n\n"},{"id":"456250","messageId":"f710d305dd41eb14039dc45fbe58d20e7427af66.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 17/30] fsm-health-win32: add polling framework to monitor daemon health","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:11Z","receivedAt":"2022-05-26T21:50:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nExtend the Windows version of the \"health\" thread to periodically\ninspect the system and shutdown if warranted.\n\nThis commit updates the thread's wait loop to use a timeout and\ndefines a (currently empty) table of functions to poll the system.\n\nA later commit will add functions to the table to actually\ninspect the system.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-\n 1 file changed, 64 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 94b1d020f25..24fc612bf02 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -4,6 +4,24 @@\n #include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n+/*\n+ * Every minute wake up and test our health.\n+ */\n+#define WAIT_FREQ_MS (60 * 1000)\n+\n+/*\n+ * State machine states for each of the interval functions\n+ * used for polling our health.\n+ */\n+enum interval_fn_ctx {\n+\tCTX_INIT = 0,\n+\tCTX_TERM,\n+\tCTX_TIMER\n+};\n+\n+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx);\n+\n struct fsm_health_data\n {\n \tHANDLE hEventShutdown;\n@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n \tFREE_AND_NULL(state->health_data);\n }\n \n+/*\n+ * A table of the polling functions.\n+ */\n+static interval_fn *table[] = {\n+\tNULL, /* must be last */\n+};\n+\n+/*\n+ * Call all of the polling functions in the table.\n+ * Shortcut and return first error.\n+ *\n+ * Return 0 if all succeeded.\n+ */\n+static int call_all(struct fsmonitor_daemon_state *state,\n+\t\t enum interval_fn_ctx ctx)\n+{\n+\tint k;\n+\n+\tfor (k = 0; table[k]; k++) {\n+\t\tint r = table[k](state, ctx);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void fsm_health__loop(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data = state->health_data;\n+\tint r;\n+\n+\tr = call_all(state, CTX_INIT);\n+\tif (r < 0)\n+\t\tgoto force_error_stop;\n+\tif (r > 0)\n+\t\tgoto force_shutdown;\n \n \tfor (;;) {\n \t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n \t\t\t\t\t\t data->hHandles,\n-\t\t\t\t\t\t FALSE, INFINITE);\n+\t\t\t\t\t\t FALSE, WAIT_FREQ_MS);\n \n \t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n \t\t\tgoto clean_shutdown;\n \n+\t\tif (dwWait == WAIT_TIMEOUT) {\n+\t\t\tr = call_all(state, CTX_TIMER);\n+\t\t\tif (r < 0)\n+\t\t\t\tgoto force_error_stop;\n+\t\t\tif (r > 0)\n+\t\t\t\tgoto force_shutdown;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\terror(_(\"health thread wait failed [GLE %ld]\"),\n \t\t GetLastError());\n \t\tgoto force_error_stop;\n@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)\n \n force_error_stop:\n \tstate->health_error_code = -1;\n+force_shutdown:\n \tipc_server_stop_async(state->ipc_server_data);\n clean_shutdown:\n+\tcall_all(state, CTX_TERM);\n \treturn;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"456251","messageId":"6976f1d45eacf03bd0936c80d875b4dc8b745879.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 21/30] t7527: FSMonitor tests for directory moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:15Z","receivedAt":"2022-05-26T21:50:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate unit tests to move a directory. Verify that `git status`\ngives the same result with and without FSMonitor enabled.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 12655958e71..3bc335b891d 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -274,6 +274,16 @@ test_expect_success 'setup' '\n \ttrace*\n \tEOF\n \n+\tmkdir -p T1/T2/T3/T4 &&\n+\techo 1 >T1/F1 &&\n+\techo 1 >T1/T2/F1 &&\n+\techo 1 >T1/T2/T3/F1 &&\n+\techo 1 >T1/T2/T3/T4/F1 &&\n+\techo 2 >T1/F2 &&\n+\techo 2 >T1/T2/F2 &&\n+\techo 2 >T1/T2/T3/F2 &&\n+\techo 2 >T1/T2/T3/T4/F2 &&\n+\n \tgit -c core.fsmonitor=false add . &&\n \ttest_tick &&\n \tgit -c core.fsmonitor=false commit -m initial &&\n@@ -356,6 +366,19 @@ directory_to_file () {\n \techo 1 >dir1\n }\n \n+move_directory_contents_deeper() {\n+\tmkdir T1/_new_ &&\n+\tmv T1/[A-Z]* T1/_new_\n+}\n+\n+move_directory_up() {\n+\tmv T1/T2/T3 T1\n+}\n+\n+move_directory() {\n+\tmv T1/T2/T3 T1/T2/NewT3\n+}\n+\n # The next few test cases confirm that our fsmonitor daemon sees each type\n # of OS filesystem notification that we care about. At this layer we just\n # ensure we are getting the OS notifications and do not try to confirm what\n@@ -660,6 +683,10 @@ do\n \t\tmatrix_try $uc_val $fsm_val file_to_directory\n \t\tmatrix_try $uc_val $fsm_val directory_to_file\n \n+\t\tmatrix_try $uc_val $fsm_val move_directory_contents_deeper\n+\t\tmatrix_try $uc_val $fsm_val move_directory_up\n+\t\tmatrix_try $uc_val $fsm_val move_directory\n+\n \t\tif test $fsm_val = true\n \t\tthen\n \t\t\ttest_expect_success \"Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end\" '\n-- \ngitgitgadget\n\n"},{"id":"456252","messageId":"8fa7f4fa9bebc7cfb8c26f54ed8076ade4015326.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 20/30] fsmonitor: optimize processing of directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:14Z","receivedAt":"2022-05-26T21:50:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach Git to perform binary search over the cache-entries for a directory\nnotification and then linearly scan forward to find the immediate children.\n\nPreviously, when the FSMonitor reported a modified directory Git would\nperform a linear search on the entire cache-entry array for all\nentries matching that directory prefix and invalidate them. Since the\ncache-entry array is already sorted, we can use a binary search to\nfind the first matching entry and then only linearly walk forward and\ninvalidate entries until the prefix changes.\n\nAlso, the original code would invalidate anything having the same\ndirectory prefix. Since a directory event should only be received for\nitems that are immediately within the directory (and not within\nsub-directories of it), only invalidate those entries and not the\nwhole subtree.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 54 insertions(+), 17 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 292a6742b4f..e1229c289cf 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n-\tif (name[len - 1] == '/') {\n+\tint pos = index_name_pos(istate, name, len);\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n+\t\t\t name, pos);\n \n+\tif (name[len - 1] == '/') {\n \t\t/*\n-\t\t * TODO We should binary search to find the first path with\n-\t\t * TODO this directory prefix. Then linearly update entries\n-\t\t * TODO while the prefix matches. Taking care to search without\n-\t\t * TODO the trailing slash -- because '/' sorts after a few\n-\t\t * TODO interesting special chars, like '.' and ' '.\n+\t\t * The daemon can decorate directory events, such as\n+\t\t * moves or renames, with a trailing slash if the OS\n+\t\t * FS Event contains sufficient information, such as\n+\t\t * MacOS.\n+\t\t *\n+\t\t * Use this to invalidate the entire cone under that\n+\t\t * directory.\n+\t\t *\n+\t\t * We do not expect an exact match because the index\n+\t\t * does not normally contain directory entries, so we\n+\t\t * start at the insertion point and scan.\n \t\t */\n+\t\tif (pos < 0)\n+\t\t\tpos = -pos - 1;\n \n \t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\t\tif (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&\n-\t\t\t starts_with(istate->cache[i]->name, name))\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n-\t\t/* Need to remove the / from the path for the untracked cache */\n+\n+\t\t/*\n+\t\t * We need to remove the traling \"/\" from the path\n+\t\t * for the untracked cache.\n+\t\t */\n \t\tname[len - 1] = '\\0';\n+\t} else if (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\tint pos = index_name_pos(istate, name, strlen(name));\n-\n-\t\tif (pos >= 0) {\n-\t\t\tstruct cache_entry *ce = istate->cache[pos];\n-\t\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t\t}\n \t}\n \n@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n \t * as it could be a new untracked file.\n \t */\n-\ttrace_printf_key(&trace_fsmonitor, \"fsmonitor_refresh_callback '%s'\", name);\n \tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n-- \ngitgitgadget\n\n"},{"id":"456253","messageId":"e255a5b710479ff85c3ce2ce54bb323f7b4b594e.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 16/30] fsmonitor--daemon: stub in health thread","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:10Z","receivedAt":"2022-05-26T21:50:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate another thread to watch over the daemon process and\nautomatically shut it down if necessary.\n\nThis commit creates the basic framework for a \"health\" thread\nto monitor the daemon and/or the file system. Later commits\nwill add platform-specific code to do the actual work.\n\nThe \"health\" thread is intended to monitor conditions that\nwould be difficult to track inside the IPC thread pool and/or\nthe file system listener threads. For example, when there are\nfile system events outside of the watched worktree root or if\nwe want to have an idle-timeout auto-shutdown feature.\n\nThis commit creates the health thread itself, defines the thread-proc\nand sets up the thread's event loop. It integrates this new thread\ninto the existing IPC and Listener thread models.\n\nThis commit defines the API to the platform-specific code where all of\nthe monitoring will actually happen.\n\nThe platform-specific code for MacOS is just stubs. Meaning that the\nhealth thread will immediately exit on MacOS, but that is OK and\nexpected. Future work can define MacOS-specific monitoring.\n\nThe platform-specific code for Windows sets up enough of the\nWaitForMultipleObjects() machinery to watch for system and/or custom\nevents. Currently, the set of wait handles only includes our custom\nshutdown event (sent from our other theads). Later commits in this\nseries will extend the set of wait handles to monitor other\nconditions.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 6 ++-\n builtin/fsmonitor--daemon.c | 39 +++++++++++++++\n compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++\n compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++\n compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++\n contrib/buildsystems/CMakeLists.txt | 2 +\n fsmonitor--daemon.h | 4 ++\n 7 files changed, 192 insertions(+), 2 deletions(-)\n create mode 100644 compat/fsmonitor/fsm-health-darwin.c\n create mode 100644 compat/fsmonitor/fsm-health-win32.c\n create mode 100644 compat/fsmonitor/fsm-health.h\n\ndiff --git a/Makefile b/Makefile\nindex 93604fe8ef7..5f1623baadd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -472,8 +472,9 @@ all::\n #\n # If your platform supports a built-in fsmonitor backend, set\n # FSMONITOR_DAEMON_BACKEND to the \"\" of the corresponding\n-# `compat/fsmonitor/fsm-listen-.c` that implements the\n-# `fsm_listen__*()` routines.\n+# `compat/fsmonitor/fsm-listen-.c` and\n+# `compat/fsmonitor/fsm-health-.c` files\n+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.\n #\n # If your platform has OS-specific ways to tell if a repo is incompatible with\n # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS\n@@ -1982,6 +1983,7 @@ endif\n ifdef FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND\n \tCOMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o\n+\tCOMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o\n endif\n \n ifdef FSMONITOR_OS_SETTINGS\ndiff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c\nindex b2f578b239a..2c109cf8b37 100644\n--- a/builtin/fsmonitor--daemon.c\n+++ b/builtin/fsmonitor--daemon.c\n@@ -3,6 +3,7 @@\n #include \"parse-options.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"simple-ipc.h\"\n@@ -1136,6 +1137,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,\n \tpthread_mutex_unlock(&state->main_lock);\n }\n \n+static void *fsm_health__thread_proc(void *_state)\n+{\n+\tstruct fsmonitor_daemon_state *state = _state;\n+\n+\ttrace2_thread_start(\"fsm-health\");\n+\n+\tfsm_health__loop(state);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n static void *fsm_listen__thread_proc(void *_state)\n {\n \tstruct fsmonitor_daemon_state *state = _state;\n@@ -1174,6 +1187,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t\t */\n \t\t.uds_disallow_chdir = 0\n \t};\n+\tint health_started = 0;\n \tint listener_started = 0;\n \tint err = 0;\n \n@@ -1201,6 +1215,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)\n \t}\n \tlistener_started = 1;\n \n+\t/*\n+\t * Start the health thread to watch over our process.\n+\t */\n+\tif (pthread_create(&state->health_thread, NULL,\n+\t\t\t fsm_health__thread_proc, state) < 0) {\n+\t\tipc_server_stop_async(state->ipc_server_data);\n+\t\terr = error(_(\"could not start fsmonitor health thread\"));\n+\t\tgoto cleanup;\n+\t}\n+\thealth_started = 1;\n+\n \t/*\n \t * The daemon is now fully functional in background threads.\n \t * Our primary thread should now just wait while the threads\n@@ -1223,10 +1248,17 @@ cleanup:\n \t\tpthread_join(state->listener_thread, NULL);\n \t}\n \n+\tif (health_started) {\n+\t\tfsm_health__stop_async(state);\n+\t\tpthread_join(state->health_thread, NULL);\n+\t}\n+\n \tif (err)\n \t\treturn err;\n \tif (state->listen_error_code)\n \t\treturn state->listen_error_code;\n+\tif (state->health_error_code)\n+\t\treturn state->health_error_code;\n \treturn 0;\n }\n \n@@ -1242,6 +1274,7 @@ static int fsmonitor_run_daemon(void)\n \tpthread_mutex_init(&state.main_lock, NULL);\n \tpthread_cond_init(&state.cookies_cond, NULL);\n \tstate.listen_error_code = 0;\n+\tstate.health_error_code = 0;\n \tstate.current_token_data = fsmonitor_new_token_data();\n \n \t/* Prepare to (recursively) watch the directory. */\n@@ -1321,6 +1354,11 @@ static int fsmonitor_run_daemon(void)\n \t\tgoto done;\n \t}\n \n+\tif (fsm_health__ctor(&state)) {\n+\t\terr = error(_(\"could not initialize health thread\"));\n+\t\tgoto done;\n+\t}\n+\n \t/*\n \t * CD out of the worktree root directory.\n \t *\n@@ -1344,6 +1382,7 @@ done:\n \tpthread_cond_destroy(&state.cookies_cond);\n \tpthread_mutex_destroy(&state.main_lock);\n \tfsm_listen__dtor(&state);\n+\tfsm_health__dtor(&state);\n \n \tipc_server_free(state.ipc_server_data);\n \ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nnew file mode 100644\nindex 00000000000..b9f709e8548\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -0,0 +1,24 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+}\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nnew file mode 100644\nindex 00000000000..94b1d020f25\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"fsmonitor.h\"\n+#include \"fsm-health.h\"\n+#include \"fsmonitor--daemon.h\"\n+\n+struct fsm_health_data\n+{\n+\tHANDLE hEventShutdown;\n+\n+\tHANDLE hHandles[1]; /* the array does not own these handles */\n+#define HEALTH_SHUTDOWN 0\n+\tint nr_handles; /* number of active event handles */\n+};\n+\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tCALLOC_ARRAY(data, 1);\n+\n+\tdata->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);\n+\n+\tdata->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;\n+\tdata->nr_handles++;\n+\n+\tstate->health_data = data;\n+\treturn 0;\n+}\n+\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data;\n+\n+\tif (!state || !state->health_data)\n+\t\treturn;\n+\n+\tdata = state->health_data;\n+\n+\tCloseHandle(data->hEventShutdown);\n+\n+\tFREE_AND_NULL(state->health_data);\n+}\n+\n+void fsm_health__loop(struct fsmonitor_daemon_state *state)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\n+\tfor (;;) {\n+\t\tDWORD dwWait = WaitForMultipleObjects(data->nr_handles,\n+\t\t\t\t\t\t data->hHandles,\n+\t\t\t\t\t\t FALSE, INFINITE);\n+\n+\t\tif (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)\n+\t\t\tgoto clean_shutdown;\n+\n+\t\terror(_(\"health thread wait failed [GLE %ld]\"),\n+\t\t GetLastError());\n+\t\tgoto force_error_stop;\n+\t}\n+\n+force_error_stop:\n+\tstate->health_error_code = -1;\n+\tipc_server_stop_async(state->ipc_server_data);\n+clean_shutdown:\n+\treturn;\n+}\n+\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)\n+{\n+\tSetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);\n+}\ndiff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h\nnew file mode 100644\nindex 00000000000..45547ba9380\n--- /dev/null\n+++ b/compat/fsmonitor/fsm-health.h\n@@ -0,0 +1,47 @@\n+#ifndef FSM_HEALTH_H\n+#define FSM_HEALTH_H\n+\n+/* This needs to be implemented by each backend */\n+\n+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND\n+\n+struct fsmonitor_daemon_state;\n+\n+/*\n+ * Initialize platform-specific data for the fsmonitor health thread.\n+ * This will be called from the main thread PRIOR to staring the\n+ * thread.\n+ *\n+ * Returns 0 if successful.\n+ * Returns -1 otherwise.\n+ */\n+int fsm_health__ctor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Cleanup platform-specific data for the health thread.\n+ * This will be called from the main thread AFTER joining the thread.\n+ */\n+void fsm_health__dtor(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * The main body of the platform-specific event loop to monitor the\n+ * health of the daemon process. This will run in the health thread.\n+ *\n+ * The health thread should call `ipc_server_stop_async()` if it needs\n+ * to cause a shutdown. (It should NOT do so if it receives a shutdown\n+ * shutdown signal.)\n+ *\n+ * It should set `state->health_error_code` to -1 if the daemon should exit\n+ * with an error.\n+ */\n+void fsm_health__loop(struct fsmonitor_daemon_state *state);\n+\n+/*\n+ * Gently request that the health thread shutdown.\n+ * It does not wait for it to stop. The caller should do a JOIN\n+ * to wait for it.\n+ */\n+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);\n+\n+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */\n+#endif /* FSM_HEALTH_H */\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex b8f9f7a0388..16ace43d1c7 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)\n \tif(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)\n \telseif(CMAKE_SYSTEM_NAME STREQUAL \"Darwin\")\n \t\tadd_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)\n+\t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)\n \n \t\tadd_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)\n \t\tlist(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)\ndiff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h\nindex 2c6fa1a5d91..2102a5c9ff5 100644\n--- a/fsmonitor--daemon.h\n+++ b/fsmonitor--daemon.h\n@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);\n void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);\n \n struct fsm_listen_data; /* opaque platform-specific data for listener thread */\n+struct fsm_health_data; /* opaque platform-specific data for health thread */\n \n struct fsmonitor_daemon_state {\n \tpthread_t listener_thread;\n+\tpthread_t health_thread;\n \tpthread_mutex_t main_lock;\n \n \tstruct strbuf path_worktree_watch;\n@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {\n \tstruct hashmap cookies;\n \n \tint listen_error_code;\n+\tint health_error_code;\n \tstruct fsm_listen_data *listen_data;\n+\tstruct fsm_health_data *health_data;\n \n \tstruct ipc_server_data *ipc_server_data;\n \tstruct strbuf path_ipc;\n-- \ngitgitgadget\n\n"},{"id":"456254","messageId":"81559df45ab6283bc0ee69ac252fd3691f057ed1.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 22/30] t/perf/p7527: add perf test for builtin FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:16Z","receivedAt":"2022-05-26T21:50:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++\n 1 file changed, 257 insertions(+)\n create mode 100755 t/perf/p7527-builtin-fsmonitor.sh\n\ndiff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh\nnew file mode 100755\nindex 00000000000..9338b9ea008\n--- /dev/null\n+++ b/t/perf/p7527-builtin-fsmonitor.sh\n@@ -0,0 +1,257 @@\n+#!/bin/sh\n+\n+test_description=\"Perf test for the builtin FSMonitor\"\n+\n+. ./perf-lib.sh\n+\n+if ! test_have_prereq FSMONITOR_DAEMON\n+then\n+\tskip_all=\"fsmonitor--daemon is not supported on this platform\"\n+\ttest_done\n+fi\n+\n+test_lazy_prereq UNTRACKED_CACHE '\n+\t{ git update-index --test-untracked-cache; ret=$?; } &&\n+\ttest $ret -ne 1\n+'\n+\n+# Lie to perf-lib and ask for a new empty repo and avoid\n+# the complaints about GIT_PERF_REPO not being big enough\n+# the perf hit when GIT_PERF_LARGE_REPO is copied into\n+# the trash directory.\n+#\n+# NEEDSWORK: It would be nice if perf-lib had an option to\n+# \"borrow\" an existing large repo (especially for gigantic\n+# monorepos) and use it in-place. For now, fake it here.\n+#\n+test_perf_fresh_repo\n+\n+\n+# Use a generated synthetic monorepo. If it doesn't exist, we will\n+# generate it. If it does exist, we will put it in a known state\n+# before we start our timings.\n+#\n+PARAM_D=5\n+PARAM_W=10\n+PARAM_F=9\n+\n+PARAMS=\"$PARAM_D\".\"$PARAM_W\".\"$PARAM_F\"\n+\n+BALLAST_BR=p0006-ballast\n+export BALLAST_BR\n+\n+TMP_BR=tmp_br\n+export TMP_BR\n+\n+REPO=../repos/gen-many-files-\"$PARAMS\".git\n+export REPO\n+\n+if ! test -d $REPO\n+then\n+\t(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)\n+fi\n+\n+\n+enable_uc () {\n+\tgit -C $REPO config core.untrackedcache true\n+\tgit -C $REPO update-index --untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+disable_uc () {\n+\tgit -C $REPO config core.untrackedcache false\n+\tgit -C $REPO update-index --no-untracked-cache\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+start_fsm () {\n+\tgit -C $REPO fsmonitor--daemon start\n+\tgit -C $REPO fsmonitor--daemon status\n+\tgit -C $REPO config core.fsmonitor true\n+\tgit -C $REPO update-index --fsmonitor\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+stop_fsm () {\n+\tgit -C $REPO config --unset core.fsmonitor\n+\tgit -C $REPO update-index --no-fsmonitor\n+\ttest_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null\n+\tgit -C $REPO status >/dev/null 2>&1\n+}\n+\n+\n+# Ensure that FSMonitor is turned off on the borrowed repo.\n+#\n+test_expect_success \"Setup borrowed repo (fsm+uc)\" \"\n+\tstop_fsm &&\n+\tdisable_uc\n+\"\n+\n+# Also ensure that it starts in a known state.\n+#\n+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time\n+# the ballast checkout, since only the first invocation does any work and the\n+# subsequent ones just print \"already on branch\" and quit, so the reported\n+# time is not useful.\n+#\n+# Create a temp branch and do all work relative to it so that we don't\n+# accidentially alter the real ballast branch.\n+#\n+test_expect_success \"Setup borrowed repo (temp ballast branch)\" \"\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO reset --hard &&\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tgit -C $REPO branch $TMP_BR $BALLAST_BR &&\n+\tgit -C $REPO checkout $TMP_BR\n+\"\n+\n+\n+echo Data >data.txt\n+\n+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With\n+# FSMonitor enabled, we can get a skewed view of status times, since\n+# the index MAY (or may not) be updated after the first invocation\n+# which will update the FSMonitor Token, so the subsequent invocations\n+# may get a smaller response from the daemon.\n+#\n+do_status () {\n+\tmsg=$1\n+\n+\ttest_perf \"$msg\" \"\n+\t\tgit -C $REPO status >/dev/null 2>&1\n+\t\"\n+}\n+\n+do_matrix () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tt=\"[uc $uc][fsm $fsm]\"\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_expect_success \"$t Setup matrix branch\" \"\n+\t\tgit -C $REPO clean -d -f &&\n+\t\tgit -C $REPO checkout $TMP_BR &&\n+\t\ttest_might_fail git -C $REPO branch -D $MATRIX_BR &&\n+\t\tgit -C $REPO branch $MATRIX_BR $TMP_BR &&\n+\t\tgit -C $REPO checkout $MATRIX_BR\n+\t\"\n+\n+\tif test $uc = true\n+\tthen\n+\t\tenable_uc\n+\telse\n+\t\tdisable_uc\n+\tfi\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstart_fsm\n+\telse\n+\t\tstop_fsm\n+\tfi\n+\n+\tdo_status \"$t status after checkout\"\n+\n+\t# Modify many files in the matrix branch.\n+\t# Stage them.\n+\t# Commit them.\n+\t# Rollback.\n+\t#\n+\ttest_expect_success \"$t modify tracked files\" \"\n+\t\tfind $REPO -name file1 -exec cp data.txt {} \\\\;\n+\t\"\n+\n+\tdo_status \"$t status after big change\"\n+\n+\t# Don't bother timing the \"add\" because _REPEAT_COUNT\n+\t# issue described above.\n+\t#\n+\ttest_expect_success \"$t add all\" \"\n+\t\tgit -C $REPO add -A\n+\t\"\n+\n+\tdo_status \"$t status after add all\"\n+\n+\ttest_expect_success \"$t add dot\" \"\n+\t\tgit -C $REPO add .\n+\t\"\n+\n+\tdo_status \"$t status after add dot\"\n+\n+\ttest_expect_success \"$t commit staged\" \"\n+\t\tgit -C $REPO commit -a -m data\n+\t\"\n+\n+\tdo_status \"$t status after commit\"\n+\n+\ttest_expect_success \"$t reset HEAD~1 hard\" \"\n+\t\tgit -C $REPO reset --hard HEAD~1 >/dev/null 2>&1\n+\t\"\n+\n+\tdo_status \"$t status after reset hard\"\n+\n+\t# Create some untracked files.\n+\t#\n+\ttest_expect_success \"$t create untracked files\" \"\n+\t\tcp -R $REPO/ballast/dir1 $REPO/ballast/xxx1\n+\t\"\n+\n+\tdo_status \"$t status after create untracked files\"\n+\n+\t# Remove the new untracked files.\n+\t#\n+\ttest_expect_success \"$t clean -df\" \"\n+\t\tgit -C $REPO clean -d -f\n+\t\"\n+\n+\tdo_status \"$t status after clean\"\n+\n+\tif test $fsm = true\n+\tthen\n+\t\tstop_fsm\n+\tfi\n+}\n+\n+# Begin testing each case in the matrix that we care about.\n+#\n+uc_values=\"false\"\n+test_have_prereq UNTRACKED_CACHE && uc_values=\"false true\"\n+\n+fsm_values=\"false true\"\n+\n+for uc_val in $uc_values\n+do\n+\tfor fsm_val in $fsm_values\n+\tdo\n+\t\tdo_matrix $uc_val $fsm_val\n+\tdone\n+done\n+\n+cleanup () {\n+\tuc=$1\n+\tfsm=$2\n+\n+\tMATRIX_BR=\"$TMP_BR-$uc-$fsm\"\n+\n+\ttest_might_fail git -C $REPO branch -D $MATRIX_BR\n+}\n+\n+\n+# We're borrowing this repo. We should leave it in a clean state.\n+#\n+test_expect_success \"Cleanup temp and matrix branches\" \"\n+\tgit -C $REPO clean -d -f &&\n+\ttest_might_fail git -C $REPO checkout $BALLAST_BR &&\n+\ttest_might_fail git -C $REPO branch -D $TMP_BR &&\n+\tfor uc_val in $uc_values\n+\tdo\n+\t\tfor fsm_val in $fsm_values\n+\t\tdo\n+\t\t\tcleanup $uc_val $fsm_val\n+\t\tdone\n+\tdone\n+\"\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"456255","messageId":"d0c8fecd1a0d622b1a6cce0aa79fdcdc2771db1d.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 23/30] fsmonitor: never set CE_FSMONITOR_VALID on submodules","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:17Z","receivedAt":"2022-05-26T21:50:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNever set CE_FSMONITOR_VALID on the cache-entry of submodule\ndirectories.\n\nDuring a client command like 'git status', we may need to recurse\ninto each submodule to compute a status summary for the submodule.\nSince the purpose of the ce_flag is to let Git avoid scanning a\ncache-entry, setting the flag causes the recursive call to be\navoided and we report incorrect (no status) for the submodule.\n\nWe created an OS watch on the root directory of our working\ndirectory and we receive events for everything in the cone\nunder it. When submodules are present inside our working\ndirectory, we receive events for both our repo (the super) and\nany subs within it. Since our index doesn't have any information\nfor items within the submodules, we can't use those events.\n\nWe could try to truncate the paths of those events back to the\nsubmodule boundary and mark the GITLINK as dirty, but that\nfeels expensive since we would have to prefix compare every FS\nevent that we receive against a list of submodule roots. And\nit still wouldn't be sufficient to correctly report status on\nthe submodule, since we don't have any space in the cache-entry\nto cache the submodule's status (the 'SCMU' bits in porcelain\nV2 speak). That is, the CE_FSMONITOR_VALID bit just says that\nwe don't need to scan/inspect it because we already know the\nanswer -- it doesn't say that the item is clean -- and we\ndon't have space in the cache-entry to store those answers.\nSo we should always do the recursive scan.\n\nTherefore, we should never set the flag on GITLINK cache-entries.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 2 +\n fsmonitor.h | 11 ++++\n t/t7527-builtin-fsmonitor.sh | 111 +++++++++++++++++++++++++++++++++++\n 3 files changed, 124 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex e1229c289cf..57d6a483bee 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)\n \t\tif (fsmonitor_enabled) {\n \t\t\t/* Mark all entries valid */\n \t\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\t\tif (S_ISGITLINK(istate->cache[i]->ce_mode))\n+\t\t\t\t\tcontinue;\n \t\t\t\tistate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;\n \t\t\t}\n \ndiff --git a/fsmonitor.h b/fsmonitor.h\nindex 3f41f653691..edf7ce5203b 100644\n--- a/fsmonitor.h\n+++ b/fsmonitor.h\n@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)\n * Set the given cache entries CE_FSMONITOR_VALID bit. This should be\n * called any time the cache entry has been updated to reflect the\n * current state of the file on disk.\n+ *\n+ * However, never mark submodules as valid. When commands like \"git\n+ * status\" run they might need to recurse into the submodule (using a\n+ * child process) to get a summary of the submodule state. We don't\n+ * have (and don't want to create) the facility to translate every\n+ * FS event that we receive and that happens to be deep inside of a\n+ * submodule back to the submodule root, so we cannot correctly keep\n+ * track of this bit on the gitlink directory. Therefore, we never\n+ * set it on submodules.\n */\n static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)\n {\n@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache\n \n \tif (fsm_mode > FSMONITOR_MODE_DISABLED &&\n \t !(ce->ce_flags & CE_FSMONITOR_VALID)) {\n+\t\tif (S_ISGITLINK(ce->ce_mode))\n+\t\t\treturn;\n \t\tistate->cache_changed = 1;\n \t\tce->ce_flags |= CE_FSMONITOR_VALID;\n \t\ttrace_printf_key(&trace_fsmonitor, \"mark_fsmonitor_clean '%s'\", ce->name);\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 3bc335b891d..cf4fb72c3f0 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -721,4 +721,115 @@ do\n \t'\n done\n \n+# Test fsmonitor interaction with submodules.\n+#\n+# If we start the daemon in the super, it will see FS events for\n+# everything in the working directory cone and this includes any\n+# files/directories contained *within* the submodules.\n+#\n+# A `git status` at top level will get events for items within the\n+# submodule and ignore them, since they aren't named in the index\n+# of the super repo. This makes the fsmonitor response a little\n+# noisy, but it doesn't alter the correctness of the state of the\n+# super-proper.\n+#\n+# When we have submodules, `git status` normally does a recursive\n+# status on each of the submodules and adds a summary row for any\n+# dirty submodules. (See the \"S...\" bits in porcelain V2 output.)\n+#\n+# It is therefore important that the top level status not be tricked\n+# by the FSMonitor response to skip those recursive calls. That is,\n+# even if FSMonitor says that the mtime of the submodule directory\n+# hasn't changed and it could be implicitly marked valid, we must\n+# not take that shortcut. We need to force the recusion into the\n+# submodule so that we get a summary of the status *within* the\n+# submodule.\n+\n+create_super () {\n+\tsuper=\"$1\" &&\n+\n+\tgit init \"$super\" &&\n+\techo x >\"$super/file_1\" &&\n+\techo y >\"$super/file_2\" &&\n+\techo z >\"$super/file_3\" &&\n+\tmkdir \"$super/dir_1\" &&\n+\techo a >\"$super/dir_1/file_11\" &&\n+\techo b >\"$super/dir_1/file_12\" &&\n+\tmkdir \"$super/dir_1/dir_2\" &&\n+\techo a >\"$super/dir_1/dir_2/file_21\" &&\n+\techo b >\"$super/dir_1/dir_2/file_22\" &&\n+\tgit -C \"$super\" add . &&\n+\tgit -C \"$super\" commit -m \"initial $super commit\"\n+}\n+\n+create_sub () {\n+\tsub=\"$1\" &&\n+\n+\tgit init \"$sub\" &&\n+\techo x >\"$sub/file_x\" &&\n+\techo y >\"$sub/file_y\" &&\n+\techo z >\"$sub/file_z\" &&\n+\tmkdir \"$sub/dir_x\" &&\n+\techo a >\"$sub/dir_x/file_a\" &&\n+\techo b >\"$sub/dir_x/file_b\" &&\n+\tmkdir \"$sub/dir_x/dir_y\" &&\n+\techo a >\"$sub/dir_x/dir_y/file_a\" &&\n+\techo b >\"$sub/dir_x/dir_y/file_b\" &&\n+\tgit -C \"$sub\" add . &&\n+\tgit -C \"$sub\" commit -m \"initial $sub commit\"\n+}\n+\n+my_match_and_clean () {\n+\tgit -C super --no-optional-locks status --porcelain=v2 >actual.with &&\n+\tgit -C super --no-optional-locks -c core.fsmonitor=false \\\n+\t\tstatus --porcelain=v2 >actual.without &&\n+\ttest_cmp actual.with actual.without &&\n+\n+\tgit -C super/dir_1/dir_2/sub reset --hard &&\n+\tgit -C super/dir_1/dir_2/sub clean -d -f\n+}\n+\n+test_expect_success 'submodule always visited' '\n+\ttest_when_finished \"git -C super fsmonitor--daemon stop; \\\n+\t\t\t rm -rf super; \\\n+\t\t\t rm -rf sub\" &&\n+\n+\tcreate_super super &&\n+\tcreate_sub sub &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\tstart_daemon -C super &&\n+\tgit -C super config core.fsmonitor true &&\n+\tgit -C super update-index --fsmonitor &&\n+\tgit -C super status &&\n+\n+\t# Now run pairs of commands w/ and w/o FSMonitor while we make\n+\t# some dirt in the submodule and confirm matching output.\n+\n+\t# Completely clean status.\n+\tmy_match_and_clean &&\n+\n+\t# .M S..U\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M S.M.\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tmy_match_and_clean &&\n+\n+\t# .M SC..\n+\techo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&\n+\tgit -C super/dir_1/dir_2/sub add . &&\n+\tgit -C super/dir_1/dir_2/sub commit -m \"SC..\" &&\n+\tmy_match_and_clean\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"456256","messageId":"41f8cbc2ae45cb86e299eb230ad3cb0319256c37.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 24/30] t7527: test FSMonitor on case insensitive+preserving file system","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:18Z","receivedAt":"2022-05-26T21:50:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTest that FS events from the OS are received using the preserved,\non-disk spelling of files/directories rather than spelling used\nto make the change.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex cf4fb72c3f0..fbb7d6aef6e 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,4 +832,40 @@ test_expect_success 'submodule always visited' '\n \tmy_match_and_clean\n '\n \n+# On a case-insensitive file system, confirm that the daemon\n+# notices when the .git directory is moved/renamed/deleted\n+# regardless of how it is spelled in the the FS event.\n+# That is, does the FS event receive the spelling of the\n+# operation or does it receive the spelling preserved with\n+# the file/directory.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n+#\ttest_when_finished \"stop_daemon_delete_repo test_insensitive\" &&\n+\n+\tgit init test_insensitive &&\n+\n+\tstart_daemon -C test_insensitive --tf \"$PWD/insensitive.trace\" &&\n+\n+\tmkdir -p test_insensitive/abc/def &&\n+\techo xyz >test_insensitive/ABC/DEF/xyz &&\n+\n+\ttest_path_is_dir test_insensitive/.git &&\n+\ttest_path_is_dir test_insensitive/.GIT &&\n+\n+\t# Rename .git using an alternate spelling to verify that that\n+\t# daemon detects it and automatically shuts down.\n+\tmv test_insensitive/.GIT test_insensitive/.FOO &&\n+\tsleep 1 &&\n+\tmv test_insensitive/.FOO test_insensitive/.git &&\n+\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\t# Verify that events were reported using on-disk spellings of the\n+\t# directories and files that we touched. We may or may not get a\n+\t# trailing slash on modified directories.\n+\t#\n+\tegrep \"^event: abc/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/?$\" ./insensitive.trace &&\n+\tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"456257","messageId":"d8949ab5df6830ca4f5556116157bbc124af9cda.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 19/30] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:13Z","receivedAt":"2022-05-26T21:50:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the listener thread to shutdown the daemon if the spelling of the\nworktree root directory changes.\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++\n 1 file changed, 25 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 07113205a61..83d38e8ac6c 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)\n \tstrbuf_release(&msg);\n }\n \n+static int ef_is_root_changed(const FSEventStreamEventFlags ef)\n+{\n+\treturn (ef & kFSEventStreamEventFlagRootChanged);\n+}\n+\n static int ef_is_root_delete(const FSEventStreamEventFlags ef)\n {\n \treturn (ef & kFSEventStreamEventFlagItemIsDir &&\n@@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\tcontinue;\n \t\t}\n \n+\t\tif (ef_is_root_changed(event_flags[k])) {\n+\t\t\t/*\n+\t\t\t * The spelling of the pathname of the root directory\n+\t\t\t * has changed. This includes the name of the root\n+\t\t\t * directory itself or of any parent directory in the\n+\t\t\t * path.\n+\t\t\t *\n+\t\t\t * (There may be other conditions that throw this,\n+\t\t\t * but I couldn't find any information on it.)\n+\t\t\t *\n+\t\t\t * Force a shutdown now and avoid things getting\n+\t\t\t * out of sync. The Unix domain socket is inside\n+\t\t\t * the .git directory and a spelling change will make\n+\t\t\t * it hard for clients to rendezvous with us.\n+\t\t\t */\n+\t\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t\t \"event: root changed\");\n+\t\t\tgoto force_shutdown;\n+\t\t}\n+\n \t\tif (ef_ignore_xattr(event_flags[k])) {\n \t\t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t\t \"ignore-xattr: '%s', flags=0x%x\",\n-- \ngitgitgadget\n\n"},{"id":"456258","messageId":"b3f5a945d3e305998966a0754a61afc48b15ca6e.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 18/30] fsm-health-win32: force shutdown daemon if worktree root moves","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:12Z","receivedAt":"2022-05-26T21:50:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nForce shutdown fsmonitor daemon if the worktree root directory\nis moved, renamed, or deleted.\n\nUse Windows low-level GetFileInformationByHandle() to get and\ncompare the Windows system unique ID for the directory with a\ncached version when we started up. This lets us detect the\ncase where someone renames the directory that we are watching\nand then creates a new directory with the original pathname.\n\nThis is important because we are listening to a named pipe for\nrequests and they are stored in the Named Pipe File System (NPFS)\nwhich a kernel-resident pseudo filesystem not associated with\nthe actual NTFS directory.\n\nFor example, if the daemon was watching \"~/foo/\", it would have\na directory-watch handle on that directory and a named-pipe\nhandle for \"//./pipe/...foo\". Moving the directory to \"~/bar/\"\ndoes not invalidate the directory handle. (So the daemon would\nactually be watching \"~/bar\" but listening on \"//./pipe/...foo\".\nIf the user then does \"git init ~/foo\" and causes another daemon\nto start, the first daemon will still have ownership of the pipe\nand the second daemon instance will fail to start. \"git status\"\nclients in \"~/foo\" will ask \"//./pipe/...foo\" about changes and\nthe first daemon instance will tell them about \"~/bar\".\n\nThis commit causes the first daemon to shutdown if the system unique\nID for \"~/foo\" changes (changes from what it was when the daemon\nstarted). Shutdown occurs after a periodic poll. After the\nfirst daemon exits and releases the lock on the named pipe,\nsubsequent Git commands may cause another daemon to be started\non \"~/foo\". Similarly, a subsequent Git command may cause another\ndaemon to be started on \"~/bar\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++\n 1 file changed, 143 insertions(+)\n\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 24fc612bf02..2ea08c1d4e8 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -29,8 +29,150 @@ struct fsm_health_data\n \tHANDLE hHandles[1]; /* the array does not own these handles */\n #define HEALTH_SHUTDOWN 0\n \tint nr_handles; /* number of active event handles */\n+\n+\tstruct wt_moved\n+\t{\n+\t\twchar_t wpath[MAX_PATH + 1];\n+\t\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\t} wt_moved;\n };\n \n+/*\n+ * Lookup the system unique ID for the path. This is as close as\n+ * we get to an inode number, but this also contains volume info,\n+ * so it is a little stronger.\n+ */\n+static int lookup_bhfi(wchar_t *wpath,\n+\t\t BY_HANDLE_FILE_INFORMATION *bhfi)\n+{\n+\tDWORD desired_access = FILE_LIST_DIRECTORY;\n+\tDWORD share_mode =\n+\t\tFILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;\n+\tHANDLE hDir;\n+\n+\thDir = CreateFileW(wpath, desired_access, share_mode, NULL,\n+\t\t\t OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);\n+\tif (hDir == INVALID_HANDLE_VALUE) {\n+\t\terror(_(\"[GLE %ld] health thread could not open '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\treturn -1;\n+\t}\n+\n+\tif (!GetFileInformationByHandle(hDir, bhfi)) {\n+\t\terror(_(\"[GLE %ld] health thread getting BHFI for '%ls'\"),\n+\t\t GetLastError(), wpath);\n+\t\tCloseHandle(hDir);\n+\t\treturn -1;\n+\t}\n+\n+\tCloseHandle(hDir);\n+\treturn 0;\n+}\n+\n+/*\n+ * Compare the relevant fields from two system unique IDs.\n+ * We use this to see if two different handles to the same\n+ * path actually refer to the same *instance* of the file\n+ * or directory.\n+ */\n+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,\n+\t\t const BY_HANDLE_FILE_INFORMATION *bhfi_2)\n+{\n+\treturn (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&\n+\t\tbhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&\n+\t\tbhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);\n+}\n+\n+/*\n+ * Shutdown if the original worktree root directory been deleted,\n+ * moved, or renamed?\n+ *\n+ * Since the main thread did a \"chdir(getenv($HOME))\" and our CWD\n+ * is not in the worktree root directory and because the listener\n+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible\n+ * for the root directory to be moved or deleted while we are still\n+ * watching it. We want to detect that here and force a shutdown.\n+ *\n+ * Granted, a delete MAY cause some operations to fail, such as\n+ * GetOverlappedResult(), but it is not guaranteed. And because\n+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the\n+ * directory, not changes *ON* the directory, our watch will not\n+ * receive a delete event for it.\n+ *\n+ * A move/rename of the worktree root will also not generate an event.\n+ * And since the listener thread already has an open handle, it may\n+ * continue to receive events for events within the directory.\n+ * However, the pathname of the named-pipe was constructed using the\n+ * original location of the worktree root. (Remember named-pipes are\n+ * stored in the NPFS and not in the actual file system.) Clients\n+ * trying to talk to the worktree after the move/rename will not\n+ * reach our daemon process, since we're still listening on the\n+ * pipe with original path.\n+ *\n+ * Furthermore, if the user does something like:\n+ *\n+ * $ mv repo repo.old\n+ * $ git init repo\n+ *\n+ * A new daemon cannot be started in the new instance of \"repo\"\n+ * because the named-pipe is still being used by the daemon on\n+ * the original instance.\n+ *\n+ * So, detect move/rename/delete and shutdown. This should also\n+ * handle unsafe drive removal.\n+ *\n+ * We use the file system unique ID to distinguish the original\n+ * directory instance from a new instance and force a shutdown\n+ * if the unique ID changes.\n+ *\n+ * Since a worktree move/rename/delete/unmount doesn't happen\n+ * that often (and we can't get an immediate event anyway), we\n+ * use a timeout and periodically poll it.\n+ */\n+static int has_worktree_moved(struct fsmonitor_daemon_state *state,\n+\t\t\t enum interval_fn_ctx ctx)\n+{\n+\tstruct fsm_health_data *data = state->health_data;\n+\tBY_HANDLE_FILE_INFORMATION bhfi;\n+\tint r;\n+\n+\tswitch (ctx) {\n+\tcase CTX_TERM:\n+\t\treturn 0;\n+\n+\tcase CTX_INIT:\n+\t\tif (xutftowcs_path(data->wt_moved.wpath,\n+\t\t\t\t state->path_worktree_watch.buf) < 0) {\n+\t\t\terror(_(\"could not convert to wide characters: '%s'\"),\n+\t\t\t state->path_worktree_watch.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\t/*\n+\t\t * On the first call we lookup the unique sequence ID for\n+\t\t * the worktree root directory.\n+\t\t */\n+\t\treturn lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);\n+\n+\tcase CTX_TIMER:\n+\t\tr = lookup_bhfi(data->wt_moved.wpath, &bhfi);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t\tif (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {\n+\t\t\terror(_(\"BHFI changed '%ls'\"), data->wt_moved.wpath);\n+\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\n+\tdefault:\n+\t\tdie(_(\"unhandled case in 'has_worktree_moved': %d\"),\n+\t\t (int)ctx);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+\n int fsm_health__ctor(struct fsmonitor_daemon_state *state)\n {\n \tstruct fsm_health_data *data;\n@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)\n * A table of the polling functions.\n */\n static interval_fn *table[] = {\n+\thas_worktree_moved,\n \tNULL, /* must be last */\n };\n \n-- \ngitgitgadget\n\n"},{"id":"456259","messageId":"c8c4c22360c306a0e94c7c6684f439a71bef0709.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 25/30] fsmonitor: on macOS also emit NFC spelling for NFD pathname","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:19Z","receivedAt":"2022-05-26T21:50:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nEmit NFC or NFC and NFD spellings of pathnames on macOS.\n\nMacOS is Unicode composition insensitive, so NFC and NFD spellings are\ntreated as aliases and collide. While the spelling of pathnames in\nfilesystem events depends upon the underlying filesystem, such as\nAPFS, HFS+ or FAT32, the OS enforces such collisions regardless of\nfilesystem.\n\nTeach the daemon to always report the NFC spelling and to report\nthe NFD spelling when stored in that format on the disk.\n\nThis is slightly more general than \"core.precomposeUnicode\".\n\nSigned-off-by: Jeff Hostetler \n---\n compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--\n 1 file changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 83d38e8ac6c..823cf63999e 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)\n \treturn ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);\n }\n \n+/*\n+ * On MacOS we have to adjust for Unicode composition insensitivity\n+ * (where NFC and NFD spellings are not respected). The different\n+ * spellings are essentially aliases regardless of how the path is\n+ * actually stored on the disk.\n+ *\n+ * This is related to \"core.precomposeUnicode\" (which wants to try\n+ * to hide NFD completely and treat everything as NFC). Here, we\n+ * don't know what the value the client has (or will have) for this\n+ * config setting when they make a query, so assume the worst and\n+ * emit both when the OS gives us an NFD path.\n+ */\n+static void my_add_path(struct fsmonitor_batch *batch, const char *path)\n+{\n+\tchar *composed;\n+\n+\t/* add the NFC or NFD path as received from the OS */\n+\tfsmonitor_batch__add_path(batch, path);\n+\n+\t/* if NFD, also add the corresponding NFC spelling */\n+\tcomposed = (char *)precompose_string_if_needed(path);\n+\tif (!composed || composed == path)\n+\t\treturn;\n+\n+\tfsmonitor_batch__add_path(batch, composed);\n+\tfree(composed);\n+}\n+\n+\n static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \t\t\t void *ctx,\n \t\t\t size_t num_of_events,\n@@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, rel);\n+\t\t\t\tmy_add_path(batch, rel);\n \t\t\t}\n \n \t\t\tif (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {\n@@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,\n \n \t\t\t\tif (!batch)\n \t\t\t\t\tbatch = fsmonitor_batch__new();\n-\t\t\t\tfsmonitor_batch__add_path(batch, tmp.buf);\n+\t\t\t\tmy_add_path(batch, tmp.buf);\n \t\t\t}\n \n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"456260","messageId":"1612dcba5035672253165a99ffba22fe78a601da.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 26/30] t/helper/hexdump: add helper to print hexdump of stdin","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:20Z","receivedAt":"2022-05-26T21:50:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCo-authored-by: Johannes Schindelin \nSigned-off-by: Johannes Schindelin \nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n t/helper/test-hexdump.c | 30 ++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 33 insertions(+)\n create mode 100644 t/helper/test-hexdump.c\n\ndiff --git a/Makefile b/Makefile\nindex 5f1623baadd..5afa194aac6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -729,6 +729,7 @@ TEST_BUILTINS_OBJS += test-getcwd.o\n TEST_BUILTINS_OBJS += test-hash-speed.o\n TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n+TEST_BUILTINS_OBJS += test-hexdump.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\n TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\ndiff --git a/t/helper/test-hexdump.c b/t/helper/test-hexdump.c\nnew file mode 100644\nindex 00000000000..811e89c1bcb\n--- /dev/null\n+++ b/t/helper/test-hexdump.c\n@@ -0,0 +1,30 @@\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * Read stdin and print a hexdump to stdout.\n+ */\n+int cmd__hexdump(int argc, const char **argv)\n+{\n+\tchar buf[1024];\n+\tssize_t i, len;\n+\tint have_data = 0;\n+\n+\tfor (;;) {\n+\t\tlen = xread(0, buf, sizeof(buf));\n+\t\tif (len < 0)\n+\t\t\tdie_errno(\"failure reading stdin\");\n+\t\tif (!len)\n+\t\t\tbreak;\n+\n+\t\thave_data = 1;\n+\n+\t\tfor (i = 0; i < len; i++)\n+\t\t\tprintf(\"%02x \", (unsigned char)buf[i]);\n+\t}\n+\n+\tif (have_data)\n+\t\tputchar('\\n');\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 0424f7adf5d..88c4b28cdfa 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -38,6 +38,7 @@ static struct test_cmd cmds[] = {\n \t{ \"getcwd\", cmd__getcwd },\n \t{ \"hashmap\", cmd__hashmap },\n \t{ \"hash-speed\", cmd__hash_speed },\n+\t{ \"hexdump\", cmd__hexdump },\n \t{ \"index-version\", cmd__index_version },\n \t{ \"json-writer\", cmd__json_writer },\n \t{ \"lazy-init-name-hash\", cmd__lazy_init_name_hash },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex c876e8246fb..511f6251bf5 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -29,6 +29,7 @@ int cmd__genzeros(int argc, const char **argv);\n int cmd__getcwd(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n int cmd__hash_speed(int argc, const char **argv);\n+int cmd__hexdump(int argc, const char **argv);\n int cmd__index_version(int argc, const char **argv);\n int cmd__json_writer(int argc, const char **argv);\n int cmd__lazy_init_name_hash(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"456261","messageId":"2f0dea304f060ea67b89fc2876fed62e40a38ac1.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 30/30] t7527: improve implicit shutdown testing in fsmonitor--daemon","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:24Z","receivedAt":"2022-05-26T21:50:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor the tests that exercise implicit shutdown cases\nto make them more robust and less racy.\n\nThe fsmonitor--daemon will implicitly shutdown in a variety\nof situations, such as when the \".git\" directory is deleted\nor renamed.\n\nThe existing tests would delete or rename the directory, sleep\nfor one second, and then check the status of the daemon. This\nis racy, since the client/status command has no way to sync\nwith the daemon. This was noticed occasionally on very slow\nCI build machines where it would cause a random test to fail.\n\nReplace the simple sleep with a sleep-and-retry loop.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 54 ++++++++++++++++++++++++++----------\n 1 file changed, 40 insertions(+), 14 deletions(-)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 19edc96fd4d..56c0dfffea6 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -124,6 +124,36 @@ test_expect_success 'implicit daemon start' '\n \ttest_must_fail git -C test_implicit fsmonitor--daemon status\n '\n \n+# Verify that the daemon has shutdown. Spin a few seconds to\n+# make the test a little more robust during CI testing.\n+#\n+# We're looking for an implicit shutdown, such as when we delete or\n+# rename the \".git\" directory. Our delete/rename will cause a file\n+# system event that the daemon will see and the daemon will\n+# auto-shutdown as soon as it sees it. But this is racy with our `git\n+# fsmonitor--daemon status` commands (and we cannot use a cookie file\n+# here to help us). So spin a little and give the daemon a chance to\n+# see the event. (This is primarily for underpowered CI build/test\n+# machines (where it might take a moment to wake and reschedule the\n+# daemon process) to avoid false alarms during test runs.)\n+#\n+IMPLICIT_TIMEOUT=5\n+\n+verify_implicit_shutdown () {\n+\tr=$1 &&\n+\n+\tk=0 &&\n+\twhile test \"$k\" -lt $IMPLICIT_TIMEOUT\n+\tdo\n+\t\tgit -C $r fsmonitor--daemon status || return 0\n+\n+\t\tsleep 1\n+\t\tk=$(( $k + 1 ))\n+\tdone &&\n+\n+\treturn 1\n+}\n+\n test_expect_success 'implicit daemon stop (delete .git)' '\n \ttest_when_finished \"stop_daemon_delete_repo test_implicit_1\" &&\n \n@@ -142,10 +172,9 @@ test_expect_success 'implicit daemon stop (delete .git)' '\n \t# This would make the test result dependent upon whether we\n \t# were using fsmonitor on our development worktree.\n \t#\n-\tsleep 1 &&\n \tmkdir test_implicit_1/.git &&\n \n-\ttest_must_fail git -C test_implicit_1 fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_1\n '\n \n test_expect_success 'implicit daemon stop (rename .git)' '\n@@ -160,10 +189,9 @@ test_expect_success 'implicit daemon stop (rename .git)' '\n \n \t# See [1] above.\n \t#\n-\tsleep 1 &&\n \tmkdir test_implicit_2/.git &&\n \n-\ttest_must_fail git -C test_implicit_2 fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_2\n '\n \n # File systems on Windows may or may not have shortnames.\n@@ -194,13 +222,11 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '\n \t#\n \tmv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&\n \n-\tsleep 1 &&\n-\t# put it back so that our status will not crawl out to our\n-\t# parent directory.\n+\t# See [1] above.\n \t# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.\n \tmv test_implicit_1s/.gitxyz test_implicit_1s/.git &&\n \n-\ttest_must_fail git -C test_implicit_1s fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_1s\n '\n \n # Here we first create a file with LONGNAME of \"GIT~1\" before\n@@ -223,12 +249,10 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '\n \t#\n \tmv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&\n \n-\tsleep 1 &&\n-\t# put it back so that our status will not crawl out to our\n-\t# parent directory.\n+\t# See [1] above.\n \tmv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&\n \n-\ttest_must_fail git -C test_implicit_1s2 fsmonitor--daemon status\n+\tverify_implicit_shutdown test_implicit_1s2\n '\n \n test_expect_success 'cannot start multiple daemons' '\n@@ -905,9 +929,11 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \t# Rename .git using an alternate spelling to verify that that\n \t# daemon detects it and automatically shuts down.\n \tmv test_insensitive/.GIT test_insensitive/.FOO &&\n-\tsleep 1 &&\n+\n+\t# See [1] above.\n \tmv test_insensitive/.FOO test_insensitive/.git &&\n-\ttest_must_fail git -C test_insensitive fsmonitor--daemon status &&\n+\n+\tverify_implicit_shutdown test_insensitive &&\n \n \t# Verify that events were reported using on-disk spellings of the\n \t# directories and files that we touched. We may or may not get a\n-- \ngitgitgadget\n"},{"id":"456262","messageId":"b27dc48a6507283418e52bc8dd47640bfaf34ac9.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 28/30] t7527: test Unicode NFC/NFD handling on MacOS","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:22Z","receivedAt":"2022-05-26T21:50:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConfirm that the daemon reports events using the on-disk\nspelling for Unicode NFC/NFD characters. On APFS we still\nhave Unicode aliasing, so we cannot create two files that\nonly differ by NFC/NFD, but the on-disk format preserves\nthe spelling used to create the file. On HFS+ we also\nhave aliasing, but the path is always stored on disk in\nNFD.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 55 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex fbb7d6aef6e..9edae3ed830 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -868,4 +868,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '\n \tegrep \"^event: abc/def/xyz$\" ./insensitive.trace\n '\n \n+# The variable \"unicode_debug\" is defined in the following library\n+# script to dump information about how the (OS, FS) handles Unicode\n+# composition. Uncomment the following line if you want to enable it.\n+#\n+# unicode_debug=true\n+\n+. \"$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh\"\n+\n+# See if the OS or filesystem does NFC/NFD aliasing/munging.\n+#\n+# The daemon should err on the side of caution and send BOTH the\n+# NFC and NFD forms. It does not know the original spelling of\n+# the pathname (how the user thinks it should be spelled), so\n+# emit both and let the client decide (when necessary). This is\n+# similar to \"core.precomposeUnicode\".\n+#\n+test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' '\n+\ttest_when_finished \"stop_daemon_delete_repo test_unicode\" &&\n+\n+\tgit init test_unicode &&\n+\n+\tstart_daemon -C test_unicode --tf \"$PWD/unicode.trace\" &&\n+\n+\t# Create a directory using an NFC spelling.\n+\t#\n+\tmkdir test_unicode/nfc &&\n+\tmkdir test_unicode/nfc/c_${utf8_nfc} &&\n+\n+\t# Create a directory using an NFD spelling.\n+\t#\n+\tmkdir test_unicode/nfd &&\n+\tmkdir test_unicode/nfd/d_${utf8_nfd} &&\n+\n+\tgit -C test_unicode fsmonitor--daemon stop &&\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\t# We should have seen NFC event from OS.\n+\t\t# We should not have synthesized an NFD event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace &&\n+\t\tegrep -v \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace\n+\telse\n+\t\t# We should have seen NFD event from OS.\n+\t\t# We should have synthesized an NFC event.\n+\t\tegrep \"^event: nfc/c_${utf8_nfd}/?$\" ./unicode.trace &&\n+\t\tegrep \"^event: nfc/c_${utf8_nfc}/?$\" ./unicode.trace\n+\tfi &&\n+\n+\t# We assume UNICODE_NFD_PRESERVED.\n+\t# We should have seen explicit NFD from OS.\n+\t# We should have synthesized an NFC event.\n+\tegrep \"^event: nfd/d_${utf8_nfd}/?$\" ./unicode.trace &&\n+\tegrep \"^event: nfd/d_${utf8_nfc}/?$\" ./unicode.trace\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"456263","messageId":"2905b3bb59e3a1a8842df86c21841f5aa17bbddb.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 29/30] fsmonitor--daemon: allow --super-prefix argument","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:23Z","receivedAt":"2022-05-26T21:50:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a test in t7527 to verify that we get a stray warning from\n`git fsmonitor--daemon start` when indirectly called from\n`git submodule absorbgitdirs`.\n\nUpdate `git fsmonitor--daemon` to take (and ignore) the `--super-prefix`\nargument to suppress the warning.\n\nWhen we have:\n\n1. a submodule with a `sub/.git/` directory (rather than a `sub/.git`\nfile).\n\n2. `core.fsmonitor` is turned on in the submodule, but the daemon is\nnot yet started in the submodule.\n\n3. and someone does a `git submodule absorbgitdirs` in the super.\n\nGit will recursively invoke `git submodule--helper absorb-git-dirs`\nin the submodule. This will read the index and may attempt to start\nthe fsmonitor--daemon with the `--super-prefix` argument.\n\n`git fsmonitor--daemon start` does not accept the `--super-prefix`\nargument and causes a warning to be issued.\n\nThis does not cause a problem because the `refresh_index()` code\nassumes a trivial response if the daemon does not start.\n\nThe net-net is a harmelss, but stray warning. Lets eliminate the\nwarning.\n\nSigned-off-by: Jeff Hostetler \n---\n git.c | 2 +-\n t/t7527-builtin-fsmonitor.sh | 50 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 51 insertions(+), 1 deletion(-)\n\ndiff --git a/git.c b/git.c\nindex 3d8e48cf555..e6fdac1f8e3 100644\n--- a/git.c\n+++ b/git.c\n@@ -537,7 +537,7 @@ static struct cmd_struct commands[] = {\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n-\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, RUN_SETUP },\n+\t{ \"fsmonitor--daemon\", cmd_fsmonitor__daemon, SUPPORT_SUPER_PREFIX | RUN_SETUP },\n \t{ \"gc\", cmd_gc, RUN_SETUP },\n \t{ \"get-tar-commit-id\", cmd_get_tar_commit_id, NO_PARSEOPT },\n \t{ \"grep\", cmd_grep, RUN_SETUP_GENTLY },\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 9edae3ed830..19edc96fd4d 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -832,6 +832,56 @@ test_expect_success 'submodule always visited' '\n \tmy_match_and_clean\n '\n \n+# If a submodule has a `sub/.git/` directory (rather than a file\n+# pointing to the super's `.git/modules/sub`) and `core.fsmonitor`\n+# turned on in the submodule and the daemon is not yet started in\n+# the submodule, and someone does a `git submodule absorbgitdirs`\n+# in the super, Git will recursively invoke `git submodule--helper`\n+# to do the work and this may try to read the index. This will\n+# try to start the daemon in the submodule *and* pass (either\n+# directly or via inheritance) the `--super-prefix` arg to the\n+# `git fsmonitor--daemon start` command inside the submodule.\n+# This causes a warning because fsmonitor--daemon does take that\n+# global arg (see the table in git.c)\n+#\n+# This causes a warning when trying to start the daemon that is\n+# somewhat confusing. It does not seem to hurt anything because\n+# the fsmonitor code maps the query failure into a trivial response\n+# and does the work anyway.\n+#\n+# It would be nice to silence the warning, however.\n+\n+have_t2_error_event () {\n+\tlog=$1\n+\tmsg=\"fsmonitor--daemon doesnQt support --super-prefix\" &&\n+\n+\ttr '\\047' Q <$1 | grep -e \"$msg\"\n+}\n+\n+test_expect_success \"stray submodule super-prefix warning\" '\n+\ttest_when_finished \"rm -rf super; \\\n+\t\t\t rm -rf sub; \\\n+\t\t\t rm super-sub.trace\" &&\n+\n+\tcreate_super super &&\n+\tcreate_sub sub &&\n+\n+\t# Copy rather than submodule add so that we get a .git dir.\n+\tcp -R ./sub ./super/dir_1/dir_2/sub &&\n+\n+\tgit -C super/dir_1/dir_2/sub config core.fsmonitor true &&\n+\n+\tgit -C super submodule add ../sub ./dir_1/dir_2/sub &&\n+\tgit -C super commit -m \"add sub\" &&\n+\n+\ttest_path_is_dir super/dir_1/dir_2/sub/.git &&\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/super-sub.trace\" \\\n+\t\tgit -C super submodule absorbgitdirs &&\n+\n+\t! have_t2_error_event super-sub.trace\n+'\n+\n # On a case-insensitive file system, confirm that the daemon\n # notices when the .git directory is moved/renamed/deleted\n # regardless of how it is spelled in the the FS event.\n-- \ngitgitgadget\n\n"},{"id":"456264","messageId":"e5a7f05d9d414a8b680114f3a71739a20088bae4.1653601644.git.gitgitgadget@gmail.com","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"[PATCH v9 27/30] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-05-26T21:47:21Z","receivedAt":"2022-05-26T21:50:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a set of prereqs to help understand how file names\nare handled by the filesystem when they contain NFC and NFD\nUnicode characters.\n\nSigned-off-by: Jeff Hostetler \n---\n t/lib-unicode-nfc-nfd.sh | 162 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 162 insertions(+)\n create mode 100755 t/lib-unicode-nfc-nfd.sh\n\ndiff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh\nnew file mode 100755\nindex 00000000000..22232247efc\n--- /dev/null\n+++ b/t/lib-unicode-nfc-nfd.sh\n@@ -0,0 +1,162 @@\n+# Help detect how Unicode NFC and NFD are handled on the filesystem.\n+\n+# A simple character that has a NFD form.\n+#\n+# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE\n+# UTF8(NFC): \\xc3 \\xa9\n+#\n+# NFD: U+0065 LATIN SMALL LETTER E\n+# U+0301 COMBINING ACUTE ACCENT\n+# UTF8(NFD): \\x65 + \\xcc \\x81\n+#\n+utf8_nfc=$(printf \"\\xc3\\xa9\")\n+utf8_nfd=$(printf \"\\x65\\xcc\\x81\")\n+\n+# Is the OS or the filesystem \"Unicode composition sensitive\"?\n+#\n+# That is, does the OS or the filesystem allow files to exist with\n+# both the NFC and NFD spellings? Or, does the OS/FS lie to us and\n+# tell us that the NFC and NFD forms are equivalent.\n+#\n+# This is or may be independent of what type of filesystem we have,\n+# since it might be handled by the OS at a layer above the FS.\n+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a\n+# collision, for example.\n+#\n+# This does not tell us how the Unicode pathname will be spelled\n+# on disk, but rather only that the two spelling \"collide\". We\n+# will examine the actual on disk spelling in a later prereq.\n+#\n+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${utf8_nfc} &&\n+\tmkdir trial_${utf8_nfd}\n+'\n+\n+# Is the spelling of an NFC pathname preserved on disk?\n+#\n+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD\n+# and on APFS, NFC paths are preserved. As we have established\n+# above, this is independent of \"composition sensitivity\".\n+#\n+test_lazy_prereq UNICODE_NFC_PRESERVED '\n+\tmkdir c_${utf8_nfc} &&\n+\tls | test-tool hexdump >dump &&\n+\tgrep \"63 5f c3 a9\" dump\n+'\n+\n+# Is the spelling of an NFD pathname preserved on disk?\n+#\n+test_lazy_prereq UNICODE_NFD_PRESERVED '\n+\tmkdir d_${utf8_nfd} &&\n+\tls | test-tool hexdump >dump &&\n+\tgrep \"64 5f 65 cc 81\" dump\n+'\n+\n+# The following _DOUBLE_ forms are more for my curiosity,\n+# but there may be quirks lurking when there are multiple\n+# combining characters in non-canonical order.\n+\n+# Unicode also allows multiple combining characters\n+# that can be decomposed in pieces.\n+#\n+# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI\n+# UTF8(NFC): \\xe1 \\xbd \\xa7\n+#\n+# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA\n+# U+0342 COMBINING GREEK PERISPOMENI\n+# UTF8(NFD1): \\xe1 \\xbd \\xa1 + \\xcd \\x82\n+#\n+# But U+1f61 decomposes into\n+# NFD2: U+03c9 GREEK SMALL LETTER OMEGA\n+# U+0314 COMBINING REVERSED COMMA ABOVE\n+# UTF8(NFD2): \\xcf \\x89 + \\xcc \\x94\n+#\n+# Yielding: \\xcf \\x89 + \\xcc \\x94 + \\xcd \\x82\n+#\n+# Note that I've used the canonical ordering of the\n+# combinining characters. It is also possible to\n+# swap them. My testing shows that that non-standard\n+# ordering also causes a collision in mkdir. However,\n+# the resulting names don't draw correctly on the\n+# terminal (implying that the on-disk format also has\n+# them out of order).\n+#\n+greek_nfc=$(printf \"\\xe1\\xbd\\xa7\")\n+greek_nfd1=$(printf \"\\xe1\\xbd\\xa1\\xcd\\x82\")\n+greek_nfd2=$(printf \"\\xcf\\x89\\xcc\\x94\\xcd\\x82\")\n+\n+# See if a double decomposition also collides.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '\n+\tmkdir trial_${greek_nfc} &&\n+\tmkdir trial_${greek_nfd2}\n+'\n+\n+# See if the NFC spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '\n+\tmkdir c_${greek_nfc} &&\n+\tls | test-tool hexdump >dump &&\n+\tgrep \"63 5f e1 bd a7\" dump\n+'\n+\n+# See if the NFD spelling appears on the disk.\n+#\n+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '\n+\tmkdir d_${greek_nfd2} &&\n+\tls | test-tool hexdump >dump &&\n+\tgrep \"64 5f cf 89 cc 94 cd 82\" dump\n+'\n+\n+# The following is for debugging. I found it useful when\n+# trying to understand the various (OS, FS) quirks WRT\n+# Unicode and how composition/decomposition is handled.\n+# For example, when trying to understand how (macOS, APFS)\n+# and (macOS, HFS) and (macOS, FAT32) compare.\n+#\n+# It is rather noisy, so it is disabled by default.\n+#\n+if test \"$unicode_debug\" = \"true\"\n+then\n+\tif test_have_prereq UNICODE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFC_PRESERVED\n+\tthen\n+\t\techo NFC maintains original spelling.\n+\telse\n+\t\techo NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_NFD_PRESERVED\n+\tthen\n+\t\techo NFD maintains original spelling.\n+\telse\n+\t\techo NFD is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE\n+\tthen\n+\t\techo DOUBLE NFC and NFD are distinct on this OS/filesystem.\n+\telse\n+\t\techo DOUBLE NFC and NFD are aliases on this OS/filesystem.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED\n+\tthen\n+\t\techo Double NFC maintains original spelling.\n+\telse\n+\t\techo Double NFC is modified.\n+\tfi\n+\n+\tif test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED\n+\tthen\n+\t\techo Double NFD maintains original spelling.\n+\telse\n+\t\techo Double NFD is modified.\n+\tfi\n+fi\n-- \ngitgitgadget\n\n"},{"id":"456487","messageId":"nycvar.QRO.7.76.6.2206021155070.349@tvgsbejvaqbjf.bet","threadId":"57414","inReplyTo":"pull.1143.v9.git.1653601644.gitgitgadget@gmail.com","subject":"Re: [PATCH v9 00/30] Builtin FSMonitor Part 3","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-02T09:56:36Z","receivedAt":"2022-06-02T09:57:03Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Thu, 26 May 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> Here is version 9 of part 3 of FSMonitor.\n>\n> This version addresses the test failure on t7527.[56] when deleting/renaming\n> the .git directory using the \"GIT~1\" spelling. Between v6 and v7, I changed\n> the way check_for_shortnames() constructed the wchar_t path to .git from a\n> pair of wcscpy() calls to a swprintf() call. However, I used a \"%s\" by\n> mistake rather than a \"%ls\".\n>\n> This caused a non-portable behavior. The tests passed with MSVC and with GCC\n> 10.1.0 on my laptop, but failed under GCC on one of the CI build machines.\n>\n> This was partially hidden by CI machines that have GCC 12 and that fail to\n> compile Git without the fixes for GCC 12.x.\n\nI was already happy with the range-diff you showed in v7, and I am even\nhappier with the `hexdump` changes you introduced in v8 and with the\n`swprintf()` format fix in v9.\n\nFrom my side, this patch series is ready for `next`.\n\nThank you,\nDscho\n"}]}